federico-jose-soerenson-sanchez.jpeg

Federico José Sörenson Sánchez

Two years ago we started the decentralization of an e-commerce project. The reason is that the project is deployed in production with a cluster of Hybris nodes in Europe, and our customer has world-wide interests. Latencies between continents are bigger than the quality we want to provide. Also we have the typical issues of a huge monolith, like long release cycles, long and exhaustive testing process. . . With an ever growing feature set, the time it took to bring new features to market was also steadily growing.

## Research
### Application

As I stated before, here at communicode we have large experience in working with Java and Spring. This is why finding the basic technology for us to use for developing the micro-services was straight forward.

From the very beginning we knew that our applications would be based on Spring boot because it delivers embedded servlet container (can be tomcat, jetty, etc. . . ), so this was one of the easiest parts to agree with the team.

“Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can ‘just run’. We take an opinionated view of the Spring platform and third-party libraries so you can get started with minimum fuss. Most Spring Boot applications need very little Spring configuration.”

Decision about the framework and technology used for creating applications is important. This is the basic stone of the project and it is important that everybody in the team feels comfortable with the tools and technologies used.

### Communication
After our first decision, we had to think about the communication between the services. Communication between micro-services can be either synchronous or asynchronous. The main communication needed for our use case was to inform other services that some actions were performed by the user. In this publish/subscriber pattern the communication could be made asynchronously, allowing us to give a fast response to the user meanwhile all the processes triggered by the interaction of the user were finished.

For achieving this, the idea was to use a message queue. We tried out two solutions: Redis (as a message queue) and RabbitMQ. One of the requisites for the new system was that we had to assure that no data is lost in any case, therefore, so far Redis does not support reliable delivery, the decision was to use RabbitMQ.

RabbitMQ is a message broker that implements the AMQP (Advanced Message Queuing Protocol). It is robust, easy to use, open source and is supported in several languages like Java, .NET, Python and Ruby.

For working with RabbitMQ it is important to understand what a message broker is. For making an analogy with the real world, a message broker is like the post office. When you have to send a package, you need the content (the paper you introduce in it), the destination address and the source address. Once you go to the post office, they take responsibility of your package and they assure that it will be delivered or, in the contrary, they can send you a notification of failure on the delivery. A message broker works the same. It assures that your message will be delivered in an asynchronous mode, where you don’t have to wait for the acknowledge from the recipient.

At this point, we have applications that can communicate between each other. Now we have to solve some further problems: deployment, testing and monitoring.

## Deployment and distribution
There are different approaches for deploying and distributing an application. In our case, we wanted to be agnostic of the environment where the application would run. We wanted to be agile on this process and it had to be easy to maintain, easy to redeploy for new releases and easy to configure. Therefore, our focus was on Docker. Why Docker? Quoting the official site from Docker: “Docker is the world’s leading software containerization platform. Docker allows you to compose your application from microservices without worrying about inconsistencies between development and production environments, and without locking into any platform or language. Docker lets you design the entire cycle of application development, testing and distribution, and manage it with a consistent user interface. Docker offers you the ability to deploy scalable services, securely and reliably, on a wide variety of platforms.”

So Docker actually allows us to run our application always in the same environment. The only thing we have to do is to configure this environment and then, the deployable is always exactly the same. As long as with Docker we can generate an image of the application, we can create containers based on that image and deploy them fast in any environment capable of running Docker. Docker has daemons for Windows, Mac and runs natively in GNU-Linux (and, since the latest release, now they can also run natively in Mac). This means that, no matter which environment you use for developing, the application will run always the same, in development and in production.

## Testing
Testing was the most interesting topic in this project. Generally we write unit tests, API tests and integration tests (within the application), but this project is composed of several applications that communicate with each other, therefore we needed to properly test the use cases from the entry point (usage from a user) to the exit point.

The graph shows all the connections that exist between the services. As you may already have imagined, for testing all the services, we needed to make an orchestration of them first, configuring them and have them up and running with the proper connections. For that we used the “docker-maven-plugin” from “fabric8io”. The plugin allows the management of the Docker images and containers. It can build images, run containers, link containers between each other and plenty other Docker functionalities like push, stop, remove, etc. . .

What we did was to configure the set up exactly as it is used in production, assuring that the deployment of the delivered services will always work and the connectivity from the entry point to the exit point is preserved and the applications are capable of sending and receiving messages.

In the graph we can see that, apart of RabbitMQ and the entry points, there is an extra box called “JLament”. This is the “Testing tool/Framework” that we built for testing. In JLament we have the orchestration previously discussed, the test suite, mocks of external services not developed by the team (mocks of backends) and some services that allow the tests to communicate with the services, reading messages from the message queue, publishing messages, keeping track of the communications, and all sort of different functionalities needed for testing all use cases.

## Storage
This topic was one of the most controversial that we had. Choosing the right storage for your application is key, so we invested a good amount of time researching to find the best solution for us. The first approach we had was to use Redis. Every time a new technology is proposed and discussed, before adapting it, we always make a research about the technology, where we test what this technology can do for us. In the case of Redis, a non relational database, we made several proposals on how to store the data, pros and cons lists of each structure and then measurements of the performance of each approach. For example, we made a comparison about the performance that Redis would give us in case of storing the data in Strings or in Redis Hashes (stored in form of a map) and a pro-cons list before making our decision. In this case the pro-cons list was:

**And the performance results were:**
By gathering all this information, we could make decisions that were mature, well thought-out and adequate for the project. The result after testing Redis as a database technology was not successful for us. The performance that we got when the amount of data stored grow was out of our boundaries (we had timing constraints to satisfy).

Approach 1 is the String approach and Approach 2 is the Hash approach.

Approach with blue line is the String approach and Approach with red line is the Hash approach.

Then the infrastructure team decided that Amazon AWS would be the offer chosen for hosting the services in the cloud. Therefore we decided to take a look at the solutions provided by Amazon AWS. Using Amazon RDS seemed to be a good idea. It provides the possibility of using technologies well known to the developers of the team like MySQL, Oracle DB or PostreSQL. So we tried that solution only to find out that the data model as we wanted to store it would have required complex queries for simple use cases, making the usage of a relational database in the cloud quite costly and more complex than necessary. So we took a look at the other service that Amazon AWS provides for non relational databases: DynamoDB. “Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document and key-value store models. It’s flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad tech, IoT, and many other applications. 
<a href="https://aws.amazon.com/documentation/dynamodb/" target="_blank" rel="nofollow">aws.amazon.com/documentation/dynamodb</a> 

Setting it up was easy and the performance provided by DynamoDB is incredibly high. Amazon also provides a Java SDK for the connection with DynamoDB, making the development much easier.

## Monitoring
The last step in all projects is to have the right tools for monitoring the services. In a big project like this, where different companies are producing different components, there is the need of having centralized tools that can be used for anyone in the project for monitoring all components. Just imagine if every component (we are talking about more than 15 at the moment) would have its own monitoring tools. There would be no possibility to cross the data for debugging purposes without having a huge overhead. Therefore we all agreed to use ELK (ELK stands for Elasticsearch) for the logging system and InfluxDB + Grafana for the metrics. ELK stack allows us to centralize all logs from all components, using Logstash to push data to Elasticsearch that can be read and filtered by Kibana. This is a powerful stack for logging systems. Kibana provides different visualizations, creations of dashboards and even graphs for monitoring, for example, how many times a log statement occurs. InfluxDB + Grafana is the second monitoring stack. In InfluxDB we store all metrics produced by any component and Grafana allows us to create graphs for visualization of the data. With this stack we can have basic monitoring (i.e.: CPU, Memory usage, Threads available / on use), custom monitoring (i.e.: count of events of one type raised in the application), infrastructure monitoring (i.e.: provisioned read/write in dynamoDB and usage of that provisioned data). . .

This combination of both stacks, provides both, the infrastructure team and the component team with all the needed information for monitoring the systems and debugging, tracing or tracking any kind of misbehavior that may be produced by the normal usage of the platform.



Prevent latencys and typical issues of monolith with microservices.

large_1920x1200px_erfolgsfaktoren_agilität.jpg

medium_1920x1200px_erfolgsfaktoren_agilität.jpg

small_1920x1200px_erfolgsfaktoren_agilität.jpg

thumbnail_1920x1200px_erfolgsfaktoren_agilität.jpg

1920x1200px_erfolgsfaktoren_agilität.jpg

From Monoliths to micro-services

When I thought about writing about „AMQP vs HTTP“ I thought on writing about differences between both protocols, describe every single header and why it is there, how it is the flow of each message/package you are sending in each protocol, etc, etc, etc… But then I realized that all that grey information is out there, anyone that wants to read about the protocol itself can just „google it“ and have plenty information about each protocol.

If you were looking for that information here, just “let me google that for you”:  AMQP, HTTP. If you were looking for a real use case and a discussion about when to use each protocol and why, then you are in the right place.

### Case of discussion
Some time ago, when we started the decentralization of the huge monolith we needed a way to synchronize data between the core and the regions. Core is the name we gave to the place where the central database and the e-commerce part are, and the regions are parts of the world where we set our components (Europe, America, Asia…). The publish-subscribe pattern was fitting perfectly in our project because all data that had to be synchronized was always in one direction. Also, the pattern allows you to turn it around and make the subscriber the publisher of new information and have the possibility of a bi-directional synchronization in the case of future needs.

Back then we faced the fact that we had to build a world wide publish-subscribe pattern, where the core had to be talking with services of which we may not know if they are up and running and, the most important thing to do was to be reliable due to the high amount of traffic that we expected (not only core to region traffic but also traffic that is generated in the region, too). We needed to have a certain queuing mechanism that would hold the communications in case that the receivers are too busy to respond, but, at the same time, we did not want to block the producer neither (by retrying requests again and again). We could not lose any communication, and with that, any synchronization of data at all. In that moment, the idea of using messaging queues came into our minds. Messaging queues allow you to have a middleware that will persist all communications and, depending on how you build them, you can add the reliability that you may need, forcing the acknowledge of each package and using a chain of trust in the communication.

So there we were, knowing what to do, but not how. Of course each of us had a great idea! You can always build a very basic “write somewhere the communications that were not acked and try again later until it works”, we even tried to use Redis as a messaging queue (and it works good, but it is not reliable). Then, among others, we found RabbitMQ and the AMQP protocol. Before this, I have to say that most of the team did not know this technology at all. We had no experience whatsoever with messaging queues, so we had to learn a lot. We went through a trial-error phase until we tamed this technology into our will, and the rest is history. For internal communications that has to be made in an asynchronous way we use RabbitMQ. We have a broker per region plus one in the core and we use shoveling for the transferring messages between core and regions.

### Discussion
But still, after all the experiences that we have accumulated in almost an entire year using this technology in production with great and undoubted performance, and having the RabbitMQ brokers as a core of our communications between components, “AMQP vs HTTP” is a daily (maybe weekly) discussion in our office. People sometimes even get angry at each other because of this topic and we have long pointless discussions about why better to use RabbitMQ and AMQP or better not. There are four kinds of opinions (that at least I found/heard) about this topic (internal asynchronous communication between components):

- I prefer to use HTTP.
- I prefer to use AMQP.
- I do not have a preference on them, I just use them where they are already in use.
- I don’t care why using one or another protocol, I just want to synchronize data.

The kind of opinions 3 and 4 I am going to leave out for the commenting part of this article (we can discuss it if you want, but it would be a drift from the main subject), so I will focus the rest of the article on the other two kinds of opinions.

And now is when the tricky part of this article comes for me, because “prefer” is probably the wrong word. Should we (as developers) have “opinions” on technical matters? Or should we better analyse the facts and use what is convenient in each case? Our profession is not about instincts but more about facts (even though instincts are important too!). So, let’s go to facts:

1. Both protocols will allow you to have communication between components.
2. Both protocols are traceable.
3. Both protocols are well documented.
4. AMQP is asynchronous.
5. HTTP is synchronous.
6. HTTP is easy to debug.
7. People are familiar with HTTP.
8. HTTP is well mapped to an interface.
9. AMQP is easy to maintain and easy to scale.
10. AMQP has guaranteed message delivery.
11. With HTTP you need to have some kind of service discovery.
12. AMQP needs to know the broker to reach the queue to read/write.
13. RabbitMQ provides fanout mechanisms, shoveling and federation of queues (“out of the box”).
14. If RabbitMQ is restarted while processing a message and it’s not done yet, it will requeue automatically and other backend services will pick it up and process it.
15. HTTP is supported in almost every programming language.
16. RabbitMQ (and therefore, AMQP) is supported in several programming languages (JAVA, .NET, RUBY, Python, PHP, Objective-C, Scala, Groovy and Grails, Clojure, Jruby, Javascript, C, C++, GO, Erlang, Haskell, OCaml, Unity 3D, Perl, Common Lisp, COBOL).
17. …

The list of facts can grow for at least a couple of pages, but I think that there are enough key facts to be able to choose one or the other one depending on your situation.

### Facts pro-HTTP
- Debugging an HTTP request is really easy and repeatable, whereas an AMQP message is harder to debug (you need connection to the queue, libraries, maybe scripting, etc…).
- HTTP is a familiar technology for the developers, so there is no need to have extra training for a new developer in the project.
- HTTP is the most supported protocol in internet, so sharing your APIs as HTTP API is a good practice.

### Facts pro-AMQP
- Delivering messages with AMQP gives you reliability and being asynchronous allows you to not worry about the delivery at all.
- Knowing the host/IP of the cluster of AMQP brokers is enough to deliver/receive messages, whereas with HTTP you may have different hosts and IP depending on regions.
- You can use fanouts, meaning that one message will be enough to inform several different components, reducing the amount of communications.

### Considerations when using AMQP
For delivering and receiving AMQP messages you need a broker. A broker is nothing more than a server that receives, stores and delivers messages. Depending on your costs and infrastructure, this fact may add extra complexity to your project, having to configure and maintain one/several/federation/… of brokers.

In our case we externalized this part using CloudAMQP. This made us not having to worry about maintenance but added an extra cost to the final solution.

### Pre-Conclusion
I would say that, if you want to talk with the world (provide an API for third party usage), HTTP is giving all you need. It is supported, well known and widely used. Also, you “do not have to care” if there is a problem in the communications, because it is the duty of the client of your API to execute the request again if something went wrong.

If you want to have internal communication, where you control every single request, I would use AMQP, because it is easy to use, supported in most of the languages used nowadays, reliable, scalable and fast.

I think both protocols are great, no doubt about that. But only one of them adds reliability on your communications “out of the box” and that is key, also, only one of them is scalable almost out of the box and the fact that communications are not blocking the services allows you to continue using your resources for other processes. Furthermore, why re-inventing the wheel creating our own reliability process “just because”? Why adding extra code that has to be maintained only because “we can”? I have it clear, if it is asynchronous, use AMQP with RabbitMQ.

### Conclusion
Probably you already realized that the entire discussion makes no sense, didn’t you? We are comparing shoes with t-shirts here, and that is a mistake that we have been doing for a long time.

**Of course, reviewing the chosen infrastructure is always good, but we have to take into consideration when and why.**

Blog

live and dev

communicode Author

Federico José Sörenson Sánchez

Articles to the keywords

Article

From Monoliths to micro-services

AMQP vs. HTTP