TechLog
Backend
Microservice
Message Queue

Message queue

A message queue is a communication mechanism used in software architecture to enable communication and data transfer between different components or systems within a distributed application. It facilitates asynchronous communication, allowing decoupling between the producer of a message and the consumer, improving system scalability, reliability, and responsiveness.

Here are the key components and features of a message queue:

  • Producer:

The component or system that generates and sends messages to the message queue. The producer is responsible for creating the message and sending it to the designated queue.

  • Consumer:

The component or system that retrieves and processes messages from the message queue. Consumers can process messages at their own pace and independently from the producers.

  • Message Queue:

The intermediary entity that holds the messages temporarily until they are retrieved and processed by consumers. Messages are typically stored in a queue in a first-in, first-out (FIFO) manner.

  • Message:

The data package that contains information to be transmitted from the producer to the consumer. Messages can include various types of data, such as text, JSON, XML, or binary data.

  • Asynchronous Communication:

Message queues enable asynchronous communication, meaning producers and consumers can operate independently and at their own speeds. Producers can send messages without waiting for immediate processing, and consumers can retrieve messages whenever they are ready to process them. Decoupling:

Message queues help decouple producers and consumers, allowing flexibility in system design and reducing dependencies. Changes to the producer or consumer do not directly affect each other, as long as the message format and contract are maintained. Message queues are widely used in various scenarios, including distributed systems, microservices architectures, enterprise applications, and real-time processing. They improve system reliability, scalability, and fault tolerance by providing a buffer for handling bursts of data or traffic and ensuring that messages are not lost even if the consumer is temporarily unavailable. Popular message queue systems include Apache Kafka, RabbitMQ, Amazon SQS (Simple Queue Service), and Apache ActiveMQ.

UseCase

In microservice communication, there are 4 possible cases when direct communication (i.e. without middleware) can cause problems.

  1. Service A sends a synchronous request to Service B, but the request takes too long so clients using Service A feels lag.
  2. Service A sends a request to Service B but Service B was shut down when it received the request.
  3. Service A sends a request to Service B but Service B was shut down while processing the request.
  4. Multiple nodes of Service A sends request to Service B (single node), so Service B crashes.

Message queue works as an asynchronous middleware that solves these issues.

RabbitMQ vs Kafka

RabbitMQ

RabbitMQ is an open-source message broker that is implemented on the AMQP(Advanced Message Queuing Protocol).

Publisher: The sender of the message.

Consumer: The receiver of the message.

Exchange: Responsible for delivering the message from the publisher to the queue.

Queue: A buffer for storing messages.

Before going directly to the queue, the message goes through an additional step called "exchange." In RabbitMQ, a message is not sent directly to the queue; instead, it is delivered to the exchange, and the exchange then puts the message into the queue.

Let's briefly summarize the distinctive features of RabbitMQ, based on this operational difference:

Key Differentiators of RabbitMQ:

  1. Focuses on a broker-centric model, emphasizing guaranteed message delivery between publishers and consumers.
  2. Easy cluster configuration, providing a Management UI and plugins for excellent extensibility.
  3. Often used for building services focusing on management aspects and implementing various functionalities rather than just data processing.

RabbitMQ's design and features prioritize reliable message delivery between publishers and consumers, making it suitable for a wide range of applications where message queuing is a crucial component. The additional step involving the exchange allows for more flexible message routing and processing.

Kafka

Kafka operates based on a publish-subscribe (pub-sub) model, primarily involving publishers (producers), subscribers (consumers), and a Kafka cluster.

Here's a brief summary of the operational principles, referring to the diagram mentioned above:

Kafka Operational Principles:

  1. Publishers categorize the message they want to transmit through topics.
  2. Subscribers read messages by subscribing to the desired topics.
  3. Publishers and subscribers only know topic information about each other.
  4. Kafka is designed with brokers forming a cluster.
  5. Within the cluster, distributed processing of brokers is handled by ZooKeeper.

Let's elaborate a bit on the mentioned points and introduce additional relevant terminology.

Additional Terminology:

  • Topic:

    • A message in Kafka is categorized by a topic.
    • Each topic is divided into multiple partitions.

  • Partitioning of Topics:

    • Topics are divided into multiple partitions to process in parallel.

Explanation of Broker and ZooKeeper:

  • Broker:
    • Refers to a Kafka server.
  • ZooKeeper:
    • Responsible for managing information about the distributed message queue and is an essential component.

Now, let's highlight the distinctive features of Kafka:

Key Differentiators of Kafka:

  1. Ensures persistence by storing messages in the file system.
  2. Operates on a pull-based approach, where subscribers (consumers) directly retrieve messages from brokers.
  3. Focuses on a publisher (producer)-centric model, particularly partitioning large amounts of data.
  4. Subscribers remember the delivery status.
  5. Suitable for processing massive amounts of data efficiently.

Kafka's design and features make it a robust solution for handling large-scale data processing and efficient message distribution within a distributed system.

Priority Queuing Pattern

Let's say a service it needs to send various notification through a message broker. It may need to send 'update of terms', 'welcome message', or 'emergency software update' notice. These messages have different priority. Thus, it would be necessary to assign priorities on messages, so that consumers can consume the messages with higher priority first.

There are two ways of implementing it.

  1. One queue with priorizied message: O(lg n) insertion.
  2. Multiple queues, each with different priority level. O(1) insertion.

Reference

TILMicroservice Dynamic Bootstrap