HLD · Queues

Message Queues

Decouple services with buffers so work can run asynchronously, absorb traffic spikes, and survive downstream failures.

A message queue is a buffer between services. It can hold work while consumers are busy or temporarily unavailable.

1. The Necessity of Message Queues

Consider a photo-sharing application like Instagram. When a user uploads a photo, the system must perform several time-intensive operations:

• Resizing the image into multiple resolutions
• Applying filters
• Running content moderation checks

The limitations of synchronous architecture

In a simple synchronous design, the server performs all these tasks before returning a response. This approach suffers from three primary flaws:

1. High latency: The user must wait for all background tasks to complete (e.g., 6+ seconds) before receiving a success confirmation.
2. Fragility: If a single sub-service (like the filter service) crashes, the entire upload fails, and previously completed work (like resizing) may be lost.
3. Inability to handle bursts: If traffic spikes from 50 to 5,000 uploads per second, a fixed-capacity server will get overwhelmed, leading to timeouts and dropped requests.

The asynchronous solution

By introducing a message queue, the server simply saves the file and writes a message (e.g., “Photo 456 needs processing”) to the queue. The server immediately responds to the user, while a pool of background workers (consumers) pulls messages from the queue to process the images at their own pace.

2. Core Concepts and Architecture

At its most basic level, a message queue is a buffer sitting between a producer and a consumer.

• Producer: The service that creates the work (e.g., the web server handling the upload). It sends a message and moves on without waiting for the work to finish. Delivery still depends on the queue's durability and acknowledgement settings.
• Consumer: The service that performs the work (e.g., the worker server). It pulls messages from the queue and processes them.
• Decoupling: Producers and consumers do not need to know about each other, allowing them to scale independently.

3. Internal Mechanisms and Reliability

Acknowledgements (ACKs)

To prevent data loss if a worker crashes mid-process, queues use ACKs:

• When a consumer pulls a message, the queue does not delete it immediately.
• The consumer must send an ACK back to the queue upon successful completion.
• If the consumer crashes before sending an ACK, the queue redelivers the message to another worker.

Controlling concurrent delivery

Queues coordinate workers so one consumer normally owns a message at a time. Redelivery can still create duplicates, so the consumer must remain idempotent:

• SQS (visibility timeouts): When a message is picked up, it becomes invisible to others for a configurable window (e.g., 30 seconds).
• Kafka (partitioning): Assigns specific partitions to exactly one consumer in a group to avoid competition.
• RabbitMQ: Uses acknowledgements and prefetch limits; an unacknowledged message can be requeued when its channel or connection closes.

4. Delivery Guarantees

In distributed systems, ensuring exactly how many times a message is delivered is a significant challenge.

The importance of idempotency

Because at-least-once is the most practical guarantee, consumers must be idempotent—processing the same message twice produces the same result.

• Non-idempotent: “Increment post count by 1.”
• Idempotent: “Set post count to 54”, or check whether the action was already done before executing it again.

5. When to Use (and Avoid) Message Queues

Usage signals

1. Async work: When the user does not need the result immediately (e.g., sending emails, generating reports).
2. Bursty traffic: To smooth out load spikes by accumulating a backlog.
3. Decoupling: When services have different hardware needs (e.g., upload servers need high throughput, while workers need GPUs).
4. Reliability: When work must be persisted even if downstream services are temporarily offline.

Anti-patterns

Avoid adding a queue when the caller must receive the completed result in the same request and direct communication already meets the load and reliability needs. A queue adds another dependency and makes the result asynchronous; it is not automatically too slow for every low-latency use case.

6. Deep Dives: Scaling and Failure Handling

Partitioning and consumer groups

A single queue has throughput limits. To scale horizontally, queues are split into partitions.

• Partitioning: Dividing a logical queue into multiple independent sequences that can be processed in parallel.
• Consumer groups: A pool of workers that divides partitions among themselves.

Note: You cannot have more active consumers than partitions; extra consumers will sit idle.

Selecting a partition key

Choosing the right key is critical and involves a trade-off between ordering and distribution:

1. Ordering: Messages with the same key go to the same partition. Vital for sequential tasks (e.g., a bank deposit must be processed before a withdrawal for the same account ID).
2. Distribution: Keys should spread work evenly. Partitioning by “City” might create a hot partition for a large city like New York while smaller cities sit idle. Partitioning by a unique ID usually provides a more even spread.

Back pressure

A queue is a buffer, not a capacity solution. If producers consistently create 300 messages/second while consumers only process 200, the queue will grow indefinitely until it runs out of memory.

Solutions: Auto-scaling consumers, adding partitions, or applying back pressure (slowing down or rejecting requests from the producer).

Failure scenarios: poison messages and DLQs

A poison message is a malformed message that causes a consumer to crash every time it is processed. To prevent infinite retry loops:

• Max retry count: Configure a limit on how many times a message can be attempted.
• Dead letter queue (DLQ): After reaching the retry limit, the message is moved to a separate DLQ for inspection, allowing the main queue to continue moving.

Concrete example: An email worker receives an event with no recipient address. After three failed attempts, the event moves to the DLQ, an alert is raised, and valid email events continue to be processed.

7. Comparison of Common Technologies

Note on Kafka durability

Kafka persists records for a configurable retention window, independent of whether a consumer has read them. This allows message replay, where a new or fixed consumer can re-process historical data if an error is discovered in previous processing logic.