System Design · Core

Message Queues

Buffer and decouple work: why queues exist, ACKs, delivery guarantees, Kafka vs SQS vs RabbitMQ, and dead-letter queues.

In simple words: A queue lets producers hand off work without waiting for consumers. It absorbs bursts and supports retries, but consumers must handle duplicates and backlog growth.

Concrete example: After an Instagram photo upload, the API stores the image and publishes a processing job. Separate workers resize it, scan it, and create thumbnails without keeping the user request open.

Short version: HLD Message Queues.

1. Why queues

A queue sits between producers and consumers so they do not have to be up, fast, or scaled together. Classic wins: absorb spikes, retry safely, and fan out work to worker pools.

Sync pain: Upload API waits for resize + virus scan + notify → slow, fragile With queue: Upload API writes object + enqueues "process:photoId" Workers pull jobs at their own pace API returns 202/200 quickly

Decoupling: producers unaware of consumer count or language.
Smoothing: backlog grows instead of dropping requests.
Reliability: durable messages survive consumer crashes (with ACKs).

2. Core actors

Producer → [ Queue / Topic ] → Consumer(s) | optional DLQ

Producer: enqueues a message (command or event).
Broker: stores and delivers.
Consumer: processes; scales horizontally.

3. Acknowledgements (ACKs)

1) Consumer receives message (not deleted yet) 2) Processes work 3) Sends ACK → broker deletes / advances offset If crash before ACK → redelivery after timeout / rebalance

Visibility timeout (SQS) or unacked redelivery (Rabbit) or offset commit (Kafka) are variants of the same idea: do not drop work on failure.

4. Delivery guarantees

Guarantee	Meaning	Practice
At-most-once	May lose; no dup	Metrics where loss is OK
At-least-once	No loss; may dup	Default; need idempotent consumers
Exactly-once	Process once end-to-end	Hard; often “effectively once” via idempotency + dedupe

Non-idempotent: balance += 10 Idempotent: apply txn_id T once; store processed_ids

5. Ordering

Single queue / single partition → total order (limits throughput).
Partition by key (user_id) → order per key, parallel across keys (Kafka).
Competing consumers on one Rabbit queue → no global order.

6. Partitioning and consumer groups

One queue or partition has a throughput limit. Split a topic into partitions so several consumers can process messages in parallel. Within one consumer group, each partition is assigned to at most one active consumer at a time.

Topic: photo-jobs Partition 0 → Consumer A Partition 1 → Consumer B Partition 2 → Consumer C If the group has 5 consumers but only 3 partitions, 2 consumers are idle.

Choose a partition key that preserves the order you need and still spreads traffic. For a payment stream, account_id keeps one account's events in order. A low-cardinality key such as city can create a hot partition when one city produces most of the traffic.

7. Back pressure and backlog

A queue delays overload; it does not create processing capacity. If producers publish 300 jobs per second and consumers finish 200, the backlog grows by 100 jobs per second. Eventually retention, disk, or the business latency limit is exhausted.

Monitor queue depth, oldest-message age, consumer lag, and processing failures.
Scale consumers and partitions when the work can be parallelized.
Rate-limit producers or reject non-critical work when lag crosses a safe threshold.
Use bounded retries with exponential backoff and jitter so failures do not create retry storms.

8. Kafka vs SQS vs RabbitMQ

	Kafka	SQS	RabbitMQ
Model	Log / topics / partitions	Managed queue	Smart broker, exchanges
Replay	Yes (retain by time/size)	No (delete on ACK)	Limited
Fan-out	Consumer groups	SNS+SQS or multiple queues	Exchanges → queues
Ops	Heavier (or managed MSK)	Serverless-ish	Moderate
Best for	Event streams, high throughput	Simple async jobs on AWS	Routing, protocols, classic MQ

Interview default: Kafka for high-volume event streams and replay; SQS for straightforward task queues on AWS; Rabbit when you need flexible routing patterns.

9. Dead-letter queues (DLQ)

Main queue --fail N times--> DLQ Ops inspects / replays / alerts Poison message does not block the partition forever

Set max receive count; alert on DLQ depth.
Fix consumer bugs; replay when safe.
Do not silently drop business-critical messages.

10. Anti-patterns

Using a queue for sub-100ms synchronous UX without a clear async story.
Huge messages — store blob in S3, queue the pointer.
No idempotency while relying on at-least-once.
Unbounded retry without DLQ or backoff.

11. Sketch for designs

Draw producer → topic/queue → consumer group → DB side effects, plus DLQ. Call out partition key, ACK/idempotency, and how you scale consumers.

Quick revision

Queues decouple, buffer spikes, and enable retries.
ACK after success; crash before ACK → redelivery.
At-least-once + idempotent consumers is the practical combo.
Kafka = log/replay/partitions; SQS = managed tasks; Rabbit = routing MQ.
Order per partition key, not globally across the cluster.
Consumer parallelism is limited by partition count within a group.
A growing backlog needs scaling or back pressure; a queue is only a buffer.
DLQ for poison messages; alert and replay.
Queue pointers to big payloads, not the payloads themselves.