Message Queue

Four mechanics provide this behavior, and each one answers a failure question.

1. The queue saves the message before confirming. A broker is the software that manages the queue. It writes each message to disk before confirming receipt to the producer. After that confirmation, a worker crash, deployment, or hour of downtime does not lose the work because the message remains stored.

2. Workers confirm after finishing, not before. A worker receives a message, completes the job, and then sends an acknowledgment called an ack. If the worker crashes during processing, the broker waits for a configured visibility timeout. When no ack arrives, it gives the message to another worker.

This behavior has an important consequence. A worker might complete the job and then crash before sending its ack. The broker cannot distinguish that case from a crash during processing, so it delivers the message again and the job runs twice. This is at-least-once delivery: each message arrives one or more times, but never zero times. Preventing lost work therefore creates occasional duplicate work.

Every consumer consequently needs idempotency. Idempotent processing produces the same result whether a job runs once or twice. A later lesson covers this property fully, but the module rule is enough for now: duplicate delivery is normal and must be planned for.

3. Many workers share one queue. Each message is assigned to exactly one worker. If the system needs greater throughput, meaning more completed messages per second, add workers. This arrangement is called competing consumers and does not require custom coordination code.

4. Queue depth is the clearest signal of health. Depth counts the waiting tasks, or arrivals minus completions accumulated over time. Rising depth during a short spike shows that buffering is working. Depth that rises for a week indicates a capacity problem, as the self-check explains.

Together, these mechanics separate three concerns that the synchronous design combined:

1. Rate: producers can burst to 5,000 per second while workers steadily process 400.
2. Time: workers can be down, deploying, or not built yet. The work waits.
3. Failure: a worker crash becomes a redelivery, not lost data. The producer never notices.

4. Walkthrough with Numbers

Now replay the flash sale with a queue between the API and the order pipeline.

10:00 to 10:02. The service receives 5,000 orders per second. For each order, the API validates the request, adds a message to the queue, and replies "Order confirmed" in about 50 ms. Meanwhile, the queue grows by 4,600 messages per second because 5,000 arrive and 400 finish. Across 120 seconds, that produces a backlog of about 552,000 messages. A backlog is work that has been accepted but not completed, and the queue can store it cheaply on disk.

10:02 to 10:25. Workers continue processing 400 messages per second. At that rate, clearing 552,000 messages takes 552,000 / 400 = 1,380 seconds, or about 23 minutes. Some fulfillment work begins 20 minutes after purchase, but the delay does not affect buyers because the confirmation promises only "shipping today." The business approved this trade-off in advance, making the delay an intentional design decision.

10:07, the failure drill. A worker stops during message processing, so its visibility timeout expires and another worker receives the message. The idempotency check prevents the partially completed operation from being applied twice. No orders are lost.

The bill. A queue service that costs pennies per million messages replaces the proposal to operate 12x the normal fleet.

The queue did not increase processing capacity. Instead, it removed the requirement for processing to match arrivals every second, which the business did not need.

5. Trade-offs

You gain:

• Spikes become backlog instead of errors, limited by disk instead of by your slowest service.
• A worker being down becomes a growing task list, not an outage. Deploys stop being risky.
• Producers and workers scale separately. Queue depth is a ready-made signal for auto-scaling.
• Redelivery gives you retry behavior without writing retry code. The broker does it.

You pay:

Latency, on purpose. Processing now completes seconds or minutes later, so work that a user actively waits for does not belong on a queue.

Duplicates. At-least-once delivery requires idempotent workers rather than treating idempotency as an optional improvement.

Ordering is lost. With twenty workers processing concurrently, messages may finish in a different order from their arrival. Key-based ordering can preserve sequence for one customer, but it reduces some throughput. The self-check demonstrates the failure caused by ignoring this constraint.

A gap between "accepted" and "done." An order can be confirmed before fulfillment begins, so the product team must define what users see during this period.

More parts to operate. The system now includes a broker, monitoring for depth and message age, and storage for repeated failures. The dead letter queue lesson covers that storage. Debugging also crosses several systems instead of following one stack trace.

6. When Not to Use It

1. The caller needs the answer. Login checks, price lookups, and search results require an immediate response because the user is waiting. These cases need request-response, not an emailed search result.

2. Tight end-to-end deadlines. If the complete operation must finish within 200 ms, a broker hop and worker wait add risk without providing useful separation.

3. Small, simple systems. A service handling 10 requests per second does not need a broker between two local functions. Use a function call until scale, traffic spikes, or failure boundaries justify the additional system.

4. As a database. A queue transfers work rather than storing application data. Use a database when messages must be queried, updated, or retained permanently.

A more subtle misuse is allowing a queue to conceal a permanent capacity shortage. When depth increases for days, arrivals consistently exceed processing capacity and disk will eventually fill. Queues absorb temporary bursts, but they do not create capacity.

7. Message Queue vs. Publish-Subscribe

Both patterns use a broker and asynchronous delivery, which makes them easy to confuse. However, they answer different questions.

"Resize this image" describes one job that one worker should perform, so it belongs on a queue. "An order was placed" describes a fact needed independently by fulfillment, email, analytics, and fraud detection, so it belongs in publish-subscribe. The patterns can also be combined by publishing the fact once and feeding each team's work queue. AWS uses this structure with SNS feeding SQS, while Kafka provides the same structure through consumer groups.

8. Real-World Examples

• AWS SQS is the pattern in its purest form: visibility timeouts, dead-letter support, near-unlimited depth, pennies per million messages.
• RabbitMQ is the classic self-hosted broker. It adds rich routing options on top of the same ack-and-redeliver core.
• Kafka is technically a replicated log rather than a queue. A replicated log is an append-only record copied across several machines. Its consumer groups behave like a queue. Its ability to replay old messages supports event-driven architecture and the stream processing patterns later in the course.
• Celery, Sidekiq, BullMQ: the same pattern packaged as an application library. This is where most engineers first meet it.

For AI engineers: GPU workloads suit queues because they are expensive, bursty, and usually not urgent. Queues commonly supply embedding pipelines, batch inference, fine-tuning jobs, and evaluation runs. Queue depth, or preferably the age of the oldest message, provides the standard signal for GPU auto-scaling. A RAG ingestion pipeline uses the same pattern, with a dead letter queue collecting failures. Anthropic and OpenAI batch APIs package this trade-off as a product, charging less when the work can finish later.

9. In the Interview

Questions about handling a 10x traffic spike or designing an order pipeline often test queue design. An interviewer may instead examine a synchronous arrow and ask, "Does the user need to wait for this?" When the answer is no, that work may belong on a queue.

The 30-second answer: "I put a durable queue between the API and the pipeline. The API validates, enqueues, and confirms in milliseconds. Workers process at their sustainable rate, so a 5,000-per-second spike becomes backlog instead of errors. The broker saves each message before confirming and redelivers if a worker dies. That means at-least-once delivery, so my consumers are idempotent, and repeatedly failing messages go to a monitored dead letter queue. Queue depth drives my worker autoscaling and alerting. The trade is latency. This only works because the business agreed fulfillment can take minutes. Anything the user actually waits for stays synchronous."

Likely follow-ups:

1. "A worker crashes halfway through a message. Walk me through it." The visibility timeout expires, and the broker gives the message to another worker. Idempotency prevents repeated effects. After several consecutive failures, the message moves to a dead letter queue and triggers an alert. This answer connects to the idempotency and dead letter queue lessons.
2. "Orders for the same customer must process in order. Now what?" Partition by customer ID with Kafka partition keys or SQS FIFO (first in, first out) message groups. A partition is one queue segment that preserves its own ordering. Messages remain ordered for each customer while different customers process in parallel. However, one very active customer is limited by a single partition's capacity.

A mistake that fails candidates: placing a queue on a path the user must wait for, such as sending a search request through Kafka. Another mistake is proposing a queue without explaining duplicate handling. The first ignores why queues are used, while the second ignores at-least-once delivery.

10. Check Yourself

Q1 (recall). A queue separates producers from consumers in three ways. Name them, and explain step by step where at-least-once delivery comes from.

Q2 (trade-off). Queue depth has grown steadily for six days. A teammate proposes raising the queue's storage limit. What is the depth actually telling you, and why is the proposal a trap?

Q3 (scenario). Charges and refunds for the same customer flow through one queue with 20 workers. Finance reports that a customer's refund was processed before the charge it refunds, leaving their balance negative. Explain how that happened, and fix it without giving up parallel processing.

A1. The queue separates rate because producers can burst while workers process steadily. It separates time because work waits safely while workers are unavailable. It separates failure because a worker crash causes redelivery instead of data loss. At-least-once delivery follows from acknowledging only after completion. The broker must redeliver both a job interrupted during processing and a completed job whose worker crashed before acknowledging, so the second case produces a duplicate.

A2. Six days of continuous growth means arrivals permanently exceed processing capacity rather than representing a temporary spike. The oldest message is already six days late, and raising the storage limit only postpones the failure. The system must add worker capacity or remove a worker bottleneck, reduce or reprioritize incoming work, or move some work elsewhere. Alerting should also track the oldest message's age because that is the delay users experience.

A3. Twenty workers removed global ordering because different workers processed the charge and refund, and the refund finished first. Partition the queue by customer ID with a Kafka partition key or an SQS FIFO message group. Each customer's messages then remain sequential while different customers still run in parallel. One active customer becomes limited to one partition's throughput, and a stuck message blocks that customer's partition. As the dead letter queue lesson explains, moving a failed message aside can break strict ordering, so some streams require pausing and repairing the failure.

11. Related Patterns

• Idempotency: at-least-once delivery makes every consumer a retry target. This is the queue's non-negotiable prerequisite.
• Dead Letter Queue: where messages go after redelivery gives up. The queue's place for failed work.
• Retry with Exponential Backoff: redelivery is retrying, done by the broker. Tune attempt counts and delays with that lesson's logic.
• Publish-Subscribe: the matching pattern for facts rather than jobs. Often composed as one event copied into per-team queues.
• Auto-Scaling: queue depth and backlog age are the cleanest scaling signals in distributed systems.

12. TL;DR