Message Queue

Four mechanics make this work. Each one answers a "what if" question.

1. The queue saves the message before confirming. The broker is the software that runs the queue. It writes the message to disk first, then confirms to the producer. From that moment, a worker crash, a deploy, or an hour of downtime loses nothing. The work waits.

2. Workers confirm after finishing, not before. A worker takes a message and does the job. Only then does it send an acknowledgment, called an ack. Suppose the worker crashes partway through. The broker waits a set time, called the visibility timeout, and hears no ack. It then hands the message to another worker.

Note the consequence. Suppose a worker finished the job but crashed just before acking. The broker cannot tell the difference, so it delivers the message again, and the job runs twice. This is where at-least-once delivery comes from: every message is delivered one or more times, never zero. The price of never losing work is sometimes doing it twice.

That is why every consumer needs idempotency. Idempotent means running the same job twice gives the same result as running it once. That lesson covers it fully. For now, remember Rule 2 from the module introduction: duplicates are normal, so plan for them.

3. Many workers share one queue. Each message goes to exactly one of them. Need more throughput, that is, more messages finished per second? Add workers. No coordination code is needed. This setup is called competing consumers.

4. Queue depth is the clearest signal of health. Depth is the number of tasks waiting: arrivals minus completions, adding up over time. Depth rising during a spike means the pattern is working. Depth rising for a week is a warning. The self-check below covers it.

Together, the queue separates three things that the synchronous design had combined into one:

1. Rate: producers can burst to 5,000 per second while workers steadily process 400.
2. Time: workers can be down, deploying, or not built yet. The work waits.
3. Failure: a worker crash becomes a redelivery, not lost data. The producer never notices.

4. Walkthrough with Numbers

Replay the flash sale, this time with a queue between the API and the pipeline.

10:00 to 10:02. 5,000 orders arrive per second. The API validates each one, adds it to the queue, and replies "Order confirmed" in about 50 ms. Buyers see no errors. Meanwhile the queue grows. 5,000 in and 400 out each second means it grows by 4,600 per second. Over 120 seconds, that is a backlog of about 552,000 messages. A backlog is the pile of work not yet done. The queue handles this easily; disk is cheap.

10:02 to 10:25. Workers keep processing at 400 per second. Clearing 552,000 messages at 400 per second takes 552,000 / 400 = 1,380 seconds, about 23 minutes. During that window, an order's fulfillment might start 20 minutes after purchase instead of instantly. Nobody notices. The confirmation email says "shipping today," and that is true. Importantly, someone agreed on this trade with the business beforehand. That is what makes it a design instead of an accident.

10:07, the failure drill. A worker dies in the middle of a message. The visibility timeout expires. Another worker picks up the message and finishes it. The idempotency check ensures the half-done work does not apply twice. Zero orders lost.

The bill. One queue service, costing pennies per million messages, replaced the plan to run 12x the fleet.

One sentence to remember: the queue did not add any capacity. It removed the requirement that processing keep up with arrivals second by second. The business could always afford that.

5. Trade-offs

You gain:

• Spikes become backlog instead of errors, limited by disk instead of by your slowest service.
• A worker being down becomes a growing task list, not an outage. Deploys stop being risky.
• Producers and workers scale separately. Queue depth is a ready-made signal for auto-scaling.
• Redelivery gives you retry behavior without writing retry code. The broker does it.

You pay:

Latency, on purpose. Work now finishes seconds to minutes later. Anything the user actively waits for does not belong on a queue.

Duplicates. At-least-once delivery means workers must be idempotent. This is a requirement, not an optional extra.

Ordering is lost. Twenty workers process in parallel, so messages no longer finish in the order they arrived. You can restore order per key, so all messages for one customer stay in sequence. That costs some throughput. The self-check shows the bug you get for ignoring this.

A gap between "accepted" and "done." The order is confirmed but not yet fulfilled. The product team has to decide what users see during that gap.

More parts to operate. A broker to run. Depth and message age to monitor. A place for failed messages, covered in the dead letter queue lesson. And debugging that spans several systems instead of one stack trace.

6. When Not to Use It

1. The caller needs the answer. Login checks, price lookups, search results: the user is waiting. That is request-response. You cannot email someone their search results.

2. Tight end-to-end deadlines. If the whole job must finish in 200 ms, a broker hop plus a waiting worker adds risk for no benefit.

3. Small, simple systems. A service doing 10 requests per second does not need a broker between two of its own functions. It needs a function call. A queue is worth its complexity at scale, during spikes, or at failure boundaries.

4. As a database. A queue passes work along; it does not store data. If you need to query messages, update them, or keep them permanently, use a real data store.

The most subtle misuse is using a queue to hide a capacity problem. If depth trends upward for days, you are not absorbing a spike anymore. Arrivals permanently exceed capacity, and the queue is hiding it until the disk fills. A queue absorbs bursts; it does not create capacity.

7. Message Queue vs. Publish-Subscribe

This is the most common point of confusion in the module, and a favorite interview question. Both use a broker. Both are asynchronous. They answer different questions.

"Resize this image" is a job. One worker should do it, once: use a queue. "An order was placed" is a fact. Fulfillment, email, analytics, and fraud detection all care, each for its own reason: use publish-subscribe. The two also compose. Publish the fact once, and let each interested team feed its own work queue from it. That is exactly what AWS's SNS-into-SQS setup does. Kafka's consumer groups provide the same thing built in. If you can explain those last two sentences, you have this comparison covered.

8. Real-World Examples

• AWS SQS is the pattern in its purest form: visibility timeouts, dead-letter support, near-unlimited depth, pennies per million messages.
• RabbitMQ is the classic self-hosted broker. It adds rich routing options on top of the same ack-and-redeliver core.
• Kafka is technically a replicated log rather than a queue. A replicated log is an append-only record copied across several machines. Its consumer groups behave like a queue. Its ability to replay old messages supports event-driven architecture and the stream processing patterns later in the course.
• Celery, Sidekiq, BullMQ: the same pattern packaged as an application library. This is where most engineers first meet it.

For AI engineers: GPU work is the workload that fits a queue best in modern infrastructure. It is expensive, bursty, and usually not urgent. Embedding pipelines, batch inference, fine-tuning jobs, and evaluation runs are all fed by queues. Queue depth, or better, the age of the oldest message, is the standard signal for GPU auto-scaling. A RAG ingestion pipeline is this pattern, with the dead letter queue collecting whatever fails. The batch APIs from Anthropic and OpenAI are this pattern sold as a product. They are cheaper because you agreed the work does not need to be fast.

9. In the Interview

"How do you handle a 10x traffic spike?" and "Design an order pipeline" are both really queue questions. The senior version arrives as a review of your own diagram. The interviewer points at a synchronous arrow and asks, "Does the user need to wait for this?" Every arrow where the honest answer is "no" is a queue you have not drawn yet.

The 30-second answer: "I put a durable queue between the API and the pipeline. The API validates, enqueues, and confirms in milliseconds. Workers process at their sustainable rate, so a 5,000-per-second spike becomes backlog instead of errors. The broker saves each message before confirming and redelivers if a worker dies. That means at-least-once delivery, so my consumers are idempotent, and repeatedly failing messages go to a monitored dead letter queue. Queue depth drives my worker autoscaling and alerting. The trade is latency. This only works because the business agreed fulfillment can take minutes. Anything the user actually waits for stays synchronous."

Likely follow-ups:

1. "A worker crashes halfway through a message. Walk me through it." The visibility timeout expires and the broker redelivers to another worker. The idempotency check prevents double-applying. If the message fails several times in a row, it goes to the dead letter queue and triggers an alert. This follow-up leads directly into the idempotency and dead letter queue lessons.
2. "Orders for the same customer must process in order. Now what?" With competing consumers, global order is gone. Partition by customer ID, using Kafka partition keys or SQS FIFO (first in, first out) message groups. A partition is one slice of the queue that keeps its own strict order. You get strict order within each customer and parallel processing across customers. The honest cost: one very busy customer is limited to a single partition's speed.

A mistake that fails candidates: putting a queue on a path the user is waiting on, like "the search request goes into Kafka." Or presenting a queue with no answer for duplicates. The first says you memorized the shape but not the reason. The second says you have not yet had to deal with a duplicate delivery in production.

10. Check Yourself

Q1 (recall). A queue separates producers from consumers in three ways. Name them, and explain step by step where at-least-once delivery comes from.

Q2 (trade-off). Queue depth has grown steadily for six days. A teammate proposes raising the queue's storage limit. What is the depth actually telling you, and why is the proposal a trap?

Q3 (scenario). Charges and refunds for the same customer flow through one queue with 20 workers. Finance reports that a customer's refund was processed before the charge it refunds, leaving their balance negative. Explain how that happened, and fix it without giving up parallel processing.

A1. Rate: producers burst while workers process steadily. Time: workers can be down; the work waits safely. Failure: a worker crash becomes a redelivery, not lost data. At-least-once delivery, step by step: workers ack only after finishing. The broker redelivers anything not acked in time. The broker cannot tell "crashed before finishing" apart from "finished but crashed before acking." So it must redeliver in both cases, and the second case produces a duplicate.

A2. Six days of steady growth means arrivals have permanently outpaced processing. This is not a spike being absorbed. It is a capacity shortfall being hidden, and the cost keeps growing: the oldest message is now six days late. Raising the storage limit only delays the discovery. The real options are three. Add worker capacity, or fix the workers' bottleneck. Reduce or reprioritize what flows in. Or change what work goes through this path at all. Also, alert on the age of the oldest message, not just the depth. Age is what users actually experience.

A3. Twenty parallel workers destroyed the ordering. The charge and the refund were picked up by different workers, and the refund's worker finished first. Fix it with key-based ordering. Partition the queue by customer ID, using a Kafka partition key or an SQS FIFO message group. Each customer's messages then process strictly in sequence, while different customers still run in parallel. Two costs to state out loud. A single very active customer is limited to one partition's throughput. And a stuck message now blocks that customer's partition. Note the dead letter queue lesson's ordering caveat. For strictly ordered streams, moving a failed message aside breaks the sequence, so this path may need pause-and-fix instead.

11. Related Patterns

• Idempotency: at-least-once delivery makes every consumer a retry target. This is the queue's non-negotiable prerequisite.
• Dead Letter Queue: where messages go after redelivery gives up. The queue's place for failed work.
• Retry with Exponential Backoff: redelivery is retrying, done by the broker. Tune attempt counts and delays with that lesson's logic.
• Publish-Subscribe: the matching pattern for facts rather than jobs. Often composed as one event copied into per-team queues.
• Auto-Scaling: queue depth and backlog age are the cleanest scaling signals in distributed systems.

12. TL;DR