Publish-Subscribe

The parts, one at a time:

1. The topic is the named stream of facts, such as order.placed. Checkout's entire job is now: publish here, once, and move on. One durable write, always the same cost, no matter how many teams listen. The partial-crash problem is gone, because there is only one write.
2. A subscription is one team's personal copy of the stream. Internally, each subscription works like that team's own message queue: it has its own backlog, its own position in the stream, its own acknowledgments and redeliveries, and its own dead letter queue. Everything the queue lesson established applies per subscription. If fraud's service is down for an hour, fraud's backlog grows. Email does not notice. Checkout does not either.
3. Joining is configuration, not code. When data science wants order events, they create a subscription. Checkout does not deploy, and checkout does not even know. With a broker that keeps history (Kafka is the best-known example), the new subscriber can start by replaying past events, feeding months of history into a brand-new model.
4. The event's format is now a public contract. This is the part teams underestimate. Once unknown consumers exist, the fields in your event are an API. Two rules follow. First, events state facts, not commands: "an order was placed, here are its details," never "send a receipt." The publisher reports what happened; each consumer decides what to do about it. Second, changes must be additive only: you may add new optional fields, but renaming or removing a field breaks subscribers you cannot see. Teams typically enforce this with a schema registry, a tool that checks event formats before they ship.

One sentence captures the idea: the producer says what happened, and every consumer decides what it means. Checkout does not know receipts exist, and that is the point.

4. Walkthrough with Numbers

The topic carries 200 orders per second. Compare the old design and the new one:

• Publishing cost. Old: five queue writes in a row, about 25 ms, growing with every new team. New: one durable write, about 5 ms, no matter how many subscribers exist.
• The data science request. They create a subscription in an afternoon. Zero checkout changes. With 30 days of history retained on the topic, they replay roughly 500 million past events to train on before switching to live traffic. The old answer was a PR, a release cycle, and no history at all.
• The failure drill. Fraud's consumer goes down for 40 minutes. Its subscription accumulates 200 x 2,400 seconds = 480,000 events of backlog, then drains when the service returns. Email sent every receipt on time. Checkout never noticed. In the old design, this same outage either slowed checkout down or made fraud silently miss 40 minutes of orders.
• The audit that now passes. Ask "which teams saw order X?" and every subscription has a precise answer: processed, still in backlog, or waiting in its dead letter queue.

5. Trade-offs

You gain:

• The producer's cost and risk stay constant no matter how many consumers exist.
• Teams ship on their own schedules. This organizational decoupling is the biggest benefit in practice.
• Failures are isolated per subscription, and with history retention, new consumers can start with the full past.

You pay:

• Your event format is a public API now. Renaming a field breaks consumers you do not know about. You traded deploy coupling for contract discipline: schema registries, additive-only changes, deprecation windows.
• Duplicates, multiplied. Each subscription is at-least-once on its own. Five subscribers means five places that must handle the same event twice. Idempotency, always.
• Invisible coupling. Without deliberate bookkeeping, nobody can list who consumes a topic. The dependency did not disappear; it became invisible. Mature teams keep a catalog of who subscribes to what.
• No "all done" signal. Pub/sub cannot tell you "everything that should happen for order X has happened." Each subscription only knows about itself. Workflows that need a completion answer need a different tool (see the comparison below).
• Slow consumers can lose data. History is not kept forever. A subscriber that falls behind the retention window does not just get slow; it loses events permanently. Alert on the age of each subscription's backlog.

6. When Not to Use It

1. It is a job, not a fact. "Resize this image" should be done once, by one worker. That is a message queue.
2. The producer needs the result. Checkout must know whether the payment authorized before it can continue. Publishing payment.requested and hoping someone handles it is not a design. Paths where the caller's next step depends on the outcome stay request-response, or use an orchestrated workflow.
3. Two services, low traffic. One producer with one consumer does not need broadcast infrastructure. A direct queue is simpler to run and to reason about.
4. Request-response rebuilt from events. Publishing user.lookup.requested and subscribing to user.lookup.completed is a synchronous call rebuilt out of events, with worse latency and much worse debugging. If the event name contains "request," it should have been a call.

The ecosystem-level misuse is topic sprawl: hundreds of topics, no owners, no schema rules, and consumers you only discover when something breaks. Pub/sub removes the natural pressure that forced teams to talk to each other. Mature organizations replace that pressure deliberately, with contracts and a catalog.

7. Choreography vs. Orchestration

Pub/sub enables a style called choreography: every service reacts to published events on its own, and the overall behavior emerges from those reactions. The alternative is orchestration: one coordinator explicitly runs the workflow step by step. Interviewers ask about this distinction often.

The test to apply: are the steps truly independent? If nobody cares whether email runs before analytics, choreography works well. But if the steps form a sequence with consequences (reserve inventory, then charge, then ship, and undo completed steps if a later one fails), that is a workflow, and it needs an orchestrator. The saga lesson covers exactly that. Real systems use both: orchestrate the order workflow itself, and have it publish facts at each step for everyone else to react to.

8. Real-World Examples

• AWS SNS feeding SQS queues is this pattern built from managed services: SNS is the topic, and each team's SQS queue is its subscription. Broadcast first, then each team gets its own task list.
• Kafka treats each consumer group as an independent subscription over a stored log, which is why it dominates this space: broadcast, replay, and per-team queue behavior in one system.
• Google Cloud Pub/Sub is the same model, fully managed, with subscriptions as first-class objects.
• Redis Pub/Sub is the cautionary example: it is fire-and-forget. If a subscriber is offline, the message is gone permanently. "Pub/sub" without durable subscriptions is a much weaker promise with the same name. Always check which one you are being offered.

For AI engineers: the interaction-event topic is the central stream of an ML platform. One user.interacted stream feeds the analytics warehouse, the feature store, the job that keeps vector embeddings fresh, and the recommender's real-time features. Four consumers, one publish, and all of it replayable when the next model needs history. Agent systems are adopting the same structure: every tool call and decision is published once, and observability, evals, and audit each consume it independently, without slowing down the agent.

9. In the Interview

The direct form of the question is "Five teams need order data. How do they get it?" The hidden form appears in any microservices design: you draw arrows between services, and the interviewer asks what happens when a sixth service needs the same data. Candidates without this pattern answer by drawing another arrow from the producer, and the producer keeps collecting arrows. There is also a trap form: you are given a genuine workflow (order, payment, shipping) and the interviewer waits to see whether you choreograph what should be orchestrated.

The 30-second answer: "Checkout publishes an order.placed fact to a topic, once. Every interested team gets its own subscription with its own backlog, retries, and dead letter queue, so a slow or crashed consumer affects nobody else. Adding a consumer is configuration, not a checkout deploy, and with a log-based broker like Kafka the new consumer can replay history. The costs I manage: the event schema is now a public API, so changes are additive-only and checked against a registry; every subscription needs idempotent processing; and I alert on backlog age so a slow consumer never falls past the retention window. If the steps are really a workflow that needs completion tracking, I orchestrate that part instead."

Likely follow-ups:

1. "One subscriber processes at half the speed of the others. What happens?" Its own backlog grows; everyone else is untouched. That is the isolation working. The real danger is silent: if it falls behind the retention window, it starts losing data. So alert on backlog age, and scale that consumer by adding workers within its subscription (the queue lesson's competing consumers, applied inside one subscription).
2. "How do you change an event's schema without breaking consumers you don't know about?" Additive changes only: new optional fields, never renames or removals. Enforce it with a schema registry that fails the build on breaking changes. For a truly breaking change, publish version 2 alongside version 1, give consumers a deprecation window, and keep a catalog so "consumers you don't know about" stops being a category.

A mistake that fails candidates: pointing several different services at one shared queue and calling it broadcast (each service just gets a random sample), or building request-response out of event pairs. Both mean the jobs-versus-facts distinction has not been understood.

10. Check Yourself

Q1 (recall). Explain the difference between a topic and a subscription. Then state exactly what it costs the producer when a sixth consumer team is added.

Q2 (trade-off). "Pub/sub decouples services." Argue the opposite for two minutes: what coupling survives, and what new discipline replaces the old deploy-time coordination?

Q3 (scenario). Fraud reports scoring only about 20% of orders. Email reports sending about 30% of receipts. The broker's metrics show nothing was lost, and checkout's publish count matches order volume exactly. Diagnose the setup and fix it.

A1. The topic is the single, durable stream of published facts. A subscription is one consumer's independent feed from that stream, tracking its own position, with its own backlog, acknowledgments, and dead letter queue: effectively a private queue filled from the topic. A sixth consumer costs the producer nothing: no code, no deploy, no added latency, no awareness. The new consumer carries the costs (build idempotent processing, watch its backlog), and the ecosystem carries one more party depending on the event schema.

A2. The data coupling survives; it changes shape. Every consumer now depends on the event schema, so renaming one field breaks systems the producer cannot even list, which is worse than a compile error unless it is managed. The discipline that replaces it: treat the schema as an API (registry, additive-only changes, deprecation windows), keep a catalog of consumers so dependencies stay visible, and give each subscription an operational owner. Pub/sub does not delete coupling. It moves coupling out of the deploy calendar and into the contract, which is cheaper only if someone actually governs the contract.

A3. All three services were pointed at the same subscription (or the same shared queue). That made them competing consumers: the broker delivered each event to exactly one of the three, so each service saw a random subset, sized by its worker count and speed. Nothing was lost and nothing errored, which is why it went unnoticed. The fix: one subscription per service, so fraud, email, and fulfillment each receive a full copy, and use competing consumers only inside a single subscription to scale that service up. The rule in one line: broadcast across services, compete within a service.

11. Related Patterns

• Message Queue: each subscription effectively is one, and the broadcast-into-queues setup (SNS into SQS) composes the two patterns directly.
• Idempotency: every subscription is at-least-once on its own, so N consumers means N chances to double-apply.
• Event-Driven Architecture: the next lesson. Pub/sub is the transport; that lesson decides what should travel over it.
• Change Data Capture: turns a database's own change log into the event stream, the disciplined version of "just read my tables."
• Webhooks: the same idea applied across company boundaries, over plain HTTP, with the guarantees renegotiated.

12. TL;DR