Publish-Subscribe

The moving parts, one at a time:

• The topic is the named stream of facts, like order.placed. Checkout's entire job is now: publish here, once, and move on. One durable write, always the same cost, no matter how many teams listen. The partial-crash lottery is gone, because there's only one write.
• A subscription is one team's personal copy of the stream. Under the hood, each subscription works like that team's own message queue: it has its own backlog, its own position in the stream, its own acknowledgments and redeliveries, and its own dead letter queue. Everything the queue lesson established applies per subscription. So if fraud's service is down for an hour, fraud's backlog grows. Email doesn't notice. Checkout doesn't either.
• Joining is configuration, not code. When data science wants order events, they create a subscription. Checkout doesn't deploy, and honestly, checkout doesn't even know. With a broker that keeps history (Kafka is the famous one), the new subscriber can even start by replaying past events, feeding months of history into a brand-new model.
• The event's format is now a public promise. This is the part teams underestimate. Once unknown consumers exist, the fields in your event are an API. Two disciplines follow. First, events state facts, not commands: "an order was placed, here are its details," never "send a receipt." The announcer reports news; each reader decides what to do about it. Second, changes must be additive only: you may add new optional fields, but renaming or removing a field breaks subscribers you can't see. Teams typically enforce this with a schema registry, a tool that checks event formats before they ship.

One sentence captures the philosophy: the producer says what happened, and every consumer decides what it means. Checkout doesn't know receipts exist. That ignorance is the feature.

4. Walkthrough with Numbers

The topic carries 200 orders per second. Compare the old world and the new one:

• Publishing cost. Old: five queue-writes in a row, about 25 ms, growing with every new team. New: one durable write, about 5 ms, forever, no matter how many subscribers exist.
• The data science request. They create a subscription in an afternoon. Zero checkout changes. With 30 days of history retained on the topic, they replay roughly 500 million past events to train on before switching to live traffic. The old answer was a PR, a release train, and no history at all.
• The failure drill. Fraud's consumer goes down for 40 minutes. Its subscription quietly accumulates 200 × 2,400 seconds = 480,000 events of backlog, then drains when the service returns. Email sent every receipt on time. Checkout never felt a thing. In the old world, this same outage either slowed checkout down or made fraud silently miss 40 minutes of orders.
• The audit that now passes. Ask "which teams saw order X?" and every subscription has a precise answer: processed, still in backlog, or parked in its dead letter queue. No more lottery.

5. Trade-offs: What You Pay

You gain:

• The producer's cost and risk stay constant no matter how many consumers exist.
• Teams ship on their own calendars. This organizational decoupling is the quiet superpower.
• Failures isolate per subscription, and (with history) new consumers can be born already knowing the past.

You pay:

• Your event format is a public API now. Renaming a field breaks consumers you've never met. You traded deploy coupling for contract discipline: schema registries, additive-only changes, deprecation windows.
• Duplicates, multiplied. Each subscription is at-least-once on its own. Five subscribers means five places that must handle the same event twice (idempotency, always).
• Invisible coupling. Without deliberate bookkeeping, nobody can list who consumes a topic. The dependency didn't disappear; it went quiet. Mature teams keep a catalog of who subscribes to what.
• No "all done" signal. Pub/sub can't tell you "everything that should happen for order X has happened." Each subscription only knows about itself. Workflows that need a completion answer want a different tool (see the face-off).
• Slow consumers can fall off the edge. History isn't kept forever. A subscriber that lags past the retention window doesn't get "slow": it gets data loss. Alert on how old each subscription's backlog is.

6. When NOT to Use It

• It's a job, not a fact. "Resize this image" should be done once, by one worker. That's a message queue, full stop.
• The producer needs the result. Checkout must know whether the payment authorized before it can continue. Publishing payment.requested and hoping someone handles it is not a design. Paths where the caller's next step depends on the outcome stay request-response, or use an orchestrated workflow.
• Two services, low traffic. One producer with one consumer doesn't need broadcast plumbing. A direct queue is simpler to run and to reason about.
• Fake request-response. Publishing user.lookup.requested and then subscribing to user.lookup.completed is a phone call rebuilt out of announcements, with worse latency and much worse debugging. If the event name contains "request," it wanted to be a call.

The big-ecosystem misuse is event soup: hundreds of topics, no owners, no schema rules, and consumers you only discover when something breaks. Pub/sub removes the natural pressure that forced teams to talk to each other. Mature organizations replace that pressure on purpose, with contracts and a catalog.

7. Pattern Face-Off: Choreography vs Orchestration

Pub/sub enables a style called choreography: every service reacts to announcements on its own, like dancers who each know their part, and the overall behavior emerges. The alternative is orchestration: one coordinator explicitly runs the workflow step by step, like a conductor pointing at each musician. Interviewers love this distinction.

The test to apply: are the steps truly independent? If nobody cares whether email runs before analytics, choreograph and enjoy the freedom. But if the steps form a sequence with stakes (reserve inventory, then charge, then ship, and undo things if a step fails), that's a workflow wearing an event costume, and it needs an orchestrator: the saga lesson picks up exactly there. Real systems use both: orchestrate the order workflow itself, and let it announce facts along the way for everyone else to react to.

8. In the Wild

• AWS SNS feeding SQS queues is this pattern drawn with managed services: SNS is the topic, and each team's SQS queue is its subscription. Broadcast first, then each team's own to-do list.
• Kafka treats each "consumer group" as an independent subscription over a stored log, which is why it dominates: broadcast, replay, and per-team queue behavior in one system.
• Google Cloud Pub/Sub is the same model, fully managed, with subscriptions as first-class objects.
• Redis Pub/Sub is the cautionary example: it's fire-and-forget. If a subscriber is offline, the message is gone, permanently. "Pub/sub" without durable subscriptions is a much weaker promise wearing the same name. Always check which one you're being offered.

For AI engineers: the interaction-event topic is the circulatory system of an ML platform. One user.interacted stream feeds the analytics warehouse, the feature store, the job that keeps vector embeddings fresh, and the recommender's real-time features. Four consumers, one publish, and all of it replayable when the next model needs history. Agent systems are adopting the same shape: every tool call and decision gets published once, and observability, evals, and audit each consume it independently, none of them slowing down the agent itself.

9. The Interview Lens

How it's asked: the direct form is "Five teams need order data. How do they get it?" The hidden form appears in any microservices design: you draw arrows between services, and the interviewer asks what happens when a sixth service needs the same data. Candidates without this pattern answer by drawing another arrow from the producer, and the interviewer watches the producer collect arrows like a pincushion. There's also a trap form: you're given a genuine workflow (order, payment, shipping) and the interviewer waits to see if you choreograph what should be orchestrated.

The 30-second answer: "Checkout publishes an order.placed fact to a topic, once. Every interested team gets its own subscription with its own backlog, retries, and dead letter queue, so a slow or crashed consumer affects nobody else. Adding a consumer is configuration, not a checkout deploy, and with a log-based broker like Kafka the new consumer can replay history. The costs I manage: the event schema is now a public API, so changes are additive-only and checked against a registry; every subscription needs idempotent processing; and I alert on backlog age so a slow consumer never falls past the retention window. And if the steps are really a workflow that needs completion tracking, I orchestrate that part instead."

Likely follow-ups:

1. "One subscriber processes at half the speed of the others. What happens?" Its own backlog grows; everyone else is untouched. That's the isolation working. The real danger is silent: if it lags past the retention window, it starts losing data. So alert on backlog age, and scale that one consumer by adding workers within its subscription (the queue lesson's competing consumers, applied inside one subscription).
2. "How do you change an event's schema without breaking consumers you don't know about?" Additive changes only: new optional fields, never renames or removals. Enforce it with a schema registry that fails the build on breaking changes. For a truly breaking change, publish version 2 alongside version 1, give consumers a deprecation window, and keep a catalog so "consumers you don't know about" stops being a real category.

Red flag that fails candidates: pointing several different services at one shared queue and calling it broadcast (each service just gets a random sample), or building request-response out of event pairs. Both mean the jobs-versus-facts distinction hasn't landed.

10. Check Yourself

Q1 (recall). Explain the difference between a topic and a subscription. Then state exactly what it costs the producer when a sixth consumer team shows up.

Q2 (trade-off). "Pub/sub decouples services." Argue the opposite for two minutes: what coupling survives, and what new discipline replaces the old deploy-time coordination?

Q3 (scenario). Fraud reports scoring only about 20% of orders. Email reports sending about 30% of receipts. The broker's metrics swear nothing was lost, and checkout's publish count matches order volume exactly. Diagnose the wiring and fix it.

<details> <summary>Answers</summary>

A1. The topic is the single, durable stream of published facts. A subscription is one consumer's independent feed from that stream, tracking its own position, with its own backlog, acknowledgments, and dead letter queue: effectively a private queue filled from the topic. A sixth consumer costs the producer nothing: no code, no deploy, no added latency, no awareness. The costs land on the new consumer (build idempotent processing, watch its backlog) and on the ecosystem (one more party depending on the event schema).

A2. The data coupling survives; it just changes shape. Every consumer now depends on the event schema, so renaming one field breaks systems the producer can't even list, which is worse than a compile error unless it's managed. The replacing discipline: treat the schema as an API (registry, additive-only changes, deprecation windows), keep a catalog of consumers so dependencies stay visible, and give each subscription an operational owner. Pub/sub doesn't delete coupling. It moves coupling out of the deploy calendar and into the contract, which is cheaper only if someone actually governs the contract.

A3. All three services were pointed at the same subscription (or the same shared queue). That made them competing consumers: the broker delivered each event to exactly one of the three, so each service saw a random slice, sized by its worker count and speed. Nothing was lost and nothing errored, which is why it went unnoticed. The fix: one subscription per service, so fraud, email, and fulfillment each receive a full copy, and use competing consumers only inside a single subscription to scale that service up. The wiring rule in one line: broadcast across services, compete within a service.

</details>

11. Composes With

• Message Queue: each subscription effectively is one, and the broadcast-into-queues setup (SNS into SQS) composes the two patterns openly.
• Idempotency: every subscription is at-least-once on its own, so N consumers means N chances to double-apply.
• Event-Driven Architecture: the next lesson; pub/sub is the transport, and that lesson decides what should travel over it.
• Change Data Capture: turns a database's own change log into the event stream, the disciplined descendant of "just read my tables."
• Webhooks: this same shape stretched across company boundaries, over plain HTTP, with the guarantees renegotiated.

12. TL;DR Card