Event-Driven Architecture

The last two lessons built the delivery trucks. This lesson decides what goes in the boxes. There are three levels, from lightest to heaviest:

  Level 1: Event Notification        "Order ord_7f3a was placed."
  (a sticky note)                    Consumers must call back for details.
        |
  Level 2: Event-Carried State       "Order ord_7f3a: 3 items, $142,
  Transfer (a detailed memo)          customer c_91, ships to Austin."
                                      Consumers act without calling anyone.
        |
  Level 3: Event Sourcing            The events ARE the data.
  (the log is the truth)             Current state is rebuilt by replaying them.

• Level 1, notification, is the sticky note: "something happened, come ask me about it." Tiny events, no copied data, and the callback storm from the incident.
• Level 2, event-carried state transfer, is the detailed memo from the module intro: the event carries the facts themselves. Each consumer then keeps a local view: its own small table of the data it cares about, updated from events. Fraud keeps its own slim orders table with just the fields fraud needs. When fraud wants an order's amount, it reads its own table, in under a millisecond, even if checkout is down. Consumers stop being callers and become independent keepers of their own copy.
• Level 3, event sourcing, goes all the way: the event log becomes the system of record, and every table is rebuilt from it. That's a storage-level commitment with its own lesson; for now just know the level exists, and that you don't need it to fix a callback storm. Level 2 does that.

Most day-to-day event-driven engineering is Level 2, plus three habits:

• Events are facts, never commands. Publish "an order was placed," never "send a receipt." The moment your event tells a specific consumer what to do, you've re-coupled yourself to that consumer's job, which is the exact thing the topic exists to prevent.
• Local views are copies, never truth. Each view can be deleted and rebuilt anytime by replaying the topic's history. One service (here, checkout) remains the official owner of the data. Everyone else holds a convenient, disposable copy.
• Freshness becomes a "when," not an "if." Every local view runs slightly behind the source: usually milliseconds, sometimes minutes during a backlog. This is called eventual consistency: all the copies will agree, eventually. Designing with that fact in mind, instead of being ambushed by it, is what separates people who use pub/sub from people who design event-driven systems.

4. Walkthrough with Numbers

The flash sale again, with detailed-memo events:

• The event grows from about 100 bytes to about 2 KB. At 5,000 events per second, that's 10 MB per second through the broker, which is nothing for any real broker. In exchange, the 25,000 GETs per second disappear. You're trading kilobytes for coupling. It's the cheapest trade in this module.
• Checkout's read traffic during the sale: flat. Zero callbacks. Its API now serves actual users, not the company's internal data hunger.
• Fraud reads locally in about 1 ms instead of making a 50 ms cross-service call wrapped in timeout-and-retry machinery. Better yet: during checkout's worst hour, fraud doesn't degrade at all. Your speed and uptime stop depending on someone else's.
• A new consumer replays history. The topic retains 30 days, roughly 500 million events. A new team builds its local view by replaying them in a few hours, then switches to live traffic. Without this, that team would be writing a backfill script against checkout's API and apologizing for the load.

5. Trade-offs: What You Pay

You gain:

• Independence: consumers read locally, stay up when the producer is down, and never storm anyone with callbacks.
• Producer load that ignores both how many consumers exist and how curious they are.
• Views that can be rebuilt from history, which makes recovery and onboarding the same operation: replay.

You pay:

• Copies everywhere. Order data now lives in six services' local stores, shaped six ways. Storage is cheap; the discipline is remembering that copies are disposable and only the owner's store is the truth.
• Eventual consistency, always. Every view lags a little. Your product has to survive a support agent looking at a view that's 40 seconds behind (a staleness question the graceful degradation lesson also wrestles with; here it's baked into the architecture).
• Personal data spreads. A detailed memo carrying the customer's email copies it into every subscriber's store and into the topic's retained history. When a deletion request arrives, there are N+1 places to clean. Q3 makes you actually do this.
• The schema carries even more weight. A fat event has more fields, and every field is now part of the public promise from the pub/sub lesson: registry, additive-only, catalog. What was recommended there becomes mandatory here.
• Harder debugging. "What's the current state of order X?" now has one answer per service, each with a timestamp. Correlation IDs and distributed tracing stop being optional.

6. When NOT to Use It

• Inside one service. A monolith calling its own functions has perfect consistency and free transactions. Event-driven architecture solves problems between teams and services. Applied inside a boundary that has neither, it's ceremony.
• When the read must be exactly current. Checking inventory before selling the last unit, checking balance before a withdrawal: these reads must see the very latest write, so they go synchronously to the owning service. Route those deliberately, and let local views serve everything else.
• Request-response in disguise. A detailed memo doesn't fix a workflow that's secretly a phone call. The pub/sub lesson's warning stands.
• Before the team is ready. Event-driven systems without schema rules, tracing, and backlog alerts aren't architecture; they're a distributed mystery. If those investments aren't realistic yet, fewer services making boring synchronous calls is the stronger engineering choice.

The classic misuse at this level: using events to avoid deciding who owns the data. Six services each hold a view, nobody is the official owner, and the copies slowly drift apart with no way to settle arguments. Every fact needs exactly one owner whose store is the truth. Events distribute the truth. They don't replace having one.

7. Pattern Face-Off: Sticky Note vs Detailed Memo

The two practical levels, head to head:

There's a practical middle: a medium memo carrying the commonly needed fields plus an ID for the rare deep lookup. Most mature systems settle there, deciding field by field with two questions: who needs this field? and what does it cost to have copies of it everywhere? Hot fields ride in the event. Sensitive fields stay home.

8. In the Wild

• LinkedIn built Kafka for exactly this: activity events as one big stream that every team taps to maintain its own view. The "central nervous system" nickname started there.
• Uber: a trip is a stream of events long before it's a database row, consumed independently by pricing, ETA prediction, fraud, and driver payments.
• Bank ledgers are the pattern's ancestor: transactions are immutable facts, and your balance is just a view computed from them. Event-driven design predates software.
• Netflix runs viewing events through the same kind of backbone into recommendations, artwork personalization, and capacity planning.

For AI engineers: detailed-memo events are the quiet backbone of feature freshness. The feature store's online features update from the interaction stream without calling any service's API. The vector index re-embeds content when a content.updated event arrives carrying the content itself. Training pipelines replay retained history as their dataset, using the very same replay that onboards a new microservice. When someone asks how the recommender knows about a purchase 200 ms after it happened without ever calling checkout, this lesson is the answer.

9. The Interview Lens

How it's asked: the direct probe is "How does service B know about service A's data without calling it?" The deeper one hides in follow-ups: you draw a topic in your design, and the interviewer asks, "What's in the event?" Then they watch whether you understand that this one decision controls the callback load, the independence of every consumer, and the privacy footprint of the whole system.

The 30-second answer: "I treat the event's contents as the main design decision. Thin notification events force every consumer to call the producer back, so the coupling and the load return through the back door. For widely consumed facts I put the state in the event, and each consumer maintains its own local view: updated from the stream, rebuildable by replay, readable in microseconds, and alive even when the producer is down. The costs I manage: every view is eventually consistent, so reads that must be exactly current go synchronously to the owning service; sensitive fields stay out of the event or get encrypted; and the fat schema is a public API under registry control. And ownership stays singular: views are copies, never the truth."

Likely follow-ups:

1. "A support agent updates an order and doesn't see the change on their own dashboard. Why, and what do you do?" The dashboard reads a local view that lags by its backlog: the agent wrote to the source and read from a stale copy. Options, from cheapest up: show the staleness honestly ("as of 12:04:31"); route the writer's next read to the source (called read-your-writes); or wait until the view catches up to the write's version. Choosing per screen, rather than one policy everywhere, is the senior move.
2. "A user invokes their right to be deleted. Their data sits in fat events across a 30-day retained topic and six local views. Go." Delete from the owning store first. Publish a tombstone: a special event meaning "purge this user," which every view consumer must honor. For the retained history, either let retention age it out (if the legal clock allows) or use crypto-shredding: encrypt each user's fields with a per-user key, and when deletion is requested, destroy the key. Every copy everywhere becomes unreadable at once, including replays and backups.

Red flag that fails candidates: publishing sticky notes and calling the system "decoupled" while every consumer synchronously fetches from the producer. Or the mirror image: fat events with no answer for staleness, deletion, or which copy is the truth. Both are the same gap: not seeing that the event's contents are the architecture.

10. Check Yourself

Q1 (recall). Name the three levels of event design, and for each, say in one line where the truth lives.

Q2 (trade-off). The fraud team asks you to add customer.lifetime_value to the fat order.placed event, "since it's already flowing." Argue both sides in four sentences, then decide.

Q3 (scenario). Eight months after going fat-event, legal forwards a deletion request. The customer's email address exists in: checkout's database, the 30-day retained order.placed topic, and the local views of six consumer services, two of which have no listed owner. Design the deletion, and name the process failure that made it hard.

<details> <summary>Answers</summary>

A1. Notification: the event announces an ID; the truth lives entirely with the producer, and consumers fetch it. Event-carried state transfer: the event carries the facts; the truth still lives with the producer, but consumers hold disposable local copies. Event sourcing: the event log itself is the truth, and every table anywhere is rebuilt from it.

A2. For: fraud genuinely needs it; adding a field is an additive, non-breaking change; and the alternative is fraud calling a customer API on every order, which recreates the callback problem. Against: lifetime value isn't a fact about this order: it's computed analytics about the customer, so embedding it turns the order event into a junk drawer, freezes a fast-staling number into a permanent record, and broadcasts a sensitive business signal to every subscriber, not just fraud. Decision: keep order.placed clean; publish lifetime value on its own stream (or let fraud read it from the feature store), and let fraud combine the two in its own view. The rule worth extracting: an event carries facts about its own subject; derived data about other things gets its own stream.

A3. The mechanics: delete from checkout, the owning store. Publish a tombstone event that every view consumer must treat as "purge this user," and verify they do. For the retained topic, check whether the 30-day window satisfies the legal deadline; if not, crypto-shredding (per-user encryption keys for personal fields, destroy the key on request) unreads every copy at once, including replays and backups. Now the real finding: the two views with no listed owner are the actual failure, and it's a process failure, not a technical one. Subscribing to a topic that carries personal data should have required a named owner and a data-handling sign-off, recorded in a catalog. You can only design a deletion when you can list the copies. And the durable fix is upstream: the email address probably never belonged in the fat event at all. Sensitive fields stay home.

</details>

11. Composes With

• Publish-Subscribe: the transport this doctrine rides on; this lesson decides what travels over it.
• Event Sourcing: the third level, where the log stops feeding the truth and becomes it.
• Change Data Capture: manufactures detailed-memo events straight from a database's change log, when the producer can't or won't publish.
• Saga: owns the multi-step workflows that events must not pretend to be.
• Idempotency: view updates arrive at-least-once; applying the same event twice must leave the view correct.

12. TL;DR Card