Event-Driven Architecture

1. The Incident

Your pub/sub migration was a success. Checkout publishes order.placed once, five teams subscribe, and checkout deploys are calm again. The events themselves are small: {event: "order.placed", order_id: "ord_7f3a"}. Just a notification. Any consumer that needs details can ask.

And they do ask. Fraud receives the event, then calls GET /orders/ord_7f3a on checkout's API to get the amount and items. Email calls the same endpoint to build the receipt. Analytics calls it. Loyalty calls it. Fulfillment calls it twice (a bug, but still). Count it up: every event you publish comes back to you as five API calls.

Now the next flash sale arrives. 5,000 orders per second means 5,000 events per second, which means 25,000 GET requests per second hitting checkout's API, in exactly the same bursts the queue was built to absorb. Checkout's read path fails. The consumers' calls start timing out, their retries add even more traffic, backlogs grow, and the fraud team ends up scoring orders minutes late during the highest-fraud hour of the year.

Notice what happened. You went asynchronous, and the load came back anyway. The events successfully announced that something happened while carrying nothing about what happened. So every consumer still has to call checkout to ask. That is not an event-driven architecture. It is a synchronous architecture with extra steps.

2. The Obvious Fixes, and Why They Fail

"Cache the lookups." Put a cache in front of checkout's API. It helps a little, but consider the timing: all five consumers ask about an order the moment it is created. The order was not in the cache one second ago, because it did not exist one second ago. So all five miss the cache together and hit the API anyway. And every consumer still depends on checkout's API being up just to do its own job.

"Slow the consumers down." Rate-limit the callbacks so checkout survives. Now every downstream team processes at whatever speed checkout permits, and backlogs grow during exactly the bursts that matter most. Fraud detection now runs at checkout's convenience. The dependency did not go away. It only gained a queue in front of it.

"Could we just put the details in the event?" Yes. That is the pattern, and the fact that it feels almost too easy is why it needs a lesson. The concern that usually follows ("isn't that copying data everywhere?") is real, is the trade-off, and appears in the self-check. But notice what the first two fixes had in common: they tried to make it cheaper for consumers to ask checkout. The better approach is to make asking unnecessary.

3. The Pattern

Event-driven architecture means services communicate mainly through events. Its central design decision is simple to state: how much information does the event carry? Enough that consumers can act on their own, without calling back?

The previous two lessons built the delivery mechanism. This lesson decides what goes inside the events. There are three levels, from lightest to heaviest:

  Level 1: Event Notification         "Order ord_7f3a was placed."
                                      Consumers must call back for details.
        |
  Level 2: Event-Carried State        "Order ord_7f3a: 3 items, $142,
  Transfer                             customer c_91, ships to Austin."
                                      Consumers act without calling anyone.
        |
  Level 3: Event Sourcing             The events ARE the data.
                                      Current state is rebuilt by replaying them.

1. Level 1, notification. The event says "something happened, ask me for details." Small events, no copied data, and the callback storm from the incident.
2. Level 2, event-carried state transfer. The event carries the facts themselves. Each consumer then keeps a local view: its own small table of the data it cares about, updated from events. Fraud keeps its own small orders table with just the fields fraud needs. When fraud wants an order's amount, it reads its own table, in under a millisecond, even if checkout is down. Consumers stop being callers and become independent owners of their own copy.
3. Level 3, event sourcing. The event log becomes the system of record, and every table is rebuilt from it. That is a storage-level commitment with its own lesson. For now, know that the level exists, and that you do not need it to fix a callback storm. Level 2 does that.

Most day-to-day event-driven engineering is Level 2, plus three rules:

1. Events are facts, never commands. Publish "an order was placed," never "send a receipt." The moment your event tells a specific consumer what to do, you have re-coupled yourself to that consumer's job, which is exactly what the topic exists to prevent.
2. Local views are copies, never the truth. Each view can be deleted and rebuilt at any time by replaying the topic's history. One service (here, checkout) remains the official owner of the data. Everyone else holds a convenient, disposable copy.
3. Freshness becomes a "when," not an "if." Every local view runs slightly behind the source: usually milliseconds, sometimes minutes during a backlog. This is called eventual consistency: all the copies will agree, eventually. Designing with that fact in mind, instead of being surprised by it, is what separates people who use pub/sub from people who design event-driven systems.

4. Walkthrough with Numbers

The flash sale again, with full-state events:

• The event grows from about 100 bytes to about 2 KB. At 5,000 events per second, that is 10 MB per second through the broker, which is a very small load for any real broker. In exchange, the 25,000 GETs per second disappear. You are trading kilobytes of bandwidth for removed coupling, and it is the cheapest trade in this module.
• Checkout's read traffic during the sale: flat. Zero callbacks. Its API now serves actual users, not internal lookups.
• Fraud reads locally in about 1 ms instead of making a 50 ms cross-service call wrapped in timeout and retry logic. Better: during checkout's worst hour, fraud does not degrade at all. Its speed and uptime no longer depend on another team's service.
• A new consumer replays history. The topic retains 30 days, roughly 500 million events. A new team builds its local view by replaying them in a few hours, then switches to live traffic. Without this, that team would be writing a backfill script against checkout's API and apologizing for the load.

5. Trade-offs

You gain:

• Independence: consumers read locally, stay up when the producer is down, and never overload anyone with callbacks.
• Producer load that does not depend on how many consumers exist or how much data they need.
• Views that can be rebuilt from history, which makes recovery and onboarding the same operation: replay.

You pay:

• Copies everywhere. Order data now lives in six services' local stores, shaped six ways. Storage is cheap; the discipline is remembering that copies are disposable and only the owner's store is the truth.
• Eventual consistency, always. Every view lags a little. Your product has to handle a support agent looking at a view that is 40 seconds behind.
• Personal data spreads. A full-state event carrying the customer's email copies it into every subscriber's store and into the topic's retained history. When a deletion request arrives, there are N+1 places to clean. The self-check makes you actually do this.
• The schema matters even more. A larger event has more fields, and every field is now part of the public contract from the pub/sub lesson: registry, additive-only changes, catalog. What was recommended there becomes mandatory here.
• Harder debugging. "What is the current state of order X?" now has one answer per service, each with a timestamp. Correlation IDs and distributed tracing stop being optional.

6. When Not to Use It

1. Inside one service. A monolith calling its own functions has perfect consistency and transactions at no cost. Event-driven architecture solves problems between teams and services. Applied inside a boundary that has neither problem, it is unnecessary complexity.
2. When the read must be exactly current. Checking inventory before selling the last unit, or checking a balance before a withdrawal: these reads must see the very latest write, so they go synchronously to the owning service. Route those deliberately, and let local views serve everything else.
3. Request-response, rebuilt from events. A full-state event does not fix a workflow that is secretly a synchronous call. The pub/sub lesson's warning still applies.
4. Before the team is ready. Event-driven systems without schema rules, tracing, and backlog alerts are very hard to debug. If those investments are not realistic yet, fewer services making simple synchronous calls is the stronger engineering choice.

The classic misuse at this level: using events to avoid deciding who owns the data. Six services each hold a view, nobody is the official owner, and the copies slowly disagree with no way to settle which is right. Every fact needs exactly one owner whose store is the truth. Events distribute the truth. They do not replace having one.

7. Notification Events vs. Full-State Events

The two practical levels, side by side:

There is a practical middle option: an event carrying the commonly needed fields plus an ID for the rare detailed lookup. Most mature systems settle there, deciding field by field with two questions: who needs this field, and what does it cost to have copies of it everywhere? Frequently used fields go in the event. Sensitive fields stay out.

8. Real-World Examples

• LinkedIn built Kafka for exactly this: activity events as one large stream that every team reads to maintain its own view. The "central nervous system" nickname for Kafka started there.
• Uber: a trip is a stream of events long before it is a database row, consumed independently by pricing, ETA prediction, fraud, and driver payments.
• Bank ledgers are the pattern's ancestor: transactions are immutable facts, and your balance is a view computed from them. Event-driven design predates software.
• Netflix runs viewing events through the same kind of central stream into recommendations, artwork personalization, and capacity planning.

For AI engineers: full-state events are what keep features fresh. The feature store's online features update from the interaction stream without calling any service's API. The vector index re-embeds content when a content.updated event arrives carrying the content itself. Training pipelines replay retained history as their dataset, using the same replay that onboards a new microservice. When someone asks how the recommender knows about a purchase 200 ms after it happened without ever calling checkout, this lesson is the answer.

9. In the Interview

The direct question is "How does service B know about service A's data without calling it?" The deeper one appears in follow-ups: you draw a topic in your design, and the interviewer asks, "What is in the event?" Then they watch whether you understand that this one decision controls the callback load, the independence of every consumer, and how much personal data spreads through the whole system.

The 30-second answer: "I treat the event's contents as the main design decision. Thin notification events force every consumer to call the producer back, so the coupling and the load return. For widely consumed facts I put the state in the event, and each consumer maintains its own local view: updated from the stream, rebuildable by replay, readable in microseconds, and available even when the producer is down. The costs I manage: every view is eventually consistent, so reads that must be exactly current go synchronously to the owning service; sensitive fields stay out of the event or get encrypted; and the larger schema is a public API under registry control. Ownership stays with one service: views are copies, never the truth."

Likely follow-ups:

1. "A support agent updates an order and doesn't see the change on their own dashboard. Why, and what do you do?" The dashboard reads a local view that lags by its backlog: the agent wrote to the source and read from a stale copy. Options, from cheapest up: show the staleness honestly ("as of 12:04:31"); route the writer's next read to the source (called read-your-writes); or wait until the view catches up to the write's version. Choosing per screen, rather than one policy everywhere, is the senior move.
2. "A user invokes their right to be deleted. Their data sits in full-state events across a 30-day retained topic and six local views. Go." Delete from the owning store first. Publish a tombstone: a special event meaning "purge this user," which every view consumer must honor. For the retained history, either let retention age it out (if the legal deadline allows) or use crypto-shredding: encrypt each user's fields with a per-user key, and when deletion is requested, destroy the key. Every copy everywhere becomes unreadable at once, including replays and backups.

A mistake that fails candidates: publishing notification events and calling the system "decoupled" while every consumer synchronously fetches from the producer. Or the opposite: full-state events with no answer for staleness, deletion, or which copy is the truth. Both show the same gap: not seeing that the event's contents are the architecture.

10. Check Yourself

Q1 (recall). Name the three levels of event design, and for each, say in one line where the truth lives.

Q2 (trade-off). The fraud team asks you to add customer.lifetime_value to the full order.placed event, "since it's already flowing." Argue both sides in four sentences, then decide.

Q3 (scenario). Eight months after moving to full-state events, legal forwards a deletion request. The customer's email address exists in: checkout's database, the 30-day retained order.placed topic, and the local views of six consumer services, two of which have no listed owner. Design the deletion, and name the process failure that made it hard.

A1. Notification: the event announces an ID; the truth lives entirely with the producer, and consumers fetch it. Event-carried state transfer: the event carries the facts; the truth still lives with the producer, but consumers hold disposable local copies. Event sourcing: the event log itself is the truth, and every table anywhere is rebuilt from it.

A2. For: fraud genuinely needs it; adding a field is an additive, non-breaking change; and the alternative is fraud calling a customer API on every order, which recreates the callback problem. Against: lifetime value is not a fact about this order. It is computed analytics about the customer, so embedding it clutters the order event, freezes a number that goes out of date quickly into a permanent record, and broadcasts a sensitive business signal to every subscriber, not just fraud. Decision: keep order.placed clean; publish lifetime value on its own stream (or let fraud read it from the feature store), and let fraud combine the two in its own view. The rule worth extracting: an event carries facts about its own subject; derived data about other things gets its own stream.

A3. The mechanics: delete from checkout, the owning store. Publish a tombstone event that every view consumer must treat as "purge this user," and verify they do. For the retained topic, check whether the 30-day window satisfies the legal deadline; if not, crypto-shredding (per-user encryption keys for personal fields, destroy the key on request) makes every copy unreadable at once, including replays and backups. Now the real finding: the two views with no listed owner are the actual failure, and it is a process failure, not a technical one. Subscribing to a topic that carries personal data should have required a named owner and a data-handling sign-off, recorded in a catalog. You can only design a deletion when you can list the copies. And the lasting fix is at the source: the email address probably never belonged in the event at all. Sensitive fields stay out.

11. Related Patterns

• Publish-Subscribe: the transport this pattern uses; this lesson decides what travels over it.
• Event Sourcing: the third level, where the log stops feeding the truth and becomes it.
• Change Data Capture: produces full-state events directly from a database's change log, when the producer cannot or will not publish.
• Saga: the pattern for multi-step workflows, which events should not be used to imitate.
• Idempotency: view updates arrive at-least-once; applying the same event twice must leave the view correct.

12. TL;DR