Webhooks

Five disciplines, in the order the request meets them. Together they're a checklist you could implement Monday morning:

• 1. Verify before you believe. Your webhook URL is public. Anyone on the internet can POST {"type": "payment.succeeded"} to it. So providers sign each delivery: they compute a stamp (an HMAC) over the message using a secret key that only you and they share, and send the stamp along. Your handler recomputes the stamp and compares. No valid stamp, no entry. They also include a timestamp so an attacker can't replay an old, real message. Skip this step and your handler is an open door that marks orders as paid.
• 2. Save it and say "got it." Nothing else. Write the raw event to storage, put it on your own message queue, and return 200. Total time: about 15 ms. The key idea: 200 means "safely received," never "fully processed." All the work the incident's handler did inline (inventory, warehouse, receipt) now happens behind your queue, at your pace, with your own retry and failure handling around it. The provider's 5-second limit stops mattering, because nothing slow lives inside their request anymore.
• 3. Dedupe on the event ID. Providers deliver at-least-once, and they say so in their docs: any doubt, they resend. Every event carries a unique ID; your worker uses it as an idempotency key, so the same event processed twice does the work once. This also retroactively fixes the incident's double shipments.
• 4. Don't trust the order, or the payload's freshness. Deliveries can arrive out of order: a "refund updated" before the "charge succeeded" it refers to. The robust habit for decisions that depend on state: treat the webhook as a hint that something changed, then fetch the current state from the provider's API before acting. The provider's API is the source of truth; the payload in your hand may have been written hours ago and retried since. Stripe's own docs recommend exactly this.
• 5. Reconcile, because silence is not success. Some events will simply never arrive: your endpoint was disabled, their outage outlasted their retry window, a network path quietly failed. So run a periodic check: "give me everything that changed since my bookmark," using a stored marker of where you last checked (often called a watermark). Webhooks give you speed. This polling loop gives you the guarantee.

If you're ever the sender (your product delivering webhooks to your customers), read that list in a mirror: sign your payloads, include unique event IDs, retry on a published schedule, keep your delivery timeout tight, disable endpoints that keep failing, and show all of it in a dashboard with a "redeliver" button. Every receiver discipline above is a sender obligation, and providers get judged on them.

4. Walkthrough with Numbers

The incident's flow, rebuilt with the five disciplines:

• The handler now takes ~15 ms: about 1 ms to verify the signature, about 10 ms to save the raw event and enqueue it, then 200. The provider sees a 100% success rate, at any load, during any deploy. The disable threshold is unreachable, because the handler no longer contains anything that can be slow.
• The real work drains from your queue exactly as the queue lesson taught: spikes become backlog, worker crashes become redeliveries, and repeatedly-failing events park in the dead letter queue.
• A duplicate delivery arrives 90 seconds after the original (your 200 got lost on the way back, so the provider resent). The worker checks the event ID, sees it's already processed, and returns. One shipment. This is the idempotency lesson's double-charge story, replayed at a company boundary.
• The hourly reconciliation job asks the provider for everything updated since its bookmark. Most hours: nothing. The week the provider has a delivery incident, it quietly finds 217 missed confirmations, feeds them into the same queue, and your on-call sleeps through what used to be a three-hour outage.
• One new alarm exists: webhook silence. If an endpoint that normally receives steady traffic hears nothing for 15 minutes, someone gets paged. The incident taught you the most dangerous failure has no errors in it at all.

5. Trade-offs: What You Pay

You gain:

• Near-real-time updates from other companies, without polling everything constantly.
• The provider's retry machinery working for you, now that your handler stops punishing it.
• One consistent front door for external events, feeding queue-and-worker machinery you already know how to run.

You pay:

• You inherit their rules. At-least-once delivery, no ordering promises, their retry schedule, their disable policy, their payload versioning. Every provider differs. Every integration is a small treaty.
• A public door to defend. Signature checks, timestamp windows, and secret rotation are now part of your security surface. A leaked signing secret means someone can forge "payment succeeded" events.
• Debugging across a boundary. "Did they send it, or did we drop it?" needs their dashboard, your raw-event table, and timestamps that line up. This is exactly why discipline 2 saves the raw event.
• The polling loop never goes away. Reconciliation is the completeness guarantee; webhooks only bought speed. You run both, forever.

6. When NOT to Use It

• Between your own services. Webhooks are HTTP calls pretending to be a broker: no durable subscriptions, no replay, hand-rolled retries. Inside your own walls you control both ends, so use real pub/sub and get real guarantees.
• When completeness matters more than speed. Billing reconciliation, financial close, compliance exports: build those poll-first, on bulk export APIs, with webhooks as an optional freshness layer at most.
• High-frequency streams. Thousands of events per second as individual HTTP POSTs is the wrong shape. Providers at that scale offer streaming or bulk interfaces; take them.
• When someone is waiting for the answer. If it's a question with a user on hold, that's request-response. Adding a webhook round-trip to an interactive path buys latency and failure modes for nothing.

And the misuse that started this lesson: treating the webhook request as an invitation to do the work, instead of to accept the news. Every pathology here (timeouts, duplicates, disables, silent gaps) grows from that one confusion.

7. Pattern Face-Off: Webhooks vs Polling

People frame these as rivals. In production they're teammates with different jobs.

Polling's weakness is cost and delay. Webhooks' weakness is sneakier: their worst failure is indistinguishable from nothing happening. So the production answer is webhooks for speed, polling for truth, running together, with the poller's bookmark catching whatever the push path drops. One boundary note: for continuous streams to browsers or apps you control, held-open connections (Server-Sent Events, next lesson) replace repeated POSTs. Webhooks are specifically the server-to-server, company-boundary member of the push family.

8. In the Wild

• Stripe is the reference implementation of everything above: signed payloads with timestamp checks, event IDs for dedupe, retries spread over days, endpoint auto-disable, and official advice to fetch current state instead of trusting payload freshness.
• GitHub signs deliveries with X-Hub-Signature-256 and gives you a delivery log with a redeliver button: the sender-side obligations, done well.
• Slack requires a 200 within 3 seconds on event deliveries. Nobody's business logic fits in 3 seconds, and that's the point: the contract forces accept-don't-work.
• Twilio, Shopify, PayPal: every integration-heavy platform converges on the same treaty terms, because the failure modes converge.

For AI engineers: webhooks are how asynchronous AI work announces it's done. Batch inference jobs (Anthropic's and OpenAI's batch APIs) and long fine-tunes report completion by webhook, and the receiving checklist applies word for word: verify, save, ack fast, process behind a queue. Agent products also consume webhooks as triggers: inbound email, calendar changes, CRM updates. The save-then-queue front door matters double there, because the "worker" is an expensive agent run, and a duplicate delivery would mean paying for the same agent run twice.

9. The Interview Lens

How it's asked: "Integrate with Stripe" is the direct form; "how do you find out when the payment settles?" is the hidden one. Interviewers who have run real integrations probe three things: what your handler does before returning 200, what you do with a duplicate, and how you discover the events that never arrived. That third question separates candidates cleanly, because most have never thought about it.

The 30-second answer: "My webhook handler does three things only: verify the signature and timestamp, save the raw event, and enqueue it, returning 200 in about 15 milliseconds, because 200 means received, not processed. Workers do the real work behind my own queue, deduping on the event ID since delivery is at-least-once, and for state-based decisions they fetch current state from the provider's API instead of trusting a possibly stale payload. Then two safety nets: an hourly reconciliation poll from a bookmark, because webhooks are best-effort and the worst failure is silence, and an alarm if a normally busy endpoint goes quiet. Speed from the push, truth from the poll."

Likely follow-ups:

1. "How do you know you missed one?" From the webhook path alone, you can't: absence sends no signal. That's why the reconciliation poller exists (it eventually catches every gap) and why the silence alarm exists (it catches big gaps fast). Saying "silence is the worst failure mode" out loud is the senior marker here.
2. "Secure the endpoint." Verify the HMAC signature over the raw body with the shared secret, using a constant-time comparison; check the timestamp to block replays; have a rotation plan for the secret; reject before parsing. IP allowlists are a garnish, not the meal. The signature is the authentication.

Red flag that fails candidates: a handler that does the business work inline before responding, or no answer for duplicates. Either one says the integration works fine until its first bad day.

10. Check Yourself

Q1 (recall). List the receiver's five disciplines in the order a request meets them, and state precisely what the 200 response means and does not mean.

Q2 (trade-off). The provider's payload already contains everything your worker needs. This lesson still recommends fetching current state from their API before acting. Defend the extra call, and name the cases where you'd skip it.

Q3 (scenario). Payment confirmations have quietly stopped. Logs show no handler errors for three weeks. The provider's dashboard shows your endpoint was auto-disabled 20 days ago, after a failure spike during a deploy. Reconstruct the chain of events, then redesign so this class of failure can't happen silently again.

<details> <summary>Answers</summary>

A1. Verify (signature and timestamp, reject before believing), save (the raw event, durably), acknowledge (200 in milliseconds), process later (your own queue, dedupe on event ID, fetch current state when order or freshness matters), reconcile (poll from a bookmark for whatever never arrived). The 200 means "this event is safely in my possession." It never means "I acted on it," and that separation is what keeps your business logic out of the sender's timeout budget.

A2. Across a company boundary, the payload is a snapshot from whenever this delivery attempt was made: possibly hours stale after retries, possibly out of order relative to other events, and, if verification ever has a gap, possibly forged. The provider's API is the current truth, so fetch-then-act bases decisions on truth rather than on history. Skip the fetch when the event is self-contained and append-only (a log line, an analytics tick), when your volume would blow the API rate limit, or when the action is idempotent and trivially cheap to correct.

A3. The chain: the deploy briefly made the handler slow or erroring; the provider's retries kept hitting the same slowness; the measured failure rate crossed their health threshold; the endpoint was auto-disabled; deliveries stopped, producing zero local errors, because absence doesn't log. The redesign, in layers: the 15 ms save-and-ack handler makes failure spikes nearly impossible (nothing slow remains in the request); a silence alarm turns any future disablement into a page within minutes; the hourly reconciliation poller bounds the data gap no matter what; and webhook endpoint health becomes a standard item in deploy verification. The principle underneath: never let your completeness depend on someone else's delivery persistence, and never let silence be an unmonitored state.

</details>

11. Composes With

• Message Queue: the wall between accepting the news and doing the work; the handler's only job is feeding it.
• Idempotency: event-ID dedupe is the boundary-crossing version of idempotency keys; at-least-once is written into every provider's contract.
• Publish-Subscribe: what a webhook is, shape-wise, once stretched over HTTP and a company boundary, with the broker's guarantees renegotiated as treaty terms.
• Retry with Exponential Backoff: running on the sender's side, both for you and against you; worth understanding from both chairs.

12. TL;DR Card