Webhooks

1. The Incident

Your payment provider confirms successful charges by webhook. For every payment, it sends an HTTP POST to https://api.yourstore.com/webhooks/payments. Your handler does the obvious thing. It receives the event, marks the order paid, reserves inventory, notifies the warehouse, sends the receipt, and then returns 200. All in one request. It takes about 7.5 seconds on a busy day.

The provider gives you 5 seconds to respond. Then it closes the connection.

Here is the sequence that follows. From the provider's side, 40% of deliveries now "fail," because you did not answer in time. So it does what any careful sender does: it retries. But your handler actually finished most of the work before the provider gave up. The retry runs it all again. Customers get two shipments, from deliveries that never really failed.

Meanwhile, your measured failure rate keeps climbing, because the handler is still slow. It climbs until it crosses the provider's health threshold. Then the provider does exactly what its documentation said it would: it disables your endpoint.

Now comes the worst part: silence. Nothing is calling you anymore. There are no errors in your logs, because nothing is happening at all. For three hours, customers pay and their orders sit "pending." You find out from support tickets, not from monitoring.

Notice what is new here. In every earlier lesson, both ends of the arrow belonged to you. A webhook is the pub/sub idea applied across a company boundary. The sender's timeout, retry schedule, and disable policy: you do not get to configure any of it. You can only be ready for it.

2. The Obvious Fixes, and Why They Fail

"Make the handler faster." Optimize the pipeline down to 4 seconds and you are under the limit, today. But one slow warehouse API call, one brief database problem, or one traffic spike puts you over it again. The whole failure sequence (slow, retry, duplicate, disable) is still waiting. You have a structural problem, not a performance problem.

"Ask the provider for a longer timeout, or fewer retries." They will say no, and they are right to. Their short timeout protects their systems from your slowness. The timeout lesson makes the same argument from the caller's side. Their retries are the only reason a brief failure on your side does not lose you money. You are asking them to remove the protections because they are inconvenient.

"Forget webhooks. We'll poll their API instead." Ask "anything new?" every minute. Now you are consuming your API rate limit, running up to a minute behind, and mostly hearing "no." But this bad idea contains a good one, which the pattern will use. Polling gives you completeness, and webhooks give you speed. The mistake is thinking you must choose one.

3. The Pattern

A webhook is an event delivered as an HTTP POST from another organization. The receiving rule: check it is authentic, save it, respond "received" within milliseconds, and do the actual work later, behind your own queue.

Five rules, in the order the request meets them. Together they form a checklist you could implement this week.

1. Verify before you believe. Your webhook URL is public. Anyone on the internet can POST {"type": "payment.succeeded"} to it. So providers sign each delivery. They compute an HMAC over the message and send it along. An HMAC is a hash made with a secret key, here a secret that only you and they share. Your handler recomputes the HMAC and compares. No valid signature, no entry. Providers also include a timestamp, so an attacker cannot replay an old, genuine message. Skip this step and anyone can mark orders as paid.
2. Save it and acknowledge. Nothing else. Write the raw event to storage, put it on your own message queue, and return 200. Total time: about 15 ms. The key idea: 200 means "safely received," never "fully processed." All the work the incident's handler did inline (inventory, warehouse, receipt) now happens behind your queue. It runs at your pace, with your own retry and failure handling. The provider's 5-second limit stops mattering, because nothing slow remains inside their request.
3. Deduplicate on the event ID. Providers deliver at-least-once, and they say so in their docs: when in doubt, they resend. Every event carries a unique ID. Your worker uses it as an idempotency key, so the same event processed twice does the work once. This also fixes the incident's double shipments.
4. Do not trust the order, or the payload's freshness. Deliveries can arrive out of order: a "refund updated" before the "charge succeeded" it refers to. For decisions that depend on state, the robust habit is this. Treat the webhook as a signal that something changed, then fetch the current state from the provider's API before acting. The provider's API is the source of truth, meaning the copy you trust. The payload you received may have been created hours ago and retried since. Stripe's own documentation recommends exactly this.
5. Reconcile, because silence is not success. Some events will simply never arrive. Your endpoint was disabled, their outage outlasted their retry window, or a network path quietly failed. So run a periodic check: "give me everything that changed since my last check," using a stored marker (often called a watermark). Webhooks give you speed. This polling loop gives you the guarantee.

If you are ever the sender (your product delivering webhooks to your customers), the same list becomes your obligations. Sign your payloads, include unique event IDs, retry on a published schedule, keep your delivery timeout tight, and disable endpoints that keep failing. Show all of it in a dashboard with a "redeliver" button. Providers are judged on these.

4. Walkthrough with Numbers

The incident's flow, rebuilt with the five rules.

The handler now takes about 15 ms. About 1 ms goes to verifying the signature. About 10 ms goes to saving the raw event and enqueueing it. Then 200. The provider sees a 100% success rate, at any load, during any deploy. The disable threshold is unreachable, because the handler no longer contains anything that can be slow.

The real work drains from your queue exactly as the queue lesson described. Spikes become backlog, and worker crashes become redeliveries. Events that keep failing go to the dead letter queue.

A duplicate delivery arrives 90 seconds after the original. Your 200 was lost on the way back, so the provider resent. The worker checks the event ID, sees it is already processed, and returns. One shipment. This is the idempotency lesson's double-charge scenario, replayed at a company boundary.

The hourly reconciliation job asks the provider for everything updated since its bookmark. Most hours: nothing. The week the provider has a delivery incident, it quietly finds 217 missed confirmations and feeds them into the same queue. Your on-call engineer sleeps through what used to be a three-hour outage.

One new alarm exists: webhook silence. If an endpoint that normally receives steady traffic hears nothing for 15 minutes, someone gets paged. The incident taught you that the most dangerous failure produces no errors at all.

5. Trade-offs

You gain:

• Near-real-time updates from other companies, without constant polling.
• The provider's retry machinery working for you, now that your handler responds quickly.
• One consistent entry point for external events, feeding queue-and-worker machinery you already know how to run.

You pay:

You inherit their rules. At-least-once delivery, no ordering promises, their retry schedule, their disable policy, their payload versioning. Every provider differs. Every integration is a separate contract to learn.

A public endpoint to defend. Signature checks, timestamp windows, and secret rotation are now part of your security surface. A leaked signing secret means someone can forge "payment succeeded" events.

Debugging across a boundary. "Did they send it, or did we drop it?" requires their dashboard, your raw-event table, and timestamps that can be compared. This is exactly why rule 2 saves the raw event.

The polling loop never goes away. Reconciliation is the completeness guarantee; webhooks only provide speed. You run both, permanently.

6. When Not to Use It

Between your own services. Webhooks are HTTP calls imitating a broker: no durable subscriptions, no replay, hand-written retries. Inside your own systems you control both ends. So use real pub/sub and get real guarantees.

When completeness matters more than speed. Billing reconciliation, financial close, compliance exports: build those on polling and bulk export APIs first. Webhooks are an optional freshness layer on top.

High-frequency streams. Thousands of events per second as individual HTTP POSTs is the wrong design. Providers at that scale offer streaming or bulk interfaces. Use them.

When someone is waiting for the answer. If it is a question with a user waiting, that is request-response. Adding a webhook round trip to an interactive path adds latency and failure modes with no benefit.

And the misuse that started this lesson: treating the webhook request as the moment to do the work. It is only the moment to accept the news. Every problem here (timeouts, duplicates, disables, silent gaps) grows from that one confusion.

7. Webhooks vs. Polling

People frame these as rivals. In production they work together, with different jobs.

Polling's weakness is cost and delay. Webhooks' weakness is more dangerous: their worst failure looks the same as nothing happening. So the production answer is webhooks for speed, polling for truth, running together. The poller's bookmark catches whatever the push path drops.

One boundary note. For continuous streams to browsers or apps you control, held-open connections replace repeated POSTs. That is Server-Sent Events, the next lesson. Webhooks are specifically the server-to-server, company-boundary form of push.

8. Real-World Examples

• Stripe is the reference implementation of everything above. It has signed payloads with timestamp checks, event IDs for dedupe, retries spread over days, and endpoint auto-disable. Its official advice is to fetch current state instead of trusting payload freshness.
• GitHub signs deliveries with X-Hub-Signature-256 and gives you a delivery log with a redeliver button: the sender-side obligations, done well.
• Slack requires a 200 within 3 seconds on event deliveries. Nobody's business logic fits in 3 seconds, and that is the point: the contract forces accept-first, work-later.
• Twilio, Shopify, PayPal: every integration-heavy platform converges on the same contract terms, because the failure modes converge.

For AI engineers: webhooks are how asynchronous AI work announces it is done. Batch inference jobs (Anthropic's and OpenAI's batch APIs) and long fine-tunes report completion by webhook. The receiving checklist applies word for word: verify, save, acknowledge fast, process behind a queue. Agent products also consume webhooks as triggers: inbound email, calendar changes, CRM updates. The save-then-queue design matters twice as much there. The "worker" is an expensive agent run, and a duplicate delivery would mean paying for the same agent run twice.

9. In the Interview

"Integrate with Stripe" is the direct form; "how do you find out when the payment settles?" is the hidden one. Interviewers who have run real integrations probe three things. What does your handler do before returning 200? What do you do with a duplicate? How do you discover the events that never arrived? That third question separates candidates cleanly, because most have never thought about it.

The 30-second answer: "My webhook handler does three things only: verify the signature and timestamp, save the raw event, and enqueue it. It returns 200 in about 15 milliseconds, because 200 means received, not processed. Workers do the real work behind my own queue. They deduplicate on the event ID, since delivery is at-least-once. For state-based decisions they fetch current state from the provider's API instead of trusting a possibly stale payload. Then two protections. An hourly reconciliation poll from a bookmark, because webhooks are best-effort and the worst failure is silence. And an alarm if a normally busy endpoint goes quiet. Speed from the push, truth from the poll."

Likely follow-ups:

"How do you know you missed one?" From the webhook path alone, you cannot: absence sends no signal. That is why the reconciliation poller exists (it eventually catches every gap). It is also why the silence alarm exists (it catches big gaps fast). Saying "silence is the worst failure mode" out loud is the senior marker here.

"Secure the endpoint." Verify the HMAC signature over the raw body with the shared secret. Use a constant-time comparison, meaning one that takes the same time whether the match is close or not, so timing gives nothing away. Check the timestamp to block replays. Have a rotation plan for the secret. Reject before parsing. IP allowlists are a minor extra layer. The signature is the authentication.

A mistake that fails candidates: a handler that does the business work inline before responding, or no answer for duplicates. Either one says the integration works fine until the first day something goes wrong.

10. Check Yourself

Q1 (recall). List the receiver's five rules in the order a request meets them, and state precisely what the 200 response means and does not mean.

Q2 (trade-off). The provider's payload already contains everything your worker needs. This lesson still recommends fetching current state from their API before acting. Defend the extra call, and name the cases where you would skip it.

Q3 (scenario). Payment confirmations have quietly stopped. Logs show no handler errors for three weeks. The provider's dashboard shows your endpoint was auto-disabled 20 days ago, after a failure spike during a deploy. Reconstruct the chain of events, then redesign so this class of failure cannot happen silently again.

A1. Verify: signature and timestamp, reject before believing. Save: the raw event, durably. Acknowledge: 200 in milliseconds. Process later: your own queue, dedupe on event ID, fetch current state when order or freshness matters. Reconcile: poll from a bookmark for whatever never arrived. The 200 means "this event is safely in my possession." It never means "I acted on it." That separation is what keeps your business logic out of the sender's timeout budget.

A2. Across a company boundary, the payload is a snapshot from whenever this delivery attempt was created. It may be hours stale after retries. It may be out of order relative to other events. And if verification ever has a gap, it may be forged. The provider's API is the current truth, so fetch-then-act bases decisions on truth rather than on history. Skip the fetch in three cases. The event is self-contained and append-only (a log line, an analytics tick). Your volume would exceed the API rate limit. Or the action is idempotent and cheap to correct.

A3. The chain: the deploy briefly made the handler slow or erroring. The provider's retries kept hitting the same slowness. The measured failure rate crossed their health threshold, and the endpoint was auto-disabled. Deliveries stopped, producing zero local errors, because absence does not log. The redesign, in layers: the 15 ms save-and-acknowledge handler makes failure spikes nearly impossible, because nothing slow remains in the request. A silence alarm turns any future disablement into a page within minutes. The hourly reconciliation poller bounds the data gap no matter what. And webhook endpoint health becomes a standard item in deploy verification. The principle underneath: never let your completeness depend on someone else's delivery persistence, and never let silence be an unmonitored state.

11. Related Patterns

• Message Queue: the separation between accepting the news and doing the work; the handler's only job is feeding it.
• Idempotency: event-ID dedupe is the cross-company version of idempotency keys; at-least-once is written into every provider's contract.
• Publish-Subscribe: what a webhook is, structurally, once carried over HTTP across a company boundary, with the broker's guarantees renegotiated as contract terms.
• Retry with Exponential Backoff: running on the sender's side, both for you and against you; worth understanding from both sides.

12. TL;DR