Retry with Exponential Backoff

1. The Incident

Your checkout service calls a payment provider over the network. The provider is healthy. Your code is correct. And yet 1 out of every 200 calls fails: a dropped packet here, a connection reset there, one unlucky server behind their load balancer. Nothing is "down." The network is just being the network.

Now scale it. At 100,000 orders a day, 1 in 200 means 500 customers a day who typed in their card number, clicked Pay, and got an error screen, for a failure that would have succeeded if the very same request had been sent again 100 milliseconds later