Order Pipeline: Failures and Operations

At 17:40 on the evening before a public holiday, the card payment provider began answering in 1.6 seconds instead of its usual 200 ms. Nothing returned an error. Every authorization still succeeded, only eight times more slowly. At 17:42 an alert fired: the payment service was more than 10 seconds behind on payment-commands.

The provider recovered at 18:05. By then 45,000 payment commands were waiting, and customers who had ordered at about 17:52 had been looking at "Confirming payment" for over 12 minutes. The backlog cleared about 4 minutes later