Saga

1. The Incident

Checkout at the store now spans three services. Inventory reserves the items. Payment charges the card, through the external provider. Shipping creates the shipment. The code calls them in order, one after another. It works, mostly.

Then shipping has a bad deploy: down for twenty minutes on a busy evening. For every checkout in that window, inventory reserved and payment charged. Then the third call failed. The code has a catch block that logs the error. That is all it has.

The damage report: 340 orders in limbo, stuck half done