Introduction

Surviving Failure: How to Read This Module

Every arrow you draw between two boxes in a system diagram is a promise that will sometimes be broken. A call across a network can hang, fail, half-succeed, or succeed so slowly it's worse than failing. None of that is unusual. It's the normal operating environment.

The mistake is thinking failures are rare, because at scale, rare means constantly. Run the numbers: at 100,000 orders a day, a failure rate of "only" 1 in 200 is 500 failures every day. You don't hope your way past that