Distributed Tracing

1. The Incident

The ticket reads: "checkout is slow. Sometimes." The p99 (the time the slowest 1% of requests exceed) says 4.2 seconds. The flow budgets 800 ms. But it is only some requests, at some times, with no pattern anyone can name.

A checkout touches twelve services across forty-plus calls: gateway, BFF (the backend-for-frontend layer that serves the mobile app), checkout, inventory, payment, fraud, pricing, and more. So the investigation begins the only way it can. Twelve engineers' worth of log files, searched in parallel.