MapReduce

1. The Incident

Ten years of business, and finance asks a simple question: "total revenue by product category, all time, by region." The data: 5 billion order rows, about 2 TB, archived in object storage.

The analyst writes the obvious script. Read every row, keep running totals per category. Do the arithmetic. One machine reads at about 100 MB/s. 2 TB / 100 MB/s = 20,000 seconds. That is five and a half hours just to read the data, before any computing. The script starts Monday at 9 AM.