System Design Patterns: From Fundamentals to Real Systems

0% completed

​

MapReduce

1. The Incident

Ten years of business, and finance asks a simple question: "total revenue by product category, all time, by region." The data: 5 billion order rows, about 2 TB, archived in object storage.

The analyst writes the obvious script: read every row, keep running totals per category. Do the arithmetic: one machine reading at ~100 MB/s takes 2 TB ÷ 100 MB/s = 20,000 seconds: five and a half hours just to read the data, before any computing. The script starts Monday at 9 AM.

At 12:40 PM: three hours in: the machine's spot instance is reclaimed

.....

.....

.....

Like the course? Get enrolled and start learning!