Processing Files Larger Than Memory

A weather website writes one line to its access log for every request it serves. One day of that log is 40 GB, and the machine that has to summarize it has 4 GB of memory. The lines in this lesson's sample file average 40 bytes, so 40 GB is about 40,000,000,000 / 40 = 1 billion lines.

Reading the whole file into a Python list takes more memory than the file itself. In the measurement below, reading an 8.1 MB file into a list of lines peaked at 25,791 KB, about 3.2 times the file size. At the same ratio, the 40 GB file would need about 128 GB, which is 32 times the memory the machine has.