Partitioning and File Sizes

A photo-editing app for phones sends 500 million events a day: app opens, photo imports, filters applied, crops, exports, and shares. Suppose an event takes about 80 bytes stored as Parquet, so one day of events is about 40 GB and a year is about 365 x 40 = 14,600 GB. A report on yesterday's exports needs one day. If the table is one undivided set of files, the engine reads all 14,600 GB to find it, 365 times more than the answer needs.

Two layout choices decide how much a query reads: how the files are grouped into folders by a column (partitioning), and how large each file is