Clickstream Pipeline: Processing Late Data and Costs

The incident below is invented for this example. In the first month after the news app's warehouse loaded 90 days of past events, the editors added a dashboard tile that showed how many devices had read an article today. The tile counted distinct device_id values in the events table and refreshed every 5 minutes. The table was not partitioned, so each refresh read those columns for all 90 days of events, and the tile alone cost about 10,337 dollars that month.