Idempotent Pipelines

The following incident is invented for this lesson, but every step in it is ordinary. A power utility runs a nightly job that totals each day's meter charges into a table called daily_revenue, and the finance report reads that table. On day 12 the job inserted its rows and then failed on a later step. The scheduler ran the whole job again, and the job inserted day 12 a second time. The next morning the report showed 4,740 cents of revenue for day 12 instead of 2,370.

No statement failed, and each row was correct on its own. The job was simply not safe to run twice