Daily Reporting Pipeline: Orchestration Quality and Backfills

When a scheduler marks a run of the cinema report as successful, it has kept one promise: every task ran in the order the workflow defines, and none of them raised an error. It makes no promise that all 120 cinemas sent their till files (the daily files from their cash registers), that the loaded lines add up to what each till recorded, or that the report reached the managers by 08:00. A task that adds up a half-loaded table also finishes without an error.

The pipeline must keep those other promises itself