Grokking the Data Engineering Interview
0% completed
Monitoring and Recovering Pipelines
The nightly load of a marketplace's orders failed, and nobody has looked at it yet. At 08:00, which tables hold yesterday's data, and which do not? Which dashboards and files were built from the missing data, and which teams read them? Once the cause is fixed, what must run again, and in what order?
Checking a pipeline so that a problem is found before its readers find it is called monitoring, and correcting the data after the cause is fixed is called recovery. The sections below answer these three questions for one failed night.
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%