Grokking the Data Engineering Interview
Vote

0% completed

​

Monitoring and Recovering Pipelines

The nightly load of a marketplace's orders failed, and nobody has looked at it yet. At 08:00, which tables hold yesterday's data, and which do not? Which dashboards and files were built from the missing data, and which teams read them? Once the cause is fixed, what must run again, and in what order?

Checking a pipeline so that a problem is found before its readers find it is called monitoring, and correcting the data after the cause is fixed is called recovery. The sections below answer these three questions for one failed night.

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content