Grokking Concurrency in Python
Vote

0% completed

​

Failure Drills

The final design of Harvest makes three promises about failure:

  • If an index worker process dies, no page is lost and no page is written twice.
  • If one site stops answering in time, the crawl still ends, and the other sites are still crawled.
  • If the process is told to stop, it finishes the pages in flight, saves the rest, and a later run continues from the saved state.

It makes no promise that a slow site leaves the crawl time unchanged, and nothing survives SIGKILL sent to the main process

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content