Grokking Concurrency in Python
Vote

0% completed

​

A Concurrent Web Crawler

When a crawler reaches its deadline, how do you make sure that no task is still running and no connection is still open? Stopping on time without leaving work behind is the hardest requirement of the classic crawler problem. It fits, with the other requirements, in one asyncio program of about 120 lines.

The Problem

Write a crawler that starts from one URL and meets six requirements:

  1. Follow links up to a maximum depth. The depth of a page is the number of links followed from the starting page to reach it, so the starting page has depth 0. 2

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content