Grokking Concurrency in Python
0% completed
Seeing Every Task
Harvest's async crawler has a frontier of ten pages and three fetch workers. Three seconds after it starts, it has fetched nine pages and makes no more progress. It uses no CPU, prints no error, and its event loop is not blocked, so a lag monitor would report nothing.
A thread dump does not help, because every task runs on one thread, which shows only the loop waiting for I/O. You need a list of every task, with what each one waits for and which task waits for which.
A Crawler That Never Finishes
The crawler puts ten page numbers into an asyncio.Queue called the frontier
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%