Grokking Concurrency in Python
0% completed
Build a Thread Pool
How many threads should a crawler start when 100,000 pages are waiting to be fetched? The simplest threaded version of Harvest calls threading.Thread(target=fetch, args=(n,)).start() once per page. With a frontier (the list of pages still waiting to be fetched) of 100,000 pages, that loop tries to start 100,000 threads, each with its own connection to one site.
The loop has a second problem even with 16 pages. A thread discards the value its function returns, so the caller cannot collect the downloaded text or learn that a download failed.
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%