Grokking Concurrency in Python
0% completed
Streaming Pipelines
pages = [t.result() for t in tasks] all_counts = [index(page) for page in pages]
These two lines from Harvest's batch crawl collect every page in a list and then index every page into a second list. For 1,000 pages, the crawl merged its first result after 6.40 seconds and needed 16.5 MB of memory at its peak. For 250 pages, it needed 4.2 MB.
At the same rate, a crawl of a million pages would need about 16.5 GB
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%