Grokking Concurrency in Python
0% completed
Bounding Concurrency and Backpressure
How many requests should a crawler send to one site at the same time? Asyncio sets no limit of its own: a program can start 200 tasks, or 200,000, and each task sends its request as soon as it runs.
The answer therefore has to come from the program. The site serves only a limited number of requests at once, and the program's memory has to hold every result that a slower step has not yet handled.
One Task per Page
A first version of a crawler often starts one task per URL
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%