Grokking Concurrency in Python
0% completed
Rate Limiting
A limit of 16 requests in flight, as in Bounding Concurrency and Backpressure, sounds modest. If each reply takes 50 milliseconds, however, 16 slots can start 16 / 0.05 = 320 requests per second, and more when the site refuses requests at once.
A site that allows 10 requests per second would refuse almost all of that traffic. Requests in flight and requests started per second are two different limits, and a crawler has to respect both.
.....
.....
.....
Like the course? Get enrolled and start learning!
Reading Progress
0%