Grokking Concurrency in Python
Vote

0% completed

​

Rate Limiting

A limit of 16 requests in flight, as in Bounding Concurrency and Backpressure, sounds modest. If each reply takes 50 milliseconds, however, 16 slots can start 16 / 0.05 = 320 requests per second, and more when the site refuses requests at once.

A site that allows 10 requests per second would refuse almost all of that traffic. Requests in flight and requests started per second are two different limits, and a crawler has to respect both.

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content