Rate Limiting with a Token Bucket

A service accepts 10 requests per second from each client and rejects every request above that with an error. A batch job has 500 invoices to send through that service. If the job starts one goroutine per invoice, about 500 requests arrive in the first few milliseconds. Ten of them succeed, and the other 490 fail.

Sending the invoices one after another avoids the errors, but it ties the job's speed to the speed of each call. With 100 ms calls the loop sends 10 requests per second. If the service slows to 400 ms per call, the loop sends only 2.5 per second while the allowance is still 10