Rebalancing

An image service reads ImageUploaded events and makes a thumbnail for each one. Earlier in the year, the team set max.poll.records=200 to limit memory use, and a thumbnail normally takes about 300 ms, so one batch of 200 finished in about a minute. One morning a photography customer uploads several thousand panoramas, and each panorama takes about 2 seconds. A batch of 200 now takes about 400 seconds.

The consumer's default limit between two polls is 300 seconds. After 300 seconds Kafka decides the consumer has failed and moves its partitions to another member