Backpressure and Consumer Lag

1. What Lag Actually Measures

Consumer lag is the most watched number in a Kafka deployment and the most misread.

Lag is the difference between the log end offset of a partition and the offset your consumer group has committed for it.

Three consequences follow from that definition, and each one surprises somebody.

• Lag is per partition, per group. The number on the dashboard is usually a sum. A sum of zero across nineteen partitions and forty thousand on the twentieth looks like two thousand, which looks fine