Measuring Consumer Lag

Consumer lag is how far a consumer group is behind the newest record in a partition. Kafka reports it as a count of records: the partition's log-end offset, which is the offset the next written record will receive, minus the group's committed offset, which is the saved position the group will resume from. A group whose committed offset is 200 on a partition whose log-end offset is 300 has a lag of 100 records.

Consider a card issuer's fraud scoring service. It reads every card transaction from the topic card-transactions, keyed by card number, and scores it for fraud