What is a Consumer Group and Rebalancing?

A consumer group in Apache Kafka is a set of consumers that work together to read data from a topic, and rebalancing is the automatic redistribution of partitions whenever group membership changes, ensuring balanced processing.

When to Use

Consumer groups are used when you need to scale message consumption across multiple consumers or provide fault tolerance.

If one consumer fails, others in the group take over automatically.

Example

Imagine three consumers reading from a Kafka topic with six partitions. Each consumer gets two partitions.

If one consumer disconnects, the partitions are reassigned to the remaining two.

For deeper learning, check out Grokking System Design Fundamentals, Grokking the System Design Interview, Grokking Database Fundamentals for Tech Interviews, or practice with Mock Interviews with ex-FAANG engineers.

Why Is It Important

Consumer groups allow distributed systems to handle large volumes of data efficiently, while rebalancing ensures high availability and resilience against failures.

Interview Tips

Be ready to explain what triggers rebalancing (e.g., new consumer joins, one leaves, or partitions change). Highlight both benefits (scaling, fault tolerance) and challenges (temporary downtime).

Trade-offs

You gain scalability and reliability, but rebalancing can introduce processing delays, duplicate reads, or require careful offset management.

Pitfalls

A common mistake is assuming more consumers always improves throughput.

If consumers outnumber partitions, some remain idle. Misconfigured session timeouts can also cause frequent, unnecessary rebalances.