Consumer Groups, Parallelism, Offsets, Rebalancing, Failover, Consumption Patterns and Production Tuning
Audience: Students and freshers with no previous Kafka experience
Goal: Start with “What is a consumer?” and finish with production-grade consumer design, reliability, scaling, recovery, and performance tuning
Training environment: Confluent Kafka Cluster
Technical version note: This tutorial is aligned with modern Apache Kafka 4.x consumer concepts. Where the newer Consumer rebalance protocol differs from the older Classic protocol, both are explained explicitly.
Image note: The infographic is a learning map. The written tutorial below is the source of truth for protocol details, especially the differences between the newer Consumer group protocol and the older Classic protocol.
1. Learning Objectives
By the end of this tutorial, you should be able to explain:
- What a Kafka Consumer is
- How a consumer reads records from Kafka
- Why consumers pull data rather than Kafka pushing records to them
- How topic partitions map to consumers
- What a Consumer Group is
- How Consumer Groups provide parallel processing
- How Kafka balances workload across consumers
- Why the number of partitions limits parallelism inside one consumer group
- What the Group Coordinator does
- What rebalancing means
- What causes a rebalance
- What happens when a consumer joins
- What happens when a consumer leaves or crashes
- What happens when topic partitions change
- How the modern Consumer rebalance protocol differs from the Classic protocol
- What Kafka offsets are
- Current position vs committed offset
- Automatic vs manual offset commits
commitSync()vscommitAsync()- How to choose an offset strategy
- How consumer failover and recovery work
- Fan-out consumption
- Load-balanced consumption
- At-most-once vs at-least-once vs exactly-once approaches
- Consumer lag
- Consumer performance tuning
- Consumer configuration settings that matter in production
- Common consumer mistakes
- A production-ready consumer checklist
2. What Is a Kafka Consumer?
A Kafka Consumer is a client application that reads records from Kafka topics.
The simplest mental model is:
Producer
|
v
Kafka Topic
|
v
Consumer
Example:
Vehicle
|
v
Producer
|
v
Topic: vehicle-telemetry
|
v
Analytics Consumer
The consumer receives records such as:
{
"vehicle_id": "CAR-101",
"speed": 88,
"battery": 72
}
and performs business work:
Read event
|
v
Deserialize
|
v
Validate
|
v
Process
|
+--> Update database
+--> Generate alert
+--> Calculate analytics
+--> Call another service
|
v
Record progress
3. Kafka Consumers Pull Data
Kafka consumers normally pull records from brokers.
Kafka does not continuously push records into your application without the consumer asking.
Conceptually:
Consumer
|
| "Give me records from Partition 2
| beginning at my current position."
v
Broker
|
v
Record batch
The consumer repeatedly calls:
consumer.poll(...)
Think of poll() as:
“Give me the records that I am currently allowed to read from my assigned partitions.”
This pull design gives the consumer control over:
How quickly it reads
How many records it handles
When it pauses
When it resumes
How it tracks progress
4. Consumer End-to-End Flow
At a high level:
Application starts
|
v
Create KafkaConsumer
|
v
Connect to Kafka
|
v
Join Consumer Group
|
v
Receive Partition Assignment
|
v
Fetch Records
|
v
Deserialize
|
v
Process Business Logic
|
v
Commit Offset
|
v
poll() again
This loop continues for the life of the consumer.
5. What Does a Consumer Read?
A Kafka consumer receives ConsumerRecord objects.
A record includes important information such as:
Topic
Partition
Offset
Timestamp
Key
Value
Headers
Example:
Topic = vehicle-telemetry
Partition = 2
Offset = 1050
Key = CAR-101
Value = {"speed":88}
This information matters because a consumer must know:
Where did the record come from?
What order was it stored in?
How far have I progressed?
6. Deserialization
The producer serialized application data into bytes.
The consumer must reverse that process.
Kafka bytes
|
v
Deserializer
|
v
Application Object
For example:
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
With Confluent Schema Registry, a consumer may instead use schema-aware deserializers for:
Avro
Protobuf
JSON Schema
A producer and consumer must agree on how data is encoded.
7. What Is a Consumer Group?
A Consumer Group is a group of consumers cooperating to read one or more topics.
Consumers belong to the same logical group by using the same:
group.id
Example:
group.id = order-processing
Consumers:
Consumer A
Consumer B
Consumer C
Together:
Consumer Group: order-processing
Kafka treats them as workers sharing one consumption workload.
8. Why Consumer Groups Exist
Suppose the topic receives:
100,000 events / second
One consumer may not be able to process all of them.
We want:
Consumer 1
Consumer 2
Consumer 3
Consumer 4
...
to divide the work.
Kafka accomplishes this by distributing partitions among consumers in the same group.
9. Consumer Group + Partitions = Parallel Processing
Suppose:
Topic: orders
Partitions:
P0
P1
P2
One consumer:
Consumer Group
Consumer A
|
+--> P0
+--> P1
+--> P2
There is only one consumer process doing all the work.
Add three consumers:
P0 --> Consumer A
P1 --> Consumer B
P2 --> Consumer C
Now the group can process three partitions in parallel.
This is one of Kafka’s most important scaling mechanisms.
10. Golden Rule of Consumer Group Parallelism
Inside a normal Kafka consumer group:
One partition is assigned to at most one consumer in that group at a time.
Therefore:
Maximum useful partition-level consumer parallelism
approximately equals
number of partitions available to the group.
Example:
6 partitions
1 consumer
Consumer 1 -> P0 P1 P2 P3 P4 P5
6 partitions
3 consumers
Consumer 1 -> P0 P3
Consumer 2 -> P1 P4
Consumer 3 -> P2 P5
6 partitions
6 consumers
1 partition per consumer
6 partitions
10 consumers
6 consumers can receive partitions
4 consumers have no partition work
Adding consumers beyond the available partitions does not create additional partition-level parallelism.
11. One Consumer Can Read Multiple Partitions
Do not misunderstand the previous rule.
Kafka does not require:
1 consumer = 1 partition
A consumer can own many partitions.
Example:
12 partitions
3 consumers
A possible assignment:
Consumer A -> P0 P3 P6 P9
Consumer B -> P1 P4 P7 P10
Consumer C -> P2 P5 P8 P11
This is normal.
12. Why Partition Count Is So Important
Topic partition count affects:
Producer parallelism
Broker distribution
Consumer parallelism
Scaling ceiling
Ordering boundaries
Operational overhead
For consumers:
Too few partitions
|
v
Cannot add useful consumers beyond partition count
|
v
Limited parallel processing
But:
Huge number of partitions
|
v
More metadata
More files
More leader/replica management
More operational overhead
Do not create partitions without capacity planning.
13. How Kafka Balances Work Across Consumers
Kafka does not balance individual records directly.
It balances partition ownership.
Suppose:
P0 P1 P2 P3 P4 P5
and:
Consumer A
Consumer B
Consumer C
Kafka determines an assignment such as:
Consumer A -> P0 P3
Consumer B -> P1 P4
Consumer C -> P2 P5
Each consumer fetches records from the leaders of its assigned partitions.
So load balancing is really:
Partition Assignment
|
v
Consumer Work Distribution
14. Important Limitation: Partitions May Not Have Equal Work
Kafka can distribute partitions reasonably, but it cannot magically guarantee equal CPU work.
Imagine:
P0 = 1,000 events/sec
P1 = 1,100 events/sec
P2 = 50,000 events/sec
If:
Consumer A -> P0
Consumer B -> P1
Consumer C -> P2
Consumer C still has much more work.
Why?
Because Kafka assigns partitions, not perfectly equal units of business computation.
This is why producer key design and partition distribution affect consumer performance.
15. Consumer Group ID
group.id identifies the consumer group.
Different group.id values create independent consumption.
Example:
Topic: orders
Group 1:
group.id = analytics
Group 2:
group.id = fraud-detection
Group 3:
group.id = notifications
All three groups can independently read the same orders topic.
This becomes our fan-out pattern later.
16. The Group Coordinator
A Kafka broker acts as the Group Coordinator for a consumer group.
The coordinator manages important group responsibilities such as:
Group membership
Partition assignment coordination
Consumer liveness
Rebalance coordination
Offset commits
Conceptually:
Consumer Group
|
v
Group Coordinator
|
v
Kafka
17. How Kafka Finds the Group Coordinator
Kafka has an internal topic:
__consumer_offsets
The consumer group’s group.id maps to a partition of this internal topic.
The broker that leads that offsets partition becomes the coordinator for the group.
Conceptually:
group.id
|
v
hash / mapping
|
v
__consumer_offsets partition
|
v
Leader Broker
|
v
Group Coordinator
This distributes consumer-group coordination across brokers.
18. What Is Rebalancing?
A rebalance is the process of changing partition assignments among consumers in a group.
Example before:
Consumer A -> P0 P1
Consumer B -> P2 P3
Consumer C joins.
After rebalance:
Consumer A -> P0
Consumer B -> P1 P2
Consumer C -> P3
The exact assignment depends on the selected protocol and assignment strategy.
The key idea is:
Group membership or subscribed-partition changes may require Kafka to redistribute partition ownership.
19. Why Rebalancing Exists
Without rebalancing:
Consumer B crashes
|
v
P2 and P3 have no active consumer
|
v
Processing stops forever
With rebalancing:
Consumer B crashes
|
v
Kafka detects membership change
|
v
Partitions reassigned
|
v
Consumer A / C take over
Rebalancing is one of the mechanisms that gives consumer groups fault tolerance and elasticity.
20. Major Rebalance Triggers
A rebalance or reassignment can happen when:
A consumer joins
A consumer leaves gracefully
A consumer crashes or becomes unreachable
Consumer membership expires
A subscribed topic gains partitions
A subscribed topic changes
A regex subscription matches a new topic
Subscription metadata changes
Administrative/manual group changes occur
The exact protocol messages differ between the modern Consumer protocol and the older Classic protocol.
21. Consumer Joins a Group
Imagine:
Group currently:
Consumer A
Consumer B
A new instance starts:
Consumer C
Kafka must decide:
Which partitions should Consumer C receive?
Which existing consumers should give up partitions?
That causes a new assignment.
22. Consumer Leaves Gracefully
A consumer can shut down cleanly.
Conceptually:
Consumer B
|
v
close()
|
v
Leaves group
|
v
Its partitions become available
|
v
Assignment updated
Graceful shutdown is preferable to simply killing processes because the system can respond more cleanly.
23. Consumer Crashes
A crash is different.
The consumer cannot politely say:
"I am leaving."
Kafka must detect that the member is no longer healthy.
After failure detection:
Consumer B declared unavailable
|
v
Its partitions reassigned
|
v
Another consumer resumes processing
This is consumer failover.
24. Topic Partition Count Changes
Suppose:
Topic: orders
Partitions: 3
becomes:
Partitions: 6
The group now has new work.
Kafka needs to assign the new partitions to consumers.
Therefore adding partitions can cause new partition assignments.
This is why changing topic partitions is an operational event, not just a metadata edit.
25. Modern Consumer Protocol vs Classic Protocol
This is extremely important for modern Kafka students.
Kafka now has two consumer-group protocol families you may encounter:
1. Consumer protocol
2. Classic protocol
The Consumer protocol is the newer generation of Kafka’s group rebalance protocol.
The Classic protocol is the older model that many existing tutorials and applications still describe.
You should understand both because real organizations may run both during migration.
26. Classic Consumer Group Protocol — Conceptual Model
In the Classic protocol, the group has a client-side leader involved in partition assignment.
A simplified flow is:
Consumers
|
v
Join Group
|
v
Coordinator
|
v
Group leader participates in assignment
|
v
Sync Group
|
v
Consumers receive assignments
Classic assignment strategies include client-side assignors such as:
Range
Round Robin
Sticky
Cooperative Sticky
Some Classic strategies use eager movement of partitions; cooperative strategies can reduce how much work must stop and move.
27. Modern Consumer Rebalance Protocol
Modern Kafka’s Consumer protocol moves more group-assignment responsibility to the broker-side coordinator.
Key ideas:
Broker-side assignment
Incremental assignment changes
No client group leader needed for assignment
Reduced global synchronization
Faster / less disruptive rebalances
The assignment strategy is managed differently from the older Classic client-side assignor model.
For students:
Do not memorize only the old “one consumer becomes leader and assigns everything” explanation as if it describes every modern Kafka consumer group.
28. Which Protocol Should Students Learn?
Learn the architecture in this order:
First:
Consumer Group concepts
Partitions
Coordinator
Offsets
Rebalancing
Then:
Modern Consumer protocol
Then:
Classic protocol
for compatibility and existing systems
The business concepts are the same:
Consumers share partitions
Membership changes
Assignments change
Offsets allow recovery
The protocol mechanics differ.
29. Rebalancing and Application Processing
Rebalances matter because consumers may temporarily change ownership of partitions.
Imagine:
Consumer A owns P0
Then:
P0 is moved to Consumer B
The application must correctly handle:
Finish or stop work on P0
Commit safe progress
Release partition-specific resources
Begin processing new assignment
Incorrect rebalance handling can cause:
Duplicate work
Incorrect offset commits
Lost application progress
Processing delays
30. ConsumerRebalanceListener
The Java consumer provides rebalance callbacks.
Important concepts:
onPartitionsRevoked(...)
onPartitionsAssigned(...)
onPartitionsLost(...)
These callbacks let applications react when partition ownership changes.
Typical uses include:
Commit safe offsets
Flush local state
Close partition-specific resources
Initialize state for new partitions
onPartitionsLost is especially important because in some failure cases the consumer may discover it has already lost ownership and should not assume it can safely perform the same actions as a normal revoke.
31. What Is a Kafka Offset?
A Kafka offset is a numerical position inside a partition log.
Example:
Partition 0
Offset 0 -> Record A
Offset 1 -> Record B
Offset 2 -> Record C
Offset 3 -> Record D
Offset 4 -> Record E
Offsets belong to partitions.
Therefore:
P0 offset 100
and:
P1 offset 100
are different positions.
Offsets are not global message IDs across the entire topic.
32. Consumer Position — Current Position
Kafka consumers have a current position.
A simple definition:
The current position is the offset of the next record the consumer will fetch/return from that partition.
Example:
Partition:
0 1 2 3 4 5 6 7
^
|
Current position = 6
This means the consumer has already advanced past earlier records and expects to continue from around offset 6.
The current position normally lives in the running consumer’s state and advances as poll() returns records.
33. Committed Offset
The committed offset is the saved recovery checkpoint for a consumer group.
Example:
Partition:
0 1 2 3 4 5 6 7
^ ^
| |
Committed Current
= 4 = 6
The consumer may currently have fetched through offset 5 but only safely committed progress up to the point represented by committed offset 4.
If the consumer crashes:
Current in-memory position is lost
Kafka uses:
Committed offset
to decide where the group should resume.
34. Critical Offset Rule: Commit the NEXT Record to Read
This is one of the most important details in Kafka.
If you successfully process records:
Offset 10
Offset 11
Offset 12
your commit should normally indicate:
13
because committed offset means:
“The next record I intend to read/process is offset 13.”
Do not think of the committed value merely as:
"The last record I processed."
Think:
"The next position I should resume from."
35. Where Are Committed Offsets Stored?
Kafka stores consumer-group commits in the internal compacted topic:
__consumer_offsets
Conceptually:
Consumer
|
v
Offset Commit
|
v
Group Coordinator
|
v
__consumer_offsets
This allows offsets to survive a consumer process restart.
36. Why __consumer_offsets Is Compacted
Kafka usually only needs the latest committed position for:
group
+
topic
+
partition
Example:
order-processing / orders / P0 -> 1050
Later:
order-processing / orders / P0 -> 1100
The newest commit becomes the important recovery state.
A compacted internal topic is a good fit for this kind of state.
37. Four Positions Worth Understanding
Students should eventually understand these concepts:
1. Current consumer position
2. Committed group offset
3. High watermark / readable end
4. Log end / latest written position
Simplified:
Committed Current Readable End
| | |
v v v
0 1 2 3 4 5 6 7 8 9 10 11 12 ...
In transactional consumption there is another important concept:
Last Stable Offset (LSO)
which matters for read_committed.
We will cover transactions separately.
38. What Is Consumer Lag?
Consumer lag answers:
“How far behind is the consumer group?”
Simplified:
End of available data
-
group's consumed/committed progress
=
lag
Example:
Latest/end position = 10,000
Committed position = 9,500
Lag ≈ 500 records
High lag means work is accumulating faster than the consumer group is completing it.
39. Lag Is a Symptom, Not the Root Cause
If lag grows, possible causes include:
Consumer processing too slowly
Too few consumers
Too few partitions
Hot partition
Slow database
Slow external API
Large records
Deserialization cost
Long GC pauses
Network latency
Broker throttling
Bad fetch configuration
Frequent rebalances
Consumer errors
Do not respond to lag by blindly adding consumers.
If:
Topic has 4 partitions
Group already has 4 active consumers
adding ten more consumers will not create more partition-level parallelism.
40. Offset Commit Strategies
There are two major approaches:
Automatic Commit
Manual Commit
Neither is “always correct.”
The right strategy depends on:
Business correctness
Failure semantics
Processing time
Idempotency
Throughput
Operational complexity
41. Automatic Offset Commit
With:
enable.auto.commit=true
the consumer periodically commits offsets in the background.
The interval is controlled by:
auto.commit.interval.ms
This is easy to use.
But it is important to understand what you are promising.
The application must ensure that records returned from polling are actually processed safely before later polling/commit behavior can move the recovery point past work that is not finished.
Automatic commit is convenient, but it gives the application less explicit control over the exact business-processing boundary.
42. Automatic Commit Does NOT Mean “Commit Every Record”
A common misunderstanding:
enable.auto.commit=true
does not mean:
Kafka commits every record immediately.
The commits happen periodically.
Therefore the gap between:
current processing position
and:
committed recovery position
can exist.
If a failure happens, some records may be read again.
This is one reason duplicate-tolerant/idempotent processing is valuable.
43. Manual Offset Commit
With:
enable.auto.commit=false
the application controls commits.
Common APIs include:
commitSync()
commitAsync()
The core production rule is:
Commit progress only when the corresponding business work is safely complete according to your required delivery semantics.
44. commitSync()
commitSync() waits for the commit result.
Conceptually:
Process records
|
v
commitSync()
|
v
Wait for broker response
|
v
Continue
Advantages:
Simple failure handling
Known commit completion point
Good at controlled boundaries
Tradeoff:
Blocking can reduce throughput if used too frequently
Do not call synchronous commit after every individual record unless the workload genuinely requires that behavior and the performance cost is acceptable.
45. commitAsync()
commitAsync() sends the commit without blocking the main processing flow.
Conceptually:
Process
|
v
commitAsync()
|
+----> Commit request continues
|
v
Keep processing
Advantages:
Lower blocking overhead
Better throughput
But you must handle:
Commit callback errors
Ordering of commit attempts
Shutdown/rebalance boundaries
A common production pattern is:
Periodic async commits during normal processing
PLUS
a final controlled sync commit at important shutdown/rebalance boundaries
when that model matches the application’s semantics.
46. Why Commit Timing Matters
Consider two sequences.
Strategy A
Commit offset
|
v
Process database write
Failure scenario:
Commit succeeds
Application crashes
Database write never happens
Kafka may resume after the record.
From the business point of view:
record may be lost
This resembles at-most-once processing.
47. Process Then Commit
Now reverse it:
Process database write
|
v
Commit offset
Failure scenario:
Database write succeeds
Application crashes before commit
After restart:
Kafka resumes from old committed offset
The record is processed again.
This creates possible duplicates.
This resembles at-least-once processing.
48. At-Most-Once
Conceptually:
Commit
|
v
Process
Potential outcome:
No duplicate processing
but possible loss if failure happens after commit and before processing.
Use cases may include data where losing an occasional record is preferable to duplicate work.
For important business events, this is often not desirable.
49. At-Least-Once
Conceptually:
Process
|
v
Commit
Potential outcome:
No intentional loss of successfully fetched work
but duplicates are possible after failures.
Therefore the downstream processing should preferably be:
Idempotent
Meaning:
Processing the same event twice
does not create an incorrect business result.
50. Exactly-Once — Be Precise
Do not teach students:
“Manual commits give exactly once.”
They do not.
Exactly-once requires a coordinated processing design.
Kafka supports transactional approaches for Kafka-to-Kafka workflows, together with transactional producers and consumers configured to read committed transactional data.
If your consumer writes to:
External SQL database
External REST API
Email provider
Payment provider
Kafka cannot magically create one atomic transaction across all of those systems.
You may need:
Idempotency keys
Database transactions
Outbox / inbox patterns
Deduplication
Kafka transactions where applicable
Application-specific coordination
Exactly-once is an architecture topic, not one checkbox.
51. auto.offset.reset
What happens when there is no usable committed offset?
Kafka uses:
auto.offset.reset
Common strategies include:
earliest
latest
none
Modern Kafka also supports a duration-based reset option.
Important:
auto.offset.resetis not the normal “where should my healthy consumer read next?” setting.
It is primarily used when Kafka cannot find a valid committed position for the group/partition.
52. earliest
auto.offset.reset=earliest
means:
Start from the earliest retained offset available.
Useful for:
New analytics consumers
Backfill processing
Replay use cases
Training
Rebuilding state
Remember:
Kafka may already have deleted older records according to retention.
“Earliest” means earliest still retained, not necessarily the first record ever produced.
53. latest
auto.offset.reset=latest
means:
Start from the latest position and consume new records from that point forward.
Useful for:
Real-time consumers
where old history is not needed
Danger:
A new group configured with latest will normally not process the existing retained history before its starting point.
Know whether that is acceptable.
54. none
auto.offset.reset=none
means:
If Kafka has no valid committed offset, fail rather than automatically choosing earliest/latest.
This can be useful when silently choosing a new position would be dangerous.
For critical applications, failing visibly can be safer than unexpectedly skipping or replaying data.
55. Duration-Based Reset
Modern Kafka can also reset based on a duration from the current time.
Conceptually:
Start from approximately N hours/days ago
This is useful for some recovery or time-window processing cases.
Treat it as an advanced option after students master:
earliest
latest
none
56. Right Offset Strategy — Decision Guide
Ask:
Question 1
Can duplicate processing happen safely?
If yes:
At-least-once
+
commit after processing
+
idempotent downstream
is often a strong practical design.
Question 2
Can any record be lost?
If no:
Avoid committing before successful processing.
Question 3
What should a new group do?
Choose intentionally:
earliest
latest
none
duration-based
Question 4
Can the business operation and offset be committed atomically?
If Kafka-to-Kafka:
Kafka transactions may help.
If external side effects:
Use application/database patterns appropriate to that system.
57. Consumer Failover and Recovery
Suppose:
P0 -> Consumer A
P1 -> Consumer B
P2 -> Consumer C
Consumer B crashes.
Before:
A -> P0
B -> P1
C -> P2
After detection and reassignment:
A -> P0 P1
C -> P2
or another valid assignment.
The new owner starts from the group’s committed position for P1.
That is why committed offsets are fundamental to recovery.
58. Failure Window and Duplicate Processing
Suppose:
Committed offset = 100
Current position = 110
The consumer has processed records near 100-109 but has not committed the newer progress.
Then it crashes.
Replacement consumer resumes from:
100
Records between the committed point and the crash-time processing position may be processed again.
This is expected for at-least-once designs.
Do not call this a Kafka bug.
It is the consequence of the chosen commit boundary.
59. Consumer Liveness
Kafka needs to distinguish:
Healthy consumer
from:
Dead or stuck consumer
There are two related concerns:
1. Membership/liveness heartbeat
2. Application processing progress / poll interval
These concepts are important but the exact configuration differs between the modern Consumer protocol and the older Classic protocol.
60. Heartbeats — Concept
Consumers maintain group membership by demonstrating that they are alive.
Conceptually:
Consumer
|
v
Coordinator
"I'm alive."
If Kafka determines the consumer is no longer alive:
Remove member
|
v
Reassign partitions
61. Classic vs Consumer Protocol Timeout Configuration
Do not blindly copy old timeout advice.
Classic protocol
Client-side settings historically include:
heartbeat.interval.ms
session.timeout.ms
Modern Consumer protocol
Heartbeat/session behavior is controlled more by broker-side group configuration.
This is one of the important operational differences between the protocols.
Always check which group protocol your client and cluster are using before tuning these settings.
62. max.poll.interval.ms
Another failure mode is:
Consumer is alive
but application takes too long between poll() calls.
Kafka uses:
max.poll.interval.ms
to place an upper bound on how long the consumer can go without making expected poll progress under group management.
Example:
poll()
|
v
Receive 500 records
|
v
Processing takes 10 minutes
|
v
max.poll.interval.ms = 5 minutes
The group may treat the consumer as unable to keep up and reassign its partitions.
This can create a rebalance loop.
63. Long Processing Is a Major Consumer Design Problem
If each record requires:
10 seconds
and:
max.poll.records=500
one poll could theoretically represent a huge amount of processing time.
Possible solutions:
Reduce max.poll.records
Optimize business processing
Batch downstream operations
Increase safe poll interval where justified
Pause/resume partitions
Use carefully designed worker-thread architecture
Increase partitions and consumers if appropriate
Do not simply increase every timeout to enormous values.
That can make real failures take too long to recover.
64. Consumer Is Not Automatically Your Worker Thread Pool
A very important Java design point:
A KafkaConsumer instance is not intended to be freely used from many application threads concurrently.
A common safe model is:
One consumer thread
|
v
poll()
|
v
process assigned records
If you hand records to worker threads:
Consumer Thread
|
v
Queue
/ | \
v v v
W1 W2 W3
you must solve:
Ordering
Backpressure
Offset coordination
Failure handling
Rebalances
Shutdown
This is an advanced architecture, not “free extra parallelism.”
65. Fan-Out Consumption Pattern
Fan-out means:
Multiple independent applications/groups consume the same topic.
Example:
-> Group: analytics
/
Topic: orders -----> Group: fraud
\
-> Group: notifications
Each group gets its own independent view of the topic.
Inside each group, partitions are shared among that group’s consumers.
This lets the same event power many independent systems.
66. Fan-Out Example
Topic:
vehicle-telemetry
Groups:
group.id = realtime-dashboard
group.id = anomaly-detection
group.id = data-warehouse
group.id = billing
All four groups independently consume vehicle telemetry.
Producer does not need to send the event four times.
Kafka retains the stream, and each group tracks its own offsets.
67. Load-Balanced Consumption Pattern
Load balancing means:
Multiple consumers use the same
group.idand divide partitions.
Example:
Topic: orders
Consumer Group: order-workers
Consumer 1
Consumer 2
Consumer 3
Consumer 4
Partitions are distributed across those consumers.
Use when:
One logical application
needs more processing capacity.
68. Fan-Out vs Load Balance
Fan-Out
Same topic
Different group IDs
Result:
Each group independently gets the data.
Use for:
Analytics
Alerts
Billing
Fraud
Data warehouse
Load Balance
Same topic
Same group ID
Result:
Consumers share the partitions/work.
Use for:
Scaling one application horizontally.
This distinction is foundational.
69. Combining Fan-Out and Load Balancing
Real production systems use both.
Example:
Topic: orders
Analytics group:
analytics
Consumer A
Consumer B
Fraud group:
fraud
Consumer A
Consumer B
Consumer C
Consumer D
Notification group:
notifications
Consumer A
Kafka allows each application to scale independently.
70. Manual assign() vs Group subscribe()
Most consumer applications use:
subscribe(...)
which participates in consumer-group partition management.
Kafka can also let an application explicitly assign partitions:
assign(...)
Example:
Consumer manually owns:
P0
P4
With manual assignment:
Kafka group rebalancing does not manage those partition assignments for you.
Use manual assignment only when you intentionally want application-controlled partition ownership.
Do not confuse it with normal Consumer Group load balancing.
71. Consumer Fetching
Consumers fetch records in batches.
Important settings include:
fetch.min.bytes
fetch.max.wait.ms
fetch.max.bytes
max.partition.fetch.bytes
max.poll.records
These influence:
Network efficiency
Throughput
Latency
Memory
Amount of work returned per poll
72. fetch.min.bytes
This asks the broker to try to return at least a certain amount of data before responding, subject to wait limits.
Higher values may:
Increase batch efficiency
Reduce request overhead
Improve throughput
but may also:
Increase latency when traffic is low
Another throughput/latency tradeoff.
73. fetch.max.wait.ms
This limits how long the broker may wait while trying to satisfy the fetch-size conditions.
Think:
fetch.min.bytes
=
"Try to give me this much data."
fetch.max.wait.ms
=
"But don't wait longer than this."
These settings work together.
74. fetch.max.bytes
Controls approximately how much data a fetch response may contain across partitions.
This matters for:
Network usage
Consumer memory
Large workloads
Large records
Do not set fetch sizes blindly without understanding record sizes and available memory.
75. max.partition.fetch.bytes
This controls the amount of data fetched per partition in a request.
If messages/batches are large, this setting becomes important.
Always coordinate large-message configuration end-to-end:
Producer
Topic/Broker
Consumer
76. max.poll.records
This controls how many records poll() returns to the application at one time.
Important:
It does not necessarily change how much data Kafka fetches internally from brokers.
It limits how many fetched records are returned to the application per poll.
This is extremely useful for controlling:
Processing batch size
Per-poll work
max.poll.interval risk
Application memory
77. Example: max.poll.records
Suppose:
Each record takes 100 ms to process
max.poll.records = 500
Worst-case serial work:
500 × 100 ms
=
50 seconds
If each record instead takes:
2 seconds
then:
500 × 2 sec
=
1000 sec
=
16+ minutes
Now poll timing becomes dangerous.
Tuning must reflect business processing time, not only Kafka throughput.
78. pause() and resume()
Consumers can temporarily pause assigned partitions without giving up ownership.
Conceptually:
P0 P1 P2 assigned
P1 downstream overloaded
|
v
pause(P1)
|
v
Continue P0/P2
Later:
resume(P1)
This can help implement controlled backpressure.
But offsets, processing state and rebalance handling must still be correct.
79. Static Membership — Concept
In some deployments, consumer instances are stable and restart temporarily.
Kafka supports static membership concepts through:
group.instance.id
This can reduce unnecessary membership churn in certain restart scenarios.
But it is not a replacement for:
Good failure detection
Correct rebalancing
Offset correctness
Use it when your deployment model benefits from stable member identities.
80. Rebalance Cost
A rebalance can cost:
Temporary reduction in consumption
Partition movement
Cache warm-up
State reconstruction
Database/client initialization
Duplicate processing near commit boundaries
Therefore:
Constant rebalancing
=
Bad operational health
Monitor rebalance frequency.
81. Rebalance Storm Example
Imagine:
Consumer starts
|
v
Processing too slow
|
v
max.poll.interval exceeded
|
v
Consumer removed
|
v
Rebalance
|
v
Consumer rejoins
|
v
Rebalance again
This can repeat.
Symptoms:
High lag
Low throughput
Frequent assignment changes
Lots of logs
Consumers appear healthy but never catch up
Fix the root cause.
82. Offset Commit During Rebalance
When a consumer is about to give up a partition, the application may need to commit the safest completed progress for that partition.
Conceptually:
P0 currently owned by Consumer A
|
v
P0 will move
|
v
Commit safe completed position
|
v
Release resources
|
v
Consumer B receives P0
Never commit work that has not actually completed.
83. onPartitionsLost vs Normal Revoke
Normal revoke means the consumer is being told:
"You are about to give up these partitions."
A lost-partition callback means:
"You may already have lost ownership."
The second situation requires more caution.
Do not assume it is safe to commit arbitrary state after ownership has already been lost.
84. Offset Commit Best Practices
Best Practice 1
Commit only safe completed progress.
Process
|
v
Business operation succeeds
|
v
Commit
Best Practice 2
Commit the next offset to read.
If records through offset 120 are complete:
commit 121
Best Practice 3
Do not commit every record unless necessary.
Prefer reasonable batches:
Process N records
|
v
Commit safe position
Balance:
Duplicate window
vs
commit overhead
Best Practice 4
Make downstream processing idempotent where possible.
Example:
Event ID = ORDER-123:PAYMENT-CAPTURED
Database stores that event ID.
Second processing attempt:
Already processed
-> do not create duplicate payment
This turns expected at-least-once replay into safe recovery.
Best Practice 5
Handle commit errors.
Do not write:
consumer.commitAsync();
and assume:
"Everything is definitely saved."
Observe failures where required.
Best Practice 6
Handle rebalance boundaries.
Before giving up a partition:
Finish safe work
Commit safe progress
Release resources
when your application design and protocol callback allow it.
Best Practice 7
Monitor lag AND commit health.
A consumer can be processing quickly but failing commits.
Then:
Current position moves
Committed position stays behind
A crash may cause large replay.
Monitor both.
85. Consumer Failover Example
Topic:
payments
P0 P1 P2 P3
Group:
payment-risk
Before:
Consumer A -> P0 P1
Consumer B -> P2 P3
Consumer B crashes.
After reassignment:
Consumer A -> P0 P1 P2 P3
If:
P2 committed = 900
P2 current before crash = 920
new processing resumes from committed progress around:
900
Records near 900-919 may be processed again.
That is why payment processing must be idempotent.
86. Fault Tolerance Depends on More Than Kafka
Kafka can reassign partitions.
But your application may depend on:
Database
HTTP service
Cache
File system
Machine learning service
Payment gateway
If those are unhealthy:
Consumer is technically alive
but business processing cannot complete.
Production health must include downstream dependencies.
87. Consumer Performance Tuning — Start With Measurement
Never start by changing ten configs.
Use:
1. Define SLO
2. Measure baseline
3. Measure lag
4. Measure processing time
5. Identify bottleneck
6. Change one controlled variable
7. Load test
8. Failure test
9. Measure again
Questions:
Is Kafka fetch slow?
Is processing slow?
Is database slow?
Are partitions skewed?
Are consumers rebalancing?
Are commits slow?
Is one partition hot?
88. Goal 1 — Increase Throughput
Potential levers:
Increase useful consumer count
Increase topic partitions when architecture requires it
Increase fetch efficiency
Tune fetch.min.bytes
Tune fetch.max.wait.ms
Tune fetch sizes
Tune max.poll.records
Batch database writes
Parallelize downstream work carefully
Reduce serialization/deserialization overhead
Avoid frequent commits
Avoid frequent rebalances
But always protect:
Ordering
Correct commits
Memory
Failure recovery
89. Goal 2 — Lower Latency
Potential levers:
Lower fetch waiting
Smaller processing batches
Fast downstream services
Appropriate consumer locality
Avoid huge max.poll.records
Reduce rebalance frequency
Fast deserialization
Monitor broker throttling
Tradeoff:
Smaller batches
=
more requests / overhead
Again:
Latency
vs
Throughput
90. Goal 3 — Improve Reliability
Focus on:
Correct offset commit timing
Idempotent business processing
Safe rebalance handling
Failure testing
Intentional auto.offset.reset
Commit monitoring
Consumer lag monitoring
Retries in downstream operations
Dead-letter/error strategy where appropriate
Transactions where appropriate
The consumer is reliable only if the business result is reliable.
91. Goal 4 — Improve Availability
Focus on:
Multiple consumer instances
Enough partitions
Healthy group coordination
Fast but sensible failure detection
Stable deployments
Controlled rolling restarts
Avoid rebalance storms
Redundant downstream services
Monitoring and alerting
Availability is not just:
"Consumer process is running."
It means:
The group continues making useful progress.
92. Consumer Configuration Study Map
Important settings include:
| Setting | What it controls | Why it matters |
|---|---|---|
bootstrap.servers | Initial Kafka endpoints | Cluster discovery |
group.id | Consumer group identity | Work sharing + offsets |
key.deserializer | Key decoding | Correct application types |
value.deserializer | Value decoding | Correct application data |
enable.auto.commit | Automatic commits | Delivery semantics |
auto.commit.interval.ms | Auto commit interval | Replay/commit window |
auto.offset.reset | Start point when no valid commit exists | Recovery/new-group behavior |
max.poll.records | Records returned per poll | Processing batch |
max.poll.interval.ms | Max time between expected poll progress | Rebalance/failure detection |
fetch.min.bytes | Minimum fetch target | Throughput vs latency |
fetch.max.wait.ms | Max broker fetch wait | Throughput vs latency |
fetch.max.bytes | Fetch response size | Network/memory |
max.partition.fetch.bytes | Per-partition fetch amount | Large records/memory |
isolation.level | Transaction visibility | Exactly-once transactional reads |
group.protocol | Group protocol selection | Modern vs Classic behavior |
Protocol-specific timeout/assignment settings must be interpreted according to whether the group is using the modern Consumer protocol or Classic protocol.
93. Modern Group Protocol Configuration Awareness
For modern Kafka, students should recognize:
group.protocol=consumer
as the setting associated with using the newer Consumer group protocol in clients that support it.
With this protocol:
Assignment becomes broker-driven
Group behavior is more incremental
Some Classic client-side assignment/heartbeat settings are no longer used the same way
Do not mix configuration advice from different group protocols.
94. Classic Group Assignment Strategies
If working with Classic groups, common assignment strategies include:
RangeAssignor
RoundRobinAssignor
StickyAssignor
CooperativeStickyAssignor
Each has different distribution and rebalance behavior.
Do not say:
"Kafka always uses Round Robin."
Assignment depends on protocol and configuration.
95. Modern Consumer Protocol Assignment
With the newer Consumer protocol:
Assignment logic is broker-side
The server provides available assignment strategies, and the consumer can participate according to the modern group protocol configuration.
The important teaching point is:
Classic:
client-led assignment mechanics
Modern Consumer protocol:
broker-side coordinator-driven assignment mechanics
96. Consumer Lag Monitoring
Monitor lag per:
Consumer Group
Topic
Partition
Why partition-level?
Because:
Overall group lag = 10,000
may hide:
P0 lag = 0
P1 lag = 0
P2 lag = 10,000
That often indicates:
Hot partition
Slow record type
Bad key distribution
Stuck processing
Partition-level visibility is essential.
97. Metrics Worth Monitoring
Examples of useful consumer-health signals:
Records consumed rate
Bytes consumed rate
Fetch latency
Fetch rate
Records per request
Assigned partitions
Consumer lag
Commit latency
Commit rate
Commit failures
Rebalance rate
Time since last successful poll
Processing latency
Application error rate
Kafka metrics alone are not enough.
Also measure:
Database latency
API latency
Queue depth
CPU
Memory
GC
Thread pool saturation
98. Common Consumer Mistakes
Mistake 1 — More Consumers Than Partitions
4 partitions
20 consumers
You do not get twenty-way partition processing.
Many consumers remain idle.
Mistake 2 — Commit Before Processing
Can create business data loss after a crash.
Mistake 3 — Never Commit
Consumer restarts may replay huge amounts of work.
Mistake 4 — Auto Commit Without Understanding It
Easy configuration can hide incorrect business semantics.
Mistake 5 — Slow Processing Between Polls
Can cause consumer-group instability and repeated rebalances.
Mistake 6 — Long Blocking API Calls in Poll Thread
Example:
poll()
|
v
Call external API
wait 8 minutes
Dangerous if poll progress requirements are exceeded.
Mistake 7 — Ignoring Rebalance Callbacks
Can lose safe local state or commit the wrong progress.
Mistake 8 — Assuming Offset = Message ID
Offset is a partition position.
Mistake 9 — Assuming Consumer Lag Means “Add Consumers”
If the topic has too few partitions, extra consumers do nothing.
If one partition is hot, extra consumers do not split that partition.
Mistake 10 — Confusing Different Consumer Groups
Different groups do not share work with each other.
They independently consume the topic.
Mistake 11 — Using One Group ID for Unrelated Applications
If analytics and notifications accidentally use the same group.id:
They will split partitions
instead of both receiving all events.
This is a serious architecture mistake.
Mistake 12 — Ignoring Protocol Version
Applying Classic timeout/assignor advice to a modern Consumer-protocol group can be wrong.
Know which protocol you are operating.
99. Java Consumer Example
A simplified teaching example:
Properties props = new Properties();
props.put("bootstrap.servers", "<BOOTSTRAP_SERVER>");
props.put("group.id", "vehicle-analytics");
props.put(
"key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer"
);
props.put(
"value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer"
);
props.put("enable.auto.commit", "false");
props.put("auto.offset.reset", "earliest");
// For modern Kafka clients/clusters where you intentionally use
// the newer Consumer group protocol:
// props.put("group.protocol", "consumer");
KafkaConsumer<String, String> consumer =
new KafkaConsumer<>(props);
consumer.subscribe(List.of("vehicle-telemetry"));
try {
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(1000));
for (ConsumerRecord<String, String> record : records) {
System.out.println(
"topic=" + record.topic()
+ " partition=" + record.partition()
+ " offset=" + record.offset()
+ " key=" + record.key()
+ " value=" + record.value()
);
// Business processing here
}
// Teaching example:
// commit after the batch is successfully processed.
consumer.commitSync();
}
}
finally {
consumer.close();
}
This example favors clarity.
A production application should additionally implement:
Error handling
Controlled shutdown
Rebalance handling
Metrics
Retry strategy
Idempotent processing
Security configuration
Appropriate commit strategy
100. Confluent Kafka Cluster Connection
For a Confluent-managed cluster, the consumer also needs authentication/security configuration.
Conceptually:
bootstrap.servers=<CONFLUENT_BOOTSTRAP>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=<CREDENTIAL_CONFIGURATION>
Then:
Consumer
|
v
Authenticate
|
v
Discover brokers
|
v
Find group coordinator
|
v
Join group
|
v
Receive partition assignment
|
v
Fetch from partition leaders
Keep credentials in secure configuration/secrets, not hardcoded source code.
101. Complete Consumer Journey
Now combine everything:
APPLICATION START
|
v
Create KafkaConsumer
|
v
bootstrap.servers
|
v
Authentication
|
v
Cluster Metadata
|
v
group.id
|
v
Find Group Coordinator
|
v
Join Consumer Group
|
v
Group Protocol
|
v
Partition Assignment
|
v
Find Partition Leaders
|
v
Fetch Requests
|
v
Record Batches
|
v
Deserialize Key / Value
|
v
poll() returns ConsumerRecords
|
v
Business Processing
|
v
Current Position Advances
|
v
Commit Safe Offset
|
v
__consumer_offsets
|
v
poll() again
If membership changes:
|
v
Rebalance / Reassignment
|
v
Partitions Move
|
v
Resume From Committed Progress
Students should be able to explain every box.
102. Lab 1 — One Consumer, Multiple Partitions
Create:
Topic: orders
Partitions: 6
Start:
1 consumer
group.id = order-workers
Observe:
One consumer owns all six partitions.
Questions:
- Can this application process partitions in parallel internally?
- What limits its throughput?
- What happens if the process crashes?
103. Lab 2 — Scale to Three Consumers
Start:
3 consumer instances
same group.id
Observe the assignment.
Expected concept:
6 partitions
3 consumers
≈ 2 partitions per consumer
Questions:
Did a rebalance occur?
Which partitions moved?
Did lag fall?
Did throughput improve?
104. Lab 3 — More Consumers Than Partitions
Keep:
6 partitions
Start:
10 consumers
same group.id
Observe:
Some consumers receive no partitions.
This demonstrates:
Partitions determine partition-level parallelism.
105. Lab 4 — Consumer Failure
Start three consumers.
Kill one abruptly.
Observe:
Failure detection
Reassignment
Remaining consumers take partitions
Consumption resumes
Record:
Rebalance duration
Lag increase
Duplicate processing
New assignments
106. Lab 5 — Graceful Shutdown vs Crash
Experiment A:
consumer.close()
Experiment B:
kill -9 / abrupt container stop
Compare:
Time to group recovery
Logs
Rebalance behavior
Duplicate window
This teaches why controlled shutdown matters.
107. Lab 6 — Automatic Commit
Configure:
enable.auto.commit=true
Process records slowly.
Stop the consumer at different times.
Restart it.
Observe:
Which records repeat?
Where does consumption restart?
The objective is not to prove “auto commit is bad.”
The objective is to understand its failure window.
108. Lab 7 — Manual Commit
Configure:
enable.auto.commit=false
Flow:
poll
process all records successfully
commitSync
Crash:
after processing
before commit
Observe duplicates after restart.
Then explain:
Why duplicates are expected
Why idempotency matters
109. Lab 8 — Commit Before Processing
In a disposable training environment only:
poll
commit
then process
Force a crash after commit but before processing finishes.
Observe:
Consumer restarts after committed position
Explain the potential lost-processing window.
This demonstrates at-most-once semantics.
110. Lab 9 — Fan-Out
Topic:
orders
Create:
Group A = analytics
Group B = notifications
Both groups consume the same events independently.
Observe separate committed offsets.
111. Lab 10 — Load Balance
Create:
Group = order-workers
Consumers = 3
Observe:
Partitions divided among consumers.
Compare with Lab 9.
Students should be able to explain:
Different groups = fan-out
Same group = workload sharing
112. Lab 11 — Consumer Lag
Make the consumer intentionally slow.
Observe lag growing.
Then test:
Add a consumer
If enough partitions exist, lag recovery may improve.
Then create a scenario where one partition is hot.
Observe that adding consumers may not solve a single hot-partition bottleneck.
113. Lab 12 — max.poll.records
Test:
max.poll.records = 500
Measure processing time.
Then:
max.poll.records = 50
Observe:
Poll frequency
Processing batch size
Memory
Latency
Throughput
Explain why the right value depends on processing cost.
114. Lab 13 — Rebalance During Processing
Run several consumers.
While processing:
Start another consumer.
Observe:
Partition assignment changes.
Add rebalance callbacks and print:
revoked partitions
assigned partitions
lost partitions
This turns rebalancing from an abstract concept into something visible.
115. Production Offset Strategy Example
Suppose we process payment events.
Requirement:
Never charge twice.
Do not silently lose a payment event.
A reasonable architecture might include:
At-least-once Kafka consumption
|
v
Stable event/payment ID
|
v
Idempotent database/payment operation
|
v
Commit Kafka offset after safe business completion
Kafka offsets alone are not enough.
The business operation itself must be designed for retries.
116. Production Analytics Example
Analytics event:
page-view
Duplicate processing may be less harmful.
A different commit/batching strategy may be acceptable.
Therefore:
Payment consumer configuration
!=
Analytics consumer configuration
There is no universal “best Kafka consumer config.”
There is only:
Best config for this workload and correctness requirement.
117. Production Consumer Checklist
Before calling a consumer production-ready:
Group Design
- [ ] Intentional
group.id - [ ] Fan-out vs load-balancing pattern understood
- [ ] Consumer count matched to expected partition-level parallelism
- [ ] Partition count capacity planned
- [ ] Hot partitions considered
- [ ] Group protocol known: Consumer or Classic
Offset Strategy
- [ ]
enable.auto.commitchosen intentionally - [ ] Commit boundary matches business correctness
- [ ] Committed value represents next offset to process
- [ ]
auto.offset.resetchosen intentionally - [ ] Duplicate processing behavior understood
- [ ] Data-loss window understood
- [ ] Commit failures monitored
Processing
- [ ] Processing is idempotent where practical
- [ ] Slow downstream dependencies handled
- [ ] Retry strategy defined
- [ ] Dead-letter/error handling defined if needed
- [ ] Long-processing design tested
- [ ] Ordering requirements documented
Rebalancing
- [ ] Rebalance callbacks handled where needed
- [ ] Safe progress committed before normal revoke
- [ ] Lost-partition behavior understood
- [ ] Rolling deployment behavior tested
- [ ] Consumer crash tested
- [ ] Frequent rebalance alerting configured
Performance
- [ ] Lag monitored
- [ ] Lag monitored per partition
- [ ]
max.poll.recordsload tested - [ ] Fetch settings load tested
- [ ] Memory measured
- [ ] CPU measured
- [ ] Deserialization cost measured
- [ ] Database/API latency measured
- [ ] Consumer throughput measured
Availability
- [ ] Multiple consumer instances used where required
- [ ] Failure detection behavior tested
- [ ] Consumers distributed appropriately
- [ ] Downstream dependencies have availability strategy
- [ ] Shutdown is graceful
- [ ] Restart/recovery tested
Security
- [ ] Credentials stored securely
- [ ] Topic READ permissions minimal
- [ ] Group permissions correctly scoped
- [ ] TLS/SASL configuration validated
118. Troubleshooting Checklist
Symptom: Consumer Lag Growing
Check:
Consumer count
Partition count
Hot partitions
Processing latency
Database latency
API latency
Fetch latency
CPU
Memory
GC
Rebalances
Commit latency
Broker throttling
Symptom: Constant Rebalances
Check:
Consumer crashes
Deployment restarts
max.poll.interval
Long processing
Membership timeouts
Network stability
Protocol configuration
Rebalance callbacks
Symptom: Duplicate Processing
Check:
Crash after processing but before commit
Commit frequency
Retry behavior
Rebalance boundaries
Idempotency
Offset reset
Manual seeks
Duplicates are often expected in at-least-once systems.
Symptom: Missing Business Work
Check immediately:
Were offsets committed before processing?
Was auto.offset.reset=latest used for a new group?
Was seek() used incorrectly?
Did application drop errors?
Was downstream work asynchronous and not awaited?
Symptom: Consumers Idle
Check:
Number of partitions
Number of consumers in same group
Topic subscription
Authorization
Assignment
If:
consumers > partitions
idle members can be completely normal.
119. Interview Questions
What is a Kafka consumer?
A client application that fetches and processes records from Kafka topic partitions.
What is a Consumer Group?
A set of consumers sharing the same group identity and cooperating to divide partition consumption.
Can two consumers in the same group read the same partition simultaneously?
Under normal group assignment, one partition is assigned to at most one consumer in that group at a time.
What if consumers are in different groups?
Each group consumes independently, enabling fan-out.
What limits consumer parallelism?
Primarily the number of partitions available to the group at the Kafka partition-assignment level.
What is a consumer offset?
A numerical position in a partition used to track reading progress.
Current vs committed offset?
Current position is where the running consumer expects to read next. Committed offset is the saved recovery checkpoint for the group.
Where are group offsets stored?
Kafka stores committed group offsets in the internal __consumer_offsets topic.
What is rebalancing?
The process of changing partition ownership among consumers when group membership or subscribed partition metadata changes.
What happens when a consumer crashes?
Kafka detects the membership failure, reassigns its partitions, and replacement consumers resume using committed group progress.
What is fan-out?
Different consumer groups independently reading the same topic.
What is load balancing?
Consumers in the same group sharing partition ownership.
Why is commit timing important?
Because it determines whether failures may lead to duplicates or lost processing.
At-least-once?
Process first, then commit; failures may cause reprocessing.
At-most-once?
Commit before processing; failures can cause processing to be skipped.
Does manual commit guarantee exactly once?
No. Exactly-once requires coordinated transaction/idempotency design.
What is consumer lag?
How far a consumer/group is behind the available end of the partition stream.
120. Master Mental Model
A beginner says:
Consumer reads messages.
A Kafka engineer says:
Consumer starts
|
v
Connects to Kafka
|
v
Finds Group Coordinator
|
v
Joins Consumer Group
|
v
Participates in Group Protocol
|
v
Receives Partition Assignment
|
v
Finds Partition Leaders
|
v
Fetches Record Batches
|
v
Tracks Current Position
|
v
Deserializes
|
v
Processes Business Logic
|
v
Commits Safe Next Offset
|
v
__consumer_offsets
|
v
Handles Rebalances
|
v
Recovers From Failures
|
v
Continues Processing
A production Kafka engineer asks:
Is partition distribution balanced?
Can processing keep up?
What is the lag?
What happens when this consumer crashes?
What happens if commit succeeds but processing fails?
What happens if processing succeeds but commit fails?
Can duplicate processing hurt us?
Can any record be skipped?
How long does a rebalance take?
Are rebalances happening too often?
Do we have enough partitions?
Are we using the correct group protocol?
Are downstream systems idempotent?
Are offsets and business state coordinated correctly?
That is Kafka consumer mastery.
121. Final Takeaways
Remember these twelve ideas:
- Kafka consumers pull records from partition leaders.
- Consumers in the same group share partitions.
- Different consumer groups independently consume the same data.
- Partition count defines the main ceiling for partition-level consumer parallelism.
- Kafka balances workload by assigning partitions, not individual messages.
- Rebalancing changes partition ownership when membership or metadata changes.
- Modern Kafka has a newer Consumer group protocol as well as the older Classic protocol.
- Current position and committed offset are different.
- Committed offset represents the next position to resume/read.
- Commit timing determines duplicate-vs-loss behavior during failures.
- At-least-once designs should expect reprocessing and use idempotent business logic.
- Gold-standard consumer operations require monitoring lag, commits, rebalances, processing time, and downstream health.
122. Suggested Next Consumer Tutorials
After this foundation, continue with:
Kafka Consumer Internals
|
v
Consumer Group Protocol Deep Dive
|
v
Classic vs Modern Consumer Rebalancing
|
v
Partition Assignment Strategies
|
v
Offsets and Delivery Semantics
|
v
At-Least-Once / At-Most-Once / Exactly-Once
|
v
Consumer Performance Tuning Lab
|
v
Consumer Lag Monitoring
|
v
Rebalance Troubleshooting
|
v
Production Consumer Failure Testing
That path takes a student from “I can read a Kafka message” to “I can design, operate and troubleshoot a production Kafka consumer system.”
Technical Source Basis
This tutorial was checked against current Apache Kafka 4.3 consumer documentation/Javadocs and current Confluent consumer documentation for consumer groups, offset tracking, group coordination, rebalancing, fetch behavior, commit APIs, and consumer lag. URLs are intentionally omitted from this learning document.
