Developer Planning Guide for Correct Mapping, Scaling, Reliability and Performance
Audience: Developers, students, freshers, architects, platform engineers
Training context: Confluent Kafka Cluster
Goal: Remove confusion around how Kafka Topics, Partitions, Producers, Consumers, Consumer Groups, Replicas and Offsets map to each other, what Kafka allows, what it does not allow, and how to plan for consumer lag and scale safely.
1. The Five Kafka Decisions Developers Commonly Mix Together
Many Kafka discussions become confusing because five different design questions are treated as if they were one question.
They are not.
BUSINESS DATA
|
v
TOPIC
"What stream of events is this?"
|
v
PARTITIONS
"How much parallelism do we need,
and where are the ordering boundaries?"
|
v
CONSUMER GROUPS
"How many independent applications/use cases
need to consume this stream?"
|
v
CONSUMERS
"How much processing capacity does each
consumer group need?"
|
v
REPLICAS
"How many copies of each partition do we need
for durability and availability?"
The shortest possible explanation is:
Topic = logical event stream
Partition = unit of storage, ordering and parallelism
Consumer Group = independent logical subscriber/application
Consumer = worker inside a consumer group
Replica = redundant copy of a partition
Offset = position inside a partition
Lag = how far a consumer group is behind
If you remember only one section from this guide, remember the above.
2. The Master Mapping
A Kafka cluster can contain many topics.
A topic contains one or more partitions.
Each partition can have multiple replicas.
Each partition has one active leader replica at a time.
Producers write records to topic partitions.
Consumers read topic partitions.
Consumers with the same group.id form one Consumer Group.
Different Consumer Groups consume the same topic independently.
Kafka Cluster
|
+-- Topic A
| |
| +-- Partition 0
| | +-- Leader Replica
| | +-- Follower Replica
| | +-- Follower Replica
| |
| +-- Partition 1
| | +-- Leader Replica
| | +-- Follower Replica
| | +-- Follower Replica
| |
| +-- Partition 2
|
+-- Topic B
|
+-- Topic C
Consumption:
Topic A
P0 P1 P2
| | |
+------------+-------+-------+------------+
| |
v v
Consumer Group: analytics Consumer Group: alerts
Consumer A -> P0, P2 Consumer X -> P0, P1
Consumer B -> P1 Consumer Y -> P2
Both groups read the same topic independently.
Within each group, however, a partition is assigned to at most one consumer at a time.
3. Topic — What Problem Does It Solve?
A Topic is a named logical stream of records.
Examples:
orders
payments
vehicle-telemetry
customer-events
inventory-events
system-events
A topic should normally represent a meaningful business/event stream.
Do not think:
1 producer = 1 topic
or:
1 consumer = 1 topic
Neither rule exists.
A producer can write to multiple topics.
Many producers can write to the same topic.
A consumer can subscribe to multiple topics.
Many independent consumer groups can subscribe to the same topic.
4. How Many Topics Should We Create?
There is no universal Kafka rule such as:
"Always create 10 topics."
Topic count should follow architecture.
Create a separate topic when the stream needs meaningfully different:
Business meaning
Data contract/schema
Retention
Compaction policy
Security/ACL policy
Partitioning strategy
Throughput scaling
Ownership/team boundary
SLA/SLO
Lifecycle
Example:
orders
payments
shipments
may deserve different topics because they have different business meaning, schemas, security and lifecycle.
5. Bad Topic Design
Bad design: One topic for everything
all-events
OrderCreated
VehicleLocationChanged
PaymentAuthorized
UserLoggedIn
ServerRestarted
Possible problems:
Mixed schemas
Mixed ownership
Different retention needs
Difficult permissions
Difficult partitioning
Difficult consumer filtering
Large operational blast radius
Bad design: Topic per tiny event variation
order-created
order-updated
order-address-updated
order-item-added
order-item-removed
order-price-updated
...
This may create unnecessary topic proliferation.
A better design may be:
orders
with event type in the event envelope:
{
"event_type": "OrderCreated",
"order_id": "O-1001"
}
The correct boundary depends on data ownership, schema, volume, retention and access requirements.
6. Topic Planning Rule
Ask:
Does this stream need a different:
1. Business meaning?
2. Schema/data contract?
3. Retention policy?
4. Security policy?
5. Partition key?
6. Throughput profile?
7. Owner/team?
8. Compaction policy?
If several answers are YES, a separate topic is often appropriate.
7. Partition — The Most Important Scaling Unit
A Kafka Topic is split into one or more Partitions.
Example:
Topic: orders
P0
P1
P2
P3
P4
P5
Partitions provide:
Storage distribution
Parallel producer writes
Parallel consumer processing
Ordering boundaries
Replication units
Scalability
8. Ordering Is Per Partition
Kafka ordering should be understood as:
Ordering within one partition
not:
One global order across every partition in a topic
Example:
Partition 0
Offset 0 -> A
Offset 1 -> B
Offset 2 -> C
Kafka preserves the partition log order.
But:
P0 offset 100
P1 offset 100
have no global ordering relationship.
9. Keys Connect Producers to Partitions
A producer key is often used to keep related events together.
Example:
Topic = orders
Key = order_id
Then events for:
ORDER-1001
are normally routed consistently to the same partition according to the partitioning strategy.
This helps preserve per-order ordering.
OrderCreated
OrderPaid
OrderPacked
OrderShipped
|
v
Same order_id key
|
v
Same partition
|
v
Ordered processing for that order
10. Partition Count Controls Consumer Parallelism
This is one of the most important Kafka rules.
Suppose:
Topic: orders
Partitions = 6
One Consumer Group can have at most approximately six consumers doing partition-level work for this topic at the same time.
11. 6 Partitions + 1 Consumer
Topic
P0 P1 P2 P3 P4 P5
\ | | | | /
|
v
Consumer 1
Possible assignment:
Consumer 1 -> P0 P1 P2 P3 P4 P5
Allowed? YES. One consumer can own many partitions.
12. 6 Partitions + 2 Consumers
Possible assignment:
Consumer 1 -> P0 P1 P2
Consumer 2 -> P3 P4 P5
Allowed? YES. Both consumers are in the same group and share the work.
13. 6 Partitions + 3 Consumers
Possible assignment:
Consumer 1 -> P0 P3
Consumer 2 -> P1 P4
Consumer 3 -> P2 P5
Allowed? YES. Three consumer instances can process partitions in parallel.
14. 6 Partitions + 6 Consumers
Conceptually:
P0 -> C1
P1 -> C2
P2 -> C3
P3 -> C4
P4 -> C5
P5 -> C6
This gives the maximum simple partition-level parallelism for that one six-partition topic inside that Consumer Group.
15. 6 Partitions + 10 Consumers
Conceptually:
P0 -> C1
P1 -> C2
P2 -> C3
P3 -> C4
P4 -> C5
P5 -> C6
C7 -> idle
C8 -> idle
C9 -> idle
C10 -> idle
Allowed? YES.
Useful? Usually NO.
Extra consumers do not split one partition between multiple consumers in the same Consumer Group.
16. Golden Consumer Group Rule
Inside one normal Consumer Group:
One partition is assigned to at most one consumer in that group at a time.
This is the rule behind most consumer scaling discussions.
Important: it does not mean one partition can only have one consumer in the whole company.
Different Consumer Groups can independently consume that same partition.
17. Same Partition + Different Consumer Groups
Suppose:
Topic: orders
Partition: P0
Three applications need the data:
Analytics
Fraud Detection
Notifications
Use three groups:
group.id = analytics
group.id = fraud
group.id = notifications
Then:
orders / P0
/ | \
v v v
Analytics Fraud Notifications
Group Group Group
The same partition can therefore have one assigned reader per Consumer Group.
This is normal Kafka fan-out.
18. Consumer Group — What Problem Does It Solve?
A Consumer Group represents one logical consumption use case/application.
Examples:
order-processing
fraud-detection
analytics
notifications
billing
data-warehouse
search-indexer
Each group has its own progress.
Conceptually:
Topic: orders
Analytics Group:
P0 committed offset = 9000
Fraud Group:
P0 committed offset = 8970
Notification Group:
P0 committed offset = 9010
Each group moves independently.
19. Consumer Groups Are NOT Copies of Consumers
Do not think:
Consumer Group = one consumer
Instead:
Consumer Group
|
+-- Consumer 1
+-- Consumer 2
+-- Consumer 3
+-- Consumer N
A group is the logical application.
Consumers are the worker instances inside it.
20. How Many Consumer Groups Can Read a Topic?
Kafka architecture allows many independent Consumer Groups to consume the same topic.
Example:
+--> analytics group
|
+--> billing group
Topic: orders -------+
+--> fraud group
|
+--> notifications group
|
+--> warehouse group
There is no design rule that says:
1 topic = 1 Consumer Group
Multiple groups are a core Kafka feature.
However, every additional active group adds:
Read traffic
Consumer processing
Offset state
Network activity
Operational monitoring
So “allowed” does not mean “free.”
Cluster/provider capacity and quotas must be considered.
21. Can One Consumer Group Read Multiple Topics?
Yes.
Example:
Consumer Group: customer-360
Subscribes to:
customers
orders
payments
shipments
Kafka assigns partitions from the subscribed topics across members of the group.
Conceptually:
customers: P0 P1
orders: P0 P1 P2
payments: P0 P1
Total partition work = 7 partitions
With 3 consumers:
Consumer A -> some of the 7 partitions
Consumer B -> some of the 7 partitions
Consumer C -> some of the 7 partitions
Exact mapping depends on the group protocol and assignment strategy.
22. Can One Consumer Belong to Multiple Consumer Groups?
A single Kafka consumer instance has one:
group.id
and therefore participates in one Consumer Group at a time.
If one application process needs to participate in multiple groups, it can run multiple consumer instances.
Application Process
|
+-- KafkaConsumer A -> group.id=analytics
|
+-- KafkaConsumer B -> group.id=notifications
These are two separate consumer instances.
23. Can Multiple Producers Write to the Same Topic?
Yes.
Producer A -----\
Producer B ------> Topic: orders
Producer C -----/
This is completely normal.
24. Can One Producer Write to Multiple Topics?
Yes.
Producer
|
+--> orders
+--> audit-events
+--> system-events
Kafka does not require one producer instance per topic.
25. Replicas Are NOT Consumer Parallelism
A common confusion:
Partitions = parallelism
Replicas = durability / availability
Suppose:
Topic: orders
Partitions = 6
Replication Factor = 3
There are:
6 logical partitions
18 partition replicas
But a Consumer Group does not suddenly get 18-way normal processing parallelism.
The main partition-level consumer parallelism remains based on the six logical partitions.
26. Leader and Follower Relationship
For one partition:
Partition P0
Broker 1 -> Leader
Broker 2 -> Follower
Broker 3 -> Follower
Producer normally writes to the leader.
Consumer normally fetches from the partition leader in the standard model.
Followers replicate data for fault tolerance.
Therefore:
Add replicas
!=
Add consumer processing slots
27. Master Allowed / Not Allowed Table
| Question | Allowed? | Explanation |
|---|---|---|
| Many producers write to one topic | YES | Normal architecture |
| One producer writes to many topics | YES | Producer can publish to multiple topics |
| One topic has many partitions | YES | Main Kafka scaling model |
| One partition has multiple replicas | YES | Used for durability/availability |
| One consumer reads many partitions | YES | Very common |
| Multiple consumers in same group read same partition at the same time | NO | A partition is assigned to at most one member per group |
| Consumers in different groups read the same partition | YES | Core fan-out model |
| One Consumer Group reads multiple topics | YES | Group can subscribe to multiple topics |
| Many Consumer Groups read one topic | YES | Each group has independent progress |
| More consumers than partitions | YES | Extra consumers are normally idle |
| More partitions than consumers | YES | Consumers own multiple partitions |
| One consumer instance belongs to multiple group IDs simultaneously | NO | One consumer instance participates in one group |
| One process hosts multiple consumer instances with different groups | YES | Each KafkaConsumer instance is independent |
| Replicas increase Consumer Group parallelism | NO | Replicas provide durability, not logical partition parallelism |
| Offsets are global across a topic | NO | Offsets are per partition |
| Different groups share committed offsets | NO | Each group tracks its own offsets |
| Same group ID for unrelated apps that both need all events | TECHNICALLY YES, ARCHITECTURALLY WRONG | They will split partitions instead of both receiving all events |
| Decrease topic partition count directly | NO in normal Kafka operation | Usually requires new topic/migration |
| Increase partition count | YES | But affects partition mapping and must be planned carefully |
| Situation | Allowed? |
|---|---|
| 1 Consumer → 1 Partition | ✅ |
| 1 Consumer → Many Partitions | ✅ |
| 1 Partition → 2 Consumers in same group | ❌ |
| 1 Partition → Consumers in different groups | ✅ |
28. Fan-Out vs Load Balancing
Fan-Out
Same topic, different Consumer Groups.
Topic: orders
|
+--> group.id=analytics
|
+--> group.id=fraud
|
+--> group.id=notifications
Result:
Each group independently receives the stream.
Use when different applications need the same events.
Load Balancing
Same topic, same Consumer Group.
Topic: orders
|
+--> Consumer 1 \
+--> Consumer 2 > group.id=order-workers
+--> Consumer 3 /
Result:
Consumers share partition ownership.
Use when one logical application needs horizontal scaling.
29. Fan-Out + Load Balancing Together
Real Kafka systems commonly use both.
Topic: vehicle-telemetry
Analytics application:
group.id = analytics
Consumer A
Consumer B
Consumer C
Alert application:
group.id = alerts
Consumer A
Consumer B
Data warehouse application:
group.id = warehouse
Consumer A
Consumer B
Consumer C
Consumer D
Each group receives the complete logical stream independently.
Inside each group, its consumers divide the partitions.
30. Concrete Mapping Example
Suppose:
Topic: vehicle-telemetry
Partitions = 12
Analytics
group.id = analytics
Consumers = 6
Possible mapping:
6 consumers
12 partitions
~2 partitions per consumer
Alerts
group.id = alerts
Consumers = 3
Possible mapping:
3 consumers
12 partitions
~4 partitions per consumer
Data Warehouse
group.id = warehouse
Consumers = 12
Possible mapping:
12 consumers
12 partitions
~1 partition per consumer
All three groups independently consume all 12 partitions.
The groups do not steal partitions from each other.
31. What Happens If Analytics Has 20 Consumers?
Topic:
12 partitions
Analytics group:
20 consumers
Maximum partition assignments:
12
Therefore approximately:
12 consumers -> receive partitions
8 consumers -> idle
Adding more consumers does not help until more partition-level work is available.
32. Multiple Topics + One Group Example
Suppose:
Topic A = 4 partitions
Topic B = 6 partitions
Topic C = 2 partitions
One Consumer Group subscribes to all three.
Total partition assignments available:
4 + 6 + 2 = 12
If the group has six consumers, Kafka’s assignment strategy distributes those topic-partitions among the six members.
The exact distribution may not be perfectly two partitions each because topic subscription and assignment strategy matter.
The core invariant remains:
One topic-partition can be assigned to at most one consumer in the same group at a time.
33. Why Consumer Count Alone Does Not Determine Capacity
Suppose:
P0 = 1,000 records/sec
P1 = 1,000 records/sec
P2 = 50,000 records/sec
Group:
Consumer A -> P0
Consumer B -> P1
Consumer C -> P2
Kafka has assigned one partition to each consumer.
But Consumer C has vastly more work.
Kafka balances:
partition ownership
not:
perfect CPU workload
This is why producer partition-key design directly affects consumer performance.
34. Hot Partitions
A hot partition is a partition receiving or processing significantly more traffic than the others.
P0 ####
P1 ####
P2 ########################################
P3 #####
Possible causes:
Poor key distribution
One very large customer/entity
Low-cardinality key
Highly skewed business traffic
Custom partitioner bug
Impact:
One broker leader becomes busy
One consumer becomes busy
Group lag becomes concentrated
Adding consumers may not help
Because P2 cannot be split across two consumers inside the same group at the same time.
35. How Many Partitions Should We Create?
There is no universal correct number.
Partition count should consider:
Required producer throughput
Required consumer parallelism
Expected future growth
Ordering requirements
Broker capacity
Record size
Consumer processing rate
Failure recovery requirements
Operational overhead
A useful planning concept is:
Required Partitions
>=
maximum of:
1. partitions needed for producer throughput
2. partitions needed for consumer parallelism
3. partitions needed for expected future scale
But per-partition throughput is workload-specific.
You must benchmark.
Do not use random internet numbers as universal limits.
36. Simple Consumer Capacity Formula
Suppose:
Incoming rate = 60,000 records/sec
One measured consumer instance can safely process:
10,000 records/sec
Approximate required active consumer capacity:
60,000 / 10,000 = 6 consumers
For those six consumers to perform useful partition-level parallel work, the subscribed topic(s) need enough partitions.
For one topic:
Partitions >= 6
would be a minimum conceptual requirement.
For production headroom you normally want more capacity than the absolute minimum.
37. Do Not Size Consumers Before Measuring Them
A consumer’s capacity depends on business logic.
Consumer A:
Deserialize
Count event
may process far more records/sec than Consumer B:
Deserialize
Validate
Call HTTP API
Write database
Run business rules
Therefore:
"One Kafka consumer can process X records/sec"
is not a universal statement.
Benchmark your actual application.
38. Increasing Partitions Has Consequences
Increasing partitions can increase potential parallelism.
But it is not free.
Possible effects:
More metadata
More partition leaders
More replica logs
More file handles
More broker work
More consumer assignments
Potential rebalances
Changes to key-to-partition mapping
The last point is extremely important.
If a key is mapped using a partition-count-dependent hashing strategy, increasing the number of partitions can cause future records with the same key to map differently.
That can affect assumptions about historical per-key ordering across the partition-count change.
Plan partition growth carefully.
39. Partition Count Cannot Normally Be Reduced
Kafka supports increasing partitions.
Reducing the number of partitions of an existing topic is not a normal supported operation.
If you need fewer partitions, a common architectural approach is:
Create a new topic
|
v
Migrate/repartition data
|
v
Move producers/consumers
Partition count is therefore an important long-term design decision.
40. How Many Consumers Should a Consumer Group Have?
A practical starting rule:
Useful Consumer Count
<=
available partition assignments
For one topic:
Useful Consumer Count
<=
number of partitions
Then size actual consumers using measured processing capacity.
Example:
Topic partitions = 12
Measured workload requires 5 consumers
Start with five consumers, not automatically twelve, unless the additional processing headroom is justified.
41. Should Consumer Count Equal Partition Count?
Not necessarily.
12 partitions
3 consumers
may be completely healthy if each consumer can process four partitions comfortably.
Benefits can include:
Fewer application instances
Lower cost
Lower connection count
Simpler operations
Scale consumers when processing capacity requires it.
42. When Should We Add Consumers?
Add consumers to a Consumer Group when:
Lag is growing because consumers cannot keep up
AND
there are unexploited partitions available
Example:
Partitions = 12
Consumers = 4
Consumers are CPU-bound
Lag is increasing
You may scale toward six, eight, ten or twelve consumers while measuring improvement.
43. When Adding Consumers Will NOT Help
Case 1 — Consumers already equal partitions
Partitions = 6
Consumers = 6
Adding more consumers does not create more partition-level concurrency.
Case 2 — One hot partition
P0 lag = 0
P1 lag = 0
P2 lag = 500,000
Adding another consumer cannot split P2 inside the same group.
Case 3 — Downstream database is the bottleneck
Kafka Consumer
|
v
Database saturated
Adding more consumers may make the database problem worse.
44. Consumer Groups and Independent Applications
Use a new group.id when another logical application needs to receive all events independently.
Example:
orders topic
Application A:
analytics
Application B:
fraud
If both use the same group ID, they will share partitions.
If both applications require the complete stream, use separate group IDs.
45. Dangerous Group ID Mistake
Wrong:
Analytics Service
group.id = orders
Notification Service
group.id = orders
Result:
They become workers in the SAME logical Consumer Group.
Therefore they share partitions.
Correct:
Analytics:
group.id = order-analytics
Notifications:
group.id = order-notifications
Now both applications independently receive the topic.
46. What Is an Offset?
An offset is a position in a partition.
Partition P0
Offset 0 Record A
Offset 1 Record B
Offset 2 Record C
Offset 3 Record D
Offset 4 Record E
Offsets are per partition, not global per topic.
47. Current Position vs Committed Offset
Current Position
Where the running consumer expects to read next.
Committed Offset
The saved recovery checkpoint for the Consumer Group.
Stored in Kafka’s internal:
__consumer_offsets
Think:
Current position = where the running consumer is now
Committed offset = where the group can safely resume after failure
48. Example — Current vs Committed
0 1 2 3 4 5 6 7 8 9 10
^ ^
| |
committed current
Suppose:
Committed = 4
Current = 8
If the consumer crashes before committing newer progress, a replacement may resume near the committed point.
Records between the two positions may be processed again.
This is normal in at-least-once designs.
49. What Is Consumer Lag?
Consumer lag means:
How far a Consumer Group is behind the available end of a partition.
Simplified:
Latest available/end position
-
Consumer Group progress
=
Lag
Example:
Partition end = 10,000
Group committed = 9,400
Lag ≈ 600 records
50. Lag Belongs to Consumer Group + Topic + Partition
Do not think only:
Topic lag
The useful unit is:
Consumer Group
+
Topic
+
Partition
Example:
Group: analytics
Topic: orders
P0 lag = 0
P1 lag = 0
P2 lag = 50,000
P3 lag = 100
Total lag is 50,100, but the total hides the real problem: P2 is hot or stuck.
Always inspect lag per partition.
51. When Is Lag Created?
Lag grows whenever records arrive faster than the Consumer Group makes durable processing progress.
Simplified:
Producer Rate
>
Consumer Processing Rate
for long enough.
Example:
Producer = 50,000 records/sec
Consumer Group = 30,000 records/sec
Backlog growth:
20,000 records/sec
After 60 seconds:
~1,200,000 records behind
assuming rates remain constant.
52. Common Reasons Lag Is Created
- Too few consumers
- Too few partitions
- Consumer processing is slow
- Slow downstream database
- Slow external API
- Hot partition
- Consumer crashes
- Frequent rebalances
- Long garbage-collection pauses
max.poll.interval.msproblems- Broker/network throttling
- Inefficient fetch configuration
- Poison/bad records
- Downstream retry storms
- Commit failures or very infrequent commits causing a large committed-position gap
53. Lag Can Exist Even When Processing Is Fast
Suppose:
Consumer current position = 10,000
Committed offset = 8,000
The application may have processed much more than the committed checkpoint.
A monitoring system based on committed offsets may report substantial lag even though in-memory/current processing is further ahead.
Therefore ask:
Is this real processing lag?
or
Is processing healthy but commits are infrequent/failing?
Monitor both processing progress and commit health where possible.
54. A Temporary Lag Is Not Always a Problem
Traffic spike
|
v
Lag grows
|
v
Spike ends
|
v
Consumers process faster than new input
|
v
Lag returns to acceptable level
This may be acceptable if your SLO allows it.
The real question is:
Can the group catch up within the required time?
55. Persistent Lag Is a Capacity Signal
Bad pattern:
Time ----->
Lag:
10k
20k
40k
80k
160k
320k
The Consumer Group never catches up.
This means steady-state processing capacity is below workload demand or a persistent bottleneck exists.
56. Master Lag Troubleshooting Decision Tree
LAG IS GROWING
|
v
Is lag concentrated in one/few partitions?
|
+-- YES --> Investigate hot partition/key skew
| slow partition-specific data
| stuck records
| broker leader hotspot
|
+-- NO --> Lag across most partitions
|
v
Are consumers CPU-bound?
|
+------+------+
| |
YES NO
| |
v v
Scale consumers Check downstream
if partitions DB/API/network
allow it fetch/rebalance
Continue:
Do we have idle partition capacity?
|
+-- YES --> Add consumer instances and measure
|
+-- NO --> Consumers already near partition count
|
v
Optimize processing/downstream
|
v
If capacity still insufficient:
consider increasing partitions
+ consumers + cluster capacity
57. Lag Troubleshooting Step 1 — Check Per-Partition Lag
Do not start with:
"Add consumers."
First ask:
Which partitions are behind?
Example A:
P0 50k
P1 52k
P2 49k
P3 51k
Likely group-wide capacity issue.
Example B:
P0 0
P1 0
P2 205k
P3 0
Likely hot/stuck partition issue.
These need different solutions.
58. Lag Troubleshooting Step 2 — Compare Input vs Processing Rate
Measure:
Producer/input rate
and:
Consumer completion rate
If:
Input = 40k/sec
Processing = 60k/sec
the group should normally catch up.
If:
Input = 60k/sec
Processing = 40k/sec
lag grows permanently.
59. Lag Troubleshooting Step 3 — Check Consumer Count vs Partitions
Example:
Partitions = 12
Consumers = 3
If each consumer is overloaded, scale upward while measuring.
But if:
Partitions = 12
Consumers = 12
consumer 13 usually does not add partition-level capacity.
60. Lag Troubleshooting Step 4 — Check Processing Time
Measure:
Time per record
Time per batch
Database latency
API latency
CPU
Memory
GC
Thread pool saturation
Often Kafka is not the bottleneck.
Example:
Kafka fetch = 15 ms
Database write = 800 ms
Tuning Kafka fetch parameters will not fix the primary problem.
61. Lag Troubleshooting Step 5 — Check Rebalances
Frequent rebalances can cause:
Temporary processing interruption
Partition movement
Cache warm-up
State reload
Duplicate processing near commit boundaries
Inspect:
Consumer crashes
Rolling deployments
max.poll.interval.ms
Long processing
Membership/session behavior
Network instability
Group protocol
Application shutdown behavior
62. Lag Troubleshooting Step 6 — Check Hot Keys
If one partition is overloaded, inspect producer key distribution.
Example:
Key = customer_id
One customer generates 40% of all traffic.
All of that customer’s records map to one partition.
Adding consumers will not split that partition.
Potential solutions depend on ordering requirements:
Improve key distribution
Shard a high-volume entity if business semantics permit
Use another partitioning strategy
Separate extreme workloads
Increase processing efficiency for that partition
Do not destroy required ordering merely to spread load.
63. Lag Troubleshooting Step 7 — Check Fetch and Poll Behavior
Important consumer settings include:
fetch.min.bytes
fetch.max.wait.ms
fetch.max.bytes
max.partition.fetch.bytes
max.poll.records
max.poll.interval.ms
These influence:
Throughput
Latency
Memory
Work per poll
Failure/rebalance behavior
Tune only after identifying a real bottleneck.
64. max.poll.records and Lag
Suppose:
max.poll.records = 500
and each record requires one second of serial processing.
One poll could represent roughly:
500 seconds
of work.
That may create long processing cycles, poll interval risk, rebalances and lag.
Possible improvements:
Reduce max.poll.records
Optimize processing
Batch downstream calls
Scale consumers
Increase partitions when justified
65. Database Batching Can Reduce Lag
Bad:
500 Kafka records
=
500 individual database network calls
Potentially better:
500 Kafka records
|
v
Batch database operation
if business correctness permits.
Kafka consumer performance often depends more on downstream batching than on Kafka itself.
66. Backpressure — Do Not Overload Downstream Systems
Suppose Kafka delivers data faster than your database can safely accept it.
Do not blindly keep increasing consumers.
You may overload the database.
Better architecture may include:
Controlled consumer concurrency
Batching
pause()/resume()
Bounded worker queues
Rate limiting
Downstream capacity scaling
A healthy Kafka consumer should not destroy the system it feeds.
67. Do Not “Fix” Lag by Resetting Offsets
Dangerous response:
"Consumer is 5 million records behind.
Reset offsets to latest."
This may make lag disappear because you skipped the backlog.
But the business data was not processed.
That is not lag remediation. That is data skipping.
Only reset/seek past data when the business explicitly decides the old data can be discarded.
68. auto.offset.reset=latest Does Not Solve Existing Healthy Group Lag
auto.offset.reset is mainly relevant when there is no usable committed offset.
It is not the normal control for a healthy group that already has committed progress.
Do not teach:
"Set latest to fix lag."
That confuses offset initialization/recovery with processing capacity.
69. Offset Commit Strategy and Lag
If you commit every record synchronously:
Process one
Commit
Wait
Process one
Commit
Wait
commit overhead can reduce throughput and increase lag.
If you never commit for a long time, processing may be fast but the recovery checkpoint stays far behind.
A balanced production pattern often commits safe progress in batches.
The exact strategy depends on:
Business correctness
Duplicate tolerance
Processing time
Commit cost
Failure recovery objectives
70. At-Least-Once and Lag Recovery
A common robust pattern is:
poll
|
v
process
|
v
business operation succeeds
|
v
commit safe next offset
After a crash, some records may be processed again.
Therefore downstream processing should be idempotent where practical.
71. Poison Messages and Lag
Suppose one record repeatedly fails:
Offset 500
Application behavior:
Read 500
Fail
Retry
Fail
Retry
Fail
...
Everything behind it may stop progressing, depending on application error handling.
Lag grows.
Production designs need a failure strategy such as:
Bounded retries
Retry topic/pattern
Dead-letter topic where appropriate
Alerting
Manual investigation
Idempotent replay
Never create an infinite silent retry loop.
72. Rebalance and Lag
During a rebalance, partition ownership changes.
Older/eager rebalance behavior can temporarily pause substantial group processing.
Modern cooperative/incremental approaches can reduce disruption, depending on the Consumer Group protocol and assignment strategy.
Regardless of protocol:
Frequent rebalances
=
less useful processing time
=
potential lag
Monitor rebalance frequency and duration.
73. Failover and Lag
Example:
P0 -> Consumer A
P1 -> Consumer B
P2 -> Consumer C
Consumer B crashes.
Until Kafka detects the failure and reassigns P1:
P1 processing pauses
P1 lag grows.
After reassignment, the new owner resumes from the Consumer Group’s committed offset.
If spare processing capacity exists, it can catch up.
74. Availability Requires Spare Capacity
If every consumer normally runs at 95% CPU and one consumer fails, its partitions move to surviving consumers.
The survivors may not have enough spare capacity.
Result:
failover succeeds technically
but lag grows dramatically
Production planning should include failure headroom.
Do not size only for healthy-state average load.
75. Developer Planning Matrix
| Decision | Primary Kafka Object | Key Question |
|---|---|---|
| Event-stream boundary | Topic | What business stream/data contract is this? |
| Ordering | Key + Partition | Which events must remain ordered together? |
| Parallelism | Partitions | How many independent partition workloads are needed? |
| Independent use cases | Consumer Groups | Which applications each need the full stream? |
| Processing scale | Consumers | How many worker instances does each group need? |
| Durability | Replicas | How many broker failures should data tolerate? |
| Recovery position | Offsets | Where should this group resume after failure? |
| Backlog health | Lag | Is processing keeping up with input? |
76. Planning Example — Order Platform
Requirements:
50,000 orders/sec peak
Ordering required per order_id
Three independent applications:
- fulfillment
- fraud
- analytics
Design:
Topic:
orders
Key:
order_id
Partitions:
capacity-planned based on throughput and consumer parallelism
Replication:
chosen for durability/availability
Consumer Groups:
order-fulfillment
order-fraud
order-analytics
Within fulfillment, consumers scale according to processing demand up to useful partition-level parallelism.
Fraud and analytics scale independently.
77. Planning Example — Vehicle Telematics
Requirements:
500,000 vehicles
Frequent telemetry
Ordering desired per vehicle
Consumers:
- real-time alerts
- trip processing
- analytics
- data warehouse
Possible architecture:
Topic:
vehicle-telemetry
Key:
vehicle_id
Partitions:
planned for ingestion + consumer concurrency
Groups:
vehicle-alerts
trip-engine
telemetry-analytics
telemetry-warehouse
Each group independently reads the topic.
A problem in warehouse processing does not directly stop alerts from consuming.
78. Example Mapping — 24 Partitions
Suppose:
vehicle-telemetry = 24 partitions
Alert group:
6 consumers
~4 partitions/consumer
Trip group:
12 consumers
~2 partitions/consumer
Analytics group:
24 consumers
~1 partition/consumer
Warehouse group:
4 consumers
~6 partitions/consumer
Allowed? YES.
Every group gets independent assignments across the same 24 partitions.
79. Can We Have 100 Consumer Groups?
Architecturally, yes: Kafka supports many independent groups.
But do not use an arbitrary number without thinking about:
Read throughput
Connections
Coordinator load
Offset state
Monitoring
Authorization
Cost
Confluent/provider quotas
Broker/network capacity
There is no universal best-practice number that applies to every cluster.
Capacity planning matters more than memorizing a magic limit.
80. Can We Have 1,000 Topics?
The same principle applies.
Kafka can support large numbers of topics/partitions depending on cluster sizing and platform limits.
But the meaningful capacity number is often influenced heavily by:
Total partitions
Replication factor
Traffic
Retention
Storage
Broker count
Controller/metadata capacity
Client count
Provider quotas
Do not ask only:
"How many topics?"
Ask:
"How many total partitions, replicas, bytes/sec,
connections and retained bytes does this design create?"
81. Total Partition Replica Count
Example:
100 topics
20 partitions each
Replication Factor = 3
Logical partitions:
100 * 20 = 2,000 partitions
Replica copies:
2,000 * 3 = 6,000 partition replicas
This is more meaningful operationally than saying:
"We only have 100 topics."
82. Do Not Confuse Broker Count With Partition Count
Example:
3 brokers
12 partitions
Perfectly normal.
Partitions are distributed across brokers.
You do not need 12 brokers for 12 partitions.
Broker count is driven by:
Throughput
Storage
Availability
Failure-domain requirements
Replication
Network
CPU
83. Do Not Confuse Consumer Count With Broker Count
There is no rule:
1 consumer per broker
Example:
3 Kafka brokers
24 topic partitions
12 consumers in a group
is completely possible.
Broker count and consumer count solve different problems.
84. Do Not Confuse Consumer Group Count With Partition Count
There is no rule:
1 group per partition
One topic with six partitions may have one, ten or fifty groups depending on independent applications.
Each group tracks its own offsets across those partitions.
85. Master Mapping Rules
Memorize these:
1 Cluster
-> many Topics
1 Topic
-> many Partitions
1 Partition
-> multiple Replicas
1 Partition
-> one Leader at a time
Many Producers
-> may write to same Topic
1 Producer
-> may write to many Topics
1 Consumer Group
-> may subscribe to many Topics
1 Topic
-> may be consumed by many Consumer Groups
1 Consumer
-> may own many Partitions
1 Partition
-> at most one Consumer per Consumer Group at a time
Same Partition
-> may be read independently by many different Consumer Groups
Consumers > Partitions
-> extra consumers idle in that group
Partitions > Consumers
-> consumers own multiple partitions
Replicas
-> durability / availability
Partitions
-> parallelism / ordering / scaling
Offsets
-> group progress
Lag
-> backlog
86. What Is Allowed vs What Is Good Design?
Kafka permits many configurations that are not good architecture.
Example:
4 partitions
40 consumers
Allowed: YES.
Good design: usually NO, because most consumers are idle.
Another example:
Same group.id for analytics and billing
Allowed: YES.
Correct if both must independently read every record: NO.
Kafka does not know your business requirement.
Developers must design the mapping correctly.
87. Best-Practice Topic Checklist
Before creating a topic:
- [ ] Is the business purpose clear?
- [ ] Is ownership clear?
- [ ] Is the event schema/data contract clear?
- [ ] Is the partition key clear?
- [ ] Is ordering requirement documented?
- [ ] Is partition count capacity-planned?
- [ ] Is retention defined?
- [ ] Is compaction required?
- [ ] Is security/ACL policy defined?
- [ ] Is replication/durability requirement defined?
- [ ] Is expected throughput documented?
- [ ] Is growth estimate documented?
- [ ] Are consumers/use cases known?
88. Best-Practice Partition Checklist
Before choosing partition count:
- [ ] Measure expected producer throughput
- [ ] Measure expected consumer processing throughput
- [ ] Determine required consumer parallelism
- [ ] Document per-key ordering requirements
- [ ] Check key cardinality and skew
- [ ] Include growth headroom
- [ ] Include failure headroom
- [ ] Consider broker capacity
- [ ] Consider replication overhead
- [ ] Consider operational overhead
- [ ] Understand that partition count is difficult to reduce
- [ ] Understand that increasing partitions may change key mapping
89. Best-Practice Consumer Group Checklist
For each Consumer Group:
- [ ] One clear logical application/use case
- [ ] Intentional and unique
group.id - [ ] Topic subscriptions documented
- [ ] Consumer count based on measured processing demand
- [ ] Consumer count compared with partition count
- [ ] Rebalance behavior understood
- [ ] Offset commit strategy defined
- [ ]
auto.offset.resetintentional - [ ] Lag SLO defined
- [ ] Lag monitored per partition
- [ ] Downstream dependencies monitored
- [ ] Failure/restart behavior tested
- [ ] Duplicate-processing behavior understood
- [ ] Idempotency implemented where appropriate
90. Best-Practice Lag Checklist
When lag increases:
- [ ] Check lag per group/topic/partition
- [ ] Check producer/input rate
- [ ] Check consumer completion rate
- [ ] Check current vs committed progress
- [ ] Check consumer count
- [ ] Check partition count
- [ ] Check for hot partitions
- [ ] Check CPU
- [ ] Check memory / GC
- [ ] Check database latency
- [ ] Check API latency
- [ ] Check fetch latency
- [ ] Check commit latency/errors
- [ ] Check rebalances
- [ ] Check broker throttling
- [ ] Check network issues
- [ ] Check poison records
- [ ] Check retry loops
- [ ] Check max poll behavior
- [ ] Scale only after identifying the bottleneck
91. Anti-Patterns
Avoid:
"More consumers always fix lag."
Wrong.
Avoid:
"Replicas give more consumer parallelism."
Wrong.
Avoid:
"Every service should use the same group.id."
Wrong when services need independent copies of the stream.
Avoid:
"One topic for every producer."
Not a Kafka rule.
Avoid:
"One consumer for every partition is always best."
Not necessarily.
Avoid:
"Reset offsets to latest whenever lag is high."
Dangerous. It can skip business data.
Avoid:
"Increase partitions whenever performance is slow."
First identify the bottleneck.
92. Developer Decision Flow
Use this sequence when designing a new Kafka stream.
STEP 1
What business events are we publishing?
|
v
Define Topic
STEP 2
Which events must remain ordered together?
|
v
Choose Key
STEP 3
What throughput and consumer parallelism do we need?
|
v
Plan Partitions
STEP 4
How much broker failure should data survive?
|
v
Choose Replication / durability policy
STEP 5
Which independent applications need the data?
|
v
Create Consumer Groups
STEP 6
How much processing capacity does each group need?
|
v
Choose Consumer Count per Group
STEP 7
What are the processing correctness requirements?
|
v
Choose Offset Commit / Delivery Strategy
STEP 8
What backlog is acceptable?
|
v
Define Lag SLO + Alerts
STEP 9
Load test and failure test
|
v
Tune based on measurements
93. Example Developer Worksheet
Topic
orders
Business owner
Order Platform Team
Key
order_id
Ordering requirement
Events for one order must stay ordered.
Peak input
80,000 records/sec
Partitions
To be selected after throughput and consumer benchmarks.
Independent groups
order-fulfillment
order-fraud
order-analytics
order-notifications
Fulfillment consumer capacity
Measured: 8,000 records/sec per consumer
Approximate consumer requirement at 80k:
80,000 / 8,000 = 10 active consumers
Therefore the topic must expose enough partition-level concurrency for the desired group scaling.
Add capacity/failure headroom after benchmarking.
94. Example Lag SLO
Do not monitor lag without defining what “bad” means.
Possible SLO:
Normal:
< 5,000 records lag per partition
Warning:
lag > 20,000 for 5 minutes
Critical:
lag > 100,000
or
estimated time-behind > 2 minutes
For some workloads, time lag is more meaningful than raw record count.
Ten thousand tiny telemetry records may be trivial, while ten thousand expensive payment events may be a serious backlog.
Measure what matters to the business.
95. Lag Recovery Capacity
A healthy group needs enough spare capacity to catch up.
Suppose:
Normal input = 50k/sec
Consumer max = 52k/sec
Spare capacity:
2k/sec
A backlog of:
1,000,000 records
would take approximately:
1,000,000 / 2,000 = 500 seconds ≈ 8.3 minutes
to clear if rates remain stable.
If your recovery SLO is two minutes, this design has insufficient catch-up capacity.
96. Plan for Failure, Not Only Average Load
Suppose:
12 partitions
6 consumers
Each consumer processes about two partitions.
If one consumer fails, remaining members may receive its partitions.
If the remaining consumers are already saturated:
failover
|
v
work gets reassigned
|
v
but capacity is insufficient
|
v
lag grows
Production sizing should include spare failure capacity.
97. The Four Most Important Distinctions
Distinction 1
Topics != Partitions
Topic is logical stream. Partition is scaling/order/storage unit.
Distinction 2
Partitions != Replicas
Partitions create parallelism. Replicas create durability/availability.
Distinction 3
Consumers != Consumer Groups
Consumer is a worker. Consumer Group is the logical application/subscriber.
Distinction 4
Lag != Kafka is broken
Lag means the group is behind.
The root cause may be Kafka, consumer code, database, API, partition skew, CPU, network, rebalancing or commit strategy.
Measure before tuning.
98. One Diagram to Remember Everything
PRODUCERS
|
| many producers allowed
v
+---------------------------------------------------+
| TOPIC: orders |
| |
| P0 P1 P2 P3 P4 |
| | | | | | |
| replicas replicas replicas replicas replicas |
+---------------------------------------------------+
| |
| |
v v
Consumer Group A Consumer Group B
"fulfillment" "analytics"
C1 -> P0,P2 C1 -> P0
C2 -> P1,P3 C2 -> P1
C3 -> P4 C3 -> P2
C4 -> P3
C5 -> P4
Each group:
- gets independent access to all partitions
- has its own committed offsets
- has its own lag
- scales consumers independently
Within one group:
- one partition -> max one assigned consumer at a time
Across different groups:
- same partition -> many independent readers are normal
99. Quick Answers to Session Questions
How many topics can a cluster have?
Many. There is no one universal correct number. Total partitions, replicas, throughput, storage, metadata and provider limits matter more than topic count alone.
How many partitions can a topic have?
One or many. Choose based on throughput, consumer parallelism, ordering and operational capacity.
How many consumers can read one topic?
Across different Consumer Groups: many.
Inside one Consumer Group: many consumers may exist, but useful partition-level concurrency is limited by the available partitions.
Can one consumer read many partitions?
Yes.
Can two consumers in the same group read one partition simultaneously?
Normally no. One partition is assigned to at most one consumer in that group at a time.
Can two consumers in different groups read the same partition?
Yes.
Can one Consumer Group read many topics?
Yes.
Can one topic have many Consumer Groups?
Yes.
Does replication factor increase consumer parallelism?
No.
Do more consumers always reduce lag?
No.
Do more partitions always improve performance?
No.
Can we reduce partitions later?
Not as a normal direct Kafka operation. Plan carefully.
What is lag?
The distance between the available end of a partition and a Consumer Group’s processing/progress position.
Where should we look first when lag grows?
Per-partition lag, processing rate, consumer count, partition count, downstream latency, rebalances and hot partitions.
100. Final Master Rules
Remember these ten rules:
- Topic = business stream.
- Partition = ordering + parallelism + storage unit.
- Replica = durability and availability, not consumer concurrency.
- Consumer Group = one independent logical subscriber/application.
- Consumer = worker inside the group.
- One partition can be assigned to only one consumer per group at a time.
- Different groups can independently consume the same partition.
- Consumer parallelism is bounded by partition-level work.
- Lag grows when production outpaces durable consumer progress.
- Fix lag by finding the bottleneck—not by blindly adding consumers or skipping offsets.
101. Recommended Planning Order for Developers
Use this order in every architecture discussion:
1. Business Event
2. Topic
3. Key
4. Ordering Requirement
5. Peak Throughput
6. Partition Count
7. Replication / Durability
8. Independent Consumer Groups
9. Consumers Per Group
10. Offset Strategy
11. Lag SLO
12. Monitoring
13. Load Test
14. Failure Test
15. Tune
This order prevents most of the confusion that happens when teams start by asking:
"How many consumers should we create?"
before they have defined topics, keys, partitions, workloads and correctness requirements.
102. Final Developer Mental Model
Do not ask these as isolated questions:
How many Kafka topics are allowed?
How many partitions are allowed?
How many consumers are allowed?
Ask:
What workload are we designing?
What ordering do we need?
What throughput do we need?
How many independent applications need the stream?
How quickly must every group process it?
What failure should the system survive?
What lag/recovery SLO do we have?
Then map:
Business streams
-> Topics
Ordering + throughput
-> Keys + Partitions
Independent use cases
-> Consumer Groups
Processing capacity
-> Consumers per Group
Durability
-> Replication
Recovery progress
-> Offsets
Capacity health
-> Lag
That is the cleanest way to plan Kafka without mixing unrelated concepts.
