{"id":1154,"date":"2026-08-19T07:47:50","date_gmt":"2026-08-19T07:47:50","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/?p=1154"},"modified":"2026-08-19T07:54:54","modified_gmt":"2026-08-19T07:54:54","slug":"kafka-master-tutorials-series-7-topics-partitions-consumers-consumer-groups-lag","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/kafka-master-tutorials-series-7-topics-partitions-consumers-consumer-groups-lag\/","title":{"rendered":"Kafka Master Tutorials Series: 7 Topics, Partitions, Consumers, Consumer Groups & Lag"},"content":{"rendered":"\n

Developer Planning Guide for Correct Mapping, Scaling, Reliability and Performance<\/h2>\n\n\n\n
\n

Audience:<\/strong> Developers, students, freshers, architects, platform engineers
Training context:<\/strong> Confluent Kafka Cluster
Goal:<\/strong> Remove confusion around how Kafka Topics, Partitions, Producers, Consumers, Consumer Groups, Replicas and Offsets map to each other, what Kafka allows, what it does not allow, and how to plan for consumer lag and scale safely.<\/p>\n<\/blockquote>\n\n\n\n

\n\n\n\n

1. The Five Kafka Decisions Developers Commonly Mix Together<\/h1>\n\n\n\n

Many Kafka discussions become confusing because five different design questions are treated as if they were one question.<\/p>\n\n\n\n

They are not.<\/p>\n\n\n\n

BUSINESS DATA\n     |\n     v\nTOPIC\n\"What stream of events is this?\"\n\n     |\n     v\nPARTITIONS\n\"How much parallelism do we need,\nand where are the ordering boundaries?\"\n\n     |\n     v\nCONSUMER GROUPS\n\"How many independent applications\/use cases\nneed to consume this stream?\"\n\n     |\n     v\nCONSUMERS\n\"How much processing capacity does each\nconsumer group need?\"\n\n     |\n     v\nREPLICAS\n\"How many copies of each partition do we need\nfor durability and availability?\"\n<\/code><\/pre>\n\n\n\n
The shortest possible explanation is:<\/p>\n\n\n\n
Topic          = logical event stream\nPartition      = unit of storage, ordering and parallelism\nConsumer Group = independent logical subscriber\/application\nConsumer       = worker inside a consumer group\nReplica        = redundant copy of a partition\nOffset         = position inside a partition\nLag            = how far a consumer group is behind\n<\/code><\/pre>\n\n\n\n
If you remember only one section from this guide, remember the above.<\/p>\n\n\n\n
\n\n\n\n
2. The Master Mapping<\/h1>\n\n\n\n
A Kafka cluster can contain many topics.<\/p>\n\n\n\n
A topic contains one or more partitions.<\/p>\n\n\n\n
Each partition can have multiple replicas.<\/p>\n\n\n\n
Each partition has one active leader replica at a time.<\/p>\n\n\n\n
Producers write records to topic partitions.<\/p>\n\n\n\n
Consumers read topic partitions.<\/p>\n\n\n\n
Consumers with the same group.id<\/code> form one Consumer Group.<\/p>\n\n\n\n
Different Consumer Groups consume the same topic independently.<\/p>\n\n\n\n
Kafka Cluster\n|\n+-- Topic A\n|   |\n|   +-- Partition 0\n|   |   +-- Leader Replica\n|   |   +-- Follower Replica\n|   |   +-- Follower Replica\n|   |\n|   +-- Partition 1\n|   |   +-- Leader Replica\n|   |   +-- Follower Replica\n|   |   +-- Follower Replica\n|   |\n|   +-- Partition 2\n|\n+-- Topic B\n|\n+-- Topic C\n<\/code><\/pre>\n\n\n\n
Consumption:<\/p>\n\n\n\n
                         Topic A\n                   P0      P1      P2\n                    |       |       |\n       +------------+-------+-------+------------+\n       |                                         |\n       v                                         v\n\nConsumer Group: analytics                Consumer Group: alerts\n\nConsumer A -> P0, P2                     Consumer X -> P0, P1\nConsumer B -> P1                         Consumer Y -> P2\n<\/code><\/pre>\n\n\n\n
Both groups read the same topic independently.<\/p>\n\n\n\n
Within each group, however, a partition is assigned to at most one consumer at a time.<\/p>\n\n\n\n
\n\n\n\n
3. Topic \u2014 What Problem Does It Solve?<\/h1>\n\n\n\n
A Topic<\/strong> is a named logical stream of records.<\/p>\n\n\n\n
Examples:<\/p>\n\n\n\n
orders\npayments\nvehicle-telemetry\ncustomer-events\ninventory-events\nsystem-events\n<\/code><\/pre>\n\n\n\n
A topic should normally represent a meaningful business\/event stream.<\/p>\n\n\n\n
Do not think:<\/p>\n\n\n\n
1 producer = 1 topic\n<\/code><\/pre>\n\n\n\n
or:<\/p>\n\n\n\n
1 consumer = 1 topic\n<\/code><\/pre>\n\n\n\n
Neither rule exists.<\/p>\n\n\n\n
A producer can write to multiple topics.<\/p>\n\n\n\n
Many producers can write to the same topic.<\/p>\n\n\n\n
A consumer can subscribe to multiple topics.<\/p>\n\n\n\n
Many independent consumer groups can subscribe to the same topic.<\/p>\n\n\n\n
\n\n\n\n
4. How Many Topics Should We Create?<\/h1>\n\n\n\n
There is no universal Kafka rule such as:<\/p>\n\n\n\n
\"Always create 10 topics.\"\n<\/code><\/pre>\n\n\n\n
Topic count should follow architecture.<\/p>\n\n\n\n
Create a separate topic when the stream needs meaningfully different:<\/p>\n\n\n\n
Business meaning\nData contract\/schema\nRetention\nCompaction policy\nSecurity\/ACL policy\nPartitioning strategy\nThroughput scaling\nOwnership\/team boundary\nSLA\/SLO\nLifecycle\n<\/code><\/pre>\n\n\n\n
Example:<\/p>\n\n\n\n
orders\npayments\nshipments\n<\/code><\/pre>\n\n\n\n
may deserve different topics because they have different business meaning, schemas, security and lifecycle.<\/p>\n\n\n\n
\n\n\n\n
5. Bad Topic Design<\/h1>\n\n\n\n
Bad design: One topic for everything<\/h2>\n\n\n\n
all-events\n\nOrderCreated\nVehicleLocationChanged\nPaymentAuthorized\nUserLoggedIn\nServerRestarted\n<\/code><\/pre>\n\n\n\n
Possible problems:<\/p>\n\n\n\n
Mixed schemas\nMixed ownership\nDifferent retention needs\nDifficult permissions\nDifficult partitioning\nDifficult consumer filtering\nLarge operational blast radius\n<\/code><\/pre>\n\n\n\n
Bad design: Topic per tiny event variation<\/h2>\n\n\n\n
order-created\norder-updated\norder-address-updated\norder-item-added\norder-item-removed\norder-price-updated\n...\n<\/code><\/pre>\n\n\n\n
This may create unnecessary topic proliferation.<\/p>\n\n\n\n
A better design may be:<\/p>\n\n\n\n
orders\n<\/code><\/pre>\n\n\n\n
with event type in the event envelope:<\/p>\n\n\n\n
{\n  \"event_type\": \"OrderCreated\",\n  \"order_id\": \"O-1001\"\n}\n<\/code><\/pre>\n\n\n\n
The correct boundary depends on data ownership, schema, volume, retention and access requirements.<\/p>\n\n\n\n
\n\n\n\n
6. Topic Planning Rule<\/h1>\n\n\n\n
Ask:<\/p>\n\n\n\n
Does this stream need a different:\n\n1. Business meaning?\n2. Schema\/data contract?\n3. Retention policy?\n4. Security policy?\n5. Partition key?\n6. Throughput profile?\n7. Owner\/team?\n8. Compaction policy?\n<\/code><\/pre>\n\n\n\n
If several answers are YES, a separate topic is often appropriate.<\/p>\n\n\n\n
\n\n\n\n
7. Partition \u2014 The Most Important Scaling Unit<\/h1>\n\n\n\n
A Kafka Topic is split into one or more Partitions<\/strong>.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
Topic: orders\n\nP0\nP1\nP2\nP3\nP4\nP5\n<\/code><\/pre>\n\n\n\n
Partitions provide:<\/p>\n\n\n\n
Storage distribution\nParallel producer writes\nParallel consumer processing\nOrdering boundaries\nReplication units\nScalability\n<\/code><\/pre>\n\n\n\n
\n\n\n\n
8. Ordering Is Per Partition<\/h1>\n\n\n\n
Kafka ordering should be understood as:<\/p>\n\n\n\n
Ordering within one partition\n<\/code><\/pre>\n\n\n\n
not:<\/p>\n\n\n\n
One global order across every partition in a topic\n<\/code><\/pre>\n\n\n\n
Example:<\/p>\n\n\n\n
Partition 0\n\nOffset 0 -> A\nOffset 1 -> B\nOffset 2 -> C\n<\/code><\/pre>\n\n\n\n
Kafka preserves the partition log order.<\/p>\n\n\n\n
But:<\/p>\n\n\n\n
P0 offset 100\nP1 offset 100\n<\/code><\/pre>\n\n\n\n
have no global ordering relationship.<\/p>\n\n\n\n
\n\n\n\n
9. Keys Connect Producers to Partitions<\/h1>\n\n\n\n
A producer key is often used to keep related events together.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
Topic = orders\nKey   = order_id\n<\/code><\/pre>\n\n\n\n
Then events for:<\/p>\n\n\n\n
ORDER-1001\n<\/code><\/pre>\n\n\n\n
are normally routed consistently to the same partition according to the partitioning strategy.<\/p>\n\n\n\n
This helps preserve per-order ordering.<\/p>\n\n\n\n
OrderCreated\nOrderPaid\nOrderPacked\nOrderShipped\n\n       |\n       v\n\nSame order_id key\n\n       |\n       v\n\nSame partition\n\n       |\n       v\n\nOrdered processing for that order\n<\/code><\/pre>\n\n\n\n
\n\n\n\n
10. Partition Count Controls Consumer Parallelism<\/h1>\n\n\n\n
This is one of the most important Kafka rules.<\/p>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Topic: orders\nPartitions = 6\n<\/code><\/pre>\n\n\n\n
One Consumer Group can have at most approximately six consumers doing partition-level work for this topic at the same time.<\/p>\n\n\n\n
\n\n\n\n
11. 6 Partitions + 1 Consumer<\/h1>\n\n\n\n
Topic\n\nP0 P1 P2 P3 P4 P5\n \\  |  |  |  |  \/\n        |\n        v\n   Consumer 1\n<\/code><\/pre>\n\n\n\n
Possible assignment:<\/p>\n\n\n\n
Consumer 1 -> P0 P1 P2 P3 P4 P5\n<\/code><\/pre>\n\n\n\n
Allowed? YES.<\/strong> One consumer can own many partitions.<\/p>\n\n\n\n
\n\n\n\n
12. 6 Partitions + 2 Consumers<\/h1>\n\n\n\n
Possible assignment:<\/p>\n\n\n\n
Consumer 1 -> P0 P1 P2\nConsumer 2 -> P3 P4 P5\n<\/code><\/pre>\n\n\n\n
Allowed? YES.<\/strong> Both consumers are in the same group and share the work.<\/p>\n\n\n\n
\n\n\n\n
13. 6 Partitions + 3 Consumers<\/h1>\n\n\n\n
Possible assignment:<\/p>\n\n\n\n
Consumer 1 -> P0 P3\nConsumer 2 -> P1 P4\nConsumer 3 -> P2 P5\n<\/code><\/pre>\n\n\n\n
Allowed? YES.<\/strong> Three consumer instances can process partitions in parallel.<\/p>\n\n\n\n
\n\n\n\n
14. 6 Partitions + 6 Consumers<\/h1>\n\n\n\n
Conceptually:<\/p>\n\n\n\n
P0 -> C1\nP1 -> C2\nP2 -> C3\nP3 -> C4\nP4 -> C5\nP5 -> C6\n<\/code><\/pre>\n\n\n\n
This gives the maximum simple partition-level parallelism for that one six-partition topic inside that Consumer Group.<\/p>\n\n\n\n
\n\n\n\n
15. 6 Partitions + 10 Consumers<\/h1>\n\n\n\n
Conceptually:<\/p>\n\n\n\n
P0 -> C1\nP1 -> C2\nP2 -> C3\nP3 -> C4\nP4 -> C5\nP5 -> C6\n\nC7  -> idle\nC8  -> idle\nC9  -> idle\nC10 -> idle\n<\/code><\/pre>\n\n\n\n
Allowed? YES.<\/strong><\/p>\n\n\n\n
Useful? Usually NO<\/strong>.<\/p>\n\n\n\n
Extra consumers do not split one partition between multiple consumers in the same Consumer Group.<\/p>\n\n\n\n
\n\n\n\n
16. Golden Consumer Group Rule<\/h1>\n\n\n\n
Inside one normal Consumer Group:<\/p>\n\n\n\n
\n
One partition is assigned to at most one consumer in that group at a time.<\/strong><\/p>\n<\/blockquote>\n\n\n\n
This is the rule behind most consumer scaling discussions.<\/p>\n\n\n\n
Important: it does not<\/strong> mean one partition can only have one consumer in the whole company.<\/p>\n\n\n\n
Different Consumer Groups can independently consume that same partition.<\/p>\n\n\n\n
\n\n\n\n
17. Same Partition + Different Consumer Groups<\/h1>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Topic: orders\nPartition: P0\n<\/code><\/pre>\n\n\n\n
Three applications need the data:<\/p>\n\n\n\n
Analytics\nFraud Detection\nNotifications\n<\/code><\/pre>\n\n\n\n
Use three groups:<\/p>\n\n\n\n
group.id = analytics\ngroup.id = fraud\ngroup.id = notifications\n<\/code><\/pre>\n\n\n\n
Then:<\/p>\n\n\n\n
                  orders \/ P0\n                 \/     |      \\\n                v      v       v\n\n            Analytics Fraud Notifications\n              Group    Group     Group\n<\/code><\/pre>\n\n\n\n
The same partition can therefore have one assigned reader per Consumer Group<\/strong>.<\/p>\n\n\n\n
This is normal Kafka fan-out.<\/p>\n\n\n\n
\n\n\n\n
18. Consumer Group \u2014 What Problem Does It Solve?<\/h1>\n\n\n\n
A Consumer Group represents one logical consumption use case\/application.<\/p>\n\n\n\n
Examples:<\/p>\n\n\n\n
order-processing\nfraud-detection\nanalytics\nnotifications\nbilling\ndata-warehouse\nsearch-indexer\n<\/code><\/pre>\n\n\n\n
Each group has its own progress.<\/p>\n\n\n\n
Conceptually:<\/p>\n\n\n\n
Topic: orders\n\nAnalytics Group:\nP0 committed offset = 9000\n\nFraud Group:\nP0 committed offset = 8970\n\nNotification Group:\nP0 committed offset = 9010\n<\/code><\/pre>\n\n\n\n
Each group moves independently.<\/p>\n\n\n\n
\n\n\n\n
19. Consumer Groups Are NOT Copies of Consumers<\/h1>\n\n\n\n
Do not think:<\/p>\n\n\n\n
Consumer Group = one consumer\n<\/code><\/pre>\n\n\n\n
Instead:<\/p>\n\n\n\n
Consumer Group\n    |\n    +-- Consumer 1\n    +-- Consumer 2\n    +-- Consumer 3\n    +-- Consumer N\n<\/code><\/pre>\n\n\n\n
A group is the logical application.<\/p>\n\n\n\n
Consumers are the worker instances inside it.<\/p>\n\n\n\n
\n\n\n\n
20. How Many Consumer Groups Can Read a Topic?<\/h1>\n\n\n\n
Kafka architecture allows many independent Consumer Groups to consume the same topic.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
                     +--> analytics group\n                     |\n                     +--> billing group\nTopic: orders -------+\n                     +--> fraud group\n                     |\n                     +--> notifications group\n                     |\n                     +--> warehouse group\n<\/code><\/pre>\n\n\n\n
There is no design rule that says:<\/p>\n\n\n\n
1 topic = 1 Consumer Group\n<\/code><\/pre>\n\n\n\n
Multiple groups are a core Kafka feature.<\/p>\n\n\n\n
However, every additional active group adds:<\/p>\n\n\n\n
Read traffic\nConsumer processing\nOffset state\nNetwork activity\nOperational monitoring\n<\/code><\/pre>\n\n\n\n
So “allowed” does not mean “free.”<\/p>\n\n\n\n
Cluster\/provider capacity and quotas must be considered.<\/p>\n\n\n\n
\n\n\n\n
21. Can One Consumer Group Read Multiple Topics?<\/h1>\n\n\n\n
Yes.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
Consumer Group: customer-360\n\nSubscribes to:\n\ncustomers\norders\npayments\nshipments\n<\/code><\/pre>\n\n\n\n
Kafka assigns partitions from the subscribed topics across members of the group.<\/p>\n\n\n\n
Conceptually:<\/p>\n\n\n\n
customers: P0 P1\norders:    P0 P1 P2\npayments:  P0 P1\n\nTotal partition work = 7 partitions\n<\/code><\/pre>\n\n\n\n
With 3 consumers:<\/p>\n\n\n\n
Consumer A -> some of the 7 partitions\nConsumer B -> some of the 7 partitions\nConsumer C -> some of the 7 partitions\n<\/code><\/pre>\n\n\n\n
Exact mapping depends on the group protocol and assignment strategy.<\/p>\n\n\n\n
\n\n\n\n
22. Can One Consumer Belong to Multiple Consumer Groups?<\/h1>\n\n\n\n
A single Kafka consumer instance has one:<\/p>\n\n\n\n
group.id\n<\/code><\/pre>\n\n\n\n
and therefore participates in one Consumer Group at a time.<\/p>\n\n\n\n
If one application process needs to participate in multiple groups, it can run multiple consumer instances.<\/p>\n\n\n\n
Application Process\n|\n+-- KafkaConsumer A -> group.id=analytics\n|\n+-- KafkaConsumer B -> group.id=notifications\n<\/code><\/pre>\n\n\n\n
These are two separate consumer instances.<\/p>\n\n\n\n
\n\n\n\n
23. Can Multiple Producers Write to the Same Topic?<\/h1>\n\n\n\n
Yes.<\/p>\n\n\n\n
Producer A -----\\\nProducer B ------> Topic: orders\nProducer C -----\/\n<\/code><\/pre>\n\n\n\n
This is completely normal.<\/p>\n\n\n\n
\n\n\n\n
24. Can One Producer Write to Multiple Topics?<\/h1>\n\n\n\n
Yes.<\/p>\n\n\n\n
Producer\n|\n+--> orders\n+--> audit-events\n+--> system-events\n<\/code><\/pre>\n\n\n\n
Kafka does not require one producer instance per topic.<\/p>\n\n\n\n
\n\n\n\n
25. Replicas Are NOT Consumer Parallelism<\/h1>\n\n\n\n
A common confusion:<\/p>\n\n\n\n
Partitions = parallelism\nReplicas   = durability \/ availability\n<\/code><\/pre>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Topic: orders\nPartitions = 6\nReplication Factor = 3\n<\/code><\/pre>\n\n\n\n
There are:<\/p>\n\n\n\n
6 logical partitions\n18 partition replicas\n<\/code><\/pre>\n\n\n\n
But a Consumer Group does not<\/strong> suddenly get 18-way normal processing parallelism.<\/p>\n\n\n\n
The main partition-level consumer parallelism remains based on the six logical partitions.<\/p>\n\n\n\n
\n\n\n\n
26. Leader and Follower Relationship<\/h1>\n\n\n\n
For one partition:<\/p>\n\n\n\n
Partition P0\n\nBroker 1 -> Leader\nBroker 2 -> Follower\nBroker 3 -> Follower\n<\/code><\/pre>\n\n\n\n
Producer normally writes to the leader.<\/p>\n\n\n\n
Consumer normally fetches from the partition leader in the standard model.<\/p>\n\n\n\n
Followers replicate data for fault tolerance.<\/p>\n\n\n\n
Therefore:<\/p>\n\n\n\n
Add replicas\n!=\nAdd consumer processing slots\n<\/code><\/pre>\n\n\n\n
\n\n\n\n
27. Master Allowed \/ Not Allowed Table<\/h1>\n\n\n\n
Question<\/th>Allowed?<\/th>Explanation<\/th><\/tr><\/thead>
Many producers write to one topic<\/td>YES<\/td>Normal architecture<\/td><\/tr>
One producer writes to many topics<\/td>YES<\/td>Producer can publish to multiple topics<\/td><\/tr>
One topic has many partitions<\/td>YES<\/td>Main Kafka scaling model<\/td><\/tr>
One partition has multiple replicas<\/td>YES<\/td>Used for durability\/availability<\/td><\/tr>
One consumer reads many partitions<\/td>YES<\/td>Very common<\/td><\/tr>
Multiple consumers in same group read same partition at the same time<\/td>NO<\/td>A partition is assigned to at most one member per group<\/td><\/tr>
Consumers in different groups read the same partition<\/td>YES<\/td>Core fan-out model<\/td><\/tr>
One Consumer Group reads multiple topics<\/td>YES<\/td>Group can subscribe to multiple topics<\/td><\/tr>
Many Consumer Groups read one topic<\/td>YES<\/td>Each group has independent progress<\/td><\/tr>
More consumers than partitions<\/td>YES<\/td>Extra consumers are normally idle<\/td><\/tr>
More partitions than consumers<\/td>YES<\/td>Consumers own multiple partitions<\/td><\/tr>
One consumer instance belongs to multiple group IDs simultaneously<\/td>NO<\/td>One consumer instance participates in one group<\/td><\/tr>
One process hosts multiple consumer instances with different groups<\/td>YES<\/td>Each KafkaConsumer instance is independent<\/td><\/tr>
Replicas increase Consumer Group parallelism<\/td>NO<\/td>Replicas provide durability, not logical partition parallelism<\/td><\/tr>
Offsets are global across a topic<\/td>NO<\/td>Offsets are per partition<\/td><\/tr>
Different groups share committed offsets<\/td>NO<\/td>Each group tracks its own offsets<\/td><\/tr>
Same group ID for unrelated apps that both need all events<\/td>TECHNICALLY YES, ARCHITECTURALLY WRONG<\/td>They will split partitions instead of both receiving all events<\/td><\/tr>
Decrease topic partition count directly<\/td>NO in normal Kafka operation<\/td>Usually requires new topic\/migration<\/td><\/tr>
Increase partition count<\/td>YES<\/td>But affects partition mapping and must be planned carefully<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Situation<\/th>Allowed?<\/th><\/tr><\/thead>
1 Consumer \u2192 1 Partition<\/td>\u2705<\/td><\/tr>
1 Consumer \u2192 Many Partitions<\/td>\u2705<\/td><\/tr>
1 Partition \u2192 2 Consumers in same group<\/strong><\/td>\u274c<\/td><\/tr>
1 Partition \u2192 Consumers in different groups<\/strong><\/td>\u2705<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
28. Fan-Out vs Load Balancing<\/h1>\n\n\n\n
Fan-Out<\/h2>\n\n\n\n
Same topic, different Consumer Groups.<\/p>\n\n\n\n
Topic: orders\n|\n+--> group.id=analytics\n|\n+--> group.id=fraud\n|\n+--> group.id=notifications\n<\/code><\/pre>\n\n\n\n
Result:<\/p>\n\n\n\n
Each group independently receives the stream.\n<\/code><\/pre>\n\n\n\n
Use when different applications need the same events.<\/p>\n\n\n\n
Load Balancing<\/h2>\n\n\n\n
Same topic, same Consumer Group.<\/p>\n\n\n\n
Topic: orders\n|\n+--> Consumer 1 \\\n+--> Consumer 2  > group.id=order-workers\n+--> Consumer 3 \/\n<\/code><\/pre>\n\n\n\n
Result:<\/p>\n\n\n\n
Consumers share partition ownership.\n<\/code><\/pre>\n\n\n\n
Use when one logical application needs horizontal scaling.<\/p>\n\n\n\n
\n\n\n\n
29. Fan-Out + Load Balancing Together<\/h1>\n\n\n\n
Real Kafka systems commonly use both.<\/p>\n\n\n\n
Topic: vehicle-telemetry\n<\/code><\/pre>\n\n\n\n
Analytics application:<\/p>\n\n\n\n
group.id = analytics\nConsumer A\nConsumer B\nConsumer C\n<\/code><\/pre>\n\n\n\n
Alert application:<\/p>\n\n\n\n
group.id = alerts\nConsumer A\nConsumer B\n<\/code><\/pre>\n\n\n\n
Data warehouse application:<\/p>\n\n\n\n
group.id = warehouse\nConsumer A\nConsumer B\nConsumer C\nConsumer D\n<\/code><\/pre>\n\n\n\n
Each group receives the complete logical stream independently.<\/p>\n\n\n\n
Inside each group, its consumers divide the partitions.<\/p>\n\n\n\n
\n\n\n\n
30. Concrete Mapping Example<\/h1>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Topic: vehicle-telemetry\nPartitions = 12\n<\/code><\/pre>\n\n\n\n
Analytics<\/h2>\n\n\n\n
group.id = analytics\nConsumers = 6\n<\/code><\/pre>\n\n\n\n
Possible mapping:<\/p>\n\n\n\n
6 consumers\n12 partitions\n~2 partitions per consumer\n<\/code><\/pre>\n\n\n\n
Alerts<\/h2>\n\n\n\n
group.id = alerts\nConsumers = 3\n<\/code><\/pre>\n\n\n\n
Possible mapping:<\/p>\n\n\n\n
3 consumers\n12 partitions\n~4 partitions per consumer\n<\/code><\/pre>\n\n\n\n
Data Warehouse<\/h2>\n\n\n\n
group.id = warehouse\nConsumers = 12\n<\/code><\/pre>\n\n\n\n
Possible mapping:<\/p>\n\n\n\n
12 consumers\n12 partitions\n~1 partition per consumer\n<\/code><\/pre>\n\n\n\n
All three groups independently consume all 12 partitions.<\/p>\n\n\n\n
The groups do not<\/strong> steal partitions from each other.<\/p>\n\n\n\n
\n\n\n\n
31. What Happens If Analytics Has 20 Consumers?<\/h1>\n\n\n\n
Topic:<\/p>\n\n\n\n
12 partitions\n<\/code><\/pre>\n\n\n\n
Analytics group:<\/p>\n\n\n\n
20 consumers\n<\/code><\/pre>\n\n\n\n
Maximum partition assignments:<\/p>\n\n\n\n
12\n<\/code><\/pre>\n\n\n\n
Therefore approximately:<\/p>\n\n\n\n
12 consumers -> receive partitions\n8 consumers  -> idle\n<\/code><\/pre>\n\n\n\n
Adding more consumers does not help until more partition-level work is available.<\/p>\n\n\n\n
\n\n\n\n
32. Multiple Topics + One Group Example<\/h1>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Topic A = 4 partitions\nTopic B = 6 partitions\nTopic C = 2 partitions\n<\/code><\/pre>\n\n\n\n
One Consumer Group subscribes to all three.<\/p>\n\n\n\n
Total partition assignments available:<\/p>\n\n\n\n
4 + 6 + 2 = 12\n<\/code><\/pre>\n\n\n\n
If the group has six consumers, Kafka’s assignment strategy distributes those topic-partitions among the six members.<\/p>\n\n\n\n
The exact distribution may not be perfectly two partitions each because topic subscription and assignment strategy matter.<\/p>\n\n\n\n
The core invariant remains:<\/p>\n\n\n\n
\n
One topic-partition can be assigned to at most one consumer in the same group at a time.<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\n
33. Why Consumer Count Alone Does Not Determine Capacity<\/h1>\n\n\n\n
Suppose:<\/p>\n\n\n\n
P0 = 1,000 records\/sec\nP1 = 1,000 records\/sec\nP2 = 50,000 records\/sec\n<\/code><\/pre>\n\n\n\n
Group:<\/p>\n\n\n\n
Consumer A -> P0\nConsumer B -> P1\nConsumer C -> P2\n<\/code><\/pre>\n\n\n\n
Kafka has assigned one partition to each consumer.<\/p>\n\n\n\n
But Consumer C has vastly more work.<\/p>\n\n\n\n
Kafka balances:<\/p>\n\n\n\n
partition ownership\n<\/code><\/pre>\n\n\n\n
not:<\/p>\n\n\n\n
perfect CPU workload\n<\/code><\/pre>\n\n\n\n
This is why producer partition-key design directly affects consumer performance.<\/p>\n\n\n\n
\n\n\n\n
34. Hot Partitions<\/h1>\n\n\n\n
A hot partition<\/strong> is a partition receiving or processing significantly more traffic than the others.<\/p>\n\n\n\n
P0  ####\nP1  ####\nP2  ########################################\nP3  #####\n<\/code><\/pre>\n\n\n\n
Possible causes:<\/p>\n\n\n\n
Poor key distribution\nOne very large customer\/entity\nLow-cardinality key\nHighly skewed business traffic\nCustom partitioner bug\n<\/code><\/pre>\n\n\n\n
Impact:<\/p>\n\n\n\n
One broker leader becomes busy\nOne consumer becomes busy\nGroup lag becomes concentrated\nAdding consumers may not help\n<\/code><\/pre>\n\n\n\n
Because P2 cannot be split across two consumers inside the same group at the same time.<\/p>\n\n\n\n
\n\n\n\n
35. How Many Partitions Should We Create?<\/h1>\n\n\n\n
There is no universal correct number.<\/p>\n\n\n\n
Partition count should consider:<\/p>\n\n\n\n
Required producer throughput\nRequired consumer parallelism\nExpected future growth\nOrdering requirements\nBroker capacity\nRecord size\nConsumer processing rate\nFailure recovery requirements\nOperational overhead\n<\/code><\/pre>\n\n\n\n
A useful planning concept is:<\/p>\n\n\n\n
Required Partitions\n>=\nmaximum of:\n\n1. partitions needed for producer throughput\n2. partitions needed for consumer parallelism\n3. partitions needed for expected future scale\n<\/code><\/pre>\n\n\n\n
But per-partition throughput is workload-specific.<\/p>\n\n\n\n
You must benchmark.<\/p>\n\n\n\n
Do not use random internet numbers as universal limits.<\/p>\n\n\n\n
\n\n\n\n
36. Simple Consumer Capacity Formula<\/h1>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Incoming rate = 60,000 records\/sec\n<\/code><\/pre>\n\n\n\n
One measured consumer instance can safely process:<\/p>\n\n\n\n
10,000 records\/sec\n<\/code><\/pre>\n\n\n\n
Approximate required active consumer capacity:<\/p>\n\n\n\n
60,000 \/ 10,000 = 6 consumers\n<\/code><\/pre>\n\n\n\n
For those six consumers to perform useful partition-level parallel work, the subscribed topic(s) need enough partitions.<\/p>\n\n\n\n
For one topic:<\/p>\n\n\n\n
Partitions >= 6\n<\/code><\/pre>\n\n\n\n
would be a minimum conceptual requirement.<\/p>\n\n\n\n
For production headroom you normally want more capacity than the absolute minimum.<\/p>\n\n\n\n
\n\n\n\n
37. Do Not Size Consumers Before Measuring Them<\/h1>\n\n\n\n
A consumer’s capacity depends on business logic.<\/p>\n\n\n\n
Consumer A:<\/p>\n\n\n\n
Deserialize\nCount event\n<\/code><\/pre>\n\n\n\n
may process far more records\/sec than Consumer B:<\/p>\n\n\n\n
Deserialize\nValidate\nCall HTTP API\nWrite database\nRun business rules\n<\/code><\/pre>\n\n\n\n
Therefore:<\/p>\n\n\n\n
\"One Kafka consumer can process X records\/sec\"\n<\/code><\/pre>\n\n\n\n
is not a universal statement.<\/p>\n\n\n\n
Benchmark your actual application.<\/p>\n\n\n\n
\n\n\n\n
38. Increasing Partitions Has Consequences<\/h1>\n\n\n\n
Increasing partitions can increase potential parallelism.<\/p>\n\n\n\n
But it is not free.<\/p>\n\n\n\n
Possible effects:<\/p>\n\n\n\n
More metadata\nMore partition leaders\nMore replica logs\nMore file handles\nMore broker work\nMore consumer assignments\nPotential rebalances\nChanges to key-to-partition mapping\n<\/code><\/pre>\n\n\n\n
The last point is extremely important.<\/p>\n\n\n\n
If a key is mapped using a partition-count-dependent hashing strategy, increasing the number of partitions can cause future records with the same key to map differently.<\/p>\n\n\n\n
That can affect assumptions about historical per-key ordering across the partition-count change.<\/p>\n\n\n\n
Plan partition growth carefully.<\/p>\n\n\n\n
\n\n\n\n
39. Partition Count Cannot Normally Be Reduced<\/h1>\n\n\n\n
Kafka supports increasing partitions.<\/p>\n\n\n\n
Reducing the number of partitions of an existing topic is not a normal supported operation.<\/p>\n\n\n\n
If you need fewer partitions, a common architectural approach is:<\/p>\n\n\n\n
Create a new topic\n     |\n     v\nMigrate\/repartition data\n     |\n     v\nMove producers\/consumers\n<\/code><\/pre>\n\n\n\n
Partition count is therefore an important long-term design decision.<\/p>\n\n\n\n
\n\n\n\n
40. How Many Consumers Should a Consumer Group Have?<\/h1>\n\n\n\n
A practical starting rule:<\/p>\n\n\n\n
Useful Consumer Count\n<=\navailable partition assignments\n<\/code><\/pre>\n\n\n\n
For one topic:<\/p>\n\n\n\n
Useful Consumer Count\n<=\nnumber of partitions\n<\/code><\/pre>\n\n\n\n
Then size actual consumers using measured processing capacity.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
Topic partitions = 12\nMeasured workload requires 5 consumers\n<\/code><\/pre>\n\n\n\n
Start with five consumers, not automatically twelve, unless the additional processing headroom is justified.<\/p>\n\n\n\n
\n\n\n\n
41. Should Consumer Count Equal Partition Count?<\/h1>\n\n\n\n
Not necessarily.<\/p>\n\n\n\n
12 partitions\n3 consumers\n<\/code><\/pre>\n\n\n\n
may be completely healthy if each consumer can process four partitions comfortably.<\/p>\n\n\n\n
Benefits can include:<\/p>\n\n\n\n
Fewer application instances\nLower cost\nLower connection count\nSimpler operations\n<\/code><\/pre>\n\n\n\n
Scale consumers when processing capacity requires it.<\/p>\n\n\n\n
\n\n\n\n
42. When Should We Add Consumers?<\/h1>\n\n\n\n
Add consumers to a Consumer Group when:<\/p>\n\n\n\n
Lag is growing because consumers cannot keep up\nAND\nthere are unexploited partitions available\n<\/code><\/pre>\n\n\n\n
Example:<\/p>\n\n\n\n
Partitions = 12\nConsumers  = 4\nConsumers are CPU-bound\nLag is increasing\n<\/code><\/pre>\n\n\n\n
You may scale toward six, eight, ten or twelve consumers while measuring improvement.<\/p>\n\n\n\n
\n\n\n\n
43. When Adding Consumers Will NOT Help<\/h1>\n\n\n\n
Case 1 \u2014 Consumers already equal partitions<\/h2>\n\n\n\n
Partitions = 6\nConsumers  = 6\n<\/code><\/pre>\n\n\n\n
Adding more consumers does not create more partition-level concurrency.<\/p>\n\n\n\n
Case 2 \u2014 One hot partition<\/h2>\n\n\n\n
P0 lag = 0\nP1 lag = 0\nP2 lag = 500,000\n<\/code><\/pre>\n\n\n\n
Adding another consumer cannot split P2 inside the same group.<\/p>\n\n\n\n
Case 3 \u2014 Downstream database is the bottleneck<\/h2>\n\n\n\n
Kafka Consumer\n     |\n     v\nDatabase saturated\n<\/code><\/pre>\n\n\n\n
Adding more consumers may make the database problem worse.<\/p>\n\n\n\n
\n\n\n\n
44. Consumer Groups and Independent Applications<\/h1>\n\n\n\n
Use a new group.id<\/code> when another logical application needs to receive all events independently.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
orders topic\n<\/code><\/pre>\n\n\n\n
Application A:<\/p>\n\n\n\n
analytics\n<\/code><\/pre>\n\n\n\n
Application B:<\/p>\n\n\n\n
fraud\n<\/code><\/pre>\n\n\n\n
If both use the same group ID, they will share partitions.<\/p>\n\n\n\n
If both applications require the complete stream, use separate group IDs.<\/p>\n\n\n\n
\n\n\n\n
45. Dangerous Group ID Mistake<\/h1>\n\n\n\n
Wrong:<\/p>\n\n\n\n
Analytics Service\n group.id = orders\n\nNotification Service\n group.id = orders\n<\/code><\/pre>\n\n\n\n
Result:<\/p>\n\n\n\n
They become workers in the SAME logical Consumer Group.\n<\/code><\/pre>\n\n\n\n
Therefore they share partitions.<\/p>\n\n\n\n
Correct:<\/p>\n\n\n\n
Analytics:\n group.id = order-analytics\n\nNotifications:\n group.id = order-notifications\n<\/code><\/pre>\n\n\n\n
Now both applications independently receive the topic.<\/p>\n\n\n\n
\n\n\n\n
46. What Is an Offset?<\/h1>\n\n\n\n
An offset is a position in a partition.<\/p>\n\n\n\n
Partition P0\n\nOffset 0   Record A\nOffset 1   Record B\nOffset 2   Record C\nOffset 3   Record D\nOffset 4   Record E\n<\/code><\/pre>\n\n\n\n
Offsets are per partition, not global per topic.<\/p>\n\n\n\n
\n\n\n\n
47. Current Position vs Committed Offset<\/h1>\n\n\n\n
Current Position<\/h2>\n\n\n\n
Where the running consumer expects to read next.<\/p>\n\n\n\n
Committed Offset<\/h2>\n\n\n\n
The saved recovery checkpoint for the Consumer Group.<\/p>\n\n\n\n
Stored in Kafka’s internal:<\/p>\n\n\n\n
__consumer_offsets\n<\/code><\/pre>\n\n\n\n
Think:<\/p>\n\n\n\n
Current position = where the running consumer is now\nCommitted offset = where the group can safely resume after failure\n<\/code><\/pre>\n\n\n\n
\n\n\n\n
48. Example \u2014 Current vs Committed<\/h1>\n\n\n\n
0 1 2 3 4 5 6 7 8 9 10\n        ^       ^\n        |       |\n   committed   current\n<\/code><\/pre>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Committed = 4\nCurrent   = 8\n<\/code><\/pre>\n\n\n\n
If the consumer crashes before committing newer progress, a replacement may resume near the committed point.<\/p>\n\n\n\n
Records between the two positions may be processed again.<\/p>\n\n\n\n
This is normal in at-least-once designs.<\/p>\n\n\n\n
\n\n\n\n
49. What Is Consumer Lag?<\/h1>\n\n\n\n
Consumer lag means:<\/p>\n\n\n\n
\n
How far a Consumer Group is behind the available end of a partition.<\/p>\n<\/blockquote>\n\n\n\n
Simplified:<\/p>\n\n\n\n
Latest available\/end position\n-\nConsumer Group progress\n=\nLag\n<\/code><\/pre>\n\n\n\n
Example:<\/p>\n\n\n\n
Partition end   = 10,000\nGroup committed = 9,400\nLag \u2248 600 records\n<\/code><\/pre>\n\n\n\n
\n\n\n\n
50. Lag Belongs to Consumer Group + Topic + Partition<\/h1>\n\n\n\n
Do not think only:<\/p>\n\n\n\n
Topic lag\n<\/code><\/pre>\n\n\n\n
The useful unit is:<\/p>\n\n\n\n
Consumer Group\n+\nTopic\n+\nPartition\n<\/code><\/pre>\n\n\n\n
Example:<\/p>\n\n\n\n
Group: analytics\nTopic: orders\n\nP0 lag = 0\nP1 lag = 0\nP2 lag = 50,000\nP3 lag = 100\n<\/code><\/pre>\n\n\n\n
Total lag is 50,100, but the total hides the real problem: P2 is hot or stuck.<\/strong><\/p>\n\n\n\n
Always inspect lag per partition.<\/p>\n\n\n\n
\n\n\n\n
51. When Is Lag Created?<\/h1>\n\n\n\n
Lag grows whenever records arrive faster than the Consumer Group makes durable processing progress.<\/p>\n\n\n\n
Simplified:<\/p>\n\n\n\n
Producer Rate\n>\nConsumer Processing Rate\n<\/code><\/pre>\n\n\n\n
for long enough.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
Producer = 50,000 records\/sec\nConsumer Group = 30,000 records\/sec\n<\/code><\/pre>\n\n\n\n
Backlog growth:<\/p>\n\n\n\n
20,000 records\/sec\n<\/code><\/pre>\n\n\n\n
After 60 seconds:<\/p>\n\n\n\n
~1,200,000 records behind\n<\/code><\/pre>\n\n\n\n
assuming rates remain constant.<\/p>\n\n\n\n
\n\n\n\n
52. Common Reasons Lag Is Created<\/h1>\n\n\n\n
    \n
  1. Too few consumers<\/li>\n\n\n\n
  2. Too few partitions<\/li>\n\n\n\n
  3. Consumer processing is slow<\/li>\n\n\n\n
  4. Slow downstream database<\/li>\n\n\n\n
  5. Slow external API<\/li>\n\n\n\n
  6. Hot partition<\/li>\n\n\n\n
  7. Consumer crashes<\/li>\n\n\n\n
  8. Frequent rebalances<\/li>\n\n\n\n
  9. Long garbage-collection pauses<\/li>\n\n\n\n
  10. max.poll.interval.ms<\/code> problems<\/li>\n\n\n\n
  11. Broker\/network throttling<\/li>\n\n\n\n
  12. Inefficient fetch configuration<\/li>\n\n\n\n
  13. Poison\/bad records<\/li>\n\n\n\n
  14. Downstream retry storms<\/li>\n\n\n\n
  15. Commit failures or very infrequent commits causing a large committed-position gap<\/li>\n<\/ol>\n\n\n\n
    \n\n\n\n
    53. Lag Can Exist Even When Processing Is Fast<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    Consumer current position = 10,000\nCommitted offset          = 8,000\n<\/code><\/pre>\n\n\n\n
    The application may have processed much more than the committed checkpoint.<\/p>\n\n\n\n
    A monitoring system based on committed offsets may report substantial lag even though in-memory\/current processing is further ahead.<\/p>\n\n\n\n
    Therefore ask:<\/p>\n\n\n\n
    Is this real processing lag?\n\nor\n\nIs processing healthy but commits are infrequent\/failing?\n<\/code><\/pre>\n\n\n\n
    Monitor both processing progress and commit health where possible.<\/p>\n\n\n\n
    \n\n\n\n
    54. A Temporary Lag Is Not Always a Problem<\/h1>\n\n\n\n
    Traffic spike\n     |\n     v\nLag grows\n     |\n     v\nSpike ends\n     |\n     v\nConsumers process faster than new input\n     |\n     v\nLag returns to acceptable level\n<\/code><\/pre>\n\n\n\n
    This may be acceptable if your SLO allows it.<\/p>\n\n\n\n
    The real question is:<\/p>\n\n\n\n
    Can the group catch up within the required time?\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    55. Persistent Lag Is a Capacity Signal<\/h1>\n\n\n\n
    Bad pattern:<\/p>\n\n\n\n
    Time ----->\n\nLag:\n10k\n20k\n40k\n80k\n160k\n320k\n<\/code><\/pre>\n\n\n\n
    The Consumer Group never catches up.<\/p>\n\n\n\n
    This means steady-state processing capacity is below workload demand or a persistent bottleneck exists.<\/p>\n\n\n\n
    \n\n\n\n
    56. Master Lag Troubleshooting Decision Tree<\/h1>\n\n\n\n
    LAG IS GROWING\n     |\n     v\nIs lag concentrated in one\/few partitions?\n     |\n     +-- YES --> Investigate hot partition\/key skew\n     |           slow partition-specific data\n     |           stuck records\n     |           broker leader hotspot\n     |\n     +-- NO --> Lag across most partitions\n                    |\n                    v\n            Are consumers CPU-bound?\n                    |\n             +------+------+\n             |             |\n            YES           NO\n             |             |\n             v             v\n      Scale consumers   Check downstream\n      if partitions     DB\/API\/network\n      allow it          fetch\/rebalance\n<\/code><\/pre>\n\n\n\n
    Continue:<\/p>\n\n\n\n
    Do we have idle partition capacity?\n     |\n     +-- YES --> Add consumer instances and measure\n     |\n     +-- NO --> Consumers already near partition count\n                    |\n                    v\n          Optimize processing\/downstream\n                    |\n                    v\n          If capacity still insufficient:\n          consider increasing partitions\n          + consumers + cluster capacity\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    57. Lag Troubleshooting Step 1 \u2014 Check Per-Partition Lag<\/h1>\n\n\n\n
    Do not start with:<\/p>\n\n\n\n
    \"Add consumers.\"\n<\/code><\/pre>\n\n\n\n
    First ask:<\/p>\n\n\n\n
    Which partitions are behind?\n<\/code><\/pre>\n\n\n\n
    Example A:<\/p>\n\n\n\n
    P0 50k\nP1 52k\nP2 49k\nP3 51k\n<\/code><\/pre>\n\n\n\n
    Likely group-wide capacity issue.<\/p>\n\n\n\n
    Example B:<\/p>\n\n\n\n
    P0 0\nP1 0\nP2 205k\nP3 0\n<\/code><\/pre>\n\n\n\n
    Likely hot\/stuck partition issue.<\/p>\n\n\n\n
    These need different solutions.<\/p>\n\n\n\n
    \n\n\n\n
    58. Lag Troubleshooting Step 2 \u2014 Compare Input vs Processing Rate<\/h1>\n\n\n\n
    Measure:<\/p>\n\n\n\n
    Producer\/input rate\n<\/code><\/pre>\n\n\n\n
    and:<\/p>\n\n\n\n
    Consumer completion rate\n<\/code><\/pre>\n\n\n\n
    If:<\/p>\n\n\n\n
    Input = 40k\/sec\nProcessing = 60k\/sec\n<\/code><\/pre>\n\n\n\n
    the group should normally catch up.<\/p>\n\n\n\n
    If:<\/p>\n\n\n\n
    Input = 60k\/sec\nProcessing = 40k\/sec\n<\/code><\/pre>\n\n\n\n
    lag grows permanently.<\/p>\n\n\n\n
    \n\n\n\n
    59. Lag Troubleshooting Step 3 \u2014 Check Consumer Count vs Partitions<\/h1>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Partitions = 12\nConsumers  = 3\n<\/code><\/pre>\n\n\n\n
    If each consumer is overloaded, scale upward while measuring.<\/p>\n\n\n\n
    But if:<\/p>\n\n\n\n
    Partitions = 12\nConsumers  = 12\n<\/code><\/pre>\n\n\n\n
    consumer 13 usually does not add partition-level capacity.<\/p>\n\n\n\n
    \n\n\n\n
    60. Lag Troubleshooting Step 4 \u2014 Check Processing Time<\/h1>\n\n\n\n
    Measure:<\/p>\n\n\n\n
    Time per record\nTime per batch\nDatabase latency\nAPI latency\nCPU\nMemory\nGC\nThread pool saturation\n<\/code><\/pre>\n\n\n\n
    Often Kafka is not the bottleneck.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Kafka fetch = 15 ms\nDatabase write = 800 ms\n<\/code><\/pre>\n\n\n\n
    Tuning Kafka fetch parameters will not fix the primary problem.<\/p>\n\n\n\n
    \n\n\n\n
    61. Lag Troubleshooting Step 5 \u2014 Check Rebalances<\/h1>\n\n\n\n
    Frequent rebalances can cause:<\/p>\n\n\n\n
    Temporary processing interruption\nPartition movement\nCache warm-up\nState reload\nDuplicate processing near commit boundaries\n<\/code><\/pre>\n\n\n\n
    Inspect:<\/p>\n\n\n\n
    Consumer crashes\nRolling deployments\nmax.poll.interval.ms\nLong processing\nMembership\/session behavior\nNetwork instability\nGroup protocol\nApplication shutdown behavior\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    62. Lag Troubleshooting Step 6 \u2014 Check Hot Keys<\/h1>\n\n\n\n
    If one partition is overloaded, inspect producer key distribution.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Key = customer_id\n<\/code><\/pre>\n\n\n\n
    One customer generates 40% of all traffic.<\/p>\n\n\n\n
    All of that customer’s records map to one partition.<\/p>\n\n\n\n
    Adding consumers will not split that partition.<\/p>\n\n\n\n
    Potential solutions depend on ordering requirements:<\/p>\n\n\n\n
    Improve key distribution\nShard a high-volume entity if business semantics permit\nUse another partitioning strategy\nSeparate extreme workloads\nIncrease processing efficiency for that partition\n<\/code><\/pre>\n\n\n\n
    Do not destroy required ordering merely to spread load.<\/p>\n\n\n\n
    \n\n\n\n
    63. Lag Troubleshooting Step 7 \u2014 Check Fetch and Poll Behavior<\/h1>\n\n\n\n
    Important consumer settings include:<\/p>\n\n\n\n
    fetch.min.bytes\nfetch.max.wait.ms\nfetch.max.bytes\nmax.partition.fetch.bytes\nmax.poll.records\nmax.poll.interval.ms\n<\/code><\/pre>\n\n\n\n
    These influence:<\/p>\n\n\n\n
    Throughput\nLatency\nMemory\nWork per poll\nFailure\/rebalance behavior\n<\/code><\/pre>\n\n\n\n
    Tune only after identifying a real bottleneck.<\/p>\n\n\n\n
    \n\n\n\n
    64. max.poll.records<\/code> and Lag<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    max.poll.records = 500\n<\/code><\/pre>\n\n\n\n
    and each record requires one second of serial processing.<\/p>\n\n\n\n
    One poll could represent roughly:<\/p>\n\n\n\n
    500 seconds\n<\/code><\/pre>\n\n\n\n
    of work.<\/p>\n\n\n\n
    That may create long processing cycles, poll interval risk, rebalances and lag.<\/p>\n\n\n\n
    Possible improvements:<\/p>\n\n\n\n
    Reduce max.poll.records\nOptimize processing\nBatch downstream calls\nScale consumers\nIncrease partitions when justified\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    65. Database Batching Can Reduce Lag<\/h1>\n\n\n\n
    Bad:<\/p>\n\n\n\n
    500 Kafka records\n=\n500 individual database network calls\n<\/code><\/pre>\n\n\n\n
    Potentially better:<\/p>\n\n\n\n
    500 Kafka records\n     |\n     v\nBatch database operation\n<\/code><\/pre>\n\n\n\n
    if business correctness permits.<\/p>\n\n\n\n
    Kafka consumer performance often depends more on downstream batching than on Kafka itself.<\/p>\n\n\n\n
    \n\n\n\n
    66. Backpressure \u2014 Do Not Overload Downstream Systems<\/h1>\n\n\n\n
    Suppose Kafka delivers data faster than your database can safely accept it.<\/p>\n\n\n\n
    Do not blindly keep increasing consumers.<\/p>\n\n\n\n
    You may overload the database.<\/p>\n\n\n\n
    Better architecture may include:<\/p>\n\n\n\n
    Controlled consumer concurrency\nBatching\npause()\/resume()\nBounded worker queues\nRate limiting\nDownstream capacity scaling\n<\/code><\/pre>\n\n\n\n
    A healthy Kafka consumer should not destroy the system it feeds.<\/p>\n\n\n\n
    \n\n\n\n
    67. Do Not “Fix” Lag by Resetting Offsets<\/h1>\n\n\n\n
    Dangerous response:<\/p>\n\n\n\n
    \"Consumer is 5 million records behind.\nReset offsets to latest.\"\n<\/code><\/pre>\n\n\n\n
    This may make lag disappear because you skipped the backlog.<\/p>\n\n\n\n
    But the business data was not processed.<\/p>\n\n\n\n
    That is not lag remediation. That is data skipping.<\/p>\n\n\n\n
    Only reset\/seek past data when the business explicitly decides the old data can be discarded.<\/p>\n\n\n\n
    \n\n\n\n
    68. auto.offset.reset=latest<\/code> Does Not Solve Existing Healthy Group Lag<\/h1>\n\n\n\n
    auto.offset.reset<\/code> is mainly relevant when there is no usable committed offset.<\/p>\n\n\n\n
    It is not the normal control for a healthy group that already has committed progress.<\/p>\n\n\n\n
    Do not teach:<\/p>\n\n\n\n
    \"Set latest to fix lag.\"\n<\/code><\/pre>\n\n\n\n
    That confuses offset initialization\/recovery with processing capacity.<\/p>\n\n\n\n
    \n\n\n\n
    69. Offset Commit Strategy and Lag<\/h1>\n\n\n\n
    If you commit every record synchronously:<\/p>\n\n\n\n
    Process one\nCommit\nWait\nProcess one\nCommit\nWait\n<\/code><\/pre>\n\n\n\n
    commit overhead can reduce throughput and increase lag.<\/p>\n\n\n\n
    If you never commit for a long time, processing may be fast but the recovery checkpoint stays far behind.<\/p>\n\n\n\n
    A balanced production pattern often commits safe progress in batches.<\/p>\n\n\n\n
    The exact strategy depends on:<\/p>\n\n\n\n
    Business correctness\nDuplicate tolerance\nProcessing time\nCommit cost\nFailure recovery objectives\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    70. At-Least-Once and Lag Recovery<\/h1>\n\n\n\n
    A common robust pattern is:<\/p>\n\n\n\n
    poll\n  |\n  v\nprocess\n  |\n  v\nbusiness operation succeeds\n  |\n  v\ncommit safe next offset\n<\/code><\/pre>\n\n\n\n
    After a crash, some records may be processed again.<\/p>\n\n\n\n
    Therefore downstream processing should be idempotent where practical.<\/p>\n\n\n\n
    \n\n\n\n
    71. Poison Messages and Lag<\/h1>\n\n\n\n
    Suppose one record repeatedly fails:<\/p>\n\n\n\n
    Offset 500\n<\/code><\/pre>\n\n\n\n
    Application behavior:<\/p>\n\n\n\n
    Read 500\nFail\nRetry\nFail\nRetry\nFail\n...\n<\/code><\/pre>\n\n\n\n
    Everything behind it may stop progressing, depending on application error handling.<\/p>\n\n\n\n
    Lag grows.<\/p>\n\n\n\n
    Production designs need a failure strategy such as:<\/p>\n\n\n\n
    Bounded retries\nRetry topic\/pattern\nDead-letter topic where appropriate\nAlerting\nManual investigation\nIdempotent replay\n<\/code><\/pre>\n\n\n\n
    Never create an infinite silent retry loop.<\/p>\n\n\n\n
    \n\n\n\n
    72. Rebalance and Lag<\/h1>\n\n\n\n
    During a rebalance, partition ownership changes.<\/p>\n\n\n\n
    Older\/eager rebalance behavior can temporarily pause substantial group processing.<\/p>\n\n\n\n
    Modern cooperative\/incremental approaches can reduce disruption, depending on the Consumer Group protocol and assignment strategy.<\/p>\n\n\n\n
    Regardless of protocol:<\/p>\n\n\n\n
    Frequent rebalances\n=\nless useful processing time\n=\npotential lag\n<\/code><\/pre>\n\n\n\n
    Monitor rebalance frequency and duration.<\/p>\n\n\n\n
    \n\n\n\n
    73. Failover and Lag<\/h1>\n\n\n\n
    Example:<\/p>\n\n\n\n
    P0 -> Consumer A\nP1 -> Consumer B\nP2 -> Consumer C\n<\/code><\/pre>\n\n\n\n
    Consumer B crashes.<\/p>\n\n\n\n
    Until Kafka detects the failure and reassigns P1:<\/p>\n\n\n\n
    P1 processing pauses\n<\/code><\/pre>\n\n\n\n
    P1 lag grows.<\/p>\n\n\n\n
    After reassignment, the new owner resumes from the Consumer Group’s committed offset.<\/p>\n\n\n\n
    If spare processing capacity exists, it can catch up.<\/p>\n\n\n\n
    \n\n\n\n
    74. Availability Requires Spare Capacity<\/h1>\n\n\n\n
    If every consumer normally runs at 95% CPU and one consumer fails, its partitions move to surviving consumers.<\/p>\n\n\n\n
    The survivors may not have enough spare capacity.<\/p>\n\n\n\n
    Result:<\/p>\n\n\n\n
    failover succeeds technically\nbut lag grows dramatically\n<\/code><\/pre>\n\n\n\n
    Production planning should include failure headroom.<\/p>\n\n\n\n
    Do not size only for healthy-state average load.<\/p>\n\n\n\n
    \n\n\n\n
    75. Developer Planning Matrix<\/h1>\n\n\n\n
    Decision<\/th>Primary Kafka Object<\/th>Key Question<\/th><\/tr><\/thead>
    Event-stream boundary<\/td>Topic<\/td>What business stream\/data contract is this?<\/td><\/tr>
    Ordering<\/td>Key + Partition<\/td>Which events must remain ordered together?<\/td><\/tr>
    Parallelism<\/td>Partitions<\/td>How many independent partition workloads are needed?<\/td><\/tr>
    Independent use cases<\/td>Consumer Groups<\/td>Which applications each need the full stream?<\/td><\/tr>
    Processing scale<\/td>Consumers<\/td>How many worker instances does each group need?<\/td><\/tr>
    Durability<\/td>Replicas<\/td>How many broker failures should data tolerate?<\/td><\/tr>
    Recovery position<\/td>Offsets<\/td>Where should this group resume after failure?<\/td><\/tr>
    Backlog health<\/td>Lag<\/td>Is processing keeping up with input?<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n
    76. Planning Example \u2014 Order Platform<\/h1>\n\n\n\n
    Requirements:<\/p>\n\n\n\n
    50,000 orders\/sec peak\nOrdering required per order_id\nThree independent applications:\n- fulfillment\n- fraud\n- analytics\n<\/code><\/pre>\n\n\n\n
    Design:<\/p>\n\n\n\n
    Topic:\norders\n\nKey:\norder_id\n\nPartitions:\ncapacity-planned based on throughput and consumer parallelism\n\nReplication:\nchosen for durability\/availability\n\nConsumer Groups:\norder-fulfillment\norder-fraud\norder-analytics\n<\/code><\/pre>\n\n\n\n
    Within fulfillment, consumers scale according to processing demand up to useful partition-level parallelism.<\/p>\n\n\n\n
    Fraud and analytics scale independently.<\/p>\n\n\n\n
    \n\n\n\n
    77. Planning Example \u2014 Vehicle Telematics<\/h1>\n\n\n\n
    Requirements:<\/p>\n\n\n\n
    500,000 vehicles\nFrequent telemetry\nOrdering desired per vehicle\nConsumers:\n- real-time alerts\n- trip processing\n- analytics\n- data warehouse\n<\/code><\/pre>\n\n\n\n
    Possible architecture:<\/p>\n\n\n\n
    Topic:\nvehicle-telemetry\n\nKey:\nvehicle_id\n\nPartitions:\nplanned for ingestion + consumer concurrency\n\nGroups:\nvehicle-alerts\ntrip-engine\ntelemetry-analytics\ntelemetry-warehouse\n<\/code><\/pre>\n\n\n\n
    Each group independently reads the topic.<\/p>\n\n\n\n
    A problem in warehouse processing does not directly stop alerts from consuming.<\/p>\n\n\n\n
    \n\n\n\n
    78. Example Mapping \u2014 24 Partitions<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    vehicle-telemetry = 24 partitions\n<\/code><\/pre>\n\n\n\n
    Alert group:<\/p>\n\n\n\n
    6 consumers\n~4 partitions\/consumer\n<\/code><\/pre>\n\n\n\n
    Trip group:<\/p>\n\n\n\n
    12 consumers\n~2 partitions\/consumer\n<\/code><\/pre>\n\n\n\n
    Analytics group:<\/p>\n\n\n\n
    24 consumers\n~1 partition\/consumer\n<\/code><\/pre>\n\n\n\n
    Warehouse group:<\/p>\n\n\n\n
    4 consumers\n~6 partitions\/consumer\n<\/code><\/pre>\n\n\n\n
    Allowed? YES.<\/strong><\/p>\n\n\n\n
    Every group gets independent assignments across the same 24 partitions.<\/p>\n\n\n\n
    \n\n\n\n
    79. Can We Have 100 Consumer Groups?<\/h1>\n\n\n\n
    Architecturally, yes: Kafka supports many independent groups.<\/p>\n\n\n\n
    But do not use an arbitrary number without thinking about:<\/p>\n\n\n\n
    Read throughput\nConnections\nCoordinator load\nOffset state\nMonitoring\nAuthorization\nCost\nConfluent\/provider quotas\nBroker\/network capacity\n<\/code><\/pre>\n\n\n\n
    There is no universal best-practice number that applies to every cluster.<\/p>\n\n\n\n
    Capacity planning matters more than memorizing a magic limit.<\/p>\n\n\n\n
    \n\n\n\n
    80. Can We Have 1,000 Topics?<\/h1>\n\n\n\n
    The same principle applies.<\/p>\n\n\n\n
    Kafka can support large numbers of topics\/partitions depending on cluster sizing and platform limits.<\/p>\n\n\n\n
    But the meaningful capacity number is often influenced heavily by:<\/p>\n\n\n\n
    Total partitions\nReplication factor\nTraffic\nRetention\nStorage\nBroker count\nController\/metadata capacity\nClient count\nProvider quotas\n<\/code><\/pre>\n\n\n\n
    Do not ask only:<\/p>\n\n\n\n
    \"How many topics?\"\n<\/code><\/pre>\n\n\n\n
    Ask:<\/p>\n\n\n\n
    \"How many total partitions, replicas, bytes\/sec,\nconnections and retained bytes does this design create?\"\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    81. Total Partition Replica Count<\/h1>\n\n\n\n
    Example:<\/p>\n\n\n\n
    100 topics\n20 partitions each\nReplication Factor = 3\n<\/code><\/pre>\n\n\n\n
    Logical partitions:<\/p>\n\n\n\n
    100 * 20 = 2,000 partitions\n<\/code><\/pre>\n\n\n\n
    Replica copies:<\/p>\n\n\n\n
    2,000 * 3 = 6,000 partition replicas\n<\/code><\/pre>\n\n\n\n
    This is more meaningful operationally than saying:<\/p>\n\n\n\n
    \"We only have 100 topics.\"\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    82. Do Not Confuse Broker Count With Partition Count<\/h1>\n\n\n\n
    Example:<\/p>\n\n\n\n
    3 brokers\n12 partitions\n<\/code><\/pre>\n\n\n\n
    Perfectly normal.<\/p>\n\n\n\n
    Partitions are distributed across brokers.<\/p>\n\n\n\n
    You do not need 12 brokers for 12 partitions.<\/p>\n\n\n\n
    Broker count is driven by:<\/p>\n\n\n\n
    Throughput\nStorage\nAvailability\nFailure-domain requirements\nReplication\nNetwork\nCPU\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    83. Do Not Confuse Consumer Count With Broker Count<\/h1>\n\n\n\n
    There is no rule:<\/p>\n\n\n\n
    1 consumer per broker\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    3 Kafka brokers\n24 topic partitions\n12 consumers in a group\n<\/code><\/pre>\n\n\n\n
    is completely possible.<\/p>\n\n\n\n
    Broker count and consumer count solve different problems.<\/p>\n\n\n\n
    \n\n\n\n
    84. Do Not Confuse Consumer Group Count With Partition Count<\/h1>\n\n\n\n
    There is no rule:<\/p>\n\n\n\n
    1 group per partition\n<\/code><\/pre>\n\n\n\n
    One topic with six partitions may have one, ten or fifty groups depending on independent applications.<\/p>\n\n\n\n
    Each group tracks its own offsets across those partitions.<\/p>\n\n\n\n
    \n\n\n\n
    85. Master Mapping Rules<\/h1>\n\n\n\n
    Memorize these:<\/p>\n\n\n\n
    1 Cluster\n    -> many Topics\n\n1 Topic\n    -> many Partitions\n\n1 Partition\n    -> multiple Replicas\n\n1 Partition\n    -> one Leader at a time\n\nMany Producers\n    -> may write to same Topic\n\n1 Producer\n    -> may write to many Topics\n\n1 Consumer Group\n    -> may subscribe to many Topics\n\n1 Topic\n    -> may be consumed by many Consumer Groups\n\n1 Consumer\n    -> may own many Partitions\n\n1 Partition\n    -> at most one Consumer per Consumer Group at a time\n\nSame Partition\n    -> may be read independently by many different Consumer Groups\n\nConsumers > Partitions\n    -> extra consumers idle in that group\n\nPartitions > Consumers\n    -> consumers own multiple partitions\n\nReplicas\n    -> durability \/ availability\n\nPartitions\n    -> parallelism \/ ordering \/ scaling\n\nOffsets\n    -> group progress\n\nLag\n    -> backlog\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    86. What Is Allowed vs What Is Good Design?<\/h1>\n\n\n\n
    Kafka permits many configurations that are not good architecture.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    4 partitions\n40 consumers\n<\/code><\/pre>\n\n\n\n
    Allowed: YES<\/strong>.<\/p>\n\n\n\n
    Good design: usually NO<\/strong>, because most consumers are idle.<\/p>\n\n\n\n
    Another example:<\/p>\n\n\n\n
    Same group.id for analytics and billing\n<\/code><\/pre>\n\n\n\n
    Allowed: YES<\/strong>.<\/p>\n\n\n\n
    Correct if both must independently read every record: NO<\/strong>.<\/p>\n\n\n\n
    Kafka does not know your business requirement.<\/p>\n\n\n\n
    Developers must design the mapping correctly.<\/p>\n\n\n\n
    \n\n\n\n
    87. Best-Practice Topic Checklist<\/h1>\n\n\n\n
    Before creating a topic:<\/p>\n\n\n\n
      \n
    • [ ] Is the business purpose clear?<\/li>\n\n\n\n
    • [ ] Is ownership clear?<\/li>\n\n\n\n
    • [ ] Is the event schema\/data contract clear?<\/li>\n\n\n\n
    • [ ] Is the partition key clear?<\/li>\n\n\n\n
    • [ ] Is ordering requirement documented?<\/li>\n\n\n\n
    • [ ] Is partition count capacity-planned?<\/li>\n\n\n\n
    • [ ] Is retention defined?<\/li>\n\n\n\n
    • [ ] Is compaction required?<\/li>\n\n\n\n
    • [ ] Is security\/ACL policy defined?<\/li>\n\n\n\n
    • [ ] Is replication\/durability requirement defined?<\/li>\n\n\n\n
    • [ ] Is expected throughput documented?<\/li>\n\n\n\n
    • [ ] Is growth estimate documented?<\/li>\n\n\n\n
    • [ ] Are consumers\/use cases known?<\/li>\n<\/ul>\n\n\n\n
      \n\n\n\n
      88. Best-Practice Partition Checklist<\/h1>\n\n\n\n
      Before choosing partition count:<\/p>\n\n\n\n
        \n
      • [ ] Measure expected producer throughput<\/li>\n\n\n\n
      • [ ] Measure expected consumer processing throughput<\/li>\n\n\n\n
      • [ ] Determine required consumer parallelism<\/li>\n\n\n\n
      • [ ] Document per-key ordering requirements<\/li>\n\n\n\n
      • [ ] Check key cardinality and skew<\/li>\n\n\n\n
      • [ ] Include growth headroom<\/li>\n\n\n\n
      • [ ] Include failure headroom<\/li>\n\n\n\n
      • [ ] Consider broker capacity<\/li>\n\n\n\n
      • [ ] Consider replication overhead<\/li>\n\n\n\n
      • [ ] Consider operational overhead<\/li>\n\n\n\n
      • [ ] Understand that partition count is difficult to reduce<\/li>\n\n\n\n
      • [ ] Understand that increasing partitions may change key mapping<\/li>\n<\/ul>\n\n\n\n
        \n\n\n\n
        89. Best-Practice Consumer Group Checklist<\/h1>\n\n\n\n
        For each Consumer Group:<\/p>\n\n\n\n
          \n
        • [ ] One clear logical application\/use case<\/li>\n\n\n\n
        • [ ] Intentional and unique group.id<\/code><\/li>\n\n\n\n
        • [ ] Topic subscriptions documented<\/li>\n\n\n\n
        • [ ] Consumer count based on measured processing demand<\/li>\n\n\n\n
        • [ ] Consumer count compared with partition count<\/li>\n\n\n\n
        • [ ] Rebalance behavior understood<\/li>\n\n\n\n
        • [ ] Offset commit strategy defined<\/li>\n\n\n\n
        • [ ] auto.offset.reset<\/code> intentional<\/li>\n\n\n\n
        • [ ] Lag SLO defined<\/li>\n\n\n\n
        • [ ] Lag monitored per partition<\/li>\n\n\n\n
        • [ ] Downstream dependencies monitored<\/li>\n\n\n\n
        • [ ] Failure\/restart behavior tested<\/li>\n\n\n\n
        • [ ] Duplicate-processing behavior understood<\/li>\n\n\n\n
        • [ ] Idempotency implemented where appropriate<\/li>\n<\/ul>\n\n\n\n
          \n\n\n\n
          90. Best-Practice Lag Checklist<\/h1>\n\n\n\n
          When lag increases:<\/p>\n\n\n\n
            \n
          • [ ] Check lag per group\/topic\/partition<\/li>\n\n\n\n
          • [ ] Check producer\/input rate<\/li>\n\n\n\n
          • [ ] Check consumer completion rate<\/li>\n\n\n\n
          • [ ] Check current vs committed progress<\/li>\n\n\n\n
          • [ ] Check consumer count<\/li>\n\n\n\n
          • [ ] Check partition count<\/li>\n\n\n\n
          • [ ] Check for hot partitions<\/li>\n\n\n\n
          • [ ] Check CPU<\/li>\n\n\n\n
          • [ ] Check memory \/ GC<\/li>\n\n\n\n
          • [ ] Check database latency<\/li>\n\n\n\n
          • [ ] Check API latency<\/li>\n\n\n\n
          • [ ] Check fetch latency<\/li>\n\n\n\n
          • [ ] Check commit latency\/errors<\/li>\n\n\n\n
          • [ ] Check rebalances<\/li>\n\n\n\n
          • [ ] Check broker throttling<\/li>\n\n\n\n
          • [ ] Check network issues<\/li>\n\n\n\n
          • [ ] Check poison records<\/li>\n\n\n\n
          • [ ] Check retry loops<\/li>\n\n\n\n
          • [ ] Check max poll behavior<\/li>\n\n\n\n
          • [ ] Scale only after identifying the bottleneck<\/li>\n<\/ul>\n\n\n\n
            \n\n\n\n
            91. Anti-Patterns<\/h1>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"More consumers always fix lag.\"\n<\/code><\/pre>\n\n\n\n
            Wrong.<\/p>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"Replicas give more consumer parallelism.\"\n<\/code><\/pre>\n\n\n\n
            Wrong.<\/p>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"Every service should use the same group.id.\"\n<\/code><\/pre>\n\n\n\n
            Wrong when services need independent copies of the stream.<\/p>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"One topic for every producer.\"\n<\/code><\/pre>\n\n\n\n
            Not a Kafka rule.<\/p>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"One consumer for every partition is always best.\"\n<\/code><\/pre>\n\n\n\n
            Not necessarily.<\/p>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"Reset offsets to latest whenever lag is high.\"\n<\/code><\/pre>\n\n\n\n
            Dangerous. It can skip business data.<\/p>\n\n\n\n
            Avoid:<\/p>\n\n\n\n
            \"Increase partitions whenever performance is slow.\"\n<\/code><\/pre>\n\n\n\n
            First identify the bottleneck.<\/p>\n\n\n\n
            \n\n\n\n
            92. Developer Decision Flow<\/h1>\n\n\n\n
            Use this sequence when designing a new Kafka stream.<\/p>\n\n\n\n
            STEP 1\nWhat business events are we publishing?\n        |\n        v\nDefine Topic\n\nSTEP 2\nWhich events must remain ordered together?\n        |\n        v\nChoose Key\n\nSTEP 3\nWhat throughput and consumer parallelism do we need?\n        |\n        v\nPlan Partitions\n\nSTEP 4\nHow much broker failure should data survive?\n        |\n        v\nChoose Replication \/ durability policy\n\nSTEP 5\nWhich independent applications need the data?\n        |\n        v\nCreate Consumer Groups\n\nSTEP 6\nHow much processing capacity does each group need?\n        |\n        v\nChoose Consumer Count per Group\n\nSTEP 7\nWhat are the processing correctness requirements?\n        |\n        v\nChoose Offset Commit \/ Delivery Strategy\n\nSTEP 8\nWhat backlog is acceptable?\n        |\n        v\nDefine Lag SLO + Alerts\n\nSTEP 9\nLoad test and failure test\n        |\n        v\nTune based on measurements\n<\/code><\/pre>\n\n\n\n
            \n\n\n\n
            93. Example Developer Worksheet<\/h1>\n\n\n\n
            Topic<\/h2>\n\n\n\n
            orders\n<\/code><\/pre>\n\n\n\n
            Business owner<\/h2>\n\n\n\n
            Order Platform Team\n<\/code><\/pre>\n\n\n\n
            Key<\/h2>\n\n\n\n
            order_id\n<\/code><\/pre>\n\n\n\n
            Ordering requirement<\/h2>\n\n\n\n
            Events for one order must stay ordered.\n<\/code><\/pre>\n\n\n\n
            Peak input<\/h2>\n\n\n\n
            80,000 records\/sec\n<\/code><\/pre>\n\n\n\n
            Partitions<\/h2>\n\n\n\n
            To be selected after throughput and consumer benchmarks.\n<\/code><\/pre>\n\n\n\n
            Independent groups<\/h2>\n\n\n\n
            order-fulfillment\norder-fraud\norder-analytics\norder-notifications\n<\/code><\/pre>\n\n\n\n
            Fulfillment consumer capacity<\/h2>\n\n\n\n
            Measured: 8,000 records\/sec per consumer\n<\/code><\/pre>\n\n\n\n
            Approximate consumer requirement at 80k:<\/p>\n\n\n\n
            80,000 \/ 8,000 = 10 active consumers\n<\/code><\/pre>\n\n\n\n
            Therefore the topic must expose enough partition-level concurrency for the desired group scaling.<\/p>\n\n\n\n
            Add capacity\/failure headroom after benchmarking.<\/p>\n\n\n\n
            \n\n\n\n
            94. Example Lag SLO<\/h1>\n\n\n\n
            Do not monitor lag without defining what “bad” means.<\/p>\n\n\n\n
            Possible SLO:<\/p>\n\n\n\n
            Normal:\n< 5,000 records lag per partition\n\nWarning:\nlag > 20,000 for 5 minutes\n\nCritical:\nlag > 100,000\nor\nestimated time-behind > 2 minutes\n<\/code><\/pre>\n\n\n\n
            For some workloads, time lag<\/strong> is more meaningful than raw record count.<\/p>\n\n\n\n
            Ten thousand tiny telemetry records may be trivial, while ten thousand expensive payment events may be a serious backlog.<\/p>\n\n\n\n
            Measure what matters to the business.<\/p>\n\n\n\n
            \n\n\n\n
            95. Lag Recovery Capacity<\/h1>\n\n\n\n
            A healthy group needs enough spare capacity to catch up.<\/p>\n\n\n\n
            Suppose:<\/p>\n\n\n\n
            Normal input = 50k\/sec\nConsumer max = 52k\/sec\n<\/code><\/pre>\n\n\n\n
            Spare capacity:<\/p>\n\n\n\n
            2k\/sec\n<\/code><\/pre>\n\n\n\n
            A backlog of:<\/p>\n\n\n\n
            1,000,000 records\n<\/code><\/pre>\n\n\n\n
            would take approximately:<\/p>\n\n\n\n
            1,000,000 \/ 2,000 = 500 seconds \u2248 8.3 minutes\n<\/code><\/pre>\n\n\n\n
            to clear if rates remain stable.<\/p>\n\n\n\n
            If your recovery SLO is two minutes, this design has insufficient catch-up capacity.<\/p>\n\n\n\n
            \n\n\n\n
            96. Plan for Failure, Not Only Average Load<\/h1>\n\n\n\n
            Suppose:<\/p>\n\n\n\n
            12 partitions\n6 consumers\n<\/code><\/pre>\n\n\n\n
            Each consumer processes about two partitions.<\/p>\n\n\n\n
            If one consumer fails, remaining members may receive its partitions.<\/p>\n\n\n\n
            If the remaining consumers are already saturated:<\/p>\n\n\n\n
            failover\n     |\n     v\nwork gets reassigned\n     |\n     v\nbut capacity is insufficient\n     |\n     v\nlag grows\n<\/code><\/pre>\n\n\n\n
            Production sizing should include spare failure capacity.<\/p>\n\n\n\n
            \n\n\n\n
            97. The Four Most Important Distinctions<\/h1>\n\n\n\n
            Distinction 1<\/h2>\n\n\n\n
            Topics != Partitions\n<\/code><\/pre>\n\n\n\n
            Topic is logical stream. Partition is scaling\/order\/storage unit.<\/p>\n\n\n\n
            Distinction 2<\/h2>\n\n\n\n
            Partitions != Replicas\n<\/code><\/pre>\n\n\n\n
            Partitions create parallelism. Replicas create durability\/availability.<\/p>\n\n\n\n
            Distinction 3<\/h2>\n\n\n\n
            Consumers != Consumer Groups\n<\/code><\/pre>\n\n\n\n
            Consumer is a worker. Consumer Group is the logical application\/subscriber.<\/p>\n\n\n\n
            Distinction 4<\/h2>\n\n\n\n
            Lag != Kafka is broken\n<\/code><\/pre>\n\n\n\n
            Lag means the group is behind.<\/p>\n\n\n\n
            The root cause may be Kafka, consumer code, database, API, partition skew, CPU, network, rebalancing or commit strategy.<\/p>\n\n\n\n
            Measure before tuning.<\/p>\n\n\n\n
            \n\n\n\n
            98. One Diagram to Remember Everything<\/h1>\n\n\n\n
            PRODUCERS\n   |\n   | many producers allowed\n   v\n+---------------------------------------------------+\n| TOPIC: orders                                     |\n|                                                   |\n| P0        P1        P2        P3        P4        |\n| |         |         |         |         |         |\n| replicas  replicas  replicas  replicas  replicas  |\n+---------------------------------------------------+\n         |                     |\n         |                     |\n         v                     v\n\nConsumer Group A          Consumer Group B\n\"fulfillment\"             \"analytics\"\n\nC1 -> P0,P2               C1 -> P0\nC2 -> P1,P3               C2 -> P1\nC3 -> P4                  C3 -> P2\n                           C4 -> P3\n                           C5 -> P4\n\nEach group:\n- gets independent access to all partitions\n- has its own committed offsets\n- has its own lag\n- scales consumers independently\n\nWithin one group:\n- one partition -> max one assigned consumer at a time\n\nAcross different groups:\n- same partition -> many independent readers are normal\n<\/code><\/pre>\n\n\n\n
            \n\n\n\n
            99. Quick Answers to Session Questions<\/h1>\n\n\n\n
            How many topics can a cluster have?<\/h2>\n\n\n\n
            Many. There is no one universal correct number. Total partitions, replicas, throughput, storage, metadata and provider limits matter more than topic count alone.<\/p>\n\n\n\n
            How many partitions can a topic have?<\/h2>\n\n\n\n
            One or many. Choose based on throughput, consumer parallelism, ordering and operational capacity.<\/p>\n\n\n\n
            How many consumers can read one topic?<\/h2>\n\n\n\n
            Across different Consumer Groups: many.<\/p>\n\n\n\n
            Inside one Consumer Group: many consumers may exist, but useful partition-level concurrency is limited by the available partitions.<\/p>\n\n\n\n
            Can one consumer read many partitions?<\/h2>\n\n\n\n
            Yes.<\/p>\n\n\n\n
            Can two consumers in the same group read one partition simultaneously?<\/h2>\n\n\n\n
            Normally no. One partition is assigned to at most one consumer in that group at a time.<\/p>\n\n\n\n
            Can two consumers in different groups read the same partition?<\/h2>\n\n\n\n
            Yes.<\/p>\n\n\n\n
            Can one Consumer Group read many topics?<\/h2>\n\n\n\n
            Yes.<\/p>\n\n\n\n
            Can one topic have many Consumer Groups?<\/h2>\n\n\n\n
            Yes.<\/p>\n\n\n\n
            Does replication factor increase consumer parallelism?<\/h2>\n\n\n\n
            No.<\/p>\n\n\n\n
            Do more consumers always reduce lag?<\/h2>\n\n\n\n
            No.<\/p>\n\n\n\n
            Do more partitions always improve performance?<\/h2>\n\n\n\n
            No.<\/p>\n\n\n\n
            Can we reduce partitions later?<\/h2>\n\n\n\n
            Not as a normal direct Kafka operation. Plan carefully.<\/p>\n\n\n\n
            What is lag?<\/h2>\n\n\n\n
            The distance between the available end of a partition and a Consumer Group’s processing\/progress position.<\/p>\n\n\n\n
            Where should we look first when lag grows?<\/h2>\n\n\n\n
            Per-partition lag, processing rate, consumer count, partition count, downstream latency, rebalances and hot partitions.<\/p>\n\n\n\n
            \n\n\n\n
            100. Final Master Rules<\/h1>\n\n\n\n
            Remember these ten rules:<\/p>\n\n\n\n
              \n
            1. Topic = business stream.<\/strong><\/li>\n\n\n\n
            2. Partition = ordering + parallelism + storage unit.<\/strong><\/li>\n\n\n\n
            3. Replica = durability and availability, not consumer concurrency.<\/strong><\/li>\n\n\n\n
            4. Consumer Group = one independent logical subscriber\/application.<\/strong><\/li>\n\n\n\n
            5. Consumer = worker inside the group.<\/strong><\/li>\n\n\n\n
            6. One partition can be assigned to only one consumer per group at a time.<\/strong><\/li>\n\n\n\n
            7. Different groups can independently consume the same partition.<\/strong><\/li>\n\n\n\n
            8. Consumer parallelism is bounded by partition-level work.<\/strong><\/li>\n\n\n\n
            9. Lag grows when production outpaces durable consumer progress.<\/strong><\/li>\n\n\n\n
            10. Fix lag by finding the bottleneck\u2014not by blindly adding consumers or skipping offsets.<\/strong><\/li>\n<\/ol>\n\n\n\n
              \n\n\n\n
              101. Recommended Planning Order for Developers<\/h1>\n\n\n\n
              Use this order in every architecture discussion:<\/p>\n\n\n\n
              1. Business Event\n2. Topic\n3. Key\n4. Ordering Requirement\n5. Peak Throughput\n6. Partition Count\n7. Replication \/ Durability\n8. Independent Consumer Groups\n9. Consumers Per Group\n10. Offset Strategy\n11. Lag SLO\n12. Monitoring\n13. Load Test\n14. Failure Test\n15. Tune\n<\/code><\/pre>\n\n\n\n
              This order prevents most of the confusion that happens when teams start by asking:<\/p>\n\n\n\n
              \"How many consumers should we create?\"\n<\/code><\/pre>\n\n\n\n
              before they have defined topics, keys, partitions, workloads and correctness requirements.<\/p>\n\n\n\n
              \n\n\n\n
              102. Final Developer Mental Model<\/h1>\n\n\n\n
              Do not ask these as isolated questions:<\/p>\n\n\n\n
              How many Kafka topics are allowed?\nHow many partitions are allowed?\nHow many consumers are allowed?\n<\/code><\/pre>\n\n\n\n
              Ask:<\/p>\n\n\n\n
              What workload are we designing?\n\nWhat ordering do we need?\n\nWhat throughput do we need?\n\nHow many independent applications need the stream?\n\nHow quickly must every group process it?\n\nWhat failure should the system survive?\n\nWhat lag\/recovery SLO do we have?\n<\/code><\/pre>\n\n\n\n
              Then map:<\/p>\n\n\n\n
              Business streams\n        -> Topics\n\nOrdering + throughput\n        -> Keys + Partitions\n\nIndependent use cases\n        -> Consumer Groups\n\nProcessing capacity\n        -> Consumers per Group\n\nDurability\n        -> Replication\n\nRecovery progress\n        -> Offsets\n\nCapacity health\n        -> Lag\n<\/code><\/pre>\n\n\n\n
              That is the cleanest way to plan Kafka without mixing unrelated concepts.<\/p>\n","protected":false},"excerpt":{"rendered":"
              Developer Planning Guide for Correct Mapping, Scaling, Reliability and Performance Audience: Developers, students, freshers, architects, platform engineersTraining context: Confluent Kafka ClusterGoal: Remove confusion around how Kafka Topics, Partitions, Producers, Consumers,… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1154","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1154","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1154"}],"version-history":[{"count":2,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1154\/revisions"}],"predecessor-version":[{"id":1156,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1154\/revisions\/1156"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1154"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1154"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1154"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}