{"id":1152,"date":"2026-08-19T02:38:14","date_gmt":"2026-08-19T02:38:14","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/?p=1152"},"modified":"2026-08-19T02:38:15","modified_gmt":"2026-08-19T02:38:15","slug":"kafka-master-tutorials-series-6-kafka-consumer-deep-dive","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/kafka-master-tutorials-series-6-kafka-consumer-deep-dive\/","title":{"rendered":"Kafka Master Tutorials Series: 6 – Kafka Consumer Deep Dive"},"content":{"rendered":"\n

Consumer Groups, Parallelism, Offsets, Rebalancing, Failover, Consumption Patterns and Production Tuning<\/h2>\n\n\n\n
\n

Audience:<\/strong> Students and freshers with no previous Kafka experience
Goal:<\/strong> Start with \u201cWhat is a consumer?\u201d and finish with production-grade consumer design, reliability, scaling, recovery, and performance tuning
Training environment:<\/strong> Confluent Kafka Cluster
Technical version note:<\/strong> This tutorial is aligned with modern Apache Kafka 4.x consumer concepts. Where the newer Consumer rebalance protocol differs from the older Classic protocol, both are explained explicitly.<\/p>\n<\/blockquote>\n\n\n\n

\"Kafka<\/figure>\n\n\n\n
\n

Image note:<\/strong> The infographic is a learning map. The written tutorial below is the source of truth for protocol details, especially the differences between the newer Consumer group protocol and the older Classic protocol.<\/p>\n<\/blockquote>\n\n\n\n

\n\n\n\n

1. Learning Objectives<\/h1>\n\n\n\n

By the end of this tutorial, you should be able to explain:<\/p>\n\n\n\n

    \n
  • What a Kafka Consumer is<\/li>\n\n\n\n
  • How a consumer reads records from Kafka<\/li>\n\n\n\n
  • Why consumers\u00a0pull<\/strong>\u00a0data rather than Kafka pushing records to them<\/li>\n\n\n\n
  • How topic partitions map to consumers<\/li>\n\n\n\n
  • What a Consumer Group is<\/li>\n\n\n\n
  • How Consumer Groups provide parallel processing<\/li>\n\n\n\n
  • How Kafka balances workload across consumers<\/li>\n\n\n\n
  • Why the number of partitions limits parallelism inside one consumer group<\/li>\n\n\n\n
  • What the Group Coordinator does<\/li>\n\n\n\n
  • What rebalancing means<\/li>\n\n\n\n
  • What causes a rebalance<\/li>\n\n\n\n
  • What happens when a consumer joins<\/li>\n\n\n\n
  • What happens when a consumer leaves or crashes<\/li>\n\n\n\n
  • What happens when topic partitions change<\/li>\n\n\n\n
  • How the modern Consumer rebalance protocol differs from the Classic protocol<\/li>\n\n\n\n
  • What Kafka offsets are<\/li>\n\n\n\n
  • Current position vs committed offset<\/li>\n\n\n\n
  • Automatic vs manual offset commits<\/li>\n\n\n\n
  • commitSync()<\/code>\u00a0vs\u00a0commitAsync()<\/code><\/li>\n\n\n\n
  • How to choose an offset strategy<\/li>\n\n\n\n
  • How consumer failover and recovery work<\/li>\n\n\n\n
  • Fan-out consumption<\/li>\n\n\n\n
  • Load-balanced consumption<\/li>\n\n\n\n
  • At-most-once vs at-least-once vs exactly-once approaches<\/li>\n\n\n\n
  • Consumer lag<\/li>\n\n\n\n
  • Consumer performance tuning<\/li>\n\n\n\n
  • Consumer configuration settings that matter in production<\/li>\n\n\n\n
  • Common consumer mistakes<\/li>\n\n\n\n
  • A production-ready consumer checklist<\/li>\n<\/ul>\n\n\n\n
    \n\n\n\n

    2. What Is a Kafka Consumer?<\/h1>\n\n\n\n

    A Kafka Consumer<\/strong> is a client application that reads records from Kafka topics.<\/p>\n\n\n\n

    The simplest mental model is:<\/p>\n\n\n\n

    Producer\n   |\n   v\nKafka Topic\n   |\n   v\nConsumer\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Vehicle\n   |\n   v\nProducer\n   |\n   v\nTopic: vehicle-telemetry\n   |\n   v\nAnalytics Consumer\n<\/code><\/pre>\n\n\n\n
    The consumer receives records such as:<\/p>\n\n\n\n
    {\n  \"vehicle_id\": \"CAR-101\",\n  \"speed\": 88,\n  \"battery\": 72\n}\n<\/code><\/pre>\n\n\n\n
    and performs business work:<\/p>\n\n\n\n
    Read event\n   |\n   v\nDeserialize\n   |\n   v\nValidate\n   |\n   v\nProcess\n   |\n   +--> Update database\n   +--> Generate alert\n   +--> Calculate analytics\n   +--> Call another service\n   |\n   v\nRecord progress\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    3. Kafka Consumers Pull Data<\/h1>\n\n\n\n
    Kafka consumers normally pull<\/strong> records from brokers.<\/p>\n\n\n\n
    Kafka does not continuously push records into your application without the consumer asking.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Consumer\n   |\n   | \"Give me records from Partition 2\n   |  beginning at my current position.\"\n   v\nBroker\n   |\n   v\nRecord batch\n<\/code><\/pre>\n\n\n\n
    The consumer repeatedly calls:<\/p>\n\n\n\n
    consumer.poll(...)\n<\/code><\/pre>\n\n\n\n
    Think of poll()<\/code> as:<\/p>\n\n\n\n
    \n
    \u201cGive me the records that I am currently allowed to read from my assigned partitions.\u201d<\/p>\n<\/blockquote>\n\n\n\n
    This pull design gives the consumer control over:<\/p>\n\n\n\n
    How quickly it reads\nHow many records it handles\nWhen it pauses\nWhen it resumes\nHow it tracks progress\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    4. Consumer End-to-End Flow<\/h1>\n\n\n\n
    At a high level:<\/p>\n\n\n\n
    Application starts\n      |\n      v\nCreate KafkaConsumer\n      |\n      v\nConnect to Kafka\n      |\n      v\nJoin Consumer Group\n      |\n      v\nReceive Partition Assignment\n      |\n      v\nFetch Records\n      |\n      v\nDeserialize\n      |\n      v\nProcess Business Logic\n      |\n      v\nCommit Offset\n      |\n      v\npoll() again\n<\/code><\/pre>\n\n\n\n
    This loop continues for the life of the consumer.<\/p>\n\n\n\n
    \n\n\n\n
    5. What Does a Consumer Read?<\/h1>\n\n\n\n
    A Kafka consumer receives ConsumerRecord<\/code> objects.<\/p>\n\n\n\n
    A record includes important information such as:<\/p>\n\n\n\n
    Topic\nPartition\nOffset\nTimestamp\nKey\nValue\nHeaders\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Topic     = vehicle-telemetry\nPartition = 2\nOffset    = 1050\nKey       = CAR-101\nValue     = {\"speed\":88}\n<\/code><\/pre>\n\n\n\n
    This information matters because a consumer must know:<\/p>\n\n\n\n
    Where did the record come from?\nWhat order was it stored in?\nHow far have I progressed?\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    6. Deserialization<\/h1>\n\n\n\n
    The producer serialized application data into bytes.<\/p>\n\n\n\n
    The consumer must reverse that process.<\/p>\n\n\n\n
    Kafka bytes\n    |\n    v\nDeserializer\n    |\n    v\nApplication Object\n<\/code><\/pre>\n\n\n\n
    For example:<\/p>\n\n\n\n
    key.deserializer=org.apache.kafka.common.serialization.StringDeserializer\nvalue.deserializer=org.apache.kafka.common.serialization.StringDeserializer\n<\/code><\/pre>\n\n\n\n
    With Confluent Schema Registry, a consumer may instead use schema-aware deserializers for:<\/p>\n\n\n\n
    Avro\nProtobuf\nJSON Schema\n<\/code><\/pre>\n\n\n\n
    A producer and consumer must agree on how data is encoded.<\/p>\n\n\n\n
    \n\n\n\n
    7. What Is a Consumer Group?<\/h1>\n\n\n\n
    A Consumer Group<\/strong> is a group of consumers cooperating to read one or more topics.<\/p>\n\n\n\n
    Consumers belong to the same logical group by using the same:<\/p>\n\n\n\n
    group.id\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    group.id = order-processing\n<\/code><\/pre>\n\n\n\n
    Consumers:<\/p>\n\n\n\n
    Consumer A\nConsumer B\nConsumer C\n<\/code><\/pre>\n\n\n\n
    Together:<\/p>\n\n\n\n
    Consumer Group: order-processing\n<\/code><\/pre>\n\n\n\n
    Kafka treats them as workers sharing one consumption workload.<\/p>\n\n\n\n
    \n\n\n\n
    8. Why Consumer Groups Exist<\/h1>\n\n\n\n
    Suppose the topic receives:<\/p>\n\n\n\n
    100,000 events \/ second\n<\/code><\/pre>\n\n\n\n
    One consumer may not be able to process all of them.<\/p>\n\n\n\n
    We want:<\/p>\n\n\n\n
    Consumer 1\nConsumer 2\nConsumer 3\nConsumer 4\n...\n<\/code><\/pre>\n\n\n\n
    to divide the work.<\/p>\n\n\n\n
    Kafka accomplishes this by distributing partitions<\/strong> among consumers in the same group.<\/p>\n\n\n\n
    \n\n\n\n
    9. Consumer Group + Partitions = Parallel Processing<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    Topic: orders\n\nPartitions:\nP0\nP1\nP2\n<\/code><\/pre>\n\n\n\n
    One consumer:<\/p>\n\n\n\n
    Consumer Group\n\nConsumer A\n   |\n   +--> P0\n   +--> P1\n   +--> P2\n<\/code><\/pre>\n\n\n\n
    There is only one consumer process doing all the work.<\/p>\n\n\n\n
    Add three consumers:<\/p>\n\n\n\n
    P0 --> Consumer A\nP1 --> Consumer B\nP2 --> Consumer C\n<\/code><\/pre>\n\n\n\n
    Now the group can process three partitions in parallel.<\/p>\n\n\n\n
    This is one of Kafka’s most important scaling mechanisms.<\/p>\n\n\n\n
    \n\n\n\n
    10. Golden Rule of Consumer Group Parallelism<\/h1>\n\n\n\n
    Inside a normal Kafka consumer group:<\/p>\n\n\n\n
    \n
    One partition is assigned to at most one consumer in that group at a time.<\/p>\n<\/blockquote>\n\n\n\n
    Therefore:<\/p>\n\n\n\n
    Maximum useful partition-level consumer parallelism\napproximately equals\nnumber of partitions available to the group.\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    6 partitions\n1 consumer\n\nConsumer 1 -> P0 P1 P2 P3 P4 P5\n<\/code><\/pre>\n\n\n\n
    6 partitions\n3 consumers\n\nConsumer 1 -> P0 P3\nConsumer 2 -> P1 P4\nConsumer 3 -> P2 P5\n<\/code><\/pre>\n\n\n\n
    6 partitions\n6 consumers\n\n1 partition per consumer\n<\/code><\/pre>\n\n\n\n
    6 partitions\n10 consumers\n\n6 consumers can receive partitions\n4 consumers have no partition work\n<\/code><\/pre>\n\n\n\n
    Adding consumers beyond the available partitions does not create additional partition-level parallelism.<\/p>\n\n\n\n
    \n\n\n\n
    11. One Consumer Can Read Multiple Partitions<\/h1>\n\n\n\n
    Do not misunderstand the previous rule.<\/p>\n\n\n\n
    Kafka does not<\/strong> require:<\/p>\n\n\n\n
    1 consumer = 1 partition\n<\/code><\/pre>\n\n\n\n
    A consumer can own many partitions.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    12 partitions\n3 consumers\n<\/code><\/pre>\n\n\n\n
    A possible assignment:<\/p>\n\n\n\n
    Consumer A -> P0 P3 P6 P9\nConsumer B -> P1 P4 P7 P10\nConsumer C -> P2 P5 P8 P11\n<\/code><\/pre>\n\n\n\n
    This is normal.<\/p>\n\n\n\n
    \n\n\n\n
    12. Why Partition Count Is So Important<\/h1>\n\n\n\n
    Topic partition count affects:<\/p>\n\n\n\n
    Producer parallelism\nBroker distribution\nConsumer parallelism\nScaling ceiling\nOrdering boundaries\nOperational overhead\n<\/code><\/pre>\n\n\n\n
    For consumers:<\/p>\n\n\n\n
    Too few partitions\n        |\n        v\nCannot add useful consumers beyond partition count\n        |\n        v\nLimited parallel processing\n<\/code><\/pre>\n\n\n\n
    But:<\/p>\n\n\n\n
    Huge number of partitions\n        |\n        v\nMore metadata\nMore files\nMore leader\/replica management\nMore operational overhead\n<\/code><\/pre>\n\n\n\n
    Do not create partitions without capacity planning.<\/p>\n\n\n\n
    \n\n\n\n
    13. How Kafka Balances Work Across Consumers<\/h1>\n\n\n\n
    Kafka does not balance individual records directly.<\/p>\n\n\n\n
    It balances partition ownership<\/strong>.<\/p>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    P0 P1 P2 P3 P4 P5\n<\/code><\/pre>\n\n\n\n
    and:<\/p>\n\n\n\n
    Consumer A\nConsumer B\nConsumer C\n<\/code><\/pre>\n\n\n\n
    Kafka determines an assignment such as:<\/p>\n\n\n\n
    Consumer A -> P0 P3\nConsumer B -> P1 P4\nConsumer C -> P2 P5\n<\/code><\/pre>\n\n\n\n
    Each consumer fetches records from the leaders of its assigned partitions.<\/p>\n\n\n\n
    So load balancing is really:<\/p>\n\n\n\n
    Partition Assignment\n        |\n        v\nConsumer Work Distribution\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    14. Important Limitation: Partitions May Not Have Equal Work<\/h1>\n\n\n\n
    Kafka can distribute partitions reasonably, but it cannot magically guarantee equal CPU work.<\/p>\n\n\n\n
    Imagine:<\/p>\n\n\n\n
    P0 = 1,000 events\/sec\nP1 = 1,100 events\/sec\nP2 = 50,000 events\/sec\n<\/code><\/pre>\n\n\n\n
    If:<\/p>\n\n\n\n
    Consumer A -> P0\nConsumer B -> P1\nConsumer C -> P2\n<\/code><\/pre>\n\n\n\n
    Consumer C still has much more work.<\/p>\n\n\n\n
    Why?<\/p>\n\n\n\n
    Because Kafka assigns partitions<\/strong>, not perfectly equal units of business computation.<\/p>\n\n\n\n
    This is why producer key design and partition distribution affect consumer performance.<\/p>\n\n\n\n
    \n\n\n\n
    15. Consumer Group ID<\/h1>\n\n\n\n
    group.id<\/code> identifies the consumer group.<\/p>\n\n\n\n
    Different group.id<\/code> values create independent consumption.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Topic: orders\n<\/code><\/pre>\n\n\n\n
    Group 1:<\/p>\n\n\n\n
    group.id = analytics\n<\/code><\/pre>\n\n\n\n
    Group 2:<\/p>\n\n\n\n
    group.id = fraud-detection\n<\/code><\/pre>\n\n\n\n
    Group 3:<\/p>\n\n\n\n
    group.id = notifications\n<\/code><\/pre>\n\n\n\n
    All three groups can independently read the same orders<\/code> topic.<\/p>\n\n\n\n
    This becomes our fan-out pattern<\/strong> later.<\/p>\n\n\n\n
    \n\n\n\n
    16. The Group Coordinator<\/h1>\n\n\n\n
    A Kafka broker acts as the Group Coordinator<\/strong> for a consumer group.<\/p>\n\n\n\n
    The coordinator manages important group responsibilities such as:<\/p>\n\n\n\n
    Group membership\nPartition assignment coordination\nConsumer liveness\nRebalance coordination\nOffset commits\n<\/code><\/pre>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Consumer Group\n      |\n      v\nGroup Coordinator\n      |\n      v\nKafka\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    17. How Kafka Finds the Group Coordinator<\/h1>\n\n\n\n
    Kafka has an internal topic:<\/p>\n\n\n\n
    __consumer_offsets\n<\/code><\/pre>\n\n\n\n
    The consumer group’s group.id<\/code> maps to a partition of this internal topic.<\/p>\n\n\n\n
    The broker that leads that offsets partition becomes the coordinator for the group.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    group.id\n   |\n   v\nhash \/ mapping\n   |\n   v\n__consumer_offsets partition\n   |\n   v\nLeader Broker\n   |\n   v\nGroup Coordinator\n<\/code><\/pre>\n\n\n\n
    This distributes consumer-group coordination across brokers.<\/p>\n\n\n\n
    \n\n\n\n
    18. What Is Rebalancing?<\/h1>\n\n\n\n
    A rebalance<\/strong> is the process of changing partition assignments among consumers in a group.<\/p>\n\n\n\n
    Example before:<\/p>\n\n\n\n
    Consumer A -> P0 P1\nConsumer B -> P2 P3\n<\/code><\/pre>\n\n\n\n
    Consumer C joins.<\/p>\n\n\n\n
    After rebalance:<\/p>\n\n\n\n
    Consumer A -> P0\nConsumer B -> P1 P2\nConsumer C -> P3\n<\/code><\/pre>\n\n\n\n
    The exact assignment depends on the selected protocol and assignment strategy.<\/p>\n\n\n\n
    The key idea is:<\/p>\n\n\n\n
    \n
    Group membership or subscribed-partition changes may require Kafka to redistribute partition ownership.<\/p>\n<\/blockquote>\n\n\n\n
    \n\n\n\n
    19. Why Rebalancing Exists<\/h1>\n\n\n\n
    Without rebalancing:<\/p>\n\n\n\n
    Consumer B crashes\n      |\n      v\nP2 and P3 have no active consumer\n      |\n      v\nProcessing stops forever\n<\/code><\/pre>\n\n\n\n
    With rebalancing:<\/p>\n\n\n\n
    Consumer B crashes\n      |\n      v\nKafka detects membership change\n      |\n      v\nPartitions reassigned\n      |\n      v\nConsumer A \/ C take over\n<\/code><\/pre>\n\n\n\n
    Rebalancing is one of the mechanisms that gives consumer groups fault tolerance and elasticity.<\/p>\n\n\n\n
    \n\n\n\n
    20. Major Rebalance Triggers<\/h1>\n\n\n\n
    A rebalance or reassignment can happen when:<\/p>\n\n\n\n
    A consumer joins\nA consumer leaves gracefully\nA consumer crashes or becomes unreachable\nConsumer membership expires\nA subscribed topic gains partitions\nA subscribed topic changes\nA regex subscription matches a new topic\nSubscription metadata changes\nAdministrative\/manual group changes occur\n<\/code><\/pre>\n\n\n\n
    The exact protocol messages differ between the modern Consumer<\/strong> protocol and the older Classic<\/strong> protocol.<\/p>\n\n\n\n
    \n\n\n\n
    21. Consumer Joins a Group<\/h1>\n\n\n\n
    Imagine:<\/p>\n\n\n\n
    Group currently:\n\nConsumer A\nConsumer B\n<\/code><\/pre>\n\n\n\n
    A new instance starts:<\/p>\n\n\n\n
    Consumer C\n<\/code><\/pre>\n\n\n\n
    Kafka must decide:<\/p>\n\n\n\n
    Which partitions should Consumer C receive?\n\nWhich existing consumers should give up partitions?\n<\/code><\/pre>\n\n\n\n
    That causes a new assignment.<\/p>\n\n\n\n
    \n\n\n\n
    22. Consumer Leaves Gracefully<\/h1>\n\n\n\n
    A consumer can shut down cleanly.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Consumer B\n   |\n   v\nclose()\n   |\n   v\nLeaves group\n   |\n   v\nIts partitions become available\n   |\n   v\nAssignment updated\n<\/code><\/pre>\n\n\n\n
    Graceful shutdown is preferable to simply killing processes because the system can respond more cleanly.<\/p>\n\n\n\n
    \n\n\n\n
    23. Consumer Crashes<\/h1>\n\n\n\n
    A crash is different.<\/p>\n\n\n\n
    The consumer cannot politely say:<\/p>\n\n\n\n
    \"I am leaving.\"\n<\/code><\/pre>\n\n\n\n
    Kafka must detect that the member is no longer healthy.<\/p>\n\n\n\n
    After failure detection:<\/p>\n\n\n\n
    Consumer B declared unavailable\n      |\n      v\nIts partitions reassigned\n      |\n      v\nAnother consumer resumes processing\n<\/code><\/pre>\n\n\n\n
    This is consumer failover.<\/p>\n\n\n\n
    \n\n\n\n
    24. Topic Partition Count Changes<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    Topic: orders\nPartitions: 3\n<\/code><\/pre>\n\n\n\n
    becomes:<\/p>\n\n\n\n
    Partitions: 6\n<\/code><\/pre>\n\n\n\n
    The group now has new work.<\/p>\n\n\n\n
    Kafka needs to assign the new partitions to consumers.<\/p>\n\n\n\n
    Therefore adding partitions can cause new partition assignments.<\/p>\n\n\n\n
    This is why changing topic partitions is an operational event, not just a metadata edit.<\/p>\n\n\n\n
    \n\n\n\n
    25. Modern Consumer Protocol vs Classic Protocol<\/h1>\n\n\n\n
    This is extremely important for modern Kafka students.<\/p>\n\n\n\n
    Kafka now has two consumer-group protocol families you may encounter:<\/p>\n\n\n\n
    1. Consumer protocol\n2. Classic protocol\n<\/code><\/pre>\n\n\n\n
    The Consumer protocol<\/strong> is the newer generation of Kafka’s group rebalance protocol.<\/p>\n\n\n\n
    The Classic protocol<\/strong> is the older model that many existing tutorials and applications still describe.<\/p>\n\n\n\n
    You should understand both because real organizations may run both during migration.<\/p>\n\n\n\n
    \n\n\n\n
    26. Classic Consumer Group Protocol \u2014 Conceptual Model<\/h1>\n\n\n\n
    In the Classic protocol, the group has a client-side leader involved in partition assignment.<\/p>\n\n\n\n
    A simplified flow is:<\/p>\n\n\n\n
    Consumers\n   |\n   v\nJoin Group\n   |\n   v\nCoordinator\n   |\n   v\nGroup leader participates in assignment\n   |\n   v\nSync Group\n   |\n   v\nConsumers receive assignments\n<\/code><\/pre>\n\n\n\n
    Classic assignment strategies include client-side assignors such as:<\/p>\n\n\n\n
    Range\nRound Robin\nSticky\nCooperative Sticky\n<\/code><\/pre>\n\n\n\n
    Some Classic strategies use eager movement of partitions; cooperative strategies can reduce how much work must stop and move.<\/p>\n\n\n\n
    \n\n\n\n
    27. Modern Consumer Rebalance Protocol<\/h1>\n\n\n\n
    Modern Kafka’s Consumer protocol moves more group-assignment responsibility to the broker-side coordinator.<\/p>\n\n\n\n
    Key ideas:<\/p>\n\n\n\n
    Broker-side assignment\nIncremental assignment changes\nNo client group leader needed for assignment\nReduced global synchronization\nFaster \/ less disruptive rebalances\n<\/code><\/pre>\n\n\n\n
    The assignment strategy is managed differently from the older Classic client-side assignor model.<\/p>\n\n\n\n
    For students:<\/p>\n\n\n\n
    \n
    Do not memorize only the old \u201cone consumer becomes leader and assigns everything\u201d explanation as if it describes every modern Kafka consumer group.<\/p>\n<\/blockquote>\n\n\n\n
    \n\n\n\n
    28. Which Protocol Should Students Learn?<\/h1>\n\n\n\n
    Learn the architecture in this order:<\/p>\n\n\n\n
    First:\nConsumer Group concepts\nPartitions\nCoordinator\nOffsets\nRebalancing\n<\/code><\/pre>\n\n\n\n
    Then:<\/p>\n\n\n\n
    Modern Consumer protocol\n<\/code><\/pre>\n\n\n\n
    Then:<\/p>\n\n\n\n
    Classic protocol\nfor compatibility and existing systems\n<\/code><\/pre>\n\n\n\n
    The business concepts are the same:<\/p>\n\n\n\n
    Consumers share partitions\nMembership changes\nAssignments change\nOffsets allow recovery\n<\/code><\/pre>\n\n\n\n
    The protocol mechanics differ.<\/p>\n\n\n\n
    \n\n\n\n
    29. Rebalancing and Application Processing<\/h1>\n\n\n\n
    Rebalances matter because consumers may temporarily change ownership of partitions.<\/p>\n\n\n\n
    Imagine:<\/p>\n\n\n\n
    Consumer A owns P0\n<\/code><\/pre>\n\n\n\n
    Then:<\/p>\n\n\n\n
    P0 is moved to Consumer B\n<\/code><\/pre>\n\n\n\n
    The application must correctly handle:<\/p>\n\n\n\n
    Finish or stop work on P0\nCommit safe progress\nRelease partition-specific resources\nBegin processing new assignment\n<\/code><\/pre>\n\n\n\n
    Incorrect rebalance handling can cause:<\/p>\n\n\n\n
    Duplicate work\nIncorrect offset commits\nLost application progress\nProcessing delays\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    30. ConsumerRebalanceListener<\/h1>\n\n\n\n
    The Java consumer provides rebalance callbacks.<\/p>\n\n\n\n
    Important concepts:<\/p>\n\n\n\n
    onPartitionsRevoked(...)\nonPartitionsAssigned(...)\nonPartitionsLost(...)\n<\/code><\/pre>\n\n\n\n
    These callbacks let applications react when partition ownership changes.<\/p>\n\n\n\n
    Typical uses include:<\/p>\n\n\n\n
    Commit safe offsets\nFlush local state\nClose partition-specific resources\nInitialize state for new partitions\n<\/code><\/pre>\n\n\n\n
    onPartitionsLost<\/code> is especially important because in some failure cases the consumer may discover it has already lost ownership and should not assume it can safely perform the same actions as a normal revoke.<\/p>\n\n\n\n
    \n\n\n\n
    31. What Is a Kafka Offset?<\/h1>\n\n\n\n
    A Kafka offset<\/strong> is a numerical position inside a partition log.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Partition 0\n\nOffset 0 -> Record A\nOffset 1 -> Record B\nOffset 2 -> Record C\nOffset 3 -> Record D\nOffset 4 -> Record E\n<\/code><\/pre>\n\n\n\n
    Offsets belong to partitions.<\/p>\n\n\n\n
    Therefore:<\/p>\n\n\n\n
    P0 offset 100\n<\/code><\/pre>\n\n\n\n
    and:<\/p>\n\n\n\n
    P1 offset 100\n<\/code><\/pre>\n\n\n\n
    are different positions.<\/p>\n\n\n\n
    Offsets are not global message IDs across the entire topic.<\/p>\n\n\n\n
    \n\n\n\n
    32. Consumer Position \u2014 Current Position<\/h1>\n\n\n\n
    Kafka consumers have a current position<\/strong>.<\/p>\n\n\n\n
    A simple definition:<\/p>\n\n\n\n
    \n
    The current position is the offset of the next record the consumer will fetch\/return from that partition.<\/p>\n<\/blockquote>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Partition:\n\n0 1 2 3 4 5 6 7\n            ^\n            |\nCurrent position = 6\n<\/code><\/pre>\n\n\n\n
    This means the consumer has already advanced past earlier records and expects to continue from around offset 6.<\/p>\n\n\n\n
    The current position normally lives in the running consumer’s state and advances as poll()<\/code> returns records.<\/p>\n\n\n\n
    \n\n\n\n
    33. Committed Offset<\/h1>\n\n\n\n
    The committed offset<\/strong> is the saved recovery checkpoint for a consumer group.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Partition:\n\n0 1 2 3 4 5 6 7\n        ^   ^\n        |   |\nCommitted Current\n   = 4     = 6\n<\/code><\/pre>\n\n\n\n
    The consumer may currently have fetched through offset 5 but only safely committed progress up to the point represented by committed offset 4.<\/p>\n\n\n\n
    If the consumer crashes:<\/p>\n\n\n\n
    Current in-memory position is lost\n<\/code><\/pre>\n\n\n\n
    Kafka uses:<\/p>\n\n\n\n
    Committed offset\n<\/code><\/pre>\n\n\n\n
    to decide where the group should resume.<\/p>\n\n\n\n
    \n\n\n\n
    34. Critical Offset Rule: Commit the NEXT Record to Read<\/h1>\n\n\n\n
    This is one of the most important details in Kafka.<\/p>\n\n\n\n
    If you successfully process records:<\/p>\n\n\n\n
    Offset 10\nOffset 11\nOffset 12\n<\/code><\/pre>\n\n\n\n
    your commit should normally indicate:<\/p>\n\n\n\n
    13\n<\/code><\/pre>\n\n\n\n
    because committed offset means:<\/p>\n\n\n\n
    \n
    \u201cThe next record I intend to read\/process is offset 13.\u201d<\/p>\n<\/blockquote>\n\n\n\n
    Do not think of the committed value merely as:<\/p>\n\n\n\n
    \"The last record I processed.\"\n<\/code><\/pre>\n\n\n\n
    Think:<\/p>\n\n\n\n
    \"The next position I should resume from.\"\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    35. Where Are Committed Offsets Stored?<\/h1>\n\n\n\n
    Kafka stores consumer-group commits in the internal compacted topic:<\/p>\n\n\n\n
    __consumer_offsets\n<\/code><\/pre>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Consumer\n   |\n   v\nOffset Commit\n   |\n   v\nGroup Coordinator\n   |\n   v\n__consumer_offsets\n<\/code><\/pre>\n\n\n\n
    This allows offsets to survive a consumer process restart.<\/p>\n\n\n\n
    \n\n\n\n
    36. Why __consumer_offsets<\/code> Is Compacted<\/h1>\n\n\n\n
    Kafka usually only needs the latest committed position for:<\/p>\n\n\n\n
    group\n+\ntopic\n+\npartition\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    order-processing \/ orders \/ P0 -> 1050\n<\/code><\/pre>\n\n\n\n
    Later:<\/p>\n\n\n\n
    order-processing \/ orders \/ P0 -> 1100\n<\/code><\/pre>\n\n\n\n
    The newest commit becomes the important recovery state.<\/p>\n\n\n\n
    A compacted internal topic is a good fit for this kind of state.<\/p>\n\n\n\n
    \n\n\n\n
    37. Four Positions Worth Understanding<\/h1>\n\n\n\n
    Students should eventually understand these concepts:<\/p>\n\n\n\n
    1. Current consumer position\n2. Committed group offset\n3. High watermark \/ readable end\n4. Log end \/ latest written position\n<\/code><\/pre>\n\n\n\n
    Simplified:<\/p>\n\n\n\n
    Committed       Current          Readable End\n    |              |                 |\n    v              v                 v\n\n0 1 2 3 4 5 6 7 8 9 10 11 12 ...\n<\/code><\/pre>\n\n\n\n
    In transactional consumption there is another important concept:<\/p>\n\n\n\n
    Last Stable Offset (LSO)\n<\/code><\/pre>\n\n\n\n
    which matters for read_committed<\/code>.<\/p>\n\n\n\n
    We will cover transactions separately.<\/p>\n\n\n\n
    \n\n\n\n
    38. What Is Consumer Lag?<\/h1>\n\n\n\n
    Consumer lag answers:<\/p>\n\n\n\n
    \n
    \u201cHow far behind is the consumer group?\u201d<\/p>\n<\/blockquote>\n\n\n\n
    Simplified:<\/p>\n\n\n\n
    End of available data\n-\ngroup's consumed\/committed progress\n=\nlag\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Latest\/end position = 10,000\nCommitted position  = 9,500\n\nLag \u2248 500 records\n<\/code><\/pre>\n\n\n\n
    High lag means work is accumulating faster than the consumer group is completing it.<\/p>\n\n\n\n
    \n\n\n\n
    39. Lag Is a Symptom, Not the Root Cause<\/h1>\n\n\n\n
    If lag grows, possible causes include:<\/p>\n\n\n\n
    Consumer processing too slowly\nToo few consumers\nToo few partitions\nHot partition\nSlow database\nSlow external API\nLarge records\nDeserialization cost\nLong GC pauses\nNetwork latency\nBroker throttling\nBad fetch configuration\nFrequent rebalances\nConsumer errors\n<\/code><\/pre>\n\n\n\n
    Do not respond to lag by blindly adding consumers.<\/p>\n\n\n\n
    If:<\/p>\n\n\n\n
    Topic has 4 partitions\nGroup already has 4 active consumers\n<\/code><\/pre>\n\n\n\n
    adding ten more consumers will not create more partition-level parallelism.<\/p>\n\n\n\n
    \n\n\n\n
    40. Offset Commit Strategies<\/h1>\n\n\n\n
    There are two major approaches:<\/p>\n\n\n\n
    Automatic Commit\nManual Commit\n<\/code><\/pre>\n\n\n\n
    Neither is \u201calways correct.\u201d<\/p>\n\n\n\n
    The right strategy depends on:<\/p>\n\n\n\n
    Business correctness\nFailure semantics\nProcessing time\nIdempotency\nThroughput\nOperational complexity\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    41. Automatic Offset Commit<\/h1>\n\n\n\n
    With:<\/p>\n\n\n\n
    enable.auto.commit=true\n<\/code><\/pre>\n\n\n\n
    the consumer periodically commits offsets in the background.<\/p>\n\n\n\n
    The interval is controlled by:<\/p>\n\n\n\n
    auto.commit.interval.ms\n<\/code><\/pre>\n\n\n\n
    This is easy to use.<\/p>\n\n\n\n
    But it is important to understand what you are promising.<\/p>\n\n\n\n
    The application must ensure that records returned from polling are actually processed safely before later polling\/commit behavior can move the recovery point past work that is not finished.<\/p>\n\n\n\n
    Automatic commit is convenient, but it gives the application less explicit control over the exact business-processing boundary.<\/p>\n\n\n\n
    \n\n\n\n
    42. Automatic Commit Does NOT Mean \u201cCommit Every Record\u201d<\/h1>\n\n\n\n
    A common misunderstanding:<\/p>\n\n\n\n
    enable.auto.commit=true\n<\/code><\/pre>\n\n\n\n
    does not mean:<\/p>\n\n\n\n
    Kafka commits every record immediately.\n<\/code><\/pre>\n\n\n\n
    The commits happen periodically.<\/p>\n\n\n\n
    Therefore the gap between:<\/p>\n\n\n\n
    current processing position\n<\/code><\/pre>\n\n\n\n
    and:<\/p>\n\n\n\n
    committed recovery position\n<\/code><\/pre>\n\n\n\n
    can exist.<\/p>\n\n\n\n
    If a failure happens, some records may be read again.<\/p>\n\n\n\n
    This is one reason duplicate-tolerant\/idempotent processing is valuable.<\/p>\n\n\n\n
    \n\n\n\n
    43. Manual Offset Commit<\/h1>\n\n\n\n
    With:<\/p>\n\n\n\n
    enable.auto.commit=false\n<\/code><\/pre>\n\n\n\n
    the application controls commits.<\/p>\n\n\n\n
    Common APIs include:<\/p>\n\n\n\n
    commitSync()\ncommitAsync()\n<\/code><\/pre>\n\n\n\n
    The core production rule is:<\/p>\n\n\n\n
    \n
    Commit progress only when the corresponding business work is safely complete according to your required delivery semantics.<\/p>\n<\/blockquote>\n\n\n\n
    \n\n\n\n
    44. commitSync()<\/code><\/h1>\n\n\n\n
    commitSync()<\/code> waits for the commit result.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Process records\n      |\n      v\ncommitSync()\n      |\n      v\nWait for broker response\n      |\n      v\nContinue\n<\/code><\/pre>\n\n\n\n
    Advantages:<\/p>\n\n\n\n
    Simple failure handling\nKnown commit completion point\nGood at controlled boundaries\n<\/code><\/pre>\n\n\n\n
    Tradeoff:<\/p>\n\n\n\n
    Blocking can reduce throughput if used too frequently\n<\/code><\/pre>\n\n\n\n
    Do not call synchronous commit after every individual record unless the workload genuinely requires that behavior and the performance cost is acceptable.<\/p>\n\n\n\n
    \n\n\n\n
    45. commitAsync()<\/code><\/h1>\n\n\n\n
    commitAsync()<\/code> sends the commit without blocking the main processing flow.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Process\n   |\n   v\ncommitAsync()\n   |\n   +----> Commit request continues\n   |\n   v\nKeep processing\n<\/code><\/pre>\n\n\n\n
    Advantages:<\/p>\n\n\n\n
    Lower blocking overhead\nBetter throughput\n<\/code><\/pre>\n\n\n\n
    But you must handle:<\/p>\n\n\n\n
    Commit callback errors\nOrdering of commit attempts\nShutdown\/rebalance boundaries\n<\/code><\/pre>\n\n\n\n
    A common production pattern is:<\/p>\n\n\n\n
    Periodic async commits during normal processing\n\nPLUS\n\na final controlled sync commit at important shutdown\/rebalance boundaries\n<\/code><\/pre>\n\n\n\n
    when that model matches the application’s semantics.<\/p>\n\n\n\n
    \n\n\n\n
    46. Why Commit Timing Matters<\/h1>\n\n\n\n
    Consider two sequences.<\/p>\n\n\n\n
    Strategy A<\/h2>\n\n\n\n
    Commit offset\n     |\n     v\nProcess database write\n<\/code><\/pre>\n\n\n\n
    Failure scenario:<\/p>\n\n\n\n
    Commit succeeds\nApplication crashes\nDatabase write never happens\n<\/code><\/pre>\n\n\n\n
    Kafka may resume after<\/strong> the record.<\/p>\n\n\n\n
    From the business point of view:<\/p>\n\n\n\n
    record may be lost\n<\/code><\/pre>\n\n\n\n
    This resembles at-most-once<\/strong> processing.<\/p>\n\n\n\n
    \n\n\n\n
    47. Process Then Commit<\/h1>\n\n\n\n
    Now reverse it:<\/p>\n\n\n\n
    Process database write\n     |\n     v\nCommit offset\n<\/code><\/pre>\n\n\n\n
    Failure scenario:<\/p>\n\n\n\n
    Database write succeeds\nApplication crashes before commit\n<\/code><\/pre>\n\n\n\n
    After restart:<\/p>\n\n\n\n
    Kafka resumes from old committed offset\n<\/code><\/pre>\n\n\n\n
    The record is processed again.<\/p>\n\n\n\n
    This creates possible duplicates.<\/p>\n\n\n\n
    This resembles at-least-once<\/strong> processing.<\/p>\n\n\n\n
    \n\n\n\n
    48. At-Most-Once<\/h1>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Commit\n   |\n   v\nProcess\n<\/code><\/pre>\n\n\n\n
    Potential outcome:<\/p>\n\n\n\n
    No duplicate processing\nbut possible loss if failure happens after commit and before processing.\n<\/code><\/pre>\n\n\n\n
    Use cases may include data where losing an occasional record is preferable to duplicate work.<\/p>\n\n\n\n
    For important business events, this is often not desirable.<\/p>\n\n\n\n
    \n\n\n\n
    49. At-Least-Once<\/h1>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Process\n   |\n   v\nCommit\n<\/code><\/pre>\n\n\n\n
    Potential outcome:<\/p>\n\n\n\n
    No intentional loss of successfully fetched work\nbut duplicates are possible after failures.\n<\/code><\/pre>\n\n\n\n
    Therefore the downstream processing should preferably be:<\/p>\n\n\n\n
    Idempotent\n<\/code><\/pre>\n\n\n\n
    Meaning:<\/p>\n\n\n\n
    Processing the same event twice\ndoes not create an incorrect business result.\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    50. Exactly-Once \u2014 Be Precise<\/h1>\n\n\n\n
    Do not teach students:<\/p>\n\n\n\n
    \n
    \u201cManual commits give exactly once.\u201d<\/p>\n<\/blockquote>\n\n\n\n
    They do not.<\/p>\n\n\n\n
    Exactly-once requires a coordinated processing design.<\/p>\n\n\n\n
    Kafka supports transactional approaches for Kafka-to-Kafka workflows, together with transactional producers and consumers configured to read committed transactional data.<\/p>\n\n\n\n
    If your consumer writes to:<\/p>\n\n\n\n
    External SQL database\nExternal REST API\nEmail provider\nPayment provider\n<\/code><\/pre>\n\n\n\n
    Kafka cannot magically create one atomic transaction across all of those systems.<\/p>\n\n\n\n
    You may need:<\/p>\n\n\n\n
    Idempotency keys\nDatabase transactions\nOutbox \/ inbox patterns\nDeduplication\nKafka transactions where applicable\nApplication-specific coordination\n<\/code><\/pre>\n\n\n\n
    Exactly-once is an architecture topic, not one checkbox.<\/p>\n\n\n\n
    \n\n\n\n
    51. auto.offset.reset<\/code><\/h1>\n\n\n\n
    What happens when there is no usable committed offset<\/strong>?<\/p>\n\n\n\n
    Kafka uses:<\/p>\n\n\n\n
    auto.offset.reset\n<\/code><\/pre>\n\n\n\n
    Common strategies include:<\/p>\n\n\n\n
    earliest\nlatest\nnone\n<\/code><\/pre>\n\n\n\n
    Modern Kafka also supports a duration-based reset option.<\/p>\n\n\n\n
    Important:<\/p>\n\n\n\n
    \n
    auto.offset.reset<\/code> is not the normal \u201cwhere should my healthy consumer read next?\u201d setting.<\/p>\n<\/blockquote>\n\n\n\n
    It is primarily used when Kafka cannot find a valid committed position for the group\/partition.<\/p>\n\n\n\n
    \n\n\n\n
    52. earliest<\/code><\/h1>\n\n\n\n
    auto.offset.reset=earliest\n<\/code><\/pre>\n\n\n\n
    means:<\/p>\n\n\n\n
    \n
    Start from the earliest retained offset available.<\/p>\n<\/blockquote>\n\n\n\n
    Useful for:<\/p>\n\n\n\n
    New analytics consumers\nBackfill processing\nReplay use cases\nTraining\nRebuilding state\n<\/code><\/pre>\n\n\n\n
    Remember:<\/p>\n\n\n\n
    Kafka may already have deleted older records according to retention.<\/p>\n\n\n\n
    \u201cEarliest\u201d means earliest still retained<\/strong>, not necessarily the first record ever produced.<\/p>\n\n\n\n
    \n\n\n\n
    53. latest<\/code><\/h1>\n\n\n\n
    auto.offset.reset=latest\n<\/code><\/pre>\n\n\n\n
    means:<\/p>\n\n\n\n
    \n
    Start from the latest position and consume new records from that point forward.<\/p>\n<\/blockquote>\n\n\n\n
    Useful for:<\/p>\n\n\n\n
    Real-time consumers\nwhere old history is not needed\n<\/code><\/pre>\n\n\n\n
    Danger:<\/p>\n\n\n\n
    A new group configured with latest<\/code> will normally not process the existing retained history before its starting point.<\/p>\n\n\n\n
    Know whether that is acceptable.<\/p>\n\n\n\n
    \n\n\n\n
    54. none<\/code><\/h1>\n\n\n\n
    auto.offset.reset=none\n<\/code><\/pre>\n\n\n\n
    means:<\/p>\n\n\n\n
    \n
    If Kafka has no valid committed offset, fail rather than automatically choosing earliest\/latest.<\/p>\n<\/blockquote>\n\n\n\n
    This can be useful when silently choosing a new position would be dangerous.<\/p>\n\n\n\n
    For critical applications, failing visibly can be safer than unexpectedly skipping or replaying data.<\/p>\n\n\n\n
    \n\n\n\n
    55. Duration-Based Reset<\/h1>\n\n\n\n
    Modern Kafka can also reset based on a duration from the current time.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Start from approximately N hours\/days ago\n<\/code><\/pre>\n\n\n\n
    This is useful for some recovery or time-window processing cases.<\/p>\n\n\n\n
    Treat it as an advanced option after students master:<\/p>\n\n\n\n
    earliest\nlatest\nnone\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    56. Right Offset Strategy \u2014 Decision Guide<\/h1>\n\n\n\n
    Ask:<\/p>\n\n\n\n
    Question 1<\/h3>\n\n\n\n
    Can duplicate processing happen safely?<\/p>\n\n\n\n
    If yes:<\/p>\n\n\n\n
    At-least-once\n+\ncommit after processing\n+\nidempotent downstream\n<\/code><\/pre>\n\n\n\n
    is often a strong practical design.<\/p>\n\n\n\n
    Question 2<\/h3>\n\n\n\n
    Can any record be lost?<\/p>\n\n\n\n
    If no:<\/p>\n\n\n\n
    Avoid committing before successful processing.<\/p>\n\n\n\n
    Question 3<\/h3>\n\n\n\n
    What should a new group do?<\/p>\n\n\n\n
    Choose intentionally:<\/p>\n\n\n\n
    earliest\nlatest\nnone\nduration-based\n<\/code><\/pre>\n\n\n\n
    Question 4<\/h3>\n\n\n\n
    Can the business operation and offset be committed atomically?<\/p>\n\n\n\n
    If Kafka-to-Kafka:<\/p>\n\n\n\n
    Kafka transactions may help.\n<\/code><\/pre>\n\n\n\n
    If external side effects:<\/p>\n\n\n\n
    Use application\/database patterns appropriate to that system.\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    57. Consumer Failover and Recovery<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    P0 -> Consumer A\nP1 -> Consumer B\nP2 -> Consumer C\n<\/code><\/pre>\n\n\n\n
    Consumer B crashes.<\/p>\n\n\n\n
    Before:<\/p>\n\n\n\n
    A -> P0\nB -> P1\nC -> P2\n<\/code><\/pre>\n\n\n\n
    After detection and reassignment:<\/p>\n\n\n\n
    A -> P0 P1\nC -> P2\n<\/code><\/pre>\n\n\n\n
    or another valid assignment.<\/p>\n\n\n\n
    The new owner starts from the group’s committed position for P1.<\/p>\n\n\n\n
    That is why committed offsets are fundamental to recovery.<\/p>\n\n\n\n
    \n\n\n\n
    58. Failure Window and Duplicate Processing<\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    Committed offset = 100\nCurrent position = 110\n<\/code><\/pre>\n\n\n\n
    The consumer has processed records near 100-109 but has not committed the newer progress.<\/p>\n\n\n\n
    Then it crashes.<\/p>\n\n\n\n
    Replacement consumer resumes from:<\/p>\n\n\n\n
    100\n<\/code><\/pre>\n\n\n\n
    Records between the committed point and the crash-time processing position may be processed again.<\/p>\n\n\n\n
    This is expected for at-least-once designs.<\/p>\n\n\n\n
    Do not call this a Kafka bug.<\/p>\n\n\n\n
    It is the consequence of the chosen commit boundary.<\/p>\n\n\n\n
    \n\n\n\n
    59. Consumer Liveness<\/h1>\n\n\n\n
    Kafka needs to distinguish:<\/p>\n\n\n\n
    Healthy consumer\n<\/code><\/pre>\n\n\n\n
    from:<\/p>\n\n\n\n
    Dead or stuck consumer\n<\/code><\/pre>\n\n\n\n
    There are two related concerns:<\/p>\n\n\n\n
    1. Membership\/liveness heartbeat\n2. Application processing progress \/ poll interval\n<\/code><\/pre>\n\n\n\n
    These concepts are important but the exact configuration differs between the modern Consumer protocol and the older Classic protocol.<\/p>\n\n\n\n
    \n\n\n\n
    60. Heartbeats \u2014 Concept<\/h1>\n\n\n\n
    Consumers maintain group membership by demonstrating that they are alive.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    Consumer\n   |\n   v\nCoordinator\n\n\"I'm alive.\"\n<\/code><\/pre>\n\n\n\n
    If Kafka determines the consumer is no longer alive:<\/p>\n\n\n\n
    Remove member\n      |\n      v\nReassign partitions\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    61. Classic vs Consumer Protocol Timeout Configuration<\/h1>\n\n\n\n
    Do not blindly copy old timeout advice.<\/p>\n\n\n\n
    Classic protocol<\/h2>\n\n\n\n
    Client-side settings historically include:<\/p>\n\n\n\n
    heartbeat.interval.ms\nsession.timeout.ms\n<\/code><\/pre>\n\n\n\n
    Modern Consumer protocol<\/h2>\n\n\n\n
    Heartbeat\/session behavior is controlled more by broker-side group configuration.<\/p>\n\n\n\n
    This is one of the important operational differences between the protocols.<\/p>\n\n\n\n
    Always check which group protocol your client and cluster are using before tuning these settings.<\/p>\n\n\n\n
    \n\n\n\n
    62. max.poll.interval.ms<\/code><\/h1>\n\n\n\n
    Another failure mode is:<\/p>\n\n\n\n
    Consumer is alive\nbut application takes too long between poll() calls.\n<\/code><\/pre>\n\n\n\n
    Kafka uses:<\/p>\n\n\n\n
    max.poll.interval.ms\n<\/code><\/pre>\n\n\n\n
    to place an upper bound on how long the consumer can go without making expected poll progress under group management.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    poll()\n   |\n   v\nReceive 500 records\n   |\n   v\nProcessing takes 10 minutes\n   |\n   v\nmax.poll.interval.ms = 5 minutes\n<\/code><\/pre>\n\n\n\n
    The group may treat the consumer as unable to keep up and reassign its partitions.<\/p>\n\n\n\n
    This can create a rebalance loop.<\/p>\n\n\n\n
    \n\n\n\n
    63. Long Processing Is a Major Consumer Design Problem<\/h1>\n\n\n\n
    If each record requires:<\/p>\n\n\n\n
    10 seconds\n<\/code><\/pre>\n\n\n\n
    and:<\/p>\n\n\n\n
    max.poll.records=500\n<\/code><\/pre>\n\n\n\n
    one poll could theoretically represent a huge amount of processing time.<\/p>\n\n\n\n
    Possible solutions:<\/p>\n\n\n\n
    Reduce max.poll.records\nOptimize business processing\nBatch downstream operations\nIncrease safe poll interval where justified\nPause\/resume partitions\nUse carefully designed worker-thread architecture\nIncrease partitions and consumers if appropriate\n<\/code><\/pre>\n\n\n\n
    Do not simply increase every timeout to enormous values.<\/p>\n\n\n\n
    That can make real failures take too long to recover.<\/p>\n\n\n\n
    \n\n\n\n
    64. Consumer Is Not Automatically Your Worker Thread Pool<\/h1>\n\n\n\n
    A very important Java design point:<\/p>\n\n\n\n
    A KafkaConsumer<\/code> instance is not intended to be freely used from many application threads concurrently.<\/p>\n\n\n\n
    A common safe model is:<\/p>\n\n\n\n
    One consumer thread\n      |\n      v\npoll()\n      |\n      v\nprocess assigned records\n<\/code><\/pre>\n\n\n\n
    If you hand records to worker threads:<\/p>\n\n\n\n
    Consumer Thread\n      |\n      v\nQueue\n   \/  |  \\\n  v   v   v\nW1  W2  W3\n<\/code><\/pre>\n\n\n\n
    you must solve:<\/p>\n\n\n\n
    Ordering\nBackpressure\nOffset coordination\nFailure handling\nRebalances\nShutdown\n<\/code><\/pre>\n\n\n\n
    This is an advanced architecture, not \u201cfree extra parallelism.\u201d<\/p>\n\n\n\n
    \n\n\n\n
    65. Fan-Out Consumption Pattern<\/h1>\n\n\n\n
    Fan-out means:<\/p>\n\n\n\n
    \n
    Multiple independent applications\/groups consume the same topic.<\/p>\n<\/blockquote>\n\n\n\n
    Example:<\/p>\n\n\n\n
                       -> Group: analytics\n                  \/\nTopic: orders -----> Group: fraud\n                  \\\n                   -> Group: notifications\n<\/code><\/pre>\n\n\n\n
    Each group gets its own independent view of the topic.<\/p>\n\n\n\n
    Inside each group, partitions are shared among that group’s consumers.<\/p>\n\n\n\n
    This lets the same event power many independent systems.<\/p>\n\n\n\n
    \n\n\n\n
    66. Fan-Out Example<\/h1>\n\n\n\n
    Topic:<\/p>\n\n\n\n
    vehicle-telemetry\n<\/code><\/pre>\n\n\n\n
    Groups:<\/p>\n\n\n\n
    group.id = realtime-dashboard\ngroup.id = anomaly-detection\ngroup.id = data-warehouse\ngroup.id = billing\n<\/code><\/pre>\n\n\n\n
    All four groups independently consume vehicle telemetry.<\/p>\n\n\n\n
    Producer does not need to send the event four times.<\/p>\n\n\n\n
    Kafka retains the stream, and each group tracks its own offsets.<\/p>\n\n\n\n
    \n\n\n\n
    67. Load-Balanced Consumption Pattern<\/h1>\n\n\n\n
    Load balancing means:<\/p>\n\n\n\n
    \n
    Multiple consumers use the same group.id<\/code> and divide partitions.<\/p>\n<\/blockquote>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Topic: orders\n\nConsumer Group: order-workers\n\nConsumer 1\nConsumer 2\nConsumer 3\nConsumer 4\n<\/code><\/pre>\n\n\n\n
    Partitions are distributed across those consumers.<\/p>\n\n\n\n
    Use when:<\/p>\n\n\n\n
    One logical application\nneeds more processing capacity.\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    68. Fan-Out vs Load Balance<\/h1>\n\n\n\n
    Fan-Out<\/h2>\n\n\n\n
    Same topic\nDifferent group IDs\n<\/code><\/pre>\n\n\n\n
    Result:<\/p>\n\n\n\n
    Each group independently gets the data.\n<\/code><\/pre>\n\n\n\n
    Use for:<\/p>\n\n\n\n
    Analytics\nAlerts\nBilling\nFraud\nData warehouse\n<\/code><\/pre>\n\n\n\n
    Load Balance<\/h2>\n\n\n\n
    Same topic\nSame group ID\n<\/code><\/pre>\n\n\n\n
    Result:<\/p>\n\n\n\n
    Consumers share the partitions\/work.\n<\/code><\/pre>\n\n\n\n
    Use for:<\/p>\n\n\n\n
    Scaling one application horizontally.\n<\/code><\/pre>\n\n\n\n
    This distinction is foundational.<\/p>\n\n\n\n
    \n\n\n\n
    69. Combining Fan-Out and Load Balancing<\/h1>\n\n\n\n
    Real production systems use both.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Topic: orders\n<\/code><\/pre>\n\n\n\n
    Analytics group:<\/p>\n\n\n\n
    analytics\n  Consumer A\n  Consumer B\n<\/code><\/pre>\n\n\n\n
    Fraud group:<\/p>\n\n\n\n
    fraud\n  Consumer A\n  Consumer B\n  Consumer C\n  Consumer D\n<\/code><\/pre>\n\n\n\n
    Notification group:<\/p>\n\n\n\n
    notifications\n  Consumer A\n<\/code><\/pre>\n\n\n\n
    Kafka allows each application to scale independently.<\/p>\n\n\n\n
    \n\n\n\n
    70. Manual assign()<\/code> vs Group subscribe()<\/code><\/h1>\n\n\n\n
    Most consumer applications use:<\/p>\n\n\n\n
    subscribe(...)\n<\/code><\/pre>\n\n\n\n
    which participates in consumer-group partition management.<\/p>\n\n\n\n
    Kafka can also let an application explicitly assign partitions:<\/p>\n\n\n\n
    assign(...)\n<\/code><\/pre>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Consumer manually owns:\n\nP0\nP4\n<\/code><\/pre>\n\n\n\n
    With manual assignment:<\/p>\n\n\n\n
    Kafka group rebalancing does not manage those partition assignments for you.\n<\/code><\/pre>\n\n\n\n
    Use manual assignment only when you intentionally want application-controlled partition ownership.<\/p>\n\n\n\n
    Do not confuse it with normal Consumer Group load balancing.<\/p>\n\n\n\n
    \n\n\n\n
    71. Consumer Fetching<\/h1>\n\n\n\n
    Consumers fetch records in batches.<\/p>\n\n\n\n
    Important settings include:<\/p>\n\n\n\n
    fetch.min.bytes\nfetch.max.wait.ms\nfetch.max.bytes\nmax.partition.fetch.bytes\nmax.poll.records\n<\/code><\/pre>\n\n\n\n
    These influence:<\/p>\n\n\n\n
    Network efficiency\nThroughput\nLatency\nMemory\nAmount of work returned per poll\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    72. fetch.min.bytes<\/code><\/h1>\n\n\n\n
    This asks the broker to try to return at least a certain amount of data before responding, subject to wait limits.<\/p>\n\n\n\n
    Higher values may:<\/p>\n\n\n\n
    Increase batch efficiency\nReduce request overhead\nImprove throughput\n<\/code><\/pre>\n\n\n\n
    but may also:<\/p>\n\n\n\n
    Increase latency when traffic is low\n<\/code><\/pre>\n\n\n\n
    Another throughput\/latency tradeoff.<\/p>\n\n\n\n
    \n\n\n\n
    73. fetch.max.wait.ms<\/code><\/h1>\n\n\n\n
    This limits how long the broker may wait while trying to satisfy the fetch-size conditions.<\/p>\n\n\n\n
    Think:<\/p>\n\n\n\n
    fetch.min.bytes\n=\n\"Try to give me this much data.\"\n\nfetch.max.wait.ms\n=\n\"But don't wait longer than this.\"\n<\/code><\/pre>\n\n\n\n
    These settings work together.<\/p>\n\n\n\n
    \n\n\n\n
    74. fetch.max.bytes<\/code><\/h1>\n\n\n\n
    Controls approximately how much data a fetch response may contain across partitions.<\/p>\n\n\n\n
    This matters for:<\/p>\n\n\n\n
    Network usage\nConsumer memory\nLarge workloads\nLarge records\n<\/code><\/pre>\n\n\n\n
    Do not set fetch sizes blindly without understanding record sizes and available memory.<\/p>\n\n\n\n
    \n\n\n\n
    75. max.partition.fetch.bytes<\/code><\/h1>\n\n\n\n
    This controls the amount of data fetched per partition in a request.<\/p>\n\n\n\n
    If messages\/batches are large, this setting becomes important.<\/p>\n\n\n\n
    Always coordinate large-message configuration end-to-end:<\/p>\n\n\n\n
    Producer\nTopic\/Broker\nConsumer\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    76. max.poll.records<\/code><\/h1>\n\n\n\n
    This controls how many records poll()<\/code> returns to the application at one time.<\/p>\n\n\n\n
    Important:<\/p>\n\n\n\n
    It does not necessarily change how much data Kafka fetches internally from brokers.<\/p>\n\n\n\n
    It limits how many fetched records are returned to the application per poll.<\/p>\n\n\n\n
    This is extremely useful for controlling:<\/p>\n\n\n\n
    Processing batch size\nPer-poll work\nmax.poll.interval risk\nApplication memory\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    77. Example: max.poll.records<\/code><\/h1>\n\n\n\n
    Suppose:<\/p>\n\n\n\n
    Each record takes 100 ms to process\nmax.poll.records = 500\n<\/code><\/pre>\n\n\n\n
    Worst-case serial work:<\/p>\n\n\n\n
    500 \u00d7 100 ms\n=\n50 seconds\n<\/code><\/pre>\n\n\n\n
    If each record instead takes:<\/p>\n\n\n\n
    2 seconds\n<\/code><\/pre>\n\n\n\n
    then:<\/p>\n\n\n\n
    500 \u00d7 2 sec\n=\n1000 sec\n=\n16+ minutes\n<\/code><\/pre>\n\n\n\n
    Now poll timing becomes dangerous.<\/p>\n\n\n\n
    Tuning must reflect business processing time<\/strong>, not only Kafka throughput.<\/p>\n\n\n\n
    \n\n\n\n
    78. pause()<\/code> and resume()<\/code><\/h1>\n\n\n\n
    Consumers can temporarily pause assigned partitions without giving up ownership.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    P0 P1 P2 assigned\n\nP1 downstream overloaded\n        |\n        v\npause(P1)\n        |\n        v\nContinue P0\/P2\n<\/code><\/pre>\n\n\n\n
    Later:<\/p>\n\n\n\n
    resume(P1)\n<\/code><\/pre>\n\n\n\n
    This can help implement controlled backpressure.<\/p>\n\n\n\n
    But offsets, processing state and rebalance handling must still be correct.<\/p>\n\n\n\n
    \n\n\n\n
    79. Static Membership \u2014 Concept<\/h1>\n\n\n\n
    In some deployments, consumer instances are stable and restart temporarily.<\/p>\n\n\n\n
    Kafka supports static membership concepts through:<\/p>\n\n\n\n
    group.instance.id\n<\/code><\/pre>\n\n\n\n
    This can reduce unnecessary membership churn in certain restart scenarios.<\/p>\n\n\n\n
    But it is not a replacement for:<\/p>\n\n\n\n
    Good failure detection\nCorrect rebalancing\nOffset correctness\n<\/code><\/pre>\n\n\n\n
    Use it when your deployment model benefits from stable member identities.<\/p>\n\n\n\n
    \n\n\n\n
    80. Rebalance Cost<\/h1>\n\n\n\n
    A rebalance can cost:<\/p>\n\n\n\n
    Temporary reduction in consumption\nPartition movement\nCache warm-up\nState reconstruction\nDatabase\/client initialization\nDuplicate processing near commit boundaries\n<\/code><\/pre>\n\n\n\n
    Therefore:<\/p>\n\n\n\n
    Constant rebalancing\n=\nBad operational health\n<\/code><\/pre>\n\n\n\n
    Monitor rebalance frequency.<\/p>\n\n\n\n
    \n\n\n\n
    81. Rebalance Storm Example<\/h1>\n\n\n\n
    Imagine:<\/p>\n\n\n\n
    Consumer starts\n    |\n    v\nProcessing too slow\n    |\n    v\nmax.poll.interval exceeded\n    |\n    v\nConsumer removed\n    |\n    v\nRebalance\n    |\n    v\nConsumer rejoins\n    |\n    v\nRebalance again\n<\/code><\/pre>\n\n\n\n
    This can repeat.<\/p>\n\n\n\n
    Symptoms:<\/p>\n\n\n\n
    High lag\nLow throughput\nFrequent assignment changes\nLots of logs\nConsumers appear healthy but never catch up\n<\/code><\/pre>\n\n\n\n
    Fix the root cause.<\/p>\n\n\n\n
    \n\n\n\n
    82. Offset Commit During Rebalance<\/h1>\n\n\n\n
    When a consumer is about to give up a partition, the application may need to commit the safest completed progress for that partition.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    P0 currently owned by Consumer A\n      |\n      v\nP0 will move\n      |\n      v\nCommit safe completed position\n      |\n      v\nRelease resources\n      |\n      v\nConsumer B receives P0\n<\/code><\/pre>\n\n\n\n
    Never commit work that has not actually completed.<\/p>\n\n\n\n
    \n\n\n\n
    83. onPartitionsLost<\/code> vs Normal Revoke<\/h1>\n\n\n\n
    Normal revoke means the consumer is being told:<\/p>\n\n\n\n
    \"You are about to give up these partitions.\"\n<\/code><\/pre>\n\n\n\n
    A lost-partition callback means:<\/p>\n\n\n\n
    \"You may already have lost ownership.\"\n<\/code><\/pre>\n\n\n\n
    The second situation requires more caution.<\/p>\n\n\n\n
    Do not assume it is safe to commit arbitrary state after ownership has already been lost.<\/p>\n\n\n\n
    \n\n\n\n
    84. Offset Commit Best Practices<\/h1>\n\n\n\n
    Best Practice 1<\/h2>\n\n\n\n
    Commit only safe completed progress<\/strong>.<\/p>\n\n\n\n
    Process\n   |\n   v\nBusiness operation succeeds\n   |\n   v\nCommit\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    Best Practice 2<\/h2>\n\n\n\n
    Commit the next offset to read<\/strong>.<\/p>\n\n\n\n
    If records through offset 120 are complete:<\/p>\n\n\n\n
    commit 121\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    Best Practice 3<\/h2>\n\n\n\n
    Do not commit every record unless necessary.<\/p>\n\n\n\n
    Prefer reasonable batches:<\/p>\n\n\n\n
    Process N records\n      |\n      v\nCommit safe position\n<\/code><\/pre>\n\n\n\n
    Balance:<\/p>\n\n\n\n
    Duplicate window\nvs\ncommit overhead\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    Best Practice 4<\/h2>\n\n\n\n
    Make downstream processing idempotent where possible.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Event ID = ORDER-123:PAYMENT-CAPTURED\n<\/code><\/pre>\n\n\n\n
    Database stores that event ID.<\/p>\n\n\n\n
    Second processing attempt:<\/p>\n\n\n\n
    Already processed\n-> do not create duplicate payment\n<\/code><\/pre>\n\n\n\n
    This turns expected at-least-once replay into safe recovery.<\/p>\n\n\n\n
    \n\n\n\n
    Best Practice 5<\/h2>\n\n\n\n
    Handle commit errors.<\/p>\n\n\n\n
    Do not write:<\/p>\n\n\n\n
    consumer.commitAsync();\n<\/code><\/pre>\n\n\n\n
    and assume:<\/p>\n\n\n\n
    \"Everything is definitely saved.\"\n<\/code><\/pre>\n\n\n\n
    Observe failures where required.<\/p>\n\n\n\n
    \n\n\n\n
    Best Practice 6<\/h2>\n\n\n\n
    Handle rebalance boundaries.<\/p>\n\n\n\n
    Before giving up a partition:<\/p>\n\n\n\n
    Finish safe work\nCommit safe progress\nRelease resources\n<\/code><\/pre>\n\n\n\n
    when your application design and protocol callback allow it.<\/p>\n\n\n\n
    \n\n\n\n
    Best Practice 7<\/h2>\n\n\n\n
    Monitor lag AND commit health.<\/p>\n\n\n\n
    A consumer can be processing quickly but failing commits.<\/p>\n\n\n\n
    Then:<\/p>\n\n\n\n
    Current position moves\nCommitted position stays behind\n<\/code><\/pre>\n\n\n\n
    A crash may cause large replay.<\/p>\n\n\n\n
    Monitor both.<\/p>\n\n\n\n
    \n\n\n\n
    85. Consumer Failover Example<\/h1>\n\n\n\n
    Topic:<\/p>\n\n\n\n
    payments\nP0 P1 P2 P3\n<\/code><\/pre>\n\n\n\n
    Group:<\/p>\n\n\n\n
    payment-risk\n<\/code><\/pre>\n\n\n\n
    Before:<\/p>\n\n\n\n
    Consumer A -> P0 P1\nConsumer B -> P2 P3\n<\/code><\/pre>\n\n\n\n
    Consumer B crashes.<\/p>\n\n\n\n
    After reassignment:<\/p>\n\n\n\n
    Consumer A -> P0 P1 P2 P3\n<\/code><\/pre>\n\n\n\n
    If:<\/p>\n\n\n\n
    P2 committed = 900\nP2 current before crash = 920\n<\/code><\/pre>\n\n\n\n
    new processing resumes from committed progress around:<\/p>\n\n\n\n
    900\n<\/code><\/pre>\n\n\n\n
    Records near 900-919 may be processed again.<\/p>\n\n\n\n
    That is why payment processing must be idempotent.<\/p>\n\n\n\n
    \n\n\n\n
    86. Fault Tolerance Depends on More Than Kafka<\/h1>\n\n\n\n
    Kafka can reassign partitions.<\/p>\n\n\n\n
    But your application may depend on:<\/p>\n\n\n\n
    Database\nHTTP service\nCache\nFile system\nMachine learning service\nPayment gateway\n<\/code><\/pre>\n\n\n\n
    If those are unhealthy:<\/p>\n\n\n\n
    Consumer is technically alive\nbut business processing cannot complete.\n<\/code><\/pre>\n\n\n\n
    Production health must include downstream dependencies.<\/p>\n\n\n\n
    \n\n\n\n
    87. Consumer Performance Tuning \u2014 Start With Measurement<\/h1>\n\n\n\n
    Never start by changing ten configs.<\/p>\n\n\n\n
    Use:<\/p>\n\n\n\n
    1. Define SLO\n2. Measure baseline\n3. Measure lag\n4. Measure processing time\n5. Identify bottleneck\n6. Change one controlled variable\n7. Load test\n8. Failure test\n9. Measure again\n<\/code><\/pre>\n\n\n\n
    Questions:<\/p>\n\n\n\n
    Is Kafka fetch slow?\nIs processing slow?\nIs database slow?\nAre partitions skewed?\nAre consumers rebalancing?\nAre commits slow?\nIs one partition hot?\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    88. Goal 1 \u2014 Increase Throughput<\/h1>\n\n\n\n
    Potential levers:<\/p>\n\n\n\n
    Increase useful consumer count\nIncrease topic partitions when architecture requires it\nIncrease fetch efficiency\nTune fetch.min.bytes\nTune fetch.max.wait.ms\nTune fetch sizes\nTune max.poll.records\nBatch database writes\nParallelize downstream work carefully\nReduce serialization\/deserialization overhead\nAvoid frequent commits\nAvoid frequent rebalances\n<\/code><\/pre>\n\n\n\n
    But always protect:<\/p>\n\n\n\n
    Ordering\nCorrect commits\nMemory\nFailure recovery\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    89. Goal 2 \u2014 Lower Latency<\/h1>\n\n\n\n
    Potential levers:<\/p>\n\n\n\n
    Lower fetch waiting\nSmaller processing batches\nFast downstream services\nAppropriate consumer locality\nAvoid huge max.poll.records\nReduce rebalance frequency\nFast deserialization\nMonitor broker throttling\n<\/code><\/pre>\n\n\n\n
    Tradeoff:<\/p>\n\n\n\n
    Smaller batches\n=\nmore requests \/ overhead\n<\/code><\/pre>\n\n\n\n
    Again:<\/p>\n\n\n\n
    Latency\nvs\nThroughput\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    90. Goal 3 \u2014 Improve Reliability<\/h1>\n\n\n\n
    Focus on:<\/p>\n\n\n\n
    Correct offset commit timing\nIdempotent business processing\nSafe rebalance handling\nFailure testing\nIntentional auto.offset.reset\nCommit monitoring\nConsumer lag monitoring\nRetries in downstream operations\nDead-letter\/error strategy where appropriate\nTransactions where appropriate\n<\/code><\/pre>\n\n\n\n
    The consumer is reliable only if the business result<\/strong> is reliable.<\/p>\n\n\n\n
    \n\n\n\n
    91. Goal 4 \u2014 Improve Availability<\/h1>\n\n\n\n
    Focus on:<\/p>\n\n\n\n
    Multiple consumer instances\nEnough partitions\nHealthy group coordination\nFast but sensible failure detection\nStable deployments\nControlled rolling restarts\nAvoid rebalance storms\nRedundant downstream services\nMonitoring and alerting\n<\/code><\/pre>\n\n\n\n
    Availability is not just:<\/p>\n\n\n\n
    \"Consumer process is running.\"\n<\/code><\/pre>\n\n\n\n
    It means:<\/p>\n\n\n\n
    The group continues making useful progress.\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    92. Consumer Configuration Study Map<\/h1>\n\n\n\n
    Important settings include:<\/p>\n\n\n\n
    Setting<\/th>What it controls<\/th>Why it matters<\/th><\/tr><\/thead>
    bootstrap.servers<\/code><\/td>Initial Kafka endpoints<\/td>Cluster discovery<\/td><\/tr>
    group.id<\/code><\/td>Consumer group identity<\/td>Work sharing + offsets<\/td><\/tr>
    key.deserializer<\/code><\/td>Key decoding<\/td>Correct application types<\/td><\/tr>
    value.deserializer<\/code><\/td>Value decoding<\/td>Correct application data<\/td><\/tr>
    enable.auto.commit<\/code><\/td>Automatic commits<\/td>Delivery semantics<\/td><\/tr>
    auto.commit.interval.ms<\/code><\/td>Auto commit interval<\/td>Replay\/commit window<\/td><\/tr>
    auto.offset.reset<\/code><\/td>Start point when no valid commit exists<\/td>Recovery\/new-group behavior<\/td><\/tr>
    max.poll.records<\/code><\/td>Records returned per poll<\/td>Processing batch<\/td><\/tr>
    max.poll.interval.ms<\/code><\/td>Max time between expected poll progress<\/td>Rebalance\/failure detection<\/td><\/tr>
    fetch.min.bytes<\/code><\/td>Minimum fetch target<\/td>Throughput vs latency<\/td><\/tr>
    fetch.max.wait.ms<\/code><\/td>Max broker fetch wait<\/td>Throughput vs latency<\/td><\/tr>
    fetch.max.bytes<\/code><\/td>Fetch response size<\/td>Network\/memory<\/td><\/tr>
    max.partition.fetch.bytes<\/code><\/td>Per-partition fetch amount<\/td>Large records\/memory<\/td><\/tr>
    isolation.level<\/code><\/td>Transaction visibility<\/td>Exactly-once transactional reads<\/td><\/tr>
    group.protocol<\/code><\/td>Group protocol selection<\/td>Modern vs Classic behavior<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Protocol-specific timeout\/assignment settings must be interpreted according to whether the group is using the modern Consumer protocol or Classic protocol.<\/p>\n\n\n\n
    \n\n\n\n
    93. Modern Group Protocol Configuration Awareness<\/h1>\n\n\n\n
    For modern Kafka, students should recognize:<\/p>\n\n\n\n
    group.protocol=consumer\n<\/code><\/pre>\n\n\n\n
    as the setting associated with using the newer Consumer group protocol in clients that support it.<\/p>\n\n\n\n
    With this protocol:<\/p>\n\n\n\n
    Assignment becomes broker-driven\nGroup behavior is more incremental\nSome Classic client-side assignment\/heartbeat settings are no longer used the same way\n<\/code><\/pre>\n\n\n\n
    Do not mix configuration advice from different group protocols.<\/p>\n\n\n\n
    \n\n\n\n
    94. Classic Group Assignment Strategies<\/h1>\n\n\n\n
    If working with Classic groups, common assignment strategies include:<\/p>\n\n\n\n
    RangeAssignor\nRoundRobinAssignor\nStickyAssignor\nCooperativeStickyAssignor\n<\/code><\/pre>\n\n\n\n
    Each has different distribution and rebalance behavior.<\/p>\n\n\n\n
    Do not say:<\/p>\n\n\n\n
    \"Kafka always uses Round Robin.\"\n<\/code><\/pre>\n\n\n\n
    Assignment depends on protocol and configuration.<\/p>\n\n\n\n
    \n\n\n\n
    95. Modern Consumer Protocol Assignment<\/h1>\n\n\n\n
    With the newer Consumer protocol:<\/p>\n\n\n\n
    Assignment logic is broker-side\n<\/code><\/pre>\n\n\n\n
    The server provides available assignment strategies, and the consumer can participate according to the modern group protocol configuration.<\/p>\n\n\n\n
    The important teaching point is:<\/p>\n\n\n\n
    Classic:\nclient-led assignment mechanics\n\nModern Consumer protocol:\nbroker-side coordinator-driven assignment mechanics\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    96. Consumer Lag Monitoring<\/h1>\n\n\n\n
    Monitor lag per:<\/p>\n\n\n\n
    Consumer Group\nTopic\nPartition\n<\/code><\/pre>\n\n\n\n
    Why partition-level?<\/p>\n\n\n\n
    Because:<\/p>\n\n\n\n
    Overall group lag = 10,000\n<\/code><\/pre>\n\n\n\n
    may hide:<\/p>\n\n\n\n
    P0 lag = 0\nP1 lag = 0\nP2 lag = 10,000\n<\/code><\/pre>\n\n\n\n
    That often indicates:<\/p>\n\n\n\n
    Hot partition\nSlow record type\nBad key distribution\nStuck processing\n<\/code><\/pre>\n\n\n\n
    Partition-level visibility is essential.<\/p>\n\n\n\n
    \n\n\n\n
    97. Metrics Worth Monitoring<\/h1>\n\n\n\n
    Examples of useful consumer-health signals:<\/p>\n\n\n\n
    Records consumed rate\nBytes consumed rate\nFetch latency\nFetch rate\nRecords per request\nAssigned partitions\nConsumer lag\nCommit latency\nCommit rate\nCommit failures\nRebalance rate\nTime since last successful poll\nProcessing latency\nApplication error rate\n<\/code><\/pre>\n\n\n\n
    Kafka metrics alone are not enough.<\/p>\n\n\n\n
    Also measure:<\/p>\n\n\n\n
    Database latency\nAPI latency\nQueue depth\nCPU\nMemory\nGC\nThread pool saturation\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    98. Common Consumer Mistakes<\/h1>\n\n\n\n
    Mistake 1 \u2014 More Consumers Than Partitions<\/h2>\n\n\n\n
    4 partitions\n20 consumers\n<\/code><\/pre>\n\n\n\n
    You do not get twenty-way partition processing.<\/p>\n\n\n\n
    Many consumers remain idle.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 2 \u2014 Commit Before Processing<\/h2>\n\n\n\n
    Can create business data loss after a crash.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 3 \u2014 Never Commit<\/h2>\n\n\n\n
    Consumer restarts may replay huge amounts of work.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 4 \u2014 Auto Commit Without Understanding It<\/h2>\n\n\n\n
    Easy configuration can hide incorrect business semantics.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 5 \u2014 Slow Processing Between Polls<\/h2>\n\n\n\n
    Can cause consumer-group instability and repeated rebalances.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 6 \u2014 Long Blocking API Calls in Poll Thread<\/h2>\n\n\n\n
    Example:<\/p>\n\n\n\n
    poll()\n  |\n  v\nCall external API\nwait 8 minutes\n<\/code><\/pre>\n\n\n\n
    Dangerous if poll progress requirements are exceeded.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 7 \u2014 Ignoring Rebalance Callbacks<\/h2>\n\n\n\n
    Can lose safe local state or commit the wrong progress.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 8 \u2014 Assuming Offset = Message ID<\/h2>\n\n\n\n
    Offset is a partition position.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 9 \u2014 Assuming Consumer Lag Means \u201cAdd Consumers\u201d<\/h2>\n\n\n\n
    If the topic has too few partitions, extra consumers do nothing.<\/p>\n\n\n\n
    If one partition is hot, extra consumers do not split that partition.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 10 \u2014 Confusing Different Consumer Groups<\/h2>\n\n\n\n
    Different groups do not<\/strong> share work with each other.<\/p>\n\n\n\n
    They independently consume the topic.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 11 \u2014 Using One Group ID for Unrelated Applications<\/h2>\n\n\n\n
    If analytics and notifications accidentally use the same group.id<\/code>:<\/p>\n\n\n\n
    They will split partitions\n<\/code><\/pre>\n\n\n\n
    instead of both receiving all events.<\/p>\n\n\n\n
    This is a serious architecture mistake.<\/p>\n\n\n\n
    \n\n\n\n
    Mistake 12 \u2014 Ignoring Protocol Version<\/h2>\n\n\n\n
    Applying Classic timeout\/assignor advice to a modern Consumer-protocol group can be wrong.<\/p>\n\n\n\n
    Know which protocol you are operating.<\/p>\n\n\n\n
    \n\n\n\n
    99. Java Consumer Example<\/h1>\n\n\n\n
    A simplified teaching example:<\/p>\n\n\n\n
    Properties props = new Properties();\n\nprops.put(\"bootstrap.servers\", \"<BOOTSTRAP_SERVER>\");\nprops.put(\"group.id\", \"vehicle-analytics\");\n\nprops.put(\n    \"key.deserializer\",\n    \"org.apache.kafka.common.serialization.StringDeserializer\"\n);\n\nprops.put(\n    \"value.deserializer\",\n    \"org.apache.kafka.common.serialization.StringDeserializer\"\n);\n\nprops.put(\"enable.auto.commit\", \"false\");\nprops.put(\"auto.offset.reset\", \"earliest\");\n\n\/\/ For modern Kafka clients\/clusters where you intentionally use<\/em>\n\/\/ the newer Consumer group protocol:<\/em>\n\/\/ props.put(\"group.protocol\", \"consumer\");<\/em>\n\nKafkaConsumer<String, String> consumer =\n    new KafkaConsumer<>(props);\n\nconsumer.subscribe(List.of(\"vehicle-telemetry\"));\n\ntry {\n    while (true) {\n\n        ConsumerRecords<String, String> records =\n            consumer.poll(Duration.ofMillis(1000));\n\n        for (ConsumerRecord<String, String> record : records) {\n\n            System.out.println(\n                \"topic=\" + record.topic()\n                + \" partition=\" + record.partition()\n                + \" offset=\" + record.offset()\n                + \" key=\" + record.key()\n                + \" value=\" + record.value()\n            );\n\n            \/\/ Business processing here<\/em>\n        }\n\n        \/\/ Teaching example:<\/em>\n        \/\/ commit after the batch is successfully processed.<\/em>\n        consumer.commitSync();\n    }\n}\nfinally {\n    consumer.close();\n}\n<\/code><\/pre>\n\n\n\n
    This example favors clarity.<\/p>\n\n\n\n
    A production application should additionally implement:<\/p>\n\n\n\n
    Error handling\nControlled shutdown\nRebalance handling\nMetrics\nRetry strategy\nIdempotent processing\nSecurity configuration\nAppropriate commit strategy\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    100. Confluent Kafka Cluster Connection<\/h1>\n\n\n\n
    For a Confluent-managed cluster, the consumer also needs authentication\/security configuration.<\/p>\n\n\n\n
    Conceptually:<\/p>\n\n\n\n
    bootstrap.servers=<CONFLUENT_BOOTSTRAP>\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=PLAIN\n\nsasl.jaas.config=<CREDENTIAL_CONFIGURATION>\n<\/code><\/pre>\n\n\n\n
    Then:<\/p>\n\n\n\n
    Consumer\n   |\n   v\nAuthenticate\n   |\n   v\nDiscover brokers\n   |\n   v\nFind group coordinator\n   |\n   v\nJoin group\n   |\n   v\nReceive partition assignment\n   |\n   v\nFetch from partition leaders\n<\/code><\/pre>\n\n\n\n
    Keep credentials in secure configuration\/secrets, not hardcoded source code.<\/p>\n\n\n\n
    \n\n\n\n
    101. Complete Consumer Journey<\/h1>\n\n\n\n
    Now combine everything:<\/p>\n\n\n\n
    APPLICATION START\n      |\n      v\nCreate KafkaConsumer\n      |\n      v\nbootstrap.servers\n      |\n      v\nAuthentication\n      |\n      v\nCluster Metadata\n      |\n      v\ngroup.id\n      |\n      v\nFind Group Coordinator\n      |\n      v\nJoin Consumer Group\n      |\n      v\nGroup Protocol\n      |\n      v\nPartition Assignment\n      |\n      v\nFind Partition Leaders\n      |\n      v\nFetch Requests\n      |\n      v\nRecord Batches\n      |\n      v\nDeserialize Key \/ Value\n      |\n      v\npoll() returns ConsumerRecords\n      |\n      v\nBusiness Processing\n      |\n      v\nCurrent Position Advances\n      |\n      v\nCommit Safe Offset\n      |\n      v\n__consumer_offsets\n      |\n      v\npoll() again\n\nIf membership changes:\n      |\n      v\nRebalance \/ Reassignment\n      |\n      v\nPartitions Move\n      |\n      v\nResume From Committed Progress\n<\/code><\/pre>\n\n\n\n
    Students should be able to explain every box.<\/p>\n\n\n\n
    \n\n\n\n
    102. Lab 1 \u2014 One Consumer, Multiple Partitions<\/h1>\n\n\n\n
    Create:<\/p>\n\n\n\n
    Topic: orders\nPartitions: 6\n<\/code><\/pre>\n\n\n\n
    Start:<\/p>\n\n\n\n
    1 consumer\ngroup.id = order-workers\n<\/code><\/pre>\n\n\n\n
    Observe:<\/p>\n\n\n\n
    One consumer owns all six partitions.\n<\/code><\/pre>\n\n\n\n
    Questions:<\/p>\n\n\n\n
      \n
    1. Can this application process partitions in parallel internally?<\/li>\n\n\n\n
    2. What limits its throughput?<\/li>\n\n\n\n
    3. What happens if the process crashes?<\/li>\n<\/ol>\n\n\n\n
      \n\n\n\n
      103. Lab 2 \u2014 Scale to Three Consumers<\/h1>\n\n\n\n
      Start:<\/p>\n\n\n\n
      3 consumer instances\nsame group.id\n<\/code><\/pre>\n\n\n\n
      Observe the assignment.<\/p>\n\n\n\n
      Expected concept:<\/p>\n\n\n\n
      6 partitions\n3 consumers\n\u2248 2 partitions per consumer\n<\/code><\/pre>\n\n\n\n
      Questions:<\/p>\n\n\n\n
      Did a rebalance occur?\nWhich partitions moved?\nDid lag fall?\nDid throughput improve?\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      104. Lab 3 \u2014 More Consumers Than Partitions<\/h1>\n\n\n\n
      Keep:<\/p>\n\n\n\n
      6 partitions\n<\/code><\/pre>\n\n\n\n
      Start:<\/p>\n\n\n\n
      10 consumers\nsame group.id\n<\/code><\/pre>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Some consumers receive no partitions.\n<\/code><\/pre>\n\n\n\n
      This demonstrates:<\/p>\n\n\n\n
      Partitions determine partition-level parallelism.\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      105. Lab 4 \u2014 Consumer Failure<\/h1>\n\n\n\n
      Start three consumers.<\/p>\n\n\n\n
      Kill one abruptly.<\/p>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Failure detection\nReassignment\nRemaining consumers take partitions\nConsumption resumes\n<\/code><\/pre>\n\n\n\n
      Record:<\/p>\n\n\n\n
      Rebalance duration\nLag increase\nDuplicate processing\nNew assignments\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      106. Lab 5 \u2014 Graceful Shutdown vs Crash<\/h1>\n\n\n\n
      Experiment A:<\/p>\n\n\n\n
      consumer.close()\n<\/code><\/pre>\n\n\n\n
      Experiment B:<\/p>\n\n\n\n
      kill -9 \/ abrupt container stop\n<\/code><\/pre>\n\n\n\n
      Compare:<\/p>\n\n\n\n
      Time to group recovery\nLogs\nRebalance behavior\nDuplicate window\n<\/code><\/pre>\n\n\n\n
      This teaches why controlled shutdown matters.<\/p>\n\n\n\n
      \n\n\n\n
      107. Lab 6 \u2014 Automatic Commit<\/h1>\n\n\n\n
      Configure:<\/p>\n\n\n\n
      enable.auto.commit=true\n<\/code><\/pre>\n\n\n\n
      Process records slowly.<\/p>\n\n\n\n
      Stop the consumer at different times.<\/p>\n\n\n\n
      Restart it.<\/p>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Which records repeat?\nWhere does consumption restart?\n<\/code><\/pre>\n\n\n\n
      The objective is not to prove \u201cauto commit is bad.\u201d<\/p>\n\n\n\n
      The objective is to understand its failure window.<\/p>\n\n\n\n
      \n\n\n\n
      108. Lab 7 \u2014 Manual Commit<\/h1>\n\n\n\n
      Configure:<\/p>\n\n\n\n
      enable.auto.commit=false\n<\/code><\/pre>\n\n\n\n
      Flow:<\/p>\n\n\n\n
      poll\nprocess all records successfully\ncommitSync\n<\/code><\/pre>\n\n\n\n
      Crash:<\/p>\n\n\n\n
      after processing\nbefore commit\n<\/code><\/pre>\n\n\n\n
      Observe duplicates after restart.<\/p>\n\n\n\n
      Then explain:<\/p>\n\n\n\n
      Why duplicates are expected\nWhy idempotency matters\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      109. Lab 8 \u2014 Commit Before Processing<\/h1>\n\n\n\n
      In a disposable training environment only:<\/p>\n\n\n\n
      poll\ncommit\nthen process\n<\/code><\/pre>\n\n\n\n
      Force a crash after commit but before processing finishes.<\/p>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Consumer restarts after committed position\n<\/code><\/pre>\n\n\n\n
      Explain the potential lost-processing window.<\/p>\n\n\n\n
      This demonstrates at-most-once semantics.<\/p>\n\n\n\n
      \n\n\n\n
      110. Lab 9 \u2014 Fan-Out<\/h1>\n\n\n\n
      Topic:<\/p>\n\n\n\n
      orders\n<\/code><\/pre>\n\n\n\n
      Create:<\/p>\n\n\n\n
      Group A = analytics\nGroup B = notifications\n<\/code><\/pre>\n\n\n\n
      Both groups consume the same events independently.<\/p>\n\n\n\n
      Observe separate committed offsets.<\/p>\n\n\n\n
      \n\n\n\n
      111. Lab 10 \u2014 Load Balance<\/h1>\n\n\n\n
      Create:<\/p>\n\n\n\n
      Group = order-workers\nConsumers = 3\n<\/code><\/pre>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Partitions divided among consumers.\n<\/code><\/pre>\n\n\n\n
      Compare with Lab 9.<\/p>\n\n\n\n
      Students should be able to explain:<\/p>\n\n\n\n
      Different groups = fan-out\nSame group = workload sharing\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      112. Lab 11 \u2014 Consumer Lag<\/h1>\n\n\n\n
      Make the consumer intentionally slow.<\/p>\n\n\n\n
      Observe lag growing.<\/p>\n\n\n\n
      Then test:<\/p>\n\n\n\n
      Add a consumer\n<\/code><\/pre>\n\n\n\n
      If enough partitions exist, lag recovery may improve.<\/p>\n\n\n\n
      Then create a scenario where one partition is hot.<\/p>\n\n\n\n
      Observe that adding consumers may not solve a single hot-partition bottleneck.<\/p>\n\n\n\n
      \n\n\n\n
      113. Lab 12 \u2014 max.poll.records<\/code><\/h1>\n\n\n\n
      Test:<\/p>\n\n\n\n
      max.poll.records = 500\n<\/code><\/pre>\n\n\n\n
      Measure processing time.<\/p>\n\n\n\n
      Then:<\/p>\n\n\n\n
      max.poll.records = 50\n<\/code><\/pre>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Poll frequency\nProcessing batch size\nMemory\nLatency\nThroughput\n<\/code><\/pre>\n\n\n\n
      Explain why the right value depends on processing cost.<\/p>\n\n\n\n
      \n\n\n\n
      114. Lab 13 \u2014 Rebalance During Processing<\/h1>\n\n\n\n
      Run several consumers.<\/p>\n\n\n\n
      While processing:<\/p>\n\n\n\n
      Start another consumer.\n<\/code><\/pre>\n\n\n\n
      Observe:<\/p>\n\n\n\n
      Partition assignment changes.\n<\/code><\/pre>\n\n\n\n
      Add rebalance callbacks and print:<\/p>\n\n\n\n
      revoked partitions\nassigned partitions\nlost partitions\n<\/code><\/pre>\n\n\n\n
      This turns rebalancing from an abstract concept into something visible.<\/p>\n\n\n\n
      \n\n\n\n
      115. Production Offset Strategy Example<\/h1>\n\n\n\n
      Suppose we process payment events.<\/p>\n\n\n\n
      Requirement:<\/p>\n\n\n\n
      Never charge twice.\nDo not silently lose a payment event.\n<\/code><\/pre>\n\n\n\n
      A reasonable architecture might include:<\/p>\n\n\n\n
      At-least-once Kafka consumption\n      |\n      v\nStable event\/payment ID\n      |\n      v\nIdempotent database\/payment operation\n      |\n      v\nCommit Kafka offset after safe business completion\n<\/code><\/pre>\n\n\n\n
      Kafka offsets alone are not enough.<\/p>\n\n\n\n
      The business operation itself must be designed for retries.<\/p>\n\n\n\n
      \n\n\n\n
      116. Production Analytics Example<\/h1>\n\n\n\n
      Analytics event:<\/p>\n\n\n\n
      page-view\n<\/code><\/pre>\n\n\n\n
      Duplicate processing may be less harmful.<\/p>\n\n\n\n
      A different commit\/batching strategy may be acceptable.<\/p>\n\n\n\n
      Therefore:<\/p>\n\n\n\n
      Payment consumer configuration\n!=\nAnalytics consumer configuration\n<\/code><\/pre>\n\n\n\n
      There is no universal \u201cbest Kafka consumer config.\u201d<\/p>\n\n\n\n
      There is only:<\/p>\n\n\n\n
      Best config for this workload and correctness requirement.\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      117. Production Consumer Checklist<\/h1>\n\n\n\n
      Before calling a consumer production-ready:<\/p>\n\n\n\n
      Group Design<\/h2>\n\n\n\n
        \n
      • [ ] Intentional\u00a0group.id<\/code><\/li>\n\n\n\n
      • [ ] Fan-out vs load-balancing pattern understood<\/li>\n\n\n\n
      • [ ] Consumer count matched to expected partition-level parallelism<\/li>\n\n\n\n
      • [ ] Partition count capacity planned<\/li>\n\n\n\n
      • [ ] Hot partitions considered<\/li>\n\n\n\n
      • [ ] Group protocol known: Consumer or Classic<\/li>\n<\/ul>\n\n\n\n
        Offset Strategy<\/h2>\n\n\n\n
          \n
        • [ ]\u00a0enable.auto.commit<\/code>\u00a0chosen intentionally<\/li>\n\n\n\n
        • [ ] Commit boundary matches business correctness<\/li>\n\n\n\n
        • [ ] Committed value represents next offset to process<\/li>\n\n\n\n
        • [ ]\u00a0auto.offset.reset<\/code>\u00a0chosen intentionally<\/li>\n\n\n\n
        • [ ] Duplicate processing behavior understood<\/li>\n\n\n\n
        • [ ] Data-loss window understood<\/li>\n\n\n\n
        • [ ] Commit failures monitored<\/li>\n<\/ul>\n\n\n\n
          Processing<\/h2>\n\n\n\n
            \n
          • [ ] Processing is idempotent where practical<\/li>\n\n\n\n
          • [ ] Slow downstream dependencies handled<\/li>\n\n\n\n
          • [ ] Retry strategy defined<\/li>\n\n\n\n
          • [ ] Dead-letter\/error handling defined if needed<\/li>\n\n\n\n
          • [ ] Long-processing design tested<\/li>\n\n\n\n
          • [ ] Ordering requirements documented<\/li>\n<\/ul>\n\n\n\n
            Rebalancing<\/h2>\n\n\n\n
              \n
            • [ ] Rebalance callbacks handled where needed<\/li>\n\n\n\n
            • [ ] Safe progress committed before normal revoke<\/li>\n\n\n\n
            • [ ] Lost-partition behavior understood<\/li>\n\n\n\n
            • [ ] Rolling deployment behavior tested<\/li>\n\n\n\n
            • [ ] Consumer crash tested<\/li>\n\n\n\n
            • [ ] Frequent rebalance alerting configured<\/li>\n<\/ul>\n\n\n\n
              Performance<\/h2>\n\n\n\n
                \n
              • [ ] Lag monitored<\/li>\n\n\n\n
              • [ ] Lag monitored per partition<\/li>\n\n\n\n
              • [ ]\u00a0max.poll.records<\/code>\u00a0load tested<\/li>\n\n\n\n
              • [ ] Fetch settings load tested<\/li>\n\n\n\n
              • [ ] Memory measured<\/li>\n\n\n\n
              • [ ] CPU measured<\/li>\n\n\n\n
              • [ ] Deserialization cost measured<\/li>\n\n\n\n
              • [ ] Database\/API latency measured<\/li>\n\n\n\n
              • [ ] Consumer throughput measured<\/li>\n<\/ul>\n\n\n\n
                Availability<\/h2>\n\n\n\n
                  \n
                • [ ] Multiple consumer instances used where required<\/li>\n\n\n\n
                • [ ] Failure detection behavior tested<\/li>\n\n\n\n
                • [ ] Consumers distributed appropriately<\/li>\n\n\n\n
                • [ ] Downstream dependencies have availability strategy<\/li>\n\n\n\n
                • [ ] Shutdown is graceful<\/li>\n\n\n\n
                • [ ] Restart\/recovery tested<\/li>\n<\/ul>\n\n\n\n
                  Security<\/h2>\n\n\n\n
                    \n
                  • [ ] Credentials stored securely<\/li>\n\n\n\n
                  • [ ] Topic READ permissions minimal<\/li>\n\n\n\n
                  • [ ] Group permissions correctly scoped<\/li>\n\n\n\n
                  • [ ] TLS\/SASL configuration validated<\/li>\n<\/ul>\n\n\n\n
                    \n\n\n\n
                    118. Troubleshooting Checklist<\/h1>\n\n\n\n
                    Symptom: Consumer Lag Growing<\/h2>\n\n\n\n
                    Check:<\/p>\n\n\n\n
                    Consumer count\nPartition count\nHot partitions\nProcessing latency\nDatabase latency\nAPI latency\nFetch latency\nCPU\nMemory\nGC\nRebalances\nCommit latency\nBroker throttling\n<\/code><\/pre>\n\n\n\n
                    \n\n\n\n
                    Symptom: Constant Rebalances<\/h2>\n\n\n\n
                    Check:<\/p>\n\n\n\n
                    Consumer crashes\nDeployment restarts\nmax.poll.interval\nLong processing\nMembership timeouts\nNetwork stability\nProtocol configuration\nRebalance callbacks\n<\/code><\/pre>\n\n\n\n
                    \n\n\n\n
                    Symptom: Duplicate Processing<\/h2>\n\n\n\n
                    Check:<\/p>\n\n\n\n
                    Crash after processing but before commit\nCommit frequency\nRetry behavior\nRebalance boundaries\nIdempotency\nOffset reset\nManual seeks\n<\/code><\/pre>\n\n\n\n
                    Duplicates are often expected in at-least-once systems.<\/p>\n\n\n\n
                    \n\n\n\n
                    Symptom: Missing Business Work<\/h2>\n\n\n\n
                    Check immediately:<\/p>\n\n\n\n
                    Were offsets committed before processing?\nWas auto.offset.reset=latest used for a new group?\nWas seek() used incorrectly?\nDid application drop errors?\nWas downstream work asynchronous and not awaited?\n<\/code><\/pre>\n\n\n\n
                    \n\n\n\n
                    Symptom: Consumers Idle<\/h2>\n\n\n\n
                    Check:<\/p>\n\n\n\n
                    Number of partitions\nNumber of consumers in same group\nTopic subscription\nAuthorization\nAssignment\n<\/code><\/pre>\n\n\n\n
                    If:<\/p>\n\n\n\n
                    consumers > partitions\n<\/code><\/pre>\n\n\n\n
                    idle members can be completely normal.<\/p>\n\n\n\n
                    \n\n\n\n
                    119. Interview Questions<\/h1>\n\n\n\n
                    What is a Kafka consumer?<\/h2>\n\n\n\n
                    A client application that fetches and processes records from Kafka topic partitions.<\/p>\n\n\n\n
                    What is a Consumer Group?<\/h2>\n\n\n\n
                    A set of consumers sharing the same group identity and cooperating to divide partition consumption.<\/p>\n\n\n\n
                    Can two consumers in the same group read the same partition simultaneously?<\/h2>\n\n\n\n
                    Under normal group assignment, one partition is assigned to at most one consumer in that group at a time.<\/p>\n\n\n\n
                    What if consumers are in different groups?<\/h2>\n\n\n\n
                    Each group consumes independently, enabling fan-out.<\/p>\n\n\n\n
                    What limits consumer parallelism?<\/h2>\n\n\n\n
                    Primarily the number of partitions available to the group at the Kafka partition-assignment level.<\/p>\n\n\n\n
                    What is a consumer offset?<\/h2>\n\n\n\n
                    A numerical position in a partition used to track reading progress.<\/p>\n\n\n\n
                    Current vs committed offset?<\/h2>\n\n\n\n
                    Current position is where the running consumer expects to read next. Committed offset is the saved recovery checkpoint for the group.<\/p>\n\n\n\n
                    Where are group offsets stored?<\/h2>\n\n\n\n
                    Kafka stores committed group offsets in the internal __consumer_offsets<\/code> topic.<\/p>\n\n\n\n
                    What is rebalancing?<\/h2>\n\n\n\n
                    The process of changing partition ownership among consumers when group membership or subscribed partition metadata changes.<\/p>\n\n\n\n
                    What happens when a consumer crashes?<\/h2>\n\n\n\n
                    Kafka detects the membership failure, reassigns its partitions, and replacement consumers resume using committed group progress.<\/p>\n\n\n\n
                    What is fan-out?<\/h2>\n\n\n\n
                    Different consumer groups independently reading the same topic.<\/p>\n\n\n\n
                    What is load balancing?<\/h2>\n\n\n\n
                    Consumers in the same group sharing partition ownership.<\/p>\n\n\n\n
                    Why is commit timing important?<\/h2>\n\n\n\n
                    Because it determines whether failures may lead to duplicates or lost processing.<\/p>\n\n\n\n
                    At-least-once?<\/h2>\n\n\n\n
                    Process first, then commit; failures may cause reprocessing.<\/p>\n\n\n\n
                    At-most-once?<\/h2>\n\n\n\n
                    Commit before processing; failures can cause processing to be skipped.<\/p>\n\n\n\n
                    Does manual commit guarantee exactly once?<\/h2>\n\n\n\n
                    No. Exactly-once requires coordinated transaction\/idempotency design.<\/p>\n\n\n\n
                    What is consumer lag?<\/h2>\n\n\n\n
                    How far a consumer\/group is behind the available end of the partition stream.<\/p>\n\n\n\n
                    \n\n\n\n
                    120. Master Mental Model<\/h1>\n\n\n\n
                    A beginner says:<\/p>\n\n\n\n
                    Consumer reads messages.\n<\/code><\/pre>\n\n\n\n
                    A Kafka engineer says:<\/p>\n\n\n\n
                    Consumer starts\n    |\n    v\nConnects to Kafka\n    |\n    v\nFinds Group Coordinator\n    |\n    v\nJoins Consumer Group\n    |\n    v\nParticipates in Group Protocol\n    |\n    v\nReceives Partition Assignment\n    |\n    v\nFinds Partition Leaders\n    |\n    v\nFetches Record Batches\n    |\n    v\nTracks Current Position\n    |\n    v\nDeserializes\n    |\n    v\nProcesses Business Logic\n    |\n    v\nCommits Safe Next Offset\n    |\n    v\n__consumer_offsets\n    |\n    v\nHandles Rebalances\n    |\n    v\nRecovers From Failures\n    |\n    v\nContinues Processing\n<\/code><\/pre>\n\n\n\n
                    A production Kafka engineer asks:<\/p>\n\n\n\n
                    Is partition distribution balanced?\n\nCan processing keep up?\n\nWhat is the lag?\n\nWhat happens when this consumer crashes?\n\nWhat happens if commit succeeds but processing fails?\n\nWhat happens if processing succeeds but commit fails?\n\nCan duplicate processing hurt us?\n\nCan any record be skipped?\n\nHow long does a rebalance take?\n\nAre rebalances happening too often?\n\nDo we have enough partitions?\n\nAre we using the correct group protocol?\n\nAre downstream systems idempotent?\n\nAre offsets and business state coordinated correctly?\n<\/code><\/pre>\n\n\n\n
                    That is Kafka consumer mastery.<\/p>\n\n\n\n
                    \n\n\n\n
                    121. Final Takeaways<\/h1>\n\n\n\n
                    Remember these twelve ideas:<\/p>\n\n\n\n
                      \n
                    1. Kafka consumers\u00a0pull<\/strong>\u00a0records from partition leaders.<\/li>\n\n\n\n
                    2. Consumers in the same group\u00a0share partitions<\/strong>.<\/li>\n\n\n\n
                    3. Different consumer groups independently consume the same data.<\/li>\n\n\n\n
                    4. Partition count defines the main ceiling for partition-level consumer parallelism.<\/li>\n\n\n\n
                    5. Kafka balances workload by assigning\u00a0partitions<\/strong>, not individual messages.<\/li>\n\n\n\n
                    6. Rebalancing changes partition ownership when membership or metadata changes.<\/li>\n\n\n\n
                    7. Modern Kafka has a newer Consumer group protocol as well as the older Classic protocol.<\/li>\n\n\n\n
                    8. Current position and committed offset are different.<\/li>\n\n\n\n
                    9. Committed offset represents the\u00a0next position to resume\/read<\/strong>.<\/li>\n\n\n\n
                    10. Commit timing determines duplicate-vs-loss behavior during failures.<\/li>\n\n\n\n
                    11. At-least-once designs should expect reprocessing and use idempotent business logic.<\/li>\n\n\n\n
                    12. Gold-standard consumer operations require monitoring\u00a0lag, commits, rebalances, processing time, and downstream health<\/strong>.<\/li>\n<\/ol>\n\n\n\n
                      \n\n\n\n
                      122. Suggested Next Consumer Tutorials<\/h1>\n\n\n\n
                      After this foundation, continue with:<\/p>\n\n\n\n
                      Kafka Consumer Internals\n        |\n        v\nConsumer Group Protocol Deep Dive\n        |\n        v\nClassic vs Modern Consumer Rebalancing\n        |\n        v\nPartition Assignment Strategies\n        |\n        v\nOffsets and Delivery Semantics\n        |\n        v\nAt-Least-Once \/ At-Most-Once \/ Exactly-Once\n        |\n        v\nConsumer Performance Tuning Lab\n        |\n        v\nConsumer Lag Monitoring\n        |\n        v\nRebalance Troubleshooting\n        |\n        v\nProduction Consumer Failure Testing\n<\/code><\/pre>\n\n\n\n
                      That path takes a student from \u201cI can read a Kafka message\u201d to \u201cI can design, operate and troubleshoot a production Kafka consumer system.\u201d<\/p>\n\n\n\n
                      \n\n\n\n
                      Technical Source Basis<\/h2>\n\n\n\n
                      This tutorial was checked against current Apache Kafka 4.3 consumer documentation\/Javadocs and current Confluent consumer documentation for consumer groups, offset tracking, group coordination, rebalancing, fetch behavior, commit APIs, and consumer lag. URLs are intentionally omitted from this learning document.<\/p>\n","protected":false},"excerpt":{"rendered":"
                      Consumer Groups, Parallelism, Offsets, Rebalancing, Failover, Consumption Patterns and Production Tuning Audience: Students and freshers with no previous Kafka experienceGoal: Start with \u201cWhat is a consumer?\u201d and finish with… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1152","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1152","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1152"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1152\/revisions"}],"predecessor-version":[{"id":1153,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1152\/revisions\/1153"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1152"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}