{"id":1148,"date":"2026-08-19T02:07:43","date_gmt":"2026-08-19T02:07:43","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/?p=1148"},"modified":"2026-08-19T02:07:44","modified_gmt":"2026-08-19T02:07:44","slug":"kafka-master-tutorials-series-5-deep-dive-into-kafka-producers","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/kafka-master-tutorials-series-5-deep-dive-into-kafka-producers\/","title":{"rendered":"Kafka Master Tutorials Series: 5 – Deep Dive Into Kafka Producers"},"content":{"rendered":"\n

From send()<\/code> to Broker ACK: Keys, Partitions, Batching, Retries, Reliability, Latency and Performance Tuning<\/h2>\n\n\n\n
\n

Audience:<\/strong> Students and freshers with no prior Kafka experience
Goal:<\/strong> Build from producer fundamentals to production-grade Kafka producer design and tuning
Training environment:<\/strong> Confluent Kafka Cluster<\/p>\n<\/blockquote>\n\n\n\n

\"Kafka<\/figure>\n\n\n\n
\n\n\n\n

1. Learning Objectives<\/h1>\n\n\n\n

By the end of this tutorial, you should understand:<\/p>\n\n\n\n

    \n
  • What a Kafka Producer is<\/li>\n\n\n\n
  • What a Kafka record contains<\/li>\n\n\n\n
  • Key vs value<\/li>\n\n\n\n
  • Which record fields are optional<\/li>\n\n\n\n
  • How a producer discovers Kafka brokers<\/li>\n\n\n\n
  • How a producer selects a topic partition<\/li>\n\n\n\n
  • What happens when a key is present<\/li>\n\n\n\n
  • What happens when a key is absent<\/li>\n\n\n\n
  • Why “round robin” is not a complete description of modern default no-key partitioning<\/li>\n\n\n\n
  • What batching does<\/li>\n\n\n\n
  • What\u00a0batch.size<\/code>\u00a0does<\/li>\n\n\n\n
  • What\u00a0linger.ms<\/code>\u00a0does<\/li>\n\n\n\n
  • What compression does<\/li>\n\n\n\n
  • How producer memory is used<\/li>\n\n\n\n
  • What retries do<\/li>\n\n\n\n
  • Why idempotence matters<\/li>\n\n\n\n
  • What\u00a0acks=0<\/code>,\u00a0acks=1<\/code>, and\u00a0acks=all<\/code>\u00a0mean<\/li>\n\n\n\n
  • How replicas and ISR affect producer reliability<\/li>\n\n\n\n
  • Who assigns Kafka offsets<\/li>\n\n\n\n
  • What\u00a0flush()<\/code>\u00a0does<\/li>\n\n\n\n
  • How to optimize a producer for throughput<\/li>\n\n\n\n
  • How to optimize a producer for low latency<\/li>\n\n\n\n
  • How to build for production reliability<\/li>\n\n\n\n
  • Which producer metrics should be monitored<\/li>\n<\/ul>\n\n\n\n
    \n\n\n\n

    2. What Is a Kafka Producer?<\/h1>\n\n\n\n

    A Kafka Producer<\/strong> is a client application that publishes records to Kafka topics.<\/p>\n\n\n\n

    The simplest mental model is:<\/p>\n\n\n\n

    Application\n    |\n    v\nKafka Producer\n    |\n    v\nKafka Topic\n<\/code><\/pre>\n\n\n\n
    For example:<\/p>\n\n\n\n
    Vehicle\n   |\n   v\nTelematics Service\n   |\n   v\nKafka Producer\n   |\n   v\nTopic: vehicle-telemetry\n<\/code><\/pre>\n\n\n\n
    The producer does much more than simply “send a message.”<\/p>\n\n\n\n
    Inside the producer, Kafka may need to:<\/p>\n\n\n\n
    Create a record\n    |\n    v\nSerialize it\n    |\n    v\nObtain cluster metadata\n    |\n    v\nSelect a partition\n    |\n    v\nPlace it in memory\n    |\n    v\nBuild a batch\n    |\n    v\nCompress the batch\n    |\n    v\nLocate the partition leader\n    |\n    v\nSend the Produce request\n    |\n    v\nHandle acknowledgement\n    |\n    v\nRetry if required\n<\/code><\/pre>\n\n\n\n
    Kafka producers are designed to work asynchronously. In normal usage, your application calls send()<\/code>, the record enters the producer pipeline, and a background I\/O process sends data to Kafka efficiently.<\/p>\n\n\n\n
    This asynchronous behavior is one of the main reasons Kafka producers can achieve high throughput.<\/p>\n\n\n\n
    \n\n\n\n
    3. Kafka Producer Record<\/h1>\n\n\n\n
    A producer sends a record<\/strong>.<\/p>\n\n\n\n
    A simplified producer record contains:<\/p>\n\n\n\n
    Producer Record\n\nTopic       required\nPartition   optional\nTimestamp   optional\nKey         optional\nValue       usually the business payload\nHeaders     optional\n<\/code><\/pre>\n\n\n\n
    Example business data:<\/p>\n\n\n\n
    {\n  \"vehicle_id\": \"CAR-101\",\n  \"speed\": 88,\n  \"battery\": 72\n}\n<\/code><\/pre>\n\n\n\n
    We might publish it as:<\/p>\n\n\n\n
    Topic = vehicle-telemetry\nKey   = CAR-101\nValue = {\"speed\":88,\"battery\":72}\n<\/code><\/pre>\n\n\n\n
    If you do not explicitly provide a partition, the producer’s partitioning logic decides where the record should go.<\/p>\n\n\n\n
    \n\n\n\n
    4. Key and Value<\/h1>\n\n\n\n
    These are two of the most important producer concepts.<\/p>\n\n\n\n
    4.1 Value<\/h2>\n\n\n\n
    The value<\/strong> usually contains the business information.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    {\n  \"speed\": 88,\n  \"battery\": 72\n}\n<\/code><\/pre>\n\n\n\n
    Think:<\/p>\n\n\n\n
    VALUE\n=\n\"What happened?\"\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    4.2 Key<\/h2>\n\n\n\n
    The key<\/strong> is commonly used to identify the entity that the event belongs to.<\/p>\n\n\n\n
    Example:<\/p>\n\n\n\n
    Key = CAR-101\n<\/code><\/pre>\n\n\n\n
    Think:<\/p>\n\n\n\n
    KEY\n=\n\"Who or what does this event belong to?\"\n<\/code><\/pre>\n\n\n\n
    Examples:<\/p>\n\n\n\n
    Vehicle telemetry\nKey = vehicle_id\n\nCustomer events\nKey = customer_id\n\nOrder events\nKey = order_id\n\nBank account events\nKey = account_id\n<\/code><\/pre>\n\n\n\n
    The key is much more important than simply giving a message an ID.<\/p>\n\n\n\n
    The key commonly influences which partition receives the record<\/strong>.<\/p>\n\n\n\n
    \n\n\n\n
    5. Why the Key Is So Important<\/h1>\n\n\n\n
    Suppose we have:<\/p>\n\n\n\n
    Topic: vehicle-telemetry\n\nPartition 0\nPartition 1\nPartition 2\n<\/code><\/pre>\n\n\n\n
    The producer sends:<\/p>\n\n\n\n
    Key = CAR-101\n<\/code><\/pre>\n\n\n\n
    Kafka’s partitioning logic may determine:<\/p>\n\n\n\n
    CAR-101\n   |\n   v\nPartition 2\n<\/code><\/pre>\n\n\n\n
    Future records using the same key normally map consistently according to the same partitioning strategy:<\/p>\n\n\n\n
    CAR-101 speed=50  -> P2\nCAR-101 speed=60  -> P2\nCAR-101 speed=75  -> P2\n<\/code><\/pre>\n\n\n\n
    Why does this matter?<\/p>\n\n\n\n
    Because Kafka ordering is fundamentally ordering inside a partition<\/strong>.<\/p>\n\n\n\n
    If all events for CAR-101<\/code> go to the same partition, Kafka can preserve the order in which those records are appended to that partition.<\/p>\n\n\n\n
    \n\n\n\n
    6. Key Design Is an Architecture Decision<\/h1>\n\n\n\n
    Imagine an order system:<\/p>\n\n\n\n
    OrderCreated\nOrderPaid\nOrderPacked\nOrderShipped\n<\/code><\/pre>\n\n\n\n
    If:<\/p>\n\n\n\n
    Key = order_id\n<\/code><\/pre>\n\n\n\n
    then all events for one order can remain together:<\/p>\n\n\n\n
    ORDER-500\n\nOrderCreated\n     |\n     v\nOrderPaid\n     |\n     v\nOrderPacked\n     |\n     v\nOrderShipped\n<\/code><\/pre>\n\n\n\n
    But a poor key can create a hot partition<\/strong>.<\/p>\n\n\n\n
    Imagine:<\/p>\n\n\n\n
    Key = country\n<\/code><\/pre>\n\n\n\n
    and 70% of your traffic uses:<\/p>\n\n\n\n
    India\n<\/code><\/pre>\n\n\n\n
    Your partition distribution might look like:<\/p>\n\n\n\n
    Partition 0    ########################\nPartition 1    ###\nPartition 2    ##\nPartition 3    ##\n<\/code><\/pre>\n\n\n\n
    One partition becomes extremely busy while others are lightly used.<\/p>\n\n\n\n
    This can overload:<\/p>\n\n\n\n
      \n
    • the partition leader<\/li>\n\n\n\n
    • the broker hosting that leader<\/li>\n\n\n\n
    • the consumers assigned to that partition<\/li>\n<\/ul>\n\n\n\n
      A good key must consider both:<\/p>\n\n\n\n
      Ordering requirements\n\nAND\n\nTraffic distribution\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      7. What If the Key Is Missing?<\/h1>\n\n\n\n
      A common simplified explanation is:<\/p>\n\n\n\n
      \n
      “If there is no key, Kafka uses round robin.”<\/p>\n<\/blockquote>\n\n\n\n
      That is not a complete description of modern producer behavior.<\/p>\n\n\n\n
      Modern Kafka producer implementations can use sticky\/adaptive behavior for records without keys so that records can remain on a partition long enough to create efficient batches.<\/p>\n\n\n\n
      A better beginner mental model is:<\/p>\n\n\n\n
      WITH KEY\n\nkey\n |\n v\npartitioning logic\n |\n v\nconsistent partition choice\n<\/code><\/pre>\n\n\n\n
      versus:<\/p>\n\n\n\n
      WITHOUT KEY\n\nproducer selects an available partition\n        |\n        v\ntries to build efficient batches\n        |\n        v\nmoves to another partition over time\n<\/code><\/pre>\n\n\n\n
      Do not assume that every individual no-key record automatically rotates:<\/p>\n\n\n\n
      P0 -> P1 -> P2 -> P0 -> P1 -> P2\n<\/code><\/pre>\n\n\n\n
      A strict round-robin partitioner can be configured explicitly, but it is not the best universal description of modern default behavior.<\/p>\n\n\n\n
      \n\n\n\n
      8. Can We Explicitly Select a Partition?<\/h1>\n\n\n\n
      Yes.<\/p>\n\n\n\n
      An application can explicitly specify a partition.<\/p>\n\n\n\n
      Example:<\/p>\n\n\n\n
      Topic     = orders\nPartition = 2\nKey       = ORDER-101\nValue     = ...\n<\/code><\/pre>\n\n\n\n
      Then the producer does not need to choose the partition automatically.<\/p>\n\n\n\n
      But be careful.<\/p>\n\n\n\n
      Manual partition selection means your application becomes responsible for understanding:<\/p>\n\n\n\n
      Partition count\nBroker distribution\nLoad balancing\nOrdering\nScaling\nFuture partition changes\n<\/code><\/pre>\n\n\n\n
      In many applications, a better design is:<\/p>\n\n\n\n
      good key strategy\n+\nKafka partitioning\n<\/code><\/pre>\n\n\n\n
      rather than hardcoding partition numbers.<\/p>\n\n\n\n
      \n\n\n\n
      9. Topic, Partition and Broker Relationship<\/h1>\n\n\n\n
      Students must understand this before producer tuning.<\/p>\n\n\n\n
      Suppose:<\/p>\n\n\n\n
      Topic: vehicle-telemetry\n\nPartitions:\nP0\nP1\nP2\n<\/code><\/pre>\n\n\n\n
      Kafka may distribute the partition leaders like:<\/p>\n\n\n\n
      Broker 1\n  P0 Leader\n\nBroker 2\n  P1 Leader\n\nBroker 3\n  P2 Leader\n<\/code><\/pre>\n\n\n\n
      If the producer selects:<\/p>\n\n\n\n
      Partition 2\n<\/code><\/pre>\n\n\n\n
      it must eventually send the record to:<\/p>\n\n\n\n
      P2 Leader\n=\nBroker 3\n<\/code><\/pre>\n\n\n\n
      So the flow is:<\/p>\n\n\n\n
      Producer\n   |\n   v\nTopic\n   |\n   v\nPartition selection\n   |\n   v\nPartition Leader\n   |\n   v\nBroker\n<\/code><\/pre>\n\n\n\n
      The producer does not<\/strong> randomly select any broker for the final write.<\/p>\n\n\n\n
      \n\n\n\n
      10. How Does the Producer Know Which Broker Is the Leader?<\/h1>\n\n\n\n
      The producer uses cluster metadata<\/strong>.<\/p>\n\n\n\n
      It starts with one or more bootstrap endpoints.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Producer Configuration\n\nbootstrap.servers\n       |\n       v\nInitial Kafka connection\n       |\n       v\nCluster metadata\n<\/code><\/pre>\n\n\n\n
      Metadata tells the producer things such as:<\/p>\n\n\n\n
      Topic: orders\n\nP0 Leader = Broker 1\nP1 Leader = Broker 3\nP2 Leader = Broker 2\n<\/code><\/pre>\n\n\n\n
      Important:<\/p>\n\n\n\n
      bootstrap server\n!=\nbroker that permanently receives every record\n<\/code><\/pre>\n\n\n\n
      Bootstrap servers are entry points that allow the client to discover the cluster.<\/p>\n\n\n\n
      After discovery, the producer communicates with the correct partition leaders.<\/p>\n\n\n\n
      \n\n\n\n
      11. Serialization<\/h1>\n\n\n\n
      Before Kafka can send your key and value over the network, they must be converted into bytes.<\/p>\n\n\n\n
      Example:<\/p>\n\n\n\n
      Application Object\n      |\n      v\nSerializer\n      |\n      v\nBytes\n<\/code><\/pre>\n\n\n\n
      Both the key and value can have serializers.<\/p>\n\n\n\n
      For example:<\/p>\n\n\n\n
      key.serializer=org.apache.kafka.common.serialization.StringSerializer\nvalue.serializer=org.apache.kafka.common.serialization.StringSerializer\n<\/code><\/pre>\n\n\n\n
      With Confluent environments, you may later use schema-aware serializers for:<\/p>\n\n\n\n
      Avro\nProtobuf\nJSON Schema\n<\/code><\/pre>\n\n\n\n
      Schema Registry is a separate major Kafka\/Confluent topic that we will cover later.<\/p>\n\n\n\n
      \n\n\n\n
      12. The Producer’s Internal Memory<\/h1>\n\n\n\n
      A simplified producer pipeline looks like:<\/p>\n\n\n\n
      Application Thread\n       |\n       v\n     send()\n       |\n       v\nSerialization\n       |\n       v\nPartition Selection\n       |\n       v\nProducer Buffer \/ Accumulator\n       |\n       v\nRecord Batches\n       |\n       v\nBackground Sender \/ I\/O\n       |\n       v\nKafka Broker\n<\/code><\/pre>\n\n\n\n
      The producer has memory used to buffer records that have not yet been transmitted.<\/p>\n\n\n\n
      A key producer setting is:<\/p>\n\n\n\n
      buffer.memory\n<\/code><\/pre>\n\n\n\n
      This controls approximately how much memory is available to buffer records waiting to be sent.<\/p>\n\n\n\n
      \n\n\n\n
      13. Why Buffer Records in Memory?<\/h1>\n\n\n\n
      Imagine sending 100,000 events.<\/p>\n\n\n\n
      Inefficient design:<\/p>\n\n\n\n
      Record 1 -> Network request\nRecord 2 -> Network request\nRecord 3 -> Network request\n...\nRecord 100000 -> Network request\n<\/code><\/pre>\n\n\n\n
      Huge overhead.<\/p>\n\n\n\n
      Kafka instead tries to do:<\/p>\n\n\n\n
      Record 1\nRecord 2\nRecord 3\nRecord 4\nRecord 5\n    |\n    v\nBatch\n    |\n    v\nEfficient Produce request\n<\/code><\/pre>\n\n\n\n
      Batching is one of Kafka producer’s most important performance mechanisms.<\/p>\n\n\n\n
      \n\n\n\n
      14. Batching Happens Per Partition<\/h1>\n\n\n\n
      This is critical.<\/p>\n\n\n\n
      Suppose the producer is sending records to:<\/p>\n\n\n\n
      P0\nP1\nP2\n<\/code><\/pre>\n\n\n\n
      The producer builds batches for destination partitions.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Producer Memory\n\nP0 Batch\n[A][B][C][D]\n\nP1 Batch\n[E][F]\n\nP2 Batch\n[G][H][I]\n<\/code><\/pre>\n\n\n\n
      A broker request can then contain batches for partitions led by that broker.<\/p>\n\n\n\n
      This is why partition selection and batching are connected<\/strong>.<\/p>\n\n\n\n
      \n\n\n\n
      15. batch.size<\/code><\/h1>\n\n\n\n
      batch.size<\/code> controls the target\/default producer batch size in bytes.<\/p>\n\n\n\n
      A conceptual example:<\/p>\n\n\n\n
      batch.size = 16 KB\n<\/code><\/pre>\n\n\n\n
      Records:<\/p>\n\n\n\n
      2 KB\n2 KB\n3 KB\n4 KB\n3 KB\n<\/code><\/pre>\n\n\n\n
      can accumulate into a larger batch.<\/p>\n\n\n\n
      Larger useful batches may improve:<\/p>\n\n\n\n
      Network efficiency\nRequest efficiency\nCompression efficiency\nThroughput\n<\/code><\/pre>\n\n\n\n
      But blindly setting a huge batch size is not automatically better.<\/p>\n\n\n\n
      Possible disadvantages:<\/p>\n\n\n\n
      Higher memory requirements\nMore unused batch allocation\nPotential additional waiting in some workloads\nNo benefit for very low traffic\n<\/code><\/pre>\n\n\n\n
      Always load-test.<\/p>\n\n\n\n
      \n\n\n\n
      16. linger.ms<\/code><\/h1>\n\n\n\n
      What if the batch is not full yet?<\/p>\n\n\n\n
      Should Kafka:<\/p>\n\n\n\n
      send immediately?\n<\/code><\/pre>\n\n\n\n
      or:<\/p>\n\n\n\n
      wait briefly for more records?\n<\/code><\/pre>\n\n\n\n
      That is where:<\/p>\n\n\n\n
      linger.ms\n<\/code><\/pre>\n\n\n\n
      comes in.<\/p>\n\n\n\n
      Example:<\/p>\n\n\n\n
      linger.ms = 5\n<\/code><\/pre>\n\n\n\n
      means the producer may allow a very small batching window rather than immediately sending every under-filled batch.<\/p>\n\n\n\n
      \n\n\n\n
      17. batch.size<\/code> and linger.ms<\/code> Work Together<\/h1>\n\n\n\n
      Think of a bus.<\/p>\n\n\n\n
      batch.size\n=\nHow much can the bus carry?\n<\/code><\/pre>\n\n\n\n
      linger.ms\n=\nHow long may the bus wait for more passengers?\n<\/code><\/pre>\n\n\n\n
      Simplified:<\/p>\n\n\n\n
      Records arrive\n     |\n     v\nAccumulator\n     |\n     +---- batch ready? ----> SEND\n     |\n     +---- linger expires? -> SEND\n<\/code><\/pre>\n\n\n\n
      This is one of the most important producer performance relationships.<\/p>\n\n\n\n
      \n\n\n\n
      18. Compression<\/h1>\n\n\n\n
      Kafka can compress record batches.<\/p>\n\n\n\n
      Common options include:<\/p>\n\n\n\n
      none\ngzip\nsnappy\nlz4\nzstd\n<\/code><\/pre>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Uncompressed Batch\n\n###########################\n\n        |\n        v\n\nCompression\n\n        |\n        v\n\n##########\n<\/code><\/pre>\n\n\n\n
      Potential benefits:<\/p>\n\n\n\n
      Less network traffic\nLess broker storage\nHigher effective throughput\nLower network cost\n<\/code><\/pre>\n\n\n\n
      Potential cost:<\/p>\n\n\n\n
      CPU used to compress and decompress\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      19. Which Compression Type Should We Use?<\/h1>\n\n\n\n
      Do not memorize:<\/p>\n\n\n\n
      \"X is always best.\"\n<\/code><\/pre>\n\n\n\n
      There is no universal winner.<\/p>\n\n\n\n
      Evaluate:<\/p>\n\n\n\n
      Compression ratio\nCPU cost\nLatency\nRecord characteristics\nProducer language\/client\nBroker workload\nNetwork cost\n<\/code><\/pre>\n\n\n\n
      A production engineer benchmarks the real workload.<\/p>\n\n\n\n
      For example, repeated JSON field names can often compress well.<\/p>\n\n\n\n
      \n\n\n\n
      20. Producer Sends a Produce Request<\/h1>\n\n\n\n
      Suppose our key maps to:<\/p>\n\n\n\n
      Topic     = vehicle-telemetry\nPartition = 2\nLeader    = Broker 3\n<\/code><\/pre>\n\n\n\n
      Then:<\/p>\n\n\n\n
      Producer\n   |\n   v\nProduceRequest\n   |\n   v\nBroker 3\n<\/code><\/pre>\n\n\n\n
      The partition leader receives the batch and appends the records to the partition log.<\/p>\n\n\n\n
      \n\n\n\n
      21. Who Assigns the Offset?<\/h1>\n\n\n\n
      Another common misunderstanding:<\/p>\n\n\n\n
      The producer does not normally choose the Kafka offset<\/strong>.<\/p>\n\n\n\n
      Suppose the partition currently contains:<\/p>\n\n\n\n
      Partition 2\n\nOffset 100\nOffset 101\nOffset 102\n<\/code><\/pre>\n\n\n\n
      The leader appends the new record:<\/p>\n\n\n\n
      Offset 103\n<\/code><\/pre>\n\n\n\n
      The producer can later receive metadata containing:<\/p>\n\n\n\n
      topic\npartition\noffset\n<\/code><\/pre>\n\n\n\n
      So:<\/p>\n\n\n\n
      Producer chooses\/supplies:\ntopic\nkey\/value\npossibly partition\n<\/code><\/pre>\n\n\n\n
      while:<\/p>\n\n\n\n
      Kafka partition leader determines:\nrecord's log position \/ offset\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      22. Replicas<\/h1>\n\n\n\n
      Suppose:<\/p>\n\n\n\n
      Replication Factor = 3\n<\/code><\/pre>\n\n\n\n
      Partition 2 may look like:<\/p>\n\n\n\n
      Broker 3 -> Leader\nBroker 1 -> Follower\nBroker 2 -> Follower\n<\/code><\/pre>\n\n\n\n
      The producer writes to:<\/p>\n\n\n\n
      Leader\n<\/code><\/pre>\n\n\n\n
      Followers replicate the leader’s data.<\/p>\n\n\n\n
      The producer does not<\/strong> normally send three separate copies directly to all replicas.<\/p>\n\n\n\n
      Think:<\/p>\n\n\n\n
      Producer\n   |\n   v\nLeader\n   |------> Follower\n   |\n   +------> Follower\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      23. Acknowledgements \u2014 acks<\/code><\/h1>\n\n\n\n
      One of the most important producer reliability settings is:<\/p>\n\n\n\n
      acks\n<\/code><\/pre>\n\n\n\n
      It controls when the producer considers the Produce request successfully acknowledged.<\/p>\n\n\n\n
      The major values are:<\/p>\n\n\n\n
      acks=0\nacks=1\nacks=all\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      24. acks=0<\/code> \u2014 Fire and Forget<\/h1>\n\n\n\n
      The correct phrase is:<\/p>\n\n\n\n
      fire and forget<\/strong><\/p>\n\n\n\n
      Flow:<\/p>\n\n\n\n
      Producer\n   |\n   v\nSend\n   |\n   v\nDoes not wait for broker acknowledgement\n<\/code><\/pre>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Producer:\n\"I sent it. I am not waiting to know whether Kafka accepted it.\"\n<\/code><\/pre>\n\n\n\n
      Advantage<\/h3>\n\n\n\n
      Very little acknowledgement overhead.<\/p>\n\n\n\n
      Major disadvantage<\/h3>\n\n\n\n
      The producer does not receive broker acknowledgement that the broker successfully accepted the record.<\/p>\n\n\n\n
      Use only where message loss is acceptable.<\/p>\n\n\n\n
      \n\n\n\n
      25. acks=1<\/code> \u2014 Leader Acknowledges<\/h1>\n\n\n\n
      Flow:<\/p>\n\n\n\n
      Producer\n   |\n   v\nLeader Broker\n   |\n   v\nLeader accepts\/appends\n   |\n   v\nACK\n<\/code><\/pre>\n\n\n\n
      Followers may still be catching up.<\/p>\n\n\n\n
      Possible failure:<\/p>\n\n\n\n
      Leader writes record\n      |\n      v\nACK sent\n      |\n      v\nLeader fails before a follower has safely replicated\n      |\n      v\nPossible record loss\n<\/code><\/pre>\n\n\n\n
      So acks=1<\/code> is stronger than acks=0<\/code>, but weaker than acks=all<\/code>.<\/p>\n\n\n\n
      \n\n\n\n
      26. acks=all<\/code> \u2014 Strongest Producer ACK Mode<\/h1>\n\n\n\n
      A common oversimplification is:<\/p>\n\n\n\n
      \n
      “acks=all<\/code> means fully replicated to every configured replica.”<\/p>\n<\/blockquote>\n\n\n\n
      A more accurate explanation is:<\/p>\n\n\n\n
      \n
      The partition leader waits for the acknowledgement conditions involving the current in-sync replicas.<\/p>\n<\/blockquote>\n\n\n\n
      This makes acks=all<\/code> the strongest normal producer acknowledgement mode.<\/p>\n\n\n\n
      However, it must be understood together with:<\/p>\n\n\n\n
      Replication Factor\nISR\nmin.insync.replicas\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      27. min.insync.replicas<\/code><\/h1>\n\n\n\n
      Now combine:<\/p>\n\n\n\n
      Replication Factor\n+\nISR\n+\nacks\n+\nmin.insync.replicas\n<\/code><\/pre>\n\n\n\n
      A common production-style example:<\/p>\n\n\n\n
      replication.factor = 3\nmin.insync.replicas = 2\nacks = all\n<\/code><\/pre>\n\n\n\n
      Healthy state:<\/p>\n\n\n\n
      Leader    - ISR\nFollower  - ISR\nFollower  - ISR\n<\/code><\/pre>\n\n\n\n
      One broker fails:<\/p>\n\n\n\n
      Leader    - ISR\nFollower  - ISR\nFollower  - unavailable\n<\/code><\/pre>\n\n\n\n
      There may still be enough in-sync replicas to satisfy the configured minimum.<\/p>\n\n\n\n
      But if:<\/p>\n\n\n\n
      ISR size < min.insync.replicas\n<\/code><\/pre>\n\n\n\n
      Kafka can reject strongly acknowledged writes.<\/p>\n\n\n\n
      That is not necessarily a bad thing.<\/p>\n\n\n\n
      Production principle:<\/p>\n\n\n\n
      Sometimes failing a write\nis safer than\naccepting a write with weaker durability.\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      28. Reliability vs Latency<\/h1>\n\n\n\n
      You can now see a major tradeoff.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      acks=0\n\nLess waiting\n|\nv\nLower acknowledgement latency\n|\nv\nWeaker reliability feedback\n<\/code><\/pre>\n\n\n\n
      versus:<\/p>\n\n\n\n
      acks=all\n\nStronger acknowledgement conditions\n|\nv\nStronger durability\n|\nv\nPotentially more acknowledgement latency\n<\/code><\/pre>\n\n\n\n
      Actual latency also depends on:<\/p>\n\n\n\n
      Network\nBroker load\nBatching\nCompression\nPartition leadership\nReplica health\nRequest queues\nQuotas\nCluster capacity\nApplication workload\n<\/code><\/pre>\n\n\n\n
      Never tune only one setting in isolation.<\/p>\n\n\n\n
      \n\n\n\n
      29. Retries<\/h1>\n\n\n\n
      Networks fail.<\/p>\n\n\n\n
      Brokers restart.<\/p>\n\n\n\n
      Partition leaders change.<\/p>\n\n\n\n
      Connections break temporarily.<\/p>\n\n\n\n
      Therefore Kafka producers can retry eligible transient failures.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Send\n |\n v\nTemporary failure\n |\n v\nRetry\n |\n v\nSuccess\n<\/code><\/pre>\n\n\n\n
      But retries create an important question.<\/p>\n\n\n\n
      \n\n\n\n
      30. The Duplicate Problem<\/h1>\n\n\n\n
      Imagine:<\/p>\n\n\n\n
      Producer sends Record A\n        |\n        v\nBroker stores A\n        |\n        v\nACK is lost on the network\n        |\n        v\nProducer thinks:\n\"Maybe it failed.\"\n        |\n        v\nProducer retries A\n<\/code><\/pre>\n\n\n\n
      Without protection:<\/p>\n\n\n\n
      A\nA\n<\/code><\/pre>\n\n\n\n
      could appear.<\/p>\n\n\n\n
      This is why idempotence is important.<\/p>\n\n\n\n
      \n\n\n\n
      31. Idempotent Producer<\/h1>\n\n\n\n
      Kafka’s idempotent producer protects against duplicate writes caused by producer retries within Kafka’s producer guarantees.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Send A\n  |\n  v\nBroker stores A\n  |\n  v\nACK lost\n  |\n  v\nRetry A\n  |\n  v\nKafka identifies retry\n  |\n  v\nDo not create another logical copy\n<\/code><\/pre>\n\n\n\n
      Important:<\/p>\n\n\n\n
      Idempotent Producer\n!=\nAutomatically exactly-once business processing\n<\/code><\/pre>\n\n\n\n
      Exactly-once behavior involving multiple Kafka writes, consumed offsets, transactions, external databases or other business side effects is a larger topic.<\/p>\n\n\n\n
      We will cover Kafka Transactions separately.<\/p>\n\n\n\n
      \n\n\n\n
      32. max.in.flight.requests.per.connection<\/code><\/h1>\n\n\n\n
      A producer can have multiple requests waiting for responses.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Request 1 ---------------->\nRequest 2 ---------------->\nRequest 3 ---------------->\n<\/code><\/pre>\n\n\n\n
      This improves throughput.<\/p>\n\n\n\n
      But historically, retries plus multiple in-flight requests could create ordering concerns when idempotence was disabled.<\/p>\n\n\n\n
      Therefore these concepts must be understood together:<\/p>\n\n\n\n
      Throughput\nRetries\nOrdering\nIdempotence\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      33. delivery.timeout.ms<\/code><\/h1>\n\n\n\n
      Instead of thinking only:<\/p>\n\n\n\n
      \"How many times should I retry?\"\n<\/code><\/pre>\n\n\n\n
      a better mental model is:<\/p>\n\n\n\n
      \"How long may this record continue trying before delivery is considered failed?\"\n<\/code><\/pre>\n\n\n\n
      That is the role of:<\/p>\n\n\n\n
      delivery.timeout.ms\n<\/code><\/pre>\n\n\n\n
      It bounds the overall delivery attempt window.<\/p>\n\n\n\n
      This can include:<\/p>\n\n\n\n
      time waiting before send\nbroker acknowledgement waiting\neligible retries\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      34. Producer Memory and Backpressure<\/h1>\n\n\n\n
      Suppose the application produces:<\/p>\n\n\n\n
      1,000,000 records\/sec\n<\/code><\/pre>\n\n\n\n
      but Kafka can currently accept:<\/p>\n\n\n\n
      300,000 records\/sec\n<\/code><\/pre>\n\n\n\n
      Records accumulate in producer memory.<\/p>\n\n\n\n
      Application\n   |||||||||||\n   vvvvvvvvvvv\nProducer Buffer\n########################\n   |||\n   vvv\nKafka\n<\/code><\/pre>\n\n\n\n
      Eventually the producer buffer may become full.<\/p>\n\n\n\n
      The application can then experience blocking and eventually send failures depending on configuration.<\/p>\n\n\n\n
      This is backpressure<\/strong>.<\/p>\n\n\n\n
      Do not “solve” it by blindly adding huge memory.<\/p>\n\n\n\n
      Ask:<\/p>\n\n\n\n
      Why is Kafka slower than the producer?\n\nBroker saturation?\nNetwork?\nQuota?\nHot partition?\nToo few partitions?\nBroker failover?\nLarge records?\nInsufficient cluster capacity?\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      35. buffer.memory<\/code><\/h1>\n\n\n\n
      buffer.memory<\/code> controls approximately how much producer memory is available for pending records.<\/p>\n\n\n\n
      Increasing it can help absorb short bursts.<\/p>\n\n\n\n
      But it does not create infinite throughput.<\/p>\n\n\n\n
      If your steady-state application rate is greater than Kafka’s sustainable rate:<\/p>\n\n\n\n
      more buffer\n=\nfailure happens later\n<\/code><\/pre>\n\n\n\n
      not:<\/p>\n\n\n\n
      problem solved\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      36. max.block.ms<\/code><\/h1>\n\n\n\n
      send()<\/code> is normally asynchronous, but application calls can still block in some cases, such as:<\/p>\n\n\n\n
      waiting for metadata\nwaiting for producer buffer space\n<\/code><\/pre>\n\n\n\n
      max.block.ms<\/code> limits how long the producer may block in these situations.<\/p>\n\n\n\n
      This matters for application responsiveness.<\/p>\n\n\n\n
      Example:<\/p>\n\n\n\n
      Web request\n   |\n   v\nproducer.send()\n   |\n   v\nbuffer completely exhausted\n   |\n   v\napplication thread waits\n<\/code><\/pre>\n\n\n\n
      Kafka producer performance can therefore affect your API latency even before the send ultimately succeeds or fails.<\/p>\n\n\n\n
      \n\n\n\n
      37. send()<\/code> Does Not Mean “Already Stored”<\/h1>\n\n\n\n
      This distinction is critical.<\/p>\n\n\n\n
      Application code:<\/p>\n\n\n\n
      producer.send(record);\n<\/code><\/pre>\n\n\n\n
      does not<\/strong> automatically mean:<\/p>\n\n\n\n
      \"The record is already safely stored in Kafka.\"\n<\/code><\/pre>\n\n\n\n
      It normally means that the record has entered the producer’s asynchronous delivery pipeline.<\/p>\n\n\n\n
      To observe the delivery result, use:<\/p>\n\n\n\n
      Future\n<\/code><\/pre>\n\n\n\n
      or:<\/p>\n\n\n\n
      Callback\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      38. Use Callbacks in Real Applications<\/h1>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      producer.send(record, (metadata, exception) -> {\n    if (exception != null) {\n        \/\/ handle\/report failure<\/em>\n    } else {\n        \/\/ success<\/em>\n        \/\/ metadata.topic()<\/em>\n        \/\/ metadata.partition()<\/em>\n        \/\/ metadata.offset()<\/em>\n    }\n});\n<\/code><\/pre>\n\n\n\n
      This provides visibility into:<\/p>\n\n\n\n
      Success\nFailure\nTopic\nPartition\nOffset\n<\/code><\/pre>\n\n\n\n
      Do not produce important events and completely ignore asynchronous failures.<\/p>\n\n\n\n
      \n\n\n\n
      39. flush()<\/code><\/h1>\n\n\n\n
      flush()<\/code> means:<\/p>\n\n\n\n
      \n
      Force currently buffered records to be made available for immediate sending and wait for their associated requests to complete.<\/p>\n<\/blockquote>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      Producer Buffer\n\nA\nB\nC\nD\n   |\n   v\nflush()\n   |\n   v\nsend pending records now\n   |\n   v\nwait for requests to complete\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      40. Should We Call flush()<\/code> After Every Record?<\/h1>\n\n\n\n
      Normally:<\/p>\n\n\n\n
      No.<\/strong><\/p>\n\n\n\n
      Bad high-throughput pattern:<\/p>\n\n\n\n
      send(A)\nflush()\n\nsend(B)\nflush()\n\nsend(C)\nflush()\n<\/code><\/pre>\n\n\n\n
      This destroys much of the benefit of:<\/p>\n\n\n\n
      asynchronous sending\nbatching\nrequest aggregation\n<\/code><\/pre>\n\n\n\n
      You effectively move toward:<\/p>\n\n\n\n
      send\nwait\n\nsend\nwait\n\nsend\nwait\n<\/code><\/pre>\n\n\n\n
      Use flush()<\/code> intentionally.<\/p>\n\n\n\n
      Typical uses:<\/p>\n\n\n\n
      controlled synchronization point\nsmall teaching\/demo application\nbefore a critical application boundary\nbefore shutdown when needed\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      41. flush()<\/code> vs close()<\/code><\/h1>\n\n\n\n
      They are not the same.<\/p>\n\n\n\n
      flush()<\/code><\/h2>\n\n\n\n
      send pending buffered records\nwait for requests\nproducer remains usable\n<\/code><\/pre>\n\n\n\n
      close()<\/code><\/h2>\n\n\n\n
      complete producer shutdown\nrelease resources\nbackground I\/O stops\nproducer should no longer be used\n<\/code><\/pre>\n\n\n\n
      A Kafka producer owns:<\/p>\n\n\n\n
      memory buffers\nnetwork connections\nbackground I\/O resources\n<\/code><\/pre>\n\n\n\n
      Always close it properly.<\/p>\n\n\n\n
      \n\n\n\n
      42. Complete Producer Journey<\/h1>\n\n\n\n
      Put everything together:<\/p>\n\n\n\n
      APPLICATION\n    |\n    v\nCreate business event\n    |\n    v\nProducerRecord\n    |\n    +--> Topic\n    +--> Key\n    +--> Value\n    +--> Headers\n    +--> Optional partition\/timestamp\n    |\n    v\nSerializer\n    |\n    v\nCluster Metadata\n    |\n    v\nPartition Selection\n    |\n    v\nProducer Buffer\n    |\n    v\nPer-Partition Batch\n    |\n    +--> batch.size\n    +--> linger.ms\n    |\n    v\nCompression\n    |\n    v\nBackground Sender\n    |\n    v\nNetwork\n    |\n    v\nPartition Leader Broker\n    |\n    v\nAppend to Partition Log\n    |\n    v\nOffset Assigned\n    |\n    v\nFollower Replication\n    |\n    v\nISR\n    |\n    v\nacks condition\n    |\n    v\nACK \/ Error\n    |\n    v\nRetry if eligible\n    |\n    v\nCallback \/ Future\n<\/code><\/pre>\n\n\n\n
      This is the producer lifecycle you should be able to explain in an interview and in a production incident.<\/p>\n\n\n\n
      \n\n\n\n
      43. Producer Performance Tuning \u2014 The Correct Approach<\/h1>\n\n\n\n
      Do not start tuning like this:<\/p>\n\n\n\n
      Increase batch.size!\nIncrease memory!\nDecrease acks!\nAdd partitions!\n<\/code><\/pre>\n\n\n\n
      Instead:<\/p>\n\n\n\n
      1. Define requirement\n2. Measure baseline\n3. Find bottleneck\n4. Change one variable\n5. Load test\n6. Measure again\n7. Check reliability impact\n<\/code><\/pre>\n\n\n\n
      Performance tuning without measurements is guessing.<\/p>\n\n\n\n
      \n\n\n\n
      44. Goal 1 \u2014 Optimize for High Throughput<\/h1>\n\n\n\n
      If your goal is:<\/p>\n\n\n\n
      Maximum events\/sec\nMaximum MB\/sec\nEfficient broker\/network usage\n<\/code><\/pre>\n\n\n\n
      consider:<\/p>\n\n\n\n
      Increase useful batching\n     |\n     v\nappropriate batch.size\n\nAllow batches to fill\n     |\n     v\nappropriate linger.ms\n\nReduce bytes transferred\n     |\n     v\ncompression\n\nDistribute traffic\n     |\n     v\nenough well-balanced partitions\n\nAvoid per-message blocking\n     |\n     v\nasynchronous sends\n\nHandle bursts\n     |\n     v\nappropriate producer memory\n<\/code><\/pre>\n\n\n\n
      Also monitor:<\/p>\n\n\n\n
      broker throttling\nhot partitions\nrequest latency\nerror rate\nretry rate\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      45. High-Throughput Mental Model<\/h1>\n\n\n\n
      Less efficient:<\/p>\n\n\n\n
      A -> Kafka\nB -> Kafka\nC -> Kafka\nD -> Kafka\n<\/code><\/pre>\n\n\n\n
      More efficient:<\/p>\n\n\n\n
      A\nB\nC\nD\nE\nF\n |\n v\nBatch\n |\n v\nCompress\n |\n v\nKafka\n<\/code><\/pre>\n\n\n\n
      Kafka achieves much of its producer efficiency by moving batches rather than treating every record as an isolated network transaction.<\/p>\n\n\n\n
      \n\n\n\n
      46. Goal 2 \u2014 Minimize Latency<\/h1>\n\n\n\n
      Suppose the requirement is:<\/p>\n\n\n\n
      Event must reach Kafka as quickly as practical.\n<\/code><\/pre>\n\n\n\n
      Consider:<\/p>\n\n\n\n
      Smaller batching wait\nLower linger.ms\nAvoid unnecessary application blocking\nHealthy nearby Kafka infrastructure\nCorrect partition distribution\nLow broker load\nFast serialization\nAppropriate compression choice\n<\/code><\/pre>\n\n\n\n
      But do not automatically set every batching value to zero.<\/p>\n\n\n\n
      Under real load, a tiny batching window may improve overall system efficiency enough to produce excellent practical latency.<\/p>\n\n\n\n
      Always benchmark your real workload.<\/p>\n\n\n\n
      \n\n\n\n
      47. Goal 3 \u2014 Maximum Reliability<\/h1>\n\n\n\n
      For events where loss is unacceptable, think about the complete chain:<\/p>\n\n\n\n
      Producer\n   |\n   v\nacks=all\n   |\n   v\nHealthy ISR\n   |\n   v\nAppropriate min.insync.replicas\n   |\n   v\nReplication factor\n   |\n   v\nIdempotence\n   |\n   v\nRetries\n   |\n   v\nError handling\n   |\n   v\nMonitoring\n<\/code><\/pre>\n\n\n\n
      A common production-oriented pattern is:<\/p>\n\n\n\n
      Replication Factor = 3\nmin.insync.replicas = 2\nacks = all\nenable.idempotence = true\n<\/code><\/pre>\n\n\n\n
      This is a useful starting concept, not a universal configuration that should be copied blindly.<\/p>\n\n\n\n
      \n\n\n\n
      48. The Throughput \/ Latency \/ Reliability Triangle<\/h1>\n\n\n\n
      A Kafka producer balances:<\/p>\n\n\n\n
                     THROUGHPUT\n                  \/\\\n                 \/  \\\n                \/    \\\n               \/      \\\n              \/        \\\n        LATENCY ------ RELIABILITY\n<\/code><\/pre>\n\n\n\n
      Examples:<\/p>\n\n\n\n
      More batching\n-> often better throughput\n-> may add waiting time\n\nMore compression\n-> less network\/storage\n-> more CPU\n\nacks=all\n-> stronger durability\n-> stronger acknowledgement conditions\n\nacks=0\n-> less acknowledgement waiting\n-> much weaker delivery feedback\n\nMore producer memory\n-> more burst absorption\n-> does not increase broker capacity\n<\/code><\/pre>\n\n\n\n
      Master-level Kafka is about understanding these tradeoffs.<\/p>\n\n\n\n
      \n\n\n\n
      49. Important Producer Tuning Parameters<\/h1>\n\n\n\n
      Parameter<\/th>Main purpose<\/th>Main tradeoff<\/th><\/tr><\/thead>
      batch.size<\/code><\/td>Build larger record batches<\/td>Memory \/ workload dependent<\/td><\/tr>
      linger.ms<\/code><\/td>Give batches time to fill<\/td>Throughput vs added waiting<\/td><\/tr>
      compression.type<\/code><\/td>Reduce bytes<\/td>CPU vs network\/storage<\/td><\/tr>
      acks<\/code><\/td>Delivery acknowledgement strength<\/td>Reliability vs ACK latency<\/td><\/tr>
      enable.idempotence<\/code><\/td>Prevent duplicate retry writes<\/td>Requires compatible reliability settings<\/td><\/tr>
      retries<\/code><\/td>Recover from transient failures<\/td>Longer time before permanent failure<\/td><\/tr>
      delivery.timeout.ms<\/code><\/td>Bound delivery attempts<\/td>Reliability window vs failure speed<\/td><\/tr>
      buffer.memory<\/code><\/td>Buffer pending records<\/td>Memory usage<\/td><\/tr>
      max.block.ms<\/code><\/td>Limit blocking on metadata\/buffer<\/td>Application responsiveness<\/td><\/tr>
      max.in.flight.requests.per.connection<\/code><\/td>Parallel outstanding requests<\/td>Throughput\/order interaction<\/td><\/tr>
      max.request.size<\/code><\/td>Limit producer request size<\/td>Large-message constraints<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
      Do not tune these independently without understanding their interactions.<\/p>\n\n\n\n
      \n\n\n\n
      50. Example Balanced Producer Profile<\/h1>\n\n\n\n
      For teaching purposes, a balanced producer might start conceptually with:<\/p>\n\n\n\n
      acks=all\nenable.idempotence=true\n\ncompression.type=zstd\n\nlinger.ms=5\nbatch.size=65536\n\ndelivery.timeout.ms=120000\n<\/code><\/pre>\n\n\n\n
      This is an example starting profile<\/strong>, not a production recipe.<\/p>\n\n\n\n
      The correct configuration depends on:<\/p>\n\n\n\n
      event size\nevent rate\nlatency SLO\ncluster location\nnumber of partitions\nnetwork bandwidth\nCPU\ncompression ratio\nbroker capacity\nfailure requirements\nclient library\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      51. Confluent Kafka Cluster Connection<\/h1>\n\n\n\n
      For Confluent Cloud-style training, the producer also needs security and connection settings.<\/p>\n\n\n\n
      Conceptually:<\/p>\n\n\n\n
      bootstrap.servers=<CONFLUENT_BOOTSTRAP_SERVER>\n\nsecurity.protocol=SASL_SSL\nsasl.mechanism=PLAIN\n\nsasl.jaas.config=<API_KEY_AND_SECRET>\n\nkey.serializer=...\nvalue.serializer=...\n<\/code><\/pre>\n\n\n\n
      Keep secrets outside application source code wherever possible.<\/p>\n\n\n\n
      After authentication, the normal producer flow remains:<\/p>\n\n\n\n
      Discover metadata\n      |\n      v\nSelect partition\n      |\n      v\nFind partition leader\n      |\n      v\nProduce batch\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      52. Practical Java Example<\/h1>\n\n\n\n
      Simplified teaching example:<\/p>\n\n\n\n
      Properties props = new Properties();\n\nprops.put(\"bootstrap.servers\", \"<BOOTSTRAP_SERVER>\");\nprops.put(\"security.protocol\", \"SASL_SSL\");\nprops.put(\"sasl.mechanism\", \"PLAIN\");\n\nprops.put(\n    \"key.serializer\",\n    \"org.apache.kafka.common.serialization.StringSerializer\"\n);\n\nprops.put(\n    \"value.serializer\",\n    \"org.apache.kafka.common.serialization.StringSerializer\"\n);\n\nprops.put(\"acks\", \"all\");\nprops.put(\"enable.idempotence\", \"true\");\nprops.put(\"compression.type\", \"zstd\");\nprops.put(\"linger.ms\", \"5\");\n\nKafkaProducer<String, String> producer =\n    new KafkaProducer<>(props);\n\nProducerRecord<String, String> record =\n    new ProducerRecord<>(\n        \"vehicle-telemetry\",\n        \"CAR-101\",\n        \"{\\\"speed\\\":88}\"\n    );\n\nproducer.send(record, (metadata, exception) -> {\n\n    if (exception != null) {\n        exception.printStackTrace();\n        return;\n    }\n\n    System.out.println(\n        \"topic=\" + metadata.topic()\n        + \" partition=\" + metadata.partition()\n        + \" offset=\" + metadata.offset()\n    );\n});\n\nproducer.flush();\nproducer.close();\n<\/code><\/pre>\n\n\n\n
      Why call flush()<\/code> here?<\/p>\n\n\n\n
      Because this is a small teaching program and we want an explicit point at which all pending sends have completed before shutdown.<\/p>\n\n\n\n
      In a high-throughput long-running service, do not call flush()<\/code> after every record.<\/p>\n\n\n\n
      \n\n\n\n
      53. Lab 1 \u2014 Keyed Events<\/h1>\n\n\n\n
      Create a topic with multiple partitions.<\/p>\n\n\n\n
      For example:<\/p>\n\n\n\n
      vehicle-telemetry\n\nP0\nP1\nP2\nP3\n<\/code><\/pre>\n\n\n\n
      Produce:<\/p>\n\n\n\n
      CAR-101 event 1\nCAR-101 event 2\nCAR-101 event 3\n<\/code><\/pre>\n\n\n\n
      Use the callback to print:<\/p>\n\n\n\n
      partition\noffset\n<\/code><\/pre>\n\n\n\n
      Observe whether records for the same key consistently go to the same partition.<\/p>\n\n\n\n
      Then try:<\/p>\n\n\n\n
      CAR-101\nCAR-102\nCAR-103\nCAR-104\n<\/code><\/pre>\n\n\n\n
      Compare the partition distribution.<\/p>\n\n\n\n
      \n\n\n\n
      54. Lab 2 \u2014 No-Key Events<\/h1>\n\n\n\n
      Produce records without a key:<\/p>\n\n\n\n
      null -> event 1\nnull -> event 2\nnull -> event 3\n...\n<\/code><\/pre>\n\n\n\n
      Print the assigned partitions.<\/p>\n\n\n\n
      Do not assume that modern no-key behavior is a naive strict per-record round robin.<\/p>\n\n\n\n
      Observe how batches and partition selection interact.<\/p>\n\n\n\n
      \n\n\n\n
      55. Lab 3 \u2014 Batching Experiment<\/h1>\n\n\n\n
      Run a load test with:<\/p>\n\n\n\n
      linger.ms = 0\n<\/code><\/pre>\n\n\n\n
      Record:<\/p>\n\n\n\n
      throughput\nrequest rate\nlatency\naverage batch size\n<\/code><\/pre>\n\n\n\n
      Then test:<\/p>\n\n\n\n
      linger.ms = 5\n<\/code><\/pre>\n\n\n\n
      Then:<\/p>\n\n\n\n
      linger.ms = 20\n<\/code><\/pre>\n\n\n\n
      Do not ask only:<\/p>\n\n\n\n
      \"Which value is best?\"\n<\/code><\/pre>\n\n\n\n
      Ask:<\/p>\n\n\n\n
      \"What changed and why?\"\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      56. Lab 4 \u2014 Compression Experiment<\/h1>\n\n\n\n
      Produce the same data using:<\/p>\n\n\n\n
      compression.type=none\n<\/code><\/pre>\n\n\n\n
      then:<\/p>\n\n\n\n
      compression.type=lz4\n<\/code><\/pre>\n\n\n\n
      then:<\/p>\n\n\n\n
      compression.type=zstd\n<\/code><\/pre>\n\n\n\n
      Measure:<\/p>\n\n\n\n
      producer CPU\nbytes sent\nthroughput\nlatency\ncompression ratio\n<\/code><\/pre>\n\n\n\n
      The correct choice depends on your workload.<\/p>\n\n\n\n
      \n\n\n\n
      57. Lab 5 \u2014 acks<\/code> Experiment<\/h1>\n\n\n\n
      In a safe training cluster, compare:<\/p>\n\n\n\n
      acks=0\nacks=1\nacks=all\n<\/code><\/pre>\n\n\n\n
      Do not only measure speed.<\/p>\n\n\n\n
      Record:<\/p>\n\n\n\n
      Producer latency\nErrors\nBehavior during broker\/leader disruption\nDurability guarantees\n<\/code><\/pre>\n\n\n\n
      The objective is to understand that configuration is about failure behavior<\/strong>, not only benchmark numbers.<\/p>\n\n\n\n
      \n\n\n\n
      58. Lab 6 \u2014 flush()<\/code> Experiment<\/h1>\n\n\n\n
      Test:<\/p>\n\n\n\n
      for each event:\n    send()\n    flush()\n<\/code><\/pre>\n\n\n\n
      Then compare with:<\/p>\n\n\n\n
      send many events asynchronously\nflush once at the end\n<\/code><\/pre>\n\n\n\n
      Compare throughput and latency.<\/p>\n\n\n\n
      This demonstrates why excessive flush()<\/code> calls can remove much of the benefit of producer batching.<\/p>\n\n\n\n
      \n\n\n\n
      59. Producer Metrics to Monitor<\/h1>\n\n\n\n
      A production producer should never be a black box.<\/p>\n\n\n\n
      Important metrics include:<\/p>\n\n\n\n
      record-send-rate\nrecord-error-rate\nrecord-retry-rate\nrequest-latency-avg\nrequest-latency-max\nbatch-size-avg\nrecords-per-request-avg\ncompression-rate-avg\nrecord-queue-time-avg\nbuffer-available-bytes\nwaiting-threads\nrequests-in-flight\nproduce-throttle-time-avg\n<\/code><\/pre>\n\n\n\n
      These help you understand whether the producer is:<\/p>\n\n\n\n
      healthy\nslow\nbuffer-constrained\nretrying\nthrottled\npoorly batched\nexperiencing broker\/network latency\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      60. How to Read Producer Metrics<\/h1>\n\n\n\n
      High record-error-rate<\/code><\/h2>\n\n\n\n
      Investigate:<\/p>\n\n\n\n
      authentication\nauthorization\nbroker availability\ninvalid records\nmessage size\nISR health\ntimeouts\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      High record-retry-rate<\/code><\/h2>\n\n\n\n
      Possible causes:<\/p>\n\n\n\n
      network instability\nleader elections\nbroker overload\ntransient broker errors\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      High request-latency-avg<\/code><\/h2>\n\n\n\n
      Possible causes:<\/p>\n\n\n\n
      network latency\nbroker overload\nreplication delay\nthrottling\ncross-region traffic\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      Very low batch-size-avg<\/code><\/h2>\n\n\n\n
      Possible interpretation:<\/p>\n\n\n\n
      batches are not filling\n<\/code><\/pre>\n\n\n\n
      Possible reasons:<\/p>\n\n\n\n
      low traffic\nvery low linger\ntoo many destination partitions for the traffic level\napplication send pattern\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      Low buffer-available-bytes<\/code><\/h2>\n\n\n\n
      Producer buffer pressure is increasing.<\/p>\n\n\n\n
      Ask:<\/p>\n\n\n\n
      Is the application producing events\nfaster than Kafka can accept them?\n<\/code><\/pre>\n\n\n\n
      \n\n\n\n
      61. Producer Troubleshooting Flow<\/h1>\n\n\n\n
      When producers become slow:<\/p>\n\n\n\n
      START\n  |\n  v\nAre sends failing?\n  |\n  v\nCheck error and retry metrics\n  |\n  v\nCheck request latency\n  |\n  v\nCheck broker throttling\n  |\n  v\nCheck partition distribution\n  |\n  v\nCheck hot partitions\n  |\n  v\nCheck batch efficiency\n  |\n  v\nCheck compression\n  |\n  v\nCheck producer buffer pressure\n  |\n  v\nCheck network latency\n  |\n  v\nCheck broker \/ ISR health\n  |\n  v\nChange one thing\n  |\n  v\nLoad test again\n<\/code><\/pre>\n\n\n\n
      Do not change ten settings simultaneously.<\/p>\n\n\n\n
      Otherwise, you will not know which change helped or hurt.<\/p>\n\n\n\n
      \n\n\n\n
      62. Common Producer Mistakes<\/h1>\n\n\n\n
      Mistake 1 \u2014 No meaningful key<\/h2>\n\n\n\n
      Later the team says:<\/p>\n\n\n\n
      \"We need strict ordering for each customer.\"\n<\/code><\/pre>\n\n\n\n
      Key design should be considered early.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 2 \u2014 Bad low-cardinality key<\/h2>\n\n\n\n
      Examples:<\/p>\n\n\n\n
      country\nregion\nevent_type\n<\/code><\/pre>\n\n\n\n
      may create very uneven traffic.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 3 \u2014 Calling flush()<\/code> for every event<\/h2>\n\n\n\n
      This destroys useful asynchronous batching.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 4 \u2014 Assuming send()<\/code> means successful durable write<\/h2>\n\n\n\n
      It does not.<\/p>\n\n\n\n
      Observe callback\/Future results.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 5 \u2014 Using acks=0<\/code> for critical business events<\/h2>\n\n\n\n
      You lose normal broker acknowledgement of delivery.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 6 \u2014 Setting huge buffers to hide slow Kafka<\/h2>\n\n\n\n
      This delays the symptom rather than fixing sustainable throughput.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 7 \u2014 Increasing partitions without thinking about keys<\/h2>\n\n\n\n
      You can still have hot partitions.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 8 \u2014 Enabling retries without understanding idempotence<\/h2>\n\n\n\n
      Retry semantics matter.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 9 \u2014 Optimizing throughput while ignoring reliability<\/h2>\n\n\n\n
      A benchmark that loses critical records under failure is not a successful production design.<\/p>\n\n\n\n
      \n\n\n\n
      Mistake 10 \u2014 Copying configuration blindly<\/h2>\n\n\n\n
      Always benchmark your workload and understand your client version.<\/p>\n\n\n\n
      \n\n\n\n
      63. Production Producer Checklist<\/h1>\n\n\n\n
      Before calling a producer production-ready, verify:<\/p>\n\n\n\n
        \n
      • [ ] Correct topic selected<\/li>\n\n\n\n
      • [ ] Key strategy intentionally designed<\/li>\n\n\n\n
      • [ ] Partition distribution tested<\/li>\n\n\n\n
      • [ ] Serialization format selected<\/li>\n\n\n\n
      • [ ] Schema evolution strategy defined where required<\/li>\n\n\n\n
      • [ ]\u00a0acks<\/code>\u00a0intentionally selected<\/li>\n\n\n\n
      • [ ] Replication \/ ISR policy understood<\/li>\n\n\n\n
      • [ ] Idempotence behavior understood<\/li>\n\n\n\n
      • [ ] Retry behavior tested<\/li>\n\n\n\n
      • [ ] Delivery timeout understood<\/li>\n\n\n\n
      • [ ] Batching load-tested<\/li>\n\n\n\n
      • [ ]\u00a0linger.ms<\/code>\u00a0load-tested<\/li>\n\n\n\n
      • [ ]\u00a0batch.size<\/code>\u00a0load-tested<\/li>\n\n\n\n
      • [ ] Compression benchmarked<\/li>\n\n\n\n
      • [ ] Producer memory monitored<\/li>\n\n\n\n
      • [ ] Backpressure behavior tested<\/li>\n\n\n\n
      • [ ] Callback\/errors handled<\/li>\n\n\n\n
      • [ ] Producer properly closed on shutdown<\/li>\n\n\n\n
      • [ ]\u00a0flush()<\/code>\u00a0used only where appropriate<\/li>\n\n\n\n
      • [ ] Hot partitions monitored<\/li>\n\n\n\n
      • [ ] Producer latency monitored<\/li>\n\n\n\n
      • [ ] Error and retry rates monitored<\/li>\n\n\n\n
      • [ ] Failure testing performed<\/li>\n\n\n\n
      • [ ] Broker failure tested<\/li>\n\n\n\n
      • [ ] Network interruption tested<\/li>\n\n\n\n
      • [ ] Application SLOs documented<\/li>\n<\/ul>\n\n\n\n
        \n\n\n\n
        64. Interview-Level Questions<\/h1>\n\n\n\n
        What is a Kafka producer?<\/h2>\n\n\n\n
        A Kafka client that publishes records to Kafka topics.<\/p>\n\n\n\n
        What does a producer record contain?<\/h2>\n\n\n\n
        A topic, key\/value and optionally headers, timestamp and an explicitly selected partition.<\/p>\n\n\n\n
        Is the key mandatory?<\/h2>\n\n\n\n
        No. But a key is extremely important when partition affinity and per-entity ordering matter.<\/p>\n\n\n\n
        Does no key always mean strict round robin?<\/h2>\n\n\n\n
        No. Modern producers can use sticky\/adaptive behavior to improve batching. Strict round robin can be explicitly configured.<\/p>\n\n\n\n
        Who assigns the offset?<\/h2>\n\n\n\n
        The Kafka partition leader assigns the log position when the record is appended.<\/p>\n\n\n\n
        Does the producer write directly to every replica?<\/h2>\n\n\n\n
        No. The producer writes to the partition leader. Followers replicate from the leader.<\/p>\n\n\n\n
        What is batch.size<\/code>?<\/h2>\n\n\n\n
        A producer setting that influences the target\/default amount of data grouped into a partition batch.<\/p>\n\n\n\n
        What is linger.ms<\/code>?<\/h2>\n\n\n\n
        A small upper waiting window that gives batches time to fill before being sent.<\/p>\n\n\n\n
        What does acks=0<\/code> mean?<\/h2>\n\n\n\n
        The producer does not wait for broker acknowledgement.<\/p>\n\n\n\n
        What does acks=1<\/code> mean?<\/h2>\n\n\n\n
        The partition leader acknowledges without waiting for the strongest ISR acknowledgement condition.<\/p>\n\n\n\n
        What does acks=all<\/code> mean?<\/h2>\n\n\n\n
        The leader waits for the current in-sync replica acknowledgement requirements. This is Kafka’s strongest normal producer acknowledgement mode.<\/p>\n\n\n\n
        Why use idempotence?<\/h2>\n\n\n\n
        To prevent duplicate Kafka writes caused by retries within producer idempotence guarantees.<\/p>\n\n\n\n
        What does flush()<\/code> do?<\/h2>\n\n\n\n
        It forces currently buffered records to become immediately eligible for sending and waits for their associated requests to complete.<\/p>\n\n\n\n
        \n\n\n\n
        65. Master Mental Model<\/h1>\n\n\n\n
        A beginner thinks:<\/p>\n\n\n\n
        Producer sends a message to Kafka.\n<\/code><\/pre>\n\n\n\n
        A Kafka engineer thinks:<\/p>\n\n\n\n
        Business Event\n     |\n     v\nProducerRecord\n     |\n     v\nKey + Value + Headers\n     |\n     v\nSerialization\n     |\n     v\nMetadata\n     |\n     v\nPartition Strategy\n     |\n     v\nProducer Buffer\n     |\n     v\nPer-Partition Batch\n     |\n     +--> batch.size\n     +--> linger.ms\n     |\n     v\nCompression\n     |\n     v\nBackground Sender\n     |\n     v\nPartition Leader\n     |\n     v\nBroker Append\n     |\n     v\nOffset Assignment\n     |\n     v\nReplication\n     |\n     v\nISR\n     |\n     v\nacks\n     |\n     v\nRetry \/ Idempotence\n     |\n     v\nDelivery Result\n<\/code><\/pre>\n\n\n\n
        A production Kafka engineer asks:<\/p>\n\n\n\n
        How does every step affect:\n\nPERFORMANCE?\nLATENCY?\nRELIABILITY?\nAVAILABILITY?\nCOST?\n<\/code><\/pre>\n\n\n\n
        That is the level of thinking we want from the Kafka Master Tutorials series.<\/p>\n\n\n\n
        \n\n\n\n
        66. Final Takeaways<\/h1>\n\n\n\n
        Remember these ten ideas:<\/p>\n\n\n\n
          \n
        1. A producer is much more than a network sender.<\/li>\n\n\n\n
        2. Keys are fundamental to partitioning and ordering design.<\/li>\n\n\n\n
        3. Modern no-key partitioning should not simply be taught as strict round robin.<\/li>\n\n\n\n
        4. Kafka batches records primarily for efficiency.<\/li>\n\n\n\n
        5. batch.size<\/code>\u00a0and\u00a0linger.ms<\/code>\u00a0work together.<\/li>\n\n\n\n
        6. Compression trades CPU for potentially lower network and storage usage.<\/li>\n\n\n\n
        7. The producer writes to the\u00a0partition leader<\/strong>, not directly to every replica.<\/li>\n\n\n\n
        8. acks<\/code>, ISR and\u00a0min.insync.replicas<\/code>\u00a0work together to define durability.<\/li>\n\n\n\n
        9. Retries should be understood together with idempotence and delivery timeout.<\/li>\n\n\n\n
        10. Producer performance tuning must be driven by measurements, not copied configuration.<\/li>\n<\/ol>\n\n\n\n
          \n\n\n\n
          67. Suggested Next Producer Tutorials<\/h1>\n\n\n\n
          After this producer foundation, the natural deep dives are:<\/p>\n\n\n\n
          Producer Internals\n        |\n        v\nPartitioning Strategies\n        |\n        v\nProducer Reliability & Delivery Semantics\n        |\n        v\nProducer Performance Tuning Lab\n        |\n        v\nProducer Errors, Retries & Idempotence\n        |\n        v\nKafka Transactions\n        |\n        v\nProduction Producer Monitoring\n<\/code><\/pre>\n\n\n\n
          That path takes students from producer fundamentals to production-grade producer mastery.<\/p>\n","protected":false},"excerpt":{"rendered":"
          From send() to Broker ACK: Keys, Partitions, Batching, Retries, Reliability, Latency and Performance Tuning Audience: Students and freshers with no prior Kafka experienceGoal: Build from producer fundamentals to production-grade Kafka producer… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1148","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1148","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1148"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1148\/revisions"}],"predecessor-version":[{"id":1149,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1148\/revisions\/1149"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1148"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1148"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}