From send() to Broker ACK: Keys, Partitions, Batching, Retries, Reliability, Latency and Performance Tuning

Audience: Students and freshers with no prior Kafka experience
Goal: Build from producer fundamentals to production-grade Kafka producer design and tuning
Training environment: Confluent Kafka Cluster

---

1. Learning Objectives

By the end of this tutorial, you should understand:

• What a Kafka Producer is
• What a Kafka record contains
• Key vs value
• Which record fields are optional
• How a producer discovers Kafka brokers
• How a producer selects a topic partition
• What happens when a key is present
• What happens when a key is absent
• Why “round robin” is not a complete description of modern default no-key partitioning
• What batching does
• What batch.size does
• What linger.ms does
• What compression does
• How producer memory is used
• What retries do
• Why idempotence matters
• What acks=0, acks=1, and acks=all mean
• How replicas and ISR affect producer reliability
• Who assigns Kafka offsets
• What flush() does
• How to optimize a producer for throughput
• How to optimize a producer for low latency
• How to build for production reliability
• Which producer metrics should be monitored

---

2. What Is a Kafka Producer?

A Kafka Producer is a client application that publishes records to Kafka topics.

The simplest mental model is:

Application
    |
    v
Kafka Producer
    |
    v
Kafka Topic

For example:

Vehicle
   |
   v
Telematics Service
   |
   v
Kafka Producer
   |
   v
Topic: vehicle-telemetry

The producer does much more than simply “send a message.”

Inside the producer, Kafka may need to:

Create a record
    |
    v
Serialize it
    |
    v
Obtain cluster metadata
    |
    v
Select a partition
    |
    v
Place it in memory
    |
    v
Build a batch
    |
    v
Compress the batch
    |
    v
Locate the partition leader
    |
    v
Send the Produce request
    |
    v
Handle acknowledgement
    |
    v
Retry if required

Kafka producers are designed to work asynchronously. In normal usage, your application calls send(), the record enters the producer pipeline, and a background I/O process sends data to Kafka efficiently.

This asynchronous behavior is one of the main reasons Kafka producers can achieve high throughput.

---

3. Kafka Producer Record

A producer sends a record.

A simplified producer record contains:

Producer Record

Topic       required
Partition   optional
Timestamp   optional
Key         optional
Value       usually the business payload
Headers     optional

Example business data:

{
  "vehicle_id": "CAR-101",
  "speed": 88,
  "battery": 72
}

We might publish it as:

Topic = vehicle-telemetry
Key   = CAR-101
Value = {"speed":88,"battery":72}

If you do not explicitly provide a partition, the producer’s partitioning logic decides where the record should go.

---

4. Key and Value

These are two of the most important producer concepts.

4.1 Value

The value usually contains the business information.

Example:

{
  "speed": 88,
  "battery": 72
}

Think:

VALUE
=
"What happened?"

---

4.2 Key

The key is commonly used to identify the entity that the event belongs to.

Example:

Key = CAR-101

Think:

KEY
=
"Who or what does this event belong to?"

Examples:

Vehicle telemetry
Key = vehicle_id

Customer events
Key = customer_id

Order events
Key = order_id

Bank account events
Key = account_id

The key is much more important than simply giving a message an ID.

The key commonly influences which partition receives the record.

---

5. Why the Key Is So Important

Suppose we have:

Topic: vehicle-telemetry

Partition 0
Partition 1
Partition 2

The producer sends:

Key = CAR-101

Kafka’s partitioning logic may determine:

CAR-101
   |
   v
Partition 2

Future records using the same key normally map consistently according to the same partitioning strategy:

CAR-101 speed=50  -> P2
CAR-101 speed=60  -> P2
CAR-101 speed=75  -> P2

Why does this matter?

Because Kafka ordering is fundamentally ordering inside a partition.

If all events for CAR-101 go to the same partition, Kafka can preserve the order in which those records are appended to that partition.

---

6. Key Design Is an Architecture Decision

Imagine an order system:

OrderCreated
OrderPaid
OrderPacked
OrderShipped

If:

Key = order_id

then all events for one order can remain together:

ORDER-500

OrderCreated
     |
     v
OrderPaid
     |
     v
OrderPacked
     |
     v
OrderShipped

But a poor key can create a hot partition.

Imagine:

Key = country

and 70% of your traffic uses:

India

Your partition distribution might look like:

Partition 0    ########################
Partition 1    ###
Partition 2    ##
Partition 3    ##

One partition becomes extremely busy while others are lightly used.

This can overload:

• the partition leader
• the broker hosting that leader
• the consumers assigned to that partition

A good key must consider both:

Ordering requirements

AND

Traffic distribution

---

7. What If the Key Is Missing?

A common simplified explanation is:

“If there is no key, Kafka uses round robin.”

That is not a complete description of modern producer behavior.

Modern Kafka producer implementations can use sticky/adaptive behavior for records without keys so that records can remain on a partition long enough to create efficient batches.

A better beginner mental model is:

WITH KEY

key
 |
 v
partitioning logic
 |
 v
consistent partition choice

versus:

WITHOUT KEY

producer selects an available partition
        |
        v
tries to build efficient batches
        |
        v
moves to another partition over time

Do not assume that every individual no-key record automatically rotates:

P0 -> P1 -> P2 -> P0 -> P1 -> P2

A strict round-robin partitioner can be configured explicitly, but it is not the best universal description of modern default behavior.

---

8. Can We Explicitly Select a Partition?

Yes.

An application can explicitly specify a partition.

Example:

Topic     = orders
Partition = 2
Key       = ORDER-101
Value     = ...

Then the producer does not need to choose the partition automatically.

But be careful.

Manual partition selection means your application becomes responsible for understanding:

Partition count
Broker distribution
Load balancing
Ordering
Scaling
Future partition changes

In many applications, a better design is:

good key strategy
+
Kafka partitioning

rather than hardcoding partition numbers.

---

9. Topic, Partition and Broker Relationship

Students must understand this before producer tuning.

Suppose:

Topic: vehicle-telemetry

Partitions:
P0
P1
P2

Kafka may distribute the partition leaders like:

Broker 1
  P0 Leader

Broker 2
  P1 Leader

Broker 3
  P2 Leader

If the producer selects:

Partition 2

it must eventually send the record to:

P2 Leader
=
Broker 3

So the flow is:

Producer
   |
   v
Topic
   |
   v
Partition selection
   |
   v
Partition Leader
   |
   v
Broker

The producer does not randomly select any broker for the final write.

---

10. How Does the Producer Know Which Broker Is the Leader?

The producer uses cluster metadata.

It starts with one or more bootstrap endpoints.

Conceptually:

Producer Configuration

bootstrap.servers
       |
       v
Initial Kafka connection
       |
       v
Cluster metadata

Metadata tells the producer things such as:

Topic: orders

P0 Leader = Broker 1
P1 Leader = Broker 3
P2 Leader = Broker 2

Important:

bootstrap server
!=
broker that permanently receives every record

Bootstrap servers are entry points that allow the client to discover the cluster.

After discovery, the producer communicates with the correct partition leaders.

---

11. Serialization

Before Kafka can send your key and value over the network, they must be converted into bytes.

Example:

Application Object
      |
      v
Serializer
      |
      v
Bytes

Both the key and value can have serializers.

For example:

key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

With Confluent environments, you may later use schema-aware serializers for:

Avro
Protobuf
JSON Schema

Schema Registry is a separate major Kafka/Confluent topic that we will cover later.

---

12. The Producer’s Internal Memory

A simplified producer pipeline looks like:

Application Thread
       |
       v
     send()
       |
       v
Serialization
       |
       v
Partition Selection
       |
       v
Producer Buffer / Accumulator
       |
       v
Record Batches
       |
       v
Background Sender / I/O
       |
       v
Kafka Broker

The producer has memory used to buffer records that have not yet been transmitted.

A key producer setting is:

buffer.memory

This controls approximately how much memory is available to buffer records waiting to be sent.

---

13. Why Buffer Records in Memory?

Imagine sending 100,000 events.

Inefficient design:

Record 1 -> Network request
Record 2 -> Network request
Record 3 -> Network request
...
Record 100000 -> Network request

Huge overhead.

Kafka instead tries to do:

Record 1
Record 2
Record 3
Record 4
Record 5
    |
    v
Batch
    |
    v
Efficient Produce request

Batching is one of Kafka producer’s most important performance mechanisms.

---

14. Batching Happens Per Partition

This is critical.

Suppose the producer is sending records to:

P0
P1
P2

The producer builds batches for destination partitions.

Conceptually:

Producer Memory

P0 Batch
[A][B][C][D]

P1 Batch
[E][F]

P2 Batch
[G][H][I]

A broker request can then contain batches for partitions led by that broker.

This is why partition selection and batching are connected.

---

15. batch.size

batch.size controls the target/default producer batch size in bytes.

A conceptual example:

batch.size = 16 KB

Records:

2 KB
2 KB
3 KB
4 KB
3 KB

can accumulate into a larger batch.

Larger useful batches may improve:

Network efficiency
Request efficiency
Compression efficiency
Throughput

But blindly setting a huge batch size is not automatically better.

Possible disadvantages:

Higher memory requirements
More unused batch allocation
Potential additional waiting in some workloads
No benefit for very low traffic

Always load-test.

---

16. linger.ms

What if the batch is not full yet?

Should Kafka:

send immediately?

or:

wait briefly for more records?

That is where:

linger.ms

comes in.

Example:

linger.ms = 5

means the producer may allow a very small batching window rather than immediately sending every under-filled batch.

---

17. batch.size and linger.ms Work Together

Think of a bus.

batch.size
=
How much can the bus carry?

linger.ms
=
How long may the bus wait for more passengers?

Simplified:

Records arrive
     |
     v
Accumulator
     |
     +---- batch ready? ----> SEND
     |
     +---- linger expires? -> SEND

This is one of the most important producer performance relationships.

---

18. Compression

Kafka can compress record batches.

Common options include:

none
gzip
snappy
lz4
zstd

Conceptually:

Uncompressed Batch

###########################

        |
        v

Compression

        |
        v

##########

Potential benefits:

Less network traffic
Less broker storage
Higher effective throughput
Lower network cost

Potential cost:

CPU used to compress and decompress

---

19. Which Compression Type Should We Use?

Do not memorize:

"X is always best."

There is no universal winner.

Evaluate:

Compression ratio
CPU cost
Latency
Record characteristics
Producer language/client
Broker workload
Network cost

A production engineer benchmarks the real workload.

For example, repeated JSON field names can often compress well.

---

20. Producer Sends a Produce Request

Suppose our key maps to:

Topic     = vehicle-telemetry
Partition = 2
Leader    = Broker 3

Then:

Producer
   |
   v
ProduceRequest
   |
   v
Broker 3

The partition leader receives the batch and appends the records to the partition log.

---

21. Who Assigns the Offset?

Another common misunderstanding:

The producer does not normally choose the Kafka offset.

Suppose the partition currently contains:

Partition 2

Offset 100
Offset 101
Offset 102

The leader appends the new record:

Offset 103

The producer can later receive metadata containing:

topic
partition
offset

So:

Producer chooses/supplies:
topic
key/value
possibly partition

while:

Kafka partition leader determines:
record's log position / offset

---

22. Replicas

Suppose:

Replication Factor = 3

Partition 2 may look like:

Broker 3 -> Leader
Broker 1 -> Follower
Broker 2 -> Follower

The producer writes to:

Leader

Followers replicate the leader’s data.

The producer does not normally send three separate copies directly to all replicas.

Think:

Producer
   |
   v
Leader
   |------> Follower
   |
   +------> Follower

---

23. Acknowledgements — acks

One of the most important producer reliability settings is:

acks

It controls when the producer considers the Produce request successfully acknowledged.

The major values are:

acks=0
acks=1
acks=all

---

24. acks=0 — Fire and Forget

The correct phrase is:

fire and forget

Flow:

Producer
   |
   v
Send
   |
   v
Does not wait for broker acknowledgement

Conceptually:

Producer:
"I sent it. I am not waiting to know whether Kafka accepted it."

Advantage

Very little acknowledgement overhead.

Major disadvantage

The producer does not receive broker acknowledgement that the broker successfully accepted the record.

Use only where message loss is acceptable.

---

25. acks=1 — Leader Acknowledges

Flow:

Producer
   |
   v
Leader Broker
   |
   v
Leader accepts/appends
   |
   v
ACK

Followers may still be catching up.

Possible failure:

Leader writes record
      |
      v
ACK sent
      |
      v
Leader fails before a follower has safely replicated
      |
      v
Possible record loss

So acks=1 is stronger than acks=0, but weaker than acks=all.

---

26. acks=all — Strongest Producer ACK Mode

A common oversimplification is:

“acks=all means fully replicated to every configured replica.”

A more accurate explanation is:

The partition leader waits for the acknowledgement conditions involving the current in-sync replicas.

This makes acks=all the strongest normal producer acknowledgement mode.

However, it must be understood together with:

Replication Factor
ISR
min.insync.replicas

---

27. min.insync.replicas

Now combine:

Replication Factor
+
ISR
+
acks
+
min.insync.replicas

A common production-style example:

replication.factor = 3
min.insync.replicas = 2
acks = all

Healthy state:

Leader    - ISR
Follower  - ISR
Follower  - ISR

One broker fails:

Leader    - ISR
Follower  - ISR
Follower  - unavailable

There may still be enough in-sync replicas to satisfy the configured minimum.

But if:

ISR size < min.insync.replicas

Kafka can reject strongly acknowledged writes.

That is not necessarily a bad thing.

Production principle:

Sometimes failing a write
is safer than
accepting a write with weaker durability.

---

28. Reliability vs Latency

You can now see a major tradeoff.

Conceptually:

acks=0

Less waiting
|
v
Lower acknowledgement latency
|
v
Weaker reliability feedback

versus:

acks=all

Stronger acknowledgement conditions
|
v
Stronger durability
|
v
Potentially more acknowledgement latency

Actual latency also depends on:

Network
Broker load
Batching
Compression
Partition leadership
Replica health
Request queues
Quotas
Cluster capacity
Application workload

Never tune only one setting in isolation.

---

29. Retries

Networks fail.

Brokers restart.

Partition leaders change.

Connections break temporarily.

Therefore Kafka producers can retry eligible transient failures.

Conceptually:

Send
 |
 v
Temporary failure
 |
 v
Retry
 |
 v
Success

But retries create an important question.

---

30. The Duplicate Problem

Imagine:

Producer sends Record A
        |
        v
Broker stores A
        |
        v
ACK is lost on the network
        |
        v
Producer thinks:
"Maybe it failed."
        |
        v
Producer retries A

Without protection:

A
A

could appear.

This is why idempotence is important.

---

31. Idempotent Producer

Kafka’s idempotent producer protects against duplicate writes caused by producer retries within Kafka’s producer guarantees.

Conceptually:

Send A
  |
  v
Broker stores A
  |
  v
ACK lost
  |
  v
Retry A
  |
  v
Kafka identifies retry
  |
  v
Do not create another logical copy

Important:

Idempotent Producer
!=
Automatically exactly-once business processing

Exactly-once behavior involving multiple Kafka writes, consumed offsets, transactions, external databases or other business side effects is a larger topic.

We will cover Kafka Transactions separately.

---

32. max.in.flight.requests.per.connection

A producer can have multiple requests waiting for responses.

Conceptually:

Request 1 ---------------->
Request 2 ---------------->
Request 3 ---------------->

This improves throughput.

But historically, retries plus multiple in-flight requests could create ordering concerns when idempotence was disabled.

Therefore these concepts must be understood together:

Throughput
Retries
Ordering
Idempotence

---

33. delivery.timeout.ms

Instead of thinking only:

"How many times should I retry?"

a better mental model is:

"How long may this record continue trying before delivery is considered failed?"

That is the role of:

delivery.timeout.ms

It bounds the overall delivery attempt window.

This can include:

time waiting before send
broker acknowledgement waiting
eligible retries

---

34. Producer Memory and Backpressure

Suppose the application produces:

1,000,000 records/sec

but Kafka can currently accept:

300,000 records/sec

Records accumulate in producer memory.

Application
   |||||||||||
   vvvvvvvvvvv
Producer Buffer
########################
   |||
   vvv
Kafka

Eventually the producer buffer may become full.

The application can then experience blocking and eventually send failures depending on configuration.

This is backpressure.

Do not “solve” it by blindly adding huge memory.

Ask:

Why is Kafka slower than the producer?

Broker saturation?
Network?
Quota?
Hot partition?
Too few partitions?
Broker failover?
Large records?
Insufficient cluster capacity?

---

35. buffer.memory

buffer.memory controls approximately how much producer memory is available for pending records.

Increasing it can help absorb short bursts.

But it does not create infinite throughput.

If your steady-state application rate is greater than Kafka’s sustainable rate:

more buffer
=
failure happens later

not:

problem solved

---

36. max.block.ms

send() is normally asynchronous, but application calls can still block in some cases, such as:

waiting for metadata
waiting for producer buffer space

max.block.ms limits how long the producer may block in these situations.

This matters for application responsiveness.

Example:

Web request
   |
   v
producer.send()
   |
   v
buffer completely exhausted
   |
   v
application thread waits

Kafka producer performance can therefore affect your API latency even before the send ultimately succeeds or fails.

---

37. send() Does Not Mean “Already Stored”

This distinction is critical.

Application code:

producer.send(record);

does not automatically mean:

"The record is already safely stored in Kafka."

It normally means that the record has entered the producer’s asynchronous delivery pipeline.

To observe the delivery result, use:

Future

or:

Callback

---

38. Use Callbacks in Real Applications

Conceptually:

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        // handle/report failure
    } else {
        // success
        // metadata.topic()
        // metadata.partition()
        // metadata.offset()
    }
});

This provides visibility into:

Success
Failure
Topic
Partition
Offset

Do not produce important events and completely ignore asynchronous failures.

---

39. flush()

flush() means:

Force currently buffered records to be made available for immediate sending and wait for their associated requests to complete.

Conceptually:

Producer Buffer

A
B
C
D
   |
   v
flush()
   |
   v
send pending records now
   |
   v
wait for requests to complete

---

40. Should We Call flush() After Every Record?

Normally:

No.

Bad high-throughput pattern:

send(A)
flush()

send(B)
flush()

send(C)
flush()

This destroys much of the benefit of:

asynchronous sending
batching
request aggregation

You effectively move toward:

send
wait

send
wait

send
wait

Use flush() intentionally.

Typical uses:

controlled synchronization point
small teaching/demo application
before a critical application boundary
before shutdown when needed

---

41. flush() vs close()

They are not the same.

flush()

send pending buffered records
wait for requests
producer remains usable

close()

complete producer shutdown
release resources
background I/O stops
producer should no longer be used

A Kafka producer owns:

memory buffers
network connections
background I/O resources

Always close it properly.

---

42. Complete Producer Journey

Put everything together:

APPLICATION
    |
    v
Create business event
    |
    v
ProducerRecord
    |
    +--> Topic
    +--> Key
    +--> Value
    +--> Headers
    +--> Optional partition/timestamp
    |
    v
Serializer
    |
    v
Cluster Metadata
    |
    v
Partition Selection
    |
    v
Producer Buffer
    |
    v
Per-Partition Batch
    |
    +--> batch.size
    +--> linger.ms
    |
    v
Compression
    |
    v
Background Sender
    |
    v
Network
    |
    v
Partition Leader Broker
    |
    v
Append to Partition Log
    |
    v
Offset Assigned
    |
    v
Follower Replication
    |
    v
ISR
    |
    v
acks condition
    |
    v
ACK / Error
    |
    v
Retry if eligible
    |
    v
Callback / Future

This is the producer lifecycle you should be able to explain in an interview and in a production incident.

---

43. Producer Performance Tuning — The Correct Approach

Do not start tuning like this:

Increase batch.size!
Increase memory!
Decrease acks!
Add partitions!

Instead:

1. Define requirement
2. Measure baseline
3. Find bottleneck
4. Change one variable
5. Load test
6. Measure again
7. Check reliability impact

Performance tuning without measurements is guessing.

---

44. Goal 1 — Optimize for High Throughput

If your goal is:

Maximum events/sec
Maximum MB/sec
Efficient broker/network usage

consider:

Increase useful batching
     |
     v
appropriate batch.size

Allow batches to fill
     |
     v
appropriate linger.ms

Reduce bytes transferred
     |
     v
compression

Distribute traffic
     |
     v
enough well-balanced partitions

Avoid per-message blocking
     |
     v
asynchronous sends

Handle bursts
     |
     v
appropriate producer memory

Also monitor:

broker throttling
hot partitions
request latency
error rate
retry rate

---

45. High-Throughput Mental Model

Less efficient:

A -> Kafka
B -> Kafka
C -> Kafka
D -> Kafka

More efficient:

A
B
C
D
E
F
 |
 v
Batch
 |
 v
Compress
 |
 v
Kafka

Kafka achieves much of its producer efficiency by moving batches rather than treating every record as an isolated network transaction.

---

46. Goal 2 — Minimize Latency

Suppose the requirement is:

Event must reach Kafka as quickly as practical.

Consider:

Smaller batching wait
Lower linger.ms
Avoid unnecessary application blocking
Healthy nearby Kafka infrastructure
Correct partition distribution
Low broker load
Fast serialization
Appropriate compression choice

But do not automatically set every batching value to zero.

Under real load, a tiny batching window may improve overall system efficiency enough to produce excellent practical latency.

Always benchmark your real workload.

---

47. Goal 3 — Maximum Reliability

For events where loss is unacceptable, think about the complete chain:

Producer
   |
   v
acks=all
   |
   v
Healthy ISR
   |
   v
Appropriate min.insync.replicas
   |
   v
Replication factor
   |
   v
Idempotence
   |
   v
Retries
   |
   v
Error handling
   |
   v
Monitoring

A common production-oriented pattern is:

Replication Factor = 3
min.insync.replicas = 2
acks = all
enable.idempotence = true

This is a useful starting concept, not a universal configuration that should be copied blindly.

---

48. The Throughput / Latency / Reliability Triangle

A Kafka producer balances:

               THROUGHPUT
                  /\
                 /  \
                /    \
               /      \
              /        \
        LATENCY ------ RELIABILITY

Examples:

More batching
-> often better throughput
-> may add waiting time

More compression
-> less network/storage
-> more CPU

acks=all
-> stronger durability
-> stronger acknowledgement conditions

acks=0
-> less acknowledgement waiting
-> much weaker delivery feedback

More producer memory
-> more burst absorption
-> does not increase broker capacity

Master-level Kafka is about understanding these tradeoffs.

---

49. Important Producer Tuning Parameters

Parameter	Main purpose	Main tradeoff
batch.size	Build larger record batches	Memory / workload dependent
linger.ms	Give batches time to fill	Throughput vs added waiting
compression.type	Reduce bytes	CPU vs network/storage
acks	Delivery acknowledgement strength	Reliability vs ACK latency
enable.idempotence	Prevent duplicate retry writes	Requires compatible reliability settings
retries	Recover from transient failures	Longer time before permanent failure
delivery.timeout.ms	Bound delivery attempts	Reliability window vs failure speed
buffer.memory	Buffer pending records	Memory usage
max.block.ms	Limit blocking on metadata/buffer	Application responsiveness
max.in.flight.requests.per.connection	Parallel outstanding requests	Throughput/order interaction
max.request.size	Limit producer request size	Large-message constraints

Do not tune these independently without understanding their interactions.

---

50. Example Balanced Producer Profile

For teaching purposes, a balanced producer might start conceptually with:

acks=all
enable.idempotence=true

compression.type=zstd

linger.ms=5
batch.size=65536

delivery.timeout.ms=120000

This is an example starting profile, not a production recipe.

The correct configuration depends on:

event size
event rate
latency SLO
cluster location
number of partitions
network bandwidth
CPU
compression ratio
broker capacity
failure requirements
client library

---

51. Confluent Kafka Cluster Connection

For Confluent Cloud-style training, the producer also needs security and connection settings.

Conceptually:

bootstrap.servers=<CONFLUENT_BOOTSTRAP_SERVER>

security.protocol=SASL_SSL
sasl.mechanism=PLAIN

sasl.jaas.config=<API_KEY_AND_SECRET>

key.serializer=...
value.serializer=...

Keep secrets outside application source code wherever possible.

After authentication, the normal producer flow remains:

Discover metadata
      |
      v
Select partition
      |
      v
Find partition leader
      |
      v
Produce batch

---

52. Practical Java Example

Simplified teaching example:

Properties props = new Properties();

props.put("bootstrap.servers", "<BOOTSTRAP_SERVER>");
props.put("security.protocol", "SASL_SSL");
props.put("sasl.mechanism", "PLAIN");

props.put(
    "key.serializer",
    "org.apache.kafka.common.serialization.StringSerializer"
);

props.put(
    "value.serializer",
    "org.apache.kafka.common.serialization.StringSerializer"
);

props.put("acks", "all");
props.put("enable.idempotence", "true");
props.put("compression.type", "zstd");
props.put("linger.ms", "5");

KafkaProducer<String, String> producer =
    new KafkaProducer<>(props);

ProducerRecord<String, String> record =
    new ProducerRecord<>(
        "vehicle-telemetry",
        "CAR-101",
        "{\"speed\":88}"
    );

producer.send(record, (metadata, exception) -> {

    if (exception != null) {
        exception.printStackTrace();
        return;
    }

    System.out.println(
        "topic=" + metadata.topic()
        + " partition=" + metadata.partition()
        + " offset=" + metadata.offset()
    );
});

producer.flush();
producer.close();

Why call flush() here?

Because this is a small teaching program and we want an explicit point at which all pending sends have completed before shutdown.

In a high-throughput long-running service, do not call flush() after every record.

---

53. Lab 1 — Keyed Events

Create a topic with multiple partitions.

For example:

vehicle-telemetry

P0
P1
P2
P3

Produce:

CAR-101 event 1
CAR-101 event 2
CAR-101 event 3

Use the callback to print:

partition
offset

Observe whether records for the same key consistently go to the same partition.

Then try:

CAR-101
CAR-102
CAR-103
CAR-104

Compare the partition distribution.

---

54. Lab 2 — No-Key Events

Produce records without a key:

null -> event 1
null -> event 2
null -> event 3
...

Print the assigned partitions.

Do not assume that modern no-key behavior is a naive strict per-record round robin.

Observe how batches and partition selection interact.

---

55. Lab 3 — Batching Experiment

Run a load test with:

linger.ms = 0

Record:

throughput
request rate
latency
average batch size

Then test:

linger.ms = 5

Then:

linger.ms = 20

Do not ask only:

"Which value is best?"

Ask:

"What changed and why?"

---

56. Lab 4 — Compression Experiment

Produce the same data using:

compression.type=none

then:

compression.type=lz4

then:

compression.type=zstd

Measure:

producer CPU
bytes sent
throughput
latency
compression ratio

The correct choice depends on your workload.

---

57. Lab 5 — acks Experiment

In a safe training cluster, compare:

acks=0
acks=1
acks=all

Do not only measure speed.

Record:

Producer latency
Errors
Behavior during broker/leader disruption
Durability guarantees

The objective is to understand that configuration is about failure behavior, not only benchmark numbers.

---

58. Lab 6 — flush() Experiment

Test:

for each event:
    send()
    flush()

Then compare with:

send many events asynchronously
flush once at the end

Compare throughput and latency.

This demonstrates why excessive flush() calls can remove much of the benefit of producer batching.

---

59. Producer Metrics to Monitor

A production producer should never be a black box.

Important metrics include:

record-send-rate
record-error-rate
record-retry-rate
request-latency-avg
request-latency-max
batch-size-avg
records-per-request-avg
compression-rate-avg
record-queue-time-avg
buffer-available-bytes
waiting-threads
requests-in-flight
produce-throttle-time-avg

These help you understand whether the producer is:

healthy
slow
buffer-constrained
retrying
throttled
poorly batched
experiencing broker/network latency

---

60. How to Read Producer Metrics

High record-error-rate

Investigate:

authentication
authorization
broker availability
invalid records
message size
ISR health
timeouts

---

High record-retry-rate

Possible causes:

network instability
leader elections
broker overload
transient broker errors

---

High request-latency-avg

Possible causes:

network latency
broker overload
replication delay
throttling
cross-region traffic

---

Very low batch-size-avg

Possible interpretation:

batches are not filling

Possible reasons:

low traffic
very low linger
too many destination partitions for the traffic level
application send pattern

---

Low buffer-available-bytes

Producer buffer pressure is increasing.

Ask:

Is the application producing events
faster than Kafka can accept them?

---

61. Producer Troubleshooting Flow

When producers become slow:

START
  |
  v
Are sends failing?
  |
  v
Check error and retry metrics
  |
  v
Check request latency
  |
  v
Check broker throttling
  |
  v
Check partition distribution
  |
  v
Check hot partitions
  |
  v
Check batch efficiency
  |
  v
Check compression
  |
  v
Check producer buffer pressure
  |
  v
Check network latency
  |
  v
Check broker / ISR health
  |
  v
Change one thing
  |
  v
Load test again

Do not change ten settings simultaneously.

Otherwise, you will not know which change helped or hurt.

---

62. Common Producer Mistakes

Mistake 1 — No meaningful key

Later the team says:

"We need strict ordering for each customer."

Key design should be considered early.

---

Mistake 2 — Bad low-cardinality key

Examples:

country
region
event_type

may create very uneven traffic.

---

Mistake 3 — Calling flush() for every event

This destroys useful asynchronous batching.

---

Mistake 4 — Assuming send() means successful durable write

It does not.

Observe callback/Future results.

---

Mistake 5 — Using acks=0 for critical business events

You lose normal broker acknowledgement of delivery.

---

Mistake 6 — Setting huge buffers to hide slow Kafka

This delays the symptom rather than fixing sustainable throughput.

---

Mistake 7 — Increasing partitions without thinking about keys

You can still have hot partitions.

---

Mistake 8 — Enabling retries without understanding idempotence

Retry semantics matter.

---

Mistake 9 — Optimizing throughput while ignoring reliability

A benchmark that loses critical records under failure is not a successful production design.

---

Mistake 10 — Copying configuration blindly

Always benchmark your workload and understand your client version.

---

63. Production Producer Checklist

Before calling a producer production-ready, verify:

• [ ] Correct topic selected
• [ ] Key strategy intentionally designed
• [ ] Partition distribution tested
• [ ] Serialization format selected
• [ ] Schema evolution strategy defined where required
• [ ] acks intentionally selected
• [ ] Replication / ISR policy understood
• [ ] Idempotence behavior understood
• [ ] Retry behavior tested
• [ ] Delivery timeout understood
• [ ] Batching load-tested
• [ ] linger.ms load-tested
• [ ] batch.size load-tested
• [ ] Compression benchmarked
• [ ] Producer memory monitored
• [ ] Backpressure behavior tested
• [ ] Callback/errors handled
• [ ] Producer properly closed on shutdown
• [ ] flush() used only where appropriate
• [ ] Hot partitions monitored
• [ ] Producer latency monitored
• [ ] Error and retry rates monitored
• [ ] Failure testing performed
• [ ] Broker failure tested
• [ ] Network interruption tested
• [ ] Application SLOs documented

---

64. Interview-Level Questions

What is a Kafka producer?

A Kafka client that publishes records to Kafka topics.

What does a producer record contain?

A topic, key/value and optionally headers, timestamp and an explicitly selected partition.

Is the key mandatory?

No. But a key is extremely important when partition affinity and per-entity ordering matter.

Does no key always mean strict round robin?

No. Modern producers can use sticky/adaptive behavior to improve batching. Strict round robin can be explicitly configured.

Who assigns the offset?

The Kafka partition leader assigns the log position when the record is appended.

Does the producer write directly to every replica?

No. The producer writes to the partition leader. Followers replicate from the leader.

What is batch.size?

A producer setting that influences the target/default amount of data grouped into a partition batch.

What is linger.ms?

A small upper waiting window that gives batches time to fill before being sent.

What does acks=0 mean?

The producer does not wait for broker acknowledgement.

What does acks=1 mean?

The partition leader acknowledges without waiting for the strongest ISR acknowledgement condition.

What does acks=all mean?

The leader waits for the current in-sync replica acknowledgement requirements. This is Kafka’s strongest normal producer acknowledgement mode.

Why use idempotence?

To prevent duplicate Kafka writes caused by retries within producer idempotence guarantees.

What does flush() do?

It forces currently buffered records to become immediately eligible for sending and waits for their associated requests to complete.

---

65. Master Mental Model

A beginner thinks:

Producer sends a message to Kafka.

A Kafka engineer thinks:

Business Event
     |
     v
ProducerRecord
     |
     v
Key + Value + Headers
     |
     v
Serialization
     |
     v
Metadata
     |
     v
Partition Strategy
     |
     v
Producer Buffer
     |
     v
Per-Partition Batch
     |
     +--> batch.size
     +--> linger.ms
     |
     v
Compression
     |
     v
Background Sender
     |
     v
Partition Leader
     |
     v
Broker Append
     |
     v
Offset Assignment
     |
     v
Replication
     |
     v
ISR
     |
     v
acks
     |
     v
Retry / Idempotence
     |
     v
Delivery Result

A production Kafka engineer asks:

How does every step affect:

PERFORMANCE?
LATENCY?
RELIABILITY?
AVAILABILITY?
COST?

That is the level of thinking we want from the Kafka Master Tutorials series.

---

66. Final Takeaways

Remember these ten ideas:

1. A producer is much more than a network sender.
2. Keys are fundamental to partitioning and ordering design.
3. Modern no-key partitioning should not simply be taught as strict round robin.
4. Kafka batches records primarily for efficiency.
5. batch.size and linger.ms work together.
6. Compression trades CPU for potentially lower network and storage usage.
7. The producer writes to the partition leader, not directly to every replica.
8. acks, ISR and min.insync.replicas work together to define durability.
9. Retries should be understood together with idempotence and delivery timeout.
10. Producer performance tuning must be driven by measurements, not copied configuration.

---

67. Suggested Next Producer Tutorials

After this producer foundation, the natural deep dives are:

Producer Internals
        |
        v
Partitioning Strategies
        |
        v
Producer Reliability & Delivery Semantics
        |
        v
Producer Performance Tuning Lab
        |
        v
Producer Errors, Retries & Idempotence
        |
        v
Kafka Transactions
        |
        v
Production Producer Monitoring

That path takes students from producer fundamentals to production-grade producer mastery.