Kafka Architecture

1. Learning objectives

By the end of this tutorial, a student should understand:

what a Kafka cluster actually contains
producer side vs Kafka cluster side vs consumer side
brokers, topics, partitions and replicas
leader and follower replicas
KRaft controllers
Kafka’s data plane and control plane
where Kafka stores data
how Kafka provides scalability
how Kafka provides reliability
how Kafka provides availability
how this architecture maps to Confluent Kafka

2. Start with the simplest Kafka architecture

At the highest level:

Producer
   ↓
Kafka Cluster
   ↓
Consumer

For example:

Vehicle
   ↓
Telematics Application
   ↓
Kafka
   ↓
Analytics Application

But a real Kafka architecture contains many more pieces:

                    KAFKA CLUSTER

               ┌────────────────────┐
Producer ─────→│ Broker 1           │
               │ Broker 2           │─────→ Consumer
               │ Broker 3           │
               └────────────────────┘
                       │
                       │
                   Topics
                       │
                   Partitions
                       │
                    Replicas

Kafka distributes topic partitions across brokers, and partitions can have replicas on multiple brokers for fault tolerance. (Apache Kafka)

3. Kafka has two major architectural areas

A very useful master-level way of understanding Kafka is to separate it into:

DATA PLANE

Producer
   ↓
Broker
   ↓
Partition
   ↓
Consumer

and:

CONTROL PLANE

KRaft Controllers
       ↓
Cluster metadata
       ↓
Partition leaders
Broker membership
Topic configuration
Replica information

Data plane

The data plane handles the actual events.

Examples:

GPS event
Payment event
Order event
User-click event

The producer writes these records to brokers and consumers fetch them.

Control plane

The control plane manages information about the Kafka cluster.

For example:

Which brokers exist?
Which topics exist?
How many partitions exist?
Which broker leads Partition 3?
Where are replicas located?
Which brokers are alive?

Modern Kafka uses KRaft for this metadata/control-plane function. Current Kafka documentation recommends separate broker and controller roles for critical deployments; combined broker/controller mode is mainly appropriate for smaller or development environments. (kafka.apache.org)

4. Kafka Cluster

A Kafka cluster is a collection of Kafka servers working together.

Those Kafka servers are called:

brokers

Example:

Kafka Cluster

+----------+
| Broker 1 |
+----------+

+----------+
| Broker 2 |
+----------+

+----------+
| Broker 3 |
+----------+

A production cluster normally contains multiple brokers because a single broker creates obvious limits for scale and availability.

5. Broker

A broker is a Kafka server.

Its important responsibilities include:

Receive producer requests
Store partition data
Serve consumer fetch requests
Participate in replication
Maintain partition replicas
Respond to metadata-related client requests

Think of a broker as:

Kafka Broker
    =
Network Server
+
Partition Storage
+
Replication Participant

6. Topic

A topic is a logical stream/category of events.

Examples:

vehicle-telemetry
orders
payments
customer-events
charging-events
system-events

Suppose:

Topic = vehicle-telemetry

It may receive:

{"vehicle_id":"V101","speed":60}
{"vehicle_id":"V205","speed":74}
{"vehicle_id":"V101","speed":63}

But the topic is only the logical view.

Internally, Kafka divides topics into partitions. (Apache Kafka)

7. Partition

Suppose we create:

vehicle-telemetry

with three partitions:

vehicle-telemetry

├── Partition 0
├── Partition 1
└── Partition 2

Each partition is an ordered log.

Partition 0

Offset
0 → Event A
1 → Event B
2 → Event C
3 → Event D

Kafka guarantees ordering within a partition, not automatically across every partition of a multi-partition topic. Events using the same key are normally routed consistently so that related records can remain within the same partition. (Apache Kafka)

8. Why partitions exist

Partitions give Kafka parallelism.

Without partitions:

Topic
 ↓
one stream
 ↓
limited parallelism

With partitions:

               Topic

       ┌────────┼────────┐
       ↓        ↓        ↓
      P0       P1       P2
       ↓        ↓        ↓
 Consumer1 Consumer2 Consumer3

This is one of Kafka’s fundamental scaling mechanisms.

More partitions can allow more parallel producers, broker work and consumers, although blindly creating huge partition counts also has operational costs.

9. Where partitions physically live

Partitions are distributed across brokers.

Example:

Broker 1
  P0
  P3

Broker 2
  P1
  P4

Broker 3
  P2
  P5

Therefore increasing broker capacity can allow Kafka to distribute storage and traffic.

10. Replication

Now imagine Broker 1 fails.

If Partition 0 exists only on Broker 1:

Broker 1
   ↓
P0
   ↓
BROKER FAILURE

DATA UNAVAILABLE

Kafka addresses this using replicas.

Example replication factor:

Replication Factor = 3

Then:

Partition 0

Broker 1 → Replica
Broker 2 → Replica
Broker 3 → Replica

Kafka replication operates at the partition level. A replication factor of three is a common production choice. (Apache Kafka)

11. Leader and Followers

Kafka does not normally allow every replica to independently process writes.

One replica is the:

leader

Other replicas are:

followers

Example:

Partition 0

Broker 1 → LEADER
Broker 2 → FOLLOWER
Broker 3 → FOLLOWER

The producer writes to:

Leader

Followers replicate the leader’s log.

Producer
   ↓
Leader
   ├────→ Follower
   └────→ Follower

Kafka’s producer sends directly to the broker leading the target partition. (Apache Kafka)

12. ISR — In-Sync Replicas

ISR means:

In-Sync Replicas

Think of it initially as:

replicas sufficiently caught up with the leader to participate in Kafka’s durability and failover guarantees.

Example:

Partition 0

Leader     Broker 1  ✓ ISR
Follower   Broker 2  ✓ ISR
Follower   Broker 3  ✓ ISR

Suppose Broker 3 becomes too far behind:

Leader     Broker 1  ✓ ISR
Follower   Broker 2  ✓ ISR
Follower   Broker 3  ✗ outside ISR

min.insync.replicas works together with producer acknowledgment settings to control how much replica health is required for strongly acknowledged writes. (Apache Kafka)

13. KRaft Controllers

Modern Kafka uses KRaft controllers rather than ZooKeeper for Kafka’s metadata management.

Conceptually:

              KRaft Controller Quorum
                       ↓
              Cluster Metadata
                       ↓
     ┌─────────────────┼─────────────────┐
     ↓                 ↓                 ↓
 Broker 1          Broker 2          Broker 3

Controllers manage information such as broker membership, topics, partitions and leadership.

For critical deployments, Kafka recommends controller redundancy; a controller quorum of 2N + 1 can tolerate N controller failures. (kafka.apache.org)

14. Architecture viewed through three goals

This is extremely important.

Performance

Kafka obtains performance through things such as:

Partitions
Parallelism
Sequential log writes
Batching
Compression
Efficient network transfer
Consumer batching

Reliability

Reliability comes from:

Replication
ISR
acks
min.insync.replicas
Retries
Idempotent producer
Offset management

Availability

Availability comes from:

Multiple brokers
Replica placement
Leader election
Multiple KRaft controllers
Availability-zone/rack awareness
Monitoring
Failover

Throughout this series, always ask:

Does this configuration affect:

PERFORMANCE?
RELIABILITY?
AVAILABILITY?

or all three?

That is how production Kafka engineers think.

Architecture knowledge check

Explain this without looking:

Producer
   ↓
Topic
   ↓
Partition
   ↓
Leader Broker
   ↓
Follower Replicas
   ↓
Consumer

If that makes sense, move to the next tutorial.

Tutorial 2 — Components of Kafka

Now let’s identify the building blocks individually.

1. Producer

A producer publishes records to Kafka.

Application
    ↓
Kafka Producer
    ↓
Topic

A record commonly contains:

Topic
Partition (optional explicit choice)
Key
Value
Headers
Timestamp

Example:

{
  "vehicleId": "V101",
  "speed": 82
}

2. Serializer

Kafka sends bytes over the network.

Your application may have:

Java Object
JSON Object
String
Integer
Protobuf Object
Avro Object

A serializer converts application data to bytes.

Object
  ↓
Serializer
  ↓
Bytes
  ↓
Kafka

With Confluent environments, Schema Registry can be added for schema-based formats and compatibility management.

3. Record key

A Kafka record may have a key.

Example:

Key = vehicle-101
Value = GPS event

Keys are extremely important because they commonly influence partition selection.

vehicle-101
    ↓
hash(key)
    ↓
Partition 2

Future events using that same key can then continue to the same partition, preserving their per-partition order. (Apache Kafka)

4. Topic

Logical stream:

vehicle-telemetry

Not a physical single file.

It consists of partitions.

5. Partition

Physical unit of:

ordering
parallelism
storage
replication

This is perhaps the most important Kafka component to master.

6. Offset

Every record in a partition has a position.

Example:

Partition 2

Offset 0 → A
Offset 1 → B
Offset 2 → C
Offset 3 → D

Offsets are unique within a partition, not globally across the whole topic.

7. Broker

Stores and serves partitions.

Broker 1
Broker 2
Broker 3

8. Cluster

Collection of brokers and the associated Kafka control plane.

9. Replica

A copy of a partition.

P0 Replica → Broker 1
P0 Replica → Broker 2
P0 Replica → Broker 3

10. Leader

The active replica that handles normal requests for its partition.

P0 Leader = Broker 2

11. Follower

Copies partition data from its leader.

Kafka’s replication design has followers consume/copy the leader’s log. (Apache Kafka)

12. ISR

The in-sync replica set.

Important when deciding whether Kafka can safely acknowledge writes and which replicas are suitable for normal failover.

13. Replication factor

Number of replicas for each partition.

Example:

RF = 3

means:

1 leader
+
2 additional replicas

= 3 copies

14. KRaft Controller

Manages cluster metadata and leadership information.

Do not confuse:

KRaft Controller

with:

Partition Leader

They solve different problems.

15. Consumer

Reads records from Kafka.

Consumers pull/fetch data from the partition leaders. (Confluent Documentation)

16. Consumer Group

Consumers can work together.

Consumer Group: analytics

Consumer 1
Consumer 2
Consumer 3

If the topic has:

P0
P1
P2

Kafka could assign:

P0 → Consumer 1
P1 → Consumer 2
P2 → Consumer 3

Within one consumer group, a partition is assigned to one consumer at a time. Consumer groups therefore provide parallel consumption. (Confluent Documentation)

17. Group Coordinator

A broker acts as coordinator for a particular consumer group.

It helps manage:

group membership
heartbeats
partition assignments/rebalancing
offset commit validation

The group ID maps to a partition of Kafka’s internal __consumer_offsets topic; the broker leading that partition becomes the coordinator for the group. (Confluent Documentation)

18. Consumer Offset

Suppose the consumer processed through:

Offset 1050

Kafka can store the group’s committed position.

If the consumer restarts, it can continue near that position instead of necessarily restarting at zero.

19. `__consumer_offsets`

Kafka has an internal topic:

__consumer_offsets

It is used to store committed consumer-group offsets and participates in consumer-group coordination. (Confluent Documentation)

Students should recognize this name immediately.

20. Retention

Kafka normally keeps records according to topic retention policies rather than deleting a record simply because one consumer read it.

This enables:

Replay
Reprocessing
New consumers
Recovery
Backfills

That behavior is a major difference from the simplistic mental model of a destructive work queue.

21. Log segments

A partition log is physically broken into segments rather than growing forever as a single giant file.

Conceptually:

Partition 0

000000.log
000001.log
000002.log
...

Segments help Kafka manage retention and storage efficiently.

22. Schema Registry

Schema Registry is part of the broader Confluent ecosystem, not a required core broker component.

It helps manage schemas such as:

Avro
JSON Schema
Protobuf

Think:

Producer
   ↓
Schema
   ↓
Kafka data
   ↓
Consumer

It is extremely valuable when many teams independently produce and consume data.

23. Kafka Connect

Kafka Connect is another surrounding component.

It moves data between Kafka and external systems.

PostgreSQL
    ↓
Kafka Connect
    ↓
Kafka

or:

Kafka
  ↓
Kafka Connect
  ↓
Data Warehouse

24. Stream processing

Kafka can feed applications such as:

Kafka Streams
Flink
ksqlDB
other stream processors

These applications read streams, perform computation and frequently write results back to Kafka.

Component relationship map

Application
    ↓
Producer
    ↓
Serializer
    ↓
Key
    ↓
Topic
    ↓
Partition
    ↓
Leader Broker
    ↓
Replica / ISR
    ↓
Consumer Group
    ↓
Consumer
    ↓
Deserializer
    ↓
Application
    ↓
Offset Commit

Memorizing isolated definitions is not enough.

Mastery comes from understanding this relationship.

Tutorial 3 — How Kafka Works

Now we put the components into motion.

1. Application creates an event

Example:

{
  "vehicleId": "V101",
  "speed": 92,
  "timestamp": "2026-08-19T06:30:00Z"
}

The application decides:

I need to publish this event.

2. Producer creates a Kafka record

Conceptually:

Topic = vehicle-telemetry
Key   = V101
Value = event

3. Serialization happens

Application Object
       ↓
   Serializer
       ↓
      Bytes

Kafka brokers primarily deal with bytes; application semantics come from producers, consumers and optionally schema-management systems.

4. Producer obtains cluster metadata

The producer is configured with bootstrap information.

Example concept:

bootstrap servers
     ↓
Connect to Kafka
     ↓
Receive metadata

The producer learns:

Which topic partitions exist?
Which broker currently leads each partition?

Bootstrap servers are entry points, not necessarily the broker that receives every record.

5. Partition selection

The producer decides the destination partition.

For example:

Key = V101
       ↓
Partitioning strategy
       ↓
Partition 2

Same-key routing is important where per-entity ordering matters. Kafka’s ordering guarantee is per topic-partition. (Apache Kafka)

6. Producer batches records

Kafka producers are designed to batch.

Instead of:

Event 1 → network
Event 2 → network
Event 3 → network

the producer can do:

Event 1
Event 2
Event 3
Event 4
   ↓
Batch
   ↓
Network request

Producer settings such as batch.size and linger.ms influence batching. Modern Kafka producer documentation describes linger.ms as an upper waiting time to allow a batch to fill before sending.

7. Compression may occur

The batch may be compressed.

Examples include:

gzip
snappy
lz4
zstd

Compression can reduce:

network bandwidth
broker network load
storage footprint

at the cost of CPU.

8. Producer sends to partition leader

Suppose:

Topic: vehicle-telemetry
Partition: 2
Leader: Broker 3

Then:

Producer
   ↓
Broker 3

Kafka intentionally has producers send to the leader directly rather than through a separate routing tier. (Apache Kafka)

9. Leader appends record

Broker 3 receives the produce request.

Conceptually:

Partition 2

0
1
2
3
4
5 ← new record

Kafka is based around append-oriented partition logs.

10. Offset is assigned

Suppose the new position is:

Offset 8127

The record can now be identified by:

Topic
+
Partition
+
Offset

Example:

vehicle-telemetry / 2 / 8127

11. Followers replicate

If:

RF = 3

we might have:

Broker 3 → Leader
Broker 1 → Follower
Broker 2 → Follower

The followers copy records from the leader. (Apache Kafka)

12. Producer acknowledgment

Now Kafka determines when to reply successfully to the producer.

Important producer setting:

acks

A simplified view:

acks=0
Producer does not wait for broker acknowledgment.

acks=1
Leader acknowledges.

acks=all
Leader waits for the required in-sync replication conditions.

acks=all provides Kafka’s strongest producer acknowledgment mode and is required when producer idempotence is enabled. (Confluent Documentation)

13. Consumers fetch data

Consumers do not normally wait for brokers to push arbitrary messages at them.

They issue fetch requests.

Consumer
   ↓
Fetch from P2 starting at offset X
   ↓
Broker
   ↓
Records

The consumer controls its position and can rewind that position to re-read retained data. (Confluent Documentation)

14. Consumer processes events

Fetch
  ↓
Deserialize
  ↓
Business logic
  ↓
Database / API / Analytics / Alert

15. Consumer commits progress

After processing records, the consumer may commit an offset.

Conceptually:

"I have safely processed records through this point."

That progress is stored using Kafka’s consumer offset machinery, including __consumer_offsets. (Confluent Documentation)

16. Kafka retains the original record

Consuming does not inherently delete the event.

Therefore:

Consumer A reads it
Consumer B can read it
Consumer C can read it later
Consumer A may replay it

This is central to Kafka’s event-log model.

Tutorial 4 — How All Kafka Components Work Together

This tutorial answers:

Why does Kafka need so many components?

Because each component solves a different scaling or reliability problem.

1. Producer + topic

Producer answers:

WHO creates the event?

Topic answers:

WHAT logical stream does it belong to?

Example:

Telematics Service
       ↓
vehicle-telemetry

2. Topic + partition

Topic gives logical organization.

Partition gives physical parallelism.

vehicle-telemetry

P0
P1
P2
P3
P4
P5

3. Partition + key

The key helps decide:

WHERE should this event go?

For vehicle systems:

Key = vehicle_id

is often useful when we want all events for one vehicle to retain ordering.

4. Partition + broker

Partitions have to physically live somewhere.

That somewhere is a broker.

P0 → Broker 1
P1 → Broker 2
P2 → Broker 3

5. Partition + replication

One physical copy is dangerous.

Therefore:

P0

Leader   Broker 1
Follower Broker 2
Follower Broker 3

6. Replication + ISR

A replica existing is not enough.

Kafka must know which replicas are sufficiently synchronized.

Hence:

ISR

7. ISR + acks + min.insync.replicas

These three are tightly connected.

Suppose:

Replication Factor = 3

ISR:
Broker 1
Broker 2
Broker 3

and:

min.insync.replicas = 2
acks = all

Kafka can require an adequate ISR before accepting strongly acknowledged writes.

If too few in-sync replicas remain, writes using this durability policy can fail rather than silently reduce the intended safety level. (Apache Kafka)

That is a feature, not merely a failure.

Kafka is saying:

I cannot currently provide the durability promise you requested.

8. Producer retries + idempotence

Networks fail.

Brokers can temporarily fail.

Responses can get lost.

So producers retry eligible failures.

But consider:

Producer sends Event A
        ↓
Broker writes A
        ↓
ACK gets lost
        ↓
Producer thinks send failed
        ↓
Producer retries A

Without protection:

A
A

may appear.

Idempotent production exists to suppress duplicate writes caused by producer retries under its guarantees. Current Kafka defaults enable producer idempotence when compatible settings are used. (Confluent Documentation)

Important:

Producer idempotence is not automatically the same as end-to-end exactly-once business processing.

Transactions and application design are separate advanced topics.

9. Consumers + partitions

Suppose we have six partitions:

P0 P1 P2 P3 P4 P5

One consumer:

Consumer 1
  ↓
all six partitions

Three consumers:

Consumer 1 → P0 P1
Consumer 2 → P2 P3
Consumer 3 → P4 P5

Six consumers:

one partition each

Eight consumers:

6 active partition assignments
2 consumers without partition work

That is why partition count influences maximum parallelism for a consumer group.

10. Consumer group + coordinator

The consumer group gives:

logical workers

The group coordinator gives:

coordination

Consumers send heartbeats so Kafka knows that members remain alive. Membership changes can cause partition reassignment/rebalancing. (Confluent Documentation)

11. Consumer + offset

Consumer says:

I am currently reading here.

Offset represents the location.

Committed offset provides a stored recovery checkpoint.

12. Offset + retention

This combination creates a powerful property.

Kafka can retain:

0 1 2 3 4 5 6 7 8 9 ...

Consumer may currently be at:

but if older records remain retained, it can intentionally move backward.

7
↓
3

Replay 3,4,5,6,7...

That is why Kafka is useful for reprocessing.

13. KRaft + broker + partition leader

KRaft controls cluster metadata.

Broker stores/serves records.

Partition leader handles a partition’s normal client traffic.

Do not mix them up:

KRaft Controller
      =
cluster metadata control

Partition Leader
      =
data-serving responsibility for one partition

14. All components together

                    CONTROL PLANE

                 KRaft Controllers
                       ↓
                 Cluster Metadata


                     DATA PLANE

Application
    ↓
Producer
    ↓
Serializer
    ↓
Partitioner
    ↓
Topic
    ↓
Partition Leader
    ↓
Broker
    ↓
Replica Followers / ISR
    ↓
Consumer Group
    ↓
Consumer
    ↓
Deserializer
    ↓
Application
    ↓
Offset Commit

That picture is worth mastering.

Tutorial 5 — Ways to Get a Kafka Cluster Up and Running

There is no single way to run Kafka.

The correct choice depends on whether your objective is:

learning
development
testing
staging
production
large enterprise operations

Current Confluent Platform supports local Docker/archives for development and several production installation paths, while Confluent Cloud provides the managed option. (Confluent Documentation)

Method 1 — Confluent Cloud

This is our primary training environment.

Student Laptop
     ↓
Internet
     ↓
Confluent Cloud
     ↓
Managed Kafka Cluster

Confluent manages much of:

broker infrastructure
platform operations
scaling mechanisms
managed security capabilities
service reliability

The current quick start supports creating a cluster, topic, producer and consumer through Confluent Cloud workflows. (Confluent Documentation)

Best for

Training
Application developers
Cloud-first architectures
Teams that do not want to operate Kafka brokers
Production workloads

For this course, this is the environment students should treat as their primary lab.

Method 2 — Confluent Platform locally

Confluent Platform can run locally for development.

Options include:

Docker
Docker Compose
ZIP/TAR installation
Confluent local tooling

Current Confluent quick-start documentation uses KRaft-based local environments and explicitly states that confluent local is for single-node development, not production. (Confluent Documentation)

Excellent for understanding what components actually run.

Method 3 — Docker Compose

Example conceptual architecture:

Laptop

Docker
 ├── Kafka
 ├── Schema Registry
 ├── Connect
 └── Control Center

Advantages:

Reproducible
Quick setup
Easy reset
Good for workshops
Good CI/dev testing

But a laptop Docker Compose environment should not teach students that “three containers = production Kafka.”

Production concerns include persistent storage, networking, resource management, security and failure-domain design. Confluent recommends persistent external volumes for Kafka data when using its Docker images. (Confluent Documentation)

Method 4 — Kafka/Confluent Platform on VMs or bare metal

Example:

VM 1 → Kafka Broker
VM 2 → Kafka Broker
VM 3 → Kafka Broker

VM 4 → KRaft Controller
VM 5 → KRaft Controller
VM 6 → KRaft Controller

This gives a team substantial control over:

CPU
RAM
disk
network
security
OS
JVM
broker configuration
failure domains

but also substantial operational responsibility.

Method 5 — Kubernetes

Kafka can run on Kubernetes.

In the Confluent ecosystem:

Confluent for Kubernetes (CFK) can deploy and manage Confluent Platform components declaratively. Current CFK documentation includes separate KRaft controllers and Kafka broker deployment. (Confluent Documentation)

Conceptually:

Kubernetes Cluster

Kafka Broker Pods
KRaft Controller Pods
Schema Registry Pods
Connect Pods
Control Center
Persistent Volumes
Services
Secrets

Best suited to organizations that already have strong Kubernetes operational expertise.

Method 6 — Automated VM/bare-metal deployment

Confluent supports Ansible-based deployment as another orchestrated installation method. (Confluent Documentation)

This can be appropriate where a company operates:

VMs
cloud instances
traditional data centers

but does not want Kubernetes.

Method 7 — Other managed Kafka services

Other cloud providers and vendors offer managed Kafka-compatible services.

Conceptually:

You manage:
Applications
Topics
Clients
Data contracts

Provider manages:
Much of broker infrastructure

The exact feature and compatibility tradeoffs should always be evaluated before selecting one.

Recommended learning progression

For students:

Stage 1
Confluent Cloud
        ↓
Learn Kafka

Stage 2
Local Docker
        ↓
See Kafka components

Stage 3
Multi-broker environment
        ↓
Understand failures

Stage 4
Production architecture
        ↓
Security
Monitoring
Capacity
HA
DR

Do not spend the first week fighting Linux networking before students understand what a partition is.

Tutorial 6 — Complete Producer → Kafka Cluster → Consumer Journey

This is the most important tutorial in this module.

We will follow one record from the moment an application creates it until a consumer processes it.

Our example:

{
  "vehicle_id": "V1001",
  "speed": 88,
  "battery": 67,
  "timestamp": "2026-08-19T06:30:00Z"
}

Topic:

vehicle-telemetry

Key:

V1001

Suppose:

Partitions = 6
Replication Factor = 3

FLOW 1 — Application creates the business event

The real event is:

Vehicle V1001 changed speed to 88.

Our software represents it as an object.

Real-world event
       ↓
Application Object

Kafka has not yet done anything.

FLOW 2 — Producer receives the record

The application calls its Kafka producer.

Conceptually:

producer.send(
    topic = vehicle-telemetry,
    key   = V1001,
    value = telemetry-event
)

Now the Kafka producer client becomes responsible for efficiently delivering that record.

FLOW 3 — Serialization

The producer cannot simply send an arbitrary programming-language object.

Object
  ↓
Serializer
  ↓
Byte representation

Possible formats:

String
JSON
Avro
Protobuf
JSON Schema
custom binary

Production lesson

Serialization affects:

payload size
CPU
network usage
schema evolution
compatibility

For enterprise systems, data format is an architectural decision—not a cosmetic one.

FLOW 4 — Schema management, when used

If using Schema Registry:

Producer
   ↓
Schema-aware serializer
   ↓
Schema Registry
   ↓
Encoded record
   ↓
Kafka

Schema Registry does not normally sit in the record’s Kafka data path like a proxy.

The producer’s serializer interacts with it for schema management; Kafka still carries the serialized event.

FLOW 5 — Producer needs metadata

The producer begins with Kafka bootstrap information.

bootstrap endpoint
      ↓
Kafka connection
      ↓
metadata

The important point:

bootstrap.servers does not mean:

send every message permanently to this one server.

It means:

give the client initial Kafka endpoints so it can discover the cluster.

After metadata discovery, the producer learns partition leaders.

FLOW 6 — Producer chooses a partition

Our key is:

V1001

The producer’s partitioning strategy selects a partition.

Suppose:

hash(V1001)
     ↓
Partition 4

Now every correctly partitioned event with key V1001 should continue mapping consistently according to the partitioning scheme, which is useful for per-vehicle ordering. Kafka guarantees event order inside a topic-partition. (Apache Kafka)

FLOW 7 — Why partition choice is critical

Suppose you use:

vehicle_id

as key.

You get:

V1001 → P4
V1001 → P4
V1001 → P4

Great for vehicle ordering.

But suppose one vehicle generates 40% of your entire traffic.

Then:

P4 = extremely hot

while:

P0 P1 P2 P3 P5

may remain comparatively quiet.

This is called a hot partition problem.

So key design affects:

ordering
load distribution
consumer parallelism
throughput

This is why partitioning is architecture.

FLOW 8 — Record enters producer buffer

The producer does not necessarily make a network request immediately for each record.

Records can enter producer memory.

Record
  ↓
Producer Buffer

The producer has a finite buffer; if applications generate data faster than Kafka can accept it for long enough, backpressure/blocking and eventually send failures can occur depending on producer configuration. (Confluent Documentation)

FLOW 9 — Producer batching

Records destined for the same partition can be grouped.

P4 Records

A
B
C
D
E
 ↓
Batch

Instead of five network requests:

A → request
B → request
C → request
D → request
E → request

we may perform:

ABCDE → one larger request

This is enormously important for Kafka throughput.

FLOW 10 — `batch.size`

batch.size influences the producer’s target batch sizing.

Larger effective batches may:

improve throughput
reduce request overhead
improve compression

but can require more buffering and should be evaluated together with workload characteristics.

FLOW 11 — `linger.ms`

What if the batch isn’t full?

The producer may wait briefly:

Record A arrives

wait...
B arrives
C arrives

send ABC

linger.ms places an upper delay on this batching opportunity; current Kafka defaults use a small linger rather than necessarily sending every underfilled batch instantly.

Tradeoff

Higher batching opportunity
        ↓
better throughput

but possibly

slightly higher record latency

Performance tuning is almost always a tradeoff.

FLOW 12 — Compression

Producer batches can be compressed.

100 KB events
    ↓
compression
    ↓
smaller network payload

Potential benefit:

less network
less storage
higher effective throughput

Potential cost:

CPU

Choose based on workload.

FLOW 13 — Producer sends to partition leader

Metadata says:

vehicle-telemetry
Partition 4
Leader = Broker 2

Therefore:

Producer
   ↓
Broker 2

No random broker forwarding is required for normal writes; Kafka clients know the partition leader and communicate with it directly. (Apache Kafka)

FLOW 14 — Broker validates the request

Before happily writing data, real production environments can involve:

Authentication
Authorization
Request limits
Quotas
Message-size limits
Protocol validation

For Confluent Cloud, client authentication and networking are part of the connection design.

FLOW 15 — Leader appends to partition log

Broker 2 owns the current leader replica for P4.

Before:

After:

100
101
102
103
104 ← new record

Kafka adds data to the partition log rather than performing arbitrary in-place record updates.

FLOW 16 — Offset assignment

Our record gets:

Topic     vehicle-telemetry
Partition 4
Offset    104

Its Kafka identity is therefore:

vehicle-telemetry / 4 / 104

Remember:

offset 104 only has meaning inside Partition 4.

Partition 2 can independently also have offset 104.

FLOW 17 — Storage

The partition log is stored using Kafka’s broker storage.

Kafka’s architecture is built around durable partition logs.

Conceptually:

Broker
  ↓
Partition directory
  ↓
Log segments
  ↓
Filesystem / storage

FLOW 18 — Replication begins

We configured:

RF = 3

Suppose:

Broker 2 → P4 Leader
Broker 1 → P4 Follower
Broker 3 → P4 Follower

The followers fetch/copy the leader’s log. (Apache Kafka)

FLOW 19 — ISR matters

Healthy state:

ISR = [Broker2, Broker1, Broker3]

One follower falls too far behind:

ISR = [Broker2, Broker1]

The replica may still physically exist, but it is no longer considered in sync.

That distinction matters enormously for availability and durability.

FLOW 20 — `min.insync.replicas`

Suppose:

RF = 3
min.insync.replicas = 2
acks = all

Healthy:

ISR = 3
write succeeds

One replica unavailable:

ISR = 2
write may still succeed

Too few ISR:

ISR = 1

Kafka can reject the strongly acknowledged write because it cannot satisfy the configured durability policy. (Apache Kafka)

Production principle

Sometimes:

Rejecting a write

is safer than:

Accepting a write without the promised durability.

FLOW 21 — `acks`

Now we decide when the producer receives success.

`acks=0`

Producer
   ↓
send
   ↓
does not wait for acknowledgment

Fastest/weakest feedback.

`acks=1`

Producer
   ↓
Leader writes
   ↓
Leader ACK

`acks=all`

Producer
   ↓
Leader
   ↓
Required ISR replication condition
   ↓
ACK

For production systems where event loss matters, acks=all together with appropriate replication and ISR settings is central to durability design. (Confluent Documentation)

FLOW 22 — ACK travels back

Kafka
   ↓
ProduceResponse
   ↓
Producer

The producer now knows whether the request succeeded or failed.

FLOW 23 — What if the network loses the ACK?

Interesting failure:

Producer → Broker
           writes record

Broker → ACK
           X
       network failure

Producer may not know whether Kafka received the record.

That creates retry ambiguity.

FLOW 24 — Retries

Producer can retry transient failures.

But retrying creates a question:

Did the first attempt actually succeed?

This is one reason idempotent production is so valuable.

FLOW 25 — Idempotent producer

With idempotent production, Kafka’s producer protocol can protect against duplicate records caused by retries within its guarantees.

Current Kafka enables idempotence by default when compatible settings are used, and idempotence requires acks=all, retries, and an allowed in-flight request configuration.

Again:

Idempotent producer
≠
automatically exactly-once business workflow

We will cover Kafka transactions later.

FLOW 26 — What if the leader fails?

Imagine:

P4 Leader = Broker 2

Broker 2
   X

Kafka’s metadata/control system can elect an eligible replacement leader.

For example:

Before

Broker2 Leader
Broker1 Follower
Broker3 Follower

After failure

Broker1 Leader
Broker3 Follower

The producer eventually refreshes metadata and learns the new leader.

This is Kafka availability in action.

FLOW 27 — KRaft’s role

KRaft controllers manage the metadata necessary to coordinate this cluster state and partition leadership.

They are not carrying every producer record.

Think:

KRaft
=
Who owns what?

Broker data path
=
Move and store actual events.

FLOW 28 — Record remains in Kafka

Now our record is safely stored:

vehicle-telemetry
P4
Offset 104

Consumer has not read it yet.

That is fine.

Producer and consumer are decoupled.

FLOW 29 — Consumer starts

Consumer configuration contains conceptually:

bootstrap information
group.id
deserializers
security
fetch-related settings
offset behavior

Suppose:

group.id = telemetry-analytics

FLOW 30 — Consumer joins group

Consumer asks Kafka:

I want to join telemetry-analytics.

Kafka identifies the group’s coordinator.

The coordinator is tied to the broker leading the relevant partition of __consumer_offsets. (Confluent Documentation)

FLOW 31 — Group membership

Suppose:

Consumer Group: telemetry-analytics

Consumer A
Consumer B
Consumer C

Topic:

6 partitions

Kafka assigns partitions among group members.

For example:

Consumer A → P0 P1
Consumer B → P2 P3
Consumer C → P4 P5

FLOW 32 — Our event goes to Consumer C

Our record is:

P4 / Offset 104

Consumer C owns P4.

So:

Consumer C
    ↓
fetch P4

FLOW 33 — Consumer polling

Kafka consumers repeatedly poll/fetch.

while running:

    poll()
      ↓
    process records
      ↓
    poll again

Consumer fetch requests include offsets, and brokers can return batches of records beginning from the requested position. (Confluent Documentation)

FLOW 34 — Consumer fetch performance

Consumers also use batching.

Important concepts include:

fetch.min.bytes
fetch.max.bytes
max.partition.fetch.bytes
max.poll.records

For example, increasing fetch.min.bytes can let a broker wait for more data before responding, potentially improving throughput at the cost of additional latency.

Again:

throughput ↔ latency

tradeoff.

FLOW 35 — Deserialization

Consumer receives bytes.

Bytes
  ↓
Deserializer
  ↓
Application Object

If schemas are involved:

Schema-aware deserializer

reconstructs data according to the serialization system.

FLOW 36 — Application processing

Now:

{
 vehicle_id: V1001,
 speed: 88
}

reaches application logic.

Maybe:

if speed > speed_limit:
    create alert

or:

update analytics

or:

store in database

Kafka’s job and your application’s job are different.

FLOW 37 — Offset commit

After processing, the consumer needs a recovery checkpoint.

Suppose processed:

P4 through Offset 104

It commits progress.

Conceptually:

Consumer
   ↓
Group Coordinator
   ↓
__consumer_offsets

Kafka stores the committed group position. (Confluent Documentation)

FLOW 38 — Why commit timing matters

Consider this dangerous order:

1. Commit offset
2. Process event

Then:

commit succeeds
application crashes before processing

On restart Kafka may think:

104 already handled

although your business operation did not occur.

Possible loss from the application’s perspective.

Now reverse it:

1. Process event
2. Commit offset

Crash between steps:

business processing succeeds
commit fails

Consumer may process it again.

Possible duplicate processing.

This leads us directly to advanced Kafka delivery semantics:

at-most-once
at-least-once
exactly-once approaches

We will cover these separately because they deserve their own tutorial.

FLOW 39 — Consumer restart

Suppose Consumer C dies.

The group can rebalance.

Before

A → P0 P1
B → P2 P3
C → P4 P5

C dies

Potential reassignment:

A → P0 P1 P4
B → P2 P3 P5

The new owner resumes according to committed offsets.

Consumer groups and rebalancing are therefore part of availability on the consumption side. (Confluent Documentation)

FLOW 40 — Heartbeats

Consumers must maintain group membership.

Conceptually:

Consumer
   ↓
heartbeat
   ↓
Coordinator

"I'm alive."

If a consumer disappears long enough, Kafka can remove it from the group and redistribute partitions. (Confluent Documentation)

FLOW 41 — Retention means replay

Imagine our analytics code had a bug for six hours.

Fixed version deployed.

If Kafka still retains the relevant data:

Reset/reposition consumer
       ↓
Older offset
       ↓
Replay events
       ↓
Rebuild analytics

This is one of Kafka’s most valuable operational capabilities.

The complete journey

Now put all 41 steps into one mental model:

BUSINESS EVENT
     ↓
Application
     ↓
ProducerRecord
     ↓
Serializer
     ↓
Schema handling
     ↓
Metadata lookup
     ↓
Partition selection
     ↓
Producer buffer
     ↓
Batching
     ↓
Compression
     ↓
Network
     ↓
Partition Leader
     ↓
Broker
     ↓
Append to Log
     ↓
Offset
     ↓
Follower Replication
     ↓
ISR
     ↓
acks / min.insync.replicas
     ↓
Producer ACK
     ↓

        EVENT STORED IN KAFKA

     ↓
Consumer Group
     ↓
Group Coordinator
     ↓
Partition Assignment
     ↓
Consumer Poll / Fetch
     ↓
Partition Leader
     ↓
Record Batch
     ↓
Deserializer
     ↓
Application Processing
     ↓
Offset Commit
     ↓
__consumer_offsets

Performance optimization map

When performance is the goal, investigate:

Area	Important concepts
Producer	batching
Producer	`linger.ms`
Producer	`batch.size`
Producer	compression
Producer	buffer capacity
Topic	partition count
Broker	capacity and distribution
Consumer	consumer count
Consumer	fetch sizing
Consumer	processing speed
Application	record size
Network	bandwidth/latency

Producer batching and consumer fetching are deliberate parts of Kafka’s throughput-oriented design. (Confluent Documentation)

Reliability optimization map

When reliability is the goal:

Area	Important concept
Topic	Replication factor
Broker	Replica health
Partition	ISR
Topic/Broker	`min.insync.replicas`
Producer	`acks=all`
Producer	retries
Producer	idempotence
Consumer	correct offset strategy
Application	idempotent processing
Schema	compatibility
Operations	monitoring

Availability optimization map

When availability is the goal:

Area	Important concept
Cluster	multiple brokers
Metadata	redundant KRaft controllers
Topic	replicas
Partition	leader election
Replica	healthy ISR
Infrastructure	rack/AZ placement
Consumer	multiple group members
Networking	redundant paths
Operations	alerts
Recovery	tested failure procedures

Current KRaft documentation recommends multiple controllers for redundant production deployments, while Confluent Cloud provides multi-zone deployment options for workloads that need zone-level availability. (Apache Kafka)

The configuration triangle students must remember

Kafka tuning is rarely:

turn setting HIGH
=
better

Instead:

                PERFORMANCE
                    /\
                   /  \
                  /    \
                 /      \
                /        \
       RELIABILITY ------ AVAILABILITY

For example:

More replication
→ better durability/availability
→ more network/storage work

More batching
→ better throughput
→ potentially more latency

More partitions
→ greater parallelism
→ more metadata and operational overhead

acks=all
→ stronger write durability
→ waits for stronger acknowledgment conditions

More consumers
→ more processing parallelism
→ useful only up to partition-level parallelism

A Kafka engineer’s job is not to maximize one setting.

It is to design the correct balance for the workload.

Production scenario exercise

Assume we have:

500,000 vehicles

Each vehicle sends:
1 event every 5 seconds

Topic:
vehicle-telemetry

Requirement:
No unacceptable loss of telemetry

Analytics latency target:
< 5 seconds

Cluster must survive:
one infrastructure failure

Students should start asking:

What event rate do we expect?

What is the average record size?

How many partitions do we need?

What key should we use?

What replication strategy?

What acknowledgment policy?

What minimum ISR?

What consumer concurrency?

What happens if a broker fails?

What happens if a consumer fails?

How long should events be retained?

How will we monitor producer lag/errors?

How will we monitor consumer lag?

How will we handle schema evolution?

That transition—from asking “What command creates a topic?” to asking “What architecture satisfies this workload?”—is the path from beginner Kafka knowledge to Kafka mastery.

Final Architecture Master Checklist

A student should now be able to explain every box below:

[Application]
     ↓
[Producer]
     ↓
[Serializer]
     ↓
[Schema]
     ↓
[Metadata]
     ↓
[Key]
     ↓
[Partitioner]
     ↓
[Producer Buffer]
     ↓
[Batch]
     ↓
[Compression]
     ↓
[Network]
     ↓
[Broker]
     ↓
[Topic]
     ↓
[Partition Leader]
     ↓
[Partition Log]
     ↓
[Offset]
     ↓
[Replica Followers]
     ↓
[ISR]
     ↓
[Replication Factor]
     ↓

[acks]

↓

[min.insync.replicas]

↓ [KRaft Controller] ↓ [Consumer Group] ↓ [Group Coordinator] ↓ [Partition Assignment] ↓ [Consumer] ↓ [Fetch/Poll] ↓ [Deserializer] ↓ [Processing] ↓ [Offset Commit] ↓ [__consumer_offsets] ↓ [Retention / Replay]

If a student truly understands why every component in this chain exists, later topics such as producer tuning, consumer tuning, replication, delivery semantics, transactions, Kafka security, monitoring, capacity planning and disaster recovery become dramatically easier.

This is the foundation we want before moving deeper into Kafka configuration.

rajeshkumar

Kafka Architecture

1. Learning objectives

2. Start with the simplest Kafka architecture

3. Kafka has two major architectural areas

Data plane

Control plane

4. Kafka Cluster

5. Broker

6. Topic

7. Partition

8. Why partitions exist

9. Where partitions physically live

10. Replication

11. Leader and Followers

12. ISR — In-Sync Replicas

13. KRaft Controllers

14. Architecture viewed through three goals

Performance

Reliability

Availability

Architecture knowledge check

Tutorial 2 — Components of Kafka

1. Producer

2. Serializer

3. Record key

4. Topic

5. Partition

6. Offset

7. Broker

8. Cluster

9. Replica

10. Leader

11. Follower

12. ISR

13. Replication factor

14. KRaft Controller

15. Consumer

16. Consumer Group

17. Group Coordinator

18. Consumer Offset

19. __consumer_offsets

20. Retention

21. Log segments

22. Schema Registry

23. Kafka Connect

24. Stream processing

Component relationship map

Tutorial 3 — How Kafka Works

1. Application creates an event

2. Producer creates a Kafka record

3. Serialization happens

4. Producer obtains cluster metadata

5. Partition selection

6. Producer batches records

7. Compression may occur

8. Producer sends to partition leader

9. Leader appends record

10. Offset is assigned

11. Followers replicate

12. Producer acknowledgment

13. Consumers fetch data

14. Consumer processes events

15. Consumer commits progress

16. Kafka retains the original record

Tutorial 4 — How All Kafka Components Work Together

1. Producer + topic

2. Topic + partition

3. Partition + key

4. Partition + broker

5. Partition + replication

6. Replication + ISR

7. ISR + acks + min.insync.replicas

8. Producer retries + idempotence

9. Consumers + partitions

10. Consumer group + coordinator

11. Consumer + offset

12. Offset + retention

13. KRaft + broker + partition leader

14. All components together

Tutorial 5 — Ways to Get a Kafka Cluster Up and Running

19. `__consumer_offsets`

FLOW 10 — `batch.size`

FLOW 11 — `linger.ms`

FLOW 20 — `min.insync.replicas`

FLOW 21 — `acks`

`acks=0`

`acks=1`

`acks=all`