{"id":1139,"date":"2026-08-19T01:20:26","date_gmt":"2026-08-19T01:20:26","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/?p=1139"},"modified":"2026-08-19T01:20:27","modified_gmt":"2026-08-19T01:20:27","slug":"kafka-master-tutorials-series-2-how-kafka-works-its-architecture-workflow","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/kafka-master-tutorials-series-2-how-kafka-works-its-architecture-workflow\/","title":{"rendered":"Kafka Master Tutorials Series: 2 \u2013 How Kafka Works &amp; its Architecture &amp; workflow"},"content":{"rendered":"\n<h1 class=\"wp-block-heading\">Kafka Architecture<\/h1>\n\n\n\n<h2 class=\"wp-block-heading\">1. Learning objectives<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">By the end of this tutorial, a student should understand:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>what a Kafka cluster actually contains<\/li>\n\n\n\n<li>producer side vs Kafka cluster side vs consumer side<\/li>\n\n\n\n<li>brokers, topics, partitions and replicas<\/li>\n\n\n\n<li>leader and follower replicas<\/li>\n\n\n\n<li>KRaft controllers<\/li>\n\n\n\n<li>Kafka&#8217;s data plane and control plane<\/li>\n\n\n\n<li>where Kafka stores data<\/li>\n\n\n\n<li>how Kafka provides scalability<\/li>\n\n\n\n<li>how Kafka provides reliability<\/li>\n\n\n\n<li>how Kafka provides availability<\/li>\n\n\n\n<li>how this architecture maps to Confluent Kafka<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"sandbox:\/mnt\/data\/kafka_architecture_training_infographic.png\" alt=\"Kafka Architecture\"\/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">2. Start with the simplest Kafka architecture<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">At the highest level:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nKafka Cluster\n   \u2193\nConsumer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Vehicle\n   \u2193\nTelematics Application\n   \u2193\nKafka\n   \u2193\nAnalytics Application<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">But a real Kafka architecture contains many more pieces:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>                    KAFKA CLUSTER\n\n               \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\nProducer \u2500\u2500\u2500\u2500\u2500\u2192\u2502 Broker 1           \u2502\n               \u2502 Broker 2           \u2502\u2500\u2500\u2500\u2500\u2500\u2192 Consumer\n               \u2502 Broker 3           \u2502\n               \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                       \u2502\n                       \u2502\n                   Topics\n                       \u2502\n                   Partitions\n                       \u2502\n                    Replicas<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka distributes topic partitions across brokers, and partitions can have replicas on multiple brokers for fault tolerance. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">3. Kafka has two major architectural areas<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A very useful master-level way of understanding Kafka is to separate it into:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>DATA PLANE\n\nProducer\n   \u2193\nBroker\n   \u2193\nPartition\n   \u2193\nConsumer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">and:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CONTROL PLANE\n\nKRaft Controllers\n       \u2193\nCluster metadata\n       \u2193\nPartition leaders\nBroker membership\nTopic configuration\nReplica information<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Data plane<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The data plane handles the actual events.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Examples:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>GPS event\nPayment event\nOrder event\nUser-click event<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer writes these records to brokers and consumers fetch them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Control plane<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The control plane manages information <strong>about the Kafka cluster<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Which brokers exist?\nWhich topics exist?\nHow many partitions exist?\nWhich broker leads Partition 3?\nWhere are replicas located?\nWhich brokers are alive?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Modern Kafka uses <strong>KRaft<\/strong> for this metadata\/control-plane function. Current Kafka documentation recommends separate broker and controller roles for critical deployments; combined broker\/controller mode is mainly appropriate for smaller or development environments. (<a href=\"https:\/\/kafka.apache.org\/43\/operations\/kraft\/?utm_source=chatgpt.com\">kafka.apache.org<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">4. Kafka Cluster<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A Kafka cluster is a collection of Kafka servers working together.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Those Kafka servers are called:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>brokers<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Kafka Cluster\n\n+----------+\n| Broker 1 |\n+----------+\n\n+----------+\n| Broker 2 |\n+----------+\n\n+----------+\n| Broker 3 |\n+----------+<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A production cluster normally contains multiple brokers because a single broker creates obvious limits for scale and availability.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">5. Broker<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A broker is a Kafka server.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Its important responsibilities include:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Receive producer requests\nStore partition data\nServe consumer fetch requests\nParticipate in replication\nMaintain partition replicas\nRespond to metadata-related client requests<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Think of a broker as:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Kafka Broker\n    =\nNetwork Server\n+\nPartition Storage\n+\nReplication Participant<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">6. Topic<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A <strong>topic<\/strong> is a logical stream\/category of events.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Examples:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry\norders\npayments\ncustomer-events\ncharging-events\nsystem-events<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic = vehicle-telemetry<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It may receive:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\"vehicle_id\":\"V101\",\"speed\":60}\n{\"vehicle_id\":\"V205\",\"speed\":74}\n{\"vehicle_id\":\"V101\",\"speed\":63}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">But the topic is only the logical view.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Internally, Kafka divides topics into <strong>partitions<\/strong>. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">7. Partition<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose we create:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">with three partitions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry\n\n\u251c\u2500\u2500 Partition 0\n\u251c\u2500\u2500 Partition 1\n\u2514\u2500\u2500 Partition 2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Each partition is an ordered log.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 0\n\nOffset\n0 \u2192 Event A\n1 \u2192 Event B\n2 \u2192 Event C\n3 \u2192 Event D<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka guarantees ordering <strong>within a partition<\/strong>, not automatically across every partition of a multi-partition topic. Events using the same key are normally routed consistently so that related records can remain within the same partition. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">8. Why partitions exist<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Partitions give Kafka parallelism.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Without partitions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic\n \u2193\none stream\n \u2193\nlimited parallelism<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With partitions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>               Topic\n\n       \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n       \u2193        \u2193        \u2193\n      P0       P1       P2\n       \u2193        \u2193        \u2193\n Consumer1 Consumer2 Consumer3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is one of Kafka&#8217;s fundamental scaling mechanisms.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">More partitions can allow more parallel producers, broker work and consumers, although blindly creating huge partition counts also has operational costs.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">9. Where partitions physically live<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Partitions are distributed across brokers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Broker 1\n  P0\n  P3\n\nBroker 2\n  P1\n  P4\n\nBroker 3\n  P2\n  P5<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore increasing broker capacity can allow Kafka to distribute storage and traffic.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">10. Replication<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Now imagine Broker 1 fails.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If Partition 0 exists only on Broker 1:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Broker 1\n   \u2193\nP0\n   \u2193\nBROKER FAILURE\n\nDATA UNAVAILABLE<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka addresses this using <strong>replicas<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example replication factor:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Replication Factor = 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 0\n\nBroker 1 \u2192 Replica\nBroker 2 \u2192 Replica\nBroker 3 \u2192 Replica<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka replication operates at the partition level. A replication factor of three is a common production choice. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">11. Leader and Followers<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka does not normally allow every replica to independently process writes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One replica is the:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>leader<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Other replicas are:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>followers<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 0\n\nBroker 1 \u2192 LEADER\nBroker 2 \u2192 FOLLOWER\nBroker 3 \u2192 FOLLOWER<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer writes to:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Leader<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Followers replicate the leader&#8217;s log.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nLeader\n   \u251c\u2500\u2500\u2500\u2500\u2192 Follower\n   \u2514\u2500\u2500\u2500\u2500\u2192 Follower<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka&#8217;s producer sends directly to the broker leading the target partition. (<a href=\"https:\/\/kafka.apache.org\/40\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">12. ISR \u2014 In-Sync Replicas<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">ISR means:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>In-Sync Replicas<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Think of it initially as:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">replicas sufficiently caught up with the leader to participate in Kafka&#8217;s durability and failover guarantees.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 0\n\nLeader     Broker 1  \u2713 ISR\nFollower   Broker 2  \u2713 ISR\nFollower   Broker 3  \u2713 ISR<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose Broker 3 becomes too far behind:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Leader     Broker 1  \u2713 ISR\nFollower   Broker 2  \u2713 ISR\nFollower   Broker 3  \u2717 outside ISR<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>min.insync.replicas<\/code> works together with producer acknowledgment settings to control how much replica health is required for strongly acknowledged writes. (<a href=\"https:\/\/kafka.apache.org\/41\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">13. KRaft Controllers<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Modern Kafka uses <strong>KRaft controllers<\/strong> rather than ZooKeeper for Kafka&#8217;s metadata management.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>              KRaft Controller Quorum\n                       \u2193\n              Cluster Metadata\n                       \u2193\n     \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n     \u2193                 \u2193                 \u2193\n Broker 1          Broker 2          Broker 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Controllers manage information such as broker membership, topics, partitions and leadership.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For critical deployments, Kafka recommends controller redundancy; a controller quorum of <code>2N + 1<\/code> can tolerate <code>N<\/code> controller failures. (<a href=\"https:\/\/kafka.apache.org\/43\/operations\/kraft\/?utm_source=chatgpt.com\">kafka.apache.org<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">14. Architecture viewed through three goals<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">This is extremely important.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Performance<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka obtains performance through things such as:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partitions\nParallelism\nSequential log writes\nBatching\nCompression\nEfficient network transfer\nConsumer batching<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Reliability<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Reliability comes from:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Replication\nISR\nacks\nmin.insync.replicas\nRetries\nIdempotent producer\nOffset management<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Availability<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Availability comes from:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Multiple brokers\nReplica placement\nLeader election\nMultiple KRaft controllers\nAvailability-zone\/rack awareness\nMonitoring\nFailover<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Throughout this series, always ask:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Does this configuration affect:\n\nPERFORMANCE?\nRELIABILITY?\nAVAILABILITY?\n\nor all three?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That is how production Kafka engineers think.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Architecture knowledge check<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Explain this without looking:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nTopic\n   \u2193\nPartition\n   \u2193\nLeader Broker\n   \u2193\nFollower Replicas\n   \u2193\nConsumer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If that makes sense, move to the next tutorial.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Tutorial 2 \u2014 Components of Kafka<\/h1>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"sandbox:\/mnt\/data\/apache_kafka_components_cheat_sheet.png\" alt=\"Kafka Components\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now let&#8217;s identify the building blocks individually.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">1. Producer<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A producer publishes records to Kafka.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Application\n    \u2193\nKafka Producer\n    \u2193\nTopic<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A record commonly contains:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic\nPartition (optional explicit choice)\nKey\nValue\nHeaders\nTimestamp<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"vehicleId\": \"V101\",\n  \"speed\": 82\n}<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">2. Serializer<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka sends bytes over the network.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Your application may have:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Java Object\nJSON Object\nString\nInteger\nProtobuf Object\nAvro Object<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A serializer converts application data to bytes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Object\n  \u2193\nSerializer\n  \u2193\nBytes\n  \u2193\nKafka<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With Confluent environments, Schema Registry can be added for schema-based formats and compatibility management.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">3. Record key<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A Kafka record may have a key.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Key = vehicle-101\nValue = GPS event<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Keys are extremely important because they commonly influence partition selection.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-101\n    \u2193\nhash(key)\n    \u2193\nPartition 2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Future events using that same key can then continue to the same partition, preserving their per-partition order. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">4. Topic<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Logical stream:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Not a physical single file.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It consists of partitions.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">5. Partition<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Physical unit of:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ordering\nparallelism\nstorage\nreplication<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is perhaps the most important Kafka component to master.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">6. Offset<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Every record in a partition has a position.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 2\n\nOffset 0 \u2192 A\nOffset 1 \u2192 B\nOffset 2 \u2192 C\nOffset 3 \u2192 D<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Offsets are unique <strong>within a partition<\/strong>, not globally across the whole topic.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">7. Broker<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Stores and serves partitions.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Broker 1\nBroker 2\nBroker 3<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">8. Cluster<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Collection of brokers and the associated Kafka control plane.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">9. Replica<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A copy of a partition.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 Replica \u2192 Broker 1\nP0 Replica \u2192 Broker 2\nP0 Replica \u2192 Broker 3<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">10. Leader<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The active replica that handles normal requests for its partition.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 Leader = Broker 2<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">11. Follower<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Copies partition data from its leader.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka&#8217;s replication design has followers consume\/copy the leader&#8217;s log. (<a href=\"https:\/\/kafka.apache.org\/34\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">12. ISR<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The in-sync replica set.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Important when deciding whether Kafka can safely acknowledge writes and which replicas are suitable for normal failover.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">13. Replication factor<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Number of replicas for each partition.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>RF = 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">means:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>1 leader\n+\n2 additional replicas\n\n= 3 copies<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">14. KRaft Controller<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Manages cluster metadata and leadership information.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do not confuse:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>KRaft Controller<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition Leader<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">They solve different problems.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">15. Consumer<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Reads records from Kafka.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Consumers <strong>pull\/fetch<\/strong> data from the partition leaders. (<a href=\"https:\/\/docs.confluent.io\/cloud\/current\/client-apps\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">16. Consumer Group<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumers can work together.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer Group: analytics\n\nConsumer 1\nConsumer 2\nConsumer 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If the topic has:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0\nP1\nP2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka could assign:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 \u2192 Consumer 1\nP1 \u2192 Consumer 2\nP2 \u2192 Consumer 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Within one consumer group, a partition is assigned to one consumer at a time. Consumer groups therefore provide parallel consumption. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">17. Group Coordinator<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A broker acts as coordinator for a particular consumer group.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It helps manage:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>group membership\nheartbeats\npartition assignments\/rebalancing\noffset commit validation<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The group ID maps to a partition of Kafka&#8217;s internal <code>__consumer_offsets<\/code> topic; the broker leading that partition becomes the coordinator for the group. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">18. Consumer Offset<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose the consumer processed through:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Offset 1050<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka can store the group&#8217;s committed position.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If the consumer restarts, it can continue near that position instead of necessarily restarting at zero.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">19. <code>__consumer_offsets<\/code><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka has an internal topic:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>__consumer_offsets<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It is used to store committed consumer-group offsets and participates in consumer-group coordination. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Students should recognize this name immediately.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">20. Retention<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka normally keeps records according to topic retention policies rather than deleting a record simply because one consumer read it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This enables:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Replay\nReprocessing\nNew consumers\nRecovery\nBackfills<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That behavior is a major difference from the simplistic mental model of a destructive work queue.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">21. Log segments<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A partition log is physically broken into segments rather than growing forever as a single giant file.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 0\n\n000000.log\n000001.log\n000002.log\n...<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Segments help Kafka manage retention and storage efficiently.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">22. Schema Registry<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Schema Registry is part of the broader Confluent ecosystem, not a required core broker component.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It helps manage schemas such as:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Avro\nJSON Schema\nProtobuf<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Think:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nSchema\n   \u2193\nKafka data\n   \u2193\nConsumer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It is extremely valuable when many teams independently produce and consume data.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">23. Kafka Connect<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka Connect is another surrounding component.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It moves data between Kafka and external systems.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>PostgreSQL\n    \u2193\nKafka Connect\n    \u2193\nKafka<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">or:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Kafka\n  \u2193\nKafka Connect\n  \u2193\nData Warehouse<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">24. Stream processing<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka can feed applications such as:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Kafka Streams\nFlink\nksqlDB\nother stream processors<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">These applications read streams, perform computation and frequently write results back to Kafka.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Component relationship map<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>Application\n    \u2193\nProducer\n    \u2193\nSerializer\n    \u2193\nKey\n    \u2193\nTopic\n    \u2193\nPartition\n    \u2193\nLeader Broker\n    \u2193\nReplica \/ ISR\n    \u2193\nConsumer Group\n    \u2193\nConsumer\n    \u2193\nDeserializer\n    \u2193\nApplication\n    \u2193\nOffset Commit<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Memorizing isolated definitions is not enough.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Mastery comes from understanding this relationship.<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Tutorial 3 \u2014 How Kafka Works<\/h1>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"sandbox:\/mnt\/data\/how_kafka_works_event_flow_infographic.png\" alt=\"How Kafka Works\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now we put the components into motion.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">1. Application creates an event<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"vehicleId\": \"V101\",\n  \"speed\": 92,\n  \"timestamp\": \"2026-08-19T06:30:00Z\"\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The application decides:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>I need to publish this event.<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">2. Producer creates a Kafka record<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic = vehicle-telemetry\nKey   = V101\nValue = event<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">3. Serialization happens<\/h1>\n\n\n\n<pre class=\"wp-block-code\"><code>Application Object\n       \u2193\n   Serializer\n       \u2193\n      Bytes<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka brokers primarily deal with bytes; application semantics come from producers, consumers and optionally schema-management systems.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">4. Producer obtains cluster metadata<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The producer is configured with bootstrap information.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example concept:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>bootstrap servers\n     \u2193\nConnect to Kafka\n     \u2193\nReceive metadata<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer learns:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Which topic partitions exist?\nWhich broker currently leads each partition?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Bootstrap servers are <strong>entry points<\/strong>, not necessarily the broker that receives every record.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">5. Partition selection<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The producer decides the destination partition.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Key = V101\n       \u2193\nPartitioning strategy\n       \u2193\nPartition 2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Same-key routing is important where per-entity ordering matters. Kafka&#8217;s ordering guarantee is per topic-partition. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">6. Producer batches records<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka producers are designed to batch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Instead of:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Event 1 \u2192 network\nEvent 2 \u2192 network\nEvent 3 \u2192 network<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">the producer can do:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Event 1\nEvent 2\nEvent 3\nEvent 4\n   \u2193\nBatch\n   \u2193\nNetwork request<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Producer settings such as <code>batch.size<\/code> and <code>linger.ms<\/code> influence batching. Modern Kafka producer documentation describes <code>linger.ms<\/code> as an upper waiting time to allow a batch to fill before sending.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">7. Compression may occur<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The batch may be compressed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Examples include:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>gzip\nsnappy\nlz4\nzstd<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Compression can reduce:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>network bandwidth\nbroker network load\nstorage footprint<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">at the cost of CPU.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">8. Producer sends to partition leader<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic: vehicle-telemetry\nPartition: 2\nLeader: Broker 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nBroker 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka intentionally has producers send to the leader directly rather than through a separate routing tier. (<a href=\"https:\/\/kafka.apache.org\/40\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">9. Leader appends record<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Broker 3 receives the produce request.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partition 2\n\n0\n1\n2\n3\n4\n5 \u2190 new record<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka is based around append-oriented partition logs.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">10. Offset is assigned<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose the new position is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Offset 8127<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The record can now be identified by:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic\n+\nPartition\n+\nOffset<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry \/ 2 \/ 8127<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">11. Followers replicate<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">If:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>RF = 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">we might have:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Broker 3 \u2192 Leader\nBroker 1 \u2192 Follower\nBroker 2 \u2192 Follower<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The followers copy records from the leader. (<a href=\"https:\/\/kafka.apache.org\/34\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">12. Producer acknowledgment<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Now Kafka determines when to reply successfully to the producer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Important producer setting:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>acks<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A simplified view:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>acks=0\nProducer does not wait for broker acknowledgment.\n\nacks=1\nLeader acknowledges.\n\nacks=all\nLeader waits for the required in-sync replication conditions.<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>acks=all<\/code> provides Kafka&#8217;s strongest producer acknowledgment mode and is required when producer idempotence is enabled. (<a href=\"https:\/\/docs.confluent.io\/platform\/7.7\/installation\/configuration\/producer-configs.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">13. Consumers fetch data<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumers do not normally wait for brokers to push arbitrary messages at them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">They issue fetch requests.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer\n   \u2193\nFetch from P2 starting at offset X\n   \u2193\nBroker\n   \u2193\nRecords<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The consumer controls its position and can rewind that position to re-read retained data. (<a href=\"https:\/\/docs.confluent.io\/cloud\/current\/client-apps\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">14. Consumer processes events<\/h1>\n\n\n\n<pre class=\"wp-block-code\"><code>Fetch\n  \u2193\nDeserialize\n  \u2193\nBusiness logic\n  \u2193\nDatabase \/ API \/ Analytics \/ Alert<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">15. Consumer commits progress<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">After processing records, the consumer may commit an offset.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\"I have safely processed records through this point.\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That progress is stored using Kafka&#8217;s consumer offset machinery, including <code>__consumer_offsets<\/code>. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">16. Kafka retains the original record<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consuming does not inherently delete the event.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer A reads it\nConsumer B can read it\nConsumer C can read it later\nConsumer A may replay it<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is central to Kafka&#8217;s event-log model.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Tutorial 4 \u2014 How All Kafka Components Work Together<\/h1>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"sandbox:\/mnt\/data\/how_kafka_components_work_together.png\" alt=\"Kafka Components Working Together\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This tutorial answers:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Why does Kafka need so many components?<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">Because each component solves a different scaling or reliability problem.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">1. Producer + topic<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Producer answers:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>WHO creates the event?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Topic answers:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>WHAT logical stream does it belong to?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Telematics Service\n       \u2193\nvehicle-telemetry<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">2. Topic + partition<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Topic gives logical organization.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Partition gives physical parallelism.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry\n\nP0\nP1\nP2\nP3\nP4\nP5<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">3. Partition + key<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The key helps decide:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>WHERE should this event go?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For vehicle systems:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Key = vehicle_id<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">is often useful when we want all events for one vehicle to retain ordering.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">4. Partition + broker<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Partitions have to physically live somewhere.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That somewhere is a broker.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 \u2192 Broker 1\nP1 \u2192 Broker 2\nP2 \u2192 Broker 3<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">5. Partition + replication<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">One physical copy is dangerous.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0\n\nLeader   Broker 1\nFollower Broker 2\nFollower Broker 3<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">6. Replication + ISR<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A replica existing is not enough.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka must know which replicas are sufficiently synchronized.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Hence:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ISR<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">7. ISR + acks + min.insync.replicas<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">These three are tightly connected.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Replication Factor = 3\n\nISR:\nBroker 1\nBroker 2\nBroker 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">and:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>min.insync.replicas = 2\nacks = all<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka can require an adequate ISR before accepting strongly acknowledged writes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If too few in-sync replicas remain, writes using this durability policy can fail rather than silently reduce the intended safety level. (<a href=\"https:\/\/kafka.apache.org\/41\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That is a <strong>feature<\/strong>, not merely a failure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka is saying:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">I cannot currently provide the durability promise you requested.<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">8. Producer retries + idempotence<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Networks fail.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Brokers can temporarily fail.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Responses can get lost.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So producers retry eligible failures.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But consider:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer sends Event A\n        \u2193\nBroker writes A\n        \u2193\nACK gets lost\n        \u2193\nProducer thinks send failed\n        \u2193\nProducer retries A<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Without protection:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>A\nA<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">may appear.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Idempotent production exists to suppress duplicate writes caused by producer retries under its guarantees. Current Kafka defaults enable producer idempotence when compatible settings are used. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/configuration\/producer-configs.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Important:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Producer idempotence is not automatically the same as end-to-end exactly-once business processing.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Transactions and application design are separate advanced topics.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">9. Consumers + partitions<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose we have six partitions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 P1 P2 P3 P4 P5<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">One consumer:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer 1\n  \u2193\nall six partitions<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Three consumers:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer 1 \u2192 P0 P1\nConsumer 2 \u2192 P2 P3\nConsumer 3 \u2192 P4 P5<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Six consumers:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>one partition each<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Eight consumers:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>6 active partition assignments\n2 consumers without partition work<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That is why partition count influences maximum parallelism for a consumer group.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">10. Consumer group + coordinator<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The consumer group gives:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>logical workers<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The group coordinator gives:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>coordination<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Consumers send heartbeats so Kafka knows that members remain alive. Membership changes can cause partition reassignment\/rebalancing. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">11. Consumer + offset<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer says:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>I am currently reading here.<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Offset represents the location.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Committed offset provides a stored recovery checkpoint.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">12. Offset + retention<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">This combination creates a powerful property.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka can retain:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>0 1 2 3 4 5 6 7 8 9 ...<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer may currently be at:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>7<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">but if older records remain retained, it can intentionally move backward.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>7\n\u2193\n3\n\nReplay 3,4,5,6,7...<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That is why Kafka is useful for reprocessing.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">13. KRaft + broker + partition leader<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">KRaft controls cluster metadata.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Broker stores\/serves records.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Partition leader handles a partition&#8217;s normal client traffic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do not mix them up:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>KRaft Controller\n      =\ncluster metadata control\n\nPartition Leader\n      =\ndata-serving responsibility for one partition<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">14. All components together<\/h1>\n\n\n\n<pre class=\"wp-block-code\"><code>                    CONTROL PLANE\n\n                 KRaft Controllers\n                       \u2193\n                 Cluster Metadata\n\n\n                     DATA PLANE\n\nApplication\n    \u2193\nProducer\n    \u2193\nSerializer\n    \u2193\nPartitioner\n    \u2193\nTopic\n    \u2193\nPartition Leader\n    \u2193\nBroker\n    \u2193\nReplica Followers \/ ISR\n    \u2193\nConsumer Group\n    \u2193\nConsumer\n    \u2193\nDeserializer\n    \u2193\nApplication\n    \u2193\nOffset Commit<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That picture is worth mastering.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Tutorial 5 \u2014 Ways to Get a Kafka Cluster Up and Running<\/h1>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"sandbox:\/mnt\/data\/kafka_cluster_deployment_options_guide.png\" alt=\"Kafka Deployment Options\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">There is no single way to run Kafka.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The correct choice depends on whether your objective is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>learning\ndevelopment\ntesting\nstaging\nproduction\nlarge enterprise operations<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Current Confluent Platform supports local Docker\/archives for development and several production installation paths, while Confluent Cloud provides the managed option. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/get-started\/platform-quickstart.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Method 1 \u2014 Confluent Cloud<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This is our primary training environment.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Student Laptop\n     \u2193\nInternet\n     \u2193\nConfluent Cloud\n     \u2193\nManaged Kafka Cluster<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent manages much of:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>broker infrastructure\nplatform operations\nscaling mechanisms\nmanaged security capabilities\nservice reliability<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The current quick start supports creating a cluster, topic, producer and consumer through Confluent Cloud workflows. (<a href=\"https:\/\/docs.confluent.io\/cloud\/current\/get-started\/index.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Best for<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>Training\nApplication developers\nCloud-first architectures\nTeams that do not want to operate Kafka brokers\nProduction workloads<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For this course, this is the environment students should treat as their primary lab.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Method 2 \u2014 Confluent Platform locally<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent Platform can run locally for development.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Options include:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Docker\nDocker Compose\nZIP\/TAR installation\nConfluent local tooling<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Current Confluent quick-start documentation uses KRaft-based local environments and explicitly states that <code>confluent local<\/code> is for single-node development, not production. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/get-started\/platform-quickstart.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Excellent for understanding what components actually run.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Method 3 \u2014 Docker Compose<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Example conceptual architecture:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Laptop\n\nDocker\n \u251c\u2500\u2500 Kafka\n \u251c\u2500\u2500 Schema Registry\n \u251c\u2500\u2500 Connect\n \u2514\u2500\u2500 Control Center<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Advantages:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Reproducible\nQuick setup\nEasy reset\nGood for workshops\nGood CI\/dev testing<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">But a laptop Docker Compose environment should not teach students that &#8220;three containers = production Kafka.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Production concerns include persistent storage, networking, resource management, security and failure-domain design. Confluent recommends persistent external volumes for Kafka data when using its Docker images. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/docker\/installation.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Method 4 \u2014 Kafka\/Confluent Platform on VMs or bare metal<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VM 1 \u2192 Kafka Broker\nVM 2 \u2192 Kafka Broker\nVM 3 \u2192 Kafka Broker\n\nVM 4 \u2192 KRaft Controller\nVM 5 \u2192 KRaft Controller\nVM 6 \u2192 KRaft Controller<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This gives a team substantial control over:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CPU\nRAM\ndisk\nnetwork\nsecurity\nOS\nJVM\nbroker configuration\nfailure domains<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">but also substantial operational responsibility.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Method 5 \u2014 Kubernetes<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka can run on Kubernetes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the Confluent ecosystem:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Confluent for Kubernetes (CFK)<\/strong> can deploy and manage Confluent Platform components declaratively. Current CFK documentation includes separate KRaft controllers and Kafka broker deployment. (<a href=\"https:\/\/docs.confluent.io\/operator\/current\/co-quickstart.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Kubernetes Cluster\n\nKafka Broker Pods\nKRaft Controller Pods\nSchema Registry Pods\nConnect Pods\nControl Center\nPersistent Volumes\nServices\nSecrets<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Best suited to organizations that already have strong Kubernetes operational expertise.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Method 6 \u2014 Automated VM\/bare-metal deployment<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent supports Ansible-based deployment as another orchestrated installation method. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/index.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This can be appropriate where a company operates:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VMs\ncloud instances\ntraditional data centers<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">but does not want Kubernetes.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Method 7 \u2014 Other managed Kafka services<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Other cloud providers and vendors offer managed Kafka-compatible services.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>You manage:\nApplications\nTopics\nClients\nData contracts\n\nProvider manages:\nMuch of broker infrastructure<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The exact feature and compatibility tradeoffs should always be evaluated before selecting one.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Recommended learning progression<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">For students:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Stage 1\nConfluent Cloud\n        \u2193\nLearn Kafka\n\nStage 2\nLocal Docker\n        \u2193\nSee Kafka components\n\nStage 3\nMulti-broker environment\n        \u2193\nUnderstand failures\n\nStage 4\nProduction architecture\n        \u2193\nSecurity\nMonitoring\nCapacity\nHA\nDR<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Do <strong>not<\/strong> spend the first week fighting Linux networking before students understand what a partition is.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Tutorial 6 \u2014 Complete Producer \u2192 Kafka Cluster \u2192 Consumer Journey<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">This is the most important tutorial in this module.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"sandbox:\/mnt\/data\/kafka_data_journey_infographic.png\" alt=\"Producer to Consumer Deep Dive\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">We will follow <strong>one record<\/strong> from the moment an application creates it until a consumer processes it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"vehicle_id\": \"V1001\",\n  \"speed\": 88,\n  \"battery\": 67,\n  \"timestamp\": \"2026-08-19T06:30:00Z\"\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Topic:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Key:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>V1001<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Partitions = 6\nReplication Factor = 3<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 1 \u2014 Application creates the business event<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The real event is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Vehicle V1001 changed speed to 88.<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Our software represents it as an object.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Real-world event\n       \u2193\nApplication Object<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka has not yet done anything.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 2 \u2014 Producer receives the record<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The application calls its Kafka producer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>producer.send(\n    topic = vehicle-telemetry,\n    key   = V1001,\n    value = telemetry-event\n)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Now the Kafka producer client becomes responsible for efficiently delivering that record.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 3 \u2014 Serialization<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The producer cannot simply send an arbitrary programming-language object.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Object\n  \u2193\nSerializer\n  \u2193\nByte representation<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Possible formats:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>String\nJSON\nAvro\nProtobuf\nJSON Schema\ncustom binary<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Production lesson<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Serialization affects:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>payload size\nCPU\nnetwork usage\nschema evolution\ncompatibility<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For enterprise systems, data format is an architectural decision\u2014not a cosmetic one.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 4 \u2014 Schema management, when used<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">If using Schema Registry:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nSchema-aware serializer\n   \u2193\nSchema Registry\n   \u2193\nEncoded record\n   \u2193\nKafka<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Schema Registry does <strong>not<\/strong> normally sit in the record&#8217;s Kafka data path like a proxy.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The producer&#8217;s serializer interacts with it for schema management; Kafka still carries the serialized event.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 5 \u2014 Producer needs metadata<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The producer begins with Kafka bootstrap information.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>bootstrap endpoint\n      \u2193\nKafka connection\n      \u2193\nmetadata<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The important point:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>bootstrap.servers<\/code> does <strong>not<\/strong> mean:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">send every message permanently to this one server.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">It means:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">give the client initial Kafka endpoints so it can discover the cluster.<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">After metadata discovery, the producer learns partition leaders.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 6 \u2014 Producer chooses a partition<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Our key is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>V1001<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer&#8217;s partitioning strategy selects a partition.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>hash(V1001)\n     \u2193\nPartition 4<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Now every correctly partitioned event with key <code>V1001<\/code> should continue mapping consistently according to the partitioning scheme, which is useful for per-vehicle ordering. Kafka guarantees event order inside a topic-partition. (<a href=\"https:\/\/kafka.apache.org\/documentation\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 7 \u2014 Why partition choice is critical<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose you use:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle_id<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">as key.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You get:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>V1001 \u2192 P4\nV1001 \u2192 P4\nV1001 \u2192 P4<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Great for vehicle ordering.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But suppose one vehicle generates 40% of your entire traffic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Then:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P4 = extremely hot<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">while:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 P1 P2 P3 P5<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">may remain comparatively quiet.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is called a <strong>hot partition problem<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So key design affects:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ordering\nload distribution\nconsumer parallelism\nthroughput<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is why partitioning is architecture.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 8 \u2014 Record enters producer buffer<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The producer does not necessarily make a network request immediately for each record.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Records can enter producer memory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Record\n  \u2193\nProducer Buffer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer has a finite buffer; if applications generate data faster than Kafka can accept it for long enough, backpressure\/blocking and eventually send failures can occur depending on producer configuration. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/configuration\/producer-configs.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 9 \u2014 Producer batching<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Records destined for the same partition can be grouped.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P4 Records\n\nA\nB\nC\nD\nE\n \u2193\nBatch<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Instead of five network requests:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>A \u2192 request\nB \u2192 request\nC \u2192 request\nD \u2192 request\nE \u2192 request<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">we may perform:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ABCDE \u2192 one larger request<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is enormously important for Kafka throughput.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 10 \u2014 <code>batch.size<\/code><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\"><code>batch.size<\/code> influences the producer&#8217;s target batch sizing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Larger effective batches may:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>improve throughput\nreduce request overhead\nimprove compression<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">but can require more buffering and should be evaluated together with workload characteristics.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 11 \u2014 <code>linger.ms<\/code><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">What if the batch isn&#8217;t full?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The producer may wait briefly:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Record A arrives\n\nwait...\nB arrives\nC arrives\n\nsend ABC<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>linger.ms<\/code> places an upper delay on this batching opportunity; current Kafka defaults use a small linger rather than necessarily sending every underfilled batch instantly.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Tradeoff<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>Higher batching opportunity\n        \u2193\nbetter throughput\n\nbut possibly\n\nslightly higher record latency<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Performance tuning is almost always a tradeoff.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 12 \u2014 Compression<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Producer batches can be compressed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>100 KB events\n    \u2193\ncompression\n    \u2193\nsmaller network payload<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Potential benefit:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>less network\nless storage\nhigher effective throughput<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Potential cost:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CPU<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Choose based on workload.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 13 \u2014 Producer sends to partition leader<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Metadata says:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry\nPartition 4\nLeader = Broker 2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Therefore:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nBroker 2<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">No random broker forwarding is required for normal writes; Kafka clients know the partition leader and communicate with it directly. (<a href=\"https:\/\/kafka.apache.org\/40\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 14 \u2014 Broker validates the request<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Before happily writing data, real production environments can involve:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Authentication\nAuthorization\nRequest limits\nQuotas\nMessage-size limits\nProtocol validation<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For Confluent Cloud, client authentication and networking are part of the connection design.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 15 \u2014 Leader appends to partition log<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Broker 2 owns the current leader replica for P4.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Before:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P4\n\n100\n101\n102\n103<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>100\n101\n102\n103\n104 \u2190 new record<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka adds data to the partition log rather than performing arbitrary in-place record updates.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 16 \u2014 Offset assignment<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Our record gets:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic     vehicle-telemetry\nPartition 4\nOffset    104<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Its Kafka identity is therefore:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry \/ 4 \/ 104<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Remember:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>offset 104 only has meaning inside Partition 4.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Partition 2 can independently also have offset 104.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 17 \u2014 Storage<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">The partition log is stored using Kafka&#8217;s broker storage.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka&#8217;s architecture is built around durable partition logs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Broker\n  \u2193\nPartition directory\n  \u2193\nLog segments\n  \u2193\nFilesystem \/ storage<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 18 \u2014 Replication begins<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">We configured:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>RF = 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Broker 2 \u2192 P4 Leader\nBroker 1 \u2192 P4 Follower\nBroker 3 \u2192 P4 Follower<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The followers fetch\/copy the leader&#8217;s log. (<a href=\"https:\/\/kafka.apache.org\/34\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 19 \u2014 ISR matters<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Healthy state:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ISR = &#91;Broker2, Broker1, Broker3]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">One follower falls too far behind:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ISR = &#91;Broker2, Broker1]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The replica may still physically exist, but it is no longer considered in sync.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That distinction matters enormously for availability and durability.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 20 \u2014 <code>min.insync.replicas<\/code><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>RF = 3\nmin.insync.replicas = 2\nacks = all<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Healthy:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ISR = 3\nwrite succeeds<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">One replica unavailable:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ISR = 2\nwrite may still succeed<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Too few ISR:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ISR = 1<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka can reject the strongly acknowledged write because it cannot satisfy the configured durability policy. (<a href=\"https:\/\/kafka.apache.org\/41\/design\/design\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Production principle<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Sometimes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Rejecting a write<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">is safer than:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Accepting a write without the promised durability.<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 21 \u2014 <code>acks<\/code><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Now we decide when the producer receives success.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><code>acks=0<\/code><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nsend\n   \u2193\ndoes not wait for acknowledgment<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Fastest\/weakest feedback.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><code>acks=1<\/code><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nLeader writes\n   \u2193\nLeader ACK<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><code>acks=all<\/code><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2193\nLeader\n   \u2193\nRequired ISR replication condition\n   \u2193\nACK<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For production systems where event loss matters, <code>acks=all<\/code> together with appropriate replication and ISR settings is central to durability design. (<a href=\"https:\/\/docs.confluent.io\/platform\/7.7\/installation\/configuration\/producer-configs.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 22 \u2014 ACK travels back<\/h1>\n\n\n\n<pre class=\"wp-block-code\"><code>Kafka\n   \u2193\nProduceResponse\n   \u2193\nProducer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer now knows whether the request succeeded or failed.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 23 \u2014 What if the network loses the ACK?<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Interesting failure:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer \u2192 Broker\n           writes record\n\nBroker \u2192 ACK\n           X\n       network failure<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Producer may not know whether Kafka received the record.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That creates retry ambiguity.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 24 \u2014 Retries<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Producer can retry transient failures.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But retrying creates a question:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Did the first attempt actually succeed?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is one reason idempotent production is so valuable.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 25 \u2014 Idempotent producer<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">With idempotent production, Kafka&#8217;s producer protocol can protect against duplicate records caused by retries within its guarantees.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Current Kafka enables idempotence by default when compatible settings are used, and idempotence requires <code>acks=all<\/code>, retries, and an allowed in-flight request configuration.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Idempotent producer\n\u2260\nautomatically exactly-once business workflow<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We will cover Kafka transactions later.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 26 \u2014 What if the leader fails?<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Imagine:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P4 Leader = Broker 2\n\nBroker 2\n   X<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka&#8217;s metadata\/control system can elect an eligible replacement leader.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Before\n\nBroker2 Leader\nBroker1 Follower\nBroker3 Follower\n\nAfter failure\n\nBroker1 Leader\nBroker3 Follower<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The producer eventually refreshes metadata and learns the new leader.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is Kafka availability in action.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 27 \u2014 KRaft&#8217;s role<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">KRaft controllers manage the metadata necessary to coordinate this cluster state and partition leadership.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">They are not carrying every producer record.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Think:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>KRaft\n=\nWho owns what?\n\nBroker data path\n=\nMove and store actual events.<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 28 \u2014 Record remains in Kafka<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Now our record is safely stored:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>vehicle-telemetry\nP4\nOffset 104<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer has not read it yet.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That is fine.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Producer and consumer are decoupled.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 29 \u2014 Consumer starts<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer configuration contains conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>bootstrap information\ngroup.id\ndeserializers\nsecurity\nfetch-related settings\noffset behavior<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>group.id = telemetry-analytics<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 30 \u2014 Consumer joins group<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer asks Kafka:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>I want to join telemetry-analytics.<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka identifies the group&#8217;s coordinator.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The coordinator is tied to the broker leading the relevant partition of <code>__consumer_offsets<\/code>. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 31 \u2014 Group membership<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer Group: telemetry-analytics\n\nConsumer A\nConsumer B\nConsumer C<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Topic:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>6 partitions<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka assigns partitions among group members.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer A \u2192 P0 P1\nConsumer B \u2192 P2 P3\nConsumer C \u2192 P4 P5<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 32 \u2014 Our event goes to Consumer C<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Our record is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P4 \/ Offset 104<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer C owns P4.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer C\n    \u2193\nfetch P4<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 33 \u2014 Consumer polling<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka consumers repeatedly poll\/fetch.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>while running:\n\n    poll()\n      \u2193\n    process records\n      \u2193\n    poll again<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer fetch requests include offsets, and brokers can return batches of records beginning from the requested position. (<a href=\"https:\/\/docs.confluent.io\/cloud\/current\/client-apps\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 34 \u2014 Consumer fetch performance<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumers also use batching.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Important concepts include:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>fetch.min.bytes\nfetch.max.bytes\nmax.partition.fetch.bytes\nmax.poll.records<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For example, increasing <code>fetch.min.bytes<\/code> can let a broker wait for more data before responding, potentially improving throughput at the cost of additional latency.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Again:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>throughput \u2194 latency<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">tradeoff.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 35 \u2014 Deserialization<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer receives bytes.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Bytes\n  \u2193\nDeserializer\n  \u2193\nApplication Object<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If schemas are involved:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Schema-aware deserializer<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">reconstructs data according to the serialization system.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 36 \u2014 Application processing<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Now:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n vehicle_id: V1001,\n speed: 88\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">reaches application logic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Maybe:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>if speed &gt; speed_limit:\n    create alert<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">or:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>update analytics<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">or:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>store in database<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka&#8217;s job and your application&#8217;s job are different.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 37 \u2014 Offset commit<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">After processing, the consumer needs a recovery checkpoint.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose processed:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P4 through Offset 104<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It commits progress.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer\n   \u2193\nGroup Coordinator\n   \u2193\n__consumer_offsets<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka stores the committed group position. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 38 \u2014 Why commit timing matters<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consider this dangerous order:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>1. Commit offset\n2. Process event<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>commit succeeds\napplication crashes before processing<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">On restart Kafka may think:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>104 already handled<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">although your business operation did not occur.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Possible loss from the application&#8217;s perspective.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now reverse it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>1. Process event\n2. Commit offset<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Crash between steps:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>business processing succeeds\ncommit fails<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer may process it again.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Possible duplicate processing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This leads us directly to advanced Kafka delivery semantics:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>at-most-once\nat-least-once\nexactly-once approaches<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We will cover these separately because they deserve their own tutorial.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 39 \u2014 Consumer restart<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Suppose Consumer C dies.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The group can rebalance.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Before\n\nA \u2192 P0 P1\nB \u2192 P2 P3\nC \u2192 P4 P5\n\nC dies<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Potential reassignment:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>A \u2192 P0 P1 P4\nB \u2192 P2 P3 P5<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The new owner resumes according to committed offsets.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Consumer groups and rebalancing are therefore part of availability on the consumption side. (<a href=\"https:\/\/docs.confluent.io\/cloud\/current\/client-apps\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 40 \u2014 Heartbeats<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Consumers must maintain group membership.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conceptually:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer\n   \u2193\nheartbeat\n   \u2193\nCoordinator\n\n\"I'm alive.\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If a consumer disappears long enough, Kafka can remove it from the group and redistribute partitions. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/clients\/consumer.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">FLOW 41 \u2014 Retention means replay<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Imagine our analytics code had a bug for six hours.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Fixed version deployed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If Kafka still retains the relevant data:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Reset\/reposition consumer\n       \u2193\nOlder offset\n       \u2193\nReplay events\n       \u2193\nRebuild analytics<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is one of Kafka&#8217;s most valuable operational capabilities.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">The complete journey<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Now put all 41 steps into one mental model:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>BUSINESS EVENT\n     \u2193\nApplication\n     \u2193\nProducerRecord\n     \u2193\nSerializer\n     \u2193\nSchema handling\n     \u2193\nMetadata lookup\n     \u2193\nPartition selection\n     \u2193\nProducer buffer\n     \u2193\nBatching\n     \u2193\nCompression\n     \u2193\nNetwork\n     \u2193\nPartition Leader\n     \u2193\nBroker\n     \u2193\nAppend to Log\n     \u2193\nOffset\n     \u2193\nFollower Replication\n     \u2193\nISR\n     \u2193\nacks \/ min.insync.replicas\n     \u2193\nProducer ACK\n     \u2193\n\n        EVENT STORED IN KAFKA\n\n     \u2193\nConsumer Group\n     \u2193\nGroup Coordinator\n     \u2193\nPartition Assignment\n     \u2193\nConsumer Poll \/ Fetch\n     \u2193\nPartition Leader\n     \u2193\nRecord Batch\n     \u2193\nDeserializer\n     \u2193\nApplication Processing\n     \u2193\nOffset Commit\n     \u2193\n__consumer_offsets<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Performance optimization map<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">When performance is the goal, investigate:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Area<\/th><th>Important concepts<\/th><\/tr><\/thead><tbody><tr><td>Producer<\/td><td>batching<\/td><\/tr><tr><td>Producer<\/td><td><code>linger.ms<\/code><\/td><\/tr><tr><td>Producer<\/td><td><code>batch.size<\/code><\/td><\/tr><tr><td>Producer<\/td><td>compression<\/td><\/tr><tr><td>Producer<\/td><td>buffer capacity<\/td><\/tr><tr><td>Topic<\/td><td>partition count<\/td><\/tr><tr><td>Broker<\/td><td>capacity and distribution<\/td><\/tr><tr><td>Consumer<\/td><td>consumer count<\/td><\/tr><tr><td>Consumer<\/td><td>fetch sizing<\/td><\/tr><tr><td>Consumer<\/td><td>processing speed<\/td><\/tr><tr><td>Application<\/td><td>record size<\/td><\/tr><tr><td>Network<\/td><td>bandwidth\/latency<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Producer batching and consumer fetching are deliberate parts of Kafka&#8217;s throughput-oriented design. (<a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/configuration\/producer-configs.html?utm_source=chatgpt.com\">Confluent Documentation<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Reliability optimization map<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">When reliability is the goal:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Area<\/th><th>Important concept<\/th><\/tr><\/thead><tbody><tr><td>Topic<\/td><td>Replication factor<\/td><\/tr><tr><td>Broker<\/td><td>Replica health<\/td><\/tr><tr><td>Partition<\/td><td>ISR<\/td><\/tr><tr><td>Topic\/Broker<\/td><td><code>min.insync.replicas<\/code><\/td><\/tr><tr><td>Producer<\/td><td><code>acks=all<\/code><\/td><\/tr><tr><td>Producer<\/td><td>retries<\/td><\/tr><tr><td>Producer<\/td><td>idempotence<\/td><\/tr><tr><td>Consumer<\/td><td>correct offset strategy<\/td><\/tr><tr><td>Application<\/td><td>idempotent processing<\/td><\/tr><tr><td>Schema<\/td><td>compatibility<\/td><\/tr><tr><td>Operations<\/td><td>monitoring<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Availability optimization map<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">When availability is the goal:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Area<\/th><th>Important concept<\/th><\/tr><\/thead><tbody><tr><td>Cluster<\/td><td>multiple brokers<\/td><\/tr><tr><td>Metadata<\/td><td>redundant KRaft controllers<\/td><\/tr><tr><td>Topic<\/td><td>replicas<\/td><\/tr><tr><td>Partition<\/td><td>leader election<\/td><\/tr><tr><td>Replica<\/td><td>healthy ISR<\/td><\/tr><tr><td>Infrastructure<\/td><td>rack\/AZ placement<\/td><\/tr><tr><td>Consumer<\/td><td>multiple group members<\/td><\/tr><tr><td>Networking<\/td><td>redundant paths<\/td><\/tr><tr><td>Operations<\/td><td>alerts<\/td><\/tr><tr><td>Recovery<\/td><td>tested failure procedures<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Current KRaft documentation recommends multiple controllers for redundant production deployments, while Confluent Cloud provides multi-zone deployment options for workloads that need zone-level availability. (<a href=\"https:\/\/kafka.apache.org\/43\/operations\/kraft\/?utm_source=chatgpt.com\">Apache Kafka<\/a>)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">The configuration triangle students must remember<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka tuning is rarely:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>turn setting HIGH\n=\nbetter<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Instead:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>                PERFORMANCE\n                    \/\\\n                   \/  \\\n                  \/    \\\n                 \/      \\\n                \/        \\\n       RELIABILITY ------ AVAILABILITY<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>More replication\n\u2192 better durability\/availability\n\u2192 more network\/storage work\n\nMore batching\n\u2192 better throughput\n\u2192 potentially more latency\n\nMore partitions\n\u2192 greater parallelism\n\u2192 more metadata and operational overhead\n\nacks=all\n\u2192 stronger write durability\n\u2192 waits for stronger acknowledgment conditions\n\nMore consumers\n\u2192 more processing parallelism\n\u2192 useful only up to partition-level parallelism<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A Kafka engineer&#8217;s job is not to maximize one setting.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It is to design the correct balance for the workload.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Production scenario exercise<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Assume we have:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>500,000 vehicles\n\nEach vehicle sends:\n1 event every 5 seconds\n\nTopic:\nvehicle-telemetry\n\nRequirement:\nNo unacceptable loss of telemetry\n\nAnalytics latency target:\n&lt; 5 seconds\n\nCluster must survive:\none infrastructure failure<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Students should start asking:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>What event rate do we expect?\n\nWhat is the average record size?\n\nHow many partitions do we need?\n\nWhat key should we use?\n\nWhat replication strategy?\n\nWhat acknowledgment policy?\n\nWhat minimum ISR?\n\nWhat consumer concurrency?\n\nWhat happens if a broker fails?\n\nWhat happens if a consumer fails?\n\nHow long should events be retained?\n\nHow will we monitor producer lag\/errors?\n\nHow will we monitor consumer lag?\n\nHow will we handle schema evolution?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That transition\u2014from asking <strong>&#8220;What command creates a topic?&#8221;<\/strong> to asking <strong>&#8220;What architecture satisfies this workload?&#8221;<\/strong>\u2014is the path from beginner Kafka knowledge to Kafka mastery.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">Final Architecture Master Checklist<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">A student should now be able to explain every box below:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;Application]\n     \u2193\n&#91;Producer]\n     \u2193\n&#91;Serializer]\n     \u2193\n&#91;Schema]\n     \u2193\n&#91;Metadata]\n     \u2193\n&#91;Key]\n     \u2193\n&#91;Partitioner]\n     \u2193\n&#91;Producer Buffer]\n     \u2193\n&#91;Batch]\n     \u2193\n&#91;Compression]\n     \u2193\n&#91;Network]\n     \u2193\n&#91;Broker]\n     \u2193\n&#91;Topic]\n     \u2193\n&#91;Partition Leader]\n     \u2193\n&#91;Partition Log]\n     \u2193\n&#91;Offset]\n     \u2193\n&#91;Replica Followers]\n     \u2193\n&#91;ISR]\n     \u2193\n&#91;Replication Factor]\n     \u2193<\/code><\/pre>\n\n\n<p>[acks]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2193<\/p>\n\n\n<p>[min.insync.replicas]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2193 [KRaft Controller] \u2193 [Consumer Group] \u2193 [Group Coordinator] \u2193 [Partition Assignment] \u2193 [Consumer] \u2193 [Fetch\/Poll] \u2193 [Deserializer] \u2193 [Processing] \u2193 [Offset Commit] \u2193 [__consumer_offsets] \u2193 [Retention \/ Replay]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If a student truly understands <strong>why every component in this chain exists<\/strong>, later topics such as producer tuning, consumer tuning, replication, delivery semantics, transactions, Kafka security, monitoring, capacity planning and disaster recovery become dramatically easier.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is the foundation we want before moving deeper into Kafka configuration.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Kafka Architecture 1. Learning objectives By the end of this tutorial, a student should understand: 2. Start with the simplest Kafka architecture At the highest level: For&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1139","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1139"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1139\/revisions"}],"predecessor-version":[{"id":1140,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1139\/revisions\/1140"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}