{"id":1146,"date":"2026-08-19T01:58:51","date_gmt":"2026-08-19T01:58:51","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/?p=1146"},"modified":"2026-08-19T01:58:51","modified_gmt":"2026-08-19T01:58:51","slug":"kafka-master-tutorials-series-4-confluent-cloud-kafka-production-checklist","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/kafka-master-tutorials-series-4-confluent-cloud-kafka-production-checklist\/","title":{"rendered":"Kafka Master Tutorials Series: 4 &#8211; Confluent Cloud Kafka \u2014 Production Checklist"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">1. Cluster Architecture<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use a <strong>Dedicated Kafka cluster<\/strong> for critical\/high-throughput production workloads.<\/li>\n\n\n\n<li>Select <strong>MULTI_ZONE \/ High Availability<\/strong>.<\/li>\n\n\n\n<li>Use at least <strong>2 CKUs<\/strong> because Confluent requires 2 CKUs for multi-zone Dedicated clusters.<\/li>\n\n\n\n<li>Do <strong>not<\/strong> size production at the absolute minimum; leave capacity headroom.<\/li>\n\n\n\n<li>Keep applications and Kafka in the <strong>same cloud provider and region<\/strong> wherever possible.<\/li>\n\n\n\n<li>Avoid application \u2192 Kafka cross-region traffic on the normal data path.<\/li>\n\n\n\n<li>Use private networking such as PrivateLink\/peering where security architecture requires it.<\/li>\n\n\n\n<li>Separate Production, Staging and Development clusters\/environments.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Recommended architecture:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>                 Availability Zone A\n                       \u2502\n                 Kafka Replica\n                       \u2502\nProducer \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Kafka Leader\n                       \u2502\n                 Kafka Replica\n                       \u2502\n                 Availability Zone B\/C\n\n                       \u2502\n                       \u25bc\n\n                  Consumer Group\n              \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n              \u25bc        \u25bc        \u25bc\n             C1       C2       C3\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">2. Capacity Planning<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Determine peak <strong>MB\/sec produced<\/strong>.<\/li>\n\n\n\n<li>Determine peak <strong>MB\/sec consumed<\/strong>.<\/li>\n\n\n\n<li>Determine messages\/sec.<\/li>\n\n\n\n<li>Determine average and maximum message size.<\/li>\n\n\n\n<li>Determine expected partition count.<\/li>\n\n\n\n<li>Determine number of producers.<\/li>\n\n\n\n<li>Determine number of consumer groups.<\/li>\n\n\n\n<li>Determine number of consumers.<\/li>\n\n\n\n<li>Determine client connection count.<\/li>\n\n\n\n<li>Determine retention requirements.<\/li>\n\n\n\n<li>Determine expected growth for at least the next 6\u201312 months.<\/li>\n\n\n\n<li>Size CKUs for <strong>peak<\/strong>, not average traffic.<\/li>\n\n\n\n<li>Maintain spare capacity for broker\/zone failures and traffic spikes.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent recommends monitoring cluster load closely. Sustained load around <strong>70\u201380% is a reason to consider adding CKUs<\/strong>, while above <strong>80% can result in increased latency and throttling<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">My production target would normally be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Normal load       &lt; 60\u201370%\nPeak load         &lt; 80%\nEmergency capacity available\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">3. Topic Replication<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Use:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>replication.factor=3\nmin.insync.replicas=2\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent Cloud&#8217;s default replication factor is <strong>3<\/strong>, and <code>min.insync.replicas<\/code> defaults to <strong>2<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Replication factor = <strong>3<\/strong><\/li>\n\n\n\n<li><code>min.insync.replicas=2<\/code><\/li>\n\n\n\n<li>Do not reduce replication simply to improve throughput.<\/li>\n\n\n\n<li>Do not use RF=1 for critical production data.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This gives the classic durable Kafka combination:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>RF = 3\n\nBroker 1     Broker 2     Broker 3\n Leader       Replica      Replica\n   \u2502             \u2502             \u2502\n   \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\nmin.insync.replicas = 2\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">4. Producer Durability<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">For critical production workloads:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>acks=all\nenable.idempotence=true\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>acks=all<\/code><\/li>\n\n\n\n<li><code>enable.idempotence=true<\/code><\/li>\n\n\n\n<li>Enable retries.<\/li>\n\n\n\n<li>Use sensible <code>delivery.timeout.ms<\/code>.<\/li>\n\n\n\n<li>Do not use <code>acks=0<\/code>.<\/li>\n\n\n\n<li>Avoid <code>acks=1<\/code> for business-critical data.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The important combination is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2502\n   \u2502 acks=all\n   \u25bc\nLeader\n   \u2502\n   \u251c\u2500\u2500\u2500\u2500 Replica\n   \u2502\n   \u2514\u2500\u2500\u2500\u2500 Replica\n\n       \u2193\n\nACK returned\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>acks=1<\/code> can improve raw throughput, and Confluent lists it as a throughput optimization, but it weakens the durability guarantee. For a system where <strong>performance and HA both matter<\/strong>, I would keep <code>acks=all<\/code>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">5. Producer Performance<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Start with approximately:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>acks=all\n\nenable.idempotence=true\n\ncompression.type=lz4\n\nbatch.size=100000\n\nlinger.ms=10\n\nbuffer.memory=67108864\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then benchmark.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable producer batching.<\/li>\n\n\n\n<li>Start <code>batch.size<\/code> around <strong>100\u2013200 KB<\/strong>.<\/li>\n\n\n\n<li>Start <code>linger.ms<\/code> around <strong>10 ms<\/strong>.<\/li>\n\n\n\n<li>Test 10\u201350 ms for throughput-oriented workloads.<\/li>\n\n\n\n<li>Consider up to ~100 ms where throughput matters far more than latency.<\/li>\n\n\n\n<li>Use <code>compression.type=lz4<\/code> for strong performance.<\/li>\n\n\n\n<li>Increase <code>buffer.memory<\/code> if producers write heavily across many partitions.<\/li>\n\n\n\n<li>Monitor producer buffer exhaustion.<\/li>\n\n\n\n<li>Avoid producing thousands of tiny requests.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent specifically recommends larger batches, increasing <code>linger.ms<\/code>, and <code>lz4<\/code> for throughput optimization. Their throughput guidance suggests approximately <code>batch.size=100000\u2013200000<\/code> and <code>linger.ms=10\u2013100<\/code>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">6. Compression<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Recommended starting point:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>compression.type=lz4\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">My preference:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Codec<\/th><th>Throughput<\/th><th>Compression<\/th><th>CPU<\/th><\/tr><\/thead><tbody><tr><td>none<\/td><td>High network usage<\/td><td>None<\/td><td>Very low<\/td><\/tr><tr><td>lz4<\/td><td>\u2b50\u2b50\u2b50\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50\u2b50\u2b50<\/td><\/tr><tr><td>snappy<\/td><td>\u2b50\u2b50\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50\u2b50<\/td><\/tr><tr><td>zstd<\/td><td>\u2b50\u2b50\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50<\/td><\/tr><tr><td>gzip<\/td><td>\u2b50\u2b50<\/td><td>\u2b50\u2b50\u2b50\u2b50\u2b50<\/td><td>\u2b50<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">For performance-oriented Kafka:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>LZ4 = excellent default\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent explicitly recommends <strong>LZ4 for performance<\/strong> rather than gzip.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">7. Partition Strategy<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Partitions are one of the <strong>most important performance decisions<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Estimate partitions before creating heavily keyed topics.<\/li>\n\n\n\n<li>Create enough partitions to support required producer\/consumer parallelism.<\/li>\n\n\n\n<li>Avoid blindly creating hundreds\/thousands of partitions.<\/li>\n\n\n\n<li>Monitor partition traffic distribution.<\/li>\n\n\n\n<li>Avoid hot partitions.<\/li>\n\n\n\n<li>Design message keys carefully.<\/li>\n\n\n\n<li>Don&#8217;t use one permanently hot key for large portions of traffic.<\/li>\n\n\n\n<li>Remember:<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>Maximum useful consumers\nin one consumer group\n\n\u2248\n\nnumber of topic partitions\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Topic = orders\n\nP0 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Consumer 1\nP1 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Consumer 2\nP2 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Consumer 3\nP3 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba Consumer 4\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent uses approximately <strong>6\u201310 partitions per CKU<\/strong> as a useful parallelism guideline when diagnosing under-parallelized Dedicated clusters, although the correct number depends on your workload.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">8. Avoid Hot Partitions<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Bad:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 80%\nP1 \u2588\u2588                     5%\nP2 \u2588\u2588                     5%\nP3 \u2588\u2588\u2588\u2588                  10%\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Good:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>P0 \u2588\u2588\u2588\u2588\u2588\u2588 25%\nP1 \u2588\u2588\u2588\u2588\u2588\u2588 25%\nP2 \u2588\u2588\u2588\u2588\u2588\u2588 25%\nP3 \u2588\u2588\u2588\u2588\u2588\u2588 25%\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Choose high-cardinality keys where ordering requirements allow.<\/li>\n\n\n\n<li>Monitor per-partition traffic.<\/li>\n\n\n\n<li>Find dominant keys.<\/li>\n\n\n\n<li>Reconsider partition-key strategy when traffic is skewed.<\/li>\n\n\n\n<li>Use key salting only where ordering semantics permit it.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent identifies skewed\/hot partitions as a direct source of throttling and performance problems.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">9. Consumer Performance<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Start around:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>fetch.min.bytes=100000\n\nfetch.max.wait.ms=500\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then tune according to latency requirements.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Increase <code>fetch.min.bytes<\/code> for throughput.<\/li>\n\n\n\n<li>Configure <code>fetch.max.bytes<\/code> appropriately.<\/li>\n\n\n\n<li>Configure <code>max.partition.fetch.bytes<\/code> according to maximum record\/batch size.<\/li>\n\n\n\n<li>Set <code>max.poll.records<\/code> according to processing capacity.<\/li>\n\n\n\n<li>Set <code>max.poll.interval.ms<\/code> longer than worst-case batch processing time.<\/li>\n\n\n\n<li>Scale consumers horizontally.<\/li>\n\n\n\n<li>Keep consumer processing fast.<\/li>\n\n\n\n<li>Move expensive processing away from the poll loop where architecture permits it.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent recommends approximately <code>fetch.min.bytes=100000<\/code> for throughput-oriented consumers.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">10. Consumer Parallelism<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">If there are:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>16 partitions\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">then a natural upper bound for active consumers in the same group is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>16 consumers\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>16 Partitions\n       \u2502\n       \u25bc\n Consumer Group\n       \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u25bc     \u25bc                \u25bc\nC1    C2 ...           C16\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Adding:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>C17\nC18\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">doesn&#8217;t create additional partition-level parallelism for that topic.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>Consumers &lt;= useful partition parallelism<\/code><\/li>\n\n\n\n<li>Scale consumers when consumer lag grows.<\/li>\n\n\n\n<li>Increase partitions when justified by sustained parallelism requirements.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">11. Consumer Rebalancing<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Avoid constantly starting\/stopping consumers.<\/li>\n\n\n\n<li>Keep consumer processing time below <code>max.poll.interval.ms<\/code>.<\/li>\n\n\n\n<li>Monitor rebalance frequency.<\/li>\n\n\n\n<li>Monitor rebalance duration.<\/li>\n\n\n\n<li>Use the newer consumer group protocol where supported and appropriate.<\/li>\n\n\n\n<li>Test application deployments for rebalance storms.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent recommends monitoring rebalance frequency and notes that the newer consumer group protocol can reduce rebalance impact compared with the classic protocol.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">12. Message Size<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Best:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Small Kafka messages\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">rather than:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>10 MB\n20 MB\n50 MB\n...\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Keep records compact.<\/li>\n\n\n\n<li>Avoid putting large binary files directly into Kafka.<\/li>\n\n\n\n<li>Store large objects in object storage such as S3\/GCS\/Azure Blob.<\/li>\n\n\n\n<li>Send object metadata\/reference through Kafka where appropriate.<\/li>\n\n\n\n<li>Make producer, broker\/topic and consumer maximum sizes compatible.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Pattern:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Kafka message\n     \u2502\n     \u251c\u2500\u2500 customerId\n     \u251c\u2500\u2500 eventType\n     \u251c\u2500\u2500 timestamp\n     \u2514\u2500\u2500 s3:\/\/bucket\/object\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">rather than putting the entire large file into Kafka.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">13. Schema Management<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use Schema Registry.<\/li>\n\n\n\n<li>Prefer Avro, Protobuf or JSON Schema.<\/li>\n\n\n\n<li>Define compatibility policy.<\/li>\n\n\n\n<li>Prefer <code>BACKWARD<\/code> compatibility for many event-streaming systems.<\/li>\n\n\n\n<li>Prevent breaking schema changes in CI\/CD.<\/li>\n\n\n\n<li>Version schemas through governance rather than application-specific conventions.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer\n   \u2502\n   \u251c\u2500\u2500\u2500\u2500 Schema Registry\n   \u2502\n   \u25bc\n Kafka\n   \u2502\n   \u25bc\nConsumer\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">14. Retention<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Never leave retention decisions accidental.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Configure according to the use case:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>retention.ms=...\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Define business retention requirements.<\/li>\n\n\n\n<li>Define replay requirements.<\/li>\n\n\n\n<li>Define regulatory requirements.<\/li>\n\n\n\n<li>Use appropriate retention for high-volume topics.<\/li>\n\n\n\n<li>Use compaction for state\/change-log use cases where appropriate.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Examples:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Transaction Events \u2192 30 days\n\nTelemetry \u2192 7 days\n\nAudit Events \u2192 1 year\n\nCurrent User State \u2192 compacted topic\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">15. Networking<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Ideal:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>AWS Application\n      \u2502\n      \u2502 same AWS region\n      \u25bc\nConfluent Cloud Kafka\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Avoid:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>AWS Tokyo\n   \u2502\n   \u2502 WAN\n   \u25bc\nKafka Virginia\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">unless the architecture genuinely requires it.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Application and Kafka in same region where possible.<\/li>\n\n\n\n<li>Minimize network hops.<\/li>\n\n\n\n<li>Avoid unnecessary proxies.<\/li>\n\n\n\n<li>Avoid repeatedly opening Kafka connections.<\/li>\n\n\n\n<li>Reuse long-lived Kafka clients.<\/li>\n\n\n\n<li>Monitor network latency.<\/li>\n\n\n\n<li>Monitor connection count.<\/li>\n\n\n\n<li>Monitor connection creation rate.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">16. Connection Management<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka clients are intended to be long-lived.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Bad:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Request\n \u2193\nCreate Producer\n \u2193\nProduce\n \u2193\nClose Producer\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Better:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Application starts\n       \u2502\n       \u25bc\nCreate Producer\n       \u2502\n       \u25bc\nReuse for millions of messages\n       \u2502\n       \u25bc\nApplication shuts down\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Reuse producers.<\/li>\n\n\n\n<li>Reuse consumers.<\/li>\n\n\n\n<li>Avoid connection churn.<\/li>\n\n\n\n<li>Monitor <code>active_connection_count<\/code>.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent notes that excessive client connections can result in sharply increased producer latency.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">17. Security<\/h1>\n\n\n\n<ul class=\"wp-block-list\">\n<li>TLS enabled.<\/li>\n\n\n\n<li>SASL authentication.<\/li>\n\n\n\n<li>Use separate service accounts per application\/team.<\/li>\n\n\n\n<li>Apply least-privilege RBAC\/ACLs.<\/li>\n\n\n\n<li>Do not share production API keys.<\/li>\n\n\n\n<li>Store secrets in Vault\/AWS Secrets Manager\/etc.<\/li>\n\n\n\n<li>Rotate credentials.<\/li>\n\n\n\n<li>Restrict network access.<\/li>\n\n\n\n<li>Maintain audit logs.<\/li>\n\n\n\n<li>Separate human identities from application identities.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">18. Monitoring<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">These metrics should absolutely be monitored.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Cluster<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cluster load %<\/li>\n\n\n\n<li>Maximum cluster load<\/li>\n\n\n\n<li>CKU utilization\/count<\/li>\n\n\n\n<li>Produce throughput<\/li>\n\n\n\n<li>Fetch throughput<\/li>\n\n\n\n<li>Request rate<\/li>\n\n\n\n<li>Client throttling<\/li>\n\n\n\n<li>Active connections<\/li>\n\n\n\n<li>Hot partitions<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Producer<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Producer request latency<\/li>\n\n\n\n<li>Produce rate<\/li>\n\n\n\n<li>Error rate<\/li>\n\n\n\n<li>Retry rate<\/li>\n\n\n\n<li>Record-send rate<\/li>\n\n\n\n<li>Buffer availability<\/li>\n\n\n\n<li>Buffer wait time<\/li>\n\n\n\n<li>Throttle time<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Consumer<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Consumer lag<\/li>\n\n\n\n<li>Consumer latency<\/li>\n\n\n\n<li>Records consumed\/sec<\/li>\n\n\n\n<li>Fetch rate<\/li>\n\n\n\n<li>Fetch latency<\/li>\n\n\n\n<li>Consumer throttle time<\/li>\n\n\n\n<li>Rebalance frequency<\/li>\n\n\n\n<li>Rebalance duration<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent specifically recommends monitoring cluster load, hot partitions, consumer lag, throttling and producer latency when evaluating Dedicated cluster performance.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">19. Consumer Lag Alerts<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">This is one of my highest-priority alerts:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Producer Rate\n      \u2502\n      \u2502\n      \u25bc\nKafka\n      \u2502\n      \u2502\n      \u25bc\nConsumer\n      \u2502\n      \u25bc\nConsumer Lag\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Alert when:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>lag continuously increases\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">rather than merely reacting to one temporary spike.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Alert on large absolute lag.<\/li>\n\n\n\n<li>Alert on continuously growing lag.<\/li>\n\n\n\n<li>Alert on consumer latency.<\/li>\n\n\n\n<li>Alert if consumer group disappears unexpectedly.<\/li>\n\n\n\n<li>Correlate lag with cluster load.<\/li>\n\n\n\n<li>Correlate lag with consumer CPU.<\/li>\n\n\n\n<li>Correlate lag with rebalances.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent explicitly recommends monitoring <strong>lag trends rather than only absolute values<\/strong>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">20. Scaling Strategy<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Use this decision flow:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Consumer lag increasing?\n       \u2502\n       \u25bc\nCluster load &gt;70%?\n       \u2502\n   \u250c\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2510\n  YES      NO\n   \u2502        \u2502\n   \u25bc        \u25bc\nAdd CKU   Enough partitions?\n             \u2502\n         \u250c\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2510\n        NO       YES\n         \u2502         \u2502\n         \u25bc         \u25bc\n Add partitions  Add consumers\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This closely follows Confluent&#8217;s troubleshooting recommendation for Dedicated clusters.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">21. Disaster Recovery<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Multi-zone is not the same thing as multi-region DR.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For zonal failures:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Region A\n\nAZ1 + AZ2 + AZ3\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Multi-zone Kafka handles infrastructure\/AZ resilience.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For regional failures:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>         Primary\n      Region Tokyo\n           \u2502\n           \u2502 Cluster Linking\n           \u25bc\n        DR Cluster\n      Region Osaka\/etc.\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Define RPO.<\/li>\n\n\n\n<li>Define RTO.<\/li>\n\n\n\n<li>Provision secondary-region Kafka where required.<\/li>\n\n\n\n<li>Use Cluster Linking where appropriate.<\/li>\n\n\n\n<li>Replicate required topics.<\/li>\n\n\n\n<li>Synchronize required consumer offsets.<\/li>\n\n\n\n<li>Synchronize required ACLs.<\/li>\n\n\n\n<li>Pre-create DR credentials.<\/li>\n\n\n\n<li>Keep bootstrap servers outside application code.<\/li>\n\n\n\n<li>Test DR failover regularly.<\/li>\n\n\n\n<li>Monitor replication\/mirror lag.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Confluent recommends Cluster Linking for multi-region DR and supports replication of topic data, consumer offsets and ACLs.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">22. Benchmark Before Production<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">Never trust configuration recommendations without measuring your actual workload.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Test:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>10K msg\/sec\n25K msg\/sec\n50K msg\/sec\n100K msg\/sec\n200K msg\/sec\n...\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Measure:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Throughput\nLatency p50\nLatency p95\nLatency p99\nProducer errors\nConsumer lag\nCluster load\nThrottle time\nCPU\nMemory\nNetwork\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Also test:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Normal traffic.<\/li>\n\n\n\n<li>2\u00d7 normal traffic.<\/li>\n\n\n\n<li>Peak traffic.<\/li>\n\n\n\n<li>Sudden burst.<\/li>\n\n\n\n<li>Producer restart.<\/li>\n\n\n\n<li>Consumer restart.<\/li>\n\n\n\n<li>Consumer scale-out.<\/li>\n\n\n\n<li>Broker\/AZ disturbance where test environment permits it.<\/li>\n\n\n\n<li>Schema changes.<\/li>\n\n\n\n<li>Large messages.<\/li>\n\n\n\n<li>Consumer slowdown.<\/li>\n\n\n\n<li>DR failover.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">23. My Recommended Production Baseline<\/h1>\n\n\n\n<h3 class=\"wp-block-heading\">Cluster<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>Confluent Cloud Dedicated\nMulti-Zone\nRF = 3\nmin ISR = 2\nCKUs = benchmark-derived with &gt;=20\u201330% capacity headroom\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Producer<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>acks=all\nenable.idempotence=true\n\ncompression.type=lz4\n\nbatch.size=100000\nlinger.ms=10\n\n# Increase where partition count \/ throughput requires it\nbuffer.memory=67108864\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Consumer<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>fetch.min.bytes=100000\nfetch.max.wait.ms=500\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then tune:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>max.partition.fetch.bytes\nfetch.max.bytes\nmax.poll.records\nmax.poll.interval.ms\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">according to message size and processing time.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\">24. The 15 Things I Would Check First<\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">If I inherited a production Confluent Cloud Kafka cluster tomorrow, these would be my first checks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Dedicated cluster<\/strong><\/li>\n\n\n\n<li><strong>Multi-zone enabled<\/strong><\/li>\n\n\n\n<li><strong>RF = 3<\/strong><\/li>\n\n\n\n<li><strong>min ISR = 2<\/strong><\/li>\n\n\n\n<li><strong>Producer <code>acks=all<\/code><\/strong><\/li>\n\n\n\n<li><strong>Producer idempotence enabled<\/strong><\/li>\n\n\n\n<li><strong>LZ4 compression<\/strong><\/li>\n\n\n\n<li><strong>Producer batching enabled<\/strong><\/li>\n\n\n\n<li><strong>Correct partition count<\/strong><\/li>\n\n\n\n<li><strong>No hot partitions<\/strong><\/li>\n\n\n\n<li><strong>Enough consumer instances<\/strong><\/li>\n\n\n\n<li><strong>Consumer lag monitored<\/strong><\/li>\n\n\n\n<li><strong>Cluster load below ~70\u201380% sustained<\/strong><\/li>\n\n\n\n<li><strong>No client throttling<\/strong><\/li>\n\n\n\n<li><strong>Multi-region DR designed separately where business RTO\/RPO requires it<\/strong><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Golden Rule<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>                 Kafka Performance\n                       \u2502\n       \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n       \u2502               \u2502                \u2502\n       \u25bc               \u25bc                \u25bc\n   Partitions       Batching        Compression\n       \u2502               \u2502                \u2502\n       \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n                       \u25bc\n                  Parallelism\n                       \u2502\n                       \u25bc\n                  More Throughput\n\n             while preserving:\n\n             RF=3\n             min ISR=2\n             acks=all\n             idempotence=true\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Do not gain performance by sacrificing reliability first.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Gain performance primarily through:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>correct partitioning \u2192 batching \u2192 compression \u2192 consumer parallelism \u2192 sufficient CKU capacity \u2192 good network placement.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"<p>1. Cluster Architecture Recommended architecture: 2. Capacity Planning Confluent recommends monitoring cluster load closely. Sustained load around 70\u201380% is a reason to consider adding CKUs, while above&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1146","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1146","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1146"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1146\/revisions"}],"predecessor-version":[{"id":1147,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1146\/revisions\/1147"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1146"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1146"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1146"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}