{"id":1141,"date":"2026-08-19T01:25:58","date_gmt":"2026-08-19T01:25:58","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/?p=1141"},"modified":"2026-08-19T01:26:00","modified_gmt":"2026-08-19T01:26:00","slug":"kafka-master-tutorials-series-3-capacity-planning-in-confluent-cloud","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/kafka-master-tutorials-series-3-capacity-planning-in-confluent-cloud\/","title":{"rendered":"Kafka Master Tutorials Series: 3 -Capacity Planning in Confluent Cloud"},"content":{"rendered":"\n

1. What is Kafka Capacity Planning?<\/h2>\n\n\n\n

Capacity planning means calculating how much Kafka capacity your application needs before production traffic arrives.<\/strong><\/p>\n\n\n\n

In simple words:<\/p>\n\n\n\n

\n

How big should my Kafka cluster be so that it can handle normal traffic, peak traffic, failures, and future growth without slowing down?<\/p>\n<\/blockquote>\n\n\n\n

For a Confluent Cloud Dedicated cluster<\/strong>, capacity is primarily expressed using CKUs \u2014 Confluent Units for Kafka<\/strong>. Adding CKUs increases the capacity of the Dedicated cluster. (Confluent Documentation<\/a>)<\/p>\n\n\n\n

Your original production checklist already identifies the right inputs: peak produce\/consume throughput, messages\/sec, message size, partitions, producers, consumer groups, connections, retention, and expected growth.<\/p>\n\n\n\n

\n\n\n\n

2. Why Capacity Planning Matters<\/h1>\n\n\n\n

Suppose normal traffic is:<\/p>\n\n\n\n

20 MB\/sec<\/code><\/pre>\n\n\n\n
but peak traffic reaches:<\/p>\n\n\n\n
100 MB\/sec<\/code><\/pre>\n\n\n\n
If you designed Kafka only for 20 MB\/sec:<\/p>\n\n\n\n
Normal Traffic\n     \u2502\n     \u25bc\n   Kafka\n     \u2502\n     \u2705<\/code><\/pre>\n\n\n\n
during peak traffic:<\/p>\n\n\n\n
Traffic Spike\n100 MB\/sec\n     \u2502\n     \u25bc\nKafka Saturation\n     \u2502\n     \u25bc\nThrottling\n     \u2502\n     \u25bc\nProducer Latency\n     \u2502\n     \u25bc\nConsumer Lag\n     \u2502\n     \u25bc\nApplication Problems<\/code><\/pre>\n\n\n\n
Therefore:<\/p>\n\n\n\n
Never size Kafka for average traffic only.\n\nSize primarily for:\n\nPeak Traffic\n     +\nCapacity Headroom\n     +\nExpected Growth\n     +\nFailure\/Burst Capacity<\/code><\/pre>\n\n\n\n
\n\n\n\n
3. What is a CKU?<\/h1>\n\n\n\n
For a Confluent Cloud Dedicated cluster<\/strong>:<\/p>\n\n\n\n
CKU\n=\nConfluent Unit for Kafka<\/code><\/pre>\n\n\n\n
Think of a CKU as a unit of Kafka capacity.<\/p>\n\n\n\n
Current Confluent documentation gives these per-CKU planning values for Dedicated clusters: (Confluent Documentation<\/a>)<\/p>\n\n\n\n
Capacity dimension<\/th>Dedicated CKU<\/th><\/tr><\/thead>
Ingress<\/td>60 MB\/sec<\/strong><\/td><\/tr>
Egress<\/td>180 MB\/sec<\/strong><\/td><\/tr>
Partitions, pre-replication<\/td>4,500<\/strong><\/td><\/tr>
Client connections<\/td>18,000<\/strong><\/td><\/tr>
New connection attempts<\/td>500\/sec<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
This means you do not<\/strong> calculate Kafka size from only one number.<\/p>\n\n\n\n
You evaluate several dimensions.<\/p>\n\n\n\n
\n\n\n\n
4. Five Important Capacity Dimensions<\/h1>\n\n\n\n
Think of cluster capacity as:<\/p>\n\n\n\n
                   Kafka Capacity\n                         \u2502\n       \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n       \u2502                 \u2502                  \u2502\n       \u25bc                 \u25bc                  \u25bc\n    Ingress            Egress           Partitions\n                                            \u2502\n                              \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n                              \u25bc                          \u25bc\n                         Connections                 Requests<\/code><\/pre>\n\n\n\n
You calculate each separately.<\/p>\n\n\n\n
Then the largest requirement<\/strong> normally determines the starting CKU count.<\/p>\n\n\n\n
\n\n\n\n
5. Dimension 1 \u2014 Ingress<\/h1>\n\n\n\n
What is ingress?<\/h2>\n\n\n\n
Ingress means:<\/p>\n\n\n\n
\n
Data being written into Kafka<\/strong>.<\/p>\n<\/blockquote>\n\n\n\n
Producer\n   \u2502\n   \u2502 Events\n   \u25bc\n Kafka<\/code><\/pre>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Messages\/sec = 100,000\nAverage message = 1 KB<\/code><\/pre>\n\n\n\n
Approximate ingress:<\/p>\n\n\n\n
100,000 \u00d7 1 KB\n\n\u2248 100 MB\/sec<\/code><\/pre>\n\n\n\n
A Dedicated CKU currently provides approximately:<\/p>\n\n\n\n
60 MB\/sec ingress<\/code><\/pre>\n\n\n\n
So:<\/p>\n\n\n\n
Required CKUs\n=\n100 \/ 60\n\n=\n1.67<\/code><\/pre>\n\n\n\n
Always round up:<\/p>\n\n\n\n
2 CKUs<\/code><\/pre>\n\n\n\n
But there is a problem.<\/p>\n\n\n\n
Two CKUs provide approximately:<\/p>\n\n\n\n
2 \u00d7 60\n=\n120 MB\/sec<\/code><\/pre>\n\n\n\n
Your peak is:<\/p>\n\n\n\n
100 MB\/sec<\/code><\/pre>\n\n\n\n
Usage:<\/p>\n\n\n\n
100 \/ 120\n\u2248 83%<\/code><\/pre>\n\n\n\n
That leaves little headroom.<\/p>\n\n\n\n
A better production candidate could therefore be:<\/p>\n\n\n\n
3 CKUs<\/code><\/pre>\n\n\n\n
Then:<\/p>\n\n\n\n
3 \u00d7 60\n=\n180 MB\/sec\n\n100 \/ 180\n\u2248 56%<\/code><\/pre>\n\n\n\n
Much healthier.<\/p>\n\n\n\n
\n\n\n\n
6. Dimension 2 \u2014 Egress<\/h1>\n\n\n\n
What is egress?<\/h2>\n\n\n\n
Egress means:<\/p>\n\n\n\n
\n
Data consumers read out of Kafka<\/strong>.<\/p>\n<\/blockquote>\n\n\n\n
Example:<\/p>\n\n\n\n
                   Kafka\n                     \u2502\n          \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n          \u25bc          \u25bc          \u25bc\n       Billing    Analytics    Fraud<\/code><\/pre>\n\n\n\n
This is extremely important because multiple consumer groups independently consume the same records<\/strong>.<\/p>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Producer throughput = 50 MB\/sec<\/code><\/pre>\n\n\n\n
And four consumer groups read every event:<\/p>\n\n\n\n
Billing      = 50 MB\/sec\nAnalytics    = 50 MB\/sec\nFraud        = 50 MB\/sec\nSearch       = 50 MB\/sec<\/code><\/pre>\n\n\n\n
Total egress:<\/p>\n\n\n\n
50 \u00d7 4\n=\n200 MB\/sec<\/code><\/pre>\n\n\n\n
Notice:<\/p>\n\n\n\n
Ingress = 50 MB\/sec\n\nEgress = 200 MB\/sec<\/code><\/pre>\n\n\n\n
Therefore:<\/p>\n\n\n\n
\n
Do not assume ingress and egress are equal.<\/strong><\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\n
7. Egress CKU Example<\/h1>\n\n\n\n
Dedicated CKU egress capacity is currently approximately:<\/p>\n\n\n\n
180 MB\/sec<\/code><\/pre>\n\n\n\n
If you need:<\/p>\n\n\n\n
200 MB\/sec<\/code><\/pre>\n\n\n\n
calculate:<\/p>\n\n\n\n
200 \/ 180\n\n=\n1.11<\/code><\/pre>\n\n\n\n
Round up:<\/p>\n\n\n\n
2 CKUs<\/code><\/pre>\n\n\n\n
Again, production headroom still needs to be considered.<\/p>\n\n\n\n
\n\n\n\n
8. Dimension 3 \u2014 Partitions<\/h1>\n\n\n\n
A Kafka topic is divided into partitions:<\/p>\n\n\n\n
orders\n\n\u251c\u2500\u2500 P0\n\u251c\u2500\u2500 P1\n\u251c\u2500\u2500 P2\n\u251c\u2500\u2500 P3\n\u251c\u2500\u2500 P4\n\u2514\u2500\u2500 ...<\/code><\/pre>\n\n\n\n
Partitions provide much of Kafka’s:<\/p>\n\n\n\n
Parallelism\nScalability\nConsumer concurrency<\/code><\/pre>\n\n\n\n
A Dedicated CKU currently supports up to 4,500 pre-replication partitions<\/strong> as the published capacity dimension. (Confluent Documentation<\/a>)<\/p>\n\n\n\n
Suppose your cluster needs:<\/p>\n\n\n\n
8,000 partitions<\/code><\/pre>\n\n\n\n
Then:<\/p>\n\n\n\n
8,000 \/ 4,500\n\n=\n1.78<\/code><\/pre>\n\n\n\n
Round up:<\/p>\n\n\n\n
2 CKUs<\/code><\/pre>\n\n\n\n
Important<\/h3>\n\n\n\n
This does not<\/strong> mean:<\/p>\n\n\n\n
\n
\u201c4,500 partitions per CKU is the recommended number of partitions.\u201d<\/p>\n<\/blockquote>\n\n\n\n
It means it is a capacity limit\/dimension.<\/p>\n\n\n\n
Your actual partition count should depend on:<\/p>\n\n\n\n
Throughput\n+\nConsumer Parallelism\n+\nOrdering Requirements\n+\nKey Distribution<\/code><\/pre>\n\n\n\n
We will cover this deeply in the dedicated Partition Strategy tutorial<\/strong>.<\/p>\n\n\n\n
\n\n\n\n
9. Dimension 4 \u2014 Connections<\/h1>\n\n\n\n
Kafka clients maintain connections to Kafka.<\/p>\n\n\n\n
Examples:<\/p>\n\n\n\n
Producers\nConsumers\nKafka Connect\nKafka Streams\nFlink\nAdmin clients\nCluster Linking<\/code><\/pre>\n\n\n\n
Current Dedicated guidance provides:<\/p>\n\n\n\n
18,000 client connections \/ CKU<\/code><\/pre>\n\n\n\n
as the per-CKU capacity dimension. (Confluent Documentation<\/a>)<\/p>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Expected connections\n=\n25,000<\/code><\/pre>\n\n\n\n
Then:<\/p>\n\n\n\n
25,000 \/ 18,000\n\n=\n1.39<\/code><\/pre>\n\n\n\n
Round up:<\/p>\n\n\n\n
2 CKUs<\/code><\/pre>\n\n\n\n
\n\n\n\n
10. Connections vs Connection Attempts<\/h1>\n\n\n\n
These are different.<\/p>\n\n\n\n
Existing connections<\/h3>\n\n\n\n
Application\n     \u2502\n     \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 persistent connection \u2500\u2500\u2500\u2500\u2500\u2500\u2500 Kafka<\/code><\/pre>\n\n\n\n
Connection attempts<\/h3>\n\n\n\n
Connect\nDisconnect\n\nConnect\nDisconnect\n\nConnect\nDisconnect<\/code><\/pre>\n\n\n\n
Repeated connection creation can create additional load.<\/p>\n\n\n\n
Kafka clients should normally be long-lived<\/strong>.<\/p>\n\n\n\n
Bad:<\/p>\n\n\n\n
HTTP request\n   \u2193\nCreate producer\n   \u2193\nProduce\n   \u2193\nClose producer<\/code><\/pre>\n\n\n\n
Better:<\/p>\n\n\n\n
Application starts\n       \u2193\nCreate Producer\n       \u2193\nReuse Producer\n       \u2193\nApplication shuts down\n       \u2193\nClose Producer<\/code><\/pre>\n\n\n\n
\n\n\n\n
11. Dimension 5 \u2014 Request Rate<\/h1>\n\n\n\n
Suppose Producer A sends:<\/p>\n\n\n\n
1 message\nper Kafka request<\/code><\/pre>\n\n\n\n
while Producer B sends:<\/p>\n\n\n\n
500 messages\nper Kafka request<\/code><\/pre>\n\n\n\n
Producer A creates far more requests.<\/p>\n\n\n\n
Tiny Messages\n+\nNo Batching\n=\nLots of Kafka Requests<\/code><\/pre>\n\n\n\n
That is one reason why producer batching matters.<\/p>\n\n\n\n
Later, in the Producer Performance tutorial<\/strong>, we will tune:<\/p>\n\n\n\n
batch.size\nlinger.ms\ncompression.type\nbuffer.memory<\/code><\/pre>\n\n\n\n
\n\n\n\n
12. The Master CKU Formula<\/h1>\n\n\n\n
Calculate each dimension separately.<\/p>\n\n\n\n
Ingress CKUs\n=\nPeak Ingress \/ CKU Ingress Capacity<\/code><\/pre>\n\n\n\n
Egress CKUs\n=\nPeak Egress \/ CKU Egress Capacity<\/code><\/pre>\n\n\n\n
Partition CKUs\n=\nPartition Count \/ CKU Partition Capacity<\/code><\/pre>\n\n\n\n
Connection CKUs\n=\nConnections \/ CKU Connection Capacity<\/code><\/pre>\n\n\n\n
Then:<\/p>\n\n\n\n
Required CKUs\n=\nMAX(\n    Ingress CKUs,\n    Egress CKUs,\n    Partition CKUs,\n    Connection CKUs,\n    Other Capacity Constraints\n)<\/code><\/pre>\n\n\n\n
Finally:<\/p>\n\n\n\n
Round Up\n    +\nHA requirement\n    +\nHeadroom\n    +\nExpected growth\n    +\nBenchmark result<\/code><\/pre>\n\n\n\n
\n\n\n\n
13. Complete Real-World Example<\/h1>\n\n\n\n
Suppose an e-commerce platform expects:<\/p>\n\n\n\n
Peak ingress    = 90 MB\/sec\nPeak egress     = 250 MB\/sec\nPartitions      = 8,000\nConnections     = 25,000<\/code><\/pre>\n\n\n\n
Ingress<\/h3>\n\n\n\n
90 \/ 60\n=\n1.5\n\n\u2192 2 CKUs<\/code><\/pre>\n\n\n\n
Egress<\/h3>\n\n\n\n
250 \/ 180\n=\n1.39\n\n\u2192 2 CKUs<\/code><\/pre>\n\n\n\n
Partitions<\/h3>\n\n\n\n
8000 \/ 4500\n=\n1.78\n\n\u2192 2 CKUs<\/code><\/pre>\n\n\n\n
Connections<\/h3>\n\n\n\n
25000 \/ 18000\n=\n1.39\n\n\u2192 2 CKUs<\/code><\/pre>\n\n\n\n
Therefore mathematical minimum:<\/p>\n\n\n\n
2 CKUs<\/code><\/pre>\n\n\n\n
But look at ingress:<\/p>\n\n\n\n
2 \u00d7 60\n=\n120 MB\/sec<\/code><\/pre>\n\n\n\n
Peak traffic:<\/p>\n\n\n\n
90 MB\/sec<\/code><\/pre>\n\n\n\n
Approximate usage:<\/p>\n\n\n\n
90 \/ 120\n=\n75%<\/code><\/pre>\n\n\n\n
That’s already fairly high for expected peak production traffic.<\/p>\n\n\n\n
So I would benchmark:<\/p>\n\n\n\n
3 CKUs<\/code><\/pre>\n\n\n\n
as the stronger candidate.<\/p>\n\n\n\n
\n\n\n\n
14. Why Multi-Zone Starts at 2 CKUs<\/h1>\n\n\n\n
For a Dedicated Multi-Zone Confluent Cloud cluster<\/strong>, the current minimum is:<\/p>\n\n\n\n
2 CKUs<\/code><\/pre>\n\n\n\n
Confluent spreads a Multi-Zone Dedicated cluster across three availability zones and requires at least two CKUs. (Confluent Documentation<\/a>)<\/p>\n\n\n\n
Therefore even if your calculation says:<\/p>\n\n\n\n
0.8 CKU<\/code><\/pre>\n\n\n\n
your architecture requirement can say:<\/p>\n\n\n\n
Dedicated\n+\nMulti-Zone\n\n\u2192 minimum 2 CKUs<\/code><\/pre>\n\n\n\n
\n\n\n\n
15. Cluster Load<\/h1>\n\n\n\n
One of the most important Confluent Dedicated metrics is:<\/p>\n\n\n\n
Cluster Load %<\/code><\/pre>\n\n\n\n
It ranges from:<\/p>\n\n\n\n
0%\n\u2502\n\u2502 No load\n\n...\n\n100%\n\u2502\n\u2514 Fully saturated<\/code><\/pre>\n\n\n\n
Confluent provides both:<\/p>\n\n\n\n
Average Cluster Load<\/code><\/pre>\n\n\n\n
and:<\/p>\n\n\n\n
Maximum Cluster Load<\/code><\/pre>\n\n\n\n
(Confluent Documentation<\/a>)<\/p>\n\n\n\n
\n\n\n\n
16. Why Average AND Maximum Matter<\/h1>\n\n\n\n
Suppose:<\/p>\n\n\n\n
Average Cluster Load = 35%\n\nMaximum Cluster Load = 92%<\/code><\/pre>\n\n\n\n
That is suspicious.<\/p>\n\n\n\n
It often suggests:<\/p>\n\n\n\n
Skewed Traffic\n     \u2193\nHot Partition\n     \u2193\nOne part overloaded\n     \u2193\nRest underutilized<\/code><\/pre>\n\n\n\n
Confluent specifically recommends investigating hot partitions when maximum cluster load is high but average load is much lower. (Confluent Documentation<\/a>)<\/p>\n\n\n\n
\n\n\n\n
17. 70\u201380% Production Rule<\/h1>\n\n\n\n
Current Confluent guidance says:<\/p>\n\n\n\n
70\u201380% sustained load\n\u2192 consider adding CKUs\n\n80%+\n\u2192 expect increased throttling \/\n   degraded performance risk<\/code><\/pre>\n\n\n\n
(Confluent Documentation<\/a>)<\/p>\n\n\n\n
Your original checklist uses the same operational idea:<\/p>\n\n\n\n
Normal load < 60\u201370%\nPeak load   < 80%\nEmergency capacity available<\/code><\/pre>\n\n\n\n
This is a sensible production target.<\/p>\n\n\n\n
\n\n\n\n
18. Why Headroom Is Important<\/h1>\n\n\n\n
Suppose your cluster normally runs at:<\/p>\n\n\n\n
95%<\/code><\/pre>\n\n\n\n
Then a 20% traffic spike occurs.<\/p>\n\n\n\n
You have almost no spare capacity.<\/p>\n\n\n\n
Compare:<\/p>\n\n\n\n
Normal load\n55%\n\nTraffic spike\n+20%\n\nResult\n75%<\/code><\/pre>\n\n\n\n
Much healthier.<\/p>\n\n\n\n
Capacity headroom protects against:<\/p>\n\n\n\n
Traffic spikes\nConsumer catch-up\nNew consumers\nDeployments\nFailures\nLarge payloads\nBusiness growth\nUnexpected workloads<\/code><\/pre>\n\n\n\n
Unused capacity is not automatically waste.<\/p>\n\n\n\n
In production it can represent resilience<\/strong>.<\/p>\n\n\n\n
\n\n\n\n
19. Growth Planning<\/h1>\n\n\n\n
Suppose today’s peak ingress is:<\/p>\n\n\n\n
60 MB\/sec<\/code><\/pre>\n\n\n\n
Expected growth:<\/p>\n\n\n\n
50%<\/code><\/pre>\n\n\n\n
Next-period forecast:<\/p>\n\n\n\n
60 \u00d7 1.50\n\n=\n90 MB\/sec<\/code><\/pre>\n\n\n\n
But don’t forecast only ingress.<\/p>\n\n\n\n
Forecast:<\/p>\n\n\n\n
Ingress growth\nEgress growth\nPartition growth\nConsumer-group growth\nConnection growth<\/code><\/pre>\n\n\n\n
independently.<\/p>\n\n\n\n
For example, ingress may grow only 20%, while adding four analytics applications may make egress grow 300%.<\/p>\n\n\n\n
\n\n\n\n
20. Real-World Confluent Capacity Lab<\/h1>\n\n\n\n
Assume:<\/p>\n\n\n\n
Peak messages\/sec = 50,000\n\nAverage message\n= 2 KB\n\nConsumer groups\n= 3\n\nPartitions\n= 500\n\nGrowth forecast\n= 30%<\/code><\/pre>\n\n\n\n
Step 1 \u2014 Calculate ingress<\/h2>\n\n\n\n
50,000 \u00d7 2 KB\n\n\u2248 100 MB\/sec<\/code><\/pre>\n\n\n\n
Step 2 \u2014 Calculate egress<\/h2>\n\n\n\n
Three complete consumer groups:<\/p>\n\n\n\n
100 \u00d7 3\n\n=\n300 MB\/sec<\/code><\/pre>\n\n\n\n
Step 3 \u2014 Apply growth<\/h2>\n\n\n\n
Ingress:<\/p>\n\n\n\n
100 \u00d7 1.30\n\n=\n130 MB\/sec<\/code><\/pre>\n\n\n\n
Egress:<\/p>\n\n\n\n
300 \u00d7 1.30\n\n=\n390 MB\/sec<\/code><\/pre>\n\n\n\n
Step 4 \u2014 CKUs from ingress<\/h2>\n\n\n\n
130 \/ 60\n\n=\n2.17\n\n\u2192 3 CKUs<\/code><\/pre>\n\n\n\n
Step 5 \u2014 CKUs from egress<\/h2>\n\n\n\n
390 \/ 180\n\n=\n2.17\n\n\u2192 3 CKUs<\/code><\/pre>\n\n\n\n
Candidate:<\/p>\n\n\n\n
3 CKUs<\/code><\/pre>\n\n\n\n
\n\n\n\n
21. But the Calculation Is NOT the Final Answer<\/h1>\n\n\n\n
This is extremely important.<\/p>\n\n\n\n
Do not say:<\/p>\n\n\n\n
\n
Formula says 3 CKUs, therefore production needs exactly 3 CKUs.<\/p>\n<\/blockquote>\n\n\n\n
Instead:<\/p>\n\n\n\n
Requirements\n    \u2193\nCapacity Calculation\n    \u2193\nCandidate CKU Count\n    \u2193\nLoad Test\n    \u2193\nObserve Metrics\n    \u2193\nTune\n    \u2193\nProduction CKU Count<\/code><\/pre>\n\n\n\n
The spreadsheet gives you a starting point<\/strong>.<\/p>\n\n\n\n
The benchmark gives you the production answer<\/strong>.<\/p>\n\n\n\n
\n\n\n\n
22. What Should We Measure During the Test?<\/h1>\n\n\n\n
Monitor:<\/p>\n\n\n\n
Cluster Load\nMaximum Cluster Load\nIngress MB\/sec\nEgress MB\/sec\nProducer latency\nConsumer lag\nClient throttling\nHot partitions\nConnections\nRequest rate<\/code><\/pre>\n\n\n\n
Current Confluent guidance recommends looking at cluster load, CKU count, hot partitions, consumer lag, throttling and producer latency together when deciding whether expansion is appropriate. (Confluent Documentation<\/a>)<\/p>\n\n\n\n
\n\n\n\n
23. Troubleshooting High Load<\/h1>\n\n\n\n
Use this flow:<\/p>\n\n\n\n
Kafka Performance Problem\n          \u2502\n          \u25bc\nCheck Cluster Load\n          \u2502\n    \u250c\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2510\n    \u2502           \u2502\n High          Low\n    \u2502           \u2502\n    \u25bc           \u25bc\nCheck CKUs   Check hot\n             partitions\n                \u2502\n          \u250c\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2510\n          \u2502           \u2502\n         Hot         Balanced\n          \u2502           \u2502\n          \u25bc           \u25bc\n      Fix key      Check client\n      strategy     behavior\n                       \u2502\n                 \u250c\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2510\n                 \u25bc     \u25bc     \u25bc\n              Batching Lag Connections<\/code><\/pre>\n\n\n\n
\n\n\n\n
24. Common Mistakes<\/h1>\n\n\n\n
Mistake 1 \u2014 Sizing from average traffic<\/h3>\n\n\n\n
Average = 20 MB\/sec\n\nPeak = 100 MB\/sec<\/code><\/pre>\n\n\n\n
Always design around realistic peak load.<\/p>\n\n\n\n
Mistake 2 \u2014 Ignoring egress<\/h3>\n\n\n\n
One producer stream may have:<\/p>\n\n\n\n
5 consumer groups<\/code><\/pre>\n\n\n\n
which can make read traffic much larger than write traffic.<\/p>\n\n\n\n
Mistake 3 \u2014 Ignoring partitions<\/h3>\n\n\n\n
A cluster can hit a partition constraint even when throughput is not saturated.<\/p>\n\n\n\n
Mistake 4 \u2014 Ignoring hot partitions<\/h3>\n\n\n\n
This:<\/p>\n\n\n\n
P0 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 80%\nP1 \u2588\u2588                 7%\nP2 \u2588\u2588                 7%\nP3 \u2588\u2588                 6%<\/code><\/pre>\n\n\n\n
cannot be understood from average cluster load alone.<\/p>\n\n\n\n
Mistake 5 \u2014 Assuming CKUs solve every issue<\/h3>\n\n\n\n
Sometimes the real issue is:<\/p>\n\n\n\n
Bad partition key\nBad batching\nSlow consumer\nToo few partitions\nConnection churn\nSlow database\nBad networking<\/code><\/pre>\n\n\n\n
Mistake 6 \u2014 Running permanently at 90\u2013100%<\/h3>\n\n\n\n
It may work today.<\/p>\n\n\n\n
It has little resilience for tomorrow.<\/p>\n\n\n\n
\n\n\n\n
25. Scaling a Dedicated Cluster<\/h1>\n\n\n\n
Dedicated clusters scale by changing the CKU count.<\/p>\n\n\n\n
For example:<\/p>\n\n\n\n
confluent kafka cluster update lkc-abc123 --cku 3<\/code><\/pre>\n\n\n\n
Confluent supports increasing or reducing CKUs on a Dedicated cluster; Multi-Zone Dedicated clusters cannot be reduced below two CKUs. (Confluent Documentation<\/a>)<\/p>\n\n\n\n
\n\n\n\n
26. Capacity Planning vs Performance Tuning<\/h1>\n\n\n\n
These are related but different.<\/p>\n\n\n\n
Capacity Planning<\/th>Performance Tuning<\/th><\/tr><\/thead>
How much Kafka capacity?<\/td>How efficiently do clients use it?<\/td><\/tr>
CKUs<\/td>batch.size<\/code><\/td><\/tr>
Ingress<\/td>linger.ms<\/code><\/td><\/tr>
Egress<\/td>Compression<\/td><\/tr>
Partitions<\/td>Fetch settings<\/td><\/tr>
Connections<\/td>Producer\/consumer concurrency<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Example:<\/p>\n\n\n\n
Bad batching\n       \u2193\nToo many requests\n       \u2193\nKafka capacity wasted inefficiently<\/code><\/pre>\n\n\n\n
So:<\/p>\n\n\n\n
Capacity Planning\n        +\nProducer Tuning\n        +\nConsumer Tuning\n        +\nPartition Design\n        =\nHigh Kafka Performance<\/code><\/pre>\n\n\n\n
\n\n\n\n
27. Production Capacity Checklist<\/h1>\n\n\n\n
Before production I want these numbers:<\/p>\n\n\n\n
Peak messages\/sec\nPeak ingress MB\/sec\nPeak egress MB\/sec\n\nAverage message size\nMaximum message size\n\nTopic count\nPartition count\n\nProducer instances\nConsumer groups\nConsumer instances\n\nClient connections\nConnection attempts\n\nRetention\n\n6\u201312 month growth\n\nExpected burst traffic\n\nLatency SLA\nAvailability SLA\n\nRPO\nRTO<\/code><\/pre>\n\n\n\n
If these numbers are unknown, Kafka sizing is largely guesswork.<\/p>\n\n\n\n
\n\n\n\n
28. GOLD Production Recommendation<\/h1>\n\n\n\n
For a critical Confluent Cloud workload:<\/p>\n\n\n\n
                Workload Forecast\n                       \u2502\n                       \u25bc\n           Calculate Peak Ingress\n                       \u2502\n                       \u25bc\n            Calculate Peak Egress\n                       \u2502\n                       \u25bc\n             Check Partitions\n                       \u2502\n                       \u25bc\n             Check Connections\n                       \u2502\n                       \u25bc\n          Determine Required CKUs\n                       \u2502\n                       \u25bc\n              Add HA Minimum\n                       \u2502\n                       \u25bc\n               Add Headroom\n                       \u2502\n                       \u25bc\n                Load Test\n                       \u2502\n                       \u25bc\n             Monitor Cluster Load\n                       \u2502\n                       \u25bc\n              Production Ready<\/code><\/pre>\n\n\n\n
Golden Rule<\/h2>\n\n\n\n
\n
Size Kafka for peak traffic, keep operational headroom, and validate the calculation with a realistic benchmark.<\/strong><\/p>\n<\/blockquote>\n\n\n\n
And remember:<\/p>\n\n\n\n
Capacity Calculation\n        \u2260\nFinal Production Size\n\nCapacity Calculation\n        \u2193\nStarting Point\n        \u2193\nBenchmark\n        \u2193\nProduction Size<\/code><\/pre>\n\n\n\n
That is the GOLD-standard mental model for Confluent Cloud Kafka capacity planning<\/strong>.<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
1. What is Kafka Capacity Planning? Capacity planning means calculating how much Kafka capacity your application needs before production traffic arrives. In simple words: How big should… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1141","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1141","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=1141"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1141\/revisions"}],"predecessor-version":[{"id":1143,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/1141\/revisions\/1143"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=1141"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=1141"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=1141"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}