1. What is Kafka Capacity Planning?

Capacity planning means calculating how much Kafka capacity your application needs before production traffic arrives.

In simple words:

How big should my Kafka cluster be so that it can handle normal traffic, peak traffic, failures, and future growth without slowing down?

For a Confluent Cloud Dedicated cluster, capacity is primarily expressed using CKUs — Confluent Units for Kafka. Adding CKUs increases the capacity of the Dedicated cluster. (Confluent Documentation)

Your original production checklist already identifies the right inputs: peak produce/consume throughput, messages/sec, message size, partitions, producers, consumer groups, connections, retention, and expected growth.

---

2. Why Capacity Planning Matters

Suppose normal traffic is:

20 MB/sec

but peak traffic reaches:

100 MB/sec

If you designed Kafka only for 20 MB/sec:

Normal Traffic
     │
     ▼
   Kafka
     │
     ✅

during peak traffic:

Traffic Spike
100 MB/sec
     │
     ▼
Kafka Saturation
     │
     ▼
Throttling
     │
     ▼
Producer Latency
     │
     ▼
Consumer Lag
     │
     ▼
Application Problems

Therefore:

Never size Kafka for average traffic only.

Size primarily for:

Peak Traffic
     +
Capacity Headroom
     +
Expected Growth
     +
Failure/Burst Capacity

---

3. What is a CKU?

For a Confluent Cloud Dedicated cluster:

CKU
=
Confluent Unit for Kafka

Think of a CKU as a unit of Kafka capacity.

Current Confluent documentation gives these per-CKU planning values for Dedicated clusters: (Confluent Documentation)

Capacity dimension	Dedicated CKU
Ingress	60 MB/sec
Egress	180 MB/sec
Partitions, pre-replication	4,500
Client connections	18,000
New connection attempts	500/sec

This means you do not calculate Kafka size from only one number.

You evaluate several dimensions.

---

4. Five Important Capacity Dimensions

Think of cluster capacity as:

                   Kafka Capacity
                         │
       ┌─────────────────┼──────────────────┐
       │                 │                  │
       ▼                 ▼                  ▼
    Ingress            Egress           Partitions
                                            │
                              ┌─────────────┴────────────┐
                              ▼                          ▼
                         Connections                 Requests

You calculate each separately.

Then the largest requirement normally determines the starting CKU count.

---

5. Dimension 1 — Ingress

What is ingress?

Ingress means:

Data being written into Kafka.

Producer
   │
   │ Events
   ▼
 Kafka

Suppose:

Messages/sec = 100,000
Average message = 1 KB

Approximate ingress:

100,000 × 1 KB

≈ 100 MB/sec

A Dedicated CKU currently provides approximately:

60 MB/sec ingress

So:

Required CKUs
=
100 / 60

=
1.67

Always round up:

2 CKUs

But there is a problem.

Two CKUs provide approximately:

2 × 60
=
120 MB/sec

Your peak is:

100 MB/sec

Usage:

100 / 120
≈ 83%

That leaves little headroom.

A better production candidate could therefore be:

3 CKUs

Then:

3 × 60
=
180 MB/sec

100 / 180
≈ 56%

Much healthier.

---

6. Dimension 2 — Egress

What is egress?

Egress means:

Data consumers read out of Kafka.

Example:

                   Kafka
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       Billing    Analytics    Fraud

This is extremely important because multiple consumer groups independently consume the same records.

Suppose:

Producer throughput = 50 MB/sec

And four consumer groups read every event:

Billing      = 50 MB/sec
Analytics    = 50 MB/sec
Fraud        = 50 MB/sec
Search       = 50 MB/sec

Total egress:

50 × 4
=
200 MB/sec

Notice:

Ingress = 50 MB/sec

Egress = 200 MB/sec

Therefore:

Do not assume ingress and egress are equal.

---

7. Egress CKU Example

Dedicated CKU egress capacity is currently approximately:

180 MB/sec

If you need:

200 MB/sec

calculate:

200 / 180

=
1.11

Round up:

2 CKUs

Again, production headroom still needs to be considered.

---

8. Dimension 3 — Partitions

A Kafka topic is divided into partitions:

orders

├── P0
├── P1
├── P2
├── P3
├── P4
└── ...

Partitions provide much of Kafka’s:

Parallelism
Scalability
Consumer concurrency

A Dedicated CKU currently supports up to 4,500 pre-replication partitions as the published capacity dimension. (Confluent Documentation)

Suppose your cluster needs:

8,000 partitions

Then:

8,000 / 4,500

=
1.78

Round up:

2 CKUs

Important

This does not mean:

“4,500 partitions per CKU is the recommended number of partitions.”

It means it is a capacity limit/dimension.

Your actual partition count should depend on:

Throughput
+
Consumer Parallelism
+
Ordering Requirements
+
Key Distribution

We will cover this deeply in the dedicated Partition Strategy tutorial.

---

9. Dimension 4 — Connections

Kafka clients maintain connections to Kafka.

Examples:

Producers
Consumers
Kafka Connect
Kafka Streams
Flink
Admin clients
Cluster Linking

Current Dedicated guidance provides:

18,000 client connections / CKU

as the per-CKU capacity dimension. (Confluent Documentation)

Suppose:

Expected connections
=
25,000

Then:

25,000 / 18,000

=
1.39

Round up:

2 CKUs

---

10. Connections vs Connection Attempts

These are different.

Existing connections

Application
     │
     └──────────── persistent connection ─────── Kafka

Connection attempts

Connect
Disconnect

Connect
Disconnect

Connect
Disconnect

Repeated connection creation can create additional load.

Kafka clients should normally be long-lived.

Bad:

HTTP request
   ↓
Create producer
   ↓
Produce
   ↓
Close producer

Better:

Application starts
       ↓
Create Producer
       ↓
Reuse Producer
       ↓
Application shuts down
       ↓
Close Producer

---

11. Dimension 5 — Request Rate

Suppose Producer A sends:

1 message
per Kafka request

while Producer B sends:

500 messages
per Kafka request

Producer A creates far more requests.

Tiny Messages
+
No Batching
=
Lots of Kafka Requests

That is one reason why producer batching matters.

Later, in the Producer Performance tutorial, we will tune:

batch.size
linger.ms
compression.type
buffer.memory

---

12. The Master CKU Formula

Calculate each dimension separately.

Ingress CKUs
=
Peak Ingress / CKU Ingress Capacity

Egress CKUs
=
Peak Egress / CKU Egress Capacity

Partition CKUs
=
Partition Count / CKU Partition Capacity

Connection CKUs
=
Connections / CKU Connection Capacity

Then:

Required CKUs
=
MAX(
    Ingress CKUs,
    Egress CKUs,
    Partition CKUs,
    Connection CKUs,
    Other Capacity Constraints
)

Finally:

Round Up
    +
HA requirement
    +
Headroom
    +
Expected growth
    +
Benchmark result

---

13. Complete Real-World Example

Suppose an e-commerce platform expects:

Peak ingress    = 90 MB/sec
Peak egress     = 250 MB/sec
Partitions      = 8,000
Connections     = 25,000

Ingress

90 / 60
=
1.5

→ 2 CKUs

Egress

250 / 180
=
1.39

→ 2 CKUs

Partitions

8000 / 4500
=
1.78

→ 2 CKUs

Connections

25000 / 18000
=
1.39

→ 2 CKUs

Therefore mathematical minimum:

2 CKUs

But look at ingress:

2 × 60
=
120 MB/sec

Peak traffic:

90 MB/sec

Approximate usage:

90 / 120
=
75%

That’s already fairly high for expected peak production traffic.

So I would benchmark:

3 CKUs

as the stronger candidate.

---

14. Why Multi-Zone Starts at 2 CKUs

For a Dedicated Multi-Zone Confluent Cloud cluster, the current minimum is:

2 CKUs

Confluent spreads a Multi-Zone Dedicated cluster across three availability zones and requires at least two CKUs. (Confluent Documentation)

Therefore even if your calculation says:

0.8 CKU

your architecture requirement can say:

Dedicated
+
Multi-Zone

→ minimum 2 CKUs

---

15. Cluster Load

One of the most important Confluent Dedicated metrics is:

Cluster Load %

It ranges from:

0%
│
│ No load

...

100%
│
└ Fully saturated

Confluent provides both:

Average Cluster Load

and:

Maximum Cluster Load

(Confluent Documentation)

---

16. Why Average AND Maximum Matter

Suppose:

Average Cluster Load = 35%

Maximum Cluster Load = 92%

That is suspicious.

It often suggests:

Skewed Traffic
     ↓
Hot Partition
     ↓
One part overloaded
     ↓
Rest underutilized

Confluent specifically recommends investigating hot partitions when maximum cluster load is high but average load is much lower. (Confluent Documentation)

---

17. 70–80% Production Rule

Current Confluent guidance says:

70–80% sustained load
→ consider adding CKUs

80%+
→ expect increased throttling /
   degraded performance risk

(Confluent Documentation)

Your original checklist uses the same operational idea:

Normal load < 60–70%
Peak load   < 80%
Emergency capacity available

This is a sensible production target.

---

18. Why Headroom Is Important

Suppose your cluster normally runs at:

95%

Then a 20% traffic spike occurs.

You have almost no spare capacity.

Compare:

Normal load
55%

Traffic spike
+20%

Result
75%

Much healthier.

Capacity headroom protects against:

Traffic spikes
Consumer catch-up
New consumers
Deployments
Failures
Large payloads
Business growth
Unexpected workloads

Unused capacity is not automatically waste.

In production it can represent resilience.

---

19. Growth Planning

Suppose today’s peak ingress is:

60 MB/sec

Expected growth:

50%

Next-period forecast:

60 × 1.50

=
90 MB/sec

But don’t forecast only ingress.

Forecast:

Ingress growth
Egress growth
Partition growth
Consumer-group growth
Connection growth

independently.

For example, ingress may grow only 20%, while adding four analytics applications may make egress grow 300%.

---

20. Real-World Confluent Capacity Lab

Assume:

Peak messages/sec = 50,000

Average message
= 2 KB

Consumer groups
= 3

Partitions
= 500

Growth forecast
= 30%

Step 1 — Calculate ingress

50,000 × 2 KB

≈ 100 MB/sec

Step 2 — Calculate egress

Three complete consumer groups:

100 × 3

=
300 MB/sec

Step 3 — Apply growth

Ingress:

100 × 1.30

=
130 MB/sec

Egress:

300 × 1.30

=
390 MB/sec

Step 4 — CKUs from ingress

130 / 60

=
2.17

→ 3 CKUs

Step 5 — CKUs from egress

390 / 180

=
2.17

→ 3 CKUs

Candidate:

3 CKUs

---

21. But the Calculation Is NOT the Final Answer

This is extremely important.

Do not say:

Formula says 3 CKUs, therefore production needs exactly 3 CKUs.

Instead:

Requirements
    ↓
Capacity Calculation
    ↓
Candidate CKU Count
    ↓
Load Test
    ↓
Observe Metrics
    ↓
Tune
    ↓
Production CKU Count

The spreadsheet gives you a starting point.

The benchmark gives you the production answer.

---

22. What Should We Measure During the Test?

Monitor:

Cluster Load
Maximum Cluster Load
Ingress MB/sec
Egress MB/sec
Producer latency
Consumer lag
Client throttling
Hot partitions
Connections
Request rate

Current Confluent guidance recommends looking at cluster load, CKU count, hot partitions, consumer lag, throttling and producer latency together when deciding whether expansion is appropriate. (Confluent Documentation)

---

23. Troubleshooting High Load

Use this flow:

Kafka Performance Problem
          │
          ▼
Check Cluster Load
          │
    ┌─────┴─────┐
    │           │
 High          Low
    │           │
    ▼           ▼
Check CKUs   Check hot
             partitions
                │
          ┌─────┴─────┐
          │           │
         Hot         Balanced
          │           │
          ▼           ▼
      Fix key      Check client
      strategy     behavior
                       │
                 ┌─────┼─────┐
                 ▼     ▼     ▼
              Batching Lag Connections

---

24. Common Mistakes

Mistake 1 — Sizing from average traffic

Average = 20 MB/sec

Peak = 100 MB/sec

Always design around realistic peak load.

Mistake 2 — Ignoring egress

One producer stream may have:

5 consumer groups

which can make read traffic much larger than write traffic.

Mistake 3 — Ignoring partitions

A cluster can hit a partition constraint even when throughput is not saturated.

Mistake 4 — Ignoring hot partitions

This:

P0 █████████████████ 80%
P1 ██                 7%
P2 ██                 7%
P3 ██                 6%

cannot be understood from average cluster load alone.

Mistake 5 — Assuming CKUs solve every issue

Sometimes the real issue is:

Bad partition key
Bad batching
Slow consumer
Too few partitions
Connection churn
Slow database
Bad networking

Mistake 6 — Running permanently at 90–100%

It may work today.

It has little resilience for tomorrow.

---

25. Scaling a Dedicated Cluster

Dedicated clusters scale by changing the CKU count.

For example:

confluent kafka cluster update lkc-abc123 --cku 3

Confluent supports increasing or reducing CKUs on a Dedicated cluster; Multi-Zone Dedicated clusters cannot be reduced below two CKUs. (Confluent Documentation)

---

26. Capacity Planning vs Performance Tuning

These are related but different.

Capacity Planning	Performance Tuning
How much Kafka capacity?	How efficiently do clients use it?
CKUs	batch.size
Ingress	linger.ms
Egress	Compression
Partitions	Fetch settings
Connections	Producer/consumer concurrency

Example:

Bad batching
       ↓
Too many requests
       ↓
Kafka capacity wasted inefficiently

So:

Capacity Planning
        +
Producer Tuning
        +
Consumer Tuning
        +
Partition Design
        =
High Kafka Performance

---

27. Production Capacity Checklist

Before production I want these numbers:

Peak messages/sec
Peak ingress MB/sec
Peak egress MB/sec

Average message size
Maximum message size

Topic count
Partition count

Producer instances
Consumer groups
Consumer instances

Client connections
Connection attempts

Retention

6–12 month growth

Expected burst traffic

Latency SLA
Availability SLA

RPO
RTO

If these numbers are unknown, Kafka sizing is largely guesswork.

---

28. GOLD Production Recommendation

For a critical Confluent Cloud workload:

                Workload Forecast
                       │
                       ▼
           Calculate Peak Ingress
                       │
                       ▼
            Calculate Peak Egress
                       │
                       ▼
             Check Partitions
                       │
                       ▼
             Check Connections
                       │
                       ▼
          Determine Required CKUs
                       │
                       ▼
              Add HA Minimum
                       │
                       ▼
               Add Headroom
                       │
                       ▼
                Load Test
                       │
                       ▼
             Monitor Cluster Load
                       │
                       ▼
              Production Ready

Golden Rule

Size Kafka for peak traffic, keep operational headroom, and validate the calculation with a realistic benchmark.

And remember:

Capacity Calculation
        ≠
Final Production Size

Capacity Calculation
        ↓
Starting Point
        ↓
Benchmark
        ↓
Production Size

That is the GOLD-standard mental model for Confluent Cloud Kafka capacity planning.