1. Cluster Architecture

• Use a Dedicated Kafka cluster for critical/high-throughput production workloads.
• Select MULTI_ZONE / High Availability.
• Use at least 2 CKUs because Confluent requires 2 CKUs for multi-zone Dedicated clusters.
• Do not size production at the absolute minimum; leave capacity headroom.
• Keep applications and Kafka in the same cloud provider and region wherever possible.
• Avoid application → Kafka cross-region traffic on the normal data path.
• Use private networking such as PrivateLink/peering where security architecture requires it.
• Separate Production, Staging and Development clusters/environments.

Recommended architecture:

                 Availability Zone A
                       │
                 Kafka Replica
                       │
Producer ───────► Kafka Leader
                       │
                 Kafka Replica
                       │
                 Availability Zone B/C

                       │
                       ▼

                  Consumer Group
              ┌────────┼────────┐
              ▼        ▼        ▼
             C1       C2       C3

---

2. Capacity Planning

• Determine peak MB/sec produced.
• Determine peak MB/sec consumed.
• Determine messages/sec.
• Determine average and maximum message size.
• Determine expected partition count.
• Determine number of producers.
• Determine number of consumer groups.
• Determine number of consumers.
• Determine client connection count.
• Determine retention requirements.
• Determine expected growth for at least the next 6–12 months.
• Size CKUs for peak, not average traffic.
• Maintain spare capacity for broker/zone failures and traffic spikes.

Confluent recommends monitoring cluster load closely. Sustained load around 70–80% is a reason to consider adding CKUs, while above 80% can result in increased latency and throttling.

My production target would normally be:

Normal load       < 60–70%
Peak load         < 80%
Emergency capacity available

---

3. Topic Replication

Use:

replication.factor=3
min.insync.replicas=2

Confluent Cloud’s default replication factor is 3, and min.insync.replicas defaults to 2.

• Replication factor = 3
• min.insync.replicas=2
• Do not reduce replication simply to improve throughput.
• Do not use RF=1 for critical production data.

This gives the classic durable Kafka combination:

RF = 3

Broker 1     Broker 2     Broker 3
 Leader       Replica      Replica
   │             │             │
   └─────────────┴─────────────┘

min.insync.replicas = 2

---

4. Producer Durability

For critical production workloads:

acks=all
enable.idempotence=true

• acks=all
• enable.idempotence=true
• Enable retries.
• Use sensible delivery.timeout.ms.
• Do not use acks=0.
• Avoid acks=1 for business-critical data.

The important combination is:

Producer
   │
   │ acks=all
   ▼
Leader
   │
   ├──── Replica
   │
   └──── Replica

       ↓

ACK returned

acks=1 can improve raw throughput, and Confluent lists it as a throughput optimization, but it weakens the durability guarantee. For a system where performance and HA both matter, I would keep acks=all.

---

5. Producer Performance

Start with approximately:

acks=all

enable.idempotence=true

compression.type=lz4

batch.size=100000

linger.ms=10

buffer.memory=67108864

Then benchmark.

• Enable producer batching.
• Start batch.size around 100–200 KB.
• Start linger.ms around 10 ms.
• Test 10–50 ms for throughput-oriented workloads.
• Consider up to ~100 ms where throughput matters far more than latency.
• Use compression.type=lz4 for strong performance.
• Increase buffer.memory if producers write heavily across many partitions.
• Monitor producer buffer exhaustion.
• Avoid producing thousands of tiny requests.

Confluent specifically recommends larger batches, increasing linger.ms, and lz4 for throughput optimization. Their throughput guidance suggests approximately batch.size=100000–200000 and linger.ms=10–100.

---

6. Compression

Recommended starting point:

compression.type=lz4

My preference:

Codec	Throughput	Compression	CPU
none	High network usage	None	Very low
lz4	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐⭐
snappy	⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐
zstd	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐
gzip	⭐⭐	⭐⭐⭐⭐⭐	⭐

For performance-oriented Kafka:

LZ4 = excellent default

Confluent explicitly recommends LZ4 for performance rather than gzip.

---

7. Partition Strategy

Partitions are one of the most important performance decisions.

• Estimate partitions before creating heavily keyed topics.
• Create enough partitions to support required producer/consumer parallelism.
• Avoid blindly creating hundreds/thousands of partitions.
• Monitor partition traffic distribution.
• Avoid hot partitions.
• Design message keys carefully.
• Don’t use one permanently hot key for large portions of traffic.
• Remember:

Maximum useful consumers
in one consumer group

≈

number of topic partitions

Example:

Topic = orders

P0 ─────────► Consumer 1
P1 ─────────► Consumer 2
P2 ─────────► Consumer 3
P3 ─────────► Consumer 4

Confluent uses approximately 6–10 partitions per CKU as a useful parallelism guideline when diagnosing under-parallelized Dedicated clusters, although the correct number depends on your workload.

---

8. Avoid Hot Partitions

Bad:

P0 █████████████████████ 80%
P1 ██                     5%
P2 ██                     5%
P3 ████                  10%

Good:

P0 ██████ 25%
P1 ██████ 25%
P2 ██████ 25%
P3 ██████ 25%

• Choose high-cardinality keys where ordering requirements allow.
• Monitor per-partition traffic.
• Find dominant keys.
• Reconsider partition-key strategy when traffic is skewed.
• Use key salting only where ordering semantics permit it.

Confluent identifies skewed/hot partitions as a direct source of throttling and performance problems.

---

9. Consumer Performance

Start around:

fetch.min.bytes=100000

fetch.max.wait.ms=500

Then tune according to latency requirements.

• Increase fetch.min.bytes for throughput.
• Configure fetch.max.bytes appropriately.
• Configure max.partition.fetch.bytes according to maximum record/batch size.
• Set max.poll.records according to processing capacity.
• Set max.poll.interval.ms longer than worst-case batch processing time.
• Scale consumers horizontally.
• Keep consumer processing fast.
• Move expensive processing away from the poll loop where architecture permits it.

Confluent recommends approximately fetch.min.bytes=100000 for throughput-oriented consumers.

---

10. Consumer Parallelism

If there are:

16 partitions

then a natural upper bound for active consumers in the same group is:

16 consumers

Example:

16 Partitions
       │
       ▼
 Consumer Group
       │
 ┌─────┼────────────────┐
 ▼     ▼                ▼
C1    C2 ...           C16

Adding:

C17
C18

doesn’t create additional partition-level parallelism for that topic.

• Consumers <= useful partition parallelism
• Scale consumers when consumer lag grows.
• Increase partitions when justified by sustained parallelism requirements.

---

11. Consumer Rebalancing

• Avoid constantly starting/stopping consumers.
• Keep consumer processing time below max.poll.interval.ms.
• Monitor rebalance frequency.
• Monitor rebalance duration.
• Use the newer consumer group protocol where supported and appropriate.
• Test application deployments for rebalance storms.

Confluent recommends monitoring rebalance frequency and notes that the newer consumer group protocol can reduce rebalance impact compared with the classic protocol.

---

12. Message Size

Best:

Small Kafka messages

rather than:

10 MB
20 MB
50 MB
...

• Keep records compact.
• Avoid putting large binary files directly into Kafka.
• Store large objects in object storage such as S3/GCS/Azure Blob.
• Send object metadata/reference through Kafka where appropriate.
• Make producer, broker/topic and consumer maximum sizes compatible.

Pattern:

Kafka message
     │
     ├── customerId
     ├── eventType
     ├── timestamp
     └── s3://bucket/object

rather than putting the entire large file into Kafka.

---

13. Schema Management

• Use Schema Registry.
• Prefer Avro, Protobuf or JSON Schema.
• Define compatibility policy.
• Prefer BACKWARD compatibility for many event-streaming systems.
• Prevent breaking schema changes in CI/CD.
• Version schemas through governance rather than application-specific conventions.

Example:

Producer
   │
   ├──── Schema Registry
   │
   ▼
 Kafka
   │
   ▼
Consumer

---

14. Retention

Never leave retention decisions accidental.

Configure according to the use case:

retention.ms=...

• Define business retention requirements.
• Define replay requirements.
• Define regulatory requirements.
• Use appropriate retention for high-volume topics.
• Use compaction for state/change-log use cases where appropriate.

Examples:

Transaction Events → 30 days

Telemetry → 7 days

Audit Events → 1 year

Current User State → compacted topic

---

15. Networking

Ideal:

AWS Application
      │
      │ same AWS region
      ▼
Confluent Cloud Kafka

Avoid:

AWS Tokyo
   │
   │ WAN
   ▼
Kafka Virginia

unless the architecture genuinely requires it.

• Application and Kafka in same region where possible.
• Minimize network hops.
• Avoid unnecessary proxies.
• Avoid repeatedly opening Kafka connections.
• Reuse long-lived Kafka clients.
• Monitor network latency.
• Monitor connection count.
• Monitor connection creation rate.

---

16. Connection Management

Kafka clients are intended to be long-lived.

Bad:

Request
 ↓
Create Producer
 ↓
Produce
 ↓
Close Producer

Better:

Application starts
       │
       ▼
Create Producer
       │
       ▼
Reuse for millions of messages
       │
       ▼
Application shuts down

• Reuse producers.
• Reuse consumers.
• Avoid connection churn.
• Monitor active_connection_count.

Confluent notes that excessive client connections can result in sharply increased producer latency.

---

17. Security

• TLS enabled.
• SASL authentication.
• Use separate service accounts per application/team.
• Apply least-privilege RBAC/ACLs.
• Do not share production API keys.
• Store secrets in Vault/AWS Secrets Manager/etc.
• Rotate credentials.
• Restrict network access.
• Maintain audit logs.
• Separate human identities from application identities.

---

18. Monitoring

These metrics should absolutely be monitored.

Cluster

• Cluster load %
• Maximum cluster load
• CKU utilization/count
• Produce throughput
• Fetch throughput
• Request rate
• Client throttling
• Active connections
• Hot partitions

Producer

• Producer request latency
• Produce rate
• Error rate
• Retry rate
• Record-send rate
• Buffer availability
• Buffer wait time
• Throttle time

Consumer

• Consumer lag
• Consumer latency
• Records consumed/sec
• Fetch rate
• Fetch latency
• Consumer throttle time
• Rebalance frequency
• Rebalance duration

Confluent specifically recommends monitoring cluster load, hot partitions, consumer lag, throttling and producer latency when evaluating Dedicated cluster performance.

---

19. Consumer Lag Alerts

This is one of my highest-priority alerts:

Producer Rate
      │
      │
      ▼
Kafka
      │
      │
      ▼
Consumer
      │
      ▼
Consumer Lag

Alert when:

lag continuously increases

rather than merely reacting to one temporary spike.

• Alert on large absolute lag.
• Alert on continuously growing lag.
• Alert on consumer latency.
• Alert if consumer group disappears unexpectedly.
• Correlate lag with cluster load.
• Correlate lag with consumer CPU.
• Correlate lag with rebalances.

Confluent explicitly recommends monitoring lag trends rather than only absolute values.

---

20. Scaling Strategy

Use this decision flow:

Consumer lag increasing?
       │
       ▼
Cluster load >70%?
       │
   ┌───┴───┐
  YES      NO
   │        │
   ▼        ▼
Add CKU   Enough partitions?
             │
         ┌───┴────┐
        NO       YES
         │         │
         ▼         ▼
 Add partitions  Add consumers

This closely follows Confluent’s troubleshooting recommendation for Dedicated clusters.

---

21. Disaster Recovery

Multi-zone is not the same thing as multi-region DR.

For zonal failures:

Region A

AZ1 + AZ2 + AZ3

Multi-zone Kafka handles infrastructure/AZ resilience.

For regional failures:

         Primary
      Region Tokyo
           │
           │ Cluster Linking
           ▼
        DR Cluster
      Region Osaka/etc.

• Define RPO.
• Define RTO.
• Provision secondary-region Kafka where required.
• Use Cluster Linking where appropriate.
• Replicate required topics.
• Synchronize required consumer offsets.
• Synchronize required ACLs.
• Pre-create DR credentials.
• Keep bootstrap servers outside application code.
• Test DR failover regularly.
• Monitor replication/mirror lag.

Confluent recommends Cluster Linking for multi-region DR and supports replication of topic data, consumer offsets and ACLs.

---

22. Benchmark Before Production

Never trust configuration recommendations without measuring your actual workload.

Test:

10K msg/sec
25K msg/sec
50K msg/sec
100K msg/sec
200K msg/sec
...

Measure:

Throughput
Latency p50
Latency p95
Latency p99
Producer errors
Consumer lag
Cluster load
Throttle time
CPU
Memory
Network

Also test:

• Normal traffic.
• 2× normal traffic.
• Peak traffic.
• Sudden burst.
• Producer restart.
• Consumer restart.
• Consumer scale-out.
• Broker/AZ disturbance where test environment permits it.
• Schema changes.
• Large messages.
• Consumer slowdown.
• DR failover.

---

23. My Recommended Production Baseline

Cluster

Confluent Cloud Dedicated
Multi-Zone
RF = 3
min ISR = 2
CKUs = benchmark-derived with >=20–30% capacity headroom

Producer

acks=all
enable.idempotence=true

compression.type=lz4

batch.size=100000
linger.ms=10

# Increase where partition count / throughput requires it
buffer.memory=67108864

Consumer

fetch.min.bytes=100000
fetch.max.wait.ms=500

Then tune:

max.partition.fetch.bytes
fetch.max.bytes
max.poll.records
max.poll.interval.ms

according to message size and processing time.

---

24. The 15 Things I Would Check First

If I inherited a production Confluent Cloud Kafka cluster tomorrow, these would be my first checks:

• Dedicated cluster
• Multi-zone enabled
• RF = 3
• min ISR = 2
• Producer acks=all
• Producer idempotence enabled
• LZ4 compression
• Producer batching enabled
• Correct partition count
• No hot partitions
• Enough consumer instances
• Consumer lag monitored
• Cluster load below ~70–80% sustained
• No client throttling
• Multi-region DR designed separately where business RTO/RPO requires it

Golden Rule

                 Kafka Performance
                       │
       ┌───────────────┼────────────────┐
       │               │                │
       ▼               ▼                ▼
   Partitions       Batching        Compression
       │               │                │
       └───────────────┼────────────────┘
                       ▼
                  Parallelism
                       │
                       ▼
                  More Throughput

             while preserving:

             RF=3
             min ISR=2
             acks=all
             idempotence=true

Do not gain performance by sacrificing reliability first.

Gain performance primarily through:

correct partitioning → batching → compression → consumer parallelism → sufficient CKU capacity → good network placement.