- \n
- Topic<\/strong>: named stream of records.<\/li>\n\n\n\n
- Partition<\/strong>: an ordered shard of a topic. Ordering is guaranteed within<\/em> a partition.<\/li>\n\n\n\n
- Consumer group<\/strong>: a logical application reading a topic. As a group, each record is processed once.<\/li>\n\n\n\n
- Consumer process (a.k.a. worker \/ pod \/ container \/ JVM)<\/strong>: a running OS process that joins a consumer group.<\/li>\n\n\n\n
- KafkaConsumer instance<\/strong>: the client object<\/strong> in your code (e.g.,
new KafkaConsumer<>(props)<\/code> in Java) that talks to brokers, polls, tracks offsets. Kafka assigns partitions to instances<\/strong>.<\/li>\n\n\n\n- Thread<\/strong>: execution lane inside a process. You may run zero, one, or many KafkaConsumer instances per process\u2014typically one per thread.<\/li>\n<\/ul>\n\n\n\n
\n
Rule of exclusivity:<\/strong> At any moment, one partition is owned by exactly one KafkaConsumer instance in a group<\/strong>. No two instances in the same group read the same partition concurrently.<\/p>\n\n\n\n
Architecture: Topic \u2192 Partitions \u2192 Consumer Group \u2192 Processes \u2192 Threads \u2192 KafkaConsumer instances<\/h2>\n\n\n\n
flowchart TB\n %% ===== Topic & Partitions =====\n subgraph T[\"Topic: orders (6 partitions)\"]\n direction LR\n P0([\"Partition P0n(offsets: 0..\u221e)\"])\n P1([\"Partition P1n(offsets: 0..\u221e)\"])\n P2([\"Partition P2n(offsets: 0..\u221e)\"])\n P3([\"Partition P3n(offsets: 0..\u221e)\"])\n P4([\"Partition P4n(offsets: 0..\u221e)\"])\n P5([\"Partition P5n(offsets: 0..\u221e)\"])\n end\n\n %% ===== Consumer Group =====\n subgraph G[\"Consumer Group: orders-service\"]\n direction TB\n\n %% ---- Process A (Worker\/Pod\/Container) ----\n subgraph A[\"Consumer Process A (Worker A \/ Pod)\"]\n direction TB\n ATh1[\"Thread 1nKafkaConsumer #1\"]\n ATh2[\"Thread 2nKafkaConsumer #2\"]\n end\n\n %% ---- Process B (Worker\/Pod\/Container) ----\n subgraph B[\"Consumer Process B (Worker B \/ Pod)\"]\n direction TB\n BTh1[\"Thread 1nKafkaConsumer #3\"]\n end\n end\n\n %% ===== Active assignment (example) =====\n P0 -- \"assigned to\" --> ATh1\n P3 -- \"assigned to\" --> ATh1\n\n P1 -- \"assigned to\" --> ATh2\n P4 -- \"assigned to\" --> ATh2\n\n P2 -- \"assigned to\" --> BTh1\n P5 -- \"assigned to\" --> BTh1\n\n %% ===== Notes \/ Rules =====\n Rules[\"Rules:\n \u2022 One partition \u2192 at most one active KafkaConsumer instance in the group.\n \u2022 A KafkaConsumer instance may own 1..N partitions.\n \u2022 KafkaConsumer is NOT thread-safe \u2192 one thread owns poll\/commit for that instance.\n \u2022 Ordering is guaranteed per partition (by offset), not across partitions.\"]\n classDef note fill:#fff,stroke:#bbb,stroke-dasharray: 3 3,color:#333;\n Rules:::note\n G --- Rules\n<\/pre><\/div>\n\n\n\n
Partition assignment scenarios (how partitions are spread across instances)<\/h2>\n\n\n\n
flowchart LR\n %% ===== Scenario A =====\n subgraph S1[\"Scenario A: 7 partitions (P0..P6), 3 consumer instances (C1..C3)\"]\n direction TB\n C1A[\"C1 \u279c P0, P3, P6\"]\n C2A[\"C2 \u279c P1, P4\"]\n C3A[\"C3 \u279c P2, P5\"]\n end\n\n %% ===== Scenario B =====\n subgraph S2[\"Scenario B: Add C4 (cooperative-sticky rebalance)\"]\n direction TB\n C1B[\"C1 \u279c P0, P6\"]\n C2B[\"C2 \u279c P1, P4\"]\n C3B[\"C3 \u279c P2\"]\n C4B[\"C4 \u279c P3, P5\"]\n end\n\n %% ===== Scenario C =====\n subgraph S3[\"Scenario C: 4 partitions (P0..P3), 6 consumer instances (C1..C6)\"]\n direction TB\n C1C[\"C1 \u279c P0\"]\n C2C[\"C2 \u279c P1\"]\n C3C[\"C3 \u279c P2\"]\n C4C[\"C4 \u279c P3\"]\n C5C[\"C5 \u279c (idle)\"]\n C6C[\"C6 \u279c (idle)\"]\n end\n\n classDef idle fill:#fafafa,stroke:#ccc,color:#999;\n C5C:::idle; C6C:::idle\n<\/pre>
![\"\"](\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2025\/09\/merpress-1.png\")
<\/div>\n\n\n\nBest practices & limitations (mind map)<\/h2>\n\n\n\n
mindmap\n root((Partitioning and assignment))\n Assignment strategies\n Range\n Contiguous ranges per topic\n Can skew when topics or partition counts differ\n Round robin\n Even spread over time\n Less stable across rebalances\n Sticky\n Keeps prior mapping to reduce churn\n Cooperative sticky\n Incremental rebalance with fewer pauses\n Recommended default in modern clients\n Best practices\n Plan partition count\n Estimate peak throughput in MB per second\n Target around 1 to 3 MB per second per partition\n Add 1.5 to 2x headroom\n Keying\n Key by entity to preserve per key order\n Avoid hot keys and ensure good spread\n Use a custom consistent hash partitioner if needed\n Stability\n Use static membership with group.instance.id\n Use cooperative sticky assignor\n Ordering safe processing\n Per partition single thread executors\n Or per key serial queues on top of a worker pool\n Track contiguous offsets and commit after processing\n Backpressure\n Bounded queues with pause and resume when saturated\n Keep poll loop fast and do IO on worker threads\n Consumer safety\n KafkaConsumer is not thread safe\n One consumer instance per poll and commit thread\n Disable auto commit and commit on completion\n Limitations and trade offs\n No global order across partitions\n Max parallelism per group equals number of partitions\n Hot partitions bottleneck throughput\n Too many partitions increase metadata and rebalance time\n Changing partition count remaps hash of key modulo N\n<\/pre><\/div>\n\n\n\n
How messages land in partitions (with\/without keys)<\/h2>\n\n\n\n
flowchart TB\n subgraph Prod[\"Producer\"]\n direction TB\n K1[\"send(key=A, value=m1)\"]\n K2[\"send(key=B, value=m2)\"]\n K3[\"send(key=A, value=m3)\"]\n K4[\"send(key=C, value=m4)\"]\n end\n\n subgraph Partitions[\"Topic: orders (3 partitions)\"]\n direction LR\n X0[\"P0noffsets 0..\u221e\"]\n X1[\"P1noffsets 0..\u221e\"]\n X2[\"P2noffsets 0..\u221e\"]\n end\n\n note1[\"Default partitioner:npartition = hash(key) % numPartitionsn(no key \u21d2 sticky batching across partitions)\"]\n classDef note fill:#fff,stroke:#bbb,stroke-dasharray: 3 3,color:#333;\n note1:::note\n\n Prod --- note1\n K1 --> X1\n K2 --> X0\n K3 --> X1\n K4 --> X2\n\n subgraph Order[\"Per-partition order\"]\n direction LR\n X1o[\"P1 offsets:n0: m1(A)n1: m3(A)\"]\n X0o[\"P0 offsets:n0: m2(B)\"]\n X2o[\"P2 offsets:n0: m4(C)\"]\n end\n\n X1 --> X1o\n X0 --> X0o\n X2 --> X2o\n<\/pre><\/div>\n\n\n\n
What is a partition?<\/h1>\n\n\n\n
- \n
- Core idea:<\/strong> A partition is one append-only log<\/strong> that belongs to a topic.<\/li>\n\n\n\n
- Physical & logical:<\/strong>\n
- \n
- Logical<\/em> shard for routing and parallelism (clients see \u201cP0, P1, \u2026\u201d).<\/li>\n\n\n\n
- Physical<\/em> files on a broker (leader) with replicas<\/strong> on other brokers for HA.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Offsets:<\/strong> Every record in a partition gets a strictly increasing offset<\/strong> (0,1,2,\u2026).
Ordering is guaranteed within that partition<\/strong> (by offset). There is no global order across partitions<\/strong>.<\/li>\n<\/ul>\n\n\n\nHow messages are placed into partitions<\/h1>\n\n\n\n
Producers<\/strong> decide the target partition using a partitioner<\/strong>:<\/p>\n\n\n\n
- \n
- If you set a key<\/strong> (
producer.send(topic, key, value)<\/code>):
\u2192 Kafka uses a hash of the key:partition = hash(key) % numPartitions<\/code>.\n- \n
- All messages with the same key<\/strong> go to the same partition<\/strong> \u2192 preserves per-key order.<\/li>\n\n\n\n
- Watch out for hot keys<\/strong> (one key getting huge traffic) \u2192 one partition becomes a bottleneck.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- If you don\u2019t set a key<\/strong> (key = null):\n
- \n
- The default Java producer uses a sticky partitioner<\/strong> (since Kafka 2.4): it sends a batch<\/em> to one partition to maximize throughput, then switches.<\/li>\n\n\n\n
- Older behavior was round-robin. Either way, distribution is roughly even over time<\/strong> but not ordered across partitions<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n
\n
You can also plug a custom partitioner<\/strong> if you need special routing (e.g., consistent hashing).<\/p>\n<\/blockquote>\n\n\n\n
Example: 7 messages into a topic with 3 partitions<\/h1>\n\n\n\n
Assume topic
orders<\/code> has partitions: P0, P1, P2<\/strong>.<\/p>\n\n\n\nA) No keys (illustrative round-robin for clarity)<\/h2>\n\n\n\n
Messages:
m1, m2, m3, m4, m5, m6, m7<\/code><\/p>\n\n\n\nA simple spread might look like:<\/p>\n\n\n
P0<\/span>: m1 (offset 0), m4 (1), m7 (2)\nP1<\/span>: m2 (offset 0), m5 (1)\nP2<\/span>: m3 (offset 0), m6 (1)\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\nNotes:<\/p>\n\n\n\n
- \n
- Each partition has its own offset sequence<\/strong> starting at 0.<\/li>\n\n\n\n
- Consumers reading P0 will always see m1 \u2192 m4 \u2192 m7.
But there\u2019s no defined order between<\/strong> (say) m2 on P1 and m3 on P2.<\/li>\n<\/ul>\n\n\n\nB) With keys (so per-key order holds)<\/h2>\n\n\n\n
Suppose keys:
A,B,A,C,B,A,G<\/code> for messagesm1..m7<\/code>.
(Using a simple illustration ofhash(key)%3<\/code>, say A\u2192P1, B\u2192P0, C\u2192P2, G\u2192P2.)<\/p>\n\n\nP0<\/span>: m2(B, offset 0), m5(B, 1)\nP1<\/span>: m1(A, offset 0), m3(A, 1), m6(A, 2)\nP2<\/span>: m4(C, offset 0), m7(G, 1)\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\nNotes:<\/p>\n\n\n\n
- \n
- All A<\/strong> events land on P1<\/strong>, so A\u2019s order is preserved (m1 \u2192 m3 \u2192 m6).<\/li>\n\n\n\n
- All B<\/strong> events land on P0<\/strong>, etc.<\/li>\n\n\n\n
- If you later change the partition count<\/strong>,
hash(key) % numPartitions<\/code> changes \u2192 future records for the same key may map to a different partition<\/strong> (ordering across time can \u201cjump\u201d partitions). Plan for that (see best practices).<\/li>\n<\/ul>\n\n\n\nWhy partitions matter (consumers)<\/h1>\n\n\n\n
- \n
- In a consumer group<\/strong>, one partition \u2192 one active consumer instance<\/strong> at a time.
Max parallelism per topic per group = number of partitions<\/strong>.<\/li>\n\n\n\n- A single KafkaConsumer instance<\/strong> can own multiple partitions<\/strong>, and it will deliver records from each partition in the correct per-partition order.<\/li>\n<\/ul>\n\n\n\n
Best practices for partitioning<\/h1>\n\n\n\n
- \n
- Choose keys deliberately<\/strong>\n
- \n
- If you need per-entity order (e.g., per user\/order\/account), use that entity ID as the key<\/strong>.<\/li>\n\n\n\n
- Ensure keys are well-distributed to avoid hot partitions<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Plan partition count with headroom<\/strong>\n
- \n
- Start with enough partitions for peak throughput + 1.5\u20132\u00d7<\/strong> headroom.<\/li>\n\n\n\n
- Rule of thumb capacity: ~1\u20133 MB\/s per partition<\/strong> (varies by infra).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Avoid frequent partition increases<\/strong> if strict per-key order must hold over long time frames.\n
- \n
- Adding partitions can remap keys \u2192 future events for a key may go to a new partition.<\/li>\n\n\n\n
- If you must scale up, consider creating a new topic<\/strong> with more partitions and migrating producers\/consumers in a controlled way (or implement a custom consistent-hash partitioner<\/strong>).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Monitor skew<\/strong>\n
- \n
- Watch per-partition bytes in\/out<\/strong>, lag<\/strong>, and message rate<\/strong>.<\/li>\n\n\n\n
- If a few partitions are hot, re-evaluate your key choice<\/strong> or increase partitions<\/strong> (with the caveat above).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
Limitations & gotchas<\/h1>\n\n\n\n
- \n
- No global ordering:<\/strong> Only within a partition. Don\u2019t build logic that needs cross-partition order.<\/li>\n\n\n\n
- Hot keys:<\/strong> One key can bottleneck an entire partition (and thus your group\u2019s parallelism).<\/li>\n\n\n\n
- Too many partitions:<\/strong> Faster parallelism, but increases broker memory\/files, metadata load, rebalance time, recovery time.<\/li>\n\n\n\n
- Changing partition count:<\/strong> Remaps keys (default partitioner) \u2192 can disrupt long-lived per-key ordering assumptions.<\/li>\n<\/ul>\n\n\n\n
TL;DR<\/h1>\n\n\n\n
- \n
- A partition<\/strong> is the storage + routing unit<\/strong> of a topic: an append-only log with its own offsets.<\/li>\n\n\n\n
- Producers<\/strong> choose the partition (by key hash<\/strong> or sticky\/round-robin<\/strong> when keyless).<\/li>\n\n\n\n
- Consumers<\/strong> read partitions independently; order only exists within a partition<\/strong>.<\/li>\n\n\n\n
- Pick keys<\/strong> and partition counts<\/strong> intentionally to balance order<\/strong>, throughput<\/strong>, and operational overhead<\/strong>.<\/li>\n\n\n\n
- \n<\/ul>\n<\/blockquote>\n\n\n\n
\n\n\n\n\u201cKafkaConsumer is NOT thread-safe\u201d \u2014 what this actually means<\/h2>\n\n\n\n
- \n
- You must not<\/strong> call
poll()<\/code>,commitSync()<\/code>,seek()<\/code>, etc. on the same<\/strong> KafkaConsumer instance from multiple threads.<\/li>\n\n\n\n- Safe patterns:\n
- \n
- One consumer instance on one thread<\/strong> (single-threaded polling).<\/li>\n\n\n\n
- One consumer instance per thread<\/strong> (multi-consumer, multi-thread).<\/li>\n\n\n\n
- One polling thread + worker pool<\/strong>: the poller thread owns the consumer and hands work to other threads; only the poller commits.<\/li>\n<\/ol>\n<\/li>\n<\/ul>\n\n\n\n
Unsafe patterns (don\u2019t do these): two threads calling
poll()<\/code> on the same instance; one thread polling while another commits on the same instance.<\/p>\n\n\n\n
\n\n\n\nHow partitions get assigned (and re-assigned)<\/h2>\n\n\n\n
- \n
- When a consumer group starts (or changes), the group coordinator<\/strong> performs an assignment<\/strong>: it maps topic partitions \u2192 consumer instances.<\/li>\n\n\n\n
- Triggers for rebalance:<\/strong> a consumer joins\/leaves\/crashes, subscriptions change, partitions change, a consumer stops heartbeating (e.g., blocked poll loop).<\/li>\n\n\n\n
- Lifecycle callbacks<\/strong> (Java):
onPartitionsRevoked<\/code> \u2192 you must finish\/flush and commit<\/strong> work for those partitions \u2192 thenonPartitionsAssigned<\/code>.<\/li>\n<\/ul>\n\n\n\nAssignment strategies (high-level)<\/h3>\n\n\n\n
- \n
- Range<\/strong>: assigns contiguous ranges per topic\u2014can skew if topics\/partitions per topic differ.<\/li>\n\n\n\n
- RoundRobin<\/strong>: cycles through partitions \u2192 consumers\u2014more even across topics.<\/li>\n\n\n\n
- Sticky<\/strong>: tries to keep prior assignments stable across rebalances (reduces churn).<\/li>\n\n\n\n
- CooperativeSticky<\/strong>: best default<\/strong> today\u2014incremental rebalances; consumers give up partitions gradually, minimizing stop-the-world pauses.<\/li>\n<\/ul>\n\n\n\n
\n
Best practice:<\/strong> Use CooperativeSticky<\/strong> + static membership<\/strong> (stable
group.instance.id<\/code>) so transient restarts don\u2019t trigger full rebalances.<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\nWhat an assignment can look like (concrete scenarios)<\/h2>\n\n\n\n
Example A: 7 partitions, 3 consumer instances (C1, C2, C3)<\/strong>
A balanced round-robin\/sticky outcome might be:<\/p>\n\n\n\n- \n
- C1: P0, P3, P6<\/li>\n\n\n\n
- C2: P1, P4<\/li>\n\n\n\n
- C3: P2, P5<\/li>\n<\/ul>\n\n\n\n
Example B: add a 4th consumer (C4) \u2192 incremental rebalance<\/strong><\/p>\n\n\n\n
- \n
- Previous owner of a partition will relinquish one:\n
- \n
- C1: P0, P6<\/li>\n\n\n\n
- C2: P1, P4<\/li>\n\n\n\n
- C3: P2<\/li>\n\n\n\n
- Previous owner of a partition will relinquish one:\n
- RoundRobin<\/strong>: cycles through partitions \u2192 consumers\u2014more even across topics.<\/li>\n\n\n\n
- Triggers for rebalance:<\/strong> a consumer joins\/leaves\/crashes, subscriptions change, partitions change, a consumer stops heartbeating (e.g., blocked poll loop).<\/li>\n\n\n\n
- One consumer instance per thread<\/strong> (multi-consumer, multi-thread).<\/li>\n\n\n\n
- Safe patterns:\n
- Producers<\/strong> choose the partition (by key hash<\/strong> or sticky\/round-robin<\/strong> when keyless).<\/li>\n\n\n\n
- Hot keys:<\/strong> One key can bottleneck an entire partition (and thus your group\u2019s parallelism).<\/li>\n\n\n\n
- If a few partitions are hot, re-evaluate your key choice<\/strong> or increase partitions<\/strong> (with the caveat above).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
- Monitor skew<\/strong>\n
- Rule of thumb capacity: ~1\u20133 MB\/s per partition<\/strong> (varies by infra).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Ensure keys are well-distributed to avoid hot partitions<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- If you need per-entity order (e.g., per user\/order\/account), use that entity ID as the key<\/strong>.<\/li>\n\n\n\n
- A single KafkaConsumer instance<\/strong> can own multiple partitions<\/strong>, and it will deliver records from each partition in the correct per-partition order.<\/li>\n<\/ul>\n\n\n\n
- All B<\/strong> events land on P0<\/strong>, etc.<\/li>\n\n\n\n
- Consumers reading P0 will always see m1 \u2192 m4 \u2192 m7.
- Older behavior was round-robin. Either way, distribution is roughly even over time<\/strong> but not ordered across partitions<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n
- Watch out for hot keys<\/strong> (one key getting huge traffic) \u2192 one partition becomes a bottleneck.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- All messages with the same key<\/strong> go to the same partition<\/strong> \u2192 preserves per-key order.<\/li>\n\n\n\n
- Physical<\/em> files on a broker (leader) with replicas<\/strong> on other brokers for HA.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Physical & logical:<\/strong>\n
- Partition<\/strong>: an ordered shard of a topic. Ordering is guaranteed within<\/em> a partition.<\/li>\n\n\n\n