{"id":51653,"date":"2025-08-20T04:44:42","date_gmt":"2025-08-20T04:44:42","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=51653"},"modified":"2025-08-20T04:44:42","modified_gmt":"2025-08-20T04:44:42","slug":"kafka-confluent-terminology-complete-glossary","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/kafka-confluent-terminology-complete-glossary\/","title":{"rendered":"Kafka & Confluent Terminology \u2013 Complete Glossary"},"content":{"rendered":"\n


I\u2019ll cover Apache Kafka core terms<\/strong>, Confluent Platform extensions<\/strong>, and Confluent Cloud additions<\/strong>.<\/p>\n\n\n\n

\n\n\n\n

Kafka & Confluent Terminology \u2013 Complete Glossary<\/strong><\/h1>\n\n\n\n

\ud83d\udd39 Core Kafka Concepts<\/strong><\/h2>\n\n\n\n
    \n
  1. Kafka Cluster<\/strong>
    A group of servers (called brokers) working together to store, process, and stream data.<\/li>\n\n\n\n
  2. Broker<\/strong>
    A single Kafka server that stores data and serves client requests (produce\/consume).<\/li>\n\n\n\n
  3. Producer<\/strong>
    An application that sends data (messages) into Kafka topics.<\/li>\n\n\n\n
  4. Consumer<\/strong>
    An application that reads data (messages) from Kafka topics.<\/li>\n\n\n\n
  5. Consumer Group<\/strong>
    A group of consumers working together to read data from a topic. Kafka ensures each message is processed by only one consumer within the group<\/strong>.<\/li>\n\n\n\n
  6. Topic<\/strong>
    A named channel where producers send messages and consumers read messages (like a folder or queue).<\/li>\n\n\n\n
  7. Partition<\/strong>
    A topic is divided into slices called partitions. Messages in a partition are ordered. Partitions allow parallelism and scalability.<\/li>\n\n\n\n
  8. Offset<\/strong>
    The position of a message in a partition (like a bookmark). Consumers use offsets to track what they\u2019ve read.<\/li>\n\n\n\n
  9. Record \/ Message<\/strong>
    A single unit of data in Kafka. It has:\n
      \n
    • Key<\/strong> (optional, used for partitioning\/order)<\/li>\n\n\n\n
    • Value<\/strong> (the actual payload)<\/li>\n\n\n\n
    • Headers<\/strong> (extra metadata)<\/li>\n<\/ul>\n<\/li>\n\n\n\n
    • Log<\/strong>
      A partition is stored as an append-only log (new messages are always written at the end).<\/li>\n\n\n\n
    • Replication<\/strong>
      Kafka keeps copies of partitions across multiple brokers for fault tolerance.<\/li>\n\n\n\n
    • Leader & Follower<\/strong>\n
        \n
      • Leader<\/strong>: The main replica of a partition that handles all reads\/writes.<\/li>\n\n\n\n
      • Follower<\/strong>: Copies the leader\u2019s data for backup.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
      • ISR (In-Sync Replicas)<\/strong>
        A set of replicas that are fully caught up with the leader.<\/li>\n\n\n\n
      • Retention Policy<\/strong>
        Defines how long Kafka keeps data (e.g., 7 days, forever, or until size limit).<\/li>\n\n\n\n
      • Compaction<\/strong>
        A cleanup policy that keeps only the latest value per key<\/strong>, deleting older duplicates.<\/li>\n\n\n\n
      • Throughput<\/strong>
        The rate at which Kafka processes messages (messages per second).<\/li>\n\n\n\n
      • Latency<\/strong>
        The time it takes for a message to travel from producer \u2192 broker \u2192 consumer.<\/li>\n<\/ol>\n\n\n\n
        \n\n\n\n

        \ud83d\udd39 Kafka Internals<\/strong><\/h2>\n\n\n\n
          \n
        1. ZooKeeper (Legacy)<\/strong>
          Used in older Kafka versions to manage cluster metadata and leader election. (Being replaced by KRaft<\/strong>).<\/li>\n\n\n\n
        2. KRaft (Kafka Raft Metadata mode)<\/strong>
          New architecture where Kafka itself manages metadata, removing the need for ZooKeeper.<\/li>\n\n\n\n
        3. Controller<\/strong>
          A special broker responsible for managing partition leaders.<\/li>\n\n\n\n
        4. Rebalancing<\/strong>
          When consumers join\/leave a group, Kafka redistributes partitions among them.<\/li>\n\n\n\n
        5. Coordinator<\/strong>
          The broker responsible for managing a consumer group.<\/li>\n\n\n\n
        6. ACL (Access Control List)<\/strong>
          Security rules defining which user\/app can access which topic or resource.<\/li>\n\n\n\n
        7. Quotas<\/strong>
          Limits on how much data a client can produce\/consume to prevent abuse.<\/li>\n\n\n\n
        8. Idempotent Producer<\/strong>
          Ensures no duplicate messages are produced even if retries happen.<\/li>\n\n\n\n
        9. Exactly-Once Semantics (EOS)<\/strong>
          Guarantee that messages are processed only once, even during failures.<\/li>\n\n\n\n
        10. Transactions<\/strong>
          A way to group multiple messages into an atomic unit of work.<\/li>\n<\/ol>\n\n\n\n
          \n\n\n\n

          \ud83d\udd39 Confluent-Specific Terms<\/strong><\/h2>\n\n\n\n
            \n
          1. Confluent Platform<\/strong>
            An enterprise distribution of Kafka with additional tools for management, monitoring, and integration.<\/li>\n\n\n\n
          2. Confluent Cloud<\/strong>
            A fully managed Kafka service hosted by Confluent on AWS, Azure, or GCP.<\/li>\n\n\n\n
          3. Schema Registry<\/strong>
            Stores and enforces schemas (data formats) for messages (e.g., Avro, JSON, Protobuf) to ensure compatibility.<\/li>\n\n\n\n
          4. kSQL \/ ksqlDB<\/strong>
            A SQL-like engine to query, process, and transform Kafka streams in real-time.<\/li>\n\n\n\n
          5. Kafka Connect<\/strong>
            A framework to move data in\/out of Kafka using connectors (e.g., JDBC, S3, Elasticsearch).<\/li>\n\n\n\n
          6. Connector<\/strong>
            A plugin used with Kafka Connect to integrate Kafka with external systems.\n
              \n
            • Source Connector<\/strong>: Pulls data into Kafka.<\/li>\n\n\n\n
            • Sink Connector<\/strong>: Pushes data out of Kafka.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
            • Confluent Hub<\/strong>
              A marketplace of prebuilt Kafka connectors.<\/li>\n\n\n\n
            • Confluent Control Center<\/strong>
              A GUI tool for monitoring Kafka clusters, topics, connectors, and schemas.<\/li>\n\n\n\n
            • Replicator<\/strong>
              A Confluent tool to copy topics from one Kafka cluster to another (useful for multi-region).<\/li>\n\n\n\n
            • Confluent REST Proxy<\/strong>
              Allows producing\/consuming data using REST APIs instead of Kafka clients.<\/li>\n\n\n\n
            • Confluent RBAC (Role-Based Access Control)<\/strong>
              Fine-grained access control for Kafka resources.<\/li>\n\n\n\n
            • Confluent CLI<\/strong>
              A command-line tool for managing Confluent Cloud clusters, topics, and connectors.<\/li>\n\n\n\n
            • Tiered Storage<\/strong>
              A Confluent feature that offloads older Kafka data to cheaper cloud storage (e.g., S3, GCS).<\/li>\n\n\n\n
            • Cluster Linking<\/strong>
              A Confluent Cloud feature to link clusters across regions\/clouds for data replication.<\/li>\n\n\n\n
            • Confluent Cloud Metrics API<\/strong>
              Provides usage and performance metrics for monitoring clusters.<\/li>\n<\/ol>\n\n\n\n
              \n\n\n\n

              \ud83d\udd39 Stream Processing Terms<\/strong><\/h2>\n\n\n\n
                \n
              1. Kafka Streams<\/strong>
                A Java library for building real-time streaming applications on top of Kafka.<\/li>\n\n\n\n
              2. Stream<\/strong>
                A continuous flow of data records in Kafka.<\/li>\n\n\n\n
              3. Stream Processor<\/strong>
                An application that transforms or processes Kafka data in real-time.<\/li>\n\n\n\n
              4. Topology<\/strong>
                The workflow (graph of processors) that defines how streams are processed.<\/li>\n\n\n\n
              5. State Store<\/strong>
                Local storage used by stream processing apps to maintain state (e.g., counts, aggregations).<\/li>\n\n\n\n
              6. Global Store<\/strong>
                A replicated state store available to all stream tasks.<\/li>\n\n\n\n
              7. Windowing<\/strong>
                Grouping data by time intervals (e.g., 5-minute sales totals).<\/li>\n<\/ol>\n\n\n\n
                \n\n\n\n

                \ud83d\udd39 Advanced Kafka Concepts<\/strong><\/h2>\n\n\n\n
                  \n
                1. Reassignment<\/strong>
                  Moving partitions across brokers for load balancing.<\/li>\n\n\n\n
                2. Throttling<\/strong>
                  Slowing down producers\/consumers to avoid overwhelming the cluster.<\/li>\n\n\n\n
                3. Backpressure<\/strong>
                  When consumers can\u2019t keep up with producers, causing slowdowns.<\/li>\n\n\n\n
                4. Dead Letter Queue (DLQ)<\/strong>
                  A special topic where failed or invalid messages are sent for later debugging.<\/li>\n\n\n\n
                5. MirrorMaker 2.0<\/strong>
                  Kafka\u2019s built-in tool for replicating data across clusters (open-source equivalent of Confluent Replicator).<\/li>\n\n\n\n
                6. Metrics & JMX<\/strong>
                  Kafka exposes metrics via JMX for monitoring cluster health.<\/li>\n\n\n\n
                7. Log Segment<\/strong>
                  Each partition\u2019s log is broken into smaller files called log segments.<\/li>\n\n\n\n
                8. Message Key Partitioning<\/strong>
                  The method Kafka uses to decide which partition a message goes to (based on key hash).<\/li>\n\n\n\n
                9. Rack Awareness<\/strong>
                  Kafka spreads replicas across different racks\/data centers for reliability.<\/li>\n<\/ol>\n\n\n\n
                  \n\n\n\n

                  <\/p>\n","protected":false},"excerpt":{"rendered":"

                  I\u2019ll cover Apache Kafka core terms, Confluent Platform extensions, and Confluent Cloud additions. Kafka & Confluent Terminology \u2013 Complete Glossary \ud83d\udd39 Core Kafka Concepts \ud83d\udd39 Kafka Internals \ud83d\udd39 Confluent-Specific Terms… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[2],"tags":[],"class_list":["post-51653","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/51653","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=51653"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/51653\/revisions"}],"predecessor-version":[{"id":51654,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/51653\/revisions\/51654"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=51653"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=51653"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=51653"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}