{"id":927,"date":"2026-04-17T04:07:34","date_gmt":"2026-04-17T04:07:34","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/oracle-cloud-streaming-with-apache-kafka-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-integration\/"},"modified":"2026-04-17T04:07:34","modified_gmt":"2026-04-17T04:07:34","slug":"oracle-cloud-streaming-with-apache-kafka-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-integration","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/oracle-cloud-streaming-with-apache-kafka-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-integration\/","title":{"rendered":"Oracle Cloud Streaming with Apache Kafka Tutorial: Architecture, Pricing, Use Cases, and Hands-On Guide for Integration"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Category<\/h2>\n\n\n\n<p>Integration<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. Introduction<\/h2>\n\n\n\n<p><strong>What this service is<\/strong><br\/>\nIn Oracle Cloud (OCI), <strong>Streaming with Apache Kafka<\/strong> refers to using <strong>Oracle Cloud Infrastructure Streaming<\/strong> as a managed event streaming platform <strong>through Kafka-compatible APIs and clients<\/strong>. You build producers and consumers with familiar Apache Kafka tooling while Oracle Cloud operates the underlying streaming infrastructure.<\/p>\n\n\n\n<p><strong>Simple explanation (one paragraph)<\/strong><br\/>\nIf you want Kafka-style publish\/subscribe messaging without running Kafka brokers yourself, Oracle Cloud lets you create streams and then connect using Kafka clients to produce and consume events. This is commonly used to connect microservices, ingest logs and telemetry, stream business events, and feed analytics pipelines.<\/p>\n\n\n\n<p><strong>Technical explanation (one paragraph)<\/strong><br\/>\nOracle Cloud Infrastructure Streaming provides a managed streaming data plane (streams, partitions, retention) and a control plane (stream pools, streams, IAM, metrics). With <strong>Streaming with Apache Kafka<\/strong>, you use Kafka protocols from standard Kafka clients to authenticate (typically using OCI IAM credentials\/auth tokens and TLS) and publish\/consume records to an OCI stream that maps to a Kafka topic name (confirm exact mapping and supported Kafka APIs in official docs).<\/p>\n\n\n\n<p><strong>What problem it solves<\/strong><br\/>\nIt solves the operational burden and risk of self-managing Kafka clusters (broker sizing, patching, scaling, availability, TLS, authentication), while enabling event-driven integration between systems\u2014often across microservices, data platforms, and SaaS\/enterprise applications\u2014using the Kafka ecosystem that teams already know.<\/p>\n\n\n\n<blockquote>\n<p>Naming note (important): In Oracle Cloud, the underlying managed service is commonly documented as <strong>Oracle Cloud Infrastructure Streaming<\/strong> (often shortened to <strong>OCI Streaming<\/strong>). \u201c<strong>Streaming with Apache Kafka<\/strong>\u201d is best understood as <strong>Kafka compatibility \/ Kafka interface<\/strong> for OCI Streaming, not a separate product. Verify the latest naming and supported Kafka versions\/APIs in the official documentation.<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">2. What is Streaming with Apache Kafka?<\/h2>\n\n\n\n<p><strong>Official purpose<\/strong><br\/>\nStreaming with Apache Kafka on Oracle Cloud is designed to let you <strong>stream events in real time<\/strong> using <strong>Kafka-compatible producer\/consumer applications<\/strong>, while OCI manages the streaming backend.<\/p>\n\n\n\n<p><strong>Core capabilities<\/strong>\n&#8211; Create and manage <strong>stream pools<\/strong> and <strong>streams<\/strong> (topics) with <strong>partitions<\/strong> and <strong>retention<\/strong>.\n&#8211; Produce and consume events with <strong>Kafka clients<\/strong> (language SDKs and CLI tools).\n&#8211; Secure access with <strong>Oracle Cloud IAM<\/strong> and <strong>TLS<\/strong>.\n&#8211; Monitor throughput\/lag\/availability with <strong>OCI Monitoring metrics<\/strong> and <strong>logs\/audit events<\/strong> (capabilities vary by configuration\u2014verify in official docs).\n&#8211; Integrate with OCI services commonly used in Integration architectures (for example: Events, Service Connector Hub, Logging, Functions, API Gateway, Object Storage, Data Flow\u2014verify supported connectors and patterns in docs and architecture center).<\/p>\n\n\n\n<p><strong>Major components (how you should think about it)<\/strong>\n&#8211; <strong>Tenancy \/ Compartment<\/strong>: OCI governance boundary where you create resources and apply policies.\n&#8211; <strong>Stream Pool<\/strong>: A logical container for streaming resources (and often the unit where endpoints\/networking are defined).\n&#8211; <strong>Stream<\/strong>: The named event stream (analogous to a Kafka topic in many workflows).\n&#8211; <strong>Partitions<\/strong>: Parallelism and ordering unit.\n&#8211; <strong>Consumer groups<\/strong>: Scale-out pattern for consumers (Kafka concept; confirm supported semantics with OCI Kafka compatibility docs).\n&#8211; <strong>IAM policies &amp; auth tokens<\/strong>: Access control for producers\/consumers.\n&#8211; <strong>Endpoints<\/strong>: Public and\/or private access endpoints depending on pool\/networking configuration (verify current options per region).<\/p>\n\n\n\n<p><strong>Service type<\/strong>\n&#8211; <strong>Managed streaming service<\/strong> in Oracle Cloud, used under <strong>Integration<\/strong> architectures for event-driven systems.<\/p>\n\n\n\n<p><strong>Scope (regional\/global\/zonal, etc.)<\/strong>\n&#8211; OCI Streaming resources are generally <strong>regional<\/strong> (created in a specific OCI region) and scoped to a <strong>compartment<\/strong> within your tenancy.<br\/>\n  Verify exact regional behavior and cross-region replication options (if any) in official docs.<\/p>\n\n\n\n<p><strong>How it fits into the Oracle Cloud ecosystem<\/strong>\n&#8211; Acts as an <strong>event backbone<\/strong> between OCI compute (Compute instances, OKE\/Kubernetes), serverless (Functions), integration (API Gateway, Service Connector Hub), and data services (Object Storage, Data Flow, Autonomous Database via custom consumers\/producers).\n&#8211; Fits well in <strong>event-driven integration<\/strong> patterns: event sourcing, streaming ETL\/ELT, operational telemetry pipelines, CDC pipelines (with Kafka Connect where compatible), and asynchronous decoupling between systems.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">3. Why use Streaming with Apache Kafka?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Business reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Faster delivery<\/strong>: Teams can implement event streaming without provisioning and operating Kafka brokers.<\/li>\n<li><strong>Lower operational overhead<\/strong>: Oracle Cloud manages much of the platform reliability and maintenance.<\/li>\n<li><strong>Standard skill reuse<\/strong>: Kafka clients and patterns are widely known; onboarding is easier than proprietary APIs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Technical reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Kafka ecosystem compatibility<\/strong>: Reuse Kafka libraries, CLI tools, and established patterns (subject to the subset of Kafka APIs supported\u2014verify in official docs).<\/li>\n<li><strong>Scalable event ingestion<\/strong>: Partitioned streams enable horizontal scale for both producers and consumers.<\/li>\n<li><strong>Decoupled architectures<\/strong>: Producers and consumers evolve independently.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Simplified scaling<\/strong>: Managed service reduces scaling complexity versus self-managed Kafka (though you still must plan partitions, retention, consumer concurrency, and throughput).<\/li>\n<li><strong>Observability hooks<\/strong>: OCI Monitoring metrics and logs\/auditing help operations teams track health and usage.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security \/ compliance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Central IAM<\/strong>: Access governed through OCI IAM policies and compartments.<\/li>\n<li><strong>Transport security<\/strong>: Kafka connections typically use TLS (SASL over SSL), reducing the need to engineer custom encryption.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scalability \/ performance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Partition-based parallelism<\/strong>: Scale throughput by increasing partitions and consumer group members (within quotas and service limits).<\/li>\n<li><strong>Durable buffering<\/strong>: Retention enables replay and resilience during downstream outages.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should choose it<\/h3>\n\n\n\n<p>Choose Streaming with Apache Kafka in Oracle Cloud when:\n&#8211; You want Kafka-style streaming without running Kafka clusters.\n&#8211; You\u2019re building an event-driven platform in OCI (microservices, integration, data pipelines).\n&#8211; You need multiple consumer applications to process the same event stream independently.\n&#8211; You have variable traffic and want managed scaling patterns (within service constraints).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should not choose it<\/h3>\n\n\n\n<p>Avoid or reconsider when:\n&#8211; You require <strong>full Kafka broker control<\/strong> (custom broker configs, plugins) or Kafka features that may not be supported by OCI\u2019s Kafka compatibility layer.\n&#8211; You need <strong>Kafka ecosystem components<\/strong> that require deep broker integration (some Kafka Connect setups, certain admin APIs, or specific Kafka protocol features)\u2014verify compatibility first.\n&#8211; You have strict requirements for <strong>on-prem-only<\/strong> or air-gapped deployments (self-managed Kafka may be better).\n&#8211; Your workload is primarily <strong>simple point-to-point queues<\/strong> (OCI Queue or messaging alternatives may fit better, depending on requirements\u2014verify OCI messaging options).<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">4. Where is Streaming with Apache Kafka used?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Industries<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Fintech \/ banking<\/strong>: Transaction events, fraud signals, audit trails, risk scoring pipelines.<\/li>\n<li><strong>Retail \/ e-commerce<\/strong>: Clickstream, order lifecycle events, inventory updates.<\/li>\n<li><strong>Telecom<\/strong>: Network telemetry streams, real-time alerting.<\/li>\n<li><strong>Gaming<\/strong>: Player events, matchmaking telemetry, anti-cheat signals.<\/li>\n<li><strong>Manufacturing \/ IoT<\/strong>: Sensor telemetry aggregation and alerting.<\/li>\n<li><strong>Media<\/strong>: Content interactions and streaming analytics.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Team types<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Platform engineering teams building internal event platforms.<\/li>\n<li>Integration teams connecting enterprise apps through event streams.<\/li>\n<li>Data engineering teams building streaming ETL\/ELT pipelines.<\/li>\n<li>DevOps\/SRE teams standardizing observability pipelines.<\/li>\n<li>Application teams implementing microservices choreography.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Workloads<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>High-volume ingestion (logs, metrics, events).<\/li>\n<li>Real-time analytics feed for data lakes\/warehouses.<\/li>\n<li>Event-driven microservices and serverless workflows.<\/li>\n<li>CDC and event sourcing (compatibility-dependent).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Architectures<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Event-driven architecture (EDA)<\/li>\n<li>Microservices with asynchronous communication<\/li>\n<li>CQRS + event sourcing<\/li>\n<li>Streaming analytics pipelines<\/li>\n<li>Hybrid integration (on-prem producers, cloud consumers)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Real-world deployment contexts<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Production<\/strong>: Multiple streams, multiple consumer groups, private endpoints, IAM least privilege, monitoring\/alerting, and automation with Terraform.<\/li>\n<li><strong>Dev\/Test<\/strong>: Lower partitions\/retention, public endpoints (where acceptable), simpler IAM, smaller traffic.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">5. Top Use Cases and Scenarios<\/h2>\n\n\n\n<p>Below are realistic use cases for Streaming with Apache Kafka on Oracle Cloud. Each is phrased as a \u201cproblem \u2192 fit \u2192 scenario\u201d.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1) Microservices event bus<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Synchronous APIs create coupling and cascading failures.<\/li>\n<li><strong>Why this fits<\/strong>: Producers publish events; consumers process independently with retries and replay.<\/li>\n<li><strong>Scenario<\/strong>: <code>orders-service<\/code> publishes <code>OrderCreated<\/code>; <code>billing-service<\/code>, <code>shipping-service<\/code>, and <code>email-service<\/code> consume in separate consumer groups.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2) Centralized application logging pipeline<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Logs overload downstream storage during traffic spikes.<\/li>\n<li><strong>Why this fits<\/strong>: Stream buffers logs; consumers batch and write to Object Storage or a SIEM.<\/li>\n<li><strong>Scenario<\/strong>: OKE workloads ship JSON logs into a stream; a consumer writes compressed objects to Object Storage hourly.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3) Security event ingestion and correlation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Security telemetry arrives from many systems and needs correlation.<\/li>\n<li><strong>Why this fits<\/strong>: High-throughput ingestion and fan-out to multiple security tools.<\/li>\n<li><strong>Scenario<\/strong>: CloudGuard findings, app events, and WAF logs go into streams; multiple consumers feed detection rules and dashboards.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4) Clickstream analytics<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Web\/mobile events must be processed in near real time.<\/li>\n<li><strong>Why this fits<\/strong>: Low-latency ingestion + replay for backfills.<\/li>\n<li><strong>Scenario<\/strong>: Frontends publish page views; consumers compute sessions and write aggregates to a database.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5) IoT telemetry ingestion<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Millions of device readings per minute; intermittent downstream systems.<\/li>\n<li><strong>Why this fits<\/strong>: Durable ingestion and scalable consumer processing.<\/li>\n<li><strong>Scenario<\/strong>: Devices send readings via an API gateway into a producer; stream partitions by device region; consumers perform anomaly detection.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6) Real-time fraud feature stream<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Fraud models need live features and signals.<\/li>\n<li><strong>Why this fits<\/strong>: Stream processing can build rolling aggregates; features can be replayed.<\/li>\n<li><strong>Scenario<\/strong>: Transaction signals published to a stream; a consumer calculates velocity metrics per account.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7) Event-driven integration between SaaS and enterprise apps<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Polling integrations are slow and expensive.<\/li>\n<li><strong>Why this fits<\/strong>: Events become the integration contract.<\/li>\n<li><strong>Scenario<\/strong>: CRM updates publish events; internal systems consume and reconcile.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8) Data lake ingestion (stream-to-object storage)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You need an append-only, replayable ingestion layer for analytics.<\/li>\n<li><strong>Why this fits<\/strong>: Streams buffer; consumers land data to Object Storage in partitioned folders.<\/li>\n<li><strong>Scenario<\/strong>: Events written as Parquet\/JSON to <code>oci:\/\/bucket\/events\/date=...\/<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">9) Operational metrics pipeline<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Metrics ingestion spikes break monitoring pipelines.<\/li>\n<li><strong>Why this fits<\/strong>: Stream absorbs bursts; consumers downsample and forward.<\/li>\n<li><strong>Scenario<\/strong>: Custom metrics emitted into stream; consumer writes summarized metrics to Monitoring or a TSDB.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">10) CI\/CD event stream for audit and automation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You need traceability and automation triggers across pipelines.<\/li>\n<li><strong>Why this fits<\/strong>: Streams capture pipeline events; consumers trigger workflows.<\/li>\n<li><strong>Scenario<\/strong>: Build events published; a consumer opens change tickets and updates deployment dashboards.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">11) Multi-tenant event streaming platform (internal)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Different teams need isolated streams and access controls.<\/li>\n<li><strong>Why this fits<\/strong>: Compartments, policies, and per-stream governance.<\/li>\n<li><strong>Scenario<\/strong>: One compartment per product team; shared platform team provides stream pools; teams manage their streams.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">12) Backpressure safety for downstream databases<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Direct writes overload databases during bursts.<\/li>\n<li><strong>Why this fits<\/strong>: Stream buffers writes; consumers process at a controlled rate.<\/li>\n<li><strong>Scenario<\/strong>: Producers publish change events; consumer writes to Autonomous Database with throttling and retry.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">6. Core Features<\/h2>\n\n\n\n<blockquote>\n<p>Note: Features may vary by OCI region and by the current implementation of Kafka compatibility. Always confirm the supported Kafka client versions, authentication format, and API coverage in the official docs.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">1) Managed streams with partitions and retention<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Provides durable streams split into partitions; data retained for a configured retention period.<\/li>\n<li><strong>Why it matters<\/strong>: Enables parallel processing, ordering per partition, and replay.<\/li>\n<li><strong>Practical benefit<\/strong>: You can scale consumers and recover from downstream outages by reprocessing.<\/li>\n<li><strong>Caveats<\/strong>: Retention and partition count affect cost; ordering is typically guaranteed only within a partition.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2) Stream pools as administrative\/networking boundaries<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Organizes streams and defines connectivity characteristics (for example, endpoint configuration).<\/li>\n<li><strong>Why it matters<\/strong>: Helps manage environments (dev\/test\/prod) and isolate networking.<\/li>\n<li><strong>Practical benefit<\/strong>: Separate pools for public vs private access, or per business unit.<\/li>\n<li><strong>Caveats<\/strong>: Changing endpoint\/network settings may require planning; verify current constraints.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3) Kafka-compatible producer\/consumer connectivity<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Lets Kafka clients publish and consume using Kafka protocol endpoints.<\/li>\n<li><strong>Why it matters<\/strong>: Reuse existing Kafka-based code and tooling.<\/li>\n<li><strong>Practical benefit<\/strong>: Faster migration from self-managed Kafka or faster adoption with existing skills.<\/li>\n<li><strong>Caveats<\/strong>: Not all Kafka APIs or ecosystem tools are necessarily supported (for example, topic creation via Kafka Admin API may not be allowed if streams are managed via OCI). Verify supported features.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4) IAM-based authentication and authorization<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Governs who can create\/manage streams and who can produce\/consume.<\/li>\n<li><strong>Why it matters<\/strong>: Centralized, auditable access control integrated with OCI governance.<\/li>\n<li><strong>Practical benefit<\/strong>: Least privilege with compartment-scoped policies and separate roles for admins vs apps.<\/li>\n<li><strong>Caveats<\/strong>: Misconfigured policies are a common cause of authentication\/authorization failures.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5) TLS encryption in transit<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Encrypts traffic between Kafka clients and OCI endpoints.<\/li>\n<li><strong>Why it matters<\/strong>: Protects credentials and message contents on the wire.<\/li>\n<li><strong>Practical benefit<\/strong>: Meets baseline security requirements for many organizations.<\/li>\n<li><strong>Caveats<\/strong>: Ensure client truststores are correct; TLS handshake errors are common when proxies\/inspection interfere.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6) Metrics and monitoring (OCI Monitoring)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Exposes service metrics (throughput, errors, potentially lag\/usage metrics depending on available telemetry).<\/li>\n<li><strong>Why it matters<\/strong>: Operations needs alerting for producer errors, consumer lag, and saturation.<\/li>\n<li><strong>Practical benefit<\/strong>: Create alarms and integrate with Notifications\/on-call.<\/li>\n<li><strong>Caveats<\/strong>: Metric names and availability can differ; verify in official docs for your region\/service version.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7) Auditing (OCI Audit)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Records control plane actions (create\/update\/delete streams\/pools, policy changes).<\/li>\n<li><strong>Why it matters<\/strong>: Security and compliance require traceability.<\/li>\n<li><strong>Practical benefit<\/strong>: Investigate \u201cwho changed what\u201d during incidents.<\/li>\n<li><strong>Caveats<\/strong>: Audit covers control plane, not necessarily every data-plane message event.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8) Integration patterns with OCI services<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Enables event pipelines where producers\/consumers run on Compute\/OKE\/Functions and land data in Object Storage or databases.<\/li>\n<li><strong>Why it matters<\/strong>: Most streaming value comes from end-to-end pipelines.<\/li>\n<li><strong>Practical benefit<\/strong>: Use Service Connector Hub where supported, or custom consumers for flexible routing.<\/li>\n<li><strong>Caveats<\/strong>: Some integrations may be pattern-based (custom code) rather than \u201cnative connectors\u201d; verify available connectors.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">7. Architecture and How It Works<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">High-level service architecture<\/h3>\n\n\n\n<p>Streaming with Apache Kafka in Oracle Cloud typically looks like:\n1. <strong>Control plane<\/strong>: You create a <strong>stream pool<\/strong> and <strong>streams<\/strong> (and configure retention\/partitions) using OCI Console, CLI, SDKs, or IaC.\n2. <strong>Data plane<\/strong>: Your applications use <strong>Kafka clients<\/strong> to connect to the streaming endpoint and produce\/consume records.\n3. <strong>Security plane<\/strong>: OCI IAM policies govern access; clients authenticate using an OCI-supported method (often involving an auth token and a Kafka SASL mechanism\u2014verify the exact required configuration in official docs).\n4. <strong>Observability plane<\/strong>: Metrics and audit logs provide operational visibility.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Request\/data\/control flow<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Create resources<\/strong> (control plane): Admin creates stream pool and stream in a compartment.<\/li>\n<li><strong>Producer flow<\/strong> (data plane): Producer app authenticates \u2192 sends records to a topic\/stream \u2192 service appends records to partitions.<\/li>\n<li><strong>Consumer flow<\/strong> (data plane): Consumer app authenticates \u2192 joins a consumer group \u2192 reads from partitions \u2192 commits offsets (Kafka semantics; verify supported behavior).<\/li>\n<li><strong>Monitoring<\/strong>: OCI metrics record usage\/health; audit logs capture administrative actions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Integrations with related services (common patterns)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>OKE (Kubernetes)<\/strong>: Run Kafka producers\/consumers as Deployments; use Kubernetes secrets for auth tokens.<\/li>\n<li><strong>Compute instances<\/strong>: Run Kafka clients on VMs for lift-and-shift integration workloads.<\/li>\n<li><strong>Functions<\/strong>: Trigger-based processing is possible through patterns (for example, a consumer that invokes Functions) or supported connectors if available.<\/li>\n<li><strong>Service Connector Hub<\/strong>: Where supported, route streaming data to Object Storage\/Logging\/Monitoring targets (verify streaming as a source and Kafka compatibility considerations).<\/li>\n<li><strong>Logging<\/strong>: Forward application logs into streams, or stream logs to storage\/analytics.<\/li>\n<li><strong>Events<\/strong>: Use Events for OCI resource events; if you need high throughput, route through streaming patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Dependency services<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>OCI IAM<\/strong>: Users, groups, dynamic groups, policies.<\/li>\n<li><strong>OCI Networking<\/strong>: VCN, subnets, security lists\/NSGs if using private connectivity patterns.<\/li>\n<li><strong>OCI Monitoring\/Audit<\/strong>: For metrics and governance visibility.<\/li>\n<li><strong>OCI Vault<\/strong> (recommended): Store secrets (auth tokens) securely.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/authentication model (conceptual)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Authentication<\/strong>: Kafka clients authenticate using an OCI-supported method (commonly SASL over TLS with OCI credentials\/auth token).<br\/>\n  Do not hardcode credentials; use Vault or secret stores.<\/li>\n<li><strong>Authorization<\/strong>: IAM policies grant permissions to manage or use stream pools\/streams within compartments.<\/li>\n<li><strong>Separation of duties<\/strong>: Separate admins (create streams\/pools) from apps (produce\/consume).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Networking model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Public endpoints<\/strong>: Simpler for labs, but must be locked down via IAM and client network controls.<\/li>\n<li><strong>Private endpoints<\/strong>: Preferred for production. Connect from VCN resources (OKE\/Compute) without exposing traffic publicly.<br\/>\n  Availability and exact configuration steps can vary; verify current stream pool endpoint options in docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Monitoring\/logging\/governance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Set alarms on key metrics (ingress\/egress, errors, throttling, consumer lag if available).<\/li>\n<li>Use Audit to track configuration changes.<\/li>\n<li>Tag resources for cost allocation (environment, owner, application, data classification).<\/li>\n<li>Implement quotas and compartment-level governance to avoid uncontrolled partition growth.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Simple architecture diagram (Mermaid)<\/h4>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart LR\n  P[Producer app&lt;br\/&gt;Kafka client] --&gt;|SASL_SSL| S[(OCI Streaming&lt;br\/&gt;Stream\/Topic)]\n  S --&gt; C[Consumer app&lt;br\/&gt;Kafka client]\n  A[OCI IAM&lt;br\/&gt;Policies &amp; Auth] -.-&gt; P\n  A -.-&gt; C\n  M[OCI Monitoring &amp; Audit] -.-&gt; S\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Production-style architecture diagram (Mermaid)<\/h4>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart TB\n  subgraph VCN[VCN (Private Networking)]\n    subgraph OKE[OKE Cluster]\n      P1[Producers&lt;br\/&gt;Microservices]:::app\n      C1[Consumers&lt;br\/&gt;Stream processors]:::app\n    end\n    VAULT[OCI Vault&lt;br\/&gt;Secrets]:::sec\n  end\n\n  subgraph OCI[Oracle Cloud (Control &amp; Observability)]\n    IAM[OCI IAM&lt;br\/&gt;Compartments\/Policies]:::sec\n    MON[OCI Monitoring&lt;br\/&gt;Metrics\/Alarms]:::ops\n    AUD[OCI Audit]:::ops\n    STREAM[(OCI Streaming&lt;br\/&gt;Stream Pool + Streams&lt;br\/&gt;Kafka endpoint)]:::svc\n    OBJ[Object Storage&lt;br\/&gt;Data Lake Landing]:::svc\n    DB[Autonomous Database&lt;br\/&gt;Operational Store]:::svc\n    NOTIF[Notifications&lt;br\/&gt;On-call]:::ops\n  end\n\n  P1 --&gt;|Kafka produce| STREAM\n  C1 --&gt;|Kafka consume| STREAM\n  VAULT -.-&gt; P1\n  VAULT -.-&gt; C1\n\n  C1 --&gt;|Batch writes| OBJ\n  C1 --&gt;|Upserts| DB\n\n  IAM -.-&gt; STREAM\n  MON -.-&gt; STREAM\n  AUD -.-&gt; STREAM\n  MON --&gt; NOTIF\n\n  classDef app fill:#eef,stroke:#335;\n  classDef svc fill:#efe,stroke:#353;\n  classDef sec fill:#fee,stroke:#533;\n  classDef ops fill:#ffd,stroke:#553;\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">8. Prerequisites<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Oracle Cloud account\/tenancy requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>An <strong>Oracle Cloud tenancy<\/strong> with permission to create and manage OCI Streaming resources.<\/li>\n<li>Access to the <strong>OCI Console<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Permissions \/ IAM roles<\/h3>\n\n\n\n<p>You typically need:\n&#8211; Ability to create\/manage <strong>stream pools<\/strong> and <strong>streams<\/strong> in a compartment.\n&#8211; Ability for an application identity (user or dynamic group) to <strong>use<\/strong> streams (produce\/consume).<\/p>\n\n\n\n<p>OCI IAM is policy-based; exact policy verbs and resource types must match current OCI Streaming policy reference. Use official docs to confirm. Start by searching OCI docs for \u201cStreaming IAM policies\u201d.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Billing requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OCI Streaming is a paid service (with possible Always Free usage in some tenancies\/regions).<br\/>\n  Confirm current free tier eligibility on the official Oracle Cloud Free Tier and Streaming pricing pages.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Tools needed<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A machine to run Kafka clients:<\/li>\n<li>Local workstation (Linux\/macOS\/Windows) <strong>or<\/strong><\/li>\n<li>OCI Compute instance <strong>or<\/strong><\/li>\n<li>Kubernetes pod in OKE.<\/li>\n<li><strong>Kafka client tools<\/strong> (one of):<\/li>\n<li>Apache Kafka binaries (includes <code>kafka-console-producer<\/code>\/<code>kafka-console-consumer<\/code>)  <\/li>\n<li>Confluent Kafka distribution tools (compatible clients)<\/li>\n<li>Language client (Java, Python, Go, .NET)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Region availability<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OCI Streaming is available in many OCI regions, but not necessarily all.<br\/>\n  Verify availability in your target region in official docs or the OCI Console service list.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas\/limits<\/h3>\n\n\n\n<p>Expect quotas\/limits around:\n&#8211; Number of stream pools\n&#8211; Number of streams\n&#8211; Partitions per stream\n&#8211; Retention bounds\n&#8211; Throughput per partition\/pool\n&#8211; Max message size and request rates<\/p>\n\n\n\n<p>Exact values change by region and service updates\u2014verify the current limits in official docs and in the OCI Console quota pages.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisite services (recommended)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>OCI Vault<\/strong> for storing auth tokens\/credentials securely.<\/li>\n<li><strong>OCI Monitoring<\/strong> (enabled by default) for alarms.<\/li>\n<li><strong>VCN + private connectivity<\/strong> for production deployments.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">9. Pricing \/ Cost<\/h2>\n\n\n\n<blockquote>\n<p>Pricing changes over time and varies by region\/currency. Do not rely on blog posts for exact numbers\u2014use Oracle\u2019s official pricing pages and the OCI Cost Estimator\/Calculator.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Current pricing model (how you are billed)<\/h3>\n\n\n\n<p>Streaming with Apache Kafka uses OCI Streaming pricing. Common pricing dimensions for managed streaming services include:\n&#8211; <strong>Partition-hours (or partition time)<\/strong>: Cost tied to how many partitions you provision and how long they exist.\n&#8211; <strong>Data ingestion (GB in)<\/strong>: Cost tied to the volume of data written to streams.\n&#8211; <strong>Data egress (GB out)<\/strong>: Cost tied to data read from streams (and possibly cross-region\/network egress depending on topology).\n&#8211; <strong>Retention\/storage<\/strong>: Some models include storage cost implicitly in partition-hours; others break it out. Verify how OCI Streaming prices retention and stored data.<\/p>\n\n\n\n<p><strong>Verify in official pricing docs<\/strong>:\n&#8211; OCI Streaming pricing page (start from Oracle pricing and navigate to Streaming):<br\/>\n  https:\/\/www.oracle.com\/cloud\/price-list\/<br\/>\n&#8211; OCI Cost Estimator \/ Calculator:<br\/>\n  https:\/\/www.oracle.com\/cloud\/costestimator.html<br\/>\n&#8211; OCI Streaming official documentation (billing\/pricing notes, if available):<br\/>\n  https:\/\/docs.oracle.com\/en-us\/iaas\/Content\/Streaming\/home.htm<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Free tier (if applicable)<\/h3>\n\n\n\n<p>Oracle Cloud has an Always Free offering, but Always Free eligibility and limits vary and can change.<br\/>\nCheck:\n&#8211; Oracle Cloud Free Tier: https:\/\/www.oracle.com\/cloud\/free\/<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Primary cost drivers<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Number of partitions<\/strong> (and how long they exist): Most common major driver.<\/li>\n<li><strong>Throughput<\/strong>: High write\/read volumes increase ingestion\/egress charges.<\/li>\n<li><strong>Retention period<\/strong>: Longer retention can increase storage footprint and\/or cost.<\/li>\n<li><strong>Consumer fan-out<\/strong>: Multiple consumer groups reading the same data multiplies egress (each group reads the full stream).<\/li>\n<li><strong>Cross-region traffic<\/strong>: If producers\/consumers are in different regions, network egress can dominate.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Hidden or indirect costs<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Compute for consumers\/producers<\/strong>: VMs\/OKE nodes running Kafka clients.<\/li>\n<li><strong>NAT Gateway \/ Load Balancer<\/strong> costs if you place clients in private subnets and need outbound internet.<\/li>\n<li><strong>Logging costs<\/strong>: If you export detailed logs to OCI Logging and retain them for long periods.<\/li>\n<li><strong>Object Storage<\/strong>: If you land data into a lake, storage and requests add cost.<\/li>\n<li><strong>Key Management\/Vault<\/strong>: Secret storage and KMS operations may add marginal costs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network\/data transfer implications<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data transfer charges may apply when traffic leaves a region or exits OCI.<br\/>\n  Keep producers and consumers in the same region and use private connectivity where possible to reduce exposure and egress surprises.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How to optimize cost<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Right-size <strong>partition count<\/strong>: Start small; increase partitions only when needed for throughput\/parallelism.<\/li>\n<li>Limit <strong>retention<\/strong> to what you actually need (e.g., 24\u2013168 hours) unless replay requirements justify more.<\/li>\n<li>Reduce <strong>consumer fan-out<\/strong>: Consolidate consumers or use downstream storage for additional consumers to read from.<\/li>\n<li>Compress payloads (application-level compression if supported by your clients and accepted by your processing).<\/li>\n<li>Use batching to reduce per-request overhead (client-side).<\/li>\n<li>Avoid cross-region reads; replicate\/forward data intentionally if needed.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example low-cost starter estimate (conceptual)<\/h3>\n\n\n\n<p>A minimal dev\/test setup typically uses:\n&#8211; 1 stream\n&#8211; 1 partition\n&#8211; Low retention (e.g., 24 hours)\n&#8211; Small ingestion\/egress volumes\n&#8211; Public endpoint (for quick testing)<\/p>\n\n\n\n<p>Use the OCI calculator to estimate with your region and expected GB\/day.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example production cost considerations (conceptual)<\/h3>\n\n\n\n<p>For production, plan for:\n&#8211; Multiple streams (per domain)\n&#8211; Higher partitions for throughput and consumer parallelism\n&#8211; Multiple consumer groups (each adds read volume)\n&#8211; Private networking (possible additional network resources)\n&#8211; Monitoring\/alerting and longer retention\n&#8211; Landing to Object Storage or databases<\/p>\n\n\n\n<p>Production estimates must be built from:\n&#8211; Expected message size \u00d7 messages\/sec\n&#8211; Number of consumer groups\n&#8211; Retention hours\n&#8211; Partition count growth plan\n&#8211; Cross-region data movement (if any)<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">10. Step-by-Step Hands-On Tutorial<\/h2>\n\n\n\n<p>This lab is designed to be <strong>realistic, low-risk, and beginner-friendly<\/strong>. It uses standard Kafka CLI tools to produce and consume messages against Oracle Cloud\u2019s Kafka-compatible streaming endpoint.<\/p>\n\n\n\n<blockquote>\n<p>Because exact Kafka authentication strings\/endpoints can change and are region-specific, this tutorial intentionally has you <strong>copy the Kafka connection settings from the OCI Console<\/strong> (or official docs) instead of hardcoding potentially outdated formats.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Objective<\/h3>\n\n\n\n<p>Create an OCI Streaming stream and use <strong>Kafka console tools<\/strong> to:\n1. Produce JSON messages into the stream\n2. Consume them from a consumer group\n3. Verify delivery and basic operations\n4. Clean up all resources to avoid ongoing cost<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Lab Overview<\/h3>\n\n\n\n<p>You will:\n1. Create a compartment (optional) for isolation\n2. Create a Stream Pool\n3. Create a Stream (topic equivalent)\n4. Create an IAM user\/group\/policy for streaming access (or use an existing controlled identity)\n5. Generate an auth token (if required by the Kafka compatibility method)\n6. Configure Kafka client properties (TLS\/SASL)\n7. Produce and consume messages\n8. Validate and troubleshoot\n9. Clean up resources<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Create (or choose) a compartment for the lab<\/h3>\n\n\n\n<p><strong>Console path<\/strong>: OCI Console \u2192 Identity &amp; Security \u2192 Compartments<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Create a compartment such as <code>lab-streaming-kafka<\/code> (or reuse an existing dev compartment).<\/li>\n<li>Note the compartment name for policy statements and resource creation.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome<\/strong>: You have a compartment dedicated to streaming lab resources.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Create a Stream Pool<\/h3>\n\n\n\n<p><strong>Console path<\/strong>: OCI Console \u2192 (Search) \u201cStreaming\u201d \u2192 Stream Pools \u2192 Create Stream Pool<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Choose the <strong>compartment<\/strong> (<code>lab-streaming-kafka<\/code>).<\/li>\n<li>Name the pool <code>kafka-lab-pool<\/code>.<\/li>\n<li>Choose endpoint\/networking:\n   &#8211; For a quick lab: <strong>public endpoint<\/strong> (if available in your tenancy\/region).\n   &#8211; For production-like: <strong>private endpoint<\/strong> in a VCN (more secure; requires VCN\/subnet setup).<\/li>\n<li>Create the stream pool.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome<\/strong>: Stream pool is in <strong>Active<\/strong> state.<\/p>\n\n\n\n<p><strong>Verification<\/strong>\n&#8211; Open the stream pool details page.\n&#8211; Look for a section such as <strong>Kafka settings \/ Kafka connection information<\/strong> (names vary).\n&#8211; You should be able to find:\n  &#8211; <strong>Bootstrap server(s)<\/strong>\n  &#8211; <strong>Security protocol requirements<\/strong> (TLS\/SASL)\n  &#8211; Any required <strong>SASL mechanism<\/strong> and <strong>username format<\/strong><\/p>\n\n\n\n<blockquote>\n<p>If you cannot find Kafka connection info in the Console, use official docs for \u201cStreaming with Apache Kafka\u201d under OCI Streaming.<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Create a Stream (Kafka topic equivalent)<\/h3>\n\n\n\n<p><strong>Console path<\/strong>: Streaming \u2192 Streams \u2192 Create Stream<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Select your stream pool <code>kafka-lab-pool<\/code>.<\/li>\n<li>Create a stream named: <code>demo-telemetry<\/code><\/li>\n<li>Choose:\n   &#8211; Partitions: <code>1<\/code> (lowest-cost starting point)\n   &#8211; Retention: keep default or set a short retention appropriate for a lab<\/li>\n<li>Create the stream.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome<\/strong>: Stream <code>demo-telemetry<\/code> exists and is Active.<\/p>\n\n\n\n<p><strong>Verification<\/strong>\n&#8211; Open the stream details page and confirm:\n  &#8211; Partitions = 1\n  &#8211; Retention is set\n  &#8211; Stream belongs to your pool<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Create an IAM identity for Kafka clients (user or workload identity)<\/h3>\n\n\n\n<p>For a beginner lab, a dedicated IAM <strong>user<\/strong> is easiest. For production on OCI Compute\/OKE, prefer <strong>instance principals<\/strong> or <strong>workload identity<\/strong> patterns where supported (verify current recommended approach in docs).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Option A (lab-friendly): Create a user + group + policy<\/h4>\n\n\n\n<p><strong>Console path<\/strong>: Identity &amp; Security \u2192 Users \/ Groups \/ Policies<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Create a group: <code>StreamingKafkaLabUsers<\/code><\/li>\n<li>Create a user: <code>streaming-kafka-lab-user<\/code><\/li>\n<li>Add the user to the group.<\/li>\n<\/ol>\n\n\n\n<p>Now create a policy in your compartment (or at tenancy level if required) that grants permissions to use the stream pool\/streams.<\/p>\n\n\n\n<p><strong>Policy example (verify exact resource types\/verbs in official docs):<\/strong><\/p>\n\n\n\n<pre><code class=\"language-text\">Allow group StreamingKafkaLabUsers to use streams in compartment lab-streaming-kafka\nAllow group StreamingKafkaLabUsers to use stream-pools in compartment lab-streaming-kafka\n<\/code><\/pre>\n\n\n\n<p>If your user will also create\/manage streams\/pools from the lab identity, you may need <code>manage<\/code> rather than <code>use<\/code>. Prefer least privilege.<\/p>\n\n\n\n<p><strong>Expected outcome<\/strong>: The lab identity can authenticate and is authorized to access streaming resources.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Generate an auth token (if required)<\/h4>\n\n\n\n<p><strong>Console path<\/strong>: Identity &amp; Security \u2192 Users \u2192 streaming-kafka-lab-user \u2192 Auth Tokens \u2192 Generate Token<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Generate an auth token.<\/li>\n<li>Copy it immediately and store it securely (password manager or Vault).<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome<\/strong>: You have a token ready to use as a Kafka password if the OCI Kafka configuration requires it.<\/p>\n\n\n\n<blockquote>\n<p>Some OCI Kafka compatibility configurations use an auth token as the SASL password. Verify the current auth approach in your Console\u2019s Kafka settings or official docs.<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Prepare Kafka client tools (local or Docker)<\/h3>\n\n\n\n<p>You need Kafka console tools. Two practical options:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Option A: Use Docker (recommended for quick labs)<\/h4>\n\n\n\n<p>This avoids installing Java\/Kafka on your machine. You can run Kafka CLI tools from a container image that includes them.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Confirm Docker is installed:<\/li>\n<\/ol>\n\n\n\n<pre><code class=\"language-bash\">docker --version\n<\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li>Pull a Kafka tooling image that includes CLI utilities. For example, Confluent images often include Kafka CLI tools.<br\/>\nVerify the image\/tag you choose includes the exact commands used below.<\/li>\n<\/ol>\n\n\n\n<pre><code class=\"language-bash\">docker pull confluentinc\/cp-kafka:latest\n<\/code><\/pre>\n\n\n\n<blockquote>\n<p>If your organization restricts Docker Hub, use an approved internal registry or install Apache Kafka binaries locally.<\/p>\n<\/blockquote>\n\n\n\n<h4 class=\"wp-block-heading\">Option B: Install Apache Kafka locally<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Download a Kafka release from the Apache Kafka official site: https:\/\/kafka.apache.org\/downloads<\/li>\n<li>Extract and ensure you can run <code>bin\/kafka-console-producer.sh<\/code> and <code>bin\/kafka-console-consumer.sh<\/code><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Create a Kafka client configuration file<\/h3>\n\n\n\n<p>Create a file named <code>client.properties<\/code>. You will fill it with values from the OCI Console\u2019s Kafka connection settings for your stream pool.<\/p>\n\n\n\n<p><strong>Important<\/strong>: Do not guess these values. Copy them from the Console or official docs.<\/p>\n\n\n\n<p>Example template (you must verify each setting):<\/p>\n\n\n\n<pre><code class=\"language-properties\"># --- Connection ---\nbootstrap.servers=&lt;PASTE_FROM_OCI_STREAM_POOL_KAFKA_SETTINGS&gt;\n\n# --- Security ---\nsecurity.protocol=SASL_SSL\nsasl.mechanism=PLAIN\n\n# If required, the username format is OCI-specific.\n# Many managed Kafka-compatible services use a structured username and an auth token as password.\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \\\n  username=\"&lt;PASTE_USERNAME_FROM_OCI_SETTINGS&gt;\" \\\n  password=\"&lt;PASTE_AUTH_TOKEN_OR_PASSWORD&gt;\";\n\n# Optional (only if OCI docs instruct you to set these)\n# ssl.endpoint.identification.algorithm=https\n# request.timeout.ms=30000\n# retry.backoff.ms=500\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>: You have a valid client config that matches OCI requirements.<\/p>\n\n\n\n<p><strong>Verification<\/strong>\n&#8211; Confirm you can reach the bootstrap host\/port from your machine\/network.\n  &#8211; If using a public endpoint, ensure outbound to the endpoint is allowed.\n  &#8211; If using a private endpoint, run the client from within the VCN (Compute\/OKE) and ensure NSG\/Security Lists allow it.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 7: Produce messages to the stream using Kafka console producer<\/h3>\n\n\n\n<p>In OCI Streaming Kafka compatibility, your <strong>stream name<\/strong> is typically used as the Kafka <strong>topic name<\/strong> (verify the mapping rules in docs).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Using Docker (example)<\/h4>\n\n\n\n<p>Run producer:<\/p>\n\n\n\n<pre><code class=\"language-bash\">docker run --rm -it \\\n  -v \"$PWD:\/work\" -w \/work \\\n  confluentinc\/cp-kafka:latest \\\n  kafka-console-producer \\\n    --bootstrap-server \"$(grep -E '^bootstrap\\.servers=' client.properties | cut -d= -f2-)\" \\\n    --producer.config \/work\/client.properties \\\n    --topic demo-telemetry\n<\/code><\/pre>\n\n\n\n<p>Now paste a few lines (each line is one Kafka record value), for example:<\/p>\n\n\n\n<pre><code class=\"language-json\">{\"ts\":\"2026-04-17T10:00:00Z\",\"service\":\"web\",\"level\":\"INFO\",\"msg\":\"startup complete\"}\n{\"ts\":\"2026-04-17T10:00:05Z\",\"service\":\"web\",\"level\":\"WARN\",\"msg\":\"high latency detected\",\"latency_ms\":180}\n{\"ts\":\"2026-04-17T10:00:10Z\",\"service\":\"web\",\"level\":\"INFO\",\"msg\":\"request served\",\"path\":\"\/api\/orders\",\"status\":200}\n<\/code><\/pre>\n\n\n\n<p>Exit with <code>Ctrl+C<\/code>.<\/p>\n\n\n\n<p><strong>Expected outcome<\/strong>: Messages are accepted by the service without authentication or authorization errors.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 8: Consume messages using Kafka console consumer<\/h3>\n\n\n\n<p>Consume from the beginning:<\/p>\n\n\n\n<pre><code class=\"language-bash\">docker run --rm -it \\\n  -v \"$PWD:\/work\" -w \/work \\\n  confluentinc\/cp-kafka:latest \\\n  kafka-console-consumer \\\n    --bootstrap-server \"$(grep -E '^bootstrap\\.servers=' client.properties | cut -d= -f2-)\" \\\n    --consumer.config \/work\/client.properties \\\n    --topic demo-telemetry \\\n    --from-beginning \\\n    --group demo-telemetry-lab-group\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>: You see the JSON lines printed in the consumer terminal.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 9: (Optional) Validate offset\/consumer group behavior<\/h3>\n\n\n\n<p>Kafka offset\/consumer group behavior is central to real systems. What you can validate depends on which Kafka admin APIs are supported.<\/p>\n\n\n\n<p>Basic validation without admin APIs:\n1. Start the consumer with <code>--from-beginning<\/code> and a new group name \u2192 you should see all retained messages.\n2. Stop the consumer and restart it with the same group name \u2192 you should not re-read already committed messages (assuming auto-commit is enabled by the CLI and supported).<\/p>\n\n\n\n<p>Example: Start with a new group:<\/p>\n\n\n\n<pre><code class=\"language-bash\"># new group name to force full replay\n--group demo-telemetry-lab-group-2\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>: Replay works based on retention and group offsets.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Validation<\/h3>\n\n\n\n<p>Use the following checklist:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>\n<p><strong>Producer succeeded<\/strong>\n   &#8211; No <code>SASL authentication failed<\/code>\n   &#8211; No <code>Not authorized<\/code>\n   &#8211; No repeated timeouts<\/p>\n<\/li>\n<li>\n<p><strong>Consumer received records<\/strong>\n   &#8211; Records appear as expected\n   &#8211; Replay works with a new consumer group<\/p>\n<\/li>\n<li>\n<p><strong>OCI Console<\/strong>\n   &#8211; Stream exists, status Active\n   &#8211; Metrics show activity (if metrics are exposed for your configuration; check Monitoring for the stream pool\/stream)<\/p>\n<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Error: <code>SASL authentication failed<\/code> \/ <code>Authentication failed<\/code><\/h4>\n\n\n\n<p>Common causes:\n&#8211; Wrong username format (OCI-specific)\n&#8211; Wrong password\/auth token\n&#8211; Token revoked\/expired (or rotated)\n&#8211; Incorrect SASL mechanism<\/p>\n\n\n\n<p>Fix:\n&#8211; Re-copy the <strong>Kafka connection settings<\/strong> from the stream pool details page.\n&#8211; Generate a new auth token if required.\n&#8211; Verify the client properties match the official \u201cStreaming with Apache Kafka\u201d documentation.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Error: <code>Not authorized to access topics<\/code><\/h4>\n\n\n\n<p>Common causes:\n&#8211; IAM policy missing <code>use<\/code> permissions for streams\/stream-pools\n&#8211; Wrong compartment in the policy\n&#8211; Using a different stream pool than the policy allows<\/p>\n\n\n\n<p>Fix:\n&#8211; Confirm the policy statements target the correct compartment.\n&#8211; Ensure the user is in the correct group (or dynamic group if using instance principals).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Error: Timeouts \/ cannot connect to bootstrap server<\/h4>\n\n\n\n<p>Common causes:\n&#8211; Network path blocked (corporate firewall)\n&#8211; Trying to reach a private endpoint from the public internet\n&#8211; NSG\/Security List rules blocking egress\/ingress inside a VCN<\/p>\n\n\n\n<p>Fix:\n&#8211; If using a private endpoint, run the client from Compute\/OKE inside the VCN.\n&#8211; Verify DNS resolution and routing.\n&#8211; Confirm allowed ports and rules per OCI docs.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Error: TLS handshake errors<\/h4>\n\n\n\n<p>Common causes:\n&#8211; TLS interception proxy\n&#8211; Outdated JRE CA bundle (if using Java-based clients)\n&#8211; Incorrect SSL settings<\/p>\n\n\n\n<p>Fix:\n&#8211; Try from a clean network path.\n&#8211; Update JRE \/ CA certificates.\n&#8211; Follow OCI docs for SSL configuration requirements.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Cleanup<\/h3>\n\n\n\n<p>To avoid ongoing costs, delete resources in reverse order:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Stop producer\/consumer terminals.<\/li>\n<li>Delete the stream <code>demo-telemetry<\/code>.<\/li>\n<li>Delete the stream pool <code>kafka-lab-pool<\/code>.<\/li>\n<li>Delete (or disable) the IAM user and auth token (or at least revoke the token).<\/li>\n<li>Remove policies\/groups if they were created only for this lab.<\/li>\n<li>If you created a VCN\/private endpoint only for this lab, delete those networking resources too.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome<\/strong>: No streaming resources remain and billing stops for partition-hours and traffic.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">11. Best Practices<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Architecture best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Design by domain<\/strong>: Create streams aligned to business domains (orders, payments, telemetry) rather than one \u201cmega topic\u201d.<\/li>\n<li><strong>Partition with intent<\/strong>: Choose partition keys that preserve needed ordering (e.g., orderId) and distribute load.<\/li>\n<li><strong>Plan for replay<\/strong>: Retention should cover your maximum downstream outage + reprocessing window.<\/li>\n<li><strong>Use multiple consumer groups<\/strong> for independent processing pipelines (but budget for the multiplied egress\/read volume).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IAM\/security best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Least privilege<\/strong>: Separate policies for:<\/li>\n<li>Admins (manage streams\/pools)<\/li>\n<li>Applications (use streams only)<\/li>\n<li><strong>Prefer workload identities<\/strong> (dynamic groups\/instance principals) where supported, over long-lived user credentials.<\/li>\n<li><strong>Rotate secrets\/tokens<\/strong> regularly; store them in <strong>OCI Vault<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Cost best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Keep partitions minimal and scale only when justified by throughput and consumer parallelism.<\/li>\n<li>Keep retention low in dev\/test.<\/li>\n<li>Reduce consumer fan-out or move \u201cextra readers\u201d to downstream storage (Object Storage) to avoid repeated reads from streaming.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Performance best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Batch producer sends; tune linger\/batch size in your client (client-specific).<\/li>\n<li>Use compression where appropriate and supported.<\/li>\n<li>Scale consumers by adding instances to the same consumer group.<\/li>\n<li>Monitor consumer lag (if exposed) and throughput; scale partitions only when necessary.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Reliability best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Make consumers idempotent; expect retries and reprocessing.<\/li>\n<li>Use dead-letter patterns:<\/li>\n<li>Send invalid\/unprocessable events to a separate stream.<\/li>\n<li>Keep raw events for later forensic replay.<\/li>\n<li>Implement backpressure: consumers should gracefully handle downstream slowdowns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operations best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Standardize naming: <code>env.domain.purpose<\/code> (example: <code>prod.orders.events<\/code>), within OCI naming constraints.<\/li>\n<li>Tag resources: <code>Environment<\/code>, <code>Owner<\/code>, <code>CostCenter<\/code>, <code>DataClassification<\/code>.<\/li>\n<li>Use alarms and on-call routing (OCI Monitoring + Notifications).<\/li>\n<li>Automate provisioning via Terraform for reproducibility (verify official OCI Terraform provider docs for Streaming resources).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Governance best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use separate compartments for dev\/test\/prod.<\/li>\n<li>Apply quotas\/limits per compartment to prevent accidental sprawl.<\/li>\n<li>Document event schemas and enforce compatibility rules in CI\/CD.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">12. Security Considerations<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Identity and access model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OCI uses <strong>IAM policies<\/strong> scoped to compartments.<\/li>\n<li>For Kafka clients, authentication is typically configured via a Kafka SASL mechanism combined with OCI credentials (often an auth token).<br\/>\n  Always follow the current official \u201cStreaming with Apache Kafka\u201d configuration instructions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Encryption<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>In transit<\/strong>: Use TLS (<code>SASL_SSL<\/code>) per OCI guidance.<\/li>\n<li><strong>At rest<\/strong>: Managed services typically encrypt at rest by default in OCI, but encryption controls and key management options can vary. Verify OCI Streaming encryption-at-rest behavior and any customer-managed key options in official docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network exposure<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Prefer <strong>private endpoints<\/strong> and VCN-only access for production to reduce attack surface.<\/li>\n<li>If using public endpoints:<\/li>\n<li>Restrict where producers\/consumers run (locked-down networks).<\/li>\n<li>Use strict IAM and short-lived credentials.<\/li>\n<li>Monitor for anomalous usage.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secrets handling<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Store auth tokens and connection strings in <strong>OCI Vault<\/strong>.<\/li>\n<li>Never commit <code>client.properties<\/code> with real secrets to Git.<\/li>\n<li>Use separate credentials per environment and per application.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Audit\/logging<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ensure <strong>OCI Audit<\/strong> is enabled (typically on by default) and retained per compliance requirements.<\/li>\n<li>Log application-level events (connectivity failures, auth failures, throttling) to OCI Logging.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compliance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Classify stream data (PII\/PCI\/PHI) and enforce:<\/li>\n<li>Encryption requirements<\/li>\n<li>Retention constraints<\/li>\n<li>Access controls and separation of duties<\/li>\n<li>Data minimization and masking if needed<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Common security mistakes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Overly broad IAM policies (<code>manage all-resources in tenancy<\/code>) for streaming apps.<\/li>\n<li>Long-lived user credentials in CI\/CD or container images.<\/li>\n<li>Public endpoints with weak network controls.<\/li>\n<li>No monitoring\/alerts on throughput spikes (potential abuse).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secure deployment recommendations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Run clients in private subnets; use private endpoints where possible.<\/li>\n<li>Use dynamic groups\/workload identity patterns.<\/li>\n<li>Rotate credentials, enforce MFA for human admins.<\/li>\n<li>Implement schema validation and input sanitization for producers.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">13. Limitations and Gotchas<\/h2>\n\n\n\n<blockquote>\n<p>Treat this section as a checklist to validate early in a design.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Kafka compatibility scope<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>OCI\u2019s Kafka compatibility may not cover <strong>every<\/strong> Kafka API feature.<\/li>\n<li>Admin operations (e.g., topic creation, ACL management) may be limited or handled via OCI control plane instead of Kafka protocol.<\/li>\n<li>Kafka Connect compatibility varies depending on connectors and required broker features\u2014verify with official docs and test with a proof of concept.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas and limits<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Maximum streams\/partitions\/retention can be capped by service limits and tenancy quotas.<\/li>\n<li>Throughput per partition\/pool may be limited; scaling requires partitioning and consumer parallelism.<\/li>\n<li>Max record size may be limited; large payloads should be stored in Object Storage with an event containing a pointer.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Regional constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Resources are region-scoped; cross-region consumption can introduce latency and egress costs.<\/li>\n<li>Not all regions may support the same endpoint\/networking features.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing surprises<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Multiple consumer groups multiply read volume.<\/li>\n<li>Long retention and high partitions increase baseline cost (partition-hours).<\/li>\n<li>Cross-region traffic or internet egress can dominate total cost.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compatibility issues<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Client TLS\/SASL configuration mismatches are common.<\/li>\n<li>Corporate TLS interception can break Kafka TLS.<\/li>\n<li>Some Kafka CLI images\/tools may not match the required Kafka protocol version\u2014use versions recommended by Oracle docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational gotchas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Misconfigured partition keys cause hot partitions and throttling.<\/li>\n<li>Consumers that are not idempotent can produce duplicate side effects during retries.<\/li>\n<li>Lack of schema governance leads to breaking changes and downstream failures.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Migration challenges<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Migrating from self-managed Kafka may require:<\/li>\n<li>Topic\/stream naming mapping<\/li>\n<li>ACL\/IAM model changes<\/li>\n<li>Retention and partition strategy redesign<\/li>\n<li>Replacing broker-level configs with managed equivalents<\/li>\n<li>Re-validating Kafka client compatibility<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Vendor-specific nuances<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Stream provisioning is often via OCI APIs, not Kafka topic auto-create.<\/li>\n<li>Authentication is OCI-specific; do not assume the same as AWS MSK or Confluent Cloud.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">14. Comparison with Alternatives<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Nearest services in Oracle Cloud<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>OCI Streaming (native API)<\/strong>: Same underlying service; Kafka compatibility is an access mode. Native SDK may offer different operational knobs or integrations.<\/li>\n<li><strong>OCI Events Service<\/strong>: Best for OCI resource event routing and lower-throughput eventing patterns (not a Kafka replacement).<\/li>\n<li><strong>OCI Notifications<\/strong>: Pub\/sub notifications (push), not a high-throughput streaming log.<\/li>\n<li><strong>Oracle Integration Cloud<\/strong>: Enterprise integration\/iPaaS for SaaS\/app workflows; complementary to streaming.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Nearest services in other clouds<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>AWS MSK<\/strong>: Managed Kafka clusters (more Kafka control, more ops responsibility).<\/li>\n<li><strong>AWS Kinesis Data Streams<\/strong>: Managed streaming but proprietary API.<\/li>\n<li><strong>Azure Event Hubs (Kafka endpoint)<\/strong>: Kafka-compatible endpoint; conceptually similar approach.<\/li>\n<li><strong>Google Cloud Pub\/Sub<\/strong>: Managed messaging with different semantics.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Open-source \/ self-managed alternatives<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Self-managed Apache Kafka<\/strong> on VMs or Kubernetes (full control, highest ops burden).<\/li>\n<li><strong>Redpanda<\/strong> (Kafka-compatible, different architecture; could be self-managed or managed via Redpanda Cloud).<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Comparison table<\/h4>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Option<\/th>\n<th>Best For<\/th>\n<th>Strengths<\/th>\n<th>Weaknesses<\/th>\n<th>When to Choose<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>Oracle Cloud \u2013 Streaming with Apache Kafka<\/strong><\/td>\n<td>OCI-native event streaming using Kafka clients<\/td>\n<td>Managed service, IAM integration, Kafka client reuse<\/td>\n<td>Kafka feature coverage may be partial; OCI-specific auth; verify compatibility<\/td>\n<td>You run workloads on OCI and want Kafka-style streaming without brokers<\/td>\n<\/tr>\n<tr>\n<td><strong>Oracle Cloud \u2013 OCI Events Service<\/strong><\/td>\n<td>Resource\/event routing, simple integration triggers<\/td>\n<td>Simple, integrates with OCI resource events<\/td>\n<td>Not built for high-throughput streaming or replay like Kafka<\/td>\n<td>You need event routing for OCI changes and lightweight automation<\/td>\n<\/tr>\n<tr>\n<td><strong>Oracle Integration Cloud<\/strong><\/td>\n<td>SaaS\/app integration workflows<\/td>\n<td>Rich connectors, transformations, process orchestration<\/td>\n<td>Not a high-throughput streaming backbone<\/td>\n<td>You need enterprise iPaaS integration more than streaming<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS MSK<\/strong><\/td>\n<td>Full Kafka cluster semantics with managed infra<\/td>\n<td>Broad Kafka compatibility, broker-level features<\/td>\n<td>Higher ops complexity and cost; cluster management still matters<\/td>\n<td>You need deeper Kafka control or existing MSK footprint<\/td>\n<\/tr>\n<tr>\n<td><strong>Azure Event Hubs (Kafka endpoint)<\/strong><\/td>\n<td>Kafka-style clients for Azure ingestion<\/td>\n<td>Kafka endpoint, strong ingestion pipeline<\/td>\n<td>Semantics differ from Kafka; compatibility boundaries<\/td>\n<td>You\u2019re on Azure and want Kafka clients without Kafka clusters<\/td>\n<\/tr>\n<tr>\n<td><strong>Self-managed Kafka (VM\/OKE)<\/strong><\/td>\n<td>Maximum control and customization<\/td>\n<td>Full Kafka features, plugins, exact versions<\/td>\n<td>Highest ops burden, patching, scaling, availability engineering<\/td>\n<td>You need features not supported by managed compatibility layers<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">15. Real-World Example<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Enterprise example: Retail order event backbone<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: A retail enterprise has tightly coupled synchronous integrations between ordering, payment, fraud, fulfillment, and CRM systems. Peak load causes timeouts and cascading failures. They also need an auditable event trail.<\/li>\n<li><strong>Proposed architecture<\/strong><\/li>\n<li>Use <strong>Streaming with Apache Kafka<\/strong> as the central event backbone.<\/li>\n<li><code>orders-service<\/code> produces <code>OrderCreated<\/code>, <code>OrderPaid<\/code>, <code>OrderShipped<\/code>.<\/li>\n<li><code>fraud-service<\/code>, <code>fulfillment-service<\/code>, and <code>crm-sync<\/code> consume in separate consumer groups.<\/li>\n<li>A \u201clake landing\u201d consumer writes raw events to <strong>Object Storage<\/strong> for audit and analytics.<\/li>\n<li>All workloads run on <strong>OKE<\/strong> in private subnets; the stream pool uses <strong>private endpoints<\/strong>.<\/li>\n<li>Auth secrets stored in <strong>OCI Vault<\/strong>; IAM policies restrict apps to <code>use<\/code> only required streams.<\/li>\n<li><strong>Why this service was chosen<\/strong><\/li>\n<li>Kafka client reuse avoids re-platforming effort.<\/li>\n<li>OCI-managed streaming reduces operational load versus running Kafka clusters.<\/li>\n<li>Compartment\/IAM integration supports enterprise governance.<\/li>\n<li><strong>Expected outcomes<\/strong><\/li>\n<li>Reduced coupling and fewer cascading failures.<\/li>\n<li>Ability to replay events during incident recovery.<\/li>\n<li>Clear ownership and auditability through IAM and logging.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Startup\/small-team example: SaaS telemetry and alerting pipeline<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: A startup SaaS product needs to ingest app telemetry from multiple services and run near-real-time alerting, but the team can\u2019t afford to run Kafka.<\/li>\n<li><strong>Proposed architecture<\/strong><\/li>\n<li>One stream per environment: <code>dev-telemetry<\/code>, <code>prod-telemetry<\/code>.<\/li>\n<li>Producers in services publish JSON telemetry.<\/li>\n<li>A single consumer group processes and sends alerts to a webhook\/on-call system.<\/li>\n<li>Optional: Land raw telemetry to Object Storage daily for debugging.<\/li>\n<li><strong>Why this service was chosen<\/strong><\/li>\n<li>Managed service reduces operational burden.<\/li>\n<li>Kafka clients\/tools speed up development with known patterns.<\/li>\n<li><strong>Expected outcomes<\/strong><\/li>\n<li>Reliable buffering during spikes.<\/li>\n<li>Faster incident detection.<\/li>\n<li>Minimal infra footprint and predictable scaling path (add partitions\/consumers as they grow).<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">16. FAQ<\/h2>\n\n\n\n<p>1) <strong>Is \u201cStreaming with Apache Kafka\u201d a separate Oracle Cloud service?<\/strong><br\/>\nUsually, it refers to using <strong>OCI Streaming<\/strong> via Kafka-compatible endpoints\/clients. Confirm the latest naming and scope in the OCI Streaming docs.<\/p>\n\n\n\n<p>2) <strong>Do I have to run Kafka brokers in Oracle Cloud to use Kafka clients?<\/strong><br\/>\nNo. With Streaming with Apache Kafka, you typically do <strong>not<\/strong> manage brokers; OCI provides a managed streaming backend.<\/p>\n\n\n\n<p>3) <strong>Can I use standard Kafka client libraries (Java\/Python\/Go\/.NET)?<\/strong><br\/>\nOften yes, provided your client version and the required security settings match OCI\u2019s Kafka compatibility requirements. Verify supported Kafka versions\/APIs in official docs.<\/p>\n\n\n\n<p>4) <strong>How do topics map to OCI resources?<\/strong><br\/>\nCommonly, an OCI <strong>stream<\/strong> maps to a Kafka <strong>topic name<\/strong>. Stream creation may be done through the OCI control plane rather than Kafka Admin APIs\u2014verify current behavior.<\/p>\n\n\n\n<p>5) <strong>Can I create topics using <code>kafka-topics --create<\/code>?<\/strong><br\/>\nNot necessarily. Some managed Kafka-compatible services restrict topic creation to their control plane. Test and verify with OCI docs.<\/p>\n\n\n\n<p>6) <strong>How do I authenticate?<\/strong><br\/>\nOCI typically uses <strong>IAM-backed authentication<\/strong>; Kafka clients may use SASL over TLS with an OCI-specific username format and an auth token\/password. Always copy settings from the OCI Console Kafka connection info.<\/p>\n\n\n\n<p>7) <strong>What is a stream pool?<\/strong><br\/>\nA stream pool is a logical container for streams and often the place where endpoints\/network access are configured.<\/p>\n\n\n\n<p>8) <strong>Is data encrypted in transit?<\/strong><br\/>\nKafka connections typically use TLS (<code>SASL_SSL<\/code>) as required by OCI Kafka configuration. Confirm current requirements in docs.<\/p>\n\n\n\n<p>9) <strong>How do I keep the endpoint private?<\/strong><br\/>\nUse a stream pool configuration that provides <strong>private access<\/strong> within a VCN (if supported in your region) and run clients inside that VCN.<\/p>\n\n\n\n<p>10) <strong>How do I monitor producer\/consumer health?<\/strong><br\/>\nUse OCI Monitoring metrics and application logs. Track throughput, errors, and consumer lag (if exposed).<\/p>\n\n\n\n<p>11) <strong>What are the main scaling levers?<\/strong><br\/>\n&#8211; Increase <strong>partitions<\/strong> for parallelism\/throughput<br\/>\n&#8211; Increase consumer instances in a <strong>consumer group<\/strong><br\/>\n&#8211; Optimize producer batching\/compression<\/p>\n\n\n\n<p>12) <strong>Does increasing partitions affect ordering?<\/strong><br\/>\nOrdering is generally per partition. If you increase partitions and change keying, global ordering is not guaranteed.<\/p>\n\n\n\n<p>13) <strong>How long can I retain events?<\/strong><br\/>\nRetention is configurable within service constraints. Longer retention can increase cost. Verify retention limits in docs.<\/p>\n\n\n\n<p>14) <strong>What\u2019s the best way to handle large payloads?<\/strong><br\/>\nStore large objects in <strong>Object Storage<\/strong> and publish an event containing a reference (bucket\/key, version, checksum).<\/p>\n\n\n\n<p>15) <strong>Can I use Kafka Connect?<\/strong><br\/>\nPossibly, but compatibility depends on supported Kafka APIs and connector requirements. Validate with a proof-of-concept and OCI\u2019s Kafka compatibility documentation.<\/p>\n\n\n\n<p>16) <strong>What is the most common cause of connection failures?<\/strong><br\/>\nIncorrect SASL\/SSL configuration and IAM policy issues, followed by network reachability problems (private endpoint vs public).<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">17. Top Online Resources to Learn Streaming with Apache Kafka<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Resource Type<\/th>\n<th>Name<\/th>\n<th>Why It Is Useful<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Official documentation<\/td>\n<td>OCI Streaming Docs<\/td>\n<td>Authoritative feature set, concepts, limits, and how-to guides: https:\/\/docs.oracle.com\/en-us\/iaas\/Content\/Streaming\/home.htm<\/td>\n<\/tr>\n<tr>\n<td>Official pricing<\/td>\n<td>Oracle Cloud Price List<\/td>\n<td>Official price list entry points (navigate to Streaming): https:\/\/www.oracle.com\/cloud\/price-list\/<\/td>\n<\/tr>\n<tr>\n<td>Pricing calculator<\/td>\n<td>OCI Cost Estimator<\/td>\n<td>Estimate region-specific costs: https:\/\/www.oracle.com\/cloud\/costestimator.html<\/td>\n<\/tr>\n<tr>\n<td>Free tier<\/td>\n<td>Oracle Cloud Free Tier<\/td>\n<td>Check Always Free eligibility\/limits: https:\/\/www.oracle.com\/cloud\/free\/<\/td>\n<\/tr>\n<tr>\n<td>Architecture center<\/td>\n<td>Oracle Architecture Center<\/td>\n<td>Reference architectures and best practices (search for streaming\/event-driven): https:\/\/docs.oracle.com\/en\/solutions\/<\/td>\n<\/tr>\n<tr>\n<td>Kafka learning<\/td>\n<td>Apache Kafka Documentation<\/td>\n<td>Kafka concepts, client configuration, semantics: https:\/\/kafka.apache.org\/documentation\/<\/td>\n<\/tr>\n<tr>\n<td>CLI tools<\/td>\n<td>Apache Kafka Downloads<\/td>\n<td>Official Kafka binaries for console producer\/consumer: https:\/\/kafka.apache.org\/downloads<\/td>\n<\/tr>\n<tr>\n<td>Community learning<\/td>\n<td>Confluent Developer (Kafka concepts)<\/td>\n<td>Practical Kafka patterns; validate OCI compatibility separately: https:\/\/developer.confluent.io\/<\/td>\n<\/tr>\n<tr>\n<td>Terraform (official)<\/td>\n<td>OCI Terraform Provider Docs<\/td>\n<td>Automate creation of streaming resources (verify resource names): https:\/\/registry.terraform.io\/providers\/oracle\/oci\/latest\/docs<\/td>\n<\/tr>\n<tr>\n<td>SDK reference<\/td>\n<td>OCI SDKs<\/td>\n<td>Integrate OCI control plane via SDKs (search for Streaming): https:\/\/docs.oracle.com\/en-us\/iaas\/Content\/API\/Concepts\/sdks.htm<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">18. Training and Certification Providers<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Institute<\/th>\n<th>Suitable Audience<\/th>\n<th>Likely Learning Focus<\/th>\n<th>Mode<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps engineers, SREs, platform teams<\/td>\n<td>DevOps + cloud automation + operationalizing streaming systems<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>ScmGalaxy.com<\/td>\n<td>Beginners to intermediate engineers<\/td>\n<td>DevOps fundamentals, tools, CI\/CD; complementary skills for integration platforms<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.scmgalaxy.com\/<\/td>\n<\/tr>\n<tr>\n<td>CLoudOpsNow.in<\/td>\n<td>Cloud engineers and operations teams<\/td>\n<td>Cloud ops practices, monitoring, cost, governance<\/td>\n<td>Check website<\/td>\n<td>https:\/\/cloudopsnow.in\/<\/td>\n<\/tr>\n<tr>\n<td>SreSchool.com<\/td>\n<td>SREs, reliability engineers<\/td>\n<td>Reliability engineering practices for production platforms<\/td>\n<td>Check website<\/td>\n<td>https:\/\/sreschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>AiOpsSchool.com<\/td>\n<td>Ops teams adopting AIOps<\/td>\n<td>Observability, automation, and operations analytics<\/td>\n<td>Check website<\/td>\n<td>https:\/\/aiopsschool.com\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">19. Top Trainers<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Platform\/Site<\/th>\n<th>Likely Specialization<\/th>\n<th>Suitable Audience<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>RajeshKumar.xyz<\/td>\n<td>DevOps\/cloud guidance and training materials<\/td>\n<td>Beginners to working professionals<\/td>\n<td>https:\/\/rajeshkumar.xyz\/<\/td>\n<\/tr>\n<tr>\n<td>devopstrainer.in<\/td>\n<td>DevOps tools and practices<\/td>\n<td>Engineers seeking practical DevOps enablement<\/td>\n<td>https:\/\/devopstrainer.in\/<\/td>\n<\/tr>\n<tr>\n<td>devopsfreelancer.com<\/td>\n<td>Independent consulting\/training engagements<\/td>\n<td>Teams needing project-based coaching<\/td>\n<td>https:\/\/devopsfreelancer.com\/<\/td>\n<\/tr>\n<tr>\n<td>devopssupport.in<\/td>\n<td>Operational support and training resources<\/td>\n<td>Ops\/DevOps teams needing troubleshooting support<\/td>\n<td>https:\/\/devopssupport.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">20. Top Consulting Companies<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Company<\/th>\n<th>Likely Service Area<\/th>\n<th>Where They May Help<\/th>\n<th>Consulting Use Case Examples<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>cotocus.com<\/td>\n<td>Cloud\/DevOps\/engineering services<\/td>\n<td>Architecture, implementation, and operations support<\/td>\n<td>Event streaming platform rollout; OKE-based consumer deployments; IAM\/policy hardening<\/td>\n<td>https:\/\/cotocus.com\/<\/td>\n<\/tr>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps consulting and training<\/td>\n<td>Enablement, migrations, operational readiness<\/td>\n<td>Kafka-client migration to OCI Streaming; CI\/CD for streaming apps; SRE playbooks<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>DEVOPSCONSULTING.IN<\/td>\n<td>DevOps and cloud consulting<\/td>\n<td>Assessments and implementation guidance<\/td>\n<td>Cost reviews for partition\/retention; observability setup; secure private endpoint deployments<\/td>\n<td>https:\/\/devopsconsulting.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">21. Career and Learning Roadmap<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn before this service<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kafka fundamentals:<\/li>\n<li>Topics\/streams, partitions, offsets<\/li>\n<li>Producer acks\/retries<\/li>\n<li>Consumer groups and rebalancing<\/li>\n<li>OCI fundamentals:<\/li>\n<li>Compartments, IAM policies, tagging<\/li>\n<li>VCN basics (subnets, NSGs, routing)<\/li>\n<li>Basic security:<\/li>\n<li>TLS, secrets management, least privilege<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn after this service<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Event-driven architecture patterns:<\/li>\n<li>Outbox pattern, sagas, idempotency<\/li>\n<li>Schema evolution strategies (Avro\/Protobuf\/JSON Schema)<\/li>\n<li>Production operations:<\/li>\n<li>Consumer lag management, backpressure<\/li>\n<li>Incident response and replay procedures<\/li>\n<li>Data engineering:<\/li>\n<li>Stream-to-lake patterns, partitioned storage layouts<\/li>\n<li>Stream processing frameworks (verify compatibility and deployment approach)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Job roles that use it<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud engineer \/ cloud developer<\/li>\n<li>Solutions architect<\/li>\n<li>DevOps engineer \/ platform engineer<\/li>\n<li>SRE<\/li>\n<li>Data engineer (streaming pipelines)<\/li>\n<li>Security engineer (telemetry pipelines)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Certification path (if available)<\/h3>\n\n\n\n<p>Oracle certification programs change over time. Verify current OCI certification paths here:<br\/>\nhttps:\/\/education.oracle.com\/<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Project ideas for practice<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Build an \u201corders\u201d event stream with 3 consumers (billing, shipping, email) and idempotent handlers.<\/li>\n<li>Implement a telemetry pipeline: producers \u2192 stream \u2192 consumer \u2192 Object Storage (compressed files) with daily partitions.<\/li>\n<li>Build a replay tool that can reprocess a time window into a new downstream store.<\/li>\n<li>Create Terraform automation for stream pools\/streams + IAM policies + monitoring alarms (verify resource support).<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">22. Glossary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>OCI (Oracle Cloud Infrastructure)<\/strong>: Oracle Cloud\u2019s IaaS platform, including networking, compute, IAM, and managed services.<\/li>\n<li><strong>Integration (category)<\/strong>: Architectural area focused on connecting services and systems via APIs, events, messaging, and automation.<\/li>\n<li><strong>Stream<\/strong>: A named log of events with partitions and retention (often analogous to a Kafka topic).<\/li>\n<li><strong>Stream Pool<\/strong>: Administrative container for streams; often includes endpoint\/network configuration.<\/li>\n<li><strong>Partition<\/strong>: Ordered subset of a stream enabling parallelism; ordering is typically guaranteed per partition.<\/li>\n<li><strong>Offset<\/strong>: Position of a record within a partition.<\/li>\n<li><strong>Consumer Group<\/strong>: A set of consumers sharing work across partitions; each partition is typically processed by one group member at a time.<\/li>\n<li><strong>Retention<\/strong>: How long events remain available for consumption\/replay.<\/li>\n<li><strong>Producer<\/strong>: Application publishing events to a stream.<\/li>\n<li><strong>Consumer<\/strong>: Application reading events from a stream.<\/li>\n<li><strong>SASL<\/strong>: Simple Authentication and Security Layer used by Kafka clients for authentication.<\/li>\n<li><strong>TLS\/SSL<\/strong>: Encryption protocol used to secure network traffic.<\/li>\n<li><strong>IAM Policy<\/strong>: OCI statement granting permissions to users\/groups\/dynamic groups within compartments.<\/li>\n<li><strong>Auth Token<\/strong>: OCI-generated secret used by some services for programmatic authentication (usage depends on service configuration).<\/li>\n<li><strong>Dead-letter stream<\/strong>: A stream used to store events that could not be processed successfully.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">23. Summary<\/h2>\n\n\n\n<p>Streaming with Apache Kafka in <strong>Oracle Cloud<\/strong> is a practical way to implement <strong>event streaming<\/strong> using familiar <strong>Kafka clients<\/strong> while relying on <strong>OCI Streaming<\/strong> as the managed backend. It matters because it reduces the operational burden of running Kafka clusters, improves decoupling in microservices and integration architectures, and supports replayable, scalable event pipelines.<\/p>\n\n\n\n<p>Cost and security are primarily driven by <strong>partition-hours<\/strong>, <strong>ingress\/egress volume<\/strong>, <strong>retention<\/strong>, <strong>consumer fan-out<\/strong>, and your choice of <strong>public vs private endpoints<\/strong>, with IAM policies and secret handling (Vault) being central to secure deployments.<\/p>\n\n\n\n<p>Use it when you want Kafka-style streaming on OCI with managed operations; avoid it when you require full Kafka broker control or Kafka features outside OCI\u2019s supported compatibility scope. The best next step is to complete the hands-on lab, then validate production requirements (compatibility, limits, networking, and pricing) using the official OCI Streaming documentation and pricing calculator.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Integration<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48,62],"tags":[],"class_list":["post-927","post","type-post","status-publish","format-standard","hentry","category-integration","category-oracle-cloud"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/927","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=927"}],"version-history":[{"count":0,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/927\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=927"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=927"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=927"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}