{"id":667,"date":"2026-04-14T23:04:50","date_gmt":"2026-04-14T23:04:50","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/google-cloud-pub-sub-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-data-analytics-and-pipelines\/"},"modified":"2026-04-14T23:04:50","modified_gmt":"2026-04-14T23:04:50","slug":"google-cloud-pub-sub-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-data-analytics-and-pipelines","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/google-cloud-pub-sub-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-data-analytics-and-pipelines\/","title":{"rendered":"Google Cloud Pub\/Sub Tutorial: Architecture, Pricing, Use Cases, and Hands-On Guide for Data analytics and pipelines"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Category<\/h2>\n\n\n\n<p>Data analytics and pipelines<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. Introduction<\/h2>\n\n\n\n<p>Pub\/Sub is Google Cloud\u2019s fully managed asynchronous messaging service for event ingestion and distribution. It\u2019s commonly used as the \u201cfront door\u201d for streaming data analytics and pipelines, and as the backbone for event-driven architectures.<\/p>\n\n\n\n<p>In simple terms: producers publish messages to a topic, and one or more consumers receive those messages through subscriptions. Producers and consumers don\u2019t need to know about each other, and they can scale independently.<\/p>\n\n\n\n<p>Technically, Pub\/Sub implements a durable publish\/subscribe messaging pattern with features such as push or pull delivery, message retention, ordering (when enabled), filtering, dead-letter topics, and integrations across Google Cloud services. It is designed for high-throughput, low-latency event ingestion and fan-out, supporting many-to-many communication.<\/p>\n\n\n\n<p>Pub\/Sub solves common problems in data analytics and pipelines such as:\n&#8211; Decoupling data producers from downstream processing systems\n&#8211; Buffering bursts of events while consumers scale out\n&#8211; Reliably delivering event streams to multiple consumers (fan-out)\n&#8211; Building resilient ETL\/ELT streaming pipelines (e.g., into BigQuery via Dataflow)\n&#8211; Enabling event-driven microservices without running messaging infrastructure<\/p>\n\n\n\n<blockquote>\n<p>Naming note: You may still see \u201cCloud Pub\/Sub\u201d in older articles and code samples. The product is commonly referenced as \u201cPub\/Sub\u201d in Google Cloud documentation and the console. Verify current naming in the official docs if you are aligning to internal standards: https:\/\/cloud.google.com\/pubsub\/docs\/overview<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">2. What is Pub\/Sub?<\/h2>\n\n\n\n<p>Pub\/Sub is a managed messaging service on Google Cloud for ingesting and delivering event streams. Its official purpose is to let applications publish events (messages) and deliver them asynchronously to subscribers, with durable storage for a configurable retention window and scalable delivery semantics.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Core capabilities<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Publish messages to a topic (events can include a payload and attributes)<\/li>\n<li>Deliver messages to subscribers via:<\/li>\n<li><strong>Pull<\/strong> subscriptions (subscribers pull messages)<\/li>\n<li><strong>Push<\/strong> subscriptions (Pub\/Sub pushes to an HTTPS endpoint)<\/li>\n<li>Scale to high throughput and large numbers of publishers\/subscribers<\/li>\n<li>Support common resilience patterns: retries, dead-letter topics, replay\/seek (snapshots), and buffering<\/li>\n<li>Integrate with many Google Cloud services (Dataflow, Cloud Run, Cloud Functions, BigQuery patterns, Logging\/Monitoring, IAM)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Major components<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Topic<\/strong>: A named resource that accepts published messages.<\/li>\n<li><strong>Subscription<\/strong>: A named resource attached to a topic that delivers that topic\u2019s messages to subscribers.<\/li>\n<li><strong>Publisher<\/strong>: Any client that sends messages to a topic.<\/li>\n<li><strong>Subscriber<\/strong>: Any client that receives and processes messages from a subscription.<\/li>\n<li><strong>Push endpoint<\/strong> (push subscriptions): A publicly reachable HTTPS service (often Cloud Run) that receives POST requests.<\/li>\n<li><strong>Dead-letter topic (DLT\/DLQ)<\/strong>: A topic to receive messages that can\u2019t be processed successfully after a configured number of delivery attempts.<\/li>\n<li><strong>Snapshot<\/strong>: A point-in-time state of a subscription, used for replay\/seek workflows.<\/li>\n<li><strong>Schema<\/strong> (optional): A definition for message validation (e.g., Avro\/Protocol Buffers). Verify current schema support details in docs: https:\/\/cloud.google.com\/pubsub\/docs\/schemas<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Service type<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fully managed, serverless messaging service (you don\u2019t manage brokers, partitions, or clusters for standard Pub\/Sub).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scope and resource model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub resources are <strong>project-scoped<\/strong> in Google Cloud.<\/li>\n<li>Topics and subscriptions are global resources within a project in the sense that you don\u2019t pick a single VM zone for them; however, Pub\/Sub provides controls such as <strong>message storage policies<\/strong> (data residency constraints) and may offer regional behaviors\/endpoints depending on configuration. Confirm the latest location and residency semantics in official docs: https:\/\/cloud.google.com\/pubsub\/docs\/locations<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How it fits into Google Cloud<\/h3>\n\n\n\n<p>In the Google Cloud ecosystem, Pub\/Sub often sits between:\n&#8211; <strong>Ingestion sources<\/strong>: applications, IoT devices, logs\/telemetry collectors, Cloud Run services, on-prem systems\n&#8211; <strong>Stream processing<\/strong>: Dataflow (Apache Beam), Dataproc (Spark), custom consumers on GKE\/Compute Engine\n&#8211; <strong>Storage\/analytics sinks<\/strong>: BigQuery, Cloud Storage, Bigtable, Spanner, Elasticsearch (self-managed), third-party platforms<\/p>\n\n\n\n<p>For Data analytics and pipelines, Pub\/Sub is frequently the ingestion layer that decouples upstream event production from downstream transformations and analytics.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">3. Why use Pub\/Sub?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Business reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Faster delivery of data products<\/strong>: teams can add new consumers without changing producer code.<\/li>\n<li><strong>Reduced operational burden<\/strong>: no need to operate a messaging cluster for many common scenarios.<\/li>\n<li><strong>Improved resilience<\/strong>: messages can buffer during spikes or downstream outages, reducing data loss risk.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Technical reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Decoupling<\/strong>: producers and consumers evolve independently.<\/li>\n<li><strong>Fan-out<\/strong>: multiple subscriptions can receive the same topic\u2019s events for different use cases (analytics, monitoring, ML features, auditing).<\/li>\n<li><strong>Backpressure handling<\/strong>: retention and subscriber scaling help absorb bursty traffic.<\/li>\n<li><strong>Delivery controls<\/strong>: acknowledgments, retries, dead-letter topics, message filtering, ordering (when enabled).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Elastic scaling<\/strong>: handle bursty ingestion and variable consumption rates.<\/li>\n<li><strong>Managed reliability<\/strong>: reduces the complexity of patching, scaling, and monitoring broker fleets.<\/li>\n<li><strong>Observability hooks<\/strong>: integrates with Cloud Monitoring metrics and Cloud Logging \/ Audit Logs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/compliance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>IAM-based access control<\/strong> for topics and subscriptions.<\/li>\n<li><strong>Encryption at rest and in transit<\/strong> (Google-managed by default; CMEK options may be available\u2014verify in docs).<\/li>\n<li><strong>Auditability<\/strong> via Cloud Audit Logs for admin and data access patterns (depending on configuration and log types enabled).<\/li>\n<li><strong>Data residency controls<\/strong> via storage policies (confirm details in official docs).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scalability\/performance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Designed for <strong>high-throughput<\/strong> ingestion and delivery.<\/li>\n<li>Supports parallelism through multiple subscribers and flow control settings in client libraries.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should choose it<\/h3>\n\n\n\n<p>Choose Pub\/Sub when you need:\n&#8211; Asynchronous event ingestion for streaming pipelines\n&#8211; Decoupled microservices and event-driven workflows\n&#8211; Multi-consumer fan-out\n&#8211; Managed messaging without running brokers<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should not choose it<\/h3>\n\n\n\n<p>Avoid or reconsider Pub\/Sub when:\n&#8211; You need <strong>strict exactly-once processing end-to-end<\/strong> across complex pipelines without carefully designed idempotency (Pub\/Sub can offer exactly-once delivery for some subscription modes, but \u201cexactly-once processing\u201d still requires application design\u2014verify latest constraints in docs).\n&#8211; You require <strong>very long-term storage<\/strong> of messages as a system of record (use BigQuery\/Cloud Storage instead; Pub\/Sub is a transport\/buffer).\n&#8211; You need <strong>complex stream reprocessing<\/strong> across large windows beyond Pub\/Sub retention (consider storing raw events durably in Cloud Storage\/BigQuery).\n&#8211; You require <strong>Kafka protocol compatibility<\/strong> (consider Kafka on GKE\/Compute Engine, or a managed Kafka offering from a partner).<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">4. Where is Pub\/Sub used?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Industries<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fintech and banking: transaction events, fraud signals, audit pipelines<\/li>\n<li>Retail\/e-commerce: clickstreams, orders, inventory updates<\/li>\n<li>Media\/ads: event tracking, near-real-time analytics<\/li>\n<li>Healthcare: event-driven integrations (with careful compliance controls)<\/li>\n<li>Manufacturing\/IoT: device telemetry ingestion<\/li>\n<li>SaaS: product analytics, webhook processing, workflow orchestration<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Team types<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data engineering teams building streaming pipelines<\/li>\n<li>Platform teams implementing event buses<\/li>\n<li>Backend application teams building microservices<\/li>\n<li>SRE\/operations teams centralizing telemetry<\/li>\n<li>Security teams building detection pipelines<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Workloads<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Streaming ETL\/ELT (Pub\/Sub \u2192 Dataflow \u2192 BigQuery)<\/li>\n<li>Event-driven microservices (Pub\/Sub \u2192 Cloud Run)<\/li>\n<li>Near-real-time monitoring\/alert enrichment<\/li>\n<li>Log ingestion pipelines<\/li>\n<li>Asynchronous task distribution (when \u201cevent messaging\u201d fits better than task queues)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Architectures<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Event-driven architecture (EDA)<\/li>\n<li>CQRS\/event sourcing adjunct pipelines (Pub\/Sub as a transport, not the event store)<\/li>\n<li>Streaming analytics<\/li>\n<li>Hybrid ingestion (on-prem \u2192 Pub\/Sub \u2192 Google Cloud analytics)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Real-world deployment contexts<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Production: multi-topic event bus, DLQs, monitoring dashboards, SLOs for consumer lag, IAM boundaries<\/li>\n<li>Dev\/test: smaller topics, shorter retention, ephemeral subscriptions, emulator usage for local tests (verify Pub\/Sub emulator capabilities in official docs)<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">5. Top Use Cases and Scenarios<\/h2>\n\n\n\n<p>Below are realistic scenarios where Pub\/Sub is commonly the right fit in Google Cloud Data analytics and pipelines.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1) Streaming ingestion into BigQuery (via Dataflow)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You need near-real-time analytics over application events.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Durable ingestion buffer; Dataflow can read from Pub\/Sub and write to BigQuery with transforms.<\/li>\n<li><strong>Example<\/strong>: Web app publishes click events to <code>clicks-topic<\/code>; Dataflow aggregates and writes to BigQuery tables for dashboards.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2) Event-driven microservices fan-out<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Multiple services need to react to the same business event.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Multiple subscriptions can consume the same topic independently.<\/li>\n<li><strong>Example<\/strong>: <code>orders-topic<\/code> feeds billing, shipping, email notifications, and analytics services via separate subscriptions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3) Decoupling batch producers from real-time consumers<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Upstream systems produce bursts; downstream systems can\u2019t keep up.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Buffers bursts; consumers scale out; retention covers downtime windows.<\/li>\n<li><strong>Example<\/strong>: Nightly job publishes thousands of \u201crecompute\u201d events; subscribers process them over hours.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4) Centralized audit\/event pipeline<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You need a unified stream of key business events for audit and compliance reporting.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Central event topic with controlled subscribers; retention allows short-term replay.<\/li>\n<li><strong>Example<\/strong>: \u201cAccount updated\u201d and \u201cPrivilege changed\u201d events published to an audit topic; consumers store to BigQuery and Cloud Storage.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5) IoT telemetry ingestion<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Many devices send small messages continuously; ingestion must scale.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Designed for high-throughput event ingestion; supports multiple processing paths.<\/li>\n<li><strong>Example<\/strong>: Device telemetry published to <code>telemetry-topic<\/code>; one subscriber detects anomalies; another stores raw events.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6) Webhook ingestion and smoothing<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Third-party webhooks arrive unpredictably and can spike.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Convert synchronous HTTP intake into asynchronous processing.<\/li>\n<li><strong>Example<\/strong>: Cloud Run endpoint validates webhook and publishes to Pub\/Sub; downstream workers process without timing out the webhook sender.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7) Dead-letter handling for \u201cpoison messages\u201d<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Some messages consistently fail processing and block progress.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Dead-letter topics isolate failures after N delivery attempts.<\/li>\n<li><strong>Example<\/strong>: A malformed JSON event is retried; after max attempts it lands in DLQ for inspection.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8) Cross-environment event distribution (dev\/test\/prod)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You want consistent event contracts across environments.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Topics\/subscriptions per environment; schemas help enforce message structure.<\/li>\n<li><strong>Example<\/strong>: <code>orders-v1<\/code> schema enforced across <code>orders-dev<\/code>, <code>orders-stage<\/code>, <code>orders-prod<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">9) Real-time feature updates for ML systems<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: ML features must be updated as events happen.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Low-latency event distribution to feature pipelines.<\/li>\n<li><strong>Example<\/strong>: \u201cUser clicked item\u201d events consumed by a service that updates an online feature store (implementation varies).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">10) Pipeline branching by message attributes (filtering)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Different consumers only need subsets of events.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Subscription filters reduce downstream load and cost.<\/li>\n<li><strong>Example<\/strong>: <code>events-topic<\/code> contains many event types; one subscription filters only <code>eventType=\"purchase\"<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">11) Near-real-time cache invalidation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Cache entries must be invalidated when data changes.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Publish invalidation events; multiple caches\/services react.<\/li>\n<li><strong>Example<\/strong>: Product updates publish <code>productId<\/code> invalidation messages; cache services subscribe.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">12) Data quality and anomaly detection sidecar<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You need real-time checks without slowing main pipeline.<\/li>\n<li><strong>Why Pub\/Sub fits<\/strong>: Add a new subscription for DQ checks without touching producers.<\/li>\n<li><strong>Example<\/strong>: A DQ service subscribes to raw events and flags schema drift or missing fields.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">6. Core Features<\/h2>\n\n\n\n<p>This section focuses on important, current Pub\/Sub capabilities. Always validate feature availability and limitations in the official docs because some features vary by subscription type, region, or client library.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Topics and subscriptions<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Topics receive published messages; subscriptions deliver them to consumers.<\/li>\n<li><strong>Why it matters<\/strong>: Clean separation enables fan-out, independent scaling, and access control.<\/li>\n<li><strong>Practical benefit<\/strong>: Add a new downstream system by creating a subscription\u2014no producer changes.<\/li>\n<li><strong>Caveats<\/strong>: Subscription configuration impacts delivery, retries, and costs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Push and pull delivery<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>:<\/li>\n<li><strong>Pull<\/strong>: subscribers poll or stream-pull messages and ack\/nack.<\/li>\n<li><strong>Push<\/strong>: Pub\/Sub sends HTTPS POST requests to an endpoint.<\/li>\n<li><strong>Why it matters<\/strong>: Lets you choose between consumer-controlled flow (pull) and simpler webhooks (push).<\/li>\n<li><strong>Practical benefit<\/strong>: Pull fits worker pools; push fits HTTP services (Cloud Run).<\/li>\n<li><strong>Caveats<\/strong>: Push requires a reachable HTTPS endpoint and careful auth; pull requires managing subscriber processes.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">At-least-once delivery (default behavior)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Messages may be delivered more than once; subscribers must handle duplicates.<\/li>\n<li><strong>Why it matters<\/strong>: Enables reliability under retries and transient failures.<\/li>\n<li><strong>Practical benefit<\/strong>: Fewer lost events; supports robust pipeline design.<\/li>\n<li><strong>Caveats<\/strong>: Consumers should implement idempotency (e.g., de-dup keys) when needed.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Exactly-once delivery (subscription feature)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Prevents acknowledged messages from being redelivered (within the exactly-once model).<\/li>\n<li><strong>Why it matters<\/strong>: Reduces duplicate processing for pull subscribers.<\/li>\n<li><strong>Practical benefit<\/strong>: Simplifies consumer logic in some cases.<\/li>\n<li><strong>Caveats<\/strong>: Availability and requirements can depend on client libraries and subscription type. Verify current constraints and how to enable it in docs: https:\/\/cloud.google.com\/pubsub\/docs\/exactly-once-delivery<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Message ordering (ordering keys)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Preserves order of messages that share an ordering key, when ordering is enabled.<\/li>\n<li><strong>Why it matters<\/strong>: Some workflows require per-entity ordering (e.g., per order ID).<\/li>\n<li><strong>Practical benefit<\/strong>: Avoids complex reordering in consumers.<\/li>\n<li><strong>Caveats<\/strong>: Ordering reduces throughput for a given key and can increase latency; requires publisher discipline and correct key selection. Verify latest ordering behavior: https:\/\/cloud.google.com\/pubsub\/docs\/ordering<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Message retention and replay (seek\/snapshots)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Retain messages for a configured duration; allow replay by seeking to a timestamp or snapshot.<\/li>\n<li><strong>Why it matters<\/strong>: Enables recovery from consumer bugs or backfills.<\/li>\n<li><strong>Practical benefit<\/strong>: Reprocess last N hours of events without needing a separate event store (within retention).<\/li>\n<li><strong>Caveats<\/strong>: Retention increases storage costs; replay can amplify downstream costs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Acknowledgments, ack deadlines, and retry behavior<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Subscriber acks confirm processing; ack deadlines define how long before redelivery if not acked.<\/li>\n<li><strong>Why it matters<\/strong>: Core reliability mechanism for handling failures and slow processing.<\/li>\n<li><strong>Practical benefit<\/strong>: Consumers can scale and manage long-running tasks by extending ack deadlines (via client libraries).<\/li>\n<li><strong>Caveats<\/strong>: Poor ack management causes duplicates and increased costs. Push subscriptions have different retry semantics\u2014verify docs: https:\/\/cloud.google.com\/pubsub\/docs\/subscriber<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Dead-letter topics (DLQ)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Routes messages that exceed max delivery attempts to a dead-letter topic.<\/li>\n<li><strong>Why it matters<\/strong>: Prevents poison messages from endlessly retrying.<\/li>\n<li><strong>Practical benefit<\/strong>: Operational clarity; separate remediation workflow.<\/li>\n<li><strong>Caveats<\/strong>: Requires IAM permissions between subscription and dead-letter topic; DLQ itself must be monitored.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Subscription filtering<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Filters messages delivered to a subscription based on attributes.<\/li>\n<li><strong>Why it matters<\/strong>: Reduces unnecessary downstream processing.<\/li>\n<li><strong>Practical benefit<\/strong>: Lower compute and simpler consumers.<\/li>\n<li><strong>Caveats<\/strong>: Requires consistent use of attributes by publishers; filtering rules must be tested.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Schemas (Avro\/Protocol Buffers) and validation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Defines and validates message structure.<\/li>\n<li><strong>Why it matters<\/strong>: Helps prevent malformed events and contract drift.<\/li>\n<li><strong>Practical benefit<\/strong>: Safer evolution of event types.<\/li>\n<li><strong>Caveats<\/strong>: Schema enforcement is optional and requires coordination; verify current support and limitations.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IAM integration<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Controls who can publish, subscribe, administer, and view resources.<\/li>\n<li><strong>Why it matters<\/strong>: Messaging is often a central integration point; it must be tightly controlled.<\/li>\n<li><strong>Practical benefit<\/strong>: Least-privilege patterns; separate publisher\/subscriber identities.<\/li>\n<li><strong>Caveats<\/strong>: Misconfigured IAM is a common cause of failures and security incidents.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Observability (metrics, logs, audit)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Exposes metrics (e.g., backlog, throughput), logs for admin activity, and operational signals.<\/li>\n<li><strong>Why it matters<\/strong>: Streaming pipelines require active monitoring and alerting.<\/li>\n<li><strong>Practical benefit<\/strong>: SLOs around consumer lag and error rates.<\/li>\n<li><strong>Caveats<\/strong>: Logging can create cost; choose log levels\/types intentionally.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">7. Architecture and How It Works<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">High-level service architecture<\/h3>\n\n\n\n<p>Pub\/Sub follows a managed broker pattern:\n1. A publisher sends messages to a topic using authenticated Google Cloud APIs.\n2. Pub\/Sub durably stores the message for the topic (within retention and policies).\n3. Each subscription on the topic tracks delivery state independently.\n4. Subscribers receive messages (push or pull).\n5. Subscribers acknowledge (ack) messages after processing; unacked messages are retried.\n6. Messages exceeding retry attempts can be moved to a dead-letter topic (if configured).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Request\/data\/control flow<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Data plane<\/strong>: publish requests, message delivery to subscribers, ack\/nack operations.<\/li>\n<li><strong>Control plane<\/strong>: create\/update topics\/subscriptions, IAM policies, schemas, DLQ configuration.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Integrations with related services<\/h3>\n\n\n\n<p>Common Google Cloud integrations in Data analytics and pipelines:\n&#8211; <strong>Dataflow<\/strong>: native Pub\/Sub IO connectors for streaming pipelines.\n&#8211; <strong>BigQuery<\/strong>: common sink via Dataflow; direct ingestion patterns exist but confirm current recommended approach in docs.\n&#8211; <strong>Cloud Run \/ Cloud Functions<\/strong>: event-driven processing, typically via push subscriptions or Eventarc (depending on trigger model\u2014verify current guidance).\n&#8211; <strong>Cloud Storage<\/strong>: store raw events or batch outputs; notifications can feed Pub\/Sub in some patterns (verify current \u201cCloud Storage notifications\u201d options).\n&#8211; <strong>Cloud Monitoring and Cloud Logging<\/strong>: metrics dashboards, alerting, audit trails.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Dependency services<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>IAM, Service Usage API (enabling Pub\/Sub API)<\/li>\n<li>Cloud Monitoring\/Logging for observability<\/li>\n<li>Optional: Cloud KMS for CMEK (verify current support and setup)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/authentication model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Publishers\/subscribers authenticate using:<\/li>\n<li>User credentials (for development in Cloud Shell)<\/li>\n<li>Service accounts (for production workloads)<\/li>\n<li>Authorization is via IAM roles on topics\/subscriptions.<\/li>\n<li>Push subscriptions to an endpoint can use authentication tokens (OIDC) to secure the HTTP receiver; verify current push authentication model: https:\/\/cloud.google.com\/pubsub\/docs\/push<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Networking model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub is accessed via Google APIs endpoints over HTTPS.<\/li>\n<li>Workloads in VPC can access Pub\/Sub using normal outbound internet routes or with Private Google Access where applicable; perimeter controls may be applied using VPC Service Controls (verify supported configurations).<\/li>\n<li>Push delivery requires the target endpoint to be reachable and able to validate auth tokens.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Monitoring\/logging\/governance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Monitor subscription backlog and oldest unacked message age to detect lag.<\/li>\n<li>Track publish and ack error rates.<\/li>\n<li>Use Cloud Audit Logs for admin actions and review IAM changes.<\/li>\n<li>Use naming conventions and labels for environment, owner, data classification, and cost center.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Simple architecture diagram<\/h3>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart LR\n  A[Publisher App] --&gt;|Publish messages| T[Pub\/Sub Topic]\n  T --&gt; S1[Subscription A]\n  T --&gt; S2[Subscription B]\n  S1 --&gt;|Pull| C1[Subscriber Service A]\n  S2 --&gt;|Push HTTPS| C2[Subscriber Service B]\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Production-style architecture diagram<\/h3>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart TB\n  subgraph Producers\n    P1[Cloud Run API]\n    P2[Batch Job]\n    P3[On-prem Connector]\n  end\n\n  subgraph Messaging\n    T[Pub\/Sub Topic: events]\n    F[Subscription Filter: purchases]\n    G[Subscription: all-events]\n    DLQ[Dead-letter Topic]\n    DLS[DLQ Subscription]\n  end\n\n  subgraph Processing\n    DF[Dataflow Streaming Pipeline]\n    CR[Cloud Run Consumer]\n    DQ[Data Quality Service]\n  end\n\n  subgraph Storage_Analytics\n    BQ[BigQuery]\n    GCS[Cloud Storage Raw Archive]\n  end\n\n  subgraph Ops\n    CM[Cloud Monitoring Alerts]\n    CL[Cloud Logging \/ Audit Logs]\n  end\n\n  P1 --&gt; T\n  P2 --&gt; T\n  P3 --&gt; T\n\n  T --&gt; F\n  T --&gt; G\n\n  F --&gt; DF\n  G --&gt; CR\n  G --&gt; DQ\n\n  DF --&gt; BQ\n  DF --&gt; GCS\n\n  CR -. failed after attempts .-&gt; DLQ\n  DLQ --&gt; DLS\n\n  T --&gt; CL\n  F --&gt; CM\n  G --&gt; CM\n  DF --&gt; CM\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">8. Prerequisites<\/h2>\n\n\n\n<p>Before starting the hands-on lab and using Pub\/Sub in Google Cloud:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Account\/project requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A Google Cloud account<\/li>\n<li>A Google Cloud project with <strong>billing enabled<\/strong> (Pub\/Sub usage is billable beyond free tier\/quotas)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Permissions \/ IAM roles<\/h3>\n\n\n\n<p>For the lab, your user or service account typically needs:\n&#8211; <code>roles\/pubsub.admin<\/code> (create\/manage topics\/subscriptions)\n&#8211; Or more limited roles:\n  &#8211; <code>roles\/pubsub.editor<\/code> (if applicable)\n  &#8211; <code>roles\/pubsub.publisher<\/code> (publish)\n  &#8211; <code>roles\/pubsub.subscriber<\/code> (consume)\n&#8211; Permission to enable APIs: <code>roles\/serviceusage.serviceUsageAdmin<\/code> (or project Owner)<\/p>\n\n\n\n<p>In production, prefer separate service accounts for publishers and subscribers with least privilege.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">CLI\/SDK\/tools<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Cloud Shell<\/strong> (recommended) or local setup with:<\/li>\n<li>Google Cloud CLI (<code>gcloud<\/code>)<\/li>\n<li>Python 3.10+ (or your language runtime)<\/li>\n<li>Pub\/Sub client library (for the tutorial): <code>google-cloud-pubsub<\/code><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">APIs to enable<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub API<br\/>\n  Official docs: https:\/\/cloud.google.com\/pubsub\/docs\/quickstart-client-libraries<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Region availability<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub is generally available across Google Cloud, but location\/residency features and CMEK support can vary. Verify location support: https:\/\/cloud.google.com\/pubsub\/docs\/locations<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas\/limits<\/h3>\n\n\n\n<p>Pub\/Sub enforces quotas (requests per second, message sizes, subscriptions per topic, etc.). Quotas can change and can be project\/region dependent:\n&#8211; Check quotas: https:\/\/cloud.google.com\/pubsub\/quotas<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisite services (optional, depending on your architecture)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud Monitoring\/Logging for dashboards and alerts<\/li>\n<li>Cloud KMS if using CMEK<\/li>\n<li>Dataflow\/BigQuery\/Cloud Run if building end-to-end pipelines<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">9. Pricing \/ Cost<\/h2>\n\n\n\n<p>Pub\/Sub pricing is usage-based. Exact prices vary by SKU and can change, so use official sources for current rates.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Official pricing page: https:\/\/cloud.google.com\/pubsub\/pricing  <\/li>\n<li>Pricing calculator: https:\/\/cloud.google.com\/products\/calculator<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing dimensions (what you pay for)<\/h3>\n\n\n\n<p>Common pricing dimensions for Pub\/Sub include (verify the latest breakdown on the pricing page):\n&#8211; <strong>Data volume<\/strong>:\n  &#8211; Data published to topics (ingress)\n  &#8211; Data delivered to subscriptions (egress within Pub\/Sub service)\n&#8211; <strong>Message delivery<\/strong>: delivery volume scales with fan-out (multiple subscriptions increase delivered bytes).\n&#8211; <strong>Message storage \/ retention<\/strong>: retaining messages longer can incur storage charges (depending on current SKUs).\n&#8211; <strong>Network egress<\/strong>: delivering to subscribers outside Google Cloud regions or to the public internet (push endpoints, on-prem consumers) can incur network egress charges.\n&#8211; <strong>Additional features<\/strong>: Some advanced capabilities may have pricing implications. Confirm on pricing page if applicable (e.g., exactly-once delivery, dedicated capacity, etc.).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Free tier (if applicable)<\/h3>\n\n\n\n<p>Google Cloud often provides free usage tiers for some services, but details and limits can change. Check the Pub\/Sub pricing page for any free tier, free operations, or monthly free volume.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cost drivers (the biggest levers)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Fan-out multiplier<\/strong>: 1 published message delivered to 5 subscriptions counts as 5 deliveries.<\/li>\n<li><strong>Message size<\/strong>: large payloads and verbose attributes cost more.<\/li>\n<li><strong>Retention<\/strong>: longer retention increases storage.<\/li>\n<li><strong>Cross-region and internet delivery<\/strong>: egress charges can dominate.<\/li>\n<li><strong>Retry behavior<\/strong>: repeatedly failing consumers increase redeliveries and costs.<\/li>\n<li><strong>Subscriber inefficiency<\/strong>: slow consumers cause backlog; may lead to longer retention usage and higher compute costs downstream.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Hidden or indirect costs<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Downstream compute<\/strong>: Dataflow\/Cloud Run\/GKE costs can exceed Pub\/Sub costs.<\/li>\n<li><strong>Logging<\/strong>: verbose application logs and audit logs retention can become significant.<\/li>\n<li><strong>Operational overhead<\/strong>: dashboards, alerting, and incident response time.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network\/data transfer implications<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pull subscribers running outside Google Cloud (or in different regions) can incur egress.<\/li>\n<li>Push subscriptions to public endpoints also involve internet egress.<\/li>\n<li>Within Google Cloud, network paths can still incur charges depending on source\/destination and region\u2014validate with pricing docs and calculator.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How to optimize cost<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Minimize message payload size (store large payloads in Cloud Storage; publish pointers\/URIs).<\/li>\n<li>Use subscription filtering to reduce downstream delivery and compute.<\/li>\n<li>Keep retention as low as your recovery requirements allow.<\/li>\n<li>Fix poison messages quickly; use DLQs to avoid infinite retries.<\/li>\n<li>Batch publish messages where appropriate (client libraries support batching).<\/li>\n<li>Use compression at the application layer if your consumers can handle it (trade CPU vs bytes).<\/li>\n<li>Avoid unnecessary fan-out; consider a single processing pipeline that writes to multiple sinks if it reduces delivery duplication.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example low-cost starter estimate (conceptual)<\/h3>\n\n\n\n<p>A small lab setup might include:\n&#8211; One topic, two subscriptions\n&#8211; A few thousand small messages (hundreds of KB to a few MB total)\n&#8211; Pull subscriber running in Cloud Shell briefly<\/p>\n\n\n\n<p>This is typically very low cost, often near free tier levels (if available), but <strong>do not assume<\/strong>: always confirm with the pricing page and your billing reports.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example production cost considerations<\/h3>\n\n\n\n<p>In production, plan for:\n&#8211; Daily\/weekly message volume (GiB) \u00d7 number of subscriptions (fan-out)\n&#8211; Peak throughput (to avoid quota issues and to size downstream compute)\n&#8211; Retention needs (hours vs days)\n&#8211; Cross-region egress\n&#8211; Error rates and retries\n&#8211; Cost attribution by team: use labels, separate projects, or billing export analysis<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">10. Step-by-Step Hands-On Tutorial<\/h2>\n\n\n\n<p>This lab builds a small but realistic Pub\/Sub workflow using:\n&#8211; A topic for events\n&#8211; A \u201cmain\u201d subscription with a dead-letter topic\n&#8211; A filtered subscription (only some events)\n&#8211; A Python pull subscriber that intentionally fails some messages so you can observe retries and DLQ behavior<\/p>\n\n\n\n<p>This is beginner-friendly, executable in Cloud Shell, and designed to stay low-cost.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Objective<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Create a Pub\/Sub topic and subscriptions.<\/li>\n<li>Publish sample messages with attributes.<\/li>\n<li>Consume messages with a Python subscriber (ack successes, nack failures).<\/li>\n<li>Observe retries and dead-letter routing.<\/li>\n<li>Validate filtered subscription behavior.<\/li>\n<li>Clean up resources.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Lab Overview<\/h3>\n\n\n\n<p>You will create:\n&#8211; Topic: <code>events-topic<\/code>\n&#8211; Dead-letter topic: <code>events-dlq-topic<\/code>\n&#8211; Subscription (main): <code>events-sub<\/code> with DLQ configured\n&#8211; Subscription (DLQ): <code>events-dlq-sub<\/code>\n&#8211; Subscription (filtered): <code>events-us-sub<\/code> that only receives messages where <code>region=\"us\"<\/code><\/p>\n\n\n\n<p>You will publish messages like:\n&#8211; Payload: JSON string\n&#8211; Attributes: <code>region<\/code>, <code>eventType<\/code>, <code>shouldFail<\/code><\/p>\n\n\n\n<p>Then you\u2019ll run a Python subscriber that:\n&#8211; If <code>shouldFail=true<\/code>, it nacks the message to trigger retries\n&#8211; Otherwise it acks<\/p>\n\n\n\n<p>After max delivery attempts, failing messages should appear in the DLQ subscription.<\/p>\n\n\n\n<blockquote>\n<p>Notes:\n&#8211; Exact timing of retries and DLQ forwarding can vary. Expect a few minutes for DLQ behavior to become visible.\n&#8211; Commands below use the <code>gcloud<\/code> CLI. Run them from Cloud Shell for easiest setup.<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Set your project and enable the Pub\/Sub API<\/h3>\n\n\n\n<p>1) Open <strong>Google Cloud Console \u2192 Cloud Shell<\/strong>.<\/p>\n\n\n\n<p>2) Set environment variables:<\/p>\n\n\n\n<pre><code class=\"language-bash\">export PROJECT_ID=\"$(gcloud config get-value project)\"\necho \"PROJECT_ID=${PROJECT_ID}\"\n<\/code><\/pre>\n\n\n\n<p>If <code>PROJECT_ID<\/code> is empty, set it:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud config set project YOUR_PROJECT_ID\nexport PROJECT_ID=\"YOUR_PROJECT_ID\"\n<\/code><\/pre>\n\n\n\n<p>3) Enable the Pub\/Sub API:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud services enable pubsub.googleapis.com\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Pub\/Sub API is enabled for the project.<\/p>\n\n\n\n<p><strong>Verify<\/strong><\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud services list --enabled --filter=\"name:pubsub.googleapis.com\"\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Create topics (main + dead-letter)<\/h3>\n\n\n\n<p>Create the main topic:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics create events-topic\n<\/code><\/pre>\n\n\n\n<p>Create the DLQ topic:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics create events-dlq-topic\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Two topics exist in your project.<\/p>\n\n\n\n<p><strong>Verify<\/strong><\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics list\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Create subscriptions (main with DLQ, DLQ subscription, and filtered subscription)<\/h3>\n\n\n\n<p>1) Create the main subscription with a dead-letter policy.<\/p>\n\n\n\n<p>Set a relatively small max delivery attempts so you can see DLQ behavior quickly (for real production you may want higher values):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions create events-sub \\\n  --topic=events-topic \\\n  --ack-deadline=20 \\\n  --dead-letter-topic=events-dlq-topic \\\n  --max-delivery-attempts=5\n<\/code><\/pre>\n\n\n\n<p>2) Create a subscription on the dead-letter topic:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions create events-dlq-sub \\\n  --topic=events-dlq-topic\n<\/code><\/pre>\n\n\n\n<p>3) Create a filtered subscription (only region=us):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions create events-us-sub \\\n  --topic=events-topic \\\n  --filter='attributes.region=\"us\"'\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; <code>events-sub<\/code> receives all messages (with DLQ handling).\n&#8211; <code>events-us-sub<\/code> receives only messages with attribute <code>region=us<\/code>.\n&#8211; <code>events-dlq-sub<\/code> receives dead-lettered messages.<\/p>\n\n\n\n<p><strong>Verify<\/strong><\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions list\ngcloud pubsub subscriptions describe events-sub\n<\/code><\/pre>\n\n\n\n<p>If <code>events-sub<\/code> creation fails due to permissions on the dead-letter topic, verify IAM. Pub\/Sub needs permission to publish to the dead-letter topic. Official DLQ docs include required roles\/permissions: https:\/\/cloud.google.com\/pubsub\/docs\/dead-letter-topics<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Publish sample messages with attributes<\/h3>\n\n\n\n<p>Publish a few messages. We\u2019ll include attributes to drive filtering and failure behavior.<\/p>\n\n\n\n<p>Publish a successful US purchase event:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics publish events-topic \\\n  --message='{\"eventId\":\"e-1001\",\"eventType\":\"purchase\",\"amount\":42.50}' \\\n  --attribute=region=us,eventType=purchase,shouldFail=false\n<\/code><\/pre>\n\n\n\n<p>Publish a successful EU signup event:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics publish events-topic \\\n  --message='{\"eventId\":\"e-1002\",\"eventType\":\"signup\",\"plan\":\"free\"}' \\\n  --attribute=region=eu,eventType=signup,shouldFail=false\n<\/code><\/pre>\n\n\n\n<p>Publish a failing US event (will be nacked by our subscriber):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics publish events-topic \\\n  --message='{\"eventId\":\"e-1003\",\"eventType\":\"purchase\",\"amount\":13.37}' \\\n  --attribute=region=us,eventType=purchase,shouldFail=true\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Three messages are now available for delivery on subscriptions.<\/p>\n\n\n\n<p><strong>Verify (quick pull from filtered sub)<\/strong>\nThis pulls one message if available and auto-acks it (good for quick checks; not how you\u2019d run production consumers):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions pull events-us-sub --limit=5 --auto-ack\n<\/code><\/pre>\n\n\n\n<p>You should see only messages where <code>region=us<\/code> (depending on timing and whether another subscriber consumed them).<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Create a Python pull subscriber (ack successes, nack failures)<\/h3>\n\n\n\n<p>1) Install the Pub\/Sub Python client library in Cloud Shell:<\/p>\n\n\n\n<pre><code class=\"language-bash\">python3 -m pip install --user --upgrade google-cloud-pubsub\n<\/code><\/pre>\n\n\n\n<p>2) Create a subscriber script:<\/p>\n\n\n\n<pre><code class=\"language-bash\">cat &gt; subscriber.py &lt;&lt;'PY'\nimport json\nimport os\nimport time\nfrom google.cloud import pubsub_v1\n\nproject_id = os.environ.get(\"PROJECT_ID\")\nsubscription_id = os.environ.get(\"SUBSCRIPTION_ID\", \"events-sub\")\n\nif not project_id:\n    raise SystemExit(\"PROJECT_ID env var is required\")\n\nsubscription_path = f\"projects\/{project_id}\/subscriptions\/{subscription_id}\"\n\nsubscriber = pubsub_v1.SubscriberClient()\n\ndef callback(message: pubsub_v1.subscriber.message.Message) -&gt; None:\n    attrs = dict(message.attributes or {})\n    data = message.data.decode(\"utf-8\", errors=\"replace\")\n\n    should_fail = attrs.get(\"shouldFail\", \"false\").lower() == \"true\"\n\n    print(\"\\n--- Received message ---\")\n    print(f\"Message ID: {message.message_id}\")\n    print(f\"Publish time: {message.publish_time}\")\n    print(f\"Attributes: {attrs}\")\n    print(f\"Data: {data}\")\n\n    # Simulate processing\n    time.sleep(1)\n\n    if should_fail:\n        print(\"Simulated failure -&gt; nack (will retry, may go to DLQ after max attempts)\")\n        message.nack()\n        return\n\n    # Example idempotency hint: use eventId from payload if present\n    try:\n        payload = json.loads(data)\n        print(f\"Parsed eventId: {payload.get('eventId')}\")\n    except Exception:\n        pass\n\n    print(\"Processed OK -&gt; ack\")\n    message.ack()\n\nstreaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)\nprint(f\"Listening on {subscription_path}... Press Ctrl+C to stop.\")\n\ntry:\n    streaming_pull_future.result()\nexcept KeyboardInterrupt:\n    streaming_pull_future.cancel()\n    print(\"Stopped.\")\nPY\n<\/code><\/pre>\n\n\n\n<p>3) Run the subscriber:<\/p>\n\n\n\n<pre><code class=\"language-bash\">export PROJECT_ID=\"$(gcloud config get-value project)\"\nexport SUBSCRIPTION_ID=\"events-sub\"\npython3 subscriber.py\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; The subscriber prints received messages.\n&#8211; Messages with <code>shouldFail=false<\/code> are acked and stop retrying.\n&#8211; Messages with <code>shouldFail=true<\/code> are nacked and retried until they reach max delivery attempts, then they should be published to the dead-letter topic.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Observe dead-letter behavior<\/h3>\n\n\n\n<p>While the subscriber is running (or after stopping it), wait a bit for retries and DLQ routing (often a couple minutes).<\/p>\n\n\n\n<p>Then pull from the DLQ subscription:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions pull events-dlq-sub --limit=10 --auto-ack\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; The failing message (<code>eventId e-1003<\/code>) eventually appears in <code>events-dlq-sub<\/code>.<\/p>\n\n\n\n<p><strong>If you don\u2019t see it yet<\/strong>\n&#8211; Wait longer and try again.\n&#8211; Confirm that <code>events-sub<\/code> has the dead-letter policy configured:\n  <code>bash\n  gcloud pubsub subscriptions describe events-sub --format=\"flattened(deadLetterPolicy)\"<\/code>\n&#8211; Confirm your subscriber is nacking the message (or not acking it).\n&#8211; Verify <code>max-delivery-attempts<\/code> is set and the message is being retried.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Step 7: Validate subscription filtering<\/h3>\n\n\n\n<p>Publish two more messages\u2014one US and one EU:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics publish events-topic \\\n  --message='{\"eventId\":\"e-1004\",\"eventType\":\"purchase\",\"amount\":9.99}' \\\n  --attribute=region=us,eventType=purchase,shouldFail=false\n\ngcloud pubsub topics publish events-topic \\\n  --message='{\"eventId\":\"e-1005\",\"eventType\":\"purchase\",\"amount\":19.99}' \\\n  --attribute=region=eu,eventType=purchase,shouldFail=false\n<\/code><\/pre>\n\n\n\n<p>Pull from the US-only subscription:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions pull events-us-sub --limit=10 --auto-ack\n<\/code><\/pre>\n\n\n\n<p>Pull from the main subscription (if your Python subscriber is stopped):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions pull events-sub --limit=10 --auto-ack\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; <code>events-us-sub<\/code> returns only the US message(s).\n&#8211; <code>events-sub<\/code> can return both US and EU messages (unless already consumed).<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Validation<\/h3>\n\n\n\n<p>Use these checks to confirm your setup is correct.<\/p>\n\n\n\n<p>1) List topics\/subscriptions:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics list\ngcloud pubsub subscriptions list\n<\/code><\/pre>\n\n\n\n<p>2) Inspect subscription configuration:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions describe events-sub\ngcloud pubsub subscriptions describe events-us-sub\n<\/code><\/pre>\n\n\n\n<p>3) Check backlog\/metrics in the Console:\n&#8211; Go to <strong>Google Cloud Console \u2192 Pub\/Sub \u2192 Subscriptions<\/strong>\n&#8211; Open <code>events-sub<\/code>\n&#8211; Look for message backlog and delivery metrics<br\/>\n(Exact metric names and UI can change; verify with Cloud Monitoring docs if needed.)<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting<\/h3>\n\n\n\n<p>Common issues and fixes:<\/p>\n\n\n\n<p>1) <strong><code>PERMISSION_DENIED<\/code> when creating DLQ subscription<\/strong>\n&#8211; Cause: missing permissions for dead-letter topic usage.\n&#8211; Fix: ensure you have admin rights for the lab, or follow required IAM bindings in DLQ docs: https:\/\/cloud.google.com\/pubsub\/docs\/dead-letter-topics<\/p>\n\n\n\n<p>2) <strong>No messages received by subscriber<\/strong>\n&#8211; Confirm you\u2019re listening to the correct subscription:\n  <code>bash\n  echo $PROJECT_ID\n  echo $SUBSCRIPTION_ID<\/code>\n&#8211; Publish a new test message and watch logs.\n&#8211; Ensure the subscription exists and is attached to the topic.<\/p>\n\n\n\n<p>3) <strong>Messages keep retrying even after \u201csuccess\u201d<\/strong>\n&#8211; Check your subscriber code calls <code>message.ack()<\/code> and does not crash before acking.\n&#8211; If your subscriber process terminates before ack, the message will be redelivered.<\/p>\n\n\n\n<p>4) <strong>DLQ message never appears<\/strong>\n&#8211; Ensure your subscriber is consistently failing the same message (nack or no ack).\n&#8211; Ensure <code>--max-delivery-attempts<\/code> is set low enough for the lab.\n&#8211; Wait longer; delivery attempts may not happen instantly.<\/p>\n\n\n\n<p>5) <strong>Local Python dependency issues in Cloud Shell<\/strong>\n&#8211; Re-run:\n  <code>bash\n  python3 -m pip install --user --upgrade google-cloud-pubsub<\/code>\n&#8211; Ensure you\u2019re using <code>python3<\/code>.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h3 class=\"wp-block-heading\">Cleanup<\/h3>\n\n\n\n<p>Delete lab resources to avoid ongoing costs.<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub subscriptions delete events-sub\ngcloud pubsub subscriptions delete events-us-sub\ngcloud pubsub subscriptions delete events-dlq-sub\n\ngcloud pubsub topics delete events-topic\ngcloud pubsub topics delete events-dlq-topic\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Topics and subscriptions are removed.<\/p>\n\n\n\n<p>Verify:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics list\ngcloud pubsub subscriptions list\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">11. Best Practices<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Architecture best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Design for idempotency<\/strong>: assume duplicate delivery (at-least-once). Use unique event IDs and de-dup logic where necessary.<\/li>\n<li><strong>Separate topics by event domain<\/strong>: e.g., <code>orders-events<\/code>, <code>payments-events<\/code>, not one giant topic for everything.<\/li>\n<li><strong>Use attributes intentionally<\/strong>: keep payload stable; use attributes for routing\/filtering metadata (e.g., <code>eventType<\/code>, <code>tenantId<\/code>, <code>region<\/code>, <code>schemaVersion<\/code>).<\/li>\n<li><strong>Apply fan-out thoughtfully<\/strong>: every additional subscription increases delivery volume and cost. Consider whether a downstream pipeline can branch internally.<\/li>\n<li><strong>Use DLQs for poison messages<\/strong>: configure dead-letter topics and build operational playbooks for DLQ remediation.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IAM\/security best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Least privilege<\/strong>:<\/li>\n<li>Publishers: <code>roles\/pubsub.publisher<\/code> on specific topics<\/li>\n<li>Subscribers: <code>roles\/pubsub.subscriber<\/code> on specific subscriptions<\/li>\n<li>Admin tasks: limited to platform team<\/li>\n<li><strong>Use separate service accounts<\/strong> per workload and environment.<\/li>\n<li><strong>Avoid user credentials in production<\/strong>; use service accounts with workload identity where appropriate.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Cost best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Control message size<\/strong>: store large payloads outside Pub\/Sub and publish references.<\/li>\n<li><strong>Tune retention<\/strong> to real recovery needs.<\/li>\n<li><strong>Prevent runaway retries<\/strong>: set sane retry\/DLQ policies and monitor error rates.<\/li>\n<li><strong>Use filtering<\/strong> to avoid delivering irrelevant events to consumers.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Performance best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Batch publishing<\/strong>: use client library batching settings to improve throughput.<\/li>\n<li><strong>Use flow control<\/strong> in subscribers: cap outstanding messages\/bytes to avoid memory pressure.<\/li>\n<li><strong>Parallelize subscribers<\/strong>: scale horizontally; ensure processing is stateless where possible.<\/li>\n<li><strong>Use ordering keys only when required<\/strong>: ordering can reduce effective parallelism per key.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Reliability best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Set SLOs<\/strong>: backlog size, oldest unacked message age, subscriber error rate.<\/li>\n<li><strong>Plan for consumer outages<\/strong>: retention should cover expected recovery times.<\/li>\n<li><strong>Test failover<\/strong>: intentionally stop subscribers and validate recovery and replay behavior.<\/li>\n<li><strong>Use DLQ + alerting<\/strong>: DLQ growth should page or create incidents.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operations best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Standardize naming<\/strong>:<\/li>\n<li><code>topic<\/code>: <code>{env}.{domain}.{event}<\/code> (example: <code>prod.orders.events<\/code>)<\/li>\n<li><code>subscription<\/code>: <code>{env}.{consumer}.{topic}<\/code> (example: <code>prod.analytics.orders.events.sub<\/code>)<\/li>\n<li><strong>Label resources<\/strong>: owner, cost center, environment, data classification.<\/li>\n<li><strong>Document contracts<\/strong>: schemas, attribute conventions, versioning strategy.<\/li>\n<li><strong>Automate provisioning<\/strong>: use Terraform or other IaC to manage topics\/subscriptions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Governance\/tagging\/naming best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use consistent labels:<\/li>\n<li><code>env=dev|stage|prod<\/code><\/li>\n<li><code>team=data-platform<\/code><\/li>\n<li><code>pii=false|true<\/code><\/li>\n<li><code>cost-center=1234<\/code><\/li>\n<li>Define a topic lifecycle process: creation, versioning, deprecation, migration.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">12. Security Considerations<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Identity and access model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub uses <strong>IAM<\/strong> for authorization.<\/li>\n<li>Common IAM patterns:<\/li>\n<li>Grant publish rights on a topic to producer service accounts only.<\/li>\n<li>Grant subscribe rights on a subscription to consumer service accounts only.<\/li>\n<li>Restrict topic\/subscription admin rights to platform\/security teams.<\/li>\n<\/ul>\n\n\n\n<p>Recommended roles (verify exact role names in IAM docs):\n&#8211; <code>roles\/pubsub.publisher<\/code>\n&#8211; <code>roles\/pubsub.subscriber<\/code>\n&#8211; <code>roles\/pubsub.viewer<\/code>\n&#8211; <code>roles\/pubsub.admin<\/code><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Encryption<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>In transit<\/strong>: Pub\/Sub uses TLS for API communication.<\/li>\n<li><strong>At rest<\/strong>: encrypted by default with Google-managed keys.<\/li>\n<li><strong>CMEK (Customer-managed encryption keys)<\/strong>: Pub\/Sub may support CMEK for certain resources\/configurations; availability can vary. Verify CMEK support and setup steps: https:\/\/cloud.google.com\/pubsub\/docs\/encryption<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network exposure<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub is accessed via Google APIs over HTTPS.<\/li>\n<li>For subscribers\/publishers in VPC environments:<\/li>\n<li>Use organization policies and egress controls as needed.<\/li>\n<li>Consider <strong>VPC Service Controls<\/strong> for data exfiltration risk reduction (verify compatibility and supported services).<\/li>\n<li>For push subscriptions:<\/li>\n<li>Endpoints must be internet-reachable (or reachable via appropriate networking) and must validate authentication tokens.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secrets handling<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Avoid embedding service account keys in code.<\/li>\n<li>Prefer:<\/li>\n<li>Workload Identity (GKE)<\/li>\n<li>Service account attached to Cloud Run \/ Compute Engine<\/li>\n<li>Short-lived credentials via ADC (Application Default Credentials)<\/li>\n<li>Use Secret Manager for application secrets unrelated to Pub\/Sub credentials.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Audit\/logging<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use Cloud Audit Logs to track:<\/li>\n<li>Topic\/subscription creation\/deletion<\/li>\n<li>IAM policy changes<\/li>\n<li>Schema changes (if used)<\/li>\n<li>Consider enabling and routing logs to a central logging project if you operate at scale.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compliance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>For regulated data, validate:<\/li>\n<li>Data residency\/location controls (message storage policies)<\/li>\n<li>Retention duration and deletion behavior<\/li>\n<li>Access boundaries (projects, folders, org policies, VPC-SC)<\/li>\n<li>Pub\/Sub is often part of a larger compliance story; ensure downstream systems also comply.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Common security mistakes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Granting <code>roles\/pubsub.admin<\/code> broadly to developers and workloads<\/li>\n<li>Using a single shared service account for many services<\/li>\n<li>Push endpoints without authentication\/authorization<\/li>\n<li>Publishing sensitive payloads without classification and retention controls<\/li>\n<li>No monitoring for DLQ growth (can hide attacks or data quality failures)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secure deployment recommendations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use separate projects for dev\/test\/prod.<\/li>\n<li>Use per-team topics only when boundaries matter; otherwise central platform-managed topics with strict IAM.<\/li>\n<li>Enforce schema validation for critical event domains.<\/li>\n<li>Use DLQs and alerts to detect abnormal failure patterns.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">13. Limitations and Gotchas<\/h2>\n\n\n\n<p>Always confirm details in official docs because limits and behaviors can change.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Known limitations \/ common constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Message size limits<\/strong> exist (payload + attributes). Verify current max size: https:\/\/cloud.google.com\/pubsub\/quotas<\/li>\n<li><strong>At-least-once delivery<\/strong> means duplicates are possible unless exactly-once delivery is enabled and used correctly (and even then, end-to-end exactly-once processing is not automatic).<\/li>\n<li><strong>Ordering<\/strong> is per ordering key; ordering across different keys is not guaranteed.<\/li>\n<li><strong>Retention is not archival<\/strong>: Pub\/Sub is not designed as a long-term event store.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Quotas exist for:<\/li>\n<li>publish rate, pull rate<\/li>\n<li>subscriptions per topic, topics per project<\/li>\n<li>outstanding messages\/bytes<\/li>\n<li>Review and request increases where needed: https:\/\/cloud.google.com\/pubsub\/quotas<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Regional constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Location\/residency features and CMEK may have constraints.<\/li>\n<li>Cross-region subscribers can incur egress costs and add latency.<\/li>\n<li>Verify Pub\/Sub location guidance: https:\/\/cloud.google.com\/pubsub\/docs\/locations<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing surprises<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fan-out multiplies delivery volume costs.<\/li>\n<li>Retention and replay can increase storage and delivery charges.<\/li>\n<li>Retries due to failing subscribers can significantly increase delivery volume.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compatibility issues<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Some features depend on client library versions (exactly-once delivery support, flow control behavior).<\/li>\n<li>Always pin and update client libraries carefully in production.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational gotchas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Poor ack handling causes redeliveries and cost spikes.<\/li>\n<li>Misconfigured push endpoints can cause repeated delivery attempts.<\/li>\n<li>DLQ without alerting can silently accumulate failures.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Migration challenges<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Migrating from Kafka\/RabbitMQ may require adapting message keying, ordering assumptions, and consumer group semantics.<\/li>\n<li>Pub\/Sub\u2019s model (topic + independent subscriptions) is different from Kafka partitions and consumer groups. Plan carefully.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Vendor-specific nuances<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub integrates deeply with Google Cloud IAM and monitoring; this is a benefit, but it also means you should design around Google Cloud operational patterns.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">14. Comparison with Alternatives<\/h2>\n\n\n\n<p>Pub\/Sub is one option in Google Cloud and among cloud providers. The best choice depends on delivery semantics, throughput, protocol needs, and operational constraints.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Comparison table<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Option<\/th>\n<th>Best For<\/th>\n<th>Strengths<\/th>\n<th>Weaknesses<\/th>\n<th>When to Choose<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>Pub\/Sub (Google Cloud)<\/strong><\/td>\n<td>Event ingestion, fan-out, streaming pipelines<\/td>\n<td>Fully managed, scalable, native Google Cloud integrations, DLQ\/filtering\/ordering options<\/td>\n<td>Not a long-term event store; duplicates possible; non-Kafka protocol<\/td>\n<td>Default choice for event-driven architectures and Data analytics and pipelines on Google Cloud<\/td>\n<\/tr>\n<tr>\n<td><strong>Pub\/Sub Lite (Google Cloud)<\/strong><\/td>\n<td>Cost-sensitive, very high-throughput streaming (where applicable)<\/td>\n<td>Lower-cost model for certain patterns; regional<\/td>\n<td>Different operational model and constraints vs Pub\/Sub; feature parity may differ<\/td>\n<td>Consider if your use case matches Lite\u2019s model and you\u2019ve verified current status and fit in official docs<\/td>\n<\/tr>\n<tr>\n<td><strong>Cloud Tasks (Google Cloud)<\/strong><\/td>\n<td>Task queueing, request scheduling<\/td>\n<td>Good for HTTP task execution, retries, scheduling<\/td>\n<td>Not a pub\/sub fan-out event bus<\/td>\n<td>When you need task execution semantics rather than event streaming<\/td>\n<\/tr>\n<tr>\n<td><strong>Eventarc (Google Cloud)<\/strong><\/td>\n<td>Event routing from Google Cloud sources to services<\/td>\n<td>Simplifies event triggers to Cloud Run; integrates with Google Cloud events<\/td>\n<td>Not a general-purpose high-throughput ingestion layer by itself<\/td>\n<td>When you want managed event routing\/triggers rather than building consumer plumbing<\/td>\n<\/tr>\n<tr>\n<td><strong>Kafka (self-managed on GKE\/Compute Engine)<\/strong><\/td>\n<td>Kafka protocol, long retention, stream replay<\/td>\n<td>Strong ecosystem, partitions, consumer groups, long retention<\/td>\n<td>Operational overhead, scaling\/patching complexity<\/td>\n<td>When you require Kafka compatibility, long replay windows, or existing Kafka tooling<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS SNS + SQS<\/strong><\/td>\n<td>Pub\/sub + queueing on AWS<\/td>\n<td>Mature services; flexible patterns<\/td>\n<td>Different ecosystem; multi-service composition<\/td>\n<td>When operating primarily on AWS<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS Kinesis<\/strong><\/td>\n<td>High-throughput streaming on AWS<\/td>\n<td>Tight integration with AWS analytics<\/td>\n<td>Different API model and cost structure<\/td>\n<td>When on AWS and needing managed streaming<\/td>\n<\/tr>\n<tr>\n<td><strong>Azure Event Hubs \/ Service Bus<\/strong><\/td>\n<td>Messaging\/streaming on Azure<\/td>\n<td>Strong Azure integration<\/td>\n<td>Different semantics<\/td>\n<td>When operating primarily on Azure<\/td>\n<\/tr>\n<tr>\n<td><strong>RabbitMQ (self-managed\/managed elsewhere)<\/strong><\/td>\n<td>Traditional messaging, routing patterns<\/td>\n<td>Flexible routing, familiar AMQP<\/td>\n<td>Operational overhead; scaling limits for very high throughput<\/td>\n<td>When you need AMQP\/routing semantics and accept operational tradeoffs<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">15. Real-World Example<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Enterprise example: Streaming analytics for a retail platform<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: A large retailer needs near-real-time analytics of purchases and browsing behavior, plus multiple downstream consumers (fraud, recommendations, BI dashboards). They want reliability, replay within a limited window, and strong IAM separation between teams.<\/li>\n<li><strong>Proposed architecture<\/strong>:<\/li>\n<li>Microservices publish events to domain topics (<code>orders-events<\/code>, <code>clickstream-events<\/code>)<\/li>\n<li>Subscription filtering routes subsets (e.g., only purchases) to a Dataflow pipeline<\/li>\n<li>Dataflow validates schemas, enriches events, writes to BigQuery<\/li>\n<li>Separate subscriptions feed fraud detection services and operational monitoring<\/li>\n<li>Dead-letter topics capture poison messages; remediation pipeline stores failed payloads to Cloud Storage with incident tickets<\/li>\n<li><strong>Why Pub\/Sub was chosen<\/strong>:<\/li>\n<li>Managed ingestion and fan-out at scale<\/li>\n<li>Tight Google Cloud integration (Dataflow + BigQuery)<\/li>\n<li>Independent subscriptions per team with IAM boundaries<\/li>\n<li>DLQs and retention for operational safety<\/li>\n<li><strong>Expected outcomes<\/strong>:<\/li>\n<li>Reduced coupling and faster onboarding of new consumers<\/li>\n<li>Near-real-time dashboards in BigQuery<\/li>\n<li>Improved reliability under traffic spikes<\/li>\n<li>Clear operational model for failures (DLQ + alerts)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Startup\/small-team example: Webhook processing for a SaaS product<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: A small SaaS team receives webhooks from payment and CRM providers. Webhooks spike during billing cycles. They need to process events reliably without timing out the webhook sender and without running message brokers.<\/li>\n<li><strong>Proposed architecture<\/strong>:<\/li>\n<li>Cloud Run service receives webhook, verifies signature, publishes normalized event to Pub\/Sub<\/li>\n<li>Cloud Run worker(s) subscribe (push or pull) and process events asynchronously<\/li>\n<li>DLQ captures events that fail repeatedly; team reviews and replays<\/li>\n<li><strong>Why Pub\/Sub was chosen<\/strong>:<\/li>\n<li>Minimal ops effort<\/li>\n<li>Scales automatically during spikes<\/li>\n<li>Clean separation between webhook intake and background processing<\/li>\n<li><strong>Expected outcomes<\/strong>:<\/li>\n<li>Fewer webhook failures\/timeouts<\/li>\n<li>Better reliability during peak spikes<\/li>\n<li>Simple path to add analytics subscription later (fan-out)<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">16. FAQ<\/h2>\n\n\n\n<p>1) <strong>Is Pub\/Sub the same as Kafka?<\/strong><br\/>\nNo. Pub\/Sub is a managed messaging service with topics and independent subscriptions; Kafka is a distributed log with partitions and consumer groups. They solve similar streaming problems but have different operational models and semantics.<\/p>\n\n\n\n<p>2) <strong>Does Pub\/Sub guarantee exactly-once processing?<\/strong><br\/>\nNot automatically. Pub\/Sub can provide exactly-once delivery in supported scenarios, but end-to-end exactly-once processing still requires idempotent design, transactional sinks, or de-duplication strategies. Verify exactly-once delivery details: https:\/\/cloud.google.com\/pubsub\/docs\/exactly-once-delivery<\/p>\n\n\n\n<p>3) <strong>What\u2019s the difference between a topic and a subscription?<\/strong><br\/>\nA topic is where messages are published. A subscription is a delivery configuration that receives messages from a topic and tracks ack state independently.<\/p>\n\n\n\n<p>4) <strong>Can multiple subscribers read from the same subscription?<\/strong><br\/>\nYes. Multiple subscriber instances can share a subscription to scale out processing (competing consumers). Each message is delivered to one of the subscribers for that subscription.<\/p>\n\n\n\n<p>5) <strong>How do I broadcast the same message to multiple systems?<\/strong><br\/>\nCreate multiple subscriptions on the same topic (fan-out). Each subscription receives a copy of the message stream.<\/p>\n\n\n\n<p>6) <strong>What happens if my subscriber crashes?<\/strong><br\/>\nUnacked messages become eligible for redelivery after the ack deadline. Your application should be idempotent and able to process duplicates.<\/p>\n\n\n\n<p>7) <strong>How do push subscriptions work?<\/strong><br\/>\nPub\/Sub sends HTTPS POST requests to your endpoint. Your service must return appropriate success responses and handle retries. Secure the endpoint using authentication (OIDC token) per docs: https:\/\/cloud.google.com\/pubsub\/docs\/push<\/p>\n\n\n\n<p>8) <strong>Should I use push or pull?<\/strong><br\/>\nUse pull for worker pools, fine-grained flow control, and many data processing pipelines. Use push for HTTP-based event handling (e.g., Cloud Run) where you want simpler delivery and don\u2019t want polling logic.<\/p>\n\n\n\n<p>9) <strong>How do I handle poison messages?<\/strong><br\/>\nUse dead-letter topics and alert on DLQ growth. Also store failing payloads and add tools\/workflows to replay after fixing bugs.<\/p>\n\n\n\n<p>10) <strong>Can I filter messages so a subscription only receives some events?<\/strong><br\/>\nYes, with subscription filtering based on message attributes. Ensure producers publish consistent attributes.<\/p>\n\n\n\n<p>11) <strong>Where should I put large payloads?<\/strong><br\/>\nPrefer storing large data in Cloud Storage (or another datastore) and publishing a reference (object path, ID) to Pub\/Sub. This reduces messaging costs and avoids size limits.<\/p>\n\n\n\n<p>12) <strong>How do I replay messages?<\/strong><br\/>\nUse retention plus seek-to-timestamp or snapshots to reset subscription state. Replay is bounded by retention. Verify replay\/seek docs: https:\/\/cloud.google.com\/pubsub\/docs\/replay-overview (verify exact URL in docs if it changed)<\/p>\n\n\n\n<p>13) <strong>How do I monitor subscriber lag?<\/strong><br\/>\nTrack backlog metrics (undelivered\/unacked messages) and oldest unacked message age in Cloud Monitoring. Create alerts for sustained growth.<\/p>\n\n\n\n<p>14) <strong>Is Pub\/Sub regional or global?<\/strong><br\/>\nPub\/Sub resources are managed by Google Cloud and are not tied to a specific VM zone. Data residency and location controls exist (message storage policy). Verify current location semantics: https:\/\/cloud.google.com\/pubsub\/docs\/locations<\/p>\n\n\n\n<p>15) <strong>What\u2019s a good retention period?<\/strong><br\/>\nLong enough to recover from expected outages and deploy rollbacks (often hours to a couple days), but not so long that cost and operational replay risk become high. Choose based on your incident recovery objectives.<\/p>\n\n\n\n<p>16) <strong>Can I use Pub\/Sub for request\/response RPC?<\/strong><br\/>\nIt\u2019s possible but usually not ideal. Pub\/Sub is optimized for asynchronous messaging. For synchronous request\/response, use HTTP\/gRPC directly.<\/p>\n\n\n\n<p>17) <strong>How do I secure access between teams?<\/strong><br\/>\nUse separate topics\/subscriptions by domain or environment and grant IAM roles only to required service accounts. Consider separate projects for stronger isolation.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">17. Top Online Resources to Learn Pub\/Sub<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Resource Type<\/th>\n<th>Name<\/th>\n<th>Why It Is Useful<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Official documentation<\/td>\n<td>Pub\/Sub Overview \u2014 https:\/\/cloud.google.com\/pubsub\/docs\/overview<\/td>\n<td>Authoritative description of concepts, features, and core behavior<\/td>\n<\/tr>\n<tr>\n<td>Official quickstart<\/td>\n<td>Pub\/Sub client library quickstarts \u2014 https:\/\/cloud.google.com\/pubsub\/docs\/quickstarts<\/td>\n<td>Step-by-step getting started for multiple languages<\/td>\n<\/tr>\n<tr>\n<td>Official pricing<\/td>\n<td>Pub\/Sub pricing \u2014 https:\/\/cloud.google.com\/pubsub\/pricing<\/td>\n<td>Current pricing dimensions and rates (don\u2019t rely on third-party summaries)<\/td>\n<\/tr>\n<tr>\n<td>Pricing calculator<\/td>\n<td>Google Cloud Pricing Calculator \u2014 https:\/\/cloud.google.com\/products\/calculator<\/td>\n<td>Estimate costs for different volumes, regions, and architectures<\/td>\n<\/tr>\n<tr>\n<td>Architecture guidance<\/td>\n<td>Cloud Architecture Center (search event-driven \/ streaming) \u2014 https:\/\/cloud.google.com\/architecture<\/td>\n<td>Reference architectures and best practices for event-driven systems<\/td>\n<\/tr>\n<tr>\n<td>Dead-letter topics<\/td>\n<td>DLQ docs \u2014 https:\/\/cloud.google.com\/pubsub\/docs\/dead-letter-topics<\/td>\n<td>Required IAM, configuration details, and operational guidance<\/td>\n<\/tr>\n<tr>\n<td>Ordering<\/td>\n<td>Message ordering \u2014 https:\/\/cloud.google.com\/pubsub\/docs\/ordering<\/td>\n<td>Correct setup and constraints for ordering keys<\/td>\n<\/tr>\n<tr>\n<td>Exactly-once delivery<\/td>\n<td>Exactly-once delivery \u2014 https:\/\/cloud.google.com\/pubsub\/docs\/exactly-once-delivery<\/td>\n<td>How it works, limitations, and supported clients<\/td>\n<\/tr>\n<tr>\n<td>Subscriber guidance<\/td>\n<td>Subscriber overview \u2014 https:\/\/cloud.google.com\/pubsub\/docs\/subscriber<\/td>\n<td>Ack deadlines, retries, flow control, and client behavior<\/td>\n<\/tr>\n<tr>\n<td>GitHub samples (official)<\/td>\n<td>GoogleCloudPlatform Pub\/Sub samples \u2014 https:\/\/github.com\/GoogleCloudPlatform<\/td>\n<td>Practical code examples (search within org for pubsub client samples)<\/td>\n<\/tr>\n<tr>\n<td>Video (official)<\/td>\n<td>Google Cloud Tech YouTube \u2014 https:\/\/www.youtube.com\/@googlecloudtech<\/td>\n<td>Product explainers and architecture talks (search for \u201cPub\/Sub\u201d)<\/td>\n<\/tr>\n<tr>\n<td>Hands-on labs<\/td>\n<td>Google Cloud Skills Boost (search Pub\/Sub labs) \u2014 https:\/\/www.cloudskillsboost.google\/<\/td>\n<td>Guided labs with temporary projects and step-by-step instructions<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">18. Training and Certification Providers<\/h2>\n\n\n\n<p>Below are training providers as requested. Verify course outlines, delivery modes, and schedules on their websites.<\/p>\n\n\n\n<p>1) <strong>DevOpsSchool.com<\/strong><br\/>\n&#8211; <strong>Suitable audience<\/strong>: DevOps engineers, SREs, cloud engineers, developers<br\/>\n&#8211; <strong>Likely learning focus<\/strong>: DevOps\/cloud automation; may include Google Cloud messaging and pipelines in broader tracks<br\/>\n&#8211; <strong>Mode<\/strong>: check website<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.devopsschool.com\/<\/p>\n\n\n\n<p>2) <strong>ScmGalaxy.com<\/strong><br\/>\n&#8211; <strong>Suitable audience<\/strong>: beginners to intermediate IT professionals<br\/>\n&#8211; <strong>Likely learning focus<\/strong>: software configuration management and DevOps fundamentals; may offer cloud-related modules<br\/>\n&#8211; <strong>Mode<\/strong>: check website<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.scmgalaxy.com\/<\/p>\n\n\n\n<p>3) <strong>CLoudOpsNow.in<\/strong><br\/>\n&#8211; <strong>Suitable audience<\/strong>: cloud operations and platform teams<br\/>\n&#8211; <strong>Likely learning focus<\/strong>: cloud ops practices, reliability, monitoring, and operational runbooks<br\/>\n&#8211; <strong>Mode<\/strong>: check website<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.cloudopsnow.in\/<\/p>\n\n\n\n<p>4) <strong>SreSchool.com<\/strong><br\/>\n&#8211; <strong>Suitable audience<\/strong>: SREs, production engineers, operations teams<br\/>\n&#8211; <strong>Likely learning focus<\/strong>: SRE practices, observability, incident response, reliability architecture<br\/>\n&#8211; <strong>Mode<\/strong>: check website<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.sreschool.com\/<\/p>\n\n\n\n<p>5) <strong>AiOpsSchool.com<\/strong><br\/>\n&#8211; <strong>Suitable audience<\/strong>: operations teams adopting AIOps practices<br\/>\n&#8211; <strong>Likely learning focus<\/strong>: AIOps concepts, monitoring automation, incident analytics<br\/>\n&#8211; <strong>Mode<\/strong>: check website<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.aiopsschool.com\/<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">19. Top Trainers<\/h2>\n\n\n\n<p>These are trainer-related sites\/platforms as requested. Verify specific Pub\/Sub or Google Cloud coverage on each site.<\/p>\n\n\n\n<p>1) <strong>RajeshKumar.xyz<\/strong><br\/>\n&#8211; <strong>Likely specialization<\/strong>: DevOps\/cloud training content (verify on site)<br\/>\n&#8211; <strong>Suitable audience<\/strong>: engineers seeking practical training<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/rajeshkumar.xyz\/<\/p>\n\n\n\n<p>2) <strong>devopstrainer.in<\/strong><br\/>\n&#8211; <strong>Likely specialization<\/strong>: DevOps and cloud training<br\/>\n&#8211; <strong>Suitable audience<\/strong>: beginners to intermediate DevOps\/cloud learners<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.devopstrainer.in\/<\/p>\n\n\n\n<p>3) <strong>devopsfreelancer.com<\/strong><br\/>\n&#8211; <strong>Likely specialization<\/strong>: DevOps consulting\/training resources (verify offerings)<br\/>\n&#8211; <strong>Suitable audience<\/strong>: teams\/individuals looking for hands-on guidance<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.devopsfreelancer.com\/<\/p>\n\n\n\n<p>4) <strong>devopssupport.in<\/strong><br\/>\n&#8211; <strong>Likely specialization<\/strong>: DevOps support and training resources<br\/>\n&#8211; <strong>Suitable audience<\/strong>: operations teams and engineers needing implementation support<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.devopssupport.in\/<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">20. Top Consulting Companies<\/h2>\n\n\n\n<p>These consulting companies are listed as requested. Descriptions below are kept generic; verify exact offerings directly with each firm.<\/p>\n\n\n\n<p>1) <strong>cotocus.com<\/strong><br\/>\n&#8211; <strong>Likely service area<\/strong>: cloud\/DevOps consulting (verify service catalog)<br\/>\n&#8211; <strong>Where they may help<\/strong>: architecture reviews, implementation support, operationalization<br\/>\n&#8211; <strong>Consulting use case examples<\/strong>:\n  &#8211; Designing an event-driven architecture with Pub\/Sub topics\/subscriptions and IAM boundaries<br\/>\n  &#8211; Implementing DLQ strategy and monitoring\/alerting dashboards<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/cotocus.com\/<\/p>\n\n\n\n<p>2) <strong>DevOpsSchool.com<\/strong><br\/>\n&#8211; <strong>Likely service area<\/strong>: DevOps and cloud consulting\/training<br\/>\n&#8211; <strong>Where they may help<\/strong>: platform enablement, CI\/CD, cloud best practices, skills development<br\/>\n&#8211; <strong>Consulting use case examples<\/strong>:\n  &#8211; Building a reference Pub\/Sub-based ingestion layer for data analytics and pipelines<br\/>\n  &#8211; Creating Terraform modules for topics\/subscriptions with standardized naming\/labels<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.devopsschool.com\/<\/p>\n\n\n\n<p>3) <strong>DEVOPSCONSULTING.IN<\/strong><br\/>\n&#8211; <strong>Likely service area<\/strong>: DevOps and cloud consulting<br\/>\n&#8211; <strong>Where they may help<\/strong>: implementation and operational support for cloud platforms<br\/>\n&#8211; <strong>Consulting use case examples<\/strong>:\n  &#8211; Migrating event workloads to Google Cloud Pub\/Sub with monitoring and cost controls<br\/>\n  &#8211; Designing subscriber scaling and reliability patterns (ack handling, retries, DLQ)<br\/>\n&#8211; <strong>Website<\/strong>: https:\/\/www.devopsconsulting.in\/<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">21. Career and Learning Roadmap<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn before Pub\/Sub<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Google Cloud fundamentals: projects, IAM, billing, APIs<\/li>\n<li>Basic networking and identity concepts: service accounts, OAuth, least privilege<\/li>\n<li>Event-driven architecture basics: async messaging, retries, idempotency<\/li>\n<li>Data analytics and pipelines fundamentals: streaming vs batch, ETL\/ELT, schema evolution<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn after Pub\/Sub<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Dataflow (Apache Beam)<\/strong> for streaming transformations and windowing<\/li>\n<li><strong>BigQuery<\/strong> for analytics, partitioning, and streaming ingestion patterns<\/li>\n<li><strong>Cloud Run<\/strong> for event-driven compute<\/li>\n<li><strong>Cloud Monitoring\/Logging<\/strong> for SLOs and alerting on pipeline health<\/li>\n<li><strong>Terraform<\/strong> for IaC management of topics\/subscriptions and IAM<\/li>\n<li><strong>Security<\/strong>: VPC Service Controls, org policies, audit logging strategies<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Job roles that use Pub\/Sub<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud engineer \/ platform engineer<\/li>\n<li>Data engineer \/ analytics engineer<\/li>\n<li>Backend engineer (microservices)<\/li>\n<li>DevOps engineer \/ SRE<\/li>\n<li>Security engineer (event-driven detections, audit pipelines)<\/li>\n<li>Solutions architect<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Certification path (Google Cloud)<\/h3>\n\n\n\n<p>Google Cloud certifications change over time. Pub\/Sub knowledge is relevant for:\n&#8211; Associate Cloud Engineer (foundational services and operations)\n&#8211; Professional Cloud Architect (architecture decisions and tradeoffs)\n&#8211; Professional Data Engineer (streaming pipelines with Pub\/Sub\/Dataflow\/BigQuery)<\/p>\n\n\n\n<p>Verify current Google Cloud certification tracks: https:\/\/cloud.google.com\/learn\/certification<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Project ideas for practice<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Build a clickstream ingestion pipeline: Pub\/Sub \u2192 Dataflow \u2192 BigQuery with dashboards.<\/li>\n<li>Implement an event-driven order workflow with 3 services consuming different subscriptions.<\/li>\n<li>Add schema validation and versioning to an event domain.<\/li>\n<li>Implement DLQ remediation tooling: DLQ \u2192 Cloud Run job to reprocess after fix.<\/li>\n<li>Add Monitoring alerts for backlog growth and DLQ spikes; define SLOs.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">22. Glossary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Ack (Acknowledgment)<\/strong>: A confirmation from subscriber to Pub\/Sub that a message was processed successfully.<\/li>\n<li><strong>Ack deadline<\/strong>: The time allowed for a subscriber to ack a message before it becomes eligible for redelivery.<\/li>\n<li><strong>At-least-once delivery<\/strong>: Delivery guarantee where messages may be delivered multiple times; consumers must handle duplicates.<\/li>\n<li><strong>Dead-letter topic (DLQ)<\/strong>: A topic where messages are sent after repeated delivery failures.<\/li>\n<li><strong>Fan-out<\/strong>: Pattern where one published message is delivered to multiple independent consumers via multiple subscriptions.<\/li>\n<li><strong>Filtering<\/strong>: Subscription feature to only receive messages matching attribute-based expressions.<\/li>\n<li><strong>Message attributes<\/strong>: Key\/value metadata attached to a message, often used for routing\/filtering.<\/li>\n<li><strong>Ordering key<\/strong>: A key used to preserve the order of messages within that key\u2019s stream (when ordering is enabled).<\/li>\n<li><strong>Publisher<\/strong>: Client that sends messages to a Pub\/Sub topic.<\/li>\n<li><strong>Pull subscription<\/strong>: Subscriber pulls messages from Pub\/Sub and controls flow\/ack.<\/li>\n<li><strong>Push subscription<\/strong>: Pub\/Sub pushes messages to an HTTPS endpoint.<\/li>\n<li><strong>Retention<\/strong>: How long Pub\/Sub stores messages for delivery\/replay.<\/li>\n<li><strong>Schema<\/strong>: A formal definition of message structure (e.g., Avro\/Protobuf) used for validation and compatibility.<\/li>\n<li><strong>Seek<\/strong>: Reset a subscription to a timestamp or snapshot to replay messages.<\/li>\n<li><strong>Snapshot<\/strong>: A saved point-in-time cursor\/state of a subscription for replay.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">23. Summary<\/h2>\n\n\n\n<p>Pub\/Sub is Google Cloud\u2019s managed publish\/subscribe messaging service and a foundational component for Data analytics and pipelines and event-driven architectures. It provides durable ingestion, scalable fan-out, and operational features like retries, dead-letter topics, retention-based replay, filtering, and integrations with services such as Dataflow and Cloud Run.<\/p>\n\n\n\n<p>Cost and security require intentional design:\n&#8211; <strong>Cost<\/strong> is driven by data volume, fan-out (deliveries per subscription), retention, retries, and network egress. Use filtering, control payload size, and monitor retry\/DLQ rates.\n&#8211; <strong>Security<\/strong> relies on IAM least privilege, service accounts, encryption controls (including CMEK where applicable), and audit logging.<\/p>\n\n\n\n<p>Use Pub\/Sub when you need decoupled, scalable event distribution and ingestion on Google Cloud. Pair it with Dataflow\/BigQuery for streaming analytics, or with Cloud Run for event-driven services. Next step: build a small streaming pipeline (Pub\/Sub \u2192 Dataflow \u2192 BigQuery) and add production-grade monitoring and DLQ remediation playbooks using the official docs: https:\/\/cloud.google.com\/pubsub\/docs\/overview<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data analytics and pipelines<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[59,51],"tags":[],"class_list":["post-667","post","type-post","status-publish","format-standard","hentry","category-data-analytics-and-pipelines","category-google-cloud"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=667"}],"version-history":[{"count":0,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/667\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}