{"id":666,"date":"2026-04-14T23:00:15","date_gmt":"2026-04-14T23:00:15","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/google-cloud-manufacturing-data-engine-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-data-analytics-and-pipelines\/"},"modified":"2026-04-14T23:00:15","modified_gmt":"2026-04-14T23:00:15","slug":"google-cloud-manufacturing-data-engine-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-data-analytics-and-pipelines","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/google-cloud-manufacturing-data-engine-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-data-analytics-and-pipelines\/","title":{"rendered":"Google Cloud Manufacturing Data Engine Tutorial: Architecture, Pricing, Use Cases, and Hands-On Guide for Data analytics and pipelines"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Category<\/h2>\n\n\n\n<p>Data analytics and pipelines<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. Introduction<\/h2>\n\n\n\n<p>Manufacturing Data Engine is a Google Cloud offering aimed at helping manufacturers unify, contextualize, and analyze manufacturing data across operational technology (OT) and information technology (IT) systems so it can be used reliably for analytics, reporting, and AI\/ML.<\/p>\n\n\n\n<p>In simple terms: Manufacturing Data Engine helps you take messy, siloed factory data (machines, sensors, production lines, quality systems, ERP\/MES) and turn it into trusted, analysis-ready datasets that teams can query and build dashboards and models on.<\/p>\n\n\n\n<p>In technical terms: Manufacturing Data Engine is best understood as a manufacturing-focused data foundation and solution pattern on Google Cloud that typically uses core \u201cData analytics and pipelines\u201d services (for ingestion, transformation, governance, and analytics) to standardize manufacturing events and time-series telemetry, enrich it with asset\/production context, and publish it to analytics systems (commonly BigQuery + Looker) while enforcing access controls, lineage, and operational monitoring. <strong>The exact packaging, components, and availability can vary\u2014verify the current official documentation for your organization\u2019s edition\/rollout status.<\/strong><\/p>\n\n\n\n<p>What problem it solves:\n&#8211; Manufacturing data is often fragmented (PLC\/SCADA historians, MES, QMS, ERP, spreadsheets).\n&#8211; Data lacks consistent identifiers and context (asset hierarchy, shift, work order, product).\n&#8211; Pipelines are brittle, hard to govern, and expensive to operate at scale.\n&#8211; Analytics and AI initiatives stall due to data quality, latency, and access challenges.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">2. What is Manufacturing Data Engine?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Official purpose (what Google Cloud positions it for)<\/h3>\n\n\n\n<p>Manufacturing Data Engine is positioned by Google Cloud as a manufacturing-oriented data capability that helps manufacturers <strong>organize, contextualize, and operationalize<\/strong> manufacturing data for analytics and downstream applications. It focuses on making manufacturing data usable across teams (operations, quality, engineering, data\/ML) by creating standardized, governed datasets.<\/p>\n\n\n\n<p><strong>Important:<\/strong> Google Cloud\u2019s manufacturing offerings can be delivered as a combination of services, solution templates, partner integrations, and reference architectures. If your organization is evaluating this service, confirm the current scope and GA\/preview status in official Google Cloud materials.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Core capabilities (conceptual, implementation-oriented)<\/h3>\n\n\n\n<p>In Manufacturing Data Engine-style implementations, the core capabilities typically include:\n&#8211; <strong>Ingestion<\/strong> of streaming telemetry and events from plant systems (directly or via gateways\/partners).\n&#8211; <strong>Data harmonization<\/strong> into consistent schemas and identifiers.\n&#8211; <strong>Contextualization<\/strong> with asset hierarchy, production orders, shift calendars, and product definitions.\n&#8211; <strong>Storage and analytics<\/strong> in a query-optimized repository (commonly BigQuery).\n&#8211; <strong>Governance<\/strong> (metadata, lineage, access control) across datasets and domains.\n&#8211; <strong>Consumption<\/strong> via dashboards (Looker\/Looker Studio), APIs, and ML pipelines (Vertex AI).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Major components (how it shows up in a Google Cloud stack)<\/h3>\n\n\n\n<p>Because Manufacturing Data Engine is closely tied to Google Cloud\u2019s broader data platform, the \u201ccomponents\u201d you\u2019ll commonly see around it are:\n&#8211; <strong>Ingestion &amp; messaging:<\/strong> Pub\/Sub\n&#8211; <strong>Stream\/batch processing:<\/strong> Dataflow, Dataproc (Spark), BigQuery SQL\n&#8211; <strong>Landing zones:<\/strong> Cloud Storage (raw files), BigQuery (curated datasets)\n&#8211; <strong>Governance:<\/strong> Dataplex, Data Catalog (capabilities vary by product evolution\u2014verify in official docs)\n&#8211; <strong>Orchestration:<\/strong> Cloud Composer (Airflow), Workflows\n&#8211; <strong>Analytics &amp; BI:<\/strong> BigQuery + Looker \/ Looker Studio\n&#8211; <strong>ML:<\/strong> Vertex AI\n&#8211; <strong>Security:<\/strong> IAM, Cloud KMS, VPC Service Controls (where applicable)\n&#8211; <strong>Ops:<\/strong> Cloud Logging, Cloud Monitoring, Error Reporting<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Service type<\/h3>\n\n\n\n<p>Manufacturing Data Engine is best treated as an <strong>industry solution\/data foundation<\/strong> rather than a single primitive compute service (like a VM) or a single database. Practically, you implement it by composing Google Cloud \u201cData analytics and pipelines\u201d services and (where available) any official manufacturing-specific accelerators\/templates.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Scope: regional vs global, project-scoped vs account-scoped<\/h3>\n\n\n\n<p>The scope depends on the underlying services you deploy:\n&#8211; <strong>Project-scoped resources:<\/strong> Pub\/Sub topics, Dataflow jobs, BigQuery datasets, service accounts.\n&#8211; <strong>Regional considerations:<\/strong> Dataflow jobs are regional; Pub\/Sub and BigQuery are multi-regional\/regional depending on configuration.\n&#8211; <strong>Data residency:<\/strong> Determined by BigQuery dataset location, Cloud Storage bucket region, and Dataflow region.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How it fits into the Google Cloud ecosystem<\/h3>\n\n\n\n<p>Manufacturing Data Engine fits as a manufacturing-oriented layer on top of Google Cloud\u2019s data platform:\n&#8211; It uses <strong>Google Cloud\u2019s data analytics and pipelines<\/strong> building blocks to create a repeatable, governed manufacturing data pipeline.\n&#8211; It integrates naturally with <strong>BigQuery<\/strong> for analytics, <strong>Looker<\/strong> for BI, and <strong>Vertex AI<\/strong> for predictive quality\/maintenance and process optimization.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">3. Why use Manufacturing Data Engine?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Business reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Faster time to insight:<\/strong> Reduce the time it takes to get from factory signals to dashboards and decisions.<\/li>\n<li><strong>Reduced integration cost:<\/strong> Consolidate point-to-point OT\/IT integrations into a governed data foundation.<\/li>\n<li><strong>Cross-site standardization:<\/strong> Apply consistent data models across plants, lines, and equipment types.<\/li>\n<li><strong>Enable AI programs:<\/strong> Predictive maintenance, yield optimization, anomaly detection, and quality prediction depend on clean, contextual data.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Technical reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Streaming + batch support:<\/strong> Manufacturing requires both (sensor telemetry is streaming; ERP\/MES often arrives in batches).<\/li>\n<li><strong>Separation of raw vs curated:<\/strong> Preserve raw signals while creating trusted curated datasets for business use.<\/li>\n<li><strong>Scale-out analytics:<\/strong> BigQuery and Dataflow patterns are proven at high throughput when designed correctly.<\/li>\n<li><strong>Schema evolution &amp; data contracts:<\/strong> Better handling of changing machine signals and vendor formats.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Observability:<\/strong> Standard Google Cloud monitoring\/logging across ingestion and processing.<\/li>\n<li><strong>Automation:<\/strong> Infrastructure-as-code, CI\/CD for pipelines, repeatable deployments.<\/li>\n<li><strong>Incident response:<\/strong> Backlog metrics, DLQs (dead-letter queues), and reprocessing strategies.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/compliance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Least-privilege IAM:<\/strong> Fine-grained access controls on datasets and pipelines.<\/li>\n<li><strong>Auditability:<\/strong> Cloud Audit Logs + lineage\/metadata practices.<\/li>\n<li><strong>Encryption:<\/strong> Default encryption at rest and in transit; customer-managed keys (CMK) where required.<\/li>\n<li><strong>Segmentation:<\/strong> Private networking patterns and VPC Service Controls (verify applicability per service).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scalability\/performance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Burst handling:<\/strong> Pub\/Sub decouples producers from consumers.<\/li>\n<li><strong>Parallel processing:<\/strong> Dataflow scales horizontally for throughput and windowing\/aggregation.<\/li>\n<li><strong>Columnar analytics:<\/strong> BigQuery is well-suited for high-volume time-series-like manufacturing event analytics when modeled properly.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should choose it<\/h3>\n\n\n\n<p>Choose Manufacturing Data Engine patterns when you need:\n&#8211; A <strong>repeatable manufacturing data pipeline<\/strong> across plants.\n&#8211; <strong>Near-real-time analytics<\/strong> (seconds to minutes) for OEE, downtime, scrap monitoring, alerts.\n&#8211; <strong>Governed self-service data<\/strong> for analysts, quality teams, and data science.\n&#8211; A platform to support <strong>predictive maintenance\/quality<\/strong> initiatives.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should not choose it<\/h3>\n\n\n\n<p>Avoid (or delay) Manufacturing Data Engine-style investment if:\n&#8211; You only need a <strong>single machine dashboard<\/strong> and can solve it with an edge historian alone.\n&#8211; You lack ownership for <strong>data governance and data modeling<\/strong> (you\u2019ll build pipelines but won\u2019t create trusted datasets).\n&#8211; Your regulatory\/data residency requirements cannot be met with your chosen regions\/services (verify early).\n&#8211; You\u2019re not ready to operate streaming systems (start with batch, then evolve).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">4. Where is Manufacturing Data Engine used?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Industries<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Discrete manufacturing (automotive, electronics, aerospace)<\/li>\n<li>Process manufacturing (chemicals, food &amp; beverage, pharmaceuticals\u2014compliance requirements are higher)<\/li>\n<li>Industrial equipment and heavy manufacturing<\/li>\n<li>Contract manufacturing and multi-plant enterprises<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Team types<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data engineering and platform teams<\/li>\n<li>Manufacturing IT\/OT integration teams<\/li>\n<li>Quality engineering and process engineering<\/li>\n<li>Reliability engineering \/ maintenance teams<\/li>\n<li>BI teams and plant operations leadership<\/li>\n<li>Data science \/ ML engineering teams<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Workloads<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Streaming telemetry ingestion and transformation<\/li>\n<li>Event correlation (downtime reason + machine state + order context)<\/li>\n<li>Batch ingestion from MES\/ERP\/QMS<\/li>\n<li>Curated analytics datasets and semantic layers<\/li>\n<li>ML feature pipelines for predictive maintenance and quality<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Architectures<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Centralized \u201chub-and-spoke\u201d data platform across plants<\/li>\n<li>Federated domain-based data products (data mesh-style) with shared governance<\/li>\n<li>Edge-to-cloud ingestion with buffering and replay (often via gateways\/partners)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Real-world deployment contexts<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Plants with inconsistent equipment vendors and protocols<\/li>\n<li>Multi-site rollouts needing standardized KPIs<\/li>\n<li>Mergers\/acquisitions where data consolidation is a priority<\/li>\n<li>Brownfield factories modernizing gradually<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Production vs dev\/test usage<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Dev\/test:<\/strong> smaller throughput, synthetic sensor generators, sampled data, short-running Dataflow jobs, limited retention.<\/li>\n<li><strong>Production:<\/strong> 24\/7 streaming, DLQs, replay pipelines, HA design for critical KPIs, strict IAM, retention and cost controls.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">5. Top Use Cases and Scenarios<\/h2>\n\n\n\n<p>Below are realistic scenarios where Manufacturing Data Engine patterns are commonly applied.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1) Near-real-time OEE dashboards<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> OEE requires high-frequency machine state + production counts + downtime categorization, usually spread across systems.<\/li>\n<li><strong>Why this fits:<\/strong> Streaming ingestion + contextualization into curated tables supports minute-level OEE.<\/li>\n<li><strong>Example:<\/strong> Pub\/Sub ingests machine states; Dataflow enriches with asset hierarchy; BigQuery powers Looker OEE dashboards by line\/shift.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2) Downtime root-cause correlation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Downtime codes in MES don\u2019t match actual machine sensor patterns; investigations take days.<\/li>\n<li><strong>Why this fits:<\/strong> Unified timeline of events across OT telemetry + MES events enables correlation.<\/li>\n<li><strong>Example:<\/strong> Join \u201cmachine stopped\u201d state with maintenance work orders and operator notes to identify recurring failure modes.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3) Predictive maintenance feature store foundation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> ML models fail because sensor data lacks consistent labeling and history.<\/li>\n<li><strong>Why this fits:<\/strong> Curated, governed telemetry tables provide consistent features and labels.<\/li>\n<li><strong>Example:<\/strong> Build rolling-window vibration features per asset and train in Vertex AI.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4) Quality yield analytics and traceability<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Quality defects are discovered late; tracing batches to conditions is manual.<\/li>\n<li><strong>Why this fits:<\/strong> Contextualize process parameters and link to batch\/lot IDs.<\/li>\n<li><strong>Example:<\/strong> Combine temperature\/pressure curves with lot genealogy to identify out-of-spec patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5) Energy monitoring and sustainability reporting<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Energy data is siloed in building systems; reporting is inconsistent.<\/li>\n<li><strong>Why this fits:<\/strong> Standardize energy telemetry across sites and integrate with production volumes.<\/li>\n<li><strong>Example:<\/strong> Compute kWh per unit by line and shift; track anomalies and savings projects.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6) Multi-plant benchmarking<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Plants measure KPIs differently; leadership lacks comparable metrics.<\/li>\n<li><strong>Why this fits:<\/strong> Standard schema and governance enable consistent cross-site queries.<\/li>\n<li><strong>Example:<\/strong> A central BigQuery dataset stores normalized KPIs with site\/line dimensions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7) Alerting on abnormal process conditions<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Operators need timely alerts, but thresholds differ per asset.<\/li>\n<li><strong>Why this fits:<\/strong> Stream processing can compute rolling statistics and trigger downstream actions.<\/li>\n<li><strong>Example:<\/strong> Dataflow detects abnormal vibration trend; publishes alert to Pub\/Sub for notification workflows.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8) ERP\/MES reconciliation with shop-floor counts<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> ERP production counts differ from sensor-based counts; finance and operations disagree.<\/li>\n<li><strong>Why this fits:<\/strong> Unified datasets allow systematic reconciliation and audit trails.<\/li>\n<li><strong>Example:<\/strong> Compare MES counts with sensor pulses and scrap signals; flag variances.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">9) Digital thread for manufacturing engineering<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Engineering changes aren\u2019t linked to performance outcomes.<\/li>\n<li><strong>Why this fits:<\/strong> Join BOM\/routing changes to quality and throughput metrics.<\/li>\n<li><strong>Example:<\/strong> Analyze defect rates before\/after a tooling change across lines.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">10) Data product publishing for partners and suppliers<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> Sharing manufacturing KPIs externally is risky and labor-intensive.<\/li>\n<li><strong>Why this fits:<\/strong> Curated datasets + controlled access + audit logs enable safer sharing.<\/li>\n<li><strong>Example:<\/strong> Provide suppliers limited access to quality trend tables with row-level restrictions (where supported).<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">6. Core Features<\/h2>\n\n\n\n<p>Because Manufacturing Data Engine is implemented using Google Cloud data services (and may be packaged differently depending on release\/edition), the most reliable way to describe \u201cfeatures\u201d is by the capabilities you implement. <strong>Verify the exact official feature list in Google Cloud\u2019s Manufacturing Data Engine documentation for your environment.<\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 1: Streaming ingestion for telemetry and events<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Accepts high-throughput event streams (machine states, sensor readings) into cloud pipelines.<\/li>\n<li><strong>Why it matters:<\/strong> Manufacturing signals are continuous and bursty; decoupling producers from processing prevents data loss.<\/li>\n<li><strong>Practical benefit:<\/strong> You can ingest once and reuse the stream for multiple consumers (analytics, alerting, ML).<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Streaming systems require backlog monitoring, retry handling, and schema\/versioning discipline.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 2: Batch ingestion for enterprise manufacturing systems<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Brings in MES\/ERP\/QMS extracts (files, database exports, APIs) on schedules.<\/li>\n<li><strong>Why it matters:<\/strong> Production orders and quality records often arrive in batches; they provide essential context.<\/li>\n<li><strong>Practical benefit:<\/strong> Enables contextual joins (telemetry + work orders + lots).<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Late-arriving data complicates time-based analytics; you need watermarking and backfill strategy.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 3: Harmonization into consistent schemas (data contracts)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Standardizes machine signals into normalized formats: timestamps, asset IDs, measurement units, and event types.<\/li>\n<li><strong>Why it matters:<\/strong> Without harmonization, every dashboard\/model becomes custom per machine\/vendor.<\/li>\n<li><strong>Practical benefit:<\/strong> Analysts can write reusable queries; ML features can be standardized.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Requires governance: naming conventions, unit conversions, and controlled schema evolution.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 4: Contextualization with asset hierarchy and production context<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Enriches raw signals with metadata: plant\/line\/cell, asset class, product, shift, work order, operator.<\/li>\n<li><strong>Why it matters:<\/strong> Telemetry alone is rarely actionable without context.<\/li>\n<li><strong>Practical benefit:<\/strong> Enables KPI rollups (by line\/shift\/product) and root-cause analysis.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Context sources (MES master data) must be accurate; mismatched IDs are a common failure mode.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 5: Curated analytics layer in BigQuery (commonly)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Publishes \u201csilver\/gold\u201d tables optimized for analytics, dashboards, and ML.<\/li>\n<li><strong>Why it matters:<\/strong> Querying raw unmodeled telemetry is expensive and slow.<\/li>\n<li><strong>Practical benefit:<\/strong> Lower query cost and faster dashboards; consistent semantics.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> BigQuery costs depend on query patterns and storage; partitioning\/clustering design is critical.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 6: Governance (metadata, lineage, access control)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Tracks datasets, ownership, definitions, and (where configured) lineage and policy controls.<\/li>\n<li><strong>Why it matters:<\/strong> Manufacturing data is sensitive (production rates, yields, downtime causes).<\/li>\n<li><strong>Practical benefit:<\/strong> Enables self-service access without losing control; supports audit\/compliance.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Governance tooling and capabilities evolve\u2014confirm which features are enabled for your org.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 7: Operational monitoring and reliability patterns<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Uses Cloud Monitoring\/Logging to observe pipeline health, lag, errors, and throughput.<\/li>\n<li><strong>Why it matters:<\/strong> Pipelines become production systems; you need SLOs and alerting.<\/li>\n<li><strong>Practical benefit:<\/strong> Faster detection of broken sensors, schema changes, and backlog growth.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Logging volume can become costly; set retention and sampling intentionally.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 8: Integration with BI and reporting<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Exposes curated datasets to BI tools (Looker\/Looker Studio) and standard SQL consumers.<\/li>\n<li><strong>Why it matters:<\/strong> Operational and leadership decisions need accessible, trusted dashboards.<\/li>\n<li><strong>Practical benefit:<\/strong> Faster KPI iteration with consistent definitions.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Semantic modeling (metrics definitions) must be governed to avoid \u201cmultiple truths.\u201d<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 9: Integration with ML workflows (optional)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Enables ML training\/serving using curated features and labeled outcomes.<\/li>\n<li><strong>Why it matters:<\/strong> Manufacturing ML depends on long historical windows and consistent labeling.<\/li>\n<li><strong>Practical benefit:<\/strong> Predictive maintenance and quality models become repeatable.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> ML success depends on label quality and intervention workflows, not just pipelines.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Feature 10: Reprocessing\/backfill patterns<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Supports replaying data for corrections, model rebuilds, and late-arriving context.<\/li>\n<li><strong>Why it matters:<\/strong> Manufacturing pipelines must handle outages and data corrections.<\/li>\n<li><strong>Practical benefit:<\/strong> Recover from errors without losing historical continuity.<\/li>\n<li><strong>Limitations\/caveats:<\/strong> Reprocessing can be expensive; design raw retention and idempotent transformations.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">7. Architecture and How It Works<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">High-level architecture<\/h3>\n\n\n\n<p>A Manufacturing Data Engine-style architecture typically follows a layered approach:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Sources (OT\/IT):<\/strong> PLC\/SCADA\/historians, sensors, MES, ERP, QMS.<\/li>\n<li><strong>Ingestion:<\/strong> Streaming via Pub\/Sub; batch via Cloud Storage imports or scheduled extracts.<\/li>\n<li><strong>Processing:<\/strong> Dataflow (stream + batch) to validate, enrich, and transform.<\/li>\n<li><strong>Storage:<\/strong> Raw landing (Cloud Storage and\/or BigQuery raw tables), curated BigQuery tables.<\/li>\n<li><strong>Governance:<\/strong> Metadata cataloging, dataset ownership, access policies, audit logs.<\/li>\n<li><strong>Consumption:<\/strong> Looker dashboards, SQL, APIs, ML pipelines.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Request\/data\/control flow<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Data flow:<\/strong> device\/gateway \u2192 Pub\/Sub \u2192 Dataflow transform\/enrich \u2192 BigQuery curated tables \u2192 BI\/ML.<\/li>\n<li><strong>Control flow:<\/strong> CI\/CD deploys pipelines; orchestration triggers batch loads; monitoring triggers alerts.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Integrations with related services (common)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub for ingestion decoupling<\/li>\n<li>Dataflow for transformations, windowing, deduplication<\/li>\n<li>BigQuery for analytics storage and SQL<\/li>\n<li>Looker for dashboards and semantic modeling<\/li>\n<li>Cloud Storage for raw file landing and replay<\/li>\n<li>Cloud Monitoring\/Logging for operations<\/li>\n<li>IAM and (optionally) KMS\/VPC Service Controls for security controls<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Dependency services<\/h3>\n\n\n\n<p>Manufacturing Data Engine implementations generally depend on:\n&#8211; A Google Cloud project with billing enabled\n&#8211; One or more data regions\/multi-regions selected for data residency\n&#8211; Service accounts, IAM bindings\n&#8211; Data services (BigQuery, Pub\/Sub, Dataflow, Cloud Storage)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/authentication model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>IAM governs who can publish to Pub\/Sub, run Dataflow jobs, and query BigQuery.<\/li>\n<li>Workloads should run with dedicated service accounts.<\/li>\n<li>Audit Logs record administrative and data access events (depending on configuration).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Networking model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Many pipelines run over Google-managed endpoints by default.<\/li>\n<li>For tighter control, use private networking patterns (VPCs, Private Google Access, and service perimeters where supported). Verify per-service support and organizational constraints.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Monitoring\/logging\/governance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Monitor: Pub\/Sub subscription backlog, Dataflow job throughput\/latency, BigQuery load\/streaming errors.<\/li>\n<li>Log: pipeline errors with correlation IDs; avoid logging full payloads if sensitive.<\/li>\n<li>Govern: dataset naming, tags\/labels, data retention, and access review processes.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Simple architecture diagram (conceptual)<\/h4>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart LR\n  A[Factory signals\\n(telemetry\/events)] --&gt; B[Pub\/Sub]\n  B --&gt; C[Dataflow\\n(stream processing)]\n  C --&gt; D[BigQuery\\ncurated tables]\n  D --&gt; E[Looker \/ SQL \/ ML]\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Production-style architecture diagram (more realistic)<\/h4>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart TB\n  subgraph OT[OT \/ Plant Systems]\n    PLC[PLCs &amp; sensors]\n    SCADA[SCADA \/ Historian]\n    MES[MES \/ QMS]\n  end\n\n  subgraph Edge[Edge \/ Integration]\n    GW[Gateway \/ Connector\\n(partner or custom)]\n  end\n\n  subgraph GCP[Google Cloud Project]\n    PS[Pub\/Sub topics]\n    DF[Dataflow streaming &amp; batch jobs]\n    GCS[Cloud Storage\\nraw landing + replay]\n    BQRaw[BigQuery raw\/bronze]\n    BQCur[BigQuery curated\/silver+gold]\n    GOV[Governance\\n(Dataplex\/Data Catalog)\\nVerify exact tooling]\n    MON[Cloud Monitoring &amp; Logging]\n    BI[Looker \/ Looker Studio]\n    ML[Vertex AI\\n(optional)]\n  end\n\n  PLC --&gt; GW\n  SCADA --&gt; GW\n  MES --&gt; GCS\n\n  GW --&gt; PS\n  PS --&gt; DF\n  DF --&gt; BQRaw\n  DF --&gt; BQCur\n  GCS --&gt; DF\n  BQRaw --&gt; GOV\n  BQCur --&gt; GOV\n\n  BQCur --&gt; BI\n  BQCur --&gt; ML\n\n  PS --&gt; MON\n  DF --&gt; MON\n  BQCur --&gt; MON\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">8. Prerequisites<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Account\/project requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A <strong>Google Cloud project<\/strong> with <strong>billing enabled<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Permissions \/ IAM roles<\/h3>\n\n\n\n<p>For the hands-on lab in this tutorial, the simplest is:\n&#8211; Project Owner (for a temporary sandbox project)<\/p>\n\n\n\n<p>If you must use least privilege, you typically need permissions to:\n&#8211; Create and manage Pub\/Sub topics\/subscriptions\n&#8211; Create and run Dataflow jobs\n&#8211; Create BigQuery datasets\/tables and run queries\n&#8211; Create service accounts and bind IAM roles<\/p>\n\n\n\n<p>Exact role combinations vary. Common roles involved:\n&#8211; Pub\/Sub: <code>roles\/pubsub.admin<\/code> (lab) or narrower publisher\/subscriber roles\n&#8211; Dataflow: <code>roles\/dataflow.admin<\/code>, <code>roles\/dataflow.worker<\/code>\n&#8211; BigQuery: <code>roles\/bigquery.admin<\/code> (lab) or <code>roles\/bigquery.dataEditor<\/code> + <code>roles\/bigquery.jobUser<\/code>\n&#8211; Service accounts: <code>roles\/iam.serviceAccountAdmin<\/code> and <code>roles\/iam.serviceAccountUser<\/code> (or admin equivalents)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Billing requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Expect small charges for Dataflow streaming runtime, Pub\/Sub messages, BigQuery storage\/queries, and Logging.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">CLI\/SDK\/tools needed<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud Console access<\/li>\n<li>Optional: Cloud Shell (includes <code>gcloud<\/code>, <code>bq<\/code>, Python)<\/li>\n<li>Optional local tools:<\/li>\n<li>Google Cloud CLI: https:\/\/cloud.google.com\/sdk\/docs\/install<\/li>\n<li>Python 3.10+ for the message publisher script<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Region availability<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Choose a region where Dataflow is available and that meets your data residency needs.<\/li>\n<li>BigQuery dataset location must be chosen up front (changing later requires data migration).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas\/limits<\/h3>\n\n\n\n<p>Quotas vary by project and region. Common quota areas:\n&#8211; Pub\/Sub throughput and message size limits\n&#8211; Dataflow worker limits and job quotas\n&#8211; BigQuery streaming inserts and load job quotas (if used)\nAlways confirm in the Google Cloud Console <strong>Quotas<\/strong> page and relevant docs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisite services\/APIs<\/h3>\n\n\n\n<p>Enable APIs in your project:\n&#8211; Pub\/Sub API\n&#8211; Dataflow API\n&#8211; BigQuery API\n&#8211; Cloud Resource Manager API (often already enabled)\n&#8211; IAM API (often already enabled)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">9. Pricing \/ Cost<\/h2>\n\n\n\n<p><strong>Manufacturing Data Engine pricing model (practical reality):<\/strong>\n&#8211; In many organizations, Manufacturing Data Engine is implemented primarily using underlying Google Cloud services (Pub\/Sub, Dataflow, BigQuery, Cloud Storage, governance, BI).\n&#8211; There may or may not be a standalone SKU or commercial packaging depending on Google Cloud\u2019s current program\/edition. <strong>Verify in official docs<\/strong> for the current licensing\/pricing model specific to \u201cManufacturing Data Engine.\u201d<\/p>\n\n\n\n<p>Because of that, cost planning should focus on the cost drivers of the underlying services.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing dimensions (what you pay for)<\/h3>\n\n\n\n<p>Common cost dimensions in a Manufacturing Data Engine deployment:<\/p>\n\n\n\n<p><strong>Pub\/Sub<\/strong>\n&#8211; Data volume (ingress\/egress), message delivery, retained messages, and regional considerations.\n&#8211; Cost grows with high-frequency telemetry across many assets.<\/p>\n\n\n\n<p><strong>Dataflow<\/strong>\n&#8211; Worker compute (vCPU\/RAM), streaming engine (if used), job runtime, and autoscaling behavior.\n&#8211; Streaming jobs run continuously, so even low throughput can cost non-trivially over time.<\/p>\n\n\n\n<p><strong>BigQuery<\/strong>\n&#8211; Storage (active and long-term)\n&#8211; Query processing (on-demand bytes processed or capacity-based pricing)\n&#8211; Streaming ingestion (if used) and other BigQuery features depending on usage<\/p>\n\n\n\n<p><strong>Cloud Storage<\/strong>\n&#8211; Raw landing storage by GB-month\n&#8211; Operations (Class A\/B) and retrieval (depending on storage class)\n&#8211; Network egress (if exporting across regions)<\/p>\n\n\n\n<p><strong>Looker \/ Looker Studio<\/strong>\n&#8211; Looker licensing is typically subscription-based (enterprise). Looker Studio has a free tier and paid capabilities; verify current offerings.<\/p>\n\n\n\n<p><strong>Cloud Monitoring \/ Logging<\/strong>\n&#8211; Metric ingestion, logs ingestion, retention, and query costs. Logging can be a hidden cost driver in noisy pipelines.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Free tier<\/h3>\n\n\n\n<p>Google Cloud has free tiers for some products, but they are limited and may not cover sustained streaming Dataflow jobs. <strong>Verify current free-tier limits<\/strong> in official pricing docs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Main cost drivers (what makes bills spike)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>24\/7 Dataflow streaming jobs left running<\/li>\n<li>Unpartitioned\/unclustered BigQuery tables with broad \u201cSELECT *\u201d queries<\/li>\n<li>High-cardinality telemetry with no sampling\/aggregation strategy<\/li>\n<li>Excessive Logging volume (logging full payloads)<\/li>\n<li>Cross-region data movement (egress)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Hidden or indirect costs<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Egress:<\/strong> exporting data to other clouds\/on-prem<\/li>\n<li><strong>Backfills:<\/strong> reprocessing months of data through Dataflow\/BigQuery<\/li>\n<li><strong>BI concurrency:<\/strong> Looker query loads and caching strategy<\/li>\n<li><strong>Security overhead:<\/strong> CMEK key operations (small) and governance tooling operational time<\/li>\n<li><strong>People\/ops:<\/strong> on-call and pipeline maintenance<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network\/data transfer implications<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Keep Pub\/Sub, Dataflow region, and BigQuery dataset location aligned to reduce latency and avoid cross-region charges.<\/li>\n<li>If you ingest from on-prem plants, consider secure connectivity (Cloud VPN\/Interconnect) and its costs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How to optimize cost (practical checklist)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Start with a <strong>small curated dataset<\/strong> (the KPIs you truly need).<\/li>\n<li>Use <strong>aggregation windows<\/strong> (e.g., 1-second or 10-second rollups) rather than storing every raw sample forever.<\/li>\n<li>Partition BigQuery tables by time and cluster by asset ID.<\/li>\n<li>Use <strong>BigQuery reservations\/capacity<\/strong> if query volume is high and predictable.<\/li>\n<li>Control Dataflow autoscaling and worker machine types; set maximum workers.<\/li>\n<li>Use Logging exclusions and set retention intentionally.<\/li>\n<li>Separate environments (dev\/test\/prod) with budgets and alerts.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example low-cost starter estimate (no fabricated numbers)<\/h3>\n\n\n\n<p>A minimal pilot often includes:\n&#8211; One Pub\/Sub topic ingesting a small sensor stream\n&#8211; One Dataflow job with a small number of workers (or minimal autoscaling)\n&#8211; One BigQuery dataset with partitioned tables\n&#8211; A few Looker Studio dashboards<\/p>\n\n\n\n<p>Total monthly cost depends heavily on:\n&#8211; Dataflow job runtime (hours\/month)\n&#8211; Data volume ingested (GB\/month)\n&#8211; BigQuery queries (bytes processed) and retention\nUse:\n&#8211; Google Cloud Pricing Calculator: https:\/\/cloud.google.com\/products\/calculator\n&#8211; Product pricing pages (Pub\/Sub, Dataflow, BigQuery) for region-specific rates<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example production cost considerations<\/h3>\n\n\n\n<p>In production, the bill is usually dominated by:\n&#8211; Streaming compute (Dataflow)\n&#8211; BigQuery query processing at scale (dashboards + ad hoc + scheduled jobs)\n&#8211; Data retention (raw telemetry storage grows fast)\n&#8211; Multi-plant network ingress and secure connectivity<\/p>\n\n\n\n<p>Plan with:\n&#8211; A data retention policy (raw vs curated)\n&#8211; Aggregation and sampling strategy per use case\n&#8211; Query governance (authorized views, cached extracts, semantic layer discipline)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">10. Step-by-Step Hands-On Tutorial<\/h2>\n\n\n\n<p>This lab does not assume special access to a proprietary \u201cManufacturing Data Engine\u201d console. Instead, it teaches a <strong>practical, executable manufacturing data pipeline<\/strong> using the same Google Cloud data analytics and pipeline building blocks that commonly underpin Manufacturing Data Engine implementations.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Objective<\/h3>\n\n\n\n<p>Build a small, low-cost manufacturing telemetry pipeline:\n&#8211; Simulate machine sensor events\n&#8211; Ingest events into Pub\/Sub\n&#8211; Stream-transform into BigQuery (via Dataflow template in the Console)\n&#8211; Query in BigQuery to validate near-real-time analytics<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Lab Overview<\/h3>\n\n\n\n<p>You will create:\n1. BigQuery dataset and tables (telemetry + dead-letter)\n2. Pub\/Sub topic for incoming telemetry\n3. Dataflow streaming pipeline from Pub\/Sub \u2192 BigQuery\n4. A Python publisher that sends JSON events to Pub\/Sub\n5. Validation queries in BigQuery\n6. Cleanup of all resources<\/p>\n\n\n\n<p><strong>Estimated time:<\/strong> 45\u201375 minutes<br\/>\n<strong>Cost note:<\/strong> Dataflow streaming jobs cost money while running. You will stop it during cleanup.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Set project variables and enable APIs<\/h3>\n\n\n\n<p>You can do this in <strong>Cloud Shell<\/strong>.<\/p>\n\n\n\n<p>1) Set your project:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud config set project YOUR_PROJECT_ID\n<\/code><\/pre>\n\n\n\n<p>2) Enable required APIs:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud services enable \\\n  pubsub.googleapis.com \\\n  dataflow.googleapis.com \\\n  bigquery.googleapis.com\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong> APIs enable successfully (may take 1\u20133 minutes).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Choose a region and create a BigQuery dataset<\/h3>\n\n\n\n<p>Pick a region that fits your requirements (example: <code>us-central1<\/code>). Your BigQuery dataset location should align with your Dataflow region where possible.<\/p>\n\n\n\n<p>Create dataset (choose a location you are allowed to use):<\/p>\n\n\n\n<pre><code class=\"language-bash\">bq --location=US mk -d \\\n  --description \"Manufacturing Data Engine tutorial dataset (demo)\" \\\n  mde_demo\n<\/code><\/pre>\n\n\n\n<p>If you need a specific region instead of multi-region US, use <code>--location=us-central1<\/code> (BigQuery supports regional datasets in many regions). <strong>Verify supported locations in BigQuery docs.<\/strong><\/p>\n\n\n\n<p><strong>Expected outcome:<\/strong> Dataset <code>mde_demo<\/code> exists.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Create BigQuery tables (telemetry + dead-letter)<\/h3>\n\n\n\n<p>Create a partitioned telemetry table. We\u2019ll store one row per sensor event.<\/p>\n\n\n\n<pre><code class=\"language-bash\">bq mk --table \\\n  --time_partitioning_field event_ts \\\n  --time_partitioning_type DAY \\\n  mde_demo.telemetry_events \\\n  event_ts:TIMESTAMP,machine_id:STRING,temperature_c:FLOAT,vibration_mm_s:FLOAT,status:STRING,source:STRING\n<\/code><\/pre>\n\n\n\n<p>Create a dead-letter table for malformed messages (so the pipeline doesn\u2019t silently drop data):<\/p>\n\n\n\n<pre><code class=\"language-bash\">bq mk --table \\\n  --time_partitioning_field event_ts \\\n  --time_partitioning_type DAY \\\n  mde_demo.telemetry_deadletter \\\n  event_ts:TIMESTAMP,raw_payload:STRING,error_message:STRING\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong> Two tables exist in BigQuery.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Create a Pub\/Sub topic for telemetry ingestion<\/h3>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics create mde-telemetry-topic\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong> Topic <code>mde-telemetry-topic<\/code> exists.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Start a Dataflow streaming pipeline (Console-based, template)<\/h3>\n\n\n\n<p>To avoid relying on possibly changing command-line template parameters, use the Cloud Console template UI.<\/p>\n\n\n\n<p>1) Open the Dataflow jobs page:<br\/>\nhttps:\/\/console.cloud.google.com\/dataflow\/jobs<\/p>\n\n\n\n<p>2) Click <strong>Create job from template<\/strong>.<\/p>\n\n\n\n<p>3) Select:\n&#8211; <strong>Region:<\/strong> choose the same region you planned for processing (example: <code>us-central1<\/code>)\n&#8211; <strong>Dataflow template:<\/strong> search for a template that streams <strong>Pub\/Sub to BigQuery<\/strong>.\n  &#8211; Template names and parameters can evolve. Use the template description in the Console to confirm it reads from a Pub\/Sub topic and writes to BigQuery.<\/p>\n\n\n\n<p>4) Configure the template parameters (use the UI prompts):\n&#8211; <strong>Input Pub\/Sub topic:<\/strong> <code>projects\/YOUR_PROJECT_ID\/topics\/mde-telemetry-topic<\/code>\n&#8211; <strong>Output BigQuery table:<\/strong> <code>YOUR_PROJECT_ID:mde_demo.telemetry_events<\/code>\n&#8211; <strong>Dead-letter output table<\/strong> (if the template supports it): <code>YOUR_PROJECT_ID:mde_demo.telemetry_deadletter<\/code><\/p>\n\n\n\n<p>5) Runtime settings (recommended for a low-cost lab):\n&#8211; Choose a small worker type (default is usually fine)\n&#8211; Set a low <strong>maximum workers<\/strong> (for example 1\u20132)\n&#8211; Use a dedicated <strong>service account<\/strong> if required by your org policy (recommended in production)<\/p>\n\n\n\n<p>6) Click <strong>Run job<\/strong>.<\/p>\n\n\n\n<p><strong>Expected outcome:<\/strong>\n&#8211; A Dataflow streaming job starts.\n&#8211; Within a few minutes, it should be in a \u201cRunning\u201d state and ready to process messages.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Publish simulated machine events to Pub\/Sub<\/h3>\n\n\n\n<p>In Cloud Shell, create a Python script to generate telemetry events.<\/p>\n\n\n\n<pre><code class=\"language-bash\">cat &gt; publish_telemetry.py &lt;&lt;'PY'\nimport json, os, random, time\nfrom datetime import datetime, timezone\nfrom google.cloud import pubsub_v1\n\nPROJECT_ID = os.environ[\"PROJECT_ID\"]\nTOPIC_ID = os.environ.get(\"TOPIC_ID\", \"mde-telemetry-topic\")\n\npublisher = pubsub_v1.PublisherClient()\ntopic_path = publisher.topic_path(PROJECT_ID, TOPIC_ID)\n\nmachines = [\"press-01\", \"press-02\", \"cnc-07\", \"robot-03\"]\n\ndef make_event():\n    machine_id = random.choice(machines)\n    temperature_c = round(random.uniform(35.0, 95.0), 2)\n    vibration = round(random.uniform(0.1, 18.0), 2)\n    status = \"RUN\" if vibration &lt; 12.0 else \"ALERT\"\n    return {\n        \"event_ts\": datetime.now(timezone.utc).isoformat(),\n        \"machine_id\": machine_id,\n        \"temperature_c\": temperature_c,\n        \"vibration_mm_s\": vibration,\n        \"status\": status,\n        \"source\": \"simulator\"\n    }\n\ndef main():\n    print(f\"Publishing to {topic_path} ... Ctrl+C to stop\")\n    while True:\n        evt = make_event()\n        data = json.dumps(evt).encode(\"utf-8\")\n        future = publisher.publish(topic_path, data=data)\n        msg_id = future.result()\n        print(msg_id, evt)\n        time.sleep(1)\n\nif __name__ == \"__main__\":\n    main()\nPY\n<\/code><\/pre>\n\n\n\n<p>Install the Pub\/Sub client library (Cloud Shell often has it, but ensure it\u2019s available):<\/p>\n\n\n\n<pre><code class=\"language-bash\">pip3 install --user google-cloud-pubsub\n<\/code><\/pre>\n\n\n\n<p>Run the publisher:<\/p>\n\n\n\n<pre><code class=\"language-bash\">export PROJECT_ID=\"$(gcloud config get-value project)\"\npython3 publish_telemetry.py\n<\/code><\/pre>\n\n\n\n<p>Let it publish for ~3\u20135 minutes.<\/p>\n\n\n\n<p><strong>Expected outcome:<\/strong>\n&#8211; You see message IDs printed in Cloud Shell.\n&#8211; Pub\/Sub receives messages continuously.\n&#8211; Dataflow begins writing rows into BigQuery.<\/p>\n\n\n\n<p>Stop the script with <code>Ctrl+C<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 7: Query BigQuery to confirm data arrived<\/h3>\n\n\n\n<p>Run a query:<\/p>\n\n\n\n<pre><code class=\"language-bash\">bq query --use_legacy_sql=false '\nSELECT\n  machine_id,\n  status,\n  COUNT(*) AS events,\n  ROUND(AVG(temperature_c),2) AS avg_temp,\n  ROUND(AVG(vibration_mm_s),2) AS avg_vibration\nFROM `mde_demo.telemetry_events`\nWHERE event_ts &gt;= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 15 MINUTE)\nGROUP BY machine_id, status\nORDER BY events DESC;\n'\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong>\n&#8211; You see rows per machine and status with event counts and averages.\n&#8211; The table continues to grow if the publisher is still running.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 8: (Optional) Create a simple \u201cgold\u201d KPI table<\/h3>\n\n\n\n<p>A typical Manufacturing Data Engine pattern is to publish curated KPI tables derived from raw events.<\/p>\n\n\n\n<p>Create a simple aggregated table (last 1 minute by machine):<\/p>\n\n\n\n<pre><code class=\"language-bash\">bq query --use_legacy_sql=false '\nCREATE OR REPLACE TABLE `mde_demo.kpi_1min`\nPARTITION BY DATE(bucket_start)\nAS\nSELECT\n  TIMESTAMP_TRUNC(event_ts, MINUTE) AS bucket_start,\n  machine_id,\n  COUNT(*) AS event_count,\n  AVG(temperature_c) AS avg_temperature_c,\n  AVG(vibration_mm_s) AS avg_vibration_mm_s,\n  SUM(CASE WHEN status=\"ALERT\" THEN 1 ELSE 0 END) AS alert_events\nFROM `mde_demo.telemetry_events`\nWHERE event_ts &gt;= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 60 MINUTE)\nGROUP BY bucket_start, machine_id;\n'\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong> Table <code>mde_demo.kpi_1min<\/code> exists and contains rollups suitable for dashboards.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Validation<\/h3>\n\n\n\n<p>Use this checklist:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Pub\/Sub<\/strong><\/li>\n<li>Topic exists: <code>gcloud pubsub topics list | grep mde-telemetry-topic<\/code><\/li>\n<li><strong>Dataflow<\/strong><\/li>\n<li>Job is running in the Console and shows processed element counts increasing.<\/li>\n<li><strong>BigQuery<\/strong><\/li>\n<li><code>mde_demo.telemetry_events<\/code> has rows.<\/li>\n<li>Recent timestamps are present:\n    <code>bash\n    bq query --use_legacy_sql=false '\n    SELECT MAX(event_ts) AS latest_event_ts, COUNT(*) AS total\n    FROM `mde_demo.telemetry_events`;'<\/code><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting<\/h3>\n\n\n\n<p>Common issues and fixes:<\/p>\n\n\n\n<p>1) <strong>No rows in BigQuery<\/strong>\n&#8211; Confirm the Dataflow job is <strong>running<\/strong> and not failed.\n&#8211; Confirm the output table spec in the template matches:\n  &#8211; <code>YOUR_PROJECT_ID:mde_demo.telemetry_events<\/code>\n&#8211; Check if the template expects <strong>attributes<\/strong> vs message body JSON. Template behavior can differ\u2014open the template details in the Console and match the input format.<\/p>\n\n\n\n<p>2) <strong>Permission denied errors<\/strong>\n&#8211; If Dataflow cannot write to BigQuery, the Dataflow worker service account needs BigQuery permissions (<code>BigQuery Data Editor<\/code> + <code>BigQuery Job User<\/code> are commonly required).\n&#8211; If Dataflow cannot read Pub\/Sub, it needs Pub\/Sub subscriber permissions.<\/p>\n\n\n\n<p>3) <strong>Schema mismatch \/ parsing errors<\/strong>\n&#8211; Check dead-letter table (if configured) for malformed payloads.\n&#8211; Ensure JSON keys match column names and types (timestamp string should be ISO-8601).<\/p>\n\n\n\n<p>4) <strong>Dataflow job won\u2019t start<\/strong>\n&#8211; Verify Dataflow API is enabled.\n&#8211; Verify region availability and quotas.\n&#8211; Organization policies may require a customer-managed service account, CMEK, or restricted networking\u2014work with your platform team.<\/p>\n\n\n\n<p>5) <strong>High cost risk<\/strong>\n&#8211; Streaming Dataflow jobs bill while running. Stop the job when finished.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cleanup<\/h3>\n\n\n\n<p>Do these steps to avoid ongoing charges.<\/p>\n\n\n\n<p>1) Stop the Dataflow job:\n&#8211; In the Console: Dataflow job \u2192 <strong>Stop<\/strong> (or <strong>Drain<\/strong>, then stop)\n  &#8211; \u201cDrain\u201d attempts to finish in-flight work gracefully; it can take longer.<\/p>\n\n\n\n<p>2) Delete Pub\/Sub topic:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud pubsub topics delete mde-telemetry-topic\n<\/code><\/pre>\n\n\n\n<p>3) Delete BigQuery dataset (deletes all tables inside):<\/p>\n\n\n\n<pre><code class=\"language-bash\">bq rm -r -d mde_demo\n<\/code><\/pre>\n\n\n\n<p>4) (Optional) Delete any service accounts you created specifically for this lab.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">11. Best Practices<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Architecture best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use a <strong>layered data model<\/strong>: raw (bronze) \u2192 standardized (silver) \u2192 KPI\/semantic (gold).<\/li>\n<li>Keep <strong>raw immutable<\/strong> when possible; fix issues in curated layers, not by rewriting raw history (except in controlled backfills).<\/li>\n<li>Design for <strong>replay<\/strong>: retain raw messages\/files long enough to reprocess.<\/li>\n<li>Use <strong>idempotent transformations<\/strong> and deduplication keys (machine_id + event_ts + sequence).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IAM\/security best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Run pipelines with <strong>dedicated service accounts<\/strong> per environment (dev\/test\/prod).<\/li>\n<li>Grant least privilege:<\/li>\n<li>Pub\/Sub: publisher vs subscriber separation<\/li>\n<li>BigQuery: dataset-level access; use views for consumers<\/li>\n<li>Separate duties: data engineering admin vs analyst query access.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Cost best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Partition and cluster BigQuery tables.<\/li>\n<li>Avoid \u201cSELECT *\u201d in dashboards; use curated KPI tables.<\/li>\n<li>Aggregate high-frequency telemetry early (seconds\/minutes) unless raw resolution is required.<\/li>\n<li>Set Dataflow autoscaling limits and choose appropriate worker sizes.<\/li>\n<li>Control Logging volume; exclude noisy logs and set retention.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Performance best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In BigQuery:<\/li>\n<li>Partition by event time<\/li>\n<li>Cluster by machine_id (and possibly plant_id\/line_id)<\/li>\n<li>Use approximate aggregations where acceptable<\/li>\n<li>In streaming:<\/li>\n<li>Prefer structured messages with stable schemas<\/li>\n<li>Use windowing and state carefully; avoid unbounded cardinality keys<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Reliability best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Implement DLQs for bad messages.<\/li>\n<li>Track pipeline SLOs:<\/li>\n<li>ingestion lag<\/li>\n<li>processing success rate<\/li>\n<li>data freshness in curated tables<\/li>\n<li>Have a backfill plan: \u201cwhat if we lose 6 hours of MES data?\u201d<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operations best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Centralize dashboards for pipeline health (Pub\/Sub backlog, Dataflow errors, BigQuery load errors).<\/li>\n<li>Document runbooks for:<\/li>\n<li>schema change<\/li>\n<li>sensor outage<\/li>\n<li>replay\/backfill<\/li>\n<li>Use labels\/tags on resources for ownership, environment, and cost center.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Governance\/tagging\/naming best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Standard naming:<\/li>\n<li>topics: <code>mde-{domain}-{env}-telemetry<\/code><\/li>\n<li>datasets: <code>{domain}_{layer}_{env}<\/code> (example: <code>plant_raw_prod<\/code>)<\/li>\n<li>Data contracts:<\/li>\n<li>define required fields, units, and timestamp semantics<\/li>\n<li>Maintain a data dictionary and KPI definitions.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">12. Security Considerations<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Identity and access model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use IAM to control:<\/li>\n<li>who can publish telemetry<\/li>\n<li>who can operate pipelines<\/li>\n<li>who can query curated datasets<\/li>\n<li>Prefer group-based access (Google Groups \/ Cloud Identity) over individual grants.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Encryption<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Google Cloud encrypts data at rest and in transit by default.<\/li>\n<li>For sensitive environments, consider CMEK with Cloud KMS where supported by the services you use (verify per service and region).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network exposure<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Avoid public endpoints for ingestion when possible; use secure connectivity patterns:<\/li>\n<li>Cloud VPN \/ Dedicated Interconnect from plants<\/li>\n<li>Private Google Access for workloads without external IPs (where applicable)<\/li>\n<li>Restrict egress using VPC firewall rules and organization policy.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secrets handling<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Don\u2019t embed credentials in edge scripts or pipeline code.<\/li>\n<li>Use Secret Manager for API keys and DB passwords (if integrating with external systems).<\/li>\n<li>Prefer Workload Identity \/ service accounts over long-lived keys.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Audit\/logging<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable and retain Cloud Audit Logs according to your compliance needs.<\/li>\n<li>Log access to curated datasets (data access logs may be optional\u2014verify configuration).<\/li>\n<li>Use log-based metrics and alerts for error spikes.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compliance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Manufacturing can involve regulated data (pharma GMP, food traceability, export controls).<\/li>\n<li>Ensure region selection and retention align to compliance requirements.<\/li>\n<li>Document data lineage and change management for KPI definitions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Common security mistakes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Overly broad roles like Project Editor for analysts<\/li>\n<li>Allowing raw telemetry topics to be readable by many users<\/li>\n<li>Logging sensitive payloads in plaintext<\/li>\n<li>Cross-environment access (dev pipelines writing to prod datasets)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secure deployment recommendations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Separate projects for dev\/test\/prod.<\/li>\n<li>Use perimeters (VPC Service Controls) where appropriate and supported.<\/li>\n<li>Apply \u201cbreak-glass\u201d admin procedures with strong auditing.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">13. Limitations and Gotchas<\/h2>\n\n\n\n<p>Because Manufacturing Data Engine implementations depend on multiple services, the \u201cgotchas\u201d are usually cross-service.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Known limitations (practical)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Schema drift<\/strong> from machines\/vendors breaks downstream parsing.<\/li>\n<li><strong>Time sync issues<\/strong> (device timestamps vs gateway timestamps) create incorrect KPIs.<\/li>\n<li><strong>Late-arriving MES\/ERP context<\/strong> complicates joins and windowing.<\/li>\n<li><strong>Cardinality explosions<\/strong> (too many unique tags\/signals) raise costs and reduce performance.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pub\/Sub throughput and subscription limits<\/li>\n<li>Dataflow job and worker quotas per region<\/li>\n<li>BigQuery streaming and load limits\nAlways verify in your project\u2019s quotas pages and official docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Regional constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Not all services\/features are available in every region.<\/li>\n<li>BigQuery dataset location choices can constrain Dataflow region alignment.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing surprises<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Leaving Dataflow streaming jobs running overnight\/weekend<\/li>\n<li>Large BigQuery scans from dashboards refreshing frequently<\/li>\n<li>Logging ingestion costs from verbose pipeline logs<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compatibility issues<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Differences in OT protocols often require specialized gateways\/partners.<\/li>\n<li>Data formats vary widely (CSV, proprietary historian exports, JSON, OPC UA mappings).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational gotchas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u201cExactly-once\u201d semantics are not automatic end-to-end; design for duplicates.<\/li>\n<li>Reprocessing\/backfills require careful idempotency and partition management.<\/li>\n<li>Upgrades\/changes to pipelines should be versioned and rolled out safely.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Migration challenges<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Migrating from historians or on-prem warehouses requires careful mapping of tag names, units, and asset identity.<\/li>\n<li>Backfilling years of raw telemetry can be expensive; decide what history is truly needed.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Vendor-specific nuances<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>BigQuery performs best with correct partitioning and clustered access patterns.<\/li>\n<li>Streaming pipelines need careful monitoring of lag to avoid \u201csilent staleness.\u201d<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">14. Comparison with Alternatives<\/h2>\n\n\n\n<p>Manufacturing Data Engine sits in the \u201cmanufacturing analytics foundation\u201d space. Alternatives depend on whether you want a cloud-native analytics platform, an IoT\/industrial platform, or a self-managed stack.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Option<\/th>\n<th>Best For<\/th>\n<th>Strengths<\/th>\n<th>Weaknesses<\/th>\n<th>When to Choose<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>Manufacturing Data Engine (Google Cloud)<\/strong><\/td>\n<td>Manufacturers building a governed analytics foundation on Google Cloud<\/td>\n<td>Leverages Google Cloud analytics\/pipelines, scales well, integrates with BigQuery\/Looker\/Vertex AI<\/td>\n<td>Exact packaging\/features may vary; requires solid data engineering\/governance<\/td>\n<td>You want a manufacturing-focused data foundation and already use (or want) Google Cloud<\/td>\n<\/tr>\n<tr>\n<td><strong>BigQuery + Dataflow (DIY)<\/strong><\/td>\n<td>Teams that want full control and can engineer the platform<\/td>\n<td>Maximum flexibility, clear pricing by component, strong ecosystem<\/td>\n<td>More design\/ops burden; requires governance discipline<\/td>\n<td>You want the patterns without relying on any higher-level solution packaging<\/td>\n<\/tr>\n<tr>\n<td><strong>Dataplex + BigQuery<\/strong><\/td>\n<td>Data governance + analytics for many domains<\/td>\n<td>Cataloging\/governance + lakehouse patterns<\/td>\n<td>Still need ingestion\/processing; governance rollout takes time<\/td>\n<td>You need data governance as a first-class requirement<\/td>\n<\/tr>\n<tr>\n<td><strong>Cloud Data Fusion<\/strong><\/td>\n<td>Low-code ETL and connectors<\/td>\n<td>Faster ingestion for some sources<\/td>\n<td>Can be costly; may not be ideal for very high-rate telemetry<\/td>\n<td>You have many enterprise sources and need faster ETL onboarding<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS IoT SiteWise + AWS Analytics<\/strong><\/td>\n<td>Industrial data modeling + IoT ingestion on AWS<\/td>\n<td>Purpose-built industrial modeling and edge options<\/td>\n<td>Different ecosystem; analytics integration varies<\/td>\n<td>You are standardized on AWS and want an industrial platform approach<\/td>\n<\/tr>\n<tr>\n<td><strong>Azure IoT + Fabric\/Synapse<\/strong><\/td>\n<td>Industrial IoT and analytics on Azure<\/td>\n<td>Strong enterprise integration<\/td>\n<td>Different services and governance model<\/td>\n<td>You are standardized on Microsoft Azure ecosystem<\/td>\n<\/tr>\n<tr>\n<td><strong>Kafka + Spark + Data Lake (self-managed)<\/strong><\/td>\n<td>Organizations with strong platform engineering<\/td>\n<td>Cloud-agnostic, flexible<\/td>\n<td>High ops burden, scaling and security complexity<\/td>\n<td>You must remain cloud-agnostic and can operate complex systems<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">15. Real-World Example<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Enterprise example (multi-plant manufacturer)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> A global manufacturer has 20+ plants with different MES systems and inconsistent downtime reporting. Leadership wants standardized OEE and scrap dashboards and a foundation for predictive maintenance.<\/li>\n<li><strong>Proposed architecture:<\/strong><\/li>\n<li>Plant gateways publish standardized machine state events to Pub\/Sub<\/li>\n<li>Dataflow streaming normalizes events and enriches with asset hierarchy and shift calendars<\/li>\n<li>BigQuery stores curated event tables and KPI aggregates<\/li>\n<li>Looker provides global dashboards with consistent metrics definitions<\/li>\n<li>Vertex AI uses curated features for predictive maintenance models<\/li>\n<li>Governance via standardized dataset ownership, metadata, and access controls<\/li>\n<li><strong>Why Manufacturing Data Engine:<\/strong> It aligns with a repeatable pattern across plants\u2014unify, contextualize, govern, and publish data products.<\/li>\n<li><strong>Expected outcomes:<\/strong><\/li>\n<li>Standardized KPI definitions across plants<\/li>\n<li>Faster downtime investigations with unified timelines<\/li>\n<li>Reduced manual reporting and better decision cadence<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Startup\/small-team example (single plant, limited staff)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> A small manufacturer wants real-time visibility into a few critical machines and early warning for abnormal vibration but has only one engineer and a limited budget.<\/li>\n<li><strong>Proposed architecture:<\/strong><\/li>\n<li>Simple telemetry publisher (gateway) \u2192 Pub\/Sub<\/li>\n<li>Minimal Dataflow streaming job \u2192 BigQuery<\/li>\n<li>Looker Studio dashboard for 1-minute KPIs<\/li>\n<li>Alerts triggered from anomaly rules (can be added later)<\/li>\n<li><strong>Why Manufacturing Data Engine:<\/strong> The team adopts the same foundational pattern but starts small: one stream, one curated table, one dashboard.<\/li>\n<li><strong>Expected outcomes:<\/strong><\/li>\n<li>A working pipeline in days, not months<\/li>\n<li>Clear path to scale when more machines come online<\/li>\n<li>Costs controlled by limiting streaming runtime and aggregating data<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">16. FAQ<\/h2>\n\n\n\n<p>1) <strong>Is Manufacturing Data Engine a single Google Cloud product with its own console?<\/strong><br\/>\nNot always in the way services like BigQuery or Pub\/Sub are. It is often implemented as a manufacturing-oriented data foundation using multiple Google Cloud services. <strong>Verify the current official documentation and availability<\/strong> for your organization.<\/p>\n\n\n\n<p>2) <strong>Do I need Pub\/Sub and Dataflow to use Manufacturing Data Engine patterns?<\/strong><br\/>\nNot strictly, but they are common for streaming telemetry. For batch-only scenarios, you might rely on Cloud Storage + scheduled BigQuery loads and transformations.<\/p>\n\n\n\n<p>3) <strong>Where should I store raw telemetry\u2014Cloud Storage or BigQuery?<\/strong><br\/>\nOften both: Cloud Storage for cheap immutable raw retention\/replay; BigQuery raw tables for queryable raw. The choice depends on access patterns and retention needs.<\/p>\n\n\n\n<p>4) <strong>How do I handle schema changes when a machine vendor adds a new signal?<\/strong><br\/>\nUse versioned schemas, keep raw payloads, and evolve curated tables via controlled releases. Consider a \u201cwide\u201d telemetry table only if governance is strong; otherwise use normalized key-value modeling carefully (tradeoffs).<\/p>\n\n\n\n<p>5) <strong>How do I prevent duplicate events in streaming pipelines?<\/strong><br\/>\nDesign idempotency using event IDs\/sequence numbers, deduplication windows, and merge\/upsert patterns in curated tables. End-to-end exactly-once is not automatic.<\/p>\n\n\n\n<p>6) <strong>Is BigQuery good for time-series manufacturing data?<\/strong><br\/>\nYes for analytics, especially when partitioned and clustered properly. For extremely high-frequency raw signals, aggregate early and keep only the necessary resolution.<\/p>\n\n\n\n<p>7) <strong>How quickly can dashboards update?<\/strong><br\/>\nTypically seconds to minutes, depending on ingestion, processing, and BI caching. Define a data freshness SLO and monitor it.<\/p>\n\n\n\n<p>8) <strong>What\u2019s the best way to model an asset hierarchy (plant\/line\/cell\/machine)?<\/strong><br\/>\nUse dimension tables with stable asset IDs and effective dating for changes. Enrich events with the asset ID and optionally the hierarchy fields for faster queries.<\/p>\n\n\n\n<p>9) <strong>How do I join MES work orders to sensor events correctly?<\/strong><br\/>\nUse time validity intervals (work order start\/end), ensure consistent timezones, and handle late corrections via backfills.<\/p>\n\n\n\n<p>10) <strong>How do I control who can see sensitive KPIs like yield and downtime causes?<\/strong><br\/>\nUse IAM at dataset\/table\/view level. Consider authorized views and row-level security where supported and appropriate. Audit access regularly.<\/p>\n\n\n\n<p>11) <strong>What are the biggest operational risks?<\/strong><br\/>\nSilent pipeline failures, backlog growth, schema drift, and cost runaway from continuous streaming compute and broad queries.<\/p>\n\n\n\n<p>12) <strong>Can I run this with multiple plants?<\/strong><br\/>\nYes. Common patterns are per-plant topics and datasets with a central curated layer, or a central ingestion bus with standardized event formats.<\/p>\n\n\n\n<p>13) <strong>How do I implement data quality checks?<\/strong><br\/>\nUse SQL-based validation, anomaly detection on distributions, and quarantine tables. Some governance tools may provide data quality features\u2014<strong>verify current Dataplex capabilities<\/strong>.<\/p>\n\n\n\n<p>14) <strong>Is this suitable for regulated manufacturing (pharma\/medical)?<\/strong><br\/>\nPotentially, but you must design for validation, auditability, change control, and data residency. Engage compliance early and verify service certifications and controls.<\/p>\n\n\n\n<p>15) <strong>How do I estimate costs before production?<\/strong><br\/>\nMeasure:\n&#8211; events\/sec \u00d7 message size\n&#8211; required retention\n&#8211; dashboard query frequency\n&#8211; streaming runtime<br\/>\nThen model Pub\/Sub + Dataflow + BigQuery in the Pricing Calculator and run a controlled pilot.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">17. Top Online Resources to Learn Manufacturing Data Engine<\/h2>\n\n\n\n<p>Because \u201cManufacturing Data Engine\u201d can be delivered as a solution pattern across multiple Google Cloud services, you should learn both the manufacturing solution materials and the core data services.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Resource Type<\/th>\n<th>Name<\/th>\n<th>Why It Is Useful<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Official manufacturing landing<\/td>\n<td>https:\/\/cloud.google.com\/manufacturing<\/td>\n<td>Entry point for Google Cloud manufacturing solutions and related offerings<\/td>\n<\/tr>\n<tr>\n<td>Official docs (service lookup)<\/td>\n<td>https:\/\/cloud.google.com\/docs<\/td>\n<td>Use to search for \u201cManufacturing Data Engine\u201d and confirm current docs, status, and scope<\/td>\n<\/tr>\n<tr>\n<td>Pub\/Sub documentation<\/td>\n<td>https:\/\/cloud.google.com\/pubsub\/docs<\/td>\n<td>Core ingestion building block for streaming manufacturing events<\/td>\n<\/tr>\n<tr>\n<td>Dataflow documentation<\/td>\n<td>https:\/\/cloud.google.com\/dataflow\/docs<\/td>\n<td>Stream\/batch processing patterns, templates, and operational guidance<\/td>\n<\/tr>\n<tr>\n<td>BigQuery documentation<\/td>\n<td>https:\/\/cloud.google.com\/bigquery\/docs<\/td>\n<td>Data modeling, partitioning, cost control, and SQL analytics<\/td>\n<\/tr>\n<tr>\n<td>BigQuery pricing<\/td>\n<td>https:\/\/cloud.google.com\/bigquery\/pricing<\/td>\n<td>Understand storage vs query pricing and editions\/capacity options<\/td>\n<\/tr>\n<tr>\n<td>Dataflow pricing<\/td>\n<td>https:\/\/cloud.google.com\/dataflow\/pricing<\/td>\n<td>Understand streaming job cost drivers<\/td>\n<\/tr>\n<tr>\n<td>Pub\/Sub pricing<\/td>\n<td>https:\/\/cloud.google.com\/pubsub\/pricing<\/td>\n<td>Understand message delivery and throughput cost drivers<\/td>\n<\/tr>\n<tr>\n<td>Architecture Center<\/td>\n<td>https:\/\/cloud.google.com\/architecture<\/td>\n<td>Reference architectures for data platforms and streaming analytics<\/td>\n<\/tr>\n<tr>\n<td>Pricing Calculator<\/td>\n<td>https:\/\/cloud.google.com\/products\/calculator<\/td>\n<td>Build realistic estimates for pilot and production<\/td>\n<\/tr>\n<tr>\n<td>Looker documentation<\/td>\n<td>https:\/\/cloud.google.com\/looker\/docs<\/td>\n<td>Semantic modeling and governed BI on BigQuery<\/td>\n<\/tr>\n<tr>\n<td>Vertex AI documentation<\/td>\n<td>https:\/\/cloud.google.com\/vertex-ai\/docs<\/td>\n<td>ML pipelines and training\/serving for predictive maintenance\/quality<\/td>\n<\/tr>\n<tr>\n<td>Google Cloud Skills Boost<\/td>\n<td>https:\/\/www.cloudskillsboost.google<\/td>\n<td>Hands-on labs for BigQuery, Dataflow, Pub\/Sub, and data engineering patterns<\/td>\n<\/tr>\n<tr>\n<td>Dataflow templates overview<\/td>\n<td>https:\/\/cloud.google.com\/dataflow\/docs\/guides\/templates\/provided-templates<\/td>\n<td>Find the current \u201cPub\/Sub to BigQuery\u201d template parameters and behavior (verify template details here)<\/td>\n<\/tr>\n<tr>\n<td>BigQuery best practices<\/td>\n<td>https:\/\/cloud.google.com\/bigquery\/docs\/best-practices-performance-overview<\/td>\n<td>Practical optimization guidance for manufacturing analytics workloads<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">18. Training and Certification Providers<\/h2>\n\n\n\n<p>The following institutes are third-party training providers. Review their sites for current course outlines, delivery modes, and pricing.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Institute<\/th>\n<th>Suitable Audience<\/th>\n<th>Likely Learning Focus<\/th>\n<th>Mode<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>Cloud engineers, DevOps, data platform teams<\/td>\n<td>Google Cloud fundamentals, DevOps practices, pipelines and operations<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>ScmGalaxy.com<\/td>\n<td>Beginners to intermediate IT professionals<\/td>\n<td>DevOps, SCM, automation foundations that support data platform delivery<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.scmgalaxy.com\/<\/td>\n<\/tr>\n<tr>\n<td>CLoudOpsNow.in<\/td>\n<td>Cloud operations and platform teams<\/td>\n<td>Cloud operations practices, monitoring, governance<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.cloudopsnow.in\/<\/td>\n<\/tr>\n<tr>\n<td>SreSchool.com<\/td>\n<td>SREs, reliability-focused engineers<\/td>\n<td>SRE practices for operating production pipelines<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.sreschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>AiOpsSchool.com<\/td>\n<td>Ops + ML\/analytics practitioners<\/td>\n<td>AIOps concepts, monitoring analytics, incident automation<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.aiopsschool.com\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">19. Top Trainers<\/h2>\n\n\n\n<p>These are trainer-related platforms\/sites. Confirm specific trainer profiles and course details directly on the sites.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Platform\/Site<\/th>\n<th>Likely Specialization<\/th>\n<th>Suitable Audience<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>RajeshKumar.xyz<\/td>\n<td>DevOps\/cloud guidance (verify specific coverage)<\/td>\n<td>Individuals and teams seeking practical coaching<\/td>\n<td>https:\/\/rajeshkumar.xyz\/<\/td>\n<\/tr>\n<tr>\n<td>devopstrainer.in<\/td>\n<td>DevOps and cloud training (verify course list)<\/td>\n<td>Beginners to professionals<\/td>\n<td>https:\/\/devopstrainer.in\/<\/td>\n<\/tr>\n<tr>\n<td>devopsfreelancer.com<\/td>\n<td>Freelance DevOps support\/training (verify offerings)<\/td>\n<td>Teams needing flexible short-term help<\/td>\n<td>https:\/\/devopsfreelancer.com\/<\/td>\n<\/tr>\n<tr>\n<td>devopssupport.in<\/td>\n<td>DevOps support and training (verify scope)<\/td>\n<td>Ops teams and engineers<\/td>\n<td>https:\/\/devopssupport.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">20. Top Consulting Companies<\/h2>\n\n\n\n<p>These are consulting\/training organizations. Validate current service offerings and engagement models on their websites.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Company<\/th>\n<th>Likely Service Area<\/th>\n<th>Where They May Help<\/th>\n<th>Consulting Use Case Examples<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>cotocus.com<\/td>\n<td>Cloud\/DevOps\/engineering services (verify specifics)<\/td>\n<td>Implementing cloud platforms and automation<\/td>\n<td>Data platform automation, CI\/CD, infrastructure operations<\/td>\n<td>https:\/\/cotocus.com\/<\/td>\n<\/tr>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps and cloud enablement<\/td>\n<td>Training + consulting for cloud adoption<\/td>\n<td>Pipeline operationalization, monitoring rollout, team upskilling<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>DEVOPSCONSULTING.IN<\/td>\n<td>DevOps consulting services (verify specifics)<\/td>\n<td>Delivery\/process improvements and platform support<\/td>\n<td>Operational readiness, SRE practices, deployment automation<\/td>\n<td>https:\/\/www.devopsconsulting.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">21. Career and Learning Roadmap<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn before this service<\/h3>\n\n\n\n<p>To be effective with Manufacturing Data Engine patterns on Google Cloud, learn:\n&#8211; Google Cloud basics: projects, IAM, service accounts, networking basics\n&#8211; BigQuery fundamentals: datasets, partitioning, clustering, query costs\n&#8211; Pub\/Sub fundamentals: topics\/subscriptions, delivery semantics, ordering keys (where needed)\n&#8211; Dataflow basics: streaming vs batch, windowing concepts, operational monitoring\n&#8211; Data modeling: dimensional modeling, event modeling, slowly changing dimensions\n&#8211; Basic security: least privilege, audit logs, key management concepts<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn after this service<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Looker semantic modeling (LookML) and governed metrics<\/li>\n<li>Dataplex and data governance patterns (verify current feature set)<\/li>\n<li>CI\/CD for data pipelines (Cloud Build, Terraform)<\/li>\n<li>Data quality and observability tooling<\/li>\n<li>Vertex AI pipelines for predictive maintenance and quality<\/li>\n<li>Edge-to-cloud architecture and secure connectivity patterns<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Job roles that use it<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data Engineer (streaming\/batch pipelines)<\/li>\n<li>Cloud Data Platform Engineer<\/li>\n<li>Solutions Architect (manufacturing analytics)<\/li>\n<li>SRE \/ Platform Engineer (operating pipelines)<\/li>\n<li>Analytics Engineer (curated models and KPI layers)<\/li>\n<li>ML Engineer \/ Data Scientist (predictive maintenance\/quality)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Certification path (if available)<\/h3>\n\n\n\n<p>Google Cloud certifications that commonly align (verify current names and availability):\n&#8211; Professional Data Engineer\n&#8211; Professional Cloud Architect\n&#8211; Associate Cloud Engineer<\/p>\n\n\n\n<p>There may not be a dedicated \u201cManufacturing Data Engine\u201d certification; teams typically certify on the underlying Google Cloud data and architecture tracks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Project ideas for practice<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Build an OEE model: machine state \u2192 downtime \u2192 production count \u2192 OEE by shift.<\/li>\n<li>Implement a replayable ingestion design: raw to Cloud Storage + curated to BigQuery.<\/li>\n<li>Add data quality checks: range checks, missing timestamp checks, anomaly checks.<\/li>\n<li>Build an alert pipeline: detect anomaly \u2192 Pub\/Sub \u2192 Cloud Run webhook.<\/li>\n<li>Create a multi-plant dataset with standardized asset IDs and site rollups.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">22. Glossary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Asset hierarchy:<\/strong> A structured representation of manufacturing assets (plant \u2192 line \u2192 cell \u2192 machine).<\/li>\n<li><strong>Batch ingestion:<\/strong> Loading data in discrete intervals (hourly\/daily files, scheduled extracts).<\/li>\n<li><strong>BigQuery:<\/strong> Google Cloud\u2019s serverless data warehouse for SQL analytics.<\/li>\n<li><strong>Curated (silver\/gold) data:<\/strong> Cleaned and modeled datasets intended for analytics and reporting.<\/li>\n<li><strong>Data contract:<\/strong> An agreed schema and semantics for events\/tables shared between producers and consumers.<\/li>\n<li><strong>Dataflow:<\/strong> Managed service for Apache Beam pipelines (streaming and batch).<\/li>\n<li><strong>Dead-letter queue (DLQ):<\/strong> A place to store invalid messages\/events for later inspection and reprocessing.<\/li>\n<li><strong>Event time vs processing time:<\/strong> Event time is when the event happened at the source; processing time is when the pipeline processed it.<\/li>\n<li><strong>Historian:<\/strong> OT system that stores time-series process data from plant equipment.<\/li>\n<li><strong>IAM:<\/strong> Identity and Access Management; controls permissions in Google Cloud.<\/li>\n<li><strong>KPI:<\/strong> Key performance indicator (OEE, yield, scrap, downtime, throughput).<\/li>\n<li><strong>MES:<\/strong> Manufacturing Execution System; tracks production orders, operations, and execution details.<\/li>\n<li><strong>OEE:<\/strong> Overall Equipment Effectiveness; availability \u00d7 performance \u00d7 quality.<\/li>\n<li><strong>OT\/IT:<\/strong> Operational technology (plant systems) vs information technology (enterprise systems).<\/li>\n<li><strong>Partitioning:<\/strong> Organizing BigQuery tables by time\/date to reduce scan cost and improve performance.<\/li>\n<li><strong>Pub\/Sub:<\/strong> Managed messaging service used for event ingestion and decoupling.<\/li>\n<li><strong>Streaming pipeline:<\/strong> Continuous processing of events as they arrive with low latency.<\/li>\n<li><strong>Telemetry:<\/strong> Automated measurements collected from devices\/machines.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">23. Summary<\/h2>\n\n\n\n<p>Manufacturing Data Engine on Google Cloud is best approached as a manufacturing-focused data foundation: ingest OT\/IT data, standardize it, enrich it with context, govern it, and publish curated datasets for analytics and ML. In practice, it aligns closely with Google Cloud\u2019s <strong>Data analytics and pipelines<\/strong> services\u2014especially Pub\/Sub, Dataflow, and BigQuery\u2014plus governance, security, and BI layers.<\/p>\n\n\n\n<p>Cost and security outcomes depend on how you implement it:\n&#8211; <strong>Cost:<\/strong> watch Dataflow streaming runtime, BigQuery query patterns, and logging volume.\n&#8211; <strong>Security:<\/strong> use least-privilege IAM, dedicated service accounts, audit logging, and region controls.<\/p>\n\n\n\n<p>Use Manufacturing Data Engine patterns when you need repeatable, scalable manufacturing analytics across assets and sites. Start small (one stream, one curated table, one dashboard), then expand with governance, data quality, and ML.<\/p>\n\n\n\n<p>Next step: deepen your core skills in Pub\/Sub, Dataflow, and BigQuery, then add governance and semantic modeling so your manufacturing KPIs remain consistent and trusted as you scale.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data analytics and pipelines<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[59,51],"tags":[],"class_list":["post-666","post","type-post","status-publish","format-standard","hentry","category-data-analytics-and-pipelines","category-google-cloud"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/666","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=666"}],"version-history":[{"count":0,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/666\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=666"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=666"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=666"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}