{ "schema": 1, "slug": "dataops", "title": "DataOps", "full_name": "DataOps Engineering", "category": "Data & AI", "tagline": "Ship trusted, tested and observable data products with software-engineering discipline and automation.", "summary": "Learn to build data pipelines the way great teams ship software: version-controlled, tested, continuously deployed and monitored. This path takes you from SQL and data modelling through lakehouse platforms, dbt, orchestration, data contracts, observability and governance to AI-ready semantic layers.", "description": "DataOps applies DevOps, Agile and lean manufacturing thinking to data. Pipelines are treated as code, every change is versioned, reviewed, tested and deployed automatically, and data quality, freshness and lineage are monitored like any other production service. The goal is a short, safe cycle time from a business question to a trusted answer, with far less late-night firefighting.\n\nThe stakes are higher in 2026. Dashboards are no longer the only consumers: AI assistants, agents and retrieval-augmented applications read the same tables and metrics, so a silently broken pipeline now becomes a confidently wrong answer delivered at scale. At the same time the modern data stack has converged on open table formats such as Apache Iceberg and Delta Lake, shared catalogs, transformation as code with dbt and asset-aware orchestration, which makes real engineering discipline both possible and expected.\n\nThis path starts with SQL, Python, Git and data modelling, moves through lakehouse platforms, ingestion, change data capture and orchestration, then covers dbt, data testing and data contracts. The advanced stages add CI/CD for data, infrastructure as code, observability and lineage, governance, privacy, cost control and the semantic layers that make data safe for AI.\n\nDataOps sits next to MLOps, which consumes the features and training data you produce, FinOps, because warehouse and lakehouse compute is often a data team's largest cloud bill, and platform engineering, which turns your tooling into a self-service data platform. It borrows heavily from DevOps and SRE practice.", "icon": "database", "accent": "#14B8A6", "level": "Beginner → Advanced", "duration_weeks": 32, "weekly_hours": 10, "audience": [ "Data engineers who want to ship pipelines with tests, CI/CD and monitoring instead of manual fixes", "Analytics engineers and BI developers moving from ad hoc SQL to dbt-style software practices", "DevOps, platform and SRE engineers who support data platforms such as Snowflake, Databricks or Airflow", "Software engineers and data analysts transitioning into data engineering", "Data team leads who need better reliability, governance and delivery speed" ], "prerequisites": [ "Basic programming concepts (variables, loops, functions) in any language", "Simple SQL SELECT queries with filtering and grouping, or willingness to learn them in Stage 1", "Comfort with the command line and installing developer tools", "A laptop that can run Docker, plus a free tier or trial account on a cloud data platform" ], "outcomes": [ "Write performant, idempotent SQL and Python for batch and incremental data processing", "Design dimensional models and implement them as Apache Iceberg or Delta Lake tables on a lakehouse or cloud warehouse", "Build ingestion pipelines from APIs and databases, including log-based change data capture with Debezium and Apache Kafka", "Orchestrate dependable workflows with Apache Airflow 3 or Dagster, including retries, backfills and alerting", "Develop modular dbt projects with unit tests, data tests and enforced data contracts", "Run CI/CD for data with slim builds, isolated environments and infrastructure as code", "Instrument pipelines with OpenLineage lineage, data observability and SLAs, and lead data incident response", "Govern, secure and cost-optimise a data platform and expose certified metrics safely to BI tools and AI agents" ], "trends": [ "Open table formats are now the default: Apache Iceberg v3 and REST catalogs such as Apache Polaris (a top-level Apache project since February 2026) and Unity Catalog let Spark, Snowflake, Databricks, Trino and DuckDB share the same tables.", "Fivetran and dbt Labs completed their merger in June 2026, and dbt's Rust-based Fusion engine runtime was open-sourced as dbt Core v2.0 (alpha) under the Apache 2.0 licence.", "Orchestration is asset-aware: Airflow 3.x brings assets, DAG versioning and human-in-the-loop tasks, Spark 4.1 introduced Spark Declarative Pipelines, and Databricks Asset Bundles are now Declarative Automation Bundles.", "Semantic layers have become AI infrastructure: the Open Semantic Interchange entered the Apache Incubator as Apache Ossie in 2026, and MCP servers expose governed metrics and lineage to AI agents.", "Quality is shifting left: data contracts based on the Open Data Contract Standard, dbt unit tests and contract checks now run in pull requests, with OpenLineage tying incidents back to upstream changes.", "Cost and compliance are first-class concerns: data teams own warehouse and lakehouse spend with FinOps practices and billing system tables, while regulation such as the EU AI Act raises expectations for documented, governed training data." ], "roles": [ { "title": "DataOps Engineer", "description": "Builds the automation around data pipelines, including CI/CD, testing, orchestration, monitoring and incident response, so data products ship quickly and reliably." }, { "title": "Analytics Engineer", "description": "Turns raw data into clean, tested, documented models and metrics with dbt and SQL, bridging data engineering and analytics." }, { "title": "Data Platform Engineer", "description": "Runs the lakehouse or warehouse as an internal product: storage, compute, catalogs, access control, infrastructure as code and cost." }, { "title": "Data Reliability Engineer", "description": "Applies SRE practice to data by defining freshness and quality SLOs, running observability and lineage tooling, and leading data incident postmortems." }, { "title": "Data Governance Engineer", "description": "Implements catalogs, classification, access policies, masking and retention so data is discoverable, compliant and safe to use for AI." } ], "stages": [ { "title": "Foundations: DataOps Mindset, SQL & Python", "level": "Beginner", "duration_weeks": 6, "description": "Learn the DataOps principles and the core hands-on skills every data pipeline depends on: SQL, Python, Git, Linux and containers.", "milestone": "You can write production-quality SQL and Python, keep it in Git with pull requests, and run a local PostgreSQL and DuckDB stack in Docker.", "topics": [ { "title": "DataOps Principles & the Data Lifecycle", "description": "Understand what DataOps is, where it came from and how it changes the way data teams work. Map the data lifecycle from source to consumption and learn to measure reliability and delivery speed so improvements are visible.", "est_hours": 8, "optional": false, "concepts": [ "DataOps Manifesto", "data products and ownership", "batch vs streaming", "medallion architecture", "data SLAs and SLOs", "cycle time and DORA-style metrics for data" ], "tools": [], "resources": [ { "title": "The DataOps Manifesto", "url": "https://dataopsmanifesto.org/en/", "type": "article", "free": true }, { "title": "DataOps Training and Certification — DevOpsSchool", "url": "https://www.devopsschool.com/courses/dataops/", "type": "course", "free": false }, { "title": "DataOps Foundation Certification — DevOpsSchool", "url": "https://www.devopsschool.com/certification/dataops-foundation-certification.html", "type": "course", "free": false } ] }, { "title": "SQL for Data Engineering", "description": "SQL is the lingua franca of every warehouse, lakehouse and transformation tool. Go beyond SELECT to window functions, CTEs, MERGE-based upserts and reading query plans so your transformations are correct, idempotent and fast.", "est_hours": 24, "optional": false, "concepts": [ "joins and set operations", "window functions", "CTEs and subqueries", "NULL semantics", "MERGE and idempotent upserts", "query plans with EXPLAIN", "semi-structured JSON data" ], "tools": [ "PostgreSQL", "DuckDB" ], "resources": [ { "title": "PostgreSQL Tutorial (official docs)", "url": "https://www.postgresql.org/docs/current/tutorial.html", "type": "docs", "free": true }, { "title": "SQLBolt interactive SQL lessons", "url": "https://sqlbolt.com/", "type": "course", "free": true }, { "title": "DuckDB Documentation", "url": "https://duckdb.org/docs/stable/", "type": "docs", "free": true } ] }, { "title": "Python for Data Pipelines", "description": "Python glues ingestion, orchestration, testing and automation together. Learn modern packaging, typed code, columnar DataFrames with Arrow and Parquet, API extraction and unit testing so your pipeline code is maintainable.", "est_hours": 20, "optional": false, "concepts": [ "virtual environments and uv", "type hints", "DataFrames with Polars and pandas", "Apache Arrow and Parquet", "calling REST APIs", "logging and error handling", "unit tests with pytest" ], "tools": [ "Python", "Polars" ], "resources": [ { "title": "The Python Tutorial", "url": "https://docs.python.org/3/tutorial/", "type": "docs", "free": true }, { "title": "Polars User Guide", "url": "https://docs.pola.rs/", "type": "docs", "free": true }, { "title": "pytest Documentation", "url": "https://docs.pytest.org/en/stable/", "type": "docs", "free": true } ] }, { "title": "Git, Linux & Containers for Data Work", "description": "DataOps starts with treating analytics as code. Use Git branches and pull requests for every change, work fluently in a Linux shell, and package local data stacks with Docker Compose so environments are reproducible and disposable.", "est_hours": 12, "optional": false, "concepts": [ "branches and pull requests", "code review", "shell basics", "environment variables and secrets", "Docker images and Compose", "pre-commit hooks" ], "tools": [ "Git", "GitHub", "Linux", "Docker" ], "resources": [ { "title": "Pro Git (free book)", "url": "https://git-scm.com/book/en/v2", "type": "book", "free": true }, { "title": "Docker Get Started", "url": "https://docs.docker.com/get-started/", "type": "docs", "free": true }, { "title": "pre-commit framework", "url": "https://pre-commit.com/", "type": "docs", "free": true } ] } ] }, { "title": "Data Modelling & the Lakehouse", "level": "Intermediate", "duration_weeks": 6, "description": "Model data for analytics and AI, then learn the cloud warehouse and lakehouse platforms, open table formats and catalogs that store it.", "milestone": "You can design a dimensional model and implement it as Iceberg or Delta tables that more than one query engine can read.", "topics": [ { "title": "Data Modelling for Analytics", "description": "Good models make every downstream query, test and metric simpler. Learn dimensional modelling, slowly changing dimensions and layered project structures, and when Data Vault or wide denormalised tables are the better fit.", "est_hours": 16, "optional": false, "concepts": [ "grain", "facts and dimensions", "star schema", "slowly changing dimensions (SCD Type 2)", "Data Vault basics", "one big table vs normalised models", "staging, intermediate and mart layers" ], "tools": [], "resources": [ { "title": "Kimball Dimensional Modeling Techniques", "url": "https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/", "type": "article", "free": true }, { "title": "How we structure our dbt projects", "url": "https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview", "type": "docs", "free": true }, { "title": "Designing Data-Intensive Applications", "url": "https://dataintensive.net/", "type": "book", "free": false } ] }, { "title": "Cloud Warehouses & Lakehouse Platforms", "description": "Most DataOps work runs on Snowflake, Databricks, BigQuery or Microsoft Fabric. Learn how storage and compute are separated, how Spark processes data at scale, and how the medallion pattern organises a lakehouse, using free tiers to practise.", "est_hours": 20, "optional": false, "concepts": [ "separation of storage and compute", "virtual warehouses and clusters", "Spark DataFrames", "bronze, silver and gold layers", "BigQuery and Microsoft Fabric equivalents", "free tiers and trial accounts" ], "tools": [ "Snowflake", "Databricks", "Apache Spark" ], "resources": [ { "title": "Snowflake in 20 Minutes", "url": "https://docs.snowflake.com/en/user-guide/tutorials/snowflake-in-20minutes", "type": "docs", "free": true }, { "title": "Databricks Free Edition", "url": "https://www.databricks.com/learn/free-edition", "type": "lab", "free": true }, { "title": "Databricks Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/databricks/", "type": "course", "free": false } ] }, { "title": "Open Table Formats: Apache Iceberg & Delta Lake", "description": "Open table formats bring ACID transactions, time travel and schema evolution to files in object storage. Understanding them is key to building lakehouses that avoid lock-in and stay fast as data grows.", "est_hours": 14, "optional": false, "concepts": [ "Parquet internals", "ACID on object storage", "snapshots and time travel", "schema and partition evolution", "deletion vectors and compaction", "Iceberg v3 features", "Delta UniForm interoperability" ], "tools": [ "Apache Iceberg", "Delta Lake" ], "resources": [ { "title": "Apache Iceberg Documentation", "url": "https://iceberg.apache.org/docs/latest/", "type": "docs", "free": true }, { "title": "Delta Lake Documentation", "url": "https://docs.delta.io/", "type": "docs", "free": true }, { "title": "Lakehouse: A New Generation of Open Platforms (CIDR 2021)", "url": "https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf", "type": "paper", "free": true } ] }, { "title": "Catalogs & Multi-Engine Interoperability", "description": "The catalog decides who can find and write which tables from which engine. Learn the Iceberg REST catalog model, Apache Polaris and Unity Catalog, and practise a local lakehouse with DuckDB so you can reason about interoperability.", "est_hours": 8, "optional": false, "concepts": [ "Iceberg REST catalog", "Apache Polaris", "Unity Catalog open source", "Iceberg tables in Snowflake", "local lakehouse with DuckDB", "avoiding engine lock-in" ], "tools": [ "Apache Polaris", "Unity Catalog", "DuckDB" ], "resources": [ { "title": "Apache Polaris", "url": "https://polaris.apache.org/", "type": "docs", "free": true }, { "title": "Unity Catalog (open source) Documentation", "url": "https://docs.unitycatalog.io/", "type": "docs", "free": true }, { "title": "Snowflake: Apache Iceberg tables", "url": "https://docs.snowflake.com/en/user-guide/tables-iceberg", "type": "docs", "free": true } ] } ] }, { "title": "Ingestion, CDC & Orchestration", "level": "Intermediate", "duration_weeks": 5, "description": "Move data reliably from APIs, SaaS tools and operational databases into the lakehouse, and orchestrate the work with retries, backfills and alerting.", "milestone": "You can land data from an API and a CDC stream into the lakehouse on a schedule, with retries, backfills and alerts handled by an orchestrator.", "topics": [ { "title": "Batch & API Ingestion (ELT)", "description": "Ingestion is where most data incidents start. Learn connector-based ELT and code-first loading, incremental extraction with state, and how to handle schema drift and backfills without duplicating or losing data.", "est_hours": 12, "optional": false, "concepts": [ "ELT vs ETL", "managed vs open-source connectors", "incremental cursors and state", "schema drift handling", "idempotent loads", "backfills" ], "tools": [ "Airbyte", "dlt" ], "resources": [ { "title": "Airbyte Open Source Quickstart", "url": "https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart", "type": "docs", "free": true }, { "title": "dlt Documentation", "url": "https://dlthub.com/docs/intro", "type": "docs", "free": true }, { "title": "AWS Certified Data Engineer Associate Training — DevOpsSchool", "url": "https://www.devopsschool.com/certification/aws-certified-data-engineer-associate.html", "type": "course", "free": false } ] }, { "title": "Change Data Capture & Streaming", "description": "Log-based change data capture streams every insert, update and delete from operational databases without hammering them with queries. Learn Debezium and Kafka fundamentals, schemas and delivery semantics, and how CDC lands in lakehouse tables.", "est_hours": 20, "optional": false, "concepts": [ "log-based CDC", "topics, partitions and consumer groups", "KRaft mode", "schema registry with Avro or Protobuf", "delivery semantics", "outbox pattern", "streaming into Iceberg tables" ], "tools": [ "Debezium", "Apache Kafka" ], "resources": [ { "title": "Debezium Tutorial", "url": "https://debezium.io/documentation/reference/stable/tutorial.html", "type": "docs", "free": true }, { "title": "Apache Kafka Documentation", "url": "https://kafka.apache.org/documentation/", "type": "docs", "free": true }, { "title": "Kafka Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/kafka/", "type": "course", "free": false } ] }, { "title": "Workflow Orchestration with Apache Airflow 3", "description": "Airflow is still the most widely deployed orchestrator, and version 3 modernised it with assets, DAG versioning and a new task execution model. Learn to write idempotent, testable DAGs and run them reliably.", "est_hours": 18, "optional": false, "concepts": [ "DAGs and tasks", "assets and data-aware scheduling", "DAG versioning", "deferrable operators", "retries and failure callbacks", "backfills", "running Airflow on Kubernetes" ], "tools": [ "Apache Airflow" ], "resources": [ { "title": "Apache Airflow Tutorials", "url": "https://airflow.apache.org/docs/apache-airflow/stable/tutorial/index.html", "type": "docs", "free": true }, { "title": "Apache Airflow Documentation", "url": "https://airflow.apache.org/docs/apache-airflow/stable/index.html", "type": "docs", "free": true } ] }, { "title": "Asset-Based Orchestration with Dagster & Prefect", "description": "Dagster models pipelines as the data assets they produce, while Prefect offers lightweight Python-native flows. Compare them with Airflow so you can choose the right orchestrator for a team.", "est_hours": 10, "optional": true, "concepts": [ "software-defined assets", "partitions and freshness policies", "Python-native flows", "local development and testing", "choosing an orchestrator" ], "tools": [ "Dagster", "Prefect" ], "resources": [ { "title": "Dagster Documentation", "url": "https://docs.dagster.io/", "type": "docs", "free": true }, { "title": "Prefect Documentation", "url": "https://docs.prefect.io/", "type": "docs", "free": true } ] } ] }, { "title": "Transformation, Testing & Data Contracts", "level": "Intermediate", "duration_weeks": 5, "description": "Build transformations as modular, documented code with dbt, then protect them with layered tests and explicit contracts between data producers and consumers.", "milestone": "You can ship a tested, documented dbt project with unit tests, data tests and enforced contracts that fails fast on bad data.", "topics": [ { "title": "Transformation as Code with dbt", "description": "dbt brings software engineering to SQL transformations through modular models, dependency graphs, documentation and environments. Learn the core workflow plus incremental models, snapshots and macros used in real projects.", "est_hours": 24, "optional": false, "concepts": [ "models, sources and ref()", "materializations", "incremental models", "snapshots", "Jinja macros and packages", "documentation and exposures", "dbt Fusion engine", "dbt Mesh" ], "tools": [ "dbt" ], "resources": [ { "title": "dbt Fundamentals (free course)", "url": "https://learn.getdbt.com/courses/dbt-fundamentals", "type": "course", "free": true }, { "title": "What is dbt? (official docs)", "url": "https://docs.getdbt.com/docs/introduction", "type": "docs", "free": true }, { "title": "dbt Incremental Models", "url": "https://docs.getdbt.com/docs/build/incremental-models", "type": "docs", "free": true } ] }, { "title": "Data Testing & Quality Engineering", "description": "Code tests alone cannot catch bad data arriving from upstream. Combine dbt unit tests for logic with data tests and quality checks for freshness, volume, schema and distributions, and stop bad records before they are published.", "est_hours": 16, "optional": false, "concepts": [ "dbt unit tests vs data tests", "freshness, volume and schema checks", "distribution and anomaly checks", "write-audit-publish pattern", "quarantining bad records", "a testing pyramid for data" ], "tools": [ "dbt", "Great Expectations", "Soda" ], "resources": [ { "title": "dbt Unit Tests", "url": "https://docs.getdbt.com/docs/build/unit-tests", "type": "docs", "free": true }, { "title": "GX Core Introduction (Great Expectations)", "url": "https://docs.greatexpectations.io/docs/core/introduction/", "type": "docs", "free": true }, { "title": "Soda Documentation", "url": "https://docs.soda.io/", "type": "docs", "free": true } ] }, { "title": "Data Contracts & Schema Change Management", "description": "Data contracts make the expectations between producers and consumers explicit and machine-checkable. Learn the Open Data Contract Standard, dbt model contracts and versions, and how to catch breaking changes in pull requests.", "est_hours": 10, "optional": false, "concepts": [ "producer and consumer agreements", "Open Data Contract Standard (ODCS)", "dbt model contracts and versions", "breaking-change detection", "schema registries", "SLAs and ownership in contracts" ], "tools": [ "Data Contract CLI", "dbt" ], "resources": [ { "title": "Open Data Contract Standard (Bitol)", "url": "https://bitol-io.github.io/open-data-contract-standard/latest/", "type": "docs", "free": true }, { "title": "dbt Model Contracts", "url": "https://docs.getdbt.com/docs/mesh/govern/model-contracts", "type": "docs", "free": true }, { "title": "Data Contract CLI", "url": "https://cli.datacontract.com/", "type": "tool", "free": true } ] }, { "title": "Declarative Pipelines with Spark & Databricks", "description": "Declarative pipelines let you describe datasets and quality expectations while the engine handles dependencies, retries and incremental processing. Learn Spark Declarative Pipelines and Structured Streaming for batch and near-real-time workloads.", "est_hours": 10, "optional": true, "concepts": [ "Spark Declarative Pipelines", "streaming tables and materialized views", "pipeline expectations", "Structured Streaming", "Lakeflow jobs" ], "tools": [ "Apache Spark", "Databricks" ], "resources": [ { "title": "Spark Declarative Pipelines Programming Guide", "url": "https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html", "type": "docs", "free": true }, { "title": "Structured Streaming Programming Guide", "url": "https://spark.apache.org/docs/latest/streaming/index.html", "type": "docs", "free": true } ] } ] }, { "title": "CI/CD, Observability & Lineage", "level": "Advanced", "duration_weeks": 5, "description": "Automate how data changes reach production and make pipelines observable end to end, so issues are caught before consumers notice and traced quickly when they do.", "milestone": "Every pull request runs a slim CI build in an isolated environment, deployments are automated, and lineage plus freshness and quality alerts cover the pipeline end to end.", "topics": [ { "title": "CI/CD for Data Pipelines", "description": "Data CI/CD builds and tests only what changed, in isolated environments, before promoting it. Learn slim CI with dbt state selection, SQL linting, ephemeral schemas and zero-copy clones, data branching and bundle-based deployments.", "est_hours": 18, "optional": false, "concepts": [ "slim CI with state:modified and defer", "ephemeral schemas and zero-copy clones", "SQL linting", "blue-green table deployments", "data versioning and branches", "promotion across dev, staging and prod" ], "tools": [ "GitHub Actions", "SQLFluff", "lakeFS" ], "resources": [ { "title": "Continuous integration in dbt", "url": "https://docs.getdbt.com/docs/deploy/continuous-integration", "type": "docs", "free": true }, { "title": "GitHub Actions Documentation", "url": "https://docs.github.com/en/actions", "type": "docs", "free": true }, { "title": "Databricks Declarative Automation Bundles", "url": "https://docs.databricks.com/aws/en/dev-tools/bundles/", "type": "docs", "free": true } ] }, { "title": "Infrastructure as Code for Data Platforms", "description": "Warehouses, catalogs, grants and compute policies drift quickly when managed by hand. Define them with Terraform so environments are consistent, access changes are reviewed and the whole platform can be rebuilt.", "est_hours": 10, "optional": false, "concepts": [ "Snowflake and Databricks Terraform providers", "RBAC and grants as code", "environment parity", "secrets management", "policy checks in CI" ], "tools": [ "Terraform" ], "resources": [ { "title": "Snowflake Terraform Provider", "url": "https://registry.terraform.io/providers/snowflakedb/snowflake/latest/docs", "type": "docs", "free": true }, { "title": "Databricks Terraform Provider", "url": "https://registry.terraform.io/providers/databricks/databricks/latest/docs", "type": "docs", "free": true }, { "title": "Terraform Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/terraform/", "type": "course", "free": false } ] }, { "title": "Data Observability & Incident Response", "description": "Data observability watches freshness, volume, schema, distribution and lineage so problems are detected before stakeholders find them. Learn to set data SLOs, route alerts and run blameless data incident reviews.", "est_hours": 12, "optional": false, "concepts": [ "five pillars of data observability", "anomaly detection", "data SLOs and error budgets", "alert routing and on-call", "data incident postmortems", "pipeline runtime metrics with OpenTelemetry" ], "tools": [ "Elementary", "OpenTelemetry" ], "resources": [ { "title": "Elementary OSS Introduction", "url": "https://docs.elementary-data.com/oss/oss-introduction", "type": "docs", "free": true }, { "title": "OpenTelemetry Documentation", "url": "https://opentelemetry.io/docs/", "type": "docs", "free": true }, { "title": "Observability Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/observability/", "type": "course", "free": false } ] }, { "title": "Lineage with OpenLineage & Marquez", "description": "Lineage shows where every dataset comes from and what breaks if it changes. Learn the OpenLineage standard, collect lineage from Airflow, Spark and dbt, and use it for impact analysis and root-cause investigation.", "est_hours": 10, "optional": false, "concepts": [ "run, job and dataset model", "facets", "column-level lineage", "Airflow, Spark and dbt integrations", "impact and root-cause analysis" ], "tools": [ "OpenLineage", "Marquez" ], "resources": [ { "title": "OpenLineage Documentation", "url": "https://openlineage.io/docs/", "type": "docs", "free": true }, { "title": "Airflow OpenLineage Provider", "url": "https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html", "type": "docs", "free": true }, { "title": "Marquez Project", "url": "https://marquezproject.ai/", "type": "tool", "free": true } ] } ] }, { "title": "Governance, Privacy, Cost & AI-Ready Data", "level": "Advanced", "duration_weeks": 5, "description": "Operate the data platform responsibly at scale: catalog and govern data, protect personal data, control spend, and publish certified metrics that BI tools and AI agents can trust.", "milestone": "You can run a governed, cost-aware data platform whose certified metrics are safely consumable by BI tools and AI agents.", "topics": [ { "title": "Catalogs, Metadata & Access Governance", "description": "People and AI agents can only use data they can find, understand and are allowed to access. Learn ownership, glossaries, classification tags and fine-grained access control with modern catalogs, and how they support data products and data mesh.", "est_hours": 12, "optional": false, "concepts": [ "ownership and stewardship", "business glossary", "classification tags", "RBAC and ABAC", "data products and data mesh", "metadata ingestion" ], "tools": [ "Unity Catalog", "DataHub", "OpenMetadata" ], "resources": [ { "title": "Databricks Unity Catalog", "url": "https://docs.databricks.com/aws/en/data-governance/unity-catalog/", "type": "docs", "free": true }, { "title": "DataHub Quickstart", "url": "https://docs.datahub.com/docs/quickstart", "type": "docs", "free": true }, { "title": "OpenMetadata Documentation", "url": "https://docs.open-metadata.org/", "type": "docs", "free": true } ] }, { "title": "Privacy, Security & Compliance", "description": "Personal and sensitive data must be protected wherever it flows. Learn to classify PII, apply dynamic masking and row filters, pseudonymise data and handle retention and erasure requests in lakehouse tables.", "est_hours": 10, "optional": false, "concepts": [ "PII discovery and classification", "dynamic data masking", "row-level filters", "tokenization and pseudonymization", "retention and right to erasure", "audit logging", "GDPR principles" ], "tools": [ "Snowflake", "Databricks" ], "resources": [ { "title": "Snowflake Dynamic Data Masking", "url": "https://docs.snowflake.com/en/user-guide/security-column-ddm-intro", "type": "docs", "free": true }, { "title": "Databricks Row Filters and Column Masks", "url": "https://docs.databricks.com/aws/en/data-governance/unity-catalog/filters-and-masks", "type": "docs", "free": true }, { "title": "NIST Privacy Framework", "url": "https://www.nist.gov/privacy-framework", "type": "docs", "free": true } ] }, { "title": "Cost Management & Performance Tuning", "description": "Compute for warehouses and lakehouses is often a data team's biggest bill. Learn to size and auto-suspend compute, tune table layout and queries, attribute spend with tags and billing tables, and apply FinOps practices.", "est_hours": 10, "optional": false, "concepts": [ "warehouse sizing and auto-suspend", "partitioning, clustering and compaction", "query profiling", "billing system tables and cost tags", "incremental processing to cut compute", "FinOps for data platforms" ], "tools": [ "Snowflake", "Databricks" ], "resources": [ { "title": "Understanding Snowflake Cost", "url": "https://docs.snowflake.com/en/user-guide/cost-understanding-overall", "type": "docs", "free": true }, { "title": "Databricks Billable Usage System Table", "url": "https://docs.databricks.com/aws/en/admin/system-tables/billing", "type": "docs", "free": true }, { "title": "FinOps Framework", "url": "https://www.finops.org/framework/", "type": "docs", "free": true } ] }, { "title": "Semantic Layers & AI-Ready Data", "description": "AI agents and text-to-SQL tools give wrong answers when metric definitions are ambiguous. Define certified metrics once in a semantic layer, expose them and their lineage through MCP servers with guardrails, and prepare documented, quality-checked data for retrieval and embeddings.", "est_hours": 16, "optional": false, "concepts": [ "metrics layer with MetricFlow", "Apache Ossie semantic interchange", "MCP servers for data", "text-to-SQL guardrails", "embeddings and vector search", "data quality for RAG and agents", "documentation as AI context" ], "tools": [ "dbt", "Apache Ossie", "Model Context Protocol", "pgvector" ], "resources": [ { "title": "dbt Semantic Layer", "url": "https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl", "type": "docs", "free": true }, { "title": "dbt MCP Server", "url": "https://docs.getdbt.com/docs/dbt-ai/about-mcp", "type": "docs", "free": true }, { "title": "Apache Ossie (Incubating)", "url": "https://ossie.apache.org/", "type": "docs", "free": true } ] } ] } ], "projects": [ { "title": "Local Lakehouse Starter", "description": "Use Docker Compose to run PostgreSQL and DuckDB, load a public dataset with dlt into Parquet files, and model it into a small star schema with SQL. Publish the repository with a README, a Makefile and pre-commit checks.", "difficulty": "Beginner", "skills": [ "SQL", "Python", "dlt", "DuckDB", "Docker", "Git" ] }, { "title": "CDC Stream into Iceberg", "description": "Capture changes from a PostgreSQL database with Debezium, stream them through Apache Kafka and land them as Apache Iceberg tables that both Spark and DuckDB can query. Demonstrate inserts, updates and deletes flowing end to end with a replay test.", "difficulty": "Intermediate", "skills": [ "Debezium", "Apache Kafka", "Apache Iceberg", "Apache Spark", "change data capture" ] }, { "title": "Tested dbt Project Orchestrated by Airflow", "description": "Build a dbt project with staging, intermediate and mart layers on Snowflake, Databricks or DuckDB, covered by unit tests, data tests and model contracts. Schedule ingestion and dbt runs with Airflow 3 assets and publish generated documentation.", "difficulty": "Intermediate", "skills": [ "dbt", "Apache Airflow", "data testing", "data contracts", "data modelling" ] }, { "title": "Data CI/CD Pipeline with Infrastructure as Code", "description": "Create a GitHub Actions workflow that lints SQL with SQLFluff, runs slim dbt CI against an ephemeral schema, validates data contracts and promotes changes to production with approval. Manage warehouse roles, grants and compute with Terraform.", "difficulty": "Advanced", "skills": [ "GitHub Actions", "dbt", "SQLFluff", "Terraform", "data contracts" ] }, { "title": "Observability and Lineage Control Tower", "description": "Emit OpenLineage events from Airflow, Spark and dbt into Marquez, add Elementary freshness and anomaly monitors with alerts to chat, and define data SLOs for key tables. Run a simulated upstream schema break and write the incident postmortem.", "difficulty": "Advanced", "skills": [ "OpenLineage", "Marquez", "Elementary", "data SLOs", "incident response" ] }, { "title": "Governed, AI-Ready Data Product", "description": "Publish a data product with an ODCS contract, catalog entries in DataHub or OpenMetadata, PII tags with masking policies and certified metrics in the dbt Semantic Layer. Expose it to an AI assistant through an MCP server and evaluate the assistant's answers against the certified metrics.", "difficulty": "Advanced", "skills": [ "semantic layer", "data governance", "privacy controls", "Model Context Protocol", "data contracts" ] } ], "certifications": [ { "name": "DataOps Certified Professional (DOCP)", "provider": "DevOpsSchool", "url": "https://www.devopsschool.com/certification/dataops-certified-professional.html", "level": "Professional", "note": "Instructor-led programme covering DataOps end to end, from pipelines, CI/CD and IaC to observability, Databricks, Delta Lake and Unity Catalog governance." }, { "name": "dbt Analytics Engineering Certification", "provider": "dbt Labs (Fivetran + dbt Labs)", "url": "https://www.getdbt.com/certifications/analytics-engineer-certification-exam", "level": "Associate", "note": "Validates modelling, testing, documentation and deployment with dbt, the core transformation skill in this path; the 2026 exam targets dbt 1.11." }, { "name": "Databricks Certified Data Engineer Associate", "provider": "Databricks", "url": "https://www.databricks.com/learn/certification/data-engineer-associate", "level": "Associate", "note": "The current exam guide has dedicated CI/CD and troubleshooting domains alongside ingestion, transformation, Lakeflow Jobs and governance; progress to the Data Engineer Professional exam afterwards." }, { "name": "SnowPro Core Certification (COF-C03)", "provider": "Snowflake", "url": "https://learn.snowflake.com/en/certifications/snowpro-core-c03/", "level": "Associate", "note": "Proves hands-on Snowflake skills in architecture, loading, transformation, performance and data protection; COF-C03 replaced COF-C02 in 2026." }, { "name": "AWS Certified Data Engineer – Associate (DEA-C01)", "provider": "Amazon Web Services", "url": "https://aws.amazon.com/certification/certified-data-engineer-associate/", "level": "Associate", "note": "Covers ingestion, transformation, orchestration, data stores, operations and governance on AWS for teams whose platform runs there." }, { "name": "Google Cloud Professional Data Engineer", "provider": "Google Cloud", "url": "https://cloud.google.com/learn/certification/data-engineer", "level": "Professional", "note": "Tests designing, building, operationalising and automating data processing systems on Google Cloud, including BigQuery-based platforms." }, { "name": "Microsoft Certified: Fabric Data Engineer Associate (DP-700)", "provider": "Microsoft", "url": "https://learn.microsoft.com/en-us/credentials/certifications/fabric-data-engineer-associate/", "level": "Associate", "note": "The current Microsoft data engineering credential, covering ingestion, transformation, security and monitoring of analytics solutions in Microsoft Fabric." } ], "tools_catalog": [ { "name": "Airbyte", "category": "Data Engineering", "website": "https://airbyte.com/", "ds_course_url": "" }, { "name": "Apache Airflow", "category": "Data Engineering", "website": "https://airflow.apache.org/", "ds_course_url": "" }, { "name": "Apache Iceberg", "category": "Data Engineering", "website": "https://iceberg.apache.org/", "ds_course_url": "" }, { "name": "Apache Kafka", "category": "Data Engineering", "website": "https://kafka.apache.org/", "ds_course_url": "https://www.devopsschool.com/courses/kafka/" }, { "name": "Apache Ossie", "category": "Data Engineering", "website": "https://ossie.apache.org/", "ds_course_url": "" }, { "name": "Apache Polaris", "category": "Data Engineering", "website": "https://polaris.apache.org/", "ds_course_url": "" }, { "name": "Apache Spark", "category": "Data Engineering", "website": "https://spark.apache.org/", "ds_course_url": "https://www.devopsschool.com/courses/spark/" }, { "name": "Dagster", "category": "Data Engineering", "website": "https://dagster.io/", "ds_course_url": "" }, { "name": "Data Contract CLI", "category": "Data Engineering", "website": "https://cli.datacontract.com/", "ds_course_url": "" }, { "name": "Databricks", "category": "Data Engineering", "website": "https://www.databricks.com/", "ds_course_url": "https://www.devopsschool.com/courses/databricks/" }, { "name": "DataHub", "category": "Data Engineering", "website": "https://datahub.com/", "ds_course_url": "" }, { "name": "dbt", "category": "Data Engineering", "website": "https://www.getdbt.com/", "ds_course_url": "" }, { "name": "Debezium", "category": "Data Engineering", "website": "https://debezium.io/", "ds_course_url": "" }, { "name": "Delta Lake", "category": "Data Engineering", "website": "https://delta.io/", "ds_course_url": "" }, { "name": "dlt", "category": "Data Engineering", "website": "https://dlthub.com/", "ds_course_url": "" }, { "name": "Docker", "category": "Containers & Kubernetes", "website": "https://www.docker.com/", "ds_course_url": "https://www.devopsschool.com/courses/docker/" }, { "name": "DuckDB", "category": "Data Engineering", "website": "https://duckdb.org/", "ds_course_url": "" }, { "name": "Elementary", "category": "Data Engineering", "website": "https://www.elementary-data.com/", "ds_course_url": "" }, { "name": "Git", "category": "Foundations & Languages", "website": "https://git-scm.com/", "ds_course_url": "https://www.devopsschool.com/courses/git/" }, { "name": "GitHub", "category": "Foundations & Languages", "website": "https://github.com/", "ds_course_url": "https://www.devopsschool.com/courses/github/" }, { "name": "GitHub Actions", "category": "CI/CD & Release", "website": "https://github.com/features/actions", "ds_course_url": "https://www.devopsschool.com/courses/github/" }, { "name": "Great Expectations", "category": "Data Engineering", "website": "https://greatexpectations.io/", "ds_course_url": "" }, { "name": "lakeFS", "category": "Data Engineering", "website": "https://lakefs.io/", "ds_course_url": "" }, { "name": "Linux", "category": "Foundations & Languages", "website": "https://www.kernel.org/", "ds_course_url": "https://www.devopsschool.com/courses/linux/" }, { "name": "Marquez", "category": "Data Engineering", "website": "https://marquezproject.ai/", "ds_course_url": "" }, { "name": "Model Context Protocol", "category": "LLMs, Agents & GenAI", "website": "https://modelcontextprotocol.io/", "ds_course_url": "" }, { "name": "OpenLineage", "category": "Data Engineering", "website": "https://openlineage.io/", "ds_course_url": "" }, { "name": "OpenMetadata", "category": "Data Engineering", "website": "https://open-metadata.org/", "ds_course_url": "" }, { "name": "OpenTelemetry", "category": "Observability", "website": "https://opentelemetry.io/", "ds_course_url": "https://www.devopsschool.com/courses/opentelemetry/" }, { "name": "pgvector", "category": "LLMs, Agents & GenAI", "website": "https://github.com/pgvector/pgvector", "ds_course_url": "https://www.devopsschool.com/courses/postgresql/" }, { "name": "Polars", "category": "Data Engineering", "website": "https://pola.rs/", "ds_course_url": "" }, { "name": "PostgreSQL", "category": "Data Engineering", "website": "https://www.postgresql.org/", "ds_course_url": "https://www.devopsschool.com/courses/postgresql/" }, { "name": "Prefect", "category": "Data Engineering", "website": "https://www.prefect.io/", "ds_course_url": "" }, { "name": "Python", "category": "Foundations & Languages", "website": "https://www.python.org/", "ds_course_url": "https://www.devopsschool.com/courses/python/" }, { "name": "Snowflake", "category": "Data Engineering", "website": "https://www.snowflake.com/", "ds_course_url": "" }, { "name": "Soda", "category": "Data Engineering", "website": "https://soda.io/", "ds_course_url": "" }, { "name": "SQLFluff", "category": "Foundations & Languages", "website": "https://www.sqlfluff.com/", "ds_course_url": "" }, { "name": "Terraform", "category": "Infrastructure as Code & Automation", "website": "https://developer.hashicorp.com/terraform", "ds_course_url": "https://www.devopsschool.com/courses/terraform/" }, { "name": "Unity Catalog", "category": "Data Engineering", "website": "https://www.unitycatalog.io/", "ds_course_url": "" } ], "faqs": [ { "q": "What is DataOps and how is it different from data engineering?", "a": "Data engineering is the work of building pipelines and data platforms. DataOps is the set of practices and automation that make that work reliable and fast to change: version control, automated testing, CI/CD, orchestration, observability and close collaboration with data consumers. Many DataOps engineers are data engineers with strong DevOps skills." }, { "q": "Is DataOps just DevOps applied to data?", "a": "It borrows heavily from DevOps, but data adds problems that code alone does not have. Data changes independently of your code, so you must test the data as well as the logic, monitor freshness and quality in production, and track lineage and governance. The DataOps Manifesto also draws on lean manufacturing and statistical process control." }, { "q": "Do I need to learn both Snowflake and Databricks?", "a": "No. Learn one platform deeply, ideally the one your target employers use, and understand the other at a conceptual level. SQL, data modelling, dbt and open table formats such as Apache Iceberg transfer between them, and both platforms now read and write Iceberg tables." }, { "q": "Should I learn Apache Airflow or Dagster first?", "a": "Start with Apache Airflow because it is the most widely deployed orchestrator and appears in the most job postings, and Airflow 3 added assets and DAG versioning. Then try Dagster or Prefect to understand asset-centric and Python-native approaches, which many newer teams prefer." }, { "q": "Is dbt still worth learning after the Fivetran and dbt Labs merger?", "a": "Yes. dbt remains the standard way to manage SQL transformations as code, and the merged company has kept investing in open source, releasing the Rust-based Fusion runtime as dbt Core v2.0 (alpha) under Apache 2.0. The dbt Analytics Engineering Certification also remains current." }, { "q": "Which certification should I take first for a DataOps career?", "a": "Pick the platform certification that matches your target stack: SnowPro Core, Databricks Data Engineer Associate, AWS Certified Data Engineer – Associate, Google Professional Data Engineer or Microsoft DP-700. Add the dbt Analytics Engineering Certification for transformation skills, and the DevOpsSchool DataOps Certified Professional for end-to-end DataOps practice." }, { "q": "How long does it take to become job-ready in DataOps?", "a": "At about 10 hours per week this path takes roughly 32 weeks, including portfolio projects. People who already know SQL and Python, or who come from DevOps, can often move faster through the first two stages." } ], "related": [ "mlops", "llmops", "devops", "finops" ], "key_tools": [ "dbt", "Apache Airflow", "Dagster", "Databricks", "Snowflake", "Apache Iceberg" ], "course_url": "https://www.devopsschool.com/courses/dataops/", "seo_title": "DataOps Learning Path 2026: Roadmap, Tools & Projects", "seo_description": "Learn DataOps in 2026: SQL, lakehouse, dbt, Airflow, data testing and contracts, CI/CD, observability, lineage, governance and AI-ready data. Free roadmap.", "url": "https://www.devopsschool.com/path/dataops/", "updated_at": "2026-10-26 08:58:45" }