Ship trusted, tested and observable data products with software-engineering discipline and automation.
Learn to build data pipelines the way great teams ship software: version-controlled, tested, continuously deployed and monitored. This path takes you from SQL and data modelling through lakehouse platforms, dbt, orchestration, data contracts, observability and governance to AI-ready semantic layers.
DataOps applies DevOps, Agile and lean manufacturing thinking to data. Pipelines are treated as code, every change is versioned, reviewed, tested and deployed automatically, and data quality, freshness and lineage are monitored like any other production service. The goal is a short, safe cycle time from a business question to a trusted answer, with far less late-night firefighting.
The stakes are higher in 2026. Dashboards are no longer the only consumers: AI assistants, agents and retrieval-augmented applications read the same tables and metrics, so a silently broken pipeline now becomes a confidently wrong answer delivered at scale. At the same time the modern data stack has converged on open table formats such as Apache Iceberg and Delta Lake, shared catalogs, transformation as code with dbt and asset-aware orchestration, which makes real engineering discipline both possible and expected.
This path starts with SQL, Python, Git and data modelling, moves through lakehouse platforms, ingestion, change data capture and orchestration, then covers dbt, data testing and data contracts. The advanced stages add CI/CD for data, infrastructure as code, observability and lineage, governance, privacy, cost control and the semantic layers that make data safe for AI.
DataOps sits next to MLOps, which consumes the features and training data you produce, FinOps, because warehouse and lakehouse compute is often a data team's largest cloud bill, and platform engineering, which turns your tooling into a self-service data platform. It borrows heavily from DevOps and SRE practice.
6 stages · 24 topics · ~340 hours of focused learning. Work through the stages in order; optional topics are electives you can come back to.
Learn the DataOps principles and the core hands-on skills every data pipeline depends on: SQL, Python, Git, Linux and containers.
Understand what DataOps is, where it came from and how it changes the way data teams work. Map the data lifecycle from source to consumption and learn to measure reliability and delivery speed so improvements are visible.
SQL is the lingua franca of every warehouse, lakehouse and transformation tool. Go beyond SELECT to window functions, CTEs, MERGE-based upserts and reading query plans so your transformations are correct, idempotent and fast.
Python glues ingestion, orchestration, testing and automation together. Learn modern packaging, typed code, columnar DataFrames with Arrow and Parquet, API extraction and unit testing so your pipeline code is maintainable.
DataOps starts with treating analytics as code. Use Git branches and pull requests for every change, work fluently in a Linux shell, and package local data stacks with Docker Compose so environments are reproducible and disposable.
Milestone: You can write production-quality SQL and Python, keep it in Git with pull requests, and run a local PostgreSQL and DuckDB stack in Docker.
Model data for analytics and AI, then learn the cloud warehouse and lakehouse platforms, open table formats and catalogs that store it.
Good models make every downstream query, test and metric simpler. Learn dimensional modelling, slowly changing dimensions and layered project structures, and when Data Vault or wide denormalised tables are the better fit.
Most DataOps work runs on Snowflake, Databricks, BigQuery or Microsoft Fabric. Learn how storage and compute are separated, how Spark processes data at scale, and how the medallion pattern organises a lakehouse, using free tiers to practise.
Open table formats bring ACID transactions, time travel and schema evolution to files in object storage. Understanding them is key to building lakehouses that avoid lock-in and stay fast as data grows.
The catalog decides who can find and write which tables from which engine. Learn the Iceberg REST catalog model, Apache Polaris and Unity Catalog, and practise a local lakehouse with DuckDB so you can reason about interoperability.
Milestone: You can design a dimensional model and implement it as Iceberg or Delta tables that more than one query engine can read.
Move data reliably from APIs, SaaS tools and operational databases into the lakehouse, and orchestrate the work with retries, backfills and alerting.
Ingestion is where most data incidents start. Learn connector-based ELT and code-first loading, incremental extraction with state, and how to handle schema drift and backfills without duplicating or losing data.
Log-based change data capture streams every insert, update and delete from operational databases without hammering them with queries. Learn Debezium and Kafka fundamentals, schemas and delivery semantics, and how CDC lands in lakehouse tables.
Airflow is still the most widely deployed orchestrator, and version 3 modernised it with assets, DAG versioning and a new task execution model. Learn to write idempotent, testable DAGs and run them reliably.
Dagster models pipelines as the data assets they produce, while Prefect offers lightweight Python-native flows. Compare them with Airflow so you can choose the right orchestrator for a team.
Milestone: You can land data from an API and a CDC stream into the lakehouse on a schedule, with retries, backfills and alerts handled by an orchestrator.
Build transformations as modular, documented code with dbt, then protect them with layered tests and explicit contracts between data producers and consumers.
dbt brings software engineering to SQL transformations through modular models, dependency graphs, documentation and environments. Learn the core workflow plus incremental models, snapshots and macros used in real projects.
Code tests alone cannot catch bad data arriving from upstream. Combine dbt unit tests for logic with data tests and quality checks for freshness, volume, schema and distributions, and stop bad records before they are published.
Data contracts make the expectations between producers and consumers explicit and machine-checkable. Learn the Open Data Contract Standard, dbt model contracts and versions, and how to catch breaking changes in pull requests.
Declarative pipelines let you describe datasets and quality expectations while the engine handles dependencies, retries and incremental processing. Learn Spark Declarative Pipelines and Structured Streaming for batch and near-real-time workloads.
Milestone: You can ship a tested, documented dbt project with unit tests, data tests and enforced contracts that fails fast on bad data.
Automate how data changes reach production and make pipelines observable end to end, so issues are caught before consumers notice and traced quickly when they do.
Data CI/CD builds and tests only what changed, in isolated environments, before promoting it. Learn slim CI with dbt state selection, SQL linting, ephemeral schemas and zero-copy clones, data branching and bundle-based deployments.
Warehouses, catalogs, grants and compute policies drift quickly when managed by hand. Define them with Terraform so environments are consistent, access changes are reviewed and the whole platform can be rebuilt.
Data observability watches freshness, volume, schema, distribution and lineage so problems are detected before stakeholders find them. Learn to set data SLOs, route alerts and run blameless data incident reviews.
Lineage shows where every dataset comes from and what breaks if it changes. Learn the OpenLineage standard, collect lineage from Airflow, Spark and dbt, and use it for impact analysis and root-cause investigation.
Milestone: Every pull request runs a slim CI build in an isolated environment, deployments are automated, and lineage plus freshness and quality alerts cover the pipeline end to end.
Operate the data platform responsibly at scale: catalog and govern data, protect personal data, control spend, and publish certified metrics that BI tools and AI agents can trust.
People and AI agents can only use data they can find, understand and are allowed to access. Learn ownership, glossaries, classification tags and fine-grained access control with modern catalogs, and how they support data products and data mesh.
Personal and sensitive data must be protected wherever it flows. Learn to classify PII, apply dynamic masking and row filters, pseudonymise data and handle retention and erasure requests in lakehouse tables.
Compute for warehouses and lakehouses is often a data team's biggest bill. Learn to size and auto-suspend compute, tune table layout and queries, attribute spend with tags and billing tables, and apply FinOps practices.
AI agents and text-to-SQL tools give wrong answers when metric definitions are ambiguous. Define certified metrics once in a semantic layer, expose them and their lineage through MCP servers with guardrails, and prepare documented, quality-checked data for retrieval and embeddings.
Milestone: You can run a governed, cost-aware data platform whose certified metrics are safely consumable by BI tools and AI agents.
39 tools and platforms used across this path. Focus on concepts first — tools change, principles don’t.
Build these to prove your skills. Publish each one on GitHub with a clear README and architecture diagram.
Use Docker Compose to run PostgreSQL and DuckDB, load a public dataset with dlt into Parquet files, and model it into a small star schema with SQL. Publish the repository with a README, a Makefile and pre-commit checks.
Capture changes from a PostgreSQL database with Debezium, stream them through Apache Kafka and land them as Apache Iceberg tables that both Spark and DuckDB can query. Demonstrate inserts, updates and deletes flowing end to end with a replay test.
Build a dbt project with staging, intermediate and mart layers on Snowflake, Databricks or DuckDB, covered by unit tests, data tests and model contracts. Schedule ingestion and dbt runs with Airflow 3 assets and publish generated documentation.
Create a GitHub Actions workflow that lints SQL with SQLFluff, runs slim dbt CI against an ephemeral schema, validates data contracts and promotes changes to production with approval. Manage warehouse roles, grants and compute with Terraform.
Emit OpenLineage events from Airflow, Spark and dbt into Marquez, add Elementary freshness and anomaly monitors with alerts to chat, and define data SLOs for key tables. Run a simulated upstream schema break and write the incident postmortem.
Publish a data product with an ODCS contract, catalog entries in DataHub or OpenMetadata, PII tags with masking policies and certified metrics in the dbt Semantic Layer. Expose it to an AI assistant through an MCP server and evaluate the assistant's answers against the certified metrics.
Certifications that validate this path’s skills. Always confirm the current exam version on the official page before booking.
DevOpsSchool
Instructor-led programme covering DataOps end to end, from pipelines, CI/CD and IaC to observability, Databricks, Delta Lake and Unity Catalog governance.
Official pagedbt Labs (Fivetran + dbt Labs)
Validates modelling, testing, documentation and deployment with dbt, the core transformation skill in this path; the 2026 exam targets dbt 1.11.
Official pageDatabricks
The current exam guide has dedicated CI/CD and troubleshooting domains alongside ingestion, transformation, Lakeflow Jobs and governance; progress to the Data Engineer Professional exam afterwards.
Official pageSnowflake
Proves hands-on Snowflake skills in architecture, loading, transformation, performance and data protection; COF-C03 replaced COF-C02 in 2026.
Official pageAmazon Web Services
Covers ingestion, transformation, orchestration, data stores, operations and governance on AWS for teams whose platform runs there.
Official pageGoogle Cloud
Tests designing, building, operationalising and automating data processing systems on Google Cloud, including BigQuery-based platforms.
Official pageMicrosoft
The current Microsoft data engineering credential, covering ingestion, transformation, security and monitoring of analytics solutions in Microsoft Fabric.
Official pageRoles that hire for these skills.
Builds the automation around data pipelines, including CI/CD, testing, orchestration, monitoring and incident response, so data products ship quickly and reliably.
Turns raw data into clean, tested, documented models and metrics with dbt and SQL, bridging data engineering and analytics.
Runs the lakehouse or warehouse as an internal product: storage, compute, catalogs, access control, infrastructure as code and cost.
Applies SRE practice to data by defining freshness and quality SLOs, running observability and lineage tooling, and leading data incident postmortems.
Implements catalogs, classification, access policies, masking and retention so data is discoverable, compliant and safe to use for AI.
Data engineering is the work of building pipelines and data platforms. DataOps is the set of practices and automation that make that work reliable and fast to change: version control, automated testing, CI/CD, orchestration, observability and close collaboration with data consumers. Many DataOps engineers are data engineers with strong DevOps skills.
It borrows heavily from DevOps, but data adds problems that code alone does not have. Data changes independently of your code, so you must test the data as well as the logic, monitor freshness and quality in production, and track lineage and governance. The DataOps Manifesto also draws on lean manufacturing and statistical process control.
No. Learn one platform deeply, ideally the one your target employers use, and understand the other at a conceptual level. SQL, data modelling, dbt and open table formats such as Apache Iceberg transfer between them, and both platforms now read and write Iceberg tables.
Start with Apache Airflow because it is the most widely deployed orchestrator and appears in the most job postings, and Airflow 3 added assets and DAG versioning. Then try Dagster or Prefect to understand asset-centric and Python-native approaches, which many newer teams prefer.
Yes. dbt remains the standard way to manage SQL transformations as code, and the merged company has kept investing in open source, releasing the Rust-based Fusion runtime as dbt Core v2.0 (alpha) under Apache 2.0. The dbt Analytics Engineering Certification also remains current.
Pick the platform certification that matches your target stack: SnowPro Core, Databricks Data Engineer Associate, AWS Certified Data Engineer – Associate, Google Professional Data Engineer or Microsoft DP-700. Add the dbt Analytics Engineering Certification for transformation skills, and the DevOpsSchool DataOps Certified Professional for end-to-end DataOps practice.
At about 10 hours per week this path takes roughly 32 weeks, including portfolio projects. People who already know SQL and Python, or who come from DevOps, can often move faster through the first two stages.
Join DevOpsSchool instructor-led programs with hands-on labs, real-world projects and certification guidance.
Roadmap reviewed October 2026 · Last updated 26 Oct 2026 · JSON