Databricks Trainer

The mental model everything else depends on. What the lakehouse actually is at the file level, how the workspace, clusters, notebooks and jobs relate, and Spark's execution model — driver, executors, jobs, stages and tasks — plus the Databricks-specific optimisations layered on top of open-source Spark.

Topics: Lakehouse architecture: object storage plus a transaction log · Workspace, notebooks, repos, clusters, jobs and workflows · Spark architecture: driver, executors, jobs, stages, tasks, partitions · Databricks Runtime and its optimisations, including Photon · Cluster types: all-purpose, job, SQL warehouse, serverless · The key components in one picture: Delta Lake, Unity Catalog, MLflow, Databricks SQL · Reading the Spark UI: stages, tasks, shuffle and storage tabs · Medallion architecture: bronze, silver, gold as a working convention

• Assignments: (1) Run one query three ways and account for the difference in the Spark UI; (2) Map an existing data flow onto a medallion layout and mark the gaps
• Capstone: Produce an architecture note for a real pipeline with layer boundaries and table contracts

The module that pays for the course. Reading logical and physical plans, understanding what triggers a shuffle and what it costs, the join strategies Spark chooses between and how to influence that choice, and the diagnosis of skew, spill and small files from evidence rather than folklore.

Topics: Catalyst optimizer and Tungsten execution: what each contributes · Logical and physical plans, and reading EXPLAIN output · Partitioning strategies: repartition, coalesce, partition sizing · Shuffle mechanics and spark.sql.shuffle.partitions · Join strategies: broadcast hash, sort-merge, shuffle hash · Adaptive Query Execution and what it fixes automatically · Data skew: detection, salting and skew join handling · Memory management, spill and executor sizing · Caching and persistence: when it helps and when it hurts · Photon: which workloads benefit and which do not

• Assignments: (1) Take a skewed join from failing to passing and document the evidence trail; (2) Halve a job's runtime by changing only partitioning and join strategy
• Capstone: Deliver a tuning report on a real job with plan analysis and before-and-after cost

Delta as a transaction log rather than a file format. How ACID guarantees are produced and what concurrent writers do to each other, schema enforcement versus evolution, time travel and what retention it requires, then the maintenance operations that decide whether a table stays fast a year from now.

Topics: The Delta transaction log: commits, checkpoints, and how readers see a version · ACID guarantees, isolation levels and concurrent write conflicts · Schema enforcement, schema evolution and column mapping · Time travel: version and timestamp queries, RESTORE, retention · OPTIMIZE, file compaction and the small-file problem · Z-ordering and liquid clustering: what each solves · VACUUM, retention windows and the cost of getting them wrong · Deletion vectors and their effect on read and write paths · Table constraints, generated columns and identity columns · Deep and shallow clones for testing and recovery

• Assignments: (1) Force a concurrent-write conflict and resolve it correctly; (2) Take a table with thousands of small files and restore its read performance
• Capstone: Deliver a table maintenance policy with schedules, retention and measured impact

Getting data in without reprocessing everything. Auto Loader for incremental file discovery at scale, COPY INTO for simpler cases, MERGE for upserts and deletes, change data feed for propagating changes downstream, and Delta Live Tables where declarative pipelines with expectations fit better than hand-written jobs.

Topics: Auto Loader: file notification vs directory listing, schema inference and evolution · COPY INTO and idempotent batch loads · MERGE INTO: upserts, deletes and matched-clause performance · Change data feed: enabling it and consuming changes downstream · CDC patterns from operational databases into bronze · Slowly changing dimensions type 1 and type 2 in Delta · Delta Live Tables: declarative pipelines, expectations, quality metrics · Streaming tables and materialized views · Idempotency and exactly-once semantics in practice

• Assignments: (1) Build an Auto Loader pipeline that survives a schema change without failing; (2) Implement SCD type 2 with MERGE and prove correctness against test data
• Capstone: Deliver a bronze-to-silver CDC pipeline with data quality expectations enforced

Continuous processing with the guarantees actually on offer. The Structured Streaming model against the older DStreams approach, checkpoints and offsets, trigger modes, watermarks and late data, stateful operations, and the Kafka integration details that determine whether a stream recovers cleanly after a restart.

Topics: Structured Streaming model versus DStreams · Sources, sinks, checkpoints and offset management · Trigger modes: continuous, fixed interval, availableNow · Event time, watermarks and handling late-arriving data · Stateful aggregations, joins and state store sizing · Kafka integration: offsets, partitions, backpressure, maxOffsetsPerTrigger · Streaming into and out of Delta tables · Restart, replay and recovery semantics · Monitoring streams: metrics, lag and progress reporting

• Assignments: (1) Build a Kafka-to-Delta stream, kill it mid-batch and prove clean recovery; (2) Add a watermark and demonstrate its effect on late records
• Capstone: Deliver a production streaming pipeline with monitoring, alerting and a restart runbook

Turning notebooks into jobs a team can rely on. Multi-task workflows with dependency graphs, parameterisation, retries and timeouts, failure notification, job clusters versus shared compute, CI/CD from Git through Databricks Asset Bundles, and the environment promotion pattern that keeps production separate.

Topics: Workflows: multi-task jobs, dependencies, conditional and looping tasks · Parameters, task values and passing state between tasks · Retries, timeouts, failure handling and alerting · Job clusters vs all-purpose clusters and the cost difference · Scheduling, concurrency limits and queueing behaviour · Databricks Asset Bundles and infrastructure as code for jobs · Git integration, Repos and CI/CD promotion across workspaces · Testing pipelines: unit tests, data tests and staging data · Observability: system tables, job run history and lineage

• Assignments: (1) Convert a notebook chain into a parameterised multi-task workflow with retries; (2) Promote a pipeline dev to prod through Asset Bundles with no manual step
• Capstone: Deliver an ETL pipeline deployed entirely from Git with tests and alerting

Access control designed before the tables exist. The metastore and its three-level namespace, storage credentials and external locations, the grant model including row filters and column masks, lineage and audit, and the practical work of migrating a workspace-scoped Hive metastore estate onto Unity Catalog.

Topics: Metastore, catalogs, schemas and the three-level namespace · Storage credentials, external locations and managed vs external tables · Users, groups, service principals and identity federation · GRANT model, ownership and privilege inheritance · Row filters, column masks and dynamic views · Lineage capture, audit logs and system tables · Delta Sharing for sharing data outside the workspace · Migrating from Hive metastore to Unity Catalog · Data classification, tagging and access review practice

• Assignments: (1) Design a catalog and grant model for three teams with least privilege; (2) Apply a row filter and prove two users see different data from one table
• Capstone: Deliver a governance design with catalog layout, grants, masking and audit reporting

Model building on the same platform as the data. MLlib and SparkML pipelines for distributed training, feature engineering and the feature store, hyperparameter tuning with grid, random and Hyperopt search, and MLflow tracking that makes an experiment reproducible rather than a notebook nobody can rerun.

Topics: MLlib and SparkML: transformers, estimators and pipelines · Feature engineering at scale and the feature store · Training distributed models on Spark and when not to · Hyperparameter tuning: grid search, random search and Hyperopt · MLflow tracking: runs, parameters, metrics, artefacts and autologging · Comparing runs and reproducing an experiment months later · Model registry: versions, aliases and stage transitions · Model evaluation, validation and fairness checks · Working with single-node libraries alongside Spark

• Assignments: (1) Build and tune a classification pipeline with tracked experiments; (2) Reproduce a colleague's model result from MLflow alone
• Capstone: Deliver a registered, evaluated model with a documented reproducible training run

Getting models into production and keeping them honest. Serving endpoints and their scaling behaviour, batch and streaming inference against Delta tables, CI/CD for model promotion, monitoring for drift and quality regression, and the retraining trigger that closes the loop.

Topics: Model Serving endpoints: deployment, scaling, cold starts and cost · Batch and streaming inference patterns against Delta tables · CI/CD for models: automated testing, approval and promotion · Champion and challenger deployments and traffic splitting · Inference tables, monitoring and drift detection · Data and model quality alerts and retraining triggers · Lineage from feature to model to prediction under Unity Catalog · Cost and latency trade-offs in serving architecture

• Assignments: (1) Deploy a registered model to an endpoint and load-test its latency; (2) Detect drift on a served model and trigger a retraining workflow
• Capstone: Deliver an end-to-end MLOps flow from training through serving to monitored retraining

Operating the platform for a bill you can defend. Connecting to cloud storage and warehouses correctly, cluster policies that constrain what users can start, autoscaling and spot behaviour, SQL warehouse sizing, and cost attribution through tags and system tables so spend can be traced to a team.

Topics: Cloud storage integration: S3, ADLS Gen2 and GCS with proper credentials · Querying and federating external warehouses such as Redshift, BigQuery and Snowflake · Databricks Connect and external development workflows · Cluster policies, instance pools and constraining user choice · Autoscaling behaviour, spot and preemptible instances, and their failure modes · Serverless SQL warehouses: sizing, scaling and concurrency · Photon and instance family selection for cost per unit of work · Tagging, budget policies, system billing tables and cost attribution · Right-sizing large jobs and identifying the top ten spenders

• Assignments: (1) Write a cluster policy that prevents the three most expensive mistakes; (2) Attribute one month of spend to teams using tags and system tables
• Capstone: Deliver a cost-optimisation review with measured savings and enforced guardrails