ModelOps Trainer

Where the two disciplines genuinely differ and where the marketing blurs them. The portfolio view, the model lifecycle as a governance object, the roles that must be separate — developer, owner, validator, approver — and the honest assessment of which controls an organisation needs given its risk profile rather than because a framework lists them.

Topics: The full population of decision models: ML, rules, optimisation, statistical, vendor-embedded, LLM-based · What MLOps solves and what it leaves open · Lifecycle stages as governance gates rather than pipeline stages · Three lines of defence applied to models, and how it works in a small organisation · Separation between model development, ownership, validation and approval · Proportionality: why one process for every model fails in both directions

• Assignments: (1) List every model making decisions in one business area, including the ones outside the ML platform; (2) Map current roles onto developer, owner, validator and approver and mark the gaps
• Capstone: Write a target operating model for governance covering roles, gates and proportional controls

The artefact everything else depends on. What a model record must contain, how registration is enforced rather than requested, linking the inventory to registries and deployment systems so it stays current, and a risk-tiering method that produces different control sets and survives challenge from both engineers and auditors.

Topics: Model record contents: purpose, owner, data, method, downstream use, dependencies · Registration enforced at deployment rather than by policy request · Linking the inventory to MLflow, SageMaker or Vertex registries and to CI/CD · Risk tiering dimensions: materiality, autonomy, reversibility, exposure to individuals · Control sets per tier — what a tier-3 model is genuinely exempt from · Finding shadow models: scheduled notebooks, spreadsheets, vendor scoring · Keeping the inventory accurate: reconciliation and staleness detection

• Assignments: (1) Build an inventory schema and populate it for at least five real models; (2) Risk-tier those models and write the justification for each tier
• Capstone: Deliver a working inventory with automated registration and a reconciliation report

Reviewing a model you did not build. Conceptual soundness, data lineage and representativeness, outcome analysis, benchmarking against a simple alternative, sensitivity and stress testing — then writing it up as a document that records limitations honestly and states the conditions under which the model may be used.

Topics: Conceptual soundness: does the method suit the problem and the data · Data validation: lineage, leakage, representativeness, label quality · Benchmarking against a trivial baseline and a challenger · Sensitivity, stress and edge-case testing · Replication and reproducibility of the approved artefact · Model documentation and model cards: intended use, limitations, conditions of use · Validation findings, severity and remediation tracking

• Assignments: (1) Validate a supplied model end to end and produce findings ranked by severity; (2) Write a model card that states limitations a business user would actually understand
• Capstone: Produce a full validation report and approval recommendation for a material model

Getting an approved model into production without either rubber-stamping or gridlock. Approval workflow tied to risk tier, reproducing exactly what was approved, staged release patterns that gather evidence before full exposure, and the change-control question of what counts as a new model rather than an update.

Topics: Approval workflow by tier, with standard changes pre-approved · Artefact immutability: proving what runs is what was approved · Shadow scoring, champion-challenger and staged rollout · Traffic splitting, guardrail metrics and automated rollback · Retraining: when it is a change requiring approval and when it is not · Emergency changes and after-the-fact review · Serving patterns — batch, online, embedded — and their governance differences

• Assignments: (1) Implement a champion-challenger deployment with a defined promotion criterion; (2) Write a change policy distinguishing retrain, refit, and new-model cases
• Capstone: Ship a model through the full approval, shadow and staged rollout path with evidence at each gate

Knowing whether a model still works. Input drift and its measures, concept drift where the relationship itself moves, prediction stability, delayed ground truth and how to monitor when labels arrive months later, segment-level degradation hidden by healthy aggregates, and the thresholds that trigger revalidation rather than another dashboard.

Topics: Data drift measures: PSI, KS, KL divergence and their failure modes · Concept drift and the delayed-label problem · Prediction stability and score distribution monitoring · Business KPI monitoring: approval rates, loss rates, conversion, override rates · Segment-level monitoring and Simpson's paradox in production · Feature pipeline failures that look like drift · Thresholds, alert routing and the revalidation trigger · Human override and feedback data as a monitoring signal

• Assignments: (1) Instrument drift, stability and one business KPI for a deployed model; (2) Replay a shifted dataset and tune thresholds so the alert fires before the KPI moves
• Capstone: Build a monitoring pack with defined triggers that would catch degradation before the business does

The controls that are increasingly non-negotiable. Fairness metrics and the trade-offs between them, explainability methods and their genuine limits, human oversight designed to be meaningful rather than nominal, and mapping obligations from the EU AI Act, NIST AI RMF, ISO/IEC 42001 and model risk supervisory guidance onto controls you already run.

Topics: Fairness definitions, their incompatibility, and choosing one defensibly · Bias testing across protected and proxy attributes · Global and local explanation methods, and where SHAP and LIME mislead · Adverse action and right-to-explanation requirements in practice · Human-in-the-loop that is meaningful rather than a rubber stamp · EU AI Act risk tiers and documentation duties · NIST AI RMF and ISO/IEC 42001 mapped to existing controls · Model risk management guidance applied to machine learning

• Assignments: (1) Run a fairness assessment on a model and document the metric chosen and why; (2) Map one regulatory obligation onto specific controls, artefacts and owners
• Capstone: Deliver a compliance evidence pack for one model against a named regulatory obligation

The newest and oldest problems together. Applying model governance to LLM and retrieval systems where output is non-deterministic and the model is a third party's, then running the portfolio: review cadence, aggregate risk reporting, and the decommissioning process that most organisations discover they have never done.

Topics: Governing a model you did not train: vendor LLMs, versions and deprecation notices · Evaluation suites, regression tests and golden datasets for generative systems · Prompt and retrieval configuration as versioned, reviewed artefacts · Guardrails, output filtering and human escalation paths · Monitoring hallucination, refusal, toxicity and cost per interaction · Portfolio reporting: aggregate risk, coverage of controls, overdue validations · Model committee cadence and decision records · Decommissioning: dependency discovery, shadow running, archival of evidence

• Assignments: (1) Build an evaluation suite for an LLM feature and run it as a regression gate; (2) Plan the retirement of a model with unknown downstream consumers
• Capstone: Deliver a portfolio governance pack: inventory, control coverage, review cadence and a retirement plan