Build, evaluate, serve, observe and secure LLM, RAG and agent applications in production at a sane cost.
A 2026 roadmap for engineers who run generative AI in production. You will select and self-host foundation models, build RAG on vector databases, evaluate and guard outputs, trace every call with OpenTelemetry, control spend with gateways and caching, and operate tool-using agents built on LangGraph and MCP.
LLMOps is the discipline of operating applications built on large language models: assistants, retrieval-augmented generation (RAG) systems, document pipelines and, increasingly, agents that plan and call tools. Unlike classical MLOps, most teams do not train their own models. The hard problems are choosing and serving foundation models, managing prompts and context, evaluating non-deterministic outputs, stopping prompt injection and data leaks, and keeping latency and token spend under control.
The field matured fast through 2025 and 2026. vLLM and SGLang now dominate self-hosted inference (Hugging Face put TGI into maintenance mode in December 2025), llm-d and the Gateway API Inference Extension bring KV-cache-aware routing to Kubernetes, and OpenTelemetry defines GenAI semantic conventions for tracing. The Model Context Protocol, now governed by the Linux Foundation's Agentic AI Foundation, is the standard way to connect agents to tools, and security teams work from the OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications.
This path assumes you can already ship software and ideally know MLOps basics such as registries, pipelines and monitoring. It focuses on operating LLM and agent workloads end to end. Pair it with the MLOps path for the classical model lifecycle, DevSecOps for application security, and Platform Engineering if you are building a shared GenAI platform for many teams.
7 stages · 26 topics · ~280 hours of focused learning. Work through the stages in order; optional topics are electives you can come back to.
Understand how LLMs behave, how to call them reliably through APIs, and how to choose the right model for a workload.
Learn the mechanics that drive operational behaviour: tokenisation, attention and context windows, sampling, reasoning models and embeddings. Cost, latency, hallucination and prompt-injection risk all follow from these basics.
Production code talks to models through APIs with real limits. Practise streaming, structured outputs with JSON Schema, tool calling, retries with backoff, timeouts and batch APIs across more than one provider so you are never locked in.
Choose models on evidence, not hype. Compare proprietary and open-weight models on your own task-specific evals plus public leaderboards, and weigh latency, price per token, context length, licence terms, data residency and the managed platforms that host them.
Milestone: You can explain tokens, context windows and sampling, call hosted and local models with structured outputs and tool calls, and defend a model choice with data.
Treat prompts and context as versioned production artifacts and learn the core application patterns and frameworks.
Design system prompts, examples and output schemas that are robust rather than clever, and manage the whole context window: instructions, retrieved documents, tool results and memory. Context engineering is now a bigger quality lever than wording tricks.
Hard-coded prompts make every wording change a redeploy and make regressions impossible to trace. Store prompts in a registry with versions and environment labels, link each version to traces and eval results, and roll back in seconds.
Know the patterns before the frameworks: prompt chaining, routing, parallelisation, orchestrator-workers and full agents. Then learn LangChain, LlamaIndex and DSPy well enough to judge when a framework helps and when plain code is easier to operate.
Milestone: Your application loads versioned prompts from a registry, logs which version served each request, and uses the simplest workflow or agent pattern that solves the problem.
Ground model answers in your own data with well-engineered ingestion, embeddings, vector search and retrieval quality techniques.
Retrieval quality is decided before any query runs. Choose embedding models with benchmarks and your own data, chunk documents along their structure, enrich chunks with metadata and context, and plan for re-embedding when models change.
Operate vector search like any production datastore: choose index types, tune recall against latency, filter on metadata, isolate tenants, back up and reindex. Start with pgvector if you already run PostgreSQL and move to Qdrant, Weaviate or Milvus when scale or features demand it.
Naive top-k retrieval fails on real questions. Combine keyword and dense search, rerank with cross-encoders, rewrite and decompose queries, use knowledge graphs where relationships matter, and force answers to cite their sources.
Most RAG bugs are ingestion bugs. Parse PDFs, tables and HTML reliably, index incrementally, deduplicate, redact PII, carry document permissions through to retrieval, and support deletion requests across the index.
Milestone: You run a RAG service with incremental ingestion, permission-aware hybrid retrieval and reranking, and you can show measurable retrieval quality.
Make quality and safety measurable: evaluation suites in CI, runtime guardrails, and a security programme built on the OWASP GenAI Top 10 lists.
Evaluation is the core LLMOps discipline. Build versioned test sets from real usage, combine deterministic checks with LLM-as-judge metrics calibrated against human labels, measure RAG faithfulness and context quality, and block regressions in CI.
Guardrails enforce policy at runtime around the model: validate inputs and outputs, keep conversations on topic, detect jailbreaks and prompt injection, redact sensitive data and enforce output schemas, all within a latency budget and failing closed.
Threat-model GenAI systems with the OWASP Top 10 for LLM Applications 2025 and the OWASP Top 10 for Agentic Applications, then test them continuously with automated red teaming. Cover indirect prompt injection through documents and tools, data leakage, excessive agency and the AI software supply chain.
Know the rules your GenAI system will be audited against: EU AI Act transparency and general-purpose AI obligations and the revised Digital Omnibus timeline, the NIST Generative AI Profile, and the documentation, logging and human-oversight evidence they expect.
Milestone: Every prompt, model or retrieval change runs an evaluation and red-team suite in CI, and the application enforces input and output guardrails mapped to OWASP risks.
Self-host open-weight models efficiently: modern inference engines, quantization and KV-cache techniques, and Kubernetes-native distributed serving.
vLLM is the most widely used open-source inference engine and a PyTorch Foundation project. Learn PagedAttention, continuous batching, prefix caching, parallelism options, multi-LoRA serving and its OpenAI-compatible server, and benchmark it properly.
Pick the right engine per workload: SGLang for high-throughput and structured generation, TensorRT-LLM and NVIDIA Dynamo for maximum NVIDIA performance and disaggregated prefill and decode, llama.cpp for CPUs and edge. Plan migrations away from TGI, which entered maintenance mode in December 2025.
Memory bandwidth, not compute, usually limits LLM serving. Size GPUs from first principles, quantize weights (FP8, INT4, AWQ, GPTQ) and the KV cache, offload or share KV cache across requests, use speculative decoding, and always re-run evals after optimising.
Run inference as a cloud-native workload: GPU node pools and Dynamic Resource Allocation, fast model loading, KServe for generative inference, llm-d and the Gateway API Inference Extension for prefix-cache-aware routing, and autoscaling on queue depth and KV-cache utilisation.
Milestone: You serve an open-weight model on Kubernetes with vLLM or SGLang behind KV-cache-aware routing, and you can quote its TTFT, throughput and cost per million tokens.
See inside every request, score quality in production, centralise model access through a gateway and engineer token spend down.
Trace each request across prompt rendering, retrieval, model calls and tool executions. Use the OpenTelemetry GenAI semantic conventions (gen_ai attributes, inference, retrieval, tool and agent spans) so data flows to Langfuse, Arize Phoenix or your existing backend without lock-in.
Offline evals cannot anticipate every user. Score sampled production traces with LLM judges, capture explicit and implicit user feedback, route hard cases to annotation queues, and turn incidents into new test cases. Watch for silent regressions when providers update models.
A gateway gives every team one OpenAI-compatible endpoint while the platform enforces virtual keys, token-based rate limits, budgets, fallbacks across providers, routing rules and audit logs. Because it holds every provider key, treat the gateway as a high-value asset with strict supply-chain controls.
Token spend scales with success, so engineer it. Use provider prompt caching, semantic caching, routing of easy requests to smaller models, batch APIs, output-length control and streaming, and track unit costs such as cost per conversation or per resolved task alongside TTFT SLOs.
Milestone: Every request is traced end to end with token and cost attribution, online evals score sampled traffic, and a gateway enforces per-team budgets, fallbacks and caching.
Adapt models when prompting and RAG are not enough, and build, connect and operate tool-using agents safely at scale.
Fine-tune only when evals prove prompting and RAG fall short, or when a smaller tuned model can replace a larger one. Curate data, run supervised fine-tuning with LoRA or QLoRA, consider preference tuning, evaluate against the baseline and serve adapters alongside base models.
Agents are long-running, stateful programs that make decisions. Model them as explicit graphs with LangGraph to get durable execution, checkpoints, human-in-the-loop interrupts, memory and controllable multi-agent designs instead of opaque loops.
MCP standardises how agents discover and call the tools, resources and prompts exposed by MCP servers. The 2026-07-28 specification makes every request stateless, which simplifies scaling servers behind load balancers, while HTTP authorization, the MCP Registry and extensions such as Tasks cover enterprise needs. Learn to build and operate MCP servers safely, and where the A2A protocol fits for agent-to-agent communication.
Agents fail differently from chatbots: loops, wrong tool calls and irreversible side effects. Evaluate full trajectories, sandbox code execution, scope credentials per tool, require approval for risky actions, cap steps and spend, trace agent spans and keep a kill switch and incident runbook.
Milestone: You can fine-tune and ship a LoRA adapter with evidence it beats the baseline, and run a governed MCP-connected agent with approvals, budgets, traces and agent-level evals.
45 tools and platforms used across this path. Focus on concepts first — tools change, principles don’t.
Build these to prove your skills. Publish each one on GitHub with a clear README and architecture diagram.
Build an evaluation harness that sends the same 50 domain questions to a hosted API model, an open-weight model running in Ollama and a model on a managed cloud platform. Score answers with exact checks plus a calibrated LLM judge and report quality, latency and cost per 1,000 requests.
Ingest a documentation set with Docling, index it in pgvector or Qdrant with hybrid search and reranking, version prompts in Langfuse and ship a chat API that cites its sources. Add a Ragas regression suite that runs in CI on every change.
Run promptfoo and garak red-team scans against your RAG app, map findings to the OWASP Top 10 for LLM Applications, then add NeMo Guardrails or Guardrails AI and report attack success rates before and after.
Deploy an open-weight model with vLLM on a GPU node pool, compare BF16 with FP8 or AWQ quantized variants, front it with llm-d or the Gateway API Inference Extension for prefix-cache-aware routing, and publish TTFT, throughput and cost-per-million-token benchmarks.
Put LiteLLM or Agent Router in front of two providers and your self-hosted model with per-team virtual keys, token budgets, fallbacks and semantic caching. Export OpenTelemetry GenAI traces to Langfuse or Phoenix and build a Grafana dashboard for cost and latency per team.
Build a LangGraph agent that uses two MCP servers, such as a ticketing system and a read-only database, with OAuth-scoped credentials, human approval for write actions, step and budget limits, trajectory-level evals and an incident runbook.
Certifications that validate this path’s skills. Always confirm the current exam version on the official page before booking.
Amazon Web Services
Validates putting foundation models into production on AWS: RAG and vector stores, agentic AI, prompt management, safety and governance, cost and performance optimisation, and troubleshooting.
Official pageDatabricks
Covers designing, building, deploying, governing, evaluating and monitoring RAG and LLM applications on Databricks.
Official pageMicrosoft
Successor to the retired AI-102 (Azure AI Engineer) and focused on generative AI, RAG and agentic solutions built with Microsoft Foundry.
Official pageNVIDIA
Entry-level validation of LLM fundamentals, experimentation, software development and trustworthy AI with NVIDIA tooling; NVIDIA's professional LLM and agentic AI exams were still listed as coming soon in October 2026.
Official pageAgentic AI Foundation / Linux Foundation
Launched in September 2026 as the first official MCP certification, covering MCP architecture, interactions, security and governance for agent tooling.
Official pageDevOpsSchool
DevOpsSchool's instructor-led AI certification programme spanning LLMs, prompt engineering, RAG, AI ethics and AI engineering with MLOps and model deployment, a guided companion to this path.
Official pageRoles that hire for these skills.
Runs the infrastructure behind GenAI applications: model serving, gateways, vector databases, tracing, evaluation pipelines and cost controls.
Builds LLM, RAG and agent features end to end, from prompt, context and retrieval design to evaluation suites and production rollout.
Provides a shared, governed GenAI platform: GPU inference clusters, an LLM gateway, approved models, MCP tool catalogs and golden paths for product teams.
Defines SLOs for latency, quality and cost, instruments traces and online evaluations, and leads incident response when model behaviour regresses.
Threat-models LLM and agent applications, runs red-team campaigns, designs guardrails and least-privilege tool access, and secures the AI supply chain.
MLOps manages the lifecycle of models you train yourself: data, features, training pipelines, registries and drift. LLMOps usually starts from a pretrained foundation model and focuses on prompts and context, retrieval, evaluation of open-ended outputs, guardrails, inference cost and agent behaviour. The infrastructure skills overlap heavily, which is why many teams run both on one platform.
Rarely at first. Most production gains come from better context, retrieval, evaluation and model choice. Fine-tune with LoRA or QLoRA when you need a consistent format or behaviour, or want a smaller, cheaper model to match a larger one, and only once an evaluation suite can prove the improvement.
Start with hosted APIs for speed, then self-host with vLLM or SGLang when data residency, sustained high volume, latency control or customisation justify running GPUs. Compare total cost per million tokens including idle GPU time and engineering effort, and keep an OpenAI-compatible gateway in front so you can switch either way.
If you already run PostgreSQL and have up to tens of millions of vectors, pgvector keeps operations simple. Qdrant, Weaviate and Milvus add built-in hybrid search, multi-tenancy, vector quantization and horizontal scaling for larger or more demanding workloads. Retrieval quality usually depends more on chunking, hybrid search and reranking than on the database itself.
Build a versioned test set from real questions with expected facts or reference answers, score outputs with deterministic checks plus LLM-as-judge metrics calibrated against human labels, and run the suite in CI on every prompt, model or retrieval change. In production, score sampled traces online and turn every incident into a new test case.
The Model Context Protocol is an open standard for connecting LLM applications and agents to tools, data and prompts through MCP servers. Since Anthropic donated it to the Linux Foundation's Agentic AI Foundation in December 2025 it has become vendor-neutral infrastructure, so platform teams now own MCP server deployment, authorization, auditing and defences against tool poisoning.
Cloud-aligned options are AWS Certified Generative AI Developer – Professional, Microsoft AI-103 (Azure AI Apps and Agents Developer Associate) and the Databricks Generative AI Engineer Associate. NVIDIA's NCA-GENL covers LLM fundamentals and the new MCPA validates Model Context Protocol knowledge. On Azure, Microsoft's AI-300 MLOps Engineer exam also covers GenAIOps.
Join DevOpsSchool instructor-led programs with hands-on labs, real-world projects and certification guidance.
Roadmap reviewed October 2026 · Last updated 26 Oct 2026 · JSON