{ "schema": 1, "slug": "llmops", "title": "LLMOps", "full_name": "LLMOps: Operating LLM, RAG and Agent Applications", "category": "Data & AI", "tagline": "Build, evaluate, serve, observe and secure LLM, RAG and agent applications in production at a sane cost.", "summary": "A 2026 roadmap for engineers who run generative AI in production. You will select and self-host foundation models, build RAG on vector databases, evaluate and guard outputs, trace every call with OpenTelemetry, control spend with gateways and caching, and operate tool-using agents built on LangGraph and MCP.", "description": "LLMOps is the discipline of operating applications built on large language models: assistants, retrieval-augmented generation (RAG) systems, document pipelines and, increasingly, agents that plan and call tools. Unlike classical MLOps, most teams do not train their own models. The hard problems are choosing and serving foundation models, managing prompts and context, evaluating non-deterministic outputs, stopping prompt injection and data leaks, and keeping latency and token spend under control.\n\nThe field matured fast through 2025 and 2026. vLLM and SGLang now dominate self-hosted inference (Hugging Face put TGI into maintenance mode in December 2025), llm-d and the Gateway API Inference Extension bring KV-cache-aware routing to Kubernetes, and OpenTelemetry defines GenAI semantic conventions for tracing. The Model Context Protocol, now governed by the Linux Foundation's Agentic AI Foundation, is the standard way to connect agents to tools, and security teams work from the OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications.\n\nThis path assumes you can already ship software and ideally know MLOps basics such as registries, pipelines and monitoring. It focuses on operating LLM and agent workloads end to end. Pair it with the MLOps path for the classical model lifecycle, DevSecOps for application security, and Platform Engineering if you are building a shared GenAI platform for many teams.", "icon": "smart_toy", "accent": "#C026D3", "level": "Beginner → Expert", "duration_weeks": 28, "weekly_hours": 10, "audience": [ "Software, backend and full-stack engineers shipping LLM features or agents to production", "MLOps, DevOps and platform engineers asked to host models, gateways and vector databases", "SREs and observability engineers who now own the reliability and cost of GenAI services", "AI engineers and data scientists whose prototypes need to survive real users, real data and audits" ], "prerequisites": [ "Solid Python, including async code, HTTP APIs and packaging", "Docker and Git; Kubernetes basics for the serving and platform stages", "Comfort with REST APIs, JSON and SQL", "Recommended: the early stages of the MLOps path or equivalent experience with CI/CD, model registries and monitoring" ], "outcomes": [ "Select foundation models on quality, latency, cost, licence and data-residency criteria and back the choice with your own evals", "Version, test and roll out prompts and context templates like code using a prompt registry", "Build and tune a production RAG pipeline with hybrid search and reranking on pgvector, Qdrant, Weaviate or Milvus", "Gate every release with offline evaluation suites and red-team tests mapped to the OWASP Top 10 for LLM Applications", "Self-host open-weight models with vLLM or SGLang on Kubernetes using quantization, prefix caching and KV-cache-aware routing", "Trace prompts, retrievals, tool calls and token usage end to end with OpenTelemetry GenAI conventions, Langfuse or Arize Phoenix", "Enforce budgets, fallbacks and rate limits through an LLM gateway and cut spend with prompt caching, semantic caching and model routing", "Operate tool-using agents built with LangGraph and MCP using least-privilege access, human approval steps and agent-level evaluations" ], "trends": [ "Agents reached production: LangChain and LangGraph hit 1.0 in October 2025, and MCP (donated to the Agentic AI Foundation in December 2025, latest spec 2026-07-28) is the default way to connect agents to tools.", "Self-hosted inference consolidated on vLLM and SGLang after Hugging Face moved TGI to maintenance mode, and llm-d joined the CNCF Sandbox in March 2026 for Kubernetes-native distributed inference.", "Gateways became AI control planes: Envoy AI Gateway was renamed Agent Router and moved to the Agentic AI Foundation in September 2026, routing MCP tools as well as models and token budgets.", "Observability is standardising on OpenTelemetry GenAI semantic conventions (still Development status, now in their own repository) while tools consolidate: ClickHouse acquired Langfuse and OpenAI announced its acquisition of promptfoo in 2026.", "AI security moved from demos to incidents and frameworks: the OWASP Top 10 for Agentic Applications (December 2025) and the March 2026 compromise of the LiteLLM PyPI package made AI supply-chain hygiene a priority.", "Context engineering, evaluation-driven development and cost engineering (prompt caching, model routing, small fine-tuned models) are replacing ad hoc prompt tweaking as the core LLMOps skills." ], "roles": [ { "title": "LLMOps Engineer", "description": "Runs the infrastructure behind GenAI applications: model serving, gateways, vector databases, tracing, evaluation pipelines and cost controls." }, { "title": "AI Engineer", "description": "Builds LLM, RAG and agent features end to end, from prompt, context and retrieval design to evaluation suites and production rollout." }, { "title": "GenAI Platform Engineer", "description": "Provides a shared, governed GenAI platform: GPU inference clusters, an LLM gateway, approved models, MCP tool catalogs and golden paths for product teams." }, { "title": "AI Reliability Engineer", "description": "Defines SLOs for latency, quality and cost, instruments traces and online evaluations, and leads incident response when model behaviour regresses." }, { "title": "AI Security Engineer", "description": "Threat-models LLM and agent applications, runs red-team campaigns, designs guardrails and least-privilege tool access, and secures the AI supply chain." } ], "stages": [ { "title": "LLM Foundations and Model Selection", "level": "Beginner", "duration_weeks": 3, "description": "Understand how LLMs behave, how to call them reliably through APIs, and how to choose the right model for a workload.", "milestone": "You can explain tokens, context windows and sampling, call hosted and local models with structured outputs and tool calls, and defend a model choice with data.", "topics": [ { "title": "How LLMs Work for Engineers", "description": "Learn the mechanics that drive operational behaviour: tokenisation, attention and context windows, sampling, reasoning models and embeddings. Cost, latency, hallucination and prompt-injection risk all follow from these basics.", "est_hours": 10, "optional": false, "concepts": [ "transformer architecture at a high level", "tokens and tokenisers", "context windows and attention cost", "temperature and top-p sampling", "reasoning models and thinking budgets", "embeddings", "hallucination" ], "tools": [ "Hugging Face" ], "resources": [ { "title": "Hugging Face LLM Course", "url": "https://huggingface.co/learn/llm-course/chapter1/1", "type": "course", "free": true }, { "title": "The Illustrated Transformer (Jay Alammar)", "url": "https://jalammar.github.io/illustrated-transformer/", "type": "article", "free": true }, { "title": "Artificial Intelligence Training and Certification Course — DevOpsSchool", "url": "https://www.devopsschool.com/courses/ai/", "type": "course", "free": false } ] }, { "title": "Working with Model APIs", "description": "Production code talks to models through APIs with real limits. Practise streaming, structured outputs with JSON Schema, tool calling, retries with backoff, timeouts and batch APIs across more than one provider so you are never locked in.", "est_hours": 10, "optional": false, "concepts": [ "chat and messages APIs", "streaming responses", "structured outputs (JSON Schema)", "tool and function calling", "rate limits, retries and timeouts", "batch APIs", "multimodal inputs" ], "tools": [ "Python" ], "resources": [ { "title": "OpenAI API Documentation", "url": "https://developers.openai.com/api/docs", "type": "docs", "free": true }, { "title": "Claude API Documentation", "url": "https://platform.claude.com/docs/en/home", "type": "docs", "free": true }, { "title": "Azure OpenAI Training and Certification Course — DevOpsSchool", "url": "https://www.devopsschool.com/courses/openai/", "type": "course", "free": false } ] }, { "title": "Foundation Model Selection", "description": "Choose models on evidence, not hype. Compare proprietary and open-weight models on your own task-specific evals plus public leaderboards, and weigh latency, price per token, context length, licence terms, data residency and the managed platforms that host them.", "est_hours": 10, "optional": false, "concepts": [ "open-weight vs proprietary models", "public leaderboards vs task-specific evals", "price, latency and throughput trade-offs", "licences and acceptable-use terms", "data residency and privacy", "managed model platforms", "local testing of open-weight models" ], "tools": [ "Hugging Face", "Ollama", "Amazon Bedrock", "Microsoft Foundry" ], "resources": [ { "title": "Artificial Analysis: Model Quality, Speed and Price Comparisons", "url": "https://artificialanalysis.ai/", "type": "tool", "free": true }, { "title": "Stanford HELM (Holistic Evaluation of Language Models)", "url": "https://crfm.stanford.edu/helm/", "type": "tool", "free": true }, { "title": "Ollama Documentation", "url": "https://docs.ollama.com/", "type": "docs", "free": true } ] } ] }, { "title": "Prompts, Context and Application Patterns", "level": "Beginner", "duration_weeks": 3, "description": "Treat prompts and context as versioned production artifacts and learn the core application patterns and frameworks.", "milestone": "Your application loads versioned prompts from a registry, logs which version served each request, and uses the simplest workflow or agent pattern that solves the problem.", "topics": [ { "title": "Prompt and Context Engineering", "description": "Design system prompts, examples and output schemas that are robust rather than clever, and manage the whole context window: instructions, retrieved documents, tool results and memory. Context engineering is now a bigger quality lever than wording tricks.", "est_hours": 10, "optional": false, "concepts": [ "system prompts and roles", "few-shot examples", "output schemas and validation", "context window budgeting", "long-context vs retrieval", "memory and conversation state", "prompt templates" ], "tools": [], "resources": [ { "title": "Prompt Engineering Overview (Claude Docs)", "url": "https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview", "type": "docs", "free": true }, { "title": "Effective Context Engineering for AI Agents (Anthropic)", "url": "https://www.anthropic.com/research/effective-context-engineering-for-ai-agents", "type": "article", "free": true }, { "title": "Prompt Engineering Training and Certification Course — DevOpsSchool", "url": "https://www.devopsschool.com/courses/prompt-engineering/", "type": "course", "free": false } ] }, { "title": "Prompt Management and Versioning", "description": "Hard-coded prompts make every wording change a redeploy and make regressions impossible to trace. Store prompts in a registry with versions and environment labels, link each version to traces and eval results, and roll back in seconds.", "est_hours": 10, "optional": false, "concepts": [ "prompts as versioned artifacts", "labels such as production and staging", "decoupling prompt releases from code deploys", "linking prompt versions to traces", "A/B testing prompts", "rollback" ], "tools": [ "Langfuse", "MLflow" ], "resources": [ { "title": "Langfuse Prompt Management", "url": "https://langfuse.com/docs/prompt-management/overview", "type": "docs", "free": true }, { "title": "MLflow Prompt Registry", "url": "https://mlflow.org/docs/latest/genai/prompt-registry/", "type": "docs", "free": true } ] }, { "title": "LLM Application Frameworks and Patterns", "description": "Know the patterns before the frameworks: prompt chaining, routing, parallelisation, orchestrator-workers and full agents. Then learn LangChain, LlamaIndex and DSPy well enough to judge when a framework helps and when plain code is easier to operate.", "est_hours": 10, "optional": false, "concepts": [ "workflows vs agents", "prompt chaining and routing", "orchestrator-worker pattern", "LangChain 1.0 agents and middleware", "LlamaIndex for data-centric apps", "DSPy programmatic prompting", "framework-free implementations" ], "tools": [ "LangChain", "LlamaIndex", "DSPy" ], "resources": [ { "title": "Building Effective Agents (Anthropic)", "url": "https://www.anthropic.com/research/building-effective-agents", "type": "article", "free": true }, { "title": "LangChain Documentation", "url": "https://docs.langchain.com/oss/python/langchain/overview", "type": "docs", "free": true }, { "title": "LlamaIndex Framework Documentation", "url": "https://developers.llamaindex.ai/python/framework/", "type": "docs", "free": true } ] } ] }, { "title": "RAG and Knowledge Pipelines", "level": "Intermediate", "duration_weeks": 4, "description": "Ground model answers in your own data with well-engineered ingestion, embeddings, vector search and retrieval quality techniques.", "milestone": "You run a RAG service with incremental ingestion, permission-aware hybrid retrieval and reranking, and you can show measurable retrieval quality.", "topics": [ { "title": "Embeddings and Chunking", "description": "Retrieval quality is decided before any query runs. Choose embedding models with benchmarks and your own data, chunk documents along their structure, enrich chunks with metadata and context, and plan for re-embedding when models change.", "est_hours": 8, "optional": false, "concepts": [ "embedding models and dimensions", "MTEB benchmark", "chunk size and overlap", "structure-aware chunking", "metadata enrichment", "contextual retrieval", "re-embedding strategy" ], "tools": [ "Hugging Face" ], "resources": [ { "title": "Sentence Transformers Documentation", "url": "https://sbert.net/", "type": "docs", "free": true }, { "title": "MTEB Embedding Leaderboard", "url": "https://huggingface.co/spaces/mteb/leaderboard", "type": "tool", "free": true }, { "title": "Introducing Contextual Retrieval (Anthropic)", "url": "https://www.anthropic.com/research/contextual-retrieval", "type": "article", "free": true } ] }, { "title": "Vector Databases", "description": "Operate vector search like any production datastore: choose index types, tune recall against latency, filter on metadata, isolate tenants, back up and reindex. Start with pgvector if you already run PostgreSQL and move to Qdrant, Weaviate or Milvus when scale or features demand it.", "est_hours": 12, "optional": false, "concepts": [ "approximate nearest neighbour indexes (HNSW, IVF)", "distance metrics", "recall vs latency tuning", "metadata filtering", "multi-tenancy", "vector quantization", "backup and reindexing" ], "tools": [ "pgvector", "PostgreSQL", "Qdrant", "Weaviate", "Milvus" ], "resources": [ { "title": "pgvector (official repository and docs)", "url": "https://github.com/pgvector/pgvector", "type": "docs", "free": true }, { "title": "Qdrant Documentation", "url": "https://qdrant.tech/documentation/", "type": "docs", "free": true }, { "title": "PostgreSQL Training and Certification Course — DevOpsSchool", "url": "https://www.devopsschool.com/courses/postgresql/", "type": "course", "free": false } ] }, { "title": "Advanced Retrieval and Grounding", "description": "Naive top-k retrieval fails on real questions. Combine keyword and dense search, rerank with cross-encoders, rewrite and decompose queries, use knowledge graphs where relationships matter, and force answers to cite their sources.", "est_hours": 10, "optional": false, "concepts": [ "hybrid search (BM25 plus dense)", "reranking with cross-encoders", "query rewriting and decomposition", "GraphRAG", "agentic retrieval", "citations and grounding checks", "freshness and recency" ], "tools": [ "LlamaIndex", "Weaviate" ], "resources": [ { "title": "Patterns for Building LLM-based Systems and Products (Eugene Yan)", "url": "https://eugeneyan.com/writing/llm-patterns/", "type": "article", "free": true }, { "title": "Microsoft GraphRAG Documentation", "url": "https://microsoft.github.io/graphrag/", "type": "docs", "free": true }, { "title": "Weaviate Documentation", "url": "https://docs.weaviate.io/", "type": "docs", "free": true } ] }, { "title": "Document Ingestion and Data Governance for RAG", "description": "Most RAG bugs are ingestion bugs. Parse PDFs, tables and HTML reliably, index incrementally, deduplicate, redact PII, carry document permissions through to retrieval, and support deletion requests across the index.", "est_hours": 10, "optional": false, "concepts": [ "document parsing and layout extraction", "incremental indexing and change detection", "deduplication", "PII detection and redaction", "permission-aware retrieval", "lineage and deletion requests", "scheduled ingestion pipelines" ], "tools": [ "Docling", "Apache Airflow", "Milvus" ], "resources": [ { "title": "Docling Documentation", "url": "https://docling-project.github.io/docling/", "type": "docs", "free": true }, { "title": "Milvus Documentation", "url": "https://milvus.io/docs", "type": "docs", "free": true }, { "title": "Apache Airflow Documentation", "url": "https://airflow.apache.org/docs/apache-airflow/stable/index.html", "type": "docs", "free": true } ] } ] }, { "title": "Evaluation, Guardrails and Security", "level": "Intermediate", "duration_weeks": 4, "description": "Make quality and safety measurable: evaluation suites in CI, runtime guardrails, and a security programme built on the OWASP GenAI Top 10 lists.", "milestone": "Every prompt, model or retrieval change runs an evaluation and red-team suite in CI, and the application enforces input and output guardrails mapped to OWASP risks.", "topics": [ { "title": "LLM Evaluation and Testing", "description": "Evaluation is the core LLMOps discipline. Build versioned test sets from real usage, combine deterministic checks with LLM-as-judge metrics calibrated against human labels, measure RAG faithfulness and context quality, and block regressions in CI.", "est_hours": 14, "optional": false, "concepts": [ "golden datasets from real traffic", "error analysis", "deterministic assertions", "LLM-as-judge and judge calibration", "RAG metrics: faithfulness, context precision and recall", "pairwise comparisons", "regression gates in CI" ], "tools": [ "Ragas", "promptfoo", "DeepEval" ], "resources": [ { "title": "Your AI Product Needs Evals (Hamel Husain)", "url": "https://hamel.dev/blog/posts/evals/", "type": "article", "free": true }, { "title": "Ragas Documentation", "url": "https://docs.ragas.io/", "type": "docs", "free": true }, { "title": "promptfoo Documentation", "url": "https://www.promptfoo.dev/docs/intro/", "type": "docs", "free": true } ] }, { "title": "Guardrails and Safety Controls", "description": "Guardrails enforce policy at runtime around the model: validate inputs and outputs, keep conversations on topic, detect jailbreaks and prompt injection, redact sensitive data and enforce output schemas, all within a latency budget and failing closed.", "est_hours": 10, "optional": false, "concepts": [ "input and output rails", "topical and dialogue rails", "jailbreak and prompt-injection detection", "PII detection and redaction", "safety classifier models", "schema validation and re-asking", "latency and fail-closed design" ], "tools": [ "NeMo Guardrails", "Guardrails AI" ], "resources": [ { "title": "NVIDIA NeMo Guardrails Documentation", "url": "https://docs.nvidia.com/nemo/guardrails/latest/index.html", "type": "docs", "free": true }, { "title": "Guardrails AI Documentation", "url": "https://www.guardrailsai.com/docs", "type": "docs", "free": true } ] }, { "title": "LLM and Agent Application Security", "description": "Threat-model GenAI systems with the OWASP Top 10 for LLM Applications 2025 and the OWASP Top 10 for Agentic Applications, then test them continuously with automated red teaming. Cover indirect prompt injection through documents and tools, data leakage, excessive agency and the AI software supply chain.", "est_hours": 12, "optional": false, "concepts": [ "prompt injection (direct and indirect)", "sensitive information disclosure", "excessive agency", "system prompt leakage", "vector and embedding weaknesses", "unbounded consumption", "automated red teaming", "AI supply-chain security" ], "tools": [ "promptfoo", "garak" ], "resources": [ { "title": "OWASP Top 10 for LLM Applications 2025", "url": "https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/", "type": "docs", "free": true }, { "title": "OWASP Top 10 for Agentic Applications", "url": "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/", "type": "docs", "free": true }, { "title": "OWASP Training and Certification Course — DevOpsSchool", "url": "https://www.devopsschool.com/courses/owasp/", "type": "course", "free": false } ] }, { "title": "AI Governance and Compliance", "description": "Know the rules your GenAI system will be audited against: EU AI Act transparency and general-purpose AI obligations and the revised Digital Omnibus timeline, the NIST Generative AI Profile, and the documentation, logging and human-oversight evidence they expect.", "est_hours": 4, "optional": true, "concepts": [ "EU AI Act transparency and GPAI obligations", "Digital Omnibus timeline changes", "NIST AI 600-1 Generative AI Profile", "AI-generated content labelling", "audit logging and retention", "human oversight" ], "tools": [], "resources": [ { "title": "NIST AI 600-1: Generative AI Profile", "url": "https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence", "type": "paper", "free": true }, { "title": "EU AI Act: Regulatory Framework for AI (European Commission)", "url": "https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai", "type": "docs", "free": true } ] } ] }, { "title": "Inference Serving and Optimization", "level": "Advanced", "duration_weeks": 5, "description": "Self-host open-weight models efficiently: modern inference engines, quantization and KV-cache techniques, and Kubernetes-native distributed serving.", "milestone": "You serve an open-weight model on Kubernetes with vLLM or SGLang behind KV-cache-aware routing, and you can quote its TTFT, throughput and cost per million tokens.", "topics": [ { "title": "Self-Hosting Models with vLLM", "description": "vLLM is the most widely used open-source inference engine and a PyTorch Foundation project. Learn PagedAttention, continuous batching, prefix caching, parallelism options, multi-LoRA serving and its OpenAI-compatible server, and benchmark it properly.", "est_hours": 14, "optional": false, "concepts": [ "PagedAttention", "continuous batching", "automatic prefix caching", "tensor and pipeline parallelism", "OpenAI-compatible server", "multi-LoRA serving", "TTFT, inter-token latency and throughput" ], "tools": [ "vLLM", "Docker" ], "resources": [ { "title": "vLLM Documentation", "url": "https://docs.vllm.ai/en/latest/", "type": "docs", "free": true }, { "title": "Efficient Memory Management for LLM Serving with PagedAttention (paper)", "url": "https://arxiv.org/abs/2309.06180", "type": "paper", "free": true } ] }, { "title": "Inference Engines Compared", "description": "Pick the right engine per workload: SGLang for high-throughput and structured generation, TensorRT-LLM and NVIDIA Dynamo for maximum NVIDIA performance and disaggregated prefill and decode, llama.cpp for CPUs and edge. Plan migrations away from TGI, which entered maintenance mode in December 2025.", "est_hours": 10, "optional": false, "concepts": [ "SGLang and RadixAttention", "TensorRT-LLM engines", "disaggregated prefill and decode", "llama.cpp and GGUF", "TGI maintenance mode and migration", "managed endpoints vs self-hosting break-even" ], "tools": [ "SGLang", "NVIDIA Dynamo", "TensorRT-LLM", "llama.cpp" ], "resources": [ { "title": "SGLang Documentation", "url": "https://docs.sglang.io/", "type": "docs", "free": true }, { "title": "NVIDIA Dynamo Documentation", "url": "https://docs.nvidia.com/dynamo/latest/", "type": "docs", "free": true }, { "title": "Text Generation Inference Documentation (maintenance-mode notice)", "url": "https://huggingface.co/docs/text-generation-inference/index", "type": "docs", "free": true } ] }, { "title": "Quantization, KV Cache and Speculative Decoding", "description": "Memory bandwidth, not compute, usually limits LLM serving. Size GPUs from first principles, quantize weights (FP8, INT4, AWQ, GPTQ) and the KV cache, offload or share KV cache across requests, use speculative decoding, and always re-run evals after optimising.", "est_hours": 12, "optional": false, "concepts": [ "GPU memory math for weights and KV cache", "FP8 and INT4 weight quantization", "AWQ and GPTQ", "KV cache quantization and offloading", "speculative decoding", "quality regression checks after quantization" ], "tools": [ "LLM Compressor", "vLLM" ], "resources": [ { "title": "vLLM Quantization Guide", "url": "https://docs.vllm.ai/en/latest/features/quantization/", "type": "docs", "free": true }, { "title": "LLM Compressor Documentation", "url": "https://docs.vllm.ai/projects/llm-compressor/en/latest/", "type": "docs", "free": true }, { "title": "Hugging Face Transformers Quantization Overview", "url": "https://huggingface.co/docs/transformers/main/en/quantization/overview", "type": "docs", "free": true } ] }, { "title": "LLM Serving on Kubernetes", "description": "Run inference as a cloud-native workload: GPU node pools and Dynamic Resource Allocation, fast model loading, KServe for generative inference, llm-d and the Gateway API Inference Extension for prefix-cache-aware routing, and autoscaling on queue depth and KV-cache utilisation.", "est_hours": 14, "optional": false, "concepts": [ "GPU node pools and Dynamic Resource Allocation", "model weight caching and fast startup", "KServe generative inference", "Gateway API Inference Extension (InferencePool, endpoint picker)", "llm-d prefix-cache-aware scheduling", "autoscaling on queue depth and KV-cache usage" ], "tools": [ "Kubernetes", "KServe", "llm-d" ], "resources": [ { "title": "llm-d Documentation", "url": "https://llm-d.ai/", "type": "docs", "free": true }, { "title": "Gateway API Inference Extension", "url": "https://gateway-api-inference-extension.sigs.k8s.io/", "type": "docs", "free": true }, { "title": "KServe Generative Inference Overview", "url": "https://kserve.github.io/website/docs/model-serving/generative-inference/overview", "type": "docs", "free": true } ] } ] }, { "title": "Observability, Gateways and Cost Control", "level": "Advanced", "duration_weeks": 4, "description": "See inside every request, score quality in production, centralise model access through a gateway and engineer token spend down.", "milestone": "Every request is traced end to end with token and cost attribution, online evals score sampled traffic, and a gateway enforces per-team budgets, fallbacks and caching.", "topics": [ { "title": "LLM Observability and Tracing", "description": "Trace each request across prompt rendering, retrieval, model calls and tool executions. Use the OpenTelemetry GenAI semantic conventions (gen_ai attributes, inference, retrieval, tool and agent spans) so data flows to Langfuse, Arize Phoenix or your existing backend without lock-in.", "est_hours": 12, "optional": false, "concepts": [ "OpenTelemetry GenAI semantic conventions", "spans for inference, retrieval, tools and agents", "token usage and cost attribution", "sessions and user context", "PII handling in traces", "sampling strategies", "dashboards for latency and cost" ], "tools": [ "OpenTelemetry", "Langfuse", "Arize Phoenix", "Grafana" ], "resources": [ { "title": "OpenTelemetry Semantic Conventions for GenAI (official repository)", "url": "https://github.com/open-telemetry/semantic-conventions-genai", "type": "docs", "free": true }, { "title": "Langfuse Observability Documentation", "url": "https://langfuse.com/docs/observability/overview", "type": "docs", "free": true }, { "title": "Arize Phoenix Documentation", "url": "https://arize.com/docs/phoenix", "type": "docs", "free": true } ] }, { "title": "Online Evaluation and Feedback Loops", "description": "Offline evals cannot anticipate every user. Score sampled production traces with LLM judges, capture explicit and implicit user feedback, route hard cases to annotation queues, and turn incidents into new test cases. Watch for silent regressions when providers update models.", "est_hours": 8, "optional": false, "concepts": [ "online LLM-as-judge scoring", "explicit and implicit feedback", "annotation queues", "turning incidents into test cases", "quality SLOs", "provider model updates and regressions" ], "tools": [ "Langfuse", "MLflow" ], "resources": [ { "title": "Langfuse Evaluation Documentation", "url": "https://langfuse.com/docs/evaluation/overview", "type": "docs", "free": true }, { "title": "MLflow for GenAI (tracing and evaluation)", "url": "https://mlflow.org/docs/latest/genai/", "type": "docs", "free": true } ] }, { "title": "LLM Gateways and Routing", "description": "A gateway gives every team one OpenAI-compatible endpoint while the platform enforces virtual keys, token-based rate limits, budgets, fallbacks across providers, routing rules and audit logs. Because it holds every provider key, treat the gateway as a high-value asset with strict supply-chain controls.", "est_hours": 10, "optional": false, "concepts": [ "unified OpenAI-compatible API", "virtual keys and per-team budgets", "token-based rate limiting", "provider fallbacks and retries", "model routing rules", "MCP tool routing", "gateway credential and dependency security" ], "tools": [ "LiteLLM", "Agent Router" ], "resources": [ { "title": "LiteLLM Documentation", "url": "https://docs.litellm.ai/", "type": "docs", "free": true }, { "title": "Agent Router (formerly Envoy AI Gateway) Documentation", "url": "https://theagentrouter.ai/docs/", "type": "docs", "free": true }, { "title": "LiteLLM Security Update: March 2026 Supply Chain Incident", "url": "https://docs.litellm.ai/blog/security-update-march-2026", "type": "article", "free": true } ] }, { "title": "Cost and Latency Engineering", "description": "Token spend scales with success, so engineer it. Use provider prompt caching, semantic caching, routing of easy requests to smaller models, batch APIs, output-length control and streaming, and track unit costs such as cost per conversation or per resolved task alongside TTFT SLOs.", "est_hours": 10, "optional": false, "concepts": [ "token economics and unit costs", "provider prompt caching", "semantic caching", "model cascades and routing", "batch processing", "output length control", "TTFT and latency SLOs", "GPU utilisation and right-sizing" ], "tools": [ "Redis", "OpenCost" ], "resources": [ { "title": "Prompt Caching (Claude Docs)", "url": "https://platform.claude.com/docs/en/build-with-claude/prompt-caching", "type": "docs", "free": true }, { "title": "Prompt Caching Guide (OpenAI)", "url": "https://developers.openai.com/api/docs/guides/prompt-caching", "type": "docs", "free": true }, { "title": "FinOps Training and Certification Course — DevOpsSchool", "url": "https://www.devopsschool.com/courses/finops/", "type": "course", "free": false } ] } ] }, { "title": "Fine-Tuning, Agents and Production GenAI Platforms", "level": "Expert", "duration_weeks": 5, "description": "Adapt models when prompting and RAG are not enough, and build, connect and operate tool-using agents safely at scale.", "milestone": "You can fine-tune and ship a LoRA adapter with evidence it beats the baseline, and run a governed MCP-connected agent with approvals, budgets, traces and agent-level evals.", "topics": [ { "title": "Fine-Tuning with LoRA and QLoRA", "description": "Fine-tune only when evals prove prompting and RAG fall short, or when a smaller tuned model can replace a larger one. Curate data, run supervised fine-tuning with LoRA or QLoRA, consider preference tuning, evaluate against the baseline and serve adapters alongside base models.", "est_hours": 14, "optional": false, "concepts": [ "when to fine-tune vs prompt vs RAG", "dataset curation and synthetic data", "supervised fine-tuning", "LoRA and QLoRA", "preference tuning (DPO)", "distillation into smaller models", "adapter registry and serving" ], "tools": [ "Hugging Face", "Unsloth", "Axolotl" ], "resources": [ { "title": "Hugging Face PEFT Documentation", "url": "https://huggingface.co/docs/peft/index", "type": "docs", "free": true }, { "title": "Hugging Face TRL Documentation", "url": "https://huggingface.co/docs/trl/index", "type": "docs", "free": true }, { "title": "QLoRA: Efficient Finetuning of Quantized LLMs (paper)", "url": "https://arxiv.org/abs/2305.14314", "type": "paper", "free": true } ] }, { "title": "Building Agents with LangGraph", "description": "Agents are long-running, stateful programs that make decisions. Model them as explicit graphs with LangGraph to get durable execution, checkpoints, human-in-the-loop interrupts, memory and controllable multi-agent designs instead of opaque loops.", "est_hours": 12, "optional": false, "concepts": [ "the agent loop and tool calling", "graphs, state and reducers", "durable execution and checkpoints", "human-in-the-loop interrupts", "short- and long-term memory", "multi-agent patterns", "timeouts and error handling" ], "tools": [ "LangGraph" ], "resources": [ { "title": "LangGraph Documentation", "url": "https://docs.langchain.com/oss/python/langgraph/overview", "type": "docs", "free": true } ] }, { "title": "Model Context Protocol and Agent Interoperability", "description": "MCP standardises how agents discover and call the tools, resources and prompts exposed by MCP servers. The 2026-07-28 specification makes every request stateless, which simplifies scaling servers behind load balancers, while HTTP authorization, the MCP Registry and extensions such as Tasks cover enterprise needs. Learn to build and operate MCP servers safely, and where the A2A protocol fits for agent-to-agent communication.", "est_hours": 10, "optional": false, "concepts": [ "MCP hosts, clients and servers", "tools, resources and prompts", "stateless requests (2026-07-28 spec)", "stdio vs HTTP transports and authorization", "Tasks extension for long-running work", "MCP Registry and tool catalogs", "tool poisoning and confused-deputy risks", "A2A agent-to-agent protocol" ], "tools": [ "Model Context Protocol", "A2A Protocol" ], "resources": [ { "title": "Model Context Protocol Documentation", "url": "https://modelcontextprotocol.io/", "type": "docs", "free": true }, { "title": "MCP Specification (latest)", "url": "https://modelcontextprotocol.io/specification/latest", "type": "docs", "free": true }, { "title": "A2A Protocol Documentation", "url": "https://a2a-protocol.org/", "type": "docs", "free": true } ] }, { "title": "Operating Agents in Production", "description": "Agents fail differently from chatbots: loops, wrong tool calls and irreversible side effects. Evaluate full trajectories, sandbox code execution, scope credentials per tool, require approval for risky actions, cap steps and spend, trace agent spans and keep a kill switch and incident runbook.", "est_hours": 14, "optional": false, "concepts": [ "task success and trajectory evals", "tool-call accuracy", "sandboxed code execution", "least-privilege, per-tool credentials", "approval gates for side effects", "step, time and budget limits", "agent spans and incident response" ], "tools": [ "LangGraph", "OpenTelemetry" ], "resources": [ { "title": "Demystifying Evals for AI Agents (Anthropic)", "url": "https://www.anthropic.com/research/demystifying-evals-for-ai-agents", "type": "article", "free": true }, { "title": "OpenTelemetry GenAI Agent Spans", "url": "https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md", "type": "docs", "free": true }, { "title": "OWASP Top 10 for Agentic Applications", "url": "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/", "type": "docs", "free": true } ] } ] } ], "projects": [ { "title": "Model Bake-Off Harness", "description": "Build an evaluation harness that sends the same 50 domain questions to a hosted API model, an open-weight model running in Ollama and a model on a managed cloud platform. Score answers with exact checks plus a calibrated LLM judge and report quality, latency and cost per 1,000 requests.", "difficulty": "Beginner", "skills": [ "model APIs", "Ollama", "LLM-as-judge", "cost analysis" ] }, { "title": "Production RAG over Internal Docs", "description": "Ingest a documentation set with Docling, index it in pgvector or Qdrant with hybrid search and reranking, version prompts in Langfuse and ship a chat API that cites its sources. Add a Ragas regression suite that runs in CI on every change.", "difficulty": "Intermediate", "skills": [ "Docling", "pgvector", "Qdrant", "Langfuse", "Ragas", "CI" ] }, { "title": "Red-Team and Guardrail a Chatbot", "description": "Run promptfoo and garak red-team scans against your RAG app, map findings to the OWASP Top 10 for LLM Applications, then add NeMo Guardrails or Guardrails AI and report attack success rates before and after.", "difficulty": "Intermediate", "skills": [ "promptfoo", "garak", "NeMo Guardrails", "OWASP LLM Top 10", "prompt injection" ] }, { "title": "Self-Hosted Inference Platform on Kubernetes", "description": "Deploy an open-weight model with vLLM on a GPU node pool, compare BF16 with FP8 or AWQ quantized variants, front it with llm-d or the Gateway API Inference Extension for prefix-cache-aware routing, and publish TTFT, throughput and cost-per-million-token benchmarks.", "difficulty": "Advanced", "skills": [ "vLLM", "Kubernetes", "llm-d", "quantization", "benchmarking" ] }, { "title": "LLM Gateway with Tracing and Budgets", "description": "Put LiteLLM or Agent Router in front of two providers and your self-hosted model with per-team virtual keys, token budgets, fallbacks and semantic caching. Export OpenTelemetry GenAI traces to Langfuse or Phoenix and build a Grafana dashboard for cost and latency per team.", "difficulty": "Advanced", "skills": [ "LiteLLM", "Agent Router", "OpenTelemetry", "Langfuse", "Redis", "Grafana" ] }, { "title": "Governed MCP Agent", "description": "Build a LangGraph agent that uses two MCP servers, such as a ticketing system and a read-only database, with OAuth-scoped credentials, human approval for write actions, step and budget limits, trajectory-level evals and an incident runbook.", "difficulty": "Advanced", "skills": [ "LangGraph", "Model Context Protocol", "OAuth", "agent evaluation", "OWASP Agentic Top 10" ] } ], "certifications": [ { "name": "AWS Certified Generative AI Developer – Professional (AIP-C01)", "provider": "Amazon Web Services", "url": "https://aws.amazon.com/certification/certified-generative-ai-developer-professional/", "level": "Professional", "note": "Validates putting foundation models into production on AWS: RAG and vector stores, agentic AI, prompt management, safety and governance, cost and performance optimisation, and troubleshooting." }, { "name": "Databricks Certified Generative AI Engineer Associate", "provider": "Databricks", "url": "https://www.databricks.com/learn/certification/genai-engineer-associate", "level": "Associate", "note": "Covers designing, building, deploying, governing, evaluating and monitoring RAG and LLM applications on Databricks." }, { "name": "Microsoft Certified: Azure AI Apps and Agents Developer Associate (AI-103)", "provider": "Microsoft", "url": "https://learn.microsoft.com/en-us/credentials/certifications/azure-ai-apps-and-agents-developer-associate/", "level": "Associate", "note": "Successor to the retired AI-102 (Azure AI Engineer) and focused on generative AI, RAG and agentic solutions built with Microsoft Foundry." }, { "name": "NVIDIA-Certified Associate: Generative AI LLMs (NCA-GENL)", "provider": "NVIDIA", "url": "https://www.nvidia.com/en-us/learn/certification/generative-ai-llm-associate/", "level": "Associate", "note": "Entry-level validation of LLM fundamentals, experimentation, software development and trustworthy AI with NVIDIA tooling; NVIDIA's professional LLM and agentic AI exams were still listed as coming soon in October 2026." }, { "name": "Model Context Protocol Associate (MCPA)", "provider": "Agentic AI Foundation / Linux Foundation", "url": "https://training.linuxfoundation.org/certification/model-context-protocol-associate-mcpa/", "level": "Associate", "note": "Launched in September 2026 as the first official MCP certification, covering MCP architecture, interactions, security and governance for agent tooling." }, { "name": "Master Artificial Intelligence Course", "provider": "DevOpsSchool", "url": "https://www.devopsschool.com/certification/master-artificial-intelligence-course.html", "level": "Professional", "note": "DevOpsSchool's instructor-led AI certification programme spanning LLMs, prompt engineering, RAG, AI ethics and AI engineering with MLOps and model deployment, a guided companion to this path." } ], "tools_catalog": [ { "name": "A2A Protocol", "category": "LLMs, Agents & GenAI", "website": "https://a2a-protocol.org/", "ds_course_url": "" }, { "name": "Agent Router", "category": "LLMs, Agents & GenAI", "website": "https://theagentrouter.ai/", "ds_course_url": "" }, { "name": "Amazon Bedrock", "category": "LLMs, Agents & GenAI", "website": "https://aws.amazon.com/bedrock/", "ds_course_url": "https://www.devopsschool.com/courses/aws/" }, { "name": "Apache Airflow", "category": "Data Engineering", "website": "https://airflow.apache.org/", "ds_course_url": "" }, { "name": "Arize Phoenix", "category": "LLMs, Agents & GenAI", "website": "https://arize.com/phoenix/", "ds_course_url": "" }, { "name": "Axolotl", "category": "LLMs, Agents & GenAI", "website": "https://docs.axolotl.ai/", "ds_course_url": "" }, { "name": "DeepEval", "category": "LLMs, Agents & GenAI", "website": "https://deepeval.com/", "ds_course_url": "" }, { "name": "Docker", "category": "Containers & Kubernetes", "website": "https://www.docker.com/", "ds_course_url": "https://www.devopsschool.com/courses/docker/" }, { "name": "Docling", "category": "LLMs, Agents & GenAI", "website": "https://docling-project.github.io/docling/", "ds_course_url": "" }, { "name": "DSPy", "category": "LLMs, Agents & GenAI", "website": "https://dspy.ai/", "ds_course_url": "" }, { "name": "garak", "category": "LLMs, Agents & GenAI", "website": "https://github.com/NVIDIA/garak", "ds_course_url": "" }, { "name": "Grafana", "category": "Observability", "website": "https://grafana.com/", "ds_course_url": "https://www.devopsschool.com/courses/grafana/" }, { "name": "Guardrails AI", "category": "LLMs, Agents & GenAI", "website": "https://www.guardrailsai.com/", "ds_course_url": "" }, { "name": "Hugging Face", "category": "LLMs, Agents & GenAI", "website": "https://huggingface.co/", "ds_course_url": "" }, { "name": "KServe", "category": "Machine Learning & MLOps", "website": "https://kserve.github.io/website/", "ds_course_url": "" }, { "name": "Kubernetes", "category": "Containers & Kubernetes", "website": "https://kubernetes.io/", "ds_course_url": "https://www.devopsschool.com/courses/kubernetes/" }, { "name": "LangChain", "category": "LLMs, Agents & GenAI", "website": "https://www.langchain.com/", "ds_course_url": "" }, { "name": "Langfuse", "category": "LLMs, Agents & GenAI", "website": "https://langfuse.com/", "ds_course_url": "" }, { "name": "LangGraph", "category": "LLMs, Agents & GenAI", "website": "https://www.langchain.com/langgraph", "ds_course_url": "" }, { "name": "LiteLLM", "category": "LLMs, Agents & GenAI", "website": "https://www.litellm.ai/", "ds_course_url": "" }, { "name": "llama.cpp", "category": "LLMs, Agents & GenAI", "website": "https://github.com/ggml-org/llama.cpp", "ds_course_url": "" }, { "name": "LlamaIndex", "category": "LLMs, Agents & GenAI", "website": "https://www.llamaindex.ai/", "ds_course_url": "" }, { "name": "LLM Compressor", "category": "LLMs, Agents & GenAI", "website": "https://github.com/vllm-project/llm-compressor", "ds_course_url": "" }, { "name": "llm-d", "category": "LLMs, Agents & GenAI", "website": "https://llm-d.ai/", "ds_course_url": "" }, { "name": "Microsoft Foundry", "category": "LLMs, Agents & GenAI", "website": "https://azure.microsoft.com/en-us/products/ai-foundry", "ds_course_url": "https://www.devopsschool.com/courses/openai/" }, { "name": "Milvus", "category": "LLMs, Agents & GenAI", "website": "https://milvus.io/", "ds_course_url": "" }, { "name": "MLflow", "category": "Machine Learning & MLOps", "website": "https://mlflow.org/", "ds_course_url": "https://www.devopsschool.com/courses/mlops/" }, { "name": "Model Context Protocol", "category": "LLMs, Agents & GenAI", "website": "https://modelcontextprotocol.io/", "ds_course_url": "" }, { "name": "NeMo Guardrails", "category": "LLMs, Agents & GenAI", "website": "https://developer.nvidia.com/nemo-guardrails", "ds_course_url": "" }, { "name": "NVIDIA Dynamo", "category": "LLMs, Agents & GenAI", "website": "https://developer.nvidia.com/dynamo", "ds_course_url": "" }, { "name": "Ollama", "category": "LLMs, Agents & GenAI", "website": "https://ollama.com/", "ds_course_url": "" }, { "name": "OpenCost", "category": "FinOps & Sustainability", "website": "https://opencost.io/", "ds_course_url": "https://www.devopsschool.com/courses/finops/" }, { "name": "OpenTelemetry", "category": "Observability", "website": "https://opentelemetry.io/", "ds_course_url": "https://www.devopsschool.com/courses/opentelemetry/" }, { "name": "pgvector", "category": "LLMs, Agents & GenAI", "website": "https://github.com/pgvector/pgvector", "ds_course_url": "https://www.devopsschool.com/courses/postgresql/" }, { "name": "PostgreSQL", "category": "Data Engineering", "website": "https://www.postgresql.org/", "ds_course_url": "https://www.devopsschool.com/courses/postgresql/" }, { "name": "promptfoo", "category": "LLMs, Agents & GenAI", "website": "https://www.promptfoo.dev/", "ds_course_url": "" }, { "name": "Python", "category": "Foundations & Languages", "website": "https://www.python.org/", "ds_course_url": "https://www.devopsschool.com/courses/python/" }, { "name": "Qdrant", "category": "LLMs, Agents & GenAI", "website": "https://qdrant.tech/", "ds_course_url": "" }, { "name": "Ragas", "category": "LLMs, Agents & GenAI", "website": "https://www.ragas.io/", "ds_course_url": "" }, { "name": "Redis", "category": "Data Engineering", "website": "https://redis.io/", "ds_course_url": "https://www.devopsschool.com/courses/redis/" }, { "name": "SGLang", "category": "LLMs, Agents & GenAI", "website": "https://www.sglang.io/", "ds_course_url": "" }, { "name": "TensorRT-LLM", "category": "LLMs, Agents & GenAI", "website": "https://developer.nvidia.com/tensorrt-llm", "ds_course_url": "" }, { "name": "Unsloth", "category": "LLMs, Agents & GenAI", "website": "https://unsloth.ai/docs", "ds_course_url": "" }, { "name": "vLLM", "category": "LLMs, Agents & GenAI", "website": "https://vllm.ai/", "ds_course_url": "" }, { "name": "Weaviate", "category": "LLMs, Agents & GenAI", "website": "https://weaviate.io/", "ds_course_url": "" } ], "faqs": [ { "q": "What is the difference between LLMOps and MLOps?", "a": "MLOps manages the lifecycle of models you train yourself: data, features, training pipelines, registries and drift. LLMOps usually starts from a pretrained foundation model and focuses on prompts and context, retrieval, evaluation of open-ended outputs, guardrails, inference cost and agent behaviour. The infrastructure skills overlap heavily, which is why many teams run both on one platform." }, { "q": "Do I need to fine-tune models to do LLMOps?", "a": "Rarely at first. Most production gains come from better context, retrieval, evaluation and model choice. Fine-tune with LoRA or QLoRA when you need a consistent format or behaviour, or want a smaller, cheaper model to match a larger one, and only once an evaluation suite can prove the improvement." }, { "q": "Should I self-host open-weight models or use hosted APIs?", "a": "Start with hosted APIs for speed, then self-host with vLLM or SGLang when data residency, sustained high volume, latency control or customisation justify running GPUs. Compare total cost per million tokens including idle GPU time and engineering effort, and keep an OpenAI-compatible gateway in front so you can switch either way." }, { "q": "Which vector database should I choose for RAG?", "a": "If you already run PostgreSQL and have up to tens of millions of vectors, pgvector keeps operations simple. Qdrant, Weaviate and Milvus add built-in hybrid search, multi-tenancy, vector quantization and horizontal scaling for larger or more demanding workloads. Retrieval quality usually depends more on chunking, hybrid search and reranking than on the database itself." }, { "q": "How do you test LLM applications when outputs are non-deterministic?", "a": "Build a versioned test set from real questions with expected facts or reference answers, score outputs with deterministic checks plus LLM-as-judge metrics calibrated against human labels, and run the suite in CI on every prompt, model or retrieval change. In production, score sampled traces online and turn every incident into a new test case." }, { "q": "What is MCP and why does it matter for LLMOps?", "a": "The Model Context Protocol is an open standard for connecting LLM applications and agents to tools, data and prompts through MCP servers. Since Anthropic donated it to the Linux Foundation's Agentic AI Foundation in December 2025 it has become vendor-neutral infrastructure, so platform teams now own MCP server deployment, authorization, auditing and defences against tool poisoning." }, { "q": "Which certifications help an LLMOps career in 2026?", "a": "Cloud-aligned options are AWS Certified Generative AI Developer – Professional, Microsoft AI-103 (Azure AI Apps and Agents Developer Associate) and the Databricks Generative AI Engineer Associate. NVIDIA's NCA-GENL covers LLM fundamentals and the new MCPA validates Model Context Protocol knowledge. On Azure, Microsoft's AI-300 MLOps Engineer exam also covers GenAIOps." } ], "related": [ "mlops", "aiops", "platform-engineering", "devsecops" ], "key_tools": [ "vLLM", "LangGraph", "LlamaIndex", "Langfuse", "promptfoo", "Hugging Face" ], "course_url": "", "seo_title": "LLMOps Learning Path 2026: RAG, Evals, vLLM & Agents", "seo_description": "2026 LLMOps roadmap: model selection, prompt management, RAG, evals, guardrails, vLLM serving, OpenTelemetry tracing, gateways, MCP agents and certs.", "url": "https://www.devopsschool.com/path/llmops/", "updated_at": "2026-10-26 08:58:45" }