{ "schema": 1, "slug": "aiops", "title": "AIOps", "full_name": "AIOps: AI for IT Operations", "category": "Reliability & Operations", "tagline": "Turn telemetry into action: cut alert noise, predict incidents, find root cause and automate fixes with AI.", "summary": "Build AI-assisted operations on a solid observability foundation: OpenTelemetry data, event correlation, anomaly detection, root-cause analysis and safe automated remediation. The path ends with 2026-era agentic operations, where MCP-connected AI SRE agents investigate incidents under human-in-the-loop guardrails.", "description": "AIOps applies machine learning, analytics and now large language models to operational data (metrics, logs, traces, events, topology and change records) so teams can detect problems earlier, cut alert noise, find root cause faster and automate the response. Done well, it turns thousands of raw alerts into a handful of context-rich incidents and measurably shortens time to detect and time to resolve.\n\nThe field changed sharply in 2025 and 2026. Gartner reframed AIOps platforms as event intelligence solutions to cut through vendor hype, and AI SRE agents became generally available from AWS, Microsoft, Datadog, PagerDuty and others, alongside open-source projects such as HolmesGPT. Platforms such as Dynatrace now pair deterministic causal AI with agentic AI so root-cause answers stay explainable while agents act within guardrails, and the Model Context Protocol has become the common way to wire agents into observability and ITSM tools.\n\nThis path is deliberately foundation-first, because agents are only as good as the telemetry, topology and runbooks they can reach. You start with OpenTelemetry and alerting hygiene, move through correlation, time-series ML and root-cause analysis, integrate with ServiceNow and runbook automation, and finish by building and governing an AI operations agent with approval gates and audit trails.\n\nAIOps sits between SRE (SLOs, incident practice), MLOps (running models in production) and LLMOps (operating LLM applications), and it feeds CloudOps and FinOps, because the same telemetry and automation drive capacity and cost decisions.", "icon": "psychology", "accent": "#EC4899", "level": "Intermediate → Expert", "duration_weeks": 27, "weekly_hours": 10, "audience": [ "SREs, DevOps and platform engineers who carry the pager and want less noise and faster resolution", "NOC and IT operations engineers moving from manual triage to automated, AI-assisted operations", "Observability engineers who want to add ML-based detection, correlation and root-cause analysis to their stack", "Data scientists and ML engineers who want to apply time-series and LLM skills to operations data", "ITSM and ServiceNow professionals modernising event and incident management" ], "prerequisites": [ "Comfortable with Linux, networking basics and at least one public cloud", "Hands-on experience running or supporting production services, including basic monitoring and on-call", "Working knowledge of Python (scripts, pandas basics) and SQL", "Familiarity with containers and core Kubernetes concepts", "Basic statistics: mean, variance, percentiles and distributions" ], "outcomes": [ "Instrument services with OpenTelemetry and design telemetry pipelines rich enough for correlation, ML and agents", "Design deduplication and correlation rules that measurably reduce alert noise without hiding real incidents", "Build, evaluate and tune anomaly-detection and forecasting models on real metrics and logs", "Perform topology- and change-aware root-cause analysis and explain the evidence behind each conclusion", "Automate remediation with Event-Driven Ansible, runbook automation and ServiceNow incident workflows", "Evaluate AIOps platforms such as Dynatrace, Datadog, Splunk ITSI, BigPanda and PagerDuty with objective proof-of-concept metrics", "Build and govern an MCP-connected AI SRE agent with least-privilege tools, human approval gates and audit logging" ], "trends": [ "Gartner reframed AIOps platforms as event intelligence solutions in 2025, so buyers now judge tools on measured noise reduction and faster response rather than AI branding.", "AI SRE agents went mainstream in 2026: AWS DevOps Agent and Azure SRE Agent reached general availability in March, joining Datadog's Bits AI investigation agent, PagerDuty's SRE Agent and the CNCF sandbox project HolmesGPT.", "The Model Context Protocol, now hosted by the Linux Foundation's Agentic AI Foundation, has become the standard way to connect agents to observability, incident and ITSM tools.", "Platforms are fusing deterministic causal AI with agentic AI, as in Dynatrace Intelligence (January 2026), so root-cause answers stay explainable while agents act within policy guardrails.", "OpenTelemetry keeps widening the data foundation: the profiles signal entered public alpha in 2026 and OpenTelemetry eBPF Instrumentation (OBI) is working towards a stable 1.0.", "Time-series foundation models such as Datadog Toto, Amazon Chronos and Google TimesFM make zero-shot forecasting and anomaly detection practical without training one model per metric." ], "roles": [ { "title": "AIOps Engineer", "description": "Builds and tunes the event pipeline, correlation rules, anomaly models and automations that turn raw telemetry into actionable incidents. Owns noise-reduction, MTTD and MTTR metrics for the operations platform." }, { "title": "Site Reliability Engineer", "description": "Defines SLOs and alerting, leads incident response and postmortems, and increasingly supervises AI agents that do first-pass investigation and propose fixes." }, { "title": "Observability Engineer", "description": "Designs OpenTelemetry instrumentation, telemetry pipelines and data models so metrics, logs, traces and topology are consistent, affordable and ready for ML and agents." }, { "title": "NOC / IT Operations Automation Engineer", "description": "Modernises the NOC by integrating monitoring with ServiceNow, automating L1 triage and runbooks, and keeping humans in control of risky changes." }, { "title": "Agentic Operations Engineer", "description": "Builds, evaluates and governs LLM-based operations agents and their MCP tool integrations, including permissions, approval gates, evaluation suites and audit trails." } ], "stages": [ { "title": "Operations & Observability Foundations", "level": "Beginner", "duration_weeks": 4, "description": "The operational and telemetry groundwork every AIOps capability depends on: incident practice, the four telemetry signals, OpenTelemetry, alerting and the Python data skills used throughout the path.", "milestone": "Instrument a microservices app with OpenTelemetry, query its metrics, logs and traces in one Grafana workspace, and alert on an SLO burn rate.", "topics": [ { "title": "Incident Management & Service Reliability Basics", "description": "Learn how incidents, problems and changes flow through an operations team and which metrics AIOps is meant to move. Without clear severities, ownership and SLOs you cannot tell whether a correlation rule or an agent actually helped.", "est_hours": 6, "optional": false, "concepts": [ "incident vs problem vs change", "severity and priority models", "MTTA, MTTD and MTTR", "SLIs, SLOs and error budgets", "on-call and escalation policies", "blameless postmortems" ], "tools": [ "PagerDuty" ], "resources": [ { "title": "Google SRE Book: Managing Incidents", "url": "https://sre.google/sre-book/managing-incidents/", "type": "book", "free": true }, { "title": "PagerDuty Incident Response Guide", "url": "https://response.pagerduty.com/", "type": "docs", "free": true }, { "title": "Site Reliability Engineering (SRE) Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/sre/", "type": "course", "free": false } ] }, { "title": "Telemetry Signals & OpenTelemetry", "description": "Understand metrics, logs, traces, events and the newer profiles signal, and how OpenTelemetry standardises them with shared resource attributes and semantic conventions. Consistent attributes such as service.name and deployment.environment are what later make correlation, topology and ML possible.", "est_hours": 10, "optional": false, "concepts": [ "metrics, logs, traces, events and profiles", "resource attributes and semantic conventions", "SDK vs zero-code and eBPF instrumentation", "context propagation", "OTLP", "Collector receivers, processors and exporters" ], "tools": [ "OpenTelemetry" ], "resources": [ { "title": "OpenTelemetry Signals", "url": "https://opentelemetry.io/docs/concepts/signals/", "type": "docs", "free": true }, { "title": "OpenTelemetry Collector Documentation", "url": "https://opentelemetry.io/docs/collector/", "type": "docs", "free": true }, { "title": "OpenTelemetry Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/opentelemetry/", "type": "course", "free": false } ] }, { "title": "Metrics, Alerting & SLO Burn Rates", "description": "Use Prometheus and Grafana to collect metrics, write PromQL and route alerts through Alertmanager. Learn why symptom-based, SLO burn-rate alerts produce far less noise than threshold alerts on every host, which is the cheapest noise reduction you will ever get.", "est_hours": 10, "optional": false, "concepts": [ "PromQL and recording rules", "Alertmanager grouping, inhibition and silences", "symptom vs cause alerting", "multi-window burn-rate alerts", "label and cardinality hygiene", "dashboards for triage" ], "tools": [ "Prometheus", "Grafana" ], "resources": [ { "title": "Prometheus Overview", "url": "https://prometheus.io/docs/introduction/overview/", "type": "docs", "free": true }, { "title": "The Site Reliability Workbook: Alerting on SLOs", "url": "https://sre.google/workbook/alerting-on-slos/", "type": "book", "free": true }, { "title": "Prometheus Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/prometheus/", "type": "course", "free": false } ] }, { "title": "Logs & Distributed Traces", "description": "Store and query logs with Loki or the Elastic Stack and traces with Tempo or Jaeger, then link them through trace IDs and exemplars. Cross-signal navigation is the manual version of what correlation engines and agents automate later.", "est_hours": 8, "optional": false, "concepts": [ "structured logging", "LogQL and log labels", "trace and span model", "trace-to-logs and exemplars", "sampling basics", "retention and cost trade-offs" ], "tools": [ "Loki", "Tempo", "Jaeger", "Elastic Stack" ], "resources": [ { "title": "Grafana Loki Documentation", "url": "https://grafana.com/docs/loki/latest/", "type": "docs", "free": true }, { "title": "Jaeger Documentation", "url": "https://www.jaegertracing.io/docs/", "type": "docs", "free": true }, { "title": "Observability Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/observability/", "type": "course", "free": false } ] }, { "title": "Python for Operations Data", "description": "Get fluent with Python, pandas and notebooks for pulling telemetry from APIs, reshaping time series and plotting results. Every ML and agent exercise later in the path is written in Python.", "est_hours": 6, "optional": false, "concepts": [ "HTTP APIs and JSON", "pandas DataFrames and time indexes", "resampling and rolling windows", "Jupyter notebooks", "virtual environments" ], "tools": [ "Python" ], "resources": [ { "title": "The Python Tutorial", "url": "https://docs.python.org/3/tutorial/", "type": "docs", "free": true }, { "title": "pandas Documentation", "url": "https://pandas.pydata.org/docs/", "type": "docs", "free": true }, { "title": "Python Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/python/", "type": "course", "free": false } ] } ] }, { "title": "Event Pipelines, Correlation & Topology", "level": "Intermediate", "duration_weeks": 4, "description": "The classic core of AIOps: getting events from many tools into one pipeline, normalising and enriching them, grouping related alerts into incidents and mapping services so impact and blast radius are visible.", "milestone": "Ingest alerts from three monitoring sources, deduplicate and correlate them into topology-aware incidents, and report the resulting noise-reduction ratio.", "topics": [ { "title": "Telemetry Pipelines & Data Quality", "description": "Design pipelines that collect, buffer, enrich and route telemetry at scale with the OpenTelemetry Collector and Apache Kafka. Garbage in is garbage out: missing service names, exploding cardinality or broken timestamps silently ruin ML and correlation.", "est_hours": 10, "optional": false, "concepts": [ "Collector gateway vs agent patterns", "tail-based sampling", "attribute enrichment and redaction", "buffering and back-pressure", "cardinality and cost control", "data quality checks on telemetry" ], "tools": [ "OpenTelemetry", "Apache Kafka" ], "resources": [ { "title": "OpenTelemetry Collector Documentation", "url": "https://opentelemetry.io/docs/collector/", "type": "docs", "free": true }, { "title": "Apache Kafka Documentation", "url": "https://kafka.apache.org/documentation/", "type": "docs", "free": true }, { "title": "Apache Kafka Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/kafka/", "type": "course", "free": false } ] }, { "title": "Event Normalisation, Deduplication & Enrichment", "description": "Map alerts from different tools into one event schema, deduplicate repeats, suppress flapping and maintenance noise, and enrich each event with owner, service and recent changes. Enrichment is what lets a responder, or an agent, act without opening five consoles.", "est_hours": 8, "optional": false, "concepts": [ "common event schema", "dedup keys and fingerprints", "flap detection", "maintenance windows and suppression", "ownership and CMDB enrichment", "change-event ingestion" ], "tools": [ "PagerDuty", "Keep" ], "resources": [ { "title": "PagerDuty AIOps / Signal Intelligence Overview", "url": "https://docs.pagerduty.com/ai-automation/aiops", "type": "docs", "free": true }, { "title": "Keep Documentation (open-source AIOps and alert management)", "url": "https://docs.keephq.dev/overview/introduction", "type": "docs", "free": true } ] }, { "title": "Alert Correlation & Noise Reduction", "description": "Group related alerts into incidents using time windows, shared attributes, topology and ML similarity, and measure the result with compression ratio and missed-incident rate. This is where most commercial AIOps value is proven or disproven.", "est_hours": 12, "optional": false, "concepts": [ "time-based grouping", "content and attribute-based grouping", "topology-based correlation", "ML and text-similarity grouping", "compression ratio and false merges", "feedback loops from responders" ], "tools": [ "PagerDuty", "BigPanda", "ServiceNow" ], "resources": [ { "title": "PagerDuty Alert Grouping", "url": "https://docs.pagerduty.com/ai-automation/aiops/noise-reduction/alert-grouping", "type": "docs", "free": true }, { "title": "BigPanda Documentation", "url": "https://docs.bigpanda.io/", "type": "docs", "free": true }, { "title": "ServiceNow Event Management", "url": "https://www.servicenow.com/products/event-management.html", "type": "docs", "free": true } ] }, { "title": "Service Topology & Dependency Mapping", "description": "Build a live map of services and dependencies from traces, service catalogs and CMDB data. Topology turns a pile of alerts into an impact story and is the backbone of root-cause analysis and agent reasoning.", "est_hours": 10, "optional": false, "concepts": [ "service graphs from traces", "service catalog and ownership", "CMDB and CSDM basics", "upstream vs downstream impact", "blast radius", "keeping topology fresh" ], "tools": [ "OpenTelemetry", "Tempo", "Backstage", "ServiceNow" ], "resources": [ { "title": "OpenTelemetry Collector Service Graph Connector", "url": "https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector", "type": "docs", "free": true }, { "title": "Grafana Tempo: Service Graphs", "url": "https://grafana.com/docs/tempo/latest/metrics-from-traces/service_graphs/enable-service-graphs/", "type": "docs", "free": true }, { "title": "Backstage Software Catalog", "url": "https://backstage.io/docs/features/software-catalog/", "type": "docs", "free": true } ] } ] }, { "title": "Machine Learning for Operations Data", "level": "Intermediate", "duration_weeks": 5, "description": "Practical time-series and log ML for operations: forecasting, anomaly detection, log pattern mining and when to reach for time-series foundation models, always evaluated against real incidents rather than demo data.", "milestone": "Build and tune a seasonal anomaly detector and a capacity forecast on real telemetry, and report precision and recall against labelled incidents.", "topics": [ { "title": "Time-Series Analysis & Forecasting", "description": "Learn trend, seasonality and decomposition, then forecast capacity and demand with classical models before trying anything fancier. Forecasts power proactive alerts such as disk-full predictions and capacity planning.", "est_hours": 12, "optional": false, "concepts": [ "trend and multiple seasonality", "STL decomposition", "ETS and ARIMA", "backtesting and forecast error metrics", "prediction intervals", "PromQL predict_linear" ], "tools": [ "Python", "StatsForecast", "Prometheus" ], "resources": [ { "title": "Forecasting: Principles and Practice, the Pythonic Way (free book)", "url": "https://otexts.com/fpppy/", "type": "book", "free": true }, { "title": "StatsForecast Documentation", "url": "https://nixtlaverse.nixtla.io/statsforecast", "type": "docs", "free": true }, { "title": "Prometheus Query Functions", "url": "https://prometheus.io/docs/prometheus/latest/querying/functions/", "type": "docs", "free": true } ] }, { "title": "Anomaly Detection on Metrics", "description": "Move from static thresholds to dynamic baselines: robust statistics, seasonal baselines, Isolation Forest and multivariate detectors. The hard part is evaluation, so learn to measure precision, recall and alert volume against labelled incidents.", "est_hours": 14, "optional": false, "concepts": [ "static vs dynamic thresholds", "z-score, MAD and seasonal baselines", "Isolation Forest and LOF", "multivariate anomaly detection", "labelling incidents for evaluation", "precision, recall and alert fatigue" ], "tools": [ "Python", "scikit-learn", "PyOD" ], "resources": [ { "title": "scikit-learn: Novelty and Outlier Detection", "url": "https://scikit-learn.org/stable/modules/outlier_detection.html", "type": "docs", "free": true }, { "title": "PyOD Documentation", "url": "https://pyod.readthedocs.io/en/latest/", "type": "docs", "free": true }, { "title": "Elastic Machine Learning: Anomaly Detection", "url": "https://www.elastic.co/docs/explore-analyze/machine-learning/anomaly-detection", "type": "docs", "free": true } ] }, { "title": "Log Analytics & Pattern Mining", "description": "Turn unstructured logs into templates and counts with log parsers such as Drain, then detect new or spiking patterns. Pattern-level signals are far cheaper and more stable than per-line ML and feed both correlation and LLM summaries.", "est_hours": 10, "optional": false, "concepts": [ "log template extraction", "pattern frequency anomalies", "new error-pattern detection", "log clustering", "embeddings for semantic log search", "PII redaction" ], "tools": [ "Python", "Elastic Stack", "Loki" ], "resources": [ { "title": "Drain3 Log Template Miner", "url": "https://github.com/logpai/Drain3", "type": "tool", "free": true }, { "title": "Loghub: Large Collection of System Log Datasets", "url": "https://github.com/logpai/loghub", "type": "lab", "free": true } ] }, { "title": "Time-Series Foundation Models", "description": "Explore pretrained forecasting models such as Amazon Chronos, Google TimesFM and Datadog Toto that work zero-shot across many metrics. Learn where they beat per-metric models and where cost, latency and explainability make classical methods the better choice.", "est_hours": 6, "optional": true, "concepts": [ "zero-shot forecasting", "probabilistic forecasts", "fine-tuning vs zero-shot", "benchmarking against baselines", "inference cost and latency" ], "tools": [ "Python", "Hugging Face" ], "resources": [ { "title": "Chronos Forecasting (Amazon Science)", "url": "https://github.com/amazon-science/chronos-forecasting", "type": "tool", "free": true }, { "title": "TimesFM (Google Research)", "url": "https://github.com/google-research/timesfm", "type": "tool", "free": true }, { "title": "Toto Time-Series Foundation Model (Datadog)", "url": "https://github.com/DataDog/toto", "type": "tool", "free": true } ] }, { "title": "Operating AIOps Models in Production", "description": "Treat detectors like any production model: version them, track experiments, watch for drift as systems change and retrain using responder feedback. This keeps models trustworthy after every re-architecture or traffic shift.", "est_hours": 8, "optional": true, "concepts": [ "experiment tracking", "model versioning", "concept drift after deployments", "feedback labels from incidents", "scheduled retraining" ], "tools": [ "MLflow" ], "resources": [ { "title": "MLflow Documentation", "url": "https://mlflow.org/docs/latest/", "type": "docs", "free": true }, { "title": "MLOps Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/mlops/", "type": "course", "free": false } ] } ] }, { "title": "Root-Cause Analysis & AIOps Platforms", "level": "Advanced", "duration_weeks": 5, "description": "How root-cause analysis really works, and how the leading platforms implement detection, correlation and RCA, so you can operate them well and choose between them with evidence.", "milestone": "Run a structured proof of concept comparing two AIOps platforms on the same incident set and recommend one using noise-reduction, MTTD and RCA-accuracy evidence.", "topics": [ { "title": "Root-Cause Analysis Techniques", "description": "Combine topology, change events and anomaly timing to rank probable causes, and learn the difference between causal analysis and correlation. Pair automated RCA with blameless postmortems so findings turn into fixes.", "est_hours": 10, "optional": false, "concepts": [ "probable cause vs root cause", "change and deployment correlation", "topology-aware fault propagation", "causal graphs", "evidence and explainability", "postmortem action tracking" ], "tools": [ "Dynatrace" ], "resources": [ { "title": "Dynatrace Intelligence: Root-Cause Analysis", "url": "https://docs.dynatrace.com/docs/dynatrace-intelligence/root-cause-analysis", "type": "docs", "free": true }, { "title": "Google SRE Book: Postmortem Culture", "url": "https://sre.google/sre-book/postmortem-culture/", "type": "book", "free": true }, { "title": "AIOps Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/aiops/", "type": "course", "free": false } ] }, { "title": "Dynatrace Intelligence (Davis AI)", "description": "Learn how Dynatrace combines Smartscape topology, the Grail data lakehouse and deterministic causal AI, now extended with agents under Dynatrace Intelligence. It is the reference example of topology-first, explainable RCA.", "est_hours": 10, "optional": false, "concepts": [ "Smartscape topology", "Grail and DQL", "automatic baselining", "problems and causal root cause", "workflows and agents with guardrails" ], "tools": [ "Dynatrace" ], "resources": [ { "title": "Dynatrace Intelligence Documentation", "url": "https://docs.dynatrace.com/docs/dynatrace-intelligence", "type": "docs", "free": true }, { "title": "Dynatrace Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/dynatrace/", "type": "course", "free": false } ] }, { "title": "Datadog Watchdog, Event Management & Bits AI", "description": "Use Watchdog for automatic anomaly and root-cause detection, Event Management for correlation, and the Bits AI investigation agent for autonomous first-pass triage. Learn to judge what the AI found versus what it guessed.", "est_hours": 10, "optional": false, "concepts": [ "Watchdog anomalies and root cause", "event correlation patterns", "unified service tagging", "AI investigation workflow", "reviewing AI findings" ], "tools": [ "Datadog" ], "resources": [ { "title": "Datadog Watchdog", "url": "https://docs.datadoghq.com/watchdog/", "type": "docs", "free": true }, { "title": "Datadog Event Management: Correlation", "url": "https://docs.datadoghq.com/service_management/events/correlation/", "type": "docs", "free": true }, { "title": "Datadog Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/datadog/", "type": "course", "free": false } ] }, { "title": "Splunk ITSI, Grafana & Elastic for Service Intelligence", "description": "Model services and KPIs with adaptive thresholds and episode grouping in Splunk ITSI, and compare the ML features in Grafana Cloud (forecasting, outlier detection, Sift) and Elastic. Many enterprises run more than one, so knowing the patterns transfers.", "est_hours": 10, "optional": false, "concepts": [ "service and KPI modelling", "adaptive thresholds", "episode grouping and review", "ML forecasting in dashboards", "automated investigation helpers" ], "tools": [ "Splunk", "Grafana", "Elastic Stack" ], "resources": [ { "title": "Splunk IT Service Intelligence Documentation", "url": "https://help.splunk.com/en/splunk-it-service-intelligence", "type": "docs", "free": true }, { "title": "Grafana Cloud Machine Learning: Forecasting", "url": "https://grafana.com/docs/grafana-cloud/ai-tools/machine-learning/forecasting/", "type": "docs", "free": true }, { "title": "Splunk Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/splunk/", "type": "course", "free": false } ] }, { "title": "Evaluating & Selecting an AIOps Platform", "description": "Run a fair proof of concept: replay the same historical alerts through each candidate, then compare noise reduction, missed incidents, RCA accuracy, integration effort, data residency and pricing. Include an open-source baseline so you know what you are paying for.", "est_hours": 10, "optional": false, "concepts": [ "build vs buy", "POC success criteria", "historical alert replay", "integration and data-residency checks", "pricing models and telemetry cost", "open-source baseline" ], "tools": [ "New Relic", "BigPanda", "PagerDuty", "Keep" ], "resources": [ { "title": "New Relic Documentation", "url": "https://docs.newrelic.com/", "type": "docs", "free": true }, { "title": "Keep (open-source AIOps platform) on GitHub", "url": "https://github.com/keephq/keep", "type": "tool", "free": true } ] } ] }, { "title": "Automated Remediation & ITSM Integration", "level": "Advanced", "duration_weeks": 4, "description": "Closing the loop: safe, auditable automation that fixes known problems, integration with ServiceNow so every action is tracked, and fault injection to prove detection and remediation actually work.", "milestone": "Detect an injected fault, open and enrich a ServiceNow incident, and run an approved automated remediation end to end with a full audit trail.", "topics": [ { "title": "Runbook Automation & Event-Driven Remediation", "description": "Codify runbooks and trigger them from events with Event-Driven Ansible rulebooks, Rundeck jobs or AWS Systems Manager Automation. Build in pre-checks, rate limits and rollback so automation never makes an incident worse.", "est_hours": 12, "optional": false, "concepts": [ "runbooks as code", "Ansible rulebooks and event sources", "idempotent remediation", "pre-checks, rate limits and circuit breakers", "approval steps for risky actions", "audit logging" ], "tools": [ "Ansible", "Rundeck", "AWS" ], "resources": [ { "title": "Ansible Rulebook (Event-Driven Ansible) Documentation", "url": "https://docs.ansible.com/projects/rulebook/en/latest/", "type": "docs", "free": true }, { "title": "AWS Systems Manager Automation", "url": "https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html", "type": "docs", "free": true }, { "title": "Ansible Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/ansible/", "type": "course", "free": false } ] }, { "title": "Self-Healing on Kubernetes", "description": "Use Kubernetes probes, autoscaling and controllers for built-in recovery, and Argo Rollouts analysis to roll back bad releases automatically using live metrics. Since a large share of incidents start with a change, automated rollback is often the highest-value remediation.", "est_hours": 8, "optional": false, "concepts": [ "liveness and readiness probes", "horizontal pod autoscaling", "operators and reconciliation", "metric-driven canary analysis", "automated rollback" ], "tools": [ "Kubernetes", "Argo Rollouts" ], "resources": [ { "title": "Argo Rollouts: Analysis & Progressive Delivery", "url": "https://argo-rollouts.readthedocs.io/en/stable/features/analysis/", "type": "docs", "free": true }, { "title": "Kubernetes Horizontal Pod Autoscaling", "url": "https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/", "type": "docs", "free": true }, { "title": "Kubernetes Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/kubernetes/", "type": "course", "free": false } ] }, { "title": "ITSM Integration with ServiceNow", "description": "Connect the AIOps pipeline to ServiceNow ITOM and ITSM: bidirectional incident sync, CMDB-backed impact, change-risk context and the agentic AIOps features in Now Assist for ITOM. This is where AIOps meets audit, compliance and the rest of the enterprise.", "est_hours": 10, "optional": false, "concepts": [ "event to alert to incident flow", "bidirectional incident sync", "CMDB, CSDM and service mapping", "change risk and change correlation", "AI agents in ITOM", "SLA and audit requirements" ], "tools": [ "ServiceNow", "PagerDuty" ], "resources": [ { "title": "ServiceNow IT Operations Management (ITOM)", "url": "https://www.servicenow.com/products/it-operations-management.html", "type": "docs", "free": true }, { "title": "Master in ServiceNow — DevOpsSchool", "url": "https://www.devopsschool.com/certification/master-in-servicenow.html", "type": "course", "free": false } ] }, { "title": "Validating the Loop with Chaos Engineering", "description": "Inject realistic faults to prove that detection fires, correlation groups correctly, RCA points at the right component and remediation succeeds. Game days give you the labelled incidents that ML evaluation needs.", "est_hours": 10, "optional": false, "concepts": [ "fault-injection experiments", "game days", "measuring MTTD and MTTR", "detection coverage gaps", "creating labelled incident data" ], "tools": [ "Chaos Mesh", "LitmusChaos" ], "resources": [ { "title": "Chaos Mesh", "url": "https://chaos-mesh.org/", "type": "docs", "free": true }, { "title": "LitmusChaos", "url": "https://litmuschaos.io/", "type": "docs", "free": true }, { "title": "Chaos Engineering Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/chaos-engineering/", "type": "course", "free": false } ] } ] }, { "title": "LLM & Agentic AI for Operations", "level": "Expert", "duration_weeks": 5, "description": "The 2026 frontier: LLM agents that investigate incidents, query observability tools through MCP and propose or execute fixes, built and governed so that humans stay in control of anything risky.", "milestone": "Deploy an MCP-connected investigation agent that drafts root cause and remediation for real alerts, with human approval gates, evaluation results and full audit logging.", "topics": [ { "title": "LLM Foundations for Operators", "description": "Learn how LLMs reason over telemetry, runbooks and postmortems: retrieval, tool calling, structured output and evaluation. Understand failure modes such as hallucinated metrics so you design around them rather than discover them during an outage.", "est_hours": 8, "optional": false, "concepts": [ "retrieval over runbooks and postmortems", "tool calling and structured output", "workflows vs autonomous agents", "hallucination and grounding", "token cost and latency budgets", "evaluation datasets" ], "tools": [ "Python" ], "resources": [ { "title": "Building Effective Agents (Anthropic)", "url": "https://www.anthropic.com/research/building-effective-agents", "type": "article", "free": true }, { "title": "Prompt Engineering Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/prompt-engineering/", "type": "course", "free": false } ] }, { "title": "MCP-Connected Operations Tooling", "description": "Expose observability, incident and ITSM systems to agents through Model Context Protocol servers, starting read-only and adding write tools deliberately. Learn authentication, scoping and how to stop a noisy log line from becoming a prompt injection.", "est_hours": 10, "optional": false, "concepts": [ "MCP clients, servers, tools and resources", "read-only vs write tools", "authorisation and scoped tokens", "remote MCP servers", "prompt injection via telemetry", "tool-call logging" ], "tools": [ "Model Context Protocol", "Grafana", "PagerDuty" ], "resources": [ { "title": "Model Context Protocol Specification", "url": "https://modelcontextprotocol.io/specification/latest", "type": "docs", "free": true }, { "title": "Grafana MCP Server", "url": "https://github.com/grafana/mcp-grafana", "type": "tool", "free": true }, { "title": "PagerDuty MCP Server", "url": "https://docs.pagerduty.com/ai-automation/advance/pagerduty-mcp-server", "type": "docs", "free": true } ] }, { "title": "AI SRE Agents in Practice", "description": "Run and compare AI SRE agents: open-source HolmesGPT and K8sGPT, and managed options such as AWS DevOps Agent, Azure SRE Agent, Datadog Bits AI and PagerDuty's SRE Agent. Measure their root-cause accuracy and time saved on your own incidents, not vendor demos.", "est_hours": 12, "optional": false, "concepts": [ "alert-triggered investigations", "hypothesis-driven troubleshooting", "custom skills and runbook grounding", "autonomy levels", "measuring RCA accuracy and time saved", "escalation with full context" ], "tools": [ "HolmesGPT", "K8sGPT", "AWS DevOps Agent", "Azure SRE Agent", "Datadog", "PagerDuty" ], "resources": [ { "title": "HolmesGPT Documentation", "url": "https://holmesgpt.dev/latest/", "type": "docs", "free": true }, { "title": "AWS DevOps Agent User Guide", "url": "https://docs.aws.amazon.com/devopsagent/latest/userguide/", "type": "docs", "free": true }, { "title": "Azure SRE Agent Overview", "url": "https://learn.microsoft.com/en-us/azure/sre-agent/overview", "type": "docs", "free": true } ] }, { "title": "Guardrails, Human-in-the-Loop & Governance", "description": "Decide what agents may do alone, what needs approval and what is forbidden, then enforce it with policy-as-code, scoped credentials and immutable audit logs. Map risks with the OWASP agentic top 10 and the NIST AI Risk Management Framework.", "est_hours": 8, "optional": false, "concepts": [ "autonomy tiers and approval gates", "least-privilege agent identities", "policy-as-code for agent actions", "excessive agency and tool misuse", "audit trails and replay", "kill switches and rollback" ], "tools": [ "OPA" ], "resources": [ { "title": "OWASP Top 10 for Agentic Applications for 2026", "url": "https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/", "type": "paper", "free": true }, { "title": "NIST AI Risk Management Framework", "url": "https://www.nist.gov/itl/ai-risk-management-framework", "type": "paper", "free": true }, { "title": "Open Policy Agent Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/opa/", "type": "course", "free": false } ] }, { "title": "Building a Custom Operations Agent", "description": "Build your own investigation agent as an explicit graph with tools, memory and human-approval interrupts, and trace every step with OpenTelemetry GenAI conventions. A custom agent lets you encode your topology, runbooks and policies rather than accept a vendor's defaults.", "est_hours": 12, "optional": false, "concepts": [ "agent graphs and state", "tool design for observability APIs", "human-approval interrupts", "checkpointing and resumability", "GenAI tracing of agent steps", "offline evaluation harness" ], "tools": [ "LangGraph", "OpenTelemetry", "Python" ], "resources": [ { "title": "LangGraph Documentation", "url": "https://docs.langchain.com/oss/python/langgraph/overview", "type": "docs", "free": true }, { "title": "OpenTelemetry Semantic Conventions for Generative AI", "url": "https://opentelemetry.io/docs/specs/semconv/gen-ai/", "type": "docs", "free": true }, { "title": "Claude AI Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/claude/", "type": "course", "free": false } ] } ] } ], "projects": [ { "title": "Observable Microservices Lab", "description": "Deploy the OpenTelemetry Demo on Kubernetes, send all signals through a Collector to Prometheus, Loki and Tempo, and build triage dashboards. Deliver multi-window SLO burn-rate alerts and a one-page alerting policy.", "difficulty": "Beginner", "skills": [ "OpenTelemetry", "Prometheus", "Grafana", "SLO alerting" ] }, { "title": "Alert Noise-Reduction Pipeline", "description": "Feed alerts from Prometheus, a cloud monitoring service and a log tool into Keep or PagerDuty, normalise and enrich them with ownership and recent deploys, and correlate them into incidents. Report compression ratio and any missed incidents over a two-week replay.", "difficulty": "Intermediate", "skills": [ "event normalisation", "deduplication", "alert correlation", "enrichment" ] }, { "title": "Seasonal Anomaly Detector & Capacity Forecast", "description": "Pull a month of service metrics, build seasonal baselines and an Isolation Forest detector, and forecast capacity with StatsForecast. Compare precision, recall and alert volume against static thresholds using incidents you labelled from chaos experiments.", "difficulty": "Intermediate", "skills": [ "time-series forecasting", "anomaly detection", "model evaluation", "Python" ] }, { "title": "Topology-Aware Root-Cause Ranker", "description": "Generate a service graph from traces, ingest deployment events and rank probable root causes for each correlated incident. Validate it with Chaos Mesh fault injection and publish an accuracy report covering top-1 and top-3 hits.", "difficulty": "Advanced", "skills": [ "service topology", "change correlation", "root-cause analysis", "chaos engineering" ] }, { "title": "Closed-Loop Remediation with ServiceNow", "description": "Using a ServiceNow developer instance, open enriched incidents automatically from correlated alerts, request approval in chat, run an Event-Driven Ansible remediation and resolve the ticket with an audit note. Include rate limits, rollback and a dry-run mode.", "difficulty": "Advanced", "skills": [ "Event-Driven Ansible", "ServiceNow integration", "runbook automation", "change control" ] }, { "title": "MCP-Connected AI SRE Agent with Guardrails", "description": "Build a LangGraph or HolmesGPT-based agent that investigates alerts through MCP servers for Grafana, Kubernetes and your incident tool, drafts RCA and a fix, and executes only after human approval. Ship an evaluation harness over 20 or more replayed incidents and OpenTelemetry traces of every agent step.", "difficulty": "Advanced", "skills": [ "agentic AI", "Model Context Protocol", "human-in-the-loop design", "LLM evaluation" ] } ], "certifications": [ { "name": "AIOps Certified Professional (AIOCP)", "provider": "DevOpsSchool", "url": "https://www.devopsschool.com/certification/aiops-certified-professional.html", "level": "Professional", "note": "Hands-on, project-based programme that maps directly onto this path, from observability and event correlation to ML-driven detection and automated remediation." }, { "name": "OpenTelemetry Certified Associate (OTCA)", "provider": "CNCF / Linux Foundation", "url": "https://training.linuxfoundation.org/certification/opentelemetry-certified-associate-otca/", "level": "Associate", "note": "Validates the OpenTelemetry instrumentation, Collector and semantic-convention skills that underpin every correlation, ML and agent use case." }, { "name": "Prometheus Certified Associate (PCA)", "provider": "CNCF / Linux Foundation", "url": "https://training.linuxfoundation.org/certification/prometheus-certified-associate/", "level": "Associate", "note": "Proves metrics, PromQL and alerting fundamentals, the most common data source for open-source AIOps pipelines." }, { "name": "Dynatrace Associate", "provider": "Dynatrace", "url": "https://university.dynatrace.com/certification", "level": "Associate", "note": "Entry certification for Dynatrace, the reference platform for topology-based causal root-cause analysis." }, { "name": "Datadog Fundamentals", "provider": "Datadog", "url": "https://www.datadoghq.com/certification/overview/", "level": "Foundational", "note": "Covers Datadog data collection, tagging and visualisation, which you need before Watchdog, event correlation and Bits AI are useful." }, { "name": "Splunk O11y Cloud Certified Metrics User", "provider": "Splunk (Cisco)", "url": "https://www.splunk.com/en_us/training/certification-track/splunk-o11y-cloud-certified-metrics-user.html", "level": "Foundational", "note": "Current Splunk observability credential covering metrics, detectors, OpenTelemetry and Kubernetes; the older ITSI admin certification became a legacy credential in 2026." } ], "tools_catalog": [ { "name": "Ansible", "category": "Infrastructure as Code & Automation", "website": "https://docs.ansible.com/", "ds_course_url": "https://www.devopsschool.com/courses/ansible/" }, { "name": "Apache Kafka", "category": "Data Engineering", "website": "https://kafka.apache.org/", "ds_course_url": "https://www.devopsschool.com/courses/kafka/" }, { "name": "Argo Rollouts", "category": "CI/CD & Release", "website": "https://argoproj.github.io/rollouts/", "ds_course_url": "https://www.devopsschool.com/courses/argo/" }, { "name": "AWS", "category": "Cloud Platforms & Governance", "website": "https://aws.amazon.com/", "ds_course_url": "https://www.devopsschool.com/courses/aws/" }, { "name": "AWS DevOps Agent", "category": "AIOps & Incident Management", "website": "https://aws.amazon.com/devops-agent/", "ds_course_url": "" }, { "name": "Azure SRE Agent", "category": "AIOps & Incident Management", "website": "https://learn.microsoft.com/en-us/azure/sre-agent/overview", "ds_course_url": "" }, { "name": "Backstage", "category": "Platform Engineering", "website": "https://backstage.io/", "ds_course_url": "" }, { "name": "BigPanda", "category": "AIOps & Incident Management", "website": "https://www.bigpanda.io/", "ds_course_url": "" }, { "name": "Chaos Mesh", "category": "Reliability & Resilience", "website": "https://chaos-mesh.org/", "ds_course_url": "https://www.devopsschool.com/courses/chaos-engineering/" }, { "name": "Datadog", "category": "Observability", "website": "https://www.datadoghq.com/", "ds_course_url": "https://www.devopsschool.com/courses/datadog/" }, { "name": "Dynatrace", "category": "Observability", "website": "https://www.dynatrace.com/", "ds_course_url": "https://www.devopsschool.com/courses/dynatrace/" }, { "name": "Elastic Stack", "category": "Observability", "website": "https://www.elastic.co/elastic-stack", "ds_course_url": "https://www.devopsschool.com/courses/elastic/" }, { "name": "Grafana", "category": "Observability", "website": "https://grafana.com/", "ds_course_url": "https://www.devopsschool.com/courses/grafana/" }, { "name": "HolmesGPT", "category": "AIOps & Incident Management", "website": "https://holmesgpt.dev/", "ds_course_url": "" }, { "name": "Hugging Face", "category": "LLMs, Agents & GenAI", "website": "https://huggingface.co/", "ds_course_url": "" }, { "name": "Jaeger", "category": "Observability", "website": "https://www.jaegertracing.io/", "ds_course_url": "https://www.devopsschool.com/courses/jaeger/" }, { "name": "K8sGPT", "category": "AIOps & Incident Management", "website": "https://k8sgpt.ai/", "ds_course_url": "" }, { "name": "Keep", "category": "AIOps & Incident Management", "website": "https://www.keephq.dev/", "ds_course_url": "" }, { "name": "Kubernetes", "category": "Containers & Kubernetes", "website": "https://kubernetes.io/", "ds_course_url": "https://www.devopsschool.com/courses/kubernetes/" }, { "name": "LangGraph", "category": "LLMs, Agents & GenAI", "website": "https://www.langchain.com/langgraph", "ds_course_url": "" }, { "name": "LitmusChaos", "category": "Reliability & Resilience", "website": "https://litmuschaos.io/", "ds_course_url": "https://www.devopsschool.com/courses/chaos-engineering/" }, { "name": "Loki", "category": "Observability", "website": "https://grafana.com/oss/loki/", "ds_course_url": "" }, { "name": "MLflow", "category": "Machine Learning & MLOps", "website": "https://mlflow.org/", "ds_course_url": "https://www.devopsschool.com/courses/mlops/" }, { "name": "Model Context Protocol", "category": "LLMs, Agents & GenAI", "website": "https://modelcontextprotocol.io/", "ds_course_url": "" }, { "name": "New Relic", "category": "Observability", "website": "https://newrelic.com/", "ds_course_url": "https://www.devopsschool.com/courses/newrelic/" }, { "name": "OPA", "category": "Policy as Code & Secrets", "website": "https://www.openpolicyagent.org/", "ds_course_url": "https://www.devopsschool.com/courses/opa/" }, { "name": "OpenTelemetry", "category": "Observability", "website": "https://opentelemetry.io/", "ds_course_url": "https://www.devopsschool.com/courses/opentelemetry/" }, { "name": "PagerDuty", "category": "AIOps & Incident Management", "website": "https://www.pagerduty.com/", "ds_course_url": "" }, { "name": "Prometheus", "category": "Observability", "website": "https://prometheus.io/", "ds_course_url": "https://www.devopsschool.com/courses/prometheus/" }, { "name": "PyOD", "category": "AIOps & Incident Management", "website": "https://pyod.readthedocs.io/en/latest/", "ds_course_url": "" }, { "name": "Python", "category": "Foundations & Languages", "website": "https://www.python.org/", "ds_course_url": "https://www.devopsschool.com/courses/python/" }, { "name": "Rundeck", "category": "AIOps & Incident Management", "website": "https://www.rundeck.com/", "ds_course_url": "https://www.devopsschool.com/courses/rundeck/" }, { "name": "scikit-learn", "category": "Machine Learning & MLOps", "website": "https://scikit-learn.org/", "ds_course_url": "https://www.devopsschool.com/courses/ml/" }, { "name": "ServiceNow", "category": "AIOps & Incident Management", "website": "https://www.servicenow.com/products/it-operations-management.html", "ds_course_url": "https://www.devopsschool.com/certification/master-in-servicenow.html" }, { "name": "Splunk", "category": "Observability", "website": "https://www.splunk.com/", "ds_course_url": "https://www.devopsschool.com/courses/splunk/" }, { "name": "StatsForecast", "category": "AIOps & Incident Management", "website": "https://nixtlaverse.nixtla.io/statsforecast", "ds_course_url": "" }, { "name": "Tempo", "category": "Observability", "website": "https://grafana.com/oss/tempo/", "ds_course_url": "" } ], "faqs": [ { "q": "What is AIOps and how is it different from observability?", "a": "Observability is about collecting and exploring telemetry so humans can understand system behaviour. AIOps applies analytics, machine learning and now LLM agents on top of that telemetry to reduce alert noise, detect anomalies, find root cause and automate the response. You cannot do AIOps well without good observability, which is why this path starts there." }, { "q": "Is AIOps still relevant now that AI SRE agents exist?", "a": "Yes. Agents are the newest layer of AIOps, not a replacement for it. They depend on clean telemetry, deduplicated and correlated events, accurate topology and codified runbooks, and they work best when deterministic detection and RCA give them reliable evidence. Teams that skip the foundations get agents that confidently investigate the wrong thing." }, { "q": "Do I need a data science background to learn AIOps?", "a": "No. You need solid Python, basic statistics and an understanding of time series, all of which this path teaches in context. Most day-to-day AIOps work is data engineering, correlation design, evaluation and automation rather than inventing new algorithms." }, { "q": "Which AIOps certification should I get first?", "a": "Start with a vendor-neutral data credential such as the OpenTelemetry Certified Associate or Prometheus Certified Associate, then add a hands-on programme like DevOpsSchool's AIOps Certified Professional. Add a vendor credential such as Dynatrace Associate or Datadog Fundamentals when your employer standardises on that platform." }, { "q": "Can I learn AIOps with only open-source tools?", "a": "Yes. OpenTelemetry, Prometheus, Grafana, Loki, Tempo, Keep, Python ML libraries, Event-Driven Ansible, HolmesGPT and K8sGPT cover the full loop from telemetry to agentic investigation. Commercial platforms add scale, polish and support, and knowing the open-source mechanics makes you much better at evaluating them." }, { "q": "Will AI SRE agents replace SREs and NOC engineers?", "a": "They are replacing much of the repetitive first-pass triage, not the people accountable for reliability. The work shifts towards curating telemetry and runbooks, setting guardrails and autonomy levels, reviewing agent findings and handling the novel incidents agents cannot. Engineers who can build and govern these systems are in high demand." }, { "q": "How long does it take to become job-ready in AIOps?", "a": "At about 10 hours a week this path takes roughly six to seven months. Engineers with existing SRE or observability experience can move faster through the first two stages, while the ML and agentic stages reward extra project time." } ], "related": [ "sre", "mlops", "llmops", "cloudops" ], "key_tools": [ "OpenTelemetry", "PagerDuty", "Dynatrace", "Datadog", "ServiceNow", "BigPanda" ], "course_url": "https://www.devopsschool.com/courses/aiops/", "seo_title": "AIOps Learning Path 2026: Roadmap, Tools & AI SRE Agents", "seo_description": "Learn AIOps in 2026: OpenTelemetry, event correlation, anomaly detection, root-cause analysis, automated remediation and MCP-connected AI SRE agents.", "url": "https://www.devopsschool.com/path/aiops/", "updated_at": "2026-10-26 08:58:45" }