{ "schema": 1, "slug": "sre", "title": "SRE", "full_name": "Site Reliability Engineering", "category": "Reliability & Operations", "tagline": "Run production like a software problem: SLOs, observability, incident response and automation at scale.", "summary": "A practical 2026 Site Reliability Engineering roadmap: Linux and code foundations, Kubernetes and cloud architecture, OpenTelemetry and Prometheus observability, SLOs and error budgets, incident response, chaos engineering, capacity planning and reliability for AI systems. Every stage ends with a hands-on checkpoint and portfolio projects.", "description": "Site Reliability Engineering (SRE) is what happens when you treat operations as a software problem. Pioneered at Google, it gives teams a shared, numeric definition of reliable enough through service level objectives (SLOs) and error budgets, then uses engineering (automation, observability, safe release practices and disciplined incident response) to stay within that target while still shipping quickly.\n\nReliability matters more in 2026 than ever. Users expect always-on services, large cloud and CDN outages in late 2025 showed how a single bad configuration change can cascade globally, and regulations such as the EU Digital Operational Resilience Act now require tested resilience in financial services. AI is changing the job from both directions: agents help triage incidents, and LLM features bring new failure modes such as GPU shortages, provider rate limits and token latency.\n\nThis path starts with Linux, networking and code, moves through Kubernetes, cloud architecture and OpenTelemetry-based observability, and then goes deep on SLOs, on-call, incident management, progressive delivery, chaos engineering, capacity planning and disaster recovery. It finishes with toil automation, AI-assisted operations and reliability for AI systems. SRE overlaps heavily with DevOps (shared ownership of delivery), platform engineering (the paved roads SREs help build) and AIOps (machine-assisted operations).", "icon": "monitor_heart", "accent": "#F59E0B", "level": "Beginner → Advanced", "duration_weeks": 25, "weekly_hours": 10, "audience": [ "Software engineers who want to own the production reliability of the services they build", "System administrators and operations engineers moving from manual operations to automation", "DevOps and cloud engineers who want to master SLOs, observability and incident response", "Engineering leads setting up on-call, incident and reliability practices for their teams" ], "prerequisites": [ "Comfort with the Linux command line and basic networking", "Programming fundamentals in at least one language (Python or Go preferred)", "Familiarity with Git and how applications are built and deployed", "Basic understanding of cloud services and containers (refreshed in Stage 2)" ], "outcomes": [ "Define user-centric SLIs and SLOs, manage them as code and run an error budget policy with product teams", "Instrument services with OpenTelemetry and build Prometheus and Grafana observability with low-noise, burn-rate alerting", "Design sustainable on-call rotations and lead major incidents as incident commander", "Write blameless postmortems that produce tracked, systemic improvements", "Ship changes safely with canary analysis, automated rollback and staged configuration rollouts", "Validate resilience with chaos experiments, load tests and timed disaster recovery drills", "Measure toil and automate it away with tested tools, operators and guarded auto-remediation", "Operate AI and LLM inference services against latency SLOs and use AI agents safely in operations" ], "trends": [ "OpenTelemetry is now the default for instrumentation: the Profiles signal entered public alpha in March 2026, and eBPF-based zero-code instrumentation (OBI) produces traces and RED metrics without code changes.", "Prometheus 3.x made native histograms stable in v3.8, giving cheaper, higher-resolution latency SLIs, and it can ingest OTLP metrics directly.", "AI SRE agents went mainstream: CNCF Sandbox projects such as HolmesGPT and K8sGPT, plus vendor agents, now investigate alerts and draft postmortems, making guardrails and evaluation an SRE responsibility.", "Reliability for AI systems became a specialism, with SLOs for time to first token and inter-token latency, GPU capacity planning and OpenTelemetry GenAI semantic conventions.", "Late-2025 outages, including the AWS us-east-1 DynamoDB DNS failure and Cloudflare's bot-management configuration incident, renewed focus on staged config rollouts, dependency isolation and static stability.", "Operational resilience is increasingly regulated: the EU Digital Operational Resilience Act (DORA) has applied to financial entities since January 2025, requiring ICT incident reporting and resilience testing." ], "roles": [ { "title": "Site Reliability Engineer", "description": "Owns SLOs, observability, on-call and automation for production services, balancing feature velocity against reliability with error budgets." }, { "title": "Production / Infrastructure Engineer", "description": "Builds and runs the compute, networking and Kubernetes platforms that services depend on, with a focus on capacity, upgrades and resilience." }, { "title": "Observability Engineer", "description": "Designs telemetry pipelines, instrumentation standards, dashboards and alerting so teams can understand their systems quickly and affordably." }, { "title": "Incident Manager / Reliability Program Lead", "description": "Runs the incident process, postmortem reviews and reliability reporting across teams, and drives systemic fixes to completion." }, { "title": "AI Infrastructure SRE", "description": "Keeps GPU clusters and LLM inference services fast and available, planning GPU capacity and defining latency SLOs for AI features." } ], "stages": [ { "title": "Foundations: Linux, Networking & Code", "level": "Beginner", "duration_weeks": 4, "description": "The systems, networking and programming skills every SRE relies on, plus the principles that make SRE different from traditional operations.", "milestone": "You can triage a slow Linux service with the USE method and replace a manual fix with a tested Python or Go tool.", "topics": [ { "title": "Linux Internals & Performance Triage", "description": "Every SRE investigation eventually lands on a host, so you must read what the kernel is telling you. Learn processes, memory, file descriptors and systemd, and practise the USE method (utilisation, saturation, errors) to find bottlenecks quickly.", "est_hours": 12, "optional": false, "concepts": [ "processes, signals and systemd", "memory, page cache and the OOM killer", "file descriptors and limits", "load average vs CPU saturation", "USE method", "disk and network I/O tools", "/proc and /sys" ], "tools": [ "Linux", "Bash" ], "resources": [ { "title": "The Linux Command Line (free book)", "url": "https://linuxcommand.org/tlcl.php", "type": "book", "free": true }, { "title": "The USE Method (Brendan Gregg)", "url": "https://www.brendangregg.com/usemethod.html", "type": "article", "free": true }, { "title": "Linux Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/linux/", "type": "course", "free": false } ] }, { "title": "Networking for Reliability", "description": "Many outages are network problems in disguise: DNS, timeouts, connection limits or a misbehaving load balancer. Understand TCP behaviour, DNS caching, TLS handshakes and L4/L7 load balancing well enough to reason about latency and failure modes.", "est_hours": 8, "optional": false, "concepts": [ "TCP handshake, retransmits and keepalives", "DNS resolution and TTLs", "TLS handshake costs", "L4 vs L7 load balancing", "HTTP/2 and gRPC", "connection pooling", "timeouts and deadlines" ], "tools": [], "resources": [ { "title": "High Performance Browser Networking (free book)", "url": "https://hpbn.co/", "type": "book", "free": true }, { "title": "Load Balancing at the Frontend (Google SRE book)", "url": "https://sre.google/sre-book/load-balancing-frontend/", "type": "book", "free": true } ] }, { "title": "Programming for Automation: Python & Go", "description": "SREs spend at least half their time engineering rather than firefighting, so solid programming is non-negotiable. Use Python for scripts, API automation and data analysis, and Go for fast tooling, exporters and Kubernetes controllers, with tests and code review like any production software.", "est_hours": 14, "optional": false, "concepts": [ "CLI tools and API clients", "error handling and retries", "unit and integration tests", "concurrency in Go", "packaging and versioning tools", "parsing logs and metrics data", "Git workflows and code review" ], "tools": [ "Python", "Go", "Git" ], "resources": [ { "title": "Python Documentation", "url": "https://docs.python.org/3/", "type": "docs", "free": true }, { "title": "Go Documentation", "url": "https://go.dev/doc/", "type": "docs", "free": true }, { "title": "Python Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/python/", "type": "course", "free": false } ] }, { "title": "SRE Principles & Culture", "description": "SRE treats operations as a software problem, accepts that 100% reliability is the wrong target and manages risk explicitly. Learn the core tenets and how SRE relates to DevOps and platform engineering so you can explain and adapt the role in any organisation.", "est_hours": 6, "optional": false, "concepts": [ "operations as a software problem", "embracing risk", "the 50% cap on operational work", "SRE vs DevOps vs platform engineering", "shared ownership with product teams", "SRE team models" ], "tools": [], "resources": [ { "title": "Site Reliability Engineering (Google SRE book, free online)", "url": "https://sre.google/sre-book/table-of-contents/", "type": "book", "free": true }, { "title": "Embracing Risk (Google SRE book)", "url": "https://sre.google/sre-book/embracing-risk/", "type": "book", "free": true }, { "title": "How SRE Relates to DevOps (SRE Workbook)", "url": "https://sre.google/workbook/how-sre-relates/", "type": "book", "free": true } ] } ] }, { "title": "Cloud-Native Production Platforms", "level": "Intermediate", "duration_weeks": 4, "description": "Run services on Kubernetes, manage infrastructure and deployments from Git, and design cloud architectures around real failure domains.", "milestone": "You can deploy a multi-zone service to Kubernetes entirely from Git, with health probes, disruption budgets and infrastructure you can rebuild on demand.", "topics": [ { "title": "Kubernetes Operations for Reliability", "description": "Kubernetes is the default runtime for production services, and most of its reliability features are configuration you must get right. Learn probes, resource requests and limits, PodDisruptionBudgets, autoscaling and safe rolling updates, and practise debugging failing pods and nodes.", "est_hours": 16, "optional": false, "concepts": [ "liveness, readiness and startup probes", "requests, limits and QoS classes", "PodDisruptionBudgets", "rolling updates and rollbacks", "topology spread and anti-affinity", "cluster upgrades", "debugging CrashLoopBackOff and OOMKilled" ], "tools": [ "Docker", "Kubernetes", "Helm" ], "resources": [ { "title": "Kubernetes Documentation", "url": "https://kubernetes.io/docs/home/", "type": "docs", "free": true }, { "title": "Kubernetes Docs: Disruptions", "url": "https://kubernetes.io/docs/concepts/workloads/pods/disruptions/", "type": "docs", "free": true }, { "title": "Kubernetes Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/kubernetes/", "type": "course", "free": false } ] }, { "title": "Infrastructure as Code & GitOps", "description": "Reproducible infrastructure is a reliability feature: if you can rebuild it from Git, you can review it, roll it back and recover it. Manage cloud resources with Terraform or OpenTofu and deliver cluster state with Argo CD so every production change is versioned and auditable.", "est_hours": 12, "optional": false, "concepts": [ "declarative infrastructure", "modules and remote state", "plan review and drift detection", "immutable infrastructure", "GitOps reconciliation", "config changes as code changes", "environment promotion" ], "tools": [ "Terraform", "OpenTofu", "Argo CD" ], "resources": [ { "title": "Terraform Documentation", "url": "https://developer.hashicorp.com/terraform/docs", "type": "docs", "free": true }, { "title": "Argo CD Documentation", "url": "https://argo-cd.readthedocs.io/", "type": "docs", "free": true }, { "title": "Terraform Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/terraform/", "type": "course", "free": false } ] }, { "title": "Cloud Reliability Architecture", "description": "Cloud providers give you failure domains, not reliability; you still have to design for them. Learn how regions and availability zones fail, when multi-AZ or multi-region is worth the cost, static stability and cell-based isolation, and how dependency SLAs compose into your own.", "est_hours": 12, "optional": false, "concepts": [ "regions, zones and failure domains", "multi-AZ vs multi-region trade-offs", "static stability", "cell-based architecture", "composite availability of dependencies", "control plane vs data plane", "well-architected reliability reviews" ], "tools": [ "AWS", "Google Cloud", "Microsoft Azure" ], "resources": [ { "title": "AWS Well-Architected Framework: Reliability Pillar", "url": "https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html", "type": "guide", "free": true }, { "title": "Google Cloud Well-Architected Framework: Reliability", "url": "https://docs.cloud.google.com/architecture/framework/reliability", "type": "guide", "free": true }, { "title": "Azure Well-Architected Framework: Reliability", "url": "https://learn.microsoft.com/en-us/azure/well-architected/reliability/", "type": "guide", "free": true } ] } ] }, { "title": "Observability & SLOs", "level": "Intermediate", "duration_weeks": 5, "description": "Collect metrics, traces and logs with Prometheus and OpenTelemetry, then turn them into SLOs, error budgets and alerts that page only when users are affected.", "milestone": "You can instrument a microservice system end to end, define SLOs as code and page only on fast error-budget burn.", "topics": [ { "title": "Metrics & Monitoring with Prometheus", "description": "Prometheus and PromQL remain the common language of cloud-native metrics. Learn the data model, the four golden signals and the RED and USE methods, recording rules and how to keep cardinality and cost under control; Prometheus 3 also brings stable native histograms for cheaper, more precise latency SLIs.", "est_hours": 12, "optional": false, "concepts": [ "four golden signals, RED and USE", "PromQL rates and aggregations", "recording rules", "native histograms and percentiles", "label cardinality control", "exporters and service discovery", "remote write and long-term storage" ], "tools": [ "Prometheus", "Grafana" ], "resources": [ { "title": "Prometheus Documentation: Overview", "url": "https://prometheus.io/docs/introduction/overview/", "type": "docs", "free": true }, { "title": "Monitoring Distributed Systems (Google SRE book)", "url": "https://sre.google/sre-book/monitoring-distributed-systems/", "type": "book", "free": true }, { "title": "Prometheus Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/prometheus/", "type": "course", "free": false } ] }, { "title": "OpenTelemetry & Distributed Tracing", "description": "OpenTelemetry is the vendor-neutral standard for traces, metrics and logs, with profiles now in alpha. Instrument services with SDKs or zero-code agents, route telemetry through the Collector, propagate context across services and choose sampling strategies that keep the traces you actually need.", "est_hours": 14, "optional": false, "concepts": [ "traces, spans and context propagation", "SDK vs zero-code instrumentation", "eBPF instrumentation (OBI)", "Collector receivers, processors and exporters", "semantic conventions", "head vs tail-based sampling", "profiles as the fourth signal" ], "tools": [ "OpenTelemetry", "Jaeger", "Tempo" ], "resources": [ { "title": "OpenTelemetry Documentation", "url": "https://opentelemetry.io/docs/", "type": "docs", "free": true }, { "title": "OpenTelemetry Collector Documentation", "url": "https://opentelemetry.io/docs/collector/", "type": "docs", "free": true }, { "title": "OpenTelemetry Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/opentelemetry/", "type": "course", "free": false } ] }, { "title": "Logs & Signal Correlation", "description": "Logs are usually the most expensive signal, so be deliberate: structure them, index only what you query and link them to traces. Learn label-based log storage, log-to-trace correlation with trace IDs and exemplars, and retention policies that balance cost with forensic needs.", "est_hours": 6, "optional": false, "concepts": [ "structured JSON logging", "label vs full-text indexing", "trace ID correlation", "exemplars", "retention tiers and cost", "LogQL basics" ], "tools": [ "Loki", "Elastic Stack" ], "resources": [ { "title": "Grafana Loki Documentation", "url": "https://grafana.com/docs/loki/latest/", "type": "docs", "free": true }, { "title": "Master Elasticsearch, Logstash and Kibana (ELK Stack) — DevOpsSchool", "url": "https://www.devopsschool.com/certification/master-elasticsearch-logstash-kibana-elk-stack-training.html", "type": "course", "free": false } ] }, { "title": "SLIs, SLOs & Error Budgets", "description": "SLOs are the heart of SRE: they turn the question of whether a service is reliable enough into a number that product and engineering agree on. Choose user-centric SLIs, set realistic targets and windows, write an error budget policy, and manage SLOs as code so they are versioned and generate alerts automatically.", "est_hours": 10, "optional": false, "concepts": [ "user-centric SLIs (availability, latency, freshness, correctness)", "SLO targets and rolling windows", "error budgets and budget policies", "SLAs vs SLOs", "SLOs as code (OpenSLO, Sloth, Pyrra)", "dependency and composite SLOs", "reviewing SLOs with stakeholders" ], "tools": [ "Sloth", "Pyrra" ], "resources": [ { "title": "Service Level Objectives (Google SRE book)", "url": "https://sre.google/sre-book/service-level-objectives/", "type": "book", "free": true }, { "title": "Implementing SLOs (SRE Workbook)", "url": "https://sre.google/workbook/implementing-slos/", "type": "book", "free": true }, { "title": "Example Error Budget Policy (SRE Workbook)", "url": "https://sre.google/workbook/error-budget-policy/", "type": "guide", "free": true } ] }, { "title": "SLO-Based Alerting & Dashboards", "description": "Page humans only when users are hurting or soon will be. Replace static threshold alerts with multi-window, multi-burn-rate SLO alerts, route and silence them sensibly, and design dashboards that answer on-call questions in seconds.", "est_hours": 8, "optional": false, "concepts": [ "symptom-based vs cause-based alerts", "multi-window, multi-burn-rate alerts", "Alertmanager routing, grouping and inhibition", "page vs ticket severity", "alert fatigue reviews", "dashboard design for on-call" ], "tools": [ "Prometheus", "Grafana" ], "resources": [ { "title": "Alerting on SLOs (SRE Workbook)", "url": "https://sre.google/workbook/alerting-on-slos/", "type": "book", "free": true }, { "title": "Prometheus Docs: Alerting best practices", "url": "https://prometheus.io/docs/practices/alerting/", "type": "docs", "free": true }, { "title": "Grafana Alerting Documentation", "url": "https://grafana.com/docs/grafana/latest/alerting/", "type": "docs", "free": true } ] } ] }, { "title": "Incident Management & On-Call", "level": "Intermediate", "duration_weeks": 3, "description": "Build humane on-call, run major incidents with clear roles, and turn every incident into lasting improvements through blameless postmortems.", "milestone": "You can lead a simulated SEV-1 as incident commander and publish a blameless postmortem with owned, tracked action items.", "topics": [ { "title": "Sustainable On-Call", "description": "On-call is where SRE promises meet human limits. Design rotations, escalation policies and runbooks that keep load sustainable, measure pages per shift, and make handoffs routine so critical knowledge does not live in one person's head.", "est_hours": 8, "optional": false, "concepts": [ "rotation design and follow-the-sun", "escalation policies", "actionable runbooks", "pages per shift and on-call load", "handoffs and shadowing", "alert review rituals", "compensation and wellbeing" ], "tools": [ "PagerDuty" ], "resources": [ { "title": "Being On-Call (Google SRE book)", "url": "https://sre.google/sre-book/being-on-call/", "type": "book", "free": true }, { "title": "On-Call (SRE Workbook)", "url": "https://sre.google/workbook/on-call/", "type": "book", "free": true } ] }, { "title": "Incident Response & Command", "description": "Major incidents are coordination problems as much as technical ones. Learn severity levels, the Incident Commander, Operations and Communications roles, mitigation-first thinking and stakeholder updates, and rehearse them in drills before you need them.", "est_hours": 10, "optional": false, "concepts": [ "severity levels and declaration criteria", "Incident Commander, Ops and Comms roles", "mitigate first, investigate later", "status pages and stakeholder updates", "incident channels and tooling", "AI assistants in incident response", "handover between responders" ], "tools": [ "PagerDuty" ], "resources": [ { "title": "PagerDuty Incident Response Documentation", "url": "https://response.pagerduty.com/", "type": "guide", "free": true }, { "title": "Google SRE Incident Management Guide", "url": "https://sre.google/resources/practices-and-processes/incident-management-guide/", "type": "guide", "free": true }, { "title": "Managing Incidents (Google SRE book)", "url": "https://sre.google/sre-book/managing-incidents/", "type": "book", "free": true } ] }, { "title": "Blameless Postmortems & Learning from Incidents", "description": "Incidents are unplanned investments, and postmortems are how you collect the return. Write blameless postmortems with clear timelines and contributing factors, track action items to completion, and share learnings so the whole organisation improves.", "est_hours": 8, "optional": false, "concepts": [ "blameless and just culture", "timelines and contributing factors", "moving beyond a single root cause", "action item tracking", "postmortem review meetings", "incident trend analysis", "sharing learnings widely" ], "tools": [], "resources": [ { "title": "Postmortem Culture: Learning from Failure (Google SRE book)", "url": "https://sre.google/sre-book/postmortem-culture/", "type": "book", "free": true }, { "title": "PagerDuty Postmortem Documentation", "url": "https://postmortems.pagerduty.com/", "type": "guide", "free": true }, { "title": "The VOID: Verica Open Incident Database", "url": "https://www.thevoid.community/", "type": "report", "free": true } ] }, { "title": "Effective Troubleshooting", "description": "Good troubleshooting is a repeatable method, not heroics. Form and test hypotheses against telemetry, ask what changed, bisect deployments and configuration, and keep working notes so others can follow and verify your reasoning.", "est_hours": 4, "optional": false, "concepts": [ "hypothesis-driven debugging", "what changed: deploys, config, traffic", "bisecting changes", "observability-driven investigation", "negative results matter", "working notes during incidents" ], "tools": [], "resources": [ { "title": "Effective Troubleshooting (Google SRE book)", "url": "https://sre.google/sre-book/effective-troubleshooting/", "type": "book", "free": true }, { "title": "Linux Performance (Brendan Gregg)", "url": "https://www.brendangregg.com/linuxperf.html", "type": "article", "free": true } ] } ] }, { "title": "Resilience Engineering: Releases, Chaos & Capacity", "level": "Advanced", "duration_weeks": 5, "description": "Make change safe, design services that degrade gracefully, prove resilience with chaos experiments and back capacity and recovery plans with measurements.", "milestone": "You can ship changes behind automated canaries, prove resilience with chaos experiments, and back capacity and recovery plans with measured data.", "topics": [ { "title": "Safe Releases & Progressive Delivery", "description": "Most outages are triggered by change, and global configuration pushes are as dangerous as code, as major 2025 cloud and CDN outages showed. Use canaries, feature flags and staged rollouts with automated analysis so bad changes roll back before most users notice.", "est_hours": 10, "optional": false, "concepts": [ "canary and blue-green releases", "automated canary analysis against SLOs", "feature flags and dark launches", "staged configuration rollouts", "fast, tested rollback", "release freezes driven by error budgets" ], "tools": [ "Argo Rollouts" ], "resources": [ { "title": "Canarying Releases (SRE Workbook)", "url": "https://sre.google/workbook/canarying-releases/", "type": "book", "free": true }, { "title": "Argo Rollouts Documentation", "url": "https://argo-rollouts.readthedocs.io/", "type": "docs", "free": true }, { "title": "Argo Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/argo/", "type": "course", "free": false } ] }, { "title": "Resilience Patterns & Overload Handling", "description": "Distributed systems fail partially, and naive retries turn small failures into cascading outages. Learn timeouts, retries with backoff and jitter, circuit breaking, load shedding and graceful degradation, and verify that each dependency failure degrades the service rather than breaking it.", "est_hours": 10, "optional": false, "concepts": [ "timeouts and deadline propagation", "retries with exponential backoff and jitter", "circuit breakers and bulkheads", "load shedding and rate limiting", "graceful degradation", "cascading failure prevention", "service mesh traffic policies" ], "tools": [ "Istio" ], "resources": [ { "title": "Handling Overload (Google SRE book)", "url": "https://sre.google/sre-book/handling-overload/", "type": "book", "free": true }, { "title": "Addressing Cascading Failures (Google SRE book)", "url": "https://sre.google/sre-book/addressing-cascading-failures/", "type": "book", "free": true }, { "title": "Timeouts, Retries and Backoff with Jitter (Amazon Builders' Library)", "url": "https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/", "type": "article", "free": true } ] }, { "title": "Chaos Engineering & GameDays", "description": "Chaos engineering verifies resilience with controlled experiments instead of hope. Define steady state, form a hypothesis, inject faults with a limited blast radius and run regular GameDays, starting in staging and graduating to production with guardrails.", "est_hours": 10, "optional": false, "concepts": [ "steady-state hypothesis", "blast radius and abort conditions", "pod, network and node fault injection", "GameDays and DR drills", "chaos experiments in CI/CD", "turning findings into fixes" ], "tools": [ "LitmusChaos", "Chaos Mesh" ], "resources": [ { "title": "Principles of Chaos Engineering", "url": "https://principlesofchaos.org/", "type": "article", "free": true }, { "title": "LitmusChaos Documentation", "url": "https://docs.litmuschaos.io/", "type": "docs", "free": true }, { "title": "Chaos Engineering Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/chaos-engineering/", "type": "course", "free": false } ] }, { "title": "Capacity Planning, Load Testing & Autoscaling", "description": "Capacity planning keeps you ahead of demand without paying for idle hardware. Forecast load from business drivers, find saturation points with load tests, keep N+2 headroom and use pod and node autoscaling, treating cost as a reliability trade-off rather than an afterthought.", "est_hours": 12, "optional": false, "concepts": [ "demand forecasting: organic and inorganic growth", "load and stress testing", "saturation points and N+2 headroom", "Little's law and queueing basics", "pod autoscaling on custom and event metrics", "node autoscaling and bin-packing", "cost-aware capacity decisions" ], "tools": [ "k6", "KEDA", "Karpenter" ], "resources": [ { "title": "Kubernetes Docs: Autoscaling Workloads", "url": "https://kubernetes.io/docs/concepts/workloads/autoscaling/", "type": "docs", "free": true }, { "title": "Grafana k6 Documentation", "url": "https://grafana.com/docs/k6/latest/", "type": "docs", "free": true }, { "title": "KEDA Documentation", "url": "https://keda.sh/", "type": "docs", "free": true } ] }, { "title": "Backup & Disaster Recovery", "description": "A backup that has never been restored is a hope, not a plan. Define RPO and RTO per service, automate backups of data and cluster state, and run timed restore and regional failover drills so recovery is a practised routine.", "est_hours": 8, "optional": false, "concepts": [ "RPO and RTO targets", "backup vs replication", "restore testing", "Kubernetes backup and migration", "multi-region failover runbooks", "recovery dependencies on control planes" ], "tools": [ "Velero" ], "resources": [ { "title": "Data Integrity: What You Read Is What You Wrote (Google SRE book)", "url": "https://sre.google/sre-book/data-integrity/", "type": "book", "free": true }, { "title": "Velero Documentation", "url": "https://velero.io/", "type": "docs", "free": true }, { "title": "Velero Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/velero/", "type": "course", "free": false } ] } ] }, { "title": "Advanced SRE: Automation, AI & Leadership", "level": "Expert", "duration_weeks": 4, "description": "Eliminate toil at scale, use AI agents safely in operations, run AI and LLM services against SLOs, and scale reliability through process and influence.", "milestone": "You can automate away a top toil source, run an LLM inference service against SLOs with safe AI-assisted triage, and lead a production readiness review.", "topics": [ { "title": "Toil Reduction & Self-Healing Automation", "description": "Toil is manual, repetitive, automatable work that grows with load, and it crowds out engineering. Measure it, prioritise the worst offenders and automate them with tested tools, Kubernetes operators and guarded auto-remediation that knows when to stop and page a human.", "est_hours": 10, "optional": false, "concepts": [ "defining and measuring toil", "automation maturity levels", "runbook automation", "Kubernetes operators and controllers", "guarded auto-remediation", "idempotency and safety checks", "tracking engineering vs operations time" ], "tools": [ "Python", "Go", "Kubernetes" ], "resources": [ { "title": "Eliminating Toil (Google SRE book)", "url": "https://sre.google/sre-book/eliminating-toil/", "type": "book", "free": true }, { "title": "Eliminating Toil (SRE Workbook)", "url": "https://sre.google/workbook/eliminating-toil/", "type": "book", "free": true }, { "title": "The Evolution of Automation at Google (Google SRE book)", "url": "https://sre.google/sre-book/automation-at-google/", "type": "book", "free": true } ] }, { "title": "AI-Assisted Operations & SRE Agents", "description": "AI agents now investigate alerts, summarise incidents and propose fixes, and SREs are responsible for making them trustworthy. Learn how agentic troubleshooting works, connect agents to telemetry safely (read-only by default, scoped credentials, human approval for actions) and evaluate their suggestions against real past incidents.", "est_hours": 8, "optional": false, "concepts": [ "LLM-based alert triage and root cause analysis", "agents querying metrics, logs and traces", "MCP servers for observability tools", "read-only by default and human approval", "evaluating agent accuracy on past incidents", "prompt injection via logs and tickets", "AI-drafted incident summaries and postmortems" ], "tools": [ "HolmesGPT", "K8sGPT" ], "resources": [ { "title": "HolmesGPT: Agentic troubleshooting built for the cloud native era (CNCF blog)", "url": "https://www.cncf.io/blog/2026/01/07/holmesgpt-agentic-troubleshooting-built-for-the-cloud-native-era/", "type": "article", "free": true }, { "title": "K8sGPT Documentation", "url": "https://k8sgpt.ai/", "type": "docs", "free": true }, { "title": "AIOps Training — DevOpsSchool", "url": "https://www.devopsschool.com/courses/aiops/", "type": "course", "free": false } ] }, { "title": "Reliability for AI & LLM Systems", "description": "AI features bring new failure modes: GPU scarcity, long-tail latency, provider rate limits and silent quality regressions. Define SLIs such as time to first token and inter-token latency, monitor inference servers and GPUs, plan GPU capacity, and design fallbacks across models and providers.", "est_hours": 12, "optional": false, "concepts": [ "SLIs: time to first token, inter-token latency, throughput", "GPU utilisation and KV-cache pressure", "inference-aware load balancing", "provider rate limits, quotas and failover", "OpenTelemetry GenAI semantic conventions", "quality regressions as reliability incidents", "token cost as a capacity metric" ], "tools": [ "vLLM", "OpenTelemetry", "Kubernetes" ], "resources": [ { "title": "OpenTelemetry Semantic Conventions for Generative AI", "url": "https://opentelemetry.io/docs/specs/semconv/gen-ai/", "type": "docs", "free": true }, { "title": "vLLM Docs: Production Metrics", "url": "https://docs.vllm.ai/en/latest/usage/metrics/", "type": "docs", "free": true }, { "title": "Kubernetes Gateway API Inference Extension", "url": "https://gateway-api-inference-extension.sigs.k8s.io/", "type": "docs", "free": true } ] }, { "title": "Production Readiness & SRE Leadership", "description": "Senior SREs scale reliability through process and influence, not just code. Run production readiness reviews, choose the right SRE engagement model for each team, negotiate error budget policies with product owners, and report reliability with SLO and DORA metrics.", "est_hours": 10, "optional": false, "concepts": [ "production readiness reviews", "launch checklists", "SRE engagement models", "error budget negotiations", "reliability roadmaps and OKRs", "DORA software delivery metrics", "building and growing an SRE team" ], "tools": [], "resources": [ { "title": "Reliable Product Launches at Scale (Google SRE book)", "url": "https://sre.google/sre-book/reliable-product-launches/", "type": "book", "free": true }, { "title": "SRE Engagement Model (SRE Workbook)", "url": "https://sre.google/workbook/engagement-model/", "type": "book", "free": true }, { "title": "DORA Research and Capabilities", "url": "https://dora.dev/", "type": "report", "free": true } ] } ] } ], "projects": [ { "title": "Linux Performance Triage Toolkit", "description": "Write a Python or Go CLI that collects USE-method data (utilisation, saturation and errors for CPU, memory, disk and network) from a Linux host and prints a ranked list of likely bottlenecks. Ship it with tests, a README and a runbook showing it diagnosing three deliberately induced problems.", "difficulty": "Beginner", "skills": [ "Linux internals", "USE method", "Python or Go", "automated testing" ] }, { "title": "SLO-Driven Observability for a Microservices Demo", "description": "Deploy the OpenTelemetry Demo on Kubernetes with Prometheus, Grafana, Tempo and Loki. Define three user-journey SLOs as code with Sloth or Pyrra, wire multi-burn-rate alerts to PagerDuty, and publish an SLO dashboard together with a written error budget policy.", "difficulty": "Intermediate", "skills": [ "OpenTelemetry", "Prometheus", "SLOs as code", "burn-rate alerting", "Grafana dashboards" ] }, { "title": "Incident GameDay and Blameless Postmortem", "description": "Run a 90-minute GameDay in which a teammate injects a hidden fault into your demo app while you act as incident commander. Deliver the incident timeline, stakeholder updates, a blameless postmortem and tracked action items, and implement at least one of the fixes.", "difficulty": "Intermediate", "skills": [ "incident command", "troubleshooting", "postmortem writing", "communication" ] }, { "title": "Progressive Delivery with SLO-Gated Rollbacks", "description": "Use Argo Rollouts to canary a new version with Prometheus-based analysis, then inject latency and errors with Chaos Mesh or LitmusChaos during the rollout. Demonstrate the automated rollback and document the analysis queries, abort thresholds and what you changed after the experiment.", "difficulty": "Advanced", "skills": [ "Argo Rollouts", "canary analysis", "chaos engineering", "PromQL" ] }, { "title": "Capacity and Disaster Recovery Plan", "description": "Load test a service with k6 to find its saturation point, configure KEDA and Karpenter (or the cluster autoscaler) to keep N+2 headroom, and back up the cluster with Velero. Run a timed restore into a second cluster or region and publish the measured RPO and RTO against your targets.", "difficulty": "Advanced", "skills": [ "load testing", "autoscaling", "capacity planning", "backup and restore", "DR drills" ] }, { "title": "Operate an LLM Inference Service against SLOs", "description": "Serve an open model with vLLM on Kubernetes, export time to first token, inter-token latency and GPU or KV-cache metrics, and define SLOs with burn-rate alerts. Add autoscaling and a fallback model, then connect HolmesGPT with read-only permissions to investigate an injected incident and evaluate its findings in a short report.", "difficulty": "Advanced", "skills": [ "LLM inference", "AI reliability SLIs", "GPU capacity", "AI-assisted operations", "OpenTelemetry" ] } ], "certifications": [ { "name": "SRE Certified Professional (SRECP)", "provider": "DevOpsSchool", "url": "https://www.devopsschool.com/certification/sre-certified-professional-srecp.html", "level": "Professional", "note": "DevOpsSchool's hands-on SRE program and open-book scenario exam covering SLOs, error budgets, observability, incident management and the supporting toolchain." }, { "name": "SRE Foundation Certified (SRE-FC)", "provider": "DevOpsSchool", "url": "https://www.devopsschool.com/certification/sre-foundation-certification.html", "level": "Foundation", "note": "Structured introduction to SRE principles, SLIs and SLOs, incident management, chaos engineering and toil reduction; a good first credential for Stages 1 to 4." }, { "name": "Certified Kubernetes Administrator (CKA)", "provider": "CNCF / Linux Foundation", "url": "https://training.linuxfoundation.org/certification/certified-kubernetes-administrator-cka/", "level": "Professional", "note": "Performance-based proof that you can run, troubleshoot and upgrade Kubernetes clusters, the platform most SRE teams operate." }, { "name": "Prometheus Certified Associate (PCA)", "provider": "CNCF / Linux Foundation", "url": "https://training.linuxfoundation.org/certification/prometheus-certified-associate/", "level": "Associate", "note": "Validates PromQL, instrumentation, exporters, alerting and dashboarding, the core of Stage 3." }, { "name": "OpenTelemetry Certified Associate (OTCA)", "provider": "CNCF / Linux Foundation", "url": "https://training.linuxfoundation.org/certification/opentelemetry-certified-associate-otca/", "level": "Associate", "note": "Covers the OpenTelemetry API, SDK and Collector and debugging telemetry pipelines, now the standard way to instrument services." }, { "name": "Google Cloud Professional Cloud DevOps Engineer", "provider": "Google Cloud", "url": "https://cloud.google.com/learn/certification/cloud-devops-engineer", "level": "Professional", "note": "Includes an explicit domain on applying SRE practices (SLOs, error budgets, toil, incidents) alongside CI/CD, observability and cost optimisation on Google Cloud." }, { "name": "AWS Certified DevOps Engineer – Professional", "provider": "Amazon Web Services", "url": "https://aws.amazon.com/certification/certified-devops-engineer-professional/", "level": "Professional", "note": "Covers resilient and highly available architectures, monitoring, logging and incident and event response on AWS (exam DOP-C02)." } ], "tools_catalog": [ { "name": "Argo CD", "category": "GitOps", "website": "https://argo-cd.readthedocs.io/", "ds_course_url": "https://www.devopsschool.com/courses/argo/" }, { "name": "Argo Rollouts", "category": "CI/CD & Release", "website": "https://argoproj.github.io/rollouts/", "ds_course_url": "https://www.devopsschool.com/courses/argo/" }, { "name": "AWS", "category": "Cloud Platforms & Governance", "website": "https://aws.amazon.com/", "ds_course_url": "https://www.devopsschool.com/courses/aws/" }, { "name": "Bash", "category": "Foundations & Languages", "website": "https://www.gnu.org/software/bash/", "ds_course_url": "https://www.devopsschool.com/courses/bashscripting/" }, { "name": "Chaos Mesh", "category": "Reliability & Resilience", "website": "https://chaos-mesh.org/", "ds_course_url": "https://www.devopsschool.com/courses/chaos-engineering/" }, { "name": "Docker", "category": "Containers & Kubernetes", "website": "https://www.docker.com/", "ds_course_url": "https://www.devopsschool.com/courses/docker/" }, { "name": "Elastic Stack", "category": "Observability", "website": "https://www.elastic.co/elastic-stack", "ds_course_url": "https://www.devopsschool.com/courses/elastic/" }, { "name": "Git", "category": "Foundations & Languages", "website": "https://git-scm.com/", "ds_course_url": "https://www.devopsschool.com/courses/git/" }, { "name": "Go", "category": "Foundations & Languages", "website": "https://go.dev/", "ds_course_url": "https://www.devopsschool.com/courses/golang/" }, { "name": "Google Cloud", "category": "Cloud Platforms & Governance", "website": "https://cloud.google.com/", "ds_course_url": "https://www.devopsschool.com/courses/googlecloud/" }, { "name": "Grafana", "category": "Observability", "website": "https://grafana.com/", "ds_course_url": "https://www.devopsschool.com/courses/grafana/" }, { "name": "Helm", "category": "Containers & Kubernetes", "website": "https://helm.sh/", "ds_course_url": "https://www.devopsschool.com/courses/kubernetes/" }, { "name": "HolmesGPT", "category": "AIOps & Incident Management", "website": "https://holmesgpt.dev/", "ds_course_url": "" }, { "name": "Istio", "category": "Networking & Network Automation", "website": "https://istio.io/", "ds_course_url": "https://www.devopsschool.com/courses/istio/" }, { "name": "Jaeger", "category": "Observability", "website": "https://www.jaegertracing.io/", "ds_course_url": "https://www.devopsschool.com/courses/jaeger/" }, { "name": "k6", "category": "Reliability & Resilience", "website": "https://k6.io/", "ds_course_url": "" }, { "name": "K8sGPT", "category": "AIOps & Incident Management", "website": "https://k8sgpt.ai/", "ds_course_url": "" }, { "name": "Karpenter", "category": "Containers & Kubernetes", "website": "https://karpenter.sh/", "ds_course_url": "" }, { "name": "KEDA", "category": "Containers & Kubernetes", "website": "https://keda.sh/", "ds_course_url": "" }, { "name": "Kubernetes", "category": "Containers & Kubernetes", "website": "https://kubernetes.io/", "ds_course_url": "https://www.devopsschool.com/courses/kubernetes/" }, { "name": "Linux", "category": "Foundations & Languages", "website": "https://www.kernel.org/", "ds_course_url": "https://www.devopsschool.com/courses/linux/" }, { "name": "LitmusChaos", "category": "Reliability & Resilience", "website": "https://litmuschaos.io/", "ds_course_url": "https://www.devopsschool.com/courses/chaos-engineering/" }, { "name": "Loki", "category": "Observability", "website": "https://grafana.com/oss/loki/", "ds_course_url": "" }, { "name": "Microsoft Azure", "category": "Cloud Platforms & Governance", "website": "https://azure.microsoft.com/", "ds_course_url": "https://www.devopsschool.com/courses/azure/" }, { "name": "OpenTelemetry", "category": "Observability", "website": "https://opentelemetry.io/", "ds_course_url": "https://www.devopsschool.com/courses/opentelemetry/" }, { "name": "OpenTofu", "category": "Infrastructure as Code & Automation", "website": "https://opentofu.org/", "ds_course_url": "https://www.devopsschool.com/courses/opentofu/" }, { "name": "PagerDuty", "category": "AIOps & Incident Management", "website": "https://www.pagerduty.com/", "ds_course_url": "" }, { "name": "Prometheus", "category": "Observability", "website": "https://prometheus.io/", "ds_course_url": "https://www.devopsschool.com/courses/prometheus/" }, { "name": "Pyrra", "category": "Reliability & Resilience", "website": "https://github.com/pyrra-dev/pyrra", "ds_course_url": "" }, { "name": "Python", "category": "Foundations & Languages", "website": "https://www.python.org/", "ds_course_url": "https://www.devopsschool.com/courses/python/" }, { "name": "Sloth", "category": "Reliability & Resilience", "website": "https://sloth.dev/", "ds_course_url": "" }, { "name": "Tempo", "category": "Observability", "website": "https://grafana.com/oss/tempo/", "ds_course_url": "" }, { "name": "Terraform", "category": "Infrastructure as Code & Automation", "website": "https://developer.hashicorp.com/terraform", "ds_course_url": "https://www.devopsschool.com/courses/terraform/" }, { "name": "Velero", "category": "Reliability & Resilience", "website": "https://velero.io/", "ds_course_url": "https://www.devopsschool.com/courses/velero/" }, { "name": "vLLM", "category": "LLMs, Agents & GenAI", "website": "https://vllm.ai/", "ds_course_url": "" } ], "faqs": [ { "q": "What does a Site Reliability Engineer do day to day?", "a": "An SRE keeps production services reliable while helping teams ship quickly. Typical work includes defining and tracking SLOs, improving monitoring and alerting, taking part in on-call, leading incident response and postmortems, and writing automation that eliminates toil. Google's model caps operational work at about 50% so the rest of the time goes to engineering improvements." }, { "q": "What is the difference between SRE, DevOps and platform engineering?", "a": "DevOps is a culture and set of practices for shared ownership of software delivery and operations. SRE is a concrete implementation of those ideas focused on reliability, using SLOs, error budgets and engineering to manage risk. Platform engineering builds internal self-service platforms; many SREs contribute to them, and the three roles overlap heavily in practice." }, { "q": "Do SREs need to know how to code?", "a": "Yes. SRE is software engineering applied to operations, so you will write automation, tooling, exporters and sometimes fixes in product code. Python and Go are the most common languages, and you should be comfortable with testing, code review and reading unfamiliar codebases during an incident." }, { "q": "What is an error budget and how is it used?", "a": "An error budget is the amount of unreliability your SLO allows: a 99.9% availability SLO over 30 days permits about 43 minutes of downtime. While budget remains, teams can ship faster and take more risk; when it is exhausted, the agreed error budget policy shifts effort toward reliability work, for example by pausing risky launches." }, { "q": "Which certification is best for SRE?", "a": "There is no single SRE exam that everyone requires, so combine a platform certification with an observability one. CKA plus the Prometheus (PCA) or OpenTelemetry (OTCA) associate exams cover the most common toolchain, the Google Professional Cloud DevOps Engineer exam explicitly tests SRE practices, and DevOpsSchool's SRE Foundation and SRECP programs cover the discipline end to end." }, { "q": "How long does it take to become an SRE?", "a": "With a background in software development or systems administration, this path takes about six months at 10 hours a week. Many SREs move in from adjacent roles after a year or two of production experience, so pair your study with real on-call duty or homelab projects. Employers value evidence of SLOs you defined and incidents you handled more than certificates alone." }, { "q": "How is AI changing SRE in 2026?", "a": "In two ways. AI agents such as HolmesGPT and vendor SRE assistants now help triage alerts, correlate telemetry and draft postmortems, so SREs must design their guardrails and measure their accuracy. And AI features themselves need reliability engineering: GPU capacity planning, inference latency SLOs and fallbacks across models and providers." } ], "related": [ "devops", "platform-engineering", "aiops", "devsecops" ], "key_tools": [ "Prometheus", "Grafana", "OpenTelemetry", "Kubernetes", "PagerDuty", "LitmusChaos" ], "course_url": "https://www.devopsschool.com/courses/sre/", "seo_title": "SRE Learning Path 2026: Roadmap, SLOs, Tools & Certs", "seo_description": "Become a Site Reliability Engineer in 2026: SLOs, error budgets, OpenTelemetry, Prometheus, incident response, chaos engineering, AI reliability and certs.", "url": "https://www.devopsschool.com/path/sre/", "updated_at": "2026-10-26 08:58:45" }