Site Reliability Engineering

Every SRE investigation eventually lands on a host, so you must read what the kernel is telling you. Learn processes, memory, file descriptors and systemd, and practise the USE method (utilisation, saturation, errors) to find bottlenecks quickly.

• Linux
• Bash

Many outages are network problems in disguise: DNS, timeouts, connection limits or a misbehaving load balancer. Understand TCP behaviour, DNS caching, TLS handshakes and L4/L7 load balancing well enough to reason about latency and failure modes.

SREs spend at least half their time engineering rather than firefighting, so solid programming is non-negotiable. Use Python for scripts, API automation and data analysis, and Go for fast tooling, exporters and Kubernetes controllers, with tests and code review like any production software.

• Python
• Go
• Git

SRE treats operations as a software problem, accepts that 100% reliability is the wrong target and manages risk explicitly. Learn the core tenets and how SRE relates to DevOps and platform engineering so you can explain and adapt the role in any organisation.

Kubernetes is the default runtime for production services, and most of its reliability features are configuration you must get right. Learn probes, resource requests and limits, PodDisruptionBudgets, autoscaling and safe rolling updates, and practise debugging failing pods and nodes.

• Docker
• Kubernetes
• Helm

Reproducible infrastructure is a reliability feature: if you can rebuild it from Git, you can review it, roll it back and recover it. Manage cloud resources with Terraform or OpenTofu and deliver cluster state with Argo CD so every production change is versioned and auditable.

• Terraform
• OpenTofu
• Argo CD

Cloud providers give you failure domains, not reliability; you still have to design for them. Learn how regions and availability zones fail, when multi-AZ or multi-region is worth the cost, static stability and cell-based isolation, and how dependency SLAs compose into your own.

• AWS
• Google Cloud
• Microsoft Azure

Prometheus and PromQL remain the common language of cloud-native metrics. Learn the data model, the four golden signals and the RED and USE methods, recording rules and how to keep cardinality and cost under control; Prometheus 3 also brings stable native histograms for cheaper, more precise latency SLIs.

• Prometheus
• Grafana

OpenTelemetry is the vendor-neutral standard for traces, metrics and logs, with profiles now in alpha. Instrument services with SDKs or zero-code agents, route telemetry through the Collector, propagate context across services and choose sampling strategies that keep the traces you actually need.

• OpenTelemetry
• Jaeger
• Tempo

Logs are usually the most expensive signal, so be deliberate: structure them, index only what you query and link them to traces. Learn label-based log storage, log-to-trace correlation with trace IDs and exemplars, and retention policies that balance cost with forensic needs.

• Loki
• Elastic Stack

SLOs are the heart of SRE: they turn the question of whether a service is reliable enough into a number that product and engineering agree on. Choose user-centric SLIs, set realistic targets and windows, write an error budget policy, and manage SLOs as code so they are versioned and generate alerts automatically.

• Sloth
• Pyrra

Page humans only when users are hurting or soon will be. Replace static threshold alerts with multi-window, multi-burn-rate SLO alerts, route and silence them sensibly, and design dashboards that answer on-call questions in seconds.

• Prometheus
• Grafana

On-call is where SRE promises meet human limits. Design rotations, escalation policies and runbooks that keep load sustainable, measure pages per shift, and make handoffs routine so critical knowledge does not live in one person's head.

• PagerDuty

Major incidents are coordination problems as much as technical ones. Learn severity levels, the Incident Commander, Operations and Communications roles, mitigation-first thinking and stakeholder updates, and rehearse them in drills before you need them.

• PagerDuty

Incidents are unplanned investments, and postmortems are how you collect the return. Write blameless postmortems with clear timelines and contributing factors, track action items to completion, and share learnings so the whole organisation improves.

Good troubleshooting is a repeatable method, not heroics. Form and test hypotheses against telemetry, ask what changed, bisect deployments and configuration, and keep working notes so others can follow and verify your reasoning.

Most outages are triggered by change, and global configuration pushes are as dangerous as code, as major 2025 cloud and CDN outages showed. Use canaries, feature flags and staged rollouts with automated analysis so bad changes roll back before most users notice.

• Argo Rollouts

Distributed systems fail partially, and naive retries turn small failures into cascading outages. Learn timeouts, retries with backoff and jitter, circuit breaking, load shedding and graceful degradation, and verify that each dependency failure degrades the service rather than breaking it.

• Istio

Chaos engineering verifies resilience with controlled experiments instead of hope. Define steady state, form a hypothesis, inject faults with a limited blast radius and run regular GameDays, starting in staging and graduating to production with guardrails.

• LitmusChaos
• Chaos Mesh

Capacity planning keeps you ahead of demand without paying for idle hardware. Forecast load from business drivers, find saturation points with load tests, keep N+2 headroom and use pod and node autoscaling, treating cost as a reliability trade-off rather than an afterthought.

• k6
• KEDA
• Karpenter

A backup that has never been restored is a hope, not a plan. Define RPO and RTO per service, automate backups of data and cluster state, and run timed restore and regional failover drills so recovery is a practised routine.

• Velero

Toil is manual, repetitive, automatable work that grows with load, and it crowds out engineering. Measure it, prioritise the worst offenders and automate them with tested tools, Kubernetes operators and guarded auto-remediation that knows when to stop and page a human.

• Python
• Go
• Kubernetes

AI agents now investigate alerts, summarise incidents and propose fixes, and SREs are responsible for making them trustworthy. Learn how agentic troubleshooting works, connect agents to telemetry safely (read-only by default, scoped credentials, human approval for actions) and evaluate their suggestions against real past incidents.

• HolmesGPT
• K8sGPT

AI features bring new failure modes: GPU scarcity, long-tail latency, provider rate limits and silent quality regressions. Define SLIs such as time to first token and inter-token latency, monitor inference servers and GPUs, plan GPU capacity, and design fallbacks across models and providers.

• vLLM
• OpenTelemetry
• Kubernetes

Senior SREs scale reliability through process and influence, not just code. Run production readiness reviews, choose the right SRE engagement model for each team, negotiate error budget policies with product owners, and report reliability with SLO and DORA metrics.