SRE Incident Leadership & Stability — Google-style Incident Command, Interrogation, and Influence Without Authority

---

What this workshop is based on (industry + Google SRE)

This agenda blends:

• Google SRE incident management mindset (3Cs: coordinate, communicate, control; clear roles like IC/Comms/Ops; single live incident state document)
• Google SRE troubleshooting + alerting philosophy (systematic troubleshooting; alerts should be actionable and symptom-oriented)
• Google SRE stability levers (SLOs + error budgets as the “authority-less” mechanism to shift priorities)
• Industry incident response practice (Incident Commander training patterns; blameless postmortems; modern incident workflow tooling)
• Modern learning-from-incidents / resilience engineering (learning-focused incident analysis; coordination costs; systemic fixes)

---

3-Day Workshop Agenda (with labs + exercises)

Daily structure (recommended)

• 60% practice (simulations, role-plays, artifacts)
• 40% concepts/tools (frameworks, playbooks, decision patterns)

---

Day 1 — Incident Command + Structured Interrogation in Unknown Systems

Goal: Enable PSREs to walk into an unfamiliar incident and still drive fast, clean outcomes.

1) The “Google-style” major incident operating system (IMAG/ICS)

• Why incident response fails (freelancing, unclear roles, weak comms, no single-threaded control)
• 3Cs: coordinate / communicate / control
• Core roles and how they work together: IC, Comms Lead, Ops Lead, Scribe
  • Why roles ignore reporting lines and focus on execution clarity
• The “single writer” and “single source of truth”: live incident state document
• Severity model and operating cadence: declare, stabilize, update, resolve, review

Exercise: “Activate in 5 minutes” drill
Set roles, create war-room channel, start incident doc, set update cadence, define first 3 objectives.

---

2) Structured incident interrogation (questioning system)

Teach a repeatable questioning framework PSREs can run without deep system knowledge:

• Impact & scope (who/what is affected, blast radius, user journeys)
• Time & change (when started; what changed; last deploy/config/infra change)
• Signals (best symptom metric; error/latency patterns; what’s normal baseline)
• Dependencies (upstream/downstream; what relies on what; isolate candidates)
• Hypotheses + tests (top 3 likely causes; fastest tests to confirm/deny)
• Mitigation decision (stop bleeding vs diagnose; safe rollback vs forward fix)

Lab: Build a “First 15 Minutes” interrogation sheet + decision checkpoints
(Participants leave with a one-page checklist and question bank.)

---

3) Running the room under ambiguity (war-room mechanics)

• Keeping tempo: time-boxing, checkpoints, and workstream split
• Preventing freelancing: only Ops executes changes; everyone else feeds evidence
• Handling SME conflict: how IC arbitrates with evidence + risk framing
• Communication discipline: what updates must include (impact, actions, ETA, risks)

Simulation #1 (tabletop): “Unknown service outage”
Limited context + conflicting SME opinions + noisy signals
Outputs: live incident doc, timeline, hypothesis board, comms updates, decision log.

---

Day 2 — Navigating Architectures & Dependencies + Observability-Driven Investigation

Goal: Make PSREs effective at “finding the system” fast and driving investigations cleanly.

1) Rapid architecture discovery (when docs are missing)

• “Whiteboard the service” method:
  • request path, data stores, queues, caches, third parties, auth, network edges
• Dependency interrogation:
  • what changed, what fans out, what is hard dependency, what degrades gracefully
• “Isolation moves” playbook:
  • shed load, disable feature, bypass dependency, traffic shift, circuit breaker, rollback

Exercise: Build a 10-minute architecture map from SMEs using structured prompts.

---

2) Observability tactics that work in war rooms

• Symptom-first investigation:
  • identify best “user pain” signal, then trace to components
• Practical workflow:
  • RED/USE signals, golden paths, service graphs, trace-to-logs
• Evidence preservation:
  • what to snapshot (dashboards, logs, traces, configs, deployments) before changes
• Choosing where to look first:
  • “top suspect list” rules to avoid random debugging

Lab: Guided investigation flow on demo system
Participants practice: symptom → isolate → confirm/deny hypotheses → mitigation choice.

---

3) Comms excellence + handoffs

• IC/Comms templates (internal + external)
• Status update cadence and message quality
• Clean handoff protocol (explicit transfer of command; state doc update)

---

Simulation #2 (hands-on, tool-based)

Run a live incident game on a demo system:

• Inject failure (latency, error spike, dependency degradation, partial outage)
• PSREs rotate roles (IC / Ops / Comms / Scribe)
• Scoring on: interrogation quality, control, speed, protocol adherence, comms clarity

End-of-day artifact pack:

• Incident channel checklist
• Incident state doc template
• Interrogation question bank
• Workstream board template
• Mitigation decision log template
• Handoff checklist

---

Day 3 — Driving Stability Without Authority + Resilient Mindset (Operating in a Matrix Org)

Goal: Turn PSREs into stability leaders who can influence priorities and create accountability.

1) The “authority-less” levers: SLOs + Error Budgets (Google SRE style)

• Defining reliability outcomes: SLIs vs SLOs (what matters to users)
• Error budgets as the alignment mechanism:
  • when budget burns, reliability work gets priority
• What an error budget policy typically includes:
  • release rules, mitigation requirements, reliability triggers, escalation path
• Using these levers diplomatically:
  • “permission to pause” with shared rules, not blame

Workshop: Draft an error budget policy skeleton suitable for your org.

---

2) Influence without authority (practical playbook)

• Stakeholder mapping:
  • owners, decision makers, blockers, allies, exec sponsors, customer-facing teams
• Persuasion with evidence:
  • incident themes, toil metrics, reliability risks, customer impact, near-misses
• Creating cross-team accountability:
  • DRIs, RACI, due dates, measurable outcomes
• Operating cadence:
  • weekly stability review, top risks register, actions tracking, escalation rituals

Role-play: “Competing priorities negotiation”
Participants practice getting buy-in when the team says: “feature work first.”

---

3) Blameless learning → stability roadmap

• Postmortem quality:
  • timeline, contributing factors, detection gaps, decision analysis, action quality
• Turning postmortems into stability work:
  • reduce recurrence, improve detection, reduce MTTR, reduce toil
• Tracking follow-ups:
  • owners, deadlines, verification, and closure criteria

Lab: Write a “blameless postmortem” from the Day 2 simulation and derive the top 5 stability actions.

---

4) Resilience mindset under pressure (personal + team)

• Cognitive traps in incidents:
  • tunnel vision, confirmation bias, authority bias, panic-driven changes
• “Calm operator” habits:
  • checkpoints, asking better questions, controlling pace, safe mitigation choices
• Sustaining effectiveness:
  • fatigue management, handoffs, psychological safety, aftercare

---

Capstone (must-do): 90-day Stability Influence Plan

Each team produces a 90-day Stability Influence Plan for a real service:

• top reliability risks + supporting evidence
• proposed SLOs + measurement plan
• error budget policy proposal
• roadmap (quick wins + structural changes)
• stakeholders + comms plan + cadence
• accountability model (DRIs, due dates, review checkpoints)

Outputs: a plan you can directly take into leadership review.

---

Prerequisites (participants)

Must-have

• Comfortable with Linux CLI basics and reading logs
• Basic understanding of microservices + HTTP behavior (latency, errors, dependencies)

Good-to-have

• Familiarity with Kubernetes concepts (pods/services) or your runtime equivalent
• Basic knowledge of metrics/logs/traces (even beginner level is fine)

Pre-reading (short, high impact)

• Incident roles and the 3Cs (coordinate/communicate/control)
• Managing incidents with a live incident state document
• Systematic troubleshooting methodology
• Error budgets and how they shift reliability vs feature priorities

---

Lab setup and tools (recommended)

Lab environment (choose one approach)

Option A (easiest): Docker-based microservices demo + built-in observability

• Run a demo microservices app locally with metrics/logs/traces enabled
• Use simple failure injection:
  • add latency, drop requests, stop a dependency container, overload a service

Best when: participants have mixed environments and you want low friction.

Option B (most realistic): Kubernetes-based microservices demo + chaos injection

• Run a demo microservices app on a local Kubernetes cluster (kind/minikube) or a shared training cluster
• Use fault injection:
  • pod kill, network latency, packet loss, CPU/memory pressure, dependency failures

Best when: your real production environment is Kubernetes and you want realism.

---

Tools involved

War-room collaboration

• Slack or Teams (incident channel + pinned checklist)
• Video call (Zoom/Meet/Teams)
• Shared incident state doc (Google Docs / Confluence / internal wiki)
• Optional: incident workflow tool (any modern “incident timeline + roles + comms” platform)

Observability (hands-on)

• Metrics: Prometheus + Grafana (or vendor equivalent)
• Logs: Elastic/Splunk/Loki (or equivalent)
• Traces: Jaeger/Tempo (or equivalent)
• Optional instrumentation pipeline: OpenTelemetry-style collection

Reliability management artifacts

• SLO dashboard (even if basic)
• Error budget burn reporting
• Postmortem template + action tracker
• Stability risk register (top risks, owners, target dates)

Failure injection (for gamedays)

• Basic: container stop/restart, load generation, latency injection
• Kubernetes: chaos tooling (pod kill, network faults, resource pressure)

---

What participants will leave with (deliverables)

• Incident Command checklist (roles, cadence, comms)
• “First 15 minutes” interrogation sheet + question bank
• Incident state doc template + decision log + handoff checklist
• Investigation workflow guide (symptom → isolate → confirm → mitigate)
• Postmortem template + action quality rubric
• Draft error budget policy
• 90-day Stability Influence Plan template + a completed plan for one service
• Simulation scorecard rubric (so you can repeat drills internally)

---