What is MTTR?
How fast you go from "it's down" to "it's back". The only metric that matters at 3 AM.
MTTR = Mean Time To Recovery
The clock starts when something breaks. It stops when it works again. Everything in between is your MTTR. The goal: make that gap as small as humanly possible.
MTTR = Total downtime ÷ Number of incidentsThe four stages of every incident
Detection
Someone (or something) notices it's broken
Diagnosis
Figure out what went wrong and where
Fix
Actually fix the damn thing
Verification
Confirm it's really back, not just pretending
Why should you care?
- •Every minute of downtime = lost money and angry users refreshing your status page
- •A 99.9% availability target still allows about 8 hours 46 minutes of downtime over a 365-day year; the exact budget depends on the contract and measurement window.
- •A single industry-average MTTR is not useful across very different services and incidents. Track your own detection, diagnosis, action, and verification time so you can see where recovery actually stalls.
Why most people's MTTR is terrible
The laptop dance
3 AM alert → find laptop → boot up → VPN → SSH → wait, which server was it again?
Log spelunking
Which service? Which log file? What am I even looking for?
Fear of making it worse
One typo in a manual command and now you have two incidents
How mttrly cuts each stage
| Stage | Without mttrly | With mttrly |
|---|---|---|
| Detection | Wait for user complaint | A configured check sends an alert after it detects the condition |
| Diagnosis | SSH → grep → scroll → guess | Run prepared checks and review current evidence |
| Fix | Type commands, pray for no typos | Review a scoped action and approve it when required |
| Verification | Open dashboards, refresh, squint | Run a current status check and record the result |
Target result: less avoidable time between evidence and verified recovery. No fixed MTTR is promised.
What about Grafana and Datadog?
Keep Grafana, Datadog, and the rest of your observability stack for telemetry, dashboards, history, and alert routing. mttrly is the separate server action layer after the alert: inspect current server evidence, request a supported next step, apply runtime policy, and record verification.
Industry Standards & Research
Site Reliability Engineering: How Google Runs Production Systems ↗
Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy · Book (Free Online)
The foundational SRE text defining MTTR, monitoring, alerting, and incident response practices used by Google and adopted industry-wide.
DORA State of DevOps Reports ↗
DevOps Research and Assessment (Google Cloud) · Annual Research
Annual research on software-delivery performance and recovery-related measures. Benchmarks and terminology change by report year, so this site does not attach one timeless MTTR number to every team.
Accelerate: The Science of Lean Software and DevOps
Nicole Forsgren, Jez Humble, Gene Kim · Book
Research-backed evidence that MTTR is one of the four key metrics predicting software delivery performance and organizational outcomes.
PagerDuty Incident Response Documentation ↗
PagerDuty · Open-source Guide
Free incident response handbook covering on-call practices, severity classification, communication during incidents, and post-mortem processes.
Try it on your next incident
Free tier. Email-first signup. No credit card.