Incident Response
Handle production incidents systematically.
Severity Levels
| Level | Definition | Response Time | Examples |
|---|---|---|---|
| SEV1 | Service down, all users affected | Immediate | Database crash, DNS failure, auth broken |
| SEV2 | Major feature broken, many users affected | < 30 min | Payments failing, search not working |
| SEV3 | Minor feature broken, workaround exists | < 4 hours | Export button broken, slow dashboard |
| SEV4 | Cosmetic or low-impact issue | Next business day | Typo in UI, minor styling bug |
Incident Workflow
1. Detect & Triage (first 5 minutes)
- Acknowledge the incident — "I'm looking into this"
- Determine severity level
- Check monitoring dashboards (error rates, latency, status page)
- Check recent deployments:
git log --oneline -10— was anything deployed recently?
2. Mitigate (next 15-30 minutes)
The goal is to stop the bleeding, not find the root cause.
Quick mitigations:
- Rollback:
git revert <commit> && deploy— fastest option if a deploy caused it - Feature flag: Disable the broken feature
- Scale up: Add more instances if it's a capacity issue
- Failover: Switch to backup/secondary if primary is down
- Block traffic: Rate-limit or block specific abusive traffic
3. Communicate
Internal:
- Open an incident channel (
#incident-2026-04-10) - Post status updates every 15-30 minutes
- Assign roles: Incident Commander, Communicator, Engineers
External:
- Update status page
- Send email/notification to affected users if the outage is extended
- Be honest: "We're experiencing issues with X. We've identified the cause and are working on a fix."
4. Resolve
- Deploy the fix
- Verify the fix works in production (check metrics, not just absence of errors)
- Close the incident channel with a summary
5. Postmortem (within 48 hours)
Write a blameless postmortem:
# Incident: Payments failing for Stripe webhook
**Date:** 2026-04-10
**Duration:** 45 minutes (14:30 — 15:15 UTC)
**Severity:** SEV2
**Impact:** ~200 users unable to complete purchases
## Timeline
- 14:30 — Alert fires: payment success rate drops to 20%
- 14:35 — On-call engineer acknowledges, begins investigation
- 14:40 — Identified: Stripe webhook endpoint returning 500
- 14:45 — Root cause: migration added NOT NULL column without default
- 14:50 — Fix deployed: added default value to migration
- 15:00 — Payment success rate recovering
- 15:15 — Metrics back to normal, incident closed
## Root Cause
Database migration #47 added a `currency` column with NOT NULL
but no DEFAULT value. Existing rows were fine (backfilled), but
new webhook events failed because the insert didn't include `currency`.
## What Went Well
- Alert fired within 5 minutes of the issue starting
- Rollback was considered but the fix was faster
## What Went Wrong
- Migration wasn't tested with live webhook payloads
- No staging test for the webhook flow
## Action Items
- [ ] Add webhook integration test to CI (@alice, due 2026-04-17)
- [ ] Require DEFAULT for all new NOT NULL columns in migration review (@bob)
- [ ] Add runbook for payment failures (@charlie, due 2026-04-14)Tips
- Rollback first, investigate later — speed matters more than elegance
- The most recent deploy is the most likely cause
- Don't assign blame in postmortems — focus on process improvements
- Maintain a runbook for common failure modes
- Practice incident response with game days before real incidents happen