Disaster Recovery

What Is a Disaster Recovery Plan?

A Disaster Recovery Plan (DRP) is a collection of documented procedures for how to react to and recover from an emergency or disaster scenario. Its goal: minimize downtime and prevent significant data loss.

A DRP isn’t just a document - it’s a tested, rehearsed playbook.

---

The Three Pillars of Disaster Recovery

1. Preventive Measures

Actions taken before a disaster to minimize impact:

• Regular, automated backups stored both on-site and off-site
• Redundant systems - no single point of failure for critical services
• Redundant power supplies, network links, and hardware
• Comprehensive monitoring and alerting
• Clear, up-to-date operational documentation

2. Detection Measures

Systems that alert you when something goes wrong:

Measure	What It Detects
Network monitoring	Service outages, latency spikes, packet loss
Server monitoring	High CPU, memory exhaustion, disk full
Environmental sensors	Temperature, humidity, water/flood
Smoke/fire alarms	Physical threat to data center
Log aggregation & alerting	Error patterns, security events

Tip

Monitoring that nobody watches is useless. Set up automated alerting (PagerDuty, Opsgenie, Slack alerts) so the right people are notified within minutes of an issue.

3. Corrective / Recovery Measures

Steps taken after a disaster to restore operations:

• Restore data from backups
• Rebuild and reconfigure damaged systems
• Failover to standby systems
• Follow the documented recovery procedures step-by-step

---

Designing a DRP

Step 1: Risk Assessment

Brainstorm scenarios and assess each one:

Scenario	Likelihood	Impact	Priority
Server hardware failure	High	Medium	P1
Data center power outage	Medium	High	P1
Ransomware attack	Medium	Critical	P0
Natural disaster (flood, fire)	Low	Critical	P1
Accidental data deletion	High	Low-Medium	P2

Step 2: Identify Critical Systems

Not everything has the same priority. Rank systems by criticality:

• Mission-critical: Authentication, billing, production databases - immediate recovery required
• Important: Email, internal tools - hours of downtime acceptable
• Non-critical: Internal wiki, staging environments - can wait

Step 3: Define Recovery Procedures

For each critical system, document:

1. How to restore from backup
2. How to failover to a standby system
3. Who is responsible for each step
4. Expected recovery time
5. Links to detailed runbooks

Step 4: Test Regularly

• Run disaster recovery simulations at least annually
• Involve all relevant teams - IT, security, management
• Time the recovery to verify you meet your RTO
• Document what worked, what didn’t, and what needs improvement

Caution

Critical DRP documents must be accessible during a disaster. If your documentation lives on the same server that just failed, it’s useless. Store copies in multiple locations - cloud, printed, USB, personal devices of key personnel.

---

Single Points of Failure

A single point of failure (SPOF) is any component whose failure would take down the entire service. Identify and eliminate them:

SPOF Example	Mitigation
One database server	Primary + replica with automatic failover
One internet connection	Dual ISP with BGP failover
One power feed	UPS + generator + dual power supplies
One load balancer	Active-passive pair
One person who knows the system	Documentation + cross-training

---

Post-Mortems

A post-mortem is a report written after an incident to document what happened, why it happened, what was done about it, and how to prevent it from happening again.

Caution

A post-mortem is about learning, not blame. Organizations that punish people for honest mistakes create a culture where people hide problems instead of fixing them.

Structure of a Post-Mortem

# Incident Post-Mortem: [Title]

## Summary
Brief description of the incident - what, when, how long, impact, resolution.

## Timeline (all times in UTC)
- 14:03 - Monitoring alert: API response time > 5s
- 14:07 - On-call engineer acknowledges alert
- 14:12 - Root cause identified: DB connection pool exhausted
- 14:15 - Fix applied: connection pool limit increased
- 14:20 - Services restored, monitoring confirms recovery

## Root Cause
Detailed, honest explanation of what went wrong.

## What Went Well
- Monitoring detected the issue within 3 minutes
- Failover to read replica kept partial service running

## What Went Wrong
- Connection pool limit hadn't been reviewed since initial deploy
- No alert existed for connection pool saturation specifically

## Action Items
- [ ] Add connection pool saturation alert (owner: @alice, due: March 20)
- [ ] Review all service limits as part of quarterly capacity planning
- [ ] Update runbook with new connection pool tuning steps

Post-Mortem Best Practices

1. Write it promptly - within 24-48 hours while details are fresh
2. Be honest - document the real root cause, even if it’s embarrassing
3. Highlight what went well - failovers that worked, monitoring that caught things early
4. Actionable items - each action item has an owner and a deadline
5. Share widely - other teams can learn from your incidents
6. No-blame culture - focus on systems and processes, not individuals

---

People Matter Too

Disaster recovery isn’t only about servers and data. It also covers:

• Employee safety - evacuation plans, physical security
• Communication - how do you notify staff, customers, and stakeholders?
• Temporary work arrangements - can people work remotely if the office is inaccessible?
• Cooperation with facilities/building management - fire suppression, access control