Cloud & Infra
A Backup You Have Never Restored Is a Wish
The file is not the capability. RPO, RTO, and the quarterly drill that turns a promise into evidence.
A backup is a file. A restore is a capability. Every incident review I have read where backups existed and the recovery still failed has the same shape: the file was there, nobody had opened it, and the credentials, the version or the schema had quietly moved on.
Two numbers to write down first
- RPO — recovery point objective. How much data can you lose? An hour of orders is a different business decision from a day of them.
- RTO — recovery time objective. How long can you be down? The answer drives whether a restore is a script or a runbook with four people on a call.
Without those two numbers, "we have backups" is a sentence with no testable meaning. With them, the backup schedule and the drill calendar fall out of the requirements.
The 3-2-1 rule, translated
Three copies, two media, one off-site. The modern translation: the live database, a snapshot in the same provider, and a copy in a different account or provider that the production credentials cannot delete. Ransomware and a fat-fingered `DROP` are the same threat model: the bad actor may already have your keys.
What a drill looks like
- Restore last night's dump into a fresh throwaway database.
- Run the application against it with a read-only flag.
- Check row counts, the latest timestamp, and one join that would fail on a partial restore.
- Record how long it took. That number is your real RTO, not the one in the document.
- Delete the throwaway and update the runbook with what surprised you.
# A dump is only useful if it restores cleanly.
pg_dump --format=custom --no-owner "$DATABASE_URL" > backup.dump
createdb restore_drill
pg_restore --dbname=restore_drill --clean --if-exists backup.dump
psql restore_drill -c "select count(*), max(created_at) from orders;"
dropdb restore_drillDo that on a calendar, in the same way you would do a fire drill. A quarterly restore is enough for a small product and is infinitely more than most teams do.
The backup you have never restored is a hope with a file extension.
The pieces people forget
- Encryption keys — an encrypted backup whose key lived only on the failed host is not a backup.
- The application version — a dump from schema v41 restored into v39 code may not boot. Keep the migration path, not only the data.
- Object storage — uploads, avatars and generated files are data too, and are often excluded from the database dump.
- Secrets and config — a restore that cannot connect to anything is a rehearsal, not a recovery.