A backup you've never restored is a guess

Ask a team whether they have backups and nearly everyone says yes. Ask when they last restored one and the room goes quiet.

That silence is the problem. A backup is only a promise until you’ve used it to get something back.

How backups fail quietly

Backups rarely fail with an alarm. They fail in ways nobody notices until the day they’re needed:

• The job has been running for months, but it backs up an empty folder after a path changed.
• The files are there, but they’re encrypted with a key nobody can find.
• The database dump is complete, but restoring it takes nine hours, not the one hour everyone assumed.
• The backup lives on the same server it’s meant to protect.

None of these show up on a dashboard. Every one of them shows up during a restore.

The monthly drill

On the first Tuesday of every month, one engineer restores every client project from its latest backup onto a fresh, throwaway server. Not a sample. Every project, every time.

The drill has four steps:

1. Fetch. Pull last night’s backup from offsite storage, using only the credentials in the runbook.
2. Restore. Rebuild the database and files on the new server.
3. Check. Log in, open a recent record, and compare row counts with production.
4. Time it. Write down how long the whole thing took, from start to a working app.

Then the throwaway server is deleted, and the result goes into the project’s Logbook entry for that month.

What we write down

Each drill leaves a short record. It looks roughly like this:

project:    fernwood-ordering
backup:     2026-08-03 02:00 UTC
restored:   ok
row check:  orders 48,212 / 48,212
time:       14 min
notes:      none

The “time” line matters as much as the “restored” line. If a restore takes three hours, the client needs to know that before an outage, not during one.

Make it boring

The goal isn’t heroics. It’s a restore so routine that the engineer doing it is mildly bored.

That only happens when the steps are written down and nobody has to remember anything. Our runbooks assume the person restoring has never seen the project before, because one day that might be true.

If a restore needs the one person who set it up, you don’t have a backup. You have a dependency.

When a drill fails

About twice a year, a drill turns something up. A changed path, an expired storage key, a step that only worked because of a cached file.

That’s the drill doing its job. We fix the cause, update the runbook, and run it again the same day. A failed drill on a Tuesday morning is a minor chore. The same failure during a real outage is a very bad week.

Start with one

If you’ve never restored a backup, pick your most important app and try it this week on a spare machine. Time it, write down what went wrong, and fix one thing.

Then put the next one in your calendar.