Design for the day something fails

Restoring yesterday’s database does not restore yesterday’s world. A customer may already have received an email. A crew may have completed a job. A payment may have happened outside the application.

Recovery has to reconcile the restored system with what continued to happen around it. Backups protect information; a recovery plan explains how the business returns to a trustworthy state without repeating completed work or losing the work done during the interruption.

Plan the return to normal

Consider a dispatch outage. A paper schedule may keep the crews working, which is a useful fallback. When the system returns, someone must reconcile changes, completions and cancellations before normal automation resumes.

If the restored system treats those jobs as untouched, it may send duplicate assignments or generate the wrong invoices. The recovery boundary is therefore wider than “the application loads.” It includes the people and external systems that acted while it was unavailable.

Different times of day can change the priority. Losing a schedule before dispatch may be more disruptive than losing it after crews have left with copies. Define the interruption in terms of the work it prevents, then decide what must be restored, reconciled and checked before the normal process takes over.

Prove that you can restore

New Zealand’s National Cyber Security Centre recommends testing backups against realistic recovery scenarios. A successful backup notification alone does not establish that a usable system can be restored.

For a custom application, restoration may require configuration, attachments and working access credentials as well as database records. Test the whole path in a safe environment. Record what was restored, what was missing and how long the exercise took.

Protect backups separately enough that one mistake or compromised account does not erase the working system and its only recovery copy. The appropriate arrangement depends on the platform and the importance of the data.

Give people a fallback

A fallback can be deliberately modest: a recent read-only schedule, a queue for incoming requests or a paper process for essential tasks. Explain what it can and cannot support.

Also decide how work completed during the interruption returns to the system. Otherwise, restoring the software can create duplicate orders, contradictory job statuses or invoices for work already handled manually.

Keep sensitive information out of casual backup spreadsheets and personal inboxes. The fallback needs an owner and an agreed storage location just as the normal process does.

Rehearse the first hour

Use a short scenario with the people who would respond. An integration has stopped, several customers are waiting and you do not yet know whether records were lost. Who checks the issue? Who communicates with staff? Who can stop further changes while someone investigates?

Capture the answers in a brief runbook that remains available if the main system is inaccessible. Include the conditions for returning to normal work and the checks that confirm recovery is complete.

After an interruption or rehearsal, fix the specific weakness you found. Observability can help you notice failure sooner; maintenance can address recurring causes. Neither removes the need for a practiced recovery route.