Backup, high availability and disaster recovery are three different things

Start with three incidents, because each one is answered by a different control, and a design that handles only one of them is not a resilience strategy.

1. A disk or a host fails at 2 a.m. The service should carry on, or come back within minutes, without anyone restoring anything.
2. Someone deletes a table, overwrites a share, or ransomware encrypts everything it can reach. The system is healthy and working perfectly, and the data is wrong. Every copy that is kept in step with the live data is wrong too.
3. The site is lost: fire, flood, a failed power feed that outlasts the generator, a supplier going out of business. Everything in one place is gone, including the things you were counting on to restore it.

The answers are, respectively, high availability, backup and disaster recovery. They overlap in the equipment they use, which is why they get confused, but they are different promises with different failure modes.

Three things with three different jobs

	High availability	Backup	Disaster recovery
Question it answers	Can the service survive a component failing?	Can I get back data as it was at an earlier moment?	Can I run the business somewhere else if this place is gone?
Typical mechanism	Redundant components, clustering, failover	Point-in-time copies kept separately from the live system	A second site or platform, with data and a tested plan to run on it
Protects against	Hardware, host and some software failure	Deletion, corruption, ransomware, bad changes	Loss of a site or platform
Does not protect against	Deletion or corruption: it replicates the mistake	A long wait: restoring takes time	Mistakes that were replicated, unless point-in-time copies also exist
Measured by	Failover time, availability	How far back, and how long to restore	Time to resume, and how much data is lost

Why replication and RAID are not backup

RAID and mirrored storage protect against a disk failing. They write every change, including a mistaken one, to every copy at once. Replication to a second site is the same: it faithfully reproduces a deleted directory or an encrypted volume within seconds. Snapshots held on the same storage array as the live data are better, because they are point-in-time, but they share the array’s failure and the array’s administrator credentials, so an attacker or a firmware bug that reaches the array reaches the snapshots. A backup is a copy that is separate in time, in place and in access.

The two numbers that make this concrete

RPO, the recovery point objective, is the amount of data you can afford to lose, expressed as time. RTO, the recovery time objective, is how long you can afford to be without the service. They are business decisions that engineering then has to meet. Asking each application owner for both numbers, in writing, is the cheapest useful thing you can do this quarter, because the answers usually disagree with what the infrastructure delivers.

A nightly backup gives an RPO of up to a day, plus however long the backup took, because an incident just before the next run loses everything since the last one. Asynchronous replication gives an RPO equal to the replication lag at the moment of failure, which is small but not zero and grows under load.

RTO is arithmetic as much as design. Restore time is the data size divided by the slowest stage in the path: the backup repository’s read speed, the network, or the target’s write speed.

The 3-2-1 rule and what it is for

A widely used rule of thumb is to keep at least three copies of the data, on at least two different kinds of storage, with at least one copy off site. It is a heuristic for avoiding common-mode failure: no single fault, accident or site loss should take every copy. Modern guidance adds a fourth consideration that the rule does not cover: at least one copy must be out of reach of whoever can administer the production environment. That means an offline copy, or storage that enforces retention so that even a privileged account cannot delete the data before a set date.

The practical test is blunt. If an attacker obtains domain administrator rights in your production environment, can they also delete your backups? If yes, the backups will be gone at the moment you need them. Separate credentials for the backup system, not joined to the same directory, and immutability on at least one copy, are the controls that answer this.

Consistent copies, not just completed jobs

A job that finishes and reports success has proven that data was copied. It has not proven that the copy is usable. Databases are the usual trap: copying live database files produces something that may not start. Take application-consistent copies, using the database’s own dump or snapshot mechanism, or volume snapshots coordinated with the application.

# MySQL/MariaDB, InnoDB tables: a consistent logical dump without locking the whole database
mysqldump --single-transaction --routines --triggers appdb > appdb.sql

# PostgreSQL: a compressed, restorable dump
pg_dump --format=custom --file=appdb.dump appdb

The --single-transaction option gives a consistent snapshot only for transactional storage engines such as InnoDB, so check which engine your tables use. And a dump is not tested until it has been restored somewhere.

Restore testing is the backup

Until you have restored it, the backup is a belief. Test at three levels and keep a record of each.

• File level. Restore a sample of files and compare them with the source. Checksums make this objective:

# On the source: record a manifest of checksums
cd /srv/data && find . -type f -print0 | sort -z | xargs -0 sha256sum > /root/data.sha256

# After restoring the same tree to /restore/data: report only mismatches
cd /restore/data && sha256sum --check --quiet /root/data.sha256

• System or database level. Restore a whole server, or a database into a scratch instance, and confirm it starts. For PostgreSQL, restore into a throwaway database with createdb scratch && pg_restore --dbname=scratch --no-owner appdb.dump, and never aim a test restore at the live database.
• Application level. Have the application owner confirm the restored service works and the data is plausible. This is the only level that tests what the business cares about.

Record the elapsed time at every level. Those measurements are your real RTO, and they are usually larger than the one in the design document.

Disaster recovery: the plan matters more than the hardware

Disaster recovery strategies sit on a spectrum, and the common names are widely understood even though vendors define them slightly differently. Backup and restore keeps copies off site and rebuilds from them: cheapest to run, slowest to recover. A pilot light keeps the minimum core running elsewhere, typically data replication and the smallest set of services, and scales up on demand. Warm standby keeps a reduced but functioning copy of the whole service running. Active-active runs the service in more than one place at once. Cost and complexity rise as recovery time falls, and a business only needs to pay for the point on the spectrum that matches the RTO and RPO it truly requires. For the related question of how much redundancy is worth buying before a disaster-recovery plan is needed at all, IT KORR has a short article on high availability versus disaster recovery.

Whichever you pick, the plan has to answer questions the hardware cannot:

• What comes first? Identity, name resolution, networking, secrets and time come before applications. A recovery that starts by restoring the application server and then discovers the directory it authenticates against is also gone has lost hours.
• Where is the plan? A runbook that lives only in the system that has failed is not a plan. Keep a printed or offline copy, along with the contact details and the credentials needed to start, held according to your own secrets policy.
• Who decides? Declaring a disaster is a decision with cost. Name who makes it, and the criteria.
• Who has practised? A DR exercise run once, by the people who built it, tells you little. Run it with the people who would be on call.

How this fails

• The backup server was in the domain it was protecting. One compromised administrator account removed the data and the backups together.
• Snapshots mistaken for backups. They were on the same array, so a firmware fault or a storage-level compromise took both.
• Backups of the wrong thing. The agent covered the operating system but not the data volume added later. Reconcile backup coverage against the inventory regularly.
• Untested restore. The first restore, during the incident, found a corrupt chain or a missing encryption key. Keep the keys somewhere the restore can reach them, separately from the data they protect.
• RTO assumed, never calculated. The restore arithmetic above was never done, so a twelve-hour restore met a four-hour objective.
• Replication treated as DR. The second site faithfully copied the incident.

What to do next

Write down RTO and RPO for each application and compare them with the arithmetic above. Then read server migration, since the same transfer arithmetic governs how fast you can move or restore data, and use the migration readiness planner to check the rollback step of any move. If a supplier will run part of this for you, managed infrastructure lists what to ask about backup, retention and exit before you sign.