Data Protection and Continuity

How to Run a Disaster Recovery Test: A Checklist

Nanda Teknoloji Engineering Team · · 8 min read

In brief

A disaster recovery test is a controlled exercise that proves whether the plan actually meets the RTO and RPO targets. Run tabletop, partial, and full failover tests in stages; measure times in each test and turn findings into an improvement list.

A disaster recovery (DR) test is an exercise that proves, in a controlled way, whether your written plan works within the targeted time and data loss limit in a real outage. It should be run at least once a year and after every significant architectural change. The test answers three questions: do systems come back within the targeted time (RTO), does data loss stay within the accepted limit (RPO), and can the team carry out the plan without prior knowledge of it? A DR plan that has not been tested is merely a well-intentioned document. This article presents the test types, the steps from preparation to evaluation, and a checklist you can copy and adapt. To see how targets are set, you can first look at the RTO and RPO article.

Why is an untested plan useless?

In real outages, problems often come from unexpected places: an expired certificate, a missing license in the DR environment, an outdated DNS record, a service account that no longer works, or a manual step that only one person knows. These problems appear even when the plan is perfect on paper. A test lets you find them on a calm day instead of at the moment of crisis. NIST’s SP 800-34 Contingency Planning Guide also treats managing the plan together with testing, training, and exercises as a condition of maturity.

Test types and when to use them

Test type What is done? Impact on production What it teaches Recommended frequency
Tabletop The team walks through the scenario step by step in discussion None Gaps in roles, communication, and decisions A few times a year
Restore from backup The backup of a selected system is brought up in an isolated environment None Validity of the backup, time Periodic, automated
Partial / isolated failover The application is brought up in the DR environment, user traffic is not redirected Low Dependencies, configuration alignment At least once a year
Full failover Traffic is actually switched to the DR environment A planned outage may be needed Real RTO, failback Once a year for critical systems
Unannounced drill The scenario starts without prior notice to the team Controlled Operational readiness In mature organizations

Proceed in stages: starting with a tabletop and then moving to restore and partial failover builds the team’s confidence and matures the documents; only after that should you move to full failover.

Preparation before the test

  1. Clarify the objective. Which systems, which scenario, which success criteria (RTO, RPO, functional verification)?
  2. Write the scope and boundaries. Will production be touched? Which data will be used? Is masked data needed?
  3. Get approval. Business owners, security, and (if necessary) the provider must be informed and give approval.
  4. Name the roles. Test manager, technical implementers, observer/record keeper, communications lead.
  5. Prepare a rollback plan. How will the test be stopped if production is affected?
  6. Define the measurement method. Who will record the start, the end time of each stage, and the total time, and how?

Test checklist

The list below is a starting point that can be adapted to your organization.

Data and backup

  • Has the latest valid backup or replica point been identified, and has the actual RPO been calculated?
  • Did the backup integrity check succeed?
  • Has restore from the immutable backup copy also been tried?
  • Is access to encryption keys and backup management accounts possible in the DR scenario?

Infrastructure and network

  • Did the authentication service, DNS, and time synchronization come up first?
  • Are the firewall, VPN, and load balancing rules ready in the DR environment?
  • Are certificate validity periods and domain name matches correct?

Application

  • Did applications start in the defined order?
  • Is the database in a consistent state, and is it consistent with the application data?
  • Did external integrations (payment, email, partner APIs) work from the DR environment?
  • Were there any license or configuration differences?

People and process

  • Did the disaster declaration and communication chain work as planned?
  • Could the steps in the runbook be carried out as written, or are there gaps that require guessing?
  • Could users access the system and complete critical workflows?

Failback and closure

  • Was failback done as planned, and did the data return to the primary environment consistently?
  • Was the test environment cleaned up and were temporary accesses closed?

Evaluating results: measure, compare, close

The value of the test lies in the report written afterward. Fill in the following table for each test:

Measure Target Actual Difference Owner and date
Actual RTO Plan target Measured time Positive/negative Who, by when
Actual RPO Plan target Data point recovered to Positive/negative Who, by when
Failed steps Zero List Root cause Who, by when
Runbook gaps Zero List Update Who, by when

If there is a difference, there are two options: improve the architecture and process and test again, or bring the target down to a realistic level with management approval. There is no third option: ignoring the difference.

Common mistakes

  • Checking only that “the backup was taken.” A successful backup job does not mean a working service.
  • Testing only the ideal scenario. Also try scenarios in which the identity server is affected as well and key personnel cannot be reached.
  • Not measuring time. An unmeasured RTO does not tell you whether the target is met.
  • Not closing findings. If a report is written but the actions are not followed up, the same error shows up again in the next test.
  • Only IT taking part. Business owners must take part in the test for functional verification.
  • Not updating the plan after the test. The runbook must be revised with each finding.

Example test scenarios

The choice of scenario determines the value of the test. The following scenarios reveal different weaknesses:

  • Primary data center unreachable. Loss of network and power is assumed; takeover to the DR environment is performed. Network routing, DNS, and user access are tested.
  • Database corruption. An inconsistent or corrupted database is rolled back to a specific point. RPO and application consistency are measured.
  • Ransomware scenario. The identity infrastructure and the backup console are assumed to be affected; recovery is done in a clean environment from the immutable copy. This scenario is the most demanding and should be worked through together with the flow in the first 24 hours after ransomware.
  • Absence of a key person. The person who knows the runbook best does not take part in the test; others carry it out by following the document. It becomes clear whether the knowledge is in one person or in the document.
  • Provider or service outage. A third-party dependency is assumed to be down; alternatives or workarounds are tried.

Tips for making the test realistic

The more realistic the test conditions, the more valuable the findings. A few practical principles:

  1. Proceed by reading the document. During the test, the team should proceed by looking only at the runbook; any extra information of the “actually, it is done like this” kind should be written into the document.
  2. Keep time. Record the start and end of each stage; the difference between the total time and the sum of the stages shows the loss to waiting and coordination.
  3. Add small surprises. Controlled mishaps such as an invalid password or an expired certificate measure the team’s ability to adapt.
  4. Show the result to the business owner. The business owner should try the critical workflow in the DR environment themselves and say “acceptable.”
  5. Repeat. The first test usually produces many findings; the real value emerges in the second test after the findings are closed. The targets themselves should be written consistently with the definitions in the RTO and RPO glossary entry.

The post-test meeting: What should be discussed?

Right after the test, while memories are fresh, hold a short review meeting. The agenda can be kept simple: the parts of the plan that worked as written, the steps that stalled, unexpected dependencies, the measured times, and the actions to be closed before the next test. It is important to use non-blaming language; the aim is to find the weak points of the process, not of the people. The meeting notes, the findings table, and the updated runbook should be stored in one place and presented to management as a summary. These records also serve as evidence for audit and business continuity governance.

Decision summary

  • Build the test in stages: tabletop, restore, partial failover, full failover.
  • In each test, measure the actual RTO and RPO and compare them with the target.
  • Track findings with their owner and date; update the plan and the runbook.
  • In scenarios such as ransomware, also test the copies that allow going back in time. The verification levels in the 3-2-1-1-0 backup rule article help with this.

Moving forward with Nanda Teknoloji

Nanda Teknoloji carries out the discovery, design, migration, and operations steps in projects like this: designing test scenarios, carrying out the drill, reporting measurements, and turning findings into an improvement plan. For the solution structure, see the disaster recovery center and backup solutions pages. For general information about DR models, the article on the disaster recovery center and DRaaS is also useful.

Frequently asked questions

How often should a disaster recovery test be run?

For critical systems, at least one full scenario per year and several partial tests per year are recommended. Additional tests should be run after major architectural changes.

Does the test affect production?

If designed correctly, it does not. Restores in an isolated network and partial tests do not interrupt production. Full failover tests, however, require a planned maintenance window and approval.

Is failing a test bad?

No, the purpose of a test is to find the weak point before a crisis. What matters is that findings are recorded, assigned to owners, and verified in the next test.

Does a restore-from-backup test count as a DR test?

Not by itself. It proves that data can be restored but does not cover network, identity, application dependencies, and user access. A DR test includes these as well.

Related solutions

Related posts

Let's plan your critical systems together

Tell us what you need; our team will work out the right architecture and migration path with you.