Delivery and Operations

Rollback and Recovery: Restoring a System Without Losing Data a Second Time

How to tell rolling back a release apart from restoring data, failing over and fixing forward, and how to bring a system back to a proven state without…

Chapter 60Delivery and Operations

A "revert to previous version" button may bring a page back, but it will not return deleted rows, reverse a charge already taken or change the format of messages already written to a queue. Recovery starts with a precise question: which state has been damaged, which state do we need, and which changes made after the chosen point must be kept or reproduced?

In this unit, rollback means returning a particular part of the system to a previous version or configuration. Recovery means returning the service and its data to a consistent working state. These are different operations: you can roll back a software artifact and leave the database, payments and queue exactly as they are. Equally, you can restore a copy of the database and still not have a system ready for users.

After this unit you can:

  • tell apart rolling back a release, reverting a change in version control, restoring a copy, failing over to a replica and fixing forward;
  • decide which parts of the system have to be restored together: the artifact, configuration, schema, data, queues, external effects and access control;
  • assess acceptable data loss and downtime using the Recovery Point Objective (RPO) and the Recovery Time Objective (RTO);
  • build a plan on an isolated copy, check invariants and reconcile new records before switching over;
  • choose between rolling back and fixing forward according to what reduces harm without destroying the only usable copy of the data.

The teaching scenarios are invented. This unit is not a procedure for restoring any particular production database: the exact steps depend on the data engine, versions, provider, application and the team's authority. The full incident response process, communication and on-call duty are covered in Production Incidents; writing step-by-step runbooks is covered in Incident Runbooks.

What "going back" actually means

In conversation, the word "rollback" often covers several operations with different meanings:

Operation What it returns or changes What it does not do on its own
Rolling back a release Runs a previously built artifact or sends traffic to the previous pool. Does not roll back the schema, rows, caches, queues, external requests or users' actions.
A revert commit in version control Adds a new change that inverts a chosen diff in the repository. Does not change instances already running, the database or published artifacts until the new change is deployed separately.
Switching a flag or configuration Turns off a behaviour or changes a runtime parameter, provided every component reads it and supports the restored value. Does not cancel operations already completed and does not guarantee that a cache or long-running process has picked up the new value.
Restoring a copy Creates a state from a backup, snapshot or log up to a defined point. Does not automatically detect new records that were missed, and does not reproduce external effects after the point.
Failing over to a replica or standby node (failover) Moves work to another data or compute instance. Does not prove that the copy is current, healthy or free of the same logical error.
Fixing forward (forward fix) Releases a change that corrects the current state without going back to the old release. Does not automatically restore data that faulty logic has already transformed.

Picture the system as several connected layers that change independently: the software artifact; configuration, secrets and access rules; the database structure and its rows; queues, caches and search indexes; object storage; and external systems such as a payment service, an email provider or a partner's API (application programming interface). To get back to a working state, you need to know not only the code version but also the consistent versions of all these dependencies.

Rollback is not a time machine. After the deployment, new orders may have come in, emails may have been sent, balances may have changed or new messages may have appeared in the queue. Returning the whole database to yesterday's snapshot will also remove the useful changes made since. If the old schema version cannot read a new field, a careless code rollback can also turn a partial failure into a complete outage.

Database Migrations has already described how old and new code stay compatible with the schema during a change. The question here is different: what do you do once a change has already caused damage? A compatible schema can make rollback safe; deleting or transforming data can make it unusable.

Find the boundary of the failure and your point of comparison

Before choosing a remedy, write down four facts:

  1. What behaviour is broken? For example, the new code returns a search error rather than the correct result.
  2. What state has already changed? Check not only the main table but also derived records, queues, caches, logs and integrations.
  3. When did the damage or incorrect behaviour begin? An unknown start means the recovery point is undefined too.
  4. Which copy or version is proven to be usable? Check the artifact identifier, schema compatibility, the integrity of the copy and the availability of all dependencies.

"The last known good state" is not simply the previous version number. It can consist of the artifact hash, configuration, infrastructure parameters, schema version, data point, log position, message versions and the secrets required. Some of these parts can be returned; others can only be re-read or created again. State who confirms each part and by what observation.

Do not confuse a symptom with a cause. If the failure is caused by configuration, rolling back the code may change nothing. If the database is correct but the search index is stale, restoring the database may be unnecessary and may even roll back useful records. At this stage you need enough evidence to choose an action; the detailed process for coordinating people and communication belongs to incident response.

Rolling back a release has compatibility conditions

The simplest rollback is to return the previous artifact or configuration. It is often appropriate when the failure lies in the behaviour of the new version and the shared state has not undergone an incompatible change. The full model of versions coexisting is covered in Deployment Strategies. Before rolling back, check:

  • Does the old version start on the current schema without expecting columns that no longer exist or old constraints?
  • Do the old readers understand the new format of messages in the queue, cache and files?
  • Have the formats of sessions, tokens, encryption keys or the addresses of a dependent service changed?
  • Will long requests and background jobs that are already active stop without repeating an external operation?
  • Can the new behaviour be switched off with a flag without interrupting a payment checkout, an import or a booking already in progress?

A change in the repository and a change in the live system are different steps. A revert commit can prepare a release, but it does not make that release active until the corresponding delivery has finished. Likewise, rolling back a container tag does not restore external configuration if that was updated separately. Use an immutable artifact identifier and a known previous manifest, not a tag like latest whose contents may have changed.

The old artifact may not be safe: it may contain a known vulnerability, an expired certificate or a contract that is no longer supported. Compare the harm from the new failure with the harm from returning to the old state. When the old code is not compatible with the current schema, it is often safer to switch off just the problematic behaviour or release a small fix forward than to force the whole application back.

Backups, logs, replicas and failover give different guarantees

A backup is a stored representation of the state at a particular moment. It can be a logical export of tables, a physical copy of the storage or a volume snapshot. They have different properties: a logical copy is easier to move or import selectively, while a physical copy can sometimes be restored faster as a consistent instance. Do not draw conclusions about whether point-in-time recovery is possible without checking the format and procedure against the documentation for your particular database.

In PostgreSQL 18, continuous archiving relies on a base backup of the cluster and an archive of the Write-Ahead Log (WAL). Point-in-time recovery (PITR) applies records from the WAL up to the moment you need; the log copies must cover the entire path from the chosen base backup without gaps. The documentation warns separately that an incomplete set of logs will prevent recovery from finishing, and that an incremental backup needs every backup it depends on. Such recovery covers an entire cluster, not an arbitrarily chosen table. In addition, WAL does not bring back changes to postgresql.conf, pg_hba.conf and pg_ident.conf: those files have to be saved separately. These are the mechanics of PostgreSQL 18, not a promise for any data store. (PostgreSQL 18: continuous archiving and PITR, accessed 10 October 2026.)

Replication maintains a separate instance for availability, reading or failover. It often transfers changes quickly — and can therefore just as quickly transfer an accidental deletion, an incorrect update or a logical error. A replica that repeats every transaction without delay is useful against a node failure, but on its own it is not an independent historical copy. A delayed replica gives you a short window to stop a faulty write from being applied, but it does not replace a long-term backup and it also needs to be checked.

Failover moves the workload to a standby node. If the cause is a node crash, this can restore service. If the cause is a bad write, a faulty configuration, compromised access or a damaged data format, failing over can carry the damage to the standby node or make it worse. Before failing over, check the replica's lag and the point up to which it has applied data; afterwards, record which node is now accepting writes.

A managed service has its own limits too. Amazon Relational Database Service (Amazon RDS), for example, creates a new instance when restoring to a point in time, and the source instance is left unchanged. This gives you the chance to check the restored copy before changing the route clients use. The service limits the choice of point to the available backup retention period, and some actions or engines have their own conditions. Check the specific engine and parameters rather than carrying RDS behaviour over to another product. (Amazon RDS documentation: restoring to a specified point in time, accessed 24 September 2026.)

Caches and indexes can often be rebuilt from the source of truth, but only if that source is correct and known. A queue holds unfinished work, so you cannot simply clear it without understanding the consequences for customers. Object storage and files may have their own backup schedule. For each layer, ask: can it be safely recreated, does it hold a unique source of truth, and is its version consistent with the chosen database point?

RPO and RTO describe acceptable loss and downtime

The Recovery Point Objective (RPO) is the maximum span of changes before an incident that the business is prepared to lose; it sets how recent the last acceptable recovery point must be. The Recovery Time Objective (RTO) is the maximum acceptable time from an interruption to the restoration of the required service. Google describes these objectives as "how long until recovery" and "how much data can be lost"; the specific values vary with the product's criticality and business needs. (Google Cloud: architecting disaster recovery for cloud infrastructure outages, reviewed 10 May 2024.)

These objectives turn a technical choice into a product decision. For a personal draft catalogue, losing a few minutes may be acceptable; for payment accounting, even one inconsistent record may require manual reconciliation. A shorter RTO usually needs more prepared infrastructure and tested switchovers; a smaller RPO needs more frequent saving of changes, a longer log or more synchronous transfer. This affects complexity, cost and operational workload. There is no universal value for all systems.

Compare the objectives with measured recovery results, not just with the backup schedule. RTO covers finding the right copy, creating it, checking it, reproducing useful new changes, changing the route, checking that things work and bringing dependent services back. "A copy every night" proves neither a daily RPO nor an RTO within a day: the recovery duration, the integrity of the file, the keys required and whether the configuration is current are all unknown.

A sequence for safe recovery

This is not a universal runbook but a line of reasoning that you need to turn into steps tested for your own system:

  1. Preserve the evidence and the state. Record the time, the release and configuration identifiers, the database and log points, signals, logs and actions already taken. Where possible, keep the damaged copy separately. Do not destroy it until you have decided it is not needed for investigation or for extracting useful new data.
  2. Stop further damage. Halt exactly the change that keeps causing harm: for example, writes from the new logic, repeated imports or a background job. Do not switch off the whole system at random if that would lead to duplicates, lost payments or a growing queue. State which operations are still allowed to complete.
  3. Choose the target state. Write down the artifact, parameters, schema, copy and log point. Check the source, time zone, database identifier, the completeness of the log chain and the match with the engine version. "Yesterday at 02:00", without a precise source and a confirmed timeline, is not an adequate plan.
  4. Create a copy for checking, separately. Use an isolated environment, prevent ordinary clients from connecting to it, restrict outbound integrations and switch off automatic emails, webhooks and payments. Do not switch production to the target just because the restore process finished without an error.
  5. Check consistency. Check that the basic invariants make sense for the product: keys reference rows that exist; totals match the accounts; the number of reserved seats does not exceed the number available; there are no duplicate unique identifiers; checksums or totals agree with an independent source. A technical status of "the server started" is not a check of the business state.
  6. Build a register of changes after the point. Cross-match transaction logs, request identifiers, business events, invoices, confirmations from the payment provider, files and emails. Distinguish between: changes that must be reproduced; records already included in the restored copy; duplicates; unconfirmed external actions; irreversible operations.
  7. Reconcile the data before writes resume. Stop or restrict competing writers, decide the order of import and deduplication, and test re-execution on the copy. If the source of truth does not allow two branches of records to be merged safely, do not guess: keep the system in a controlled mode until the data owner decides.
  8. Switch the write source once. Check the target's name and address, schema, access and the writable node immediately before acting. After switching, run a small control business scenario, check the actual records, and only then gradually restore normal load, if the architecture allows it.
  9. Reconcile the consequences and keep the copy. Check error logs, queues, counters, integrity and repeated effects. Record what loss was accepted, which records were reproduced and who approved it. Keep the previous instance and the evidence long enough to be able to correct a wrong choice of point.

Google Cloud documentation advises evaluating a recovery test on at least data integrity, RPO and RTO, and checking the application and its critical dependencies on the restored data in a non-production environment. It stresses that a successful backup does not yet mean the whole application works. In Google's book Site Reliability Engineering (SRE), the team likewise warns that replication does not protect against many sources of data loss, and that a backup strategy should be judged by its ability to restore the state you need. (Google Cloud Well-Architected: perform testing for recovery from data loss, reviewed 30 December 2024; Google SRE: Data Integrity, 2016.) This is not a requirement to use Google products, but a set of evidence criteria for any recovery plan.

For physical backups, PostgreSQL 18 has the pg_verifybackup utility: it checks the manifest, the set of files, the checksums and the logs available to it for a pg_basebackup copy. But the documentation itself says plainly that this check does not cover every condition the server will check during recovery, and it still recommends a test restore and a check of the contents. So a successful automated check is a useful part of the evidence, not final proof that the backup works. (PostgreSQL 18: pg_verifybackup, accessed 24 September 2026.)

Code, database and external effects may need different decisions

After a change, consider several decisions separately. They can be combined, provided each one has a defined state and owner:

Situation Likely first option The check without which the option is risky
The new artifact is failing; state has not changed Roll back to the previous artifact or switch the feature off. Compatibility of the old code with the current schema, cache and messages.
A configuration change broke behaviour Restore the previous value or make a narrow parameter change. Whether the change is needed only for one group; which processes have already re-read the configuration.
Incorrect data is still limited, and the cause is understood Fix the data forward with a tested rule; possibly release code. Whether the original values have been kept and whether damaged rows can be told apart correctly.
A large part of the data is corrupted up to a known point Restore a copy to that point, with separate reconciliation of new records. Whether the copy and its log are complete, whether changes after the point can be brought back, whether there is a plan for external effects.
It is unknown whether the replica contains the error Stop automatic failover; compare the state and the time of the last write. Replication does not by itself separate correct transactions from faulty ones.
Money has been charged, emails sent, events published Separate recovery of internal data, plus a compensating action and coordinated communication. Repeating the effect may be undesirable; you need receipts and the external system's identifiers.

Rollback is justified when the previous code is compatible with the current state and quickly stops the harm. Fixing forward is a better fit when the previous version cannot read data that has already changed, or when the failure is local and the data can be corrected in a verifiable way. Restoring from a copy is needed when the source of truth is so damaged that a targeted fix is unreliable; but it opens a separate task — bringing back the correct records created after the point.

An external action cannot be considered "cancelled" just because the program has gone back to the old version. Returning a payment needs a separate refund operation; a sent email cannot be recalled; a message delivered to a partner may already have started their workflow. An idempotency key, an intent log and a receipt help detect repeats, but they do not turn back time. Before compensating, establish whether the external system actually accepted the original operation.

Example: the catalogue returns an error, but the data is still correct

An invented example. A product catalogue service received a release that changed how the index is read. Search started failing, but the commands that write products did not change; a check of the main table shows the previous prices and descriptions, and the reindexing queue contains no unusual jobs. The old web artifact uses the same schema and index format.

The service owner stops the rollout from widening and cross-checks the exact artifacts and configuration. Since there is no sign that the data source is damaged, the team can roll back the web code or switch off the new search path, leaving the database in its current state. Before returning traffic, it checks more than a Hypertext Transfer Protocol (HTTP) 200 response code: sample queries with different filters must find the expected products, and a newly written product must remain visible in search.

If the problem lies only in the derived index, the team may need to rebuild it from the tables once the faulty code has been removed. Restoring the whole database to an old snapshot would be unjustified: it would discard correct product changes and would not guarantee that the query logic itself was fixed. Here, "roll back the release" and "restore the data" are different decisions with different costs.

Example: a wrong price has already reached orders and payments

An invented example. A new version of the catalogue applied a rounding rule incorrectly and changed the price of 240 products. Within twenty minutes, customers placed 40 orders. The external payment service confirmed the charge for some of them; for a few requests, confirmation has not yet arrived.

The team switches off the faulty calculation of new prices and stops the bulk update. Rolling back the artifact may restore the old formula for future page views, but the orders already have a fixed amount. And if the database were restored to the moment before the release, the orders themselves, the payment confirmations and the statuses created later might disappear from it. The payment provider will not cancel a charge just because an internal record has vanished.

The owner of the financial data cross-matches the price-change audit, the version of the rule, the unaltered order log, request identifiers, payment receipts and webhook responses. They mark each order as: not sent to the provider; confirmed; declined; or with an unknown outcome. For an unknown outcome, the team first queries the status by the external identifier — a repeated payment request could create a second charge.

Then the team compares two options. If the set of damaged prices can be identified exactly from the event log and the original values, a controlled corrective recalculation going forward can preserve the 40 orders and other new records. If the damage has spread to a large number of rows and the start is known precisely, the backup can be restored into a separate database, the correct prices extracted from it and matched against the current orders. That is still not a decision to replace the whole current database.

In every option, a separate decision is needed for orders already paid: fulfil them at the displayed price, notify the customer of the change, or refund the difference or the full amount, in line with the shop's policy and the applicable customer-service process. The team does not repeat charges automatically, and records the receipt or the result of the compensation in the log. The success criterion is consistent prices, orders and actual payments — not merely a page that opens again.

Example: the backup file exists, but recovery is not yet proven

An invented example. The team finds a nightly base backup taken at 01:00. Some WAL files are also visible in storage, and the backup job dashboard shows "successful". At 11:40, the application began deleting records by mistake. The database owner wants to restore the state as of 11:35.

During the check, it turns out that the WAL directory has a gap in the segments between 10:50 and 11:10, and the team has found no other complete source of those segments. In addition, the recovery configuration, access credentials and the path to the secrets are stored outside WAL; the required parameters and key have not yet been confirmed. In PostgreSQL, the data logs do not bring back manually changed configuration files. So the existence of the base file proves only that some file has been saved. It does not prove that it can be fully replayed up to 11:35 and used together with a compatible configuration and access.

The team does not replace the main database on a guess. It preserves the current files and metadata, checks the chain of backups, logs, keys and versions, and then tries to restore an isolated instance to the latest point actually covered by continuous records. If the log after 10:50 is missing, PostgreSQL cannot magically reproduce the history up to 11:35 from a single base backup; the proven point may for now be no later than 10:50. The team will have to evaluate other copies, event sources and the cost of losing roughly 50 minutes of changes, or choose a separate fix.

Even when the server comes up, the owner checks the schema, keys, application accounts, control totals and a critical action in an environment not connected to real customers. Only then can anyone speak of a verified point. The backup became a candidate for recovery; what proved it usable was the complete recovery path, the configuration and a successful check of data and behaviour.

Real case: several layers of recovery failed at GitLab

On 31 January 2017, GitLab.com lost part of the data in its primary database. While trying to resynchronise a lagging replica, an engineer was supposed to clear the secondary node's data directory, but ran the dangerous command on the primary. That mistake alone was not enough to cause a long outage: the pg_dump backups never appeared in storage, because the job was using pg_dump 9.2 against PostgreSQL 9.6 and was failing; the emails reporting the failure were being rejected because of a problem with Domain-based Message Authentication, Reporting, and Conformance (DMARC). The WAL change log was not being archived, and the replica had also been wiped during the attempt to recreate it. (GitLab: postmortem of the database outage of 31 January, 10 February 2017.)

The option that remained was a manual snapshot made with the Logical Volume Manager (LVM) about six hours before the outage, for a test environment. To recover, GitLab copied the database from staging to the production node; slow network disks meant the copy took about 18 hours. Webhooks were restored separately from another instance of that copy, and the ID sequences were incremented so as not to reuse numbers that might already have existed after the lost interval. The postmortem estimates the loss of records between 17:20 and about 23:30 Coordinated Universal Time (UTC) that day; another part of the report rounds the end of the window to 00:00, so the time is given approximately. Git repositories and wikis were stored separately and were not affected. (GitLab: recovery details and the estimate of data loss.)

This is not an example of a failed release rollback: it is an example of database recovery in which every assumed fallback had its own gap — the wrong target for a destructive action, replication without a log archive, a backup that was never being created, undelivered alerts and recovery from a slow copy in a different environment. The lesson is not to blame a single operator, nor to treat any single measure as universal. GitLab itself recorded problems with node visibility, backup monitoring, recovery testing and ownership of data preservation. The technical details describe its 2017 system and are not a recommendation about GitLab's current configuration.

Limits on what a software agent may do

An agent can help collect version identifiers, find a backup, calculate the difference between states, check metrics or prepare a reconciliation plan. That does not mean it should choose the recovery point on its own or overwrite the working database. Before making a change, the agent must show the exact target, the scope, the expected loss after the point, the uncertainties it has found and how the result will be checked.

For a potentially destructive action, the agent must:

  • confirm the identity of the environment, instance, region, storage and chosen copy immediately before running anything;
  • work with the least privileges required and on a separate copy; prepare the plan and a comparison report first;
  • not delete original files, erase the log or restore over the live source unless that is a clearly permitted and tested action;
  • obtain explicit approval from an authorised person before an irreversible production switchover or before accepting the loss of records;
  • stop if the point cannot be determined, if access checks or logging are unavailable, or if an external operation has an unknown outcome; and not repeat the action at random.

OpenAI's documentation for the Agents SDK (Software Development Kit) describes a pattern in which a sensitive tool call is paused for human approval, and resuming it uses the same run state. Claude Code's documentation separately shows that permission modes set the basic boundary between autonomy and oversight, and that a command being available does not mean it is safe. These are examples of how agent limits are implemented, not a substitute for the data owner's policy. (OpenAI Agents SDK: guardrails and human review, accessed 24 September 2026; Claude Code: permission modes, accessed 24 September 2026.)

Summary

Rolling back code, failing over to a replica, fixing data, restoring a copy and compensating for an external effect all change different parts of the system. First determine exactly what is damaged and to which proven point the state must return. Check the compatibility of versions, logs, configuration and dependencies; create an isolated copy; reconcile invariants and the changes made after the point; only then decide on the write source. RPO and RTO belong to the product and the business. A backup file or a green job status does not prove that the application can be restored.

Sources and further reading

Materials were checked on 24 September 2026. The technical behaviour in the examples is tied to the versions and providers named below; before a real recovery it must be checked against the actual system.