Deployment Strategies: Limiting the Risk of a Release

Database Migrations looked at changing a schema and its data in a controlled way; Continuous Delivery and Deployment followed a verified artifact from build to the production environment; and Environments explained why the result of a check depends on configuration, data and external resources. This unit focuses on exactly how a new version starts working: all at once, gradually, side by side with the old one, or only for a chosen group of users. This is not a matter of picking a button in a dashboard. The strategy has to fit the consequences of a failure, the compatibility of versions, the load, the way you observe the system and the team's ability to stop the next step.

After this unit you can:

• tell deploying code apart from releasing it and from switching a feature on;
• compare a planned pause, a rolling update, a blue/green deployment, a canary and feature flags;
• decide whether simultaneously active versions of code, data schema, application programming interface and background worker are compatible;
• choose signals and stop conditions that will genuinely expose the risk for a particular system;
• explain why a small or unrepresentative canary group does not provide enough evidence;
• prepare a transition plan and a recommendation for the accountable owner, without an agent switching production on its own.

This unit describes strategies for introducing a change and the limits of their compatibility. It is not a guide to restoring data or undoing irreversible operations: those questions are covered in Rollback and Recovery. No deployment option, on its own, guarantees uninterrupted availability, correct data or fast recovery.

Deployment, release and user impact

A deployment changes the code or configuration running in an environment. A release makes new behaviour available to users or to other systems. Switching on a feature opens up one particular capability, for example to a specific organisation or a share of accounts. These events can coincide, but they do not have to.

For instance, a server may already contain a new checkout screen but hide it behind a feature flag that is switched off. The code is deployed; the feature has not been released. Conversely, changing a setting can open up a feature that already exists in production without any new deployment. This separation is only useful if the hidden code does not break the compatibility of data and of older clients.

To assess the risk, record not just the name of the method but the following facts:

Question	What to establish before the first step
What is changing?	Code, configuration, database schema, queue messages, a flag, or a combination.
Who will get the change?	All requests, a chosen region, test accounts, a defined cohort or a specific internal consumer.
Which versions will coexist?	Old and new web versions, two types of background worker, old mobile clients, the previous and the new schema.
What could be damaged or delayed?	Records, bookings, payments, queues, reports, external calls, unfinished jobs and user access.
What evidence will allow us to move on?	Signals of technical health and of a correct business process, a sufficient volume of observations, a named person responsible.
What does "stop" mean?	Stop increasing the share, switch off the flag, stop the new process or preserve the state for the decision owner.

Continuity is not the same as zero downtime. For example, a new version might answer requests but write an incompatible format, while an old process is still working through the queue under the previous rules. You need to assess everything that takes part in the business flow, not just the web server.

Five ways to start a change

There is no universally "best" strategy. A product with a single static page does not need the same levels of protection as a booking system where the same time slot must never be sold twice. The matrix below compares common options, but it does not replace checking your platform and your own constraints. A feature flag controls whether a behaviour is available and can be combined with rolling, blue/green or canary.

Approach	How availability changes	What it makes easier	Main risk or cost
Planned maintenance pause	During the transition, users see a notice or the service does not accept operations.	A simple system can change in one clear step when a short pause is acceptable.	You need to agree the window, inform users and check that unfinished operations are not left in an ambiguous state.
Rolling update	Instances are replaced in batches; old and new ones may run at the same time.	You do not have to keep a full second copy of the system; existing resources can be reused in parts.	While requests are being processed, both versions must be compatible with each other and with shared data.
Blue/green	A prepared second group of instances is checked and then traffic is switched to it.	Separates preparation from the switch; the previous group can be kept for a while.	Needs extra resources and correct handling of routing, sessions, queues and shared records.
Canary	A limited, defined segment gets the new version first; after evaluation it is enlarged.	Lets you see how the new version behaves on real but limited traffic.	A small, short-lived or unrepresentative cohort may miss a defect; the share of traffic is not a measure of the quality of the evidence.
Feature flag	Code is deployed separately, and the feature is switched on for a group or on a condition.	Separates deploying code from the availability of a particular behaviour.	A flag does not make incompatible code safe; old code paths, access rules and temporary switches also need maintenance.

A method can be implemented differently in different systems. In Kubernetes, for example, during a rolling update the Deployment controller starts new pods and terminates old ones in batches; maxUnavailable limits the number or share of pods that may be unavailable during the update, and maxSurge limits how many pods may temporarily exceed the desired count. This manages instances; it does not prove that business operations are correct. The controller reports when progress has stalled, but it does not roll back code automatically on its own. Kubernetes documentation on Deployments, accessed 24 September 2026.

For a planned pause, be specific: which operations are temporarily unavailable, whether a request already in progress can complete, how the user will retry, and how the team will confirm that the service is working again. If even a short pause creates significant risk — for taking urgent orders, say — that is an argument for a safe gradual transition, not proof that one has already been prepared.

A blue/green deployment does not mean automatic rollback. It can make it easier to send traffic back to the old group, as long as that group is still there and compatible with data already written. Switching back does not cancel payments, emails, processed messages or format changes. Some managed platforms offer "blue/green", "canary" and linear traffic-shifting modes, each with its own requirements and limits. Amazon Web Services (AWS) Elastic Container Service (ECS), for example, supports such switching options for a service, but its requirements for the load balancer and resources are not a rule for other platforms. AWS: service deployment options, accessed 24 September 2026.

To choose, use a trade-off lens, the author's adaptation of ideas from Software Architecture: The Hard Parts: weigh acceptable downtime, the reach of a possible failure, the compatibility of shared state, extra resources, coordination between the owners of different components and the complexity of maintenance. A strategy only moves cost and risk between stages; a more elaborate scheme is not automatically more reliable. It is justified when it reduces a specific cost of failure by more than it costs the team to build, coordinate and operate. Information about the book and its authors, accessed 24 September 2026.

Feature flags can also control an experiment or access for a single organisation. Make sure the same entity consistently lands in the same group if that matters for the scenario. If requests are assigned randomly rather than by a stable account identifier, for instance, a user may see different behaviour on neighbouring pages. The details of percentage allocation depend on the flag provider. Cloudflare Flagship explains its percentage rollout rules, accessed 24 September 2026.

Compatibility is what lets versions coexist

Rolling, canary and blue/green approaches can all create a period in which different versions of the code run at the same time. The same period can arise without any special strategy: a mobile client stays on someone's phone, a job waits in a queue for several hours, or a background worker began a long piece of processing before the server was replaced. The question is not only "does the new version start?" but "do the old and new participants in the system agree on the format and meaning of the data?"

Databases and programming interfaces

An application programming interface (API) is the set of rules for requests and responses between programs. If a new server demands a required field that an old client does not yet send, some requests will fail even though the deployment process itself is healthy. The same thing happens if a new client expects a response field that the old server version does not return.

For a database schema change, use several compatible steps when the risk justifies it: first add the new field or table so that the old code keeps working; deploy code that reads both formats or temporarily writes both the old and the new; check that the data has been moved across completely; and only after the transition period is over remove the old field. This family of approaches is often called "expand–migrate–contract". Database Migrations explains this transition for schema and data changes in more detail. Such a sequence supports the coexistence of old and new consumers, but it does not replace a recovery plan if the migration damages data; those limits are covered in Rollback and Recovery. The DevOps Research and Assessment (DORA) programme on loosely coupled teams and compatible changes, Martin Fowler's site on parallel change, accessed 24 September 2026.

Decide in advance whether the old code really has to run in parallel with the new. A narrow maintenance window may allow a one-off incompatible change, for example, but only if all consumers are stopped, the queues have been drained under control and users understand that the service is unavailable. Otherwise the team is relying on a short "window" with no evidence that old requests are gone.

Queues, workers and long-running operations

A background worker is a separate process that reads jobs from a queue or a schedule and completes them outside the user's page request. It can outlive a change of web servers. If a new release changes the format of a queue message, the old worker may not understand the new message, and the new worker may not understand an old one that has not yet been processed. Solutions include a compatible format with a message version, updating readers and writers step by step, or a controlled pause while the queue is processed.

An operation can be interrupted between an external effect and the recording of its confirmation: for example, the payment provider received the request, but the process ended before the response was saved. On retry, the worker may charge the customer a second time. An idempotent action, when repeated, does not duplicate the intended state change or external effect, although the interface's response to the repeat may differ. For a payment, the provider may support an idempotency key that ties retries of the same request together, but this guarantee must be checked in the documentation of that particular provider. It is a property of the operation's logic, not of a canary or a load balancer.

Check how the process shuts down: does it stop taking new jobs, does it finish the current one, does it have a time limit, and what happens after a crash? In Kubernetes, the typical lifecycle sends the process a termination signal and may force-stop it after a set period; it is the application that must handle the completion of a request or background job correctly. Kubernetes describes pod termination, accessed 24 September 2026. Do not treat a worker restart as proof that its job completed exactly once: the Kubernetes documentation warns, for instance, that the Job controller may start the same program again even when a single successful completion is configured. Kubernetes Jobs, accessed 24 September 2026.

External consumers and delayed compatibility

Webhooks, export files, caches and mobile clients can live outside the control of the current deployment. Before removing a field or an old format, find out who reads it and how long such copies stick around. "The servers have been updated" does not prove that all clients or queued messages have been updated.

What a canary has to prove

A canary is not simply "roll out ten per cent". It is a limited trial of the new version on a defined part of the system, after which the team compares the observed consequences and decides whether coverage can be widened. The result depends on who ended up in the sample, which operations they perform, how much data has accumulated and whether the metrics are capable of showing the failure.

Before you start, define:

1. The cohort and the control. By what stable key are users, organisations, regions or instances split? What does the old version receive for comparison? All paths must remain compatible with shared state.
2. The main hypothesis. Which failure should become visible: more failed bookings, slower search, wrong amounts, a delayed queue, a rise in request errors?
3. Signals and thresholds for this system. Compare the new group's results with a historical baseline or a control segment. Choose indicators tied to the expected harm, and decide in advance who judges an ambiguous signal.
4. The volume of evidence. The minimum number of relevant operations, the types of client required and the load periods. A few requests without errors may say nothing about a rare but expensive case.
5. The rule for stopping and promoting. Who is entitled to stop the next stage? Which failure blocks expansion, what can be investigated without changing traffic, and when is the evidence gathered enough?
6. The fallback path. How do you stop sending new operations to the new version without creating duplicates, losing the queue or causing conflicts in shared records?

Do not carry a universal "five minutes" or "one per cent" from one product to another. A few minutes may reveal a syntax error on a popular page, but not a nightly import, a monthly invoice or the small share of bookings that involve refunds. Google's Site Reliability Engineering (SRE) workbook advises giving the cohort enough representativeness and observation time for the specific metric; a weak signal from a small sample can be missed or lost in the overall statistics. Google SRE Workbook: Canarying Releases, accessed 24 September 2026.

An instance readiness check does not cover the whole business scenario either. A Kubernetes readiness probe, for example, reports whether the configured check passes; it does not automatically know whether the last seat can be booked twice, whether the amount is correct or whether a webhook was processed. Kubernetes explains its health probes, accessed 24 September 2026. Combine the technical signal with observation of the business process.

Example: a product catalogue and a booking service

An invented example. The same team maintains two products: a catalogue with product filters and a system for booking seats at a limited-capacity event. For the catalogue, a brief search outage is inconvenient, but the product page stays available and the data can be re-read. In the booking system, a failure could sell the same seat to two people or lose the payment state.

Option	Catalogue	Booking system
Planned pause	May be the cheapest option for a small change, if users see a clear notice.	Often unacceptable during sales or event entry; unfinished bookings need separate protection.
Rolling update	Acceptable if old and new code read the shared index and schema in the same way.	Needs evidence that both versions lock a seat and record its state in the same way.
Blue/green switch	Can move read traffic quickly, but caches and indexing must be considered.	Switching servers alone does not remove conflicts between concurrent writes or old workers.
Canary	A good way to watch search latency and error rates on real requests, if the cohort covers typical filters.	Needs separate signals for correct bookings, duplicates, cancellations and payments; a small sample may hide a rare last-seat conflict.
Feature flag	The new filter can first be switched on for the team or a group of customers.	An interface flag does not stop an old client or a background worker from making a competing write.

For the catalogue, the team chooses a rolling update and a flag for the new filter. For booking, it first checks schema compatibility and the reservation logic with two versions running at once, adds a signal for the same seat being assigned twice, and limits the canary to a small group of events with controlled support. If the volume of operations is too small for a reliable conclusion, the team does not call it proof of "zero risk": it keeps the old path or waits for further review by the product owner.

The choice follows from the cost of failure, not from how prestigious the method is. The catalogue may have more room for a fast transition. For booking, extra isolation, manual approval and longer observation may cost less than fixing double sales.

Example: the canary missed the failure

An invented example. A document-import service sends a small share of requests to the new version; most users stay on the old path, and only a few simple documents reach the canary. At ordinary times they contain no nested tables, and all of them complete successfully. A few hours later, large documents with non-standard encoding arrive at the same system; the new version exceeds its memory limit and puts the jobs back in the queue.

The overall average latency barely changed, because the canary's share was small. But the "0 errors" signal was weak: the cohort included no complex files, and the system counted a re-queued job as a new job rather than as a failed import. The team stops increasing the share, adds a separate indicator of import completion by file type and checks that every job has an identifier that makes a repeat detectable. Before the next step, it waits for enough relevant cases rather than for an arbitrary timer.

The important lesson: a canary is not a safety property in itself. You need a representative sample, a comparable control, a metric that measures the outcome you care about, and an owner who can stop the promotion. Expanding without signals and stop criteria defined in advance increases the potential impact without guaranteeing better evidence.

Example: a blue/green switch and a message format

An invented example. The team built a green version of the system, checked the web page and switched traffic to it. Along with the code, the green group began writing a new order-message format to the queue. The blue group of background workers stayed active for a few more minutes and does not understand the new field. Once the failure is spotted, web traffic is switched back to the blue version, but the messages have already been written in the incompatible format; the old worker rejects them, and the new format has partly affected order accounting.

Switching the route back restored compatible web code, but it did not return the queue to its previous state or cancel actions already carried out. Before deploying, the team should have either made both versions compatible with both formats, separated the stages of changing the message writer and the message reader, or stopped processing and coordinated the transition with the queue's owner. After the change, it needs evidence about the number of old and new messages, rejected jobs and successful operations.

This example shows the limit of a deployment strategy. It controls where new traffic goes, but it is not an automatic way to undo recorded data or an external effect. Recovery is covered in Rollback and Recovery.

Real case: Cloudflare's global configuration

Cloudflare has described a roughly 62-minute outage of its 1.1.1.1 resolver on 14 July 2025. On 6 June, a configuration error for the future Data Localization Suite (DLS) service, which was not yet running in production, included the resolver's prefixes but had no immediate effect. On 14 July, a change to that service's topology triggered a global configuration update and withdrew the 1.1.1.1 routes. In the postmortem, Cloudflare notes that the legacy components did not support gradual deployment; the company plans to retire them in order to move to staged propagation of changes with health checks. This is a description of a plan, not confirmation that the move has been completed. Recovery was also staged: the fix was checked in test locations before being applied more widely. Cloudflare: the 1.1.1.1 incident of 14 July 2025.

This is an example of releasing service configuration and topology, not a typical canary for a web application. It shows that a delayed trigger can propagate a change made earlier, and that a global operation with no intermediate stages can reach a large part of the system before anyone sees the consequence. The postmortem also describes that recovery was not instant: the configuration was restored in stages, with individual locations checked before full traffic was restored.

Practical lessons for a system owner:

• check which changes have accumulated and could be activated by a later event;
• limit the blast radius of a configuration change and check intermediate segments;
• include availability checks from different vantage points and of different dependencies among your signals;
• do not call a return to the previous setting a full recovery until the user path has been checked.

A postmortem is one organisation's account of one incident, not proof that staged deployment by itself would prevent any outage. The outcome depends on which signals, segments and dependencies the controls cover.

The decision owner and the limits of agent automation

Automation can prepare an artifact, check its identifier, run permitted checks, collect metrics, compare them with conditions defined in advance and report exactly what does not match. This cuts down manual cross-checking and gives the decision owner reproducible evidence. What an agent can actually do is determined by its tools, its environment and the rights it has been granted; an instruction in a chat does not by itself prove those limits are in place. In OpenAI's documentation for the Agents SDK (Software Development Kit), an approval can pause a specific tool call until a human decides, and checks are recommended at the boundary of the tool that creates a side effect. Anthropic's documentation for Claude Code likewise describes permission modes and rules for tool actions. These are examples of specific products, not universal behaviour of all agents. OpenAI: guardrails and human review, Anthropic: Claude Code permission modes, accessed 24 September 2026.

Switching production traffic, enlarging a cohort, running an incompatible migration or changing real data must each be delegated and bounded separately. If the agent has not been given explicit authority for that operation, it prepares a recommendation: the target artifact identifier, the environment, the signals checked, open uncertainties, the stop condition and the plan for the next check. "All checks are green", without naming the checks and their results, is not enough for such a decision.

A flag can look like an easy way to release, but changing it can still open behaviour to real users, send an external call or start data processing. A kill switch is a separate flag or controlled switch that quickly turns off one defined capability; it does not undo actions already performed, and it needs an owner, a tested access path and a known behaviour if the flag service is unavailable. Check the target group, the flag's owner, access to its controls and the audit record. Several flags increase the number of behaviour combinations that can coexist, and with it the complexity of testing. That does not mean you must test every possible combination: first check the configuration expected in production, and the safe path with the new behaviour switched off. Temporary flags must be removed once the transition is complete: over time, unknown defaults and old code paths become a permanent source of confusion. LaunchDarkly distinguishes deployment from release and discusses the flag lifecycle, Pete Hodgson explains the cost of maintaining and testing flags, accessed 24 September 2026.

Summary

A deployment strategy limits the manner and the blast radius of a transition, but it does not replace version compatibility, control over data, business signals or a decision owner. A pause may be the simplest option for a system where a short outage is acceptable. Rolling and blue/green approaches create a period in which versions coexist; a canary provides useful evidence only with a sufficient and representative sample. A feature flag separates code from its availability, but it does not remove the risk of incompatible behaviour. Before each transition, record what is changing, which signals allow you to continue, who stops the process and what remains outside the reach of a simple switch back.

Sources and further reading

Online documentation and postmortem pages were checked on 24 September 2026. The behaviour of a particular controller, cloud service or flag depends on configuration and changes over time; check your platform's current limits before adopting anything.

• Kubernetes, Deployments — rolling updates, availability parameters and reporting of stalled progress; current documentation, accessed 24 September 2026.
• Kubernetes, Pod lifecycle — process termination, signals and the grace period; current documentation, accessed 24 September 2026.
• Kubernetes, Jobs — repeated execution and the limits of the "one job, one run" assumption; current documentation, accessed 24 September 2026.
• Kubernetes, Liveness, Readiness, and Startup Probes — what configured health probes check; current documentation, accessed 24 September 2026.
• AWS, Amazon ECS service deployment options — platform-specific deployment options and requirements; accessed 24 September 2026.
• Google SRE Workbook, Canarying Releases — control groups, cohort representativeness and choosing sufficient signals; accessed 24 September 2026.
• Cloudflare, Percentage rollouts — one specific implementation of percentage-based feature allocation; page updated 24 June 2026, accessed 24 September 2026.
• LaunchDarkly, Deployment strategies — separating deployment from release and managing flags; current page, accessed 24 September 2026.
• DORA, Loosely coupled teams — compatibility between components and independent changes; current page, accessed 24 September 2026.
• Cloudflare, 1.1.1.1 incident on July 14, 2025 — postmortem of a global configuration outage and staged recovery; published 15 July 2025.
• Danilo Sato, Parallel Change, Martin Fowler's site, 13 May 2014 — a compatible transition between two interfaces or versions in expand, migrate and contract phases.
• Pete Hodgson, Feature Toggles (aka Feature Flags), MartinFowler.com, 9 October 2017 — managing the lifetime of flags and selecting which behaviour combinations to test.
• OpenAI, Guardrails and human review, current documentation, accessed 24 September 2026 — approval of tool calls and checks at the side-effect boundary.
• Anthropic, Choose a permission mode, current Claude Code documentation, accessed 24 September 2026 — one specific implementation of permission modes and rules.
• Neal Ford, Mark Richards, Pramod Sadalage and Zhamak Dehghani, Software Architecture: The Hard Parts, O'Reilly edition, October 2021 — trade-offs between architectural decisions. The treatment in this unit is the author's adaptation of the book's ideas about the cost of decisions and trade-offs; citing the title does not mean every strategy described here comes from the book.