Delivery and Operations

From a Verified Change to a Controlled Release

How a verified change becomes a controlled release — one traceable artifact promoted through environments, explicit gates, compatible data changes,…

Chapter 57Delivery and Operations

Continuous integration (CI) explained how automated checks report on a specific revision of code. Now we follow the path further: how a team prepares a change for release, delivers it to the right environment, checks its behaviour and decides whether to show it to users. In this topic the abbreviation CD has two meanings. Continuous delivery means that changes can be released on demand; continuous deployment means that every change that passes the defined conditions is deployed automatically to the production environment. The DevOps Research and Assessment (DORA) programme, which studies software delivery and operations, distinguishes the two practices: a team can keep itself permanently ready to release and still wait for an explicit decision before production. DORA describes both meanings of CD separately.

After this unit you will be able to:

  • tell a build, an artifact, a code release, a deployment and the enabling of a feature for users apart;
  • trace one verified artifact from the build through the environments, and check that it is exactly that artifact being promoted;
  • explain which automated and human conditions should stop or allow a release;
  • spot the risk of an incompatible data change, even when the new feature is hidden behind a flag;
  • name the evidence you need after deployment, and decide on the first safe action when that evidence is bad;
  • outline which preparatory tasks can be delegated to an agent without giving it undefined access to production.

Six different events

A merge incorporates a verified change into the target branch of the code. It may trigger a further workflow, but on its own it neither deploys a new version nor makes it available to users. A build turns the code and its declared dependencies into a package that can be run or hosted: for example, a web application archive or a container image. An artifact is that package together with its identifier and evidence of where it came from. A deployment installs the artifact in an environment or changes that environment's configuration. A release means the change has become available to its intended users. Enablement is a separate decision to make a specific feature available — for instance, flipping a flag or adding a group of users to a rollout. A release can happen at deployment time, but it can also happen later — for example, when the person responsible opens access through a feature flag.

So a team can deploy unfinished code to production, leave the new capability switched off and not release it to users. Likewise, a phone app can be ready for publication but wait for a decision about its release date in the app store. These examples show different transitions, not separate universal stages that every pipeline must have.

A delivery pipeline is the agreed path that changes take from version control to behaviour available to users. Its specific checks depend on the risk, the platform, security requirements and the way releases are made. A typical path might include a build and automated checks, publishing the artifact, verification in a pre-production environment, deployment, a test of the main scenario, monitoring the health of the system, permission to move to the next stage and a gradual widening of access.

The goal here is not to release changes as often as possible regardless of quality. DORA warns that increasing frequency without improving the process and the architecture can raise the failure rate and burn the team out. The value of continuous delivery comes from a release-ready state, small changes, fast feedback, reliable checks and monitoring of the outcome. DORA: Continuous delivery.

The presence of a step in an automation script does not mean it is a mandatory condition for merging or releasing. The condition has to be actually configured as a gate — a check that must pass before the next action can happen. In GitHub Actions, for example, you can do this by binding a job to an environment with approval rules or branch restrictions. The mere fact that a workflow refers to a named environment says nothing about which protections are actually configured in the repository. GitHub documentation on environments and protection rules.

Build one artifact and trace its path

DORA recommends using the same build for every environment: production then receives the very package the team verified earlier. The build must have an immutable identifier — for example, the full identifier of the code revision and a cryptographic digest of the package. A human-friendly tag such as latest is handy for finding things, but it can be moved to point to another package; on its own it does not prove which content was deployed.

The build and the deployment must also leave evidence that can be cross-checked: which revision the package was built from, with which workflow and tools, which checks passed, where it was deployed, who approved the transition and what outcome was observed. A signed provenance statement (an attestation) can cryptographically link an artifact to its workflow, repository, revision, environment and the event that triggered the build. It confirms where the package came from, but it does not guarantee that the package is safe or behaves correctly: the consumer must verify the signature and apply its own acceptance rules. GitHub explains what an attestation contains and where its limits lie.

A working artifact from a check run is not necessarily a long-term release store. Workflow artifacts are kept according to a retention policy, which the repository, organisation or enterprise may restrict; GitHub's current default settings keep them for 90 days. For a product that needs to restore an old release or pass an audit a year later, choose a package registry or another store with the required retention period and policy. GitHub: artifacts and their retention, configuring the retention period.

Environment configuration and the secrets a particular run needs can be kept separate: the address of an external service, a scaling parameter or an integration password. Such portability does not happen automatically, however. Some values are baked into the package at build time; changing the environment may then require a new build. The team must know which configuration applies at build time and which at deployment time, and check that it does not substitute for the identity of the artifact itself.

As an example, imagine a booking service that adds email confirmations. A build from revision 92ac1bf produced the image registry.example/booking@sha256:8a2d…; sha256 here denotes a digest computed with the SHA-256 algorithm (the 256-bit version of the Secure Hash Algorithm). The checks confirmed the main scenarios, and the workflow recorded links to the results and to the image's provenance. In the preview environment, the main scenario is tested on this image; in the pre-production (staging) environment, the same digest is deployed, but with a test database and a test email service. After successful verification, production receives this very same image, and the release owner separately approves the timing and authorises the gradual opening of the feature.

If production instead rebuilds the code from the same branch, the image is already different. It may contain new dependencies, different generated code or different configuration. The staging test is then no longer direct evidence for the production package. Controlled delivery promotes exactly the verified object between environments, rather than relying on version names that look alike.

Stage What to cross-check
Build Code revision, package identifier, results of required checks, available provenance attestation
Preview or staging The same package digest, configuration version, success of the critical scenario and known limits of the test integrations
Production The same digest, approval by the required people or rules, configuration version, time and environment of deployment
After deployment Key user scenarios, errors and latency, differences from the baseline, the decision to continue or stop

What exactly unlocks the next stage

A decision to promote a package can rest on several different kinds of evidence. Do not reduce them to a single "it's green" message.

  • An automated check confirms a specific property: for example, that the main booking scenario passes in the test environment.
  • A human review checks context that code cannot reliably determine: the scope of impact, who is responsible and the timing of deployment.
  • Environment protection technically holds back the production job until a set condition is met. In GitHub Actions this can be approval by a reviewer, a wait timer or a branch restriction. Whether these rules can be configured and are available depends on the type of repository and the GitHub pricing plan.
  • Post-deployment monitoring shows how the real system behaves for users. It should measure not just the fact that a process started, but the outcomes that matter for the product.
  • A feature release decision determines when and for whom the behaviour becomes available. It may require product readiness, compatibility with partners or staged access.

Define clearly who can stop the transition and on which signs: a spike in errors, increased response time, a failed critical scenario, an unavailable dependency, incorrect data or a mismatch in the package digest. Numerical thresholds must be set based on the normal level and the acceptable risk of this particular system. A universal percentage or number of minutes will not help here.

Manual approval is only one of the conditions; it does not replace verifying the artifact, the health of the system or data compatibility. Conversely, if the policy requires human permission, a successful automated check alone is not enough. In GitHub, an environment with required reviewers pauses the job until approval, and that environment's secrets become available to the job only after it. GitHub: deployment protection rules and environment secrets.

The platform can also give a workflow a short-lived token instead of a permanent cloud account key. GitHub Actions supports OpenID Connect (OIDC) — an open protocol through which the workflow requests an identity token and the cloud provider exchanges it for a temporary access token. The id-token: write permission on its own only allows a token to be requested from GitHub's provider; it does not by itself grant the right to change cloud resources. The owner of the cloud account must separately configure the trust relationship, check the claims about the repository and workflow, and grant the role only the operations it needs. GitHub: OpenID Connect in workflows and token exchange.

Compatibility of code, configuration and data

Old and new code often run at the same time: one instance is still serving users while another is being deployed; background jobs may run independently; a data migration may take longer than a job. The new version must coexist with all current consumers of the data until the team has retired them.

Consider a booking service that stored the start time in a field called start_time. The new version wants to move to a separate field, starts_at_utc, where UTC stands for Coordinated Universal Time. A safe transition might look like this:

  1. Expand the schema: add the new field while keeping the old one. Make sure the previous version still starts.
  2. Teach the code compatibility: the new code reads both the old and the new value and temporarily writes both. Old processes can continue to read the old field.
  3. Move and verify the data: fill the new field in batches, compare record counts and invariants; separately identify values that cannot be converted safely.
  4. Switch reads: after verification, make the code read the new field. Keep the old one for now, so that going back to a compatible version does not break anything.
  5. Contract the old schema: remove the obsolete field only after the team has confirmed that old versions, background consumers and the required recovery path no longer use it.

This is a teaching version of the "expand — migrate — contract" approach; the specific order and transactional behaviour depend on the storage system, the driver, the data volume and the load. The detailed risks of locking, batch backfills and backward compatibility are covered in database migrations.

Check configuration the same way as code. A partner's production address, account permissions, an enabled encryption format or the value of a feature flag can change the behaviour of the same package. Keep configuration in a versioned or audited place, assign an owner, restrict rights and never put secrets into images, logs or public artifacts. For a workflow that gains access to a cloud system, prefer a short-lived token with clear trust conditions, if the provider and the architecture support it.

A feature flag is not a time machine

A feature flag is a managed conditional switch in a program that determines whether a particular behaviour is available. It lets you deploy code separately from the moment users see the feature. The switch can limit a launch to the internal team or a share of the audience, and switching it off can hide the new path if its behaviour is bad. That is not the same as restoring the previous artifact.

A flag affects only the code that checks it. It does not undo a migration that has already been applied, does not restore deleted data, does not stop a background job that ignores the flag, and does not reverse changes made in a third-party system. If the new code wrote data in an incompatible format, simply switching the flag off may hide a screen but leave the system in an inconsistent state.

Do not leave transitional flags without an owner. Record who flips it and when, which value counts as the old behaviour, how each branch is tested, which users should see it and when the flag will be removed. Martin Fowler describes release, experiment and ops flags separately: these categories have different purposes and lifetimes, and mixing them without control creates long-lived code that nobody understands. Martin Fowler, Feature Toggles.

Worked case: the flag is off, but the migration is already incompatible

In a teaching delivery service, the team adds a new address type for a pickup point. It deploys code with the flag pickup-address-v2 switched off for all users. The same version already runs a migration: it changes the old address field in a way that old processes can no longer read. A background job starts writing the new format regardless of whether the flag hides the new user interface (UI) element.

The team sees errors in the old instance and switches pickup-address-v2 off. The form disappears for users, but the background writes continue, and the old code crashes on the new values. Simply redeploying the previous artifact does not help either: it cannot read the changed schema. The switch changed the visibility of the feature, but it did not change the data already written or restore the old compatibility rules.

Before deploying, the team should first have added the new field without breaking the old format, taught both versions to coexist, verified the records and the background processes, and only then switched the user-facing behaviour behind the flag. The old schema can be contracted only after the old code has been retired and the period during which a return to it might be needed has ended. If incompatible data has already appeared, the first step is to stop any more of it being created and to determine whether recovery is safe; depending on the consequences, a fix forward rather than a simple rollback may be needed. This is an invented example; it applies the compatibility principles covered in database migrations.

After deployment, check the outcome for the user

A job can finish with a success status because the file-copying command ran and the container started. That does not yet prove that a user can do what they need to. Post-deployment checks should verify:

  • process readiness: it has started, can see the resources it needs and can accept requests;
  • the main scenario: a new booking, payment or import completes with the expected result;
  • external dependencies: integrations are reachable with the production credentials and have the necessary permissions;
  • behaviour metrics: error rate, latency and the share of successful scenarios stay within acceptable limits;
  • data integrity: new records have the expected format, and background jobs do not create discrepancies.

Worked case: a green job and the wrong partner key

In a synthetic example, a booking service receives a new version without any errors. The process check passes: the health endpoint is reachable, the web page opens, the container is running. But the production integration with the payment partner has the wrong key: the partner rejects the requests, and new paid bookings are not confirmed.

Why did the green job not catch the problem? Its smoke test only checked that the application started, and the staging test used a different key and a test account. The team did not run a real, harmless end-to-end scenario in production and did not monitor the partner's success rate. So the automation proved the technical deployment, but not the correctness of the production configuration or the outcome for users.

The first safe action is to stop opening the feature further or halt the new rollout, and not to retry payments without control, so as not to create duplicates. Next, the owner of the integration checks the environment, the identity of the key, its permissions and its expiry, without printing the secret into a log. The team reviews the failed bookings and a plan for handling them safely, then replaces the secret in the appropriate store and runs the critical scenario with a safe test transaction or a controlled production check. If the error is only in configuration, rebuilding the code may be unnecessary; if the code mixes up environments, a new verified package will be needed. This is an invented example that shows why evidence about the state of a process differs from evidence that the product works.

Define the stopping threshold before the release, not during an argument in the middle of an incident. For example, the owner of payments may decide that any authorisation failure in the new flow stops its expansion. Another system will have its own normal fluctuations and its own criteria. After deployment, it is also worth checking the logs and metrics of external dependencies: one reachable web server does not prove that the whole chain succeeds.

Limit the impact and define the stop in advance

Not every unknown problem will show up in a test environment. That is why changes that can significantly affect users are often opened first to a limited group or a share of traffic. This is a gradual release: the new version can be compared with the previous one before it serves the whole product. For that comparison to be useful, choose signals that users will feel, distinguish the new and previous versions, and wait for sufficiently representative data. A metric that summarises an hour will not necessarily catch a problem in a five-minute stage.

Ways of rolling out a deployment are explored in more depth in the unit on deployment strategies. For now it is enough to remember that "deploy to a small number" without a separate check of that segment is not a canary rollout. A canary deploys a change to a limited part of the system for a defined time and compares its behaviour with a control group. If the new version's metrics get worse, the pipeline should stop the promotion and allow a rollback or call on the person responsible. Google's Site Reliability Engineering (SRE) practitioners recommend choosing measurable signals and taking into account the length of the stage, the volume of traffic, representativeness and shared dependencies. Google SRE Workbook, Canarying Releases.

Recovery has to be planned before the first production release: know the last verified version, who owns the decision, the command to stop and the consequences of running things again. A rollback returns the previous software artifact or configuration in order to restore the system's previous state; a forward fix changes the current state with a new change. These are different paths, not interchangeable buttons. If the change only replaced a software artifact, returning the previous one may be simple. If the schema, stored data, messages already sent to outside systems or access rights changed, returning the old binary will not undo those consequences. Keep checkpoints, confirm that the old version is compatible and test the recovery procedure in advance. More on this in rollback and recovery.

Primary-source case: YouTube's configuration and the lesson about canary rollouts

In a retrospective, Google SRE describes a global YouTube outage in 2016 caused by an error in a distributed memory caching system. The "Lessons learned" page summarises the outage as lasting fifteen minutes; the related lesson "Canary all changes" describes a cache configuration change that seriously disrupted the service for thirteen minutes. It states that a gradual rollout could have limited the global impact. The published source does not include the exact configuration file, every step of the change or a full timeline; the remark about the value of a canary rollout is an engineering conclusion by the authors, not measured evidence of how an alternative release would have gone. Google SRE: Lessons learned from two decades of SRE, section on canary releases.

This case shows that configuration can also be a release with a large blast radius. Passing a short job or having no syntax error does not prove that a new value is safe for all traffic. Roll such changes out gradually, compare the relevant metrics of the new and control groups, define the stopping criterion in advance and rehearse the recovery procedure. A separate Google SRE publication about this incident also warns that untested emergency recovery can make an outage worse instead of fixing it.

Where an agent can help

An agent can gather context for the person responsible for a release: what changed, which digest was built, which checks were run, which package is currently in staging, which metrics changed and what is missing for a decision. It can prepare release notes, an evidence table and a draft monitoring plan. The agent's conclusions must be checked against the primary checks, artifact identifiers and measurements, and must not be treated as a separate permission to deploy.

GitHub explicitly recommends using agentic workflows to prepare release notes, analyse changes and report on rollouts, while keeping the deployment itself in a deterministic workflow. GitHub: Continuous deployment. The Codex GitHub Action can run codex exec with explicitly set permissions and sandbox mode and pass the final message on as an output; OpenAI separately recommends restricting who can trigger the workflow, sanitising the text of a pull request (PR) and protecting the OPENAI_API_KEY secret. OpenAI: Codex GitHub Action. The Claude Code GitHub Action can be given actions: read to read continuous integration (CI) results, and the documentation also describes GitHub App write permissions for making changes and responding to requests. These rights depend on how the workflow and the installed app are configured, not on the mere fact that a model is being used. Anthropic: Claude Code GitHub Actions.

For a teaching release-assistant setup, read-only access to changes, checks and artifact metadata is enough. Do not give it production secrets, a cloud token or the right to bypass environment protection if its job is only to prepare a decision. If an organisation deliberately automates deployment, the workflow's rights must be limited to a specific repository, branch, environment, action and token lifetime; the production gate must remain explicit and be verified in the platform settings.

Summary

Continuous delivery keeps a change in a release-ready state; continuous deployment automates the transition under defined conditions. Trace the immutable identifier of one artifact from the build through the environments, keep configuration separate, and cross-check the package's provenance against the results of the checks. Before production, plan gates, compatibility between old and new code, migrations, post-deployment metrics and a real way to stop. A feature flag helps control the visibility of behaviour, but it does not roll data or the schema back. An agent can prepare evidence and a plan; the right to take a production action must be determined by a specific workflow policy and the minimum necessary permissions.

Sources and further reading

The materials were checked on 24 September 2026. Documentation pages that change without version numbers are cited as current on that date; before configuring a production system, re-check the features, pricing plans, permissions and behaviour of your chosen platform.

  • DORA, Continuous delivery — the distinction between continuous delivery and continuous deployment; CI, automated deployment, monitoring and managing data changes as separate capabilities.
  • DORA, Deployment automation — the same packages for different environments, environment configuration kept separate from the package, testing after deployment and preserving environment state.
  • GitHub Actions, Continuous deployment — CD trigger scenarios, environments, concurrency, OIDC and the role of agentic workflows in preparing a release.
  • GitHub Actions, Deployments and environments — required reviewers, timers, branch restrictions and access to environment secrets.
  • GitHub Actions, Workflow artifacts and artifact retention settings — the purpose of workflow artifacts and the limits of their retention period.
  • GitHub Actions, Artifact attestations and Using attestations — signed provenance data, the option of linking a package to a Software Bill of Materials (SBOM), generation permissions and verification by the consumer; an attestation on its own is not a guarantee of security.
  • GitHub Actions, OpenID Connect — exchanging a workflow's identity token for short-lived cloud tokens, and configuring trust.
  • Google SRE Workbook, Canarying Releases — the control group, choosing metrics and duration, and the decision to stop or promote a candidate.
  • Google SRE, Lessons learned from two decades of SRE — published lessons about the YouTube outage, the risk of configuration changes and testing recovery.
  • Google SRE, Configuration Design and Best Practices — gradual rollout of configuration, recovery and the limits of rollback.
  • Martin Fowler, Feature Toggles (aka Feature Flags) — categories of flags, separating deployment from release and maintaining transitional flags.
  • OpenAI, Codex GitHub Action — sandbox, workflow permissions, the agent's output messages and the security checklist.
  • Anthropic, Claude Code GitHub Actions — run modes, the ability to read continuous integration (CI) results, GitHub App permissions and protecting secrets.
  • Database migrations — more on schema compatibility, backfills, checks and choosing between rollback and a forward fix.