Delivery and Operations

Continuous Integration: How a Change Proves It Is Ready to Merge

How continuous integration turns frequent, small changes into a trustworthy signal of readiness to merge, and how to read, secure and debug that signal…

Chapter 56Delivery and Operations

Testing fundamentals established what individual tests can prove, and the test pyramid and test strategy showed how to combine levels of checking according to risk and time. This unit moves on to the automated process of regularly checking small changes before they are merged. That process is continuous integration (CI). Application security basics and AI agent security covered the security of applications and of agents respectively; here we focus on protecting automated workflows. How to promote a verified build between environments is explained in continuous delivery and deployment.

CI does not mean "the repository has a configuration file". It is the habit of frequently merging small changes into a shared main branch and quickly checking whether the shared code still works. The automation file describes only part of that practice. The signal also depends on which revision was checked, which actions were actually performed, how the team interprets their results and which checks block a merge.

Why a team integrates changes often

The DevOps Research and Assessment (DORA) research programme describes CI as merging small changes frequently, with fast feedback from builds and tests. If a short check breaks soon after a few lines of code were written, it is usually easier to find the cause than in a large batch of changes. A long-lived branch postpones integration: the longer the code drifts away from the shared version, the more mutual assumptions and conflicts the team has to reconcile. In the Accelerate State of DevOps 2021 report, DORA describes a build and automated tests after every commit, with a signal within a few minutes, and examines how that practice relates to team performance. These are research findings and recommendations, not a guarantee that a CI tool on its own will improve every product in the same way (DORA guidance, DORA 2021 report).

For a teaching example, take a booking service. The team can merge small changes daily: the slot-availability check in one change, the cancellation rule in another, logging in a third. After each pull request (PR), checks run that can quickly detect the risk of those particular changes; a slow end-to-end payment scenario can run separately or in parallel, if that gives the right balance of speed and cost.

DORA's recommendation is to shorten the time from a change to a useful signal, not to chase the largest possible number of green check marks. A forty-minute check can force people to wait for a long time or push the team to look for a workaround; a set of checks that is too narrow, on the other hand, stops catching important errors. Which checks run on every change is decided by the risk, the limits of what they prove and the acceptable feedback time.

What a workflow is made of

The terms below refer to examples from GitHub Actions. Other automation platforms use similar concepts, but names, limits and exact semantics must be checked in the documentation of the specific product.

Term Plain meaning What it is for
Event An action or schedule that triggers automation For example, a new pull request, an update to its branch or a nightly schedule
Workflow A configuration file in YAML format (YAML Ain’t Markup Language, a data serialisation format), usually in .github/workflows/ Describes the triggers, permissions and jobs of the automation
Job A group of steps assigned to a runner Lets checks run in parallel or defines their dependencies
Step One command or reusable action inside a job For example, fetch files, install dependencies, run tests
Runner The machine that executes a job Can be a short-lived cloud virtual machine (VM) or the team's own server
Environment Within CI, a set of execution conditions; in GitHub Actions it is also a separate mechanism for controlling access to a deployment target Do not confuse test variables and services with the publishing or deployment stage
Action A ready-made, reusable step or set of steps Helps, for example, to check out code or set up a runtime
Workflow artifact A file saved after a job, or for handing over to another job For example, a test report, a log or a build
Cache Stored dependencies or intermediate data that speed up the next run Cuts repeated work, but must never be a required source of correctness

In GitHub Actions, the workflow file describes events, jobs and steps; the version of the workflow that runs is tied to the revision or reference that triggered the event. A runner executes jobs. A GitHub-hosted runner is usually a fresh machine allocated per job and cleaned up when the job finishes, while the steps of one job share a file system. A self-hosted runner can keep state between runs, so keeping its environment clean becomes the team's own responsibility (GitHub: Workflows, Workflow syntax, GitHub-hosted runners).

In this unit, "workflow artifact" means a file saved after a job — a log or a test report, for example. In continuous delivery and deployment, an artifact is the package that the team verifies and promotes between environments.

Which code exactly was checked

A green result means something only together with the revision of code it checked. On GitHub, a SHA (named after the Secure Hash Algorithm, a family of cryptographic hash functions) is the short identifier of a commit. To evaluate a check on a pull request, you need at least its number, the exact SHA, the event, the ref — the branch or other reference — and the result of every required job.

In the standard pull_request scenario, GitHub provides a merge revision that combines the current base branch with the PR branch. So the check usually answers the question "does this proposal work together with the current base?", not merely "does the author's branch work on its own?". At the same time, the check must cover the latest relevant SHA: a green status on an old commit does not check changes added later. If the repository uses a merge queue, the workflow must also be connected to the merge_group event; otherwise the required check may never appear for the combination the queue is about to merge. GitHub describes the SHA and ref for the pull_request and merge_group events.

The workflow fetches the required snapshot of the code, restores dependencies and runs the checks the team has chosen as its readiness signal. Often the fast static checks run first, then the tests, and then the product is built. There is no universal mandatory order, though: a test may require a built binary, and a scanner may need an already built dependency. The sequence follows from the dependencies between jobs and the measured cost of feedback.

What a check result means

Do not reduce every state to "green or red". Reports can show a successful, failed, cancelled, timed-out or skipped result, or no run at all. Each answers a different question.

State What you can already say Next step
Success These steps finished without signalling an error Look at what exactly they did, on which SHA and with what limits
Failure A command returned a failure code, a test assertion did not match or another step ended in an error Read the first relevant error and match it against the code, the test and the environment
Timed out The job did not finish within the set interval Work out whether it is a slow product, a hung process, infrastructure or a limit that is too tight
Cancelled The run was stopped before it finished Find out whether a new run replaced it, or whether the cancellation hid a missing check
Skipped A condition prevented a step or job from running Check that condition and the merge rules; a skipped workflow and a skipped job are not the same thing
Not run / no check There is no check for the required event or revision Make sure the trigger matches the event and branch; do not call this a success

In GitHub Actions, a job skipped by an if condition can count as successful for a required check. If, however, the workflow did not start because of a branch or path filter, or a skip instruction in the commit message, the required status can remain pending. These are different cases: check the job's condition and the reason the run is missing (GitHub: status checks, skipped workflow runs).

Naming a job test does not make it mandatory: required checks are defined in branch rules or a ruleset, where you can also specify the expected GitHub App that must report them (rules for required status checks).

A green run is evidence about specific checks, not a certificate that nothing is wrong. A build can succeed even though the test suite never checked an important product condition. A test may use a fake service instead of the real provider. A scanner finds only the classes of problems it is designed to find. The limits of what was proved must be described next to the result, as testing fundamentals recommends.

Repeatability, caches and artifacts

To tell a change in the product apart from a difference in the environment, pin down the reproducible ingredients: the language or runtime version, the build toolchain, the lockfile, run parameters, fixtures, time zone, the source of test data and the external services required. Variable network responses, the current time, random test order and shared resources can change the result even on an unchanged commit. Pinning is not always possible for a complete real-world environment, but in that case the influence should be named and contained.

In a Node.js project, the npm package manager runs npm ci to install dependencies exactly as recorded in the lockfile; the command fails if the lockfile does not agree with the package description, and it does not rewrite the lockfile itself. This is an example of npm's behaviour specifically, not a universal rule for all package managers. GitHub's official guide to building and testing Node.js shows the use of a lockfile and npm ci in a workflow.

A cache should only speed work up: without it, the team must still be able to fetch dependencies again or regenerate intermediate files. A cache key usually takes into account the operating system and a hash of the lockfile. If someone changes the dependency configuration, a different key should lead to the matching set of data. Do not put secrets or trusted data from a privileged environment into the cache: data restored from a cache should be checked like input from a previous run. GitHub explains the purpose and limits of caching separately from storing workflow artifacts.

A workflow artifact preserves a job's result after it finishes, whereas a cache speeds up the next run. A saved file does not, on its own, confirm the origin of a build or its fitness for production. That requires separately configured evidence, such as an attestation, with its own permissions and verification (GitHub: Workflow artifacts).

Parallelism, cancellation and retries

A run matrix checks several runtime versions, operating systems or configurations — for example, the current and the minimum supported versions of Node.js. It increases coverage, time and cost. If every variant of the matrix is required for merging, its result and its success criterion must be clear in the protection rules. Jobs can run in parallel if they do not depend on each other through needs.

A concurrency rule can cancel an earlier run when a new commit has made it obsolete. This saves resources, but the team must still get a result for the latest SHA. A time limit is useful as protection against hanging; it does not cure a slow test and does not by itself explain the cause of a timeout.

A successful retry after a first run failed points to instability; it does not cancel the earlier result. Keep both logs, the SHA and the differences in environment: retrying until a chance success hides defects. Such unstable results were discussed separately in testing fundamentals.

Security boundaries in a workflow

Any workflow that runs code from a PR must treat that code as potentially untrusted. Set minimal permissions for GITHUB_TOKEN; an ordinary lint or test job usually needs only read access to the repository. Do not pass deployment keys, publishing keys or production database credentials into an ordinary PR run. For third-party Actions, use the full commit SHA after checking its origin: GitHub calls the full SHA the only immutable reference to a specific version of an action. Do not rely on a moving tag such as @v3 alone if the immutability of the code matters. GitHub: Secure use reference.

Do not combine obtaining a privileged token with running code from a fork PR. The pull_request_target event uses the base repository's context with higher privileges; checking out and running the changed PR code in such a workflow can give an attacker a way to use those privileges. An ordinary PR workflow for tests usually does not need this trigger. Artifacts from a workflow that ran fork code, and its text outputs, must also be treated as untrusted data. GitHub: secure use of pull_request_target.

A commit message or PR description does not become a safe instruction for an agent just because it reached a workflow. If an agent receives such texts, restrict how the job is launched, separate secrets from untrusted input and grant minimal permissions. This continues the threat model from AI agent security.

How an agent should analyse a failed run

An agent can speed up the initial review, but its first explanation is only a hypothesis. Start with the facts: the PR and exact SHA, the workflow, the runner, the first failed step and its exit code, the full relevant log, the command and tool versions, the test data and any retries. Then separate at least four explanations: a product regression, a mistake in the test or its expectation, a difference in environment, or unavailable infrastructure.

Change the smallest area that matches the confirmed cause. If a scenario depended on data from a neighbouring test, isolate the state or make the initial data explicit; do not disable the test condition just to get a green indicator. If the failure is in the product, keep the first log and add a check that demonstrates the same defect. If the agent is asked to rerun the check until it happens to pass, it should explain that a retry did not refute the first failure, and it should not report a fix without a confirmed cause.

According to OpenAI's official documentation, the Codex GitHub Action (openai/codex-action@v1) runs Codex as a GitHub Actions step: it can summarise a PR and save a result file. The final-message text does not automatically become a required status; the condition for such a status must be configured explicitly in the workflow. Anthropic describes claude-code-action@v1 and the actions: read permission for reading a PR's CI results. In both cases, the result required for merging is defined by the GitHub configuration, and access to secrets and triggers by the boundaries of the workflow and repository. OpenAI: Codex GitHub Action, Anthropic: Claude Code GitHub Actions.

Worked example: checking a change to the booking service

Situation. The team is adding a check that two customers cannot confirm the last free slot at the same time. The repository has a .node-version file with an exact Node.js version, consistent package.json and package-lock.json files, and the lint, test and build commands are defined in package.json. The team wants to check every PR to main, every push to main, and the revision that the GitHub merge queue is about to merge.

Decision and action. The workflow gets read-only access to the contents, runs one job for each required revision, finishes within a set time limit and cancels an obsolete run of the same PR. The steps and their order in this teaching example are as follows; a real repository should specify its own verified commands.

name: booking-ci

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  merge_group:
    types: [checks_requested]

permissions:
  contents: read

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

jobs:
  verify:
    name: booking-verify
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - name: Check out the event revision
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

      - name: Install the Node.js version from the repository
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version-file: .node-version
          cache: npm

      - name: Restore exact dependencies from the lockfile
        run: npm ci

      - name: Check code style and errors
        run: npm run lint

      - name: Run the tests
        run: npm test

      - name: Build the application
        run: npm run build

Observed result. The PR page shows booking-verify and the SHA it checked. A ruleset can require exactly this job; the YAML file on its own does not create a merge gate. For the merge queue, a separate build is triggered through merge_group. Success means that the specified lint, tests and build passed on this revision and this runner.

Why this helps. The team gets an early signal with minimal permissions, and the full commit SHAs pin the code of third-party actions. The npm cache speeds up repeat runs; npm ci restores dependencies from the lockfile.

Limits. The workflow does not prove that npm test checks the race for the last slot. Nor does it check the payment provider, production secrets or deployment. Make sure the run belongs to the latest SHA and that the ruleset requires the correct job. In the example, the Actions are pinned to full SHAs; their releases were checked on 26 September 2026 — checkout v7.0.1 and setup-node v7.0.0. Before starting a new project, check the current versions and update them in a controlled way.

Worked example: the green status belongs to an old SHA

Situation. A PR with the new booking check had a green booking-verify on commit a17c9e2. The author then added b38ad41, which changed the transaction. In the list of runs, the last completed workflow still belongs to a17c9e2, and there is no check for b38ad41. For a while the PR page shows the earlier success, and the team does not know whether it is safe to merge.

Decision and action. The team compares the PR's list of commits with the SHA of the check and the triggering event: the old result is evidence only about a17c9e2. For the new commit they check the pull_request and synchronize triggers, any branches or paths filters, the state of the workflow on the Actions tab and the required-check entry in the repository rules. If a job was skipped because of an if condition, they find out whether that is the intended behaviour for that particular job. If the whole workflow did not start because of a path filter or a skip instruction, the required status can remain pending; these are different causes and need different fixes.

Result. The team fixes the trigger or filter and runs the workflow for b38ad41. That new run, not the old green status, becomes the evidence for the changed transaction. Until it finishes, the PR waits for the check. If after the fix the status does not appear or stays pending, they check its source and the required-check rules, rather than switching off branch protection. A check is tied to a revision, not simply to the name of a PR or workflow.

Limits. A new green run still does not confirm properties the workflow does not check. The cause of a missing check may also lie in the event configuration, a permission, a runner limit or a temporarily unavailable service; the mere absence of a result cannot be declared a product defect.

Worked example: an integration test that sometimes fails

Situation. A booking-service test creates two free hours in a shared test database and expects to see both. On the same runner, another parallel test clears the table's records before its own scenario. One run shows one hour, the next shows two. A green local run on the same source files does not prove that the intermittent CI failure was harmless.

Decision and action. The team keeps the first failure, the SHA, the logs of Structured Query Language (SQL) queries to the database and the order of the parallel jobs. Comparing a solo run with a run of the same group of tests shows that the test passes reliably on its own but sometimes loses a record when run in parallel. Tracing confirms that clearing the shared table affects the neighbouring scenario's data, and the product function reads only the set that remains. The team isolates each test with its own schema, its own customer space (tenant) or a unique identifier, and ensures that clean-up changes only that test's own state. The assertion "two slots are available" stays the same; a parallel run with the same order checks whether the isolation worked. The timeout is not increased at random, and the test is not disabled.

Result. If, after isolation, the test consistently sees both records and the parallel scenario no longer erases them, the observed instability was caused by shared test state. If the service still returns one slot from a database that holds two, the product query or transaction must be investigated as a possible regression. The same external symptom can have different causes; a repeat green run on its own proves nothing. The test still checks the booking rule, but it no longer depends on the order of other scenarios.

Limits. This is a teaching situation, not a report of a real incident. Isolating test data does not remove a bug in the live system, where simultaneous customers really do compete for a slot; that product risk needs a separate check of the transaction. If CI uses an unstable external database, extra logs and observation will be needed to separate a network error from a product error.

Real case: pandas and strict expected failures

Situation. In November 2018 the pandas project was preparing to make xfail_strict = True the default for its tests. In pytest, the xfail marker means a test is expected to fail; if a marked test suddenly passes, strict mode turns that unexpected pass (XPASS) into an error. In a comment on PR #23721, a maintainer noticed that on CI with Python 3.7 some tests were passing, even though a local check on Python 3.6 did not show this. The difference in environments changed the result, so the PR's first failure did not by itself prove a defect in new product code.

Decision and action. The author kept strict behaviour overall, so that an unexpected fix of an old defect would also become visible. After reviewing the signals from different CI runs, the team marked one known-unstable test as strict=False and added another to the list of expected failures. The PR updated many test markers and settings rather than switching checks off entirely. The discussion on PR #23721 shows separate test and lint runs and the difference between Python versions; issue #23057 explains the intention to make xfail strict and to sort out the tests that sometimes passed.

Result. The PR, with eight commits, was merged on 3 December 2018. CI revealed an environmental difference that was not visible from a local run on a single Python version. Refining the expected state of specific tests preserved the unexpected-pass signal instead of silencing all such cases.

Limits. This is a historical example from Travis CI, not a modern GitHub Actions template. It shows the value of a matrix and of interpreting statuses precisely, but it does not prove that any failure between runtime versions can be fixed by changing a marker: pandas's supported versions and testing rules may since have changed.

Summary

CI makes regular checking of changes observable: an event triggers a workflow, jobs run on a runner, steps issue commands, and results are tied to an exact SHA. Trust a status only after you have checked which code, which conditions and which exact checks it represents. A cache speeds up repeated work, an artifact preserves a result, and a required check comes from a specific branch rule or ruleset. When a run fails, the first log and the environment hold more important data than a chance green retry. If the cause is not yet known, record the uncertainty instead of disabling the test or the protection.

Sources and further reading

Access to the sources was checked on 26 September 2026. Documentation pages without a version number are moving targets; the Actions pins in the example correspond to the releases listed below.