Quality and Security

Test Strategy: Building a Useful Set of Checks

How to build a test strategy that starts from product risks, places each piece of evidence where the risk actually lives, keeps the suite trustworthy and…

Chapter 49Quality and Security

Testing Fundamentals separated local checks, integration checks and end-to-end scenarios. The question now is a different one: which of them does a particular product actually need so that the team can change it with justified confidence?

Neither the number of test files nor the figure in a code coverage report answers that. A useful strategy starts from possible failures: what could go wrong, who would be harmed, and what evidence would catch the problem in time. Only then does it weigh the cost of each check, the precision of its signal and the moment by which the result must arrive while it can still influence the release decision.

Strategy, plan and test case: decisions at different scales

Three closely related terms answer different questions.

A test strategy sets out the standing principles of a team or product: which risks matter, where the main evidence comes from, how the levels of checking fit together, which requirements must pass before a release, and how the team maintains its test suite. It changes when the architecture, the product's risks or the way the team delivers change.

A test plan applies those principles to a specific release or change. It names the scope, objectives, owners, environments, data, sequence, schedule and exit criteria. The International Software Testing Qualifications Board (ISTQB) describes a test plan as a document of the objectives, resources and processes for a test project and its timeline. Not every change needs a separate formal document: a short plan can live in the task description or in the pull request (the request to merge a change), as long as the decisions it needs are clear to the team.

A test case describes one concrete check: preconditions, an action or input, and the expected result. For example: "a user who is not signed in opens a private invoice; the response contains none of the invoice data." A test case can be an automated test or a manual check; writing the case down does not, by itself, prove that the check was carried out.

So: the strategy defines principles, the plan defines the scope and order of a specific piece of work, and the test case defines a single instance of checking. This is a useful working distinction, not a requirement to keep three separate files. ISTQB also connects the test strategy to the plan: the plan shows how the team follows the strategy, or why it departs from it in a particular situation. ISTQB Certified Tester Foundation Level (CTFL) Syllabus v4.0.1, 2024.

A good strategy does not try to check everything in the same way. It answers at least four questions:

  1. Which errors would do the most harm?
  2. Which level or kind of check is capable of catching them?
  3. How fast and how reliable will that signal be?
  4. By what moment is the result needed so that the team can act safely?

The pyramid is a model of cost and scale, not a percentage quota

The test pyramid is a well-known metaphor for a set of checks. In its classic form it reminds us that narrow checks of a single function are usually cheaper and faster, so a team can often afford more of them, while checks that pass through a large part of the system cost more and depend more on the environment, so they are chosen selectively. Martin Fowler describes the pyramid as a way of thinking about a balanced suite and states an important premise outright: broad tests are usually slower and more brittle. If, in a particular system, high-level checks are fast, reliable and cheap to maintain, the need for a larger number of low-level checks may be different. Test Pyramid, Martin Fowler, 2012.

It does not follow that there is a universal ratio of unit, integration and end-to-end tests. Two teams can have suites of different shapes and control their risks equally well. What matters is knowing which assumptions lie behind the chosen shape, where the longest waits occur, and whether each more expensive test adds new confidence.

The level of a check and the way it interacts with an interface are not the same thing. A user interface (UI) is the screen or other means by which a person interacts with the product; a test of such an interface may check a small, local behaviour of one component. An end-to-end (E2E) test checks a path through several parts of the system, and it may talk to a service through its Application Programming Interface (API) without ever opening a browser. So the question "is this a UI test?" does not replace the questions "how much of the system does it cover?" and "which risk does it check?". The practical guide published on Fowler's site names a visible user interface and end-to-end scope as overlapping characteristics, not synonyms. The Practical Test Pyramid, Ham Vocke, 2018.

Other models put the emphasis elsewhere:

  • The Testing Trophy was proposed by Kent C. Dodds as a personal rule of thumb about the return on different kinds of checking in JavaScript applications. Above static analysis and local tests, it gives a wider share to integration checks. The author himself ties the image to his own experience as a front-end developer, not to a universal standard for every product. The Testing Trophy and Testing Classifications, Kent C. Dodds, 2021.
  • The Testing Honeycomb is how Spotify describes testing for a microservice architecture, where the risk often lies in the interaction of a service with its database and its API. Their recommendation is to concentrate on integration checks inside the service, keep a few tests of implementation details, and minimise tests that depend on other real systems. In their example, a small test suite takes a few seconds instead of the milliseconds that narrow tests need. That is a measurement of a whole suite in a specific architecture, not advice to abandon unit tests mechanically in an ordinary web application. Testing of Microservices, Spotify Engineering, 2018.

Google also recommends a solid base of narrow tests, not neglecting integration checks, and automating a small number of critical end-to-end paths. The author of the article notes separately that the depth required depends on the product's purpose, its target audience and the consequences of an error. This is a useful reference point, not a prescription with a fixed ratio. How Much Testing is Enough?, Google Testing Blog, 2021.

All of these images help a team think about cost, speed, scope and trust. None of them measures quality automatically. If a team argues only about whether its suite is "pyramid-shaped enough", it can miss the real question: which consequence is still left without evidence?

Start from the risk, then choose the check

A product risk is the possibility that a defect will harm a user, the business or another system. To compare risks, you can make a qualitative estimate of the likelihood of a failure and its impact. ISTQB describes these two dimensions as the basis of risk assessment; in a quantitative model they are sometimes multiplied, while a qualitative model presents them as a matrix. For day-to-day strategy, a consistently agreed scale of "low / medium / high" is often enough. A number such as 12 out of 25 does not become an accurate forecast just because someone calculated it. The goal is to notice important gaps and to check the risks with severe consequences first. ISTQB CTFL v4.0.1, section 5.2.

These categories most often deserve explicit attention:

What can go wrong What evidence can help
A business invariant is broken: a discount is applied twice, a slot is booked twice, a status moves into an impossible state Local checks of the rule and, if the rule depends on storage, an integration test of the storage's real behaviour
One customer's data reaches another, or a user performs a forbidden action Negative permission checks at the API or interface boundary, with authentication, the relevant role and data belonging to several owners
Our client and a service provider understand the format or meaning of a response differently A contract check on the consumer side and, where available, verification of that contract by the provider
A critical sequence of actions cannot be completed even though the individual components work A small set of end-to-end checks of the most important user journeys
A data migration corrupts records, or a restore does not bring back a working service A rehearsal of the migration on a controlled copy of the data, plus checks of the invariants, of backup restore and of the rollback plan

A single check does not always provide enough evidence for a serious risk. A local test might confirm that a fee formula is correct, for example, yet prove neither that the transaction does not store the wrong amount nor that a user cannot see someone else's invoice. In that case you need different, ideally independent, pieces of evidence: a test of the rule, a check of the storage, an access check at the boundary and a review of the critical decision. It is the risk, not a wish to duplicate everything at every level, that explains when several checks are justified.

Put the evidence where the risk lives

You choose the level according to the mechanism that can fail, not according to the prestige of a technology.

  • A simple rule, such as whether a cancellation is allowed before a booking starts, is convenient to check locally at the boundary values of time. That gives a fast, precise signal with no browser and no network.
  • Interaction with a database needs a check against a real test database of the same type, or a sufficiently close environment. A substituted repository cannot prove that the database itself enforces uniqueness constraints or transactions.
  • A contract between a consumer and a provider describes an external agreement: fields, types, required values, statuses and errors. A test with a predefined response checks how our own code behaves, but on its own it does not prove that the third-party provider still returns that response.
  • An end-to-end scenario checks that several parts of the product genuinely form the required user journey. It is worth reserving for a small number of journeys with the biggest consequences, rather than using it for every combination of data or every variant of one rule.

A mock (a test substitute for a dependency) is an object or service programmed to give an expected response or interaction. It is useful when you need to check, under control, how your component reacts to a network error, a timeout or a particular provider response. But if the mock repeats the same wrong contract that the code was built on, both tests can agree with each other and still both be wrong.

That is why mocks should be combined with evidence at the boundary: a contract check, the provider's official isolated test environment (a sandbox), or a controlled integration test. For such a test you decide in advance on the data, the clean-up and the consequences of running it again. Testing a third-party system's production environment is acceptable only with the direct permission of its owner and with agreed test data. In an ordinary strategy, use a sandbox or another test setup, not real payments, messages to customers or other people's live records.

A higher level should add new evidence. If a unit test already checks every variant of a pricing formula and dozens of slow browser tests repeat the same numbers, the browser checks add little. Keep the local variants of the formula where they catch errors quickly and clearly, and at the higher level check what the module cannot see: whether the correct amount travels the whole path and is shown to the user.

Example 1: a compact strategy for a booking service

Imagine a service where a user books a room for a given time interval. A calendar displayed incorrectly is an inconvenience; a double booking destroys trust; leaking a customer's address or confirming a booking that does not exist has more serious consequences. The team writes down the rules and the evidence for each risk:

Risk Where to check What the result must show
Time boundaries behave incorrectly: two intervals that only touch at their edges falsely conflict, or real overlaps are allowed Local test cases for the function that compares intervals; tested with the same time zone and the necessary daylight-saving transitions Two records are allowed for adjacent intervals; for an actual overlap, they are not
Two requests at the same moment both get the same free slot An integration check through the service and a test database, with two concurrent requests Storage holds one confirmed booking; the second request received a consistent conflict, not a hidden error
A user reads or cancels another person's booking A check at the API boundary with two accounts and data belonging to both owners Access by the owner is allowed; access by another user is refused according to product policy; the response does not reveal private data
The confirmation email is sent before the booking is saved, or in the wrong format A local check of how the event is built; an integration check that the event is recorded after the booking transaction commits; a contract check of the message against a test mail setup No email is sent for a rejected booking, and a confirmed one carries the right date, place and recipient
A visitor cannot complete the main journey in the product One end-to-end scenario: find a slot, confirm the booking, see the result The user sees the same confirmed booking that was stored in the system

Separate local test cases cover the time boundaries and the business rule quickly. The concurrency check launches two attempts that both start before either has committed its result; otherwise the test would accidentally check only sequential behaviour. The database must uphold the invariant even under simultaneous access; two mocks succeeding provides no such evidence.

For access, the team records exactly who is attempting the action and exactly which action. A test of an administrator opening their own booking does not cover the case "an ordinary user opens someone else's record". For the mail integration, the test setup does not send real messages to customers and does not need production credentials. If the test setup is unavailable, the strategy honestly marks that evidence as not yet obtained, instead of counting a mock as a real check of the provider.

The result is not "one test per layer" but a compact set in which each part covers a defined risk. Simple logic runs locally. Database integration, concurrent writes and permissions run in the check on each change. The contract and the critical end-to-end scenario must also pass reliably before a release; broader variants and data recovery run on an appropriate test setup before the release. If a team has a fast, isolated test environment and every scenario on it is stable, it can run more checks earlier. The schedule depends on the actual cost of each check and on how much the signal can be trusted.

Check the suite like a program: repeatability and cost

An automated test is code too. It can contain its own bug, a hidden shared state or a dependency on an unstable environment. A test is called flaky when, on the same revision of the code, it sometimes passes and sometimes fails without any corresponding change in the product. Typical sources are waiting a fixed amount of time instead of waiting for a specific state, dependence on execution order, shared records in a database, the local time zone, uncontrolled random input, the network, or incomplete clean-up after a previous scenario.

To make the signal repeatable, the team controls exactly the factors that could change the result:

  • it passes the clock or time zone into the code rather than relying on the machine's local time;
  • it sets the random seed explicitly, or records the input, when a failing scenario uses randomness;
  • it creates isolated data for every run and removes it even after a failure;
  • it waits for a visible condition or a readiness signal, not for an arbitrary "two seconds" pause;
  • it makes every test independent of whichever test happened to run before it, and checks behaviour when tests run in parallel;
  • it limits network access to test dependencies and records the environment version when that version can affect the result.

Rerunning a test helps diagnose the cause, but a later pass does not erase the first failure. In a report it is useful to distinguish a product defect, a test bug, an unstable environment and an inconclusive result. If a flaky test is temporarily put into quarantine, the team must show its status, name an owner, record the cause and the deadline for the fix, and keep other evidence for the critical risk. A test with no owner and no plan quickly turns into a requirement that has been quietly dropped.

Meta's experience shows the scale at which test stability becomes an engineering problem in its own right: the company described a probabilistic flakiness score (PFS) that it could apply to millions of tests, and used the score to find and repair flaky checks. Its metric and thresholds are Meta's tool for its own infrastructure, not a ready-made indicator that every team should copy. The general lesson is more modest: trust in a test suite requires observation and upkeep. Probabilistic flakiness: How do you test your tests?, Meta Engineering, 2020.

Cost, likewise, has to be measured rather than guessed. Shopify's report on continuous integration (CI) for its main monolith described, at the time, more than 170,000 tests and more than 41 hours to run the full suite on a single machine. The team measured the duration of every step and found that 31% of CI time went on preparing build agents and another 37% on preparing dependencies before the tests ran. In other words, in their context 68% of CI time was spent before any testing happened. They removed unnecessary rebuilding, ran a selected set of related checks, and added links between tests and fixture files (prepared sets of test data). The author reported that the share of builds that did not select the whole suite rose from 45% to more than 60%, and stability from 88% to 97%; several changes together cut the 95th percentile (p95, the time within which 95% of builds finished) from 45 to 18 minutes. This is Shopify's retrospective on a specific infrastructure. Selective execution is only as sound as the dependency map behind it: if a change is not captured in that map, the relevant test may simply never start. Keeping Developers Happy with a Fast CI, Shopify Engineering, 2021.

Example 2: hundreds of slow, flaky end-to-end tests

Imagine a mobile app for merchants. A suite of several hundred end-to-end tests opens screens, signs in to a test account, edits the catalogue and goes through checkout. The check often takes longer than writing the change itself. Some of the failures are real, but some appear only when a screen loads slowly or the simulator wakes up late. The team gets used to rerunning the suite, and then starts ignoring red results.

The first decision is neither to delete everything end-to-end nor to move the whole suite into a nightly run. The team classifies each expensive test by risk and by evidence:

  1. Repeated local rules. Dozens of browser scenarios check the same tax calculation, order status or discount format, changing only the numbers. The boundary values of the rule move into a narrow module test, and the module's connection to the database moves into a fast integration test. One end-to-end journey keeps the check that the final amount is actually visible before the order is confirmed.
  2. Real user journeys. The team keeps the scenarios that check the app's full interaction across sign-in, catalogue and order completion. With test payment details they create no real charge. Reading the user interface checks what the merchant can actually see and use, not merely that an invisible element exists in the internal component tree.
  3. The cause of flakiness. The team replaces blind pauses with waits for a specific visible state, isolates the test accounts, checks clean-up, and adds video or a step log. For each quarantined test it names an owner and a return date. Until a critical scenario returns, the release relies on other evidence that does not let the corresponding risk slip through unnoticed.
  4. Running order. Fast checks of rules and the most important integration scenarios run first. A small set of core end-to-end scenarios must pass before release. The full device matrix and broader variants run after every change or before a release, depending on their cost; but if the release happens sooner, the slow matrix is not evidence obtained before shipping.

A primary source from Shopify gives a practical example of exactly this kind of triage of flaky mobile end-to-end tests. In Shopify's largest mobile app these tests had started blocking more good changes than bad ones, and they were removed from the required checks on pull requests. Some tests could tap an element that existed in the internal component tree even when the merchant could not see it. Shopify built a stricter wrapper around the Appium driver, required a verification after every action, and used computer vision to find visible elements. A few weeks after the new system returned to the blocking checks, the company reported 98% stability, against 50% for the old test-framework API. This is Shopify's internal measurement across two mobile platforms, including background network errors and simulator start-up failures; it shows the outcome of their rebuild, not a guarantee for another app. How we raised mobile end-to-end test stability to 98%, Shopify Engineering, 2026.

After the triage, the team can delete a duplicated scenario if the risk genuinely has cheaper evidence and the main journey is still checked. The mere fact that a test is slow or occasionally flaky is not enough: that is a signal of a problem in the test, the environment or the product's design, and the problem needs to be understood. Deletion is appropriate once the team has named which risk the test covered, where it is now checked, when the new evidence runs and passes, and who is responsible for bringing back a critical check. For this hypothetical team, the result shows up in before-and-after measurements: how long it takes to get a reliable result, what share of red runs revealed a real bug, and whether the critical journeys pass before release. If the duration dropped but the real checkout remains only in the nightly suite, after the team has already shipped its changes, the strategy has not closed the risk. None of this splitting is needed if all the end-to-end checks are already fast, stable and each adds unique evidence.

Code coverage is not risk coverage

Line coverage shows which lines of code were executed during the tests. Branch coverage shows whether the different outcomes of conditions and loops were exercised. Requirements coverage shows whether each important requirement has a linked test case and a visible result. Risk coverage shows whether there is enough evidence against a specific possible harm. These measures can overlap, but none of them replaces another.

If a test ran the line return invoice, it proved that the line executed, not that the user had the right to read the invoice. Even 100% line coverage means only that every line was executed at least once. The strength of a test's assertion is determined by which conditions it checked and which wrong result it would reject. The ISTQB syllabus warns explicitly that executing a statement does not guarantee that a defect will be detected, and that full branch and statement coverage still does not reveal every error that depends on a particular path or particular data. ISTQB CTFL v4.0.1, section 4.3.

Property-based testing states a rule that must stay true across a whole described range of data, and checks it against many generated examples. For instance: for any permitted list of intervals, sorting does not change which items are in the list; for any valid amount, a percentage discount does not increase it. The Hypothesis library explains the approach as a property that must hold for all values in a chosen domain, combined with input generation that also finds non-obvious edge cases. It is a way to widen the search for examples, not a proof over all possible values: the author still chooses both the domain and the property itself. Hypothesis Documentation, version 6.168.1.

Mutation testing changes the code with small, controlled edits (for example, replacing > with >=) and reruns the tests. If the suite still passes, the change may have exposed a gap in the checks or a redundant branch of code. Cosmic Ray for Python describes the approach as a way to check whether the tests really react to a change in behaviour. But a "surviving" mutation needs to be examined: it may reveal a weak assertion, irrelevant code, or a change that makes no actual difference in that context. Large numbers of mutations are also expensive, so these checks are usually run selectively on important rules rather than made a mandatory gate for every project. Cosmic Ray Documentation.

A coverage figure is useful when it answers a specific question: which code is still never executed; which branches are untested; whether every critical requirement has a test case; whether we test failure as well as success. If a team sets a goal of "90% of lines" without tying it to behaviour, the number can encourage tests that execute code but assert nothing important.

Example 3: high line coverage misses a missing authorisation check

Imagine an invoicing service API. The team sees 97% line coverage and believes that reading invoices is protected. The existing test signs in as the invoice owner, sends GET /invoices/83 (a request to read the record with identifier 83) and receives the correct amount. The successful scenario goes through the route, the storage query, the response formatting and the reply to the client, so most of the lines genuinely did execute.

But the implementation forgot to check the record's owner. The storage query looks up the invoice by its identifier alone, not by the pair "invoice identifier plus account identifier". The owner's test stays green even with this bug present. A high percentage of lines does not answer the question "can another customer read this invoice?"

The team goes back to the risk and adds specific test cases:

Condition Expected behaviour
The invoice owner signs in and reads their own record Receives the fields that the product policy allows
An unauthenticated visitor requests the invoice Receives an authentication refusal; the response contains no invoice data at all
An authenticated owner of a different account requests invoice 83 Receives a refusal or "not found", depending on the policy about hiding whether the record exists; no private fields are returned

The key check goes through the real authentication mechanism and a test store, instead of mocking both the user and the access check at once. The test then checks the linked pair "who is asking / which record they are asking for". It should ideally be written so that it turns red if the owner check is removed. An additional independent check of the role matrix, or a review of the access rules, is appropriate because the cost of a leak is high. At the same time, the team does not copy every role variant into dozens of slow end-to-end scenarios when an integration test at the boundary catches these errors quickly and reliably.

After the change, the team can report two different facts: "97% of lines executed" and "verified that another customer receives no invoice data". The second is a concrete claim about a risk. The first is a useful but limited description of code execution. Neither should be passed off as the other.

Spread checks over time without hiding missing evidence

A strategy defines not only the kind of test but also when it must return a result. The benefit of a fast check disappears if it does not influence the decision before the change reaches users.

When What usually gives the best signal
Locally, during development Fast checks of the rules that changed, static analysis, and those integration tests that are easy to run in the same environment
On every change or pull request Regressions in changed code, business invariants, access permissions, data integrations and required contract checks; a small critical set of end-to-end journeys
After merge, or in an extended cycle Fuller combinations of environments, compatibility, concurrency, a larger set of integration checks and long-running load scenarios
Before release Checks of the release candidate, of migrations and recovery, of external providers' permitted sandboxes, and of those critical journeys that are not part of the fast checks
After release Monitoring of real metrics, errors and availability; isolated synthetic checks only with controlled data and permitted side effects

This is a guideline, not a mandatory number of pipeline stages. If the whole suite is fast and stable enough, the team can run all of it on every change. If some checks are expensive, they can be spread out in time, but then the team must know exactly which risk is left without a signal before release. Selective execution based on a map of changed files can also speed up the cycle; how safe it is depends on the quality of that map and on how it handles configuration, fixtures and shared components. A check that the selection algorithm skipped is not a check that passed.

DORA (DevOps Research and Assessment) describes continuous testing as checks that run throughout delivery, not as a separate phase at the end. Its materials link a short feedback loop to the ability to notice and fix problems sooner. The "under ten minutes" figure in DORA's guidance is best read as a target for fast feedback to the team, not as a universal threshold beyond which quality is guaranteed. DORA also recommends reviewing the suite regularly: tests should catch defects and stay understandable, while their complexity and cost stay under control. Test Automation, Continuous Delivery, checked 24 September 2026.

In the excerpt of Accelerate that was read for this unit, Nicole Forsgren, Jez Humble and Gene Kim connect shorter lead time for changes with faster feedback, course correction and fixing of problems. The practical conclusion for a test strategy is that the team shares responsibility for quality and builds checks into the work so that the necessary evidence reaches the release decision. Developers maintain the automated suite, and quality control continues throughout development and delivery. This is a conclusion about how the feedback flow is organised, not a prescription for the number of tests or the share of each level. Authors' excerpt from Accelerate, publisher's page.

Speeding up does not mean deleting "slow" checks that protect an important risk. First measure the full time to a trusted result: environment preparation, dependency downloads, the queue for a runner, the tests themselves and recovery from flakiness. Shopify's story shows that a large part of the delay in their pipeline came from preparatory operations, not from running tests; optimising rebuilds and the environment produced the gain without weakening any test assertions. In this sense fast feedback is an investment in the development process itself, not a compromise on the evidence you need.

An agent changes the implementation, not the definition of success

The criteria of a test strategy should be the same whether the change is written by a person, by Codex or by Claude Code. The executor may propose a new test or notice an outdated one, but it must not weaken the regression suite automatically just because a test fails.

A request to an agent should tie the change to a check: what exactly must become true, which negative scenario matters, which commands produce the evidence, and what to report if the evidence is not available. OpenAI's guide to working with goals in Codex recommends stating the outcome, the method of verification and the constraints explicitly, and names tests and other verifiable artefacts among the possible sources of evidence. Anthropic's recommendations for Claude Code likewise advise giving the agent a check it can run and showing the command's output instead of an unsupported claim of success. These are recommendations for specific products, not a guarantee that the agent chose the right test or that the test covers every risk. Using Goals in Codex, OpenAI; Best practices for Claude Code, Anthropic.

If an agent proposes disabling a flaky test, ask it to state: which risk the test was checking; whether there is cheaper or independent evidence; what will remain unchecked before release; and who will bring the test back or fix its cause, and when. If an agent adds a test, check that it fails precisely on the wrong behaviour, and does not merely increase coverage. An agent can generate many similar tests quickly, but the volume of test code does not, on its own, increase confidence.

Summary

A test strategy links the product's risks to evidence that the team can obtain by the moment it is needed. The pyramid, the trophy and the honeycomb help people talk about the trade-off between speed, scope, cost and precision of signal; they do not set a mandatory proportion of tests. The level of a check should match the mechanism of the possible defect: simple rules are often best checked locally, data interactions in a test environment with a real database, external agreements with contracts, and the key user journey with a small number of genuine end-to-end scenarios.

Line or branch coverage shows what was executed, but on its own it does not confirm that a business requirement or an access permission is protected. A test's stability, its owner, control over its data and the timing of its result matter as much as its name. If the suite is slowing the team down, measure the cause, separate duplicates from real journeys, and keep evidence for every significant risk.

Sources and further reading

All sources listed were accessed on 24 September 2026.