Planning Checks Before Writing Code

Suppose the work already has a boundary from decomposition and shaping, a chosen vertical slice, mapped dependencies and an implementation sequence. Before the first line of code changes, there is a different question to answer: which observations will tell us that the required behaviour has appeared and that important neighbouring scenarios have not broken?

A test plan is an agreement about risks, scenarios, expected results, environment and evidence. It is not a wish list of libraries, a promise to automate everything, or a report on work already done. Before implementation it holds expectations and ways of checking them; after the checks run, the actual results and their limitations are added to it.

This unit continues the planning thread that runs from acceptance criteria through implementation plans. The detailed mechanics of different kinds of tests are covered in testing fundamentals and the test pyramid and test strategy.

The booking and import examples below are for teaching. Service names, records, rules and results are invented. Where an example adopts a specific product rule, it is marked as a condition of the example, not as a universal requirement.

Three different things: criterion, test and evidence

An acceptance criterion states what the behaviour or result must be. It answers: "What must be true for the user, the data or the operator?"

A test describes the action or observation we use to check the criterion. It answers: "What exactly will we do, in what state, and at which level of the system?"

Evidence is the actual result of running the test: what was observed, on which version, in which environment, and when. It answers: "What happened this time?" Before a run you can plan the evidence, but you cannot already have the result of a run that has not happened.

Level	Example
Criterion	If a slot is already taken by another customer, the reschedule does not happen and the customer's existing booking stays unchanged.
Test	Create booking A for a customer, occupy the target slot with booking B, and try to move A into that slot.
Test expectation	The server returns a defined refusal; A keeps its original time; B remains the holder of the target slot; no success notification is sent.
Evidence after the run	The result of running that specific test, the actual state of the records before and after, the build or revision identifier, the environment settings and a timestamp.

These records cannot stand in for one another. The text "test passed", without a criterion, does not explain which promise was checked. A criterion without a test may stay unchecked. A list of tests without results does not prove they were run. And a green result for a different code revision does not prove how the current candidate behaves.

In the same way, do not merge the criterion with the mechanism you expect. "The user will see the current booking time" is behaviour. "The screen will call refreshBooking()" is a hypothesis about implementation. Before the code is written the mechanism may still change; the expectation for the user should stay fixed.

A test as an observable experiment

A useful test can be reproduced. It specifies:

1. the starting state;
2. the action or event;
3. the environment conditions that affect the outcome;
4. the expected observation;
5. how the actual observation will be recorded;
6. cleaning up the data or restoring the environment, if needed.

A vague "check that everything works" defines neither an action nor a correct result. Rewrite it: "As a user with the customer role, open the booking confirmation for the test record's identifier; the page must show the same time and time zone as the data source returns; attach the screenshot and the response from the test application programming interface (API) to the report." Where private data must not be shown, the evidence should contain a synthetic record or a safe generalisation, not hidden personal details.

From problem to coverage

A plan starts from the problem and the promised result, not from whichever checks are easy to write. It connects to the earlier planning work like this:

• decomposition and shaping sets the boundary of the change and the unknowns that need to be resolved;
• vertical slices define a useful end-to-end slice and the behaviour available to the user;
• dependency graphs show the dependencies, the shared state and the places where work happens concurrently;
• implementation plans set out the implementation steps, the transitions between stages and the evidence for each;
• this unit turns every promise and every significant risk into a planned way of observing it.

A plan should make gaps visible. If a criterion has not been agreed yet, record the open question and the owner of the decision. Do not invent the answer yourself and then turn it into a test as if it had always been a requirement. If a behaviour cannot be checked because the environment is unavailable, name that limitation separately.

Scope of the checks

Start with what is changing, then check the surroundings of the change:

• new or changed behaviour;
• neighbouring scenarios that use the same function or data;
• permissions, roles and boundaries between users;
• the contract with an external service;
• database records, migration and old clients;
• events, retries, queues and parallel requests;
• messages, logs, metrics and support instructions;
• speed, interface accessibility and recovery after an error;
• what is deliberately not being checked, and why.

The scope should be proportional to the consequences and to the likelihood of failure. A bug that exposes someone else's data or loses someone's only booking deserves more attention than a change to decorative text. But the size of a change does not by itself guarantee low risk: two lines of configuration can change how every device image is compiled.

A handy rule of thumb is consequence × likelihood. It is not an exact formula and not an excuse to invent decimal scores. It helps you compare "what happens if this breaks?" with "how realistic is this failure path?". Consider separately whether the failure can be noticed and the state recovered. An irreversible action may demand stronger evidence before launch.

Coverage matrix

A starting working format:

ID	Criterion or risk	Scenario and starting state	Level and data	Expected result	Planned evidence	Owner
T-01	Criterion C-1	The action under normal conditions	Component level; a record with synthetic data	The described behaviour and state	Result of the automated run, revision	Author of the change
T-02	Risk: external timeout	The remote action may have completed but the response was lost	Integration level, with a controlled stub service	No duplicate and no false success	Event log and the final state of the record	Integration owner
T-03	Criterion C-2	A person uses the changed screen	Manual exploration in the test environment	The user finds and understands the result	Observation notes without personal data	Researcher

The matrix should not grow rows for the sake of numbers. For a simple change, one row per criterion with its own check may be enough. For a complex operation, one criterion may need several rows: from a local check of the rule to integration, a manual scenario and recovery after a partial failure.

For each criterion, mark one of three options:

• covered by a specific test;
• covered by a deliberately chosen manual check;
• not yet covered — with the reason, the owner of the risk and the next action.

A gap named in the plan does not automatically become safe. It has to be accepted by a person authorised to decide on that risk. An uncovered scenario must not disappear behind the general word "tested".

Normal, negative, boundary and concurrent scenarios

The usual scenario is only the first row

The usual, or "happy", path is a single typical scenario that ends in the expected success. It is necessary, but it does not show how the system behaves on an error, at a limit, on a repeat, or when there is contention for a shared resource.

For every critical criterion, ask:

• What changes when the input is invalid, incomplete or stale?
• What happens at the edge of an allowed value, and immediately on either side of it?
• Could the user lack the required role, or lose it during the operation?
• What happens if the action arrives twice?
• What happens if two actions happen at the same time?
• Which parts might have finished before the error?
• What does the system show if the external service is slow, unavailable or returns an incomplete response?
• Does an old client stay compatible with the new contract?

Apply these questions where they fit the domain. For a payment amount, zero and the upper limit may matter. For viewing a list: an empty result, the last page and a reload. For access: an allowed role, a forbidden role and a change of data owner. There is no need to bolt every kind of case onto every feature mechanically.

Negative and boundary cases

A negative case checks how the system refuses without an unwanted side effect. A boundary case checks behaviour at the edge of the rules.

Depending on the domain, consider:

• an empty, missing or overly long field;
• the minimum value, the largest allowed value and the value one above it;
• a date on a day boundary, a time zone, or a change of calendar date;
• a record that is already used, expired or cancelled;
• a repeated token, identifier or file;
• an unexpected sequence of statuses;
• access denied;
• too much data or a long delay;
• wrong, outdated or missing configuration.

It is not enough to expect just "an error". You have to say where the error is visible, whether data was changed, whether the action can safely be repeated, whether an outsider received information, and which signal support sees.

Retries and idempotency

An idempotent operation keeps the same business result if it is accidentally repeated. For example, a second delivery of a message about the same payment must not charge the payment a second time. This does not necessarily mean an identical response on every repeat: the second response may say plainly "already processed", as long as the state is not duplicated.

For a risky operation, check that:

1. the first call completes successfully;
2. the client repeats it with the same key or identifier;
3. two identical requests arrive in parallel;
4. a retry arrives after a timeout, when the result of the first attempt is unknown;
5. a new key really does mean a new business action, if that is allowed.

The criterion should specify what counts as "one action": the identifier of the event, the operation, the record or the request. Automatic retrying does not on its own make a system safe. A retry may simply multiply the number of charges or external records.

Concurrent actions

A concurrency scenario checks actions that overlap in time. A sequential test may pass even though, in reality, two customers grab the same slot at once or two processes handle the same event.

Write down the exact precondition: how many resources there are, who is competing for them, which requests start before others finish, and which invariant must hold. An invariant is a rule that must remain true in every permitted outcome — for example, "at any given time a slot has at most one active booking as its holder".

Establish where the action is synchronised: a database transaction, a lock, a uniqueness constraint, the external supplier or some other mechanism. A plan written before implementation can raise the question without dictating the implementation: "Which layer guarantees a single holder of the slot if both customers send a request at the same moment?" If the answer is unknown, decomposition and shaping treats it as an unknown to be resolved before betting on a full implementation.

Timeouts and partial failures

A partial failure happens when one part of an operation has completed and the next has not. Example: the external calendar created a new entry, but the connection dropped before the response arrived; the local system does not know whether to repeat the command. Another example: the change has already been saved, but the notification was not sent.

The plan should break the operation down into visible transitions:

• what the system treats as confirmed;
• where the intent is recorded;
• what can be repeated without duplication;
• which state marks an unknown result;
• who or what reconciles it with the external party;
• what we tell the user;
• how an operator recovers or completes the action.

Do not expect one general error test to cover every partial state. A controlled stub or a test supplier can reproduce "accepted the operation but lost the response", "rejected before changing state", "slow but completed", "partially changed the response" and "unavailable". The real environment may differ from the stub; record which property each level actually proves.

The level of a check should match where the risk lives

Choosing a test level means choosing the cheapest place that is able to detect this particular defect. It is not a contest to have as many full end-to-end scenarios as possible.

Level	What is checked	Especially useful for	What it does not prove on its own
Unit rule check	One function or a pure data transformation	Value boundaries, access rules, calculations	How components are wired together, or the real external contract
Interaction check	Several components together, often with a controlled database or stub service	Transactions, data writes, error handling, retries	How the real supplier behaves in its own environment
Contract check	The mutual agreement between a service and its consumer	Format, types, statuses, version compatibility	That the real supplier is available, or the full user journey
End-to-end check	The full path through several systems to a visible result	Critical integrations and release risks	Completeness across all values and rare states
Manual exploration	A person tries scenarios and watches for unforeseen problems	Clarity of the interface, exploring unfamiliar behaviour, context	Repeatable checking of every combination, or a lasting regression guarantee

"We only test end to end" sounds convincing, but it often produces a slow, unstable suite in which it is hard to tell why something failed. "We tested every function separately" can also hide incompatibilities between the parts. Place checks where they will see the relevant violation earliest and most reliably — and still check critical end-to-end risks along the full path.

In the Playwright documentation, the examples of page checks use assertions that re-read an element until it shows the expected text or a timeout expires. This mechanism helps with ordinary asynchronous interface updates, but it does not fix a wrong expectation or a slow environment. The same documentation warns that assertions without re-checking can cause instability in web scenarios with delayed updates. Playwright: Assertions, accessed 26 September 2026.

Choosing a tool does not determine the right strategy. Before writing tests, be clear about what the tool can actually measure: one browser or several; isolated records or a shared account; a controlled service or the real external supplier. The Playwright documentation, for example, describes projects as a way to configure different browsers and environments. The number of parallel workers can be limited when they would otherwise use one shared resource. Playwright: Projects and Parallelism, accessed 26 September 2026.

How a test knows the right answer

A test oracle is the source we use to decide whether a result is correct. It can be an agreed business rule, a formula, a contract, a law, an independently calculated expected result, a previously approved version, or a physical observation.

The oracle must not quietly copy the very logic it is checking. If a function mistakenly adds a fee twice and the test calls that same function to compute the expected total, both can be wrong in the same way and the test stays green. It is better to define the example independently: base price 100, fee 5, expected total 105. For a complex algorithm you can compare small examples calculated by hand, properties, reference data, a previous implementation or an independent library; name the source of every expectation.

Examples of oracles:

• Requirement: a user without the administrator role cannot change another owner.
• Contract: the service accepts a date in a defined format and returns one of the agreed statuses.
• Data invariant: after one payment the balance is debited once, even if the event is delivered twice.
• Independent calculation: a receipt total calculated by hand with its components shown explicitly.
• Human observation: test users can find the reschedule button without a hint and understand what it will do.

"Obviously" is not an oracle. If the product owner, a document and the current system give different rules, the test must not settle the dispute on their behalf. Return the question to whoever has the authority to approve the behaviour, and mark which work depends on the answer.

Data and environment are part of the plan

The same test can give different results because of different records, a time zone, a feature flag, the version of an external contract, permissions or configuration. That is why "it passed on my machine", with no description of the conditions, is weak evidence.

Describe the data you need

For every important scenario, state the states of the records, not just the general phrase "test data":

• the record belongs to the current user;
• the same record belongs to another user;
• the target resource is free, occupied or deleted;
• the operation has not yet run, has already succeeded, or has an unknown result;
• a dependent record is missing, stale or archived;
• the volume is normal, empty or at the limit.

Do not use real private data without permission and a justified need. Synthetic data must preserve the property being tested. If you need a locale where payment succeeds, an invented customer and record may do. If you are testing performance against the distribution of real file sizes, a convenient artificial file will not replace a representative set; you need a safe procedure for obtaining or generating the data.

Record how the data is created, whether it has stable identifiers, who can see it and how it is deleted. State whether the test database is shared. Two parallel runs must not delete each other's records, unless that collision is exactly what the test is about.

Describe the configuration

What matters for a run:

• the application version, branch or revision hash;
• the version of the interpreter, browser and dependencies, where they have an effect;
• the environment and the scope of network access;
• feature flags and the roles of the test users;
• the version of the contract or stub for the external service;
• locale, time zone, clock and regional settings;
• how the data schema is prepared and the order of migrations;
• limits on quotas, throughput or parallel sessions;
• cleanup and recovery steps.

Do not copy production secrets into a report or a data set. To connect, use test credentials with minimal rights, where the supplier allows it. Name which external systems have been replaced with stubs: this affects what exactly the test proves.

Setup and cleanup

The plan should say how to start from a known state. That might be a fresh isolated database, a transaction that is rolled back, a record factory, a separate organisation, or a disposable environment. Say also whether there is a side effect outside the system: an email, a message, a payment, a webhook (a call to a URL in response to an event), or quota consumption.

Cleanup can itself cause a problem. Say a test deletes a booking after its run while another test is still checking it. Or a repeated cleanup deletes evidence needed for analysis. Define the data owner, the isolation rules and how long artifacts are retained.

Regression: what outside the change might have broken

A regression is the loss, because of a new change, of behaviour that used to work. You do not need to rerun every check in the product after every edit. You do need to name the neighbouring areas that have a real path of impact.

Ask:

• Has a shared contract or storage format changed?
• Do other pages or clients use the same function?
• Are there old client versions, background processes or integrations?
• Does the change touch access rights, filtering or identification of the owner?
• Could the migration affect old and new code at the same time?
• Is performance affected on other devices or data sizes?
• Do create, read, cancel and restore still work?

In the plan it is enough to name the check and the reason for the connection. If the booking schema changes, check at least the old read path, creation, cancellation and the migration. If only a button label changed, a full check of historical tables may be out of place. If the change is global, in the compiler for instance, the regression area may be far wider than two files.

A regression plan does not have to be fully automated. Keep a list of reusable automated checks, and choose individual manual checks by risk. Record it if the matrix of devices, locales or roles has been reduced, and on what grounds.

Automation, code review and exploratory testing

Different kinds of checking answer different questions:

• An automated test checks a given behaviour or invariant quickly and repeatably. It does not discover a requirement that nobody has stated.
• A code review looks for contradictions, missed branches, dangerous assumptions, excessive changes and mistakes in the test itself. It does not automatically execute every behaviour.
• Manual exploration helps find out whether a scenario is understandable, what a person notices in an unusual situation, and where the product's assumptions do not match the person's expectations. It is not repeatable evidence for every combination.
• A security, accessibility or performance review applies specialised methods and expert criteria that should not be reduced to an ordinary green build.

DORA (DevOps Research and Assessment, a research programme on software delivery) describes automated testing as a practice that gives fast feedback throughout the delivery process. At the same time, DORA's materials also describe manual exploratory testing, usability testing and acceptance testing as activities carried out throughout development. The conclusion is not that every click should be automated, but that a separate final quality-assurance phase at the end can delay what you learn about risk. DORA: Test automation and Continuous delivery, no update date shown on the pages; accessed 26 September 2026.

The book Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble and Gene Kim describes the results of research into practices associated with software delivery performance. Here it is used as research context; the specific recommendations about automation and fast feedback are tied to the current DORA pages and are not presented as a guarantee for every team. IT Revolution: Accelerate, published 27 March 2018; publisher's page accessed 26 September 2026.

The Shape Up materials advise that, while shaping work, you identify "rabbit holes" — technical unknowns, unsolved design problems and tangled dependencies — before committing to the full scope. A test plan helps turn such an unknown into a small experiment: for instance, before implementing the whole flow, check whether the test supplier guarantees an unambiguous outcome after a timeout. This is a way to reduce the risk of the bet, not a promise that the experiment will remove every risk. Shape Up, "Risks and Rabbit Holes", page with no visible update date; accessed 26 September 2026.

Do not promise full coverage from a single kind of check

Automation suits repeatable rules, critical regressions, contracts and stable invariants. Manual checking suits exploration, visual perception, an unfamiliar device, or a decision for which no oracle has been defined yet.

Some kinds of manual work can later be automated, once the team understands the exact rule and the test will be maintainable. Others will keep needing regular human observation. Distinguish automated confirmation that steps were carried out from a real check of usability: a script in which the browser clicked five buttons does not prove that a person understands the error message.

Entry, exit and honest status

A test plan defines not only a list of scenarios but also the conditions under which they can produce a useful result.

Entry conditions

Examples of conditions to check before a run:

• the required revision is deployed;
• the test database has the required schema and the data is loaded;
• the external dependency is reachable, or a stub is in place;
• the account has the defined role;
• the required feature flag is on;
• the browser or device is in scope;
• artifacts can be saved without exposing secrets;
• earlier tests have not left a shared resource in an unknown state.

If a precondition is not met, the outcome may be "blocked" — not "passed" and not "product defect". Record exactly which precondition is missing, who can provide it, and what remains unchecked.

Exit conditions

Define a concrete exit criterion before implementation starts. For example:

• every criterion defined as critical has at least one executed piece of evidence;
• no critical scenario has the status "not run" or "blocked" without explicit acceptance of the risk;
• partial failures do not lead to duplication or hidden data loss;
• the mandatory neighbouring regressions passed on the current revision;
• known unstable results are documented and someone is assigned to analyse them;
• manually checked scenarios have notes, an author, conditions and a result;
• test data and external state have been cleaned up or handed to the person responsible.

"All the tests we ran are green" is not a sufficient criterion if some were not run, or were run against a different revision. "The reviewer approved" does not replace behavioural evidence. The conditions can be tuned to the risk and the release, but they must be clear before the decision is made.

Distinguish the statuses

Status	What it means	What to do next
Planned	The check is described but has not been run yet	Draw no conclusion about code quality
Passed	The observation matches the defined expectation on the recorded revision	Keep the result and the conditions
Failed	The observation contradicts the expectation	Find out whether it is a defect, or a fault in the test, the environment or the criterion
Flaky	Repeated runs give different results with no change that explains it	Keep both the failed and the successful runs; look for the cause
Blocked	The run is impossible because of a missing precondition, access or dependency	Name the blocker, its consequence and its owner
Not run	The check was not executed	Do not treat it as passed; record the reason and the residual risk
Not applicable	The scenario does not apply to this change	Record the justification so it can be reviewed

A flaky test does not become fully passed because a rerun came back green. In the Playwright documentation, a failed test that passes on retry is classified separately as flaky, and by default no retries happen at all. A retry can be a diagnostic tool, but it does not erase the first failure and does not establish its cause. Playwright: Test retries, accessed 26 September 2026.

Keep the link between evidence and revision. For an automated run this can be a link to the specific continuous integration (CI) run, the commit (a recorded change), the report and the configuration. For a manual observation: the date, the test environment, the steps, the result, and, for a visual case, a screenshot or short screen recording without secrets. If the code changes afterwards, decide which checks need to be repeated.

Do not add results to the plan that do not exist yet. Before a run, write "expected result". After the run, write "actual result", a link to the artifact, and who checked that it matches. An agent can produce a table of scenarios, but it cannot mark that table as verified just because it generated the rows itself.

The coding agent's role and access to the environment

A coding agent can help turn requirements into a coverage matrix, find contradictions, suggest negative and boundary scenarios, prepare test data, add automated checks and run whatever commands are available. A responsible person still has to approve the product rules, judge which risks have been missed and verify that the evidence is genuine.

Before a test run, the agent should establish:

1. which commands and external services are actually available;
2. whether the environment is isolated, and which data and secrets it can see;
3. whether network access to the test supplier is allowed;
4. whether the agent is permitted to run this particular command;
5. whether logs or screenshots can be saved without leaking private information.

OpenAI's official documentation describes the agent's harness and its environment separately — the environment being an isolated sandbox where commands run; for hosted environments, the network can be configured as allowed, disabled, or restricted to a list of hosts. This means that a product's name alone does not prove access to a particular database or to an external supplier's sandbox. You have to check the actual settings of the session. OpenAI Developers: OpenAI-hosted sandboxes and Sandbox security, accessed 26 September 2026.

Anthropic describes separate filesystem and network isolation boundaries for Claude Code, which can restrict commands' access to directories and external hosts. Whether such restrictions exist, and in what mode, depends on the configuration and the environment; the plan should confirm that the required test resource is actually reachable rather than assume it from the agent's name. Anthropic: Making Claude Code more secure and autonomous with sandboxing, published 20 October 2025; accessed 26 September 2026.

The agent has no right to:

• swap the criterion so that the test passes;
• change an expected value after seeing a failure without the agreement of the rule's owner;
• invent a test environment, data or a confirmed run;
• expose real secrets or private data for convenience;
• present a stub as evidence of a real integration;
• hide the first failed run behind a later green one;
• declare a manual check done when it has only written down the steps;
• mark an unavailable test as unimportant without a decision by the risk owner.

When a check cannot be run, the agent should report the exact blocking condition and propose safe alternatives: a contract test, a local stub, a manual scenario for whoever holds the access, a short experiment, or a documented acceptance of the risk. Each option proves different properties; the limitation should be kept next to the evidence.

Worked example 1: rescheduling a booking

This example continues the booking scenario from vertical slices and implementation plans. Suppose the product owner has approved the following rules for this teaching scenario:

• A customer can reschedule only their own future, active booking.
• The customer chooses an available slot for the same service.
• The old slot stays held for the customer until the new one is confirmed.
• If the reschedule fails, the old booking does not disappear and the customer is shown what happened.
• A slot cannot be taken by two active bookings.
• After a successful reschedule, exactly one confirmation with the new time is sent.
• The confirmation shows the time in the customer's saved time zone; for test B-08 the profile is set to America/New_York.

This assumption has to be approved before the test plan. If the business wanted to release the old slot earlier, that would be different behaviour and a different risk.

First, pin down the criteria

ID	Criterion
C-1	The customer can see and reschedule their own future, active booking to an available slot for the same service.
C-2	The customer cannot read or change another customer's booking; an inactive booking cannot be rescheduled.
C-3	If the chosen slot is taken, or stops being available before confirmation, the reschedule is rejected and the original time is kept.
C-4	Repeating the same operation does not create a duplicate reschedule or a repeated confirmation.
C-5	When the external calendar's result is unknown, the system does not report success until the state has been reconciled.
C-6	After success, the confirmation contains the new time in the customer's saved time zone (America/New_York in test B-08).

Criteria C-1 to C-6 define observable rules. They do not determine whether the implementation uses a lock, an idempotency key, a queue or some other mechanism. The technical solution can be chosen during planning, but the test oracle stays the same.

Now the test plan

Test	Scenario and data	Level	Expectation	Evidence
B-01	The owner of booking A chooses a free slot for the same service.	Interaction, then end to end	A has the new slot and the old one was released after confirmation; other records are unchanged; one confirmation was sent.	Automated report; state of both bookings before/after; the notification event; revision.
B-02	Customer X tries to open and reschedule booking Y.	Access rules and API integration	Access is denied; the response reveals neither the time nor any data of Y; record Y is unchanged.	Role test and a log without private content.
B-03	The booking is completed or cancelled; rescheduling into a past time is also checked.	Rule check	The server refuses; the state does not change; no success signal is emitted.	Results for each status, initial and final state.
B-04	Two customers try to take the only free slot at the same time.	Concurrency check with controlled request start	Exactly one active booking holds the slot; the second action gets a clear refusal; neither old booking is lost.	Count of active holders, both responses, transaction logs with test identifiers.
B-05	The slot becomes taken between the list being shown and confirmation.	Interaction, concurrent change	The server rechecks availability; the customer sees a refusal; the old time remains valid.	Sequence of events and the state of the records at the moment of confirmation.
B-06	The calendar accepted the new slot, but the response was lost before local confirmation.	Integration check with a controlled stub	The user does not see a false success; a retry does not create a second external entry; the state is flagged for reconciliation until the outcome is established.	Request trace with the test key, the state of the external stub and of the local record.
B-07	The request succeeds, but the notification service does not respond.	Partial failure	The booking result is not lost; the notification is not duplicated on redelivery; the failure is visible to an operator.	State of the booking, the notification queue and the error log.
B-08	Server value 2026-01-15T00:30:00Z; the customer's profile is set to America/New_York.	Local format check and manual observation	The confirmation shows 14 January 2026, 19:30 in the America/New_York zone: this is the same UTC instant, and the local calendar date is the previous day.	The original timestamp, the profile setting, the value captured from the interface, the browser's time zone and the result of the automated formatting.
B-09	Regression: view, create and cancel a booking without rescheduling.	Interaction and a short manual pass	Unchanged scenarios keep their agreed behaviour.	Results of the existing automated suite, and manual notes only for the changed neighbouring screen.

In this table, C-1 and C-2 have their own success and negative cases. C-3 relates both to a boundary state and to contention for the slot. C-4 and C-5 need a scenario where a retry follows a lost response; a test of a single successful request does not prove them. C-6 needs a clearly defined time zone and an independent expected value.

B-06 step by step

Starting state: the test customer has an active booking A at 10:00; the 11:00 slot is free; the calendar stub knows the state of A. The booking's owner is identified by a test user, not by the name of a real person.

1. The customer sends a reschedule request with operation identifier R.
2. The stub accepts the 11:00 slot and records its external identifier.
3. The stub simulates a network loss before the application gets the response.
4. The customer sees the message "checking the result", or another agreed state of uncertainty. They do not see "reschedule successful" unless the system has sufficient evidence.
5. The system reconciles the state using identifier R or an agreed external key.
6. After reconciliation, the system has exactly one current slot; the old time is not wrongly restored; the confirmation contains 11:00.
7. The same R is sent again. The number of external entries and confirmations does not increase.

The expected result cannot be judged by the response code or a single message alone. You have to reconcile the external and local states and the side-effect message. The test does not prove how the real calendar behaves: it proves a property of the application under a precisely modelled stub response. A check against the real calendar sandbox remains a separate task, if such a sandbox is available.

Regression and environment

You need synthetic users X and Y, one calendar manager, two future slots, one occupied slot, and a cancelled and a past booking. The concurrency scenario needs a way to fire two requests synchronously, not just two clicks a moment apart. B-06 needs a stub that can lose the response after the external action has been committed. B-08 needs an environment with a controlled time zone. Before the tests, clean up the records and check that there is no earlier event with the same key.

Before the run, settle one more product question: what should be shown if the calendar's result is still unknown after repeated reconciliation? For example, can the customer try again, should they wait, or does an operator reconcile the action manually? The plan may include a pending_review status only if that name and behaviour have been agreed; otherwise record the open decision without inventing a contract.

Exit condition: the critical tests B-02, B-04, B-05 and B-06 have actual evidence; the data never shows two active holders of a slot; the success message appears only after a confirmed state; B-07 does not produce a duplicate notification. If the calendar sandbox is unavailable, that does not stop the local and stub-based checks, but the integration risk in B-06 is flagged separately and not reported as closed.

Worked example 2: a repeated import after a network failure

An import receives an event from an external partner, creates or updates one invoice record, and stores the fact that it was processed. The partner may deliver the same event again. The network may drop after the local commit — that is, after the record has been saved — but before the response reaches the partner.

For this example, the integration owner has approved these rules:

• the same external event identifier has one business effect;
• repeating the same event does not add a second copy of the invoice and does not repeat the side-effect notification;
• if the state after a failure is unknown, the system reconciles the result before creating anything again;
• a different event for the same invoice may update the amount only under an agreed versioning rule;
• a data validation error does not change an existing, correct record.

These rules distinguish redelivery of one event from a genuine new update. If the product has not defined that difference, a decision is needed before any code is written.

Criterion, test, evidence

ID	Criterion	Test	Expectation	Evidence
I-1	One event creates one invoice	Import event E once into an initially empty database	One record with external identifier E and the expected amount	Test report, record count, exact amount
I-2	A repeat does not repeat the business effect	Repeat E sequentially and in parallel	One invoice; one side effect; the repeat has a clear outcome	Delivery history and record/notification counts
I-3	A response lost after commit does not produce duplicates	Make the import commit the record, then break the response	The next delivery of E finds the previous result and does not create a second record	State before/after, key E, reconciliation log
I-4	An invalid amount is rejected without changing the existing invoice	Event E2 contains an amount outside the allowed range	The refusal is visible; the previous invoice and the event log are consistent	Validation result and an unchanged control record
I-5	A newer agreed version updates; an older one does not roll back	First deliver version 5, then, after a delay, version 4	The amount stays at version 5's value; the stale event does not overwrite it	Version number and state after each delivery
I-6	A partial failure is visible and recoverable	Stop the database or the queue at different points in processing	The event is either safely not accepted or kept for retry; neither silent loss nor a double invoice is allowed	Queue log, retry counters, reconciliation report of records

What a single success test proves — and what it does not

One successful run of I-1 proves only that a normal delivery can create a record. It does not test a repeat, a duplicate, parallelism, a lost response, recovery after the queue goes down, or a stale version. These are separate risks with different starting states.

To check I-3, the test transport has to create a precisely defined failure: the base record and the event key are already committed, but the partner has not received the response. Then the same key E arrives again. The oracle is the number of business records, the state of the invoice and the number of notifications — not merely a message saying the function returned true.

For I-2, the simultaneous deliveries must start before the first one finishes. Otherwise you have two sequential repeats, not a concurrency test. If the platform cannot synchronise requests in a controlled way, record that limitation, add another confirmation of uniqueness at a lower level, and do not claim that a real race has been tested.

For I-5, generate two messages with the same external identifier but different version numbers and amounts — for example, 4 → 100 and 5 → 125. Apply version 5 first, then the delayed version 4. The expectation is 125. The numbers here are arbitrary; the oracle is the rule "a newer accepted version cannot be overwritten by an older one", approved for this example.

Data, environment and cleanup

Create a synthetic partner; an invoice with one external key; duplicate events with the same key; a different key for the invalid amount; and two versions of the same event. Keys must be isolated between test runs and must never reach the production environment. External messages and notifications are sent only to the stub. Clean up the test invoices, events and queue after the report has been written; keep the minimal log needed for diagnosis.

The database schema, the queue and the external stub must match the agreed version of the contract. Record whether the test model of timeouts matches the real partner. If the upper limit on redelivery is unknown, do not pick an arbitrary number of attempts, and do not call a limited test complete evidence.

Exit conditions: one business effect per E; no repeat increases the number of invoices or notifications; the state after a partial failure can either be unambiguously restored or honestly shown as unknown; an old version does not overwrite a new one; the mandatory interaction with the partner has either been checked separately or is marked as unchecked.

When the sandbox you need is unavailable

Suppose the supplier allows its API to be tested only through its own sandbox, but the agent's or the team's account does not have access yet. Checking behaviour against a stub is possible; checking that the partner really accepts the event is not.

An honest status record:

Check	Status	What is known	What this result does not prove	Next step
Contract format and validation	Passed locally on agreed examples	Our side forms the expected request	That the real partner accepts the request	Keep the commit and the version of the examples
Redelivery against a stub	Passed; no duplicate was created locally	The application behaves correctly under the given response model	The supplier's real retry order and time limits	Check against the documented contract
Call to the supplier's sandbox	Blocked: no access	The account and the verified reason for the block are recorded	Real creation, timeout, retry or authorisation error	The integration owner obtains access or agrees the test
Manual recovery	Not run	A step-by-step procedure has been written	Whether an operator will actually carry it out without error	Schedule an observed operator run

The plan records who will request access, whom exactly this blocks, which criterion remains unchecked and which decision is needed to proceed. Alternatives can reduce the risk:

1. check the contract format locally with independent examples;
2. obtain a signed or documented sandbox contract from the supplier;
3. model specific responses with a stub and test local recovery;
4. ask an authorised access holder to perform the operation in the sandbox and provide safe evidence;
5. run a limited manual pilot once recovery and monitoring have been defined.

These options close different parts of the risk. A local stub does not prove the real integration. Help from the access holder does not prove every error case. A manual pilot does not automate a repeatable check. The limitation must stay visible in the test report and in the launch decision.

"Not run" is not "passed". "Blocked" is not a product defect. If a test depends on access you do not have, do not write down an invented response, do not ask the agent to get around the isolation, and do not quietly lower the criterion. Ask the legitimate owner of the environment to grant the minimum access needed, or to decide whether the residual risk is accepted.

A real pull request: a test plan and a missing check

In the public OpenIPC repository, pull request (PR) #2260 changes the optimisation flag for two GK7102 targets. The author described the test plan in two items: rebuild userspace and verify that the toolchain wrapper passes the parameter; then verify SSH (Secure Shell, a protocol for secure remote access) and WPA2 (Wi-Fi Protected Access 2, a Wi-Fi security protocol) stability on a GK7102 with the rebuilt SDK (software development kit) toolchain. Both items in the description were left unticked. OpenIPC/firmware #2260, created 11 August 2026; accessed 10 October 2026.

The plan is useful in that it names real targets and a manual check on hardware. But it does not name a check that the previous optimisation level is preserved. A review bot on the same pull request pointed out that the parameter value replaced the optimisation string with only -mno-unaligned-access, potentially removing the expected -O* mode, which could affect the size and speed of other packages. That is a different risk from "did the compiler receive the flag" and "does the Wi-Fi work".

A check through the GitHub API on 10 October 2026 showed, for the pull request's head commit, zero check runs and an empty list of commit statuses, with an overall state of pending; so no green rebuild result is available. Reviews were requested from the code owners, and the pull request was closed on 16 August after the comment "Closing this since no reply". Checks page for #2260. The PR was closed and not merged; this does not prove that the code reached the product or that the bot's concern was ultimately confirmed. It is an instructive example of the difference between planned observations, checks actually carried out, and a gap found independently.

How to improve the plan before coding:

• add a check that the system's optimisation parameters kept the required values;
• give the commands or the exact build artifacts needed to reproduce the result;
• do not tick an item as done until there is a log or a result identifier;
• name the board, firmware, driver, connection method and duration of the Wi-Fi check;
• after reviewer feedback, update the plan and check the current revision again;
• record which build targets were not run, and why.

Code review helped spot a risk that was not in the plan. It did not replace compilation, a check of size and speed, or running the device. And conversely: even a green build would not have proved that a user can work reliably with the network on the hardware board. Each kind of evidence has its own limit.

A short plan template to use before a PR

This template is meant for a working plan. Delete the fields you do not need, but keep unresolved risks visible.

## Promise of the change
Who is it for, and what will become possible? What is the boundary of the work?

## Acceptance criteria
| ID | Expected behaviour | Source/owner of the rule |
| --- | --- | --- |

## Scope and risks
- Changing:
- Neighbouring scenarios:
- High consequence or likelihood:
- Out of scope:
- Open question and decision owner:

## Checks
| ID | Criterion/risk | Starting state and action | Level | Expectation | Data/environment | Evidence |
| --- | --- | --- | --- | --- | --- | --- |

## Scenarios considered
- Usual:
- Negative:
- Boundary:
- Repeat:
- Concurrent action:
- Timeout or partial failure:
- Regression:
- Manual exploration:

## Conditions
- Before the run:
- Exit condition:
- Responsible:
- Cleanup and recovery:

## Not checked
| Check | Why | Risk | Next action/owner |
| --- | --- | --- | --- |

## Actual results
| Check | Status | Revision and environment | Link to evidence | Note |
| --- | --- | --- | --- | --- |

An empty "Actual results" section before the run is an honest state. After the run, add the exact results. Do not mix them with the plan's expectations.

A quick quality check for the plan

Before handing the plan over for implementation, answer:

1. Can each criterion be linked to a specific check or to a visible gap?
2. Does every test have an unambiguous oracle?
3. Are the relevant negative, boundary, repeat, concurrent and partial-failure scenarios covered?
4. Are the data state, environment, configuration and external dependencies named?
5. Is regression included for real neighbouring paths?
6. Are automated checking, code review and manual exploration kept distinct?
7. Are the entry, exit and cleanup conditions known?
8. Are the statuses of all tests not run visible, and do residual risks have an owner?
9. Could another person repeat the test and verify its evidence?
10. Has the agent avoided rewriting a criterion so that the code would pass?

A "no" does not always mean stopping all work. For an obvious small change, a brief clarification may be enough. For a payment, authorisation, migration, booking, import or any other operation with partial consequences, the unknown may be a reason to run a small experiment first or to go back to the owner of the rule.

Summary

A test plan translates criteria and risks into observable scenarios before implementation details start substituting for product behaviour. The criterion says what must be true; the test says how we observe it; the evidence says what was actually seen on a specific revision, with specific data, in a specific environment.

Start with the normal path, but go beyond it: check refusals, edge values, repeats, simultaneous contention, time limits and partial failures. Place each test where it best detects the defect: in a function, between components, at a contract, end to end, or through human observation. Check the regressions, permissions, data and external boundaries that the change really touches.

Record the starting state and how to clean up; safe test data; the environment and dependencies; entry and exit conditions; the decision owner; and the status of every check. A flaky, blocked or unrun test stays visible. A green rerun does not erase an earlier failure. A stub does not prove the real supplier is available. Review is not execution, and a list of scenarios written by an agent is not the result of a run.

DORA helps you think about fast feedback throughout delivery; Shape Up, about removing dangerous unknowns before a big bet. A coding agent can propose coverage and run the checks available to it, but the criteria, the safe boundaries and the acceptance of residual risk must remain explicit human decisions.

Sources and further reading

1. DORA, Test automation, no update date shown on the page; accessed 26 September 2026. The approach to fast, reliable automated feedback, and the mention of manual exploratory testing, usability testing and acceptance testing.
2. DORA, Continuous delivery, no update date shown on the page; accessed 26 September 2026. Fast feedback, test automation, and continuous checking throughout the delivery cycle.
3. IT Revolution, Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps, published 27 March 2018. The publisher's page identifies the book and its research context; the practical recommendations in the text are supported by the current DORA pages.
4. Basecamp, Ryan Singer, Shape Up, Chapter 5: Risks and Rabbit Holes, no update date shown on the page; accessed 26 September 2026. Finding technical unknowns, unsolved design and tangled dependencies before starting a full bet.
5. Playwright, Assertions, documentation checked 23 September 2026. Assertions that re-read web elements, and an explanation of the risk on unstable asynchronous pages.
6. Playwright, Retries, documentation checked 23 September 2026. Retries, the passed/flaky/failed classification, and the significance of the first result.
7. Playwright, Projects and Parallelism, documentation checked 26 September 2026. Configuring browsers and environments; limiting parallelism when tests use a shared resource.
8. OpenAI Developers, OpenAI-hosted sandboxes and Sandbox security, documentation checked 26 September 2026. Configuring the execution environment and network boundaries for agents. This describes product capabilities; it does not confirm that a particular user has access to a particular service.
9. Anthropic, Making Claude Code more secure and autonomous with sandboxing, published 20 October 2025; accessed 26 September 2026. Describes Claude Code's filesystem and network isolation. A source on environment boundaries, not a guarantee that a generated check is correct.
10. OpenIPC, Pull request #2260: gk7102 optimization flag and its checks page, created 11 August 2026 and closed without merging on 16 August 2026; accessed 10 October 2026. A real plan for a rebuild and a manual network check, unticked items, a review comment about preserving optimisation, and the absence of check runs and commit statuses, with an overall state of pending. Not used as evidence of a release or of a confirmed production bug.

Limits of these conclusions. DORA's recommendations generalise its own research programme and do not guarantee the outcome of any individual test. Shape Up describes its own method for shaping a bet, not a quality standard. The Playwright, OpenAI and Anthropic documentation confirms only the stated properties of their tools and environments; it has to be checked against the project's actual configuration. The booking example and all its results are invented. The analysis of the OpenIPC pull request is limited to its description, the visible review messages and the checks page as of the date accessed.