Testing Fundamentals: What a Check Actually Proves

Test plans before coding showed how to describe a criterion, a scenario, an expected result, an environment and the evidence before implementation begins. Worktrees and isolated environments showed that a separate working tree does not automatically create an isolated server-side database or isolated test data. This unit looks at how different checks find different defects, why a green result always has limits, and how to avoid replacing the expected behaviour with the test author's assumption.

In this unit a test means a repeatable action with known preconditions and an expected observation. A test case describes one such scenario: the starting state, the action and the expected result. A test suite groups cases so that they can be run and reported together. An assertion is an explicit check of an expected fact, for example "the status has become cancelled". The source from which the expected fact is determined is called the test oracle. It may be an accepted product rule, a contract, a data invariant, an independently calculated result, or a human observation. The plan, the oracle and the environment are covered in more detail in Test plans before coding; here the focus is on choosing the kind of check and interpreting its result.

A defect is a flaw in a requirement, a design decision or the code that can cause incorrect behaviour. A symptom is what an observer sees in a particular run: for example, the wrong status or an error message. Expected behaviour is the agreed result under defined conditions, not whatever the application currently returns. For a bug in a cancellation rule, the symptom might be that the owner is refused, while the expectation is that cancellation is allowed up to a time limit; the test case should name those input conditions and that result.

A test can reveal a defect; a successful run shows only that particular assertions matched the result under particular conditions. It does not prove that there are no unknown scenarios or undetected defects. Exhaustively checking every input, sequence and environment is usually impossible.

Four questions to ask of any test

Before trusting a green indicator, work through these:

1. Which defect can it notice? Tie the check to a specific behaviour or risk, not to a file name.
2. Which system boundary does it cross? A single rule, several components, an agreement between services, or a user's path through the deployed system?
3. Where did the expectation come from? The test must contain the correct answer independently of the code it checks.
4. Which conditions and substitutions affect the conclusion? Data, configuration, time, the network, the database, the browser, a replaced service, and parallel runs.

If we cannot answer these questions, the word "passed" does not yet explain which risk has been removed. A test can be useful and still fail to cover the neighbouring scenario.

Checking a single rule, and the three steps of a test

A unit test checks one small responsibility under controlled conditions. "Unit" does not necessarily mean one method or one file: what matters is that this particular check does not depend on the real database, the network or another external system. Such a test is cheap and localises an error in a rule well. But it does not confirm that the application has correctly connected the rule to the route, the database, the permissions and the external provider.

A common way to read such a test is Arrange–Act–Assert (AAA): first set up the state, then invoke the behaviour once, and then compare the result with the expectation. So that the time comparison does not depend on local settings, the example uses Coordinated Universal Time (UTC) for both values. Each line of the example below explains what it sets up or proves. Booking and can_cancel_booking are the names of a teaching model and function; in a real project they would correspond to specific code.

from datetime import datetime, timezone  # Import the time and time-zone types from the Python library.

def test_owner_can_cancel_confirmed_booking_before_deadline():  # The name states the rule being checked.
    owner_id = "client-17"  # Arrange: create a stable synthetic owner identifier.
    actor_id = "client-17"  # Arrange: the actor must be that same owner.
    starts_at = datetime(2026, 10, 10, 10, 0, tzinfo=timezone.utc)  # Arrange: start time is 10 October at 10:00 UTC.
    now = datetime(2026, 10, 9, 9, 0, tzinfo=timezone.utc)  # Arrange: "now" is 9 October at 09:00 UTC.
    booking = Booking(owner_id, "confirmed", starts_at)  # Arrange: the booking exists and is still confirmed.
    allowed = can_cancel_booking(booking, actor_id, now)  # Act: call the rule under test once.
    assert allowed is True  # Assert: the product rule lets the owner cancel before the deadline.

The oracle here is a separately stated requirement: the owner may cancel a confirmed booking if there are more than 24 hours until it starts. The example has 25 hours, so we independently expect True. We do not call the function under test a second time to compute the expected value. If both calculations repeated the same defect, the test could be wrong together with the code.

The word "before" has to be defined by the product: what happens at exactly 24 hours? Record this in the rule and add a case for the exact boundary; otherwise different tests may unknowingly expect > and >=. Two more independent scenarios check that another user cannot cancel someone else's booking, and that an already cancelled booking does not move into a new state a second time. A single typical success does not answer these questions.

pytest lets you check expectations with Python's familiar assert statement and shows the details of a failed comparison. When comparing numbers with an allowed tolerance, state the tolerance explicitly. You can also run existing unittest suites: both approaches can coexist while a project migrates gradually. The choice of testing framework does not, by itself, make the oracle correct. pytest: Assertions, pytest: unittest compatibility, Python: unittest — documentation checked 24 September 2026.

Example 1: cancelling a booking — the rule versus the full operation

Take the rule: only the owner may cancel a confirmed booking, and only if more than 24 hours remain before it starts. The owner should see the updated status, and the change should be saved.

Unit tests of the permission function can quickly check the owner, another account, a different status, edge values of time and the boundary itself. They localise errors in the access and time logic. But True in such a test does not yet prove that the application's route called the function with the current user's identifier; nor does it say that the updated state reached the database.

An integration test checks the interaction of several real components. For the same rule, it is worth wiring up a test route, the real service and a controlled test database, creating a synthetic booking, sending a request as the owner, and then reading the record back from storage. The expectations are set before the run:

Scenario	Action	Independent expectation	What a unit test alone would miss
Owner, confirmed booking, 25 hours before the start	Send a cancellation request	The response confirms success; the state in the database is cancelled	That the change was not saved, or that the route passed the wrong user
Another user, the same booking	Send the request from a different account	Refusal; the record's state stays confirmed	A broken permission check in the route or in the database query
Owner, exactly 24 hours before the start	Send the request at the defined boundary	The result matches the agreed boundary rule; the state does not change if refused	Differences in time, precision or clock configuration between layers

This kind of integration coverage can reveal a transaction bug, a mismatch between model fields and the schema, or access control applied only in the user interface. It still does not check real email or a payment service if those have been replaced. The test database must be controlled and safe to write to and clear; shared data or the time zone can change the result.

The words "in-memory database" do not guarantee that the route and the test see the same state either. In SQLite, each separate opening of :memory: creates a separate database by default; several connections can share one only with the appropriate Uniform Resource Identifier (URI) and shared-cache mode. So an integration test may be reading a different, empty database from the one the application wrote to. This is a property of that particular way of using SQLite, not a general rule for all storage. SQLite: In-memory databases, viewed 10 October 2026.

Integration and contracts between systems

The term "integration test" is used for different boundaries: two modules, an application with its database, or interaction with another service. For the name to be useful, state which components are real, what has been replaced, where the request travels and which side effect is checked. The word "integration" without these details does not describe the evidence.

A contract test checks an agreement between a consumer and a provider of messages: which requests the consumer sends and which responses it must understand, or which message format both sides use. For example, a calendar client must send a time zone and receive the agreed time and status fields. Such a test can catch different field names, statuses, headers or serialisation rules earlier than a full deployed scenario.

A contract is not the whole behaviour of the system. A schema can confirm that the status field is a string, but not that the cancellation actually freed the slot. A message contract test also does not check every business requirement of the provider. The Pact documentation describes contract checks as agreement on messages at a boundary and distinguishes them explicitly from functional tests of side effects. So for a booking service, the contract and the test of saving to the database answer different questions. Pact: Introduction, Pact: Contract tests vs functional tests — viewed 10 October 2026.

Example 2: the local fake is compatible, the provider is not

Suppose the booking service asks an external calendar to "reserve the 10:00 slot". A local test replaces the calendar with a fixture that always returns {"ok": true} in JavaScript Object Notation (JSON), a text format for structured data. Meanwhile the provider has changed its real response to {"status": "reserved"}. The consumer code still reads only ok, treats any unexpected response as a refusal, and does not save the confirmed booking. The local test is green because the fixture literally repeats the test's old expectation; no real exchange of messages took place.

The fix here does not necessarily mean "remove all substitutes". First make the agreement explicit: which specific response counts as success, which fields are mandatory, what a slot conflict means and how the client handles an unknown status. The contract test should call the real calendar client, send the defined request to a controlled test provider (a mock provider) and check how the agreed response is handled; the same contract must then be verified against the provider's code. A separate integration test with a real test database confirms that success or conflict changes the local state correctly. A full test environment, where needed, checks that the deployed services are reachable.

This invented scenario should not be presented as a real incident. But the problem of a test fake's fidelity has a concrete counterpart: the maintainers of controller-runtime explained that its fake Kubernetes client allowed an ordinary update of a resource to change its status as well, whereas the real client separates ordinary updates from status updates. Pull request (PR) #2259 changed the fake client and added assertions about which state must remain unchanged. For a related case, issue #2362 gives a reproduction in which a test using an ordinary Update call did not check for the conflict error the author expected from Status().Update. These are specific limits of that particular test client, not proof that all fake clients are unreliable. The PR page shows 0 checks; we do not conclude that the change had a green automated build. controller-runtime issue #2230, PR #2259, issue #2362 — viewed 10 October 2026.

Pact stresses that a consumer contract test must call the actual client code, not bypass it and talk to the test server directly. Otherwise the test confirms the behaviour of the test request, not the behaviour of your integration. Pact: Writing consumer tests, viewed 10 October 2026.

End-to-end scenarios and value to the user

An end-to-end (E2E) test goes through an important path across the system's boundaries the way a user or another client sees it: the interface, the application, storage and, where needed, controlled external services. It can notice that components that are correct individually do not, together, add up to a completed action.

The price of this coverage is more preconditions and more causes of instability: the browser, the deployment, test accounts, data, the network, timing and dependent services. A failure can come from any layer, and diagnosing it is often more expensive than with a unit test. Use E2E tests for a small number of the most important paths, and check variations of rules and boundaries closer to the code, where the cause is easier to localise.

The Playwright documentation advises checking visible behaviour, isolating each browser context and avoiding dependence on internal implementation details. A new BrowserContext isolates browser state such as cookies and session storage, but it does not, by itself, create a separate server-side database or clean up a shared external account. (The Flask session example later in this unit uses HTTP, the Hypertext Transfer Protocol.) For an asynchronous interface, a retrying assertion waits for the expected condition; reading the value instantly may see a state that has not been updated yet. Retrying does not make a wrong expectation right and does not remove network instability. Playwright: Best practices, Browser contexts, Assertions — viewed 10 October 2026.

A smoke test is a short check that critical functions work after a deployment. A regression test looks for the loss of behaviour that existed before a change. These are labels for the purpose of a check, not necessarily separate technical levels: a unit test can be a regression test, and an E2E check can be part of a short smoke suite. The International Software Testing Qualifications Board (ISTQB) distinguishes test levels, test types and change-related testing. Its model is useful as a vocabulary, but it does not impose a fixed linear pipeline on every team. Different practices use slightly different names; describe the boundary rather than argue about the label. ISTQB Foundation Level Syllabus v4.0.1, §2.2, dated 15 September 2024; viewed 10 October 2026.

Manual checking and usability research

Manual testing is not only for situations that "nobody has got round to automating yet". A person can notice unexpected behaviour, confusing wording, a visual obstacle or a contradiction with the real context. Quality assurance (QA) covers the practices that reduce the risk of error; it is broader than running tests and does not necessarily mean a separate final role or phase.

Exploratory testing combines learning about the product, forming questions and trying to find a problem. A person can change the route, a value or the sequence of actions on seeing a new clue. This is valuable when the behaviour is poorly understood or the expectations have not yet been articulated. For the result to be usable, record the goal, the data, the sequence, the observations and what was left unchecked. An automated scenario is better at repeating a sequence that has already been chosen, but it does not invent an unknown requirement.

Steve Krug emphasises human observation of how people use an interface. For example, clicks in a test browser may succeed even though a real person does not notice the button, does not understand the message, or cannot recover after an error. A model or a scenario generated by artificial intelligence (AI) can help prepare questions, but it is not a user and does not replace observing real behaviour. This does not make human observation infallible: the observer, the tasks and the participants have their own limitations. It adds a kind of evidence that is absent from a check of a selector or of code. Steve Krug: Rocket Surgery Made Easy, sample chapter, Sensible: downloadable testing resources — the author's own materials; viewed 10 October 2026.

Example 3: a successful path hides an unclear refusal

An automated E2E test signs in to the application as the owner of a booking, clicks "Cancel", confirms the action and sees "Booking cancelled". For the ordinary scenario this is useful evidence: the visible element, the route and the record interacted well enough to complete the action.

During manual exploration of the scenario "another user opens a link to someone else's booking", the tester sees the response code 403 (access denied), but the message in the interface says "An unknown error occurred. Please try again." The server handled the data correctly, but the user does not understand that they have no access, and may repeat the action or contact support. The requirement to deny access is met; the criterion of a clear refusal is not.

Once the expected wording is agreed, it can be automated as a check of visible behaviour. Human observation helped find the criterion; an automated regression test will help keep it from being lost later. This is an example of a sequence, not a mandatory replacement of manual checking with automation.

Fixtures, test doubles and what they hide

A fixture provides a reusable starting state or resource for a test: a synthetic user, an isolated database, a test browser or a controlled clock. A fixture should define both how data is created and how it is cleaned up. The longer a shared resource lives across tests, the greater the chance of a hidden coupling between them. In pytest, a fixture's scope can be a function, a class, a module or the whole session; choose a wider scope deliberately, especially if the resource changes. The documentation warns that a failed fixture setup can leave partially created state and that automatic safe clean-up cannot be assumed. pytest: Fixtures, viewed 10 October 2026.

A test double is the general name for a stand-in for a dependency. Its subtypes describe different roles, although in conversation they are often confused:

• A stub returns a predefined response to steer the test into the desired scenario.
• A mock expects a particular interaction and checks whether it happened, for example that a message was sent once with the given data.
• A fake has simplified working behaviour, for example temporary in-memory storage instead of a full database.
• A spy records calls for later inspection while letting the behaviour continue.

The definitions differ slightly between sources. For assessing risk, the more important questions are: what did the test replace, what remained real, and what discrepancy could the replacement hide? A network stub does not prove compatibility with the provider. A fake database may not reproduce transactions, uniqueness constraints or permissions. A mock can confirm that a method was called, but not that the called service actually carried out the action. A substitute can be right for isolating a single rule and still be false evidence of an external integration.

A controlled clock helps check boundaries without actually waiting a day; a fixed random seed makes generation reproducible; unique identifiers and clean-up are part of data isolation. At the same time, a clock in a fake will not reveal a time-zone bug in the database, and a serialiser in a stub will not confirm the message format on the real network. Worktrees and isolated environments covered the physical separation of working trees for parallel agents; it does not automatically isolate a shared database or an external account. Test data needs its own boundaries and owners.

Which scenarios to add

A sensible suite does not check every possible combination. It covers the classes of input and the risks that could break the promised rule. For important behaviour, ask whether there is a check of:

• the ordinary successful scenario;
• a refusal, and the side effect that must not happen when refused;
• the boundary value of the rule;
• a repeated action, or recovery after an interruption;
• parallel requests for the same slot, if that is realistically possible: after two users try at the same time, no more than one confirmed booking should remain;
• different roles and owners of data;
• serialisation and actual persistence, if they are on the path;
• behaviour visible to the user, if the criterion concerns clarity.

For access to a booking it is not enough to check that the owner gets their own record. Add another user who requests the same identifier. The expectation is a refusal and an unchanged record. Otherwise the evidence gathered concerns only successful reading, not confidentiality.

If the calendar has exactly one free slot, send two requests to book it at the same time. Afterwards exactly one booking must be confirmed; the second request receives the agreed conflict response. Check the result in storage, not just the statuses of the two responses. Sequential requests do not reproduce the race for the last slot and may miss a locking or transaction defect.

Checks can be functional (does the function work), non-functional (performance, reliability, accessibility, security), or focused on the structure of the implementation. They can run at different levels. A local call to a function can check access logic, while load testing can check a service's performance limit. One kind does not replace another just because they all have the status "passed".

Code checks and metrics are not evidence of behaviour

A formatter checks the style of the text. A linter looks for defined categories of suspicious constructs. Static type checking finds some type inconsistencies without running the code. A build shows whether the project could be turned into an artefact. A security scanner looks for known classes of risk. All of these are useful checks, but none of them, on its own, answers the question: "Can another user cancel this booking?"

Code coverage counts which lines or branches were executed during a run. It helps find areas the test suite never reached. It does not measure whether the oracle is right, whether the right state was checked, or whether an assertion is capable of failing on a defect. One hundred per cent of lines executed is compatible with a test that called every function and checked nothing. Conversely, lower coverage can be acceptable if the critical risks are checked independently; the decision should rest on risk, not on a magic number. The coverage.py documentation describes branch counting specifically as a measurement of executed control flow, not of semantic correctness. coverage.py: Branch coverage measurement, viewed 10 October 2026.

Flaky tests and accurate reporting

A flaky test sometimes passes and sometimes fails on what seems to be the same revision, because the result depends on time, randomness, ordering, shared data, parallel execution or an external system. It is a symptom of an undefined condition or of a genuine race for a resource, not evidence that the failure is unimportant.

Rerunning helps gather evidence, but several successful reruns do not prove that the first failure was false. Keep the first failure, the revision, the command, the environment and the logs; check whether the same resource was reused or the order changed. Do not switch the status to "passed" until the cause of the failure is explained. Test plans before coding distinguishes PASS (the expectation was confirmed), FAIL (the result differs), FLAKY (the result changes between repetitions), BLOCKED (a precondition prevented the run) and NOT RUN (it was not run); this vocabulary states the condition of the evidence, not a person's judgement.

Separate out the cause of an unsuccessful run as well. FAIL means the checked result did not match the expectation; this may be a regression in the product, but first check the expectation itself and the test code. BLOCKED means a precondition or the infrastructure prevented a result from being obtained. NOT RUN means the scenario was not run. A skipped check is not a successful one either. A wrong oracle can produce red despite a correct implementation, and a real defect can pass the test if the expectation or the boundary of the check was poorly chosen. None of these causes is fixed by simply renaming the status.

The report should name the code revision, the exact command or scenario, the preconditions, the substitutions and the actual result. For example: "On commit abc1234, 18 unit tests of the cancellation rule passed with a fixed clock; the calendar integration was not run because there is no access to the test provider." Such a report is more useful than "all green", because it does not turn missing external evidence into a promise.

ISTQB stresses that testing shows the presence of defects but does not prove their complete absence. DORA (DevOps Research and Assessment, a DevOps research programme) describes fast automated checks as an important feedback practice, while also recognising the role of exploratory, acceptance and usability testing carried out by people. The book Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble and Gene Kim provides the research context for this discussion. None of this is grounds for declaring any particular test pyramid a law for every team. DORA: Test automation, DORA: Continuous delivery, IT Revolution: Accelerate — viewed 10 October 2026.

A test written by an agent needs checking too

When an agent fixes a defect and writes the test itself, it can repeat the same mistaken interpretation in two places: the code behaves wrongly, and the test approves exactly that behaviour. Before reviewing the implementation, check the source of the expectation: a requirement, an invariant or an example agreed independently. Ask whether the test would fail on the old defect, whether the target code is really executed, and which neighbouring scenario is not checked.

If it can be done without side effects, it is useful to run the new check on the revision before the fix and confirm that it fails for exactly the expected reason. After the change, run it on the changed revision. This is stronger evidence of the link between the defect and the test, but it still does not prove the absence of other defects or the correctness of the requirement itself. Not every release pipeline keeps the previous code available for such a run; in that case, review the quality of the oracle, the old path in the diff and the reproduction in the issue.

Django's documentation for contributors recommends a regression test that fails before the fix and passes after it. OpenAI and Anthropic, in their current guidance on working with coding agents, likewise advise naming the check and reporting what was actually run. These are vendor guidelines for particular tools and repositories; turn them into verifiable conditions for the report, not into an assumption that the agent is infallible. Django: Writing your first patch, OpenAI: Prompting for Codex, Anthropic: Best practices for Claude Code — viewed 10 October 2026.

A real defect: IPv6 in Flask

In July 2026, Flask issue #6093 reported incorrect address parsing for Internet Protocol version 6 (IPv6). In the example, the server expected an address such as [::1]:8080, but splitting the string on the colon interpreted the segments incorrectly. The author of the issue gave example inputs and stated that the proposed checks fail before the fix; the issue does not contain saved command output or a full traceback, so this should be attributed to the author's report and not presented as an independently saved run log.

In PR #6096, Flask changed the address parsing and added several targeted assertions. A parameterised case checks that [::1]:8080 is turned into host ::1 and port 8080 in a server-start test where the network call is intercepted. Another test runs a session operation through the application's test client with the base address http://[::1]:8000/, stores a value and expects a response containing the text 42. These tests show the expected result for a specific IPv6 syntax on two Flask paths. The PR was merged; the PR page reports 15 passing checks, and the checks page shows a matrix of Python versions, platforms and static checks.

The limits of the evidence matter too: the first check is isolated and intercepts the server start, so it does not prove that a real network connection was established. The session test covers the interaction of Flask components in its test client, but not every IPv6 configuration of an operating system or network. Do not attribute to them a broader promise than the data in them actually checks. This is the discipline of testing: a specific defect, an input, an independent expectation, a targeted check and a bounded conclusion. Flask issue #6093, Flask PR #6096, PR files, PR checks — viewed 10 October 2026.

Summary

Choose a test according to where the risk lives: a unit check localises a rule, an integration check tests the interaction of real components, a contract check tests the messages between consumer and provider, an end-to-end check walks an important user path, and manual exploration finds unknown criteria and problems of understanding. For each one, state what is real, what has been replaced and how the expectation was defined.

The criterion and the test should be independent of the implementation, and the report should be tied to an exact run and revision. Formatting, types, the build and coverage give useful signals, but they do not measure the completeness of the promised behaviour. A green result applies to what was actually executed; an honest "not run" or "blocked" is more accurate than a general "everything works".

Sources and further reading

• International Software Testing Qualifications Board, Certified Tester Foundation Level Syllabus v4.0.1 — models of test levels, test types and change-related testing; version dated 15 September 2024.
• Python, unittest; pytest, Assertions, unittest compatibility, Fixtures — test cases, assertions and the fixture lifecycle.
• SQLite, In-memory databases — the boundaries of separate in-memory databases across connections.
• Playwright, Best practices, Browser contexts, Assertions — visible criteria, browser isolation and asynchronous assertions.
• Pact, Introduction, Writing consumer tests, Contract tests vs functional tests — the contract at the boundary between consumer and provider.
• DORA, Test automation, Continuous delivery; Nicole Forsgren, Jez Humble, Gene Kim, Accelerate — the context of automation and feedback.
• Steve Krug, Rocket Surgery Made Easy, sample chapter, downloadable testing resources — human observation of how interfaces are used.
• Ned Batchelder, Branch coverage measurement — what branch coverage actually counts.
• Kubernetes, controller-runtime issue #2230, PR #2259, issue #2362 — specific limits of a fake test client; the PR shows 0 checks.
• Flask, issue #6093, PR #6096 — the IPv6 defect, targeted assertions and the limits of the tests.
• Django, Writing your first patch; OpenAI, Prompting for Codex; Anthropic, Best practices for Claude Code — guidance on reproducing a regression and reporting checks in specific projects.

All changeable pages and the states of issues and PRs were checked on 24 September 2026.