Product and Requirements

Acceptance Criteria: Deciding Whether a Result Is Right

How to write acceptance criteria that are observable and bounded, choose a check that actually observes each one, and record honest evidence, including…

Chapter 10Product and Requirements

Once the need has been agreed and the requirement described, the team needs a shared rule: which observable signs let us say that a particular change is acceptable? Without one, the word "done" tends to mean different things to the client, the author of the code, the reviewer and the person who will actually use the product.

An acceptance criterion describes a condition that a specific result must satisfy before it can be accepted. In the plural, acceptance criteria are often abbreviated to AC. A criterion links the problem and the expected behaviour to a check, but it is not a test in itself, and it does not prove that any test was actually run.

After this unit you will be able to tell apart a need, a requirement, a criterion, a check and a piece of evidence; formulate the main scenario, the forbidden action and the important boundaries; choose a method that fits the kind of expectation; and honestly mark an unknown result as not verified.

Five different things that often share one word

Imagine the request: "Make the voting card easier to use." It contains a desired outcome, but no precise behaviour yet, no threshold of acceptability and no evidence. Separate these notions:

Level Question it answers Example
Need What job or problem lies behind the request? It is hard for a visitor to understand which pair they are rating and whether the system has saved their choice.
Requirement What behaviour must the product provide? The visitor can choose one option in the current pair by click, by touch or with the keyboard.
Acceptance criterion What condition must be true for this result to be acceptable? The chosen option of the current pair is saved exactly once, after which the next pair is shown.
Check What do we do to assess the criterion? Choose an option by touch and by keyboard; confirm that repeating the action does not create a second vote.
Evidence What does the check leave behind? A record of the scenario's outcome and of the code version that was checked.

A criterion is not just a rephrased requirement. "The visitor can vote" does not say what happens if a double click records the choice twice. Likewise, "there is a vote button" describes an interface detail but does not confirm that the choice was saved.

A criterion must not slip in a feature that was absent from the agreed requirement. If correct acceptance needs a new rule (for example, whether a vote can be changed after it is cast), that is a product decision. It should be agreed and recorded, and only then included in the criteria.

A criterion must be observable and bounded

A strong formulation lets two people who know the context reach the same verdict. It describes the situation, the action and the expected observation, plus the boundaries that matter: who has access, which data counts, what "on time" means, how a repeated request behaves.

Compare:

Vague What cannot be decided More testable wording
"Search is fast" Which query, what data volume, environment, time and acceptable share of slow responses? On an agreed set of 10,000 test products, the 95th percentile of time-to-results does not exceed 700 ms in the specified environment.
"The design looks good" Who judges, and on what basis? The agreed layout opens at a width of 390 pixels without horizontal scrolling; the reader can complete the named scenario with no button obscured. A designer checks visual conformance against attached screenshots and the stated screen size.
"The system is secure" Which data, role, threat and path are being checked? An authorised owner sees their own record; a user of a different account cannot read it through the page, a direct request or an export.
"The error is handled correctly" What does the person see, and what state remains? If the time slot is already taken, the system leaves the previous booking unchanged, reports the conflict and shows the available alternatives.

The example with 700 ms and 10,000 test products is an agreed teaching assumption for one specific demonstration, not a universal performance norm. Another team must choose its threshold from people's real expectations, a baseline measurement and the technical conditions.

A quantitative criterion needs at least the following to be specified:

  • what quantity is measured, and in what units;
  • on which data set and over what interval;
  • in which environment, with what hardware or network conditions;
  • how gaps, timeouts and errors are counted;
  • whether it is a mean, a median or a percentile;
  • who agreed the threshold, and on what basis.

In the search example, the team measures the time from pressing "Search" to the results appearing in a test environment. The 95th percentile is the value at or below which at least 95% of the measured times lie; it is a statistic of an ordered set, not the name of a time interval. NIST describes several methods of computing a percentile, which can give slightly different values, especially for a small sample (NIST handbook, about NIST). That is why the criterion names the chosen method. A timeout is a failed response: it is neither discarded as "incomplete" nor counted as zero.

When Given / When / Then helps

The structure of a Given, When, Then scenario

Cucumber calls Given, When and Then keywords of the Gherkin language. Together they let you present a scenario in three parts:

  • Given: the starting state of the system;
  • When: the action or event that occurred;
  • Then: the result that can be observed.

For example:

Given the visitor sees the pair "Coffee vs Tea" and has not yet voted on it

When they select "Coffee" and activate the button with the Enter key

Then one vote is recorded for this pair and a confirmation is shown; if pairs remain, the next one appears

This form helps agree the context, the action and the result without hidden steps. It is especially useful for roles, states, boundary conditions and scenarios with several possible outcomes. A Gherkin scenario can be a specification, documentation and an automated check all at once, if its steps are wired to the program.

The format does not automatically make vague content precise. "Given everything is set up; when the user continues; then everything works" is the same uncertainty split across three lines. And scenarios that are too fine-grained can turn into a script of clicking every element and hide the business rule.

Gherkin describes expectations, not internal implementation. In a scenario, show the result that is visible to the user or to an external system. A database check can help with diagnosis or security, but it does not replace observation at the boundary of the product.

For a simple condition, natural language or a table can be clearer. For instance, a table of pricing rules at headcount boundaries is more compact than ten nearly identical scenarios. Choose the format that leaves the least room for different interpretations. Cucumber specifically warns that Gherkin steps should express the language of the domain rather than tie the scenario to the implementation.

The main path, the forbidden action, the boundary and the repeat

How acceptance criteria cover different scenarios: success, forbidden action, boundary and repeat

For every important requirement, ask four questions:

  1. What happens in the ordinary successful case?
  2. Which action must be forbidden or rejected?
  3. Which edge state could change the result?
  4. What happens if the request is repeated, delayed or executed concurrently?

This is not a call to list every theoretically possible state. Add a scenario if a different interpretation would change money, access, privacy, irreversibility, data integrity, the visible experience or the cost of rework.

When a booking is changed, separate criteria might cover:

  • a successful action: the new time is free and the confirmation is unambiguous;
  • a conflict: another client booked that time between the calendar being shown and the confirmation;
  • someone else's data: the user substitutes a different booking identifier in the request;
  • a repeat: the same request is sent twice after a slow response;
  • a time boundary: the request arrives exactly as the consultation begins;
  • stale state: the client is looking at an old page although the booking was already changed from another device.

These are examples to select by risk, not an obligation to implement every possible variant. The full example below shows how some of these risks become criteria when a different interpretation would change the outcome.

Assign a suitable check to each expectation. A check for a repeated request can be automated; checking that a message is understandable may need a person; checking a legally sensitive rule may need confirmation from the responsible party. No single method proves every aspect of the result.

Data permissions need both a positive and a negative criterion

A positive condition says who should get access: "the owner sees their record". A negative one says who must not: "another client cannot read the same record". You need both: the absence of a button on screen does not prove that a direct request or an export file won't disclose the data.

In a permissions criterion, name the role, the object, the access boundary and the observation. For personal data, the observation must cover every available channel that falls within the requirement: the page; an endpoint of an application programming interface (API), that is, an address through which one program requests data or an action from another; an export; or a notification. If a particular channel was not checked, the result is not marked as fully confirmed.

Acceptance criteria, Definition of Ready and Definition of Done

Task readiness, result acceptance and completion of work as three different gates

A Definition of Ready (DoR) is an agreement about when a task is understood well enough to start work on it. A Definition of Done (DoD) is a shared description of what the team considers a finished change or increment. A DoD is not the same as the criteria of a particular story (Atlassian).

These notions answer different questions:

Artefact Question Where it applies
DoR Do we have enough context to take on this task? Before work on the task starts.
Acceptance criteria Does this particular change meet its specific expectations? During development and acceptance of this feature or fix.
DoD Has the shared completion bar agreed by the team been met? For completing various tasks or a whole increment, depending on the agreement.

A DoR must not forbid investigation. If there is a significant unknown, the team can create a separate short research task: find out what is unknown, record the decision, and then refine the criteria for the future work. An unknown is no reason to pretend it has already been agreed.

A DoD such as "code reviewed, required checks passed, documentation updated" can be a useful team completion condition, but it does not prove that the new behaviour helped the user. The team should define its own bar rather than copy someone else's template as a standard. Atlassian distinguishes a general completion bar from checking a specific implementation against the story's criteria.

A Shape Up pitch can record the time budget (appetite), the risks and the no-gos, the things the team deliberately will not do. Narrowing the scope must be explicit and agreed; you cannot remove a mandatory condition and declare the same result accepted (Shape Up, Write the Pitch).

A criterion does not dictate a single possible test: one expectation may need several checks, and one check may cover several criteria. A traceability map makes this visible:

Requirement Criterion Check Evidence and result
The client reschedules an upcoming consultation The client can choose a free time and see the confirmed new time Walk through the ordinary scenario in a test environment Scenario record, code version, PASS or FAIL
Someone else's booking is inaccessible Another client cannot read the owner's booking Check the interface and a direct request under a separate test role Access-check report with masked data
After the change, both sides see the same state The client and the administrator see the same new time Check both views after reloading Two screenshots, or recorded values and the time of the check

The table helps find gaps, but it does not require separate automation for every criterion. A financial or security boundary usually needs stronger evidence than a cosmetic change.

Measurability is a contract with context, not decoration with a number

When a result is measured, define the metric and the conditions before you see a convenient result. Otherwise, after a failed check, the sample, the threshold or the counting method can be quietly changed. "p95 no more than 700 ms" is incomplete without the start and end points of the measurement, the set of requests, how timeouts are counted, the configuration and the calculation method. A number without that methodology can create no more than an appearance of precision.

Not every kind of value reduces to a technical metric. Response time can be measured automatically; the clarity of a text has to be researched with people; conformance with agreed terms has to be checked by the responsible party. If a metric needs a product decision, record the owner, the source and the date. Don't substitute a mean, a threshold or a time period just because it is easy to measure.

A technical metric does not prove that a product is useful or understandable. Marty Cagan separates the risks of value, usability, feasibility and viability; an automated check can show that search fits within the defined time, but it does not prove that people find the product they want or that the business reaches its goal (Cagan). Steve Krug ties the clarity of an interface to whether a person can orient themselves and find the next action (Krug). For a practical check, ask a participant, without hints, to find a product through the navigation menu and return to the results; separately, show them an empty catalogue and ask what happened and what to do next. The message should explain why there are no products and offer a next step, such as returning to categories or clearing a filter. Record whether the person completed both tasks and how they explained the message in their own words. The session reveals a specific problem, but on its own it does not prove how widespread that problem is among users, or whether the business goal has been achieved.

"Not verified" is a legitimate check result

For every criterion the team needs at least three possible states:

  • PASS: the expectation has been checked and the evidence supports it.
  • FAIL: the check contradicts the expectation.
  • Not verified: there is no evidence, or the evidence does not answer the question being asked.

"No time to run the check" is not a PASS. A test that was never run is neither successful nor failed. Instead, record what exactly remains unknown, why, and who decides the next step. Depending on the risk, the team can fix the problem, add a check, postpone acceptance, or make an explicit decision to waive the criterion. A condition may be changed after a failure only through an explicit revision of the decision, never by editing the checklist until it turns green.

Evidence must be tied to the version being assessed. A note saying "everything works" doesn't say which code, test data or environment was checked. Useful evidence names the verification step, the date, the environment, and the commit or another stable identifier of the version. If a new commit appears after the check, verify whether the evidence still matches the new state.

Worked example 1: Rescheduling a consultation

The initial request: "Let the client change the time of their booking."

The request does not define the time boundary, conflicts, access or repeated actions. We read the policy and the existing behaviour, and refer any unknown that changes a product rule to the decision owner. In this teaching example, the owner approved the following rules: the client can change an upcoming consultation only if the server received the request before it starts, in the Europe/Oslo time zone; the new time is re-checked at the moment of saving; if the time is taken, the existing booking is not lost and the system offers available alternatives; confirming the same change again does not create a second booking; and an invalid time format is rejected without changing the booking.

Requirement: the client can choose an available time for an upcoming consultation. If availability changed before saving, the system keeps the old booking and offers alternatives. The new time is shown identically to the client and the administrator.

So that the server does not apply the same change twice after a timeout, the client resends an idempotency key: an identifier for the request by which the server recognises a repeat and returns the previously stored result.

Criteria:

Criterion ID Scenario and acceptance condition How to check
AC-01 Given the client is viewing their upcoming booking; when they choose an available time and confirm the action; then the booking moves to the new time and the confirmation names that time. Perform the action as a test client; compare the confirmation with the booking card.
AC-02 Given another client took the chosen time after the calendar was shown; when the first client confirms; then their old booking remains in force, and the system reports the conflict and offers current alternatives. Change availability between reading the calendar and confirming; check both states and the offered times.
AC-03 Given a client tries to change someone else's booking; when they send a direct request or open a link to the booking; then the system discloses no data and does not change the booking. Check the page and the request from a different test account.
AC-04 Given the server saved the new time but the response to the client timed out; when the client resends the same action with the same idempotency key; then the client sees the already confirmed new time, and the repeat does not create a second rescheduling. Delay the response after saving, repeat the identical request with the same key, and compare the booking and its history.
AC-05 Given the server received the request at or after the start of the consultation in the Europe/Oslo time zone; then the action is rejected and the existing booking is unchanged. Check a request just before the boundary, at the moment it is reached, and after it.
AC-06 Given the form contains an invalid time that is not in the permitted format; when the client submits it; then the system explains the error, does not change the old booking and does not create a new one. Submit an invalid time format and separately check the message, the stored booking and the absence of a duplicate.
AC-07 Given the rescheduling has completed; when the client and the administrator reopen the booking; then both see the same new time. Check both views after refreshing the page.

AC-05 rests on two agreed boundaries: the Europe/Oslo time zone and the moment the server receives the request. This is a specific policy for the demonstration, not a universal rule.

Coverage map: AC-01 checks the ordinary path; AC-02, the race for availability; AC-03, the permission boundary; AC-04, the timeout and a safe repeat; AC-05, the time boundary; AC-06, invalid input; AC-07, consistent state for client and administrator. A single integration scenario can cover several conditions, but the report must show which ones. Checking the button does not cover AC-03 if the direct request remains open.

What is not a criterion here: "create a button with a calendar icon" does not confirm that rescheduling works; it is an interface detail, unless the agreed design makes it a separate requirement. "It must be convenient" is not a self-sufficient condition until the observable task, the context and the method of assessment are named.

Requirement: catalogue search must let people find a product without a noticeable delay.

The team should not invent a number just for the sake of measurability. First find out the typical data and what users expect. In this teaching example, the product owner approved a fixed set of 10,000 products and 1,000 recorded queries. After 20 warm-up queries, the same set is run five times in the search-bench-01 environment: two application nodes, app-01 and app-02 (each with 2 vCPU and 4 GiB of memory), a separate db-01 running PostgreSQL (4 vCPU, 16 GiB, SSD), and a load-generating bench-client-01 with Chromium (2 vCPU, 4 GiB). All nodes are in the eu-north-1 region and talk over a private network; the p95 network latency between client and service does not exceed 5 ms. The report must preserve the exact versions of the application, PostgreSQL and Chromium, the node configuration, the query set and the measuring tool, so that someone else can repeat the run. All names, numbers and locations are an invented configuration for the teaching example, not a recommendation for a real product.

Time is measured from pressing "Search" to the first page being shown. So five runs give 5,000 measurements; using the nearest-rank method, p95 is the value at position 4,750, and it must be no greater than 700 ms. Errors and timeouts are recorded separately and counted as failed requests. The threshold and the method belong to this teaching example only.

Criterion: p95 of time to first page does not exceed 700 ms, and all 5,000 requests across the five runs complete without an error or a timeout. The report contains the measurements, the configuration and the exact versions of the client and the application.

Criterion for empty data: if the reference catalogue contains no products, search shows the message "There are no products in the catalogue yet" and an available action to return to categories; the screen does not stay in a loading state and does not show invented results. If the catalogue has products but an exact query has no matches, the message explains specifically that nothing matched and offers to clear the query or the filters. These are two different states that should not be collapsed into one empty list.

Check: in search-bench-01, run 20 warm-up queries, then replay the same set of 1,000 queries five times. Record all 5,000 times, count errors and timeouts separately, sort the successful measurements, and take the value at position 4,750 using the nearest-rank method. In addition, run a control for the empty catalogue and, separately, for a catalogue with products but no matches, because these are different interface states.

Evidence: the report preserves the environment ID and composition, the region and network profile, the exact versions of the application, database and browser, the version of the load client, the query set, all measurements, the number of failed attempts, the p95 calculation, and the results for both empty states. If the run used a different region or configuration, collected fewer measurements or did not preserve the calculation method, it is not evidence that the criterion passed. A run on a developer's laptop without the agreed configuration can be a signal worth investigating, but it is not reproducible evidence.

This criterion checks the speed of one specific operation under the named conditions. It does not prove that people find the product they need, or that the business has reached its goal.

Worked example 3: An agent changed the criterion after a failure

The original criterion: if the client has already used a one-time link, opening it again does not change the profile settings and shows a message that the action has already been completed.

The automated check fails: the repeated request saves the new value. The agent proposes replacing the criterion with "opening it again does not create a second one-time link", because the test on the number of links passes.

Such an edit changes the expected behaviour: the criterion constrained both re-application of the operation and the clear message, while the new text mentions only the creation of a link. This is not fixing the implementation; it is narrowing the commitment after a negative result.

The correct process: the agent keeps the criterion and records the actual failure. If the requirement is unambiguous, it fixes the code or the test without changing the condition. If the policy turns out to be different, the decision owner approves a new wording with a reason, a date and a version; then the check is run again. The history of the earlier result is not rewritten.

The practical sequence:

  1. Keep the criterion and the FAIL result exactly as they stood at the time of the check.
  2. Check whether the test really observed the named behaviour, and whether its data state matched the scenario.
  3. If the requirement is clear, fix the implementation or the faulty test; do not change the criterion.
  4. If the source did not define the policy, hand the question to the owner instead of choosing a convenient interpretation.
  5. Keep the old and new wordings, the reason, the author, the date and the version to which the new rule applies.
  6. Re-check the current implementation against the criterion in force and state its revision in the report.

The difference between clarification and substitution: if a criterion contains an ambiguous word, the owner clarifies it before assessment or documents a new version. If it was clear and the code violated it, you cannot afterwards strike out the inconvenient condition and declare success. A faulty test is also possible, but that conclusion requires evidence that the test was checking something other than the observation named in the criterion.

A fully worked real example: a visual criterion that a style test did not prove

In the open repository chamaya00/analytics-practice, task #47 defined five criteria for a voting card: one vote per drag gesture, selection by click or keyboard, page scrolling, support for reduced motion, and conformance of the wide-screen view to the mockup (issue #47).

From the outset the task distinguished one vote after a gesture, activation by mouse and by keyboard, ordinary vertical scrolling, reduced motion, and a visual comparison of the card with the mockup on a wide screen. These are different expectations, so one green set of Document Object Model (DOM) checks does not confirm all of them.

In PR #52 the author openly stated that they had no live screenshot, had done only an indirect comparison of styles, and marked the visual criterion as not verified by an actual screenshot. The reviewer opened the page in Chromium and found that the card lacked a dominant title for the pair, so the criterion was not met. They also checked touch, swipe and vertical scrolling. The automated checks covered clicking and CSS, but the report contains no actual keyboard check; a CSS rule for reduced motion does not, on its own, prove browser behaviour either (PR #52 and the reviewer's comment).

The limits of the checks were specific. poll-dom.test.ts checked clicking on the panel and protection against a double event after a gesture, but the report named no keyboard check, even though the criterion required Enter and Space. To confirm that condition, a test would have to focus the element, activate it with both keys and record exactly one event, or a reviewer would have to go through the page with the keyboard. swipe-card-css.build.test.ts looked for a prefers-reduced-motion: reduce rule in the CSS; that is evidence about the built stylesheet, not about how the browser behaves with the system setting turned on. happy-dom performed no real page layout, so the check did not prove scrolling. These criteria needed a separate browser run with the settings, actions and outcome recorded.

PR #54 added the pair title and a progress line. The reviewer checked the layout in mobile and wide windows, re-tested touch, swipe and scrolling, and then merged the change. They separately flagged a small textual inaccuracy outside the criteria instead of quietly expanding the current task (PR #54).

Fixing the implementation made it possible to repeat exactly the observations the criterion required: first change the card's composition, then check it on a real page at the required sizes. The reviewer split out the minor textual inaccuracy because the current criterion did not require it.

The conclusion is limited to this case: a CSS test does not confirm what appears on screen, and the presence of a criterion in a description does not mean it was checked. The check must match the expectation; "not verified" remains an honest state.

An agent helps refine criteria but does not rewrite expectations at will

Before implementation, an agent can spot a vague condition or propose a check that fits the criterion. Useful questions to put to criteria:

  • Is the starting condition known?
  • Is there an observable result?
  • Are the boundary, role, data and time state named, where they matter?
  • Which negative scenario or repeat could change the result?
  • What evidence will confirm the criterion, and does it fit the kind of expectation?
  • Is the criterion a hidden requirement for a specific architecture or a list of files?
  • Does the wording promise a conclusion broader than the check?

Before asking a person, the agent reads the task, the requirements document, the mockup, the policy, the code and earlier decisions. It distinguishes the program's current behaviour from the desired behaviour. If authoritative requirements conflict, or a product, financial, privacy or irreversible decision is needed, the agent names the owner and formulates a specific question. Plan mode in Codex helps draw up a plan first and ask clarifying questions where needed; Claude Code describes Plan mode as investigation before editing, during which changes may be blocked until approval. This is the behaviour of specific tools and may change; it does not give the agent authority to change what counts as an acceptable result (OpenAI Codex, Anthropic Claude Code).

The agent can hand over the gaps it found, before implementation starts, in a table:

Criterion Ambiguity or risk Proposed check Decision needed?
The client changes an upcoming booking The time zone at the start boundary is not defined Check the policy and record the boundary Yes, if the source does not define it
Another client cannot see the booking Unknown whether the requirement covers export Find the privacy policy and the export schema Yes, if the channel is not covered
Search is fast No agreed data set or threshold Propose measurement on agreed data Yes, the owner must approve the threshold

The agent does not mark a criterion as met on the strength of a pull request description, an author's comment or a green overall test suite, if the evidence does not cover that specific condition. After a FAIL it reports the actual result and keeps the criterion. If the policy was different, the owner records a new wording with a reason and a version; the agent does not change the criterion to make the check turn green.

Anti-patterns that hide unfinished work

"No bugs." This is hard to check and says nothing about the system's boundaries. Name the important scenarios and the acceptable observations.

A criterion as a list of changed files. "Changed booking.ts and booking.test.ts" does not describe a user outcome. Files can be evidence of implementation, but they are not the criterion itself.

Only the positive scenario. A successful rescheduling does not prove that another client cannot change the booking, that a conflict won't wipe out the old time, or that a repeat won't create a duplicate.

A vague metric. "Fast" plus an arbitrary number, without data, environment and a definition of the measurement, is false precision.

An automated test standing in for the need. A check that a button appeared does not prove that the client completed the action. A check of a changed database table may be useful to engineers, but on its own it does not confirm that the user saw a clear confirmation.

Weakening after a FAIL. If a condition was changed after a negative result, the history must show who changed the requirement and why. Otherwise it is a hidden shift in expectations.

Green checks presented as proof of everything. Tests confirm only what they checked, under their own conditions; they do not automatically prove value, accessibility, clarity or legal compliance.

How the author checks that criteria are ready

Before handing over, check that:

  1. The criterion follows from the agreed problem or requirement and does not silently add a new feature.
  2. There is enough context for two people to understand success the same way.
  3. The main scenario, the important prohibitions, the boundaries and the repeats that follow from the risk are named.
  4. For a number, the data, units, environment, period and error handling are specified.
  5. For permissions and personal data, there are positive and negative scenarios and the relevant channels.
  6. The check really observes the expected result.
  7. The evidence is tied to a version; without it, the state is "not verified".
  8. The DoR, the change's criteria and the team's DoD have not been collapsed into one checklist.
  9. Where the result depends on policy, the owner, the source and the date are identified.
  10. A reviewer can trace the path requirement → criterion → check → evidence.

Summary

A criterion sets the acceptable condition for a specific change; a check assesses it, and evidence records the result against a defined version. Choose the format and the method according to what exactly needs to be seen. If the condition changed, record the owner's decision, and mark the absence of evidence as "not verified".

Now the team can name the desired behaviour and the evidence that it is acceptable. The natural continuation is non-functional requirements: how to describe quality, reliability, performance, security and the other system properties that cannot be reduced to one button or one scenario.

Sources and further reading