Maintenance and Efficiency

Agent Observability and Delivery Performance

How to trace an agent's task through to a confirmed result, define metrics with explicit denominators, compare cost and time honestly, and recognise…

Chapter 71Maintenance and Efficiency

When is an agent's work actually finished, if the dashboard says DONE but a required check has not passed yet? This unit shows how to trace a task all the way to a confirmed result, define metrics together with their denominators, compare cost and time, and spot the limits of the conclusions you draw. It builds on the idea of product health from observability, on agent evaluation, and on the full cost of agent work from the economics of an AI studio.

The unit ties those topics together at the level of evidence: which task started, what the agent did, what a tool verified, who accepted the result, and what reached the user. DevOps Research and Assessment (DORA), a research programme that studies software development and operations practices, defines five software delivery performance metrics in its current DORA guide. They describe the flow of changes into a specific application or service, not the productivity of an individual person.

Observability lets you take a single run apart and find the bottleneck. Metrics summarise how a system behaves over a chosen time window. Evaluation tests the agent process on a defined set of tasks. Cost adds human labour and invoices. None of these measurements, on its own, proves that a delivered feature is useful, safe or accepted.

Measure the path to an accepted result

Separate four planes of measurement

Before adding a chart to a dashboard, name the question it is supposed to answer. Mixed indicators do not become meaningful just because they share a screen, and they should not be collapsed into a single "AI productivity" score.

Plane Question Examples of evidence
Product in production Does the service work for users? Errors, latency, availability, attainment of a service-level objective (SLO), the agreed target level of service for users
Quality of the agent process Did the agent complete the assigned task against the defined criteria? Share of accepted results, missed defects, repeats on a control set, score of the verified result
Economics of execution What did the accepted result cost? Tokens, tools, infrastructure, result verification, fixes and support; see the economics of an AI studio
Delivery flow How long did a change take to travel from commit to release, and what happened afterwards? Lead time to release, release frequency, failures after a change, unplanned fixes

A single production failure may be connected to the agent's code, to a human's review, to the release, or to a third-party dependency. Traceability helps you gather the causes of one incident, but it does not prove causation for a whole time series. Compare similar tasks and identical services first; when the process changes, write down what else was different.

Useful telemetry has keys that let you walk from the request to the result. A minimal record might include:

  • an internal task identifier (task_id) and stable acceptance criteria;
  • a run identifier (run_id), the attempt number, start and end times;
  • where available, the session identifier, the conversation turn and the provider's trace identifier;
  • the versions of the instructions, model, tool configuration and execution environment;
  • the sources of input data or documents, by stable identifier, together with access rights;
  • the tools and checks used, with their statuses and durations;
  • the commit, branch, pull request (PR; a proposal to merge a branch after review), continuous integration (CI; automated checks that run on code after changes) results, the release and any related incident;
  • the acceptance or rejection event, the criterion, the responsible role and a link to the evidence.

Do not copy the full model prompt, tool output and code into every log by default. Stable identifiers and links to systems that have proper access control and retention periods are often enough. If a message body is genuinely needed to investigate a defect, define access, masking of secrets and personal data, the purpose and the deletion date; see the OpenTelemetry guidance on sensitive data.

Do not record or demand the model's hidden chain of reasoning. For audit purposes, keep verifiable artefacts: events retained according to your data policy, tool calls, test results, code revisions and the basis for a human's decision.

A task state scheme is a convention of your orchestrator, not a vendor standard. A simple scheme might include started, blocked, retrying, needs_review, accepted, failed, cancelled and partial_success. Record each state transition with its time, reason and evidence. States such as running or idle in a provider's session describe whether that session is executing or waiting; by themselves they do not mean the customer has accepted the result.

Define each metric together with its denominator

Before aggregating anything, write down the exact definition, the unit of counting, the time slice and the event source. For example, a session can contain several turns, a turn can contain several model calls, and one customer task can involve several runs. If you count these as if they were the same thing, a change in how you group them will move the chart without any change in the outcome.

Here a cohort is a group of tasks started within one defined time window and judged against the same criteria and deadline. An agent cycle is one repeating unit of work counted by the orchestrator, for example an agent response together with the execution of the associated tool call; define where it starts and ends in your own event scheme. The median splits the observations in half. A high percentile, such as the 95th, is the value below which 95% of observations fall; it reveals long delays that an average can hide.

Outcome, quality and safety

Indicator One workable definition What must stay visible
Acceptance by the control deadline Tasks with an acceptance event before the deadline ÷ all tasks started in the cohort Unreviewed tasks shown separately; they are neither accepted nor rejected yet
Time to acceptance Time from task start to the evidenced accepted event Where possible, show active work, waiting, queueing, fixes and integration separately
Attempts per accepted task All runs of the cohort's tasks ÷ tasks eventually accepted Calculate it for a cohort whose review deadline has passed. A rerun is a new attempt; show cancelled, partial and unreviewed results separately. With zero accepted tasks the indicator is undefined
Share of tasks with human intervention Tasks with at least one manual intervention ÷ all started tasks Explain whether reviewing the result, granting a tool permission and making a fix count as interventions
Actions executed outside permission Actions outside the agreed set ÷ all executed actions Count attempts stopped by policy separately from actions actually carried out
Recall on known defects Verified defects found ÷ all known defects in the control set Have the set checked independently; an unknown denominator gives you no right to report recall
False-finding rate Findings judged false in independent review ÷ all reviewed findings Show the sample size, and separate genuine misses from inconclusive results

Flow and system behaviour

Indicator One workable definition What must stay visible
Tool call errors Tool calls that ended in an error ÷ all executed calls Distinguish a tool error from a correct refusal or a wait for approval
Cycles without progress Cycles with no new verifiable artefact, tool result or state change ÷ all cycles Define the duration or number of cycles before a signal fires, and do not treat ordinary waiting for a human as being stuck
Time in the review queue Time from needs_review to a decision; report the median and a high percentile Give the number of tasks and define where the interval ends; an average hides long delays

Acceptance must correspond to the real criterion for the work. For a code change, that might be a passing required CI check, a review by the owner and a merge of the specified revision. For a support agent, it might be a verified resolution of the request according to service policy. Never create an accepted event merely because the agent wrote "done". If a task has several independent outcomes, decide whether partial success can be accepted on its own, and do not change that rule in the middle of a comparison.

Trace an agent run without mistaking it for the result

Logs and metrics depend on the specific product

An application programming interface (API) defines which requests and responses programs can exchange with each other. A software development kit (SDK) is a set of libraries and helpers for building those requests. An API provider, an SDK, a managed agent platform and your own orchestrator may each have different events and access rules. Name the specific product and documentation version rather than generalising one SDK's example to every agent.

OpenAI Agents API and SDK. In the current OpenAI Agents API documentation, a session groups a conversation and the agent's work; it can contain several turns, and a trace shows the steps of a single turn: model responses, tool calls and handoffs of subtasks to other agents. Traces are available in the platform dashboard once a turn has finished; export through the API returns the data in the OpenTelemetry Protocol (OTLP), a format for exchanging telemetry. The organisation has to enable export, and the project key needs permission to read traces (api.traces.read) or the broader permission to read agents (api.agents.read). Exporting does not switch on automatic delivery of future traces.

As of 10 October 2026, the official quickstart requires the beta header OpenAI-Beta: agents=v1 on HTTP requests; the client SDKs add it automatically. The description above applies specifically to this version of the Agents API surface as documented on the date it was checked.

OpenAI notes in its observability documentation that recorded token counts may appear after a turn, may be missing, or may still change; an empty value is not zero and is not a final bill. A parent agent span accounts for its own usage, not that of its subordinate agents. If you are calculating cost, aggregate the records of the run and its sub-runs by their identifiers, and reconcile the financial figures with the cost source described in the economics of an AI studio.

OpenAI also documents separate built-in tracing for the Agents SDK: in the standard server-side scenario it is on by default and can record model and tool calls, handoffs, guardrails and custom trace spans. That is not a promise for every client or browser. The Agents SDK surface and Agents API tracing work in different ways; do not assume the parameters or logs of one are available in the other. These specific capabilities were checked on 10 October 2026 against the SDK guide and the Agents API reference.

Anthropic Managed Agents and the administrative APIs. The Claude Managed Agents beta has its own stream of session and agent events. As of 10 October 2026, requests to /v1/agents, /v1/sessions and /v1/environments require the header anthropic-beta: managed-agents-2026-04-01 (see the beta header rules). The session.status_idle event can indicate that a turn has finished, that the agent is waiting for a human action, or that a budget has been reached; each of these calls for a different next step. The session usage field contains cumulative tokens, server-side tool calls, active time and a cost estimate at public list prices. A session limit stops new model requests once the limit is reached, but a request already in progress is allowed to finish, so the final cost can slightly exceed the limit (session budget rules). These states and the cost estimate apply to Managed Agents.

For historical reconciliation of Claude API usage, Anthropic describes the Usage and Cost Admin API for a Claude Console organisation; it requires administrator credentials, and a workspace key alone will not work. Claude Enterprise has a separate Analytics API with a different key type. The Usage and Cost API examples use anthropic-version: 2023-06-01; for Claude Platform on Amazon Web Services (AWS, Amazon's cloud platform), programmatic endpoints are not yet available. Anthropic's rate limit documentation treats rate-limit errors, which carry a retry-after header, separately from errors caused by exhausting a monthly spend limit: these are different states and should not be handled by the same retry loop.

Look at the whole life cycle, not the DONE label

Fully worked example 1: the run announced completion, but a required check failed. An agent has to change the validation for repeat orders. Below is an illustrative timeline; success for this task requires the specified CI check to pass, an owner review, and a merge of a specific commit.

Time Event from the source Task state and evidence
10:03 The orchestrator created task_id=T-204 and recorded the criteria and start time started; the task has a defined set of checks
10:14 The agent's turn ended with a response saying "done"; a PR was opened with commit c41 needs_review; an agent turn has finished, not customer acceptance
10:17 The required CI check on revision c41 failed blocked; the event holds the run identifier, the check name and a link to the log
10:39 The agent made another attempt and produced commit c52 retrying, then needs_review again; the link to the first attempt is preserved
10:46 The same required check passed on c52 The check has passed, but there is no decision from the code owner yet
11:02 The owner accepted the reviewed revision and the PR was merged accepted; the revision, person/role, time and merge link are recorded

If the dashboard shows DONE because it counts the word "done" in the model's reply, it will record success at 10:14: ahead of the evidence, and even before the PR check failed. A correct log keeps the distinct events run_completed, check_failed, check_passed and accepted. For the acceptance rate, the numerator is a task with final confirmation, not the number of completed turns. The measurement point for the series is a cohort of tasks whose criteria and acceptance deadline were recorded before the run.

Count the price of retries together with human attention

Fully worked example 2: a cheaper call does not guarantee a cheaper result. Take the two teaching scenarios from the economics of an AI studio for the same ten tasks and the same acceptance criteria. The scenario figures are not measurements of models and do not rank providers. What matters here are the orchestrator's indicators and the total labour:

Indicator for ten accepted tasks Illustrative route Luna Illustrative route Sol
All attempts 14 11
Attempts per accepted task 14 / 10 = 1.4 11 / 10 = 1.1
Model and tools across all attempts US$0.64 US$4.39
Model and tools per attempt 0.64 / 14 ≈ US$0.046 4.39 / 11 ≈ US$0.399
Review, fixes and support reserve 348 min = 34.8 min per accepted task 178 min = 17.8 min per accepted task
Direct cost per accepted task in this example, excluding overheads US$34.86 US$18.24

Under these assumptions, route Luna has the lower model cost per attempt but needs more attempts and more human time; the higher token rate in the Sol scenario is outweighed by the shorter time spent on review and fixes. The data are invented to demonstrate the calculation; they do not show that any particular model is generally faster or better. For a decision like this in your own log, record the model version, the acceptance criteria, the full set of tasks, rejected and pending results, review time, fixes and final costs. The full unit-cost formula and current rates are in the economics of an AI studio.

Keep agent quality separate from the production change flow

Read the five current DORA metrics in the context of a service

The current DORA guide groups its five metrics into change throughput and deployment instability. It recommends applying them to a single application or service and watching how the results change over time. Measure the same production boundary under an unchanging counting rule. Use the indicators to find changes and bottlenecks, not as a target for competition between teams or as a single number for success.

DORA metric The question it answers Definition
Change lead time How long does a change take to reach release? Time from a commit in version control to the change being deployed to production
Deployment frequency How often does the service receive released changes? Number of production deployments in a chosen period, or the interval between them
Failed deployment recovery time How long does it take to restore the service after a failure that needs immediate intervention? Time from a deployment-related degradation to restoration of the service
Change fail rate What share of releases needs an immediate rollback, fix or other intervention? Production deployments that need immediate intervention ÷ all such deployments
Deployment rework rate How many releases did the team add to resolve a production incident? Unplanned deployments made to resolve a production incident ÷ all production deployments

DORA has changed its model over the years. According to the historical overview of DORA metrics:

  • 2014–2015: in 2014 the research looked at deployment frequency, lead time for changes, mean time to recover (MTTR) and change failure rate; the statistical model relied on the first three. By 2015 the model had been framed as a combination of throughput and stability.
  • 2018: the research moved from the broad notion of "information technology performance" to Software Delivery and Operational performance (SDO performance), and added availability as a measure of service health.
  • 2021: availability was broadened into reliability, which also covers latency, performance and scalability; DORA notes that reliability is a broader measure of operational work, not a fifth software delivery metric.
  • 2023: MTTR was renamed and redefined as Failed Deployment Recovery Time, to focus the measure on degradation caused by a change in production.
  • 2024: Deployment Rework Rate was added as the fifth metric.

The book by Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps, came out in 2018; the official publisher's page confirms the first print edition. It belongs to an earlier stage of the model. This unit does not attribute the modern fifth metric or the later redefinition of recovery to the book: for the current definitions it relies on the dated DORA guide and DORA's historical overview.

Reconcile metrics with events and data limitations

The open-source Four Keys repository shows how to derive metrics from source events instead of from manual surveys about "speed". Its webhook can receive events from GitHub or GitLab, but the built-in parsers extract data from Cloud Build and GitHub; GitLab needs a custom parser extension.

Cloud Run passes events to Pub/Sub, a handler normalises them and writes them to BigQuery, and Structured Query Language (SQL) queries prepare the data for Grafana. The repository keeps the event handlers and the queries, so you can trace where an indicator comes from.

This is a historical example, not a ready-made recommendation for a new service: the repository is unmaintained and implements the four older metrics. It warns that a squash merge can break the link between commits on a working branch and the revision on the main branch (trunk), which is needed to measure the time from commit to release. If the source did not preserve the link between revisions, or a timestamp, the number can still be wrong. The collection by itself does not prove that a team's results have improved.

Worked example 3: two metrics from a small cohort of production releases. An illustrative team makes ten deployments of one service over two weeks. Under the agreed counting rule, seven planned releases needed no immediate intervention; two planned releases caused a problem and needed immediate intervention; and after a production incident one separate unplanned deployment was made to fix it. In this example the groups of events do not overlap:

Event within the window Number of deployments
Planned, no immediate intervention 7
Planned, immediate intervention needed afterwards 2
Unplanned, to fix a production incident 1
All production deployments in the denominator 10

On this small illustrative set, the change fail rate is 2 / 10 = 20% and the deployment rework rate is 1 / 10 = 10%. In both cases the denominator is all production deployments of this service within the same time window, not the number of tasks, PRs or agent attempts. The separate unplanned fix counts in the denominator; in this example it is not added to the two deployments that triggered immediate intervention.

With only ten events, a single deployment moves either figure by ten percentage points, so these percentages do not describe a stable level of risk. Before comparing, check whether the service, traffic, release size, incident classification rules or merge method have changed. Over several months, show the raw numerator and denominator next to the percentage. Do not turn them into an assessment of an individual commit author.

Use signals to improve the system

Count accepted flow, the queue and the consequences together

Fully worked example 4: completions went up, but accepted flow got worse. An illustrative team compares two periods of equal length for the same service. In both cohorts every task had five working days for independent review. In the table, "accepted" means tasks confirmed within that deadline, and "pending" means not yet classified, which is not the same as rejected.

Cohort indicator Before the process change After the process change
Tasks started 20 40
Marked complete by the agent (DONE) 20 40
Accepted within the same five-day deadline 16 20
Acceptance rate within the deadline 16 / 20 = 80% 20 / 40 = 50%
Rejected after review 2 8
Pending a decision at the cut-off 2 12
Median wait for review 1 day 4 days
Production deployments 10 15
Deployments needing immediate intervention 1 4
Share of deployments needing immediate intervention (change fail rate) 1 / 10 = 10% 4 / 15 ≈ 26.7%

The agent process marked twice as many tasks complete, 40 instead of 20, yet only 20 were confirmed as accepted within the deadline instead of 16, and the acceptance rate fell from 80% to 50%. The review queue grew from two tasks to twelve, the median wait became four times longer, and the share of production releases needing immediate intervention rose.

This is a signal to investigate review capacity, task criteria, change size and the quality of control. It is not proof that the agent caused every regression: the periods may have differed in other changes or in the difficulty of the work. Pending tasks stay a separate category until a decision arrives for them.

Make every alert a trigger for a specific action

An alert should name the symptom, the owner and the next step. Practical examples:

  • an agent cycle has produced many attempts with no new revision or progress: notify the responsible person and limit or stop the run according to the budget rules for agent runs;
  • the share of rejected reviews or tool errors has risen sharply: pause automatic acceptance and check the source;
  • the needs_review queue is older than the agreed window: assign an owner or reduce concurrent flow;
  • the Usage API is missing the latest interval, or the final streaming response never arrived: mark the costs as incomplete, not as zero;
  • the rate of failed production releases has changed: check the cohort, the event source and the real incidents before changing policy.

The link between a chart and an action matters more than the number of charts. Do not page anyone just because one attempt needs a permission or is waiting for a human decision: that may be a normal control step. For requests, distinguish a short-term rate limit from an overall spend limit by the error type and the provider's headers; do not raise the budget automatically after a failed call. The other budget and stopping rules are described in the economics of an AI studio.

Respect privacy and do not turn metrics into a ranking of people

Put task_id, run_id, the model, the instruction version and event types with a small, known-in-advance set of values into a controlled store. Do not attach the identifier of every task or user as a label on every metric: unique values make aggregation harder and can create yet another copied trail of data. Keep the details in access-controlled logs or traces, and aggregate metrics by service and time window.

In the OpenTelemetry Semantic Conventions for agent traces, the conventions are marked as Development. In particular, the text of input and output messages is marked as data that should be recorded only after explicit opt-in, and a conversation identifier should not be invented by hashing the text when no real identifier exists. The schema may change; keep your own stable events and a version of your mapping.

Metrics should help the team find the bottleneck, not turn the number of runs, commits, lines or tokens used into a ranking of engineers. A review queue may point to the process or to a lack of time, not to a "slow reviewer". Peopleware: Productive Projects and Teams stresses the cost of interruptions and the importance of protecting time for focused work; this unit draws on the publisher's available excerpt. Use team-level aggregates and the quality of the accepted product; when individual personnel conclusions are needed, a chart alone does not justify them.

Summary

Completion means a confirmed result against a criterion set in advance, not a count of turns or an agent's label. Linked events let you separate active work from waiting, review and fixes; explicit numerators and denominators show what actually changed. Cost and delivery metrics must be read within a single service, taking into account quality, the queue, privacy and the limits of the data.

Sources and further reading