Production Incidents: From the First Signal to Verified Recovery

An alert tells you that something has changed. It does not yet tell you who has been harmed, what the first action should be, or whether the service is back. Observability showed how to connect metrics, logs and traces to what users actually experience. Now we put that evidence to work under time pressure: agreeing on severity by impact, organising the people involved, reducing harm, verifying recovery and handing over the risks that are still open.

Learning outcome. After this unit, the reader can declare and describe an incident without jumping to a conclusion about its cause; tell an established fact from a hypothesis; assign a coordinator, a technical lead, and owners for communication, the timeline and risk; choose a bounded mitigation; verify the user outcome, the data and the queues after it; and decide what must stay on the open-actions list after the incident is closed.

The scope here is deliberately practical. Observability covers signals, thresholds and alerts; this unit covers live coordination after the signal fires. Incident runbooks deal with procedures prepared in advance, and postmortems with the full review and learning after an incident. Release decisions, deployment strategies and rollback are covered in Deployment Strategies and Rollback and Recovery. During an incident a team may use a known way to stop a release or restore safe behaviour, but this unit should not turn into a general deployment guide.

The booking, external-API and agent scenarios below are invented teaching examples. Their numbers, system names and timestamps are made up; they are not product limits or universal thresholds. Separately, we examine a real public report from Amazon Web Services (AWS) about an event in its Northern Virginia region in October 2025; facts about that event are attributed to the AWS report itself.

An incident is a state of work, not the name of a culprit

A production incident is an unplanned state of a service that requires a coordinated response because it affects, or may soon affect, people, data, financial operations, security or some other important property of the product. A team can declare an incident before it knows the cause. Widespread checkout timeouts after a release, for example, are reason enough to gather the on-call engineers; nobody has to prove first that the latest commit is to blame.

Not every error is an incident. One user's failed attempt may be an ordinary input mistake. But a run of identical failures, an unconfirmed payment state or suspected data corruption all call for coordination, even when overall availability of the home page has not changed. Conversely, a loud server-error graph is not always a serious event if other paths work, retrying is safe and users are not affected.

Severity is set by the scale and nature of the impact, not by who changed the code or how hard the failure is to explain. At a minimum, assess:

• how many people or customers cannot complete an important action;
• whether the action is completely stopped or only slower or partly available;
• whether there is a workaround, and who can use it;
• whether the consequences involve money, private data, security, integrity or irreversible operations;
• whether a queue is growing, a backlog is accumulating, or the ability to restore state is being lost;
• whether one feature, one geography or several independent products are affected;
• whether the situation is getting worse, and how long people will wait for the next useful piece of information.

For one organisation "high severity" might mean that payments cannot be made at all; for another, an unacceptable lag in a critical data flow. Google Site Reliability Engineering (SRE) also cautions that severity depends on the product and its users rather than on one set of thresholds for every service. Labels such as "critical", "major" or "minor" are therefore useful only when a team has tied them in advance to a specific impact, response time and escalation route. Do not assume there is a universally accepted meaning of "Sev 1": "Sev" is short for severity, and the number is a local scale. Google SRE: Product-focused reliability for SRE

A convenient opening statement reads: "We are seeing [symptom] in [part of the product] since [time]; [group or volume] may be affected; [important action] is currently [state]; the cause is not yet known; we will give the next assessment at [time in Coordinated Universal Time (UTC)]." It lets you start the response honestly without substituting a theory for an observation.

Give the work a structure before more people arrive

In a small event one person can sometimes investigate, change the system and answer questions all at once. Once several channels, dependencies or risky decisions appear, that person easily loses the overall picture. SRE practice at Google and at PagerDuty adapts ideas from the Incident Command System (ICS) to separate coordination from technical work. It is a useful model, not a mandatory organisational chart for every team. Google SRE Workbook: Incident Response

Distribute the key responsibilities among the participants. One person may hold several roles when that is safe; a single coordinator must hold the coherent decision about priorities and next steps. Assign one executor to each change in the production system: that person owns the exact target, the boundary, the sequence of steps and the report on the result. Specialists can advise, and the coordinator approves the risk; for a risky change, a separate reviewer checks preconditions and side effects without performing the same change.

Responsibility	What it does during the incident	Typical statement
Response coordinator	Declares the mode of work, decides who is needed, sets priorities, approves risky steps, watches the clock and hands over command.	"We are investigating; until 14:20 we do not increase the share of the new release."
Technical lead	Maintains the technical model, assigns individual checks, proposes a mitigation and verifies its result against a defined signal.	"Check the new route in region B; do not trigger repeat charges."
Communications owner	Prepares clear, regular updates for support, management and, if needed, users.	"Only delays are confirmed; payment status is still being checked."
Timeline keeper	Records UTC time, observations, hypotheses, decisions, their owners and outcomes. Does not transcribe every message indiscriminately.	"14:11 UTC — new flag turned off; error rate has not changed yet."
Change executor	Performs one approved change within set limits and reports what actually changed.	"14:13 UTC — the new revision no longer receives traffic; the controller confirms it."
Risk owner / independent reviewer	Checks the preconditions and limits of a risky change, the presence of a stop condition and the plan for checking side effects; does not perform the same change.	"Before retrying, confirm the external system did not accept the previous request."

In a small team, roles can be combined, but the coordinator must remain the single centre of priorities and risk approval. Each change has one named executor; for a risky change, the executor and the independent reviewer are different people. External communication is handled by a designated person, not by every participant at once. If the technical lead performs a dangerous change, the coordinator separately confirms its scope and how it will be verified.

Dividing roles is not bureaucracy. It frees the technical specialist from constantly switching between diagnosis and explaining status, and it lets the coordinator notice when the team is making too many simultaneous changes. When thinking about people, the lens of Peopleware helps: what conditions let specialists concentrate and make decisions together? An incident team's productivity is its ability to reduce user harm and restore trust in the service, not a ranking of individuals by the number of messages or commits. DeMarco and Lister, Peopleware, 3rd edition

Make the timeline usable for decisions

A live incident log is not there to make a nice report. It must answer a newcomer's questions: what do we know, what do we suspect, what have we already changed, and what will we check next. Align every timestamp in the log to a single UTC scale, and record the release, configuration or other revision number wherever it matters.

The minimum entry for each important turn:

Time UTC	Entry type	Content	Owner / source	Next check
14:02	Observed fact	31 of 210 booking-confirmation attempts timed out in the last 5 minutes; the previous 5 minutes: 2 of 198.	Checkout success dashboard	Check the breakdown by revision and region at 14:07
14:05	Fact from logs	Failed traces show a repeated availability check after the reservation is written; it is unknown whether the repeated request modifies that same record.	Technical lead	Reconcile the transaction contract with the booking owner
14:08	Hypothesis	The new status check may have increased transaction time.	Release engineer	Compare old and new revision on the same path
14:10	Decision	Stop the gradual rollout of the new release until the state of records is confirmed.	Coordinator	Reassess at 14:15

A fact is what is directly visible in a source, with a stated time, sample and limits. A hypothesis is a testable explanation that may be wrong. "The error appeared after the release, so the release is to blame" is not a fact; it is a causal hypothesis. "On revision 4.8.2 the timeout rate is higher than on 4.8.1, for the same feature over the same interval" is an observation, provided that is exactly how the query and its sample were built. Note whether a metric includes successful operations, retries, delayed jobs and all regions; otherwise the percentage itself can be read in different ways.

Time also matters for reconstructing causal order. The monitoring system, the client, the database and an external provider may have different clocks and time zones. Keep both the source time and the normalised time when the source has an unknown margin of error; do not pretend that events within the same second happened strictly in log order. Add a link to the graph, log or trace, but do not copy private content or secrets into a shared channel.

The timeline must distinguish two events: "the command was given" and "the target state was confirmed". If the coordinator asked to stop the release at 14:10 and the controller showed zero traffic on the new revision at 14:13, that is two rows. Likewise, "the script reported success" does not prove that the user flow has recovered.

Triage: establish scale first, then choose an action

Triage is the rapid ordering of uncertain problems by urgency and impact, so the team directs its limited attention to where delay is most expensive. It does not mean finding the root cause in five minutes. The opening sequence is often:

1. Declare the event and name a coordinator. State the symptom, start time, potential impact, coordination channel and time of the next update.
2. Check what the user is actually experiencing. Take the metric for the important action and segment it by feature, region and revision. Do not draw conclusions from web-server availability alone.
3. Flag the risk of an irreversible consequence. Could a retried request charge money twice? Could an import overwrite the correct state? Are people waiting for a critical response?
4. Review a short timeline of changes. A new release, configuration change, secret rotation, provider change or growth in load are important correlations, but not automatic proof of cause.
5. Assess whether harm is growing. Look at new failures, queue length and age, processing time, retries, replica lag and partial success.
6. Choose a next step that can be verified and safely limited. Separate "reduce impact" from "find the cause"; do not wait for a complete explanation if a reversible way exists to stop harm that is clearly growing.

A service level objective (SLO, the target level of service quality) and a service level indicator (SLI, the measure of actual quality) can help you understand a deviation from the expected experience, if the product has already defined them for this action. But the numbers must be translated into consequences: how many requests remain unconfirmed; whether an alternative action is available; whether the delay is getting longer. One team's thresholds do not become a standard for another.

In a partial failure, a service may answer some requests, accept operations but defer them, or show stale state. Such a failure is harder than complete unavailability because the overall average can hide a small but important segment. Consider breaking the data down by a criterion that matches the action's path: payment type, provider, zone, customer segment, queue type. Do not add unbounded personal identifiers to telemetry labels: specific operations are better looked up by a controlled identifier in a system with appropriate access.

Mitigation is a separate decision from finding the cause

Mitigation is a bounded step that reduces harm, or the speed at which it grows, even when the underlying defect has not yet been found. It might be pausing a deployment, turning off a new feature, temporarily rate-limiting incoming traffic, routing to a proven alternative or stopping automatic retries. The specific step depends on the system and the consequences.

Before a change, state four things:

• target: what exactly must be restored or protected;
• boundary: which components, data, traffic or people will be affected;
• expected signal: what sign will show improvement, and when;
• stop or revert condition: what we do if the side effect turns out to be worse.

If it is possible that an external call to an application programming interface (API) has completed but the response was lost, do not treat the timeout as "the operation did not happen". First check the operation identifier or the state in the source system; repeat the action only if an idempotency contract or another duplicate control permits it. An idempotent action is an operation whose safe repetition creates no additional effect, provided the system guarantees that property for the relevant key and period. The AWS Builders' Library explains why a stable token and a clear contract are needed for safe retries; the mere presence of such a token in a request does not guarantee that the whole operation is correct. AWS Builders' Library: Making retries safe with idempotent APIs

When a dependency is degraded, unbounded retries can increase its load and make things worse. AWS documentation recommends controlling and limiting retry calls; a particular architecture may use limits on the number of attempts, back-off with growing intervals, random jitter in retry timing, concurrency limits, queues and circuit breakers. These are engineering mechanisms, not a magic set of parameters: check how many layers retry the call, whether there is a deadline, whether the provider might execute the request despite the timeout, and whether the state of the result is actually known. One client library's settings are not a universal rule for every system; libraries and versions change. AWS Well-Architected Framework: Control and limit retry calls

An incident rarely has a single variable, but during live diagnosis it helps to change one factor at a time, as far as safety allows. If you roll back the release, change the concurrency limit and restart the queue all at once, an improvement will not tell you which step helped, and a deterioration will not tell you which one caused the side effect. This does not forbid a coordinated emergency cut-off of several flows when stopping the harm matters more than a causal conclusion. In that case, record each action separately with its time and scope; do not present a bundle of changes as a clean experiment.

Diagnose from evidence, not from the first convincing story

Once harm is contained, look for an explanation that is consistent with several independent signals and that could be disproved. The tools from Observability now work together:

• metrics show scale, breakdown, duration and direction of change;
• logs give the events of a specific operation and its state transitions;
• traces help you see where time was spent and which dependency returned an error;
• versions and configurations show what changed nearby in time;
• business checks answer whether the intended action actually happened, not just whether the server returned a response.

For each hypothesis, ask: which signal should change if it is correct; what would disprove it; which segment is not covered; which operation might have been committed despite a timeout; is there lag between the primary copy and a replica; will an accumulated queue need its own recovery? When testing, choose the safest comparison — for example, the old and new revision on the same route and region, rather than an overall before-and-after average across a sharp rise in traffic.

Dependencies create partial failures: some nodes or operations work, while others are delayed or fail. If a job does not finish before its lease expires, the system may assign it again; if the first worker is still running, parallel attempts appear. If a client cannot tell a timeout from a failure, it may retry creating an order. If the library, the service and the queue all retry, the multiplication grows layer by layer. So diagnose not only the final dependency but also the place where the decision to retry was made.

A queue is useful as a buffer, but it does not absorb infinite work. Compare the arrival rate with the processing rate and with the age of the oldest item. A shrinking queue does not always mean success: it may reflect mass rejection or deletion of jobs. If the service has partly recovered, releasing deferred work too quickly can overload it again. Retried jobs must preserve idempotency, concurrency limits and visibility of final-processing errors. The lesson here is not that "the queue should always be drained faster", but that its impact must be measured separately from the availability of the main request.

A systemic cascade occurs when the degradation of one component raises the load on others; they slow down and trigger retries, and their queues or resource limits cause fresh degradation. An operational plan must therefore answer two questions at once: which input to throttle temporarily so the queue stops growing, and under what condition throughput can be restored gradually. Do not mask the original error with endless retries. Distributed systems have ambiguous outcomes, delays and independent component states; Designing Data-Intensive Applications, 2nd edition, can be read as a reference framework for these problems. It is recommended reading here, not the source of a specific procedure or quotation. Martin Kleppmann and Chris Riccomini: Designing Data-Intensive Applications, 2nd edition

Recovery is the state of users and data, not a green service

After each mitigation, check the end-to-end path that people saw. If the interface is available again, does the full action complete through to confirmation? Does the person receive a truthful state, and are they not being nudged to press the button again? If a background job is being picked up again, are new items completing, and are old ones finishing with a known outcome?

Recovery has at least four independent planes:

Plane	What to confirm
New operations	Success rate, latency and errors for the important user action have returned to a state acceptable for this product.
Operations already in flight	No double charges, duplicate records, stuck transitions or unknown outcomes without an owner.
Deferred work	The queue, the oldest job and the backlog are moving without renewed overload; failures are not hidden by deletion.
Data and dependencies	Replicas, indexes, caches, the external provider and control totals are as consistent as the product requires.

Leave incident mode not after one successful request but when the change has been stable for an interval the team has defined, the main impact has stopped, important unfinished operations are known, and an independent reviewer or the data owner has confirmed the critical invariants. The interval and the acceptable level of failed operations depend on the product; no example in this unit will set them for you.

"Rollback complete" is a report of an engineer's action, not of user success. The new service may stop receiving an error from a dependency while the queue built up over the preceding hours remains. Similarly, a recovered replica may catch up on writes but not yet restore a report that people saw in a cache. Keep infrastructure state, the business operation and the data apart. If a re-import or a compensating correction is needed, record a separate owner, time limit, permission to make the change and check of the result.

Maintain a shared picture and communicate without exaggeration

Google SRE describes three goals of incident management: coordinate the work, communicate information and keep the process under control. In practice this means short, regular communication aimed at a clear audience. The technical channel holds precise observations and the diagnostic plan; the internal audience receives the impact, current state, workaround and time of the next update; an external notice is published by an authorised person according to the organisation's rules. Google SRE Workbook: Incident Response

A useful update answers four questions: what is already known; who or what is affected; what action has been taken or is being tested; when the next update will come. For example: "Since 14:02 UTC some new bookings are delayed after the form is submitted. We are checking confirmations of bookings already created separately. Rollout of the new release has been stopped; we will provide the next update at 14:20 UTC. For now, please do not resubmit a request from a page showing the message 'checking the result'." Such a text describes the current observation; it does not promise an exact time for full recovery without evidence.

Do not write "all data is safe" if only some requests have been reconciled. Do not call a hypothesis the cause. Do not hide partial impact behind the phrase "some users" when the team already has a well-founded segment or an approximate scale. If the scale is unknown, say so plainly and say when it will be assessed. Internal communication must be truthful too: the coordinator needs to hear about a failed action or a side effect, even if the first assumption turned out to be wrong.

Make on-call handover explicit: who is taking the coordinator role from whom, what the current state is, which decisions have already been made, what must not be repeated, what will be checked next and at what time. The new coordinator confirms taking the role before the previous one leaves. This reduces the chance that two specialists independently issue conflicting instructions or that a critical hypothesis gets lost between shifts.

The perspective of Accelerate is useful as a framework for a team's shared capability to deliver changes safely and restore service. This unit should not be turned into a table of individual culprits or rankings during an incident. Some metrics can help a team see where its delivery or response path gets stuck, but the incident channel is not a performance review of a particular employee. The book is cited as further reading on organisational outcomes and team improvement, not as the source of a universal response protocol. Forsgren, Humble and Kim: Accelerate

An agent can gather evidence, but authority belongs to the environment

An agent is useful for preparing a timeline, summarising logs, segmenting errors and formulating several testable hypotheses. In its conclusion it must clearly separate observed facts from interpretation and state the source, the time window and the gap. For example: "In 18 selected traces after revision 4.8.2, waiting time for the confirmation service increased; the sample covers region A, not all regions. This is consistent with the hypothesis of a dependency delay but does not establish the cause. The next safe check is to compare a pre-release trace on the same route."

Read-only access to telemetry, operation state and the audit log is enough for this kind of investigation. By default an agent does not need write access to production: if a change really is necessary, its exact target, environment and consequences must be shown to a named responsible person before it runs. A policy that can approve a "safe" tool call on its own is not the same as human approval. So for a high-risk action, require explicit confirmation from an authorised person; if no confirmation channel is available, do not perform the action.

To change production data, shift traffic back, delete records, flip flags in bulk or launch an expensive re-import, an agent must not rely on how convincing its own conclusion sounds. It needs explicit authority from the owner, a specific target and scope of action, the prior state, a recovery plan and a check for side effects. If the responsible environment grants read-only access, that is the boundary of authority, not a reason to find another tool to make the same write.

Behaviour depends on the product and its configuration. The documentation for the OpenAI Agents SDK (software development kit) describes interrupting a run to approve a sensitive action and resuming the same run state after the decision. It advises placing the check next to the tool that creates the side effect. That is guidance for a specific SDK and application, not a guarantee about every OpenAI agent, Codex or any environment. OpenAI: Guardrails and human review

Anthropic's documentation for Managed Agents labels these rules as a beta capability of a specific platform. It distinguishes automatic approval, waiting for confirmation and server-side evaluation; custom tools that the application executes itself are governed by the application, not by these policies. So the name of a permission mode in one product does not prove that a similar barrier exists in another. Check the configuration of the actual environment the agent runs in and where the tool executes. Anthropic: Managed Agents permission policies

If a tool call returned a timeout, or the agent was interrupted after a write, the outcome may be unknown. Do not repeat a side-effecting action blindly. First establish the state using the operation identifier, the audit log or the source of truth. Then decide whether the previous call completed, and whether a retry is safe under a defined key and contract. Delivering a response and executing a command are not the same thing: a lost response does not cancel a payment or a process that has already been created. The same principle applies to a human executor who loses their connection.

Case 1: bookings are not confirmed after a release

Invented teaching scenario. An online booking service released revision 4.8.2 at 14:00 UTC. Before that, the client screen showed successful completion in 98% of attempts. A few minutes later the on-call engineer received a signal of 15% confirmation timeouts over five minutes. The exact numbers in this scenario are illustrative; they do not set a threshold for a real product.

At 14:06 the coordinator declares an incident and asks for the rollout to stop expanding. An engineer on the technical side checks the end-to-end completion metric, the booking state log and traces of slow requests. They see that the errors are concentrated in the new revision, but do not yet know whether every timeout means a booking was not created. The data owner checks the log of confirmed and pending records without triggering repeat order creation.

At first the team does not tell users to press "book" again: with the state unknown, that could create a duplicate. Support receives a short response script: say that the request status is being checked, ask the customer not to submit a duplicate, and record the support ticket's reference number. The coordinator notes in the timeline that the revision has been halted, but does not call it the proven cause.

At 14:14 a check shows that the new revision adds a synchronous check of an external schedule after a temporary reservation is created. In some traces this check drags on; the transaction has not yet confirmed the booking, but the temporary reservation already exists. The team rolls back only the new behaviour that adds the check after the reservation, leaving the write code and the security check unchanged. This is a bounded hypothesis about a useful mitigation, not proof of the final cause.

After the change, the technical lead checks new bookings on the old path, and the data owner separately reconciles temporary reservations with confirmations. Over an invented 15 minutes, the team establishes that new requests are being confirmed again, unfinished reservations are known, no duplicate active bookings were found in the checked set, and a few requests need manual reconciliation. Users are told only verified results and the time of the next update.

Why this decision fits here. The team mitigated the impact, did not repeat an operation with an unknown outcome, and checked new and already-started requests separately. Only once the route has recovered can it calmly compare traces and code. If timeouts had also appeared on the old revision, or on all routes at once, changing the client release might not have helped; the next step would then have been to investigate the schedule dependency or the database. The results of manual reconciliation, support follow-up and duplicate prevention move into open actions rather than disappearing along with the green graph.

Case 2: a slow provider turned latency into a retry storm

Invented teaching scenario. A reporting service fetches customer profiles through an external API. The provider responds slowly; on the first attempt the client waits 8 seconds, while the queue consumer has its own 10-second deadline. The library retries the call twice, the background worker retries the job once more, and the scheduler, seeing the deadline expire, assigns it again. All numbers and the sequence are invented.

At 09:20 UTC the incident is declared not because of a single graph of external latency, but because of three signs: only 61% of new reports complete within the product's expectation, the oldest job in the queue keeps getting older, and the same report key has several active attempts. The coordinator assigns an owner for external communication, the technical lead separates client requests from re-import, and the risk owner checks whether a retried request to the provider creates a paid operation.

The first measure is a temporary limit on new reports to a third of normal volume — a purely invented limit for this example. The team does not change the retry rules yet. It keeps reads of data already received, marks new reports as "delayed", does not tell users that "the data is invalid" without evidence, and gives the time of the next update. Within five minutes the inflow of new items into the queue falls, but the age of the oldest job keeps rising; the limit reduced pressure, but did not remove the cause.

After the initial limit, the technical lead reviews the retry logs. They show that one report key can be started simultaneously by the client library, the background worker and the scheduler; the provider sometimes completes a slow request after the client has already timed out. So the problem is not just the slow response: several local layers treat an unknown outcome as permission to create another attempt. At 09:32 UTC the shift lead changes one factor — turning off only the scheduler's reassignment after a timeout. The client library's bounded retry is left unchanged. After five minutes the record shows that the average number of concurrent attempts per key has dropped from three to about one, and the inflow into the queue has stabilised; report completion is still slow, so the team has no grounds to declare recovery.

The team does not extend the overall timeout or add a deduplication key during the live incident: those changes would affect resources and retry semantics, and their consequences have not been checked. Instead, before any repeat action it checks the state at the provider by request identifier. When the external system stabilises, workers resume processing the queue in small batches. Extending the deadline, an end-to-end idempotency key and an additional metric for active attempts become separate post-incident changes with owners.

The queue is not drained "at full speed": once the provider's stability is confirmed, consumption resumes in small batches while the team watches the age of the oldest item, concurrency and the share of successful reports. Because of the provider delay, a few items need a final reprocessing; one report cannot be safely retried until the external state is confirmed. Users see a "processing in progress" message with an up-to-date time, not a fictitious "ready" status.

Why this decision fits here. The team separated the initial load limit from the change to retry policy, waited for an observable result after the second action, and deferred further changes until they could be checked. As a result, the timeline gives limited but useful information about what reduced duplication; the slow completion of reports still needs its own explanation. If the provider call is a pure read, a different retry policy is appropriate than for paid or irreversible operations. The latter need a status check and an idempotency contract; increasing back-off (the delay before a retry) does not by itself make a retry safe.

Case 3: an agent proposes deleting "stuck" rows

Invented teaching scenario. An agent read the queue logs of an internal catalogue and saw seven records that had been in the processing state for more than 40 minutes. It recommends deleting those rows with a Structured Query Language (SQL) command and then rerunning the synchronisation. This is only an example: the table names, count and ages are invented.

The on-call engineer does not run the command and does not ask the agent to retry it with different permissions. The coordinator asks for a read-only check: what are the job identifiers; when did they move into processing; who owns them; what attempts and errors are visible; are there corresponding objects in the external storage; has the lease (a worker's temporary right to process the job) been renewed; have completion records appeared after the tool's timeout? The agent may prepare a read query and explain its conditions, but it has no right to delete.

The timeline shows that six records are indeed inactive and have a matching log of completed processing, but the seventh is still running in a slow external process. Had it been deleted, the external operation would have completed without a local record, and a repeat job could have duplicated the action. The owner of the data system defines a supported state-reconciliation procedure: keep the original record, restore the status of confirmed items and set the seventh aside until the provider's result is checked. If a manual fix requires a database write, the responsible person reviews the exact change, the fallback plan and the output of a control query; the agent does not perform it itself.

A separate check returns a timeout after the agent sent a request to change the state of a test job. The team does not rerun the same call: it first reads the audit log and finds the operation identifier, and then establishes that the test job has already been updated. In production this check would run under separate permissions and access rules; the mere fact that a second call can be constructed does not mean it is safe to repeat.

Why this decision fits here. The processing status describes a local observation; it does not prove that the record is redundant. Reading the source of truth and the external effect comes before any fix; for each change the owner, exact target and verification are known. If the system provides an official mechanism to reclaim a stale lease, it can be considered with the data owner under a clear precondition. It is safer not to try to manually "clean up" everything that has not changed for a long time.

Real case: DynamoDB recovered before the dependent AWS flows

The AWS report on the event of 19–20 October 2025 describes several related but distinct periods of impact in the us-east-1 region. AWS reports that the problem began with a race condition in the automation that manages Domain Name System (DNS) records, which led to an empty DNS record for DynamoDB's regional endpoint and stopped automatic repair. The report's timestamps are in Pacific Daylight Time (PDT); they are kept here exactly as AWS publishes them. According to AWS, engineers identified the DNS state as the source of the main outage at 00:38 PDT on 20 October; the records were restored at 02:25, and global table replicas had caught up by 02:32. Because of DNS caching, client connectivity was recovering until 02:40. AWS: Summary of the Amazon DynamoDB Service Disruption

Restoring DNS, however, did not mean that every dependent path was working again. AWS writes that in the system managing physical hosts for Amazon Elastic Compute Cloud (EC2), state checks were being interrupted, after which the hosts' temporary leases began to expire. When DynamoDB became available again, re-establishing the leases created a large queue; because of the queue and retries, components could not make progress. AWS notes that at 04:14 engineers throttled incoming work and selectively restarted some management hosts, which helped restore lease progress. By 05:28 the leases had been restored and new EC2 launches began to succeed, although request throttling and delayed propagation of network configuration were still affecting recovery. The EC2 system was fully recovered at 13:50 PDT. AWS: Summary of the Amazon DynamoDB Service Disruption

The report also describes impact on other dependent flows, including message processing and delayed queues. This marks an important limit of the idea that "the service has recovered": the DNS endpoint answers, but leases, queues, network propagation, limits and the operations that accumulated during the outage live in their own subsystems. Even when the primary component is available again, these states must be checked separately and restored at a pace that does not trigger a new cascade. AWS: Summary of the Amazon DynamoDB Service Disruption

This case supports one specific point: detecting and fixing the primary failure is not the same as restoring every consumer of its dependencies. That is a conclusion drawn from the sequence AWS describes, not a claim that every incident has this structure. In your own system, check separately the root service, its consumers, queues, retries, replicas, and new and existing operations. Do not attribute to AWS a number of affected customers or an independently verified duration: the source of the facts given here is the company's own public report.

Closing the incident and handing over the remaining work

The coordinator can close active response mode when the team has confirmed sufficient stability of the user path through the recovery checks described above, the main risk is no longer growing, and unfinished operations and side effects have owners. Closing does not mean that every fix has already been shipped in code or that the cause has been fully explained. It means that every known piece of remaining work has an explicit decision: do it now, move it to a separate ticket, have a designated owner accept the risk, or defer it with a justification.

Before closing, agree on the following:

• which change or action mitigated the current impact, and when its result was verified;
• what exactly users see now, including known limitations;
• whether there are unconfirmed, duplicate, stuck or deferred transactions;
• what backlog remains in the queues, and whether it is safe to restore normal load;
• which schedules or subsystems have not yet returned to normal;
• who owns each deferred action, when it will be checked and what its completion condition is;
• who needs a final notice, and which channel stays open for support.

Leave a brief summary for people who were not on the call: the initial impact; important timestamps; what mitigated the situation; what is still unknown; the state of data and queues; open decisions. A full analysis of causes and systemic improvements belongs to postmortems. This summary only preserves material that would otherwise be lost once the on-call shift ends. Assign follow-up work to teams or system owners, not to names that have been blamed.

Summary

An incident is declared on the basis of impact and the need for coordination, even when the cause is still unknown. One person holds the shared priorities; technical diagnosis, communication, the timeline and risk ownership have designated owners, although roles can be combined in a small team. Every statement separates observation from hypothesis and carries a time and a source. First limit the growth of harm, then test explanations against metrics, logs, traces and revisions. A timeout does not prove that an operation did not happen; a retry is safe only under well-understood semantics. A service coming back does not prove that user operations, data, queues and dependencies are consistent. An agent can prepare evidence and propose a check, but the right to make a side-effecting change and to confirm its result stays with the designated owner and the rules of the specific environment.

Sources and further reading

Online sources were checked on 24 September 2026; the AWS report and the OpenAI and Anthropic documentation were rechecked on 10 October 2026. The AWS report presents the provider's own account of its event; the facts cited are given with that attribution. The behaviour of SDKs, tools and platforms may change and depends on version and configuration; check the current documentation for your environment before relying on it.

• Google SRE Workbook, Incident Response — an example of coordination structure, roles, a working record and early declaration; the Google/PagerDuty model is presented as an adapted pattern, not a mandatory scheme for every team.
• Google SRE, Managing Incidents — division of responsibility, a live state record and role handover.
• Google SRE, Effective Troubleshooting — testing hypotheses, systematic diagnosis and caution about premature conclusions.
• Google SRE, Product-focused reliability for SRE — how reliability and severity criteria depend on product impact.
• AWS, Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region — the official report on the time intervals and dependent consequences of the event of 19–20 October 2025; checked 10 October 2026.
• AWS Well-Architected Framework, Control and limit retry calls — recommendations to control retries; the specific mechanisms depend on the client and the system.
• AWS Builders' Library, Making retries safe with idempotent APIs — retry design and the idempotency contract for APIs.
• OpenAI, Guardrails and human review — approvals in a specific Agents SDK, interruptions and placing the check at the boundary of the side effect; checked 10 October 2026.
• Anthropic, Managed Agents permission policies — Anthropic's specific beta model of tool policies; configuration of custom tools remains on the application side; checked 10 October 2026.
• Tom DeMarco and Tim Lister, Peopleware, 3rd edition, Addison-Wesley, 2013 — the human conditions of development and teamwork; used as a conceptual lens, not as a response protocol.
• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate, IT Revolution, 2018 — organisational capabilities and delivery outcomes; cited as further reading, without attributing to the book any particular scheme of incident roles.
• Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd edition — further reading on distributed systems and their limits; the book is not used as evidence for any specific claim in this unit.