Postmortems: Turning an Incident into a Better System

After an incident the page loads again, the queue has drained and the on-call engineer has moved on to other work. Does that mean the team now understands what happened? Perhaps the recovery merely masked a symptom. Perhaps another customer, under the same conditions, has already received incorrect data but has not reported it yet. Or perhaps the system really has returned to normal, yet its weak points are exactly as weak as before.

The units on Production Incidents and Incident Runbooks deal with the response during an incident and with the procedures prepared in advance. Here the focus shifts to what happens afterwards: assembling a verifiable account of what affected users, which signals and decisions were available at the time, what helped or hindered detection and recovery, and which changes should reduce the risk. An incident is a reason to ask questions about how the system works. The report itself neither proves the cause nor fixes it.

Learning outcome. After this unit the reader can separate facts from hypotheses and unknowns; build a timeline that records time zones and keeps the time of the event, the time of detection and the time of recording apart; describe several interacting factors without reducing the explanation to "someone made a mistake"; and turn conclusions into concrete actions, each with an owner, a priority, an acceptance criterion and a later check of its effect.

In this unit a postmortem (a post-incident review) means a structured analysis of an event once the system has been stabilised. Its output is not just a document but a set of changes whose completion can be tracked and verified. Different teams use the word for a short note, a meeting, a detailed report, or all three. More important than the name is agreeing in advance on the triggers, the depth of analysis, the audience and how conclusions turn into work. Google's Site Reliability Engineering (SRE) describes a similar practice as a written record of the impact, the mitigation or recovery actions, the causes and the follow-up actions; Google stresses that the format should be adapted to the organisation rather than copied as a universal standard. Google SRE: Postmortem Culture

Decide when to review an event, and how deeply

An incident is an unexpected condition that affects, or may affect, users, data, security or another important property of a service, and that requires a coordinated response. Getting the system working again does not mean every incident deserves an equally large report. A ten-minute outage of a secondary feature with a simple, known fix can get a short review. Lost or contradictory data, a long hidden failure, a payment error, a failed security control, or harm that spans several systems all call for deeper scrutiny.

Near misses are worth reviewing too: situations in which dangerous conditions had already arisen but harm was avoided by luck, by a last manual step or by a backup control. For example, before a bulk import a team might notice that the file uses a different character encoding and stop the import before it overwrites the catalogue. If a team analyses only actual failures, it will not see the warning that a safeguard is broken until that safeguard lets an error through.

Define the triggers before an event happens. Google SRE lists possible triggers such as a user-visible outage or degradation, data loss, manual intervention by on-call engineers, recovery that took too long, and a monitoring failure that meant users found the problem first. These are examples of Google's criteria, not normative thresholds for every company. A small team can start with two levels: a short review of every incident that needed manual intervention, and a full cross-team review for user harm, risk to data integrity, a repeat failure or a significant gap in detection. Any participant may also ask for a full review when they think it is needed. Google SRE: postmortem criteria and culture

A short review fits when the symptoms are known, the scope is limited and the corrective action is clear. It should still contain verified boundaries of the impact, the cause or an explicit "cause not yet established", a link to the change and the date on which its result will be checked. Brevity does not excuse leaving out the unknowns or the owner.

A deep review is needed when several conditions interacted, people had different signals, the recovery itself created a new hazard, the harm is uncertain, or earlier actions did not reduce the risk. Invite the component owners, support, or anyone else whose information explains another side of the event. Do not gather dozens of people "just in case": an oversized meeting can once again interrupt the very people who need to collect evidence. In Peopleware, Tom DeMarco and Tim Lister give separate attention to time for focused work and to interruptions; in the publisher's excerpt they discuss why chopping work into short fragments makes conditions worse for difficult thinking. We apply that idea to investigation: give participants time to gather facts, collect short written contributions, and hold only the meeting needed to reach agreement. This is not a claim that the book proposes a format for post-incident reviews. Pearson: Peopleware, 3rd edition, contents and excerpt of the chapter on focused work

A review is not a trial, a disciplinary procedure or a defence of the team. At the same time, "blameless" does not mean that nobody takes on the follow-up work. Here a blameless approach means three things: do not replace an explanation of the system with a hunt for a guilty person; find the conditions, information, tools and constraints that made a particular action understandable or likely; and assign owners for the improvements. The question "who will change this control and check the result?" is necessary. The question "who deserves to be ashamed?" does nothing for the technical analysis.

Separate impact and facts from assumptions

A strong report does not open with a theory of the cause. It opens with a short description of what can be confirmed right now: which function behaved incorrectly, over what period, which users or data might have been affected, how the team found out and what it did to stabilise the system. Avoid statements such as "all orders were wrong" when only three support tickets have been checked. Likewise, do not write "nobody was affected" if the measurement does not cover every route and every retry.

Mark every significant statement with one of three states:

Status	What it means	Example wording
Confirmed fact	There is a source from which another person can repeat the check.	"The processing log contains 18 rejected records from the 14:00 Coordinated Universal Time batch."
Estimate or hypothesis	The best explanation of the available signals, but it depends on assumptions.	"Up to 40 more records may have arrived again if the provider ran its automatic retries."
Unknown	The available data is insufficient, or it can no longer be collected.	"We cannot establish whether every user repeated the request after the timeout."

Describe impact together with the limits of the measurement. Separate:

• Time window: when the first confirmed incorrect state began, when it stopped occurring, and whether consequences continued after the main recovery.
• Scope: the users, regions, operations, record types, releases or integrations that were actually checked.
• Consequence: a refused action, a delay, duplication, a wrong value, a security risk, extra manual work or lost trust.
• Sources and limitations: metrics, logs, base records, support tickets, reconciliation with a supplier, gaps in retention or privacy constraints.

If the numbers are incomplete, state the estimated range and the method. "At least 7 double confirmations are known; the upper bound has not been established because the gateway log deletes requests after 24 hours" is more informative than a falsely precise "there were 7 incidents". A zero in a report may mean not the absence of a consequence but the absence of a signal.

A public message to customers, an internal engineering review and the full protected record have different audiences. A customer needs to know what happened to their action or data, how to check their own state and what the team has done to recover. The full technical report may contain exact steps, internal addresses, configuration or personal records that must not be published. Keep technical detail in a suitably protected place, and agree the public version so that it stays honest about impact and about what is unknown. Do not hide harm behind euphemisms, and do not publish secrets to prove your transparency.

Build a timeline that can be verified

A timeline must not merge at least three different moments into one:

1. Event time: when the system first entered an incorrect state or performed an action.
2. Detection time: when an alert, a user or the on-call engineer saw the signal.
3. Recording time: when someone added a note about it to a log, a chat or a document.

For example, an automated transaction might create a duplicate at 10:14:02, an integrity check might find it at 10:17, and the on-call engineer might note the finding in the coordination log at 10:20. If the document shows only "duplicate found at 10:20", the team may wrongly put the duration of the impact at six minutes, rather than three minutes to detection plus the time it took to remove the consequences.

Use one standard time zone, such as Coordinated Universal Time (UTC), but do not delete the original timestamps if they matter for verification. Keep the source's time zone and date so that you do not make mistakes around daylight-saving changes or when matching logs from different countries. For approximate times, write "approximately", "no earlier than" or a range, rather than implying precision to the second.

Useful fields for one entry are: the time with its zone, the type of time (event, detection, decision or recording), the source, the observed fact or decision, the owner of the action, and the confidence in the accuracy. When different sources disagree on the time, keep both values and explain the discrepancy. The timeline should show what people knew at each moment, not only how the event looks after full analysis.

This helps avoid hindsight bias: once the cause is known, people tend to believe it was easy to foresee before the event. For every decision, ask: which signal was visible at the time? Which alternatives were considered? Was it clear that one of the options was dangerous? What constraints of time, access, documentation or tooling applied? Do not credit a participant with knowledge that appeared only later.

Describe causal conditions, not a single culprit

The word cause can hide different levels of explanation. Distinguish at least:

• Defect or hazardous condition: the state of code, data, configuration or process that made the incorrect result possible.
• Trigger: the event or sequence that activated the defect this particular time.
• Contributing factors: conditions that raised the chance of the event, widened its scope, or delayed detection and recovery.
• Protective barriers: checks, constraints, tests, redundancy or alerts that could have stopped the error or reduced the harm, together with the reason they were missing, did not fire or were bypassed.

In a complex system it is rarely useful to insist on a single "root cause". The statement "the availability check is not atomic" may be correct at the code level, but it does not explain why two requests could be confirmed at the same time, why the database did not reject an impossible state, why the retry was not idempotent, why the test did not reproduce the race, or why the operator received no signal. A useful causal model shows how several specific conditions came together, rather than presenting the first weakness found as the whole story.

"Human error" describes an action; it is not a sufficient explanation. Record the action neutrally: "the operator ran the job again after the timeout because there was no visible status for the first run". Then examine the surrounding system. Was the first action still running? Did the interface explain the state? Did the procedure describe a retry? Did the database protect the invariant? Did the operator have a way to check the result? This does not remove responsibility for improvement. It turns a simple judgement about a person into possible changes to controls.

Blameless does not mean "nobody is responsible". If an employee broke a clear, accessible and current rule, and the system had working safeguards, the review can still record that specific decision and separately route the management question into the appropriate private process. A technical report must not draw conclusions about someone's competence, motives or character. The owner of each change should understand the task, the deadline, the access required and the escalation path if resources run short.

This practice connects to Peopleware through its attention to working conditions, coordination and organisational learning. That does not mean every incident is caused by the work environment, or that people cannot be held to requirements. It means that workload, interruptions, shift handovers, unclear ownership and fear of reporting a problem are facts about the system of work, provided they can be confirmed. DevOps Research and Assessment (DORA) describes a high-trust culture with an open flow of information as one that supports learning and better sharing of risk; its materials treat "failure leads to scapegoating" and "failure leads to inquiry" as different cultural responses. This is a generalised model drawn from research, not a promise that one postmortem will change a culture automatically. DORA: Generative Organizational Culture

Analyse detection and recovery separately

The question "what caused the defect?" is different from "why did we find out about it exactly then?" and "why did recovery take exactly that long?". For example, an incorrect record might have been created during an import, while the queue-lag alert covered only the average age of the queue. The average stayed acceptable even though one customer group was not receiving updates. That is a gap in the visibility of scope, even if the cause of the import problem is already known.

For detection, check:

• which signal first revealed the problem: an internal metric, an external probe, support, a partner, or the users themselves;
• whether the signal reflected the user outcome and not just the state of a process;
• when the condition appeared and when it triggered an alert, and whether the thresholds suited this type of harm;
• which checks stayed silent, were switched off, showed an average instead of a segment, or had no owner;
• whether a control fired on a near miss that could have stopped a larger impact.

For recovery, separate the action that stabilised the system from the full removal of the cause. Disabling an integration might stop repeated errors but leave the queue incomplete. A rollback might bring the screen back but not restore records created by the new version. Reprocessing might catch up with the source but double the payments or notifications. Record which consequences still remained after the main metric turned green.

Do not judge the quality of the response by duration alone. A longer recovery is sometimes a sign of careful state reconciliation that prevented irreversible harm. Equally, a fast recovery may simply have hidden the problem. Useful questions are: which decision reduced the impact? Which one delayed recovery, and why? Which temporary step must be removed? Which additional check was done before the incident was declared over?

Turn conclusions into verifiable actions

A follow-up action is a separate task with a concrete change or investigation tied to an identified risk. A weak action sounds like "add more tests", "be more careful", "improve monitoring" or "rewrite the complicated service". It does not say which behaviour should change, who will do it or how the result will be checked.

For each action, record:

1. Risk: which specific failure or delay mechanism must be contained.
2. Outcome: what will be different in the system or process once it is done.
3. Owner and helpers: who is accountable for the outcome, and which other teams need to be involved.
4. Priority and reason: what happens if it is postponed, and which deadline is agreed in line with the level of risk and the resources available.
5. Acceptance criterion: what can be shown so that an independent reviewer can confirm the change.
6. Effectiveness criterion: which signal after release or after a rehearsal will show that the risk has actually gone down.

Priorities only mean something within a local agreement. In the etcd report, for example, P0 meant work that blocked the reliability of the v3.5 branch, P1 meant a critically important longer step for the v3.6 release, and P2 meant a goal that could be deferred. Do not copy these levels into another team without defining who assigns them and what each level changes in practice.

Weak action: "Add a test against double booking."

Verifiable version: "The owner of the booking service adds an integration scenario in which two independent requests read the last free seat before confirmation; after it runs, the two requests cannot both receive a confirmation. The scenario is repeated after a timeout and after a temporary hold is released. Acceptance criterion: the store holds no more than one active booking per seat, and the other request receives a refusal or is offered an alternative. After release the team checks the booking-conflict counter and reconciles it with support over an agreed period."

The first part checks that this particular class of defect has not returned; the second checks that the test really catches the race in question; the third checks that the production signal does not miss a similar problem. If the test cannot show that it catches the old incorrect behaviour, first make it sensitive to a control defect or another reproducible break. "The test exists" does not prove that it tests the right thing.

Link the postmortem to a trackable issue or change request in GitHub, a task tracker or another work store. On GitHub an issue usually records a problem or a scope of work; a pull request (PR) proposes a concrete change to the repository for review and merging. A link to a task does not by itself prove the task was effective. You need the status, the acceptance criterion, the owner and a link to the verification. If a task was closed automatically after a PR, make sure the closure means the change is complete, not merely that a document exists.

Check the outcome separately, using a sufficient post-action signal. That might be another rehearsal of the procedure, an automated regression test, a check of the user journey, or observation of the failure class. For a rare event, a few weeks without a repeat is not strong evidence; add a direct engineering check rather than waiting for new harm. Where possible, record the date, the method of verification, the owner and what remains unknown.

Do not equate the number of closed tasks with reliability. A team can close a dozen small items and leave the main risk in place; it can also reasonably defer an action that has become irrelevant, provided it records the new decision and the residual risk it has accepted. DORA currently describes five software delivery performance metrics and stresses that they should be read in the context of an individual service, should not be set as targets for competition between teams, and that improvement should not be reduced to building the measurement itself. The current five-metric model differs from the four "key metrics" described in the original 2018 edition of Accelerate, published by IT Revolution. Metrics can point to a place worth investigating; they do not replace confirmation that a specific follow-up is complete and has reduced the risk. IT Revolution: Accelerate, 2018; DORA: software delivery performance metrics

Agent assistance: gathering evidence, not inventing a story

An artificial intelligence (AI) agent can help group timestamps, find discrepancies between versions of a document, draw up a list of open questions, or turn verified facts into a draft. It can also add a persuasive but unconfirmed cause; confuse text that was in the repository with text the agent actually saw; or invent an exact quotation from an incomplete log. Shared instructions for Codex and Claude Code, and the limits that come with their versions and configuration, are covered in Documentation Maintenance, in the sections on shared agent instructions and on what to do when Claude Code's product documentation changes.

So separate the agent's output into:

• extracted observations, each with a reference to a log, commit, timestamp or document;
• a conclusion or hypothesis that connects several observations;
• questions for people whose information cannot be reconstructed from technical artefacts;
• unknowns that the agent must not fill with its own plausible text.

If the review concerns an agent's work, check the exact version of the instruction, its scope, what exactly was available in that particular run, the tool permission settings, the operations invoked, and the subsequent human review. Do not treat an instruction file in the repository as proof that it was in the context of that run. Loading rules differ between products and environments: the Claude Code documentation describes CLAUDE.md/AGENTS.md as a mechanism for project context, and the OpenAI Codex documentation describes its own rules for AGENTS.md. That confirms the need to check the specific environment and its records, rather than making assumptions about the agent from the product name. The date and the rules for instruction availability in Claude Code and Codex were checked on 10 October 2026; in every review, reconcile against the log of the specific run. Claude Code: How Claude remembers your project; OpenAI Codex: Custom instructions with AGENTS.md

In its guidance for building applications with its models, OpenAI advises constraining outputs and checking results before acting on them, especially when an error has significant consequences. This is a recommendation for OpenAI application programming interface (API) scenarios; it should not be transferred automatically to Codex, Claude Code or other agents. For any tool, the concrete requirement stays the same: every important conclusion must be checked against the primary source or against a person who knows the actual context. OpenAI: Safety best practices

Fully worked examples

The first two examples below are invented for teaching. All numbers, system names and timestamps are part of the scenarios, not measured thresholds or facts about real products. The third is a public postmortem by the etcd team; its facts are kept separate from our teaching conclusions.

Example 1: two customers were confirmed for the same seat

Situation. This is a separate invented scenario, not a continuation of the booking case in Production Incidents, where the team checked state after a timeout once a temporary hold already existed. The service had one seat left for an evening show. At 10:14 two requests received the message "booking confirmed" almost simultaneously. At 10:17 support noticed that both customers had sent the same seat identifier. At 10:21 the on-call engineer stopped sales for that show and began reconciling records. Times in this teaching scenario are in UTC.

What is confirmed. The bookings table holds two active rows for the same seat and show. The log shows that both requests read the value "free" before writing. The interface confirmed both attempts. Support has two tickets. The team does not yet have a reliable count of whether both customers pressed "confirm" once each, or whether one of them repeated the request after a delay. So the report speaks of two confirmed records and notes separately what is unknown about retries and potential scope.

Timeline. 10:14:02 and 10:14:03: two booking-creation events, according to the server log. 10:17: support receives the first ticket; this is when people detected the problem, not when it arose. 10:19: the on-call engineer finds two active rows. 10:21: new bookings for the show are paused. 10:32: the customers are offered a seat at another show or a refund; the final resolution is recorded separately. If the log did not keep the exact server seconds for one of the requests, the timeline should show an approximate time rather than guessing one.

Causal model. The confirmed technical fact is that two requests were able to read the remaining capacity before either write marked the seat as taken. That explains the path to the duplicate, but not the whole mechanism: the team is still checking whether the operation should have run in a single transaction, whether the database schema allows two rows with the same key, and what role the retry after the timeout played. Contributing conditions: the interface showed a fast confirmation, and the system had no independent check for a second active seat. Do not record "customers clicked too fast" as a cause. Two near-simultaneous requests are the expected behaviour of concurrent users, and a booking system has to handle them.

Detection and recovery. The alert showed the total number of seats, but not conflicts at the level of an individual show; support was the one that spotted the problem. Pausing sales prevented new confirmations but did not fix the ones already created. After agreeing with both customers, the team rebooked one of them onto an alternative show and reconciled the payment before declaring the incident over. There is not enough data to claim that no other conflicts exist, so the team searches for the same key across the period for which logs are still available.

Actions. On this invented service's local scale, P1 means that protection against double confirmation must be accepted before sales reopen for this flow; that is a decision based on booking risk, not a universal definition of P1. The storage owner adds an invariant that forbids more than one active booking per seat. The service owner adds a concurrent-confirmation test and a separate scenario for a retry after timeout; acceptance criterion: exactly one request can create an active booking, and the other sees a safe refusal. The observability owner adds a signal for conflicting keys and checks it against an artificial control case before release. The product owner approves the message to customers and the method of reconciling payments. After an agreed time the team checks that no conflicting records are appearing, but does not treat silence as sufficient evidence: the test must show that it catches the old class of defect.

Why this is better. The description concerns the conditions of the system, not the predictable behaviour of users. The report states what is known, how it is known, what is missing, and who is changing which barrier. When this review is incomplete: if reconciliation finds money already charged, hidden bookings, or messages sent to both customers, the corresponding consequences and corrective actions must be added; do not close the report simply because new requests have been stopped.

Example 2: "the agent ignored the instruction" is a hypothesis to test

Situation. After a nightly export runs, the first draft of the report states: "the agent did not read the instruction, so it published a file containing personal data". The team manages to switch off public download before there is any confirmation that outsiders accessed it. The export itself contained extra fields. Making data available in an unsafe state is already an incident, even though whether anything actually leaked still has to be established.

First check. The review does not accept the sentence about the "unread instruction" as fact. In this invented scenario the run service keeps a manifest of each execution: the task revision, the list of context files, the outcome of tool permissions, and a link to the publication event. The manifest shows that the current export rule lived in another document that this run did not receive; it also shows that the agent was allowed to write to a public directory without any check of the output file's schema. In a real product such a manifest must be verified first: not every environment records the list of context in the same way.

Timeline and impact. In this invented scenario all timestamps belong to 2026-09-23 and are given in UTC. At 2026-09-23 02:10 UTC the run starts the export with the new fields. At 2026-09-23 02:14 UTC the file lands in time-limited public storage. At 2026-09-23 02:16 UTC an independent check finds the forbidden field and alerts the on-call engineer. At 2026-09-23 02:18 UTC public access is switched off; at 2026-09-23 02:25 UTC the team completes its initial review of the access log. Public availability lasted four minutes. The system recorded three requests for the object; the log is still being reconciled to establish the time, result and source of each one. How long the file itself was retained after public access was switched off has not been established separately. Do not call the requests leaks, or harmless views, until the security owner has reconciled the audit log.

What contributed. The scenario contained a gap between the current rule and the context the run actually received. The output file had no mandatory check of permitted fields. Publication happened before the result was reviewed, even though the consequence was a change in who could access the data. The report can say that the agent produced and handed over a file with impermissible fields, because that is visible in the artefacts. Without evidence it cannot claim that the agent "decided to ignore", "understood the risk" or acted carelessly. We do not know the model's intent, and we do not replace a technical record with a story about character.

Actions and verification. The export owner moves the list of permitted fields into a schema check at the publication boundary; criterion: a file with any unknown or forbidden field cannot become public. The owner of the agent run configures an explicit context list and records the instruction version for every execution; criterion: a reviewer can reconstruct which revision was available in a given session. The access owner removes automatic public writing for this type of operation until the output passes the schema and a human review. Finally the team reproduces the same case in a test environment and confirms that the barrier fires before publication; it then checks the access log for the original file.

Why it is done this way. The agent's model, the command-line interface (CLI), the application and the server runtime are different things. An instruction file in Git does not prove that every run received it, and the agent's text reply is not proof that its file was published. Here the claims rest on the scenario's manifest, the access log and the artefact itself. When this is not enough: if the manifest did not record the context, or the access logs are incomplete, write that the specific rule is not confirmed as having been available and that the extent of viewing is unknown. Pin down the source; do not fill gaps with a confident tone.

Example 3: a real etcd review and the chain from conclusion to PR

Situation. A report by the etcd developers dated 20 April 2022 describes a state-consistency problem in the v3.5 branch. A refactoring in v3.5.0 allowed the consistency index to be written in a way that could run ahead of applying the corresponding entry from the write-ahead log (WAL, a log of changes that is persisted before they are applied). If the process happened to crash in a narrow window, then on recovery the database could skip an entry, believing it had already been applied. In a system with several members this could produce divergent states. The review states that the team knew of no user reports of actual data corruption in production. It names trust in reliability as the main consequence for the community, rather than claiming proven large-scale data loss. Primary source: etcd v3.5 data inconsistency postmortem

What the report treats as factors. The causal description separates the unsafe change to the index from the moment of the crash. Separately, the authors list weak consistency checking for slow members and for log compaction, functional tests that were insufficiently maintained and flaky, manual release qualification that was no longer common knowledge, and the complexity of the code that applies changes. These are not synonyms for "a refactoring mistake": some factors explain the defect, others explain why it was hard to detect, reproduce and safely rule out before release.

Care with dates. In the postmortem's own timeline, one user report for issue #13654 is dated 28 January 2021, even though it is placed after a December 2021 report. GitHub shows that the issue was created on 28 January 2022; its description reports a revision inconsistency in production but does not confirm data corruption. The wording "data corruption report" in the postmortem's timeline is stronger than the content of that issue; the postmortem itself says separately that its authors knew of no user reports of confirmed data corruption in production. There is a second discrepancy: the postmortem's timeline, dated 20 April, says v3.5.3 was released on 24 April, whereas the official release page shows 13 April 2022. A teaching report should show both contradictions and separate the reported symptom from confirmed corruption; it should not silently copy either date. etcd issue #13654; etcd v3.5.3 release notes

Actions and subsequent outcome. In the postmortem these are two separate actions. P0 proposes reproducing historical inconsistencies; issue #14045 breaks that work down into specific reproduction scenarios. Separately, P1 calls for continuous release qualification. PR #14911, merged on 7 December 2022, implements precisely the sub-item about a nightly long run of linearizability tests: it refers directly to #14045, but it is not evidence that it added a historical regression reproducer on its own. Initially the schedule covered only the main branch; the discussion also raised questions about the duration of the run and the scope of the tests. This lets you trace one specific chain: "conclusion → trackable action → change in the repository". The fact that the PR was merged does not prove that every branch is covered or that the incident risk has gone. A claim about effect needs a separate check of the current configuration, the stability and results of that run, and later incident reports. etcd issue #14045; etcd PR #14911

The v3.5.3 release announced fixes for critical correctness bugs that could lead to a committed transaction not being applied on all replicas after a server crash. That confirms a specific release outcome, not a claim that no data-consistency problem could ever occur afterwards. etcd v3.5.3 release notes

Why this example is useful. The primary report provides the technical mechanism, the boundaries of the impact, the level of uncertainty, the shortcomings of detection, and concrete actions. The issue and PR history provides a more independent check that at least part of the plan really was implemented and that its risks were discussed. But this is the case of one particular distributed store, not a template for all incidents, and not a causal study proving that every follow-up action was effective. The technical details of etcd, the task statuses and the behaviour of its tests may have changed; they were checked on 24 September 2026.

What a finished report should contain

No single order of headings fits every review. For an event that affected data or several teams, a minimal form might be:

1. Brief summary: what happened and what has already been done to contain the harm.
2. Impact: users, functions, data, time window, known boundaries and unknowns.
3. Timeline: event, detection, decision, action, result and recording time.
4. Background and mechanism: the concepts, versions and component interactions the reader needs.
5. Causes, trigger and contributing factors: every statement tied to evidence or marked as a hypothesis.
6. Detection: the signal, the gap, the delay and the first reliable way to see the impact.
7. Recovery: what worked, what caused delay, residual risk and side effects.
8. What helped, what hindered and where we got lucky: without dramatisation, each with concrete grounds.
9. Actions: owners, priority, acceptance and effectiveness criteria, link to the risk.
10. Communication: who has already been told, which data is private, who must approve external wording.
11. Post-change verification: date, method, actual result and the questions that remain.

A reviewer can run through a short checklist. Is it clear what exactly affected users? Is fact separated from estimate? Is it clear what participants knew at the time of each decision? Have times been reconciled with their zones? Are both detection and recovery analysed? Does the explanation avoid ending with a personal mistake? Does every important action have an owner and a criterion? Is there direct evidence that it was carried out? Has the reduction in risk been checked separately from closing the issue? If the answer to any of these is "no", add a limitation or an action; do not mask the gap with a polished conclusion.

Summary. A good postmortem is a verifiable model of how conditions in the system led to impact, how people noticed and contained it, and what will change next. A blameless tone helps surface more accurate information, but it does not remove the owner or the need to carry out the changes. The number of reports written or tasks closed does not prove that risk has fallen: each action has to pass its own acceptance criterion and a check of its effect.

Sources and further reading

• Google Site Reliability Engineering, John Lunney and Sue Lueder, Postmortem Culture: Learning from Failure, Site Reliability Engineering, chapter 15. Source for the definition of a postmortem, its goals, typical triggers, blameless analysis, prioritised actions and the sharing of lessons. The page shows no update date; accessed 24 September 2026. Open the chapter
• Google Site Reliability Engineering, Daniel Rogers et al., Postmortem Practices for Incident Management, The Site Reliability Workbook, chapter 10. An extended example report and the difference between weak and verifiable analysis. The page shows no update date; accessed 24 September 2026. Open the chapter
• etcd-io, v3.5 data inconsistency postmortem, author serathius, dated 20 April 2022. Primary technical description of the mechanism, impact, detection limits, actions and timeline; the facts used here are limited to the report's own wording. Checked 24 September 2026. Open the report
• etcd-io, Introduce etcd robustness tests, issue #14045, opened 16 May 2022. The list contains separate sub-items for regression reproduction of historical inconsistencies and for continuous release qualification. Checked 24 September 2026. Open the issue
• etcd-io, Create a nightly job for linearizability tests, PR #14911, merged 7 December 2022. An example of an implemented follow-up change and of the discussion about frequency and coverage limits; the merge itself is not evidence that the risk was fully removed. Checked 24 September 2026. Open the PR
• etcd-io, etcd revision occurs Inconsistent, issue #13654, opened 28 January 2022. Its description reports a revision inconsistency in production without establishing confirmed data corruption. Used to check the date and the limits of the claim in the postmortem's timeline. Checked 24 September 2026. Open the issue
• etcd-io, Release v3.5.3. The official GitHub page shows the date 13 April 2022; the postmortem timeline dated 20 April gives 24 April, so the sources contradict each other. The release notes state fixes for critical correctness problems that could lead to a committed transaction not being applied after a server crash; this is not a claim that no future defects exist. Checked 24 September 2026. Open the release notes
• Tom DeMarco and Tim Lister, Peopleware: Productive Projects and Teams, 3rd edition, Addison-Wesley Professional, 2013/2014. The publisher's page is used for bibliographic data and structure; the free excerpt of chapter 10 is used for a limited link between interruptions and focused work. The full text was not reviewed. Accessed 24 September 2026. Pearson page; Pearson excerpt
• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps, IT Revolution, 2018. The publisher's page confirms authorship, the research basis and publication data. The book was not reviewed in full; current claims about the five-metric model were checked against DORA. Accessed 24 September 2026. Publisher's page
• DORA, Learning culture and Generative organizational culture. Current explanations of the link between learning, an open flow of information and review practices; presented as a generalisation of research, not as a guarantee of causality in an individual case. Accessed 24 September 2026. Learning culture; Generative organizational culture
• DORA, DORA's software delivery performance metrics. The current five-metric model, the context for applying it, and caveats about targets and comparing different systems. Accessed 24 September 2026. Open the guide
• Anthropic, How Claude remembers your project, official Claude Code documentation. Supports only the statement about how project instructions are documented in Claude Code; it does not prove that a particular file was available in a given session. Checked 24 September 2026. Open the documentation
• OpenAI, Custom instructions with AGENTS.md, official Codex documentation. Source on Codex's instruction mechanism; its behaviour is not transferred to other agents. Checked 24 September 2026. Open the documentation
• OpenAI, Safety best practices, OpenAI API documentation. Used as a limited recommendation for applications built on the OpenAI API about checking results that have significant consequences; it is not a product guarantee and not an instruction for Codex or Claude Code. Checked 24 September 2026. Open the guide