Incident Response Procedures: Playbooks and Runbooks

During an incident, a team often knows only the symptom: messages arrive late, data is not updating, or some requests fail. The on-call engineer's memory may suggest where to look, but it does not guarantee that the next person will repeat the same safe steps. A prepared procedure moves verified knowledge out of one specialist's head and into a shared description that others can check.

Production Incidents covered live incident coordination: how to assess impact, assign roles, record facts and choose a bounded mitigation. This unit focuses on the material a team prepares before an event: how to describe the search for a cause, how to lay out a sequence of actions for a known state, how to state the limits of permissions and risk, and how to check that the procedure still matches the system. Postmortems cover the post-incident review and turning its conclusions into changes. Here it is enough to understand that a gap found in a procedure gets an owner and becomes a piece of work, rather than staying a note in a chat.

Learning outcome. After this unit, the reader can tell an investigation guide from a procedure for a known action; define the boundary of a safe launch; write a step-by-step description with a result check and a stop condition; connect it to an alert, an owner and an escalation path; and keep the procedure current through review, rehearsal and change control.

The terms playbook and runbook do not form a single mandatory vocabulary across the industry. In Amazon Web Services (AWS) operations documentation, a playbook helps investigate a problem and scope its impact, while a runbook describes a sequence of steps for a specific result. Google Site Reliability Engineering (SRE) uses playbook for instructions on responding to an alert, which may include both diagnosis and possible mitigations. Teams may use the terms differently. So first agree on what each term means in your product, and do not argue about the word when what matters more is understanding the safe action. AWS Well-Architected: Use playbooks to investigate issues; AWS Well-Architected: Use runbooks to perform procedures; Google SRE Workbook: What It Means to Be On-Call

Separate investigation from a known operation

A playbook is a scenario guide that helps you work out, step by step, what is happening: which signals to review, how to determine scope, whom to involve, what to communicate and under what condition to switch to another branch of the investigation. Its result may be a hypothesis, a refined scope or a decision to escalate. It must not pretend to be an exact recipe when the cause is still unknown.

A runbook is a documented procedure for reaching a specific state: for example, safely stopping a queue worker, restoring data from a checkpoint or switching a defined component to a standby configuration. In a simple case it is a short checklist; in a more complex one, a sequence with checks, approvals, expected results and branches. AWS frames a runbook as a documented process with steps, the required tools and permissions, error handling and escalation. The AWS recommendation also suggests storing runbooks centrally, assigning an owner, updating them as the process changes and testing them by asking another team member to follow the instructions. This is an example of AWS practice, not a universal rule for every organisation. AWS Well-Architected: Use runbooks to perform procedures

An incident response plan defines the wider system: impact classes, roles, permissions, communication, escalation channels and closing rules. Playbooks and runbooks usually implement part of that plan for a specific signal or type of work. A playbook might help check a suspected import lag, for instance, and a runbook might pause one of its stages or reprocess a safe batch. Having a link to a procedure does not remove the need to appoint a coordinator, as described in Production Incidents.

There is a connection here with Michael Gerber's The E-Myth Revisited. Official EMyth material describes three entrepreneurial roles — the technician, the manager and the entrepreneur: the technician does the work, the manager produces results through people and systems, and the entrepreneur sets the direction. One founder can move between all three roles; they are ways of thinking, not labels for different kinds of people. A runbook is a small piece of managerial work: it turns "I remember what needs to be done" into a process with boundaries, an expected result and an owner, so the team does not have to track down the founder every time. This is our own application of the model to operations, not a claim that the book prescribes a format for incident runbooks. Standardisation also does not replace the technician's judgement: if the actual state falls outside the procedure, the right move is to stop and escalate, not to improvise under the guise of a repeatable process. EMyth: Business Personalities — Entrepreneur, Manager & Technician

Artifact	Question it answers	Typical result
Response plan	Who coordinates the event, how is impact determined, and who has the right to make decisions?	A coordinated response across people and teams.
Playbook	What should be checked, and whom should be notified, when this signal or scenario appears?	A confirmed symptom, a bounded scope, a hypothesis or an escalation.
Runbook	How do you perform a specific known operation without missing a critical step?	A measurable state: the action is complete, the result is verified, or execution has been stopped.
Post-incident review	Which factors influenced the event, and which changes will reduce the chance of recurrence or the harm?	Owners and tracked improvements, including updated procedures where needed.

This boundary matters because a wrong diagnosis can turn a correct step from someone else's scenario into a dangerous one. If the team has not yet confirmed what exactly is broken, first run observable diagnostic checks and branch the scenario on their results. If the state is known and the action has been tested in advance, then the runbook can be started. When an operation is irreversible or may touch money, personal data, security or record integrity, its mere presence in a procedure is not permission: you need defined authority and a current decision by the responsible person.

Not every step should be automated. Google describes the tension between general playbook entries, which go stale more slowly, and precise steps, which reduce variation in what people do. A practical compromise is to use the same minimum fields everywhere and choose the depth of the instructions according to the risk and stability of the procedure. When a list has already become a deterministic sequence of commands that the on-call engineer repeats every time, it is worth considering safe automation. But a scenario with an uncertain cause needs decision points and human oversight, not a blind run of every possible change. Google SRE Workbook: What It Means to Be On-Call

Design the procedure around its result

A bad start for a runbook: "restart the service". Which process? Where? After which check? Who is allowed to do it? What confirms success? Is it safe to repeat? Without answers, the team may carry out the step mechanically and increase the harm.

Start from the result you need, not from the tool. "Make sure new messages are processed without duplication" is a verifiable result. "Click Restart" is only an action, and not necessarily a needed one. Stating the goal helps you decide whether the procedure fits the current circumstances and what sign tells you to stop.

A practical procedure should contain these fields:

Field	What to record
Name and identifier	A short name by which the on-call engineer finds the right scenario.
Purpose and scope	Which state or symptom the instructions cover; which similar cases they do not cover.
Trigger condition	The signal, observation or confirmed decision that makes the procedure appropriate.
Expected result	A state of the system, data or user flow that can be verified independently of the fact that a command was run.
Owner and review	The responsible team, the date of the last check and the event that should force a review of the text.
Prerequisites	Required access, tools, verified context, dependencies and environment constraints.
Roles and approvals	Who performs the steps, who authorises a risky change and who is informed.
Steps and checkpoints	One action or check per step; the expected result and the next decision.
Errors and escalation	When to stop the procedure, whom to involve and what evidence to hand over.
Post-execution check	How to confirm recovery, side effects, integrity and completion of the queue.
Source and version	A link to the system, configuration or tool the text was written for.

This is not a mandatory form approved by a standard. It is a synthesis of the fields needed for instructions to be understandable under stress. AWS documentation specifically emphasises the goal, the required tools and permissions, error handling, escalation and having another team member test the instructions. For playbooks, AWS also advises explicitly naming the scenario, the expected alerts, the communication channels and the escalation contact. AWS Well-Architected: playbooks; AWS Well-Architected: runbooks

Each step should leave the executor with a clear choice:

1. What to observe or change. Open the named dashboard, check the relevant segment, or change a bounded setting.
2. What the expected result should be. For example, the number of successful completions rises and the age of the oldest job falls.
3. What to do if the result is different. Do not carry on "according to plan" if a precondition did not hold; stop and go to the named owner.
4. Who is responsible for the next step. The assigned executor reports to the incident coordinator; the procedure text does not hand out commands to an undefined audience.

Observing the system, launching a change and checking it again are three different events. Record them separately. A message saying "command accepted" does not mean "the target state has been reached". If the tool provides no confirmation, state plainly how to obtain an independent signal.

Build safety into the steps themselves

During an incident people tend to read the verbs and jump into action. So dangerous assumptions must not hide between paragraphs. Write the precondition next to the action, not only in another document.

Distinguish checks from state changes

Mark whether a step only reads telemetry or changes production state. Low-risk checks are often useful first: identify the active version, look at the queue age, compare a checksum, check that a dependency is available. Resetting a password, switching traffic, replaying payments, deleting a queue or updating data are already state changes. For these, state the required approval and the stopping point.

Write conditions and limits of execution

Instead of "restart all workers", specify the component, environment, limit and verification method. Instead of "rerun the import", record the permitted range, how to avoid duplicates, the checkpoint and the condition under which a rerun is dangerous. Vague words — "approximately", "latest", "all", "normal" — need to be made precise if safety depends on them.

Name the stop conditions

The procedure must say "stop" when:

• the actual environment or version does not match the one stated;
• the required signal is missing or contradicts expectations;
• a step goes beyond the described service or region;
• the state of a write is unknown and a retry could create a second effect;
• the result gets worse after a bounded change;
• the time agreed for the check has run out and the next step is to escalate.

A stop condition is not a refusal to work. It prevents a known scenario from turning into a dangerous experiment when the system behaves differently from what was expected.

Define access; do not copy secrets

State which role is entitled to perform the step, where to obtain the permitted access and how to request it urgently. Do not embed a password, key, full personal record or long-lived account token in Markdown. Excessive permissions harm both normal operations and recovery after a compromise. If emergency ("break-glass") access is provided for, its use should have its own rules, auditing and a review after use.

Check repeatability and side effects

Do not assume that a timeout means a failed operation. If the remote system might have accepted the request, first check its state and its retry contract. Adding an automatic retry to a procedure without duplicate control can create a second payment, notification or record. The same applies to increasing the number of workers: it may shorten the queue but overload the database or a third-party dependency. The procedure must set a limit, an observable effect and a way to return to the previous safe state.

Prepare a fallback path and communication

If the main channel, region or documentation system might be unavailable during an outage, keep an independent way to find the escalation contact and the necessary instructions. For an emergency failover it is not enough to know the Domain Name System (DNS) control panel: you must decide in advance who confirms activation, where to check the data, how to notify the right people and how to decide that the system is truly ready. Microsoft describes this set as a combination of a clear runbook, a communication plan and an escalation path; the details of its Azure example depend on the architecture of a particular organisation. Microsoft Azure Well-Architected: Develop a disaster recovery plan for multi-region deployments

Connect the signal, the scenario and the owner

A useful procedure sits on the path of real work. Google recommends linking alert conditions to the corresponding playbook entries; for a data pipeline, it recommends documenting dependencies, processes and recovery steps, and adding the playbook link to the alert message. These are examples of Google's practice, not a claim that every team must automate the same set of signals. Google SRE Workbook: What It Means to Be On-Call; Google SRE Workbook: Improve and Optimize Data Processing Pipelines

An alert should lead to a specific document and help distinguish similar states. A link to "incidents" or to the home page of an internal wiki forces the on-call engineer to search when every minute counts. At the other extreme is a page where dozens of teams have collected every possible setting without saying which section relates to the signal. It is better to have a short entry point: the name of the symptom, the main playbook, an important alternative branch and the escalation contact.

Split the instructions by the most important decision, not by the name of the authoring team. For example, "the queue is growing but the provider system is accepting requests" and "the queue is growing and the provider is returning rate limits" may call for different actions. Each branch must explain which signal distinguishes it from the other and where to go next. If there can be many causes, the playbook should gather the most useful observations and lead to escalation, rather than masking the unknown with a dozen unconditional commands.

Find a balance between detail and how often the environment changes. Host names, buttons, access policies, console paths and alert thresholds go out of date quickly. A description of the principle ("check the state of every dependency needed for the write") may survive an interface change, but may be too abstract for a newcomer. An exact menu path reduces guesswork but goes stale once the product changes. The owner must choose a level of detail the team can maintain, and state when to check that it is still current.

A short index page can hold links to the owner, the dependency diagram, the playbooks and the runbooks. Do not keep the same rule in several places. If a link in the alerting system leads to an old copy, two mutually exclusive versions appear. Name a single source of truth from which copies or automated procedures take the current version.

Signs of a weak procedure

Defect	What can happen	How to fix it
Unclear trigger condition	Someone runs the right procedure for the wrong state.	Name the observable symptom, the scope and the conditions that rule this scenario out.
A step has no expected result	The executor moves on even though the system has not changed.	After an important action, add an independent signal and the decision it determines.
The instructions contain an action without an owner or authority	A person with access to the button treats access as permission.	Separate the executor, the person who approves the risk and the person who must be notified.
No stop condition	The on-call engineer repeats steps even though the actual state does not match the scenario.	Name the mismatch after which the change must stop and be escalated.
Links or contacts are out of date	During an event, people lose time searching for the right system or owner.	Check links and contacts as part of every service change and rehearsal.
The only check is that the command completed	The tool may have accepted the request while the user result or the data remained wrong.	Check the actual business action, side effects and processed records.

A common anti-pattern is adding another paragraph every time someone asks a question. The document grows but does not get easier to follow. If two steps differ because of an important condition, make that condition an explicit branch. If a rare variant requires an expert, say when to stop and whom to call, rather than burying the expert decision in a long list of steps.

Test procedures before an incident and after the system changes

A text seems clear to its author, who remembers the details that were left out. The first check is to have a colleague who did not write the instructions follow them. This is work for the team that runs the service, not an exercise for the reader. Watch where the colleague gets stuck on undefined fields, how they interpret the expected result and whether they know when to stop. AWS recommends exactly this kind of check; the US National Institute of Standards and Technology (NIST) notes that procedures can be exercised periodically to check their accuracy and train staff. The NIST Special Publication (SP) 800-61 Rev. 3 recommendation concerns responding to cybersecurity incidents in the context of the NIST Cybersecurity Framework 2.0 (CSF 2.0), so it should be applied in that context rather than presented as a rule for every operational procedure. AWS Well-Architected: Use runbooks to perform procedures; NIST Special Publication 800-61 Rev. 3

Testing does not have to start with a real outage. You can walk through the text in a test environment, hold a tabletop rehearsal without changing the system, check links and access, practise one critical step on a safe copy, or run a scheduled disaster-recovery rehearsal with a defined impact. The method depends on the risk of the operation. If the instructions touch data, payments, identities, routing or regional failover, a test in production without an agreed plan can itself become an incident.

Give the test explicit goals:

• whether the colleague could find the right procedure from the alert message;
• whether the owner, the trigger condition and the approving role are clear;
• whether the diagnostic steps can be followed without special verbal explanation;
• whether every change has an expected state and a way to observe it;
• whether the stop condition fired in a controlled wrong scenario;
• whether the procedure and the escalation path are accessible if the main region or corporate system is down;
• whether integrity can be confirmed after recovery, not just page availability.

Distinguish steps completed from result proven. A restored process does not necessarily mean that old jobs have caught up with the flow, that retries created no duplicates, that the data is intact and that the dependent feature meets its requirements. For a recovery procedure, state who checks the actual result and for how long they watch it.

Documentation must change together with the system. A runbook review can be included in the change that renames parameters or changes the data schema, an access policy, a traffic route or the alerting mechanism. An owner and a review date do not guarantee correctness, but they show who must check it after a significant change. If a procedure is no longer used, mark it as deprecated and redirect old links, rather than leaving two sets of instructions side by side without explanation.

Reviews are not triggered by the calendar alone. Re-check the relevant sections after a change to an alert, the dependency architecture, the data schema, permissions, an external provider, on-call contacts or the deployment mechanism. If a runbook invokes an automation script, the versions of the text and of the automated implementation must stay aligned. For a critical procedure, note the last successful rehearsal the team saw and what exactly it tested; a date on its own does not prove that the recovery path will work.

After each rehearsal, record the specific gap: not "the documentation is out of date" but "step 4 refers to the old name of the setting", "the backup team's contact is no longer valid" or "there is no data-completeness check after failover". Postmortems place these problems within the wider system of post-incident improvements.

Automate only when you control the consequences

Automation reduces repetitive manual work, but it does not make an action safe by default. A wrong condition, broad permissions, a repeated run or a partial error can turn one wrong step into a change for every user.

It helps to develop a procedure gradually:

1. Observation first. Create a link to the right dashboard, a prepared query or a clear list of what to check.
2. Then reproducible manual steps. Record the entry conditions, a bounded scope, the result and the stop criterion.
3. Next, safe execution without state changes. Automatically gather evidence or compute an action plan, but require a person to check the conclusion.
4. Then a narrow automated action. Start with a frequent, well-understood scenario with limited consequences; add explicit approval if needed.
5. Finally, automatic response where it has predictable limits. Define permissions, repeatability, limits, result checks, an event log and a way to turn off or stop the automation.

Diagnostic automation and automatic remediation are different levels of authority. Collecting metrics may be open to a wide on-call group, while deleting data or disabling an account requires a restricted role and approval. Make this separation visible in the procedure itself, instead of relying on a general promise that "the tool restricts everything".

When a set of commands is always the same, automation can be more reliable than copying and pasting. When a decision depends on an unclear cause, an uncertain state or business context, automate the observations and checks and keep the human decision explicit. Nobody should press "confirm" just because the system asks for it, without understanding the scope and the possible consequence.

Fully worked examples

The examples about a sports statistics service and a remote learning platform are invented. Their system names and numeric thresholds are illustrative; they are teaching scenarios, not product norms. The third example describes a PagerDuty event as it is presented in the openly available Google SRE Workbook.

Example 1: some sports data falls behind after an import error

Situation. A sports statistics service receives match results from an external provider through several processing stages. An alert reports that updates for one league have not appeared for longer than the team expects. The overall queue keeps working, so it is unclear whether a dependency has failed, one stage is rejecting a new format, or only one section has been delayed.

Investigation following the playbook.

1. Determine the scope: which leagues and time intervals the alert covers; whether other leagues are available; when the last complete result arrived.
2. Check the dashboard for each stage — ingestion, validation, normalisation, storage, publication — and the logs for one controlled example. Do not conclude from one faulty record that the whole import has stopped.
3. Compare the version of the incoming format with the accepted schema. Distinguish an unknown format change from a temporary delay or a lack of resources.
4. If the data is only delayed and its integrity is confirmed, announce the lag and monitor the age of the oldest item.
5. If records are corrupted or wrongly rejected, stop automatic publication of that specific bounded batch and bring in the schema owner and the technical coordinator. Do not extend the pause to other leagues without evidence that they have the same defect.

Runbook for reprocessing after the fix. Precondition: the cause is confined to the batch, and the fix has already been accepted through the appropriate change process. First, record the last confirmed checkpoint and the volume of records in which the problem was found. Check that operation identifiers protect against duplication. Prepare a small trial range without publication, compare counts and key fields with the source, and then obtain the data owner's approval for a controlled replay. After it, reconcile completeness, ordering, the absence of duplicates and the correctness of what users see. Only then widen the range and resume publication.

Observable result. The last processed time interval catches up with the source; control counts match; no invalid or duplicate records have appeared; the lag graph stabilises; the data owner confirms that the display matches the accepted format.

Why it is done this way. The Google SRE Workbook chapter on data pipelines describes the experience of Spotify's event delivery system: for incomplete data they used redelivery from the last good checkpoint, and end-to-end checks are needed because a healthy individual stage does not prove that the final data is correct. Here those bounded conclusions are applied to an invented system, not to a real internal product. Google SRE Workbook: Improve and Optimize Data Processing Pipelines

When this runbook does not fit. If the ordering in the source is unknown, identifiers may repeat, or incorrect records already published need to be withdrawn, replaying from the checkpoint may not correct the state on its own. Stop, preserve the evidence, determine the required recovery range with the data owner and create a separate recovery or reconciliation procedure. An import being available is not proof that its result is correct.

Example 2: failing over a learning platform during a regional outage

Situation. An online learning platform stores lesson progress in a primary region and has a standby region. A group of alerts shows that the progress application programming interface (API) has stopped responding in the main location. Losing one region does not by itself prove that the standby is ready to accept all writes. Some users can keep reading lessons they have already loaded, but saving results needs a separate check.

Playbook for assessment and activation.

1. The coordinator confirms that this is not an isolated client problem: they cross-check the progress API signals, the state of dependencies and user reports.
2. The disaster-recovery owner checks the activation criterion and authority, and the technical lead gives the coordinator a recommendation. If the threshold has not been reached, the team continues a bounded investigation; one graph is not enough to fail over.
3. The data owner checks replication lag and the maximum acceptable loss of progress defined for the product. The communications owner announces what exactly is available, without promising that all records are preserved until this has been checked.
4. The operations team prepares the standby region, confirms the sign-in, storage and queue dependencies, and checks that there is enough capacity for the expected traffic.
5. The coordinator approves the risky failover based on the technical lead's recommendation. An independent reviewer checks the boundaries and preconditions; the assigned executor moves a limited share of traffic. The team checks writing and reading of test progress, not just page availability.
6. If the state is stable and integrity has been verified, the team moves traffic over gradually and gives updates through the agreed channel. If the data does not reconcile, it stops increasing traffic, preserves the logs and escalates to the storage owner.
7. Separately, the team records the criteria for returning to the primary region. It does not switch back automatically just because the primary location responds again: data synchronisation and a safe ordering of writes must be checked first.

Observable result. A new session can complete a lesson and save progress; reading it back shows exactly that progress; errors and queues are not growing; the teams confirm that actual data loss is within the agreed threshold; operators know whether the service is in a temporarily limited mode or fully restored.

Why it is done this way. Microsoft's Azure guide to multi-region recovery explicitly separates declaring a disaster, preparing the standby, verifying data integrity, starting the application, switching traffic and closing; it advises planning health checks after failover and a separate plan for failing back. This is an example of planning for a specific cloud environment, and the scenario above is a generalisation. Another organisation may have a different technology, threshold or order of steps. Microsoft Azure Well-Architected: Develop a disaster recovery plan for multi-region deployments

When such a failover does not fit. If both regions depend on a shared authentication system, the standby lacks the needed capacity, or the data lag exceeds the acceptable limit, failing over may move the outage elsewhere, lose records or make the state worse. The alternative is to put reads temporarily into a limited mode, show a known delay or stop writes until the data owner decides. The availability strategy is an architectural decision made earlier; this procedure does not create standby capacity that the system does not have.

Example 3: when the runbook does not work — the PagerDuty clock incident

Situation and source. In a real example from the Google SRE Workbook, an on-call engineer at PagerDuty received an alert about clock drift on internal Network Time Protocol (NTP) servers. He checked the automatic recovery and performed the mitigation steps from the relevant runbooks, but the problem persisted and began to show up in services that depended on those clocks. The on-call SRE declared a major incident, and the Incident Commander brought the on-call engineers of the dependent services and customer support into a joint response. Over the following hours the existing procedures did not fix the problem, so the team methodically tested other recovery options. After the NTP server configuration was changed, the coordinator, together with the service owners, confirmed that the dependent systems had recovered and closed the active response. Google SRE Workbook: Incident Response

What this case shows. The runbook reduced guesswork and provided the first bounded steps, but it did not turn an unknown cause into a known fact. After declaring a major incident and bringing in the owners of dependent services, the team used the existing procedures and, when they did not restore the service, moved on to a methodical check of other options. It did not repeat the same actions endlessly: wider context and coordination helped it manage the changes. This is exactly the boundary every set of instructions should contain: "if there is no result, or the scale of impact has changed, stop this procedure and move to incident coordination".

Observable result. In the account given in the report, the coordinator confirmed with the service owners that the dependent systems had recovered, and then declared the incident over. This does not prove that every company needs the same set of roles or tools. It shows the runbook's role as an initial, bounded aid within a wider response, not as a substitute for a responsible decision.

Example 4: reviewing a runbook from a public GitHub repository

Situation and source. To see the difference between a useful structure and instructions ready to copy, consider webapp/OPERATIONS.md from the public repository aws-samples/sample-devops-agent-orchestrator. The pinned version is commit a857b7b006b6a3e4f2773627fcefd57f76d66167 from 13 August 2026; it was reviewed on 24 September 2026. This lets you find exactly the text we read, even if the main branch changes later. The file at the pinned commit

What the procedure covers. The document lists the deployment prerequisites and the required AWS permissions, and describes configuration values, hosting settings, two deployment stages and a separate data-refresh procedure. Only an administrator can trigger the refresh; the document also explains how to poll the status, not to start a parallel refresh, and how to recognise success: Last_Sync_Date changes after success and stays the same on failure. Typical symptoms and a first direction for diagnosis are given separately. As a teaching example, it shows well that prerequisites and result checks can sit next to the procedure rather than living in someone's memory.

How to check fitness for use. These are instructions for a specific sample on AWS Amplify, not a universal scheme for another system. Resource names, account identifiers, the region, role permissions, the branch and the configuration must match the target environment. Before any real use, an operator would have to reconcile these values with the code, the current permissions and the service owner; the fact that a command can be copied from GitHub is not permission to run it.

What remains weakly supported. The reviewed file does not name a responsible owner for the runbook or the date of the last exercise or successful test of the whole procedure. After a refresh failure, the document advises reading the stage message and checking the logs, but does not say under what condition to stop, whether it is safe to rerun after partial success, or to whom to escalate. Nor does it define a final check of the quality of the resulting data or of the user path. Rollback is described as redeploying the previous commit for the front end and back end together; before applying it, a team would need to check version compatibility, the side effects of back-end changes and the sign of recovery after rollback. In addition, APP_ORIGIN is listed with a default value of *, and setting the exact origin is suggested after the first deployment. This is a warning for the reader: a sample's default value should not be taken as a finished production decision.

Conclusion. This version has concrete prerequisites, a refresh sequence, a status signal and short actions for typical errors. But the file contains no evidence of a current rehearsal, a named owner or a complete criterion for business recovery. The age of the source matters too: on the review date, the last commit that changed this particular file was about six weeks old. So the sound conclusion is to borrow the way it is structured and the list of questions for checking, but not to carry ready-made values, permissions or steps over to another environment. The file's change history

Summary

A good procedure turns verified knowledge into a repeatable action, but it does not pretend that every situation is known in advance. A playbook guides investigation and decisions; a runbook describes a known operation with a defined result; an incident plan sets out general roles, permissions and communication. The terms depend on the team, so their local meaning must be stated.

Start with the result; set out the prerequisites, the owner and the boundary of the action; show the expected signal and the stop condition. Test the instructions with someone who did not write them; update them after the system changes; rehearse in a safe mode; measure not whether a command ran but the state of the user flow and the data. Automate only those steps for which the permissions, consequences, limits and a way to verify the result are known.

Sources and further reading

• Amazon Web Services, OPS07-BP03 Use runbooks to perform procedures, AWS Well-Architected Framework. Supports the definition of a runbook, the procedure fields, errors, escalation, central storage and testing by another person. Accessed 24 September 2026. Open source
• Amazon Web Services, OPS07-BP04 Use playbooks to investigate issues, AWS Well-Architected Framework. Supports the diagnostic role of a playbook, its link to runbooks, and defining tools and escalation. Accessed 24 September 2026. Open source
• Amazon Web Services, SEC10-BP04 Develop and test security incident response playbooks, AWS Well-Architected Security Pillar. An example of playbook structure for security events; to be applied in the context of AWS security scenarios. Accessed 24 September 2026. Open source
• Google Site Reliability Engineering, What It Means to Be On-Call, SRE Workbook. Supports the link between an alert and a playbook, on-call training and the trade-off between general and step-by-step entries. The page shows no separate update date; checked 24 September 2026. Open source
• Google Site Reliability Engineering, Incident Response, SRE Workbook. The public PagerDuty example involving NTP, the escalation sequence and the limits of ready-made procedures. The page shows no separate update date; checked 24 September 2026. Open source
• Google Site Reliability Engineering, Improve and Optimize Data Processing Pipelines, SRE Workbook. Supports end-to-end data quality, checkpoints, reprocessing and documenting dependencies; describes Spotify's event delivery system, presented as a case study in the Google SRE Workbook. The page shows no separate update date; checked 24 September 2026. Open source
• National Institute of Standards and Technology (NIST), SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management, April 2025. Supports testing procedures and preparing playbooks for common and critical cybersecurity scenarios. Publication page
• Microsoft, Develop an incident management practice to recover from disruptions, Azure Well-Architected Framework. Supports roles, escalation criteria, authority, the failed-deployment scenario and the need to test procedures; the product examples are specific to Azure. Page updated 26 November 2025; accessed 24 September 2026. Open source
• Microsoft, Develop a disaster recovery plan for multi-region deployments, Azure Well-Architected Framework. Supports the stages of failing over to a standby and failing back to the primary environment, checking data and dependencies, rehearsals and runbook availability during an outage; the concrete examples concern multi-region Azure architectures. Accessed 24 September 2026. Open source
• AWS Samples, Operations Runbook — DevOps Observatory Web App, sample-devops-agent-orchestrator/webapp/OPERATIONS.md, pinned commit a857b7b006b6a3e4f2773627fcefd57f76d66167, 13 August 2026. A public example with AWS prerequisites, configuration, deployment, a refresh operation, status checks, rollback and first actions for typical errors; the text evaluates the limits of this version and the missing evidence of a current rehearsal. Checked 24 September 2026. Open the pinned file
• EMyth, Business Personalities: Entrepreneur, Manager & Technician, the company's own material on the three roles, repeatable systems and working on the business. Used as an accessible explanation of Gerber's idea; the model is applied to runbooks as our own editorial synthesis, not as a full summary of the book. The page shows no separate publication date; checked 24 September 2026. Open source