Observability: From Signal to Explanation

Non-Functional Requirements argued that product quality has to be described through a person's experience: did they complete the action they needed, how long did it take, how fresh was the data they saw. The units on continuous integration, continuous delivery and deployment, environments and deployment strategies showed how changes pass their checks and reach an environment, and Rollback and Recovery explained why reverting code does not always revert data and external side effects. This unit picks up after the release: how to tell an ordinary change in load from a genuine degradation, find the stretch of a request's path where things went wrong, notice stale data, and give the responsible person a signal they can act on.

Learning outcome. After this unit the reader can start from the consequence for the user, choose the metrics, logs and traces that will test an assumption, tell a symptom from a possible cause, and write a useful alert. The reader also understands why "the server responds" does not mean "the product works correctly", and why an agent may analyse the evidence it is permitted to see but must not change how a production system runs on its own initiative because of a coincidence in time or an incomplete log.

Here observability serves one purpose: understanding the state of a service and making decisions about how it runs. Metrics of development effectiveness, delivery speed and agent usage belong to Agent Observability and Delivery Performance. Coordinating people during a real incident is the subject of Production Incidents; ready-made procedures and learning after the event are covered in Incident Runbooks and Postmortems.

A green signal is not yet proof of success

Monitoring checks for conditions defined in advance: whether the web server responds, whether a queue has crossed a known threshold, whether a scheduled import finished. Observability helps you investigate the state of a system when the cause of a problem is not known beforehand: an engineer can line up events, measurements and the path of an individual request to check what was happening inside the components. This is a useful working distinction, not a claim that every tool belongs to only one category: the same system can collect telemetry both for routine monitoring and for diagnosing new situations. The distinction matches the description from DORA (DevOps Research and Assessment): monitoring relies on a predefined set of metrics or logs, while observability is about exploring properties and patterns that were not known in advance. DORA: Monitoring and observability

A system can serve its login page with a 200 status over the Hypertext Transfer Protocol (HTTP) even though no new import has arrived for hours. That successful response describes an exchange between client and server, not whether the data on the page is complete or current. Checking that a route's address is reachable answers a narrow question: "Does this page open?" It does not answer the product owner's question: "Does the reader see the current team line-up?"

So do not start from a list of charts; start from the consequence. For example:

• A person cannot complete a payment — count failed payment attempts, not only exceptions in the server process.
• Data must be updated before kick-off — measure the age of the last confirmed import, not only whether the page is available.
• A user expects a report by the end of the working day — watch the time from accepting the request to the finished file, not only how fast the job is put on the queue.

A signal with no defined action often turns into noise. If a value changes but nobody knows whom to tell or what to check, the mere existence of the metric does not make the product any easier to manage.

Three kinds of signal, three different questions

A trace is a linked record of one operation's path through the components; its parts are described in more detail below. OpenTelemetry (often shortened to OTel) is an open set of specifications and tools for collecting and transmitting telemetry. This unit looks at three common signals: logs, metrics and traces. OpenTelemetry also describes baggage for passing context along, and a separate Profiles signal is at an early stage of development with Alpha status. These signals can be linked to one another, but they are not interchangeable. OpenTelemetry: Signals, Profiles: Alpha status

Signal	Question it answers	Example
Log	What event happened, in what specific context?	The import finished partially and rejected 18 records.
Metric	How often, or how strongly, does a property of the system change?	Over the last 5 minutes 92% of imports succeeded; the data is 74 minutes old.
Trace	What path did a specific operation take, and where did the time go?	The report request waited 1.1 seconds on the access-policy service.

"Three" does not mean every system needs a full, expensive set for every component. A small internal tool may get by with a few metrics and structured logs. But if one user action passes through a queue, a background worker and several remote services, a trace can dramatically speed up finding the boundary where behaviour changed. The choice is driven by risk and by the decision you need to make, not by fashion in tooling.

Logs: short context instead of a pile of text

A log is a sequence of records about events: a request was accepted, a check passed, a call to a dependency timed out, an import was committed. A record is useful when it answers a specific question and carries enough context to find the operation it belongs to.

A structured record usually contains the time, a severity level, the service name, the event name, the version or revision, the identifier of the request or background job, and a few safe dimensions. The labels DEBUG, INFO, WARN and ERROR are a team convention about how important a record is, not a universal standard that automatically determines the response. Times are best kept on a single timescale; the examples below use Coordinated Universal Time (UTC).

Below are some invented, shortened lines in JSON (JavaScript Object Notation, a text format for structured data). An identifier (ID) lets you link records without exposing the content of the request:

{"timestamp":"2026-09-24T13:04:10Z","level":"INFO","service":"feed-worker","event":"import.started","release":"2026.09.24.3","job_id":"job-42"}
{"timestamp":"2026-09-24T13:04:18Z","level":"WARN","service":"feed-worker","event":"source.timeout","release":"2026.09.24.3","job_id":"job-42","attempt":2,"duration_ms":8000}
{"timestamp":"2026-09-24T13:04:19Z","level":"ERROR","service":"feed-worker","event":"import.failed","release":"2026.09.24.3","job_id":"job-42","reason":"source_timeout"}

These records read as a short chain: the job started, the second call to the source exceeded its 8-second timeout, and the job failed. If you add the client's exact address, the full Uniform Resource Locator (URL) with its parameters, the whole request JSON, authorisation headers and dozens of repetitions of the same text, understanding may not improve at all, while privacy risk and cost go up.

In a real system, what helps most when searching is usually not just the time and the error text but stable fields: which component, which environment, which revision of the program, which type of operation, whether it finished, how long it took. A version or release field helps you line an event up with releases of the kind described in Deployment Strategies, but it is not proof that this particular release caused the error.

Do not rely on the request ID alone. A background job may have no originating web request, and a message that is redelivered will have several attempts. There you need a job or message identifier of its own, an attempt number and the outcome of each attempt. For a long-running process, the log should let you reconstruct the main state transitions without copying the whole content of the data that was passed along.

Metrics: measure the distribution and name the window

A metric is a numeric measurement of a system's behaviour over time. Instrumentation is the code or library that makes a program produce such measurements and signals. OpenTelemetry provides an application programming interface (API) and software development kits (SDKs); the maturity of individual signals differs by programming language, so before you use one, check the status and documentation of that signal for your SDK version. OpenTelemetry: Language APIs & SDKs When a metric's name does not explain exactly what it counts, where the value comes from and over what period it is aggregated, the chart is easy to misread.

• A counter increases as observations come in: for example, the number of completed imports or of errors. To get a rate of events you take the counter's increase over a time window, such as events per minute. The absolute accumulated counter on its own does not show the current error level.
• A gauge can go up and down: queue length, number of active connections, age of the oldest job.
• A histogram stores an aggregated picture of observations: the count, the sum, and counters of measurements falling into defined buckets. It lets you estimate the shape of a distribution, such as response times, without recording every measurement as a separate line.

Resource saturation describes how busy a limited capacity is: for example, how much room is left in a connection pool, or how much of the time the processor was busy. A utilisation percentage on its own is not always a problem; what matters is whether the waiting queue, latency or failed user actions are rising along with it.

Teams often watch four families of indicators: request rate, latency, error rate and resource saturation. But these categories do not replace measuring the product outcome. Low processor usage can coexist with an empty stream of new data, and an average response time can miss a slow minority of requests.

Site Reliability Engineering (SRE) is a body of practices for keeping services running reliably; the name is also used for a series of Google's educational materials. A percentile describes the value below which a given share of measurements in a chosen set falls. The 95th percentile (p95) is the value that 95% of measurements do not exceed; the 99th percentile (p99) is the same for 99%. They do not say that every request took exactly that long. The arithmetic mean answers a different question and can hide a long tail: many fast requests pull it down even when a small share of people wait much longer. Google SRE: Service Level Objectives

The window matters too. "p95 is 1.2 seconds" without a time range, an operation type and a number of measurements is an incomplete statement. With 10 measurements, p95 almost coincides with the largest value and is unstable; a larger sample reduces the estimation error, provided the requests represent comparable load and a comparable time period. A five-minute window reacts quickly to change but can fluctuate; an hourly one smooths out the fluctuations but delays detection. The right window depends on how quickly a degradation hurts the user and how much data you need for a meaningful estimate.

A histogram does not give unlimited precision either. If a percentile is estimated from buckets, the result depends on their width and placement. Histograms with identical bucket boundaries can be merged, but changing the boundaries over time can make comparisons invalid; and from percentiles that were computed in advance you often cannot correctly recover the percentile of a combined group. So state the limits of precision and check the aggregation type in the specific tool you use. Prometheus: Histograms and summaries, OpenTelemetry: Metrics data model

Matching a question to a metric

Question	Example metric	What else you need to know
Are requests completing?	share of valid actions that succeed	which actions are in the denominator; whether rejections and timeouts are counted
Is the system responding in time?	p95, or the share of requests under a set time	route type, client-side or server-side time, the window and the number of requests
Is background processing falling behind?	age of the oldest event, or the delay from arrival to processing	what "processed" means; whether the data has already been committed
Is the risk of overload growing?	queue, active connections or resource usage	how much of the resource the system really has; whether there is headroom and there are limiters

All values in the examples that follow are invented teaching numbers, not recommendations or measurements of real systems.

Traces: one path through several components

A trace is a sequence of linked operations that together form the path of one action. A single segment of that sequence is called a span: the work of one component, such as the gateway handling the request, a database read or a call to an external service. A child span shows work that its parent span started. Their time boundaries let you see where the waiting was longest and which pieces of work overlapped.

A schematic trace for loading a report might look like this:

client → gateway API (1,310 ms)
                └─ reports application (1,274 ms)
                     ├─ storage (27 ms)
                     └─ access-rights service (1,120 ms)

The total time is close to the sum of the sequential waits plus overhead; if components work in parallel, a simple sum will overstate the total latency. So do not add up the figures of individual spans mechanically unless you have checked how they nest and overlap on the timeline.

To see a single operation across several services, systems pass trace context across the boundaries. OpenTelemetry describes a context that contains the trace ID and the parent span ID; the typical mechanism carries it in headers. A log record can store the trace and span IDs so that the matching records can be opened next to the trace. Context propagation — passing these links between processes and services — is only as useful as every link in the chain that preserves it; a boundary at an external provider or in a queue can break the causal chain. OpenTelemetry: Context propagation

A trace should not become a copy of the user's request. To find latency you usually need the route, the operation type, the outcome, the revision, the dependency and a few safe attributes. The full request body, a password or payment details add nothing to a span's timing and can end up in more systems and in front of more people. The context passed between services is not a place for secrets either: the recipient of the next hop may be less trusted, and service attributes can be kept in logs. OpenTelemetry: Context propagation security, OpenTelemetry: Handling sensitive data

Not every request will have a stored trace. Sampling keeps only a portion of traces to control data volume and cost. Deciding at the start of the path is cheap but cannot yet know whether the request will end in an error or a long delay; deciding after completion can keep such traces, but it needs buffering and state in the processing system. The sampling policy should fit the task: keep enough ordinary requests for a baseline, and have a deliberate way of not losing the rare slow or failed cases. Keeping every error simply cannot be guaranteed if the decision is made before the error is known. OpenTelemetry: Sampling

Combining signals without mistaking coincidence for cause

A practical search sequence looks like this:

1. A metric shows the symptom. Identify the type of operation where errors or latency rose, and the period when it happened.
2. Narrow the set of requests. Pick a few representative operations by time, route, environment or revision; do not try to open every log.
3. Read a short chain of logs and the trace. Match up the request ID, the trace, the events and the durations of the child spans.
4. Form a hypothesis. For example: "the delay comes from the access-rights check service".
5. Look for what could refute it. Are calls to the access-rights service slow in every revision? Do the region, report type, data or timeout differ? Did retries, or the queue in front of this particular dependency, grow?
6. Change only one factor, or check an independent piece of evidence. A release, an error and a restart coinciding in time is a reason to investigate, not proof of causation.

If these signals are built on a shared context, the metric gives the scale of the problem and the logs and traces give concrete examples for testing the mechanism. If context is not propagated, you will have to match up time windows, versions and other fields — and the conclusion will be weaker. Say so explicitly.

Worked example: latency after a release

In a teaching customer portal, after revision 2026.09.24.3 is released, the team sees the monthly report opening slowly. All numbers below are invented. Metrics over five-minute windows show this change:

Period	Requests	Error rate	p95 full response	p95 storage read
Before release	10,000	0.2%	240 ms	31 ms
After release	10,000	2.4%	1,360 ms	33 ms

Each five-minute window contains 10,000 valid requests, so the error rates mean 20 and 240 failures respectively. The p95 is computed over completed responses; timeouts are counted separately in the error rate. The average latency moved only from 170 to 215 ms, because most reports still open quickly. The percentile and the error rate show that the problem is concentrated in a minority of operations. But the metrics on their own do not yet say whether the cause is new code, different load, an external dependency or a partial failure.

The team takes several slow requests from the relevant time slice and compares their traces. In the traces of slow reports the storage read takes about 30 ms, while the child span for the call to the access-rights service takes 1.0 to 1.2 seconds. The short structured logs contain the time, route, version and final outcome, but not the content of the report:

{"timestamp":"2026-09-24T13:05:14Z","level":"WARN","service":"report-api","event":"dependency.slow","dependency":"access-policy","release":"2026.09.24.3","trace_id":"7fa4…","duration_ms":1120}
{"timestamp":"2026-09-24T13:05:14Z","level":"INFO","service":"report-api","event":"report.completed","route":"monthly-summary","release":"2026.09.24.3","trace_id":"7fa4…","duration_ms":1274}

This localises the delay, for these traces, to waiting on the dependency, but it does not prove that the release caused it. The team checks the same dependency for the previous revision and for other routes over the same interval. In the teaching outcome, the access-rights service's response time is elevated for both revisions, and the slow cases tend to coincide with an increased number of repeated calls to it on the monthly-report route. That weakens the initial story, "the new release is simply slow"; an engineer still has to find out why the dependency responds slowly and why this particular route makes more calls.

The team's action is not to declare the diagnosis complete, and not to remove the access-rights check for the sake of a faster chart. The dependency's owner receives a sample of traces and a comparison of routes; the product owner sees the scale of impact and the share of failed reports. Once the cause is confirmed, the team can choose a fix or a workaround. If traces are sampled, the results describe the examples that were observed, not guaranteed all slow requests. If the system has no per-dependency metric, or the relevant spans were not kept, you can localise the problem only to the remainder between the total time and the known segments.

When the page opens but the data is stale

For background processing, the simple question "is the server working?" is often asked at the wrong level for the product. What you need to measure is:

• the time of the last successful and committed completion, not just the last run;
• the age of the data currently being shown to the user;
• the age of the oldest event in the queue and the number of events waiting;
• replica lag, if reads come from a copy of the database;
• the number and type of rejected records, if the import finishes partially.

These indicators answer different questions. An empty queue does not prove the import is fresh: the scheduler may never have created the job. A new job completion does not prove completeness: the importer may have rejected some of the records. A "just updated" timestamp does not prove correctness either: the import may have accepted a new but wrong response from the source. Compare rejected rows and the expected versus actually committed record counts, and for critical fields check value validity and control totals. A record can be committed in the primary database but not yet visible on the replica the page reads from. So define the boundary of the product outcome: "synchronised" might mean that the source responded, that the message was read, or that the required data is actually stored and available to the reader.

Worked example: the site opens, the sync has stopped

Consider an invented football statistics site that receives line-up and result updates from an external source. The pages open, the server responds, the overall share of successful HTTP requests is 99.8% and the p95 load time is 190 ms. Yet the morning line-ups should already be fresh, and the last successful, committed data update happened 81 minutes ago.

In the original product assumption, the owner agreed on a freshness limit of 40 minutes for data about scheduled matches. These thresholds are conditions of the teaching example, not a universal target. The signals provide additional facts:

Indicator	Teaching value	What it points to
Age of last successful save	81 minutes	the user sees data older than the agreed limit
Import jobs in the queue	0	the scheduler has not put new work up for processing
Rejected rows in the last import	0	unknown whether there is a format problem, but this signal does not explain the pause
Read-replica lag	2 seconds	the replica alone does not explain an 81-minute gap

The team compares the most recent events and finds that after a schedule change the scheduler skipped two expected cycles and created no import jobs. Meanwhile the background worker's liveness check shows that the process is running, and the import queue is empty precisely because no new work has arrived. The cause was neither "the server is down" nor "the worker is stuck on the queue": it was an unnoticed failure in how the scheduler started after a configuration change. In this invented story the operator restores the scheduler, then checks the time of the last save and a control sample of line-ups. Only after a new successful commit of changes to storage and a read of the updated data does the freshness age return below the agreed limit.

A useful alert reports the consequence, not an incidental internal symptom:

Line-up data has not been updated for 81 minutes; the agreed limit is 40 minutes. Owner: import team. Next action: check the scheduler's last run and confirm a successful save of the new sample.

The "40 minutes" threshold is an assumption of this teaching case. In a real product it is determined by how often the source updates, the consequence for the reader, the import delay and the acceptable recovery time. For data about a match that has already been played, a stale minute may hardly matter; for a line-up change before kick-off, even a shorter delay can change how useful the feature is. If the data legitimately updates only a few times a day, an alert every 40 minutes will only create noise. If there is no agreed rule, the system should show the user how old the data is and require a decision from the product owner, rather than pretend there is a universal number.

This difference between server availability and the state of the data matters for systems with replication, queues, imports and derived datasets. Designing Data-Intensive Applications, second edition, gives broader context for these trade-offs through its treatment of distributed systems and data processing. The numbers and the specific teaching example above were created for this unit; they are neither a retelling of a case from the book nor a quotation. The author's page confirms the current second edition (O'Reilly Media, March 2026). Designing Data-Intensive Applications, 2nd edition, author's bibliography page

Measurement, objective and external reliability promise

When talking about a system's reliability, it helps to distinguish three roles:

• SLI — service level indicator: a measurement of what the user experiences, for example the share of successful exports.
• SLO — service level objective: an internal or agreed target for that indicator over a period.
• SLA — service level agreement: a commitment to a customer that may define consequences or compensation if it is not met.

The terms are related but not identical. The SLI says what was measured; the SLO says what the product wants to achieve; the SLA says what was agreed with the customer. A team may have an internal objective stricter than the contractual one, or a contract whose specific terms and calculation method differ from the internal indicator. Google SRE stresses that choosing an objective is not a purely technical matter: it has to take the product's expectations and risks into account. Google SRE: Service Level Objectives

A teaching example: a team reports on the share of user-accepted exports that complete within two seconds. The SLI has to define which attempts go into the denominator: all valid requests to create an export, including server errors and timeouts. The SLO might read "99.9% of such requests over 30 days complete correctly within two seconds". That numeric wording is an invented example, not a recommendation. The SLA has to be checked against the contract: it may use a different time limit, period, exclusions or compensation.

If the SLO is 99.9% for the chosen population of actions over 30 days, the gap to 100% — 0.1% — is the allowed share of non-conformance under that definition. An error budget is a way of tracking how much of that allowance has been used up over the defined period. It helps make product and operational decisions; it is not permission to make the user's experience deliberately worse. Decisions to pause releases or invest in reliability are made by the responsible owners, taking consequences and evidence into account. The Google SRE Workbook shows how the burn rate — how fast the budget is being spent — can drive useful alerts across several time windows; its value has to be read together with the number of requests, the consequence and the SLO period itself. Google SRE Workbook: Alerting on SLOs

A metric that does not reflect the outcome the user needs breeds false confidence. For example, a successful HTTP response to a request to create a report may only mean the job was accepted onto the queue, not that the report is ready to download. If the SLI counts job acceptance but the product promises a finished file within a certain time, the indicator has to cover the path to the finished file, or measure the queue-to-completion delay separately.

An alert that has a next action

An alert should answer: what happened, whom it affected, which threshold or window was breached, who is responsible, and what to check first. The number of alerts is not a measure of monitoring quality. If every ERROR record sends a separate notification, a single failure can bury the signal that really needs the operator's attention. Google SRE explicitly warns that noisy pages wear people out and can delay recovery; the Prometheus guidelines recommend alerting on the user-facing symptom and not creating a page that nobody can act on. Google SRE: Monitoring Distributed Systems, Prometheus: Alerting

Depending on severity, a notification may go to the on-call engineer, to a work queue, or only to a log. For each important alert, define:

• Condition: the indicator, the acceptable threshold, the measurement window and how long to wait before signalling. A short one-off spike may be expected, and for a background job the threshold often depends on its schedule.
• Consequence: which user action is not working, or which data is becoming incomplete.
• Owner: the team or specific on-call role that can take the first step.
• Deduplication and grouping: which repeats are manifestations of the same problem, and over what interval they are merged.
• Escalation: whom to hand the problem to if there is no response within the agreed time.
• Investigation hint: a link or short instruction for where to find the metrics, recent logs and a representative trace.

Worked example: thousands of notifications about one failure

In an invented import, the external source stops responding. Within two minutes the service has written 4,800 identical timeouts and started retrying the request with a delay between attempts. If every event generates a page to the on-call engineer, that person will see hundreds of copies instead of the scale of the problem and the next step. They may spend their time closing alerts rather than checking the state of the source, the retries and the freshness of the data.

The team keeps all the necessary records in the log but builds a single grouped alert on stable fields: event type, service, dependency and environment. The request, the trace or an individual user's ID are not used as the grouping key: they would turn every retry into a separate alert. The notification keeps the number of events, the first and last time, the number of attempts, the age of the data and a few short links to representative logs. One alert goes to the import's owner and suggests the first safe action: check the source's availability and the retry limits.

For this teaching example, a failure-rate page fires if, over a rolling 10 minutes, failed valid updates exceed 5% of all the updates that should have been performed in that window; an overdue job, or one that was never created, also counts as a failure. The rule applies only when there are at least 20 expected updates, and the condition has to hold for two minutes before the page goes out. A separate freshness alert fires when the age of the last successfully committed import exceeds the product limit of 40 minutes; it works even when the scheduler has created no jobs at all. Repeats are grouped for ten minutes by service, dependency and environment. If the on-call engineer does not acknowledge the page within five minutes, the shift lead receives it. All these numbers are invented and are not a universal configuration.

Deduplication must not hide a different impact. For example, a simultaneous loss of payments in a second region, or corruption of a different class of data, may need its own alert even if the underlying dependency is the same. So group by the cause being checked, but keep separate the independent user-facing functions and owners. Notifications can be muted while a confirmed fix is in progress, but that silence must have an owner, a time limit and a visible state.

Cost, privacy, cardinality and retention

Telemetry is also product data: it has to be collected, transmitted, indexed, protected and deleted according to clear rules. OpenTelemetry cannot decide on its own what is sensitive in a particular product; responsibility for minimisation, access and reviewing instrumentation lies with the team. OpenTelemetry: Handling sensitive data

Cardinality is the number of distinct values of a field or combination of fields. In metrics, each unique combination of labels can create a separate time series. A label with a small set of values, such as the environment or the route type, can be useful; a label carrying a request ID, an email address or a user ID grows quickly and is expensive to index. The Prometheus documentation explicitly warns against labels with unbounded sets of values, such as user IDs and email addresses. To find a specific operation, use log or trace context with an access and retention policy, rather than turning every request into a new metric series. Prometheus: Metric and label naming

A large number of attributes increases volume for the other signals too, although the cost mechanism differs: logs are expensive to store and index, traces to ingest, store and search, and metrics with many labels multiply time series. Start with the question "What decision will this field make possible?" If a field does not help you detect a problem, narrow it down or confirm an action, its benefit may be smaller than its cost and risk.

Practical minimisation rules:

1. Do not record tokens, passwords, integration secrets, full payment details or the private content of requests.
2. Instead of the full event body, store the type, size, validation result, controlled attributes and a link to a source with restricted access.
3. For personal data, use the least sensitive format the investigation needs; a pseudonymous identifier can still be personal data.
4. Separate access roles: not everyone who can view a chart should be able to read private logs or the raw content of transmitted data.
5. Set retention periods according to usefulness and risk. A short-lived diagnostic record and an aggregated performance history can have different retention periods, if the product's needs and the organisation's rules allow it.
6. Review not only your own code but also the instrumentation libraries: an automatic collector can add attributes the team never planned to store.

Sampling helps keep costs down, but it has a price. If only a small share of random traces is kept, you may never see a rare error; if the decision to keep a trace is made at the start of the request, it is not yet known whether the request will be slow. Choosing after the trace completes can keep slow and failed examples, but it needs a buffer and separate attention to the collector itself. Regulatory conditions, low signal volume or the need for complete accounting can make sampling unsuitable. Do not apply sampling to basic quality counters as if they still counted every valid action: if sampling affects an SLI, the estimation method has to account for it and be verified. OpenTelemetry: Sampling

Watch the watcher

Telemetry collection can fall behind, drop records, lose access to storage, or lack context for a particular component. So it pays to monitor the signal's own path as well: the time of the last successful export, the size of the telemetry queue, a counter of dropped records, the collector's availability and ingestion errors. Otherwise silence on a chart can mean either "nothing happened" or "we cannot see what is happening". An additional external check of the monitoring system's availability helps catch the situation where the internal system can no longer report its own failure. Prometheus: Alerting — metamonitoring

A real example of the limits of telemetry is the Cloudflare data-centre failure of November 2023. Cloudflare describes how its tooling did not detect the switch to generator power; the first indication of a problem for Cloudflare was two routers disappearing at 11:44 UTC. The company had expected that during a complete loss of the site, logs would accumulate at the edge and arrive later, but the log processing and analytics systems also depended on services in that site. A complete shutdown of the whole site had never been tried in earlier tests, so the hidden dependencies never showed up. The report contains no numeric chart of queue age and no universal alert threshold; those should not be invented. The case shows two concrete gaps: the change of power source did not produce the signal that was needed, and part of the planned telemetry depended on the very failure it was supposed to report. The company itself marked part of its explanation of the electrical cause as an unconfirmed assumption. Cloudflare: Post mortem on the Control Plane and Analytics Outage, 4 November 2023

Real cases suggest questions to ask about your own system, but they do not hand you a ready configuration for someone else's service. Decide which independent signal will tell you that an important observation channel has been lost, and where that signal will be stored if the main site or the telemetry system is unavailable.

What an agent reading the signals can and cannot do

An agent can line up the revision, logs, metrics and several traces, prepare a short timeline and propose testable explanations. For example: "After version 2026.09.24.3, latency on the monthly-report route went up; in the selected slow traces most of the waiting was on the access-rights service; the previous revision at the same time also shows slow calls. That weakens the hypothesis of a regression in the whole version. Next check: compare the number of dependency calls per route." Such a result separates what was observed from what was concluded, and names the evidence that could refute the hypothesis.

What an agent can access is determined by the environment, the tools, the permissions and the organisation's policy; the model does not receive telemetry automatically. OpenAI's public description of its own Codex deployment shows how an organisation can allow read-only commands such as viewing Kubernetes logs, restrict network destinations, and keep the agent's events for audit. That is a description of a specific internal practice, not a general claim that every Codex installation has such permissions or is already connected to service logs. Codex settings differ by interface, environment and managed policies. OpenAI: Running Codex safely at OpenAI

For Claude Code (Anthropic's agentic interface for working with a repository and command-line tools), the official documentation describes permission rules for individual tools and paths: for example, allowing reads of certain files or restricting a Bash command. That does not make every command-line command safe, nor does it mean a rule for one command will cover other ways of invoking it. Likewise, the configuration of permissions and available tools depends on the installation and the organisation's settings. Anthropic: Configure permissions for Claude Code

Useful boundaries for safe analysis:

• the agent receives only the time windows, services and fields it needs, not full access to all logs;
• service data is presented as untrusted text for analysis: a log line can contain user-controlled content or instructions, which must be treated as data, not executed;
• read permissions are separated from the right to change configuration, the database, access rules or production traffic;
• every hypothesis comes with a link or identifier for the evidence, a time boundary, the revision and a confidence level;
• before retrying after a timeout, the agent checks whether the first action already happened, so as not to create a duplicate;
• a risky action requires specific approval from the owner; a correlation in an incomplete sample is no grounds for automatically shipping a change.

The agent's own telemetry answers an additional question: what it read, which tools it called, which permissions it was or was not granted, and what it based its conclusion on. This does not replace the product's logs and metrics. In the example OpenAI describes, Codex logging can include user prompts, permission decisions, tool results and connector usage; such records can themselves contain sensitive content. They have to be designed, stored and made available under the same least-access principle as any other telemetry. For the current behaviour of OpenAI and Anthropic products, check the specific product, version, interface and configuration you actually use; the documentation behind these statements in this unit was last re-checked on 10 October 2026. OpenAI: Agent-native telemetry and audit trails, Anthropic: Permissions

A short feedback loop for the product

Observability should shorten the time between a change, the appearance of its consequence and the action of the person responsible, and it should help confirm whether the user scenario that matters has recovered. This is consistent with the ideas in Accelerate about improving software delivery, and with DORA's guidance to measure behaviour that helps decision-making; no chart on its own creates a short feedback loop. You need a clear question, a reliable measurement and an assigned action. The publisher's page describes Accelerate as a book about research findings on how to measure delivery performance and the practices that support it. Accelerate, IT Revolution Press, DORA: Monitoring and observability

Do not turn a service indicator into a ranking of people or agents. A high error rate in production is a fact about the product's behaviour over the measured period; the number of commands an agent executed is a different quantity with a different context. Mixing them produces wrong conclusions, which is why software-delivery and agent-work metrics are treated separately in Agent Observability and Delivery Performance.

Key takeaways

1. Monitoring checks for known conditions, while observability helps investigate things that do not yet have a ready explanation. Both start from the consequence for the user.
2. A log explains individual events, a metric shows change over time, and a trace describes an operation's path between components. Together they give scale, examples and the segments where time is spent waiting.
3. For a metric, name the measurement, the denominator, the unit, the version and the time window. An average does not replace a percentile, and an estimated percentile depends on sample size and how it was aggregated.
4. A successful server response, a running background worker or an empty queue do not prove that background processing has finished, that the records are complete or that the replica is up to date.
5. SLI is the measurement, SLO the objective, SLA the external commitment. An error budget helps discuss non-conformance; it does not justify deliberate harm.
6. An alert should signal a user-facing symptom and lead to an owner's action; repeated log entries can be grouped while preserving volume, context and the separate manifestations of a different impact.
7. Access, retention, sampling and cardinality are part of the design, not an afterthought to data collection.
8. An agent can gather facts and prepare a hypothesis from permitted evidence. An incomplete log and a correlation on their own do not justify a production change.

Sources and further reading

1. OpenTelemetry, Signals. Official explanation of logs, metrics and traces as different types of telemetry. Profiles is a fourth signal with Alpha status. Checked 10 October 2026. https://opentelemetry.io/docs/concepts/signals/ ; https://opentelemetry.io/docs/concepts/signals/profiles/
2. OpenTelemetry, Context propagation. Context between components, linking trace and log identifiers, security of context propagation. Checked 24 September 2026; page updated 10 August 2026. https://opentelemetry.io/docs/concepts/context-propagation/
3. OpenTelemetry, Metrics data model. The model of metric points, counters and histograms; aggregation and bucket limitations. Specification version current as of the check date. https://opentelemetry.io/docs/specs/otel/metrics/data-model/
4. OpenTelemetry, Sampling. Trace sampling, the difference between deciding at the start and after the end of the path, trade-offs between cost and lost evidence. Checked 24 September 2026. https://opentelemetry.io/docs/concepts/sampling/
5. OpenTelemetry, Handling sensitive data. The team's responsibilities for minimisation and for sensitive telemetry attributes. Checked 24 September 2026. https://opentelemetry.io/docs/security/handling-sensitive-data/
6. Google Site Reliability Engineering, Monitoring Distributed Systems. Benefits and limits of monitoring, alerting, noise and fatigue from excessive pages. Checked 24 September 2026. https://sre.google/sre-book/monitoring-distributed-systems/
7. Google Site Reliability Engineering, Service Level Objectives. Definitions of SLI/SLO, percentiles, and the role of product judgement in choosing an objective. Checked 24 September 2026. https://sre.google/sre-book/service-level-objectives/
8. Google Site Reliability Workbook, Alerting on SLOs. Budget burn rate, time windows, the trade-off between alert speed and precision. Checked 24 September 2026. https://sre.google/workbook/alerting-on-slos/
9. Prometheus, Histograms and summaries. The difference between histograms and summaries, percentile estimation, aggregation and bucket error. Official documentation, checked 24 September 2026. https://prometheus.io/docs/practices/histograms/
10. Prometheus, Metric and label naming. Metric naming and warnings about high label cardinality. Checked 24 September 2026. https://prometheus.io/docs/practices/naming/
11. Prometheus, Alerting. Alerting on user-facing symptoms, background jobs, and checking the monitoring system itself. Checked 24 September 2026. https://prometheus.io/docs/practices/alerting/
12. Cloudflare, Post mortem on the Cloudflare Control Plane and Analytics Outage, 4 November 2023. Primary source on the unregistered power change, late detection, and the dependence of logs/analytics on the unavailable site. Checked 24 September 2026. https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/
13. Cloudflare, Cloudflare incident on June 20, 2024, 20 June 2024. Primary source on detection via processor load, HTTP error and latency indicators, and how a concurrent network incident affected the initial hypothesis. Checked 24 September 2026. https://blog.cloudflare.com/cloudflare-incident-on-june-20-2024/
14. OpenAI, Running Codex safely at OpenAI. Description of a specific internal Codex deployment: execution boundaries, approval rules, read-only commands and agent logs. This is an example of organisational practice, not universal defaults for every Codex user. Checked 10 October 2026. https://openai.com/index/running-codex-safely/
15. Anthropic, Configure permissions for Claude Code. Official rules for tool and path permissions in Claude Code; behaviour depends on the specific product and configuration. Checked 10 October 2026. https://code.claude.com/docs/en/permissions
16. Martin Kleppmann and Chris Riccomini, Designing Data-Intensive Applications, 2nd edition, O'Reilly Media, March 2026. The current edition and authorship are confirmed by the author's page and the book's site; the full text was not reviewed for this research, so this unit attributes to it no page quotations and no specific teaching example. https://martin.kleppmann.com/ ; https://dataintensive.net/
17. Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps, IT Revolution Press, 27 March 2018. The publisher's page confirms the title, authors and date; used as a source on measuring and improving delivery, not as evidence that a single correlation caused an improvement. https://itrevolution.com/product/accelerate/
18. OpenTelemetry, Language APIs & SDKs. Maturity status of individual signals by programming language. Checked 10 October 2026. https://opentelemetry.io/docs/languages/
19. DORA, Monitoring and observability. The distinction between checking predefined conditions and searching for properties not known in advance. Checked 24 September 2026. https://dora.dev/capabilities/monitoring-and-observability/