Observability That Answers Questions:
RED, USE, Traces, and SLOs

Picture a checkout service on a Friday evening. The dashboard is green. CPU sits at 31 percent, memory at 52 percent, and every health check returns 200.

The support queue says otherwise. Customers click Pay and get an error. About 12 percent of requests fail, roughly one in eight, because the payment provider's API started timing out and the service returns a 502.

Nothing on the dashboard measures that. The health check asks whether the process is alive, and it is. The first signal anyone gets is a customer email, 47 minutes in.

This is a composite scenario, not one client's outage. With a 99.9 percent availability SLO, a 12 percent failure rate burns the entire 30-day error budget in 6 hours. An alarm on that burn rate pages in about 7 minutes.

Same outage, forty minutes sooner. The dashboard showed causes, and customers feel symptoms.

Green Dashboards, Angry Customers

Most teams alarm on what is easy to collect: host CPU, memory, disk. They log unstructured text and search it with grep. They learn about outages from customers.

That setup answers questions about machines, not the one that matters: are customers getting what we promised? Resource metrics explain an outage. They rarely define one.

Two gaps sit behind that. Without a trace, nobody can say which dependency made a request slow, so the war room guesses. Without an agreed target, nobody can say whether last month met the promise.

Monitoring can fail with the thing it watches. AWS's summary of the December 7, 2021 event in US-EAST-1 says network congestion cut the visibility of real-time monitoring data for its operations teams, who fell back on logs. It also says congestion kept the Service Health Dashboard tooling from failing over to its standby Region, and that the dashboard could be updated by 8:22 AM PST, 52 minutes after the 7:30 AM trigger (AWS summary). In the February 28, 2017 S3 event, the dashboard's administration console depended on S3, so per-service status could not be updated until 11:37 AM PST (AWS summary).

Observability is a workload too, and it needs the same failure planning.

Symptoms, Causes, and One Request ID

The pattern has one rule: page on what customers feel, and use resource metrics to explain it.

Three signals describe every service. Rate is how many requests arrive. Errors is how many fail. Duration is how long they take, as percentiles, not averages. That is RED, per operation.

Three signals describe every resource: a queue, a pool, a disk. Utilization is how busy it is. Saturation is how much work waits. Errors is how often it fails. That is USE, the place to look after RED says something is wrong.

Then connect them. Every request carries one ID, every log line is structured JSON that includes it, and every hop adds a span to a distributed trace. A bad RED number leads to a trace, the trace names the slow dependency, and the dependency's USE metrics say why.

The last piece is the promise. An SLO states it as a number: 99.9 percent of checkout requests succeed over 30 days. The gap between that and 100 percent is the error budget, and you alarm on how fast you spend it.

The Error Budget, With Arithmetic

A 99.9 percent SLO over 30 days:

Window:        30 days x 24 hours x 60 minutes = 43,200 minutes
Allowed bad:   0.1 percent of 43,200            = 43.2 minutes
At 1,000 requests a second:
               2,592,000,000 requests in 30 days
               0.1 percent                      = 2,592,000 failed requests

That is the whole budget. One bad deploy can spend half of it.

Burn-Rate Alerts That Page for the Right Reasons

Burn rate is how fast you spend the budget, as a multiple of the rate that would spend it exactly over the window.

burn rate = error rate over a look-back window / (100% - SLO)
          = error rate / 0.1%   for a 99.9% SLO

Burn rate 1 spends the budget in exactly 30 days. Burn rate 14.4 spends it in 50 hours. AWS's threshold formula is the share of budget you will tolerate losing, times the interval length, divided by the look-back window: 2 percent of a 30-day (720-hour) budget in one hour is 2% x 720 / 1 = 14.4, a 1.44 percent error rate (burn-rate alarms).

One window is not enough. A short window pages on blips, and a long one keeps paging after the fire is out. So pair them: the long window shows that real budget has gone, and the short one, a twelfth as long, shows that the problem is still happening. The Google SRE workbook recommends three pairs for a 99.9 percent SLO (SRE workbook), and the CloudWatch documentation builds them as composite alarms.

Run three failures through it. A total outage (100 percent errors) crosses 1.44 percent in the one-hour window after 0.0144 x 60, about 52 seconds, so the first pair pages almost at once. A 2 percent error rate takes 43 minutes to cross the same line, and that is right: it would take 36 hours to spend the budget. A steady 0.2 percent never reaches a page threshold and opens a ticket after about 36 hours, which suits something that would take 15 days to hurt. The short window also lets an alert clear within minutes of the errors stopping.

Building It on AWS

CloudWatch holds the metrics, logs, and alarms, and OpenTelemetry carries the traces. The trap is cardinality: never put a request ID in a metric dimension, because each unique combination becomes its own custom metric. Request IDs belong in the log line and the trace.

When Not to Use It

• A tool nobody depends on. An admin page used twice a month does not need an SLO. Use a canary with an availability alarm.
• Full tracing of every request. Storing every trace costs money and buries the interesting ones. Sample, and keep every trace that errored or ran slow.
• Paging on USE metrics alone. A hot CPU is a cause. If customers are fine, send a ticket. Page when RED or the SLO says they are hurting.
• A service with almost no traffic. At a few requests an hour, one failure swings the error rate by double digits. Add a canary for steady traffic before you trust the burn rate.

How We Check for It

The Well-Architected Framework asks in two pillars. Operational Excellence OPS04, "How do you implement observability in your workload?", covers key performance indicators (OPS04-BP01), application, user experience, and dependency telemetry (OPS04-BP02 to BP04), and distributed tracing (OPS04-BP05). OPS08, "How do you utilize workload observability in your organization?", covers using it, including actionable alerts (OPS08-BP04). Reliability REL06, "How do you monitor workload resources?", includes REL06-BP07, end-to-end tracing of requests.

In a review we look at these in live accounts:

• SLOs on the top three or four user journeys, each with an attainment goal and a burn-rate alarm.
• The alarm list: how many page on symptoms (availability, burn rate) and how many on CPU or memory.
• One trace from a real request that crosses every hop of the critical path, with no gaps at queues.
• Structured logs where one request ID appears in every service, checked with a Logs Insights query.
• Canaries on the same journeys the SLOs cover, running from outside the workload.

The review runs on this telemetry, so a workload without it gets findings of its own. See the Well-Architected Review.

How the Other Nine Patterns Show Up on the Dashboard

Every pattern in this series works silently until it does not. Each emits one signal that says which state it is in. Put these on the dashboard next to RED.

A global error rate of 0.5 percent can hide one cell failing at 12 percent.

Related Patterns

• Static Stability: the monitoring stack needs it too.
• Cell-Based Architecture: RED per cell shows whether the blast radius held.
• Load Shedding and Backpressure: rejections are intended, so count them apart from failures.