SLOs and burn-rate alerts for your services
An alert on "error rate above 1%" pages you for a two-minute blip and stays quiet through a week of 0.5% errors. The question that matters is whether a service keeps its promise over weeks, and how fast it is breaking that promise right now.
A service-level objective (SLO) and its error budget answer that. This post covers SLOs in ZipLogger APM: how to create one, what each number on the page means, burn rate in plain arithmetic, and alerts that page for a real outage but not for a blip.
Create an SLO
Open APM and pick the SLOs tab, the fourth one after Services, Service map and Regressions. Click + New SLO and fill in:
- Name: anything, for example
Checkout availability. - Service: as shown on the Services tab,
namespace/namewhen the service has a namespace. - Endpoint (optional): one endpoint exactly as listed on the service page, such as
POST /checkout/{cartId}. Empty means the whole service. - Measure: Availability: requests without error or Latency: requests faster than a threshold. The latency one adds Faster than (ms).
- Target (% good): at least 50 and below 100, for example
99.9. - Window: 7, 28 or 30 days, always rolling.
The service and endpoint are not checked against what APM has seen. A typo is saved, and the SLO then shows No data.
An SLO belongs to the environment you are viewing, and an environment can have up to 100. Creating and deleting SLOs needs the Editor or Admin role. Everyone else in the workspace can view them.
An SLO stores no measurements of its own. Every number is computed from APM's rollups when you open the page, so a new SLO shows the past days straight away, as far back as the data goes.
What counts as a good request
For availability, a good request is one without an error. An error means the same as everywhere in APM: HTTP 500 or above, or a span with error status. A 4xx answer is a good request.
For latency, a good request is one faster than the threshold. Errors are not looked at, so a fast 500 is good for a latency SLO. If you care about both, create two SLOs. Durations are stored in histogram buckets, not one by one, so a request within about 2.4% of the threshold can land on either side of it.
The counts come from the same place as every other APM number:
- Request metrics from the ZipLogger ASP.NET Core middleware, or the request count of OpenTelemetry's
http.server.request.duration, when the service sends them. These count every request. - Otherwise spans. If you sample traces, the ratio is then an estimate from the traces that were kept. APM does not scale it up.
OpenTelemetry request counts carry no durations, so a latency SLO on such a service reads its spans.
Reading the table
Each SLO is one row with these columns:
| Column | Meaning |
|---|---|
| Status | OK when Current is at or above the target over the window, Breached when below, No data when there were no requests |
| SLO | The name, the measure and the window |
| Service | The service, with the endpoint under it if you set one |
| Current | Good requests divided by all requests over the window |
| Target | Your target |
| Error budget left | 1 − bad / (total × (1 − target)) |
| Burn (1 h) | The bad fraction over the last hour, divided by (1 − target) |
| Requests | All requests in the window |
A worked example. The target is 99.9% and the window held 1,000,000 requests, so the budget is 0.1% of them: 1,000 bad requests. With 400 bad requests, 1 − 400 / 1,000 leaves 60% of the budget. With 1,200, the column reads exhausted (-20%) and Status is Breached.
Burn rates leave out the newest two minutes, because those are not aggregated yet.
History on a row opens a chart with one point per UTC day of the window. The line is the error budget left at the end of each day, counted from the start of the window, so its last point matches the table. The dots are each day's own result against a dashed target line, and days below the target are drawn in another color. It shows whether the budget went in one bad day or leaked away a little every day.
Burn rate in plain numbers
Burn rate compares how fast you spend the budget with spending it evenly across the window. At 1× the budget runs out exactly at the end of the window. At 2× it runs out halfway.
Take a 99.9% target over 30 days, which is 720 hours. The budget is 0.1% of requests.
- 1×: 0.1% of requests fail. The budget lasts 30 days.
- 6×: 0.6% fail. The budget lasts 30 / 6 = 5 days. Six hours at this rate spend 6 × 6 / 720 = 5% of it.
- 14.4×: 1.44% fail. The budget lasts 30 / 14.4 ≈ 2.1 days, about 50 hours. One hour at this rate spends 14.4 / 720 = 2% of it.
The same burn rate means something different on a shorter window. A 7-day window is 168 hours, so one hour at 14.4× spends 14.4 / 168 ≈ 8.6% of that budget, and the whole budget is gone in about 11.7 hours. The usual alert thresholds below were worked out for 30 days.
Alerts that page on a fast burn
Go to Alerts → Rules → + New alert and pick the condition SLO error budget burn rate (APM SLOs). The form asks for:
- SLO: a list of the SLOs in this environment. Create the SLO first.
- Burn rate at least: the threshold, for example
14.4. - Window (min): the long window, 5 to 4,320 minutes.
- Alert severity: low, medium, high or critical, included in every notification.
Interval (sec) and Notify via work as on every other rule, see the alert rules post.
The rule fires only when two windows both burn at or above the threshold: your long window, and a short window of 1/12 of it, at least 5 minutes. A 60-minute window gets a 5-minute short window. A 360-minute window gets a 30-minute one.
The two windows solve two problems. A long window alone keeps firing after a short outage has ended, because the bad minutes are still inside it. A short window alone fires on every blip. With both, the rule pages when the last hour was bad and it is still bad now, and it resolves soon after the errors stop, because the short window drops first.
The rule fires on the first check where both windows are at or above the threshold, and resolves on the first check where either one is below. There is no "two checks in a row" step like the APM: P95 latency above N ms and APM: error rate above N % conditions have. With fewer than 20 requests in the long window, the rule keeps whatever state it had.
The usual setup is two rules on the same SLO:
| Rule | Burn rate at least | Window (min) | Short window | Share of a 30-day budget spent in the long window |
|---|---|---|---|---|
| Fast burn | 14.4 | 60 | 5 minutes | 2% |
| Slow burn | 6 | 360 | 30 minutes | 5% |
Send the fast one to an escalation policy, because at that rate the month's budget is gone in about two days. Send the slow one to Slack or email. The notification gives the long window's share of good requests, the request count, the target, and both burn rates.
An SLO that burn alerts watch cannot be deleted. The delete is refused with the number of alert rules that watch it, and you delete those first.
Changing an SLO
The SLOs page creates and deletes, but has no edit button. The API has PUT /api/v1/apm/slos/{id} for changes. Deleting and creating the SLO again loses nothing, since the numbers are computed from the rollups, but alert rules point at the SLO's id, so you would need new rules too.
What it does not do
- Only APM requests. There are no SLOs on logs, custom metrics, product events or uptime checks. For alerts on your own metrics, see metric threshold and anomaly monitors.
- Rolling windows of 7, 28 or 30 days only. No calendar month and no other length.
- One service or one endpoint per SLO. No group of endpoints, and no SLO per customer.
- One measure per SLO. Availability and latency are separate SLOs.
- Spans-only services get an estimate. If you keep only slow and failed traces, the estimate leans toward bad, and APM cannot detect that from the spans alone.
- The window only reaches back as far as your workspace keeps traces. APM's hourly rollups follow trace retention, at most 90 days. With 7 days of trace retention, a 30-day SLO counts 7 days.
- No alert on budget left or on Breached. Burn rate is the only SLO alert condition.
- History is per UTC day. There is no hourly view of the budget.
To try it, create a free workspace, send traces or request metrics from one service, and add an SLO under APM → SLOs. The SLO section of the docs has the reference.