Metric monitors and anomaly alerts

You send metrics to ZipLogger and want to hear when one of them goes wrong: a host pinned at full CPU, a queue that stops draining, traffic that drops on a Tuesday morning. Log rules count log lines, so they cannot see a number that only exists as a metric.

Metric monitors are alert rules on any metric you ingest. They come in two kinds: a fixed line (metric above threshold, metric below threshold) and "unusual for this time" (the three anomaly conditions). This post sets up both, with CPU per host as the worked example.

What a metric monitor can read

Any metric with points in the environment the rule belongs to:

• OpenTelemetry metrics sent to /v1/metrics by an SDK or the OpenTelemetry Collector.
• Metrics from your own code sent to /ingest/v1/metrics, such as the request.duration metric from ZipLogger.Metrics.AspNetCore.
• Metrics made from logs by a metric rule in a log pipeline, for example a count of access-log lines with status 502. The log pipelines post covers those.

The rule needs the metric name exactly as it was reported. To find it, open the Metrics page: the Metric picker lists every name with recent points. Chart it there first, with the same aggregation, label filter and group-by you plan to alert on. If the chart is empty, the rule will have nothing to read.

Worked example: CPU per host

The OpenTelemetry Collector's hostmetrics receiver reads CPU, memory, disk and network figures from the machine it runs on. The resourcedetection processor adds host.name to every point. This config sends CPU utilization to ZipLogger:

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
          system.cpu.time:
            enabled: false

processors:
  resourcedetection:
    detectors: [system]
  batch:

exporters:
  otlphttp/ziplogger:
    endpoint: https://app.ziplogger.ai
    headers:
      X-Api-Key: ${env:ZIPLOGGER_API_KEY}
    compression: gzip

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/ziplogger]

A few things about it:

• system.cpu.utilization is off by default in the receiver, so the config turns it on. system.cpu.time is on by default; the config turns it off because nothing below uses it.
• resourcedetection is in the contrib build of the Collector (otelcol-contrib), not the core one.
• The otlphttp exporter adds /v1/metrics to the endpoint itself.
• The key decides the environment. Points sent with a production key are only seen by rules in production.
• This assumes the Collector runs on the host. In a container it measures the container, unless you mount the host's filesystem and set the receiver's root_path.

On the way in, ZipLogger keeps the point's attributes as labels, and adds the resource's host.name as a label too. For this metric that gives three labels: cpu (cpu0, cpu1, ...), state (user, system, idle, wait and a few more) and host.name. Other resource attributes are not kept as labels. The Collector sends no service.name here, so the service shows as unknown. Leave the rule's Service empty.

Utilization is a ratio from 0 to 1. ZipLogger does not convert units, so 90% CPU is 0.9.

Check it on the Metrics page before you write the rule. This URL opens the same query:

/metrics?name=system.cpu.utilization&agg=max&groupBy=host.name&filter=state:user

The rule

Go to Alerts, then Rules under Detection, and click + New alert:

• Name: Host CPU
• Condition: metric above threshold
• Metric: system.cpu.utilization
• Aggregation: max
• Above: 0.9
• Window (min): 5
• Group by (optional): host.name
• Label filter (optional): state=user
• Interval (sec): 60

Once a minute, ZipLogger takes every point of system.cpu.utilization with state = user from the last 5 minutes, splits them by host, and takes the largest value per host. Any host above 0.9 fires.

The label filter matters more than it looks. Without it, the points include state=idle, and on a quiet host idle is close to 1. The rule would fire on the hosts doing nothing.

The aggregation decides what "busy" means:

• max fires when one core spent more than 90% of one scrape in user code. That catches a single-threaded process stuck in a loop, and also catches short spikes.
• avg fires when the host's cores averaged above 0.9 over the whole window. Fewer alerts, and only for sustained load.
• To count every kind of busy (user, system, waiting on I/O), filter on state=idle instead and use metric below threshold with min below 0.1: some core was less than 10% idle.

The monitor takes one label filter. The Metrics page allows several, but a rule does not.

Keep the window to a few collection intervals or more. With a 30-second interval, a 1-minute window holds about two scrapes per core, and one late batch can leave it empty, which the rule treats as no data.

One incident per host

With a Group by, each value of that label is its own episode. A host that crosses the line opens its own incident, titled Host CPU: web-1, and resolves on its own when it drops back. The notification names the host and its value, for example max(system.cpu.utilization) for host.name=web-1 is 0.95 over the last 5m (alert when > 0.9).

The rule as a whole shows FIRING while any host is, and its value is the worst host's. Incidents, escalation policies, maintenance windows and notification channels work as they do for every other rule; the log alert rules post covers those.

Two limits keep a large fleet in check:

• At most 100 groups are read per check, the ones with the most points first.
• At most 25 incidents are open for one rule at a time. When more groups breach, the worst ones open first and the rest wait for a free slot.

Group by works on text labels. OpenTelemetry attributes are stored as text, so any OTLP label works. A label your code sends as a number to /ingest/v1/metrics has no groups.

No data, and hosts that stop reporting

Silence is handled on purpose:

• Every aggregation except count, over a window with no points, is unknown. The rule keeps its state. It does not fire, and it does not resolve.
• count of no points is 0. So an ungrouped rule with Aggregation count, Below 1 fires when the metric stops arriving. On system.cpu.utilization with no group-by, it fires when no Collector is sending at all.
• In a grouped rule, a host that goes silent while other hosts still report has its incident closed, with a note that it has no data. It is not counted as zero.
• When no group reports at all, nothing changes. Open incidents stay open.

So a grouped CPU rule will not tell you that a host died. Pair it with a count below 1 rule, filtered to one host with host.name=web-1 if that host matters on its own.

Anomaly monitors

Some metrics have no good fixed line. Request counts are high at noon and low at 3 a.m., so any threshold is either too low for the day or too high for the night. An anomaly monitor compares the metric with its own past at the same time.

The Condition menu has three:

• metric unusually high (anomaly)
• metric unusually low (anomaly)
• metric unusual either way (anomaly)

They take Metric, Aggregation, Service (optional) and Label filter (optional) like a metric monitor, plus two fields of their own.

Normal is picks the history:

• same time on previous days: the same window, at the same time of day, on each of the last 7 days. It needs 5 of those days with data.
• same time and weekday on previous weeks: the same window and weekday over the last 4 weeks. It needs 3 with data. Use it when weekends or Mondays look different: the daily choice compares Monday morning with the weekend, the weekly one with previous Monday mornings.

"Normal" is the median of those values, and the spread is the median absolute deviation times 1.4826. Unlike a mean, neither moves much when one past day had an incident. The spread is never less than 5% of the median, so a nearly flat metric does not fire on tiny changes.

Sensitivity is how many spreads away from the median counts as unusual. 3 is a good start; higher fires less. It can be above 0 and up to 20. Because the spread has that 5% floor, at sensitivity 3 the value has to be at least 15% away from normal before anything fires.

Window (min) is 5 to 360 minutes for anomaly rules.

Example: traffic that drops

Say a log pipeline has a count metric rule named logs.web.requests that counts your web server's access-log lines. A count rule writes the number of matching logs per minute, so the right aggregation is sum:

• Name: Web traffic drop
• Condition: metric unusually low (anomaly)
• Metric: logs.web.requests
• Aggregation: sum
• Sensitivity: 3
• Normal is: same time and weekday on previous weeks
• Window (min): 15

Every check sums the last 15 minutes and compares the result with the same 15 minutes on the same weekday over the past four weeks. The notification gives the current value, the usual value with its spread, and how many deviations away it is.

There is a gap here. When traffic stops completely, the pipeline writes no points, and sum of no points is "no data", so the anomaly rule keeps its state instead of firing. Add a metric below threshold rule on the same metric with count below 1 for that case.

What metric monitors do not do

• No group-by on anomaly monitors. An anomaly rule watches one series. Use the label filter to pick one host or one endpoint, and one rule per series that matters.
• No forecasting. Nothing predicts "the disk fills in 6 hours". Rules look at the current window only.
• No math across metrics. There is no "errors divided by requests" rule, and no rule that combines two other rules.
• No "true for N minutes" setting. Each check compares one window with the line. A longer window, or avg instead of max, is how you ask for sustained trouble.
• Fewer aggregations than the Metrics page. Rules offer avg, max, min, sum, count and p95. The page's p50, p90, p99 and rate are not available in a rule.
• No unit conversion. The threshold is in the metric's own unit. Utilization is a ratio, and OpenTelemetry durations are usually seconds, so 500 ms is 0.5.
• Weekly history needs retention. The weekly baseline reads raw points from up to four weeks ago. If your plan keeps raw metrics for less than about three weeks, a weekly rule never has enough history and never fires.
• New metrics need a week. A daily anomaly rule stays quiet until 5 of the last 7 days have points.

The SLO burn-rate post covers a different question: whether errors are using up an error budget faster than planned.

To set one up, create a free workspace, point a Collector or SDK at /v1/metrics, check the metric on the Metrics page, and go to Alerts → Rules → + New alert. The metric monitors section of the docs has the reference.