Parse log lines into fields and metrics

Not every log arrives with neat fields. An nginx access line, a third-party library or an old service often sends one string, like 10.0.4.17 POST /api/orders 502 123.5ms, and the status code and duration you care about are stuck inside it.

ZipLogger's log processing rules fix that as each log arrives: parse rules pull fields out of the message, and metric rules turn matching logs into metric points you can chart and alert on.

Where the rules live

Open Search and click Log processing in the toolbar. The page lists the rules for the environment you are viewing. Rules in staging do not touch production logs, and the other way round.

Changing rules needs the Editor or Admin role. There are two buttons:

• + Parse rule adds fields to a log.
• + Metric rule turns matching logs into metric points.

Parse rules run first, in the Order you give them, lowest number first. Metric rules run after that, so a metric rule can use a field a parse rule just added. An environment can have at most 50 rules. After you save, the change reaches every server within 30 seconds.

Both kinds have the same two filters:

• Only service (optional): the log's source, the service name you see in Search. Empty means all services.
• Only messages containing (optional): a piece of text the message must contain, ignoring case.

Rules apply to logs that arrive after you save them. Logs already stored are not parsed again, and deleting a rule does not remove the fields it already added.

A parse rule with a pattern

Pick Pattern in the Parse menu. A pattern is the literal text of the line with %{TYPE:field} tokens where the values are. For the access line above:

%{IP:client.ip} %{HTTPMETHOD:http.method} %{URIPATH:url.path} %{INT:http.status} %{NUMBER:duration_ms}ms

Paste a real line into Test with a log line and the form shows the result as you type. For 10.0.4.17 POST /api/orders 502 123.5ms the rule adds:

These are all the types there are:

Everything outside the tokens is a regular expression, and a named group (?<field>...) also becomes a field. So this works too:

user (?<user.id>\w+) paid %{NUMBER:amount} (?<currency>[A-Z]{3})

A few things to know about patterns:

• The pattern can match anywhere in the message. A timestamp or prefix before the part you describe does not stop a match.
• URIPATH stops at ?. In the pattern above, a line with /api/orders?page=2 does not match, because a space is expected right after the path. Add \S* after the token (%{URIPATH:url.path}\S* ) to skip the query string.
• Field names start with a letter or _ and use letters, digits, _ and .. Each field can appear once in a pattern, and a pattern needs at least one.
• Matching gives up after 50 ms per log. A pattern that backtracks badly extracts nothing for that log instead of slowing ingest down.

JSON in the message

Some loggers write a JSON object into the message text, often after a sentence. Pick JSON in the message and the object becomes fields. ZipLogger takes the whole message, or the text from its first { to its last }, so this works:

order failed {"order": {"id": "A-1", "total": 42.5}, "retry": true}

It adds order.id (text A-1), order.total (number 42.5) and retry (text true). The rules:

• Nested keys are joined with dots, up to 3 levels deep and 50 fields per log.
• Numbers stay numbers. Strings and true/false are stored as text, so a field does not switch type from one log to the next.
• Values longer than 1,000 characters are cut.
• Arrays, nulls and keys that are not valid field names (a space or a - in the key) are skipped.
• A { in the text before the JSON breaks the match, because the object is read from the first {.

key=value pairs

Pick key=value pairs for lines like this:

payment declined user=42 amount=19.99 reason="card expired"

Pairs can be anywhere in the message, separated by spaces, commas or semicolons. Put a value with spaces in double quotes. Plain decimal numbers such as 42 or -1.5 are stored as numbers. Values like 1.2.3, 07:30 or 0x1f stay text. Up to 50 fields per log.

Searching the new fields

Extracted fields sit next to the fields your SDK sends, under fields. in Search:

fields.http.status:502 AND fields.duration_ms:>500

A parse rule never overwrites a field the log already has. If your logger sends http.status itself, that value wins.

Search keeps the type a field had first that day. If duration_ms arrived as a number in the morning and a later log has duration_ms=slow, that log is still stored and shows the value when you open it, but a search on that field will not find it. When a field is text in one log and an object in another, the later one is kept as text under the field name plus _conflict. Either way the log keeps its message and every other field.

Turn logs into a metric

Click + Metric rule. Besides the two filters, it has three fields:

• Metric name: for example logs.http.requests. Letters, digits, ., _ and -.
• Value field (optional): leave it empty to count matching logs. Fill it in with a numeric field, such as duration_ms, to record that value.
• Group by field (optional): a field whose value becomes a label on each point, for example http.status. A log without the field gets the value (none).

The two kinds of metric behave differently:

For the access log, two rules cover most questions:

• logs.http.requests, no value field, grouped by http.status.
• logs.http.duration_ms, value field duration_ms, no group.

Each point carries the log's service, so the service filter on the Metrics page and in alerts works on these metrics too.

Only accepted logs are counted. A log refused at ingest with a 429, because the workspace was over its quota or the server was busy, never becomes a point.

See it on the Metrics page

In the rules table, the metric name links to the Metrics page with that metric selected. For a count metric, set Aggregation to sum. The default is avg, which shows the average size of a batch, not how many logs there were. Choose http.status under Group by to get one line per status code. Add to dashboard… saves the chart as a dashboard widget.

Points from a rule go through the same limits as any metric you send, including the daily cap on distinct series. Every value of the group-by field opens a new series. Group by a status code or a region, not a user id, an order id or a raw URL path with ids in it, or new series will be refused once the cap is reached.

If the metric store cannot keep up, points are dropped rather than slowing ingest down. The Log processing page then shows how many were dropped on that server.

Alert on it

A log-derived metric is an ordinary metric, so a metric monitor can watch it. Go to Alerts, then Rules under Detection, and click + New alert:

• Condition: metric above threshold
• Metric: logs.http.requests
• Aggregation: sum
• Above: 20
• Window (min): 5
• Label filter (optional): http.status=502

That fires when more than 20 requests returned 502 in the last 5 minutes. Put http.status in Group by (optional) instead to get a separate incident per status code. For latency, use logs.http.duration_ms with p95 or max. The metric and anomaly monitors post covers the rest of the form.

What it does not do

• It does not reparse stored logs. A new rule only affects logs that arrive after it is saved.
• It does not drop, sample, rename or mask logs. Rules only add fields and emit metric points.
• It has no custom pattern library. The twelve types above and plain regular expressions are all there is.
• It does not change a log's time or severity. A parsed level or timestamp is just another field.
• Filters only look at the service and the message text. A metric rule cannot be limited to logs where a parsed field has a certain value. Group by the field, then filter on the label in the chart or the alert.

Next step

The log processing docs list every rule setting and limit. To try it on your own logs, create a workspace, send a few lines, and open Search, then Log processing.