Query ZipLogger from Grafana, no plugin

You already have Grafana dashboards and people who know how to use them. You don't want a second place to look, and you don't want to install and maintain a community plugin on your Grafana server either.

ZipLogger answers the same HTTP API that Grafana's own Loki, Prometheus and Tempo data sources call. You add three built-in data sources, point them at ZipLogger, and your logs, request metrics and traces show up in Grafana. The catch is that ZipLogger is not Loki or Prometheus underneath, so only part of each query language works. This post says exactly which part.

The three URLs

Create a read key

Grafana needs an API key that can read. An ingest-only key is refused, because ingest keys are meant to sit in browser bundles where anyone can see them.

In ZipLogger, go to Settings → API keys, type a name, tick Can read data, and click Create key. Copy the zk_… value; it is shown once. Only Admins can create keys.

The key decides the workspace. A query cannot name another one, so one Grafana can hold several ZipLogger workspaces as separate data sources, and none of them can read another's data.

Before touching Grafana, check the key:

curl -s -H "X-Api-Key: YOUR_API_KEY" https://app.ziplogger.ai/grafana/health

It returns your workspace, the three paths, the metric catalog and the log labels. Without a key, or with an ingest-only key, every path answers 401 with "Send a ZipLogger API key as the X-Api-Key header, or as the password of HTTP basic auth."

Add the data sources

In Grafana, Connections → Data sources → Add, pick Loki, set the URL, and add a custom HTTP header X-Api-Key with your key. Repeat for Prometheus and Tempo. Basic auth with any user name and the key as the password works too, and so does Authorization: Bearer.

Or provision all three:

# /etc/grafana/provisioning/datasources/ziplogger.yaml
apiVersion: 1
datasources:
  - name: ZipLogger logs
    type: loki
    access: proxy
    url: https://app.ziplogger.ai/grafana/loki
    jsonData:
      httpHeaderName1: X-Api-Key
    secureJsonData:
      httpHeaderValue1: YOUR_API_KEY

  - name: ZipLogger metrics
    type: prometheus
    access: proxy
    url: https://app.ziplogger.ai/grafana/prometheus
    jsonData:
      httpHeaderName1: X-Api-Key
    secureJsonData:
      httpHeaderValue1: YOUR_API_KEY

  - name: ZipLogger traces
    type: tempo
    access: proxy
    url: https://app.ziplogger.ai/grafana/tempo
    jsonData:
      httpHeaderName1: X-Api-Key
    secureJsonData:
      httpHeaderValue1: YOUR_API_KEY

Every request Grafana makes, including label lookups for the query editor, counts as one API request against your plan's daily allowance, the same meter the REST API uses. A dashboard with many panels and a short refresh interval adds up. See pricing for the allowances.

Logs: the LogQL that works

A stream selector over ZipLogger's own labels:

All four matchers work (=, !=, =~, !~), and so do the four line filters (|=, !=, |~, !~), which apply to the message:

{service="checkout-api", level="error"} |= "timeout"
{service=~"checkout-.*"} != "healthcheck"
{field_orderId="83112"}
count_over_time({service="checkout-api", level="error"}[5m])
rate({service="checkout-api"}[1m])

Grafana's label browser lists the fixed labels above. It does not list the field_ ones, so type those yourself.

Each line comes back as the message, then stack_trace=… if there is one, then up to 40 structured fields as logfmt, for example orderId=83112 customer="acme corp". Grafana's field detection works on that. One query returns at most 5,000 lines, and 1,000 when Grafana doesn't ask for a number.

For count_over_time and rate, the number in brackets is also the bucket width. [5m] gives one point per 5 minutes, whatever step the panel asks for. rate divides by that width, so it is per second.

What is refused

We ran these through ZipLogger's LogQL parser, with a key for a workspace called acme. Each is refused with a 400 and the message shown after the arrow:

{service="checkout-api"} | json
  -> Parser and formatting stages (| json, | logfmt, | unwrap, …) are not supported.
     ZipLogger already indexes structured fields: select them with field_<key> in the stream selector.

bytes_over_time({service="checkout-api"}[5m])
  -> Expected a stream selector, e.g. {service="checkout-api"}.

topk(5, count_over_time({level="error"}[5m]))
  -> Expected a stream selector, e.g. {service="checkout-api"}.

{namespace="prod"}
  -> Unknown label "namespace". Available: service, source, severity, level, release,
     commit_sha, commitSha, template, tenant, or field_<key> for a structured field.

sum by (detected_level) (count_over_time({service="checkout-api"}[1m]))
  -> Cannot group by "detected_level". Available labels: service, source, severity, level,
     release, commit_sha, commitSha, template, or field_<key>.

{tenant="someone-else", service="checkout-api"}
  -> This API key belongs to workspace "acme"; it cannot query "someone-else".

The same goes for | logfmt, | pattern, | regexp, | line_format, | label_format, unwrap, binary operations between queries, and every range function other than count_over_time and rate.

One trap: sum by (service) (count_over_time({level="error"}[5m])) is accepted, but the grouping is not applied. You get a single series for all services together. For one line per service, put service="…" in the selector and use one query per service.

Metrics: a fixed catalog

The Prometheus side is not a time-series database. It serves six metrics that ZipLogger computes from your spans and logs:

The four request metrics need traces. If you don't send spans yet, the OpenTelemetry post shows how.

The only label is service, and only as an exact match:

ziplogger_request_p95_ms{service="checkout-api"}
ziplogger_log_error_count

Grafana's habit of wrapping queries is tolerated. rate(…[5m]), increase(…), sum(…), avg(…), max(…), min(…) and the *_over_time functions are accepted and then ignored, because each point is already computed for its step. Two consequences:

• rate(ziplogger_request_count[5m]) returns requests per step, not per second.
• sum by (service) (…) does not split by service. It is one series.

Refused, each with the catalog in the error:

• a regex or negative matcher: {service=~"checkout.*"} gets "Only an exact service matcher is supported, e.g. service="checkout-api"."
• any other label, such as {env="prod"}
• any other metric name, such as http_requests_total
• arithmetic like ziplogger_request_error_count / ziplogger_request_count, and histogram_quantile

A wide time range is not refused. ZipLogger caps a series at 1,500 points and widens the step to fit.

Traces: search and the waterfall

Opening a trace by ID works as it does with Tempo. ZipLogger returns up to 1,000 spans in Tempo's OTLP-shaped JSON, grouped by service, with span kind, status and attributes. Numeric attributes keep their type instead of turning into strings.

Search is much narrower. There is no TraceQL engine. The search reads these parameters and nothing else:

• service.name (or service)
• status=error
• limit, 20 by default and at most 200
• start and end, the last hour by default
• a service.name= value inside Tempo's tags parameter

A TraceQL expression is not read at all. It is not refused either: ZipLogger answers with the most recent traces in the time range, unfiltered. Don't build a panel on TraceQL filters. To look up a trace, use its ID. To search spans by attribute, use the Traces page in ZipLogger.

Tag lookups return service.name, name, status and kind, plus up to 200 attribute keys your spans actually carry, with their values.

Jump from a log line to its trace

If your logs carry a traceId field, the line Grafana gets contains traceId=…. A derived field on the Loki data source turns that into a link to the Tempo data source:

    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: 'traceId=(\w+)'
          url: '$${__value.raw}'
          datasourceUid: YOUR_TEMPO_DATASOURCE_UID

Limits worth knowing

• Read-only. Nothing in Grafana can change or delete ZipLogger data.
• No Grafana-side processing of raw logs. Structured fields are already indexed, so select them with field_<key> instead of parsing lines.
• Series listing is approximate. Grafana's log browser gets every service and severity combination seen in the window, not a real label index.
• Not the only way in. For scripts and CI, the query API post is simpler than going through Grafana.

The reference is in the Grafana docs section. If you don't have a workspace yet, create one free and make a read key under Settings → API keys.