Follow one error across logs, traces and events

A user reports that checkout failed. You find the error log line. Then you search the traces for the same minute, guess which request it was, and look up the user's events on a third screen. Each step is a guess based on timestamps.

A shared id removes the guessing. If the log line, the spans and the event all carry the same trace id, they can be joined exactly. ZipLogger's Investigate page does that join, and it tells you which records are joined by an id and which only happen to be nearby.

Open an investigation

You start from one record. There are four entry points:

• Search: expand a log line and click ⌖ Investigate.
• Events: expand an event and click ⌖ Investigate.
• Traces: in the waterfall, click a span and choose ⌖ Investigate this span or ⌖ Investigate the whole trace.

The page lists everything connected to that record in time order. Timeline shows spans as bars with their real duration, and logs, events, releases and incidents as marks. Diagram draws the same records in columns by service, with the links between them. A details panel shows why each record is there, its ids, fields and stack trace, with ⌖ Investigate from here to re-anchor on it and Open in … to go to its own screen.

You can hide record kinds and relation types, and filter by service, the environment field your SDK sends, and severity. The page only shows records from the environment you are viewing. All of it lives in the URL, so ⧉ Copy link gives a teammate the exact view you are looking at.

Direct, same session, inferred

Every record on the page is labelled with how it got there. This is the part that matters.

Releases and incidents are the inferred records. A release of an involved service first seen in the 24 hours before the request, or during it, gets a confidence of 0.6 if it landed within an hour, otherwise 0.4. An incident on an involved service triggered within an hour either side gets 0.5. Each comes with a sentence that ends "not proof".

A log line from the same service in the same second, but with no trace id, is not shown at all. Time alone is not a relationship, so the page does not pretend it is one.

The ids the join uses

With Operation scope, ZipLogger takes the anchor's trace id and fetches the spans, logs and events that share it. With Session scope, it first finds the anchor's session id, collects the other trace ids that session touched (from its events and from spans carrying session.id), then does the same join for each. If nothing on the operation carries a session id, the Session button is disabled and says so.

What you need to send

The page can only join what arrives with the ids on it. Here is what each piece needs.

Traces. Your backend exports spans over OTLP to ZipLogger and continues incoming traceparent headers, which OpenTelemetry's HTTP server instrumentation does by default. The OpenTelemetry post has the setup.

Logs with the trace id. The .NET packages do this for you. ZipLogger.Extensions.Logging and ZipLogger.Serilog add traceId and spanId to a log's fields whenever there is a current Activity, which there is inside a traced request. Logs sent over OTLP carry the trace id in the record itself. For other languages, put the active trace id into the log's fields yourself as traceId. The traces docs show it for Python, Node and Go.

Events with a request id. In the browser, call instrumentFetch() from @ziplogger/browser. An event you track() within 5 seconds of an instrumented request gets that request's trace id as requestId (change the window with requestCorrelationTtlMs). On the server, events sent through ZipLogger.Client take the current request's trace id automatically. Anything else can send requestId itself.

Sessions across services. This needs @ziplogger/browser 0.6.0 or later, which sends the visitor's session id as W3C baggage next to traceparent. The backend has to copy it onto its spans. In ASP.NET Core, app.UseZipLoggerMetrics() from ZipLogger.Metrics.AspNetCore 0.6.0 does it, as long as it has an endpoint and API key configured. In other languages, use an OpenTelemetry baggage span processor. If your API is on another origin, its CORS config must allow both traceparent and baggage. The browser-to-backend tracing post walks through the headers and CORS.

What it looks like on the wire

I ran the real browser SDK source (0.6.0) under Node against two local servers: one standing in for the app's API, one for ZipLogger. The page made one failing request and then tracked an event:

zl.instrumentFetch({ propagateTo: [apiOrigin] })
await fetch(`${apiOrigin}/api/orders/pay`, { method: 'POST' })   // the API answers 502
zl.track('payment_failed', { provider: 'card' })

What each side received, shortened:

API got POST /api/orders/pay
  traceparent: 00-fb4d410e4c5c58dda5717868956205d2-73ea3cd74a5409b8-01
  baggage:     session.id=sess_d01241cc1f7776abed94

ZipLogger got POST /v1/traces
  span: traceId=fb4d410e4c5c58dda5717868956205d2 attrs={"http.request.method":"POST","session.id":"sess_d01241cc1f7776abed94","http.response.status_code":"502", ...}

ZipLogger got POST /ingest/v1/logs
  log: severity=error message="POST .../api/orders/pay failed with HTTP 502" fields={"traceId":"fb4d410e4c5c58dda5717868956205d2","spanId":"73ea3cd74a5409b8","httpStatus":502, ...}

ZipLogger got POST /ingest/v1/events
  event: name=payment_failed requestId=fb4d410e4c5c58dda5717868956205d2 sessionId=sess_d01241cc1f7776abed94

One trace id on the request header, the browser span, the failure log and the event. That is all the page needs to put them on one timeline. When the API continues that trace and logs with ZipLogger.Serilog, its spans and its error line join the same timeline as Direct. The session id on the span and the event is what Session scope follows.

The API behind it

The page calls one endpoint, which you can call too with the same sign-in token the app uses:

GET /api/v1/investigation?kind=log&id=LOG_ID&scope=operation
Authorization: Bearer YOUR_ACCESS_TOKEN

kind is event, log, trace or span. A span id is written {traceId}-{spanId}. scope is operation or session. Without from and to, the window is 15 minutes either side of the anchor for an operation and 12 hours either side for a session. A window you pass yourself can be at most 31 days. A record from another workspace, or one that has aged out, answers 404.

Limits

• Caps. One investigation reads at most 2,000 spans, 1,000 log lines and 500 events, and a session expands into at most 100 other traces (the ones nearest the anchor). When a cap is hit the page shows Partial data and names the limit.
• Logs join on trace id only. A log line without fields.traceId shows up only when you open the investigation from it.
• Old spans have no session id. Only spans sent after the SDK update carry it. Nothing is back-filled or guessed.
• A hop that drops baggage ends the session trail there. A proxy, a third-party service or an SDK without a baggage propagator. The trace itself keeps working.
• Browser check runs are not on the page. Monitor runs are left out of the timeline.
• Inferred confidence is a fixed rule of time and service, not something learned.
• Clock skew is reported, not fixed. If a child span starts more than a second before its parent, the page warns that the two clocks disagree and shows the times as reported.
• Removed duplicates are counted, not redrawn. If duplicate cleanup removed repeated error lines, the page shows their counts and first and last times. It does not invent rows for them.

To get the ids flowing, start with the traces docs and the browser SDK docs. If you don't have a workspace yet, sign up, send one traced request, and open it from the Traces page.