Kubernetes monitoring with one Helm chart

A pod is crash-looping, or a Deployment is stuck because a Secret is missing, and you find out when a customer does. kubectl get pods would have told you, but only if someone was looking. This post walks through ZipLogger's Kubernetes monitoring: one Helm chart per cluster, what it collects, what it needs from you, and what it does not do.

What gets installed

The chart installs two collectors in your cluster. Both are the upstream OpenTelemetry Collector image otel/opentelemetry-collector-k8s, pinned by digest, with a configuration ZipLogger writes. There is no ZipLogger agent binary.

Collector	Runs as	Job
`ziplogger-cluster`	A Deployment, one replica	Watches Kubernetes objects and events across the cluster, re-lists everything every 5 minutes to catch what a watch missed, sends workload and node state metrics
`ziplogger-agent`	A DaemonSet, one pod per node	Reads CPU, memory and disk usage from the node's kubelet, ships container logs, runs an in-cluster OTLP endpoint

Both only read from Kubernetes, and both only connect outbound over HTTPS to ZipLogger. Nothing in your cluster has to accept inbound connections.

Before you install

You need Kubernetes 1.24 or newer (the chart refuses older clusters), Helm 3 or 4, kubectl rights to create a namespace, cluster roles and bindings, a Deployment and a DaemonSet, and outbound HTTPS from your nodes to app.ziplogger.ai. A proxy is supported. Nodes pull the collector image from Docker Hub, or from a mirror you set with image.repository and imagePullSecrets. Someone with the Admin role in ZipLogger creates the key.

If your cluster enforces Pod Security admission, container log collection needs the release namespace to allow Privileged, because the agent mounts /var/log/pods read-only from each node. If you can't allow that, set collection.logs=false. Without container logs, both collectors meet the Restricted level. On EKS Fargate, DaemonSets don't run, so Fargate pods get no usage metrics or container logs, but their state, events and problems still come through.

The chart uses only standard Kubernetes APIs and the kubelet's stats endpoint. It has been tested on kind clusters with Linux nodes, not yet on EKS, GKE, AKS or other managed clouds.

Step 1: create a collector key

Open Kubernetes → Install in ZipLogger. The page names the environment the cluster will report into. Check it: a key belongs to one environment and can't be moved later.

Click Create collector key. Copy the key now, because it is shown once. The page also puts it into the install command for you.

This is not the ingestion key you put in a browser app. Cluster state can open incidents, so a public key must not be able to report it. The collector key carries the kubernetes and ingest scopes and counts toward your plan's API key limit. Keep it in the cluster.

Step 2: install the chart

helm repo add ziplogger https://ziplogger.ai/charts
helm repo update
helm install ziplogger ziplogger/ziplogger \
  --namespace ziplogger --create-namespace \
  --set apiKey=zk_... \
  --set cluster.name=YOUR_CLUSTER_NAME

That is the whole configuration. There is no environment setting in the chart: the key decides the workspace and the environment.

cluster.name is optional with helm install. Without it, the chart reads the UID of your kube-system namespace and the cluster shows up as cluster- plus the first eight characters of that UID. You can rename it in ZipLogger later.

With Argo CD, Flux or helm template, set cluster.name. They render without cluster access, so the chart can't read the UID and stops with "Could not read the kube-system namespace". Pick one way of installing a cluster and keep it: switching between helm install and an offline renderer changes the identity, and the cluster appears as a new one.

Keep the key out of Helm

With --set apiKey, the key is also stored in the Helm release, and helm get values shows it. To avoid that, create the Secret yourself and point the chart at it:

kubectl create namespace ziplogger
kubectl -n ziplogger create secret generic ziplogger-key \
  --from-literal=api-key=zk_...

helm install ziplogger ziplogger/ziplogger -n ziplogger \
  --set ziplogger.existingSecret.name=ziplogger-key \
  --set cluster.name=YOUR_CLUSTER_NAME

The chart reads the data key api-key. If yours is named differently, set ziplogger.existingSecret.key.

Settings you might change

Every other value has a default. The chart validates its values, so a misspelled setting fails the install instead of being ignored. The ones you are most likely to touch:

Value	Default	Use it to
`collection.logs`	`true`	Turn off container log shipping if your logs already reach ZipLogger another way
`collection.traces`	`true`	Turn off the in-cluster OTLP endpoint if you don't use it
`namespaces.include`	all	Monitor only some namespaces on a shared cluster
`namespaces.exclude`	none	Skip namespaces such as `kube-system`
`proxy.httpsProxy`	none	Send through an outbound proxy
`nodeAgent.checkpoints`	`emptyDir`	`hostPath` keeps log positions across agent upgrades, so no lines are skipped

With collection.traces on, each node runs an OTLP endpoint on 4317 (gRPC) and 4318 (HTTP). It adds pod, workload and node names to what your applications send and forwards it to ZipLogger:

OTEL_EXPORTER_OTLP_ENDPOINT=http://ziplogger-otlp.ziplogger:4318

Your applications need no key for it, because the agent adds the collector key when it forwards. So any pod that can reach the Service can send telemetry into your workspace. If you don't use the endpoint, turn it off.

Step 3: check it works

helm test ziplogger -n ziplogger
kubectl -n ziplogger get pods
kubectl -n ziplogger logs deploy/ziplogger-cluster --tail=50

helm test checks the cluster collector is healthy from inside the cluster, not that data reaches ZipLogger. get pods should show one ziplogger-cluster pod and one ziplogger-agent pod per node, all Running. The log should have no 401, 403 or Exporting failed lines. A 401 means the key is wrong, revoked, or not a collector key. A 403 with plan_feature means your plan does not include Kubernetes monitoring.

In ZipLogger, the Install page shows Waiting for the first data from your cluster, then Connected. The cluster usually appears under Kubernetes → Clusters within a minute. Problem detection starts once the first full listing of the cluster has arrived.

What you see

Kubernetes → Clusters has one card per cluster with its state (healthy, degraded, failing or not reporting), nodes ready, pods running, pods failing and open problems. A cluster's page has Overview, Workloads, Pods, Nodes and Events tabs. The overview shows active problems, a namespace summary, the pods with the most restarts, node usage, and a Collectors table with each collector's sent, failed and dropped data.

Open a workload, pod or node and you get its state, containers, CPU and memory against limits, recent events, past problems, recent container logs, and a timeline of the last two hours that puts events, rollouts, problems, incidents, error logs and error traces in one order.

Problems and incidents

ZipLogger reads failures from Kubernetes' own state and events. A problem names the thing you would fix (the Deployment, not each of its pods), how many pods are affected, the reason Kubernetes gave, and the evidence. When Kubernetes did not state a cause, the problem says so instead of guessing.

These open an incident by default, among others: Container crash-looping, Image cannot be pulled, Missing Secret / ConfigMap (config error), Container OOMKilled, Pod cannot be scheduled, Pod stuck in Pending (after 15 minutes), Rollout stuck, Node not ready, Job failed and CronJob failing. Some are detected but off by default: Excessive container restarts, Health probe failing, Pod evicted, High CPU usage and High memory usage.

Monitors decide which problems open incidents. Under Kubernetes → Monitors, Editors and Admins can switch them on or off, change their thresholds and severity, and add monitors for one cluster or one namespace.

One problem is one incident, however many pods and repeated events it involves, and the incident resolves on its own once the problem has not been seen for 3 minutes. Nothing is raised for a namespace, workload, pod or Job that is being deleted.

It can say Secret "database-password" was not found without reading any Secret: the name comes from Kubernetes' own error message and the pod's reference to it.

Installing the chart does not set up paging. A Kubernetes incident is recorded, but nobody is told until you route it. In Kubernetes → Monitors, edit a monitor and choose an Escalation policy, or connect a PagerDuty or Opsgenie destination.

Permissions and data

Each collector has its own ServiceAccount and ClusterRole, and every permission is get, list or watch. Not granted: write verbs, wildcards, Secrets, ConfigMaps, pods/exec, pods/attach, pods/portforward and nodes/proxy. The containers run as non-root with a read-only root filesystem and all Linux capabilities dropped.

Objects pass an allow-list inside the cluster before anything is sent. Names, labels, owners, workload settings, images, resource requests and limits, status fields and events leave the cluster, and so do the names of environment variables and of the Secrets and ConfigMaps a container refers to. Environment variable values, annotations (except a Deployment's revision number), container commands and arguments, and Secret or ConfigMap contents never leave. Event messages and container logs are sent as written, so they contain whatever your applications printed.

Upgrade, rotate, uninstall

To upgrade:

helm repo update
helm upgrade ziplogger ziplogger/ziplogger -n ziplogger --reuse-values

helm rollback ziplogger -n ziplogger goes back a revision. To rotate the key, create a new one on Kubernetes → Install in the same environment. Then:

helm upgrade ziplogger ziplogger/ziplogger -n ziplogger --reuse-values \
  --set apiKey=zk_NEW...

Both collectors restart on their own. With your own Secret, update it and run kubectl -n ziplogger rollout restart deploy/ziplogger-cluster ds/ziplogger-agent. When the Last report column on the cluster's page is current, revoke the old key under Settings → API keys. The cluster keeps its history.

To uninstall:

helm uninstall ziplogger -n ziplogger
kubectl delete namespace ziplogger

This removes only what the chart created and changes none of your workloads. ZipLogger can't tell an uninstall from an outage, so about 10 minutes later Cluster stopped reporting opens an incident. To remove the cluster and its stored data, open it under Kubernetes → Clusters, click rename next to its name, then Remove cluster (Admins only). That also resolves its open incidents.

What it does not do

No remediation. It never restarts, scales, deletes or changes anything. It tells you; you fix it.
No control plane monitoring. API server, etcd and scheduler metrics are not collected, nor anything the Kubernetes API and kubelet don't report.
No network usage. Per-container usage is kept as the latest value only; node and pod usage are kept as history.
No old logs. Container logs start when the agent starts on a node. Lines over 256 KiB are cut.
Limits. Up to 50 clusters per environment and 100,000 live objects per cluster. Container logs and in-cluster OTLP telemetry count toward your plan like any other ingestion.
Retention. Events are kept 7 days, deleted objects 3 days, resolved problems 30 days. A cluster that sends nothing for 30 days is removed. Container logs follow your log retention.

If that fits, install it on one non-production cluster first and break something on purpose: deploy an image tag that doesn't exist and watch Image cannot be pulled open. The Kubernetes docs list every value and permission, or create a workspace to get a collector key.