Kubernetes agent

EZGH Obs Agent is a Helm chart that sends a Kubernetes cluster’s telemetry to a project. It runs the OpenTelemetry Collector contrib distribution, version 0.162.0: a DaemonSet with one agent on every Linux node, and a single-replica Deployment that reads cluster state and events. It needs no operator, CRDs or changes to your applications. Observability’s Kubernetes section shows what it sends.

In the console, Observability > Send data > Kubernetes creates the ingest key and writes the commands below for your cluster. The chart’s source, values and changelog are at github.com/ezgamehost/ezgh-obs-agent.

Requirements

• Helm 3.17 or later, or Helm 4.
• Kubernetes 1.28 or later, with IPv4 Linux nodes that write container logs to /var/log/pods. Request and limit utilization needs 1.33 or later (see Pod specs and Kubernetes 1.33).
• Permission to create a DaemonSet, a Deployment and cluster RBAC.
• An ingest key that allows obs.logs.ingest, obs.metrics.ingest and obs.traces.ingest (the signals you send). Use one key per cluster, so you can revoke it on its own.

Windows nodes, serverless nodes without host paths, and namespaces with the restricted Pod Security level aren’t supported.

Install

Save the ingest key in a file named ingest-key.txt, without a trailing newline, and store it as a Secret. Then install the chart from GHCR:

kubectl create namespace ezgh-obs
kubectl -n ezgh-obs create secret generic ezgh-obs-ingest \
  --from-file=ingest-key=./ingest-key.txt

helm upgrade --install ezgh-obs-agent \
  oci://ghcr.io/ezgamehost/charts/ezgh-obs-agent --version 0.3.1 \
  --namespace ezgh-obs \
  --set-string cluster.name=production-us-west \
  --set-string ezgh.environment=production \
  --set-string auth.existingSecret=ezgh-obs-ingest \
  --wait --timeout 5m

• cluster.name is stored with every record as k8s.cluster.name. Keep it unique in the project and the same across upgrades: the Kubernetes section lists clusters by these names.
• ezgh.environment sets ezgh.metadata.environment on every record, which chooses the dataset; see Datasets are environments.
• The chart never creates a Secret or keeps the key in Helm values. Delete ingest-key.txt once the Secret exists, or create the Secret with your secret manager.
• Helm returns once the agents run. Data arriving is checked separately: open Send data > Kubernetes and select Verify, or search the environment’s logs.

For networks that can’t reach ghcr.io, each GitHub release has the same package: replace the oci:// reference and --version with the release’s .tgz URL.

What it collects

Telemetry without a service.name is named after its workload: its Deployment, StatefulSet, DaemonSet, CronJob or Job, or else its container. A Deployment’s container logs and pod metrics arrive as one service, with no change to the application. A service.name the application set is kept.

Cluster state

The chart’s cluster-state collector is a Deployment with one replica (<release>-cluster). It reads the Kubernetes API for what the kubelet doesn’t report: pod phase and status reasons, container restarts and readiness, why a container is waiting or terminated (such as CrashLoopBackOff or OOMKilled), CPU and memory requests and limits, Deployment, StatefulSet, DaemonSet, ReplicaSet, Job, CronJob and HorizontalPodAutoscaler replica counts, node conditions, allocatable CPU, memory and pods, namespace phase, resource quotas and PersistentVolumeClaims. It also watches events.

• One replica sends each series once. It’s replaced, not rolled, on upgrade, so cluster state has a gap of about 30 seconds. Its export queue is kept in the pod (an emptyDir), so it survives a restart of the collector but not a new pod: data queued when the pod is replaced is lost, and events that happen meanwhile aren’t sent.
• Its own RBAC. It runs as a separate ServiceAccount with a ClusterRole that allows only get, list and watch: on namespaces, nodes, pods, services, replication controllers, resource quotas and persistent volume claims, on Deployments, ReplicaSets, StatefulSets and DaemonSets, on Jobs and CronJobs, and on HorizontalPodAutoscalers (with metrics on), and on events.k8s.io events (with logs on). It can’t read Secrets or ConfigMaps, and cluster state adds nothing to the DaemonSet’s permissions.
• Turn it off with --set clusterState.enabled=false. That removes the Deployment, its ServiceAccount and its RBAC. The DaemonSet keeps sending logs and kubelet metrics, and the Kubernetes section’s panels that need cluster state say so.

Events

Each Kubernetes event (events.k8s.io) is one log record:

• An event that repeats is sent again, with its higher count.
• An event’s time is when the agent received it.
• Node events are usually in the default namespace.

Search them like any logs:

service:kubernetes-events level:warn resource.k8s.namespace.name:payments
service:kubernetes-events k8s.event.reason:FailedScheduling

Kubernetes labels on metrics

The agent copies these resource attributes onto every metric point whose resource has them, so PromQL can filter and group by them. It does this for kubelet, cluster-state and agent metrics, and for application metrics sent to the agent.

sum by (k8s_namespace_name) (k8s_pod_cpu_usage{k8s_cluster_name="production-us-west"})

• A point that already has the attribute keeps its own value.
• Resource attributes stay as they are, so logs and traces don’t change: filter them with resource.k8s.pod.name:… as before.
• Each node, pod and container is its own series. Data from agents before 0.3.0 has no such labels: series that differ only in their pod or node read as one series (see Names and labels).
• Every metric the agent sends has k8s_cluster_name, application metrics included. They get the pod and workload labels when their resource has them: when the application sets them, or when the agent adds them, which it does with otlp.internalTrafficPolicy: Local.
• job is the workload, as the agent names its service, on both kubelet and cluster-state pod and container metrics, so it means the same workload in both. A Deployment’s or other workload’s own metrics (k8s_deployment_desired, …) have job set to it. Node, namespace, HorizontalPodAutoscaler, ReplicaSet, quota and claim metrics have no job. Of the agent’s own metrics, only agent telemetry has instance.

In your own Collector

A Collector you run yourself, with k8s_attributes (k8sattributes before Collector Contrib 0.146), gets the same labels from this transform processor. Add it to your metrics pipeline after k8s_attributes:

processors:
  transform/k8slabels:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["k8s.cluster.name"], resource.attributes["k8s.cluster.name"]) where resource.attributes["k8s.cluster.name"] != nil and attributes["k8s.cluster.name"] == nil
          - set(attributes["k8s.namespace.name"], resource.attributes["k8s.namespace.name"]) where resource.attributes["k8s.namespace.name"] != nil and attributes["k8s.namespace.name"] == nil
          - set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"]) where resource.attributes["k8s.node.name"] != nil and attributes["k8s.node.name"] == nil
          - set(attributes["k8s.pod.name"], resource.attributes["k8s.pod.name"]) where resource.attributes["k8s.pod.name"] != nil and attributes["k8s.pod.name"] == nil
          - set(attributes["k8s.container.name"], resource.attributes["k8s.container.name"]) where resource.attributes["k8s.container.name"] != nil and attributes["k8s.container.name"] == nil
          - set(attributes["k8s.deployment.name"], resource.attributes["k8s.deployment.name"]) where resource.attributes["k8s.deployment.name"] != nil and attributes["k8s.deployment.name"] == nil
          - set(attributes["k8s.statefulset.name"], resource.attributes["k8s.statefulset.name"]) where resource.attributes["k8s.statefulset.name"] != nil and attributes["k8s.statefulset.name"] == nil
          - set(attributes["k8s.daemonset.name"], resource.attributes["k8s.daemonset.name"]) where resource.attributes["k8s.daemonset.name"] != nil and attributes["k8s.daemonset.name"] == nil
          - set(attributes["k8s.replicaset.name"], resource.attributes["k8s.replicaset.name"]) where resource.attributes["k8s.replicaset.name"] != nil and attributes["k8s.replicaset.name"] == nil
          - set(attributes["k8s.job.name"], resource.attributes["k8s.job.name"]) where resource.attributes["k8s.job.name"] != nil and attributes["k8s.job.name"] == nil
          - set(attributes["k8s.cronjob.name"], resource.attributes["k8s.cronjob.name"]) where resource.attributes["k8s.cronjob.name"] != nil and attributes["k8s.cronjob.name"] == nil
          - set(attributes["k8s.hpa.name"], resource.attributes["k8s.hpa.name"]) where resource.attributes["k8s.hpa.name"] != nil and attributes["k8s.hpa.name"] == nil
          - set(attributes["k8s.persistentvolumeclaim.name"], resource.attributes["k8s.persistentvolumeclaim.name"]) where resource.attributes["k8s.persistentvolumeclaim.name"] != nil and attributes["k8s.persistentvolumeclaim.name"] == nil
          - set(attributes["k8s.volume.name"], resource.attributes["k8s.volume.name"]) where resource.attributes["k8s.volume.name"] != nil and attributes["k8s.volume.name"] == nil
          - set(attributes["k8s.volume.type"], resource.attributes["k8s.volume.type"]) where resource.attributes["k8s.volume.type"] != nil and attributes["k8s.volume.type"] == nil
          - set(attributes["k8s.resourcequota.name"], resource.attributes["k8s.resourcequota.name"]) where resource.attributes["k8s.resourcequota.name"] != nil and attributes["k8s.resourcequota.name"] == nil
          - set(attributes["k8s.replicationcontroller.name"], resource.attributes["k8s.replicationcontroller.name"]) where resource.attributes["k8s.replicationcontroller.name"] != nil and attributes["k8s.replicationcontroller.name"] == nil

service:
  pipelines:
    metrics:
      processors: [memory_limiter, k8s_attributes, resource, transform/k8slabels]

The Kubernetes section also needs k8s.cluster.name on every resource; set it with the resource processor (key: k8s.cluster.name, action: insert), before transform/k8slabels as above, so it’s copied too.

Agent telemetry

Each agent sends its own Collector metrics every 60 seconds, through its normal metrics pipeline, as service.name ezgh-obs-agent. In PromQL they’re job="ezgh-obs-agent", with instance the node’s name, or cluster-state for the cluster-state collector:

count(otelcol_process_uptime_seconds_total{job="ezgh-obs-agent", instance!="cluster-state"})

counts the agents reporting. A cluster’s Agent tab charts them. Each agent also serves the same metrics in Prometheus format on its pod’s port 8888, for a scraper of your own.

Metrics reference

Names are as PromQL reads them (see Names and labels). Every series also has the Kubernetes labels that apply to it.

k8s_container_restarts is a gauge: increase() works on it, and delta() doesn’t warn that it might not be a counter. CPU usage (*_cpu_usage) is computed from two readings, so an agent’s first 30 seconds have none.

Pod specs and Kubernetes 1.33

The request and limit utilization metrics, and the volume type and claim labels, come from pod specs the agent reads from each node’s kubelet. Kubelets allow that with the nodes/pods permission from Kubernetes 1.33. Earlier versions require nodes/proxy, which also allows running commands in pods, so the chart never grants it: on Kubernetes 1.28 to 1.32 these metrics and labels are missing.

metrics.kubelet.podSpecs is auto: on when Helm reports Kubernetes 1.33 or later at install or upgrade. A cluster upgraded to 1.33 afterwards needs another helm upgrade to turn it on. Pod specs and volume metrics are read separately from the rest, so a kubelet that refuses them, such as a node pool still behind its upgraded control plane, loses only the utilization and volume metrics (and the agent logs a 403 each time), never its node, pod and container metrics. Set false if your kubelets have the KubeletFineGrainedAuthz feature gate off, which keeps volume metrics without their type and claim labels, and true on an earlier cluster only if its kubelets have it on.

Settings

Set these with --set or a values file. The chart’s values.yaml lists every setting, and its schema refuses unknown ones.

Send application telemetry

Applications in the cluster send OTLP to the agent’s Service. They don’t need the ingest key:

OTEL_EXPORTER_OTLP_ENDPOINT=http://ezgh-obs-agent.ezgh-obs.svc:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=checkout

For gRPC, use port 4317 and protocol grpc. The Service is reachable only inside the cluster, without authentication: any pod that can reach it can send to the project. Restrict callers with a NetworkPolicy.

Upgrade from 0.2

Run the install command again with the new version, and the same release name, namespace, cluster.name, environment and Secret:

helm upgrade --install ezgh-obs-agent \
  oci://ghcr.io/ezgamehost/charts/ezgh-obs-agent --version 0.3.1 \
  --namespace ezgh-obs \
  --set-string cluster.name=production-us-west \
  --set-string ezgh.environment=production \
  --set-string auth.existingSecret=ezgh-obs-ingest \
  --wait --timeout 5m

Pass the other settings you installed with too, or keep them with --reset-then-reuse-values. Don’t use --reuse-values: it leaves out settings that are new in 0.3.

0.3.1 changes nothing you configure: it uses the Collector’s current component names (k8s_attributes, file_log), so the agents start without deprecation warnings. Log positions carry over, so an upgrade from 0.3.0 doesn’t send any log line twice.

0.3.0 adds the cluster-state Deployment and its RBAC (on by default), events, the Kubernetes labels on metrics, more kubelet metrics (uptime, node inodes, and on Kubernetes 1.33 and later request and limit utilization) and agent telemetry. On 1.33 and later, the DaemonSet’s ClusterRole also gets get on nodes/pods. The Kubernetes section needs 0.3.0.

Volume. 0.3.0 sends more metric series: about 50% more for a typical cluster. A 10-node cluster with 300 pods goes from about 12,000 series to about 18,000, at 30 seconds. The Kubernetes labels add no series of their own. Events are one log record per event, usually far fewer than container logs. A dataset’s active series shows the change. Turning cluster state off keeps the kubelet additions and agent telemetry.

Operate

kubectl -n ezgh-obs get daemonset,deployment -l app.kubernetes.io/instance=ezgh-obs-agent
kubectl -n ezgh-obs logs -l app.kubernetes.io/instance=ezgh-obs-agent --tail=50

• Ready pods mean the agents run, not that EZGH accepted data. The cluster’s Agent tab in Kubernetes shows what each agent sends and fails to send.
• Delivery is at least once: each node agent queues to the node’s disk (64 MiB per signal by default), the cluster-state collector to its pod, and both retry rate limits and outages. Node loss, a full disk or a long outage can lose records.
• No utilization or volume metrics, and the agents log 403 for the kubelet’s /pods: the kubelets don’t accept nodes/pods. Upgrade the node pool, or set metrics.kubelet.podSpecs=false to get volume metrics without their type.
• No cluster state or events: check that the ezgh-obs-agent-cluster Deployment is running and clusterState.enabled is on.
• After rotating the key, restart the agents, which read it at startup: kubectl -n ezgh-obs rollout restart daemonset/ezgh-obs-agent deployment/ezgh-obs-agent-cluster.
• Uninstall with helm uninstall ezgh-obs-agent -n ezgh-obs. The Secret and the agents’ node directories stay; revoke the ingest key when you retire the cluster.