The Kubernetes section of Observability shows every cluster that runs EZGH Obs Agent 0.3.0: its nodes, namespaces, workloads and pods, what’s wrong with them now, the cluster’s events and whether the agents are healthy. It needs nothing to set up, and EZGH maintains it: it improves with console releases. Pages mix metrics with the workload’s logs and the cluster’s events and, where an application sends traces, its requests, errors and latency.
Open it
In the console, open Observability > Kubernetes. You can also get there from:
- Send data > Kubernetes: after Verify, Open cluster opens the cluster you just installed the agent on.
- The Overview page’s Kubernetes card, with nodes, running pods and problems.
- The ⋯ menu of a log record or span from a pod: Open pod.
- The console’s search: each cluster has an entry.
Clusters lists each cluster with its environment, nodes (ready of total), running pods, CPU and memory used of allocatable, open problems and when it last reported. With one cluster, the section opens it. With none, it links to Send data > Kubernetes, and says when the project has data from an older agent that needs upgrading.
A cluster’s environment is the one its agent sends to, so pages need no environment picker. Keep
each cluster’s cluster.name unique in the project, as the agent requires.
Pages and URLs
Every page and tab has its own URL, so a copied link opens the same view. The time range
(?range=, or ?from= and ?to=) carries across tabs and links. Names in URLs are the Kubernetes
names.
URL, under /obs/kubernetes |
Page | Tabs |
|---|---|---|
| (none) | Clusters | |
/clusters/<cluster> |
A cluster | Overview, Nodes, Namespaces, Workloads, Pods, Problems, Events, Agent |
/clusters/<cluster>/nodes/<node> |
A node | Overview, Pods, Logs, Events |
/clusters/<cluster>/namespaces/<namespace> |
A namespace | Overview, Workloads, Pods, Events |
/clusters/<cluster>/workloads/<namespace>/<workload> |
A workload | Overview, Pods, Logs, Events, Traces |
/clusters/<cluster>/pods/<namespace>/<pod> |
A pod | Overview, Logs, Events |
A cluster’s tabs are at /clusters/<cluster>/nodes, /namespaces, /workloads, /pods,
/problems, /events and /agent.
A workload is named after its Deployment, StatefulSet, DaemonSet, CronJob or Job, the way the
agent names its service (see What it collects),
and its page shows its kind. Its page shows the application’s traces and OTLP logs when the
application sends them under the same service.name, or with none set, so the agent names them
after the workload.
Lists
The Nodes, Namespaces, Workloads and Pods tabs list what’s there now: what reported in the last
5 minutes, at the end of the time range. Charts are on each resource’s page. Lists have a search
box and sortable columns, filters such as ?namespace=, ?kind=, ?status= and ?node= are kept
in the URL, and each row opens the resource’s page.
| List | Columns | Status |
|---|---|---|
| Nodes | Node, status, CPU, memory and disk used, pods, uptime | NotReady when its Ready condition is false; Unknown when it’s unknown; the pressure (memory, disk or PID) when there is one; else Ready. A node whose kubelet stopped reporting stays listed, with dashes |
| Namespaces | Namespace, workloads, pods running, CPU (cores), memory, restarts in the range, problems | No status: the problems count is colored by the worst one |
| Workloads | Workload, kind, namespace, ready of desired, CPU (cores), memory, restarts in the range, status | Deployments, StatefulSets and DaemonSets: Healthy, Degraded when fewer than desired are ready, Unavailable when none are, or Scaled to zero. Jobs and CronJobs: Failed when the last Job failed, Running, Complete or Scheduled |
| Pods | Pod, namespace, workload, node, status, ready containers, restarts, CPU, memory, age | A container’s reason when one is set, in this order: CrashLoopBackOff, ImagePullBackOff, ErrImagePull, CreateContainerConfigError, ContainerCannotRun, OOMKilled, Error, ContainerCreating, Completed. Else the pod’s phase: Pending, Running, Succeeded, Failed or Unknown |
Without cluster state, lists keep their CPU, memory and disk columns and show dashes for status, ready and restarts, with a note saying why.
Problems
The Problems tab lists what’s wrong in the cluster now, critical first. Each row has the severity, the resource (linked to its page), what’s wrong, since when in the time range, and a detail such as restarts, a reason or a percentage. The cluster’s Overview tab starts with the problem count and the top five, and Clusters shows each cluster’s count.
| Problem | Severity |
|---|---|
| Node not ready | Critical |
| Node readiness unknown | Warning |
| Node under memory, disk or PID pressure | Warning |
| Node without an agent (the node reports, but no agent runs on it) | Warning |
Container waiting or crashing: CrashLoopBackOff, ImagePullBackOff, ErrImagePull or CreateContainerConfigError |
Critical |
Container OOM-killed (its reason was OOMKilled) in the range |
Warning |
| Pod pending for over 5 minutes, unless one of its containers is waiting | Critical |
| Pod failed, outside a Job (a Job’s failed pod counts as its Job’s problem) | Critical |
| Pod phase unknown | Warning |
| Deployment, StatefulSet or DaemonSet below its desired replicas | Warning; critical when none are ready |
| Job failed, with no successful pod | Critical |
| Volume over 85% full | Warning; critical over 95% |
| Container over 90% of its memory limit | Warning |
| Node disk over 85% full | Warning; critical over 95% |
Most problems need cluster state, including the memory-limit problem, which reads containers’ limits from it. Volume and node disk problems need kubelet metrics.
Events
The Events tab lists the cluster’s Kubernetes events, newest first, and filters them by type (Warning or Normal), namespace, reason and object kind. Each row links to its object’s page. Node, namespace, workload and pod pages have their own Events tab. Events are logs, so you can also search them on the Logs page; see Events.
Overview tabs
A cluster’s, node’s, namespace’s, workload’s or pod’s Overview tab is a dashboard, set to that resource. The time range picker and refresh work as on any dashboard; there are no variable pickers, because the URL sets them. Rows in the tables and top lists of nodes, namespaces, workloads and pods open that resource’s page.
| Overview of | Shows |
|---|---|
| A cluster | Health: nodes, nodes not ready, running pods, pods not running, restarts, warning events. Capacity: CPU and memory used, requested and allocatable. CPU and memory by namespace, a namespaces table, a nodes table, restarting containers, unavailable replicas, pods not running, warning events, and the fullest PersistentVolumeClaim volumes |
| A namespace | The cluster’s panels, for that namespace |
| A node | CPU, memory and filesystem used, running pods and uptime; CPU, memory, filesystem and network over time; node conditions; the node’s pods; error logs |
| A workload | Pods ready of desired, restarts, CPU, memory and error logs; CPU and memory per pod against requests and limits; containers closest to their memory limit; restarts per pod; why containers are waiting or terminated; network; autoscaling; the workload’s pods; log volume by level, warnings and errors; events in the namespace; requests, error rate and p95 latency from span metrics when the application sends traces |
| A pod | Phase, ready containers, restarts, CPU, memory and uptime; a containers table; why containers are waiting or terminated; CPU and memory per container; filesystem; network; volumes; events; logs |
Panels that need cluster state show why when it’s off, instead of “No data in this range”. The ⋯ menu shows its revision, such as “Kubernetes / Cluster, revision 1”, which changes when EZGH updates it.
Agent tab
A cluster’s Agent tab shows whether the agents are healthy and your data is arriving, from
the agent telemetry each agent sends: agents
reporting, nodes without an agent, whether the cluster-state collector runs, records, points and
spans sent per second, send failures, export queue fill (a rising queue means EZGH is unreachable
or limiting the agent’s rate), refusals by the memory limiter, agent memory and CPU, and a table of
agents. The agent doesn’t collect its own logs; read them with
kubectl -n ezgh-obs logs -l app.kubernetes.io/instance=ezgh-obs-agent.
Duplicate as dashboard
To change an Overview tab, select Duplicate as dashboard. It creates an ordinary dashboard in the project from the tab, with its variables set to this resource, named after the page, such as “Kubernetes / Cluster: production-us-west”. You own the copy and can edit, share, version and manage it as code, with pickers for its cluster, namespace or other variables. A copy doesn’t get later improvements. The ⋯ menu also has View dashboard JSON.
Rows in a copy still open the section’s pages: a row link to builtin:kubernetes-pod opens the
pod’s page. To open dashboards of your own instead, change each table’s and top list’s
viz.rowLink.dashboard to your dashboard’s id (see Row links).
Requirements
| The section needs | For |
|---|---|
| Agent 0.3.0 or later | Everything. Earlier agents send no k8s_cluster_name or other Kubernetes labels, so their nodes and pods can’t be told apart. See Upgrade to 0.3.0 |
Cluster state (clusterState.enabled, on by default) |
Status, phase, readiness, restarts, requests and limits, allocatable capacity, node conditions, replica counts, container reasons, most problems, and events. See Cluster state |
| Kubelet metrics (on by default) | CPU, memory, filesystem, network and volume usage |
| Container logs (on by default) | The Logs tabs and log panels |
| Traces from your applications, named after their workload | A workload’s request, error and latency panels, and its Traces tab |
| Kubernetes 1.33 or later | The fullest PersistentVolumeClaim volumes, which need pod specs from the kubelet; see Pod specs and Kubernetes 1.33 |
Data from before an upgrade to 0.3.0 doesn’t appear.
Access
The section isn’t a resource, so viewing it needs no extra action. Every page is read as you, on
the cluster’s environment: the section needs obs.metrics.query, the Events and Logs tabs and log
panels obs.logs.search, and the Traces tab and trace panels obs.traces.search.
ObsFullAccess, ObsReadOnlyAccess and ReadOnlyAccess include all three. Without one, the page
or tab names the action you need. Reading isn’t recorded in Trails.
Duplicate as dashboard needs obs.dashboards.create, counts toward the dashboards quota, and
is recorded as CreateDashboard, like creating any dashboard.
Limits
Pages run queries like any other, so the metrics query budgets and search limits apply to each. Long time ranges on large clusters, such as 7 days with thousands of pods, can be over a panel’s budget. The panel then shows “Too much data. Narrow the time range or filters.”, and the rest of the page still loads. Pick a shorter range.