--- title: "Kubernetes" description: "Explore clusters that run EZGH Obs Agent, from the cluster down to nodes, namespaces, workloads and pods, with their problems, events and agent health." --- > Documentation Index > Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Kubernetes The **Kubernetes** section of Observability shows every cluster that runs [EZGH Obs Agent](/observability/kubernetes-agent) 0.3.0: its nodes, namespaces, workloads and pods, what's wrong with them now, the cluster's events and whether the agents are healthy. It needs nothing to set up, and EZGH maintains it: it improves with console releases. Pages mix metrics with the workload's logs and the cluster's events and, where an application sends traces, its requests, errors and latency. ## Open it In the [console](https://console.ezghcloud.com), open **Observability** > **Kubernetes**. You can also get there from: - **Send data** > **Kubernetes**: after **Verify**, **Open cluster** opens the cluster you just installed the agent on. - The **Overview** page's **Kubernetes** card, with nodes, running pods and problems. - The ⋯ menu of a log record or span from a pod: **Open pod**. - The console's search: each cluster has an entry. **Clusters** lists each cluster with its environment, nodes (ready of total), running pods, CPU and memory used of allocatable, open problems and when it last reported. With one cluster, the section opens it. With none, it links to **Send data** > **Kubernetes**, and says when the project has data from an older agent that needs upgrading. A cluster's environment is the one its agent sends to, so pages need no environment picker. Keep each cluster's `cluster.name` unique in the project, as the agent requires. ## Pages and URLs Every page and tab has its own URL, so a copied link opens the same view. The time range (`?range=`, or `?from=` and `?to=`) carries across tabs and links. Names in URLs are the Kubernetes names. | URL, under `/obs/kubernetes` | Page | Tabs | | --- | --- | --- | | (none) | Clusters | | | `/clusters/` | A cluster | **Overview**, **Nodes**, **Namespaces**, **Workloads**, **Pods**, **Problems**, **Events**, **Agent** | | `/clusters//nodes/` | A node | Overview, Pods, Logs, Events | | `/clusters//namespaces/` | A namespace | Overview, Workloads, Pods, Events | | `/clusters//workloads//` | A workload | Overview, Pods, Logs, Events, Traces | | `/clusters//pods//` | A pod | Overview, Logs, Events | A cluster's tabs are at `/clusters//nodes`, `/namespaces`, `/workloads`, `/pods`, `/problems`, `/events` and `/agent`. A workload is named after its Deployment, StatefulSet, DaemonSet, CronJob or Job, the way the agent names its service (see [What it collects](/observability/kubernetes-agent#what-it-collects)), and its page shows its kind. Its page shows the application's traces and OTLP logs when the application sends them under the same `service.name`, or with none set, so the agent names them after the workload. ## Lists The Nodes, Namespaces, Workloads and Pods tabs list what's there **now**: what reported in the last 5 minutes, at the end of the time range. Charts are on each resource's page. Lists have a search box and sortable columns, filters such as `?namespace=`, `?kind=`, `?status=` and `?node=` are kept in the URL, and each row opens the resource's page. | List | Columns | Status | | --- | --- | --- | | Nodes | Node, status, CPU, memory and disk used, pods, uptime | **NotReady** when its `Ready` condition is false; **Unknown** when it's unknown; the pressure (memory, disk or PID) when there is one; else **Ready**. A node whose kubelet stopped reporting stays listed, with dashes | | Namespaces | Namespace, workloads, pods running, CPU (cores), memory, restarts in the range, problems | No status: the problems count is colored by the worst one | | Workloads | Workload, kind, namespace, ready of desired, CPU (cores), memory, restarts in the range, status | Deployments, StatefulSets and DaemonSets: **Healthy**, **Degraded** when fewer than desired are ready, **Unavailable** when none are, or **Scaled to zero**. Jobs and CronJobs: **Failed** when the last Job failed, **Running**, **Complete** or **Scheduled** | | Pods | Pod, namespace, workload, node, status, ready containers, restarts, CPU, memory, age | A container's reason when one is set, in this order: `CrashLoopBackOff`, `ImagePullBackOff`, `ErrImagePull`, `CreateContainerConfigError`, `ContainerCannotRun`, `OOMKilled`, `Error`, `ContainerCreating`, `Completed`. Else the pod's phase: **Pending**, **Running**, **Succeeded**, **Failed** or **Unknown** | Without [cluster state](/observability/kubernetes-agent#cluster-state), lists keep their CPU, memory and disk columns and show dashes for status, ready and restarts, with a note saying why. ## Problems The **Problems** tab lists what's wrong in the cluster now, critical first. Each row has the severity, the resource (linked to its page), what's wrong, since when in the time range, and a detail such as restarts, a reason or a percentage. The cluster's Overview tab starts with the problem count and the top five, and **Clusters** shows each cluster's count. | Problem | Severity | | --- | --- | | Node not ready | Critical | | Node readiness unknown | Warning | | Node under memory, disk or PID pressure | Warning | | Node without an agent (the node reports, but no agent runs on it) | Warning | | Container waiting or crashing: `CrashLoopBackOff`, `ImagePullBackOff`, `ErrImagePull` or `CreateContainerConfigError` | Critical | | Container OOM-killed (its reason was `OOMKilled`) in the range | Warning | | Pod pending for over 5 minutes, unless one of its containers is waiting | Critical | | Pod failed, outside a Job (a Job's failed pod counts as its Job's problem) | Critical | | Pod phase unknown | Warning | | Deployment, StatefulSet or DaemonSet below its desired replicas | Warning; critical when none are ready | | Job failed, with no successful pod | Critical | | Volume over 85% full | Warning; critical over 95% | | Container over 90% of its memory limit | Warning | | Node disk over 85% full | Warning; critical over 95% | Most problems need cluster state, including the memory-limit problem, which reads containers' limits from it. Volume and node disk problems need kubelet metrics. ## Events The **Events** tab lists the cluster's Kubernetes events, newest first, and filters them by type (Warning or Normal), namespace, reason and object kind. Each row links to its object's page. Node, namespace, workload and pod pages have their own **Events** tab. Events are logs, so you can also search them on the Logs page; see [Events](/observability/kubernetes-agent#events). ## Overview tabs A cluster's, node's, namespace's, workload's or pod's **Overview** tab is a dashboard, set to that resource. The time range picker and refresh work as on any dashboard; there are no variable pickers, because the URL sets them. Rows in the tables and top lists of nodes, namespaces, workloads and pods open that resource's page. | Overview of | Shows | | --- | --- | | A cluster | Health: nodes, nodes not ready, running pods, pods not running, restarts, warning events. Capacity: CPU and memory used, requested and allocatable. CPU and memory by namespace, a namespaces table, a nodes table, restarting containers, unavailable replicas, pods not running, warning events, and the fullest PersistentVolumeClaim volumes | | A namespace | The cluster's panels, for that namespace | | A node | CPU, memory and filesystem used, running pods and uptime; CPU, memory, filesystem and network over time; node conditions; the node's pods; error logs | | A workload | Pods ready of desired, restarts, CPU, memory and error logs; CPU and memory per pod against requests and limits; containers closest to their memory limit; restarts per pod; why containers are waiting or terminated; network; autoscaling; the workload's pods; log volume by level, warnings and errors; events in the namespace; requests, error rate and p95 latency from [span metrics](/observability/span-metrics) when the application sends traces | | A pod | Phase, ready containers, restarts, CPU, memory and uptime; a containers table; why containers are waiting or terminated; CPU and memory per container; filesystem; network; volumes; events; logs | Panels that need cluster state show why when it's off, instead of "No data in this range". The ⋯ menu shows its revision, such as "Kubernetes / Cluster, revision 1", which changes when EZGH updates it. ## Agent tab A cluster's **Agent** tab shows whether the agents are healthy and your data is arriving, from the [agent telemetry](/observability/kubernetes-agent#agent-telemetry) each agent sends: agents reporting, nodes without an agent, whether the cluster-state collector runs, records, points and spans sent per second, send failures, export queue fill (a rising queue means EZGH is unreachable or limiting the agent's rate), refusals by the memory limiter, agent memory and CPU, and a table of agents. The agent doesn't collect its own logs; read them with `kubectl -n ezgh-obs logs -l app.kubernetes.io/instance=ezgh-obs-agent`. ## Duplicate as dashboard To change an Overview tab, select **Duplicate as dashboard**. It creates an ordinary [dashboard](/observability/dashboards) in the project from the tab, with its variables set to this resource, named after the page, such as "Kubernetes / Cluster: production-us-west". You own the copy and can edit, share, version and manage it as code, with pickers for its cluster, namespace or other variables. A copy doesn't get later improvements. The ⋯ menu also has **View dashboard JSON**. Rows in a copy still open the section's pages: a row link to `builtin:kubernetes-pod` opens the pod's page. To open dashboards of your own instead, change each table's and top list's `viz.rowLink.dashboard` to your dashboard's `id` (see [Row links](/observability/dashboards#row-links)). ## Requirements | The section needs | For | | --- | --- | | Agent **0.3.0** or later | Everything. Earlier agents send no `k8s_cluster_name` or other Kubernetes labels, so their nodes and pods can't be told apart. See [Upgrade to 0.3.0](/observability/kubernetes-agent#upgrade-to-030) | | **Cluster state** (`clusterState.enabled`, on by default) | Status, phase, readiness, restarts, requests and limits, allocatable capacity, node conditions, replica counts, container reasons, most problems, and events. See [Cluster state](/observability/kubernetes-agent#cluster-state) | | Kubelet metrics (on by default) | CPU, memory, filesystem, network and volume usage | | Container logs (on by default) | The Logs tabs and log panels | | Traces from your applications, named after their workload | A workload's request, error and latency panels, and its Traces tab | | Kubernetes **1.33** or later | The fullest PersistentVolumeClaim volumes, which need pod specs from the kubelet; see [Pod specs and Kubernetes 1.33](/observability/kubernetes-agent#pod-specs-and-kubernetes-133) | Data from before an upgrade to 0.3.0 doesn't appear. ## Access The section isn't a resource, so viewing it needs no extra action. Every page is read as you, on the cluster's environment: the section needs `obs.metrics.query`, the Events and Logs tabs and log panels `obs.logs.search`, and the Traces tab and trace panels `obs.traces.search`. `ObsFullAccess`, `ObsReadOnlyAccess` and `ReadOnlyAccess` include all three. Without one, the page or tab names the action you need. Reading isn't recorded in [Trails](/trails). **Duplicate as dashboard** needs `obs.dashboards.create`, counts toward the dashboards quota, and is recorded as `CreateDashboard`, like creating any dashboard. ## Limits Pages run queries like any other, so the [metrics query budgets](/observability/metrics#query-budgets) and [search limits](/observability/limits#search-and-async-queries) apply to each. Long time ranges on large clusters, such as 7 days with thousands of pods, can be over a panel's budget. The panel then shows "Too much data. Narrow the time range or filters.", and the rest of the page still loads. Pick a shorter range. Source: https://docs.ezghcloud.com/observability/kubernetes/index.mdx