---
title: "Observability"
description: "Control-plane logs and metrics, and node and workload telemetry, in Obs."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

## Control-plane logs

A cluster can send any of five control-plane log types to an [Obs](/observability) project you name:

| Log type | What it sends |
| --- | --- |
| `api` | The Kubernetes API server's logs |
| `audit` | The API server's audit log: who did what in the cluster |
| `authenticator` | One entry per sign-in with an EZGH cluster token or a node's instance identity: time, credential type, whether it was authenticated, username, uid, groups and error. Never the token |
| `controllerManager` | The controller manager's logs |
| `scheduler` | The scheduler's logs |

Choose them when you create the cluster, or later: in the console on the cluster's Observability tab,
with `--logging-type` (repeat it for more) and `--logging-project`, or with `logging` in
[the API](/kubernetes-api/UpdateCluster/).

```sh
ezgh kubernetes clusters update kcl_3k9d0a1m2x7q --logging-type audit --logging-type authenticator \
  --logging-project prj_o11y00000000 --wait
```

A change applies to the running control plane; a log type turned off stops once the update finishes.
`--no-logging` stops them all.

Each record has `service.name` `ezgh.kubernetes`, the attribute `cluster` set to the cluster's ID and
`kubernetes.log_type` set to its log type. Logs are billed as log ingest in that project. Sending logs
to a project needs `obs.logs.ingest` on it.

## Control-plane metrics

A cluster can send its API server, etcd, scheduler and controller manager metrics to Obs, free. It's
off by default. Turn it on in the console with Send metrics, on Create cluster or the cluster's
Observability tab, with `--control-plane-metrics` on `ezgh kubernetes clusters create` or `update`,
or with `controlPlaneMetrics` on [create](/kubernetes-api/CreateCluster/) or
[update](/kubernetes-api/UpdateCluster/).
`--control-plane-metrics=false` turns it off.

```sh
ezgh kubernetes clusters update kcl_3k9d0a1m2x7q --control-plane-metrics --wait
```

Metrics arrive once a minute in the cluster's own project, in the `production` environment, with
`service.name` `ezgh.kubernetes`. Each series is labelled `cluster` (the cluster's ID), `component`
(`apiserver`, `etcd`, `scheduler` or `controllerManager`) and `member` (a number for each control-plane server; a replacement gets a new number). Other labels
are summed away, so only those listed below are kept. Each `member` sends at most 2,000 series a
minute; past that, API server series are left out first.

| Metric | Labels kept |
| --- | --- |
| `apiserver_request_total` | `verb`, `code` |
| `apiserver_request_duration_seconds` | `verb` |
| `apiserver_current_inflight_requests` | `request_kind` |
| `apiserver_resource_objects` | `group`, `resource` |
| `apiserver_requested_deprecated_apis` | `group`, `version`, `resource`, `removed_release` |
| `apiserver_admission_webhook_rejection_count` | `name`, `operation`, `rejection_code` |
| `etcd_server_has_leader` | None |
| `etcd_server_leader_changes_seen_total` | None |
| `etcd_server_proposals_failed_total` | None |
| `etcd_mvcc_db_total_size_in_bytes` | None |
| `etcd_mvcc_db_total_size_in_use_in_bytes` | None |
| `etcd_disk_wal_fsync_duration_seconds` | None |
| `etcd_disk_backend_commit_duration_seconds` | None |
| `etcd_network_peer_round_trip_time_seconds` | `To` |
| `scheduler_pending_pods` | `queue` |
| `scheduler_schedule_attempts_total` | `result`, `profile` |
| `scheduler_scheduling_attempt_duration_seconds` | `result`, `profile` |
| `leader_election_master_status` | `name` |
| `workqueue_depth` | `name` |
| `workqueue_adds_total` | `name` |
| `rest_client_requests_total` | `code`, `method` |

## Nodes and workloads

Nodes are instances, so their [instance metrics](/compute/monitoring) flow to Obs as any instance's do.

For logs, traces and metrics from your workloads and Kubernetes itself, add the `obs-agent` add-on. It
runs the [EZGH Obs agent](/observability/kubernetes-agent) on every node and sends with [pod
identity](/kubernetes/pod-identity), so no ingest key is stored in the cluster.

1. Create a bot that may send to your Obs project: `obs.logs.ingest`, `obs.metrics.ingest` and
   `obs.traces.ingest` on it.

2. Associate the bot with the agent's two service accounts, in namespace `ezgh-obs`:

   ```sh
   for sa in ezgh-obs-agent ezgh-obs-agent-cluster; do
     ezgh kubernetes pod-identity-associations create --cluster kcl_3k9d0a1m2x7q \
       --namespace ezgh-obs --service-account $sa --bot bot_5t7y9u1i3oqa
   done
   ```

3. Add the add-on, naming the project:

   ```sh
   echo '{"obsProject": "prj_o11y00000000"}' > obs-agent.json
   ezgh kubernetes addons create obs-agent --cluster kcl_3k9d0a1m2x7q \
     --configuration-file obs-agent.json --wait
   ```

| Configuration | |
| --- | --- |
| `obsProject` | The Obs project (`prj_…`) the cluster's logs, metrics and traces go to. Required |
| `environment` | The environment every record is tagged with, which selects its dataset. Default `production` |

The agent's pods wait to start until their service accounts have an association.

Source: https://docs.ezghcloud.com/kubernetes/observability/index.mdx
