Skip to content

Observability

Control-plane logs and metrics, and node and workload telemetry, in Obs.

Updated View as Markdown

Control-plane logs

A cluster can send any of five control-plane log types to an Obs project you name:

Log type What it sends
api The Kubernetes API server’s logs
audit The API server’s audit log: who did what in the cluster
authenticator One entry per sign-in with an EZGH cluster token or a node’s instance identity: time, credential type, whether it was authenticated, username, uid, groups and error. Never the token
controllerManager The controller manager’s logs
scheduler The scheduler’s logs

Choose them when you create the cluster, or later: in the console on the cluster’s Observability tab, with --logging-type (repeat it for more) and --logging-project, or with logging in the API.

ezgh kubernetes clusters update kcl_3k9d0a1m2x7q --logging-type audit --logging-type authenticator \
  --logging-project prj_o11y00000000 --wait

A change applies to the running control plane; a log type turned off stops once the update finishes. --no-logging stops them all.

Each record has service.name ezgh.kubernetes, the attribute cluster set to the cluster’s ID and kubernetes.log_type set to its log type. Logs are billed as log ingest in that project. Sending logs to a project needs obs.logs.ingest on it.

Control-plane metrics

A cluster can send its API server, etcd, scheduler and controller manager metrics to Obs, free. It’s off by default. Turn it on in the console with Send metrics, on Create cluster or the cluster’s Observability tab, with --control-plane-metrics on ezgh kubernetes clusters create or update, or with controlPlaneMetrics on create or update. --control-plane-metrics=false turns it off.

ezgh kubernetes clusters update kcl_3k9d0a1m2x7q --control-plane-metrics --wait

Metrics arrive once a minute in the cluster’s own project, in the production environment, with service.name ezgh.kubernetes. Each series is labelled cluster (the cluster’s ID), component (apiserver, etcd, scheduler or controllerManager) and member (a number for each control-plane server; a replacement gets a new number). Other labels are summed away, so only those listed below are kept. Each member sends at most 2,000 series a minute; past that, API server series are left out first.

Metric Labels kept
apiserver_request_total verb, code
apiserver_request_duration_seconds verb
apiserver_current_inflight_requests request_kind
apiserver_resource_objects group, resource
apiserver_requested_deprecated_apis group, version, resource, removed_release
apiserver_admission_webhook_rejection_count name, operation, rejection_code
etcd_server_has_leader None
etcd_server_leader_changes_seen_total None
etcd_server_proposals_failed_total None
etcd_mvcc_db_total_size_in_bytes None
etcd_mvcc_db_total_size_in_use_in_bytes None
etcd_disk_wal_fsync_duration_seconds None
etcd_disk_backend_commit_duration_seconds None
etcd_network_peer_round_trip_time_seconds To
scheduler_pending_pods queue
scheduler_schedule_attempts_total result, profile
scheduler_scheduling_attempt_duration_seconds result, profile
leader_election_master_status name
workqueue_depth name
workqueue_adds_total name
rest_client_requests_total code, method

Nodes and workloads

Nodes are instances, so their instance metrics flow to Obs as any instance’s do.

For logs, traces and metrics from your workloads and Kubernetes itself, add the obs-agent add-on. It runs the EZGH Obs agent on every node and sends with pod identity, so no ingest key is stored in the cluster.

  1. Create a bot that may send to your Obs project: obs.logs.ingest, obs.metrics.ingest and obs.traces.ingest on it.

  2. Associate the bot with the agent’s two service accounts, in namespace ezgh-obs:

    for sa in ezgh-obs-agent ezgh-obs-agent-cluster; do
      ezgh kubernetes pod-identity-associations create --cluster kcl_3k9d0a1m2x7q \
        --namespace ezgh-obs --service-account $sa --bot bot_5t7y9u1i3oqa
    done
  3. Add the add-on, naming the project:

    echo '{"obsProject": "prj_o11y00000000"}' > obs-agent.json
    ezgh kubernetes addons create obs-agent --cluster kcl_3k9d0a1m2x7q \
      --configuration-file obs-agent.json --wait
Configuration
obsProject The Obs project (prj_…) the cluster’s logs, metrics and traces go to. Required
environment The environment every record is tagged with, which selects its dataset. Default production

The agent’s pods wait to start until their service accounts have an association.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close