Control-plane logs
A cluster can send any of five control-plane log types to an Obs project you name:
| Log type | What it sends |
|---|---|
api |
The Kubernetes API server’s logs |
audit |
The API server’s audit log: who did what in the cluster |
authenticator |
One entry per sign-in with an EZGH cluster token or a node’s instance identity: time, credential type, whether it was authenticated, username, uid, groups and error. Never the token |
controllerManager |
The controller manager’s logs |
scheduler |
The scheduler’s logs |
Choose them when you create the cluster, or later: in the console on the cluster’s Observability tab,
with --logging-type (repeat it for more) and --logging-project, or with logging in
the API.
ezgh kubernetes clusters update kcl_3k9d0a1m2x7q --logging-type audit --logging-type authenticator \
--logging-project prj_o11y00000000 --waitA change applies to the running control plane; a log type turned off stops once the update finishes.
--no-logging stops them all.
Each record has service.name ezgh.kubernetes, the attribute cluster set to the cluster’s ID and
kubernetes.log_type set to its log type. Logs are billed as log ingest in that project. Sending logs
to a project needs obs.logs.ingest on it.
Control-plane metrics
A cluster can send its API server, etcd, scheduler and controller manager metrics to Obs, free. It’s
off by default. Turn it on in the console with Send metrics, on Create cluster or the cluster’s
Observability tab, with --control-plane-metrics on ezgh kubernetes clusters create or update,
or with controlPlaneMetrics on create or
update.
--control-plane-metrics=false turns it off.
ezgh kubernetes clusters update kcl_3k9d0a1m2x7q --control-plane-metrics --waitMetrics arrive once a minute in the cluster’s own project, in the production environment, with
service.name ezgh.kubernetes. Each series is labelled cluster (the cluster’s ID), component
(apiserver, etcd, scheduler or controllerManager) and member (a number for each control-plane server; a replacement gets a new number). Other labels
are summed away, so only those listed below are kept. Each member sends at most 2,000 series a
minute; past that, API server series are left out first.
| Metric | Labels kept |
|---|---|
apiserver_request_total |
verb, code |
apiserver_request_duration_seconds |
verb |
apiserver_current_inflight_requests |
request_kind |
apiserver_resource_objects |
group, resource |
apiserver_requested_deprecated_apis |
group, version, resource, removed_release |
apiserver_admission_webhook_rejection_count |
name, operation, rejection_code |
etcd_server_has_leader |
None |
etcd_server_leader_changes_seen_total |
None |
etcd_server_proposals_failed_total |
None |
etcd_mvcc_db_total_size_in_bytes |
None |
etcd_mvcc_db_total_size_in_use_in_bytes |
None |
etcd_disk_wal_fsync_duration_seconds |
None |
etcd_disk_backend_commit_duration_seconds |
None |
etcd_network_peer_round_trip_time_seconds |
To |
scheduler_pending_pods |
queue |
scheduler_schedule_attempts_total |
result, profile |
scheduler_scheduling_attempt_duration_seconds |
result, profile |
leader_election_master_status |
name |
workqueue_depth |
name |
workqueue_adds_total |
name |
rest_client_requests_total |
code, method |
Nodes and workloads
Nodes are instances, so their instance metrics flow to Obs as any instance’s do.
For logs, traces and metrics from your workloads and Kubernetes itself, add the obs-agent add-on. It
runs the EZGH Obs agent on every node and sends with pod
identity, so no ingest key is stored in the cluster.
-
Create a bot that may send to your Obs project:
obs.logs.ingest,obs.metrics.ingestandobs.traces.ingeston it. -
Associate the bot with the agent’s two service accounts, in namespace
ezgh-obs:for sa in ezgh-obs-agent ezgh-obs-agent-cluster; do ezgh kubernetes pod-identity-associations create --cluster kcl_3k9d0a1m2x7q \ --namespace ezgh-obs --service-account $sa --bot bot_5t7y9u1i3oqa done -
Add the add-on, naming the project:
echo '{"obsProject": "prj_o11y00000000"}' > obs-agent.json ezgh kubernetes addons create obs-agent --cluster kcl_3k9d0a1m2x7q \ --configuration-file obs-agent.json --wait
| Configuration | |
|---|---|
obsProject |
The Obs project (prj_…) the cluster’s logs, metrics and traces go to. Required |
environment |
The environment every record is tagged with, which selects its dataset. Default production |
The agent’s pods wait to start until their service accounts have an association.