Skip to content

Monitoring

Per-instance CPU and per-volume metrics that every instance sends to Observability, free, and how to query them with PromQL.

Updated View as Markdown

Every instance’s CPU and volume metrics go to its project in Observability, free and automatically: there’s nothing to install or turn on, and they aren’t billed or counted against Observability’s limits. They arrive every 60 seconds, in the project’s production environment, with job="ezgh.compute" and the instance’s ID as instance.

Metrics

Metric (PromQL) Labels Meaning
compute_instance_cpu_utilization_ratio Share of the instance’s vCPU capacity used over the minute, 0 to 1
compute_instance_cpu_time_seconds_total Time the instance’s vCPUs ran
compute_instance_cpu_wait_seconds_total Time its vCPUs were ready to run but waited for a host CPU (steal)
compute_instance_storage_throttled_seconds_total Time its volumes together were at the instance type’s storage limit
compute_instance_status_check_failed check (system, instance) 1 when the status check failed, 0 when it passed; nothing while it’s initializing or not applicable
blockstore_volume_operations_total ezgh_volume_id, direction (read, write) Operations the volume completed
blockstore_volume_bytes_total ezgh_volume_id, direction Bytes it read or wrote
blockstore_volume_latency_seconds ezgh_volume_id, direction Average completion time over the minute, when there were operations
blockstore_volume_throttled_seconds_total ezgh_volume_id Time the volume was at its IOPS or throughput limit

The OTLP names, as sent, are compute.instance.cpu.utilization, compute.instance.cpu.time, compute.instance.cpu.wait, compute.instance.storage.throttled, compute.instance.status_check.failed, blockstore.volume.operations, blockstore.volume.bytes, blockstore.volume.latency and blockstore.volume.throttled; see Names and labels.

The metrics cover running instances, and the volumes attached to them. Memory and disk usage inside the guest aren’t measured from outside it: send those yourself, for example with the OpenTelemetry Collector’s host metrics receiver.

Queries

# CPU utilization of one instance
compute_instance_cpu_utilization_ratio{instance="ins_7k2m9x0a4qpe"}

# Instances above 90% CPU
compute_instance_cpu_utilization_ratio > 0.9

# A volume's IOPS, by direction
sum by (direction) (rate(blockstore_volume_operations_total{ezgh_volume_id="vol_4k2m9x0a7qpe"}[5m]))

# A volume's read and write throughput, in bytes per second
sum by (direction) (rate(blockstore_volume_bytes_total{ezgh_volume_id="vol_4k2m9x0a7qpe"}[5m]))

# Volumes that spent time at their limit in the last hour
increase(blockstore_volume_throttled_seconds_total[1h]) > 0

# Instances whose instance status check failed (alert on this for 5 minutes)
max by (instance) (compute_instance_status_check_failed{check="instance"}) == 1

Time spent at a limit means the volume or instance needed more than it’s provisioned with: modify the volume for more IOPS or throughput, or resize the instance for a higher storage limit. Steal (cpu_wait) on a shared instance type means its vCPUs waited for host CPUs.

Query them in Observability, build dashboards and set alerts on them like any other metrics.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close