Every warehouse sends its metrics to its project in Observability, free and
automatically: there’s nothing to install or turn on, and they aren’t billed or counted against
Observability’s limits. They arrive every 30 seconds, in the project’s production environment, with
the warehouse’s ID as the warehouse label, and shard and replica labels where they apply.
| Metric | |
|---|---|
ClickHouseProfileEvents_Query, ClickHouseProfileEvents_FailedQuery |
Queries and failed queries (counters) |
ClickHouseProfileEvents_QueryTimeMicroseconds |
Time spent in queries (counter) |
ClickHouseProfileEvents_SelectedRows, ClickHouseProfileEvents_SelectedBytes |
Rows and bytes read (counters) |
ClickHouseProfileEvents_InsertedRows, ClickHouseProfileEvents_InsertedBytes |
Rows and bytes inserted (counters) |
ClickHouseMetrics_Query |
Queries running now |
ClickHouseMetrics_MemoryTracking |
Memory in use, in bytes |
ClickHouseMetrics_Merge, ClickHouseMetrics_PartMutation |
Merges and mutations running now |
ClickHouseAsyncMetrics_MaxPartCountForPartition |
The most parts in one partition |
ClickHouseAsyncMetrics_FilesystemMainPathUsedBytes, ClickHouseAsyncMetrics_FilesystemMainPathTotalBytes |
Storage used and its size, in bytes |
warehouse_replica_queue_size, warehouse_replica_delay_seconds |
A replica’s replication queue and how far it’s behind |
warehouse_readonly_replicas |
Replicas that can’t take writes |
warehouse_keeper_healthy |
1 while the warehouse’s Keeper coordination is healthy, one series per warehouse |
warehouse_table_rows, warehouse_table_bytes |
Rows and bytes of each table, with database and table labels |
Dashboard
The warehouse’s Monitoring tab shows the built-in Warehouse dashboard, by shard and replica.
Events
The Events tab, and ezgh warehouse warehouses events whs_4k2m9x0a7qpe, list what happened to the
warehouse, newest first: scaling, node replacements, maintenance, rolling changes, upgrades, low
storage, failed backups, a restore, and a database found that EZGH didn’t create.
Query history
ClickHouse’s own system.query_log holds the warehouse’s queries, readable by users with the admin
preset:
SELECT event_time, user, query_duration_ms, read_rows, query
FROM clusterAllReplicas('default', system.query_log)
WHERE type = 'QueryFinish'
ORDER BY event_time DESC
LIMIT 20;