Skip to content

Monitoring

Warehouse metrics in Observability, free and automatic, the built-in dashboard, events and query history.

Updated View as Markdown

Every warehouse sends its metrics to its project in Observability, free and automatically: there’s nothing to install or turn on, and they aren’t billed or counted against Observability’s limits. They arrive every 30 seconds, in the project’s production environment, with the warehouse’s ID as the warehouse label, and shard and replica labels where they apply.

Metric
ClickHouseProfileEvents_Query, ClickHouseProfileEvents_FailedQuery Queries and failed queries (counters)
ClickHouseProfileEvents_QueryTimeMicroseconds Time spent in queries (counter)
ClickHouseProfileEvents_SelectedRows, ClickHouseProfileEvents_SelectedBytes Rows and bytes read (counters)
ClickHouseProfileEvents_InsertedRows, ClickHouseProfileEvents_InsertedBytes Rows and bytes inserted (counters)
ClickHouseMetrics_Query Queries running now
ClickHouseMetrics_MemoryTracking Memory in use, in bytes
ClickHouseMetrics_Merge, ClickHouseMetrics_PartMutation Merges and mutations running now
ClickHouseAsyncMetrics_MaxPartCountForPartition The most parts in one partition
ClickHouseAsyncMetrics_FilesystemMainPathUsedBytes, ClickHouseAsyncMetrics_FilesystemMainPathTotalBytes Storage used and its size, in bytes
warehouse_replica_queue_size, warehouse_replica_delay_seconds A replica’s replication queue and how far it’s behind
warehouse_readonly_replicas Replicas that can’t take writes
warehouse_keeper_healthy 1 while the warehouse’s Keeper coordination is healthy, one series per warehouse
warehouse_table_rows, warehouse_table_bytes Rows and bytes of each table, with database and table labels

Dashboard

The warehouse’s Monitoring tab shows the built-in Warehouse dashboard, by shard and replica.

Events

The Events tab, and ezgh warehouse warehouses events whs_4k2m9x0a7qpe, list what happened to the warehouse, newest first: scaling, node replacements, maintenance, rolling changes, upgrades, low storage, failed backups, a restore, and a database found that EZGH didn’t create.

Query history

ClickHouse’s own system.query_log holds the warehouse’s queries, readable by users with the admin preset:

SELECT event_time, user, query_duration_ms, read_rows, query
FROM clusterAllReplicas('default', system.query_log)
WHERE type = 'QueryFinish'
ORDER BY event_time DESC
LIMIT 20;
Navigation

Type to search…

↑↓ navigate↵ selectEsc close