Real-Time Metrics Monitoring
Real-time monitoring of ClickHouse metrics, events, and asynchronous metrics.
Diagnostics
Run all queries from the file checks.sql and analyze the results.
Ad-Hoc Query Guidelines
Key Tables
system.metrics- Current gauge valuessystem.events- Cumulative counters since restartsystem.asynchronous_metrics- System-level metricssystem.metric_log- Historical metricssystem.asynchronous_metric_log- Historical async metrics
Useful Patterns
-- Find metrics by pattern
select * from system.metrics where metric like '%pattern%'
select * from system.asynchronous_metrics where metric like '%pattern%'
select * from system.events where event like '%pattern%'Cross-Module Triggers
| Finding | Load Module | Reason |
|---|---|---|
| High memory metrics | altinity-expert-clickhouse-memory | Memory analysis |
| High replica delay | altinity-expert-clickhouse-replication | Replication issues |
| High parts count | altinity-expert-clickhouse-merges | Merge backlog |
| High load average | altinity-expert-clickhouse-reporting | Query analysis |
| High connections | altinity-expert-clickhouse-reporting | Connection analysis |
Monitoring Recommendations
Key Metrics to Alert On
| Metric | Warning | Critical |
|---|---|---|
ReadonlyReplica | - | > 0 |
Query | > 75% max | > 90% max |
MemoryResident | > 80% RAM | > 90% RAM |
MaxPartCountForPartition | > parts_to_delay | > parts_to_throw |
ReplicasMaxAbsoluteDelay | > 5 min | > 1 hour |
LoadAverage1 | > CPU count | > 2x CPU count |
Prometheus/Grafana Export
ClickHouse exposes metrics at :9363/metrics in Prometheus format when enabled.