Server metrics architecture

TurboPanel collects host metrics from connected daemons via authenticated POST /api/daemon/v1/metrics — approximately one sample per minute per server. Metrics is disposable, statistical, and may be sampled; it is not a billing ledger or audit trail.

Overview

One versioned wire contract (METRICS_SCHEMA_VERSION = 1) is mirrored in the daemon and instance repos (not build-coupled). Storage is dual-runtime:

Store selection: resolveServerMetricsStore (always on — Workers → Analytics Engine, Deno → ClickHouse).

HTTP request body (protocol v1):

{
  "type": "metrics",
  "version": 1,
  "at": "2026-07-12T12:00:00.000Z",
  "intervalSeconds": 60,
  "sequence": 42,
  "metrics": { "cpuUsagePercent": 12.5, "load1": 0.8, "..." : "..." },
  "dimensions": { "schemaVersion": 1, "daemonVersion": "…", "operatingSystem": "…", "architecture": "…", "kernelRelease": "…" }
}

Scheduling: 60 s interval, ≤5 s deterministic per-serverId phase jitter, monotonic process-local sequence (resets on daemon restart). Independent of cell ping/heartbeat liveness — see daemon AGENTS.md. Host metrics are ingested only via authenticated POST /api/daemon/v1/metrics; WebSocket { type: "metrics" } frames are no longer accepted.

20-metric contract

Order is fixed — positional AE doubles and ClickHouse columns depend on it (HOST_METRIC_KEYS).

Missing values are null on the wire — never coerced to 0. First-sample rate metrics (disk/network per-second) are null until a second delta exists.

Analytics Engine mapping (Workers)

AE doubles have no null. Missing metrics store AE_MISSING_METRIC_SENTINEL = -1e308 — never 0. All host metrics are ≥ 0, so the sentinel cannot collide. On the AE SQL read path, compare against -pow(10, 308) (scientific-notation literals and NULLIF are not documented for AE SQL).

Connection-status event stream (blob1 = "status")

Every genuine online/offline transition on a server row (not a heartbeat or identity-only touch) also writes one row into this same dataset/table, discriminated from host samples by blob1:

AE stamps its own ingestion timestamp for status rows; ClickHouse stores the event's own timestamp directly (the same host-vs-ClickHouse asymmetry host samples already have). Both backends expose this via GET /api/client/v1/servers/:id/metrics/connection, which resolves prior state before the range plus in-range transitions into shared uptime/downtime math so AE and ClickHouse report identical numbers for the same query.

Volume impact is negligible against the write-volume formulas below — status rows are written only on a state flip (at most a handful per server per day in normal operation), not once per sampling interval like host rows, so they do not materially change the write-volume math in this doc.

Query aggregation (SQL API): filter blob1 = 'host' and supported blob2; rows under result.data (Cloudflare v4 envelope). Weighted average per bucket:

SUM(if(doubleN = -pow(10, 308), 0.0, _sample_interval * doubleN))
/ SUM(if(doubleN = -pow(10, 308), 0.0, _sample_interval * 1.0))

Metrics is sampled — always weight by _sample_interval; do not assume one row per minute.

Hosted retention: Cloudflare Analytics Engine retains data for 3 months. UI/API max range is 90 days on Workers; do not imply long-term persistence beyond AE retention on hosted deployments.

ClickHouse schema (self-hosted)

Database turbopanel_metrics; DDL is idempotent (ensureSchema once per process): CREATE TABLE IF NOT EXISTS plus ALTER … MODIFY SETTING (compact-part thresholds) and ALTER … MODIFY TTL so retention changes apply to existing tables.

ClickHouse stores the same positional column layout as Analytics Engine — timestamp, index1 (serverId), double1..double20 (in HOST_METRIC_KEYS order), and blob1..blob20 (identity dimensions + reserved slots) — in a single physical table named turbopanel_server_metrics (same as the AE dataset). Missing metrics use the same AE_MISSING_METRIC_SENTINEL (-1e308) as Analytics Engine — never null or coerced 0. Query aggregates exclude the sentinel with the same filtering semantics as AE; resolution is chosen at query time (no rollup tables or materialized views).

Late arrivals / duplicates: all inserts accepted (no ReplacingMergeTree / FINAL).

Query API & caching

Session + server read grant required. One combined backend query per (server, range) — not per metric or chart.

Resolution ladder: range ≤ 6 h → 60 s; ≤ 24 h → 300 s; ≤ 30 d → 3600 s; else 86400 s (ClickHouse maps 60 → 300). Max 1500 points; range ≤ 90 days.

Chart cache: key includes authorized serverId, bucket-rounded range, sorted metrics, resolution, backend, schema version. TTL: live 45 s / historical 300 s.

Cost model

Metrics ingestion runs on the normal Worker isolate (or Deno process) and no longer incurs Durable Object GB-sec — see Daemon Cell for DO billing. The Analytics Engine price constants, limits, and formulas below remain the single source of truth for AE cost planning.

Cloudflare Analytics Engine limits & prices

All 20 metrics = one writeDataPoint per sample (one AE data point per minute per server).

Write volume formulas

Read volume (dashboards)

Reads scale with dashboard views ÷ cache TTL (live 45 s / historical 300 s), not chart count or fleet size — one combined query per server/range/resolution, cache-collapsed.

Worked examples (60 s interval, all servers connected 24×7)

Cloudflare currently does not bill Analytics Engine — treat overage dollars as planning figures until billing activates.