Server metrics architecture

TurboPanel collects host metrics from connected daemons via authenticated POST /api/daemon/v1/metrics — one sample per minute per server, every family on every sample. Metrics is disposable, statistical, and may be sampled; it is not a billing ledger or audit trail.

Overview

One versioned wire contract (METRICS_SCHEMA_VERSION = 6) is mirrored in the daemon and instance repos (not build-coupled). Ingest rejects any sample whose metadata.version is not 6; there is no dual-accept and no migration. The contract groups metrics by entity — host, network device, filesystem, block device, GPU, hardware signal, ingress source, database proxy — plus four host-wide singletons (diagnostics, router, storage accounting, Docker usage) that carry no entity id. Every leaf value is number | null (missing is always null, never coerced to 0), and every entity carries a stable logical id instead of relying on positional membership in an allowlist. Storage is dual-runtime:

Store selection: resolveServerMetricsStore (always on — Workers → Analytics Engine, Deno → DuckDB + Parquet).

Every sample writes two mandatory rows — host.system and host.io — plus one row for each family actually present on that host this tick. See Metrics v6 contract below.

HTTP request body (schema v6 — illustrative subset). The minimal shape (a VM with one NIC, no GPU, no managed-service sidecars, before the daemon's first storage walk completes):

{
  "type": "metrics",
  "metadata": {
    "version": 6,
    "sampledAt": "2026-09-07T12:00:00.000Z",
    "intervalSeconds": 60,
    "sequence": 42,
    "topologyGeneration": 1,
    "bootGeneration": 1
  },
  "host": {
    "cpu": { "busyPercent": 8.1, "userPercent": 5.2, "systemPercent": 2.9, "saturatedCoreCount": 0, "...": "..." },
    "kernel": { "fileHandlesUsedPercent": 3.1, "conntrackUsedPercent": null },
    "memory": { "usedBytes": 1610612736, "cachedFilesBytes": 536870912, "swapUsedBytes": 0, "...": "..." },
    "storage": { "diskReadBytesPerSecond": 102400, "diskWriteBytesPerSecond": 51200, "diskLatencyMs": 0.4, "...": "..." },
    "network": { "tcpRetransmitPercent": 0.02, "softnetDropsPerSecond": 0 }
  },
  "networks": [{ "deviceId": "eth0", "receiveBytesPerSecond": 12000, "transmitBytesPerSecond": 8000, "...": "..." }],
  "filesystems": [],
  "blockDevices": [],
  "gpus": [],
  "hardwareSignals": [],
  "ingressSources": [],
  "databaseProxies": [],
  "events": [],
  "diagnostics": {
    "cpu": { "averageFrequencyMHz": 2400, "contextSwitchesPerSecond": 1800, "...": "..." },
    "memory": { "memoryFreeBytes": 268435456, "dirtyBytes": 4096, "...": "..." }
  }
}

A bare-metal hosting box with a GPU, a site Caddy, the shared hosting router, and hardware sensors declares the corresponding entity arrays and singletons — presence is what drives extra rows, not a declared allowlist. GPU and drive temperatures are signals, not fields on the gpus[] / blockDevices[] entries:

{
  "networks": [{ "deviceId": "eth0" }, { "deviceId": "eth1" }],
  "gpus": [{ "gpuId": "gpu0", "utilizationPercent": 34.2, "memoryUsedBytes": 8589934592, "...": "..." }],
  "hardwareSignals": [
    { "signalId": "coretemp:Package id 0", "kind": "temperature", "value": 58.5 },
    { "signalId": "gpu0:temperature", "kind": "temperature", "value": 61 },
    { "signalId": "nvme0n1:temperature", "kind": "temperature", "value": 41 },
    "..."
  ],
  "ingressSources": [{ "sourceId": "caddy", "sourceKind": "caddy", "requests": 744, "bucket10ms": 512, "...": "..." }],
  "router": { "backendsUp": 11, "backendsTotal": 12, "retries": 0, "tlsCertSoonestExpiryDays": 45, "...": "..." },
  "storage": { "hostingUsedBytes": 53687091200, "hostingFreeBytes": 429496729600, "dockerUsedBytes": 21474836480, "...": "..." },
  "dockerUsage": { "layersBytes": 17179869184, "imagesCount": 42, "volumesReclaimableBytes": 0, "...": "..." }
}

Scheduling: first sample POSTs immediately on connect (including the co-located self-hosted daemon), a primed second sample ~2 s later fills rate metrics (CPU / disk / net), then a 60 s baseline interval with ≤5 s deterministic per-serverId phase jitter. On top of the baseline, a UI-requested live-mode lease switches the daemon to a 10 s sampling cadence for the duration of the lease — capped at 60 minutes by default, with expiry enforced daemon-side so an abandoned browser tab cannot hold a server in fast sampling. Live samples are never stored on either backend: while any lease is active the sample lands in a short-lived buffer the live chart reads from, and the durable write is skipped. The intervalSeconds on the wire is the only cadence marker — there is no collection-mode flag. Monotonic process-local metadata.sequence (resets on daemon restart). Independent of cell ping/heartbeat liveness — see daemon AGENTS.md. Host metrics are ingested only via authenticated POST /api/daemon/v1/metrics; WebSocket { type: "metrics" } frames are not accepted.

Metrics v6 contract

The contract has no MetricPart/19-slot-per-part allowlist coupling. Metrics are grouped into families (HostedFamily in metric-descriptors.ts), each one either universal (always emitted, even if every value is null), presence-gated (emitted only when the source entity array or singleton is populated), or capability-gated (emitted only up to what the resolved MetricsCapabilityPlan allows). There is one cadence: every family writes on every 60 s sample. No family is decimated, demoted to a slower tier, or held over across empty buckets.

Entitlement vs. emission are distinct. A capability-plan slot count (gpuSlots: 2) is what a server is allowed to report; whether it emits rows depends entirely on whether the daemon found that hardware. A server entitled to two GPUs with none installed emits zero gpu rows. Entitlement caps how many entities may be reported; presence decides whether a family is emitted at all this tick. The same split applies everywhere — physicalHardwareSignalSlots is 19 on a physical machine (11 on the entry tier) and 0 on a VM, but a physical host with no working sensors still emits nothing for hardware.physical. Hosted ingest enforces the ceiling with truncateSampleToCapabilityPlan; self-hosted ingest skips truncation entirely.

Pressure Stall Information (PSI)

The five *Pressure*Percent fields on host.system and host.io are Linux Pressure Stall Information. They are not utilization: each is the share of that sample's wall clock during which tasks were stalled on CPU, memory, or disk (from the kernel total counter in /proc/pressure/, never the kernel's avg10/avg60/avg300 windows).

Some is contention. Full (memory and I/O only) is a stall. CPU has no Full series — a CPU is never completely stalled the way memory and disk can be. Unsupported kernels omit /proc/pressure/ and those fields stay null. How to read the charts: Pressure Stall Information.

Memory diagnostics

The memory half of host.diagnostics is the Memory detail group — what RAM is made of, and whether the kernel is working to free some. These are not a second used-%: they are /proc/meminfo gauges and /proc/vmstat reclaim rates, reported on every Linux host. Operator reading: Memory detail.

Missing values are null on the wire — never coerced to 0. First-sample rate metrics are null until a second delta exists; a metric a host genuinely doesn't support (no temperature probe, no swap configured, no /proc/pressure/) stays null permanently. The twelve per-engine census fields on managed.storage are null for an engine the host does not run, and for every engine until the daemon's first managed-engine census lands; an engine with instances present reports real counts, including 0 running.

Retired since v4: host.cpu.maxCoreBusyPercent → saturatedCoreCount (cores at or above 90% busy); host.memory.availableBytes → usedBytes + cachedFilesBytes; host.storage.diskReadLatencyMs + diskWriteLatencyMs → one diskLatencyMs; host.storage.maxBlockDeviceUtilPercent removed; gpu.temperatureCelsius / memoryTemperatureCelsius / powerWatts and block.temperatureCelsius moved to hardware.physical; cpu.detail + memory.detail merged into host.diagnostics (seven memory fields dropped: page tables, kernel stack, commit limit, and the four active/inactive splits); cpu.core.live and the NUMA reservation deleted outright; collectionMode removed from the metadata.

Topology generations

Every sample carries metadata.topologyGeneration and metadata.bootGeneration. topologyGeneration bumps only when the daemon's enumerated entity set (NICs, GPUs, filesystems, block devices, hardware signals) actually changes shape — a NIC replaced, a GPU added, a disk removed — not on every value change. The ingest route resolves a SlotMapping for the sample's own topologyGeneration (client/servers/topology-slot-mapping.ts) and threads it through packing so:

• host.io's two embedded NIC slots (see below) stay pinned to SlotMapping.normalNicSlot1/.normalNicSlot2's deviceId, not raw array position — a 2-NIC host with a TurboFabric mesh interface stays at 2 rows even though networks[] has 3 entries, because the fabric device is in SlotMapping.fabricDeviceIds and never pages as a network row.
• Every paged family (gpu/network/filesystem/block/hardware.physical) orders entities by the matching SlotMapping.*PageOrder list, and stamps each page's contributing entity ids so a stored row is self-describing regardless of which generation produced it — a query never needs to reinterpret a page's identity under a later generation's mapping.
• With no resolved generation yet (first sample, resync pending), packing falls back to positional order gracefully rather than dropping data.

Every stored row also carries the sample's own topologyGeneration, so a UI can segment charts around a topology change the same way v3 segmented around a hardware-profile-generation change. Capacity totals (memory, swap, root filesystem) live on the generation record, never in a sample, so a percentage from last week is still divided by last week's capacity after a RAM upgrade.

Events

A closed catalog of ~37 discrete state-change/fault kinds (MetricEventKind) is distinct from the continuous numeric families above — OOM kills, hung tasks, filesystem state changes, SMART/NVMe/RAID faults, NIC link flaps, TurboFabric mesh state, fan/thermal/PSU/voltage/ECC faults, GPU Xid/ECC/retirement/thermal faults, clock-sync loss, and topology/boot-generation bumps. Each kind is classified once as a physical-hardware-health signal or not (HARDWARE_HEALTH_EVENT_KIND) — the split hardwareHealthEventsEnabled in the capability plan actually gates; OS/kernel conditions, filesystem state, TurboFabric state, clock-sync state, and generation bumps always survive a plan that disables hardware-health events. Every event writes its own "event"-kind AE row (see the envelope table below), and self-hosted Deno writes them to a dedicated server_metric_events table.

Analytics Engine mapping (Workers)

One AE data point holds only 20 doubles, so a sample writes two or more data points depending on which families are present. host.system and host.io are always emitted — one row each, even if every value is missing. Every paged family packs AE_METRIC_DOUBLE_SLOT_COUNT = 19 metric-value slots per row: entitiesPerPage = floor(19 / fieldsPerEntity), so a family whose entities need more double slots fits fewer per page.

managed.ingress and managed.database_proxy have no paging concept — one unpaged row per source entry, since a managed-service sidecar count is small and unbounded paging would add no value. host.diagnostics, managed.router, managed.storage, and managed.docker are single fixed-shape rows per sample — one hosting root, one Docker daemon, one shared router per host — and carry no entity identity.

The dataset name is turbopanel_server_metrics_v6. The Wrangler binding is SERVER_METRICS.

AE doubles have no null. Missing metrics store AE_MISSING_METRIC_SENTINEL = -1e308. All host metrics are ≥ 0, so the sentinel cannot collide. On the AE SQL read path, compare against -pow(10, 308).

Connection-status event stream (blob1 = "status")

Every genuine online/offline transition on a server row also writes one status event. On Analytics Engine it lands in the same turbopanel_server_metrics_v6 dataset, discriminated by blob1; on self-hosted Deno it lands in a separate typed server_status_events DuckDB table (never shared with sample tables).

Hosted retention: Cloudflare Analytics Engine retains data for 3 months. UI/API max range is 90 days on Workers; do not imply long-term persistence beyond AE retention on hosted deployments.

DuckDB + Parquet schema (self-hosted)

The Deno path uses the embedded @duckdb/node-api engine — no external service, no port, no credentials. Everything lives under the metrics state root (TURBOPANEL_METRICS_DIR, default <stateDir>/metrics): metrics.duckdb (hot database), parquet/ (sealed daily partitions, one subtree per family), tmp/ (spill + in-flight exports), and a sidecar schema marker (DUCKDB_SCHEMA_MARKER_VERSION 8). The marker number is intentionally not the wire schema version: METRICS_SCHEMA_VERSION = 6 counts contract revisions, while the marker counts on-disk layout generations of the self-hosted store and bumps whenever DDL widens or adds a table (every statement is CREATE TABLE IF NOT EXISTS, so an unbumped marker would silently keep an older, narrower table). v6 wire samples land in layout 8 — the two numbers are expected to differ and are never reconciled. A missing, corrupt, or non-matching marker discards those files and the current store is created on the next open — there is no in-place migration. See Metrics deployment.

One real typed table per family — no positional layout, no sentinel. Missing metrics are genuine SQL NULLs. DuckDB writes one row per entity with named nullable columns — 24 drives is 24 rows, not 12 packed pages; AE paging is a Cloudflare-only concern. The current layout:

The v5 server_cpu_hotspot_samples, server_cpu_core_samples, and server_memory_detail_samples tables no longer exist.

Daily Parquet archive: each completed UTC day is sealed independently per family into parquet/<family-subdir>/year=YYYY/month=MM/day=DD/metrics.parquet — export to tmp/, validate the produced file's row count by re-reading it, atomic rename into the partition tree, and only then delete that family's hot rows. Resealing a day that already has a partition (late arrivals) merges the existing sealed rows with the new hot rows, keeping archiving idempotent and one-file-per-day per family. Live samples never reach these tables, so a day's partition holds baseline-cadence rows only.

Resource caps and configuration:

Query API & caching

Session + server read grant required. One combined backend query per (server, range) — not per metric or chart.

Derived at read time, never stored: used-percent figures (memory, swap, root filesystem) divide by the capacity of the generation each bucket was sampled under; ingress latency mean and p50/p90/p99 come from the stored duration sum and six bucket counters. Sums and counts add cleanly across any window, so a 5-minute view and a 24-hour view use identical math — a stored per-minute percentile could not be re-bucketed that way.

Resolution ladder: range ≤ 10 min → 10 s; ≤ 1 h → 60 s; ≤ 6 h → 300 s; ≤ 24 h → 900 s; ≤ 7 d → 3600 s; ≤ 30 d → 21600 s; else 43200 s. Max MAX_METRICS_POINTS = 1500 points; range ≤ 90 days.

Chart cache: key includes authorized serverId, bucket-rounded range, sorted metrics, resolution, backend, and v{schemaVersion} (schema v6) — plus the sample's topologyGeneration when a caller scopes a request to one generation, so cache entries never blur rows from two different topology shapes together. TTL: live 45 s / historical 300 s.

Cost model

Metrics ingestion runs on the normal Worker isolate (or Deno process) and incurs no Durable Object GB-sec — see Daemon Cell for DO billing. The Analytics Engine price constants and formulas below are the single source of truth for AE cost planning; no other page restates them.

Cloudflare Analytics Engine limits & prices

Rows per sample by tier

Every sample writes rowsPerSample data points, every minute, with no cadence discount: rowsPerSample starts at 2 (the universal baseline) and grows by one row per populated family page. What scales with the tier is entity cardinality — how many NICs, drives, GPUs, and extra filesystems a server may report — not depth: host.diagnostics and managed.storage are on at every tier because doubles inside a row are free.

The table assumes a fully loaded server: every entitled slot filled, a site Caddy (managed.ingress), the shared hosting router (managed.router), a ProxySQL sidecar (managed.database_proxy), the storage-accounting row, and the Docker breakdown from S2 up (the entry tier does not grant managed.docker, and its extra-filesystem entitlement is carved out to zero). Two NICs embed in host.io; the daemon's monitored-NIC ceiling is 11, so S7 and SX page at most nine NICs. A physical machine adds one hardware.physical page (11 signal slots on S1, 19 above); a VM never emits that family.

Row by row, a fully loaded S1 on a physical machine is: host.system + host.io + host.diagnostics + managed.storage + one block page (2 drives) + one gpu page (2 GPUs) + managed.ingress + managed.router + managed.database_proxy + one hardware.physical page = 10 rows/min, or $0.108/mo at the unsubsidised write rate. The same machine as a VM drops the sensor page: 9 rows/min, $0.097/mo. Real machines are usually cheaper — a plain VM with no sidecars is 4 rows (host.system, host.io, host.diagnostics, managed.storage), and no single knob costs more than one row per page it opens.

The 250-point invocation limit

Analytics Engine accepts at most 250 data points per Worker invocation, and every family writes on every tick, so the worst sample is also the typical sample. The inventory splits three ways, mirroring the tier model's write-limit check: hardware only is every entitled NIC, drive, GPU, and extra-filesystem page plus the always-on host rows (host.system, host.io, host.diagnostics, managed.storage, and managed.docker from S2 up) and the physical sensor page — no managed-service rows at all; + sources adds the three fixed traffic rows (managed.ingress, managed.router, managed.database_proxy), which is the fully loaded physical figure from the table above; + 128 events stacks a full event burst on top of that. Even the largest ladder shape stays far below the ceiling:

Hardware alone never exceeds 29 points. On the ladder as entitled it tops out at 25 (SX physical, every slot filled); 29 is the conservative bound the budget is planned against, and even that leaves more than 220 points spare before a single sidecar or event is counted. With the three traffic rows the numeric ceiling is 28, and with a full event burst it is 156 — the 250-point limit does not constrain the ladder at any tier. Self-hosted ingest is not metered and not truncated, so an operator's own disk can hold whatever their hardware reports.

Representative machines

Exact, test-verified row counts for the fixed machine shapes in turbopanel/src/daemon/metrics/testing/representative-machines.ts (these fixtures carry only the families named, so a shape without host.diagnostics or managed.storage does not pay for them):

Write volume formulas

Break-even connected-server count (60 s interval, 24×7) by rows/sample:

Live 10 s mode adds no durable writes: leased samples are buffered for the live chart and never written to Analytics Engine or DuckDB, so a live session costs nothing here regardless of how long it runs.

Read volume (dashboards)

Reads scale with dashboard views ÷ cache TTL (live 45 s / historical 300 s), not chart count or fleet size — one combined query per server/range/resolution, cache-collapsed.

Worked examples (60 s interval, all servers connected 24×7)

Cloudflare currently does not bill Analytics Engine — treat overage dollars as planning figures until billing activates.