Server metrics operations

Host/server metrics are a core control-plane feature — always on, disposable, and stored outside Postgres. There is no enable/disable switch. For what each console chart means, see Server metrics — including Pressure Stall Information (PSI), the five stall-time percentages on CPU, memory, and disk. For the wire contract, family catalog, backend mappings, and the dated Cloudflare cost model, see Server metrics architecture.

Configuration

The self-hosted Deno path constructs the DuckDB store directly against an internal path (resolveMetricsDir()), so it needs no operator-supplied credentials — it falls back to a temporary no-op store (with a one-time boot warning) only when the metrics directory cannot be created or opened.

Workers (hosted): add the SERVER_METRICS binding + CLOUDFLARE_ACCOUNT_ID + TURBOPANEL_ANALYTICS_ENGINE_API_TOKEN (Account Analytics Read) for chart queries. Missing either the account id or the token → soft available: false on reads, not a hard failure. Optionally tune the hosted query window with TURBOPANEL_SERVER_METRICS_AE_MAX_RANGE_SECONDS.

Deno (self-hosted): metrics persist to embedded DuckDB + Parquet automatically. Query failures return 503 metrics_backend_unavailable.

There is no analytics Ansible role — the daemon provisions nothing for metrics beyond the instance's own state directory; the instance creates and manages the DuckDB metrics directory itself on first use.

Temperature display unit

Organizations set a display unit independent of storage: GET/PUT /api/client/v1/organizations/:id/temperature-unit accepts "celsius" or "fahrenheit" (default celsius when unset). This setting only changes how temperature charts render — every temperature metric is always stored in Celsius; conversion to Fahrenheit happens at render/query-response time, never at collection or storage.

Topology & capability plan

Two backend-neutral models replace v3's hardware-profile/sensors round trip:

• Topology (daemon-side, turbopaneld/src/metrics/topology/) assigns every NIC, GPU, filesystem, block device, and hardware signal a stable logical id and tracks generation bumps — a topologyGeneration counter that advances only when the enumerated entity set's shape actually changes, plus a bootGeneration for daemon restarts. The control plane resolves a SlotMapping from a sample's topologyGeneration (client/servers/topology-slot-mapping.ts) and threads it through packing so paged families stay identity-addressed instead of positional.
• MetricsCapabilityPlan (control-plane, src/daemon/metrics/capability-plan.ts) is the backend-neutral entitlement model for how much of the contract a server is allowed to report/store — slot counts and feature toggles only, no pricing-tier names. On TurboPanel High Availability the base plan resolves from the server's license → tier: the tier row's entitlement columns (nic_slots, drive_slots, gpu_slots, filesystem_slots, plus its rank) map onto normalNicSlots, detailedBlockDeviceSlots, gpuSlots, extraFilesystemSlots, and managedDockerEnabled; the entry tier (rank 1) carves extra filesystems to 0, physical sensor slots to 11, and Docker off. An unassigned seat falls back to the platform default (normalNicSlots: 2, detailedBlockDeviceSlots: 2, gpuSlots: 1, physicalHardwareSignalSlots: 19 on physical / 0 on virtual, every feature toggle on). An org-wide override in organization.options.metricsCapabilityPlan and an optional per-server override in server.options.metricsCapabilityPlan still layer on top of that base. POST /api/daemon/v1/metrics resolves the effective plan for the reporting server and calls truncateSampleToCapabilityPlan on the incoming sample before it reaches the store — a plan or tier change takes effect on the very next ingested sample, no daemon restart required. Self-hosted is uncapped: the branch happens before truncation, so a self-hosted instance stores whatever the daemon reports — no truncation, no decimation, and the monitored-NIC default is the ceiling (11) rather than the hosted 2. There is no cpuDetailEnabled/memoryDetailEnabled any more: diagnostics ride the always-on host.diagnostics family at every tier, hosted or self-hosted. See License tiers for what each tier entitles and how a server is placed.

machineClass ("physical" | "virtual") is required, not defaulted, specifically so a virtual server can never silently inherit the physical hardware-signal baseline. The declared server.machine_class column wins; only while it is null does the daemon's topology snapshot (any host-level sensor discovered → physical) decide.

A server whose plan grants a slot but whose daemon never reports that entity (e.g. gpuSlots: 2 on a host with no GPU) simply emits nothing for that family — this is expected, not a misconfiguration to chase down. See Server metrics architecture for the entitlement-vs-emission distinction.

Ingress, router & database-proxy adapters

Three independent, loopback-only adapter domains feed managed.ingress / managed.router / managed.database_proxy — each source resolves independently, and one being unreachable never blocks another:

• Ingress (turbopaneld/src/metrics/collector/ingress/) is Caddy only since schema v6. The Caddy adapter (caddy.ts) scrapes the site-caddy role's admin listener (site_caddy_admin_addr, 127.0.0.1:2039 — the unprivileged per-site Caddy, not the separate hosting-ingress Caddy) via its Prometheus /metrics exposition. No-double-count rule: every handler in a request's Caddy handler chain (the site's top-level subroute plus any nested handle/reverse_proxy/file_server terminal) is independently instrumented, so summing every (server, handler) group would double- or n-count each request. The adapter instead picks, per server, the (server, handler) group with the largest caddy_http_requests_total — the outermost handler every request passes through — as the sole source for every normalized field; ties (the normal shape of a single-terminal site) break deterministically by handler name. The six latency buckets come straight off Caddy's default histogram bounds; the daemon ships the raw duration sum and the bucket counts, never a percentile.
• Router (turbopaneld/src/metrics/collector/router/) reads the shared hosting Traefik (traefik.ts) — the loopback proxy behind the hosting Caddy that fronts Docker-hosted sites — from the dedicated metrics entrypoint the deploy configures. It is a host-wide singleton, not an ingress source: v5 forced Traefik onto the Caddy layout, which fit neither side. The collector's own scrape traffic on the metrics entrypoint is excluded from open-connection counts.
• Database proxy (turbopaneld/src/metrics/collector/database-proxy/) reads the ProxySQL adapter (proxysql.ts) against its unauthenticated REST metrics endpoint, published loopback-only (admin-restapi_enabled, 127.0.0.1:6070) the same way the admin MySQL-protocol port is.

An adapter that throws or returns nothing for a tick omits that source entirely (never an all-null placeholder row) and invalidates that source's counter-baseline namespace so the next successful read re-origins instead of diffing across the gap.

Three further host-wide readings are collected on their own slow timers and read synchronously by the sample tick: storage accounting (managed.storage, collector/directory-usage.ts, a 15-minute walk of the hosting, backup, and log roots, or a plain statfs when the path is its own mount), Docker usage (managed.docker, collector/docker-usage.ts, Docker's GET /system/df every 5 minutes), and the managed-engine census (the per-engine fields of managed.storage, collector/managed-engines.ts, every 5 minutes: the managed engine containers found by their tp.managed.engine label over the Docker socket, each running instance probed for readiness and connection count through docker exec). None runs on the 60 s tick; a bounded walk that hits its limit reports null rather than a partial total, and an engine the host does not run reports null rather than zero.

Self-hosted schema

On every open the self-hosted store checks the sidecar schema-version marker against DUCKDB_SCHEMA_MARKER_VERSION (8). That number is the on-disk layout generation of the self-hosted store, not the metrics wire schema: the daemon ↔ instance contract is v6 (METRICS_SCHEMA_VERSION = 6), and v6 samples are stored in layout 8. The two counters are independent by design — the marker bumps whenever DDL adds or widens a table (the managed.storage / managed.docker tables took it from 7 to 8, which is why a v6 store is marker 8), while the wire version only moves when the sample shape changes. A missing, corrupt, or non-matching marker discards metrics.duckdb, parquet/, tmp/, and schema-version before creating the current layout. There is no in-place migration and no supported operator path for older DuckDB files. Layout 8 is the first carrying the server_storage_samples, server_docker_samples, and server_router_samples tables, the merged diagnostics columns, and the widened ingress and database-proxy tables.

Configured retention still expires old points (TURBOPANEL_SERVER_METRICS_RETENTION_DAYS, default 90). The sidecar schema-version marker is written after a successful open of the current store.

DuckDB + Parquet resources (self-hosted)

The self-hosted metrics store is embedded in the instance process — no container, no port, no separate daemon process, no credentials, and no analytics database service to install or monitor.

Everything lives under the metrics state root (TURBOPANEL_METRICS_DIR, default <stateDir>/metrics):

Resource caps come from the env vars above: TURBOPANEL_SERVER_METRICS_DUCKDB_THREADS (default 2) and TURBOPANEL_SERVER_METRICS_DUCKDB_MEMORY_LIMIT (default 128 MiB) bound the embedded engine's footprint.

Retention

The same daily archive tick prunes both expired Parquet partitions and any hot/status rows past the cutoff, independently per family — change retention via env on the instance unit and it applies on the next tick.

Live samples are never stored. While a live lease (10 s cadence) is active, each sample is held in a short-lived buffer the live chart reads from and the durable write is skipped on both backends. Retention, archive sizes, and the cost figures in the architecture doc therefore describe baseline-cadence data only — a long live session adds nothing to disk or to the Analytics Engine write count.

Troubleshooting ingestion

Writes are fire-and-forget at the ingest boundary — ingestion errors do not block daemon liveness.

Troubleshooting missing charts (UI)

Use sampleCount, expectedSampleCount, and gapCount from the series API — do not treat null/zero the same.

Privacy

Metrics stores:

• serverId (authenticated UUID index) — from server context after auth, not client-supplied identity
• Low-cardinality dimensions: topologyGeneration and the capability-plan generation — daemon version, OS name, architecture, kernel release, and resolved entity identities are not stored as free-form metrics rows; stable logical entity ids (e.g. deviceId, gpuId) are the only per-entity identifiers persisted

Metrics does not store organization names, account emails, hostnames, client IPs, license tokens, or secrets. Query endpoints require session + server read grant — never bare UUID access. (Internally, DuckDB stores missing values as plain SQL NULL columns while Analytics Engine uses a missing-value sentinel — a backend implementation detail with no user-facing impact.)

Related

• Server metrics — what each console chart means
• Server metrics architecture — contract, mappings, cost model
• License tiers — the S1–S7/SX ladder, placement rules, and what each tier watches
• Control plane deployment — instance + co-located daemon
• Daemon setup — remote node install