Server metrics
Everything TurboPanel tracks
This is the complete list of what a TurboPanel server reports about itself: each reading, its unit, how often it is read, how long it is kept, and where you see it. The pages in this section explain how to read each chart. This page is the inventory.
Plain words
Names here are the ones you see in the web interface. Storage details (how readings are packed on disk or in the hosted store) are internal and are not documented.
The rules that apply to everything
| Topic | What happens today |
|---|---|
| Reading rhythm | The daemon reads the host and sends one sample a minute. The first sample after it connects goes out at once, and rates (CPU busy, disk and network speed) follow about two seconds later, once there are two readings to subtract. |
| Live view | While someone watches a short range, the daemon also sends a 10 second stream. The stream feeds the chart and is not stored; the minute sample is what history keeps. |
| Slower readings | A few readings are expensive, so they are refreshed less often and the latest value rides along with every minute sample. The refresh times are in the tables below and summed up in How often each thing is read. |
| How long it is kept | TurboPanel High Availability keeps about 3 months. Self-hosted keeps 90 days by default (TURBOPANEL_SERVER_METRICS_RETENTION_DAYS). Charts look back at most 90 days. Events and connection history are kept as long as the numbers. |
| Where it is stored | Each table has a Kept on column. Both means TurboPanel High Availability and Self-hosted keep the reading in history, and the two keep exactly the same readings. Live view only means neither keeps the reading in history: it appears in the 10 second live view and is empty in history on both. |
| Plans | Every group on this page is collected on every plan, Docker included. A plan never switches a metric on or off, and a plan does not limit how many network interfaces, drives, GPUs, extra filesystems or sensors are stored: a server keeps what it has, up to 64 of each kind. See License tiers. |
| Sizes | Every minute sample carries the sizes its percentages are measured against (below), so a resize or a memory balloon shows up as a step, not as a wrong percentage. |
| Limits on a server | A sample may carry at most 16 events; the 16 most severe are kept and the rest are held for the next sample (up to 64 extra events are carried; beyond that they are dropped). A sample dated more than 5 minutes away from the control plane's clock, in either direction, is refused. |
| Coming | A stored-sample check (exactly one per minute per server), a limit of 120 events an hour, a tighter 60 second bound on samples dated in the future, and a 5 GiB cap on the self-hosted metrics store are written but not released yet. They are listed under Coming. |
What “Not reported” means
A value is blank, never zero, when TurboPanel does not have it. Blank has four causes:
- The host cannot measure it. No swap, no GPU, no interrupt-pressure file in an older kernel, a virtual machine with no real sensors. The chart or the group is hidden or empty.
- The daemon is older than the control plane. A daemon that has not updated yet sends the plain sample, without the newer readings (OOM kills, container health, the sizes, host facts). They fill in once the daemon updates.
- The reading was out of range. A percentage is held between 0 and 100; any other impossible value (a negative size, for example) is stored as blank.
- TurboPanel does not store it on purpose. Some readings are not stored on either store, so that TurboPanel High Availability and Self-hosted keep exactly the same readings. 68 contract values stay in the wire contract and the descriptors but are stored by neither edition: the hosted layout has no slot for them and the self-hosted schema has no column. They show as no data in history on both, and are still shown in the 10 second live view. Nine of them were dropped to make room for the sizes, and each of those can be worked out another way:
| Not stored | Get it another way |
|---|---|
| Cores saturated | The top CPU process text names a hog; CPU pressure shows the effect. |
| Hosting free space | Free space on the disk that holds hosting (Root filesystem available or the extra filesystem that holds it). |
| Docker used space | Add the four Docker groups: image layers, container writable layers, volumes, build cache. |
| Logs used space | Not available. |
| Major page faults | Memory pressure is the better signal. |
| Unreclaimable kernel slab | Not available. |
| Caddy 2xx responses | Requests minus 4xx, minus 5xx. |
| Router backends up | Backends total minus the names of unhealthy backends. |
| Router requests | Caddy counts every request first. |
The other readings marked Live view only in the tables are among those 68 too: empty in history on both stores, and still shown in the live view.
Sizes and totals
These come with every minute sample. A chart divides each used figure by the size from its own sample.
| Size | Unit | Used to show | Kept on |
|---|---|---|---|
| Total RAM | bytes | Memory used % | Both |
| Total swap | bytes | Swap used % | Both |
| Commit limit (the most memory the kernel will promise to processes) | bytes | Committed memory vs limit | Both |
| Logical CPU count | count | The CPU count beside the CPU charts | Both |
| Root disk size and inode total | bytes, count | Root filesystem used % and inodes used | Both |
| Each extra filesystem's size and inode total | bytes, count | That filesystem's used % | Both |
| Each GPU's memory size | bytes | That GPU's memory used % | Both |
| Each network interface's link speed | megabits per second | Link speed shown with the interface | Both |
CPU
Read every minute. Detail: CPU.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| CPU busy (all modes) | percent | Both | CPU utilization, overview tile |
| User, system, I/O wait, steal (time taken by the host machine this server runs on), soft interrupts | percent | Both | CPU utilization |
| CPU pressure (some tasks waited for a CPU) | percent of the minute | Both | CPU pressure |
| Interrupt pressure (hardware interrupts stalled the machine). Not reported when the kernel has no interrupt-pressure file | percent of the minute | Both | Not charted yet; see Coming |
| Processes running, processes blocked, total processes | count | Live view only | Processes, overview tile |
| Process-limit use (share of the kernel's process ID limit) | percent | Both | Not charted yet; see Coming |
| CPU frequency (average, minimum, maximum) | MHz | Live view only | CPU frequency |
| Context switches, interrupts | per second | Live view only | Context switches and interrupts |
| Process forks | per second | Live view only | Process forks |
| Hardware-interrupt time | percent | Live view only | IRQ time |
| Cores saturated | count | Not stored (see Not reported) | Cores saturated |
Host facts for CPU (text, refreshed every 5 minutes, kept on both): load average, the busiest process by CPU, the CPU model.
Memory and swap
Read every minute. Detail: Memory.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Memory used | bytes, and percent of total | Both | Memory breakdown, Memory used, overview tile |
| Page cache (cached file data) | bytes | Both | Memory breakdown |
| Swap used | bytes, and percent of total | Both | Swap used, Swap used % |
| Memory pressure, some and full | percent of the minute | Both | Memory pressure |
| Processes killed for memory (OOM kills) | count in the minute | Both | Not charted yet; see Coming. The same kill also raises an event |
| Committed memory | bytes | Both | Committed memory |
| Swap in and swap out | bytes per second | Live view only | Swap I/O |
| Free memory, anonymous pages, shared memory | bytes | Live view only | Free and cached memory, Shared memory |
| Reclaimable kernel slab | bytes | Live view only | Slab memory |
| Dirty pages, pages being written back | bytes | Live view only | Dirty and writeback pages |
| Page scans (direct and by the background reclaimer) | per second | Live view only | Page reclaim |
| Compaction stalls | per second | Live view only | Compaction stalls |
| Major page faults | per second | Not stored | Major page faults |
| Unreclaimable kernel slab | bytes | Not stored | Slab memory |
Host facts for memory (text, kept on both): the biggest process by memory, the last process the kernel killed for memory.
Disk, filesystems and disk I/O
Read every minute unless noted. Detail: Storage, Filesystems, Block devices.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Disk I/O pressure, some and full | percent of the minute | Both | I/O pressure |
| Disk read and write speed (whole host) | bytes per second | Both | Disk throughput |
| Disk latency (whole host) | milliseconds | Both | Disk latency |
| Root disk queue depth, root disk operations | count, per second | Both | Not charted yet; see Coming |
| Root filesystem available, size | bytes, and percent used | Both | Root filesystem available, Root filesystem used |
| Root filesystem free inodes, inode total | count | Both | Root filesystem free inodes |
| Each extra (non-root) filesystem: available bytes, free inodes, size, inode total | bytes, count | Both | Filesystems |
| Per drive: read speed, write speed, operations, read latency, write latency, queue depth | bytes per second, per second, milliseconds, count | Both. With a single drive the whole-host figures above are the drive's figures, so this group appears with two or more drives | Block devices |
| Per drive: busy percent (utilization) | percent | Live view only | Block devices |
| Per drive: model and SMART verdict (text, refreshed every 30 minutes) | text | Both | Coming; see Coming |
| Software-RAID arrays degraded | count | Both | Not charted yet; see Coming. A degraded array also raises an event |
| Software-RAID arrays resyncing | count | Live view only | Not charted yet |
| Read-only filesystems, RAID state (text, refreshed every 5 minutes) | text | Both | Coming |
Storage accounting
The daemon walks the hosting, backup and log folders every 15 minutes and re-sends the latest result with each minute sample. A walk that hits its time limit reports blank, not a partial total.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Hosting used | bytes | Both | Hosting root, overview tile |
| Backups used, backups free | bytes | Both | Backups |
| Logs free | bytes | Live view only | Logs |
| Hosting free, Docker used, logs used | bytes | Not stored | See Not reported |
| Largest sites on disk (text, refreshed every 5 minutes) | text | Both | Coming |
Managed databases on the host
The daemon counts managed database containers every 5 minutes, found by their label, and probes each running one for readiness.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Postgres, MySQL and MariaDB instances running and healthy | count | Both | Managed databases |
| Connections used and the connection limit, per engine | count | Live view only | Managed database connections |
| Engine versions (text, refreshed every 30 minutes) | text | Both | Coming |
Network
Read every minute. Detail: Network.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| TCP retransmits | percent of segments | Both | TCP retransmit rate |
| Softnet drops (packets the kernel dropped before a program saw them) | per second | Live view only | Softnet drops |
| Connection-tracking table use, open file handles use | percent | Both | Kernel resource usage |
| Per interface: receive and transmit speed | bytes per second | Both | Network |
| Per interface: receive and transmit errors, receive and transmit drops | per second | Both from the third interface on. For the first two interfaces both keep one combined problems figure (errors plus drops) | Network |
| Per interface: link state and speed (text) | text | Both | Coming |
Interfaces that belong to the TurboFabric mesh are not listed as ordinary interfaces.
GPU
Read every minute, on physical servers and on virtual machines with a real passed-through GPU. A virtual display adapter does not count as a GPU. Detail: GPU.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| GPU utilization | percent | Both | GPU |
| GPU memory used | bytes, and percent of the GPU's memory | Both | GPU |
| GPU memory activity | percent | Both | GPU |
| PCIe receive and transmit | bytes per second | Both | GPU |
| Throttling | percent | Both | GPU |
| GPU model, driver version, memory size (text) | text | Both | Coming |
GPU temperature, memory temperature and power are listed under Temperatures and power.
Temperatures and power
Read every minute, on physical servers only (a virtual machine reports none, because its probes are not real). A server reports up to 19 signals. Temperatures are stored in Celsius and shown in your organization's unit. Detail: Physical signals.
| Signal | Unit | Kept on | Screen |
|---|---|---|---|
| CPU package temperature and power | degrees, watts | Both | Physical signals |
| Hottest core, CPU thermally throttled | degrees, percent | Both | Physical signals |
| Board and ambient temperatures the daemon trusts | degrees | Both | Physical signals |
| Each drive's temperature | degrees | Both | Physical signals |
| Each GPU's core temperature, memory temperature and power | degrees, watts | Both | Physical signals |
Fan speed is not tracked as a number. Fan faults and alarms arrive as events.
Docker and containers
Docker metrics are on every plan. Container health and totals are refreshed every 30 seconds; disk use comes from Docker every 5 minutes (and backs off if Docker is slow to answer). Detail: Docker.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Containers running, unhealthy, restarting | count | Both | Not charted yet; see Coming |
| Containers killed for memory, containers that exited unexpectedly | count in the minute | Both | Not charted yet; see Coming |
| Containers' CPU (share of the whole host), containers' memory | percent, bytes | Both | Not charted yet; see Coming |
| Image layers, container writable layers, volumes, build cache | bytes | Both | Image layers, Container writable layers, Volumes, Build cache |
| What a prune would reclaim, in total | bytes | Both | Not charted yet |
| What a prune would reclaim from images, volumes and build cache separately | bytes | Live view only | Build cache, Volumes |
| Images, containers and volumes | count | Live view only | Object counts |
| Docker version, names of unhealthy containers (text, refreshed every 5 minutes) | text | Both | Coming |
An unexpected exit is one that is not a deploy and not a stop you asked for.
Web traffic: Caddy
Read every minute. TurboPanel reads the hosting Caddy for totals only; there are no per-site traffic numbers. Detail: Ingress (Caddy).
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Requests | count in the minute | Both | Ingress (Caddy) |
| 4xx responses, 5xx responses, request errors | count in the minute | Both | Ingress (Caddy) |
| Request bytes, response bytes | bytes in the minute | Both | Ingress (Caddy) |
| Time spent on requests (sum), so the average is derived | seconds | Both | Ingress (Caddy) |
| Requests finished within 100 ms, 500 ms, 1 s | count in the minute | Both | Ingress (Caddy) |
| Requests in flight | count | Both | Ingress (Caddy) |
| Days until the soonest hosting certificate expires (read every 6 hours) | days | Both | Soonest TLS expiry |
| 3xx responses, requests within 10 ms, 50 ms and 5 s, retries, upstreams healthy and total | count | Live view only | Ingress (Caddy) |
| 2xx responses | count | Not stored | See Not reported |
Percentiles (p50, p90, p99) are worked out when you open the chart; they are never stored, because a stored percentile cannot be combined across time.
Host facts for the web layer (text, refreshed every 5 minutes, kept on both): Caddy version, the soonest-expiring certificate's name, PHP versions, web engine versions, the busiest PHP pool.
Web traffic: the hosting router (Traefik)
Present only when Docker is on the host. Detail: Router (Traefik).
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Backends in total | count | Both | Backends and services |
| Backend 5xx errors | count in the minute | Both | Backend requests |
| Backend latency (average) | milliseconds | Both | Backend latency |
| Services, routers, retries | count | Live view only | Backends and services |
| Open connections | count | Live view only | Open connections |
| Config reloads, time since the last reload | count, seconds | Live view only | Config reloads, Since last config reload |
| Backends up, requests | count | Not stored | See Not reported |
Database proxy (ProxySQL)
Present only when managed databases run. Read every minute. Detail: Database proxy.
| Metric | Unit | Kept on | Screen |
|---|---|---|---|
| Queries, slow queries | count in the minute | Both | Database proxy |
| Query latency, backend latency (averages) | milliseconds | Both | Database proxy |
| Active transactions, client connections, backend connections | count | Both | Database proxy |
| Clients aborted, connections refused at the limit, connection errors | count in the minute | Both | Database proxy |
| Backends up, backends in total | count | Both | Database proxy |
| Client connections opened, backend connections opened and aborted, bytes to and from the backends | count, bytes | Live view only | Database proxy |
Host health and facts
Short pieces of text and a few counts that describe the host rather than chart over time. They are refreshed every 5 minutes (drive health and engine versions every 30 minutes) and travel with every minute sample.
| Fact | Unit | Kept on |
|---|---|---|
| Failed system services (count) | count | Both |
| Names of failed services | text | Both |
| Kernel release, operating system release, boot identity | text | Both |
| Virtualisation (for example a virtual machine or not), cloud provider | text | Both |
| TurboPanel daemon version | text | Both |
| Clock sync state | text | Both |
| Pending updates, reboot required | text | Both |
| Drive models and SMART verdicts, GPU model and driver, network link speeds | text | Both |
| Software versions: Docker, Caddy, Traefik, PHP, web engines, database engines | text | Both |
The newest of these is meant to show as About this host in the web interface, and the counts as Host health. Neither is in the web interface yet; see Coming. Self-hosted keeps the same host text in history as TurboPanel High Availability.
Events
Events are things that happened at a moment: an out-of-memory kill, a disk fault, a network link going down, a GPU fault, the clock losing sync. TurboPanel has 38 kinds in a fixed list. The full list with explanations is on Metric events.
| Group | Kinds |
|---|---|
| Kernel and operating system | out-of-memory kill, hung task, connection-tracking table full |
| Filesystems | read-only, disappeared, remount |
| Disks and RAID | SMART critical, NVMe critical, NVMe media error, RAID degraded, rebuild started, completed, failed |
| Network and TurboFabric | link down, link up, flapping, mesh peer change, mesh unavailable, mesh recovered |
| Thermal, power and memory errors | fan fault, fan alarm, temperature alarm, temperature critical, power supply fault, voltage alarm, memory error corrected, memory error uncorrected |
| GPU | Xid fault, memory error, row remap, page retirement, fallen off the bus, thermal critical, disappeared |
| Clock and restarts | clock sync lost, clock sync restored, hardware list changed, boot count changed |
Checks run at different rhythms: most on every sample, hung tasks every 2 minutes, clock sync every 5 minutes, drive SMART every 10 minutes.
- Kept: as long as the numbers (about 3 months on TurboPanel High Availability, 90 days by default on Self-hosted).
- Limit today: at most 16 events per sample; over that, the 16 most severe are kept and the rest are held for the next sample (up to 64 extra events are carried; beyond that they are dropped).
- Where: the Recent events panel on the Metrics screen. The panel is left out when nothing happened in the range.
- Hardware-health events (the disk, thermal, power, memory and GPU groups) are on for physical servers by default.
How often each thing is read
| Cadence | What |
|---|---|
| Every minute | CPU, memory, disk and network numbers, GPU, temperatures and power, pressure, Caddy, router and database proxy figures |
| Every 10 seconds (live view only, never stored) | The same numbers, while someone watches a short range |
| Every 30 seconds | Container health and totals |
| Every 2 minutes | Hung-task check |
| Every 5 minutes | Docker disk use, managed database count, clock-sync check, host text (software versions, load, top processes, largest sites) |
| Every 10 minutes | Drive SMART events |
| Every 15 minutes | Hosting, backup and log folder sizes |
| Every 30 minutes | Drive health text, database engine versions |
| Every 6 hours | Hosting certificate expiry |
A slower reading shows the same value for several minutes in a row. That is not a stuck chart.
What shows on which screen
| Screen | What it shows |
|---|---|
| Metrics on a server | One chart per reading in the tables above, grouped as on this page, with a range from 5 minutes to 90 days. Gaps in a line are missing samples, not zero. |
| Overview tiles at the top of Metrics | The latest CPU utilization, memory used, hosting storage used, running and blocked tasks, and uptime. See Server metrics. |
| Recent events on Metrics | The events above, newest first. |
| Physical signals on Metrics | Temperatures and power, physical servers only. |
| Servers list | Whether a server is online right now. It does not come from metrics history. |
Coming
These are written but not released. This page will drop the label when they are.
- Host health and About this host in the web interface, and a chart for each health number that has none yet (OOM kills, interrupt pressure, process-limit use, root disk queue and operations, failed services, degraded RAID arrays, container health, containers' CPU and memory).
- A host-facts route that returns the newest facts.
- Exactly one stored sample a minute per server: a duplicate is refused, a sample sent too early is refused with a wait time.
- 120 events an hour per server: events over the budget are dropped, most severe kept first, and the sample itself is still stored.
- A 60 second bound on samples dated in the future. Today the bound is 5 minutes either way.
- A 5 GiB cap on the Self-hosted metrics store; the oldest days are removed first.
Related
- Server metrics — how to read the Metrics screen and pressure stall information
- Server metrics architecture — contract, limits, cost
- Metrics operations — retention and troubleshooting
- License tiers — box sizes and what each watches
Edit on GitHub
Last updated on