Ingress metrics

This group appears when the daemon can scrape the site Caddy on the host. The title is Caddy. A host with no site proxy has no group.

These are HTTP-level counters for the interval, not host network bytes. The shared hosting router behind Caddy is a separate group — Router — because its backend / service view has no Caddy equivalent.

Caddy counts each request once

Caddy instruments every handler in the chain. TurboPanel uses the outermost handler (the largest request total) so a request is not counted once per nested handle / reverse_proxy.

Requests

Series	Wire name	What it is
Requests	`ingress.requests`	Requests finished in the interval
Errors	`ingress.requestErrors`	Requests that failed before a normal response (connect errors, cancelled, adapter-defined errors)

A jump in errors with a flat request line is failures, not traffic. Pair with 5xx and retries.

Responses by status

Stacked counts in the interval:

Series	Wire name	What it is
2xx	`ingress.responses2xx`	Success
3xx	`ingress.responses3xx`	Redirects
4xx	`ingress.responses4xx`	Client errors (auth, not found, bad request)
5xx	`ingress.responses5xx`	Server / upstream errors, as the client saw them

A 4xx pile after a deploy is often a bad path or a probe. A 5xx pile is the host or an upstream failing — check upstreams, the router's container 5xx, and the app, not just CPU. Caddy's 5xx minus the router's 5xx is failures in the proxy path rather than in the application.

Throughput

Series	Wire name	What it is
Request bytes	`ingress.requestBytes`	Request body bytes in the interval
Response bytes	`ingress.responseBytes`	Response body bytes in the interval

Large response bytes with modest request counts is downloads or big API payloads. This is not the same as NIC throughput (headers, other ports, and non-HTTP sit on the NIC).

Request latency

Three series — p50, p90, p99 — in milliseconds. Half of requests finished under the p50 line, nine in ten under p90, ninety-nine in a hundred under p99.

None of these is stored. The daemon ships one raw number, ingress.requestDurationSecondsSum (total request time in the interval), plus the six bucket counters below. The control plane computes the percentiles — and the mean, sum / requests — at read time from those.

Why percentiles are computed, never stored

Percentiles do not add up across time. If each minute stored its own p99, a five-minute chart would have to average five p99s, and the average of per-minute p99s is not the five-minute p99 — it is a number with no meaning. A duration sum and a set of bucket counts both add cleanly, so a 5-minute view and a 24-hour view derive from the same math. The same reasoning is why the old stored average is gone.

Six buckets topping out at five seconds cannot resolve a p999 — the number would be dominated by the last bound — so no p999 is offered at this resolution.

A rising p50 with a stable request rate is the typical "the app got slower" signal. A rising p99 with a flat p50 is a tail: a few slow endpoints, a lock, or garbage collection. Confirm which endpoint with the router's backend latency.

Requests under threshold

Counts of requests that finished faster than each cutoff. A request under 10 ms is also under 50 ms, 100 ms, and every larger bound — the bands are cumulative, not exclusive buckets. These are the raw counters the percentiles above are derived from.

Series	Wire name	Cutoff
≤10ms	`ingress.bucket10ms`	10 milliseconds
≤50ms	`ingress.bucket50ms`	50 milliseconds
≤100ms	`ingress.bucket100ms`	100 milliseconds
≤500ms	`ingress.bucket500ms`	500 milliseconds
≤1s	`ingress.bucket1s`	1 second
≤5s	`ingress.bucket5s`	5 seconds

The two smallest bounds are new in schema v6: with 100 ms as the first cutoff, every request on a healthy site landed in the first bucket and the p50 was meaningless. When total requests stay flat but the ≤10ms line falls while ≤100ms holds, work moved from fast to merely okay. When even ≤5s falls relative to requests, a growing share is slower than five seconds.

Upstreams & in-flight

Series	Wire name	What it is
In-flight	`ingress.requestsInFlight`	Requests the proxy is handling right now
Healthy upstreams	`ingress.upstreamsHealthy`	Backends the proxy currently considers up
Total upstreams	`ingress.upstreamsTotal`	Backends configured

In-flight climbing with latency is a backup forming — the proxy is holding connections. Healthy falling below total is an upstream marked down; 5xx usually follows.

Retries

ingress.retries — proxy retries in the interval.

Retries hide backend blips from clients and multiply load on the remaining healthy upstreams. A retry spike with falling healthy upstreams is the cascade to watch.

Router — the shared hosting Traefik behind Caddy
Network — TCP retransmits and kernel tables
Database proxy — ProxySQL, not HTTP

On this page