Deployment Troubleshooting

This guide helps resolve common issues when deploying or running TurboPanel in production. For security-related issues, see Security. For setup verification, see Control plane and Daemon setup.

Introduction

Production issues typically fall into TLS/Caddy misconfiguration, instance socket permissions, daemon WebSocket connectivity, Postgres socket access, or Ansible playbook failures. Self-hosted browser entrypoint is HTTPS on port 8443 by default (Caddy). lets_encrypt serves :443 (plus :80 for HTTP-01). On co-located dev hosts only, Caddy also serves a plaintext mirror at http://<host>:8880 when TURBOPANEL_DEV_HTTP_CONTROL_PLANE=1.

Port conflicts (8443 / 8880 / 80 / 443)

Problem: Caddy cannot bind :8443 or (dev) :8880. In lets_encrypt, it cannot bind :80/:443.

Solutions:

1. Change CADDY_PORT / CADDY_HTTP_PORT in instance/Caddy env and update firewall rules
2. Check for existing listeners: ss -tlnp | grep -E '8443|8880|80|443'
3. Stop conflicting services or remap the published port in your reverse proxy

UI or API not loading

Problem: Browser shows errors at https://<host>:8443 by default, https://<hostname>/ in lets_encrypt, or (co-located dev) http://<host>:8880.

Solutions:

1. Check instance: systemctl status turbopanel-instance
2. Check Caddy: systemctl status turbopanel-caddy
3. Test health over HTTPS: curl -k https://localhost:8443/api/health (omit -k when TURBOPANEL_TLS_PUBLIC is set)
4. Co-located dev only: test plaintext mirror: curl http://localhost:8880/api/health (requires TURBOPANEL_DEV_HTTP_CONTROL_PLANE=1; 403 when the flag is off)
5. Logs: journalctl -u turbopanel-instance -u turbopanel-caddy -f
6. Static UI: ensure ui/dist exists if TURBOPANEL_UI_MODE=static
7. Dev UI: ensure turbopanel-ui is running when TURBOPANEL_UI_MODE=dev

Daemon shows offline but service is running

Problem: UI badge is Offline (or Update fails) while systemctl status turbopaneld is active and the Cell panel shows recent activity.

Cause: The server list uses a Postgres presence projection that can drift from live WebSocket state after a failed upgrade or reconnect.

Solution: Re-run the release installer — see Refresh a stuck daemon.

Daemon not connecting

Problem: Remote server daemon fails to register.

Solutions:

1. Verify instance URL: TURBOPANEL_INSTANCE_URL=https://<host>:8443 by default, or https://<hostname>/ in lets_encrypt. On co-located dev only, http://<host>:8880 when TURBOPANEL_DEV_HTTP_CONTROL_PLANE=1 (no CA required)
2. Test from node over HTTPS: curl -k https://<host>:8443/api/daemon/v1/version (omit -k when the origin is publicly trusted)
3. Co-located dev only: test plaintext: curl http://<host>:8880/api/daemon/v1/version
4. Confirm platform CA (HTTPS path, default mode): compare /etc/turbopanel/instance-ca.pem with GET /api/daemon/v1/instance/ca (bundle, current CA first). A 404 means public TLS — use the system trust store and do not install instance-ca.pem.
5. Firewall: allow 8443 by default from remote daemons to instance (443 in lets_encrypt; 8880 is dev-only and not for remote production nodes)
6. Logs: journalctl -u turbopaneld -f

Stale platform CA (tls-trust parked)

Problem: Daemon log shows tls-trust parked (dialed host, CA path, fingerprint) and reconnects only every 5 minutes to 1 hour — not a silent 30 s loop.

Cause: The host still trusts an old platform CA after the control plane rotated (or the leaf SAN no longer matches the dialed hostname). Control-plane identity (JWKS JWT) is unchanged; only the transport anchor is stale.

Solutions:

1. Prefer the overlap path: on the control plane, keep the old CA in ca-bundle.pem and enqueue server.tls.trust.reconcile while the existing WSS session is still valid.
2. If the session is already dead, re-run the installer with --instance-ca (or --insecure-tls for bootstrap fetch). run.sh pins --cacert first; on HTTP 000 it retries once unpinned and installs only if the fetched PEM validates the live leaf. Old and new fingerprints are printed when $CA_PATH is replaced.
3. If verification fails, keep the existing CA and fix the instance URL / SAN list rather than blindly replacing the file.

Postgres connection failures (self-hosted)

Problem: Instance cannot connect to Postgres.

Solutions:

1. Check Postgres container: docker ps | grep postgres
2. Verify Unix socket under /var/run/turbopanel/postgres/
3. Confirm TURBOPANEL_DATABASE_URL in turbopanel-instance.service — a full Postgres URL, typically injected by the instance-launch role
4. Socket directory permissions: /run/turbopanel should be 2770 tp:tp

Install wizard / PAM failures (Deno)

Problem: Install bootstrap rejects host credentials.

Solutions:

1. Install pamtester on the host
2. Confirm tpctrl user sudoers allows pamtester login * authenticate
3. Use a host account in sudo / wheel / admin, or root

UI Update stuck on "update already in progress"

Problem: After clicking Update, Ansible runs but the daemon keeps the same PID for a long time. A second update attempt returns update already in progress.

Cause: The UI update path uses run.sh --no-start (does not stop the daemon during reconcile). The daemon then tries to restart itself with plain systemctl enable --now as the tp user — that fails silently (non-root cannot restart systemd units; enable --now also skips restart when already active). An in-memory lock then blocks further updates.

Check logs on the server:

sudo tail -30 /var/log/turbopanel/daemon.err.log
sudo grep -i 'systemctl\|sudo\|update' /var/log/turbopanel/daemon.log | tail -20

Immediate recovery on the server:

sudo systemctl restart turbopaneld

Or reconcile by re-running run.sh — see Refresh a stuck daemon.

Ansible / upgrade failures

Problem: Upgrade System or daemon orchestration fails.

Solutions:

1. Check daemon logs — on managed hosts Ansible runs as the tp user; on co-located dev it runs as the current dev user
2. Dirty git checkouts block upgrade; commit or stash changes in $HOME source repos (~/turbopaneld, ~/turbopanel, ~/ui, ~/website)

General debugging

Related documentation

• Security — TLS and authentication
• Control plane — Service layout
• Daemon setup — Node installer
• Dev console troubleshooting — Local dev issues