Troubleshooting

Diagnose and fix common VaultPanel problems by symptom, from installation and agents to websites, SSL, apps, databases, backups and the assistant.

Last updated

Problems are grouped by where they show up. Most failures in the panel carry an error code (such as DOMAIN_NOT_REACHABLE) and a request id; search this page for the code.

Start here

Three commands on the panel's server answer most questions:

vaultpanel status                 # panel, worker, agent, database and Redis
sudo vaultpanel logs panel -f     # the API's log; also: worker, agent
sudo journalctl -u vaultpanel-agent -n 100 --no-pager

On any other server, the agent's state is in its own services:

systemctl status vaultpanel-agent vaultpanel-agent-helper
journalctl -u vaultpanel-agent -u vaultpanel-agent-helper -n 100 --no-pager

Installation

The installer stops with a plain message when something is wrong. The common ones:

MessageFix
this installer supports Ubuntu 22.04/24.04/26.04 and Debian 12/13Use a supported system. See System requirements.
systemd is required (containers without systemd are not supported)Install on a VM or a server, not in a container without systemd.
port 8090 is already in useStop what uses it, or install on another port: --port 8095.
port 8091 (the panel's loopback port) is already in useSet VAULTPANEL_LOOPBACK_PORT=8095 in /opt/vaultpanel/.env, then sudo vaultpanel update --force.
port 3307 or port 6379 … is already in use on 127.0.0.1Something else (often a panel started by hand from a copy of the repository) holds the panel's database or Redis port. Stop its containers first.
port 3306 is already in use, so MariaDB cannot start thereStop what uses it, or install with DB_ENGINE=NONE.
MySQL is not available yetMySQL 8.4 and 8.0 come in a later version. Choose a MariaDB version, or NONE.
… is not available on <system>The DB_ENGINE you chose is not offered for this system. The message lists the ones that are.
Docker is installed but not runningsystemctl start docker, then run the installer again.
the vaultpanel package did not installFix what the lines above it say, then sudo dpkg --configure -a.
the panel did not startThe log above the message says why; more with vaultpanel logs.

Running the installer again is safe: it keeps .env and all data.

Setup and signing in

The setup token is refused

The setup token creates the owner account and stops working once it exists. On an installed panel, sudo vaultpanel setup-token prints the current one. A panel started by hand prints it in the API log ("No accounts exist yet…"); it changes on every restart unless VAULTPANEL_SETUP_TOKEN is set.

The API exits with "Invalid configuration"

A required setting is missing or malformed, and the message lists which. VAULTPANEL_SECRET_KEY must decode to exactly 32 bytes; generate one with openssl rand -hex 32.

"Set up two-factor sign-in" on every page

An owner requires two-factor sign-in for your role (Security > Authentication). Set it up, or ask an owner to change the policy. If you lost your phone and your recovery codes, an owner or admin can turn two-factor sign-in off for you under Users (Edit) or on Security > Authentication; you are then signed out everywhere.

"The panel does not accept your address" (IP_NOT_ALLOWED)

Panel access by address is on and your address is not on the list. From an allowed address, add yours under Security > Authentication.

If you are locked out completely, on the panel's server:

Turn the check off

Add VAULTPANEL_PANEL_ACCESS=off to /opt/vaultpanel/.env and run sudo vaultpanel restart.

Fix the list

Sign in and correct the addresses under Security > Authentication.

Turn the check back on

Remove the line from .env and run sudo vaultpanel restart again.

Behind a reverse proxy or Cloudflare, everyone appears with the proxy's address unless VAULTPANEL_TRUST_PROXY is set.

The panel's HTTPS and domain

The browser warns that the connection is not private

The panel's port uses a self-signed certificate made for your server. Check that the SHA-256 fingerprint the browser shows matches the one sudo vaultpanel https prints, then continue. For a certificate browsers trust, give the panel a domain. See Panel HTTPS and domain.

"Cannot read the panel's TLS certificate or key" after an update

The files named by VAULTPANEL_TLS_CERT and VAULTPANEL_TLS_KEY are missing or not readable by the vaultpanel group. sudo vaultpanel https on makes a new certificate and fixes the permissions. It prints the command to run on each other server, so their agents trust the new certificate.

Agents went Offline after vaultpanel https on or https renew

They still talk plain HTTP, or trust the old certificate. Run the command that vaultpanel https on (or renew) printed on each of them; sudo vaultpanel https shows the pin again. Their registration is kept.

The panel stopped loading after turning on the firewall

VaultPanel opens the port in VAULTPANEL_PUBLIC_URL on the panel's own server. If you open the panel on another port, add a firewall rule for it, or correct the address with sudo vaultpanel url <address>.

Panel maintenance

"EACCES" or "read-only file system" in vaultpanel logs

The panel runs as the vaultpanel user and may only write to VAULTPANEL_DATA_DIR (/var/lib/vaultpanel/panel). sudo vaultpanel update --force puts the ownership right. A VAULTPANEL_DATA_DIR set by hand elsewhere must belong to vaultpanel.

vaultpanel backup says the MariaDB container is not running

Start it with sudo vaultpanel start. vaultpanel backup needs the panel's own database from the installer; a panel on your own database server is backed up with your own database backups, plus .env and /etc/vaultpanel/tls.

vaultpanel restore fails at "the database update failed"

The backup comes from a newer VaultPanel than the one installed. Install that version first with sudo vaultpanel update --version vX.Y.Z (the backup's version is printed when the restore starts), then restore again.

vaultpanel rollback says no earlier version is recorded

Rollback reverses one vaultpanel update. To go to any release, run sudo vaultpanel update --version vX.Y.Z, then restore a backup from that time if the database changed in between. See Updating.

Servers and agents

A server stays "Waiting for agent"

Check the output of the install command on the server. Enrollment tokens work once and expire after 60 minutes by default (VAULTPANEL_ENROLLMENT_TTL_MINUTES); create a new one from the server's Settings tab. "Agents must connect over HTTPS" means the panel's address is plain HTTP.

"does not match the pin in the install command"

The panel answering at that address does not have the certificate the command was made for: it was replaced, or something between the servers intercepts the connection. Copy a new install command from the panel and check the address.

A server shows Offline

The agent has not reported for 90 seconds (VAULTPANEL_AGENT_OFFLINE_SECONDS). On the server, run systemctl status vaultpanel-agent and journalctl -u vaultpanel-agent.

ErrorCauseFix
CLOCK_SKEWThe server's clock is off by more than 5 minutes.Turn on NTP: sudo timedatectl set-ntp true.
AGENT_REVOKEDThe server was removed or registered again.Run the install command again with a new token.

A server went Offline after setting outgoing connections to Deny

The agent keeps the panel's address reachable, but resolves it once, when the outgoing default changes. If the panel's IP address changed since, add an outgoing allow rule for the new address from the server's console (ufw allow out to <panel IP> port <port> proto tcp), or run ufw default allow outgoing. Programs that reach other services, such as a remote database, need their own outgoing rules.

"The agent runs as root" in the security checks

The reason follows the message:

  • "did not reach the panel as the vaultpanel-agent user": the move to its own user was undone after 10 minutes. journalctl -u vaultpanel-agent -u vaultpanel-agent-helper shows why, often a firewall or proxy setting only root had. Fix it, then sudo rm /var/lib/vaultpanel/agent-mode.json && sudo systemctl restart vaultpanel-agent.
  • "runs from …": the agent was not installed by the install script (a development checkout) and stays as it was started.

HELPER_UNAVAILABLE

The agent runs as its own user but its root helper does not answer. Check systemctl status vaultpanel-agent-helper and journalctl -u vaultpanel-agent-helper; sudo systemctl restart vaultpanel-agent-helper starts it again. To go back to one root service, add "runAsRoot": true to /etc/vaultpanel-agent/config.json and restart the helper.

"Update agent" says the Node.js agent is replaced by running the command

The server still runs the old Node.js agent (a version without -rs, such as 0.42.0). It cannot install the current agent from an update job, so Update agent is refused and bulk updates skip the server. Run the command shown on the server's page once, on the server, as root:

curl -fsSL https://panel.example.com/api/v1/agent/install.sh   | sudo bash -s -- --upgrade --panel https://panel.example.com

A panel with a self-signed certificate adds the pin options; copy the command from the page. It installs /usr/local/bin/vaultpanel-agent, keeps the registration and policy, moves cron jobs and Certbot's wildcard hook to the new program, and removes /opt/vaultpanel-agent once the new agent runs. If the agent does not start, the old folder is kept: check journalctl -u vaultpanel-agent. See Moving from the Node.js agent.

AGENT_PACKAGE_MISSING

The panel has no agent program for the server's CPU type, so it cannot install or update the agent there. The agent runs on x86-64 and arm64; a panel installed from the package carries both.

The panel's own server does not show up

Look for lines starting with [vaultpanel] in the API's log when it starts; they say why. VAULTPANEL_LOCAL_AGENT=false turns the automatic connection off. See Connecting servers.

Jobs

ErrorMeaningFix
AGENT_UNREACHABLENo agent picked the job up within 5 minutes.Check the server is online, then try again.
AGENT_OUTDATEDThe server's agent predates the operation.Click Update agent on the server's page, or run the upgrade command shown there. Registration is kept.
NOT_ALLOWEDThe agent's local policy refuses it (allowedServices, allowReboot).Edit /etc/vaultpanel-agent/config.json and restart the agent.
TIMEOUTThe command did not finish in time.The job page shows the output collected so far.
PROCESS_CHANGEDThe process exited and its pid was reused before the job ran.Refresh the process list and try again.

For This server, restarting the panel (sudo vaultpanel restart) also updates its agent: the panel connects its own server before the API starts.

Websites and SSL

A website stays "Failed"

Its page shows the agent's message and a link to the log. Fix the cause, then click Try again.

ErrorFix
NGINX_NOT_INSTALLED, PHP_NOT_INSTALLEDInstall them from the server's Websites tab, Web stack card.
NGINX_CONFIG_INVALIDNGINX refused the new configuration, often because another site or a hand-edited file is broken. Run sudo nginx -t on the server.
SITE_USER_CONFLICTA Linux user with the site's name already exists for something else. Pick another site name.
WEBSITE_BUSYAnother change to the same website is still running. Wait for it and try again.

The site is created but "Not answering"

For a reverse proxy, start your app on the port shown: NGINX answers 502 until something listens there. For PHP, check the PHP-FPM service on the server's Services tab.

Certificate requests fail with DOMAIN_NOT_REACHABLE

Let's Encrypt was not asked: the agent could not fetch a test file through the domain. Check on the Domains tab that each domain points at this server, and that ports 80 and 443 are open (ufw allow 80,443/tcp). Use the staging option while testing, to stay clear of Let's Encrypt's rate limit of 5 failures per hour per domain.

Certificate requests fail with ACME_NOT_SERVED

The server's own NGINX could not serve the Let's Encrypt check file, often with HTTP 403. Run sudo nginx -t and read /var/log/nginx/error.log. On agents before 0.12.1, /var/lib/vaultpanel could be left closed; update the agent and issue again, or run sudo chmod 711 /var/lib/vaultpanel.

See SSL certificates.

File manager

SymptomFix
OUTSIDE_SITEThe path goes through a symbolic link to somewhere outside the site's folder. The file manager never follows such links; use SFTP as the server's owner instead.
"Cannot find module '/opt/vaultpanel-agent/src/services/fileWorker.js'"Old Node.js agents only, before 0.37.0: the file worker runs as the site's user, which cannot read the agent's folder. Move the server to the current agent, or run sudo chmod -R u=rwX,go=rX /opt/vaultpanel-agent.
"Could not start the file worker: spawn … EACCES"Agents before 0.9.1 could not run file requests when Node.js lives in a home folder (nvm). Update the agent, then Try again on the website.

Server self-check

The Self-check card on a server's Overview (agent 0.39.0) shows the command that fixes each failed check, and the Server self-check failed alert fires when a check stays failed for 10 minutes. See Server self-check.

CheckFix
Root helperSee HELPER_UNAVAILABLE.
Agent privileges (warning)See "The agent runs as root".
Agent filesThe command on the card: sudo chmod -R u=rwX,go=rX /usr/local/bin/vaultpanel-agent for the program, or sudo chmod o+x on the folder the check names.
File managerIts worker did not start as a site user. After fixing the agent files, click Check now.
NGINX configuration, Apache configurationsudo nginx -t or sudo apache2ctl -t shows the file and line. Until it passes, every website change is refused. A file VaultPanel wrote (vp-<site>.conf) is rewritten by saving the website's settings again.
Certificate renewal (warning)sudo systemctl enable --now certbot.timer, or issue a certificate from the panel. The panel renews due certificates itself either way.
Clock (warning)sudo timedatectl set-ntp true
Failed services (warning)systemctl --failed, then journalctl -u <unit> for each.

Applications and deployments

An app stays "Failed"

The Deployments tab shows which stage failed and its log. The previous release keeps running.

Stage or errorFix
fetch (CLONE_FAILED)The repository URL or branch is wrong, or a private repository needs the app's deploy key (on its Settings tab) added to the repository.
install or buildRun the same command in the repository on your computer to see the error.
start (HEALTH_CHECK_FAILED)The app did not listen on its port in time. It must listen on process.env.PORT (Node.js) or $PORT (Python).
APPLICATION_BUSY from a webhookA deployment is already running; the next push deploys.

A command field says it uses shell syntax

Install, build and start commands run without a shell, so &&, |, ; and > are refused. Put the steps on separate lines, or in a script in your repository. CONFIGURE_FAILED saying a program is not installed means the start command's program (for example gunicorn) is not in the app's dependencies.

The app runs but its domain shows 502

The linked website points NGINX at the app's port. If you changed the port, use Apply on the app so the app and NGINX move together.

SOFTWARE_MISSING or NODE_IN_HOME_FOLDER

SOFTWARE_MISSING means Git, PM2, Python or MariaDB is not installed; install it from the server's Software card. NODE_IN_HOME_FOLDER means the only Node.js on the server lives under /root or /home (for example through nvm), which app users cannot read. Install Node.js system-wide:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - && sudo apt-get install -y nodejs

See Deployments.

Containers

SymptomFix
"Update the agent to manage containers"Containers need agent 0.10.0 or newer.
Docker shows "Stopped", or DOCKER_NOT_RUNNINGClick Start Docker on the Containers page. By hand: snap start --enable docker for the snap (which docker shows /snap/bin/docker), or systemctl enable --now docker for Ubuntu's package.
"Could not pull … toomanyrequests"Docker Hub limits anonymous pulls. Wait, or use the same image from a mirror such as mirror.gcr.io/library/nginx:alpine.
CONTAINER_EXITED or "did not stay up"The container stopped during its health check; the error shows its last output. Usual causes: a missing environment variable, a wrong command, or too little memory. A container it replaced was started again.
A port "is used by …"Another app, container, SSH, the panel or VaultPanel's database uses that host port. Pick another host port.
A public port is reachable without a firewall ruleDocker adds its own rules for published ports, ahead of UFW. Keep ports on This server only and add a reverse-proxy website for them.

See Containers.

Databases

ErrorFix
DATABASE_SERVER_NOT_INSTALLEDInstall MariaDB from the server's Software card.
DATABASE_SERVER_DOWNStart the mariadb service on the server's Services tab. If root cannot sign in over the socket, set mysqlDefaultsFile in the agent's config.
DATABASE_PORT_IN_USESomething else holds port 3306, usually the panel's own development MariaDB from an older docker/compose.yml. Move it to 3307, then try again.

See Databases.

Email

Mail from other providers never arrives, or is refused

  • Port 25 blocked. Some providers block port 25 on new servers. Ask yours whether incoming and outgoing port 25 is open; ss -ltn | grep ':25 ' shows whether Postfix listens.
  • Reverse DNS. Check DNS on the Email page shows where the server's IP address points back to. Set its reverse DNS (PTR) to the mail host name with your provider, or Gmail and others may refuse your mail.
  • DNS records. On the domain's DNS records tab, MX, SPF, DKIM and DMARC should show In place. When the domain's DNS is elsewhere, copy the records there.

The panel's own email (invites, resets, alerts) is not sent

On System > Panel Email, Send test email shows a plain reason for a failure: a wrong password, a port that does not answer, or the wrong security option for the port (SSL/TLS goes with 465, STARTTLS with 587). Many providers block outgoing port 25, so use 587 or 465. Alert emails are sent by the worker, so it must be running (vaultpanel status).

See Email hosting and Panel email.

Backups and cron jobs

Error or symptomFix
BACKUP_DISK_FULLLess than 512 MB is free where backups are kept (/var/backups/vaultpanel). Delete old backups, lower a schedule's keep count, or change backupMinFreeBytes in the agent's config.
BACKUP_CORRUPTThe file no longer matches the checksum taken when it was made, so it is not restored. Use another backup.
A backup shows MissingIt was removed from the server by hand.
BACKUP_TOO_LARGE on a downloadThe backup is bigger than VAULTPANEL_BACKUP_DOWNLOAD_MAX_MB (10 GB by default). Copy it from the server with scp.
A restored app does not start (HEALTH_CHECK_FAILED)The files are back but the health check failed. Restore the safety copy taken just before, or deploy again.
"Cron is not installed"Install Cron from the Scheduler card on the server's Automation tab, then save the job again.
A cron job never runsCheck its next run on the Cron Jobs page: schedules use the server's time zone, shown in the form.

See Backups and Cron jobs.

Webhooks and alerts

A webhook delivery says "which webhooks may not reach"

Webhooks never call the panel's own machine, link-local or cloud metadata addresses. Private networks (10.x, 172.16-31.x, 192.168.x) are refused too unless VAULTPANEL_WEBHOOK_ALLOW_PRIVATE=true. New installs set it to false; panels installed before that default changed keep true.

An alert never fires

A metric must stay over the threshold for the rule's whole duration, and the server must be online with fresh data; while it is offline only the "server offline" rule is evaluated. The worker checks rules every 15 seconds, so it must be running (vaultpanel status).

Firewall, SSH and Fail2ban

LOCKOUT_REFUSED

VaultPanel is protecting your way in: a firewall rule that would block SSH, the panel or your own address, or an SSH change that would leave no administrator able to sign in. Add the replacement first (another rule for the port, or your SSH key), then try again.

Locked out of SSH anyway

For example by a rule added with ufw by hand. Use your provider's web console, then run ufw allow 22/tcp or ufw disable. For SSH settings, remove /etc/ssh/sshd_config.d/00-vaultpanel.conf and run systemctl reload ssh.

SSH errors

ErrorFix
Cannot connect after changing the portUse ssh -p <port>. A firewall at your hosting provider may need the port opened too. Keep the old port open until you have signed in on the new one.
PORT_IN_USEAnother program listens on that port. Pick another.
SSH_NOT_LISTENINGSSH did not answer on the new port, so the old setting came back. journalctl -u ssh says why (SELinux or AppArmor rules can refuse other ports).
SSHD_OVERRIDDENA line elsewhere in /etc/ssh/sshd_config or its sshd_config.d folder sets the same option and wins. Remove or comment it out and save again. VaultPanel's file is 00-vaultpanel.conf.
SSHD_CONFIG_INVALIDShows what sshd -t said. The previous settings stay.
KEYS_NOT_WRITABLEThe agent's service cannot write /root/.ssh. Update the agent (0.6.0 and newer).
JAIL_FAILEDFail2ban's SSH jail did not start, usually because python3-systemd is missing: apt install python3-systemd, then save again. /var/log/fail2ban.log has the details.

See Firewall and SSH.

The AI assistant

SymptomFix
It keeps "thinking" and never answersAnswers are written by the worker, so it must be running (vaultpanel status). After 10 minutes the question is marked failed and Try again appears.
The provider rejected the key, or the model was not foundOpen System > Assistant Settings and use Test connection. For Ollama and similar services, choose the OpenAI-compatible provider with an address ending in /v1 (for example http://127.0.0.1:11434/v1) and a model that supports tool calling.
A server's agent is too old to list portsUpdate that server's agent (0.9.0 adds listening ports).

See AI assistant.

Development checkouts

connect ECONNREFUSED 127.0.0.1:3307

From npm run db:migrate or the backend: the panel's development MariaDB is not running. Bring the containers up with docker compose -f docker/compose.yml up -d, wait until docker compose -f docker/compose.yml ps shows MariaDB healthy, then migrate and restart the backend. The containers restart after a reboot as long as Docker starts at boot (systemctl enable docker).

See Development.

On this page

Start here
Installation
Setup and signing in
The setup token is refused
The API exits with "Invalid configuration"
"Set up two-factor sign-in" on every page
"The panel does not accept your address" (IP_NOT_ALLOWED)
Turn the check off
Fix the list
Turn the check back on
The panel's HTTPS and domain
The browser warns that the connection is not private
"Cannot read the panel's TLS certificate or key" after an update
Agents went Offline after vaultpanel https on or https renew
The panel stopped loading after turning on the firewall
Panel maintenance
"EACCES" or "read-only file system" in vaultpanel logs
vaultpanel backup says the MariaDB container is not running
vaultpanel restore fails at "the database update failed"
vaultpanel rollback says no earlier version is recorded
Servers and agents
A server stays "Waiting for agent"
"does not match the pin in the install command"
A server shows Offline
A server went Offline after setting outgoing connections to Deny
"The agent runs as root" in the security checks
HELPER_UNAVAILABLE
"Update agent" says the Node.js agent is replaced by running the command
AGENT_PACKAGE_MISSING
The panel's own server does not show up
Jobs
Websites and SSL
A website stays "Failed"
The site is created but "Not answering"
Certificate requests fail with DOMAIN_NOT_REACHABLE
Certificate requests fail with ACME_NOT_SERVED
File manager
Server self-check
Applications and deployments
An app stays "Failed"
A command field says it uses shell syntax
The app runs but its domain shows 502
SOFTWARE_MISSING or NODE_IN_HOME_FOLDER
Containers
Databases
Email
Mail from other providers never arrives, or is refused
The panel's own email (invites, resets, alerts) is not sent
Backups and cron jobs
Webhooks and alerts
A webhook delivery says "which webhooks may not reach"
An alert never fires
Firewall, SSH and Fail2ban
LOCKOUT_REFUSED
Locked out of SSH anyway
SSH errors
The AI assistant
Development checkouts
connect ECONNREFUSED 127.0.0.1:3307