# Troubleshooting

URL: https://docs.vaultpanel.net/troubleshooting

> Diagnose and fix common VaultPanel problems by symptom, from installation and agents to websites, SSL, apps, databases, backups and the assistant.

Problems are grouped by where they show up. Most failures in the panel carry an error code (such as `DOMAIN_NOT_REACHABLE`) and a request id; search this page for the code.

## Start here

Three commands on the panel's server answer most questions:

```bash
vaultpanel status                 # panel, worker, agent, database and Redis
sudo vaultpanel logs panel -f     # the API's log; also: worker, agent
sudo journalctl -u vaultpanel-agent -n 100 --no-pager
```

On any other server, the agent's state is in its own services:

```bash
systemctl status vaultpanel-agent vaultpanel-agent-helper
journalctl -u vaultpanel-agent -u vaultpanel-agent-helper -n 100 --no-pager
```

> **Tip:**
Every error in the panel shows a request id. Search the API log for it (`sudo vaultpanel logs panel`) to find the full error. Errors returned to the browser never include stack traces.

## Installation

The installer stops with a plain message when something is wrong. The common ones:

| Message | Fix |
| --- | --- |
| `this installer supports Ubuntu 22.04/24.04/26.04 and Debian 12/13` | Use a supported system. See [System requirements](https://docs.vaultpanel.net/getting-started/system-requirements). |
| `systemd is required (containers without systemd are not supported)` | Install on a VM or a server, not in a container without systemd. |
| `port 8090 is already in use` | Stop what uses it, or install on another port: `--port 8095`. |
| `port 8091 (the panel's loopback port) is already in use` | Set `VAULTPANEL_LOOPBACK_PORT=8095` in `/opt/vaultpanel/.env`, then `sudo vaultpanel update --force`. |
| `port 3307` or `port 6379` `… is already in use on 127.0.0.1` | Something else (often a panel started by hand from a copy of the repository) holds the panel's database or Redis port. Stop its containers first. |
| `port 3306 is already in use, so MariaDB cannot start there` | Stop what uses it, or install with `DB_ENGINE=NONE`. |
| `MySQL is not available yet` | MySQL 8.4 and 8.0 come in a later version. Choose a MariaDB version, or `NONE`. |
| `… is not available on <system>` | The `DB_ENGINE` you chose is not offered for this system. The message lists the ones that are. |
| `Docker is installed but not running` | `systemctl start docker`, then run the installer again. |
| `the vaultpanel package did not install` | Fix what the lines above it say, then `sudo dpkg --configure -a`. |
| `the panel did not start` | The log above the message says why; more with `vaultpanel logs`. |

Running the installer again is safe: it keeps `.env` and all data.

## Setup and signing in

### The setup token is refused

The setup token creates the owner account and stops working once it exists. On an installed panel, `sudo vaultpanel setup-token` prints the current one. A panel started by hand prints it in the API log ("No accounts exist yet…"); it changes on every restart unless `VAULTPANEL_SETUP_TOKEN` is set.

### The API exits with "Invalid configuration"

A required setting is missing or malformed, and the message lists which. `VAULTPANEL_SECRET_KEY` must decode to exactly 32 bytes; generate one with `openssl rand -hex 32`.

### "Set up two-factor sign-in" on every page

An owner requires two-factor sign-in for your role (**Security > Authentication**). Set it up, or ask an owner to change the policy. If you lost your phone and your recovery codes, an owner or admin can turn two-factor sign-in off for you under **Users** (Edit) or on **Security > Authentication**; you are then signed out everywhere.

### "The panel does not accept your address" (`IP_NOT_ALLOWED`)

**Panel access by address** is on and your address is not on the list. From an allowed address, add yours under **Security > Authentication**.

If you are locked out completely, on the panel's server:

### Turn the check off

Add `VAULTPANEL_PANEL_ACCESS=off` to `/opt/vaultpanel/.env` and run `sudo vaultpanel restart`.

### Fix the list

Sign in and correct the addresses under **Security > Authentication**.

### Turn the check back on

Remove the line from `.env` and run `sudo vaultpanel restart` again.

Behind a reverse proxy or Cloudflare, everyone appears with the proxy's address unless `VAULTPANEL_TRUST_PROXY` is set.

## The panel's HTTPS and domain

### The browser warns that the connection is not private

The panel's port uses a self-signed certificate made for your server. Check that the SHA-256 fingerprint the browser shows matches the one `sudo vaultpanel https` prints, then continue. For a certificate browsers trust, give the panel a domain. See [Panel HTTPS and domain](https://docs.vaultpanel.net/getting-started/panel-https-and-domain).

### "Cannot read the panel's TLS certificate or key" after an update

The files named by `VAULTPANEL_TLS_CERT` and `VAULTPANEL_TLS_KEY` are missing or not readable by the `vaultpanel` group. `sudo vaultpanel https on` makes a new certificate and fixes the permissions. It prints the command to run on each other server, so their agents trust the new certificate.

### Agents went Offline after `vaultpanel https on` or `https renew`

They still talk plain HTTP, or trust the old certificate. Run the command that `vaultpanel https on` (or `renew`) printed on each of them; `sudo vaultpanel https` shows the pin again. Their registration is kept.

### The panel stopped loading after turning on the firewall

VaultPanel opens the port in `VAULTPANEL_PUBLIC_URL` on the panel's own server. If you open the panel on another port, add a firewall rule for it, or correct the address with `sudo vaultpanel url <address>`.

## Panel maintenance

### "EACCES" or "read-only file system" in `vaultpanel logs`

The panel runs as the `vaultpanel` user and may only write to `VAULTPANEL_DATA_DIR` (`/var/lib/vaultpanel/panel`). `sudo vaultpanel update --force` puts the ownership right. A `VAULTPANEL_DATA_DIR` set by hand elsewhere must belong to `vaultpanel`.

### `vaultpanel backup` says the MariaDB container is not running

Start it with `sudo vaultpanel start`. `vaultpanel backup` needs the panel's own database from the installer; a panel on your own database server is backed up with your own database backups, plus `.env` and `/etc/vaultpanel/tls`.

### `vaultpanel restore` fails at "the database update failed"

The backup comes from a newer VaultPanel than the one installed. Install that version first with `sudo vaultpanel update --version vX.Y.Z` (the backup's version is printed when the restore starts), then restore again.

### `vaultpanel rollback` says no earlier version is recorded

Rollback reverses one `vaultpanel update`. To go to any release, run `sudo vaultpanel update --version vX.Y.Z`, then restore a backup from that time if the database changed in between. See [Updating](https://docs.vaultpanel.net/getting-started/updating).

## Servers and agents

### A server stays "Waiting for agent"

Check the output of the install command on the server. Enrollment tokens work once and expire after 60 minutes by default (`VAULTPANEL_ENROLLMENT_TTL_MINUTES`); create a new one from the server's **Settings** tab. "Agents must connect over HTTPS" means the panel's address is plain HTTP.

### "does not match the pin in the install command"

The panel answering at that address does not have the certificate the command was made for: it was replaced, or something between the servers intercepts the connection. Copy a new install command from the panel and check the address.

### A server shows Offline

The agent has not reported for 90 seconds (`VAULTPANEL_AGENT_OFFLINE_SECONDS`). On the server, run `systemctl status vaultpanel-agent` and `journalctl -u vaultpanel-agent`.

| Error | Cause | Fix |
| --- | --- | --- |
| `CLOCK_SKEW` | The server's clock is off by more than 5 minutes. | Turn on NTP: `sudo timedatectl set-ntp true`. |
| `AGENT_REVOKED` | The server was removed or registered again. | Run the install command again with a new token. |

### A server went Offline after setting outgoing connections to Deny

The agent keeps the panel's address reachable, but resolves it once, when the outgoing default changes. If the panel's IP address changed since, add an outgoing allow rule for the new address from the server's console (`ufw allow out to <panel IP> port <port> proto tcp`), or run `ufw default allow outgoing`. Programs that reach other services, such as a remote database, need their own outgoing rules.

### "The agent runs as root" in the security checks

The reason follows the message:

- **"did not reach the panel as the vaultpanel-agent user"**: the move to its own user was undone after 10 minutes. `journalctl -u vaultpanel-agent -u vaultpanel-agent-helper` shows why, often a firewall or proxy setting only root had. Fix it, then `sudo rm /var/lib/vaultpanel/agent-mode.json && sudo systemctl restart vaultpanel-agent`.
- **"runs from …"**: the agent was not installed by the install script (a development checkout) and stays as it was started.

### `HELPER_UNAVAILABLE` [#helper-unavailable]

The agent runs as its own user but its root helper does not answer. Check `systemctl status vaultpanel-agent-helper` and `journalctl -u vaultpanel-agent-helper`; `sudo systemctl restart vaultpanel-agent-helper` starts it again. To go back to one root service, add `"runAsRoot": true` to `/etc/vaultpanel-agent/config.json` and restart the helper.

### "Update agent" says the Node.js agent is replaced by running the command

The server still runs the old Node.js agent (a version without `-rs`, such as 0.42.0). It cannot install the current agent from an update job, so **Update agent** is refused and bulk updates skip the server. Run the command shown on the server's page once, on the server, as root:

```bash
curl -fsSL https://panel.example.com/api/v1/agent/install.sh   | sudo bash -s -- --upgrade --panel https://panel.example.com
```

A panel with a self-signed certificate adds the pin options; copy the command from the page. It installs `/usr/local/bin/vaultpanel-agent`, keeps the registration and policy, moves cron jobs and Certbot's wildcard hook to the new program, and removes `/opt/vaultpanel-agent` once the new agent runs. If the agent does not start, the old folder is kept: check `journalctl -u vaultpanel-agent`. See [Moving from the Node.js agent](https://docs.vaultpanel.net/guides/servers/connecting-servers#moving-from-the-nodejs-agent).

### `AGENT_PACKAGE_MISSING`

The panel has no agent program for the server's CPU type, so it cannot install or update the agent there. The agent runs on x86-64 and arm64; a panel installed from the package carries both.

### The panel's own server does not show up

Look for lines starting with `[vaultpanel]` in the API's log when it starts; they say why. `VAULTPANEL_LOCAL_AGENT=false` turns the automatic connection off. See [Connecting servers](https://docs.vaultpanel.net/guides/servers/connecting-servers).

## Jobs

| Error | Meaning | Fix |
| --- | --- | --- |
| `AGENT_UNREACHABLE` | No agent picked the job up within 5 minutes. | Check the server is online, then try again. |
| `AGENT_OUTDATED` | The server's agent predates the operation. | Click **Update agent** on the server's page, or run the upgrade command shown there. Registration is kept. |
| `NOT_ALLOWED` | The agent's local policy refuses it (`allowedServices`, `allowReboot`). | Edit `/etc/vaultpanel-agent/config.json` and restart the agent. |
| `TIMEOUT` | The command did not finish in time. | The job page shows the output collected so far. |
| `PROCESS_CHANGED` | The process exited and its pid was reused before the job ran. | Refresh the process list and try again. |

For **This server**, restarting the panel (`sudo vaultpanel restart`) also updates its agent: the panel connects its own server before the API starts.

## Websites and SSL

### A website stays "Failed"

Its page shows the agent's message and a link to the log. Fix the cause, then click **Try again**.

| Error | Fix |
| --- | --- |
| `NGINX_NOT_INSTALLED`, `PHP_NOT_INSTALLED` | Install them from the server's **Websites** tab, **Web stack** card. |
| `NGINX_CONFIG_INVALID` | NGINX refused the new configuration, often because another site or a hand-edited file is broken. Run `sudo nginx -t` on the server. |
| `SITE_USER_CONFLICT` | A Linux user with the site's name already exists for something else. Pick another site name. |
| `WEBSITE_BUSY` | Another change to the same website is still running. Wait for it and try again. |

### The site is created but "Not answering"

For a reverse proxy, start your app on the port shown: NGINX answers 502 until something listens there. For PHP, check the PHP-FPM service on the server's **Services** tab.

### Certificate requests fail with `DOMAIN_NOT_REACHABLE`

Let's Encrypt was not asked: the agent could not fetch a test file through the domain. Check on the **Domains** tab that each domain points at this server, and that ports 80 and 443 are open (`ufw allow 80,443/tcp`). Use the staging option while testing, to stay clear of Let's Encrypt's rate limit of 5 failures per hour per domain.

### Certificate requests fail with `ACME_NOT_SERVED`

The server's own NGINX could not serve the Let's Encrypt check file, often with HTTP 403. Run `sudo nginx -t` and read `/var/log/nginx/error.log`. On agents before 0.12.1, `/var/lib/vaultpanel` could be left closed; update the agent and issue again, or run `sudo chmod 711 /var/lib/vaultpanel`.

See [SSL certificates](https://docs.vaultpanel.net/guides/websites/ssl-certificates).

## File manager

| Symptom | Fix |
| --- | --- |
| `OUTSIDE_SITE` | The path goes through a symbolic link to somewhere outside the site's folder. The file manager never follows such links; use SFTP as the server's owner instead. |
| "Cannot find module '/opt/vaultpanel-agent/src/services/fileWorker.js'" | Old Node.js agents only, before 0.37.0: the file worker runs as the site's user, which cannot read the agent's folder. [Move the server to the current agent](https://docs.vaultpanel.net/guides/servers/connecting-servers#moving-from-the-nodejs-agent), or run `sudo chmod -R u=rwX,go=rX /opt/vaultpanel-agent`. |
| "Could not start the file worker: spawn … EACCES" | Agents before 0.9.1 could not run file requests when Node.js lives in a home folder (nvm). Update the agent, then **Try again** on the website. |

## Server self-check

The **Self-check** card on a server's **Overview** (agent 0.39.0) shows the command that fixes each failed check, and the **Server self-check failed** alert fires when a check stays failed for 10 minutes. See [Server self-check](https://docs.vaultpanel.net/guides/servers/self-check).

| Check | Fix |
| --- | --- |
| **Root helper** | See [`HELPER_UNAVAILABLE`](https://docs.vaultpanel.net/troubleshooting#helper-unavailable). |
| **Agent privileges** (warning) | See ["The agent runs as root"](https://docs.vaultpanel.net/troubleshooting#the-agent-runs-as-root-in-the-security-checks). |
| **Agent files** | The command on the card: `sudo chmod -R u=rwX,go=rX /usr/local/bin/vaultpanel-agent` for the program, or `sudo chmod o+x` on the folder the check names. |
| **File manager** | Its worker did not start as a site user. After fixing the agent files, click **Check now**. |
| **NGINX configuration**, **Apache configuration** | `sudo nginx -t` or `sudo apache2ctl -t` shows the file and line. Until it passes, every website change is refused. A file VaultPanel wrote (`vp-<site>.conf`) is rewritten by saving the website's settings again. |
| **Certificate renewal** (warning) | `sudo systemctl enable --now certbot.timer`, or issue a certificate from the panel. The panel renews due certificates itself either way. |
| **Clock** (warning) | `sudo timedatectl set-ntp true` |
| **Failed services** (warning) | `systemctl --failed`, then `journalctl -u <unit>` for each. |

## Applications and deployments

### An app stays "Failed"

The **Deployments** tab shows which stage failed and its log. The previous release keeps running.

| Stage or error | Fix |
| --- | --- |
| `fetch` (`CLONE_FAILED`) | The repository URL or branch is wrong, or a private repository needs the app's deploy key (on its **Settings** tab) added to the repository. |
| `install` or `build` | Run the same command in the repository on your computer to see the error. |
| `start` (`HEALTH_CHECK_FAILED`) | The app did not listen on its port in time. It must listen on `process.env.PORT` (Node.js) or `$PORT` (Python). |
| `APPLICATION_BUSY` from a webhook | A deployment is already running; the next push deploys. |

### A command field says it uses shell syntax

Install, build and start commands run without a shell, so `&&`, `|`, `;` and `>` are refused. Put the steps on separate lines, or in a script in your repository. `CONFIGURE_FAILED` saying a program is not installed means the start command's program (for example `gunicorn`) is not in the app's dependencies.

### The app runs but its domain shows 502

The linked website points NGINX at the app's port. If you changed the port, use **Apply** on the app so the app and NGINX move together.

### `SOFTWARE_MISSING` or `NODE_IN_HOME_FOLDER`

`SOFTWARE_MISSING` means Git, PM2, Python or MariaDB is not installed; install it from the server's **Software** card. `NODE_IN_HOME_FOLDER` means the only Node.js on the server lives under `/root` or `/home` (for example through nvm), which app users cannot read. Install Node.js system-wide:

```bash
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - && sudo apt-get install -y nodejs
```

See [Deployments](https://docs.vaultpanel.net/guides/applications/deployments).

## Containers

| Symptom | Fix |
| --- | --- |
| "Update the agent to manage containers" | Containers need agent 0.10.0 or newer. |
| Docker shows "Stopped", or `DOCKER_NOT_RUNNING` | Click **Start Docker** on the **Containers** page. By hand: `snap start --enable docker` for the snap (`which docker` shows `/snap/bin/docker`), or `systemctl enable --now docker` for Ubuntu's package. |
| "Could not pull … toomanyrequests" | Docker Hub limits anonymous pulls. Wait, or use the same image from a mirror such as `mirror.gcr.io/library/nginx:alpine`. |
| `CONTAINER_EXITED` or "did not stay up" | The container stopped during its health check; the error shows its last output. Usual causes: a missing environment variable, a wrong command, or too little memory. A container it replaced was started again. |
| A port "is used by …" | Another app, container, SSH, the panel or VaultPanel's database uses that host port. Pick another host port. |
| A public port is reachable without a firewall rule | Docker adds its own rules for published ports, ahead of UFW. Keep ports on **This server only** and add a reverse-proxy website for them. |

See [Containers](https://docs.vaultpanel.net/guides/applications/containers).

## Databases

| Error | Fix |
| --- | --- |
| `DATABASE_SERVER_NOT_INSTALLED` | Install MariaDB from the server's **Software** card. |
| `DATABASE_SERVER_DOWN` | Start the `mariadb` service on the server's **Services** tab. If root cannot sign in over the socket, set `mysqlDefaultsFile` in the agent's config. |
| `DATABASE_PORT_IN_USE` | Something else holds port 3306, usually the panel's own development MariaDB from an older `docker/compose.yml`. Move it to 3307, then try again. |

See [Databases](https://docs.vaultpanel.net/guides/databases).

## Email

### Mail from other providers never arrives, or is refused

- **Port 25 blocked.** Some providers block port 25 on new servers. Ask yours whether incoming and outgoing port 25 is open; `ss -ltn | grep ':25 '` shows whether Postfix listens.
- **Reverse DNS.** **Check DNS** on the **Email** page shows where the server's IP address points back to. Set its reverse DNS (PTR) to the mail host name with your provider, or Gmail and others may refuse your mail.
- **DNS records.** On the domain's **DNS records** tab, MX, SPF, DKIM and DMARC should show **In place**. When the domain's DNS is elsewhere, copy the records there.

### The panel's own email (invites, resets, alerts) is not sent

On **System > Panel Email**, **Send test email** shows a plain reason for a failure: a wrong password, a port that does not answer, or the wrong security option for the port (SSL/TLS goes with 465, STARTTLS with 587). Many providers block outgoing port 25, so use 587 or 465. Alert emails are sent by the worker, so it must be running (`vaultpanel status`).

See [Email hosting](https://docs.vaultpanel.net/guides/dns-and-email/email-hosting) and [Panel email](https://docs.vaultpanel.net/guides/dns-and-email/panel-email).

## Backups and cron jobs

| Error or symptom | Fix |
| --- | --- |
| `BACKUP_DISK_FULL` | Less than 512 MB is free where backups are kept (`/var/backups/vaultpanel`). Delete old backups, lower a schedule's keep count, or change `backupMinFreeBytes` in the agent's config. |
| `BACKUP_CORRUPT` | The file no longer matches the checksum taken when it was made, so it is not restored. Use another backup. |
| A backup shows **Missing** | It was removed from the server by hand. |
| `BACKUP_TOO_LARGE` on a download | The backup is bigger than `VAULTPANEL_BACKUP_DOWNLOAD_MAX_MB` (10 GB by default). Copy it from the server with `scp`. |
| A restored app does not start (`HEALTH_CHECK_FAILED`) | The files are back but the health check failed. Restore the safety copy taken just before, or deploy again. |
| "Cron is not installed" | Install Cron from the **Scheduler** card on the server's **Automation** tab, then save the job again. |
| A cron job never runs | Check its next run on the **Cron Jobs** page: schedules use the server's time zone, shown in the form. |

See [Backups](https://docs.vaultpanel.net/guides/backups-and-automation/backups) and [Cron jobs](https://docs.vaultpanel.net/guides/backups-and-automation/cron-jobs).

## Webhooks and alerts

### A webhook delivery says "which webhooks may not reach"

Webhooks never call the panel's own machine, link-local or cloud metadata addresses. Private networks (10.x, 172.16-31.x, 192.168.x) are refused too unless `VAULTPANEL_WEBHOOK_ALLOW_PRIVATE=true`. New installs set it to `false`; panels installed before that default changed keep `true`.

### An alert never fires

A metric must stay over the threshold for the rule's whole duration, and the server must be online with fresh data; while it is offline only the "server offline" rule is evaluated. The worker checks rules every 15 seconds, so it must be running (`vaultpanel status`).

## Firewall, SSH and Fail2ban

### `LOCKOUT_REFUSED`

VaultPanel is protecting your way in: a firewall rule that would block SSH, the panel or your own address, or an SSH change that would leave no administrator able to sign in. Add the replacement first (another rule for the port, or your SSH key), then try again.

### Locked out of SSH anyway

For example by a rule added with `ufw` by hand. Use your provider's web console, then run `ufw allow 22/tcp` or `ufw disable`. For SSH settings, remove `/etc/ssh/sshd_config.d/00-vaultpanel.conf` and run `systemctl reload ssh`.

### SSH errors

| Error | Fix |
| --- | --- |
| Cannot connect after changing the port | Use `ssh -p <port>`. A firewall at your hosting provider may need the port opened too. Keep the old port open until you have signed in on the new one. |
| `PORT_IN_USE` | Another program listens on that port. Pick another. |
| `SSH_NOT_LISTENING` | SSH did not answer on the new port, so the old setting came back. `journalctl -u ssh` says why (SELinux or AppArmor rules can refuse other ports). |
| `SSHD_OVERRIDDEN` | A line elsewhere in `/etc/ssh/sshd_config` or its `sshd_config.d` folder sets the same option and wins. Remove or comment it out and save again. VaultPanel's file is `00-vaultpanel.conf`. |
| `SSHD_CONFIG_INVALID` | Shows what `sshd -t` said. The previous settings stay. |
| `KEYS_NOT_WRITABLE` | The agent's service cannot write `/root/.ssh`. Update the agent (0.6.0 and newer). |
| `JAIL_FAILED` | Fail2ban's SSH jail did not start, usually because `python3-systemd` is missing: `apt install python3-systemd`, then save again. `/var/log/fail2ban.log` has the details. |

See [Firewall](https://docs.vaultpanel.net/guides/security/firewall) and [SSH](https://docs.vaultpanel.net/guides/security/ssh).

## The AI assistant

| Symptom | Fix |
| --- | --- |
| It keeps "thinking" and never answers | Answers are written by the worker, so it must be running (`vaultpanel status`). After 10 minutes the question is marked failed and **Try again** appears. |
| The provider rejected the key, or the model was not found | Open **System > Assistant Settings** and use **Test connection**. For Ollama and similar services, choose the OpenAI-compatible provider with an address ending in `/v1` (for example `http://127.0.0.1:11434/v1`) and a model that supports tool calling. |
| A server's agent is too old to list ports | Update that server's agent (0.9.0 adds listening ports). |

See [AI assistant](https://docs.vaultpanel.net/guides/ai-assistant).

## Development checkouts

### `connect ECONNREFUSED 127.0.0.1:3307`

From `npm run db:migrate` or the backend: the panel's development MariaDB is not running. Bring the containers up with `docker compose -f docker/compose.yml up -d`, wait until `docker compose -f docker/compose.yml ps` shows MariaDB healthy, then migrate and restart the backend. The containers restart after a reboot as long as Docker starts at boot (`systemctl enable docker`).

See [Development](https://docs.vaultpanel.net/developers/development).
