# Architecture

URL: https://docs.vaultpanel.net/developers/architecture

> How VaultPanel's control plane, background worker and server agents fit together, from an API request through the job queue to a command on a server.

VaultPanel follows a **control plane and agent** model. The panel (an Express API, a background worker, MariaDB and Redis) holds every record and decides what may happen. A small agent on each server carries out a fixed list of operations that the panel signs. Agents connect out to the panel, so servers need no inbound port.

```text
Browser ── React UI (frontend/)
              │  HTTPS (self-signed on the panel port, or a panel domain),
              │  session cookie + CSRF token
              ▼
        Express API (backend/src/server.js, as the vaultpanel user) ─── MariaDB (users, servers, jobs, audit log…)
              │                              └── Redis (agent nonces, locks, BullMQ)
              │  HTTPS, Ed25519-signed requests      ▲
              ▼                                      │
        Agent on each server (agent/)        Worker (backend/src/worker.js)
              │  fixed commands, no shell           maintenance every 15 s
              ▼
        systemd, ps, /proc …
```

## Components

| Component | Path | Role |
| --- | --- | --- |
| Web interface | `frontend/` | React, Vite and Tailwind. Pages in `src/pages`, a small design system in `src/components/ui`. |
| API | `backend/src/server.js` | Express. Routes, validation, authentication, services, the agent endpoints. |
| Worker | `backend/src/worker.js` | BullMQ. The maintenance pass and the AI assistant's answers. |
| Shared package | `shared/` | Permissions, roles, the operation catalog and validation rules, used by the API and the interface. |
| Agent | `agent/` | Rust, one static program for x86-64 and arm64. Heartbeats, jobs, the terminal channel. Versioned on its own (`agent/Cargo.toml`). |
| SDK and CLI | `sdk/`, `cli/` | `@vaultpanel/sdk` and the `vaultpanel` command, built on the SDK. |
| Migrations | `database/migrations/` | Knex migrations. |

## The panel on its server

On a server installed with the [installer](https://docs.vaultpanel.net/getting-started/installation), the API and the worker run as the unprivileged `vaultpanel` user in a systemd sandbox. Only `backend/src/cli/connectLocal.js`, which connects the panel's own server before the API starts, runs as root.

- The API serves HTTPS on `VAULTPANEL_PORT` with the certificate in `VAULTPANEL_TLS_CERT` (the installer makes a self-signed one), and plain HTTP on `127.0.0.1:VAULTPANEL_LOOPBACK_PORT` for callers on the same machine: the local agent, the panel domain's NGINX and the `vaultpanel` command.
- Agent install commands carry the certificate's public key pin while the panel's address is served by it, so servers can be added safely without a domain.
- The panel's files are in `/var/lib/vaultpanel/panel`; `/var/lib/vaultpanel` itself belongs to the agent.
- With `VAULTPANEL_UI_DIR` set, the API also serves the built interface, so one port is the whole panel.

## Backend layers

Requests flow through the same layers whoever makes them:

```text
routes → middleware (request id, rate limits, auth, CSRF, panel access, tenancy, validation)
       → controllers → services → database
```

- **Controllers** are thin: they read validated input, call one service and send the response.
- **Services** (`backend/src/services`) hold all business rules, authorization and audit logging. They take an **actor** (a signed-in user, an API token, a plugin, a billing key, an agent or the system) instead of an HTTP request, so the same code serves the interface, the CLI, automation and the AI assistant.
- **Authorization** happens in one place, `services/authorization.js`, which checks a permission and records denials in the audit log.
- **Tenancy**: a hosting customer's actor carries its customer id, and every query is limited to that account's records. Anything else reads as not found.

### Authentication order

1. `/api/v1/agent` authenticates by agent signature, `/api/v1/hooks` by the token in the URL and `/api/v1/billing` by a `vpk_` key.
2. Everything else first resolves a Bearer token (`vpt_` API token or `vpp_` plugin token), which skips the cookie and CSRF checks, then the `vp_session` cookie with its CSRF token.
3. Panel access by IP address applies, then `/auth`, then the two-factor policy, then the customer areas, then the resource routes.

### Response envelope

Every response has the same shape, and the request id is also in the `X-Request-Id` header:

```json
{ "success": true, "data": {}, "message": "OK", "requestId": "…" }
```

```json
{ "success": false, "error": { "code": "VALIDATION_ERROR", "message": "…", "details": [{ "field": "email", "message": "…" }] }, "requestId": "…" }
```

Errors carry a stable `code` such as `NOT_FOUND`, `QUOTA_REACHED` or `AGENT_OUTDATED`. See [API overview](https://docs.vaultpanel.net/api-reference).

### Rate limits

| Limit | Window | Requests |
| --- | --- | --- |
| API (per address) | 1 minute | 600 |
| Sign-in | 15 minutes | 20 |
| First-run setup | 15 minutes | 10 |
| Password reset | 15 minutes | 20 |
| Agent registration | 1 hour | 30 |
| Agent install script and package downloads | 1 minute | 60 |
| Git push webhooks | 1 minute | 30 |
| Billing API | 1 minute | 120 |

## Servers, agents and jobs

### Enrollment

An admin adds a server. The panel creates a one-time enrollment token (stored hashed, valid 60 minutes by default) and an install command.

### Registration

The install script downloads the agent from the panel, and `vaultpanel-agent register` generates an Ed25519 key pair, sends the public key with the token, and receives its agent id and the panel's job-signing public key, which it pins.

### Signed requests

Every later agent request is signed over the method, path, timestamp, nonce and body hash. The panel checks the signature, a 5-minute clock window and that the nonce was not used before.

### Heartbeat and polling

The agent sends a heartbeat every 15 seconds (system information, metrics, service states, the operations it supports) and long-polls `GET /api/v1/agent/jobs/next` for work. A server silent for 90 seconds (`VAULTPANEL_AGENT_OFFLINE_SECONDS`) is marked offline.

### Queuing an operation

A user, script or service queues an operation. The panel checks that it is in the catalog, the caller's permission, the parameters and that the agent supports it (`AGENT_OUTDATED` otherwise), and refuses a duplicate of an active job.

### Running it

When the agent claims the job, the panel sends a signed envelope. The agent verifies the signature, the server id and the expiry, checks the parameters against its own allow-list and local policy, runs fixed commands without a shell, streams log lines and reports the result.

### Job states

| State | Meaning |
| --- | --- |
| `queued` | Waiting for the agent to pick it up. |
| `dispatched` | Handed to the agent, not yet started. |
| `running` | The agent is running it and streaming logs. |
| `succeeded`, `failed`, `timed_out`, `cancelled` | Final. |

Jobs are rows in the `jobs` table: the durable record people see and the queue agents pull from. Job results the panel only waits on, such as file listings and log reads, are **routine** jobs, hidden from job lists by default. Passwords, environment values and file contents in job parameters are replaced with `{ redacted: true, length }` when the job ends.

### Records changed by jobs

Changes to websites, apps and databases are jobs tied to their record (`jobs.resource_type` and `resource_id`). The record keeps its live settings and a pending action while the job runs. When the job finishes, the service that owns the record applies the result, or the error, before the agent's next poll is answered, so a follow-up job (create an app, then its website, then the first deploy) always sees the result of the one before.

Websites follow this model: the panel saves the request (a new site, or a change in `pending_spec`), queues one agent operation that does the whole change (user, folders, PHP-FPM, NGINX, health check), and applies the result to the website row when the job finishes.

## The worker

The worker runs two BullMQ queues on Redis.

**`vaultpanel-maintenance`** runs a maintenance pass every 15 seconds:

- marks silent servers offline, and runs website uptime checks;
- sweeps jobs: fails jobs no agent picked up, retries jobs an agent claimed but never started, times out stuck jobs;
- advances bulk actions and evaluates alert rules;
- rolls raw metrics up into hourly rows and prunes old metrics, alert history and sessions;
- takes backups whose schedule is due, then deletes the oldest beyond the schedule's `keepCount`;
- asks a server to renew any SSL certificate Certbot has not renewed a day into its renewal window, and raises expiry events;
- sends webhook deliveries and panel email, with retries;
- deletes expired download copies, uploads and terminal recordings;
- fails AI assistant answers stuck too long and expires old proposals;
- keeps hosting customers' app ports in line with their plans, and refreshes plugin directories.

**`vaultpanel-assistant`** answers AI assistant questions. See [AI Assistant](https://docs.vaultpanel.net/guides/ai-assistant).

> **Important:**
The worker must run next to the API. Without it, schedules, alerts, webhooks, timeouts and the assistant stop.

## Background flows

### Backups and cron jobs

Backups and restores are agent jobs tied to their `backups` row. Cron jobs are installed on the server by a job but run by the server's own cron daemon, so they keep running while the panel is down; the agent reports each run with its next request.

### Webhook events

Events come from three places: the backup and cron services, the audit log (alerts, and servers going offline or online) and finished deployment jobs. Each event becomes one `webhook_deliveries` row per webhook that wants it, so a delivery survives a restart.

### Bulk actions

A bulk action is one row in `bulk_actions` and one row per server in `bulk_action_targets`; each server's part is an ordinary agent job linked back with `jobs.resource_type = 'bulk_action'`. Progress moves forward from the job finish handler, the agent heartbeat (a server back after a reboot, or with a new agent version) and the maintenance pass. Each step locks the action row, so a rolling action never starts two servers at once.

### Alerts

Alerts work like Prometheus rules. When a rule's condition becomes true, an event is `pending`; if it stays true for the rule's duration, it becomes `firing` (audited as `alert.fire`); when the condition clears, it is `resolved`. A pending event whose condition clears is deleted. When a server has no fresh data, metric and service rules are left as they are rather than resolved.

### AI assistant

Asking a question saves it and puts the conversation on the `vaultpanel-assistant` queue. The answer is a loop of at most 12 rounds: the provider gets a fixed system prompt and a fixed set of tools, and each tool call runs through the normal services with an actor for the person who asked. The `propose_action` tool only stores a row in `assistant_actions`; it runs when a person approves it, through the same service call as the matching button.

## Frontend

React Router pages in `frontend/src/pages`, state in React contexts (`stores/`) and data loading through `hooks/useApi.js`. The design is defined by semantic tokens in `theme/index.css` with light and dark values; type is IBM Plex Sans with JetBrains Mono, bundled with the app (no external font requests). The header holds the Ctrl/Cmd+K command menu, the firing-alerts and running-jobs indicators, the theme switch and the account menu. Navigation groups live in `layouts/navigation.js`.

## Security model

- There is no endpoint that runs arbitrary commands, on the panel or the agent. The [terminal](https://docs.vaultpanel.net/guides/servers/terminal) is a separate, owner-only channel.
- Every change is a permission check in a service, and every change is audited.
- The agent re-validates everything the panel sends, and the server owner's [local policy](https://docs.vaultpanel.net/developers/agent#local-policy) wins over anything the panel asks.

See [Security](https://docs.vaultpanel.net/guides/security) for the full list.
