Architecture

How VaultPanel's control plane, background worker and server agents fit together, from an API request through the job queue to a command on a server.

Last updated

VaultPanel follows a control plane and agent model. The panel (an Express API, a background worker, MariaDB and Redis) holds every record and decides what may happen. A small agent on each server carries out a fixed list of operations that the panel signs. Agents connect out to the panel, so servers need no inbound port.

Browser ── React UI (frontend/)
              │  HTTPS (self-signed on the panel port, or a panel domain),
              │  session cookie + CSRF token
              ▼
        Express API (backend/src/server.js, as the vaultpanel user) ─── MariaDB (users, servers, jobs, audit log…)
              │                              └── Redis (agent nonces, locks, BullMQ)
              │  HTTPS, Ed25519-signed requests      ▲
              ▼                                      │
        Agent on each server (agent/)        Worker (backend/src/worker.js)
              │  fixed commands, no shell           maintenance every 15 s
              ▼
        systemd, ps, /proc …

Components

ComponentPathRole
Web interfacefrontend/React, Vite and Tailwind. Pages in src/pages, a small design system in src/components/ui.
APIbackend/src/server.jsExpress. Routes, validation, authentication, services, the agent endpoints.
Workerbackend/src/worker.jsBullMQ. The maintenance pass and the AI assistant's answers.
Shared packageshared/Permissions, roles, the operation catalog and validation rules, used by the API and the interface.
Agentagent/Rust, one static program for x86-64 and arm64. Heartbeats, jobs, the terminal channel. Versioned on its own (agent/Cargo.toml).
SDK and CLIsdk/, cli/@vaultpanel/sdk and the vaultpanel command, built on the SDK.
Migrationsdatabase/migrations/Knex migrations.

The panel on its server

On a server installed with the installer, the API and the worker run as the unprivileged vaultpanel user in a systemd sandbox. Only backend/src/cli/connectLocal.js, which connects the panel's own server before the API starts, runs as root.

  • The API serves HTTPS on VAULTPANEL_PORT with the certificate in VAULTPANEL_TLS_CERT (the installer makes a self-signed one), and plain HTTP on 127.0.0.1:VAULTPANEL_LOOPBACK_PORT for callers on the same machine: the local agent, the panel domain's NGINX and the vaultpanel command.
  • Agent install commands carry the certificate's public key pin while the panel's address is served by it, so servers can be added safely without a domain.
  • The panel's files are in /var/lib/vaultpanel/panel; /var/lib/vaultpanel itself belongs to the agent.
  • With VAULTPANEL_UI_DIR set, the API also serves the built interface, so one port is the whole panel.

Backend layers

Requests flow through the same layers whoever makes them:

routes → middleware (request id, rate limits, auth, CSRF, panel access, tenancy, validation)
       → controllers → services → database
  • Controllers are thin: they read validated input, call one service and send the response.
  • Services (backend/src/services) hold all business rules, authorization and audit logging. They take an actor (a signed-in user, an API token, a plugin, a billing key, an agent or the system) instead of an HTTP request, so the same code serves the interface, the CLI, automation and the AI assistant.
  • Authorization happens in one place, services/authorization.js, which checks a permission and records denials in the audit log.
  • Tenancy: a hosting customer's actor carries its customer id, and every query is limited to that account's records. Anything else reads as not found.

Authentication order

  1. /api/v1/agent authenticates by agent signature, /api/v1/hooks by the token in the URL and /api/v1/billing by a vpk_ key.
  2. Everything else first resolves a Bearer token (vpt_ API token or vpp_ plugin token), which skips the cookie and CSRF checks, then the vp_session cookie with its CSRF token.
  3. Panel access by IP address applies, then /auth, then the two-factor policy, then the customer areas, then the resource routes.

Response envelope

Every response has the same shape, and the request id is also in the X-Request-Id header:

{ "success": true, "data": {}, "message": "OK", "requestId": "…" }
{ "success": false, "error": { "code": "VALIDATION_ERROR", "message": "…", "details": [{ "field": "email", "message": "…" }] }, "requestId": "…" }

Errors carry a stable code such as NOT_FOUND, QUOTA_REACHED or AGENT_OUTDATED. See API overview.

Rate limits

LimitWindowRequests
API (per address)1 minute600
Sign-in15 minutes20
First-run setup15 minutes10
Password reset15 minutes20
Agent registration1 hour30
Agent install script and package downloads1 minute60
Git push webhooks1 minute30
Billing API1 minute120

Servers, agents and jobs

Enrollment

An admin adds a server. The panel creates a one-time enrollment token (stored hashed, valid 60 minutes by default) and an install command.

Registration

The install script downloads the agent from the panel, and vaultpanel-agent register generates an Ed25519 key pair, sends the public key with the token, and receives its agent id and the panel's job-signing public key, which it pins.

Signed requests

Every later agent request is signed over the method, path, timestamp, nonce and body hash. The panel checks the signature, a 5-minute clock window and that the nonce was not used before.

Heartbeat and polling

The agent sends a heartbeat every 15 seconds (system information, metrics, service states, the operations it supports) and long-polls GET /api/v1/agent/jobs/next for work. A server silent for 90 seconds (VAULTPANEL_AGENT_OFFLINE_SECONDS) is marked offline.

Queuing an operation

A user, script or service queues an operation. The panel checks that it is in the catalog, the caller's permission, the parameters and that the agent supports it (AGENT_OUTDATED otherwise), and refuses a duplicate of an active job.

Running it

When the agent claims the job, the panel sends a signed envelope. The agent verifies the signature, the server id and the expiry, checks the parameters against its own allow-list and local policy, runs fixed commands without a shell, streams log lines and reports the result.

Job states

StateMeaning
queuedWaiting for the agent to pick it up.
dispatchedHanded to the agent, not yet started.
runningThe agent is running it and streaming logs.
succeeded, failed, timed_out, cancelledFinal.

Jobs are rows in the jobs table: the durable record people see and the queue agents pull from. Job results the panel only waits on, such as file listings and log reads, are routine jobs, hidden from job lists by default. Passwords, environment values and file contents in job parameters are replaced with { redacted: true, length } when the job ends.

Records changed by jobs

Changes to websites, apps and databases are jobs tied to their record (jobs.resource_type and resource_id). The record keeps its live settings and a pending action while the job runs. When the job finishes, the service that owns the record applies the result, or the error, before the agent's next poll is answered, so a follow-up job (create an app, then its website, then the first deploy) always sees the result of the one before.

Websites follow this model: the panel saves the request (a new site, or a change in pending_spec), queues one agent operation that does the whole change (user, folders, PHP-FPM, NGINX, health check), and applies the result to the website row when the job finishes.

The worker

The worker runs two BullMQ queues on Redis.

vaultpanel-maintenance runs a maintenance pass every 15 seconds:

  • marks silent servers offline, and runs website uptime checks;
  • sweeps jobs: fails jobs no agent picked up, retries jobs an agent claimed but never started, times out stuck jobs;
  • advances bulk actions and evaluates alert rules;
  • rolls raw metrics up into hourly rows and prunes old metrics, alert history and sessions;
  • takes backups whose schedule is due, then deletes the oldest beyond the schedule's keepCount;
  • asks a server to renew any SSL certificate Certbot has not renewed a day into its renewal window, and raises expiry events;
  • sends webhook deliveries and panel email, with retries;
  • deletes expired download copies, uploads and terminal recordings;
  • fails AI assistant answers stuck too long and expires old proposals;
  • keeps hosting customers' app ports in line with their plans, and refreshes plugin directories.

vaultpanel-assistant answers AI assistant questions. See AI Assistant.

Background flows

Backups and cron jobs

Backups and restores are agent jobs tied to their backups row. Cron jobs are installed on the server by a job but run by the server's own cron daemon, so they keep running while the panel is down; the agent reports each run with its next request.

Webhook events

Events come from three places: the backup and cron services, the audit log (alerts, and servers going offline or online) and finished deployment jobs. Each event becomes one webhook_deliveries row per webhook that wants it, so a delivery survives a restart.

Bulk actions

A bulk action is one row in bulk_actions and one row per server in bulk_action_targets; each server's part is an ordinary agent job linked back with jobs.resource_type = 'bulk_action'. Progress moves forward from the job finish handler, the agent heartbeat (a server back after a reboot, or with a new agent version) and the maintenance pass. Each step locks the action row, so a rolling action never starts two servers at once.

Alerts

Alerts work like Prometheus rules. When a rule's condition becomes true, an event is pending; if it stays true for the rule's duration, it becomes firing (audited as alert.fire); when the condition clears, it is resolved. A pending event whose condition clears is deleted. When a server has no fresh data, metric and service rules are left as they are rather than resolved.

AI assistant

Asking a question saves it and puts the conversation on the vaultpanel-assistant queue. The answer is a loop of at most 12 rounds: the provider gets a fixed system prompt and a fixed set of tools, and each tool call runs through the normal services with an actor for the person who asked. The propose_action tool only stores a row in assistant_actions; it runs when a person approves it, through the same service call as the matching button.

Frontend

React Router pages in frontend/src/pages, state in React contexts (stores/) and data loading through hooks/useApi.js. The design is defined by semantic tokens in theme/index.css with light and dark values; type is IBM Plex Sans with JetBrains Mono, bundled with the app (no external font requests). The header holds the Ctrl/Cmd+K command menu, the firing-alerts and running-jobs indicators, the theme switch and the account menu. Navigation groups live in layouts/navigation.js.

Security model

  • There is no endpoint that runs arbitrary commands, on the panel or the agent. The terminal is a separate, owner-only channel.
  • Every change is a permission check in a service, and every change is audited.
  • The agent re-validates everything the panel sends, and the server owner's local policy wins over anything the panel asks.

See Security for the full list.