Backups & Automation

Off-server backup storage

Copy backups to Amazon S3, Backblaze B2, Wasabi, Cloudflare R2, MinIO or another server over SFTP, so they survive the loss of a server.

Last updated

A backup on the same server does not help when the server is gone. Storage destinations copy each backup off the server: to an S3-compatible bucket, or to another server over SFTP. Owners and admins set them up once; a backup or a schedule then names one.

Storage needs agent 0.13.0 or newer on each server whose backups use it, and the settings.manage permission to set up.

How it works

  1. The agent writes the backup on the server as usual.
  2. It encrypts a copy with the storage's backup key (agent 0.42.0 or newer), copies it to the storage as <backup>.enc and confirms the copy.
  3. Unless the storage keeps local copies, it removes the backup from the server.

If the copy fails, the backup stays on the server, completed, and shows why the copy failed. Restoring or downloading a backup that is only in storage fetches it back first, decrypts it and checks its SHA-256. Deleting a backup deletes the stored copy too.

Credentials never stay on your servers: the panel stores them encrypted and sends them to the agent only inside the job that needs them, and they are removed from the job once it ends.

Encryption

Copies are encrypted before they leave the server, with AES-256-GCM in 1 MiB authenticated chunks: whoever can read the bucket or the backup server sees only random bytes, and a copy that was changed, cut short or added to is refused when it comes back. Each storage has its own 256-bit key, which the panel makes and keeps encrypted with its other secrets; agents get it only inside the job that needs it.

The backups list shows a lock next to copies that are encrypted. Copies made before agent 0.42.0, or by a server whose agent is older, are not encrypted and say so; they restore as before. Update the agent and back up again to replace them.

Save the backup key

If the panel server is lost, its copy of the key goes with it, and the copies in storage can no longer be read. Download each storage's key once, and keep it in a password manager or another safe place that is not the panel server:

  1. Open Backups, choose the Storage tab.
  2. Click Key on the storage. A text file vaultpanel-backup-key-<storage>.txt downloads.

The Storage tab reminds you until each key has been downloaded. Downloading it is recorded in the audit log, and works only while signed in to the panel, never with an API token. Anyone with the key and access to the storage can read the backups, so treat it like a password.

To decrypt a copy without the panel, download it from the bucket or backup server and use the CLI:

vaultpanel backups decrypt 7f3c….sql.gz.enc --key vaultpanel-backup-key-offsite.txt

It writes the backup next to it (7f3c….sql.gz) and prints its SHA-256, which matches the backup's checksum in the panel when the copy is intact.

Add S3-compatible storage

Open Backups, choose the Storage tab, click Add storage and keep Kind on S3-compatible bucket.

FieldWhat it does
ServiceAmazon S3, Backblaze B2, Wasabi, Cloudflare R2, or MinIO or another S3-compatible service. Fills in the endpoint and region.
NameHow the storage is listed in the panel.
RegionThe bucket's region, such as us-east-1.
EndpointThe service's address, without a bucket or folder in it. Filled in for Amazon S3, Backblaze B2 and Wasabi from the region; typed in for R2 and other services.
Bucket3 to 63 letters, digits, dots and hyphens. Create it with the service first.
Folder in the bucketBackups go under this folder. Default vaultpanel; empty for the top of the bucket.
Access key ID, Secret access keyA key for the bucket. The secret is stored encrypted and never shown again; leave it blank when editing to keep the saved one.
Also keep each backup on its serverRestores are faster, but backups also use disk on the server. Off: each backup is removed from the server once the storage confirms its copy.
Path-style addressesPuts the bucket in the path (endpoint/bucket/...) instead of the host name. MinIO and most self-hosted services need it.

Each service's defaults:

ServiceEndpointRegionPath-style
Amazon S3https://s3.<region>.amazonaws.comus-east-1Off
Backblaze B2https://s3.<region>.backblazeb2.comus-west-004 (the part of the bucket's S3 endpoint after s3.)Off
Wasabihttps://s3.<region>.wasabisys.comus-east-1Off
Cloudflare R2https://<account id>.r2.cloudflarestorage.comautoOn
MinIO or another serviceYour service's addressus-east-1 unless the service says otherwiseOn

Backups are stored as <folder>/<server id>/<type>/<name>/<backup id>.<ext>, where the type is site, app or database.

Add another server over SFTP

Backups can go to any Linux server you can reach over SSH. The panel makes its own Ed25519 key pair for the destination; nothing secret is typed in.

Make a user on the backup server

Create a user for backups on the other server, for example:

adduser --disabled-password --gecos "" backups

The panel refuses root. The backup server's firewall must let your servers in on its SSH port.

Add the storage

Click Add storage and choose Another server, over SFTP:

FieldWhat it does
NameHow the storage is listed in the panel.
HostThe backup server's host name or IP address.
UserA Linux user on that server, not root.
SSH portDefault 22.
FolderIn the user's home, or a full path such as /srv/backups. Default vaultpanel-backups; made when needed.
Also keep each backup on its serverAs for S3 storage.

Authorize the panel's key

Add the public key the panel shows to the user's ~/.ssh/authorized_keys on the backup server. Signed in as that user there:

mkdir -p ~/.ssh && echo 'ssh-ed25519 AAAA... ' >> ~/.ssh/authorized_keys

Copy the exact line from the panel instead of the example above.

Check it

Click Check and choose a server under From server. The first check trusts the backup server's host key and shows its fingerprint. An SFTP storage must pass this check once before backups can use it.

From then on, a different host key is refused, so nobody can pose as the backup server. If the backup server was reinstalled or its keys changed on purpose, edit the storage and tick the option to trust its new key; changing the host or port also forgets the old key. The next check then trusts the new one.

Check a storage

Check asks the agent of the server you choose to write a small file to the storage, read it back and delete it: the same path its backups take. Check every server that will back up to it.

Use a storage

Choose the storage under Store in when you click Back up now or set a schedule. See Backing up and restoring.

Hosting customers neither see nor choose storage: their backups stay on the server unless staff give them a schedule that uses one.

Remove a storage

Removing a storage makes the panel forget it, its access key and its backup key. Nothing in the bucket or on the backup server is deleted. A storage that still holds backups or is used by a schedule cannot be removed: delete those backups and change those schedules first.