Servers

Server groups and bulk actions

Group servers such as Production or Europe, watch their health together, and run one operation on many servers at once or one at a time.

Last updated

Server groups are named sets of servers you filter by, watch together and act on. Bulk actions run one allow-listed operation on several servers, either all at once or one server at a time, with each server's part tracked as an ordinary job.

Server groups

Infrastructure > Server Groups lists your groups. Each card shows how many of its servers are online, firing alerts, and average CPU, memory and disk.

To create one, click New group and set:

FieldRules
NameUp to 60 characters: letters, numbers, spaces and . _ - ( ) &
DescriptionOptional, up to 300 characters
ColorBlue, green, amber, red, purple, teal or gray
ServersUp to 500 per group

A server can be in several groups. You can also set a server's groups from its Settings tab. Creating, changing and deleting groups needs servers.manage; deleting a group never touches its servers or the bulk actions run on it.

Open a group to see its servers side by side with Online, Firing alerts, CPU average, Memory average, Disk average and Agents to update, and its recent bulk actions.

Bulk actions

Start a bulk action in either of two ways:

  • On Infrastructure > Servers, tick two or more servers and click Run on selected (up to 200 servers).
  • Open a group and click Run on group.

Running a bulk action needs servers.bulk (owners and admins) in addition to the permission of the operation itself.

Operations

Only these operations can be run as a bulk action:

OperationWhat it doesDefault modeChecked after the job
Restart a serviceRestarts the same systemd service on every serverAll at once
Start a serviceStarts a stopped systemd service on every serverAll at once
Check a serviceReads the state of a systemd service on every serverAll at once
Install security updatesInstalls waiting security updates with unattended-upgradesOne server at a time
Update the agentInstalls the agent version this panel serves; servers already on it are skippedAll at onceThe server reports the new version, within 3 minutes
RebootRestarts the serversOne server at a timeThe server comes back with a newer boot time, within 10 minutes

The service operations take the systemd service name, the same on every server. Each agent still only manages the services its policy allows.

Modes

ModeBehavior
All at onceEvery server starts right away. Fastest.
One server at a timeEach server must finish before the next one starts, including coming back after a reboot or reporting a new agent version. With Stop if a server fails (on by default), the servers after a failed one are not touched.

Skipped servers

Servers that cannot run the operation are listed as Skipped with the reason instead of failing the action: offline servers, servers without an agent, agents too old for the operation, and, for Update the agent, servers already on that version, servers whose CPU type the panel has no agent program for, and servers still on the old Node.js agent, which move with the command on their page. If no server can run it, the action is refused.

The panel's own server

This server always runs last, so rebooting or restarting its agent cannot cut off the servers after it. A reboot of the panel's own server waits until every other server is done, in either mode. The panel is then unreachable until it starts again on that server, and that server is given a day, instead of 10 minutes, to report back.

Following and cancelling

Infrastructure > Bulk Actions lists every bulk action with its progress; viewing them needs jobs.read. Open one to see each server in the order it runs, its state (waiting, running, verifying, succeeded, failed, skipped or cancelled) and a log link to exactly what the agent did.

Cancelling stops the servers that have not started, and cancels jobs still waiting for their agent. Anything already running on a server finishes.

Who it runs as

Each server's part is created as the person who started the action, with the same permission checks, duplicate checks, logs and audit entries as running it by hand. Later servers in a one-at-a-time action re-check that person's current access, so deactivating them or removing their permission stops the rest. Bulk actions are recorded in the audit log as bulk_action.create, bulk_action.cancel and bulk_action.finish.