Backup and restore¶
What the portal backs up, where it puts the backups, and how to bring the portal back on a new VM. Settings are in Settings → Backups (super admin). The restore checklist is at the end of the page.
Standard and Enterprise
A Standard installation's backup settings are pinned and cannot be changed: local backups to
~/claugine/backups (~ is the account that ran the installation), once a day at 03:06, keeping 3 days,
S3 off. An Enterprise installation chooses the folder, the time and how long backups are kept, and can copy
them to S3. The rest of this page applies to both unless it says otherwise.
What you must keep to be able to restore¶
Two things, kept apart from each other:
- A backup (a backup folder with its two files, below).
- The archive password, which opens the backup's archive.
Every secret in the portal's database (platform passwords, SSH keys, BGP passwords, S3 keys) is
encrypted with FIELD_ENCRYPTION_KEY, which lives only in portal.env. A backup carries portal.env inside its
archive, which is encrypted with the archive password. Without that password the key cannot be recovered, and the stored secrets cannot
be read: keep it in a password manager, not with the backups.
To change the key, a super admin opens Settings → Portal → Change encryption key. The
portal does it step by step on the active node and shows each step: a backup; the new key on every node, the old one
kept beside it; every stored secret encrypted again with the new key; the old key removed from every node; a backup
with the new key. The portal restarts twice, for seconds each. If a step fails, the old key stays in place, nothing is
lost, and the change can be started again. A password or credential saved while the key is being changed is kept: the
change never writes back an older value. On a pair the other node receives the new key first and this node second;
if either step fails, both keep the keys they had and the change can be run again. A switch over is refused while the
two nodes do not hold the same key. A backup made before keeps the key it was made with, so it still restores.
A portal backup and a change of the key never overlap: a backup takes the database and the key together, wholly
before or wholly after the key changes, so every backup can read the secrets it holds.
Break-glass only (the portal's UI is down): the key is FIELD_ENCRYPTION_KEY in portal.env on each node, a
comma-separated list whose first key encrypts.
What a backup contains¶
Each portal backup is a folder claugine_<portal address>_<edition>_YYYYmmdd-HHMM, for example
claugine_portal.example.com_enterprise_20261116-0306, holding the archive and its restore script of the same name.
The edition (standard or enterprise) says which installation the backup restores to. Backups made before
v01.01.13 have no edition in the name (claugine_<portal address>_YYYYmmdd-HHMM) and are listed and kept in the same
way. The backups are listed newest first by the date in the name.
| File | What it is |
|---|---|
<name>.7z |
the data, in a 7z archive encrypted with AES-256 and the archive password; the file names are encrypted too. It holds the whole database and the portal's settings file portal.env (the database connection and FIELD_ENCRYPTION_KEY, the key to the stored secrets) |
<name>.sh |
the script that restores it (Restore the portal). Its first lines say when and where the backup was made and by which release. It carries the archive's checksum and checks the archive before it does anything. Its header repeats the backup's edition |
A portal backup carries portal.env, which holds FIELD_ENCRYPTION_KEY, the key to the stored secrets. The backup
fails, before anything is published or older backups are removed, when that copy has no valid key or when its key
does not open every stored secret the portal opens.
The script is written last: a folder without it is an incomplete backup. A backup does not hold the software: a restore installs the current release your licence entitles you to and migrates the restored database forward onto it. The release recorded in the script is for reference only. Each backup records its edition inside the encrypted archive, and its restore script's header repeats it.
A backup is written into a hidden .partial folder and renamed when it is complete, so a folder with the final name
is always whole. Folders are made 750 and files 640. Every folder and file a backup creates is given to the
service account named in Settings → Backups, so a backup taken by the portal (which runs with full rights) stays
readable and removable by the operator who owns the folder.
When, and by which node¶
- Once a day: Standard at 03:06; Enterprise at the time set in Backup at (by default 00:45, in the portal's time zone). Back up now makes one as soon as nothing holds it back (another backup running, or maintenance: see the bullets below).
- Only the active node backs up (a Standard installation has only that one). The appliance timer (
claugine-appliance, as root, every 5 minutes) makes the backups that are due. - One at a time: on a node, one portal backup and one run of the platforms' backups run at a time. A backup asked for while one runs (the daily one and the one made before an update, say) waits for it, and can be named with its seconds when the first has the same minute (in a bucket it may be reported as a clash instead: see One folder).
- Backups and maintenance wait for each other: a running portal or platform backup counts as running work, so an update or a switch over of the pair waits until it ends, like a deployment. While maintenance runs (an update, a switch over), no scheduled backup and no Back up now starts: the backup is made once the maintenance has ended; the backup an update makes before it installs is part of that update and runs. A Back up now asked for while the platforms' backups run is made after them.
- No backups without the archive password: until it is set, each run fails and says so.
- A failed backup is e-mailed to the super admins; a good one is silent. The last result and the available backups are shown on the tab.
- Before a release is installed on the active node from the portal, a backup is made first; if it fails, the release is not installed. See Software delivery and updates.
Where: a local folder and an S3 bucket¶
Each destination has its own switch; one run writes both.
One folder, names that say what they are¶
Every backup goes into one folder, named in Settings: the portal's own backups and the platforms' backups side by side. What a backup is, is in its name:
| Name | What it is |
|---|---|
claugine_<portal address>_<edition>_<YYYYmmdd-HHMM> |
a backup of this portal (before v01.01.13 without the edition) |
platform_<platform>_<address>_<YYYYmmdd-HHMM> |
a backup of a platform's controller |
A backup made in the same minute as the one before it can be named with its seconds as well. In a bucket, a folder that is already there is never written over: instead of a folder named with its seconds, the backup may be reported as a clash, and then the bucket has no copy of it.
The portal lists and prunes only its own backups (the ones whose name carries its own address), and prunes the platforms' backups only among those. One kind can never delete the other's, and anything else in the folder is left alone. "Kept for N days" counts from each backup's own time.
A local folder¶
- The backups go straight into the folder set (on a new installation,
/var/backups/claugineby default). - Keep, days (by default 10): older backups of this portal in the local folder are removed.
- The local folder is on the app VM itself, so it does not survive the loss of the VM: use it with a bucket, or copy the folder off the VM.
An S3 bucket¶
- AWS S3 or S3-compatible storage (for example Ceph RGW or MinIO): the endpoint (HTTPS), the bucket, an access key and secret key, and path-style or virtual-hosted addressing.
- The S3 row of each block (Portal, Platforms) has an optional Region. Empty, the portal takes the region from the
endpoint when the endpoint names one in the form s3.
. - e.g. Backblaze B2's s3.eu-central-003.backblazeb2.com gives eu-central-003, AWS's s3.eu-west-2.amazonaws.com gives eu-west-2 - and otherwise signs for us-east-1, which most self-hosted S3 stores (MinIO, Ceph) accept. Set it only for a store that wants another region name. - A backup file larger than 64 MiB is uploaded in 64 MiB parts (multipart upload); a failed upload is aborted.
- The portal's backups are uploaded to the S3 bucket only while the licence includes S3 backups, even when S3 was set up before; the S3 backups listed and the backup schedule follow the same rule.
- One folder per backup, directly in the bucket. When the local folder is on too, the same backup folder is copied to the bucket; when it is off, the backup is made in a temporary folder, uploaded, and the temporary copy removed.
- A backup in the bucket is listed as available only when both its archive and its restore script are there. A backup is never written over a folder the bucket already has: of two backups in the same minute, the second is named with its seconds, or reported. With a local folder, the backup is made there first and S3 problems are reported without touching it.
- The portal only adds backups and never deletes from the bucket. How long backups stay is the bucket's own
business. The tab shows the recommended bucket settings:
- versioning on, and object lock with a default retention (for example 45 days, governance or compliance mode): during that time nobody can delete or overwrite a backup, not even with a stolen key;
- a lifecycle rule that removes objects some days after the retention ends (for example 60 days) and prior versions after their days (for example 50), otherwise the bucket only grows;
- CORS off, public access blocked, server-side encryption on;
- a user of its own for the portal, allowed only to add backups and read them back (
s3:PutObject,s3:GetObjectands3:AbortMultipartUploadunder the prefix, and listing its backups; aborting a failed upload deletes no backup), denied every delete, lifecycle and object-lock change, and HTTPS only. The tab shows the bucket policy to use.
Backups of the platforms' front ends¶
The same tab has a second block, Platforms: backups of the platforms' OpenNebula front ends.
- Off by default, with switches of its own. from a first time of the day (by default 05:25), every so many hours (by default 24), or Back up now; Keep, days for the local copies; the archive password (the portal's, unless one is set here); the platforms to include (none ticked: every platform with a front-end VIP).
- Renaming a platform keeps it chosen for backups under its new name. If a platform chosen for backups cannot be backed up - it was deleted, or has no front-end VIP - the backup run is marked failed and names it, no older backup is deleted by that run, and the failure is e-mailed; choose the platforms again in Settings.
- A local folder (no default): the backups go straight into it, beside the portal's (see One folder); an S3 bucket can be the portal's or one of its own.
- Each backup is a folder
platform_<platform>_<VIP>_YYYYmmdd-HHMMwith the same two files:<name>.7z(AES-256, the archive password, file names encrypted) holding one folder per front-end node — the node's configuration folders (by default/etc/one,/var/lib/one/.ssh,/var/lib/one/.oneand/var/lib/one/remotes) and, from the leader, the OpenNebula database (onedbbackup of the zone, and of the federation on its primary) — and<name>.sh, written last. - The database is dumped on the front end's Raft leader, found by asking each front-end node, or on the only node of a front end without HA. A platform backup without its database dump - no leader found, or the dump failed - is a failed backup: it is not kept, and older backups are not removed.
-
A node's configuration archive counts only when it is complete. Two cases leave it incomplete:
- a file in a backed-up folder cannot be read, or changes or disappears while it is read: the archive is kept but does not count - the backup run is marked failed and names the file;
- sudo refused to read the folders on a front end: that node is named in the run's result and its configuration is not kept.
Either way no older backup is deleted by that run, and the failure is e-mailed like any other backup failure. - A platform backup is only unpacked, on any Linux with 7-Zip:
bash <name>.sh [<folder>]checks the archive, asks for the archive password, unpacks it (by default into./<name>) and prints how to put a node back by hand. - Connect to Front-Ends using: the account the portal logs in to each front-end node with. By default each platform's SA secret - the same login the portal uses for every host and VM; or one account secret (with an ssh key) chosen for all platforms. The user needs sudo without a password on the front ends: the folders are read with sudo (so files only root may read are in the backup too), and the OpenNebula database is dumped asoneadminthrough sudo. A front end where that user cannot use sudo without a password is named in the run's result. - A platform is backed up when it has a front-end VIP;oneadmin's key in its front-end secret is no longer needed. - The portal only reads on the front ends, and removes only its own temporary dump. Old local backups are removed only after a run without errors.
Restore the portal¶
A restore puts a backup onto a freshly installed Ubuntu 26.04 machine. The portal you restore onto does not have to be the machine that failed.
- Copy the backup's two files (
<name>.7zand<name>.sh) to the new machine, into the home of the ordinary user with passwordless sudo that will run the installation (from the local folder withscp, or from the bucket with any S3 client). -
As that user (not root), run the script with your licence key:
-
The script checks the archive against the checksum it carries, downloads the installation script and restores: the installer asks once for the archive password and checks it on the whole archive before anything is changed, so a wrong password costs you nothing. Right after that it reads the backup's edition, before anything is installed, and stops if it is not the licence's edition.
- It installs the current release your licence is entitled to, as a bundle from the licensing portal, then loads the backup over it and migrates it forward to that release.
Settings → Backups shows these steps, with the newest backup's name filled in.
A backup is restored from its own script (the 7z format). The earlier folder format (database.sql.gz +
portal.env.zip) still restores while such backups are kept; the formats before it are no longer read.
A restore (the installer, or restore.sh) refuses a backup that has no portal.env with a valid key, before
anything is installed.
A backup restores only to an installation of the same edition: a Standard installation's backup restores with a Standard licence, an Enterprise installation's backup with an Enterprise licence. A backup made before backups recorded their edition is accepted and the installer says it is a legacy backup.
The restored installation keeps its original identity: its installation id is the portal's name (the host of
--site-url), so restoring under the same name is the same installation. The licence binding survives, nothing has
to be released on the licensing portal, and the licence is not consumed a second time.
On an Enterprise installation whose standby survived, the standby holds the database up to the moment of the loss, which is newer than the last backup: making it the active node loses less than a restore.
When both Enterprise nodes are lost¶
Restore onto one new VM and name the other one, in a single command (both files copied to the first VM):
bash claugine_portal.example.com_enterprise_20261116-0306.sh --licence-key CLG-… --second admin@192.0.2.20
The backup is restored onto the first VM, which keeps the installation's identity; the second VM is then set up from it exactly as in a new installation (see Installation).
When only one node is lost, there is nothing to restore: use Replace node on the node that is still running (see Setting up the second node).
Restore checklist¶
After a restore, check: the portal answers on its address over HTTPS; Settings → Portal shows the licence as
active with the installation id you expect; the four timers are running; and the next backup appears in
~/claugine/backups the following night.