Portal architecture¶
How a Claugine portal installation is built: its VMs and their roles, the database, the traffic path, the application and its timers, the HA agent, secrets, mail and certificates. It describes the design, not any one installation; names and addresses below are examples.
A Standard installation¶
A Standard installation is one virtual machine running Ubuntu 26.04. On it:
- Apache serves the portal over HTTPS on the machine itself and hands the requests to the application:
gunicorn runs Django from the portal's own Python environment (
claugine-app). Apache also serves the static files, the HTTP redirect and the Let's Encrypt challenge. There is no load balancer in front of it. - A TLS certificate. The installation always writes a self-signed certificate so the portal serves HTTPS from the first minute. If you give the installer an e-mail address, it then requests a Let's Encrypt certificate for your portal's name. If that request fails — most often because the name does not yet resolve to this machine — the installation finishes anyway and keeps the self-signed certificate, and you can request the real one later.
- MySQL, on the same machine, with no replication.
- Four scheduled jobs: reading the platforms (every five minutes; each refresh also checks the platform's consistency with the portal), checking platform endpoints and images, nightly housekeeping, and an appliance job that keeps the node's mail server in line with Settings and makes the backups when they are due.
There is no HA agent: the component that holds roles and switches between nodes is only installed when there are two nodes. The rest of this page describes the Enterprise design: two app VMs, on one or two sites. See Standard and Enterprise.
An Enterprise installation¶
An installation is one portal on one or two sites:
- two app VMs at most: one active, one standby (on one site there may be a single app VM);
- HAProxy on each site in front of them, which ends TLS: on the app VM itself or on a VM of its own;
- one database (MySQL), kept on both app VMs by replication from the active to the standby.
There are never more than two app VMs.
flowchart TB
C([Users, browsers, API clients]) --> DNS["Public name, e.g. my.provider.example<br/>resolves to both sites"]
DNS --> H1
DNS --> H2
subgraph S1["Site dc1"]
H1["HAProxy<br/>TLS, Let's Encrypt"]
A1["App VM: active<br/>Apache, gunicorn, Django<br/>MySQL, writable"]
end
subgraph S2["Site dc2"]
H2["HAProxy<br/>TLS, Let's Encrypt"]
A2["App VM: standby<br/>serves nothing<br/>MySQL, super_read_only"]
end
H1 -- "to the node whose /active answers 200" --> A1
H2 -- "over the link between the sites" --> A1
A1 -- "GTID replication, TLS" --> A2
A1 <-. "HA agents, internal port, token" .-> A2flowchart TB
C([Users, browsers, API clients]) --> DNS["Public name, e.g. my.provider.example<br/>resolves to both sites"]
DNS --> H1
DNS --> H2
subgraph S1["Site dc1"]
H1["HAProxy<br/>TLS, Let's Encrypt"]
A1["App VM: active<br/>Apache, gunicorn, Django<br/>MySQL, writable"]
end
subgraph S2["Site dc2"]
H2["HAProxy<br/>TLS, Let's Encrypt"]
A2["App VM: standby<br/>serves nothing<br/>MySQL, super_read_only"]
end
H1 -- "to the node whose /active answers 200" --> A1
H2 -- "over the link between the sites" --> A1
A1 -- "GTID replication, TLS" --> A2
A1 <-. "HA agents, internal port, token" .-> A2
Setting up the second node¶
The first VM sets the second one up itself, over ssh, and pairs it: the installation script does it when it is run
with --second (see Installation), and an installation that runs one node can do it later
from the portal.
Settings → Portal → Add second node. On an Enterprise installation that runs one node, the active node can set
up the second one from the portal. The page shows this node's key: add it to the other VM's user as above, enter
that VM as user@address, and choose Check only to run the checks without changing anything, or Add second
node to do it. The page shows each step as it runs. The address is the other VM's internal IPv4 address (for
example admin@192.0.2.10; no host name). The node checks the request itself before
it does anything, and refuses one that is not in this form. Setting up the second node is refused while another
operation runs on the node (an update, a key change, a switch over), and none of those starts until it has finished.
If a run of Add second node or Replace node stops before it finishes (the node restarts or the service is stopped),
the section shows the last run as interrupted, with its log, and accepts a new request. Run it again with the same VM:
a half-finished node of the same installation is carried on with, not refused.
Add node and Replace node wait for running work: while tasks run (see Tasks), the portal answers "Not now: tasks are
running" and nothing is started; Check only still works. The provisioning script refuses in the same case. While
they run, the portal starts no new work (see Maintenance).
Replace node. When the other node stops answering, the same card offers Replace node: give a new, clean VM, and the active node installs the portal there, copies the database to it and makes it the standby. The lost VM is not needed for this.
Roles of the app VMs¶
Each app VM has its own node.env (its site, its MySQL server ID, its address and the other node's, the addresses
of the HAProxy that may reach it, its ports and its role). It is never replicated.
| Role | What the node does |
|---|---|
| active | serves the portal; its database is writable |
| standby | a read-only copy that follows the active node's database; it serves nothing but its status page |
| maintenance | out of service for work on it |
There is no role for a version under test.
The role addresses. Each app VM answers GET /active, /standby and /maintenance on its internal address -
200 when it has that role, 503 when not. HAProxy checks /active and sends the portal's traffic to the node that
answers 200, and shows a maintenance page when no node does. When HAProxy runs on the app VMs themselves, the role
addresses have a port of their own (HAProxy holds 80 and 443 there).
Nothing switches by itself. A switch-over is an operation an operator starts (below).
When a switch over stops¶
- A switch over that stops with no answer: when a switch over stops and the other node does not answer, nobody
can tell whether it took over, so the node that was serving stays in maintenance (read only) - never two active
nodes. Every 5 minutes it asks the other node again, and as soon as that node answers it recovers by itself: if
the other node took over, it follows it as the standby; if not, it fences the other node (read only, and it
cannot become active until it is promoted again) and serves again. The super admins get an e-mail either way.
When a recovery fences the other node, that node is made the standby and the portal checks, on its regular timer,
that it replicates from the active node. A node whose database was not changed and which replicates needs no new
copy; otherwise its database is copied from the active node. The pair counts as having a standby only once this
check has passed. When a switch over stops after the other node was promoted, this node serves again and fences
the other node. As after a recovery, the portal then checks on its regular timer that the fenced node is the
standby and replicates from this node; it copies the database again only when it does not.
The same holds when the switch over is cut off - the node restarts or loses power while taking the other node's
place: on its next round the node finds the interrupted switch over, and recovers by itself or offers
/recovery, as after a switch over that stopped. If the node that handed over cannot start following the other node at once (its replication cannot be set up then), it stays out of service and keeps trying by itself every 5 minutes; it becomes the standby as soon as it follows the serving node. A switch over ends only once the node that handed over replicates from the serving node. If replication does not start, the switch over reports it; the serving node keeps serving, and the other node stays out of service and keeps trying by itself every 5 minutes until it follows. When a node out of service finds that the other node serves, it follows it and becomes the standby only once replication runs; until then it stays out of service and keeps trying by itself every 5 minutes. - Serve from this node anyway: when the other node is down for good, a super admin opens
/recoveryon the portal's address (the maintenance page links to it while no node serves). It shows both nodes and asks for the super admin's e-mail, password and authenticator code, and a confirmation that the other node is down. The node then serves alone. Do this only when the other node is really down: if it still serves users somewhere, both nodes accept changes and the changes made on one of them are lost. Wrong attempts on/recoveryare counted per client address and per e-mail: five from one address, or ten for one e-mail, wait 15 minutes. Someone else's wrong attempts do not keep a super admin out. Serving alone is recorded before this node starts to serve, so even if the portal stops right then, the other node is checked and brought back as the standby when it answers again, and the super admins are told if it comes back active too. - The other node back: when it answers again, it copies the serving node's database and follows it as the standby, by itself. If it comes back active too, nothing is copied: Settings → Portal says both nodes are active and the super admins get an e-mail, because only a person can decide which node's changes to keep. A super admin settles it there with Keep serving from this node: the other node stops serving at once and becomes the standby, and its database is replaced by a copy of this node's - the changes made on it since the two parted are lost. To keep the other node instead, the same button is pressed on that node's own Settings → Portal. Until then, Update OS, Update portal, Change encryption key and Switch over wait. The decision is recorded before the other node is stood down, so if the portal stops in between, or the other node does not answer, the portal finishes it by itself: the other node is stood down and gets a new copy of this node's database. When the portal finishes an interrupted "Keep serving from this node", it fences the other node before giving it a new copy of this node's database. Once the other node has its new copy and follows as the standby, it is an ordinary standby: a later switch over between the two nodes needs no further copy. If "Keep serving from this node" is chosen on both nodes at the same moment, the decision made first wins on both: that node keeps serving and the other becomes the standby and gets a new copy; the later decision fails with "the other node was kept first".
- Settings → Portal shows what happened: "Recovered by itself at …", or "Serving from this node since … without the other node (… confirmed it was down)" until the other node is back.
The database¶
- MySQL (the Ubuntu 26.04 package), database
claugine_portal. - GTID replication from the active node to the standby, over TLS, with a replication user that each node makes
for the other node's address only. The standby is
super_read_only. Users and grants are each node's own, made outside the binary log. Replication between the two nodes is encrypted and checked: each node's database has its own certificate for its internal address, the other node receives it over the agents' authenticated link, and replicates only from a node that presents it. A pair set up before v01.01.06 switches to the check by itself once both nodes run v01.01.06. - The standby loads a new copy of the database within the time the active node waits for it: the copy must have arrived before the standby's own database is replaced, a second copy asked for while one loads is refused, and a copy that cannot be loaded in time is given up with the standby's database untouched.
- Switch-over is refused while tasks run, when the versions differ, or when the standby is not replicating.
- The database is the desired state and the single source of truth: users and their photos, organizations, projects, the infrastructure, templates, settings, tasks, the log and every secret. OpenNebula is the actual state; the portal reads it and flags differences.
- Nothing is hard-deleted: rows are versioned or blocked.
Traffic: HAProxy on each site¶
- Each site forwards the ports of its public address to its HAProxy (on the app VM, or on a VM of its own).
- 443: the portal's public name → the active node (on this site or over the link between the sites). A maintenance page when no node can take it.
- 80: a redirect to HTTPS, and the Let's Encrypt HTTP-01 checks. A check may reach either site: a token not found on one HAProxy is served by the other.
- The site's OpenNebula front end is published through the same HAProxy, with TLS ended there, under its
platform's public FQDN on 443: FireEdge (
/fireedge/sunstone, the name alone opens it too), the XML-RPC API (/RPC2) and OneFlow (/oneflow), to any address, each asking for the OpenNebula login itself; they work when the name is proxied by Cloudflare too. The gRPC API is not published (OpenNebula's own gRPC clients connect without TLS), nor is OneGate (below). See OpenNebula front ends. - What HAProxy serves comes from Settings → Customer → Provider (the public URL and the Let's Encrypt e-mail) and each site's front end: its shared platform that has a Public FQDN (the first by name when there are several; Infrastructure → the site → Platforms → Edit). The values live in the database and replicate. HAProxy is the load balancer the provider runs in front of a two-node installation; the installer opens the app VMs' firewall to it when it is given its addresses.
- With the VM's firewall (ufw) on, the app VMs take HTTPS from HAProxy only.
OneGate¶
Each platform chooses how its VMs reach OneGate: Infrastructure → the site → Platforms → Edit → OneGate through
hosts (in the Zone name row). The portal records the choice and shows the address in the platform's details; the
front end is set up by its operator to match. OneGate is never published by HAProxy: /onegate on the front end's
public name is FireEdge's like any other path. The other services on the VIP (FireEdge 2616, XML-RPC 2633, gRPC
2634, OneFlow 2474) are the same in both modes; gRPC is used internally only.
Ticked: through the VM's host. VMs reach OneGate at http://169.254.16.9:5030, OpenNebula's transparent proxy:
every hypervisor answers on that link-local address inside each virtual network's bridge and forwards the connection
to the front end over its own network. Nothing about OneGate is published on the internet, and it works for VMs on
private networks too.
- A VM that uses OneGate needs
TOKEN="YES"in its context, a route169.254.16.9 dev <its NIC> scope link, and, if its security groups limit outbound traffic, TCP 5030 allowed out. - On the front end (the operator): the
:tproxy:entry for port 5030 inOpenNebulaNetwork.confpointing at the front end's VIP,onehost sync -f, andONEGATE_ENDPOINT = "http://169.254.16.9:5030"inoned.conf. - The front end's firewall: port 5030 on the VIP is open only to the hypervisors' network; VMs never reach the
VIP directly. For example, with the hypervisors on 198.51.100.0/24 and the VIP 192.0.2.30:
ufw allow in on eth0 from 198.51.100.0/24 to 192.0.2.30 port 5030 proto tcp.
Cleared: at the VIP. VMs reach OneGate at the front end's VIP, http://<VIP>:5030, on the internal network.
- A VM that uses OneGate needs
TOKEN="YES"in its context and a route to the VIP; if its security groups limit outbound traffic, TCP 5030 to the VIP allowed out. - On the front end (the operator):
ONEGATE_ENDPOINT = "http://<VIP>:5030"inoned.conf, and the front end's firewall lets the VMs' networks reach port 5030 on the VIP.
The application¶
- Ubuntu 26.04; Apache 2.4 in front of gunicorn; Django 5.2 on Python 3.14 in a virtual environment. Every Python package is pinned to an exact version and hash; a node installs exactly the tested set.
- Code in
/opt/claugine-portal(the installed commit in/opt/claugine-portal/COMMIT); configuration in/etc/claugine-portal/(portal.env,node.env); management commands withsudo claugine-manage. - The application's own messages are in the systemd journal (
journalctl -u claugine-app). portal.envis written by the installer and holds only what the portal needs to start: the database connection, the key to the stored secrets and the node's identity. Everything an operator changes later is in Settings, read live from the database; the key is changed from Settings → Portal (see Changing the key under Secrets and encryption). Editing the file by hand and restarting the application is a break-glass step, for when the portal's UI is down.- Django apps:
| App | Owns |
|---|---|
accounts |
people, staff roles, invitations, two-factor sign-in, the audit log, the dashboard, reports, settings of the appliance (mail, backups, updates), the HA agent |
tenants |
organizations, projects, orders and approvals, quota and connectivity records, address assignments, naming |
inventory |
the infrastructure: sites, platforms, clusters, networks and IP subnets, secrets, templates, core services, capacity |
adoption |
the collection from OpenNebula: what the platforms run, checked against the portal's records at every refresh (quotas, objects without a project, addresses) |
content |
news, announcements, service status |
- Timers (systemd, on every app VM; what they do runs on the active node):
| Timer | When | Does |
|---|---|---|
claugine-collect |
every 5 minutes, collects when due | reads the platforms (Settings → Portal → Timers sets the periods) and checks each against the portal: quotas and unbound objects (Site → Platforms, Consistency with the portal) |
claugine-health |
every 5 minutes | the platforms' health, for those that are due |
claugine-cleanup |
nightly | removes finished tasks and log entries older than the history settings |
claugine-appliance |
every 5 minutes, as root | keeps the node's mail server in line with Settings, makes the backups that are due, the platforms' front-end backups, automatic updates |
The HA agent¶
claugine-ha runs as root on each app VM, on the VM's internal address and an internal port, over TLS. It answers
only requests that carry the token shared by both nodes (from portal.env); with the VM's firewall on, only the
other app VM reaches it.
- It reports the node's role, site, version, database state (read only or writable, GTIDs, replication) and running tasks.
- It runs the operations, step by step, talking to the other node: switch-over (the standby becomes active; about a minute of maintenance; refused while tasks run or when the versions differ), maintenance on/off, and the updates of the portal and of the OS, one node at a time (see Software delivery and updates).
- It can clone the database from the other node and start replicating from it.
- It never passes code between the nodes: when a node needs the release the other one runs (an update, or the second
node being set up), it fetches that release itself as a bundle from my.claugine.com - authorised by the licence
key, its checksum verified before it is unpacked - and installs it (
claugine-portal-update). Nothing is installed from source, and the other node's code never comes over (see Software delivery and updates).
The operations are started in Settings → Portal (the actions on top), or as root with claugine-manage-root ha_agent.
Secrets and encryption¶
- Everything is in the database, secrets included (platform passwords, SSH keys, BGP passwords, the mail account,
S3 keys, the archive password), each encrypted with
FIELD_ENCRYPTION_KEY(Fernet). - Outside the database, on each VM:
portal.env(the Django secret key, the database connection, the HA token and replication password, andFIELD_ENCRYPTION_KEY), and the VM's own files:node.env, the last HAProxy options and the Let's Encrypt certificates. A node gets every release as a bundle from my.claugine.com. The key cannot live beside the data it protects: whoever had the database would then have every secret. - Changing the key: a super admin opens Settings → Portal → Change encryption key. The portal does it step by step on the active node and shows each step: a backup; the new key on every node,
the old one kept beside it; every stored secret encrypted again with the new key; the old key removed from every
node; a backup with the new key. The portal restarts twice, for seconds each. If a step fails, the old key stays in
place, nothing is lost, and the change can be started again. While it runs, the portal starts no new work (see
Maintenance). A backup made before keeps the key it was made with,
so it still restores. Break-glass only (the portal's UI is down): the key is
FIELD_ENCRYPTION_KEYinportal.envon each node, a comma-separated list whose first key encrypts. - So a restore needs a backup and
portal.env(or at leastFIELD_ENCRYPTION_KEY). The backups carryportal.envin their archive, encrypted with the archive password: keep that password apart from the backups. See Backup and restore. - Every user signs in with a password and a second step: a TOTP code (required) or a passkey (each code is accepted
once, and after a wrong code the next attempt waits a moment, longer after each further failure). Every significant action
is written to the log. The technical admin site (
/django-admin/) has no sign-in of its own: it sends to the portal's sign-in, with its second factor, and opens for a staff session made there. The technical administration pages show people and invitations but do not change them: a person's roles, e-mail, sign-in and two-factor authentication are changed only on the portal's own pages.
Mail and certificates¶
-
Mail (Settings → Mail): each portal node sends the portal's mail with its own mail server, straight to the recipients' servers, under its mail name (set when the node is installed), signed with DKIM for the domain of the sender address. The portal makes the DKIM key itself; a node that already signed with a key keeps it. An installation that used a relay or an external SMTP server sends this way from this version on. Under DNS records Settings → Mail lists the records to add, with what DNS has now: the A record of each node's mail name, the sender domain's SPF, DKIM and DMARC records, and the PTR record of each node's public address (set per node under Mail sender public IPs). A record is ok when it does its job, not only when it is written as the page writes it (see Settings → Mail). The appliance timer keeps each VM's mail server in line with the settings.
-
Certificates: Let's Encrypt, requested on each site's HAProxy for the portal's names and the front ends' names, renewed by
certbot.timer. Until a certificate is issued a self-signed one is used. The Let's Encrypt account's e-mail is set in Settings → Customer → Provider and applied to both VMs within 5 minutes.