Network¶
The network functions the platforms use and how projects are connected: exchange networks to the Internet and to the corporate network, L2-over-L3 channels and the EVPN fabric, plus the address rules the portal applies. Based on the Claugine reference design. The data network itself (switches, uplinks, routing to the outside) is designed and run outside Claugine.
1. L3 layout¶
- Site network — provided by the data network; connects the site to the Internet and to other sites.
- Management gateway — routes the management networks: out-of-band, host management, platform core.
- Payload gateway — connects the exchange networks (IX) to the outside.
- Every landscape is closed the same way: by an external gateway (EGW) in the IX and an internal gateway behind it. Projects have their own EGW and internal gateway; the provider's published services have
cloud-egw(services network to the IX) andcloud-igw(platform core to the services network), defined as one core service of the site,cloud-gw, with its two VMs.
2. Fabrics and host network stack¶
| Fabric | Carries |
|---|---|
| data | everything except storage: platform, services, transit, VXLAN transport |
| storage | hosts ↔ storage systems |
Both fabrics have no single point of failure; the platform tolerates a network outage of up to one second.
| Bond | Ports | Fabric | MTU | Segments |
|---|---|---|---|---|
| bond0 | NIC1 port 1 + port 2 (25G), aliases nic-data1, nic-data2 | data | 9100 | host management, platform core, services, exchange networks, VXLAN transport |
| bond1 | NIC2 port 1 + port 2 (25G), aliases nic-storage1, nic-storage2 | storage | 9000 | NFS, iSCSI |
LACP, 802.1Q, plus an out-of-band port. A server profile with one fabric carries data and storage in bond0. VM networks are Linux bridges created by OpenNebula per network: VLAN on the management platform, VXLAN on the payload platforms. Two NTP servers per site come from the data network.
3. Segments¶
| Kind | Segment | Purpose |
|---|---|---|
| internal | oob | IPMI of all servers |
| internal | host management | host OS management |
| internal | platform core | services that run the platform |
| routed | services (cloud-services) |
services published to users: FE end points, DNS, … |
| routed | project main network | delegated to one project, /31 and up |
| transit | exchange networks (cloud-ix, corpnet-ix, portal-ix) |
IX to the Internet, IX to a corporate network |
| isolated | VXLAN transport | carries the EVPN overlay between payload hosts |
| isolated | storage | NFS / iSCSI |
| transit | s2s transport (cloud-s2s) |
L3 between the legs of L2-over-L3 channels |
| overlay | project subnets | created by projects, VXLAN/EVPN, any addressing |
4. Project connections¶
Every project has at least one connection; the customer chooses them when ordering:
| Connection | Who may order | Combines with |
|---|---|---|
inet — Internet through the exchange network |
everyone | L2 |
corpnet — routed connection to the corporate network |
corporate projects of corporate organizations only | L2 |
l2 — external L2-over-L3 channel |
everyone | inet, corpnet, or alone (alone: cloud projects only) |
The internet and the corporate network never exist in one project. The connectivity record stores the routed connection (inet or corpnet), or none when the project has only L2 channels. Internal L2 channels (§7) join a project's own networks on two sites and are not a connection to the outside.
5. Exchange network to the Internet¶
The IX is a dedicated transit VLAN. Only routers are connected to it, they talk only to each other, nothing outside the VLAN can reach their addresses, and only permitted transit traffic passes.
| Router | Where | Role |
|---|---|---|
| Internet gateways | data network | exchange traffic with the Internet; announce the default route; receive and aggregate the projects' public subnets |
| Route reflectors | management platform, 3 VyOS VMs | receive the projects' public subnets from the EGWs and the routes of the Internet gateways; send the full table to all BGP peers |
| EGW — project external gateway | payload platform, one VyOS VM per project | announces the project's public subnets, connected and statically routed |
Project side¶
- EGW — controlled by the provider, not reachable by the project:
- outside leg in the IX, inside leg in the project's main network (below). The edge gateway limits the project's internet with a shaper (fq-codel) on the egress of both its sides — towards the exchange (the project's upload) and towards the project's main network (its download) — at the project's bandwidth (10–1000 Mbit/s); traffic inside the provider network is not limited. When a bandwidth change is approved, the record changes, the edge gateway gets the new value at once and the project group's quota record follows;
- the inside leg is in the project's main public network, /31 minimum; the first address is the EGW, the second the project's internal gateway (IGW) — the project's default gateway is on the EGW;
- lets in and out only the project's public addresses (main network and additional public subnets);
- drops addresses not routed in the Internet (RFC 1122, 1918, 3927, 5737, 6598 and other special-purpose blocks);
- management only on its own NIC in the management VRF, from the management hosts.
- Internal gateway (IGW) — the project's router between its overlays and the main public network, a VM owned by the project:
vyos— advanced (VyOS), the default, created by the portal;onevr— native (Service Virtual Router), the OpenNebula virtual router; orcustom— the project's own VM; does source NAT for private subnets. The type is chosen with the order: advanced (the default), the platform's native router, or your own; with your own, nothing is created: the address is reserved and the project's virtual machine takes it. Additional public subnets are routed through the IGW. The type is changed by the project without approval; the IGW is then replaced and its configuration is not kept; it keeps its address. The internet gateway is made with 1 vCPU and 1 GB RAM by default and accepts password login (its user and a generated password) when it is made without an ssh key. Its details in the portal give its VM (host name, platform, id, cluster, host), its size, its NICs one per line (network and address), the ssh command and its password; a VyOS internet gateway has an Edit button on its row: its configurations, its networks and its size — applying a new size restarts the internet gateway's VM (the project's internet stops for about a minute). - A project has one internet connection per site, each with its own speed, internet gateway type and subnets. The main public subnet of a site is replaced only by an approved request (Main on the site's row of the project's Internet table).
6. Exchange network to the corporate network¶
The same model as the Internet IX: a dedicated transit VLAN with routers only, corporate network gateways on the data network side, the project's cgw (corporate network edge: a provider-controlled VyOS VM on the payload platform) with its outside leg in this exchange network and its inside legs in the project's corporate subnets, and routes exchanged over BGP through the corporate exchange's route reflectors. It connects a project to the corporate network instead of the Internet; internet access of a corporate-network project is controlled by the corporate network. The corporate network needs no IGW.
7. L2-over-L3 channels¶
A project's L2 channels are of two kinds; its L2 list shows for each the type · endpoint 1 · endpoint 2 · state:
| Type | Endpoint 1 | Endpoint 2 | Approval |
|---|---|---|---|
| external — a VLAN from the client's side brought into a project network over VXLAN | the VLAN | the project network | a resource manager approves; the support engineer sets the VLAN |
| internal — two of the project's own networks on two different sites, bridged over VXLAN | site 1 · its network | site 2 · its network | none to add (removal is a request); only for a project with resources on two sites; at most one per pair of sites |
External channels¶
Extend a project L2 segment to a client site over an L3 network: client systems and project VMs share one broadcast domain and one subnet. A channel has two provider-controlled legs (VyOS VMs), named after the project, the VLAN and the site:
| leg 1 — external side | leg 2 — project side | |
|---|---|---|
| VM | _<org>_<project>_l2srcNNNN_<site> |
_<org>_<project>_l2dstNNNN_<site> |
| platform | management | the project's platform |
| faces | the client site | the project network |
| eth0 | management (management VRF) | management (management VRF) |
| eth1 | the s2s transport network: WireGuard with the other leg only | the same |
| eth2 | the VLAN network to the client site (bridge s2s-NNNN), no IP |
the project's network |
| inside | Linux bridge of eth2 + a VXLAN tunnel inside WireGuard (one VNI per channel) | the same |
- The legs exchange the client's Ethernet frames in a VXLAN tunnel carried in WireGuard over the s2s transport network; the WireGuard keys are made at random per channel and kept nowhere else. Neither leg has an address in the client segment.
- The project network is an ordinary project VXLAN network on the EVPN fabric; leg 2 is one more VM in it.
- When the project's shared platform has its own edge cluster, one VM on that cluster (
_<org>_<project>_l2brNNNN_<site>) bridges the VLAN and the project network instead of the two legs. - In the portal: the project asks for a channel; a resource manager approves it; the support engineer enters the VLAN number (
l2_vlans; the VLAN number is also the VNI when it is free); the project chooses the network it connects to (l2_vnets), and changes it later without approval. The channel is made as soon as it has both.
Internal channels¶
An internal channel joins a project network on one site with a project network on another site (each site one of the project's resource pools). It is added on the project's L2 table without approval (+ Internal channel: a name, the two sites and a network on each); the portal refuses a second internal channel between the same two sites and a channel on a project with resources on one site only.
8. EVPN fabric (project overlays)¶
- Data plane — every project subnet is one VXLAN with a unique VNI. On a host, VMs of one VXLAN share a Linux bridge created with the first VM of that network on the host and removed after the last one. Between hosts, frames are carried over the isolated VXLAN transport segment, which has no L3 path outside.
- Control plane — BGP EVPN, FRR on every payload host. Hosts peer only with three route reflectors (VyOS VMs on the management platform) which distribute every VM's MAC/IP to all hosts; adding or removing a host changes nothing elsewhere.
- OpenNebula — the VTEP of every VXLAN network must be the host's local IP (
VXLAN_TEP = local_ip), on every VNet, on the VNet template projects create networks from, and in the VXLAN driver of the FE nodes.
9. Addresses: IP subnets, service networks, allocation¶
- IP subnets (the registry) are defined per site, under Infrastructure on the site's IP subnets tab: subnet, prefix, the max allowed (delegated) prefix, routing (BGP through an exchange, a static gateway, or the organization) and the type, detected from the address (private: corporate network, public: internet). A subnet itself is immutable once added, and it is deleted only while unused (no exchange network carries it and no project block lies in it).
- The max allowed prefix is the largest block a project may be given from the subnet. It is enforced when addresses are handed out, not only when the subnet is defined: a request for a larger block than it allows is refused. It is at most the subnet's prefix + 2 and never bigger than a /24 (a /20 allows /24, a /26 allows /28), and it can't be set smaller than a block already delegated from the subnet; with nothing delegated, any value from that limit to /31. It is set when the subnet is added (by default the largest allowed) and changed with Edit: for a subnet routed by BGP (ix) it is the only field that can change; for the others the routing and the static gateway can change too. The prefix lengths offered when ordering go down to the most permissive max of the site's subnets of that type.
- Delegated ranges: on the Networks tab each ix network picks the IP subnets it carries: several, all of its own type; one subnet belongs to one ix network only. The ix network announces them to its remote AS, and they are the pools a project's public and corporate prefixes are carved from (the delegated subnets).
- Public and corporate subnets are always bound to an IX network, and each IX network serves exactly one shared front end; front ends never share addresses or hosts.
- Typically each site has a public pool on its cloud-ix and a private pool on its corpnet-ix; a secondary zone uses the pools of its own site. portal-ix pools belong to the ext-managed platforms and are never used for projects.
- Service networks: inventory networks inside a public pool (for example
cloud-services, the first /29 of each cloud-ix pool) carry the services on the management clusters. The portal records them in the address registry as platform services after every inventory change and never gives them to projects. - Allocation: customers choose prefix sizes from a list — /31, then /29, /28 … down to the most permissive max of the site's IP subnets of that type (while none is defined: /31, /29, /28, /27, /26) — and, for each public prefix, a base subnet among the internet subnets delegated to the exchange networks of the platform the cluster is on, or auto. On approval the portal takes the first free aligned block, largest first, from the IX networks of the platform the project is placed on — cloud-ix for internet, corpnet-ix for the corporate network: from the chosen base subnet first, then from the others; with auto, cloud pools first. The resource manager can change the main prefix's base subnet (the internet address pool) when approving. The same choice is offered for an additional public subnet and for a new main subnet of an existing project. Everything in the registry, in OpenNebula group metadata and in the address ranges of the platforms' virtual networks counts as taken.
- The address registry (each site's IP subnets tab) lists the subnets, their use, the assignments (main, additional, corporate, platform services, reserved) and flags overlaps or assignments outside every subnet.
10. Open points¶
- Corporate network: address plan vs project subnets (overlaps, NAT), route policy, bandwidth and filtering on the cgw.
- L2 channels: MTU of the extended segment (VXLAN adds 50 bytes on the s2s transport), redundancy of the legs.
- Bandwidth step: the reference design shapes in 10 Mbit/s steps; the order form currently accepts 1 Mbit/s steps.