Skip to content

Network

The network functions the platforms use and how projects are connected: exchange networks to the Internet and to the corporate network, L2-over-L3 channels and the EVPN fabric, plus the address rules the portal applies. Based on the Claugine reference design. The data network itself (switches, uplinks, routing to the outside) is designed and run outside Claugine.

1. L3 layout

  • Site network — provided by the data network; connects the site to the Internet and to other sites.
  • Management gateway — routes the management networks: out-of-band, host management, platform core.
  • Payload gateway — connects the exchange networks (IX) to the outside.
  • Every landscape is closed the same way: by an external gateway (EGW) in the IX and an internal gateway behind it. Projects have their own EGW and internal gateway; the provider's published services have cloud-egw (services network to the IX) and cloud-igw (platform core to the services network), defined as one core service of the site, cloud-gw, with its two VMs.

2. Fabrics and host network stack

Fabric Carries
data everything except storage: platform, services, transit, VXLAN transport
storage hosts ↔ storage systems

Both fabrics have no single point of failure; the platform tolerates a network outage of up to one second.

Bond Ports Fabric MTU Segments
bond0 NIC1 port 1 + port 2 (25G), aliases nic-data1, nic-data2 data 9100 host management, platform core, services, exchange networks, VXLAN transport
bond1 NIC2 port 1 + port 2 (25G), aliases nic-storage1, nic-storage2 storage 9000 NFS, iSCSI

LACP, 802.1Q, plus an out-of-band port. A server profile with one fabric carries data and storage in bond0. VM networks are Linux bridges created by OpenNebula per network: VLAN on the management platform, VXLAN on the payload platforms. Two NTP servers per site come from the data network.

3. Segments

Kind Segment Purpose
internal oob IPMI of all servers
internal host management host OS management
internal platform core services that run the platform
routed services (cloud-services) services published to users: FE end points, DNS, …
routed project main network delegated to one project, /31 and up
transit exchange networks (cloud-ix, corpnet-ix, portal-ix) IX to the Internet, IX to a corporate network
isolated VXLAN transport carries the EVPN overlay between payload hosts
isolated storage NFS / iSCSI
transit s2s transport (cloud-s2s) L3 between the legs of L2-over-L3 channels
overlay project subnets created by projects, VXLAN/EVPN, any addressing

4. Project connections

Every project has at least one connection; the customer chooses them when ordering:

Connection Who may order Combines with
inet — Internet through the exchange network everyone L2
corpnet — routed connection to the corporate network corporate projects of corporate organizations only L2
l2 — external L2-over-L3 channel everyone inet, corpnet, or alone (alone: cloud projects only)

The internet and the corporate network never exist in one project. The connectivity record stores the routed connection (inet or corpnet), or none when the project has only L2 channels. Internal L2 channels (§7) join a project's own networks on two sites and are not a connection to the outside.

5. Exchange network to the Internet

The IX is a dedicated transit VLAN. Only routers are connected to it, they talk only to each other, nothing outside the VLAN can reach their addresses, and only permitted transit traffic passes.

Router Where Role
Internet gateways data network exchange traffic with the Internet; announce the default route; receive and aggregate the projects' public subnets
Route reflectors management platform, 3 VyOS VMs receive the projects' public subnets from the EGWs and the routes of the Internet gateways; send the full table to all BGP peers
EGW — project external gateway payload platform, one VyOS VM per project announces the project's public subnets, connected and statically routed

Project side

  • EGW — controlled by the provider, not reachable by the project:
    • outside leg in the IX, inside leg in the project's main network (below). The edge gateway limits the project's internet with a shaper (fq-codel) on the egress of both its sides — towards the exchange (the project's upload) and towards the project's main network (its download) — at the project's bandwidth (10–1000 Mbit/s); traffic inside the provider network is not limited. When a bandwidth change is approved, the record changes, the edge gateway gets the new value at once and the project group's quota record follows;
    • the inside leg is in the project's main public network, /31 minimum; the first address is the EGW, the second the project's internal gateway (IGW) — the project's default gateway is on the EGW;
    • lets in and out only the project's public addresses (main network and additional public subnets);
    • drops addresses not routed in the Internet (RFC 1122, 1918, 3927, 5737, 6598 and other special-purpose blocks);
    • management only on its own NIC in the management VRF, from the management hosts.
  • Internal gateway (IGW) — the project's router between its overlays and the main public network, a VM owned by the project: vyos — advanced (VyOS), the default, created by the portal; onevr — native (Service Virtual Router), the OpenNebula virtual router; or custom — the project's own VM; does source NAT for private subnets. The type is chosen with the order: advanced (the default), the platform's native router, or your own; with your own, nothing is created: the address is reserved and the project's virtual machine takes it. Additional public subnets are routed through the IGW. The type is changed by the project without approval; the IGW is then replaced and its configuration is not kept; it keeps its address. The internet gateway is made with 1 vCPU and 1 GB RAM by default and accepts password login (its user and a generated password) when it is made without an ssh key. Its details in the portal give its VM (host name, platform, id, cluster, host), its size, its NICs one per line (network and address), the ssh command and its password; a VyOS internet gateway has an Edit button on its row: its configurations, its networks and its size — applying a new size restarts the internet gateway's VM (the project's internet stops for about a minute).
  • A project has one internet connection per site, each with its own speed, internet gateway type and subnets. The main public subnet of a site is replaced only by an approved request (Main on the site's row of the project's Internet table).

6. Exchange network to the corporate network

The same model as the Internet IX: a dedicated transit VLAN with routers only, corporate network gateways on the data network side, the project's cgw (corporate network edge: a provider-controlled VyOS VM on the payload platform) with its outside leg in this exchange network and its inside legs in the project's corporate subnets, and routes exchanged over BGP through the corporate exchange's route reflectors. It connects a project to the corporate network instead of the Internet; internet access of a corporate-network project is controlled by the corporate network. The corporate network needs no IGW.

7. L2-over-L3 channels

A project's L2 channels are of two kinds; its L2 list shows for each the type · endpoint 1 · endpoint 2 · state:

Type Endpoint 1 Endpoint 2 Approval
external — a VLAN from the client's side brought into a project network over VXLAN the VLAN the project network a resource manager approves; the support engineer sets the VLAN
internal — two of the project's own networks on two different sites, bridged over VXLAN site 1 · its network site 2 · its network none to add (removal is a request); only for a project with resources on two sites; at most one per pair of sites

External channels

Extend a project L2 segment to a client site over an L3 network: client systems and project VMs share one broadcast domain and one subnet. A channel has two provider-controlled legs (VyOS VMs), named after the project, the VLAN and the site:

leg 1 — external side leg 2 — project side
VM _<org>_<project>_l2srcNNNN_<site> _<org>_<project>_l2dstNNNN_<site>
platform management the project's platform
faces the client site the project network
eth0 management (management VRF) management (management VRF)
eth1 the s2s transport network: WireGuard with the other leg only the same
eth2 the VLAN network to the client site (bridge s2s-NNNN), no IP the project's network
inside Linux bridge of eth2 + a VXLAN tunnel inside WireGuard (one VNI per channel) the same
  • The legs exchange the client's Ethernet frames in a VXLAN tunnel carried in WireGuard over the s2s transport network; the WireGuard keys are made at random per channel and kept nowhere else. Neither leg has an address in the client segment.
  • The project network is an ordinary project VXLAN network on the EVPN fabric; leg 2 is one more VM in it.
  • When the project's shared platform has its own edge cluster, one VM on that cluster (_<org>_<project>_l2brNNNN_<site>) bridges the VLAN and the project network instead of the two legs.
  • In the portal: the project asks for a channel; a resource manager approves it; the support engineer enters the VLAN number (l2_vlans; the VLAN number is also the VNI when it is free); the project chooses the network it connects to (l2_vnets), and changes it later without approval. The channel is made as soon as it has both.

Internal channels

An internal channel joins a project network on one site with a project network on another site (each site one of the project's resource pools). It is added on the project's L2 table without approval (+ Internal channel: a name, the two sites and a network on each); the portal refuses a second internal channel between the same two sites and a channel on a project with resources on one site only.

8. EVPN fabric (project overlays)

  • Data plane — every project subnet is one VXLAN with a unique VNI. On a host, VMs of one VXLAN share a Linux bridge created with the first VM of that network on the host and removed after the last one. Between hosts, frames are carried over the isolated VXLAN transport segment, which has no L3 path outside.
  • Control plane — BGP EVPN, FRR on every payload host. Hosts peer only with three route reflectors (VyOS VMs on the management platform) which distribute every VM's MAC/IP to all hosts; adding or removing a host changes nothing elsewhere.
  • OpenNebula — the VTEP of every VXLAN network must be the host's local IP (VXLAN_TEP = local_ip), on every VNet, on the VNet template projects create networks from, and in the VXLAN driver of the FE nodes.

9. Addresses: IP subnets, service networks, allocation

  • IP subnets (the registry) are defined per site, under Infrastructure on the site's IP subnets tab: subnet, prefix, the max allowed (delegated) prefix, routing (BGP through an exchange, a static gateway, or the organization) and the type, detected from the address (private: corporate network, public: internet). A subnet itself is immutable once added, and it is deleted only while unused (no exchange network carries it and no project block lies in it).
  • The max allowed prefix is the largest block a project may be given from the subnet. It is enforced when addresses are handed out, not only when the subnet is defined: a request for a larger block than it allows is refused. It is at most the subnet's prefix + 2 and never bigger than a /24 (a /20 allows /24, a /26 allows /28), and it can't be set smaller than a block already delegated from the subnet; with nothing delegated, any value from that limit to /31. It is set when the subnet is added (by default the largest allowed) and changed with Edit: for a subnet routed by BGP (ix) it is the only field that can change; for the others the routing and the static gateway can change too. The prefix lengths offered when ordering go down to the most permissive max of the site's subnets of that type.
  • Delegated ranges: on the Networks tab each ix network picks the IP subnets it carries: several, all of its own type; one subnet belongs to one ix network only. The ix network announces them to its remote AS, and they are the pools a project's public and corporate prefixes are carved from (the delegated subnets).
  • Public and corporate subnets are always bound to an IX network, and each IX network serves exactly one shared front end; front ends never share addresses or hosts.
  • Typically each site has a public pool on its cloud-ix and a private pool on its corpnet-ix; a secondary zone uses the pools of its own site. portal-ix pools belong to the ext-managed platforms and are never used for projects.
  • Service networks: inventory networks inside a public pool (for example cloud-services, the first /29 of each cloud-ix pool) carry the services on the management clusters. The portal records them in the address registry as platform services after every inventory change and never gives them to projects.
  • Allocation: customers choose prefix sizes from a list — /31, then /29, /28 … down to the most permissive max of the site's IP subnets of that type (while none is defined: /31, /29, /28, /27, /26) — and, for each public prefix, a base subnet among the internet subnets delegated to the exchange networks of the platform the cluster is on, or auto. On approval the portal takes the first free aligned block, largest first, from the IX networks of the platform the project is placed on — cloud-ix for internet, corpnet-ix for the corporate network: from the chosen base subnet first, then from the others; with auto, cloud pools first. The resource manager can change the main prefix's base subnet (the internet address pool) when approving. The same choice is offered for an additional public subnet and for a new main subnet of an existing project. Everything in the registry, in OpenNebula group metadata and in the address ranges of the platforms' virtual networks counts as taken.
  • The address registry (each site's IP subnets tab) lists the subnets, their use, the assignments (main, additional, corporate, platform services, reserved) and flags overlaps or assignments outside every subnet.

10. Open points

  • Corporate network: address plan vs project subnets (overlaps, NAT), route policy, bandwidth and filtering on the cgw.
  • L2 channels: MTU of the extended segment (VXLAN adds 50 bytes on the s2s transport), redundancy of the legs.
  • Bandwidth step: the reference design shapes in 10 Mbit/s steps; the order form currently accepts 1 Mbit/s steps.