From 72fc1144eee68da629a91e853b057feda9daa3c1 Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Thu, 4 Dec 2025 15:22:14 +0100 Subject: [PATCH 1/4] chore: adding gw node to the existing fabric Signed-off-by: Pau Capdevila --- docs/user-guide/gateway-expand.md | 269 ++++++++++++++++++++++++++++++ 1 file changed, 269 insertions(+) create mode 100644 docs/user-guide/gateway-expand.md diff --git a/docs/user-guide/gateway-expand.md b/docs/user-guide/gateway-expand.md new file mode 100644 index 00000000..0f5efcfd --- /dev/null +++ b/docs/user-guide/gateway-expand.md @@ -0,0 +1,269 @@ +# Adding Gateway Node + +This guide covers adding a gateway node to an existing Fabric deployment. Gateway nodes provide advanced network services +(NAT, PAT, firewalling) by handling traffic between VPCs or between VPCs and externals using Gateway Peerings. + +## Required Resources + +Three types of resources must be created: + +1. **FabNode** - Defines the gateway node at the fabricator level +2. **Gateway** - Defines gateway configuration in the fabric namespace +3. **Connection (type=gateway)** - Establishes uplinks to fabric switches + +## Step 1: Gather Configuration Values + +When adding a gateway to an existing Fabric, automatic IP hydration does not apply. You must manually allocate all IP +addresses and ASNs based on your existing subnet configuration. + +Extract subnet information from `fab.yaml`: + +* `config.control.managementSubnet` - Management network range +* `config.control.dummySubnet` - Dummy interface subnet +* `config.fabric.managementDHCPStart` - DHCP range start +* `config.fabric.protocolSubnet` - Protocol IP subnet +* `config.fabric.vtepSubnet` - VTEP IP subnet +* `config.fabric.fabricSubnet` - Fabric link IP subnet +* `config.gateway.asn` - Gateway ASN (typically 65534) + +Example configuration: + +```yaml +config: + control: + managementSubnet: 172.30.30.0/24 + dummySubnet: 172.30.50.0/24 + fabric: + managementDHCPStart: 172.30.30.10 + protocolSubnet: 172.30.40.0/24 + vtepSubnet: 172.30.41.0/24 + fabricSubnet: 172.30.42.0/24 + gateway: + asn: 65534 +``` + +## Step 2: Allocate IP Addresses + +Select unique IP addresses from the configured subnets: + +### Management IP + +Must be in management subnet, below DHCP start range: + +``` +If DHCP starts at 172.30.30.10, choose from 172.30.30.2-172.30.30.9 +Example: 172.30.30.5/24 +``` + +### Dummy IP + +Unique /31 from dummy subnet: + +``` +Example: 172.30.50.4/31 +``` + +Verify uniqueness by checking existing allocations: + +```bash +kubectl get fabnodes -n fab -o yaml | grep "dummy:" -A 1 +kubectl get controlnodes -n fab -o yaml | grep "dummy:" -A 1 +``` + +### Protocol IP + +Unique /32 from protocol subnet: + +``` +Example: 172.30.40.3/32 +``` + +### VTEP IP + +Unique /32 from VTEP subnet: + +``` +Example: 172.30.41.3/32 +``` + +### Fabric Link IPs + +Unique /31 pairs from fabric subnet (one pair per uplink): + +``` +Example for two uplinks: +- Spine-01 link: Switch 172.30.42.0/31, Gateway 172.30.42.1/31 +- Spine-02 link: Switch 172.30.42.2/31, Gateway 172.30.42.3/31 +``` + +Check existing fabric IPs: + +```bash +kubectl get connections -o yaml | grep "ip:" | grep -v "vtepIP\|protocolIP" +``` + +## Step 3: Create FabNode Resource + +Create `include/gateway-1-node.yaml` in your fabricator working directory: + +```yaml +apiVersion: fabricator.githedgehog.com/v1beta1 +kind: FabNode +metadata: + name: gateway-1 +spec: + roles: + - gateway + bootstrap: + disk: /dev/sda # Adjust to match your hardware + management: + ip: 172.30.30.5/24 # Below DHCP start, from managementSubnet + interface: enp1s0 # Management interface name on gateway hardware + dummy: + ip: 172.30.50.4/31 # Unique /31 from dummySubnet +``` + +## Step 4: Create Gateway Resource + +Create `include/gateway-1.yaml`: + +```yaml +apiVersion: gateway.githedgehog.com/v1alpha1 +kind: Gateway +metadata: + name: gateway-1 + namespace: fab +spec: + asn: 65534 # Must match config.gateway.asn from fab.yaml + protocolIP: 172.30.40.3/32 # Unique /32 from protocolSubnet + vtepIP: 172.30.41.3/32 # Unique /32 from vtepSubnet + interfaces: + enp2s1: {} # Interface for first uplink + enp2s2: {} # Interface for second uplink + workers: 8 # Dataplane worker threads +``` + +Interface names must match physical network interfaces on the gateway node. For kernel driver (default), use standard Linux +interface names (enp2s1, enp2s2, etc.). For DPDK driver, configure PCI addresses in the interfaces section. + +## Step 5: Create Gateway Connections + +Gateway connections establish uplinks to Fabric switches. For spine-leaf topology, connect to spines. For mesh topology, +connect to leaves. + +!!! note "Mesh Topology Limitation" + Gateway connections to TH5 leaf switches are not supported. + +Add to `include/gateway-1.yaml`: + +```yaml +--- +apiVersion: wiring.githedgehog.com/v1beta1 +kind: Connection +metadata: + name: spine-01--gateway--gateway-1 + namespace: default +spec: + gateway: + links: + - switch: + port: spine-01/E1/8 # Physical port on spine switch + ip: 172.30.42.0/31 # Unique /31 from fabricSubnet + gateway: + port: gateway-1/enp2s1 # Must match interface in Gateway spec + ip: 172.30.42.1/31 # Pair with switch IP from fabricSubnet +--- +apiVersion: wiring.githedgehog.com/v1beta1 +kind: Connection +metadata: + name: spine-02--gateway--gateway-1 + namespace: default +spec: + gateway: + links: + - switch: + port: spine-02/E1/8 + ip: 172.30.42.2/31 + gateway: + port: gateway-1/enp2s2 + ip: 172.30.42.3/31 +``` + +Key points: + +* Each link requires a /31 IP pair from fabricSubnet +* Gateway port must match an interface defined in the Gateway spec +* Switch port must be a valid port on the switch (see [Switch Profiles](profiles.md)) +* Connection name follows pattern: `--gateway--` + +## Step 6: Validate Configuration + +From the fabricator working directory: + +```bash +hhfab validate +``` + +Fix any validation errors before proceeding. + +## Step 7: Build Gateway Installer + +Build the gateway node installer: + +```bash +hhfab build --gateways +``` + +This creates a bootable ISO at `result/gateway-1.iso`. + +## Step 8: Install Gateway Node + +1. Boot the gateway node from the generated ISO using virtual media +2. Configure server for UEFI boot without secure boot +3. Installation proceeds automatically +4. System reboots automatically after installation +5. Remove installation media during reboot + +For detailed installation steps, see [Install Gateway Node](../install-upgrade/install.md#install-gateway-node). + +## Step 9: Verify Installation + +Check gateway node status: + +```bash +kubectl get nodes +``` + +Expected output: + +``` +NAME STATUS ROLES AGE VERSION +control-1 Ready control-plane,etcd 2d2h v1.34.1+k3s1 +gateway-1 Ready 5m v1.34.1+k3s1 +``` + +Check Gateway resource: + +```bash +kubectl get gateway -n fab +``` + +Expected output: + +``` +NAME ASN PROTOCOL IP VTEP IP AGE +gateway-1 65534 172.30.40.3/32 172.30.41.3/32 5m +``` + +Check FabNode: + +```bash +kubectl get fabnode -n fab +``` + +Expected output: + +``` +NAME ROLES MGMTIP AGE +gateway-1 ["gateway"] 172.30.0.6/21 5m +``` From e5ea1800d4bb64fc7207870b8fa74d4105bfd34c Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Thu, 4 Dec 2025 15:57:02 +0100 Subject: [PATCH 2/4] chore: adding gw section improvements Signed-off-by: Pau Capdevila --- docs/user-guide/gateway-add.md | 310 ++++++++++++++++++++++++++++++ docs/user-guide/gateway-expand.md | 269 -------------------------- 2 files changed, 310 insertions(+), 269 deletions(-) create mode 100644 docs/user-guide/gateway-add.md delete mode 100644 docs/user-guide/gateway-expand.md diff --git a/docs/user-guide/gateway-add.md b/docs/user-guide/gateway-add.md new file mode 100644 index 00000000..fd76361f --- /dev/null +++ b/docs/user-guide/gateway-add.md @@ -0,0 +1,310 @@ +# Adding Gateway Node + +Adding a gateway node to an existing Fabric. Gateway nodes provide advanced network services (NAT, PAT, firewalling) by +handling traffic between VPCs or between VPCs and externals using Gateway Peerings. + +## Understanding IP Allocation and Hydration + +When building a new Fabric from scratch, IP addresses and ASNs can be automatically allocated through **hydration** - a process +that fills in missing IP addresses and ASNs based on configured subnets. However, **hydration does not work when adding resources +to an already running Fabric**. The Kubernetes admission webhooks require all IP addresses to be present and valid when creating +resources, preventing the use of hydration for incremental additions. + +This means when adding a gateway to an existing deployment, you must manually: + +* Select unique IP addresses from configured subnets +* Determine BGP ASNs for gateway neighbors +* Ensure no IP or ASN conflicts with existing resources + +## Required Resources + +Two types of resources must be created in the running cluster: + +1. **Connection (type=gateway)** - Establishes uplinks to Fabric switches (must be created first) +2. **Gateway** - Defines gateway configuration in the fabric namespace (created after Connections) + +Additionally, for building the gateway installer: + +3. **FabNode** - Defines the gateway node at the fabricator level (required for `hhfab build --gateways`) + +## Step 1: Gather Configuration Values + +Extract subnet information from the Fabricator configuration: + +* `config.control.managementSubnet` - Management network range +* `config.control.dummySubnet` - Dummy interface subnet +* `config.fabric.managementDHCPStart` - DHCP range start +* `config.fabric.protocolSubnet` - Protocol IP subnet +* `config.fabric.vtepSubnet` - VTEP IP subnet +* `config.fabric.fabricSubnet` - Fabric link IP subnet +* `config.gateway.asn` - Gateway ASN (typically 65534) + +Example configuration: + +```yaml +config: + control: + managementSubnet: 172.30.30.0/24 + dummySubnet: 172.30.50.0/24 + fabric: + managementDHCPStart: 172.30.30.10 + protocolSubnet: 172.30.40.0/24 + vtepSubnet: 172.30.41.0/24 + fabricSubnet: 172.30.42.0/24 + gateway: + asn: 65534 +``` + +## Step 2: Allocate IP Addresses + +Select unique IP addresses from the configured subnets: + +### Management IP + +The management IP provides connectivity to the gateway node for SSH access and cluster communication. It must be: + +* In the management subnet (`config.control.managementSubnet`) +* Below the DHCP start range (`config.fabric.managementDHCPStart`) +* Not the control VIP or already assigned to another node + +To safely choose a management IP, check existing allocations: + +```bash +# List all control and gateway nodes with their management IPs +kubectl get controlnodes -n fab -o yaml | grep "ip:" -A 1 +kubectl get fabnodes -n fab -o yaml | grep "ip:" -A 1 + +# Check the control VIP +kubectl get fabricator -n fab -o yaml | grep controlVIP +``` + +Example: If management subnet is 172.30.0.0/21 and DHCP starts at 172.30.4.0, choose from 172.30.0.2 through 172.30.3.254 +(avoiding 172.30.0.1 which is typically the control VIP). + +### Dummy IP + +The dummy IP is used for internal K3s cluster communication between control and gateway nodes. Each node requires a unique /31 +subnet from the dummy subnet range. The /31 provides a point-to-point link between the node and the control plane. + +Verify uniqueness by checking existing allocations: + +```bash +kubectl get fabnodes -n fab -o yaml | grep "dummy:" -A 1 +kubectl get controlnodes -n fab -o yaml | grep "dummy:" -A 1 +``` + +Example: If dummy subnet is 172.30.90.0/24, and control-1 uses 172.30.90.0/31, choose 172.30.90.2/31 for gateway-1. + +### Protocol IP + +Unique /32 from protocol subnet: + +``` +Example: 172.30.40.3/32 +``` + +### VTEP IP + +Unique /32 from VTEP subnet: + +``` +Example: 172.30.41.3/32 +``` + +### Fabric Link IPs + +Unique /31 pairs from fabric subnet (one pair per uplink): + +``` +Example for two uplinks: +- Spine-01 link: Switch 172.30.42.0/31, Gateway 172.30.42.1/31 +- Spine-02 link: Switch 172.30.42.2/31, Gateway 172.30.42.3/31 +``` + +Check existing fabric IPs: + +```bash +kubectl get connections -o yaml | grep "ip:" | grep -v "vtepIP\|protocolIP" +``` + +## Step 3: Create Gateway Connections + +Gateway connections must be created before the Gateway resource. Connections establish uplinks to Fabric switches and define +the IP addresses used by gateway interfaces. For spine-leaf topology, connect to spines. For mesh topology, connect to leaves. + +!!! note "Mesh Topology Limitation" + Gateway connections to TH5 leaf switches are not supported. + +Apply the Connection resources to the cluster: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: wiring.githedgehog.com/v1beta1 +kind: Connection +metadata: + name: leaf-01--gateway--gateway-1 + namespace: default +spec: + gateway: + links: + - switch: + port: leaf-01/E1/8 + ip: 172.30.128.0/31 + gateway: + port: gateway-1/enp2s1 + ip: 172.30.128.1/31 +--- +apiVersion: wiring.githedgehog.com/v1beta1 +kind: Connection +metadata: + name: leaf-02--gateway--gateway-1 + namespace: default +spec: + gateway: + links: + - switch: + port: leaf-02/E1/9 + ip: 172.30.128.10/31 + gateway: + port: gateway-1/enp2s2 + ip: 172.30.128.11/31 +EOF +``` + +Key points: + +* Each link requires a /31 IP pair from fabricSubnet +* Gateway port names will be referenced in the Gateway resource +* Switch port must be a valid port on the switch (see [Switch Profiles](profiles.md)) +* Connection name follows pattern: `--gateway--` +* Connections must be in `default` namespace + +## Step 4: Create Gateway Resource + +After Connections are created, create the Gateway resource. The Gateway resource requires interface IPs and BGP neighbor +information derived from the Connections. + +First, determine the BGP ASNs of the switches you're connecting to: + +```bash +kubectl get switches -o yaml | grep -E "name: (leaf|spine)" -A 2 | grep asn +``` + +Then apply the Gateway resource: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: gateway.githedgehog.com/v1alpha1 +kind: Gateway +metadata: + name: gateway-1 + namespace: fab +spec: + asn: 65534 + protocolIP: 172.30.8.3/32 + vtepIP: 172.30.12.3/32 + vtepMAC: CA:FE:BA:BE:01:02 + interfaces: + enp2s1: + ips: + - 172.30.128.1/31 + enp2s2: + ips: + - 172.30.128.11/31 + neighbors: + - asn: 65101 + ip: 172.30.128.0 + source: enp2s1 + - asn: 65102 + ip: 172.30.128.10 + source: enp2s2 + workers: 8 +EOF +``` + +Key fields: + +* `asn` - Must match `config.gateway.asn` from Fabricator configuration (typically 65534) +* `protocolIP` - Unique /32 from protocol subnet for BGP router ID +* `vtepIP` - Unique /32 from VTEP subnet for VXLAN tunnel endpoint +* `vtepMAC` - MAC address for VTEP (use format CA:FE:BA:BE:XX:XX with unique last two octets) +* `interfaces` - Each interface must have IPs matching the gateway side of the Connections +* `neighbors` - BGP neighbors with switch ASNs and IPs from switch side of Connections +* `workers` - Number of dataplane worker threads (typically 8) + +The interface names (enp2s1, enp2s2) must match physical network interfaces on the gateway hardware. For kernel driver +(default), use standard Linux interface names. For DPDK driver, configure PCI addresses. + +## Installing the Gateway Node + +If the gateway node hardware is not yet installed, you need to create a FabNode resource and build the installer ISO. If the +gateway node is already running and joined to the cluster, skip to the Verification section. + +### Create FabNode Resource + +The FabNode resource defines the gateway node for the fabricator and is required to build the installer. Add it to your +fabricator configuration in `include/gateway-1-node.yaml`: + +```yaml +apiVersion: fabricator.githedgehog.com/v1beta1 +kind: FabNode +metadata: + name: gateway-1 + namespace: fab +spec: + roles: + - gateway + bootstrap: + disk: /dev/sda + management: + ip: 172.30.0.6/21 + interface: enp2s0 + dummy: + ip: 172.30.90.2/31 +``` + +### Build and Install + +From the fabricator working directory: + +```bash +hhfab build --gateways +``` + +This creates a bootable ISO at `result/gateway-1.iso`. Follow the [Install Gateway Node](../install-upgrade/install.md#install-gateway-node) +guide to boot the gateway hardware from the ISO and complete installation. + +## Verification + +After creating the Gateway resource, verify it was created successfully: + +```bash +kubectl get gateway -n fab -o wide +``` + +Expected output: + +``` +NAME PROTOIP VTEPIP AGE +gateway-1 172.30.8.3/32 172.30.12.3/32 10s +``` + +Check that BGP sessions are established (may take a few moments): + +```bash +kubectl get gateway -n fab -o yaml | grep -A 5 "status:" +``` + +Verify the gateway node joined the Kubernetes cluster: + +```bash +kubectl get nodes +``` + +Expected output showing the gateway node: + +``` +NAME STATUS ROLES AGE VERSION +control-1 Ready control-plane,etcd 2d2h v1.34.1+k3s1 +gateway-1 Ready 5m v1.34.1+k3s1 +``` diff --git a/docs/user-guide/gateway-expand.md b/docs/user-guide/gateway-expand.md deleted file mode 100644 index 0f5efcfd..00000000 --- a/docs/user-guide/gateway-expand.md +++ /dev/null @@ -1,269 +0,0 @@ -# Adding Gateway Node - -This guide covers adding a gateway node to an existing Fabric deployment. Gateway nodes provide advanced network services -(NAT, PAT, firewalling) by handling traffic between VPCs or between VPCs and externals using Gateway Peerings. - -## Required Resources - -Three types of resources must be created: - -1. **FabNode** - Defines the gateway node at the fabricator level -2. **Gateway** - Defines gateway configuration in the fabric namespace -3. **Connection (type=gateway)** - Establishes uplinks to fabric switches - -## Step 1: Gather Configuration Values - -When adding a gateway to an existing Fabric, automatic IP hydration does not apply. You must manually allocate all IP -addresses and ASNs based on your existing subnet configuration. - -Extract subnet information from `fab.yaml`: - -* `config.control.managementSubnet` - Management network range -* `config.control.dummySubnet` - Dummy interface subnet -* `config.fabric.managementDHCPStart` - DHCP range start -* `config.fabric.protocolSubnet` - Protocol IP subnet -* `config.fabric.vtepSubnet` - VTEP IP subnet -* `config.fabric.fabricSubnet` - Fabric link IP subnet -* `config.gateway.asn` - Gateway ASN (typically 65534) - -Example configuration: - -```yaml -config: - control: - managementSubnet: 172.30.30.0/24 - dummySubnet: 172.30.50.0/24 - fabric: - managementDHCPStart: 172.30.30.10 - protocolSubnet: 172.30.40.0/24 - vtepSubnet: 172.30.41.0/24 - fabricSubnet: 172.30.42.0/24 - gateway: - asn: 65534 -``` - -## Step 2: Allocate IP Addresses - -Select unique IP addresses from the configured subnets: - -### Management IP - -Must be in management subnet, below DHCP start range: - -``` -If DHCP starts at 172.30.30.10, choose from 172.30.30.2-172.30.30.9 -Example: 172.30.30.5/24 -``` - -### Dummy IP - -Unique /31 from dummy subnet: - -``` -Example: 172.30.50.4/31 -``` - -Verify uniqueness by checking existing allocations: - -```bash -kubectl get fabnodes -n fab -o yaml | grep "dummy:" -A 1 -kubectl get controlnodes -n fab -o yaml | grep "dummy:" -A 1 -``` - -### Protocol IP - -Unique /32 from protocol subnet: - -``` -Example: 172.30.40.3/32 -``` - -### VTEP IP - -Unique /32 from VTEP subnet: - -``` -Example: 172.30.41.3/32 -``` - -### Fabric Link IPs - -Unique /31 pairs from fabric subnet (one pair per uplink): - -``` -Example for two uplinks: -- Spine-01 link: Switch 172.30.42.0/31, Gateway 172.30.42.1/31 -- Spine-02 link: Switch 172.30.42.2/31, Gateway 172.30.42.3/31 -``` - -Check existing fabric IPs: - -```bash -kubectl get connections -o yaml | grep "ip:" | grep -v "vtepIP\|protocolIP" -``` - -## Step 3: Create FabNode Resource - -Create `include/gateway-1-node.yaml` in your fabricator working directory: - -```yaml -apiVersion: fabricator.githedgehog.com/v1beta1 -kind: FabNode -metadata: - name: gateway-1 -spec: - roles: - - gateway - bootstrap: - disk: /dev/sda # Adjust to match your hardware - management: - ip: 172.30.30.5/24 # Below DHCP start, from managementSubnet - interface: enp1s0 # Management interface name on gateway hardware - dummy: - ip: 172.30.50.4/31 # Unique /31 from dummySubnet -``` - -## Step 4: Create Gateway Resource - -Create `include/gateway-1.yaml`: - -```yaml -apiVersion: gateway.githedgehog.com/v1alpha1 -kind: Gateway -metadata: - name: gateway-1 - namespace: fab -spec: - asn: 65534 # Must match config.gateway.asn from fab.yaml - protocolIP: 172.30.40.3/32 # Unique /32 from protocolSubnet - vtepIP: 172.30.41.3/32 # Unique /32 from vtepSubnet - interfaces: - enp2s1: {} # Interface for first uplink - enp2s2: {} # Interface for second uplink - workers: 8 # Dataplane worker threads -``` - -Interface names must match physical network interfaces on the gateway node. For kernel driver (default), use standard Linux -interface names (enp2s1, enp2s2, etc.). For DPDK driver, configure PCI addresses in the interfaces section. - -## Step 5: Create Gateway Connections - -Gateway connections establish uplinks to Fabric switches. For spine-leaf topology, connect to spines. For mesh topology, -connect to leaves. - -!!! note "Mesh Topology Limitation" - Gateway connections to TH5 leaf switches are not supported. - -Add to `include/gateway-1.yaml`: - -```yaml ---- -apiVersion: wiring.githedgehog.com/v1beta1 -kind: Connection -metadata: - name: spine-01--gateway--gateway-1 - namespace: default -spec: - gateway: - links: - - switch: - port: spine-01/E1/8 # Physical port on spine switch - ip: 172.30.42.0/31 # Unique /31 from fabricSubnet - gateway: - port: gateway-1/enp2s1 # Must match interface in Gateway spec - ip: 172.30.42.1/31 # Pair with switch IP from fabricSubnet ---- -apiVersion: wiring.githedgehog.com/v1beta1 -kind: Connection -metadata: - name: spine-02--gateway--gateway-1 - namespace: default -spec: - gateway: - links: - - switch: - port: spine-02/E1/8 - ip: 172.30.42.2/31 - gateway: - port: gateway-1/enp2s2 - ip: 172.30.42.3/31 -``` - -Key points: - -* Each link requires a /31 IP pair from fabricSubnet -* Gateway port must match an interface defined in the Gateway spec -* Switch port must be a valid port on the switch (see [Switch Profiles](profiles.md)) -* Connection name follows pattern: `--gateway--` - -## Step 6: Validate Configuration - -From the fabricator working directory: - -```bash -hhfab validate -``` - -Fix any validation errors before proceeding. - -## Step 7: Build Gateway Installer - -Build the gateway node installer: - -```bash -hhfab build --gateways -``` - -This creates a bootable ISO at `result/gateway-1.iso`. - -## Step 8: Install Gateway Node - -1. Boot the gateway node from the generated ISO using virtual media -2. Configure server for UEFI boot without secure boot -3. Installation proceeds automatically -4. System reboots automatically after installation -5. Remove installation media during reboot - -For detailed installation steps, see [Install Gateway Node](../install-upgrade/install.md#install-gateway-node). - -## Step 9: Verify Installation - -Check gateway node status: - -```bash -kubectl get nodes -``` - -Expected output: - -``` -NAME STATUS ROLES AGE VERSION -control-1 Ready control-plane,etcd 2d2h v1.34.1+k3s1 -gateway-1 Ready 5m v1.34.1+k3s1 -``` - -Check Gateway resource: - -```bash -kubectl get gateway -n fab -``` - -Expected output: - -``` -NAME ASN PROTOCOL IP VTEP IP AGE -gateway-1 65534 172.30.40.3/32 172.30.41.3/32 5m -``` - -Check FabNode: - -```bash -kubectl get fabnode -n fab -``` - -Expected output: - -``` -NAME ROLES MGMTIP AGE -gateway-1 ["gateway"] 172.30.0.6/21 5m -``` From 03f5d6aafbf16dc59047722c8e0564f849cb6cfa Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Thu, 4 Dec 2025 16:43:05 +0100 Subject: [PATCH 3/4] chore: fix adding gw examples Signed-off-by: Pau Capdevila --- docs/user-guide/gateway-add.md | 57 +++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 25 deletions(-) diff --git a/docs/user-guide/gateway-add.md b/docs/user-guide/gateway-add.md index fd76361f..4aded330 100644 --- a/docs/user-guide/gateway-add.md +++ b/docs/user-guide/gateway-add.md @@ -1,6 +1,6 @@ # Adding Gateway Node -Adding a gateway node to an existing Fabric. Gateway nodes provide advanced network services (NAT, PAT, firewalling) by +This section covers adding a gateway node to an existing Fabric. Gateway nodes provide advanced network services (NAT, PAT, firewalling) by handling traffic between VPCs or between VPCs and externals using Gateway Peerings. ## Understanding IP Allocation and Hydration @@ -69,13 +69,22 @@ The management IP provides connectivity to the gateway node for SSH access and c To safely choose a management IP, check existing allocations: +List all control nodes with their management IPs: + +```bash +kubectl get controlnodes -n fab -o custom-columns=NAME:.metadata.name,MGMT_IP:.spec.management.ip,DUMMY_IP:.spec.dummy.ip +``` + +List all gateway nodes with their management IPs: + ```bash -# List all control and gateway nodes with their management IPs -kubectl get controlnodes -n fab -o yaml | grep "ip:" -A 1 -kubectl get fabnodes -n fab -o yaml | grep "ip:" -A 1 +kubectl get fabnodes -n fab -o custom-columns=NAME:.metadata.name,MGMT_IP:.spec.management.ip,DUMMY_IP:.spec.dummy.ip +``` + +Check the control VIP: -# Check the control VIP -kubectl get fabricator -n fab -o yaml | grep controlVIP +```bash +kubectl get fabricator -n fab -o jsonpath='{.items[0].spec.config.control.controlVIP}' ``` Example: If management subnet is 172.30.0.0/21 and DHCP starts at 172.30.4.0, choose from 172.30.0.2 through 172.30.3.254 @@ -88,43 +97,44 @@ subnet from the dummy subnet range. The /31 provides a point-to-point link betwe Verify uniqueness by checking existing allocations: +Check control node dummy IPs: + +```bash +kubectl get controlnodes -n fab -o custom-columns=NAME:.metadata.name,DUMMY_IP:.spec.dummy.ip +``` + +Check gateway node dummy IPs: + ```bash -kubectl get fabnodes -n fab -o yaml | grep "dummy:" -A 1 -kubectl get controlnodes -n fab -o yaml | grep "dummy:" -A 1 +kubectl get fabnodes -n fab -o custom-columns=NAME:.metadata.name,DUMMY_IP:.spec.dummy.ip ``` Example: If dummy subnet is 172.30.90.0/24, and control-1 uses 172.30.90.0/31, choose 172.30.90.2/31 for gateway-1. ### Protocol IP -Unique /32 from protocol subnet: +Unique /32 from protocol subnet. -``` -Example: 172.30.40.3/32 -``` +Example: 172.30.8.3/32 ### VTEP IP -Unique /32 from VTEP subnet: +Unique /32 from VTEP subnet. -``` -Example: 172.30.41.3/32 -``` +Example: 172.30.12.3/32 ### Fabric Link IPs -Unique /31 pairs from fabric subnet (one pair per uplink): +Unique /31 pairs from fabric subnet (one pair per uplink). -``` Example for two uplinks: - Spine-01 link: Switch 172.30.42.0/31, Gateway 172.30.42.1/31 - Spine-02 link: Switch 172.30.42.2/31, Gateway 172.30.42.3/31 -``` -Check existing fabric IPs: +Check existing gateway connection IPs: ```bash -kubectl get connections -o yaml | grep "ip:" | grep -v "vtepIP\|protocolIP" +kubectl get connections -o custom-columns=NAME:.metadata.name,SWITCH_IP:.spec.gateway.links[0].switch.ip,GW_IP:.spec.gateway.links[0].gateway.ip 2>/dev/null | grep gateway ``` ## Step 3: Create Gateway Connections @@ -132,9 +142,6 @@ kubectl get connections -o yaml | grep "ip:" | grep -v "vtepIP\|protocolIP" Gateway connections must be created before the Gateway resource. Connections establish uplinks to Fabric switches and define the IP addresses used by gateway interfaces. For spine-leaf topology, connect to spines. For mesh topology, connect to leaves. -!!! note "Mesh Topology Limitation" - Gateway connections to TH5 leaf switches are not supported. - Apply the Connection resources to the cluster: ```bash @@ -187,7 +194,7 @@ information derived from the Connections. First, determine the BGP ASNs of the switches you're connecting to: ```bash -kubectl get switches -o yaml | grep -E "name: (leaf|spine)" -A 2 | grep asn +kubectl get switches -o custom-columns=NAME:.metadata.name,ASN:.spec.asn ``` Then apply the Gateway resource: From a997dc703d6d4da537a1e41ff5a862a43d3786a6 Mon Sep 17 00:00:00 2001 From: Pau Capdevila Date: Thu, 11 Dec 2025 19:37:21 +0100 Subject: [PATCH 4/4] chore: review adding gw section Signed-off-by: Pau Capdevila --- docs/user-guide/gateway-add.md | 155 +++++++++++++++++++-------------- 1 file changed, 89 insertions(+), 66 deletions(-) diff --git a/docs/user-guide/gateway-add.md b/docs/user-guide/gateway-add.md index 4aded330..6cdb97d8 100644 --- a/docs/user-guide/gateway-add.md +++ b/docs/user-guide/gateway-add.md @@ -25,7 +25,7 @@ Two types of resources must be created in the running cluster: Additionally, for building the gateway installer: -3. **FabNode** - Defines the gateway node at the fabricator level (required for `hhfab build --gateways`) +3. **FabNode** - Defines the gateway node at the fabricator level (required for `hhfab build --gateways`). See [Install Gateway Node](../install-upgrade/install.md#install-gateway-node) for details on installing the bare metal machine. ## Step 1: Gather Configuration Values @@ -44,13 +44,14 @@ Example configuration: ```yaml config: control: - managementSubnet: 172.30.30.0/24 - dummySubnet: 172.30.50.0/24 + managementSubnet: 172.30.0.0/21 + controlVIP: 172.30.0.1 + dummySubnet: 172.30.90.0/24 fabric: - managementDHCPStart: 172.30.30.10 - protocolSubnet: 172.30.40.0/24 - vtepSubnet: 172.30.41.0/24 - fabricSubnet: 172.30.42.0/24 + managementDHCPStart: 172.30.4.0 + protocolSubnet: 172.30.8.0/24 + vtepSubnet: 172.30.12.0/24 + fabricSubnet: 172.30.128.0/24 gateway: asn: 65534 ``` @@ -92,8 +93,9 @@ Example: If management subnet is 172.30.0.0/21 and DHCP starts at 172.30.4.0, ch ### Dummy IP -The dummy IP is used for internal K3s cluster communication between control and gateway nodes. Each node requires a unique /31 -subnet from the dummy subnet range. The /31 provides a point-to-point link between the node and the control plane. +The dummy IP is used for internal K3s cluster communication between control and gateway nodes. We use dummy network devices +and IPs to ensure K3s has a stable default network route. Each node requires a unique /31 subnet from the dummy subnet range. +The /31 provides a point-to-point link between the node and the control plane. Verify uniqueness by checking existing allocations: @@ -113,23 +115,23 @@ Example: If dummy subnet is 172.30.90.0/24, and control-1 uses 172.30.90.0/31, c ### Protocol IP -Unique /32 from protocol subnet. +Unique /32 from protocol subnet (`config.fabric.protocolSubnet`). -Example: 172.30.8.3/32 +Example: 172.30.8.3/32 (from 172.30.8.0/24) ### VTEP IP -Unique /32 from VTEP subnet. +Unique /32 from VTEP subnet (`config.fabric.vtepSubnet`). -Example: 172.30.12.3/32 +Example: 172.30.12.3/32 (from 172.30.12.0/24) ### Fabric Link IPs -Unique /31 pairs from fabric subnet (one pair per uplink). +Unique /31 pairs from fabric subnet (`config.fabric.fabricSubnet`, one pair per uplink). -Example for two uplinks: -- Spine-01 link: Switch 172.30.42.0/31, Gateway 172.30.42.1/31 -- Spine-02 link: Switch 172.30.42.2/31, Gateway 172.30.42.3/31 +Example for two uplinks (from 172.30.128.0/24): +- Leaf-01 link: Switch 172.30.128.0/31, Gateway 172.30.128.1/31 +- Leaf-02 link: Switch 172.30.128.10/31, Gateway 172.30.128.11/31 Check existing gateway connection IPs: @@ -142,49 +144,59 @@ kubectl get connections -o custom-columns=NAME:.metadata.name,SWITCH_IP:.spec.ga Gateway connections must be created before the Gateway resource. Connections establish uplinks to Fabric switches and define the IP addresses used by gateway interfaces. For spine-leaf topology, connect to spines. For mesh topology, connect to leaves. -Apply the Connection resources to the cluster: +Create a YAML file with your Connection resources: -```bash -kubectl apply -f - <<'EOF' +```{.yaml .annotate linenums="1" title="gateway-connections.yaml"} apiVersion: wiring.githedgehog.com/v1beta1 kind: Connection metadata: - name: leaf-01--gateway--gateway-1 + name: leaf-01--gateway--gateway-1 # (1)! namespace: default spec: gateway: links: - switch: - port: leaf-01/E1/8 - ip: 172.30.128.0/31 + port: leaf-01/E1/8 # (2)! + ip: 172.30.128.0/31 # (3)! gateway: - port: gateway-1/enp2s1 - ip: 172.30.128.1/31 + port: gateway-1/enp2s1 # (4)! + ip: 172.30.128.1/31 # (5)! --- apiVersion: wiring.githedgehog.com/v1beta1 kind: Connection metadata: - name: leaf-02--gateway--gateway-1 + name: leaf-02--gateway--gateway-1 # (6)! namespace: default spec: gateway: links: - switch: - port: leaf-02/E1/9 - ip: 172.30.128.10/31 + port: leaf-02/E1/9 # (7)! + ip: 172.30.128.10/31 # (8)! gateway: - port: gateway-1/enp2s2 - ip: 172.30.128.11/31 -EOF + port: gateway-1/enp2s2 # (9)! + ip: 172.30.128.11/31 # (10)! ``` -Key points: +1. Connection name - we typically use the pattern `--gateway--` but any name will work +2. Switch port must be a valid port on the switch (see [Switch Profiles](profiles.md)) +3. Allocate a unique /31 IP from `config.fabric.fabricSubnet` for the switch side +4. Gateway physical network interface name (will be referenced in Gateway resource) +5. Allocate the paired /31 IP from `config.fabric.fabricSubnet` for the gateway side +6. Connection name for the second uplink +7. Switch port for the second uplink +8. Allocate another unique /31 IP pair from `config.fabric.fabricSubnet` for the switch side +9. Gateway physical network interface name for the second uplink +10. Allocate the paired /31 IP for the gateway side + +Apply the Connection resources to the cluster: + +```bash +kubectl apply -f gateway-connections.yaml +``` -* Each link requires a /31 IP pair from fabricSubnet -* Gateway port names will be referenced in the Gateway resource -* Switch port must be a valid port on the switch (see [Switch Profiles](profiles.md)) -* Connection name follows pattern: `--gateway--` -* Connections must be in `default` namespace +!!! note + Connections must be created in the `default` namespace and must be applied before the Gateway resource. ## Step 4: Create Gateway Resource @@ -197,50 +209,61 @@ First, determine the BGP ASNs of the switches you're connecting to: kubectl get switches -o custom-columns=NAME:.metadata.name,ASN:.spec.asn ``` -Then apply the Gateway resource: +Create a YAML file with your Gateway resource: -```bash -kubectl apply -f - <<'EOF' +```{.yaml .annotate linenums="1" title="gateway.yaml"} apiVersion: gateway.githedgehog.com/v1alpha1 kind: Gateway metadata: name: gateway-1 namespace: fab spec: - asn: 65534 - protocolIP: 172.30.8.3/32 - vtepIP: 172.30.12.3/32 - vtepMAC: CA:FE:BA:BE:01:02 + asn: 65534 # (1)! + protocolIP: 172.30.8.3/32 # (2)! + vtepIP: 172.30.12.3/32 # (3)! + vtepMAC: 02:00:00:00:01:02 # (4)! interfaces: - enp2s1: + enp2s1: # (5)! ips: - - 172.30.128.1/31 - enp2s2: + - 172.30.128.1/31 # (6)! + enp2s2: # (7)! ips: - - 172.30.128.11/31 + - 172.30.128.11/31 # (8)! neighbors: - - asn: 65101 - ip: 172.30.128.0 - source: enp2s1 - - asn: 65102 - ip: 172.30.128.10 - source: enp2s2 - workers: 8 -EOF + - asn: 65101 # (9)! + ip: 172.30.128.0 # (10)! + source: enp2s1 # (11)! + - asn: 65102 # (12)! + ip: 172.30.128.10 # (13)! + source: enp2s2 # (14)! + workers: 8 # (15)! ``` -Key fields: +1. Must match `config.gateway.asn` from Fabricator configuration (use `kubectl get fabricator -n fab -o jsonpath='{.items[0].spec.config.gateway.asn}'`) +2. Allocate a unique /32 from `config.fabric.protocolSubnet` (BGP router ID) +3. Allocate a unique /32 from `config.fabric.vtepSubnet` (VXLAN tunnel endpoint) +4. MAC address for VTEP - any valid MAC address (e.g., 02:00:00:00:01:02) +5. Interface name must match the physical network interface on the gateway hardware and the `gateway.port` in the Connection resource +6. Must match the `gateway.ip` from the corresponding Connection resource (gateway-connections.yaml line 11) +7. Interface name for the second uplink +8. Must match the `gateway.ip` from the corresponding Connection resource (gateway-connections.yaml line 26) +9. Switch ASN - get from `kubectl get switches -o custom-columns=NAME:.metadata.name,ASN:.spec.asn` for leaf-01 +10. Must match the `switch.ip` from the corresponding Connection resource (gateway-connections.yaml line 10) +11. Must match the interface name defined above +12. Switch ASN for the second uplink - get from `kubectl get switches` for leaf-02 +13. Must match the `switch.ip` from the corresponding Connection resource (gateway-connections.yaml line 25) +14. Must match the interface name for the second uplink +15. Number of dataplane worker threads (typically 8) + +Apply the Gateway resource to the cluster: -* `asn` - Must match `config.gateway.asn` from Fabricator configuration (typically 65534) -* `protocolIP` - Unique /32 from protocol subnet for BGP router ID -* `vtepIP` - Unique /32 from VTEP subnet for VXLAN tunnel endpoint -* `vtepMAC` - MAC address for VTEP (use format CA:FE:BA:BE:XX:XX with unique last two octets) -* `interfaces` - Each interface must have IPs matching the gateway side of the Connections -* `neighbors` - BGP neighbors with switch ASNs and IPs from switch side of Connections -* `workers` - Number of dataplane worker threads (typically 8) +```bash +kubectl apply -f gateway.yaml +``` -The interface names (enp2s1, enp2s2) must match physical network interfaces on the gateway hardware. For kernel driver -(default), use standard Linux interface names. For DPDK driver, configure PCI addresses. +!!! note + The interface names (enp2s1, enp2s2) must match physical network interfaces on the gateway hardware. For kernel driver + (default), use standard Linux interface names. For DPDK driver, configure PCI addresses. ## Installing the Gateway Node