Public IPv4 addresses are becoming less and less available, IPv6 addresses will still not be widespread or dynamically assigned to end customers in 2024 and many applications still communicate with each other unencrypted today.
A wide variety of technologies and methods can be used to securely connect different networks. I would like to present one such approach here:
Dynamic site to site networking using WireGuard and BGP
Why should I do this?
There are many different reasons to use these technologies:
- Security: WireGuard is a modern VPN protocol that is easy to set up and secure by default, directly integrated into the Linux kernel.
- Performance: WireGuard is designed to be fast and efficient, with low latency and high throughput.
- Flexibility: With dynamic routing, you can easily add new networks or change existing ones.
- Redundancy: If a connection fails, the network can automatically switch to another path.
- High granularity: You can control which networks are allowed to communicate with each other.
- No NAT: You can avoid NAT and use your private IP addresses to communicate with other networks.
How does it work?
In this example, I will use a simple setup with three networks:
- Network A: The primary site with a public IP address, data center and private networks.
- Network B: A remote site with a static IP address, a few servers and clients.
- Network C: A remote site with a dynamic non-public IP address and clients.
For the sites we assume the following networks:
- Network A:
- 10.10.0.0/16 – Data center network
- 2001:db8:10::/48 – Data center network (IPv6)
- 10.20.0.0/16 – Office network
- 2001:db8:20::/48 – Office network (IPv6)
- Network B:
- 10.100.0.0/16 – Location Network B
- 2001:db8:100::/48 – Location Network B (IPv6)
- Network C:
- 10.101.0.0/16 – Location Network C
- 2001:db8:101::/48 – Location Network C (IPv6)
- Transfer networks:
- 172.31.10.0/24 – /32 subnets for direct connections between peers
- 2001:db8::/32 – /112 subnets for direct connections between peers (use your own IPv6 prefix and /64 subnet if possible)
We will use WireGuard to create a secure tunnel between the sites and FRR (Free Range Routing) to provide dynamic routing between them using BGP (Border Gateway Protocol). This allows us to automatically exchange routing information between the sites and use the best path for each connection. We can also use BGP to filter which networks are allowed to communicate with each other.
With this setup, every network can communicate with each other directly, without the need for NAT or port forwarding. If one connection fails, the network can automatically switch to another path. We can also easily add new networks or change existing ones without reconfiguring all routers.
First, all dependencies need to be installed on the routers. For Ubuntu 24.04 LTS, you can use the following commands:
| |
WireGuard Setup
First, we need to set up WireGuard on all routers. We will use a simple configuration with a single interface and a pre-shared key for authentication. You can find more information about WireGuard in the official documentation.
In this setup, we will use Ubuntu 24.04 LTS as the operating system for all routers. You can use any other Linux distribution that supports WireGuard. (This setup also works in LXC containers or virtual machines like Proxmox.)
To generate a new private key for your local server, you can use the following command:
| |
You can then use this private key to generate a public key for the remote server:
| |
To generate a preshared key which is used on the local and remote server, you can use the following command:
| |
Here is an example configuration for Network A (s2svp00) to which Network B (s2svp00) will connect, which is saved in the file /etc/wireguard/s2svp00.conf:
| |
The following configuration is used on Network B (s2svp00) to connect to Network A (s2svp00), which is saved in the file /etc/wireguard/s2svp00.conf:
| |
With the above configuration, both sites will try to directly connect to each other using the public IP address of the other site. If the public IP address changes, the connection will be lost. To avoid this, you can use a dynamic DNS service or a VPN server with a static IP address.
To start the WireGuard interface, a systemd service can be used which executes a script with the following content on the router of Network A:
| |
On the router of Network B, the following script can be used:
| |
Using the command wg show you can check if the WireGuard interface is up and running.
| |
The WireGuard interface should now be up and running on both routers. You can now ping the other router using the IP address of the WireGuard interface.
FRR Setup (BGP)
To provide dynamic routing between the sites, we will use FRR (Free Range Routing) with BGP (Border Gateway Protocol). FRR is a fork of Quagga and provides a modern routing stack with support for BGP, OSPF, RIP, and other routing protocols.
First, we need to configure FRR on all routers. You can find more information about FRR in the official documentation.
To enable BGP within FRR, the file /etc/frr/daemons needs to be edited and the following line needs to be modified:
| |
After changing the configuration, the FRR service needs to be restarted:
| |
Next, the BGP configuration needs to be added using vtysh on the router of Network A. On this site we will use the AS number 65001 and on the router of Network B the AS number 65002.
| |
On the router of Network B, the following configuration needs to be added using vtysh (basically only the AS number and local networks need to be changed):
| |
With the above configuration, the routers will automatically exchange routing information using BGP. You can use the command show ip bgp summary within vtysh to check if the BGP session is up and running.
| |
If the field State/PfxRcd shows a number greater than 0 and Up/Down does not show never, the BGP session is up and running.
With this setup, you can now communicate between the networks using the private IP addresses of the routers. To add an additional site like Network C, you can simply repeat the steps above and add the new network to the BGP configuration.
The main difference to Network C is that the IP address of the site is dynamic and not public, or even behind a NAT. Network A and B have to remove the Endpoint line from the WireGuard configuration and rely on Network C to initiate the connection, as the public IP address of Network C is not known to the other sites.
Example of a configuration for Network A for a connection coming from Network C (s2svp01) is saved in the file /etc/wireguard/s2svp01.conf:
| |
Example of a configuration for Network C for a connection to Network A (s2svp01) is saved in the file /etc/wireguard/s2svp01.conf:
| |
The WireGuard script does only need to be adjusted to use the new interface name and IP addresses.
| |
The BGP configuration on Network A needs to be adjusted to include the new network and the new peer.
| |
The configuration on Network C is similar to the configuration on Network B, with the AS number 65003 and the local networks of Network C.
With this setup, you can now communicate between all networks using the private IP addresses of the routers. Idially, you can add a communication between Network B and C by adding the necessary configuration to the routers of Network B and C. This way, one site like Network A can be down without affecting the communication between the other sites.
Advanced: Inbound and outbound route filtering
To control which networks are allowed to communicate with each other, you can use route filtering in BGP. This allows you to filter inbound and outbound routes based on the network prefix, AS path, or other criteria. You can also use route maps to modify the routes before they are advertised to other routers.
Here is an example of how to filter inbound and outbound routes in BGP using FRR on Network A with route maps:
| |
With the above configuration, only the networks defined in MY-NETWORKS are allowed to be announced to the BGP peer. The route map FILTER-INBOUND filters inbound routes based on the INBOUND-FILTER prefix list, which allows only private IP addresses to be accepted (Except too big RFC1918 address ranges like 10.0.0.0/8).
You can use the command show ip bgp within vtysh to check if the routes are filtered correctly.
Example Homelab
In my homelab environment, I use a similar setup to connect different networks using WireGuard and provide dynamic routing between them using BGP. I have a primary site with a public IP address and multiple IPv4 and IPv6 networks, as well as three remote sites with static and dynamic IP addresses. Additionally, some friends and family members have access to the network using WireGuard.
Conclusion
With this setup, you can easily connect multiple networks using WireGuard and provide dynamic routing between them using BGP. This allows you to automatically exchange routing information between the sites and use the best path for each connection. You can also use BGP to filter which networks are allowed to communicate with each other.
Why s2svp01?
s2svp01 is th short form of “Server 2 Server VPN Peering 01” and is used to identify the different VPN connections between the sites. This way, you can easily add new connections and using the following scheme it’s easy to identify the connection and IP addresses used:
- s2svp00: Connection 00
- 172.31.10.0/31 with 172.31.10.0 primary and 172.31.10.1 secondary
- 2001:db8::0:0/112 with 2001:db8::0:0 primary and 2001:db8::0:1 secondary
- s2svp01: Connection 01
- 172.31.10.2/31 with 172.31.10.2 primary and 172.31.10.3 secondary
- 2001:db8::1:0/112 with 2001:db8::1:2 primary and 2001:db8::1:3 secondary
- s2svp02: Connection 02
- 172.31.10.4/31 with 172.31.10.4 primary and 172.31.10.5 secondary
- 2001:db8::2:0/112 with 2001:db8::2:4 primary and 2001:db8::2:5 secondary
- s2svp03: Connection 03
- 172.31.10.6/31 with 172.31.10.6 primary and 172.31.10.7 secondary
- 2001:db8::3:0/112 with 2001:db8::3:6 primary and 2001:db8::3:7 secondary
This way, you can easily identify the connection and IP addresses used for each VPN connection. Using IPv6, you can also easily identify the connection interface.