I just bought a new switch for my homelab, a Cisco Catalyst 2960-X Series (WS-C2960X-48TD-L), and wanted to configure IEEE 802.3ad Link Aggregation Policy and LACP to bundle 4 existing ethernet ports of my HP Proliant DL360p G8 to be connected to the switch.
I am running Proxmox 7 on my servers, but after upgrading from Proxmox 6 to 7 I encountered problems with LACP causing no link at all.
After some debugging, I found out that using auto <iface> within the /etc/network/interfaces configuration causes an interface to start individually and blocking any possibility to add the interface into an LACP bond afterwards.
With the following configuration, LACP was working fine and was discoveralble within Debian and the Cisco switch:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| # HPE 4 Port PCIe ethernet interface
# DO NOT USE "auto <iface>" because this breaks LACP as of Proxmox 7
iface enp1s0f0 inet manual
iface enp1s0f1 inet manual
iface enp1s0f2 inet manual
iface enp1s0f3 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp1s0f0 enp1s0f1 enp1s0f2 enp1s0f3
bond-miimon 100
bond-updelay 200
bond-downdelay 200
bond-mode 802.3ad
bond-lacp-rate 1
bond-xmit-hash-policy layer3+4
auto vmbr0
iface vmbr0 inet static
address 10.20.30.1/24
gateway 10.20.30.254
bridge-ports bond0
bridge-stp off
bridge-fd 0
# Example Network
|
On the Cisco side, I am using the following configuration to bundle 4 ports into a port channel group. These ports provide a trunk with some VLANs configured.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| interface Port-channel1
description IX7 LACP
switchport trunk allowed vlan 1,2,4,8,16-32
switchport mode trunk
switchport nonegotiate
speed 1000
spanning-tree portfast edge
interface GigabitEthernet2/0/1
description IX7 LACP
switchport trunk allowed vlan 1,2,4,8,16-32
switchport mode trunk
switchport nonegotiate
load-interval 60
speed 1000
spanning-tree bpdufilter enable
channel-protocol lacp
channel-group 1 mode active
interface GigabitEthernet2/0/2
description IX7 LACP
switchport trunk allowed vlan 1,2,4,8,16-32
switchport mode trunk
switchport nonegotiate
load-interval 60
speed 1000
spanning-tree bpdufilter enable
channel-protocol lacp
channel-group 1 mode active
[...]
|
After setting up the configuration on both devices, the Cisco device can see the servers LACP status information using show lacp neighbors. This shows an active bundle of 4 ports in fast mode:
1
2
3
4
5
6
7
8
| Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi2/0/1 FA 255 7005.7d03.bfb9 25s 0x0 0x9 0x1 0x3F
Gi2/0/2 FA 255 7005.7d03.bfb9 25s 0x0 0x9 0x2 0x3F
Gi2/0/3 FA 255 7005.7d03.bfb9 24s 0x0 0x9 0x3 0x3F
Gi2/0/4 FA 255 7005.7d03.bfb9 24s 0x0 0x9 0x4 0x3F
|
On the linux side, the active configuration can be viewed using cat /proc/net/bonding/bond0:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
| Ethernet Channel Bonding Driver: v5.15.35-3-pve
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Peer Notification Delay (ms): 0
802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 70:05:7d:03:bf:b9
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 4
Actor Key: 9
Partner Key: 1
Partner Mac Address: 2b:c5:3e:72:7f:00
Slave Interface: enp1s0f0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: c7:5b:39:a6:36:66
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 70:05:7d:03:bf:b9
port key: 9
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: 2b:c5:3e:72:7f:00
oper key: 1
port priority: 32768
port number: 514
port state: 61
[...]
|
Ethtool now shows an available bandwidth of 4000Mb/s using the command ethtool bond0. This verifies that the link to the switch is up and bundled to 4Gb/s.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| Settings for bond0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 4000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Other
PHYAD: 0
Transceiver: internal
Link detected: yes
|
Happy port bonding 😉
