LACP, VLANs, always stay connected.

I was bored last weekend, so I configured a two-port LACP bonded trunk from my FreeBSD-running NAS connected to my HP Procurve switch.

Why?

  • I could?
  • I had all these spare Ethernet ports on my NAS, and they seemed bored.
  • More seriously: high availability. One interface serving all my storage traffic just seemed ripe for failure. Imagine serving all your VMs over NFS to a VM server across the network over one NIC, and that one dies. Bad news bears.
  • I also wanted to set up VLANs on top of the trunk. Why? So if I wanted to add a network segment for my NAS on another Layer 3 domain, I don’t have to walk down to the basement to patch another cable.

On to the configuration.

First, I configured the NAS box. Relevant /etc/rc.conf configuration:

# LACP Group
# https://www.freebsd.org/doc/handbook/network-aggregation.html
ifconfig_igb0="up"
ifconfig_igb1="up"
cloned_interfaces="lagg0 vlan12"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1"

# VLAN Configuration
# https://www.freebsd.org/doc/handbook/network-vlan.html
vlans_lagg0="12"
ifconfig_lagg0_12="inet 10.10.2.31/24"

In the first stanza, define the interfaces (igb0, igb1) that comprise the LAGG (link aggregation) interface, simply up’ing them. Then configure your LAGG interface. It’s important to specify the LAGG proto (LACP in my case). Since I’ll be assigning IPs to the tagged VLAN interface, I don’t assign an IP to the raw trunk interface. If I wanted to handle untagged traffic on the trunk, I would specify an IP address.

In the second stanza, configuring the tagged VLAN interface. “vlans_lagg0” specifies a list of VLANs which traverse the lagg0 interface. Then, configure the IP address which will be tagged with VLAN 12 traffic, riding on lagg0.

Ifconfig output should look like the following:

nas1:~ % ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
 ether 0c:c0:7a:54:84:12
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet autoselect
 status: active
 groups: lagg 
 laggproto lacp lagghash l2,l3,l4
 laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
 laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

nas1:~ % ifconfig lagg0.12
lagg0.12: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=303<RXCSUM,TXCSUM,TSO4,TSO6>
 ether 0c:c0:7a:54:84:12
 inet 10.10.2.31 netmask 0xffffff00 broadcast 10.10.2.255 
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet autoselect
 status: active
 vlan: 12 vlanpcp: 0 parent interface: lagg0
 groups: vlan

With the NAS’s ethernet cables disconnected from the switch, we configure the static LACP trunk. To note, I run an HP Procurve 2910al-24G switch, so my commands will not work for instance on a Cisco switch running IOS. This is the point in the exercise where I wish I had command history on the switch. I can’t remember the commands and in what order I did them. It probably went something not far from the following:

switch1$ enable

switch1# config t

switch1(config)# trunk ethernet 14,16 trk1 lacp 

switch1(config)# vlan 12

switch1(vlan-12)# tagged trk1

What does this do? Creates a two port, static LACP trunk on ports 14 and 16. We then create VLAN 12, and assigning lagg0 to this trunk group.

Now plug those Ethernet cables in and wait a minute for LACP neogtiation to take place.

switch1# show trunk

Load Balancing

Port  | Name Type              | Group Type
 ---- + -------------------------------- 
 14   | nas1-trunk-1 100/1000T | Trk1 LACP
 16   | nas1-trunk-2 100/1000T | Trk1 LACP

switch1# show lacp

LACP

LACP Trunk Port LACP
 Port Enabled Group   Status  Partner Status
 ---- ------- ------- ------- ------- -------
 14   Active  Trk1    Up      Yes     Success
 16   Active  Trk1    Up      Yes     Success

Hooray. A working, two-port Ethernet LACP trunk.

I did some testing, removing Ethernet cables from the two-trunk pair seeing how many packets were dropped, or what the logs of the switch looked like.

64 bytes from 10.10.2.31: icmp_seq=204 ttl=63 time=5.204 ms
Request timeout for icmp_seq 207
64 bytes from 10.10.2.31: icmp_seq=208 ttl=63 time=5.223 ms
....
64 bytes from 10.10.2.31: icmp_seq=224 ttl=63 time=4.502 ms
Request timeout for icmp_seq 225
Request timeout for icmp_seq 226
Request timeout for icmp_seq 227
64 bytes from 10.10.2.31: icmp_seq=228 ttl=63 time=7.078 ms

Four packets dropped twice. Not bad! What does this look like from the switch?

switch1# log Trk1
 Keys: W=Warning I=Information
 M=Major D=Debug E=Error
 ---- Event Log listing: Events Since Boot ----
 I 10/22/16 14:00:51 00078 ports: trunk Trk1 is now active
 I 10/22/16 14:00:53 00076 ports: port 14 in Trk1 is now on-line
 I 10/22/16 14:00:59 00079 ports: trunk Trk1 is now inactive
 I 10/22/16 14:00:59 00078 ports: trunk Trk1 is now active
 I 10/22/16 14:01:02 00076 ports: port 16 in Trk1 is now on-line
 I 10/22/16 14:02:01 00077 ports: port 14 in Trk1 is now off-line
 I 10/22/16 14:02:20 00076 ports: port 14 in Trk1 is now on-line
 I 10/22/16 14:04:50 00077 ports: port 16 in Trk1 is now off-line
 I 10/22/16 14:05:02 00076 ports: port 16 in Trk1 is now on-line
 I 10/22/16 14:05:09 00077 ports: port 14 in Trk1 is now off-line
 I 10/22/16 14:05:21 00076 ports: port 14 in Trk1 is now on-line

Hopefully this end-to-end configuration example helps others who might find it useful.

References: