Aug 122012

Let’s review how to configure network interface on Solaris 10 (x86 version or sparc). This is what I want to do:

  1. First, I would like to aggregate two network interfaces so it will become one. More bandwidth available and a fail-over will be in place.
  2. Second, I want to add virtual interfaces so I would have dedicated IP address for some services. (Examples: web servers, a NFS server, a Samba server, DNS, Sendmail, …).

Aggregation

To aggregate my two onboard interfaces bnx0 and bnx1 which speed are at 1 gigabyte each, will become aggregated to be one big 2 gigabyte interfaces having only one MAC address.

Show all interface on the system with dladm because. This can’t be done with ifconfig -a because the second interface bnx1 is not yet connected. It will need to be plumb before we see it.

Note-1: that plumb come from plumber who plumb some pipes, it needs to be plumbed before water can go in the pipe. So here interface needs to be plumb before network bit can go in the interface.

Note-2: Often people ask to know: How do we see the speed and duplex of Solaris network interfaces. On Solaris 10+ this is how.

% dladm show-dev

bnx0            link: up        speed: 1000  Mbps       duplex: full
bnx1            link: up        speed: 0     Mbps       duplex: unknown
bnxe0           link: up        speed: 10000 Mbps       duplex: full

% dladm show-link

bnx0            type: non-vlan  mtu: 1500       device: bnx0
bnx1            type: non-vlan  mtu: 1500       device: bnx1
bnxe0           type: non-vlan  mtu: 1500       device: bnxe0

We can see that bnx1 is available on the system, let’s plumb it so we can play with it.

% ifconfig bnx1 plumb

% ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1
       inet 127.0.0.1 netmask ff000000
bnx0: flags=1000843 mtu 1500 index 2
       inet 192.168.108.15 netmask ffffff00 broadcast 132.207.108.255
       ether 00:2b:2b:66:c1:99
bnx1: flags=1000802 mtu 1500 index 4
        inet 0.0.0.0 netmask 0
        ether 84:2b:2b:6:c1:9a
bnxe0: flags=1000843 mtu 1500 index 3
        inet 192.168.108.2 netmask ffffff00 broadcast 192.168.108.255
        ether 0:10:18:30:fc:fa

And then unplumb both interfaces so we can setup aggregation on them. Interface need to be free to be reconfigured for aggregation.

Note: Be aware that when doing this all network connections to bnx0 interfaces will be lost!!! If you are connected remotely with SSH you will loose access. Be on the console.

% ifconfig bnx0 unplumb
% ifconfig bnx1 unplumb

This next line will create the link aggregation and the next one will set it up with the IP address we want.

% dladm create-aggr -d bnx0 -d bnx1 1

I sometime had an issue… devices where kept in a busy state.

% dladm create-aggr -d bnx0 -d bnx1 -1
dladm: create operation failed: Device busy (invalid interface name)

To solve this I disable the network service and rebooted and re-did again the create-aggr.

% svcad disable svc:/network/physical:default
% reboot

After the reboot all interface will not be present, no network connection will be available.
So when ready you will need to re-enable the service.
% svcad enable -sr svc:/network/physical:default

% ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
bnxe0: flags=1000843 mtu 1500 index 3
        inet 192.168.108.2 netmask ffffff00 broadcast 192.168.108.255
        ether 0:10:18:30:fc:fa

% ifconfig  aggr1  plumb  netmask 255.255.255.0  192.168.108.15 up
% ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
bnxe0: flags=1000843 mtu 1500 index 3
        inet 192.168.108.2 netmask ffffff00 broadcast 192.168.108.255
        ether 0:10:18:30:fc:fa
aggr1: flags=1000843 mtu 1500 index 6
        inet 192.168.108.15 netmask ffff0000 broadcast 132.207.255.255
        ether 00:2b:2b:66:c1:99

Voila, that’s it. One big interface offering more bandwidth and a fail-over is in place. Any of the two cables can be unplug and re-plug at will or almost twice the bandwidth is now available.

% dladm show-aggr
key: 1 (0x0001) policy: L4      address: 84:2b:2b:6:c1:99 (auto)
           device       address                 speed           duplex  link    state
           bnx0         84:2b:2b:6:c1:99          1000  Mbps    full    up      attached
           bnx1         84:2b:2b:6:c1:9a          1000  Mbps    full    up      attached

If you want you can check the aggregation statistics for each interfaces.

% dladm show-aggr -s
key:1           ipackets   rbytes       opackets   obytes       %ipkts  %opkts
        Total   4038       524867       847        324837
        bnx0    2310       345467       765        317395         57.2    90.3
        bnx1    1728       179400       82         7442           42.8     9.7

Question: Are those modifications permanent to the system? No! Not yet.

If you reboot your system will not like it and you won’t have running network access. To make it permanent you will need to remove your previous interface name’s in /etc/hostname.bnx0. Assuming you want to keep the same name you had before, simply move the file to its new name.

% mv /etc/hostname.bnx0 /etc/hostname.aggr1
% dladm show-aggr
key: 1 (0x0001) policy: L4      address: 00:2b:2b:66:c1:99 (auto)
           device       address                 speed           duplex  link    state
           bnx0         00:2b:2b:6:c1:99          1000  Mbps    full    up      attached
           bnx1         00:2b:2b:6:c1:9a          1000  Mbps    full    up      attached

This -L option will show detailed LACP information. Since the switch also need to be configured to do port
bounding, I found that LACP was not enable on Solaris even after creating the aggregation. I decided to set LACP mode to active (both on the switch and Solaris).

% dladm show-aggr -L
key: 1 (0x0001) policy: L4      address: 00:2b:2b:66:c1:99 (auto)
                LACP mode: off  LACP timer: short
    device    activity timeout aggregatable sync  coll dist defaulted expired
    bnx0      passive  short   yes          no    no   no   no        no     
    bnx1      passive  short   yes          no    no   no   no        no     
% dladm modify-aggr -PL4 -l active 1

% dladm show-aggr -L
key: 1 (0x0001) policy: L4      address: 00:2b:2b:66:c1:99 (auto)
                LACP mode: active       LACP timer: short
    device    activity timeout aggregatable sync  coll dist defaulted expired
    bnx0      active   short   yes          yes   yes  yes  no        no     
    bnx1      active   short   yes          yes   yes  yes  no        no

Setting up Virtual interfaces on Solaris 10.

You can plumb reel interfaces or you can plumb a virtual interface on top of existing interfaces, which will add a “:1” and be called aggr1:0  or number from 0 to 255. Because there is 256 possible virtual interface on Solaris.

For real interfaces bnx0, ce0, hme0, bnxe0, …

% ifconfig bnx0:1 plumb
% ifconfig bnx0:1 192.168.1.80 netmask 255.255.255.0 up

and for aggregated interfaces on a single command:

% ifconfig aggr1:1 plumb netmask 255.255.255.0  192.168.1.80  up

**updated**
Question: Is this permanent?
No. You will also need to create the hostname file in /etc/hostname.aggr1 with the ip address for the aggregation, and remove /etc/hostname.*** (from the previous interface).

You can remove one network interface from the aggregation.

dladm remove-aggr -d bnx0 1

Or you can completely remove aggregation from the system:

dladm delete-aggr  1

To stop using aggregation you will also need to rename back /etc/hostaname.aggr1 to /etc/hostaname.bnx0 and any other virtual interface you defined (hostaname.aggr1:1 hostaname.aggr1:2 …).

Hope this help.
Rejean

3 Responses to “Virtual interfaces and network card aggregation with Solaris 10 – “revised june 2012””

  1. Leslie Abanda says:

    Hello Rejean,
    This is really a very helpful information. The explanation was very detailed.

    Thanks

    Leslie

  2. Rahul Monde says:

    The information shared above about link aggregation is really helpful. 🙂
    Thanks a lot.

  3. Matlacha Fishing says:

    This excellent website truly has all the info I wanted concerning this subject and didn’t know who to ask.

Leave a Reply

(required)

(required)

Return to Panoramic Solution