Ubuntu Bonding vs PPPoE multilink (1/3)

I had a dream, to connect two computers on my Home network with two gigabit ethernet connections and have a 2 gigabit connection to use. However even after a lot of experimenting I could not get real benefits in the everyday use. The closest I could get was the PPPoE multilink setup. In this series of post I will share same of the test results for a normal one gigabit network, a bonded 2 gigabit network and a PPPoE multilink 2 gigabit network.

This is the first article in a series, the second part is Bonding measurements and the third is the PPPoE Multilink measurements.


There are many technologies to bundle more ethernet connections into one, but this is typically used in enterprise networks where the individual connections are always handled on different devices and only the resulting overall throughput is higher. In a home network you expect that each connection is speeded up, and if you are copying a file from your NAS, you expect to have the double speed.

The reason why individual connections are routed through different path is the out of order packages, which in case of normal TCP connections will result in dropped packets which on turn will lead to tcp congestion controll kicking in, slowing down the whole connection. In some cases you even get slower connections, then what you would get from a simple gigabit network.

Some technologies available for handling multiple links


Interface Bonding 

This is a standard Linux solution, with different configuration options. The roud robin transmit policy suits our need, to have one connection distributed over two connections, but the out of order packets are a problem. In the second article we will test this setup.

Here is a good article about setting up bonding in Ubuntu.

Interface Teaming

Roughly has the same features as Bonding, but a fresher implementation.

Look here for a comparation of Bonding and Teaming on Linux.

TEQL loadsharing

It is a simple technology, to distribute packets to 2 or more connections, I have tested it on a virtual machine and the performance was around 300 Mbit/sec so it was not suitable for aggregating 2 gigabit connections. The out of ordered packets are not handled by TEQL.

Here is the original description on how to set it up.

Multipath TCP

Multipath TCP is an IETF experimental standard RFC 6824 and is mainly intended to increase the robustness of the connections when more than one path is available. There is a Linux implementation at multipath-tcp.org. I have tested it on a Virtual Machine, but the performance was again about 300 Mbit/sec, which is less than the performance of a single link. There are a lot of possibilities to tune the system, but the installation installed a complete new kernel and the tweaking is quite advanced, so I did not experimented more with it.

Multipath GRE

GRE is a commonly used tunnel protocol in Linux and in the University of Debrecen/Hungary they created an extension, where more than one path can be used, between the endpoints. The implementation looked nice, because it had solutions for fixing packages received out of order, and you could finetune, the bandwidth allocated to the different connections. In practice again, the speed of the solution was between 200-400 Mbit/sec, because the implementation was in user space so it was not usable to create the 2 gig connection. 


ECMP - Equal Cost Multipath Routing

This is supported on recent Linux kernels, but as most other solution the packets within one connection are sent over the same route, so there is no single connection gain.


PPPoE Multilink

PPP supported multilink communication to bundle 2 or more modem connections and the same applies to PPPoE. The good thing is that it handles the out of order arrival of packets. We will do some more tests on this.

PPPoE in Wikipedia

Test environment

To do proper testin the test environment from the diagram below was implemented:

In the center of the experiments we have Orca, which is a Server, running virtual machines and has a 10G connection to the Backup server. It is also connected to the internal 1G network, and the 10G card and the 1G card connected to the rest of the network is connected in one Bridge. Our multilink test server NAS2 is set up with 2 gigabit ethernet connection, and it is located in a separate subnet. In Orca we have 2 virtual machines, one is running Windows and the other Linux.

Reference benchmarks



Before testing the Bond and PPPoE performance we start with some test using only one 1 G connection to check if there are some other bottlenecks in the network.

Setting up one 1G link between Orca and NAS2

On the NAS2 side:

sudo ip link set dev enp4s0 up
sudo ip address add 192.168.190.11/24 dev enp4s0
iperf3 -s

On Orca side:

sudo ip link set dev enp4s0 up
sudo ip addr add 192.168.190.10/24 dev eth2card0
ping 192.168.190.11


For checking you can use:

ip link
ip addr
ip route


To measure the speed issue on Orca:

iperf3 -c 192.168.190.11
iperf3 -c 192.168.190.11 -R


The results are:

942 Mbits/sec from Orca to NAS2 and 940 Mbits/sec on the oposite direction.

Now let's measure the speed from other computers to NAS2 with iperf3:

Computer To(Mb/s) From(Mb/s) To(MB/s) From(MB/s)

Orca       942       940       118        118
Boss       937       937       117        117
Blue       949       948       119        119
Backup     945       933       118        117
VM Windows 946       940       118        118
VM Ubuntu  945       934       118        117


We can see that the speed is stable between 930 and 950 Mbits/sec, witch is close to the theoretical 1000 Mbits/sec.

Samba speeds:

For the test, we create a 1 G ram disk on NAS2 a copy a 750 megabytes file to it.

smbclient \\\\192.168.190.11\\ramdisk
smb:> get proxmox.iso

We delete the file from the server and then:

smb:> put proxmox.iso

We see that the transfer speed is 103 MegaBytes/sec in one direction and 110 MegaBytes/sec in the other.

On Windows you can use robocopy, but the syntax is a bit strange, first you have to specify the source and destination directory and then the files to be copied.

robocopy "\\192.168.190.11\ramdisk\" . proxmox.iso
robocopy . "\\192.168.190.11\ramdisk\" proxmox.iso

For windows (Blue) the speed was 110 MB/sec in one direction and 115 MB/sec in the other direction.

Result summary:

Computer   To(MB/s) From(MB/s)

Orca         110      103
Boss          96      114
Blue         115      110
Backup       110       92
VM Windows   109      108
VM Ubuntu    111       95

These results are infected by the local disk speeds, so let's check the local disk speeds.

Local disk speeds

On Ubuntu:

iozone -s 950M -r 16M -I -i 0 -i 1 -f ./iozone.tmp

On Windows:

CrystalDiskMark 3.0.3 x64 1000MB sequential

Results:

Computer   Write(MB/s) Read(MB/s)

NAS2        1082       1186
Orca        2159       3097
Boss         525       1662
Blue         231        254
Backup       171        228
VM Windows   755        423
VM Ubuntu    738        588

On NAS2 and Boss we use RAM disks, but they are older slow computers, on Orca there is a NVMe SSD with rather good speeds, Blue and Backup are older computers with older SATA SSDs and the VMs are using a 2 SATA SSD connected in a RAID0 setup.

Samba speed without local disk influence

To check the raw results, let's mount the samba share on Linux and measure it with iozone, and use CrystalDiskMark on Windows. Results:

sudo mount -t cifs -o user=user //192.168.190.11/ramdisk /media/test1

Results:

Computer   To(MB/s) From(MB/s)

NAS2        231      377
Orca        113      113
Boss        107      113
Blue         96      100
Backup      112      112
VM Windows  104       97
VM Ubuntu   107      108

As we can see over the network speeds around 100 MBytes/sec can be achieved. Internally the Samba server is capable of delivering above 200 MBytes/sec speeds.

Other measurements

Just for the comparation, let's check the speed between Backup and Orca, where we have a 10G connection:

Iperf3 to Orca:

Computer   To(MB/s) From(MB/s)

Backup     1175      1143
VM Windows 1065       316 (Intel PRO/1000MT network emulation)
VM Windows 1222       165 (virtio-net network emulation)
VM Ubuntu  1238       350 (Intel PRO/1000MT network emulation)
VM Ubuntu  1762       285 (virtio-net network emulation)

Virtio-net is faster in sending data, but slower in receiving data, so I switched it off. Generally the virtual machines are performing below expectation in the receiving direction.

Samba with IOZone to Orca:

Computer   To(MB/s) From(MB/s)

Orca        1506     2030
Backup       394      377
VM Windows   641      280
VM Ubuntu    325      183


Here we can see that even if the ethernet speed is above 1000 MB/sec, over samba the speed is reduced to around 400 MB/sec.

Conclusion

What I have learnt from this is, that beside the iprf3 test, I will do the samba test with mounting the samba and using IOZone or CrystalDiskMark, and that when using 1 gigabit connection, the speeds are roughly 940 Mbit/sec in Iperf3 and 105 MByte/sec in samba.

Comments

Popular posts from this blog

Setting ethernet interface speed in /etc/network/interfaces