Ubuntu Bonding vs PPPoE multilink (1/3)
I had a dream, to connect two computers on my Home network with two gigabit ethernet connections and have a 2 gigabit connection to use. However even after a lot of experimenting I could not get real benefits in the everyday use. The closest I could get was the PPPoE multilink setup. In this series of post I will share same of the test results for a normal one gigabit network, a bonded 2 gigabit network and a PPPoE multilink 2 gigabit network.
This is the first article in a series, the second part is Bonding measurements and the third is the PPPoE Multilink measurements.
There are many technologies to bundle more ethernet connections into one, but this is typically used in enterprise networks where the individual connections are always handled on different devices and only the resulting overall throughput is higher. In a home network you expect that each connection is speeded up, and if you are copying a file from your NAS, you expect to have the double speed.
The reason why individual connections are routed through different path is the out of order packages, which in case of normal TCP connections will result in dropped packets which on turn will lead to tcp congestion controll kicking in, slowing down the whole connection. In some cases you even get slower connections, then what you would get from a simple gigabit network.
Interface Bonding
This is a standard Linux solution, with different configuration options. The roud robin transmit policy suits our need, to have one connection distributed over two connections, but the out of order packets are a problem. In the second article we will test this setup.
Here is a good article about setting up bonding in Ubuntu.
Interface Teaming
Roughly has the same features as Bonding, but a fresher implementation.
Look here for a comparation of Bonding and Teaming on Linux.
TEQL loadsharing
It is a simple technology, to distribute packets to 2 or more connections, I have tested it on a virtual machine and the performance was around 300 Mbit/sec so it was not suitable for aggregating 2 gigabit connections. The out of ordered packets are not handled by TEQL.
Here is the original description on how to set it up.
Multipath TCP
Multipath TCP is an IETF experimental standard RFC 6824 and is mainly intended to increase the robustness of the connections when more than one path is available. There is a Linux implementation at multipath-tcp.org. I have tested it on a Virtual Machine, but the performance was again about 300 Mbit/sec, which is less than the performance of a single link. There are a lot of possibilities to tune the system, but the installation installed a complete new kernel and the tweaking is quite advanced, so I did not experimented more with it.
PPP supported multilink communication to bundle 2 or more modem connections and the same applies to PPPoE. The good thing is that it handles the out of order arrival of packets. We will do some more tests on this.
PPPoE in Wikipedia
In the center of the experiments we have Orca, which is a Server, running virtual machines and has a 10G connection to the Backup server. It is also connected to the internal 1G network, and the 10G card and the 1G card connected to the rest of the network is connected in one Bridge. Our multilink test server NAS2 is set up with 2 gigabit ethernet connection, and it is located in a separate subnet. In Orca we have 2 virtual machines, one is running Windows and the other Linux.
Before testing the Bond and PPPoE performance we start with some test using only one 1 G connection to check if there are some other bottlenecks in the network.
Setting up one 1G link between Orca and NAS2
On the NAS2 side:
sudo ip link set dev enp4s0 up
sudo ip address add 192.168.190.11/24 dev enp4s0
iperf3 -s
On Orca side:
sudo ip link set dev enp4s0 up
sudo ip addr add 192.168.190.10/24 dev eth2card0
ping 192.168.190.11
For checking you can use:
ip link
ip addr
ip route
To measure the speed issue on Orca:
iperf3 -c 192.168.190.11
iperf3 -c 192.168.190.11 -R
The results are:
942 Mbits/sec from Orca to NAS2 and 940 Mbits/sec on the oposite direction.
Now let's measure the speed from other computers to NAS2 with iperf3:
Computer To(Mb/s) From(Mb/s) To(MB/s) From(MB/s)
Blue 949 948 119 119
Backup 945 933 118 117
VM Windows 946 940 118 118
VM Ubuntu 945 934 118 117
We can see that the speed is stable between 930 and 950 Mbits/sec, witch is close to the theoretical 1000 Mbits/sec.
Samba speeds:
For the test, we create a 1 G ram disk on NAS2 a copy a 750 megabytes file to it.
smbclient \\\\192.168.190.11\\ramdisk
smb:> get proxmox.iso
We delete the file from the server and then:
smb:> put proxmox.iso
We see that the transfer speed is 103 MegaBytes/sec in one direction and 110 MegaBytes/sec in the other.
On Windows you can use robocopy, but the syntax is a bit strange, first you have to specify the source and destination directory and then the files to be copied.
robocopy "\\192.168.190.11\ramdisk\" . proxmox.iso
robocopy . "\\192.168.190.11\ramdisk\" proxmox.iso
For windows (Blue) the speed was 110 MB/sec in one direction and 115 MB/sec in the other direction.
Result summary:
Computer To(MB/s) From(MB/s)
Orca 110 103
Boss 96 114
Blue 115 110
Backup 110 92
VM Windows 109 108
VM Ubuntu 111 95
These results are infected by the local disk speeds, so let's check the local disk speeds.
Local disk speeds
On Ubuntu:
iozone -s 950M -r 16M -I -i 0 -i 1 -f ./iozone.tmp
On Windows:
CrystalDiskMark 3.0.3 x64 1000MB sequential
Results:
Computer Write(MB/s) Read(MB/s)
NAS2 1082 1186
Orca 2159 3097
Boss 525 1662
Blue 231 254
Backup 171 228
VM Windows 755 423
VM Ubuntu 738 588
On NAS2 and Boss we use RAM disks, but they are older slow computers, on Orca there is a NVMe SSD with rather good speeds, Blue and Backup are older computers with older SATA SSDs and the VMs are using a 2 SATA SSD connected in a RAID0 setup.
Samba speed without local disk influence
To check the raw results, let's mount the samba share on Linux and measure it with iozone, and use CrystalDiskMark on Windows. Results:
sudo mount -t cifs -o user=user //192.168.190.11/ramdisk /media/test1
Results:
Computer To(MB/s) From(MB/s)
NAS2 231 377
Orca 113 113
Boss 107 113
Blue 96 100
Backup 112 112
VM Windows 104 97
VM Ubuntu 107 108
As we can see over the network speeds around 100 MBytes/sec can be achieved. Internally the Samba server is capable of delivering above 200 MBytes/sec speeds.
Other measurements
Just for the comparation, let's check the speed between Backup and Orca, where we have a 10G connection:
Iperf3 to Orca:
Computer To(MB/s) From(MB/s)
Backup 1175 1143
VM Windows 1065 316 (Intel PRO/1000MT network emulation)
VM Windows 1222 165 (virtio-net network emulation)
VM Ubuntu 1238 350 (Intel PRO/1000MT network emulation)
VM Ubuntu 1762 285 (virtio-net network emulation)
Virtio-net is faster in sending data, but slower in receiving data, so I switched it off. Generally the virtual machines are performing below expectation in the receiving direction.
Samba with IOZone to Orca:
Computer To(MB/s) From(MB/s)
Orca 1506 2030
Backup 394 377
VM Windows 641 280
VM Ubuntu 325 183
Here we can see that even if the ethernet speed is above 1000 MB/sec, over samba the speed is reduced to around 400 MB/sec.
Conclusion
What I have learnt from this is, that beside the iprf3 test, I will do the samba test with mounting the samba and using IOZone or CrystalDiskMark, and that when using 1 gigabit connection, the speeds are roughly 940 Mbit/sec in Iperf3 and 105 MByte/sec in samba.
This is the first article in a series, the second part is Bonding measurements and the third is the PPPoE Multilink measurements.
There are many technologies to bundle more ethernet connections into one, but this is typically used in enterprise networks where the individual connections are always handled on different devices and only the resulting overall throughput is higher. In a home network you expect that each connection is speeded up, and if you are copying a file from your NAS, you expect to have the double speed.
The reason why individual connections are routed through different path is the out of order packages, which in case of normal TCP connections will result in dropped packets which on turn will lead to tcp congestion controll kicking in, slowing down the whole connection. In some cases you even get slower connections, then what you would get from a simple gigabit network.
Some technologies available for handling multiple links
Interface Bonding
This is a standard Linux solution, with different configuration options. The roud robin transmit policy suits our need, to have one connection distributed over two connections, but the out of order packets are a problem. In the second article we will test this setup.
Here is a good article about setting up bonding in Ubuntu.
Interface Teaming
Roughly has the same features as Bonding, but a fresher implementation.
Look here for a comparation of Bonding and Teaming on Linux.
TEQL loadsharing
It is a simple technology, to distribute packets to 2 or more connections, I have tested it on a virtual machine and the performance was around 300 Mbit/sec so it was not suitable for aggregating 2 gigabit connections. The out of ordered packets are not handled by TEQL.
Here is the original description on how to set it up.
Multipath TCP
Multipath TCP is an IETF experimental standard RFC 6824 and is mainly intended to increase the robustness of the connections when more than one path is available. There is a Linux implementation at multipath-tcp.org. I have tested it on a Virtual Machine, but the performance was again about 300 Mbit/sec, which is less than the performance of a single link. There are a lot of possibilities to tune the system, but the installation installed a complete new kernel and the tweaking is quite advanced, so I did not experimented more with it.
Multipath GRE
GRE is a commonly used tunnel protocol in Linux and in the University of Debrecen/Hungary they created an extension, where more than one path can be used, between the endpoints. The implementation looked nice, because it had solutions for fixing packages received out of order, and you could finetune, the bandwidth allocated to the different connections. In practice again, the speed of the solution was between 200-400 Mbit/sec, because the implementation was in user space so it was not usable to create the 2 gig connection.
ECMP - Equal Cost Multipath Routing
This is supported on recent Linux kernels, but as most other solution the packets within one connection are sent over the same route, so there is no single connection gain.
PPPoE Multilink
PPP supported multilink communication to bundle 2 or more modem connections and the same applies to PPPoE. The good thing is that it handles the out of order arrival of packets. We will do some more tests on this.
PPPoE in Wikipedia
Test environment
To do proper testin the test environment from the diagram below was implemented:
Reference benchmarks
Before testing the Bond and PPPoE performance we start with some test using only one 1 G connection to check if there are some other bottlenecks in the network.
Setting up one 1G link between Orca and NAS2
On the NAS2 side:
sudo ip link set dev enp4s0 up
sudo ip address add 192.168.190.11/24 dev enp4s0
iperf3 -s
On Orca side:
sudo ip link set dev enp4s0 up
sudo ip addr add 192.168.190.10/24 dev eth2card0
ping 192.168.190.11
For checking you can use:
ip link
ip addr
ip route
To measure the speed issue on Orca:
iperf3 -c 192.168.190.11
iperf3 -c 192.168.190.11 -R
The results are:
942 Mbits/sec from Orca to NAS2 and 940 Mbits/sec on the oposite direction.
Now let's measure the speed from other computers to NAS2 with iperf3:
Computer To(Mb/s) From(Mb/s) To(MB/s) From(MB/s)
Orca 942 940 118 118
Boss 937 937 117 117Blue 949 948 119 119
Backup 945 933 118 117
VM Windows 946 940 118 118
VM Ubuntu 945 934 118 117
We can see that the speed is stable between 930 and 950 Mbits/sec, witch is close to the theoretical 1000 Mbits/sec.
Samba speeds:
For the test, we create a 1 G ram disk on NAS2 a copy a 750 megabytes file to it.
smbclient \\\\192.168.190.11\\ramdisk
smb:> get proxmox.iso
We delete the file from the server and then:
smb:> put proxmox.iso
We see that the transfer speed is 103 MegaBytes/sec in one direction and 110 MegaBytes/sec in the other.
On Windows you can use robocopy, but the syntax is a bit strange, first you have to specify the source and destination directory and then the files to be copied.
robocopy "\\192.168.190.11\ramdisk\" . proxmox.iso
robocopy . "\\192.168.190.11\ramdisk\" proxmox.iso
For windows (Blue) the speed was 110 MB/sec in one direction and 115 MB/sec in the other direction.
Result summary:
Computer To(MB/s) From(MB/s)
Orca 110 103
Boss 96 114
Blue 115 110
Backup 110 92
VM Windows 109 108
VM Ubuntu 111 95
These results are infected by the local disk speeds, so let's check the local disk speeds.
Local disk speeds
On Ubuntu:
iozone -s 950M -r 16M -I -i 0 -i 1 -f ./iozone.tmp
On Windows:
CrystalDiskMark 3.0.3 x64 1000MB sequential
Results:
Computer Write(MB/s) Read(MB/s)
NAS2 1082 1186
Orca 2159 3097
Boss 525 1662
Blue 231 254
Backup 171 228
VM Windows 755 423
VM Ubuntu 738 588
On NAS2 and Boss we use RAM disks, but they are older slow computers, on Orca there is a NVMe SSD with rather good speeds, Blue and Backup are older computers with older SATA SSDs and the VMs are using a 2 SATA SSD connected in a RAID0 setup.
Samba speed without local disk influence
To check the raw results, let's mount the samba share on Linux and measure it with iozone, and use CrystalDiskMark on Windows. Results:
sudo mount -t cifs -o user=user //192.168.190.11/ramdisk /media/test1
Results:
Computer To(MB/s) From(MB/s)
NAS2 231 377
Orca 113 113
Boss 107 113
Blue 96 100
Backup 112 112
VM Windows 104 97
VM Ubuntu 107 108
As we can see over the network speeds around 100 MBytes/sec can be achieved. Internally the Samba server is capable of delivering above 200 MBytes/sec speeds.
Other measurements
Just for the comparation, let's check the speed between Backup and Orca, where we have a 10G connection:
Iperf3 to Orca:
Computer To(MB/s) From(MB/s)
Backup 1175 1143
VM Windows 1065 316 (Intel PRO/1000MT network emulation)
VM Windows 1222 165 (virtio-net network emulation)
VM Ubuntu 1238 350 (Intel PRO/1000MT network emulation)
VM Ubuntu 1762 285 (virtio-net network emulation)
Virtio-net is faster in sending data, but slower in receiving data, so I switched it off. Generally the virtual machines are performing below expectation in the receiving direction.
Samba with IOZone to Orca:
Computer To(MB/s) From(MB/s)
Orca 1506 2030
Backup 394 377
VM Windows 641 280
VM Ubuntu 325 183
Here we can see that even if the ethernet speed is above 1000 MB/sec, over samba the speed is reduced to around 400 MB/sec.
Conclusion
What I have learnt from this is, that beside the iprf3 test, I will do the samba test with mounting the samba and using IOZone or CrystalDiskMark, and that when using 1 gigabit connection, the speeds are roughly 940 Mbit/sec in Iperf3 and 105 MByte/sec in samba.
Comments