Issue
After installing a new Hypervisor, very poor TCP performance was experienced on the VMs residing on that Hypervisor.
Environment
OnApp 3.x
Troubleshooting
1. Download speed tests where 4 Kb/s, a 500MB file estimated a 62 hour download time.
[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 500M 0 35944 0 0 2324 0 62:39:57 0:00:15 62:39:42 2350^C
2. The same test from the Hypervisor showed much faster download speed 16MB/s
3. While checking the logs, the following was found on /var/log/dmesg and dmesg
bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond1.175: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled bond0.200: received packets cannot be forwarded while LRO is enabled
Resolution
1. Disable LRO on each NIC/NICS
ethtool -K eth2 lro off
2. After disabling LRO TCP performance was fine on the VMS
[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 500M 100 500M 0 0 13.2M 0 0:00:37 0:00:37 --:--:-- 17.2M
3. To persist the config across reboots, use ETHTOOL_OPTS (initscripts-9.03.27-1 or later)on the rc.local of the HV
#Disabling LRO after poor TCP performance identified
ETHTOOL_OPTS="-K eth0 lro off" ETHTOOL_OPTS="-K eth1 lro off" ETHTOOL_OPTS="-K eth2 lro off" ETHTOOL_OPTS="-K eth3 lro off"
4. If the version of initscripts is earlier than initscripts-9.03.27-1 then on rc.local
/sbin/ethtool -K eth0 lro off /sbin/ethtool -K eth1 lro off /sbin/ethtool -K eth2 lro off /sbin/ethtool -K eth3 lro off ..... etc
Comments
0 comments
Please sign in to leave a comment.