Poor TCP Performance on VMs; dmesg:"received packets cannot be forwarded while LRO is enabled"

Issue

After installing a new Hypervisor, very poor TCP performance was experienced on the VMs residing on that Hypervisor.

Environment

Any OnApp version

Troubleshooting

1. Download speed tests where 4 Kb/s, a 500MB file estimated a 62-hour download time:

[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip    
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  500M    0 35944    0     0   2324      0 62:39:57  0:00:15 62:39:42  2350^C

2. The same test from the Hypervisor showed much faster download speed - 16MB/s.

3. While checking the logs, the following was found on /var/log/dmesg and dmesg:

bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled

Resolution

1. Disable LRO on each NIC:

ethtool -K eth2 lro off

2. After disabling LRO, the TCP performance was fine on the VMs:

[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip 
 % Total % Received % Xferd Average Speed Time Time Time Current 
 Dload Upload Total Spent Left Speed 
100 500M 100 500M 0 0 13.2M 0 0:00:37 0:00:37 --:--:-- 17.2M

3. To have the config persist across reboots, use ETHTOOL_OPTS (initscripts-9.03.27-1 or later)on the rc.local of the HV.

#Disabling LRO after poor TCP performance identified:

ETHTOOL_OPTS="-K eth0 lro off" 
ETHTOOL_OPTS="-K eth1 lro off" 
ETHTOOL_OPTS="-K eth2 lro off" 
ETHTOOL_OPTS="-K eth3 lro off"

4. If the version of initscripts is earlier than initscripts-9.03.27-1, then on rc.local:

/sbin/ethtool -K eth0 lro off
/sbin/ethtool -K eth1 lro off
/sbin/ethtool -K eth2 lro off
/sbin/ethtool -K eth3 lro off
..... etc