Poor TCP Performance on VMs; dmesg:"received packets cannot be forwarded while LRO is enabled"

Issue

After installing a new Hypervisor, very poor TCP performance was experienced on the VMs residing on that Hypervisor.

Environment

OnApp 3.x

Troubleshooting

1. Download speed tests where 4 Kb/s, a 500MB file estimated a 62 hour download time. 

[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip    
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  500M    0 35944    0     0   2324      0 62:39:57  0:00:15 62:39:42  2350^C

2. The same test from the Hypervisor showed much faster download speed 16MB/s

3. While checking the logs, the following was found on /var/log/dmesg and dmesg

bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond1.175: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled 
bond0.200: received packets cannot be forwarded while LRO is enabled

Resolution

1. Disable LRO on each NIC/NICS

ethtool -K eth2 lro off

2. After disabling LRO TCP performance was fine on the VMS

[root@tes-ppa ~]# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip 
 % Total % Received % Xferd Average Speed Time Time Time Current 
 Dload Upload Total Spent Left Speed 
100 500M 100 500M 0 0 13.2M 0 0:00:37 0:00:37 --:--:-- 17.2M

3. To persist the config across reboots, use ETHTOOL_OPTS (initscripts-9.03.27-1 or later)on the rc.local of the HV

#Disabling LRO after poor TCP performance identified

ETHTOOL_OPTS="-K eth0 lro off" 
ETHTOOL_OPTS="-K eth1 lro off" 
ETHTOOL_OPTS="-K eth2 lro off" 
ETHTOOL_OPTS="-K eth3 lro off"

4. If the version of initscripts is earlier than initscripts-9.03.27-1 then on rc.local

/sbin/ethtool -K eth0 lro off
/sbin/ethtool -K eth1 lro off
/sbin/ethtool -K eth2 lro off
/sbin/ethtool -K eth3 lro off
..... etc