Performance Document + perftest.sh

OnApp Storage: High-level architecture and technical details
Updated: March 27th 2013
Original document created: March 19th 2013
Authors: John Thomson, Michal Belczyk and Julian Chesterfield

Introduction:
OnApp Storage is a new product released as part of the OnApp 3.0 GA release.
It is  utilises the storage available on existing cloud platform hardware in datacentres to offer a scalable, resilient and efficient Storage Area Network (SAN). Low-cost SANs are adequate for low-performance workloads and managing data that is not critical.
Where the performance throughput or the scalability is important, OnApp Storage is an alternative to high-cost Enterprise SAN hardware.

This document describes the performance of the OnApp Storage system. Some of the features may have changed since this document was created. It is important to review the change-logs from subsequent platform releases to determine the changes.


Steps for measuring performance

Dom0 Network performance
In this first test we will analyse the performance between Dom0 on two Hypervisors.
The testing will check the asymmetric network performance to see if there is an issue in either direction of the network communication.
Firstly - log into the two HVs through the CP server.
ssh root@<ipaddr_cpserver>
[root@<ipaddr_cpserver> ~]# ssh <ipaddr_hv>
Check the network configuration is set up correctly for both HVs.

Firstly, please run the cloudboot healthcheck script and check results in case there are any anomalies. Also please verify that all SAN NICs across all Hvs have the default MTU settings of 1500 and are not enabled with jumbo frames.
Now its necessary to run the iperf binary between all Hvs to measure the network throughput and verify that network performance is as expected. Run this twice for each pair of Hvs, reversing the source and destination (ipaddr_hv_1 and ipaddr_hv_2) each time:
[root@<ipaddr_hv_1> ~]#  iperf -s -N
This runs iperf in server mode with a flag to avoid delays.
[root@<ipaddr_hv_2> ~]#  iperf -c <ipaddr_hv_1> -N

This will then provide the speed of the connection from the client to the server.
This value should be recorded as Perf_D0_H2_D0_H1.
Reversing the roles of the two iperfs (change from client to server and vice-versa) the performance from HV1 to HV2 will be calculated and this should be recorded as Perf_D0_H1_D0_H2.
If the two values differ or the performance is lower than expected then this may be indicative that something at the network level is not configured correctly.


Bonded interfaces
[root@<ipaddr_hv_1> ~]# brctl show
Will show the NICs that are bonded together as onappstorebond. If there is no bond then it will show ethx.
If there is a network bond then
[root@<ipaddr_hv_1> ~]# cat /proc/net/bonding/onappstorebond
Check that the links are up and that they are running at the same speed.


Measuring Raw Disk Performance
To check the raw speed of the disks follow these steps.
These tests will need to be adapted for custom bond scripts. Please note also that all Hvs should be in default PV mode, not hardware passthrough.

Log into the HV without creating a datastore
Stop any running OnApp groupmon services
[root@<ipaddr_hv_1> ~]# service groupmon stop

At this point run the test that is included in the script perftest.sh blockdevice against all the storage drives that are added to the integrated storage platform. N.B. This is a destructive test, so be very careful not to use drives that have not been added to the storage platform:

After the destructive testing has been performed run the following for all drives that are being added to the integrated storage platform:
formatandconfigure /dev/sd<x>
This will get the drive(s) prepared again for usage in onappstorage.
Then need to restart the OnApp storage service:
[root@<ipaddr_hv_1> ~]# service groupmon start

Datastore performance
Using the OnApp web interface or the CLI create a datastore with a single drive member.
From CLI:
[root@<ipaddr_hv_1> ~]# onappstore createds name=<datastore_name> membership_count=1  owners=<node UUID of one of the local drives>
Once the datastore is created - verify it is showing from CLI
[root@<ipaddr_hv_1> ~]# onappstore listds
Create a 4GB VDisk
[root@<ipaddr_hv_1> ~]# onappstore create name=<vdisk_name> size=4096 Datastore=<ds_uuid>
A successful creation will return the vdisk_uuid, the owners of the disk, some other details and result=SUCCESS.
e.g. uuid=<tcm3ly5fi46ebj> owners=<992993183> Datastore=<1ag3m5dqoh86yc> stripesize=<256> result=SUCCESS

Next, locate the frontend UUID of the first HV for testing:
[root@<ipaddr_hv_1> ~]# onappstore getid
result=SUCCESS uuid=<3654210369>

and online the vdisk created earlier for testing the data path from that HV:
[root@<ipaddr_hv_1> ~]# onappstore online frontend_uuid=<value from getid command> uuid=<vdisk uuid created above>
result=SUCCESS

Check that it has been onlined successfully.
[root@<ipaddr_hv_1> ~]# ps auxww | grep -w bdevclient | grep  tcm3ly5fi46ebj
root <10218> 0.0 0.0 850 80 ? Ss 12:12 0:00 /usr/sbin/bdevclient <10.200.1.1> <59051> /dev/nbd0 -b 65536 -U  <tcm3ly5fi46ebj>

Now write to the vdisk to check the local write performance:
[root@<ipaddr_hv_1> ~]# dd if=/dev/zero of=/dev/mapper/tcm3ly5fi46ebj bs=64M oflag=direct
dd: writing `/dev/mapper/tcm3ly5fi46ebj': No space left on device
17+0 records in
16+0 records out
1073741824 bytes (1.1GB) copied, <5.61974> s, <191> MB/s
Record as Perf_w_VD1_HV1_2R_1S_0Over
(2R_1S_0Over should be the data store configuration used for the set up)

For local read performance:
[root@<ipaddr_hv_1> ~]# dd of=/dev/null if=/dev/mapper/tcm3ly5fi46ebj bs=64M iflag=direct
16+0 records in
16+0 records out
1073741824 bytes (1.1GB) copied, <3.80509> s, <282> MB/s
Record as Perf_r_VD1_HV1_2R_1S_0Over
(2R_1S_0Over should be the data store configuration used for the set up)

Change to HV2 and offline the VDisk
[root@<ipaddr_hv_2> ~]# onappstore offline uuid=tcm3ly5fi46ebj

We then need to run the online on the remote HV
[root@<ipaddr_hv_2> ~]# onappstore getid
result=SUCCESS uuid=<3156021174>
[root@<ipaddr_hv_2> ~]# onappstore online uuid=<tcm3ly5fi46ebj> frontend_uuid=<3156021174>
result=SUCCESS

Run the same commands for remote performance
[root@<ipaddr_hv_2> ~]# dd if=/dev/zero of=/dev/mapper/tcm3ly5fi46ebj bs=64M oflag=direct
-> Perf_w_HV2_VD1_HV1_2R_1S_0Over
[root@<ipaddr_hv_2> ~]# dd of=/dev/null if=/dev/mapper/tcm3ly5fi46ebj bs=64M iflag=direct
-> Perf_r_HV2_VD1_HV1_2R_1S_0Over

You will get different performance levels depending on the configuration.

Note that the local path test initially should produce read and write throughput numbers that are very similar to the raw disk performance for the same disk that you measured previously. The remote performance tests should be somewhat similar to the raw network throughput measurement you ran previously between the 2 HVs.

VM Disk Performance
To test the disk performance that would be found in a Linux VM there is a tool that OnApp uses to create a small VM instance with performance tests. This is called ddwrapper.
To run it log into HV and run the following command:
[root@<ipaddr_hv_1> ~]# ddwrapper device=/dev/mapper/ej8f2sc74p6wgv bs=64M count=8

result=SUCCESS write_results=536870912_bytes_(537_MB)_copied,_2.73083_seconds,_197_MB/s write_time=2.73083 write_mbps=197 read_results=536870912_bytes_(537_MB)_copied,_2.32336_seconds,_231_MB/s read_time=2.32336 read_mbps=231


Run it from another HV:
[root@<ipaddr_hv_2> ~]# ddwrapper device=/dev/mapper/ej8f2sc74p6wgv bs=64M count=8

result=SUCCESS write_results=536870912_bytes_(537_MB)_copied,_2.74059_seconds,_196_MB/s write_time=2.74059 write_mbps=196 read_results=536870912_bytes_(537_MB)_copied,_1.03714_seconds,_518_MB/s read_time=1.03714 read_mbps=518

ddwrapper creates a VM that reports a device /dev/mapper/<vdisk> and runs performance tests on the VM. The drive is para-virtualised (handled slightly differently for xen and kvm). This performance then checks the block performance of the VM drive.

Results
If you have issues with the performance please add your results from above into a table with as many entries completed as possible for analysis.

Notes
Tests like bonnie++ (http://www.googlux.com/bonnie.html) and hdparm perform quite specific tests that may not perform optimally on virtual disk drives. Some OnApp templates do not come with virtIO. The templates that come with VirtIO are specifically named in the template name. Templates with VirtIO tend to perform much better.
We currently do not perform ext3 or other filesystem tests.