Customize Failover Scripts
Currently, the failure response on OnApp is not at an acceptable production level. The loss of the management network, and possibly the VM networks could cause a situation where the management station is not aware whether the HV is still connected to storage. In this situation, there is no way to ensure that recovery is done without booting up VMs twice.
Right now, in current scenario of failover, only software checks(ssh, snmp, ping, etc) are implemented and all checks are being done via HV mgmt and VMs network(ping).
I’d suggest adding option to run some custom (self written script) or command from CP and wait for the output string or exit code as it’s implemented in Onapp Recipe and this script of command should be used as a last step in Onapp failover before considering HV as completely offline and start migrating VMs from it. Goal of this command or script is to execute it and make sure that node is completely offline or restarted (not via MGMT interface). It could ipmitool command for SuperMicro Servers, but could be any others remote management platform interfaces, so onapp won’t be tied to Supermicro servers. Another way to consider HV as completely offline before migrating VMs from it: Run custom “Power cycle Command" from CP without logging to HV via mgmt network, but in this scenario output needs to be checked or response code(again similar to Onapp Recepi). So, this command will be only run in Failover Steps if it’s defined.
Ideally, the HA functionality would be performed without the need for the CP server, and there should be some level of coordination between the HV’s.
Storage heart beating should be enforced to ensure that an HV is not connected to the storage before booting up those VMs to avoid corruption.
-
Using Integrated Storage you already can not start a VM somewhere if the downed HV still sees storage, since the storage controllers are managing things and prevent activating in 2 places already.
For LVM datastore - perhaps taking ~1GB and formatting as some cluster-aware filesystem the HVs could create locks on the VG itself for the VM disks that it has activated as a primary check / using the LVM datastore itself to pass messages? or extending storageAPI so that LVM acts like IS - trying to activate LVM disk requires checking against all the other HVs using management, storage, and backup networks as configured, and only proceeding when the other HVs fail to reply or reply the VM disk isn't active there.
Please sign in to leave a comment.
Comments
1 comment