LVM logical volumes on FC Data Stores are activated automatically after hypervisor reboot

Issue

All logical volumes on FC Data Stores are activated on boot. Delaying the boot and risking the corruption of VS Data.

Troubleshooting/Sypmtoms

1. Logs on your Control Panel will be fill up the error WrongActivatedLogicalVolume to the point that may hung your CP.

2. The HV may delay boot as it will be activating the Logical volumes. As soon as the HV is up you will see that *all LVs are active, check if the same LVs are active in another Hypervisor.

[root@hyper1 ~]# lvscan
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/pceruad7sq7jm1' [8.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/yy0islmkopqq18' [16.00 GiB] inherit
 ACTIVE          '/dev/onapp-nb0kc3bizfa8ma/eedobaydl93rlm' [16.00 GiB] inherit
 ACTIVE         '/dev/onapp-nb0kc3bizfa8ma/foqei4zhds0l5i' [8.00 GiB] inherit
 ACTIVE          '/dev/onapp-nb0kc3bizfa8ma/eic26j80s14mma' [1.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/t5neq6e4m46vpg' [4.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/inez6gptdtp4kt' [1.00 GiB] inherit
 ACTIVE         '/dev/onapp-nb0kc3bizfa8ma/i8sqc2mfy2ctnc' [1.00 GiB] inherit
 ACTIVE          '/dev/onapp-nb0kc3bizfa8ma/qfml7nyllkr0ve' [2.00 GiB] inherit
 ACTIVE         '/dev/onapp-nb0kc3bizfa8ma/f30w8gnxvnlnio' [1.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/egy4c56dkkddma' [8.00 GiB] inherit
 ACTIVE      '/dev/onapp-nb0kc3bizfa8ma/pc4ockfav6lfev' [8.00 GiB] inherit

[root@hyper2 ~]# lvscan
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/pceruad7sq7jm1' [8.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/yy0islmkopqq18' [16.00 GiB] inherit
 ACTIVE          '/dev/onapp-nb0kc3bizfa8ma/eedobaydl93rlm' [16.00 GiB] inherit
 ACTIVE         '/dev/onapp-nb0kc3bizfa8ma/foqei4zhds0l5i' [8.00 GiB] inherit
 ACTIVE          '/dev/onapp-nb0kc3bizfa8ma/eic26j80s14mma' [1.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/t5neq6e4m46vpg' [4.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/inez6gptdtp4kt' [1.00 GiB] inherit
 ACTIVE         '/dev/onapp-nb0kc3bizfa8ma/i8sqc2mfy2ctnc' [1.00 GiB] inherit
 ACTIVE          '/dev/onapp-nb0kc3bizfa8ma/qfml7nyllkr0ve' [2.00 GiB] inherit
 ACTIVE         '/dev/onapp-nb0kc3bizfa8ma/f30w8gnxvnlnio' [1.00 GiB] inherit
 ACTIVE        '/dev/onapp-nb0kc3bizfa8ma/egy4c56dkkddma' [8.00 GiB] inherit
 ACTIVE      '/dev/onapp-nb0kc3bizfa8ma/pc4ockfav6lfev' [8.00 GiB] inherit

Root Cause

This is a Bug that applies to CentOS/RHEL and Fedora19/20 it's been identified on 6.4 5.5 upwards

As per https://bugzilla.redhat.com/show_bug.cgi?id=1009812#c10

"In FC system, physical volumes are connected to the system early on boot, when rc.sysinit or netfs run. These scripts perform auto-activation of all lvm volume groups, which activates all lvs on shared storage."

Solution

There is a proposed workaround which is to use auto_activation_volume_list filter on lvm.conf here is what the filter does:

auto_activation_volume_list — This acts as a filter through which all requests to autoactivate a logical
volume on this machine are passed. A logical volume is autoactivated if it matches an item in the list.
Volumes must also pass the volume_list filter, if present. Tags must be preceded by @ and are checked
against all tags defined in the logical volume and volume group metadata for a match. @* is short-hand to
check every tag set on the host machine (see tags above). Logical volume and volume groups can also be
included in the list by name e.g. vg00, vg00/lvol1.

By specifying that parameter on your lvm.conf file we will only activate "vg" on boot. The others will be ignored.

auto_activation_volume_list = [ "vg" ]

Please note this is a Centos/RHEL BUG, it's not related to OnApp.

*vg refers to any volume group used to boot the system, otherwise your system will not boot as it will not activate his logical volumes. Example

auto_activation_volume_list = [ "vg" ]

VG is the volume group  that contain root/swap/tmp

  ACTIVE            '/dev/vg/root' [41.78 GiB] inherit
  ACTIVE            '/dev/vg/tmp' [1.00 GiB] inherit
  ACTIVE            '/dev/vg/swap' [31.50 GiB] inherit

 1. Specify the volume group to filter in /etc/lvm.conf, please note the parmeter will only work under the hierarchy of activation{

}

activation {
    # Set to 1 to perform internal checks on the operations issued to
    # libdevmapper.  Usefula for debugging problems with activation.
    # Some of the checks may be expensive, so it's best to use this
    # only when there seems to be a problem.
    checks = 1
       
    auto_activation_volume_list = [ "vg" ]
    # Set to 0 to disable udev synchronisation (if compiled into the binaries).
    # Processes will not wait for notification from udev.