Failed to detect device mapper node

Issue

 This error message in the OnApp UI log:

Fatal: OnApp::Actions::Fatal Storage API Call failed: {"result"=>"FAILURE", "error"=>"Failed to offline the VDisk ############## with error {'result': 'FAILURE', 'error': \"onappstore offlineDisk failed for vdisk /##############] with error map: [('123456789', 'Failed to detect device mapper node [/dev/mapper/##############] disappear. out:, err:None')] and optional error: Failed to perform operation 15 for vdisk ############## on node ['123456789'] with error map: [('123456789', 'Failed to detect device mapper node [/dev/mapper/##############] disappear. out:, err:None')]\"}."}

Environment

OnApp 5.x

Integrated Storage

Resolution

This generally occurs when the dmsetup mapped device is is present somewhere else.  You will want to go through each hypervisor and backup server to check for the disk with the following command:

[root@hv]# dmsetup ls | grep ##############

 Once that has been found you should be able to bring the disk offline so you can bring it online on the hypervisor you need it to be on. To make sure, the following steps should be followed:

1. Make sure the virtual machine is offline.  Even if the interface says it's offline, it may be running on the server which the disk was active. Check it with this:
   

   [root@hv]# ps auxf | grep vmidentifierhere

   Alternatively, you can check using xen or kvm with these commands:

   [root@hv]# xm list | grep vmidentifierhere  #for xen
   
   [root@hv]# virsh list | grep vmidentifier here  #for kvm

   If the virtual machine is running, shut it down with the respective command:

   [root@hv]# xm shutdown vmidentifierhere #for Xen
   
   [root@hv]# virsh shutdown vmidentifierhere #for KVM

2. Run xm list or virsh list every few seconds to check for the domain to go offline.
3. Once the VM is off, try to use onappstore to offline the disk, departition, and remove the mapped device:

   [root@hv]# onappstore offline uuid=##############
   This command will usually fail with the same error as above
   [root@hv]# kpartx -d -p X /dev/mapper/##############
   
   [root@hv]# dmsetup remove ##############

4. Verify that the mapped device is gone with dmsetup ls | grep ##############

 Once you have determined that the problem hypervisor no longer has the device mapped, you may attempt the original transaction again.