===== Live Migration howto =====

Live migration of a virtual machine allow a VM executed by a certain compute node (which is going to fail or to be shutoff for maintenance), to be executed by another compute node with virtually zero downtime experienced by the virtual machine's users.

By default this operation can be performed only by the OpenStack admin user, which is the only one who has access to the mapping of virtual machines to the physical hypervisors (in fact admin can even force the creation of a virtual machine on a certain compute node; a normal user can't).

Other info are reported in the [[http://wiki.infn.it/progetti/cloud-areapd/operations/production_cloud/remove_a_compute_node_howto|Remove a compute node HowTo]]

==== Assumptions ====
It is assumed that the nova instance lib directory (usually ''/var/lib/nova/instances'') is remotely mounted by a gluster endpoint (or other shared file system), and that [[http://wiki.infn.it/progetti/cloud-areapd/keystone-glance_high_availability/openstack_ha/temporary_compute|this]] guide has been followed to configure the compute node. 

==== The command ====
The command to live migrate a virtual machines from a hypervisor to another one is:

<code bash>
nova live-migration <instance_name_or_ID> <target_hypervisor_hostname>
</code>

   * Note 1: ''<target_hypervisor_hostname>'' MUST be different from the current node running the virtual instances (use fully qualified name).
   * Note 2: the virtual instance files (''/var/lib/nova/instances'') must be on a file system shared between the two hypervisors.
   * Note 3: the status of the VM must be ACTIVE
   * Note 4: it is strongly reccommended that both hypervisors have the same OS and libvirt version. It's possible to update an old libvirt with yum, but then you have to restart manually libvirtd, then the openstack-nova-compute service
   * Note 5: it is possible to overcome the openstack "invtsc" CPU feature bug, deleting the invtsc section from the file /usr/share/libvirt/cpu_map.xml


If the migration fails, before retrying the migration command, reset the state of the VM

<code bash>
nova reset-state --active  11f98ac3-ee40-4735-b644-d2f2c7c176e7
</code>

If a VM is in SHOUTOFF status the command to be used is different and doesn't allow to choose the destination hypervisor:

<code bash>
nova migrate e14e276d-4594-481d-8cd1-c92831f06632
</code>

If the VM is in the VERIFY_RESIZE status, you must confirm through the dashboard the resize request; the VM now is ready to be migrated as a SHOUTOFF VM.

If the VM is PAUSED, the migration IS NOT POSSIBLE: the VM must be terminated or resumed via dashboard, then migrated; it is strongly recommended to contact the user and sync with him.

If the hypervisor of the VM is dead, use the command:

<code bash>
nova evacuate --on-shared-storage e14e276d-4594-481d-8cd1-c92831f06632 cld-nl-03.cloud.pd.infn.it
</code>

==== Helpful commands ====
Pre-source code:
<code bash>
. keystone_admin.sh
</code>

Close a Compute Node:
<code bash>
nova-manage service disable cld-nl-02.cloud.pd.infn.it nova-compute
nova-manage service list | grep cld-nl
</code>


Compute Node VM list (Live Migration Helper Script):
<code bash>
[root@cld-ctrl-01 ~]# bin/vmListLiveMigHelper.sh cld-nl-01
</code>

Compute Node VM list:
<code bash>
nova list --host cld-nl-02.cloud.pd.infn.it --all-tenants
</code>

Compute Node resources:
<code bash>
nova host-describe cld-nl-02.cloud.pd.infn.it
</code>

VM details:
<code bash>
nova show 2ac6ec4b-eaf4-440a-aac4-46d3cdf6e19b
</code>
<code bash>
nova diagnostics <istanceName>
</code>