For quite some time, I have been trying to figure out, how to gracefully shutdown the KVM-based virtual machines running on a Ubuntu 10.04 LTS (Lucid Lynx) host system. This problem consists of two parts: First you have to make the virtual machines support the shutdown event from libvirt and second you have to call the shutdown action for each virtual machine on system shutdown.
The first part is very easy for Linux VMs and also not too hard for Windows VMs. I described the necessary steps in my wiki
The seconds part is harder to accomplish: On Ubuntu 8.04 LTS (Hardy Heron) I just modified the /etc/init.d/libvirt-bin script to call a Python script in the stop action. This solution was not perfect, as it meant that the virtual machines were also shutdown, when libvirtd was just restarted, however it was a quick and easy solution.
For Ubuntu 10.04, the init script has been converted to an Upstart job. So the easiest way was to create a upstart job that is starting on the stopping libvirt-bin event. However, this did not solve the problem, because the system powered off or rebooted before the shutdown of the virtual machines was finished. As it turns out, Ubuntu 10.04 uses an odd combination of Upstart jobs and traditional init scripts. This leads to a situation, where /etc/init.d/halt or /etc/init.d/reboot are called, before all upstart jobs have stopped, when one of the upstart jobs needs a significant amount of time to stop. This can be solved by adding an init script, than runs before the halt or reboot scripts and waits for the respective Upstart job to finish. In fact, it is best to run this script before the sendsigs script to avoid processes started by one of the upstart jobs to receive a SIGKILL.
I added the complete scripts and configuration files needed for this feature to my wiki. In fact, this solution also ensures, that the virtual machines are only shutdown if libvirt is stopped because of a runlevel change. Thus, the libvirt-bin package can now be upgraded without resulting in a restart of the VMs.
For me, automatically shutting down the virtual machines is very important. The KVM hosts I manage are connected to an uninterruptible power supply with limited battery time. Although in the past years I remember only a single time, the host systems were shutdown because the battery was nearly empty (most power interrupts are very short), I want to make sure that all virtual machines are in a safe, consistent state, when the power finally goes off. So I hope that the scripts in the wiki are also helpful to other people, having the same problem.
A real problem to solve ! Unbelievable existing on a server system (ubuntu).
Another problem with shutting down VM is the order of shutting down (app server first please).
It is evident that stopping the host's VMs is a task for the virtual manager.
vmware (in 2006) had a simple "autostop.order = x" field. May not solve every case.
in libvirt world, the discussion is 'it does not solve every case so it is not implemented' !
A workaround could be autostart links to xml files prefixed (10- as init scripts) and your script following that order for killing VM, then remaining VM.
Franck
The script I wrote is mainly intended for emergency-shutdown situations (like the UPS becoming low on battery). Therefore, the main goal is to get all virtual machines into a consistent state as soon as possible.
I think there is no real need for a specific shutdown order. To use your example, if you have an application server and a webserver running the frontend, if the application server is shutdown first, the web frontend will stop working. However, if the web server is shutdown first, the web frontend will not be available either. But the state of the whole system will always be consistent, independent of the shutdown order. If you need to maximize the availability of certain services, I suggest using a Linux-HA setup.
For example, I am using a Heartbeat/Pacemaker setup consisting of two virtual machines running on two different host systems. Pacemaker ensures that the services running in the seconds virtual machine take over, if the first virtual machine fails for some reasons.
I'm trying to implement your scripts as a workarround on this libvirt issue, but i'm using debian squeeze.
It seems that upstart is not installed on my box, afaik upstart is a replacement for sysV scripts, is it safe to install upstart without breaking my system?
Thanks in advance!
Regards,
Diego Bendlin
However, the good news is, that you can use nearly the same approach with SysV-Init as well:
Before Ubuntu moved libvirt to upstart, I just modified /etc/init.d/libvirt-bin in order to call the shutdown-domains script on the stop action. You use the same /usr/local/bin/libvirt-shutdown-domains script, and add the section from the Upstart job between "script" and "end script" to the stop section of /etc/init.d/libvirt-bin. As this effectively blocks the shutdown process, you do not need the /etc/init.d/await-libvirt-bin-stopped script, which you need when using Upstart. Just make sure, that the code you add to /etc/init.d/libvirt-bin is before libvirtd is stopped, because otherwise the Python script cannot work.