Skip to content

KVM and Graceful Shutdown on Ubuntu

For quite some time, I have been trying to figure out, how to gracefully shutdown the KVM-based virtual machines running on a Ubuntu 10.04 LTS (Lucid Lynx) host system. This problem consists of two parts: First you have to make the virtual machines support the shutdown event from libvirt and second you have to call the shutdown action for each virtual machine on system shutdown.

The first part is very easy for Linux VMs and also not too hard for Windows VMs. I described the necessary steps in my wiki

The seconds part is harder to accomplish: On Ubuntu 8.04 LTS (Hardy Heron) I just modified the /etc/init.d/libvirt-bin script to call a Python script in the stop action. This solution was not perfect, as it meant that the virtual machines were also shutdown, when libvirtd was just restarted, however it was a quick and easy solution.

For Ubuntu 10.04, the init script has been converted to an Upstart job. So the easiest way was to create a upstart job that is starting on the stopping libvirt-bin event. However, this did not solve the problem, because the system powered off or rebooted before the shutdown of the virtual machines was finished. As it turns out, Ubuntu 10.04 uses an odd combination of Upstart jobs and traditional init scripts. This leads to a situation, where /etc/init.d/halt or /etc/init.d/reboot are called, before all upstart jobs have stopped, when one of the upstart jobs needs a significant amount of time to stop. This can be solved by adding an init script, than runs before the halt or reboot scripts and waits for the respective Upstart job to finish. In fact, it is best to run this script before the sendsigs script to avoid processes started by one of the upstart jobs to receive a SIGKILL.

I added the complete scripts and configuration files needed for this feature to my wiki. In fact, this solution also ensures, that the virtual machines are only shutdown if libvirt is stopped because of a runlevel change. Thus, the libvirt-bin package can now be upgraded without resulting in a restart of the VMs.

For me, automatically shutting down the virtual machines is very important. The KVM hosts I manage are connected to an uninterruptible power supply with limited battery time. Although in the past years I remember only a single time, the host systems were shutdown because the battery was nearly empty (most power interrupts are very short), I want to make sure that all virtual machines are in a safe, consistent state, when the power finally goes off. So I hope that the scripts in the wiki are also helpful to other people, having the same problem.

Problems when Upgrading to GRUB 2

Yesterday, I upgraded the last systems, which where still running on GRUB Legacy (GRUB 1) to GRUB 2 (the grub-pc package on Ubuntu).

On one of the systems, I experienced the problem, that upgrade-from-grub-legacy could not install GRUB. When I tried to install GRUB manully using grub-install, I got a strange error message:

/usr/sbin/grub-setup: warn: Your embedding area is unusually small.  core.img won't fit in it..
/usr/sbin/grub-setup: warn: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and its use is discouraged..
/usr/sbin/grub-setup: error: if you really want blocklists, use --force.

It took me some time to figure out, that this system had an unusual disk geometry, resulting in not enough space being available before the start of the first partition. The output of fdisk -lu /dev/sda looked like this:

Disk /dev/sda: 320.1 GB, 320071884800 bytes
64 heads, 32 sectors/track, 305244 cylinders, total 625140400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009153c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              32      585727      292848   83  Linux
/dev/sda2          585728   625139711   312276992    5  Extended
/dev/sda5          585760   215429119   107421680   8e  Linux LVM
/dev/sda6       215429152   430272511   107421680   8e  Linux LVM
/dev/sda7       430272544   625139711    97433584   8e  Linux LVM

Usually, the first partition starts at sector 63 (for systems installed more recently, 2048 is also common), but for this system it started at sector 32. I guess that this was cause by the strange geometry, which did not specify the usual 255 heads and 63 sectors/track. I am not sure, whether the disks (two Western Digital 320 GB drives) really have this strange geometry, or the RAID-Controller, a 3ware Escalade 8006-2LP, reports a different geometry. However, for sure the first partition being to close to the start of the disk was the cause of the problem with GRUB.

Unfortunately, I could not move the partition using parted and gparted does not allow to specify the start and end of a partition in sectors, but only in MiB (and only supports aligning to MiB or cylinders, which did not help considering the odd geometry). Therefore I decided to delete the partition (after all it was only the boot partition and its content could easily be stored on the root partition) and create a new partition using fdisk.

In fact, when (re-)creating the partition, fdisk suggested to start the partition at sector 2048, which I accepted. No surprise, after that change GRUB could be installed to the MBR without any problems.

The good thing about this problem was, that the time I invested to get a high-availability setup running for most important services, finally paid of. I for sure would have been less relaxed, while figuring out and fixing the GRUB problem, when the services usually running on this machine would have been offline.