About two years ago, I wrote an article about how to make libvirt on Ubuntu 10.04 LTS to shutdown the virtual machines gracefully, when the host system is shutdown or rebooted.
Now I recently found out, that they implemented a similar approach in Ubuntu 12.04 LTS. The only problem with this is, that the default timeout is too short (30 seconds) for virtual machines running complex services. Therefore, I documented how to change this timeout in my wiki.
In an earlier blog post I claimed that I was seeing problems with VMs using the virtio driver for networking on an Ubuntu 12.04 LTS KVM host using DHCP.
However, as far as I am concerned, this claim was wrong. I now figured out, that the messages about bad UDP checksums had nothing to do with my problem. I was rather experiencing the problems caused by a configuration that did not list the VLAN network interface (eth0.X) on which the DHCP relay agent received the answers from the DHCP server.
The mean thing is, that switch away from virtio fixed this problem. However, this was not because of the UDP checksums now being right (this was merely a side effect). It fixed the problem, because when not using the virtio driver, the DHCP relay agent would receive the answer packets, even if they were received on a VLAN interface it was not listening to. I can only guess that the implementation for VLAN-tagged interfaces is slightly different when using the virtio driver.
After adding the interface to the list of interfaces used by the DHCP relay agent, the DHCP packets are relayed correctly, even if using the virtio driver. The messages about bad UDP checksums now reappeared in the log file, but obviously this is not causing any trouble.
On the other hand, according to a bug report some users really seem to have problems with DHCP when using the virtio driver. However, this might only affect Ubuntu 12.04 LTS guests but not VMs on a Ubuntu 12.04 LTS host.
Lately I experienced a problem with KVM-based virtual machine running a DHCP server and another one running a DHCP relay (for both I use the ISC implementations). The DHCP relay was complaining about "bad udp checksums". Using tcpdump and wireshark I quickly found out, that the software was right and the UDP checksums were in fact wrong. After some searching, I found a bug report, that basically described the same problem.
Although I cannot verify this, I think the problem might be related to the fact that I recently upgraded the host machine from Ubuntu 10.04 LTS (Lucid) to Ubuntu 12.04 LTS (Precise). As a workaround, I deactivated the use of the "virtio" support for the network interface in both virtual machines, which seems to fix the problem, because then the UDP checksums are correct.
However, when I performed the same change for a virtual machine still running on an Ubuntu 10.04 LTS host, this actually caused a problem: If VLAN interfaces are used inside the virtual machine, the normal non-virtio driver will screw things up on a Ubuntu 10.04 LTS host.
Long story short: For virtual machines running on a Ubuntu 10.04 LTS host you should use and for a Ubuntu 12.04 LTS you should avoid virtio networking.
Update [2012-07-08]: It seems like the conclusions I draw in this article are actually wrong. Therefore I posted an update clarifying the situation.
Some time ago Ubuntu 10.04 LTS received a Firefox update from the 3.6 branch to the more recent versions (10.0+).
After that upgrade, Firefox on one of my Ubuntu systems suddenly appeared in English instead of the correct locale (German in my case). I first thought that this might be a problem with some localization packages not being installed correctly. However, the problem persisted after upgrading the system to Ubuntu 12.04 LTS.
Some globally extensions (in particular the language pack) showed up in an old version in the add-ons list, although the newest version was installed. Finally I found out, that Firefox looks for the extensions in /usr/lib/firefox/extensions. The new language packs however had been installed in /usr/lib/firefox-addons/extensions. On other systems, /usr/lib/firefox/extensions is a symbol link to /usr/lib/firefox-addons/extensions. In my case however the directory existed and contained the files from the old versions of the language packs.
For some reasons, the old language packs (which had different package names) had not been removed and thus the Firefox upgrade did not place the symbol link (because the directory was not empty). After manually removing the old versions of the language packs, deleting the directory and reinstalling Firefox, the symbol link was created automatically and suddenly the globally installed Firefox add-ons worked again.
For quite some time, I have been trying to figure out, how to gracefully shutdown the KVM-based virtual machines running on a Ubuntu 10.04 LTS (Lucid Lynx) host system. This problem consists of two parts: First you have to make the virtual machines support the shutdown event from libvirt and second you have to call the shutdown action for each virtual machine on system shutdown.
The first part is very easy for Linux VMs and also not too hard for Windows VMs. I described the necessary steps in my wiki
The seconds part is harder to accomplish: On Ubuntu 8.04 LTS (Hardy Heron) I just modified the /etc/init.d/libvirt-bin script to call a Python script in the stop action. This solution was not perfect, as it meant that the virtual machines were also shutdown, when libvirtd was just restarted, however it was a quick and easy solution.
For Ubuntu 10.04, the init script has been converted to an Upstart job. So the easiest way was to create a upstart job that is starting on the stopping libvirt-bin event. However, this did not solve the problem, because the system powered off or rebooted before the shutdown of the virtual machines was finished. As it turns out, Ubuntu 10.04 uses an odd combination of Upstart jobs and traditional init scripts. This leads to a situation, where /etc/init.d/halt or /etc/init.d/reboot are called, before all upstart jobs have stopped, when one of the upstart jobs needs a significant amount of time to stop. This can be solved by adding an init script, than runs before the halt or reboot scripts and waits for the respective Upstart job to finish. In fact, it is best to run this script before the sendsigs script to avoid processes started by one of the upstart jobs to receive a SIGKILL.
I added the complete scripts and configuration files needed for this feature to my wiki. In fact, this solution also ensures, that the virtual machines are only shutdown if libvirt is stopped because of a runlevel change. Thus, the libvirt-bin package can now be upgraded without resulting in a restart of the VMs.
For me, automatically shutting down the virtual machines is very important. The KVM hosts I manage are connected to an uninterruptible power supply with limited battery time. Although in the past years I remember only a single time, the host systems were shutdown because the battery was nearly empty (most power interrupts are very short), I want to make sure that all virtual machines are in a safe, consistent state, when the power finally goes off. So I hope that the scripts in the wiki are also helpful to other people, having the same problem.
Yesterday, I upgraded the last systems, which where still running on GRUB Legacy (GRUB 1) to GRUB 2 (the grub-pc package on Ubuntu).
On one of the systems, I experienced the problem, that upgrade-from-grub-legacy could not install GRUB. When I tried to install GRUB manully using grub-install, I got a strange error message:
/usr/sbin/grub-setup: warn: Your embedding area is unusually small. core.img won't fit in it.. /usr/sbin/grub-setup: warn: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and its use is discouraged.. /usr/sbin/grub-setup: error: if you really want blocklists, use --force.
It took me some time to figure out, that this system had an unusual disk geometry, resulting in not enough space being available before the start of the first partition. The output of fdisk -lu /dev/sda looked like this:
Disk /dev/sda: 320.1 GB, 320071884800 bytes 64 heads, 32 sectors/track, 305244 cylinders, total 625140400 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0009153c Device Boot Start End Blocks Id System /dev/sda1 32 585727 292848 83 Linux /dev/sda2 585728 625139711 312276992 5 Extended /dev/sda5 585760 215429119 107421680 8e Linux LVM /dev/sda6 215429152 430272511 107421680 8e Linux LVM /dev/sda7 430272544 625139711 97433584 8e Linux LVM
Usually, the first partition starts at sector 63 (for systems installed more recently, 2048 is also common), but for this system it started at sector 32. I guess that this was cause by the strange geometry, which did not specify the usual 255 heads and 63 sectors/track. I am not sure, whether the disks (two Western Digital 320 GB drives) really have this strange geometry, or the RAID-Controller, a 3ware Escalade 8006-2LP, reports a different geometry. However, for sure the first partition being to close to the start of the disk was the cause of the problem with GRUB.
Unfortunately, I could not move the partition using parted and gparted does not allow to specify the start and end of a partition in sectors, but only in MiB (and only supports aligning to MiB or cylinders, which did not help considering the odd geometry). Therefore I decided to delete the partition (after all it was only the boot partition and its content could easily be stored on the root partition) and create a new partition using fdisk.
In fact, when (re-)creating the partition, fdisk suggested to start the partition at sector 2048, which I accepted. No surprise, after that change GRUB could be installed to the MBR without any problems.
The good thing about this problem was, that the time I invested to get a high-availability setup running for most important services, finally paid of. I for sure would have been less relaxed, while figuring out and fixing the GRUB problem, when the services usually running on this machine would have been offline.
Beim Versuch EDM auf einem openSUSE-11.2-System (x86_64) zum Laufen zu bekommen, traten in bestimmten Fällen immer wieder Speicherzugriffsfehler auf. Nun ist es so, dass ich wirklich kein Freund von RPM-basierten Distributionen im allgemeinen und SUSE im speziellen bin. Aber es wäre zu einfach gewesen dieses Problem auf die Distribution zu schieben. Eine Suche im EPICS-Mailinglisten-Archiv ergab dann auch relativ schnell, dass dieses Problem auch schon auf anderen 64-Bit-Systemen aufgetreten war.
Zunächst untersuchte ich das Problem also mit dem GDB. Damit konnte ich aber lediglich herausfinden, dass der Fehler beim "delete[]" eines Arrays auftrat. Mit etwas Debug-Code konnte ich schnell feststellen, dass es (trotz entsprechender Meldungen der glibc) kein Double-Free-Problem war, kam aber ansonsten nicht wirklich weiter.
Dann entdeckte ich das Werkzeug Valgrind. Valgrind wurde speziell entwickelt um Speicher-Probleme, wie Buffer-Overflows und Memory-Leaks, in C/C++-Programmen zu finden. Tatsächlich konnte ich damit sehr schnell herausfinden, dass ein Array über die Grenzen hinweg beschrieben wurde.
Die Ursache war dann auch schnell gefunden. Der Code, der den Speicher für den Array reserviert, hatte die Größe verschiedener Datentypen fest kodiert. Die Annahme, dass ein "long"-Wert 4 Bytes hat, ist allerdings nur auf 32-Bit-Systemen richtig. Auf 64-Bit-Systemen kann ein "long" 8 Bytes lang sein.Nachdem die festen Größen durch den "sizeof"-Operator ersetzt wurden, verschwand dann auch der Speicherzugriffsfehler.
Valgrind kann also ein wertvolles Werkzeug bei der Lösung von Speicherproblem in C- oder C++-Code sein, deren Ursache andernfalls nur schwer zu finden wäre.
Nachdem ich auf einem Ubuntu 10.04 LTS (Lucid Lynx) System ein reguläres System-Update durchgeführt hatte, welches auch ein Update von libvirt enthielt, konnte ich merkwürdigerweise eine bestimmte VM nicht mehr starten.
Die ausgegebene Meldung
error: Failed to start domain mydomain error: monitor socket did not show up.: Connection refusedwar leider auch nicht besonders aussagekräftig. In den Logfiles fanden sich auch keine weiteren Informationen und libvirtd mit der Option "-v" zu starten brachte mich auch nicht weiter.
Nach viel herumprobieren stellte sich letztendlich heraus, dass die folgenden Zeilen in der Domain-Konfiguration für das Problem verantwortlich waren:
<disk type='block' device='cdrom'>
<source dev='/dev/cdrom'/>
<target dev='hdc' bus='ide'/>
<readonly/>
</disk>
Warum genau die gleiche Konfiguration vor dem Update noch funktioniert hat, kann ich auch nicht sagen. Da ich für die betroffene VM das CD-Laufwerk sowieso nicht benötige, habe ich die Zeilen einfach nur entfernt.
Squid 3.1 introduced support for IPv6, so there is a very good reason for an upgrade from Squid 3.0.
Unfortunately, there is no Ubuntu package for this version, not even in the upcoming Ubuntu release 10.04 Lucid Lynx.
For a while, I have been using the Debian experimental packages of Squid 3.1.0, compiled from source on Ubuntu 8.04. However, now a stable version of Squid 3.1 has been released, but there is no Debian package for Squid 3.1.1.
Therefore, I used the Debian package for Squid 3.1.0.14 as the base for building a package for Squid 3.1.1. I have published this package in source as well as built for Ubuntu 8.04 on amd64. There you can also find a package for libcap2, which is required by Squid 3.1 but not included in Ubuntu 8.04.
Of course, there is no kind of warranty for this packages, so use them at your own risk.
I experiences a problem, when trying to use Scala IDE for Eclipse and Spring IDE at the same time in Eclipse 3.5 Galileo on Mac OS X:
Scala IDE need JDT weaving support, however the JDT weaving seems to be broken by Spring IDE. If Spring IDE is installed and JDT Weaving is enabled, on startup the following problem occurrs:
[org.eclipse.equinox.weaving.hook] info adding AspectJ hooks ...
[org.eclipse.equinox.weaving.aspectj] info Starting AspectJ weaving service ...
21.03.2010 18:29:10 org.aspectj.weaver.tools.Jdk14Trace error
SCHWERWIEGEND: register definition failed
java.lang.ClassCastException: org.aspectj.weaver.loadtime.definition.Definition cannot be cast to org.aspectj.weaver.loadtime.definition.Definition
at ...
This problem seems to be related to the fact that the com.springsource.org.aspectj.weaver_1.6.5.RELEASE.jar plugin brings a second class file for org.aspectj.weaver.loadtime.definition.Definition in addition to the class file already shipped with Eclipse. This probably cause the ClassCastException on startup.
By uninstalling the Spring IDE features, this problem can be resolved and JDT Weaving can be enabled.
At the moment I am working on an OSGi-related project. Therefore I was looking for a logging solution.
If found a blog article which describes a very nice approach for integrating all different kinds of logging (OSGi Logging Service, log4j, jcl, etc.).
In the previous article I described how you can reduce the number of bounce mails generated by your mail server. In this article, I will describe a method to automatically distinguish between legitimate and unsolicited bounce mails, thus reducing the number of bounce mails you receive.
Bounce mails should always be sent to the envelope sender (aka "return path") and the bounce mail itself should use an empty envelope sender (some misconfgured systems however use "postmaster@...").
As the user usually does not see the envelope sender, we can set it to anything different from the true sender's address. In fact we can append some kind of signature to the local part of the address, which proves that the mail has really been sent by our mail server. The spammer does not now this signature (which contains a timestamp), unless he has received a mail from us very recently, and therefore will use the normal address (which might have been collected from some webpage or whois information).
Then our mail server can just reject any bounce mails to an address with an invalid signature, because it is probably not a reaction to a mail from our server. This mechanism is called Bounce Address Tag Validation (BATV) and supported by newer Exim versions.
In the first section of our Exim configuration, we define a macro containing the secret signing key and the list of addresses, we want to enable BATV for:
PRVS_SECRET=VerySecretKeyThatYouHaveToChange
addresslist prvs_senders = an-address@example.com : another-address@example.com
If you want to enable BATV for all addresses, you could skip this step and enable BATV for a list of whole domains. However, usually only a few addresses are affected by spam bounces and enabling BATV only for them reduces the risk of breaking something.
Then you have to extend the acl_smtp_rcpt ACL with the following rules:
deny message = This address does not send an unsigned return path
senders = : postmaster@*
recipients = +prvs_senders
deny message = Invalid return path signature
senders = : postmaster@*
condition = ${prvscheck {$local_part@$domain}{PRVS_SECRET}{1}}
!condition = $prvscheck_result
The first rule will deny bounce mails (actually any mails with an empty envelope sender or postmaster@ in the envelope sender) which are targeted at a BATV enabled address and do not have a signature. The second rule checks any bounce mails that have a signature for the correctness of this signature.
In the next step we have to add a router that rewrites a BATV address to the original address:
prvs_redirect:
driver = redirect
data = ${prvscheck {$local_part@$domain}{PRVS_SECRET}}
This router should be placed following any remote but preceding any local routers.
Finally, we have to add the signing configuration to the SMTP transport:
remote_smtp:
driver = smtp
return_path = ${if match_address{$return_path}{+prvs_senders} \
{${prvs {$return_path}{PRVS_SECRET}}}{$return_path}}
This will cause the sender address to be signed, if it is listed in prvs_senders.
Since enabling this, I receive virtually no spam bounces any longer.
In a perfect world there would be no spam and no bounce mails generated by spam. However, in the real world there is spam and there are badly configured mail servers, which generate bounce mails for invalid addresses.
Fighting spam is a complex task and there are many different approaches but no real solution. However, there are two simple and effective measures to fight bounce mails generated by spam:
First, you should take care to adjust your Exim configuration in order to minimize the number of bounce mails generated by your mail server. This will not reduce the amount of unsolicited bounce mails you receive, but it will dramatically reduce the number of bounce mails your mail server sends to the poor guy, whose address is abused by spammers for the "return path" (aka "envelope sender").
The solution is very simple: Just tell your mailserver to verify a recipient's address before accepting mail for it. The mailserver usually does this for local mail addresses, but you have to tell the server to do it for remote addresses, too. You do this by adding the line
require verify = recipient/callout=10s,defer_ok
to your Exim configuration. This way the mailserver will connect to the destination mail server and check the address before accepting a mail. If the destination mail server does not accept the local part, Exim will decide that the address is invalid and reject the mail without generating a bounce mail. If the destination mail server is down, Exim will still accept any local part. In fact this will cause bounce mails for invalid addresses again, however this will happen much more infrequently, because the destination mail server is up most of the time.
Now this article is longer than I expected, so I will describe the second measure (which will dramatically reduce the number of bounce mails you receive) in a second article.
During the last days I had some fun with a Xen DomU: The virtual machine had a very bad network performance, although the well-known ethtool line was present in /etc/network/interfaces like in any other DomU on the same host. But only this single DomU had these problems.
After some time, I found out that although the line was present, tx checksumming was not disabled in this DomU. Then I found some other strange problems, when I tried to restart networking. I had to perform a lot of debugging of the startup process before I discovered that udev tried to bring up the interface, failed and then left the interface configuration in an inconsistent state. This problem only occurred in this DomU as in contrast to the other DomUs /usr has its own partition in this DomU, which is not yet mouted when udev tries to startup the interface.
Now I have found a workaround (documented in my wiki) and filed a bug in Launchpad, so that ethtool is moved from /usr/sbin to /sbin.
I just found a very interesting video demonstrating the Mylyn feature for the Eclipse platform.
Even if you are not a Eclipse user, the video is still interesting, as it shows why the human brain conflicts with the way most traditional user interfaces are build and how a task-focused user interface can support our brain and boost our productivity.