Skip to content

Trouble with globally installed Firefox extensions after software upgrade in Ubuntu

Some time ago Ubuntu 10.04 LTS received a Firefox update from the 3.6 branch to the more recent versions (10.0+).

After that upgrade, Firefox on one of my Ubuntu systems suddenly appeared in English instead of the correct locale (German in my case). I first thought that this might be a problem with some localization packages not being installed correctly. However, the problem persisted after upgrading the system to Ubuntu 12.04 LTS.

Some globally extensions (in particular the language pack) showed up in an old version in the add-ons list, although the newest version was installed. Finally I found out, that Firefox looks for the extensions in /usr/lib/firefox/extensions. The new language packs however had been installed in /usr/lib/firefox-addons/extensions. On other systems, /usr/lib/firefox/extensions is a symbol link to /usr/lib/firefox-addons/extensions. In my case however the directory existed and contained the files from the old versions of the language packs.

For some reasons, the old language packs (which had different package names) had not been removed and thus the Firefox upgrade did not place the symbol link (because the directory was not empty). After manually removing the old versions of the language packs, deleting the directory and reinstalling Firefox, the symbol link was created automatically and suddenly the globally installed Firefox add-ons worked again.

KVM and Graceful Shutdown on Ubuntu

For quite some time, I have been trying to figure out, how to gracefully shutdown the KVM-based virtual machines running on a Ubuntu 10.04 LTS (Lucid Lynx) host system. This problem consists of two parts: First you have to make the virtual machines support the shutdown event from libvirt and second you have to call the shutdown action for each virtual machine on system shutdown.

The first part is very easy for Linux VMs and also not too hard for Windows VMs. I described the necessary steps in my wiki

The seconds part is harder to accomplish: On Ubuntu 8.04 LTS (Hardy Heron) I just modified the /etc/init.d/libvirt-bin script to call a Python script in the stop action. This solution was not perfect, as it meant that the virtual machines were also shutdown, when libvirtd was just restarted, however it was a quick and easy solution.

For Ubuntu 10.04, the init script has been converted to an Upstart job. So the easiest way was to create a upstart job that is starting on the stopping libvirt-bin event. However, this did not solve the problem, because the system powered off or rebooted before the shutdown of the virtual machines was finished. As it turns out, Ubuntu 10.04 uses an odd combination of Upstart jobs and traditional init scripts. This leads to a situation, where /etc/init.d/halt or /etc/init.d/reboot are called, before all upstart jobs have stopped, when one of the upstart jobs needs a significant amount of time to stop. This can be solved by adding an init script, than runs before the halt or reboot scripts and waits for the respective Upstart job to finish. In fact, it is best to run this script before the sendsigs script to avoid processes started by one of the upstart jobs to receive a SIGKILL.

I added the complete scripts and configuration files needed for this feature to my wiki. In fact, this solution also ensures, that the virtual machines are only shutdown if libvirt is stopped because of a runlevel change. Thus, the libvirt-bin package can now be upgraded without resulting in a restart of the VMs.

For me, automatically shutting down the virtual machines is very important. The KVM hosts I manage are connected to an uninterruptible power supply with limited battery time. Although in the past years I remember only a single time, the host systems were shutdown because the battery was nearly empty (most power interrupts are very short), I want to make sure that all virtual machines are in a safe, consistent state, when the power finally goes off. So I hope that the scripts in the wiki are also helpful to other people, having the same problem.

Problems when Upgrading to GRUB 2

Yesterday, I upgraded the last systems, which where still running on GRUB Legacy (GRUB 1) to GRUB 2 (the grub-pc package on Ubuntu).

On one of the systems, I experienced the problem, that upgrade-from-grub-legacy could not install GRUB. When I tried to install GRUB manully using grub-install, I got a strange error message:

/usr/sbin/grub-setup: warn: Your embedding area is unusually small.  core.img won't fit in it..
/usr/sbin/grub-setup: warn: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and its use is discouraged..
/usr/sbin/grub-setup: error: if you really want blocklists, use --force.

It took me some time to figure out, that this system had an unusual disk geometry, resulting in not enough space being available before the start of the first partition. The output of fdisk -lu /dev/sda looked like this:

Disk /dev/sda: 320.1 GB, 320071884800 bytes
64 heads, 32 sectors/track, 305244 cylinders, total 625140400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009153c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              32      585727      292848   83  Linux
/dev/sda2          585728   625139711   312276992    5  Extended
/dev/sda5          585760   215429119   107421680   8e  Linux LVM
/dev/sda6       215429152   430272511   107421680   8e  Linux LVM
/dev/sda7       430272544   625139711    97433584   8e  Linux LVM

Usually, the first partition starts at sector 63 (for systems installed more recently, 2048 is also common), but for this system it started at sector 32. I guess that this was cause by the strange geometry, which did not specify the usual 255 heads and 63 sectors/track. I am not sure, whether the disks (two Western Digital 320 GB drives) really have this strange geometry, or the RAID-Controller, a 3ware Escalade 8006-2LP, reports a different geometry. However, for sure the first partition being to close to the start of the disk was the cause of the problem with GRUB.

Unfortunately, I could not move the partition using parted and gparted does not allow to specify the start and end of a partition in sectors, but only in MiB (and only supports aligning to MiB or cylinders, which did not help considering the odd geometry). Therefore I decided to delete the partition (after all it was only the boot partition and its content could easily be stored on the root partition) and create a new partition using fdisk.

In fact, when (re-)creating the partition, fdisk suggested to start the partition at sector 2048, which I accepted. No surprise, after that change GRUB could be installed to the MBR without any problems.

The good thing about this problem was, that the time I invested to get a high-availability setup running for most important services, finally paid of. I for sure would have been less relaxed, while figuring out and fixing the GRUB problem, when the services usually running on this machine would have been offline.

Squid 3.1 on Ubuntu 8.04 Hardy Heron

Squid 3.1 introduced support for IPv6, so there is a very good reason for an upgrade from Squid 3.0.

Unfortunately, there is no Ubuntu package for this version, not even in the upcoming Ubuntu release 10.04 Lucid Lynx.

For a while, I have been using the Debian experimental packages of Squid 3.1.0, compiled from source on Ubuntu 8.04. However, now a stable version of Squid 3.1 has been released, but there is no Debian package for Squid 3.1.1.

Therefore, I used the Debian package for Squid 3.1.0.14 as the base for building a package for Squid 3.1.1. I have published this package in source as well as built for Ubuntu 8.04 on amd64. There you can also find a package for libcap2, which is required by Squid 3.1 but not included in Ubuntu 8.04.

Of course, there is no kind of warranty for this packages, so use them at your own risk.

Spring IDE for Eclipse breaks JDT weaving

I experiences a problem, when trying to use Scala IDE for Eclipse and Spring IDE at the same time in Eclipse 3.5 Galileo on Mac OS X:

Scala IDE need JDT weaving support, however the JDT weaving seems to be broken by Spring IDE. If Spring IDE is installed and JDT Weaving is enabled, on startup the following problem occurrs:

[org.eclipse.equinox.weaving.hook] info adding AspectJ hooks ...
[org.eclipse.equinox.weaving.aspectj] info Starting AspectJ weaving service ...
21.03.2010 18:29:10 org.aspectj.weaver.tools.Jdk14Trace error
SCHWERWIEGEND: register definition failed
java.lang.ClassCastException: org.aspectj.weaver.loadtime.definition.Definition cannot be cast to org.aspectj.weaver.loadtime.definition.Definition
    at ...

This problem seems to be related to the fact that the com.springsource.org.aspectj.weaver_1.6.5.RELEASE.jar plugin brings a second class file for org.aspectj.weaver.loadtime.definition.Definition in addition to the class file already shipped with Eclipse. This probably cause the ClassCastException on startup.

By uninstalling the Spring IDE features, this problem can be resolved and JDT Weaving can be enabled.

Thunderbird with Exchange Server 2007

The university I am studying at recently migrated the mail system from Cyrus IMAP to Exchange Server 2007.

Unfortunately Mozilla Thunderbird does not work well with Exchange's IMAP server, unless you do some manual configuration:

First in order to make Thunderbird delete e-mails on the IMAP server correctly, you have to set the mail.imap.expunge_after_delete option to true (see KB article).

Besides, you want to set the mail.server.server#.trash_folder_name property to the name of the "deleted items" folder on the server (see KB article).

With this options changed, Thunderbird will interact with Exchange's IMAP server quite well.

Fighting SPAM Bounces with Exim (Part II)

In the previous article I described how you can reduce the number of bounce mails generated by your mail server. In this article, I will describe a method to automatically distinguish between legitimate and unsolicited bounce mails, thus reducing the number of bounce mails you receive.

Bounce mails should always be sent to the envelope sender (aka "return path") and the bounce mail itself should use an empty envelope sender (some misconfgured systems however use "postmaster@...").

As the user usually does not see the envelope sender, we can set it to anything different from the true sender's address. In fact we can append some kind of signature to the local part of the address, which proves that the mail has really been sent by our mail server. The spammer does not now this signature (which contains a timestamp), unless he has received a mail from us very recently, and therefore will use the normal address (which might have been collected from some webpage or whois information).

Then our mail server can just reject any bounce mails to an address with an invalid signature, because it is probably not a reaction to a mail from our server. This mechanism is called Bounce Address Tag Validation (BATV) and supported by newer Exim versions.

In the first section of our Exim configuration, we define a macro containing the secret signing key and the list of addresses, we want to enable BATV for:

PRVS_SECRET=VerySecretKeyThatYouHaveToChange
addresslist prvs_senders = an-address@example.com : another-address@example.com

If you want to enable BATV for all addresses, you could skip this step and enable BATV for a list of whole domains. However, usually only a few addresses are affected by spam bounces and enabling BATV only for them reduces the risk of breaking something.

Then you have to extend the acl_smtp_rcpt ACL with the following rules:

  deny    message     = This address does not send an unsigned return path
          senders     = : postmaster@*
          recipients  = +prvs_senders

  deny    message     = Invalid return path signature
          senders     = : postmaster@*
          condition   = ${prvscheck {$local_part@$domain}{PRVS_SECRET}{1}}
         !condition   = $prvscheck_result

The first rule will deny bounce mails (actually any mails with an empty envelope sender or postmaster@ in the envelope sender) which are targeted at a BATV enabled address and do not have a signature. The second rule checks any bounce mails that have a signature for the correctness of this signature.

In the next step we have to add a router that rewrites a BATV address to the original address:

prvs_redirect:
  driver = redirect
  data = ${prvscheck {$local_part@$domain}{PRVS_SECRET}}

This router should be placed following any remote but preceding any local routers.

Finally, we have to add the signing configuration to the SMTP transport:

remote_smtp:
  driver = smtp
  return_path = ${if match_address{$return_path}{+prvs_senders} \
{${prvs {$return_path}{PRVS_SECRET}}}{$return_path}}

This will cause the sender address to be signed, if it is listed in prvs_senders.

Since enabling this, I receive virtually no spam bounces any longer.

Fighting SPAM Bounces with Exim (Part I)

In a perfect world there would be no spam and no bounce mails generated by spam. However, in the real world there is spam and there are badly configured mail servers, which generate bounce mails for invalid addresses.

Fighting spam is a complex task and there are many different approaches but no real solution. However, there are two simple and effective measures to fight bounce mails generated by spam:

First, you should take care to adjust your Exim configuration in order to minimize the number of bounce mails generated by your mail server. This will not reduce the amount of unsolicited bounce mails you receive, but it will dramatically reduce the number of bounce mails your mail server sends to the poor guy, whose address is abused by spammers for the "return path" (aka "envelope sender").

The solution is very simple: Just tell your mailserver to verify a recipient's address before accepting mail for it. The mailserver usually does this for local mail addresses, but you have to tell the server to do it for remote addresses, too. You do this by adding the line

require verify = recipient/callout=10s,defer_ok

to your Exim configuration. This way the mailserver will connect to the destination mail server and check the address before accepting a mail. If the destination mail server does not accept the local part, Exim will decide that the address is invalid and reject the mail without generating a bounce mail. If the destination mail server is down, Exim will still accept any local part. In fact this will cause bounce mails for invalid addresses again, however this will happen much more infrequently, because the destination mail server is up most of the time.

Now this article is longer than I expected, so I will describe the second measure (which will dramatically reduce the number of bounce mails you receive) in a second article.

Fun with Xen, Ubuntu and ethtool

During the last days I had some fun with a Xen DomU: The virtual machine had a very bad network performance, although the well-known ethtool line was present in /etc/network/interfaces like in any other DomU on the same host. But only this single DomU had these problems.

After some time, I found out that although the line was present, tx checksumming was not disabled in this DomU. Then I found some other strange problems, when I tried to restart networking. I had to perform a lot of debugging of the startup process before I discovered that udev tried to bring up the interface, failed and then left the interface configuration in an inconsistent state. This problem only occurred in this DomU as in contrast to the other DomUs /usr has its own partition in this DomU, which is not yet mouted when udev tries to startup the interface.

Now I have found a workaround (documented in my wiki) and filed a bug in Launchpad, so that ethtool is moved from /usr/sbin to /sbin.

Task-focused UI with Mylyn

I just found a very interesting video demonstrating the Mylyn feature for the Eclipse platform.

Even if you are not a Eclipse user, the video is still interesting, as it shows why the human brain conflicts with the way most traditional user interfaces are build and how a task-focused user interface can support our brain and boost our productivity.

Ubuntu 8.04 Hardy Heron breaks Xen

Yesterday I upgraded the Xen Dom0 hosting various DomUs (for example my Jabber server) from Ubuntu 7.10 to 8.04.  However, after upgrading Xen stopped working.

I tried to use the old 2.6.22 kernel instead of the new 2.6.24 one, but no chance. Xen did not show any error messages and started the DomUs, however neither networking nor "xm console" were working. In fact "xm console" did not show anything and did not react on keyboard input and even a SIGTERM caused no effect. I had to kill the instance with a SIGKILL or SIGHUP.

Now, after I have restored a backup of the old system, everything is working fine again. Therefore I advise everyone using Xen on Ubuntu not to upgrade to 8.04. Upgrading the DomUs however did not cause any problems, Ubuntu Hardy Heron seems to be broken for Dom0 only.

FreePBX: Adding Extensions does not work

After installing trixbox and updating the system I could not add new extensions using FreePBX.

Although the solution is quite easy, it took some time to find out. So here it is (I found it in the FreePBX-Forum):

Simply delete all contents in the files /etc/asterisk/vm_general.inc and /etc/asterisk/vm_email.inc. Then perform an amportal restart and everything works fine again.

Installing trixbox 2.4 on an IBM Netfinity 5000

I wanted to give Asterisk a second chance and thus decided to install trixbox 2.4 on a test machine. This machine is an old IBM Netfinity 5000 (dual Pentium II, only 128 MB memory but a nice SCSI RAID). Unfortunately the installation failed because the system got locked, when the driver for the Adaptec SCSI controller (not the RAID controller) was loaded.

Surprisingly, when I used the "linux" boot command instead of "default", the installation went fine. However, trixbox did not work (freePBX was not installed correctly). So it took me some extra time to find out, that using the "linux" boot mode the wrong installation script was used.

So finally I figured out, that actually trixbox can be installed correctly by using the boot command "linux ks=cdrom:/ks.cfg skipddc". Basically this is the same as using "default", but "default" also includes the "acpi=off" option which causes the problems with the SCSI driver mentioned above.

Sometimes small problems like a wrong boot option can literally take hours to be solved...

Now I am looking forward to get a working Asterisk installation, however there are probably more problems to come as I want to use an ISDN card with mISDN. So it looks that I will be busy with Asterisk for some more hours :-)