Today I had a very strange problem: While Totem (with gstreamer backend) was playing a VOB test file with correct 5.1 surround sound output, Kaffeine (with Xine backend) mixed up the channels incorrectly.
After looking some time around and trying different hints found on the web, I finally got the right hint in a bugreport: Some magic settings in the ~/.asoundrc or /etc/asound.conf solve the problem:
Of course you have to make sure this settings do not collide with other settings (which might have been created automatically by some tool).
Forget what I have written above: This solution works for Kaffeine, however it creates other problems. The right solution is to set the setting "audio.device.alsa_default_device" to "plug:surround51:0". The reason is that Xine does all checks (like how many channels are supported) on this device, even if configured for 5.1 surround mode. However I had to read the corresponding Xine source code to find out. This behaviour seems to be documented nowhere and one would usually expect Xine to use the device defined in "audio.device.alsa_surround51_device" for 5.1 sound output.