Interesting, I'm not able to get Vista to have multichannel audio through HDMI at all.
And ATI/nVidia supply their own drivers for HDMI sound.
If you have a card that has HDMI chances are good you have 2 sound devices, one on the motherboard (if it has onboard sound), and one on the gfx card.
Installing drivers for the onboard sound will not also work for the GFX card sound.
I read a bit on AMD support pages and found that XP does support HDMI audio.
If you haven't done this I would try the following:
1. Uninstall all sound-related drivers and graphics drivers.
2. Install the latest Catalyst driver suite from ATI. At this point you should have 2 audio devices: a Microsoft HD sound device (using default drivers from XP)--this is your motherboard device and an ATI HDMI sound device.
3. Don't bother installing drivers for the onboard device, you aren't gonna be using it.
4. This is where I might be wrong since I don't run vista. Make sure Windows is using the ATI device as the primary sound device. Go to the speaker settings and toy with those. Verify that windows can output to a 5.1 setup. Check for any kind of options that tell Windows to "pass through" DTS/DD5.1 audio (there may not be any options). There may even be some kind of setting in the Catalyst suite.
5. In XBMC verify that its using either the primary sound device (what windows is using) or the ATI HDMI audio device specifically.
6. Enable pass through of DTS and DD5.1 in XBMC.
If none of this works. The final step is to google for UAA HDMI driver for Microsoft. In a support page on AMD for Catalyst 9.1 this was needed to get sound over HDMI.
EDIT:
I don't think AC3Filter is gonna help in this case. Reason is that XBMC is not using direct show filters. AC3Filter uses direct show to encode on-the-fly AC3 streams if I remember correctly.
EDIT 2:
Googling about HDMI audio for ATI. The device as shown under "Audio and Speaker Settings" in the control panel may be called something like "ATI HD Rear Output". This is the one you will want to select for windows.
EDIT 3:
See this
link about what your ATI card supports. Also pay attention to the notes at the bottom, particularly #2 as I think your Yamaha receiver might be affected by this.