I am also having issues with 5.1. I have been using an XBOX with XBMC for years and decided to try it on an Apple TV. I upgraded my ATV to 3.0.1 and installed the latest version of XBMC.
I changed to audio to digital and turned off downmix multichannel audio to stereo. I rebooted the ATV and ran XBMC.
My receiver does not show 5.1 coming in and the sound is not correct. I lose the center channel, all I here is the surround audio coming from the front speakers.
When I reboot the ATV and the start-up animation plays, my receiver shows it is receiving 5.1 and you can definitely tell its Dolby Digital so the receiver is setup correctly.
The movies I am trying to play are direct rips from my own discs and only have 5.1 audio in them. They play perfect on the XBOX in 5.1 but not on the ATV.
I have posted a log file here
http://www.lammers.ca/downloads/xbmc.log
here are some details. The stream is found to be 5.1
20:14:44 T:43323904 M: 26165248 DEBUG: Open - av_find_stream_info finished
20:14:44 T:43323904 M: 26181632 INFO: ffmpeg[2951200]: Input #0, mpeg, from 'smb://SYNERGY/Videos/Movies/Animation/UP/VTS_01_2.VOB':
20:14:44 T:43323904 M: 26181632 INFO: ffmpeg[2951200]: Duration: 00:21:32.50, start: 77.984811, bitrate: 6645 kb/s
20:14:44 T:43323904 M: 26181632 INFO: ffmpeg[2951200]: Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x480 [PAR 32:27 DAR 16:9], 9800 kb/s, 59.94 tbr, 90k tbn, 59.94 tbc
20:14:44 T:43323904 M: 26181632 INFO: ffmpeg[2951200]: Stream #0.1[0x80]: Audio: ac3, 48000 Hz, 5.1, s16, 384 kb/s
Then there are some errors here
20:14:54 T:44225024 M: 18866176 NOTICE: Opening audio stream: 1 source: 256
20:14:54 T:44225024 M: 18866176 NOTICE: Finding audio codec for: 86020
20:14:54 T:44225024 M: 18853888 DEBUG: FactoryCodec - Audio: passthrough - Opening
20:14:54 T:44225024 M: 18853888 DEBUG: Loading: /mnt/Scratch/Users/frontrow/Applications/XBMC.app/Contents/Resources/XBMC/system/players/dvdplayer/liba52-x86-osx.so
20:14:54 T:44225024 M: 18841600 DEBUG: FactoryCodec - Audio: passthrough - Opened
20:14:54 T:44225024 M: 18841600 NOTICE: Creating audio thread
20:14:54 T:43586048 M: 18837504 DEBUG: thread start, auto delete: 0
20:14:54 T:43586048 M: 18837504 NOTICE: running thread: CDVDPlayerAudio:
rocess()
20:14:54 T:43586048 M: 18837504 DEBUG: CDVDPlayerAudio - CDVDMsg::GENERAL_RESYNC(0.000000, 0)
20:14:54 T:43586048 M: 18837504 DEBUG: ParseFrame - source flags changed flags:17 sr:48000 br:384000
20:14:54 T:43586048 M: 18833408 NOTICE: Creating audio device with codec id: 86020, channels: 2, sample rate: 48000, pass-through
20:14:54 T:43586048 M: 18833408 DEBUG: Error: Requested setting (audiooutput.passthroughdevice) was not found. It must be case-sensitive
20:14:54 T:43586048 M: 18833408 INFO: AudioRendererFactory: not a explicit device, trying to autodetect.
20:14:54 T:43586048 M: 18833408 DEBUG: SetActiveDevice - SetActiveDevice from 1 to 4
20:14:54 T:43586048 M: 18833408 DEBUG: RemoveActiveDevice - Removing device 1
20:14:54 T:43586048 M: 18833408 DEBUG: CCoreAudioHardware::FindAudioDevice: Searching for device - Default.
20:14:54 T:43586048 M: 18399232 DEBUG: CCoreAudioDevice::GetTotalOutputChannels: Found 4 channels in 2 buffers
20:14:54 T:43586048 M: 18399232 DEBUG: CCoreAudioHardware::FindAudioDevice: Device[0x0102] - Name: 'Built-in Line Output', Total Ouput Channels: 4.
20:14:54 T:43586048 M: 18395136 DEBUG: CCoreAudioDevice::Close: Closed device 0x0102
20:14:54 T:43586048 M: 18395136 WARNING: CoreAudioRenderer::Initialize: Unable to locate configured device, falling-back to the system default.
20:14:54 T:43586048 M: 18395136 DEBUG: CCoreAudioStream::Open: Opened stream 0x0103.
20:14:54 T:43586048 M: 18366464 DEBUG: CoreAudioRenderer::InitializeEncoded: Found Output stream - id: 0x0103, Terminal Type: 0x0000
20:14:54 T:43586048 M: 18366464 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 16-bit Signed Integer (48000Hz)
20:14:54 T:43586048 M: 18358272 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 16-bit Signed Integer (44100Hz)
20:14:54 T:43586048 M: 18358272 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 20-bit Signed Integer (48000Hz)
20:14:54 T:43586048 M: 18358272 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 20-bit Signed Integer (44100Hz)
20:14:54 T:43586048 M: 18358272 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 24-bit Signed Integer (48000Hz)
20:14:54 T:43586048 M: 18358272 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 24-bit Signed Integer (44100Hz)
20:14:54 T:43586048 M: 18341888 DEBUG: CCoreAudioStream::Close: Closed stream 0x0103.
20:14:54 T:43586048 M: 18341888 DEBUG: CCoreAudioStream::Open: Opened stream 0x0104.
20:14:54 T:43586048 M: 18341888 DEBUG: CoreAudioRenderer::InitializeEncoded: Found Output stream - id: 0x0104, Terminal Type: 0x0000
20:14:54 T:43586048 M: 18333696 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 16-bit Signed Integer (48000Hz)
20:14:54 T:43586048 M: 18333696 DEBUG: CoreAudioRenderer::InitializeEncoded: Considering Physical Format: [lpcm] Mixable 2 Channel 16-bit Signed Integer (44100Hz)
20:14:54 T:43586048 M: 18333696 DEBUG: CCoreAudioStream::Close: Closed stream 0x0104.
20:14:54 T:43586048 M: 18333696 ERROR: CoreAudioRenderer::InitializeEncoded: Unable to identify suitable output format.