Okay, I tried to have a good go at this today...
-------------------------
Rebuild current XBMC to check that some unintended magic hasn't happened (f8e99f8654) ...
fail
-------------------------
Build current FernetMenta build to check that his VDPAU routines don't solve the issue (89c1de156d)...
fail. (Ah, how far I've come... when that was originally suggested, I didn't know how to compile XBMC, no I just clone the git code and get on with it :-) ).
-------------------------
So, nothing new there. Let's try mplayer as the test environment (it's faster to compile if nothing else - and will still help isolate the fault).
Install mplayer (SVN-r1.0~rc3+svn20090426-4.4.3) binary from repository (not sure which one!) ... play with -vo and -vc options: mplayer -vo vdpau -vc ffh264vdpau)
Code:
xbmc@XBMCLive:~/setup/mplayer.old$ which mplayer
/usr/bin/mplayer
xbmc@XBMCLive:~/setup/mplayer.old$ ls -la /usr/bin/mplayer
-rwxr-xr-x 1 root root 4193524 2010-09-27 11:52 /usr/bin/mplayer
xbmc@XBMCLive:~/setup/mplayer.old$
xbmc@XBMCLive:~/setup/ffmpeg-dev$ mplayer -vo vdpau -vc ffh264vdpau ~/Sample.flv
MPlayer SVN-r1.0~rc3+svn20090426-4.4.3 (C) 2000-2009 MPlayer Team
mplayer: could not open config files /home/xbmc/.lircrc and /etc/lirc//lirc/lircrc
mplayer: No such file or directory
Failed to read LIRC config file ~/.lircrc.
Playing /home/xbmc/Sample.flv.
libavformat file format detected.
[h264 @ 0xa092ef0]brainfart cropping not supported, this could look slightly wrong ...
[lavf] Video stream found, -vid 0
[lavf] Audio stream found, -aid 1
VIDEO: [H264] 1280x724 0bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
==========================================================================
Forced video codec: ffh264vdpau
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
[VD_FFMPEG] XVMC-accelerated MPEG-2.
Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU))
==========================================================================
==========================================================================
Opening audio decoder: [faad] AAC (MPEG2/4 Advanced Audio Coding)
FAAD: compressed input bitrate missing, assuming 128kbit/s!
AUDIO: 48000 Hz, 2 ch, s16le, 128.0 kbit/8.33% (ratio: 16000->192000)
Selected audio codec: [faad] afm: faad (FAAD AAC (MPEG-2/MPEG-4 Audio))
==========================================================================
waitpid(): No child processes
AO: [pulse] Init failed: Internal error
Failed to initialize audio driver 'pulse'
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Starting playback...
[h264_vdpau @ 0x18d3e00]brainfart cropping not supported, this could look slightly wrong ...
[VD_FFMPEG] XVMC-accelerated MPEG-2.
VDec: vo config request - 1280 x 724 (preferred colorspace: H.264 VDPAU acceleration)
VDec: using H.264 VDPAU acceleration as output csp (no 0)
Movie-Aspect is 1.77:1 - prescaling to correct movie aspect.
VO: [vdpau] 1280x724 => 1280x724 H.264 VDPAU acceleration
MPlayer interrupted by signal 2 in module: sleep_timer
Exiting... (Quit)
=====>
Works - mplayer renders the file perfectly. I'm not interested in the LIRC and audio errors. Note that it detects the file size correctly (1280x724).
-------------------------
Okay, so let's see if we can get anywhere with the ffmpeg version that's internal to XBMC 10.1 onwards. This is git c3beafa0f1.
This version of ffmpeg compiles okay
Code:
FFmpeg version git-c3beafa, Copyright (c) 2000-2011 the FFmpeg developers
built on Sep 18 2011 15:00:32 with gcc 4.4.3
-------------------------
Compile mplayer on top... errors, wants to download its own ffmpeg so that it's in a subdirectory. Stopped, symlink back to c3beafa. Gets through this but then errors in libdvdread. Installed libdvdread-dev to correct this.
Still won't compile, picking up the wrong include files (samplefmt.h).
-------------------------
So, let it download the current ffmpeg code, let's see if that works.
Nope, lacks yet more dependencies - more missing code.
Code:
svn checkout svn://svn.mplayerhq.hu/dvdnav/trunk/libdvdread/src libdvdread4
Code:
svn checkout svn://svn.mplayerhq.hu/dvdnav/trunk/libdvdnav/src libdvdnav
Off we go again...
Compiles correctly. Time to test:
Code:
xbmc@XBMCLive:~/setup/mplayer$ ./mplayer -vo vdpau -vc ffh264vdpau ~/Sample.flv
MPlayer UNKNOWN-4.4.3 (C) 2000-2011 MPlayer Team
Playing /home/xbmc/Sample.flv.
libavformat file format detected.
[lavf] stream 0: video (h264), -vid 0
[lavf] stream 1: audio (aac), -aid 0
VIDEO: [H264] 1280x720 0bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s)
Clip info:
aacaot: 2
audiochannels: 2
audiocodecid: mp4a
audiosamplerate: 24000
avclevel: 41
avcprofile: 100
duration: 1735
height: 720
moovPosition: 32
videocodecid: avc1
videoframerate: 25
width: 1280
Load subtitles in /home/xbmc/
==========================================================================
Forced video codec: ffh264vdpau
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffh264vdpau] vfm: ffmpeg (FFmpeg H.264 (VDPAU))
==========================================================================
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, s16le, 0.0 kbit/0.00% (ratio: 0->192000)
Selected audio codec: [ffaac] afm: ffmpeg (FFmpeg AAC (MPEG-2/MPEG-4 Audio))
==========================================================================
[AO OSS] audio_setup: Can't open audio device /dev/dsp: Device or resource busy
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Starting playback...
[VD_FFMPEG] Trying pixfmt=0.
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vdpau] 1280x720 => 1280x720 H.264 VDPAU acceleration
[VD_FFMPEG] XVMC-accelerated MPEG-2.
A: 1.4 V: 1.4 A-V: -0.001 ct: -0.039 0/ 0 51% 12% 3.1% 8 0
MPlayer interrupted by signal 2 in module: sleep_timer
Exiting... (Quit)
=====>
Fail - mplayer renders the file exactly as XBMC does, with corruption at the bottom of the display. Note that it detects the file size
incorrectly (1280x720).
-------------------------
Soooo.... that suggests to me that there's clearly a problem in the underlying ffmpeg code, introduced since the version included in XBMC prior to 10.1 (r24229). I haven't checked the dates, but I'd put money on c3beafa post-dating the binary build of mplayer (dated 20090426 in the binary, and timestamped 2010-09-27 on the filesystem).
My conclusion: as of the version mplayer downloaded for itself - ba5ff1b31e, dated Sun Sep 18 19:02:37 2011 +0200 - ffmpeg seems to have problems.
I did drop a note to the ffmpeg-users mailing list to see if anyone understood the problem... while I was underwhelmed with the response, I did get something from one of the ffmpeg devs, so I'll maybe follow up with him.
-------------------------
Next steps/other thoughts:
1. Throw it to the ffmpeg devs to see if they have any ideas, as above. Maybe try a different mailing list.
2. Try to fix the compilation errors to see if I can build the current mplayer code with ancient ffmpeg - both r24229 [worked in XBMC] and the later c3beafa [didn't work]. If we could get this working, we could work towards finding the commit that broke everything. The include file mismatch (samplefmt.h) may just be an artefact of my system.
3. Try to compile a version of mplayer close to the c3beafa ffmpeg commit to see if that works or not. Closest seem to be 653eeee869/r32841 Wed Feb 2 18:30:20 2011 +0000 and cb1ff47ea2/r32840 Tue Feb 1 20:45:41 2011 +0000). I'd only try this because I can't get the later mplayer to compile with earlier ffmpeg.
4. Try to compile current xbmc with --enable-external-ffmpeg on the old (r24229) ffmpeg... perhaps the old APIs would slide through the compilation process more easily, since I failed to get it to work on the current ffmpeg release.