[typhoe]

I think you miss a few things :-)
First of all, be sure to use the vaapi enabled mplayer. If you executed sudo make install, you can normally use simply mplayer. But if you are in the build directory, be sure to use ./mplayer.

Also it is not enough to use -vo vaapi. You should use it like this (use the -va switch as well)

Code:

./mplayer -vo vaapi -va vaapi moviename.mkv

I am really curious the results you have with mplayer. On my machine the files which are stuttering in xbmc, play well in mplayer.

Well, I used mplayer directly because I previously remove the ubuntu packaged mplayer (sudo apt-get remove mplayer) and installed the compiled mplayer using checkinstall (sudo checkinstall -D make install).
So trying to prefixe mplayer to launch the binary from the compilation directory does the same.
I tried the -va option too, no more results...

Code:

typhoe@ns10:~/gpu_accel/mplayer/mplayer-vaapi-20100713/mplayer-vaapi$ ./mplayer -vo vaapi -va vaapi -ao pulse /media/USB_HD_1To/sample_720p.mkv
MPlayer SVN-r31722-4.4.5 (C) 2000-2010 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing /media/USB_HD_1To/sample_720p.mkv.
libavformat file format detected.
[matroska @ 0x2056570] Estimating duration from bitrate, this may be inaccurate
[lavf] stream 0: audio (ac3), -aid 0
[lavf] stream 1: video (h264), -vid 0
VIDEO:  [H264]  1280x720  0bpp  24.000 fps    0.0 kbps ( 0.0 kbyte/s)
Error opening/initializing the selected video_out (-vo) device.
==========================================================================
Opening audio decoder: [ffmpeg] FFmpeg/libavcodec audio decoders
AUDIO: 48000 Hz, 2 ch, s16le, 384.0 kbit/25.00% (ratio: 48000->192000)
Selected audio codec: [ffac3] afm: ffmpeg (FFmpeg AC-3)
==========================================================================
AO: [pulse] 48000Hz 2ch s16le (2 bytes per sample)
Video: no video
Starting playback...
A:  20.2 (20.2) of 2464.2 (41:04.2) 27.0%


MPlayer interrupted by signal 2 in module: play_audio
A:  20.3 (20.2) of 2464.2 (41:04.2) 26.9%

Exiting... (Quit)
typhoe@ns10:~/gpu_accel/mplayer/mplayer-vaapi-20100713/mplayer-vaapi$

So... I really don't know...