I've been also having problems with xbmc crashing to the desktop when trying to play h264 vids with vdpau set as the render method. It happens randomly (sometimes at the 1st try, sometimes at the 3rd try... haven't needed four tries yet to reproduce the crashing to the desktop) and just with vdpau enabled, with software render method enabled i haven't had a single crash, at least for now.
The errors also seem to be "random", at least, i've got 3 different errors when crashing to desktop:
a) xbmc.bin: ../../src/xcb_io.c:445: _XReply: Assertion `!dpy->xcb->reply_data' failed.
b) xbmc.bin: ../../src/xcb_io.c:176: process_responses: Assertion `!(req && current_request && !(((long) (req->sequence) - (long) (current_request)) <= 0))' failed.
c) XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":0.0"
after XYZ requests (ABC known processed) with KY events remaining.
(different XYZ, ABC and KY values)
I'm now using svn -r19930. Jaunty, 2GB RAM, 256MB 8500GT, and a quad core @2.40Ghz.
Yesterday I started digging a little bit more into this issue and found that the culprit, at least of my crashes, seems to be the commit
#19833 ... if I change both lines back to "if(dwMilliseconds > 10)" then i dont get a crash to the desktop (at least in 40 tries that I've done so far)
Hope this info helps,
Let me know if you need me to provide with further info/logs