I saw what looks like the same problem we've been seeing on the Mac Mini. Poor frame rates, in my case the video lagging behind the audio. However see later; it's not *quite* the same...
I had some additional problems I haven't seen with the Mini: For the first time I had to use the video calibration page in XBMC to get all the frame onto the screen. Overscan was enabled in the display manager, but when I turned it off, it picture-framed the desktop so it wouldn't fill the whole screen. (ooh checking something... nope, TV's aspect ratio/zoom settings didn't fix it.)
I also had an intermittent vertical-stretching mode-change happen, but I think that might have been an issue with the connector hanging out of the machine; it stopped when it was properly supported on the desk. Although when I was getting it, it only happened a few seconds into playing a video in xbmc, and only went away when xbmc was quit... oh well. :-)
As before, turning off VDADecoder, so dc:ff-h264 resolves playback speed - it plays correctly except that it's *just* too slow to actually do it reliably.
As per davilla's most recent, on the macbook pro the a/v was within plus-or-minus 0.050 on *software* playback as far as I could tell (the digits change a bit fast). When using VDADecoder it was often there too but sometimes went up as far as 0.2, 0.3.
Throughout the display mode selected was 1080p at 60Hz, which matches the 60Hz claimed by XBMC when viewing on the laptop's own screen.
So it seems there's something about being connected to an actual TV which is hurting VDADecoder? I didn't expect this and I don't have a clue how to explain it, which is why I didn't try it before. After all, the Aspire Revo (also an NVidia 9400M) running XBMC under Ubuntu with VDPAU enabled works just fine with this TV. Better in fact because it can switch refresh rate. :-P Also on the mac mini, playing h.264 movies via quicktime, which presumably is using hardware decoding, is also working fine.
Back to the Mac Mini:
looking again at display preferences there isn't an Overscan checkbox; instead an Underscan slider control going from "Off" to "More". (We love Apple but sometimes...) It's set to Off, which is the default, I haven't touched it.
In playback with VDADecoder on, the differences I saw in comparison to the Macbook Pro connected to the TV (with VDADecoder on):
The video lagging behind the sound was worse. a/v error just kept going up as time went on, still to three decimal places but value quickly went over 1, 2, 3, 4, 5 - up to 12 or so by the time I stopped playback. If those numbers are seconds, it more or less matches my own sense of how far the video is lagging. On the MBP the a/v error rate was worse than with VDADecoder off but more or less stable at below 0.2 for the most part, and the video, while lagging, didn't lag as badly, and presumably just started skipping frames to more or less keep up.
Am now uploading the first 100M of one of my test videos, that's never failed to fail, if you see what I mean. :-) This one's encoded by HandBrake with its High Profile preset, output as MKV, and the first 100M extracted with dd like this:
dd if=The\ Colour\ of\ Magic\ -\ part\ 1\,\ chap1\,\ high\ profile.mkv of=tcom-100m.mkv bs=100M count=1
It can be downloaded here: http://electronpusher.org/~rachel/tcom-100m.mkv. Here's a screenshot of it playing on the Mac Mini with VDADecoder enabled:
For comparison, the same thing being played on the TV by the macbook pro. fps is low, but a/v is currently showing 0.012; and though it does peak up to 0.2 or so it's more or less stable:
Check out the error% on both as well. Again, on the mac mini that number just keeps going up...