I've tried to set this up, and have learned a few things, but it still doesn't work.
On Windows 64 bit, I was running as a user account, but installed PS3MS (UPnP server) as an admin. I couldn't get the UPnP server to appear on the network at all, even if, from the user account, it was started as admin. Everything looked fine in the log that it should be working (whether as admin or user). Anyhow, I decided to directly login as an administrator, and sure enough, when the server was started, it appeared on the network and the xbox could see it. Long story short, if your UPnP server doesn't appear on the network and not visible by your xbox, login as an admin and run from there.
Now that I can see the server and the shares and files fine, when I attempt to play the MKVs (720p and they play fine on the PC directly), the xbox crashes and I have to restart. Sometimes I see the "Playing" dialog, but the progress bar is empty. If I attempt to play a non-HD xvid, it works fine, but when I look at the PS3 server, there is no buffering -- I guess because there is no transcoding at work. Just to ensure it wasn't just streaming the file directly, I stopped the server and the playback stopped too.
I've read that you have to use DVDPlayer (Play Using...) for this to work? If this is true, anyone know why? By the way, when I attempted to play the non-HD file using DVDPlayer, xbox crashed -- I had to use MPlayer to get it playing. That said, DVDPlayer works fine when just streaming files directly (without the PS3MediaServer).
I have also tried various transcoding settings, as well as other settings recommended by others -- still no go.
I'm running an xbox outputting as 1080i. The XBMC build is November 6th, 2009. Using PM3HD skin for now. Using build 366 of PS3MediaServer.
Any other ideas to try?
Thanks!