[ihaterich]

XBMC will play pretty much anything just fine (.rm, .avi, .mp4, .mkv, .wmv, .mov, etc) As far as compatibility, they all work the same.

MKV with h264 video is popular now because it offers great quality at reasonable sizes.

For audio, it depends I guess. If you have a nice surround system you might wanna use 5.1 Dolby Digital. If not, 2.0 AAC will save some space.

You can rip to MKV with a mac.

I use MakeMKV to decrypt and save the DVD to a mkv file(s) then use Handbrake to re-encode those files to reduce space.

http://www.makemkv.com/download/
http://handbrake.fr/downloads.php

I think Handbrake is capable of decrypting DVDs on its own, but in my experience it takes too long.