Vsync is to prevent tearing:
http://en.wikipedia.org/wiki/Screen_tearing
Nobody likes tearing, so turn on vsync.
Render method you can leave on auto.
"Adjust display refresh rate to match video" automatically adjusts the refreshrate to the video fps to prevent telecine judder:
http://en.wikipedia.org/wiki/Telecine#Telecine_judder
It currently only works on windows and linux.
"Sync playback to display" is to solve small timing differences between the monitor and playback fps, which can cause periodic video jitter, you can read about it more here:
http://wiki.xbmc.org/index.php?title=Videos_Settings
Depending on which version of xbmc you're using and which operating system you have, there are also options to turn on hardware decoding using VDPAU, DXVA, VDA or Crystal HD, preferably you want that on.
When changing these settings, don't change them all at once, change one at a time and check if video playback still works by playing some different videos (possibly an xvid, a dvd and 1080p h264), that way if something doesn't work, you know which setting caused it.