[lmyllari]

With SW decode, RGB set to "full" and 16-235 set in XBMC I see BTB and WTW in the test pattern, the image looks as close as I've seen to the PS4... but I do have some video overscan issues going on.

This tells you that your TV ignores the range in infoframes and will take limited range signal. This is good for your purposes.

With HW decode I am having a hard time getting it to look the same;

Did you enable software filtering? My guess is that might fix it.

With RGB set to "full" and 16-235 turned on, I get black crush. No BTB and no WTW in test pattern.

Assuming I was right above, you are actually getting full range signal from VAAPI, and your TV is still showing it as limited range. This leads to loss of near black and near white.

With RGB set to "full" and 16-235 turned OFF, I get even worse black crush. No BTB and no WTW in test pattern.

Not sure why this is different. Video is probably getting rendered with the wrong assumption about its range.

With RGB set to default and 16-235 turned off, I get the closest picture to the SW decode (at least I think). No BTB and no WTW in test pattern.

I think you're getting full range video scaled to limited range by your GPU. You'll notice banding in the grayscale ramp (in misc patterns)

With RGB set to default and 16-235 turned ON, the XBMC menus look noticeably washed out... video though is pretty close to RGB = limited and 16-235 set to off... which I guess is the default... weird.

The menus look washed out because they are first scaled to limited range by XBMC, then again by the GPU - this puts black too high and white too low. Video goes through a different path (which may expect limited range output from the decoder when VAAPI is giving full range).

So, at the end of the day it looks like possibly best PQ is with software decode which is kind of a bummer. Will have to tinker some more.

It sounds like your TV behaves the same as mine. I am happily using hardware decoding with correct levels, so there's just some setting missing.