Almost the same hardware here (Intel/Asus/RadeonHD5770/Denon) and I noticed if I use just the HDMI connection to the AVR, DTSHD-MA, DolbyTrueHD and all other formats are perfect.
If I
also have the DVI connection to my monitor connected, no DTSHD or DolbyTrueHD, gets to the AVR. But all the other formats do passthrough.
I have not tried to resolve this, since after my last clean install, it has run great!
5-6 year old system with el'cheapo card and the memory and CPU usages are barely noticeable even with 40Mbps reference disks.
Great work !!!
Gotham's new video rendering has negated the need for an external player/renderer setup. This makes calibration so much easier. On that note, the color range should be set to Limited 16-235...the same range all movies use. While the picture will immediately not look as "black", this is only because of the need to reset the brightness (black level) and contrast (white level). You should notice a significant increase in dynamic range (lower black level and higher white levels) once set up correctly.