[lmyllari]

I would prefer hardware decoding (uses less power -> less heat; more CPU available for other tasks). At the moment I'm using software decoding for better quality. My aim is to get the same quality with hardware decoding.

Yes, the vaapi decoding is lower quality than software. It sometimes shows some blockiness in the picture.

I am only using vaapi because the i3 is not strong enough to decode the 2D streams in the 3D blurays. With the i5 I would use software all the way.

What do you mean with "My aim is to get the same quality with hardware decoding". Does this mean you think you can do something about this?

Yes.

The decoding itself should be bit exact; I would be very surprised if it wasn't. The problems are with the RGB range expansion. Video decoder outputs YCbCr data, where Y (luminance) is in the range 16-235. Both Intel hardware decoder and FFmpeg software decoder work the same here. The difference lies in what happens next. With software decoding, xbmc gets the decoded frame in YCbCr format, calculates coefficients taking into account the colorspace and desired output range, and converts to RGB using these. When xbmc requests the decoded frame from VA-API, it specifies the colorspace (bt.601 or bt.709) and gets RGB data. There is no parameter for RGB output range, so VA-API always converts to full range. This means that luminance gets expanded from 16-235 to 0-255 and this is where the problem lies. This creates banding in the output signal.

With software decoding, we can avoid the conversion from 16-235 to 0-255 and get nice clean output.

My plan is to add a range parameter to VA-API and modify it to support limited range (16-235) RGB output. Then xbmc rendering and gpu drivers need to be checked to make sure they pass this through unmodified. The latter part I have already prototyped using the "Video 16:235 pass-through" xrandr rgb range setting.

All of this is for TV users. PC monitors usually want full range signal. That can also be improved by using limited range output from decoder, and dithering when expanding to full range. For software decoding this should be pretty straightforward by adding dithering to the colorspace conversion shader. For hardware decoding I'm not yet sure how it would work. The dithering should not be done before scaling. There are also changes coming with the VA-API post processing. If the decoder output goes through deinterlacing, other post processing, colorspace conversion and scaling, it gets interesting. If all of that could be done with 10bit resolution and dithered at the end, that would be the best for accuracy. I'm not sure if the hardware supports that, and I'm pretty sure the drivers don't.