[lmyllari]

I've done more A/B screenshots of color movies... there is literally no perceptible difference between dithering ON or OFF. After doing a difference comparison of screenshots of the identical frame there is, however, an imperceptible difference: a scattering of pixels throughout the image that are all slightly changed in color. And by slight, I mean something like an invisible change from RGB 69-91-130 to 69-90-129. There's no rhyme or reason to the changes, however.

Single-step changes at seemingly random places is exactly what dithering should look like AFAIK.

Correct. It is not meant to fix any issues in the input but to avoid issues from limited precision output.

Short version: set the GPU (with xrandr) to full range so it doesn't mess with the output and enable dithering with 8bit depth. This will add a small amount of noise when needed (for limited range setting in Kodi it may not be needed). If you need to use limited range GPU setting, try dithering with 7 or even 6 bit depth.

Longer version:

Consider the following input values (say in a greyscale ramp)

Code:

Y       Cb      Cr
16      128     128
17      128     128
18      128     128
19      128     128
20      128     128

Convert to limited range RGB and we get this

Code:

R       G       B
16      16      16
17      17      17
18      18      18
19      19      19
20      20      20

Convert to full range RGB

Code:

R       G       B
0       0       0
1.164   1.164   1.164
2.329   2.329   2.329
3.493   3.493   3.493
4.658   4.658   4.658

8bit output is integers between 0-255. If we take the full range RGB values and simply round them, we get a sequence of 0, 1, 2, 3, 5 - this is how the banding is created.

Dithering solves this by adding noise (output = input + x, 0 <= x < 1) and rounding down the output. This leaves the limited range values above untouched, since the noise added is always less than 1 and the ouput is rounded down. The full range values are sometimes shifted up by 1. Sunflux posted a picture that demonstrates this nicely.


Here are the possible steps the video frame goes through:

1. decoder output (YCbCr)
   
2. colorspace conversion to RGB
   
3. output to an 8bit texture for advanced scaling
   
4. scaling and filtering
   
5. expansion to full range RGB
   
6. dithering
   
7. output to an 8bit framebuffer
   
8. conversion to limited range RGB by the GPU
   

This is the worst case (Kodi is set to full range, advanced scaling is used and GPU is set to limited range output).

At step 3, RGB values are rounded to 8bit. I changed the colorspace conversion to always output limited range to avoid banding at this step.

Step 7 does rounding to 8bit again, and at this point dithering is required to avoid banding from step 5. With dithering applied, the input values are already 8bit integers (with even fewer values if dither depth < 8).

Step 8 is a problem. The GPU just takes the values 0-255 and scales them to 16-235.

Code:

full    limited range   rounded
0       16              16
1       16.8588235294   17
2       17.7176470588   18
3       18.5764705882   19
4       19.4352941176   19
5       20.2941176471   20
6       21.1529411765   21
7       22.0117647059   22
8       22.8705882353   23
9       23.7294117647   24
10      24.5882352941   25

Notice how both output values 3 and 4 map to 19.

Dithering to lower output depth may help with this since there are fewer output values from dithering. I'm not sure how exactly the GPU scaling does rounding so I can not provide detailed guidance. I don't recommend the limited range GPU setting since it will always lead to loss of quality.