[mclingo]

My RX 460 is about 10-20 % slower than yours, best thing to do when you are having problems is go back to default and keep it really simple, dial in my settings and once you get it running well then you can have a tinker with everything.

1. Run your PC in 4:2:0 10 bit, ignore what people have told you, this is only to ensure max compatibility while you are testing this, set you LAV decoder to D3D11 but make sure that you GFX card is selected and that it doesnt say "native", i've noticed that you get a little more processing umpf from this decoder now over straight COPYBACK. Set KODI DS to 2160p same as your desktop, tick full screen window.

!!!!! MADVR !!!!!!

DISPLAY

2. reset MADVR completely and leave all settings as default.
3. setup your display to PC levels (0-255), set it as 10 bit
4. Set calibration off for now. (see notes at bottom)
5. put nothing at all in display modes unless you are only playing 2D movies, if you are playing 3D stuff use KODI refresh rate switching instead.

RROCESSING

6. do nothing.

SCALING

7. Chroma - lanczos 3 tapes, do not change anything else
8. Leave down scale as you wont be downscaling anything if you have a 4k display
9. Image upscaling, NGI sharp MEDIUM, mine can handle this so your will be able to.
10. upscaling refinement - do nothing

RENDERING

11. General - tick use Direct3D / present a frame for every Vsync, tick full screen exclusive only if you are playing 3D movies.
12. stereo 3D, tick everything for 3D playback
13. dithering - ordered dithering, you will find that this card wont be able to use error diffusion for 4k HDR movies, it will hammer the GPU. - use Ordered dithering.
14. Trade quality for performance - untick everything except - optimize subtitle quality.

Obviously this assumes you havent setup any profiles, you could do this for dithering for instance where you could use error diffusion for non HDR movies, or ramp up processing for 1080p stuff, for not just use the above settings, you can break it as much as you like later when you've got a decent playback benchmark to strart form.

NOTES - you may find that colour output might be wrong when using madvr, stuff for some movies might look very red, if so turn calibration on and set it to BT2020 with a BT709 curve, you can experiment with this though.

Let me know how this goes.