[Eneko]

The Fix from memory:

In Audio Settings > Keep Audio Device Alive > OFF

That wasn't it but it did point me into the right direction. The problem is caused by the Audio Output Configuration option. Anything other then "Fixed", so the "Best Match" and "Optimized" options cause this behaviour. If I use one of these two options most videos start about 8 seconds in and most of them stutter for a few seconds more before starting correctly with sound.

Using "Fixed" prevents this from happening but this outputs any file as 2.0 PCM. So that really isn't an option.

EDIT: This problem is almost 99% certain the same thing that causes the audio track changes to work so bad. It creates the same weird stutters and shift in time. I'll do some more testing this afternoon and I'll try if the Log has something to offer and I might open a dedicated thread about it. Maybe better suited in the Odroid Forum??