[Jalau]

The jellyfin addon actually writes directly into the kodi database as if the movies folders were added to kodi themselves. Jellyfin just makes sure that my library is synced across all kodi devices and the jellyfin webinterface. It's still only network mounted drives with the media on it.
Therefor Jellyfin also does only refresh when changes in the database have been made. It uses some sort of last changed algorithm for that so it only applies changes if needed. Therefor the jellyfin addon definitely isn't the issue, however I will try and see if the advanced settings change anything! Thank you for the tip!

EDIT:
I also went through the general settings and disabled the youtube trailer stuff. This definitely seems to have improved performance. The widgets still take like a bit to load, like 10 seconds but not a whole minute anymore.
Althoug the similar media widgets still take a long time  to load.