Kodi Community Forum

Full Version: @Devs, please review performance issues
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3
Firstly, thx to all devs for their big affords.
You are doing awesome work here!
I just have a general request. Please test Kodi with big databases while developing. There are some major performance issues when huge collections are used.
Playlists are damn slow, nodes are unusable in case of "group" parameters etc. Even on high end hardware.
If you need sample databases I can upload something for tests.
First, how about properly doing a thread description.
Further logs, used version and whatever else are missing.
(2019-01-19, 23:31)quickmic Wrote: [ -> ]I just have a general request. Please test Kodi with big databases while developing
FYI: general requests don't really help: "work harder, be faster...". And what is 'big'? Or huge? It's all in the eye of the beholder. A big database in general for me starts at more than a million records are in a single table. How many videos or music files does your database have?

We are computer people. We need detailed information, accurate requests, direct data. A bit "more of this or that" doesn't compute for us.

Let's start with a debug log (wiki) and see what sort of contraption you have installed Kodi on.
I don't know how a better thread title should reflect a imho general development problem.
Version? Anything from 17 up to nightly 18 which I'm currently using. Not tested previous versions.
Logs? There are no errors, just slow in case of big databases. e.g. Trying to play a song from a couple thousands selection takes minutes.
But if you think this thread is inappropriate, just delete it and I zip it.
big databases means:
Audio: 332844
Music videos: 22578
Movies: 3840
TVShows: 12136
One fix... 15284 (PR)

It's all a work in progress. Some issues are known, some are yet to be discovered. But your post was too general and severely lacking in information to respond to properly.
To be honest, I tried to investigate that performance issue a long time ago, but then I stopped focusing of my own other projects.
Anyway, latest I figured out was this:
2629661 (post)
I think this is completely legacy, but maybe it helps tracing...
So are you referring to the Music Library. If yes, then this is a conversation better had in the Music forum so the music developer, @DaveBlake, will have a better chance of noticing it.

But, yes, there is slowness related to how Kodi sorts and loads music files
I think this in not a music-only issues. I assume this is a general issue, just because of the of the size of my music collection it's best traceable there.
I assume it's a problem with playlist creation. Somewhere here maybe: GUIMediaWindow.cpp in "bool CGUIMediaWindow::OnPlayAndQueueMedia" here -> "g_playlistPlayer.Add(iPlaylist, nItem);"
But I didn't dig that deep.
Currently my Musicvideos are also not really snappy, but my db for videos is 10 times less than audio.
I'm almost sure, the issue appears on every collection.
(2019-01-19, 23:55)quickmic Wrote: [ -> ]But if you think this thread is inappropriate, just delete it and I zip it.
I never said any of that, I simply asked for a log file...
All because we have no crystal ball here, and any details still can be useful.

(2019-01-19, 23:55)quickmic Wrote: [ -> ]big databases means
Details mean things like which database server, which database server version. We want to do those affords you talk about.
Things do not work by magic. Only a week ago we had to implement a bug fix for older MySQL databases.

(2019-01-19, 23:55)quickmic Wrote: [ -> ]Trying to play a song from a couple thousands selection takes minutes.
Sure... so who needs a log file, right? For all I know you installed Kodi on an ancient HAM radio.
Sorry, but general yammering without actually giving us information as requested isn't helping us.

(2019-01-20, 00:13)quickmic Wrote: [ -> ]I think this is completely legacy, but maybe it helps tracing...
Not for me.
I'm using sqlite, but the DB-Size is not the problem. I ruled that out.
Hardware is not the problem. Tested on osmc vero 4k and Ubtuntu. Vero 4k is lower of course, but on i5 amd64 ubuntu also slow.
Anyway, the only thing I wanna report/suggest/offer.
I would upload my DB and my nodes for test cases. Everybody who will use that, will immediately spot the problem.
I've uploaded my complete setup.
@Karellen- database link removed

Tracing general performance issue:
music - > widget genre -> pop -> try play a song ... it will stall

Tracing node group issue:
music - > widget genre-slow -> pop ... it will stall
Hmmm, you do have massive databases, just looking at their size, especially music. It might perk the interest of @DaveBlake

I have also removed your link from public view, but the mods can still access it
btw, I created that DB with emby for kodi plugin, but this is not relevant. The behaviour will not be different on native kodi db, just in case the content looks a bit different...
Yes, Kodi is slow when trying to populate the linked lists that sit behind the GUI and current playlist queue with a large number of items. It is a known issue, and nothing to do with the actual databases and how they are accessed. There probably could be some db optimisation done but it is not the main contributor. Using MySQL (clent/server setup) is slower than (local) SQLite, and a slow processor and data/network access will obviously makes things worse, but there is a design limitation.

What this means in practice is that if you have a large collection of music, for example, then that looking at the songs node (all the songs), or a big playlist, or clicking play from one of those (hence filling the queue with all those songs) is going to be slow. It is often hit in the music library more than video because users have more songs (332844 in this case) than movies (3840), but the issue effects both and pictures too come to that.

To fix this issue means a total redesign of how the GUI works and memory  is used to use some kind of lazy loading, not the current approach which waits until all the items are loaded into the list before displaying any of them. Likewise sorting needs moving into the db (which is quick at it) rather than being done in memory with some algoithmic mistakes.

Frankly the active dev team is very small at the moment so tackling these fundamental redesign issues is slow progress. All we can say is it is known issue and hopefully can be fixed one day.

In the meantime as a workaround make use of smaller nodes e.g. genre, year etc. , smart playlists or party mode (to randomly play songs from a playlist) to limit how many items are displayed or added to the queue at one time. FWIW my prefered work around, since I have a large music collection and like to listen with TV off, is to use a remote app to browse my music and control what I play. It also uses a SQLite db just on an Android tablet and yet UI speed is fine, which anecdotally confirms the slowness is not having lots of songs or using SQLite but what Kodi does with the data once it is fetched.
Pages: 1 2 3