2017-05-08, 15:00
Split this out to own thread so we can focus discussion on speed/size issues. I suspect that there is more than one problem involved, so perhaps we can usefully break this down and at least come up with some best practices for those with large music collections to get the best from Kodi.
There is a known fundamental design flaw that makes the songs node particularly slow if like @marantz you have a splendid library with 130k of songs. The UI does not show anything until all the song data has been fetched from the db and them manipulated in local memory, fetching every thumbnail etc. first. Everytime you leave and return to the songs node it repeats this time consuning processing. On investigation fetching the data from the DB takes only 5% of the total time (even on MySQL), the rest is memory manipulation inside Kodi.
Those that have read the Devcon blog report will know that there is work in progress to refactor both the video and music databases into one combined db using an ODB interface. In principle ODB supports lazy loading, and so could ultimately help to improve how Kodi handles large volumes of data. However it is lazy loading on the UI side of the processing that is needed to overcome the slow songs node. Simply switching to using ODB will not resolve this issue at all, there is much additional work to be done elsewhere away from the database. I have ideas, but it is more work than I can cope with alone. No doubt it's time will come one day.
All nodes use the same "fetch all and sort locally" design approach and so they can be slow just like the songs node if they contain enough items relative to processor power, and memory access speeds. I don't know how many artists or albums you have in your library @marantz (could you share those counts), but do you find all nodes slow?
I know from working with @Uatschitchun on this thread http://forum.kodi.tv/showthread.php?tid=306766 that there are other things that can make perfromance slow too. Some of these are things that can be avoided perhaps.
The other work around is to try to avoid large nodes using custom nodes and smart playlists as @docwra suggests. It is possibly easier for me to make accessing the library in smaller chunks available as new features than fix the fundamental large node issue. It also seems more useful to me to be able to get to the 20 items you want than have to scroll through hundreds of pages of items.
But is it just large nodes that are slow, or are there other issues?
There is a known fundamental design flaw that makes the songs node particularly slow if like @marantz you have a splendid library with 130k of songs. The UI does not show anything until all the song data has been fetched from the db and them manipulated in local memory, fetching every thumbnail etc. first. Everytime you leave and return to the songs node it repeats this time consuning processing. On investigation fetching the data from the DB takes only 5% of the total time (even on MySQL), the rest is memory manipulation inside Kodi.
Those that have read the Devcon blog report will know that there is work in progress to refactor both the video and music databases into one combined db using an ODB interface. In principle ODB supports lazy loading, and so could ultimately help to improve how Kodi handles large volumes of data. However it is lazy loading on the UI side of the processing that is needed to overcome the slow songs node. Simply switching to using ODB will not resolve this issue at all, there is much additional work to be done elsewhere away from the database. I have ideas, but it is more work than I can cope with alone. No doubt it's time will come one day.
All nodes use the same "fetch all and sort locally" design approach and so they can be slow just like the songs node if they contain enough items relative to processor power, and memory access speeds. I don't know how many artists or albums you have in your library @marantz (could you share those counts), but do you find all nodes slow?
I know from working with @Uatschitchun on this thread http://forum.kodi.tv/showthread.php?tid=306766 that there are other things that can make perfromance slow too. Some of these are things that can be avoided perhaps.
- MySQL is upto 10 times slower than SQLite on some queiries, so it may be worth sacrificing client/server setup for stand alone installations.
- MariaDB seems to be considerably faster than MySQL (not sure of the versions, so this could be misleading conclusion)
- Skins that use the Library Data Provider or skin.widget addon invoke inefficient additional data access, so disabling those and using skins like Estuary or Confluence may be quicker.
- For random playback of large nodes, party mode playlist is the way to go.
The other work around is to try to avoid large nodes using custom nodes and smart playlists as @docwra suggests. It is possibly easier for me to make accessing the library in smaller chunks available as new features than fix the fundamental large node issue. It also seems more useful to me to be able to get to the 20 items you want than have to scroll through hundreds of pages of items.
But is it just large nodes that are slow, or are there other issues?
