[quickmic]

Thanks Dave for your statement from a dev perspective.

Actually, this more or less what I've expected.
Rewriting and changing the concept of such a fundamental part is a pain in the a..
Anyway, I think sooner or later this has to be done.

For the moment, I think some improvement is still possible even under the current approch.
e.g. Subset of music loaded via node...

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="530" type="filter">
    <label>Funk</label>
    <content>songs</content>
    <match>one</match>
    <rule field="genre" operator="is">funk</rule>
    <rule field="genre" operator="contains">funk - </rule>
    <rule field="genre" operator="contains"> - funk</rule>
    <order direction="ascending">artist</order>
</node>

...takes about 12 seconds (on low end hardware).
I think this delay is mostly based on db performance, so nothing to improve here.
After this I have list with around 1937 items. Now I click on a item (song). As far as I know, the playlist will be created at this moment.
This takes between 20 seconds and 3 minutes depending when I click.
I assume the playlist-creation is running in background thread asynchron and if first thread is not finished, while I select a new song (generate new playlist), the first thread will not be cancelled.
Anyway, as you mentioned here is room for improvement but hard work.

Based on that observation:
If I select in the list (not playlist) another song Kodi starts over recreating the playlist.
This is imho absolutely unnecessary, cause nothing has changed. Here is room for improvement cause this is just a decition, when you trigger the create-playlist subroutine and maybe cancel previous thread.

So performance issue is not only the playlist creation itself, it's also when you trigger the routines.


Another spot for improvement:
At the previous example I select the subset of songs via node and it took around 12 seconds.
Now, if I use the group function...

Code:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="10" type="filter">
        <label>Genre-SLOW</label>
        <icon>DefaultMusicGenres.png</icon>
        <content>artists</content>
        <group>genres</group>
</node>

...it is quite fast (almost instantly) showing me the genres list, but:
If I select e.g. Funk as genre it takes 1 minutes. (compared to 12 seconds before)
Here is also imho something wrong cause this is not a DB performance issue. I think here is also a design problem the actual reason.



Additional information:
When playlist creation is in progress, the CPU is not really busy. 4 cores, and peak of one core is maybe 50-60%. Combining load of 4 cores would be approx. 1 core at 100%. The rest would be idle. Not sure if there is some load-balancing and playlist-creation is a single thread, or as it looks multithread but not using the full cpu power.
I gave pop-genre also a shot, but I stopped after 35 minutes playlist creation and still pending.

Final statement:
Good to know, that you (the devs) aware of those issues.
When I have time, I will also investigate on those again to remove the most painful delays. Unfortunately, the code is very complex and as you mentioned, this would need lots of work and really experienced developers to fix the design issues.
And please excuse if I'm wrong on some assumptions. Mostly is based on observation and not too much code review.

[DaveBlake]

Since I have your music db I'll take a closer look at your examples.

EDIT::But I have to note that not only is your music collection huge, but it is very unusual e.g. you have albums with hundreds of songs each (all track 1) , you have 18946 albums (out of 56878) with 1 song but not flagged as a single. Albums to not match their mbid values, songs don't have them. No idea if Embly contributes to any of this. Perhaps useful as a volume test but not respresentative structurally.
 

For the moment, I think some improvement is still possible even under the current approch.
e.g. Subset of music loaded via node...

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="530" type="filter">
    <label>Funk</label>
    <content>songs</content>
    <match>one</match>
    <rule field="genre" operator="is">funk</rule>
    <rule field="genre" operator="contains">funk - </rule>
    <rule field="genre" operator="contains"> - funk</rule>
    <order direction="ascending">artist</order>
</node>

...takes about 12 seconds (on low end hardware).
I think this delay is mostly based on db performance, so nothing to improve here.

Impossible to know how long the db query is taking on your "low end hardware" without a debug log. Testing on my dev env the query against 332k of songs takes ~2s. If I swap from corrulated subquery syntax to joins it is 10% faster, which is a possible optimisation, and even more query specific clever tricks could get the result in a lightening quick 100ms, but either would need a redesign of the way queries are built from smart playlist rules, and hard to do for the possible rule combinations.
 

After this I have list with around 1937 items. Now I click on a item (song). As far as I know, the playlist will be created at this moment.This takes between 20 seconds and 3 minutes depending when I click.

Yes that is when the internal list of songs to play is populated, in this case with the 1937 songs. But I am really surprized by the 3mins and the variability, are you sure and can you provide a debug log of it doing that (turn on debugging, reproduce, post debug.log in pastebin and post link).
 

I assume the playlist-creation is running in background thread asynchron and if first thread is not finished, while I select a new song (generate new playlist), the first thread will not be cancelled.

Actually not. If the queue is taking a long time to populate then the GUI freezes while it does so and you can't select another song or move to another node etc., and there is no way to cancel. That's crap isn't it, but it is how things currently work.

However the player is asynchronous, so if you click on another song (once you can) when something is playing it will continue to play while the queue is cleared and repopulated possibly with the same songs as before.
 

If I select in the list (not playlist) another song Kodi starts over recreating the playlist.
This is imho absolutely unnecessary, cause nothing has changed. Here is room for improvement cause this is just a decition, when you trigger the create-playlist subroutine and maybe cancel previous thread.

I agree room for improvement, e.g. knowing that the list of songs on display has not changed since the queue was populated and hence not clearing it just switching song.
 

Another spot for improvement:
At the previous example I select the subset of songs via node and it took around 12 seconds.
Now, if I use the group function...
...it is quite fast (almost instantly) showing me the genres list, but:
If I select e.g. Funk as genre it takes 1 minutes. (compared to 12 seconds before)
Here is also imho something wrong cause this is not a DB performance issue. I think here is also a design problem the actual reason.

What you quoted "use the group function" was the default genre node, did you mean that? It takes you 12s to show all the genres? Are you then saying that clicking on a specific genre e.g. "funk", it takes 1min to show all the artists with albums or songs with that genre? If so that sounds very odd, again a debug log please.
 

Additional information:
When playlist creation is in progress, the CPU is not really busy. 4 cores, and peak of one core is maybe 50-60%. Combining load of 4 cores would be approx. 1 core at 100%. The rest would be idle. Not sure if there is some load-balancing and playlist-creation is a single thread, or as it looks multithread but not using the full cpu power.
I gave pop-genre also a shot, but I stopped after 35 minutes playlist creation and still pending.

Not sure about processor use TBH, but what you are doing is more memory allocation limited than processing itself. I can't comment on what was happen with "pop-genre" without a debug log.
 

When I have time, I will also investigate on those again to remove the most painful delays. Unfortunately, the code is very complex and as you mentioned, this would need lots of work and really experienced developers to fix the design issues.

Yes. People with the skills and interest are always welcome to volunteer their effort, but the learning curve to be able to contribute to core changes effectively is steep, and there is a lack of experienced developers with spare time to be able guide and mentor. Patience and persistence are needed, and lots of good will on all sides.

I have huge respect for the work of previous devs, but what they originally designed was a media player that could run on an old Xbox. This has limitations given what Kodi has grown organically into, and some design weaknesses, but on the other hand Kodi is an amazing media player.

[quickmic]

I recorded videos for better documentation and I go step by step through each issue:

I start with the node performance problem:
https://1drv.ms/u/s!AsC6yIANTPi5aVlHMQNLrVEbftM

What I did:
I switched for tests to my PC, so high end hardware and plenty of ram.
My music database has songs with single genre and multi-genres e.g.:
A song can have "Alternative" as genre or "Alternative - Country" and so on if you are wondering why there is a difference in the selections.

The first run was with my own created node which also combines the multi-genres:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="40" type="filter">
        <label>Alternative</label>
        <content>songs</content>
        <match>one</match>
        <rule field="genre" operator="is">alternative</rule>
        <rule field="genre" operator="contains">alternative - </rule>
        <rule field="genre" operator="contains"> - alternative</rule>
        <order direction="ascending">artist</order>
</node>

This takes a few seconds...


The second run was with grouped node (not combining the multigenres):

Code:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="10" type="filter">
        <label>Genre-SLOW</label>
        <icon>DefaultMusicGenres.png</icon>
        <content>artists</content>
        <group>genres</group>
</node>

This is much slower.

[wsnipex]

still no Debug Log. Are you ignoring the repeated requests on purpose?

[quickmic]

ok, I will make a new sample with logs.

[quickmic]

Now with logs.
Additional information:
I'm currently rebuilding my music-database to stock (not emby mod). Progress is at the moment 25%, so the effect is smaller but still there.
Also I switched to stock skin and uninstalled plugins.
I selected Pop-Genre to get a bigger subset, cause of the not fully synced db.

With node (no grouping):

Code:

<node order="530" type="filter">
    <label>Pop</label>
    <content>songs</content>
    <match>one</match>
    <rule field="genre" operator="is">pop</rule>
    <rule field="genre" operator="contains">pop - </rule>
    <rule field="genre" operator="contains"> - pop</rule>
    <order direction="ascending">artist</order>
</node>

https://1drv.ms/u/s!AsC6yIANTPi5bZt2EHdpNjhO78Q
https://1drv.ms/u/s!AsC6yIANTPi5apKQ50Kjs0VQTxc


With node (group):

Code:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<node order="10" type="filter" visible="Library.HasContent(Music)">
        <label>135</label>
        <icon>DefaultMusicGenres.png</icon>
        <content>artists</content>
        <group>genres</group>
</node>

https://1drv.ms/u/s!AsC6yIANTPi5bCx0fegqboEDnjg
https://1drv.ms/u/s!AsC6yIANTPi5ay2kNNDY44guJ34

I assume the performance lag comes from the sql queries.

Another thing...
Actually, this is more a feature request... Not important at the moment, but I wanna mention it here and maybe I'll open another thread for that.
It would be great if group-subselections could be defined.
At the moment it's Artist -> Album -> Song.
imho, this should be adjustable in general (not only for music-db).
e.g. Artist -> Song or Artist -> Year -> Song or whatever.

[Lunatixz]

From my experience Kodi MySQL performance issues are always directly related to the hardware and configuration of the SQL server.

[quickmic]

I'm using sqlite.

[Lunatixz]

I'm using sqlite.

 for an extremely large collection? I'd consider moving to MySQL db. You are possibly expecting too much out of your hardware? which was? sorry if I missed it in a prior post.
Assuming you are using a standard arm device... Memory and system I/O are usually already taxed... If we are taking about a PC make sure you have a high-speed storage device and I would also recommend storing the db on a drive outside of the OS.

[quickmic]

For test-cases I run kodi on my workstation. i5 3.2ghz 16gb-ram 256gb ssd.
It should run like a charm, but doesn't.
Yes, the db-size has an impact on the performance but only a minor one.
The delays are mainly based on ineffective code design in some situations.
e.g. the video about the node issue shows that very clear.
My created node does more or less the same as the group-parameter node, but the difference is huge.
I think the "group" parameter triggers ineffective unnecessary sql queries.

[DaveBlake]

My created node does more or less the same as the group-parameter node

No it does not, they are totally different.
For one, your custom "pop" songs node, the results are songs with genre "pop", or containing "pop -" or " - pop".
The other, what you are calling the "group-parameter node", is genres.xsp the default genre node, the results are a list of all genres.

There are radically different numbers of such songs to the total number of genres, so no surprize one is much quicker to display than the other. I have to asume that you are trying to discuss something else, but I don't undertstand what.

[quickmic]

Ok, let me describe differently.
genres.xsp shows the list of genres instantly. So all good till here.
But when I select in this list a genre it starts loading the Artist-list and that is damn slow.

How it imho should work:
genres.xsp shows the list of genres (exactly as it does and fast).
Now I select POP:
...this should trigger:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="530" type="filter">
        <label>Pop</label>
        <content>artists</content>
        <match>one</match>
        <rule field="genre" operator="is">pop</rule>
        <rule field="genre" operator="contains">pop - </rule>
        <rule field="genre" operator="contains"> - pop</rule>
        <order direction="ascending">artist</order>
</node>

also fast and would create a Artist list:

Then the Album list:

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="530" type="filter">
        <label>Pop</label>
        <content>albums</content>

        match one of the following parameters (logic OR)
       
        <rule field="genre" operator="is">pop</rule>
        <rule field="genre" operator="contains">pop - </rule>
        <rule field="genre" operator="contains"> - pop</rule>

        and match artist "abba (logic AND)
        <rule field="artist" operator="contains">abba</rule>
        <order direction="ascending">artist</order>
</node>

and same for finally songs:


So the logic would be, if we stick to genres.xsp...
Create genre list as genres.xsp does already very fast.

then something like
Select genre e.g. pop
select * from songs where genre=pop group by artist

Select artist e.g. ABBA
select * from songs where genre=pop and artist=abba group by album

Select album e.g. GOLD
select * from songs where genre=pop and artist=abba and album=gold

I think each sql query would be very fast
I tested a few of them via nodes e.g.
show all album from ABBA where genre is POP

Code:

<?xml version='1.0' encoding='UTF-8'?>
<node order="530" type="filter">
        <label>Pop</label>
        <content>albums</content>
        <match>all</match>
        <rule field="genre" operator="is">pop</rule>
        <rule field="artist" operator="contains">abba</rule>
        <order direction="ascending">artist</order>
</node>

[DaveBlake]

OK, your renaming a default node and reusing the name had me confused!!

You clicked on the default Genres node, in your setup label "Genre-Grouped" and genres are shown. Then click on a specific genre, in the example "Alternative", and after a 16s delay a list of artists with songs of that genre are shown.

In comparison you click on your own custom menu node called "Genres", which shows a list of custom nodes. Then click on the specific custom node called  "Alternative" and after less than 1s or so see a list of songs with gerne or "Aternative" or conrtaning "Alternative -" or "- Alternative

Your question is why does getting the list of artists (via default nodes) so very slow compared to getting the list of songs (via custom node), especially since there are more songs than artists. Is that right?

I have to say I don't know, but I don't think it is the SQL queries. When I repeat the same queries against a your data provided previously (for "Pop" same as in the logs) the times are reasonable, with fetching the songs taking 4 times longer than the artists as I would expect. The time in the logs includes processing the resulting dataset in memory, so will try and look there for an explanation.

That is of course if I have understood what you are trying to draw attention to.

[quickmic]

And final question...
I started again debugging kodi in Eclipse...
I start from where I stopped while last session here: https://forum.kodi.tv/showthread.php?tid...pid2629661
The main reasons while playlist loads that slow are still valid.
As I'm not really familiar with the code, one final question.
What's the deal with that Announcement Manager. It seems slowing down playlist-creation.

[quickmic]

Your question is why does getting the list of artists (via default nodes) so very slow compared to getting the list of songs (via custom node), especially since there are more songs than artists. Is that right?

Yes, exactly! I think it's the sql queries. My 2 cents, maybe something is preloading all genres even not selected.