[DaveBlake]

Code:

2020-12-22 08:05:33.494 T:2859860848 DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/z/z1444/' due to no change
2020-12-22 08:18:17.390 T:2859860848 NOTICE: My Music: Scanning for music info using worker thread, operation took 24:08
2020-12-22 08:18:17.391 T:2859860848 DEBUG: Process - Finished scan

This seems to suggest to me that once we are done scanning (logline for z1444 is printed), it still takes 12:43:896 before the next logline is printed. Do you know what is happening in those 12 minutes?

No, I don't know where those 12 mins go, it is odd. I have tested and checked old logs from other runs that I happened to have, I don't get to see anything with a gap like that. But I'm not running on RPi with your SD card etc.

There are a couple of things it could be doing in certain circumstances - cleaning orphans from the database and compressing the database - neither make log entries. I would not expect either of those to take 12 minutes, but I suppose there could be an issue with the SD card causing file based SQLite to be slow (generally SQLite is much faster than client server set-up with MySQL or MariaDB).

You could try cleaning the music library from system > media > library clean music Library and see if that takes 12mins. I realise that isn't possible without a UI, could you plug in keyboard and screen to the RPi? Beware it does a deeper clean and will remove entries for any inaccessible media, unlike orphan cleaning that allows for removeable drives etc. so take a backup copy of the database file first just in case. It makes similar operations in SQLite so could at least give a clue where 12 minutes could go.

Grasping at straws a little, and I understand if you are content just to let it be slow and carry on with your life, but I am curious about what is happening.

[Yponac]

Kodi runs on my RPI which is connected to my receiver, as such, I don't have a screen/mouse/keyboard attached to the PI to help me reconfigure Kodi. Is it possible via the webgui to scan new items to my kodi library?

Ah so kind of running "headless". For music I often run TV off and use Yatse for controlling Kodi, having added a custom command to it to scan the music library. But there are both JSON methods and a builtin command (used by Python scripts) to trigger library scan. I would take one of those approaches rather than have it happen every restart, see https://kodi.wiki/view/HOW-TO:Remotely_update_library

Yeah, it's mostly headless. My receiver is connected to the TV so I have a screen, but I don't have a keyboard/mouse there. I can of course for testing purposes attach one, but for daily use I don't have it in place.

I will try with Yatse, I do have it installed on my phone but last time I tried to do a clean/update music library, it did not fetch the new folders.

-Yp

[Yponac]

Code:

2020-12-22 08:05:33.494 T:2859860848 DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/z/z1444/' due to no change
2020-12-22 08:18:17.390 T:2859860848 NOTICE: My Music: Scanning for music info using worker thread, operation took 24:08
2020-12-22 08:18:17.391 T:2859860848 DEBUG: Process - Finished scan

This seems to suggest to me that once we are done scanning (logline for z1444 is printed), it still takes 12:43:896 before the next logline is printed. Do you know what is happening in those 12 minutes?

No, I don't know where those 12 mins go, it is odd. I have tested and checked old logs from other runs that I happened to have, I don't get to see anything with a gap like that. But I'm not running on RPi with your SD card etc.

There are a couple of things it could be doing in certain circumstances - cleaning orphans from the database and compressing the database - neither make log entries. I would not expect either of those to take 12 minutes, but I suppose there could be an issue with the SD card causing file based SQLite to be slow (generally SQLite is much faster than client server set-up with MySQL or MariaDB).

You could try cleaning the music library from system > media > library clean music Library and see if that takes 12mins. I realise that isn't possible without a UI, could you plug in keyboard and screen to the RPi? Beware it does a deeper clean and will remove entries for any inaccessible media, unlike orphan cleaning that allows for removeable drives etc. so take a backup copy of the database file first just in case. It makes similar operations in SQLite so could at least give a clue where 12 minutes could go.

Grasping at straws a little, and I understand if you are content just to let it be slow and carry on with your life, but I am curious about what is happening.

Hmm, if I were to put a USB drive in, could I configure Kodi to use that for it's DB? And do you think it would be faster?

I can try and do a clean, I'm also very curious what would cause such a long delay!

-Yp

[DaveBlake]

Hmm, if I were to put a USB drive in, could I configure Kodi to use that for it's DB? And do you think it would be faster?

I think SD card use generally is fine, up until recently I was running Kodi on RPi and SD card daily, and don't expect USD to be better. But I do wonder if your SD card has an issue, a few other users have experienced some very odd faults that vanish with a new SD card.

Do that full clean test and see if it is slow.

[JohnnyL]

I use a similar setup to the OP and was having a similar problem. After reading a comment somewhere on the Kodi Forum, I changed the link between the NAS to RPi from SMB to NFS. The speed improvement was quite dramatic, going from around 30 minutes for a full scan, down to around 5 minutes.

I also use Yatse as a remote control, set up with custom commands for scan and clean. I only scan the library after new albums are added, and clean then scan after albums are removed.

This is with an Asustor NAS and an RP3B+ linked by WiFi.

Hope this helps.

[Yponac]

I have done the clean of the music library, but it does not seem like much is logged, I can see the clean being initiated in the log, but no mention of it being finished.

Debug log snipped

Shouldn't it be logging when it's done? 

-Yp

[Yponac]

I use a similar setup to the OP and was having a similar problem. After reading a comment somewhere on the Kodi Forum, I changed the link between the NAS to RPi from SMB to NFS. The speed improvement was quite dramatic, going from around 30 minutes for a full scan, down to around 5 minutes.

I also use Yatse as a remote control, set up with custom commands for scan and clean. I only scan the library after new albums are added, and clean then scan after albums are removed.

This is with an Asustor NAS and an RP3B+ linked by WiFi.

Hope this helps.

Thanks you for your insight! I'll do an attempt tomorrow to see if I can have it over NFS instead of SMB!

-Yp

[Yponac]

Disregard the post regarding the clean, I hadn't scrolled down far enough and did a clean of the video library instead of the music one. Music scan goes fairly fast:

NOTICE: Cleanup: Cleaning musicdatabase done. Operation took 01:52

So the 12 minutes after scanning are not spent doing a cleanup.

-Yp

[DaveBlake]

Disregard the post regarding the clean, I hadn't scrolled down far enough and did a clean of the video library instead of the music one.

That is a common mistake, the UI is poorly layed out IMO

Music scan goes fairly fast:
NOTICE: Cleanup: Cleaning musicdatabase done. Operation took 01:52
So the 12 minutes after scanning are not spent doing a cleanup.

If a major clean takes under 2 mins then the minor clean up that happens after scanning should be even quicker. So there are at least 10 mins to account for.

Interested if the change to NFS helps, it should be generally faster than SMB all round but interesting if that delay after scanning vanishes entirely.

[Yponac]

Interested if the change to NFS helps, it should be generally faster than SMB all round but interesting if that delay after scanning vanishes entirely.

So it took me some time to get around to doing this. The initial complete rescan after moving from SMB to NFS took a bit over 2 hours, which is understandable, it's 250GB+.

However, a rescan afterwards actually performed worse than SMB! From the last album being scanned, until the logline stating that it was finished showed up over 24 minutes passed! (note that this was a rescan started from Yatse, I assume it's the same rescan as when I restart Kodi).

End of log snippet here: https://paste.kodi.tv/omageqasup.kodi

I'll try another restart tomorrow and see if it goes any faster. If there's anyway to see whats happening after the last folder is scanned (I can build/compile/run another version of Kodi with more logging if such a thing is available) let me know.

Cheers,
-Yp

[Yponac]

Replying again here with some more data for you  @DaveBlake , I've switched to NFS and now done a restart of Kodi. The Artist Information Folder (and all it's subfolders) seems to get rescanned on restart since they are "not in the database". The proper music folders however are getting skipped.

Good news is the following, once the last folder was scanned, there was almost no time until it was logged as done. Also, the total time to scan when skipping everything was much much faster.

Here the last few lines:

2021-01-09 22:51:57.489 T:2875597680   DEBUG: DoScan Skipping dir 'nfs://XXX/volume1/Music/FLAC/beets/z/zuriaake/2019-Shen_Ting_resentment_in_the_ancient
_courtyard/' due to no change
2021-01-09 22:52:18.898 T:2875597680  NOTICE: My Music: Scanning for music info using worker thread, operation took 03:51
2021-01-09 22:52:18.900 T:2875597680   DEBUG: Process - Finished scan

4 minutes instead of 30+? Yeah, NFS seems to work better in my setup than SMB. Still begs the question what the hell is happening when using SMB! Again, if you want further debug info or so, let me know what I can do to help, I'm fairly comfortable building things.

Cheers,
-Yp

[DaveBlake]

Sorry @Yponac I am at a bit of a loss with this.

Thanks for trying the switch to NFS fom SMB

However, a rescan afterwards actually performed worse than SMB! From the last album being scanned, until the logline stating that it was finished showed up over 24 minutes passed!

I'm not so sure that the log snippet you posted does show that, or at least it is not an exact comparions with what was happening on SMB. It is the time between the last "DoScan Skipping dir..." and "My Music: Scanning for music info using worker thread, operation took 47:05" but there is also information for an album being scraped and a directory being skipped because it doesn't exist in the meantime.

Your next post rescanning (with nothing new to scan) may be a fairer comparison.

The Artist Information Folder (and all it's subfolders) seems to get rescanned on restart since they are "not in the database". The proper music folders however are getting skipped.

So you have added the Artist Information Folder as a music source and said that you want it scanned to library? Or I guess it may lie as a subfolder of a music source. Anyway you could put it somewhere away from the folders with music files that you have as a source, and then that won't happen. Not problem with it being scanned of course, but it does take time that doing needless hash checking.
 

Good news is the following, once the last folder was scanned, there was almost no time until it was logged as done. Also, the total time to scan when skipping everything was much much faster.

Faster hash checking is as expected.

Code:

2021-01-09 22:51:57.489 T:2875597680 DEBUG: DoScan Skipping dir 'nfs://XXX/volume1/Music/FLAC/beets/z/zuriaake/2019-Shen_Ting_resentment_in_the_ancient
_courtyard/' due to no change
2021-01-09 22:52:18.898 T:2875597680 NOTICE: My Music: Scanning for music info using worker thread, operation took 03:51
2021-01-09 22:52:18.900 T:2875597680 DEBUG: Process - Finished scan

Great no gap between last folder hash check and process finishing.
 

Still begs the question what the hell is happening when using SMB!

Yes indeed.

NFS working as expected also rules out further the possibility of it being database related. I have had another user in the last week with an odd and completely different issue also with LibreElec and access to SMB, but I don't see a common cause.

Thanks for the offer to investigate further, but when you say comfortable building things I am going to assume you don't mean compiling your own builds of LibreElec as that is what it would need to really dig into this. I am tempted to tip-toe away

[Yponac]

Thanks for the offer to investigate further, but when you say comfortable building things I am going to assume you don't mean compiling your own builds of LibreElec as that is what it would need to really dig into this. I am tempted to tip-toe away

Thats actually exactly what I meant, I work a software engineer and while I haven't built LibreElec myself, how hard can it be? (famous last words, I know).

Cheers,
-Yp

[DaveBlake]

I work a software engineer and while I haven't built LibreElec myself, how hard can it be? (famous last words, I know).

Well that changes everything

Kodi is much in need of more volunteers with dev skills, the active dev team is really tiny compared to the millions of users Kodi has and we are running as fast as we can and never keep up. So if contributing to Kodi could be interesting to you then let's talk some more.

I used LibreElec myself for a number of years, only recently abandoning my RPi3 and basing my domestic setup on a cheap (still 2 x RPi price) Windows 10 system. I wanted to run v19 beta, and LibreElec is nowhere near releasing a stable version for RPi2/3 (having dropped the clever bespoke code to use open source KMS drivers, mesa and GBM/V4L2 instead for v19 onwards). Their effort is on RPi4 and better HD etc., older hardware may eventually get support too but when depends on RPi Foundation providing a solution to some video issues (not team LE or team Kodi). I dev on Windows and never got around to working out how to build LE from that environment, and with the Milhouse builds available nightly there was no need anyway. So I can't advise on how to build LE yourself, but I know it is possible and if you have a Linux system it is fairly straight forwards. However done on the RPi itself it will take forever.