[Yponac]

I have Kodi running on Libreelec on a Raspberry-pi 4 (4GB). My music is stored on my NAS and contains some ~1000 albums in the following structure:

a/<artistName>/<albumName>/tracks
b/<artistName>/<albumName>/tracks
etc..

The tracks are mostly (95%) in FLAC format and takes up roughly 250GB at the moment.

I have configured Kodi to rescan my audio library on startup, it takes just under 30 minutes to do so. The NAS is wired, however the raspberry-pi
is connected via WiFi. 

I am guessing that the long scan time is mainly due to the RPI being on wifi, however, I was a bit surprised to see the following in the logs:

DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/' due to no change
2020-12-21 10:15:06.512 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/' due to no change
2020-12-21 10:15:06.798 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/a_loathing_requiem/' due to no change
2020-12-21 10:15:07.183 T:3011417424   DEBUG: ------ Window Deinit (DialogNotification.xml) ------
2020-12-21 10:15:07.459 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/a_loathing_requiem/2015-acolytes_eternal/' due t
o no change
2020-12-21 10:15:07.593 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/aara/' due to no change
2020-12-21 10:15:07.964 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/aara/2020-en_ergo_einai/' due to no change
2020-12-21 10:15:08.464 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/aardvarks/' due to no change
2020-12-21 10:15:08.862 T:2858812272   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/a/aardvarks/1994-eyes/' due to no change
...

NOTICE: My Music: Scanning for music info using worker thread, operation took 26:41

Why does it continue to go travel into directories if it has determined that the parent directory does not have any changes?

Thanks
-Yp

[Klojum]

Two possible speed bumps on your RPi 4 device are:

- Wifi: The RPi's antenne is tiny, and may be hampered as well by the Rpi's casing. Every obstacle between the RPi and your router antennas will result is a (small) speed drop.

- If you are not using a MySQL/MariaDB on your NAS but use the RPi's SDcard instead, you will have write speed restrictions because of the slowness of the SDcard. Read speeds are usually fine on an SDcard, write speeds not so much. Also continuous writing is not an SDcard's favorite hobby. Sooner or later there will be file corruption. Using an SSD on the USB3.0 port via a proper USB3/SATA cable (you can even boot your RPi from USB3.0), will ensure data safety.

Please provide the full debug log (wiki) next time via a pastebin website, so we get the full picture.

[Yponac]

Two possible speed bumps on your RPi 4 device are:

- Wifi: The RPi's antenne is tiny, and may be hampered as well by the Rpi's casing. Every obstacle between the RPi and your router antennas will result is a (small) speed drop.

- If you are not using a MySQL/MariaDB on your NAS but use the RPi's SDcard instead, you will have write speed restrictions because of the slowness of the SDcard. Read speeds are usually fine on an SDcard, write speeds not so much. Also continuous writing is not an SDcard's favorite hobby. Sooner or later there will be file corruption. Using an SSD on the USB3.0 port via a proper USB3/SATA cable (you can even boot your RPi from USB3.0), will ensure data safety.

Please provide the full debug log (wiki) next time via a pastebin website, so we get the full picture.

Thank you for your reply. I do suspect that the problem is due to wifi, which is something I would have to look into getting the RPI wired up.
With regards to your second point, right now there is only an SD card in the RPI, another point on my list of items to tackle. 

However, my initial question remains unanswered. Is there a reason why Kodi searches into subfolders when it has detected no changes in the parent folder? Is it only an unfortunate logline? Does 

2020-12-22 07:54:12.295 T:2859860848   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/' due to no change

mean that it found no new files in that folder, not that no changes were done in any subfolder?

I have uploaded a debug log here: https://paste.kodi.tv/pefenatiha.kodi

I removed a large number of lines stating which folders were being skipped due to no change. I see no reason in putting my entire music library up publicly. The rest of the log should be enough to determine if there are any other problems I hope.

Cheers,
-Yp

Note:

$ grep -c "DoScan Skipping dir" kodi.log
1472

[Klojum]

However, my initial question remains unanswered. Is there a reason why Kodi searches into subfolders when it has detected no changes in the parent folder? Is it only an unfortunate logline?

Scanning and scrapers isn't really my strong suit, so hopefully @Karellen or @DaveBlake (or others) can step in.

[Yponac]

Just some additional information, it looks like the scanning of the folders took 00:11:21:199

Start: 07:54:12:295
End: 08:05:33:494
Diff: 00:11:21:199

Interestingly the last lines show this:

2020-12-22 08:05:33.494 T:2859860848   DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/z/zuriaake/2019-Shen_Ting_resentment_in_the_ancient_courtyard/' due to no change
2020-12-22 08:18:17.390 T:2859860848  NOTICE: My Music: Scanning for music info using worker thread, operation took 24:08

Meaning there are about 12 minutes from the last skipping dir logline, till when the scanning is done.

Start: 08:05:33:494
End: 08:18:17:390
Diff: 00:12:43:896

Is this normal?

[Karellen]

I have uploaded a debug log here: https://paste.kodi.tv/pefenatiha.kodi

The log does not show you scanning any music. Please provide another log.

[Yponac]

I have uploaded a debug log here: pefenatiha.kodi (paste)

The log does not show you scanning any music. Please provide another log.

Please re-read my post. I removed the roughly 1200 lines saying "DoScan Skipping dir" as I did not see the need to publicly post my entire library. I looked at all the timestamps from those lines and the maximum time to scan one folder was about 3 seconds, 90% were under 1 second to scan. Average was 0.463s. Total time scanning (skipping) folders was a bit over 11 minutes.

If you also read the next post I made, there was a clarification regarding where the 28 minutes are spent. Could you explain what happens in the 14 minutes _after_ the last folder was scanned? 

I would also like to reiterate my initial question, why does Kodi search into subfolders when it has declared that it found no changes in the parent folder?

Thanks again,
-Yp

[Karellen]

Please re-read my post.

Please re-read my post. Provide me a FULL and unedited Debug Log which captures the problem you are reporting

[Yponac]

Please re-read my post.

Please re-read my post. Provide me a FULL and unedited Debug Log which captures the problem you are reporting

https://paste.kodi.tv/jofehomizu.kodi

[Karellen]

I would also like to reiterate my initial question, why does Kodi search into subfolders when it has declared that it found no changes in the parent folder?

I understand this is your question, but it is one that @DaveBlake will need to answer as he is the developer of Music in Kodi.

I can clearly see the lengthy time in scanning, but you do have approx 1,500 folders that need scanning.
DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/z/z1443/ - is that one album? Out of curiosity, how do you know what album z1443 is?

Do you need to run an Update Library at startup? Wouldn't it be a lot quicker to add a new album to your Source, then in Kodi navigate to it and select "Scan item to library'?

[Yponac]

I would also like to reiterate my initial question, why does Kodi search into subfolders when it has declared that it found no changes in the parent folder?

I understand this is your question, but it is one that @DaveBlake will need to answer as he is the developer of Music in Kodi.

I can clearly see the lengthy time in scanning, but you do have approx 1,500 folders that need scanning.
DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/z/z1443/ - is that one album? Out of curiosity, how do you know what album z1443 is?

Do you need to run an Update Library at startup? Wouldn't it be a lot quicker to add a new album to your Source, then in Kodi navigate to it and select "Scan item to library'?

Thank you for your quick reply,

Just out of curiosity, would putting all my flac files into one gigantic folder, make scanning faster? (The files themselves are are tagged proper, so the structure is mainly something for my own convenience).

Yes, z1443 is a single album, 9 FLAC files, 377 MB.  If I may ask, why point out this album? From the logs it looks like it scanned at a nominal time? I was looking at the lines right after that:

 
2020-12-22 08:05:33.494 T:2859860848 DEBUG: DoScan Skipping dir 'smb://XXX/Music/FLAC/beets/z/z1444/' due to no change
2020-12-22 08:18:17.390 T:2859860848 NOTICE: My Music: Scanning for music info using worker thread, operation took 24:08
2020-12-22 08:18:17.391 T:2859860848 DEBUG: Process - Finished scan

This seems to suggest to me that once we are done scanning (logline for z1444 is printed), it still takes 12:43:896 before the next logline is printed. Do you know what is happening in those 12 minutes?

Regards,
-Yp

[Yponac]

I would also like to reiterate my initial question, why does Kodi search into subfolders when it has declared that it found no changes in the parent folder?

Do you need to run an Update Library at startup? Wouldn't it be a lot quicker to add a new album to your Source, then in Kodi navigate to it and select "Scan item to library'?

Kodi runs on my RPI which is connected to my receiver, as such, I don't have a screen/mouse/keyboard attached to the PI to help me reconfigure Kodi. Is it possible via the webgui to scan new items to my kodi library? Right now, since the RPI has SSH, I can login and restart Kodi which forces an update library. Not the best solution, but it works. (And to be honest, I don't add albums _that_ often, I can live with the scan time although if there's a bug in how Kodi scans, then it'd be great to have that solved of course!).

-Yp

[Karellen]

If I may ask, why point out this album?

No reason. I had scanned the log and that is where the log was open when I copied the text to ask my question.

Do you know what is happening in those 12 minutes?

Unfortunately not, but it was a question I was going to ask the developer in our team forum. He is away for a couple of days (Christmas) so it may take a couple of days for him to respond.

[DaveBlake]

First a question for you @Yponac - why do you want to do a library scan on every start-up? Do you add new music that often? I only do a rescan when I have added new music, and sometimes I just scan the new folder(s) rather than the entire library paths (I need TV on to initiate that, but it is useful if I want to see the entries quickly).

I think other replies have covered why it could be slow generally - wifi, SD card access.

Why does it continue to go travel into directories if it has determined that the parent directory does not have any changes?

It needs to examine each subdirectory, saying that a parent has not changed just covers any files in that directory not the subdirectories. The initial scan is a relatively quick hash check of size and timestamp values, so every path gets checked but file metadata is only rescanned when the hash has changed.

As for the apparently lost time during scanning, I don't know but I will poke though the log

[DaveBlake]

Kodi runs on my RPI which is connected to my receiver, as such, I don't have a screen/mouse/keyboard attached to the PI to help me reconfigure Kodi. Is it possible via the webgui to scan new items to my kodi library?

Ah so kind of running "headless". For music I often run TV off and use Yatse for controlling Kodi, having added a custom command to it to scan the music library. But there are both JSON methods and a builtin command (used by Python scripts) to trigger library scan. I would take one of those approaches rather than have it happen every restart, see https://kodi.wiki/view/HOW-TO:Remotely_update_library