v17 Music Scanning issue
#1
Hi,

So I am generally adding new albums to Kodi on a daily basis and for most times the library update works as it should i.e. it ignores existing albums and scrapes the new ones or picks up any changed. With this general processing it only takes a couple of mins to add the new music to the library. But there seems to be a scenario where library update initiates a info lookup itself on the entire library and seems to be around say once a month. This causes the update to effectively take at least a day or more on a library of 85k+ songs 5,300+ albums. If it's not doing a info lookup then something additional is going on as it seems to check every library entry being scanned. Is this doing a monthly check of hashes or something?

I've got 'Prefer online info' and 'fetch additional info' but set to 'off' as well.

Is there a reason this seems to happen?, couldn't see anything to config to stop it if so?.

Would just like to know if it's intended and if there is a time frame so I can better manage it's execution.

I don't have logs as it only happens now and then and it just happened to initiate yesterday when adding latest purchases.

Mike.
SKIN: Aeon Madnox
RIP Schimi2k, we miss you.
When I grow up I want to be a skilled skinner
Thank me by ⬇ adding to my + reputation
Reply
#2
I've not seen this myself but I did post about this in another thread recently, as my large music library updates take 25mins just just to add a single album. The time it takes to scan my NAS for unchanged files seems to be my issue.

In my opinion a nice fix for this would be to add the feature that video scanning has where you can click on the source dialogue and select "Exclude Path from Library Updates".
Reply
#3
Instead of using the global Library Update option, use one of the other methods available... https://kodi.wiki/view/Update_Music_Library

You can scan the entire library, by source or individual item.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#4
Yep thats actually a nice workaround, just a bit too many clicks for my liking Smile

I wonder if I could scheduled it with an Add-on. Interesting thought..
Reply
#5
These are available...

https://kodi.wiki/view/Add-on:XBMC_Library_Auto_Update

https://kodi.wiki/view/Add-on:Library_watchdog

I have never used them though, so can't comment on them
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#6
Yeh had a quick look at both.

- Library Auto Update can only do a global update
- Library watchdog does not work on android

The search for a fully automated music system continues... Big Grin
Reply
#7
Thanks for the info and suggestions everyone. I closed down Kodi while doing the update, had got to 'S' and restarted and went and did a scan and this is doing exactly them same, bypassed everything already done, got to 'S' and doing something on every entry, not just finding the new additions.
SKIN: Aeon Madnox
RIP Schimi2k, we miss you.
When I grow up I want to be a skilled skinner
Thank me by ⬇ adding to my + reputation
Reply
#8
Mike, you're not the only one, I've noticed the same thing. Once updated, if I run a library update again soon after, it completes quickly. If I wait a few weeks or months, when I update it rescans the tags on every item in my library, which in my case takes several hours.

I've experienced this for as long as I can remember, always assumed it's just how Kodi works. Would be great if it only scanned for new items.
Reply
#9
(2018-03-29, 13:57)Mike_Doc Wrote: Thanks for the info and suggestions everyone. I closed down Kodi while doing the update, had got to 'S' and restarted and went and did a scan and this is doing exactly them same, bypassed everything already done, got to 'S' and doing something on every entry, not just finding the new additions.
Do you still have the log for the second run? Can you post it?
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#10
I know I run library updates via UpdateLibrary(music) command, and even if nothing is changed, from watching the extended progress dialog progress bar, sometimes the update is very fast but every now and then it is slow.  It's been like that for as long as I can remember.  I assumed it was recalculating and storing a hash in the db.  It also seems like there are two phases.  The first is the one subject to the periodic slow scan, the second is always lightning fast.  This is before the clean which is announced separately in the dialog.  If you have fetch addl info I can always see in the dialog when it is doing the scrape.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix see: Aeon MQ5 Mod Matrix release thread
Nexus see: Aeon MQ5 Mod Nexus release thread
Aeon MQ 5 skin and addon repo 11.1.0
Reply
#11
Firstly there are changes to scanning in v18, so I need to know for sure which version you are using and experiencing this behaviour. Mike is v17.6, but Braz or Scott have you seen this in the v18 nightlies? Secondly a debug log (with debugging on) of update lib being slow when there are few or no new things to scan would be helpful if either @scott967,  @Mike_Doc  or @braz can catch that. I know not easy if it is a matter of waiting weeks/months!

@docwra as you suspected you don't have this issue, just a very large collection that Kodi takes 25 mins to check all the hashes. It is in the background, but if you can't wait then use the selective scan facility to target the item(s) you are adding.

Then a little about the update library facility (when "fetch additional info" is disabled):
  • Update library always looks briefly at every folder and file in your library checking the hash of name, size and timestamp for changed or new files. When it finds one it scans the tags of that file, a process that takes longer.
  • There is no time limit or hash refresh setting, whenever you run update library is does the same checking
  • The progress bar is slow to start as a lower priority task, not every file processed gets reported.
That is core Kodi, it is possible that addons e.g. skin helper service maybe, cause some kind of periodic hash refresh and thus longer scan. I really don't know, but I have never had such an experience either in testing or on my family system.

My other guess would be a clock related issue causing Kodi to see the timestamp on many folders/files has having changed hash. Again, not sure how or what.

Finally something else is runing that slows the scanning processes. I have seen that happen in v16 with a very large db on MySQL, with an addon that was making very inefficient db queries. I fixed that issue, but I guess it could have reappeared in another form.

But bottomline that behaviour is not by design or expected and I can't reproduce it, so provide more evidence please.
Reply
#12
Good news, @robweber is going to look into just updating the individual source instead of full library in his Add-on

https://github.com/robweber/xbmclibrarya.../issues/37

This can be done via the JSON-RPC
Reply
#13
(2018-03-30, 13:29)docwra Wrote: Good news, @robweber is going to look into just updating the individual source instead of full library in his Add-on
This has nothing to do with the OP.
Also the facility to update just one source already exists via Kodi GUI, nothing exciting about having an addon do it. So to be clear for any other user reading this , you do not need an addon (from an unofficial repo) to be able to selectively update only part of your music library.
Reply
#14
<offtopic> All very true of course... but this does give the OP or anyone else an option to update their library without the the delay mentioned via a workaround.

I use this technique with the video library to only update a "Latest" source while leaving the majority of my audio media in a cold storage kinda state. 95% of my media never changes and hasn't for 10 years Smile</offtopic>

Back on topic... Could the OP's error have something to to with the metadata sites changing a URL or new data. Perhaps Kodi recognizes this and prompts a full exta info scan on all items. Just a random theory as I have not seen this behaviour myself with latest nightlies..
Reply
#15
@DaveBlake I'm using v18 but I've experienced this intermittently for as long as I can remember, at least going back to v16 betas. I'll keep looking for it to happen again and will post the debug log, though like you say it might be a few weeks or more. Thanks for all you do to support music!
Reply

Logout Mark Read Team Forum Stats Members Help
Music Scanning issue0