[HeresJohnny]

Since a couple of nightlies music library update is slow. Before, the update would run through all the folders and only read changed files. Judging from the display, it now seems as if each file is read in-depth at the start of each Kodi session. When the initial run is completed, further updates in the same session run at the expected speed. Could that be a side effect of the recent ffmpeg bump?

[DaveBlake]

I don't think this relates to ffmpeg. There have been recent changes related to system time use, and that is more likely the source of any time/hash checking issues which could possibly lead to full scanning every day. However I have not been able to repeat this, so all wild guesses from me.

Do you have it set to run library update on start-up @HeresJohnny ? What platform are you running on?

[black_eagle]

Can't say I can reproduce that.  I built a fresh master, started it and re-scanned my largest source (~23000 files).  I answered 'No' to do a full tag scan even if files unchanged.  Result.

xml:
2020-02-21 09:23:09.521 T:16776  NOTICE: My Music: Scanning for music info using worker thread, operation took 01:16

Set it to scan on startup and restarted.  Forgot that it would find a bunch of test stuff and scan it in .  Even so, this was the result.

sql:
2020-02-21 09:29:56.459 T:15658  NOTICE: My Music: Scanning for music info using worker thread, operation took 01:20

OS - Ubuntu 18.04.1 x86_64

Grab yourself a debug log and have a look inside to see what Kodi is doing.  It should show a bunch of 'skipping dir <whatever> due to no change'.

[DaveBlake]

Your testing is done same day @black_eagle and that could be significant, hence why subsequent updates in the same session are reported as OK in the OP. Platform could also contribute (time handling varies), but thanks for trying.

[DaveBlake]

I'm going to correct that, I suspect it is using an older build to populate and scan the files into the library that matters. My hypothesis  is that the hash of timestamp and size is being formed different lyin  newer nightlies, hence when it does the hash check against the old data it sees it as different and so rescans the files as if they had changed. @black_eagle perhaps you could test that and confirm?

I am also hoping that doing a library  update check on subsequent days does not rescan, it is just a once off issue between nightlies. Need to let time pass to test that unless @HeresJohnny can clarify what his experience is.

The unexpected rescan is a slow process especially if you have "fetch additional info on update" enabled and no NFO files, because it will scrape too.

[DaveBlake]

Call off the testing cavalry (@black_eagle )

The cause of the full rescan of all files is https://github.com/xbmc/xbmc/pull/17358 - a change to the internal data structure used for timestamps. This inadvertently impacts the way the path hash values are created, so the first library update performed using a nightly build with this change (merged 3 days ago) will rescan all files even though they have not changed.

There are plans for internal data structure used for timestamps to change again before v19 is released, which will cause similar full rescanning.

Those with large music libraries that are trying the nighly builds for v19 beware update library could trigger a full rescan

[black_eagle]

@DaveBlake I don't think I can confirm that, or at least, not on Linux.  I dropped my database, restarted the current master build which upgraded an older db and then scanned all my music sources.  As the scan was set on startup, it scanned all my sources that I have added recently for testing purposes and also some albums that I added recently that weren't in the previous database.  It did not rescan anything that was already in the db.

eg

xml:
2020-02-21 15:08:04.328 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/' due to no change

2020-02-21 15:08:04.332 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Back in Black/' due to no change

2020-02-21 15:08:04.334 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Black Ice/' due to no change

2020-02-21 15:08:04.346 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Blow Up Your Video/' due to no change

2020-02-21 15:08:04.378 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Dirty Deeds Done Dirt Cheap/' due to no change

2020-02-21 15:08:04.416 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/extrafanart/' due to no change

2020-02-21 15:08:04.423 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Flick of the Switch/' due to no change

2020-02-21 15:08:04.425 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Fly on the Wall/' due to no change

2020-02-21 15:08:04.428 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/For Those About to Rock (We Salute You)/' due to no change

2020-02-21 15:08:04.431 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/High Voltage/' due to no change

2020-02-21 15:08:04.433 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Highway to Hell/' due to no change

2020-02-21 15:08:04.437 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/If You Want Blood You've Got It/' due to no change

2020-02-21 15:08:04.439 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Live/' due to no change

2020-02-21 15:08:04.443 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Powerage/' due to no change

2020-02-21 15:08:04.446 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Rock or Bust/' due to no change

2020-02-21 15:08:04.449 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Stiff Upper Lip/' due to no change

2020-02-21 15:08:04.452 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/The Razors Edge/' due to no change

2020-02-21 15:08:04.454 T:6050   DEBUG: DoScan Skipping dir 'nfs://192.168.1.50:2049/media/sde1/Music/ACDC/Who Made Who/' due to no change

xml:
2020-02-21 15:11:25.182 T:6050  NOTICE: My Music: Scanning for music info using worker thread, operation took 03:32

Not that I do not have 'Fetch additional information' enabled but in any case all that stuff is held locally in nfo files.

Just for ref - build version, platfrom etc

xml:
2020-02-21 15:05:55.957 T:30026  NOTICE: Starting Kodi (19.0-ALPHA1 Git:20200221-4831b46cf5). Platform: Linux x86 64-bit

2020-02-21 15:05:55.957 T:30026  NOTICE: Using Debug Kodi x64 build

2020-02-21 15:05:55.958 T:30026  NOTICE: Kodi compiled 2020-02-21 by GCC 7.5.0 for Linux x86 64-bit version 4.15.18 (266002)

2020-02-21 15:05:55.958 T:30026  NOTICE: Running on Ubuntu 18.04.4 LTS, kernel: Linux x86 64-bit version 5.0.0-21-generic

2020-02-21 15:05:55.986 T:30026  NOTICE: FFmpeg version/source: 4.2.2-Kodi

2020-02-21 15:05:55.986 T:30026  NOTICE: Host CPU: AMD FX™-8350 Eight-Core Processor           , 8 cores available

[DaveBlake]

Ah cavalry got there anyway, thanks @black_eagle

@DaveBlake I don't think I can confirm that, or at least, not on Linux. 

Yes, looking again it will be platform specific - Windows that is effected not Linux.

Those with large music libraries that are trying the nighly builds for v19 on Windows beware update library could trigger a full rescan

[black_eagle]

Yes, looking again it will be platform specific - Windows that is effected not Linux.

Ummm, still don't regret my decision 7 or 8 years ago to ditch Windows on all my kit... Do you know if affects MacOS or Android or is it purely a Windows thing ?  Just out of curiosity...

[HeresJohnny]

Win64 here and yes, full rescan happens every day since then. "Fetch additional info" is not switched on and I have no nfo files. I got rid of all that to accelerate library scans a while ago.

[HeresJohnny]

As a layman, I have a horrible suspicion: I actually run library update from 2 different Windows computers subsequently. Both used the same Matrix nightly build, but both runs triggered a deep update. Could this even happen if the computer time is apart by a few milliseconds?

[DaveBlake]

@HeresJohnny are your different computers running different nightly builds e.g. one older than 3 days ago the other newer?

No, the current time on the system clock will not make a difference. What matters is where the media files are located and the timesstamp and size that is returned for each source and subfolder. I am unable to reproduce anything in the current nightly that does a full rescan every day, so you are going to have to provide more evidence if it continues to do that your end.

Do you know if affects MacOS or Android or is it purely a Windows thing ? 

I don't know for sure, more a guess because xbmc/platform/win32/XTimeUtils.cpp changed, and I have no way to test.

[HeresJohnny]

@HeresJohnny are your different computers running different nightly builds e.g. one older than 3 days ago the other newer?

No, the current time on the system clock will not make a difference. What matters is where the media files are located and the timesstamp and size that is returned for each source and subfolder. I am unable to reproduce anything in the current nightly that does a full rescan every day, so you are going to have to provide more evidence if it continues to do that your end.

I started scanning the library and it started slow again, so I stopped the scan. My library is just too big for that kind of test, updating takes at least an hour. For the time being, I'm reverting to a prior nightly 20200217-05d55687 to avoid this.

I don't think the deep scan is triggered daily, but by different windows installations. Just a thought... could each windows installation tag a hard drive internally somehow differently and could this difference be picked up by the Kodi patch? Even if it's just a prefix it would explain if different hashes are seen by different windows installations, even if the nightly Kodi build is the same.

[DaveBlake]

I started scanning the library and it started slow again, so I stopped the scan. My library is just too big for that kind of test, updating takes at least an hour. For the time being, I'm reverting to a prior nightly 20200217-05d55687 to avoid this.

No need to revert, just disable "Do library update on start-up".

If there is an issue with it doing a rescan every seesion then it would help to know more about it. You don't need to wait for complete a full scan to confirm that the files are being deeply scanned every day. Just produce logs from separate days that show the same folder path being scanned
"DoScan Rescanning dir <folder path> due to change"
and not
"DoScan Skipping dir <folder path> due to no change"

 And you did not confirm if you had same nightly builds on all computers, I'm betting that you did not.

[HeresJohnny]

No need to revert, just disable "Do library update on start-up".

I never claimed that was turned on and it isn't. Updates are only ever triggered manually.

 And you did not confirm if you had same nightly builds on all computers, I'm betting that you did not.

Both used the same Matrix nightly build, but both runs triggered a deep update.

I already had the problem of the whole library being rescanned in Leia when done from a CoreElec and a Windows installation alternatively and it was never solved. I guess I'll just wait if someone else comes forward and complains. Until then I'll use workarounds.