Solved Library update does full rescan first time v19 used (Windows)
#16
It would be unsurprising for a Windows version to produce different folder path hash values from a non-Windows version (CoreElec is on RPi?).  This cross-platform path hash sensitivity is something that might eventually be resolved with the futher changes to system time functionality that are coming.

But are you experiencing repeated deep updates from same nightly and same flavour of platform?

If you have multiple Kodi clients on various platforms connecting to a database on a server then I recommend that you only run library update from the client with the fastest server access and most powerful processor.
Reply
#17
I had some time to investigate this further and installed two Kodi v19 portable nightlies of 2020-03-02 on two different Windows 10 machines with the same Windows version and revision. While I was at work, I let one of the machines update the library which lets it go through the deep scan that occurs after that patch. All albums were shown as newly added after that. Then I started the library update again to confirm that everything was scanned in correctly and indeed, that second run was as fast as expected.

Next I opened Kodi on the other machine and triggered a library update. The update went through a deep scan again but no albums were shown as newly added. On running the library update on the first machine again, yet another deep scan ocurred.

This means, that hashes calculated vary from machine to machine. I don't know how small a time-fraction is involved in generating hashes but if it's microseconds or atomic values the smallest time difference might trigger a deep scan. If the display of newly added albums uses a more coarse time-scale like minutes, that could explain why no new albums were shown between the various runs.

My recommendation at this point would be to use a more coarse time to generate hashes (if time is indeed a factor), at least for some tests. I'd be willing to run test builds on my windows machines.
Reply
#18
Is there any chance to revert this patch https://github.com/xbmc/xbmc/pull/17358 until the rescan issue is resolved? This is holding me back from updating to a nightly newer than 2020-02-17.
Reply
#19
Sorry but no that will not be reverted. Why not simply use one client to initiate the library update rather than whichever one you happen to be using at the time. It is a very simple user workaround.

It is behaving as if the calculated hash values vary from machine to machine, but I have not been able to reproduce this myself.

Also I don't think hash calculation is dependant on the system time of the machine doing the scanning, but on the file name, size and timestamp returned for the music files. I assume that the music files are on a server or NAS drive, do both your machines see the files as having the same name, size and timestamp? I can't think why they wouldn't but it is worth checking.
Reply
#20
I may not have been clear enough, but the rescan also happens on the same machine after some time. The "server" is simply a networked Windows computer with attached storage. The external drive is a WD MyBook 8TB with the file system formatted as exFAT.
Reply
#21
(2020-03-14, 15:34)HeresJohnny Wrote: I may not have been clear enough, but the rescan also happens on the same machine after some time.

The files are untouched in the meantime ?  No tag editing or anything that might alter the timestamp of the files ?
Learning Linux the hard way !!
Reply
#22
Quote: the rescan also happens on the same machine after some time
Are you certain about that? Or could different nightly versions of Kodi have been involved?  Hashes produced after PR17358 are different from those produced before it, so switching versions will cause a deep rescan.

Or as Black_eagle asks, any file changes? Or has clock change happened in your region? Have the file/folder timestamps returned by your WD MyBook changed?

Perhaps you could archive your debug log files daily (manually copy them as Kodi will overwrite next restart) , and then have  the logs from two consecutive deep scans. Comparing then might show something. But at a loss because I am unable to reproduce this.
Reply
#23
Well, chances are that it's something on my side since noone else has come forward... I'll keep on digging. The first thing I'm going to do is re-format the drive as NTFS.
Reply
#24
No change. I've captured two (partial) library updates (huge!) done straight after each other. Kodi thinks folders change all the time, for example look at Alicia Keys.

The only reason I could think of this happens is if Kodi looked at atime instead of mtime. mtime for most files is a few months back. ctime is obviously newer since I've just moved all files in and out of the drive. Even so that wouldn't explain why Kodi continuously sees files and folders as changed.

Also, not sure if applicable: https://dfir.ru/2018/12/08/the-last-acce...most-back/
Reply
#25
OK, that gives something to work with. First general thing I notice is that not everything is scanned both times, it does not see every hash has having changed. I'm not sure what that means, but it is significant.

Using MariaDB, so all the SQL gets logged too which does make for large logs but is useful as it gives the hash values it calculated each time. That could tell something too.

@HeresJohnny can you provide the times - ctime, ctime, atime -  that Windows file explorer sees for the Alicia Keys folder and files.

Kodi does not access the atime, although an interesting idea as it is a time that varies, I will double check that.
Reply
#26
I don't actually know if a Last Access exists for a folder or how to access that. Otherwise:
Image

Image

Image
Reply
#27
I will also add that the external WD drives automatically add a "WD SES Device" to device manager. WD is quite mum about what that does or how it may interfere with the regular operation, see https://support-en.wd.com/app/answers/de..._id/19581/
I had already disabled those devices a few days back.
Reply
#28
Detailed explanation of which timestamp is for what and how to see all of them in Windows -> https://superuser.com/questions/973547/h...timestamps
Learning Linux the hard way !!
Reply
#29
I'm going to tentatively say that this patch has solved the problem, but I will need to watch the behaviour for the next few days.
Reply
#30
I was wondering if that PR would help it sounded related, but tracing through could not see it being used during hash creation. So yes, do try it and report back.
Reply

Logout Mark Read Team Forum Stats Members Help
Library update does full rescan first time v19 used (Windows)0