(2016-10-22, 20:13)DarkHelmet Wrote: [ -> ]Scheduling just in the sense of when I enter the music library after before having said "Rescan later" I get asked again if I want to rescan now. If possible I think that's a good way to go with this.
Great, that is simple to do.
Quote:You brought up the case where a rescan is interrupted (loss of internet connection for example). What actually happens then? Can Kodi detect that (% of music files scanned?), fall back to db 56 from Jarvis and try again? The advanced user can of course manually delete the db version 60 but not everybody might have that knowledge.
I feel there is a misuderstanding happening. When you upgrade from one Kodi version to the next the database is migrated, old data is read into new database format. By time you get a UI to use that is all done, and if the process is interrupted then Kodi won't work and you would need to delete the scrambled db and start again.It can happen if the migration is slow because the db is large and the user impatient, or I guess you had a power cut, but it is not a common event (I hope).
What I am talking about here is once Kodi is running (and new db exists) getting it rescan music files because the new version gets more data from those files, or does different things with it. Now changes that require that often also relate to a db structural change too, but not always. It is not a matter of using an old database, just that you have not got everything out the tags that you could now get.
If scanning is interrupted then the tags are not read from all the files. This is also true if you stop a normal library update. But with a normal update you can just re-run and if will find any remaining new/changed files and scan them. What we want here is to scan files that have not changed, because Kodi can get more from them now. If that is interrupted then Kodi has no way to know what files it has not rescanned. To complete the task it would need to go through all the files again. This will just take time, but is not a problem, just need a way to initiate such a thing. But there is a way to get it to do this should such a situation arise.
Hope that is a bit clearer DH.
(2016-10-22, 22:36)scott967 Wrote: [ -> ]I don't see any reason to run the scrapers due to updating to v17, though in some cases if user changed MBIDs it might make a difference (mainly to artists, not so much albums). But these I think could be handled individually, not in a programmed re-scan.
To scrape as well or not, that is the question.....
Scarping a large collection of artists and albums can be very slow. Hammering the servers for info we have already got is not a good design, and could result in a greater chance of timeouts from server overload. Since nothing in the change from v16 to V17 does something different with the scraped data there is no need to repeat the process of fetching it. Obviously the rescan can create new artists because it handles composer tags etc. but it is also better at identifying separate artists e.g. "artist1 feat. artist2" into "artist1", "artist2". If new artists are added then the user can elect to scrape additional data for them if and when they want.
So let's not scrape too.
But in testing rescanning (or unchanged files) I found that while artist entries keep their last scraped date, for albums it is set to null despite the scraped information still being there. I need to dig some more into why this happens.
(2016-10-23, 00:24)DaveBlake Wrote: [ -> ]If scanning is interrupted then the tags are not read from all the files. This is also true if you stop a normal library update. But with a normal update you can just re-run and if will find any remaining new/changed files and scan them. What we want here is to scan files that have not changed, because Kodi can get more from them now. If that is interrupted then Kodi has no way to know what files it has not rescanned. To complete the task it would need to go through all the files again. This will just take time, but is not a problem, just need a way to initiate such a thing. But there is a way to get it to do this should such a situation arise.
That it was I meant. Sorry if my wording isn't always correct. I was talking about an interruption during scanning the files for new information that Krypton can process but Jarvis could not. It has been a while since I scanned all my files into kodi so I might be wrong here. Is there some kind of progress bar when kodi does this?
And to clarify once and for all. During this rescan no scraping of online sources is done at all, right?
(2016-10-23, 13:23)DarkHelmet Wrote: [ -> ]That it was I meant. Sorry if my wording isn't always correct. I was talking about an interruption during scanning the files for new information that Krypton can process but Jarvis could not. It has been a while since I scanned all my files into kodi so I might be wrong here. Is there some kind of progress bar when kodi does this?
Good, I wasn't sure and the more people that have an idea what is happening the better
Yes there is a progress bar during scanning that names the artist and album it is working on.
Quote:And to clarify once and for all. During this rescan no scraping of online sources is done at all, right?
During normal library update scanning scraping happens too if you have "fetch online information during update" enabled.
I am proposing that during this special scanning to collect tag data we didn't process before we ignore this setting and do not scrape too. I am testing to ensure this does not leave the library in an odd state, and got some unexpected results. No doubt I will resolve this, but this thread stands in for design documentation should I fall under a bus before the task is complete!
To be clear, my intention is not scrape online sources and I will post if this view changes.
More on the online scraping question
When albums are updated by rescanning tags from music files, the last scraped date is set to null. Any previously scraped values for mood, style, theme, review, rating or votes will remain, but the genre(s), year, compilation, label and type will revert to what is determined from the song file tags.
If the last scraped date is set then rescraping of additional information can only be invoked album at a time. This would be very tedious, so it is just as well that it is set to null by rescanning of tags, and the album will be rescraped whenever scraping of additional album info is initiated. It means that if users subsequent to a forced rescan want to override tag derrived data with information from online sources or NFOs they will need to "query info for all" from the albums node context menu.
I think it is OK to leave this to user choice, not something to force on them lengthening the rescan process. But you could argue that the user won't know to do this, so do we need to make that an option too? What I have in mind is some thing that checks their system settings to see if they have "fetch online information on update" enabled and then ask if they want to do that as part of rescanning.
Does that mean that you have something in mind to offer a dialog with something like :-
Code:
Do you want to rescan your music library (Kodi V17 uses more tags than V16)
<NO> <YES> <YES and include online information>
Dropping the last button if online info is disabled in the settings.
I was thinking more of
Code:
Music library needs to rescan tags from files. Would you like to scan now?
<Yes> <No>
and if you have "Fetch additional information on update" enabled then it is followed by a 2nd dialog
Code:
Would you like to fetch additional information for albums and artists now?
This could take some time, and you can do so at a more convenient time using "Query Info for all"
<Yes> <No>
although I image we could discuss the right wording for ages.
The other option would be to replace the first dialog with this
Code:
Music library needs to rescan tags from files. Would you like to
<Scan Now> < Scan Later> <Never scan>
but I would still have an separate 2nd dialog, or not mention scraping at all.
Yeah, seems OK, although I was just trying to avoid a scenario with multiple pop-up dialogs. Your second option of now/later/never would seem to be the better of the two.
Did a test upgrading from Kodi 16.0 to 17 1112 nightly. I thought the prompts looked good and worked as expected. Only thing is I forgot to edit my advancedsettings file beforehand to add my <artistseparators> setting.
scott s.
.
Thanks for feedback Scott. Not sure how best we can advise users to get advancedsettings sorted first. But I note that changes to advanced settings is one reason why users could benefit from the facility to start a complete tag rescan whenever they want.
And then it was used on a MySQL server set up with multiple devices......
I had not designed for this situation, and because the prompting to rescan is determined using a value in GUIsettings.xml every device prompts to rescan even when the first device has already caused the tags to be scanned. The immediate workround is to let scanning on subsequent devices start and then stop it (from the side blade).
Having the flag that a rescan is needed outside the library is attractive because it does not require a DB bump to implement, and an xml is easy to edit and force a scan if needed. But it does face the issue of being client specific, and users get pestered to additional and unnecessary rescans. I'm open to suggestions how we avoid this issue.
For MySQL, the options:
Code:
[Yes] [No] [Not this client]
would be more appropriate, where "Not this client" simply resets the "need scan" GUISettings flag.
Ideally, the "need scan" flag should be in the database, but as you point out this almost certainly requires a db bump. However any other non-db solution is likely to be a hack, or need hack workarounds.
My forced rescan was interrupted by loss of power. Is there any way to do this again?
When not finished it should simply ask again
(2016-12-31, 14:49)lexi81 Wrote: [ -> ]My forced rescan was interrupted by loss of power. Is there any way to do this again?
Yes, although a little hands-on, you can modify the guisettings.xml file to trick it into rescanning.
Stop kodi. Edit the guisettings.xml file found in the userdata folder to have the following key value in <mymusc> section
Code:
<mymusic>
...
<needsupdate>60</needsupdate>
...
</mymusic>
When you restart Kodi and enter the music library it will prompt again to do the full scan.
Come back if you need any more clarification.