[phonix6]

Hi,

OS: Ubuntu 18.04.3 LTS
Kodi: 18.5

I have just upgraded from Kodi 16.1 to 18.5. I have a shared MySQL database. The MySQL database seems to upgrade successfully, myvideos116 and mymusic72 databases are created, all movies are in the library, watched status etc. is all fine. However, when scanning the library for new content, Kodi begins to scrape every file as if it were new. The result is two entries for each file that is already in the library.

All sources are identical to before as I copied the sources.xml and mediasources.xml from the old Kodi 16.1 installation to the new 18.5 installation.

Any ideas would be most welcome as I would like to avoid rescraping the whole library if possible.

Many Thanks.

[Klojum]

To receive meaningful assistance you will need to provide a full debug log.

The instructions are here... debug log (wiki)

If you are using the Basic Method, then ensure the following is applied...
1.Enable debugging in Settings>System Settings>Logging,
2.Restart Kodi
3.Replicate the problem.
4.Upload the log to Kodi Paste Site manually or use the Kodi Logfile Uploader. (wiki) With either method post the link to the log back here.

If you are using the Advanced Method ensure you have correctly created and applied the advancedsettings.xml file (wiki)

In both instances, you should see the word DEBUG throughout the log.

Note: Full logs only. No partial or redacted logs
Do NOT post your logs directly into the forum. Use the Kodi Paste Site. Post the link to your pasted log in the forum

[phonix6]

To receive meaningful assistance you will need to provide a full debug log.

The instructions are here... debug log (wiki)

If you are using the Basic Method, then ensure the following is applied...
1.Enable debugging in Settings>System Settings>Logging,
2.Restart Kodi
3.Replicate the problem.
4.Upload the log to Kodi Paste Site manually or use the Kodi Logfile Uploader. (wiki) With either method post the link to the log back here.

If you are using the Advanced Method ensure you have correctly created and applied the advancedsettings.xml file (wiki)

In both instances, you should see the word DEBUG throughout the log.

Note: Full logs only. No partial or redacted logs
Do NOT post your logs directly into the forum. Use the Kodi Paste Site. Post the link to your pasted log in the forum

Hi, thank you, the Kodi Paste Site wouldn't work, so have used the ubuntu paste site as mentioned in the Debug instructions. The log is here: https://paste.ubuntu.com/p/5h37pw7wzX/ . Hope this is okay.

Many Thanks

[Klojum]

However, when scanning the library for new content, Kodi begins to scrape every file as if it were new. The result is two entries for each file that is already in the library.

I hope that a 3rd scrape doesn't add the videos for a 3rd time, because that would be a real problem.

Theoretically, having 2 entries for one video means there must be a difference somewhere in the total (network) path to that video file.
It could even be an extra space in the filename. You can verify the full path of each video via the Information page, and scroll through the options at the option.
The last option should also display the full path on your system to that specific video.

The log file is a large one, so it's a tough cookie.

BTW, which video/movie is showing more than once? Something we can perhaps track in the log file.

[phonix6]

However, when scanning the library for new content, Kodi begins to scrape every file as if it were new. The result is two entries for each file that is already in the library.

I hope that a 3rd scrape doesn't add the videos for a 3rd time, because that would be a real problem.

Theoretically, having 2 entries for one video means there must be a difference somewhere in the total (network) path to that video file.
It could even be an extra space in the filename. You can verify the full path of each video via the Information page, and scroll through the options at the option.
The last option should also display the full path on your system to that specific video.

The log file is a large one, so it's a tough cookie.

BTW, which video/movie is showing more than once? Something we can perhaps track in the log file. 

Thank you for your response. No, a 3rd scrape doesn't add it a third time. The network paths haven't changed, they are on a NAS box and I copied the sources.xml file so exactly the same. This seems to be an issue with migrating to Kodi 18.5 as migrating to 17.6 is fine and does not have this issue.

I have been looking at this further and have tested the install on a Windows machine with the same result. The database migrates successfully, but the library rescrapes. I have also tested going from Kodi 16 to 17, this is fine, the library does not rescrape. What I have noticed is that when migrating the database to Kodi 18.5 the folder source content is set to 'None', which means I have to select 'Movies', Kodi sees this as a content change and asks if I want to update items in the folder. This can be cancelled, but when scanning the library again, Kodi wants to rescrape the whole library. I believe this may be where the issue is. When the database migrates to version 18.5 it doesn't keep the source as 'Movie', manually changing this means Kodi will rescrape the library.

You can look for a Harry Potter one, that is one that rescrapes, I believe they all would if I let it finish.

Many Thanks.

[fritsch]

For the record: We don't supporting upgrading from 16.1 to 18.5 - only from 16.x to 17.x and from 17.x to 18.x

[phonix6]

For the record: We don't supporting upgrading from 16.1 to 18.5 - only from 16.x to 17.x and from 17.x to 18.x

Hi, I tried that also, I went from 16.1 to 17.6 to 18.5. 16.1 to 17.6 is fine, going to 18.5 is the issue even from 17.6 to 18.5. I have looked at the MySQL database tables and can see that when Kodi upgrades from 16.1 to 17.6 it uses the same 'Path' table line in the database. When you go to the folder \ content the scraper and settings are retained. However, when going to 18.5 it seems that it can not use the old 'Path' table entry, so the content is set to 'None'. When setting this back up again, Kodi creates new lines in the database. Setting it back up essentially means that I am changing the content, so Kodi will re-scrape everything.

The only way I see around this at the moment would be to set the current folders to 'Exclude path from library updates', then create new folders and set them as the ones to be added to the library.

[Klojum]

Kodi's database is not the easiest to hack manually, but in the end the least amount of time spent when there are such 'double' troubles, is usually a video library export with the old version, and a subsequent import via the new version. Then there is also no hassling with various content settings.

Finding out the why and the how is only for those who have lots of time on their hands.

[Preacher]

Kodi's database is not the easiest to hack manually

It's not the most well-designed schema, either - with column names of c00, c01, c02 etc which require some deciphering.

Nonetheless, there's a set of triggers that handle cascade deletes well, so someone's thought about off-loading data sanity operations onto the DB.

I'm guessing it was never intended for manual tweaking - but I've done some cleanups via SQL hackery; I worked out the schema just through trial and error and it's not difficult to understand, just not particularly intuitive.

I'll also concede that a scrub + rescan is the most surefire way of fixing duff entries, IME.

[phonix6]

Thank you all for your replies. I have ended up setting the current media sources not the be scanned by the library and set-up new folders for any new files. The new sources are set to be scanned. This seems to be the best work around other than scraping everything from scratch.