Bug Several problems related to finding and scraping movies to library
#1
Hi,

A topic that has popped up several times but yet I have not found a solution for my problem - not from here, from vero4k forum/discussion, any web pages google can find..

Background:
- two Vero4k players: Vero4k and Vero4k+. Both up-to-date.
   - e.g. using Kodi 19.5 (compiled 2022-12-26)
- both connected via NFS to server (Linux) where all drives are located. NFS shares are mounted first locally and then linked to library as local device. This improves the speed to data transmission, in case you wonder why (was a problem with first vero4k whom was not able to play 60fps UHD via 100Mbit/s network - this helped a lot but was not sufficient in the end)
- all drives and folders use same privileges (these are set automatically correctly when a new movie is stored to any of the drives/folders)
   - folders: drwxdrwxr-x
   - files: -rw-rw-r--
- filenaming convention follows Kodi rules. Sometimes years are causing issues: IMDB having a different year than MovieDB but these are easy to fix. There are some peculiar problems related to naming conventions but that's not the problem here.

Problems:
- it has happened now few times that new movies are not scraped. There have been periods of time that none (? - not 100% sure) of the new movies were scraped. After the period (maybe update related), scraping started to work again. The strange thing is that this does not happen with both devices. The movie can be manually scraped into library but that's not very convenient - also, difficult to remember which have been added and where (there are over 6300 mkv-files; movies, TV shows, documents etc.).
- one example recently was Thor Love and Thunder. It did not get scraped by vero4k+ but vero4k did scrape it. In the first device the movie is "visible" under files thus it is possible to scrape. Although, there have been cases where plain folder is visible, video file is not.
- there starts to be major difference in number of movies (total number of movies at server: 2605 + 223 files named with filetype 'disc') that these devices see with same library settings:
   - vero4k: 3376 (5817+ hours)  (this database was removed completely few months ago because the device refused to scrape any new movie - probably DB got corrupted)
   - vero4k+: 2340 (4290+ hours)
   - what are the differences? Impossible to say. MissingMovies show only few missing - and damn, this is slow plugin (might be related to library size though). First one showing way too many and latter too little.
- there have been times where movie have been scraped but the movie art was not fetched. Manual fetching fixed that. But this did not happen in both devices the same way.

There are probably several bugs instead of one regarding above listing. What would be really helpful to start solving these issues would be to have proper "Missing movies" script that shows what video-files have been scanned in the library, which were not accessible and why (e.g. if the file is broken - should not be, but this can be related to communication between client and server), which files cannot be scraped (movie not found or something else) and finally a comparison to the Kodi's DB. Plain text file would be more than enough for this one (maybe later as a proper plugin).

What can be seen from Kodi's logs is that sometimes Kodi has not updated the path where it tries to look for files. Every now and then movies are relocated. Sometimes removed. Should Kodi be restarted before it's able to update folder paths or does it update the DB of paths regularly?

These problem cases may not be the most trivial one to solve but advises/tips etc. are most welcome.
Reply
#2
Oh, forgot completely that with vero4k+ it stopped adding the type of folder (e.g. movie) to some folders. Have to be manually changed to e.g. movie before anything can be found. This happened to existing shared folders that already contained movies. So, only some of the folders in shared folder were not typed in library.
Reply
#3
(2022-12-27, 23:46)paju Wrote: What would be really helpful to start solving these issues would be to have proper "Missing movies" script that shows what video-files have been scanned in the library,
No, what would be really helpful is the Debug Log that captures the problem.

If you want to see missing movies, check the Event Log... https://kodi.wiki/view/Event_Log
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#4
Log shows a lot of private info so I'm quite concerned about that. Also, filenames may indicate that they are not from normal sources. I can take parts of it but would be good to know what is needed.
Reply
#5
(2022-12-28, 01:28)paju Wrote: Log shows a lot of private info so I'm quite concerned about that
Post the log to Kodi Paste Site and provide the link.
Your account is in Moderation, so the next post will be hidden from the general public. I will be able to see it, grab the log and delete the link.

(2022-12-28, 01:28)paju Wrote: Also, filenames may indicate that they are not from normal sources
I am not interested where they came from. It only becomes an issue if you are using a Kodi addon download them.

(2022-12-28, 01:28)paju Wrote: I can take parts of it but would be good to know what is needed.
Nope. Best to provide the whole thing.

But first check item 1d here... https://forum.kodi.tv/showthread.php?tid=336969
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#6
Here's a link to vero4k log:

vero4k+ log is bigger than accepted by paste.kodi.tv. How should I cut it to make it still useful?
Reply
#7
(2022-12-29, 09:30)paju Wrote: Here's a link to vero4k log:
You did not enable debug mode. So, while I can see there are problems, there is not enough info to tell me why. Try again.

The paste site accepts approx 15,000 lines. If you have more than that, split it into two uploads.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#8
Oh, forgot that. Debug mode enabled and log-file split into 10000 line chunks. (12000, 13000, 14000 and 15000 were too long)

vero4k+:
Part 1 (10000 lines)
Part 2 (9000 lines)
Part 3 (10000 lines)
Part 4 (rest)

vero4k:
Complete log
Reply
#9
vero4k was probably not fully complete. Here's updated log.

Vero4k: kodi.log
Reply
#10
(2022-12-29, 22:25)paju Wrote: Debug mode enabled
Great, thanks. I see a few problems...

1. You are using the most current version of UMS.
You also have a Parsing nfo file to point the scraper to the correct title.
This would normally work quite well, but recently UMS has had problems with IMDB making changes to their site. In this case, the link in the nfo file is not being accepted.
I don't know if @olympia is around to look at this issue.
The quick workaround is to change scrapers to TheMovieDB-Python movie scraper and it should work, even with the parsing nfo file.

2. The second problem is your setup.
You seem to have mixed your TV Shows with your movies. Or you have set the TV Show folder as a Movie source. This will not work.
You need to use a TV Show scraper for TV Shows which is why all your Anime and other shows are not scraping.

3. You need to fix your tv show naming. Seeing this just makes me cringe...
/mnt/media01/HD piirretyt/Neon Genesis Evangelion 01-26 [Renewal + DirectorsCut + End of EVA + Rebuild]/Neon Genesis Evangelion 01-26 [Renewal + DirectorsCut + End of EVA]/zx.death.true2.renewal/zx.death.true2.renewal.divx511.mkv
There is absolutely no hope this show would ever be found named this way.
It should be as simple as this...
mnt/media01/TV shows/Neon Genesis Evangelion (1995)/Season 01/Neon Genesis Evangelion (1995) S01E01.mkv
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#11
Thanks for the tips and help.

1) Actually, I'm not that interested in nfo-files so it'd be better to just turn them off. TheMovieDB is missing some movies which is one reason I'd like all found movies to be shown in the catalogue with or without proper info from IMBD or TheMovieDB. Now these are just not added and it's not easy to find them. Especially when I'm adding sometimes 10-30 movies at time to the library. But, I'll try to change that.

2) This is strange. Need to check it out. I have separated series to own drives and paths and not added them into movies - nor added their type to be movies. But if they still are then that's quite interesting. Should be easy to fix.

3) Yes, I know that some of them have completely incorrect naming. That particular file I've not been bothered to fix yet. But e.g. the latest Thor was correctly named, if can be scraped when done manually but it does not get found if I just update the scanning. This I find strange.

Also, none of the findings yet explain why some folders (movies) just missed the type definition of the folder. These movies are added to HD movies or UHD movies folders like any other and yet they do not get the folder type set. Access rights are the same etc. When I set the type manually, the movie gets found. Haven't seen this issue now for a while though but I still have missing movies because of this but it's not easy to find what is missing.

There used to be a schedule update of movie DB but cannot find this functionality anymore. Has this been removed or am I just blind? It was possible to schedule e.g. every day, every 8h etc.
Reply
#12
(2023-01-03, 00:58)paju Wrote: Actually, I'm not that interested in nfo-files so it'd be better to just turn them off
You can't "turn them off". If they are there Kodi will use them. If they are deleted, well Kodi can't use them.

(2023-01-03, 00:58)paju Wrote: Now these are just not added and it's not easy to find them
Run an Update Library, then check the Event Log... https://kodi.wiki/view/Event_Log

(2023-01-03, 00:58)paju Wrote: 2) This is strange. Need to check it out.
/mnt/media01/HD piirretyt/ is using /home/osmc/.kodi/addons/metadata.universal/', content: 'movies', version: '5.5.12' as the scraper, and this is only possible if you have set the source as Movies instead of tv shows.
And if it is meant to be movies, then maybe you shouldn't be putting tv shows in there.

(2023-01-03, 00:58)paju Wrote: the latest Thor was correctly named
That is not in your log, so I can't comment.

(2023-01-03, 00:58)paju Wrote: and yet they do not get the folder type set
I don't really know what you mean by this.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply

Logout Mark Read Team Forum Stats Members Help
Several problems related to finding and scraping movies to library0