[ronswanson]

Hi everyone,

I've been struggling with this problem for a while now, and I couldn’t find a solution in the wiki or forum. My Movies library only contains about one fourth of all my movies. They are all in the same folder, and thus part of the same source. I thought there might be a naming problem, but I checked a few of the missing movies and they were named correctly (exact name from TheMovieDB, no additional information). I also tried adding they year, but to no avail. For example, I have a movie called “8 Mile.mkv” that’s missing in the library. I renamed it “8 Mile (2002).mkv”, re-scanned the source and it’s still not showing up. The problem seems to occur for both .mkv and .mp4 files.

When I access the movies through Videos/Files/myMoviesFolder, they all show up and can be played without a problem. I also noticed that the movies not showing up in the Movies library show up with the file extension (.mp4, .mkv, ...) in the list under Videos/File/myMoviesFolder, but the rest don't have a file extension there (which I guess must mean they were correctly matched and their file name was replaced by the title).

Does anyone have an idea what could be the problem? Thanks in advance!

FYI: I'm using Kodi 17.3 on Windows 10, the files are stored on a Samba NAS.

Edit: the problem even seems to be inconsistent. I just removed the source from my library and added it again. This time it was only able to correctly scan about one fifth of all the movies.

[Karellen]

Hello ronswanson and welcome,

Could you go to Videos>Files then bring up the Context Menu for the Source with your missing movies. Select "Change Content" and either detail here the settings you have, or take a screenshot and post it. https://imgbb.com/

How many movies do you have in that Source?

Also, can you provide a debug log. Instructions in my signature below. Use the Basic method. Once you Enable debugging, restart Kodi, then perform an Update library. Once complete, post the log to pastebin and link back here.

[ronswanson]

Hello ronswanson and welcome,

Could you go to Videos>Files then bring up the Context Menu for the Source with your missing movies. Select "Change Content" and either detail here the settings you have, or take a screenshot and post it. https://imgbb.com/

How many movies do you have in that Source?

Also, can you provide a debug log. Instructions in my signature below. Use the Basic method. Once you Enable debugging, restart Kodi, then perform an Update library. Once complete, post the log to pastebin and link back here.

Hi Karellen

Thanks for your reply! Here's the debug log (I re-imported all the movies, to reproduce the problem): http://paste.ubuntu.com/24791475/. Here are the links to the

first and second screenshot of the source details. The source contains 566 movies, 120 of which show up in the Movies library.

[Karellen]

Hello ronswanson,

It turns out that TMDB has made some changes to their API. The Kodi team have just today released a fix for this issue. I am wondering if you were a victim of this issue as the symptoms seem similar.

Here is some further reading on the issue if you are interested... https://forum.kodi.tv/showthread.php?tid=315813

Can you check your TMDB addon and ensure you have updated to the latest version 3.9.4. Go to Add-Ons>My Add-Ons>Information Providers>Movie Information. Click "The Movie Database" and then select update.

Also, 566 movies in one directory is not really advisable. You should divide that number into at least 4 separate Sources.

Let us know if that resolves the problem

---Edit---
If you don't have the update yet, you can download from here and install as a zip via Add-ons... https://kodi.tv/addons/browse?keyword=tm...=title+ASC

[globalist]

Hello ronswanson,

It turns out that TMDB has made some changes to their API. The Kodi team have just today released a fix for this issue. I am wondering if you were a victim of this issue as the symptoms seem similar.

Here is some further reading on the issue if you are interested... https://forum.kodi.tv/showthread.php?tid=315813

Can you check your TMDB addon and ensure you have updated to the latest version 3.9.3. Go to Add-Ons>My Add-Ons>Information Providers>Movie Information. Click "The Movie Database" and then select update.

Also, 566 movies in one directory is not really advisable. You should divide that number into at least 4 separate Sources.

Let us know if that resolves the problem

Hi,

I have a similar issue - incidentally I have today deleted my myvideos107.db file and re-added my movies source which contains cca. 150 movies. But only 14 of them are getting scraped now (whereas prior to my messing with things most of them were scrapped correctly). Also, upon finding this thread I checked the Movie DB addon and it's already on 3.9.3 (despite my setting to only notify me of addon updates, not install them automatically, weird...)

In my log I'm noticing these troubling lines:

Code:

09:58:29.847 T:3508   ERROR: CCurlFile::Open failed with code 404 for http://akas.imdb.com/title//|accept-language=en-us
09:58:29.847 T:3508   ERROR: ADDON::CScraper::Run: Unable to parse web site
09:58:30.035 T:3508   ERROR: Previous line repeats 1 times.
09:58:30.035 T:3508   ERROR: ADDON::CScraper::GetVideoDetails: Invalid XML file (want <details>)

Any ideas? Fortunately I saved a copy of my movies107.db so if all else fails I can just go back to that, but this seems like there's a deeper underlying issue?

Many thanks!

[Karellen]

Hello globalist,

Can you post your full log please. Remember to Enable debugging in Settings>System Settings>Logging. Restart Kodi, the perform an update library. Post at pastebin.

Thanks

[globalist]

Ok I sorted it by realizing there's actually an even newer version of the addon - 3.9.4 - which was not showing as available in the Kodi repo. But I was able to force an update in the GUI and all is fine now.

Edit: Choosing Upgrade on the addon's page didn't update the 3.9.3. But choosing Upgrade in the context menu from the addons list where MovieDB was showing, did the trick.

P.S. Don't we all just love unannounced (or unchecked?) API changes from data providers!

[Karellen]

Ok I sorted it by realizing there's actually an even newer version of the addon - 3.9.4 - which was not showing as available in the Kodi repo. But I was able to force an update in the GUI and all is fine now.

Ok good to hear. Enjoy!!

[ronswanson]

Hello ronswanson,

It turns out that TMDB has made some changes to their API. The Kodi team have just today released a fix for this issue. I am wondering if you were a victim of this issue as the symptoms seem similar.

Here is some further reading on the issue if you are interested... https://forum.kodi.tv/showthread.php?tid=315813

Can you check your TMDB addon and ensure you have updated to the latest version 3.9.4. Go to Add-Ons>My Add-Ons>Information Providers>Movie Information. Click "The Movie Database" and then select update.

Also, 566 movies in one directory is not really advisable. You should divide that number into at least 4 separate Sources.

Let us know if that resolves the problem

---Edit---
If you don't have the update yet, you can download from here and install as a zip via Add-ons... https://kodi.tv/addons/browse?keyword=tm...=title+ASC

Hi Karellen

Great tip, thank you so much! I performed the update using the ZIP, now it was able to scrape 556 out of 566 movies. The remaining 10 movies look like they aren’t named correctly, so that’s my fault =). I haven’t yet split the files into multiple, smaller sources, but I will do that.

Just out of interest: where did you find the information about having less elements per source? I'm curious as to how and why too many files in a single source can cause problems.

Thank you very much for your help! Also, thanks to globalist, for your input.

[DecoStein]

Done, (via ZIP). Tks!

[PatK]

According to this https://technet.microsoft.com/en-us/libr...10%29.aspx Microsoft article, the lookup time of a directory increases proportional to the square of the number of entries. Folders with such a large number of files are exceedingly inefficient given the file systems used on Windows (even NTFS will take its sweet time to find a file in such a monster folder). Once the directory structure becomes larger than the filesystem cache performs drops like a ball of lead. programs like Kodi suffer much sooner than 10s of thousands, because they try to access all files in a given directory to get meta-data, etc. If the folder is too large, you're looking at more rotations of a hard drive platter to just fetch the listing, then when you throw in networking and those speed limitations, you can get some serious slow-downs.

Just using the file system in Kodi, list a folder with +1000 files, compare that fetch time with 100 files; no difference?

[ronswanson]

According to this https://technet.microsoft.com/en-us/libr...10%29.aspx Microsoft article, the lookup time of a directory increases proportional to the square of the number of entries. Folders with such a large number of files are exceedingly inefficient given the file systems used on Windows (even NTFS will take its sweet time to find a file in such a monster folder). Once the directory structure becomes larger than the filesystem cache performs drops like a ball of lead. programs like Kodi suffer much sooner than 10s of thousands, because they try to access all files in a given directory to get meta-data, etc. If the folder is too large, you're looking at more rotations of a hard drive platter to just fetch the listing, then when you throw in networking and those speed limitations, you can get some serious slow-downs.

Just using the file system in Kodi, list a folder with +1000 files, compare that fetch time with 100 files; no difference?

A very interesting insight, thanks for the answer! Assuming I currently have my 550 movies in a folder called "Movies". Now I split them up into sub-folders of "Movies", called "1", "2", ..., each containing at most 100 elements. Then I add the "Movies" folder as a single movie source in Kodi and enable recursive searching. Would this solve the performance problem, or do I need to add these sub-folders as separate sources each?

[PatK]

According to this https://technet.microsoft.com/en-us/libr...10%29.aspx Microsoft article, the lookup time of a directory increases proportional to the square of the number of entries. Folders with such a large number of files are exceedingly inefficient given the file systems used on Windows (even NTFS will take its sweet time to find a file in such a monster folder). Once the directory structure becomes larger than the filesystem cache performs drops like a ball of lead. programs like Kodi suffer much sooner than 10s of thousands, because they try to access all files in a given directory to get meta-data, etc. If the folder is too large, you're looking at more rotations of a hard drive platter to just fetch the listing, then when you throw in networking and those speed limitations, you can get some serious slow-downs.

Just using the file system in Kodi, list a folder with +1000 files, compare that fetch time with 100 files; no difference?

A very interesting insight, thanks for the answer! Assuming I currently have my 550 movies in a folder called "Movies". Now I split them up into sub-folders of "Movies", called "1", "2", ..., each containing at most 100 elements. Then I add the "Movies" folder as a single movie source in Kodi and enable recursive searching. Would this solve the performance problem, or do I need to add these sub-folders as separate sources each?

There's not going to be a big performance hit between 550 and a hundred, but probably enough with the way Kodi cache's, that it would be worthwhile, at least I saw this on my local system, sub folders aside, it's the list length.

[frznto]

Hey Ron,

Awhile back i had a similar issue to you with folder structure and keeping alot of movies. My server is pretty good sized, about 12TB of Movies and TV Shows. Living where I do, the internet has been in general crappy and so the best way to watch stuff was to keep things offline. I keep my 2000+ plus movies organized via alphabetical folders. In the second to last post is my custom library node you can possibly adapt to help you. Working in my off time to look into a more updated version but hopefully this might help you out.

https://forum.kodi.tv/showthread.php?tid=299721