[mrob]

Apologies if this is the wrong forum, but I'm not sure where to ask about this.

If I add one movie to my Kodi server, I need to Update Video Library in order to see it in the Remote UI. For some reason, though, Kodi seems to read and check every single movie in my collection, and it's quite slow. I would think it only needs to check new files.

Is there a setting somewhere to change this?

[Buschel]

Thanks for reaching out. From the API documentation I do not see any other command to trigger this (the App uses "VideoLibrary.Scan" for this). Is the Kodi server behaving different when you're triggering the scan from there?

[mrob]

Hey, thanks for your help with this.

Actually, I'm not sure how to trigger the scan other than with the Remote app. I've got Kodi installed on an Ubuntu server instance.

I have Webmin installed, but I'm not sure there are any(?) Kodi modules.

I can start a shell via ssh, though I've never tried to use the command line to talk with Kodi.

Any suggestions?

[Buschel]

Hmm, I am not sure if there is a keyboard shortcut to do this. I am using headless Linux setup but can log in with a remote desktop to use the UI. Is this an option for you? Command line could to my knowledge only use the same JSON command as the App does.

[mrob]

I'm not sure how to get a remote desktop going. It's a headless server. For the setup, I followed some install instructions posted on this forum, and as I recall it was pretty hairy.

Tbh, I'm a bit hesitant to mess around with display drivers, as it works now and it took several days of fiddling to get Kodi talking to my AV setup in the first place.

I do have Chorus installed, and it has a "Scan video library" menu pick. It's also quite slow, though. The Chorus UI is kind of wonky and doesn't give much feedback.

Actually, I'm not clear on something fairly basic: if I do a video scan, will Kodi only check new items, or does it always perform a more extensive scan?

[Buschel]

Ok, then you have no X-window server installed on the Linux setup at all? If this is a special version, which Kodi version is this? For my setup I just used standard Linux Mint distributions and installed Kodi via apt. There is simply no monitor or TV connected, just audio via USB.

To my understanding Kodi scans everything when triggered it.

[mrob]

Right, there's no X-windows server installed or running. I'm using Kodi 19.4 on Ubuntu Server 18.04.6.

At some point, maybe I should re-install everything to get a simpler setup (e.g., with a distro like Mint, as you describe), but that's a bigger project.

If Kodi really scans everything, then should I go to a different forum and ask devs about that (and if so, which one)?

It seems like this has become a question about the behavior of the app itself, not just the Remote.

[Buschel]

If Kodi really scans everything, then should I go to a different forum and ask devs about that (and if so, which one)?

It seems like this has become a question about the behavior of the app itself, not just the Remote.

@kambala, do you have a good idea whom to ask on the topic of library scanning?

[kambala]

@Karellen says the following:

I would imagine he's probably got a thousand movies in a single folder and he runs a global update library which scans every folder and file each time.

But hard to confirm without a debug log.

[mrob]

@Karellen says the following:

I would imagine he's probably got a thousand movies in a single folder and he runs a global update library which scans every folder and file each time.

Hi, well, it's certainly possible I've got it set up wrong, but I tried to exactly follow the recommended method. For example one of my sources looks like this, 

Movies 01 (Kodi Source)
 |-- 2001 (1968) -- Stanley Kubrick
     |-- 2001_A_Space_Odyssey_t02.mkv

 |-- A Christmas Story (1983) -- Bob Clark
     |-- A Christmas Story (1983).mp4
     |-- A Christmas Story (1983).eng.srt

etc.

Each movie file is in a separate folder, like above.

Am I doing something wrong here, or is there something else that could cause Kodi to re-scan everything?

BTW, I do have a couple of NFO files at the top level, alongside these folders (if I understood the instructions for those correctly). Dunno if that matters for this.

[Karellen]

Each movie file is in a separate folder, like above.

That looks good.
Without seeing a log or your setup, I think it has to do with the quantity of media you have. How many movies, tv shows and episodes do you have?

You ask why is Kodi rescanning everything? If you add a new movie, how can Kodi find it unless it scans your source folders?
The Library Update is a very useful feature... up to a point. There comes a cut off point where libraries are too large to use this because of exactly what you are querying.

There are workarounds. But before getting into it, how about providing a Debug Log that captures the slow scanning. Maybe it is a completely different problem shown in the log.

[mrob]

Each movie file is in a separate folder, like above.

That looks good.
Without seeing a log or your setup, I think it has to do with the quantity of media you have. How many movies, tv shows and episodes do you have?

You ask why is Kodi rescanning everything? If you add a new movie, how can Kodi find it unless it scans your source folders?
The Library Update is a very useful feature... up to a point. There comes a cut off point where libraries are too large to use this because of exactly what you are querying.

There are workarounds. But before getting into it, how about providing a Debug Log that captures the slow scanning. Maybe it is a completely different problem shown in the log.

Thanks for your help with this. To answer the first question: around 3700 movies across three sources, and maybe 20 to 30 TV series.

I guess I'm unclear on what's happening during the scan that makes it so slow. The way the titles are displayed slowly on the upper-right corner of the screen, it seems like maybe Kodi is sending Internet queries for details on movies that have been scanned before. I don't know the details of the scanning process, but my expectation would be that Kodi maintains a database of all the movies in the sources, with a column that contains a timestamp for a successful scan of each movie. If the movie folder/file on the disk has the same timestamp, then it's not necessary to run a web query.

I'm generalizing and simplifying, of course, but it seems like there should(?) be a way to add movies to a source and not have to update every single movie in the source to get one new movie to appear in the UI. Does this make sense?

A log file is here: https://dpaste.com//FDQNL7J9H

[Karellen]

around 3700 movies across three sources

Yes, I thought you may have had a larger library.
You have 3 sources. How do you separate the content between them? Are two sources "full" and you only add new media to the third? Or are the sources split alphabetically, or by resolution, or some other means? How do you decide which source you add the new movie to?

I guess I'm unclear on what's happening during the scan that makes it so slow.

Kodi does scan every file and folder in your source. It then keeps a hash to track changes. When you add a new movie, Kodi has no idea where it is, so it searches for it by looking for new folders and files and comparing hashes for changes. Basically you are scanning 3,700+ folders each time you run an library update so Kodi can find the new movies.

Your log does not have debug mode enabled, so there is not enough information for me to see what is happening.
But I do see in that small log there are 147 movies that are not scraped into the library. I don't know why they are not in the library. A new log with Debug mode enabled might reveal why.
So, every time you run an Update Library, Kodi is scanning 3,700 folders and also trying to scrape these 147 movies into the library and failing for some reason. So, yes it will take a long time.

[mrob]

Yes, I thought you may have had a larger library.
You have 3 sources. How do you separate the content between them? Are two sources "full" and you only add new media to the third? Or are the sources split alphabetically, or by resolution, or some other means? How do you decide which source you add the new movie to?

First, thanks for looking at the log file. As you guessed, I separate content between the sources by adding files until the disk is full. I need to upgrade to some larger HDDs.

If there's a better way to organize them that would speed up the scan, I'm all ears, though it could be a lot of work to move all the files around between drives.

Kodi does scan every file and folder in your source. It then keeps a hash to track changes. When you add a new movie, Kodi has no idea where it is, so it searches for it by looking for new folders and files and comparing hashes for changes. Basically you are scanning 3,700+ folders each time you run an library update so Kodi can find the new movies.

OK, though if it's not querying over the Internet, I assume these hash checks are fairly fast, right?

Your log does not have debug mode enabled, so there is not enough information for me to see what is happening.
But I do see in that small log there are 147 movies that are not scraped into the library. I don't know why they are not in the library. A new log with Debug mode enabled might reveal why.

I think what's happening is that a number of these never resolved to TMDB, either because the names/dates don't match, or because there's simply no entry at TMDB. I should probably clean all of these up and see how much of a difference that makes, and if it still seems slow, I'll enable Debug mode and post a log file here to seek further investigation.

One more question, if you don't mind...

Some moves were split into two files, (e.g., Golden_bat_cd1.avi and Golden_bat_cd2.avi). It seems like these are not getting recognized during the Kodi scan.

I tried to fix some of these before, but I couldn't get it to work. How does Kodi parse/recognize split movies? What should the filenames look like?

Thanks again!

[Karellen]

First, thanks for looking at the log file. As you guessed, I separate content between the sources by adding files until the disk is full. I need to upgrade to some larger HDDs.

Great. That is an easy fix. Because they are full, it means you won't be adding any new movies in those sources. Now you can set those two sources to ignore the Library Update.
See image 9a... https://kodi.wiki/view/Adding_video_sources#Set_Content
There is a setting named Exclude path from library updates. Enable this on the two full sources and when you run the Library Update, this source will be ignored. It does not affect your library in any way.

OK, though if it's not querying over the Internet, I assume these hash checks are fairly fast, right?

Yes. Most of the delay is coming from the 147 movies not in your library which Kodi is trying to scrape online.

I think what's happening is that a number of these never resolved to TMDB, either because the names/dates don't match, or because there's simply no entry at TMDB

Yes, I searched 3 of your movies, and they don't exist in TheMovieDB.
I did not check IMDB, but if they exist there, you can scrape them with the Universal Movie Scraper.
You can also check TVDB as they now have movies on their site, and there is a new TVDB movie scraper you can use, if they exist there.

What I would do is move those 147 movies out of your Source, into another source until you sort out what you will do with them. You can then set that Source to ignore library updates, like above.
The 147 movies are listed in your log, so should be easy to find. Search for it won't be added to the library and they will all be highlighted.

Some moves were split into two files, (e.g., Golden_bat_cd1.avi and Golden_bat_cd2.avi). It seems like these are not getting recognized during the Kodi scan.

Split movies need a particular naming as shown here... https://kodi.wiki/view/Naming_video_file...ideo_Files
But cd1 should work. But of course this only works if they are scraped into the library.
If you are using File View, then you need to enable the setting 2.1 here... https://kodi.wiki/view/Settings/Media/Videos#Files

If the above does not solve the problem (and it should), then provide a Debug Log.