v19 Library cleaning takes FOREVER
#1
Yes i do have a very very large library but the cleaning process is longer than with 18.x versions. I am using MySQL for the DB

What has really made it worse now is that it seems that it cleans the WHOLE library even when you update a single show. 

I have the <cleanonupdate> setting to true in my advancedsettings.xml and in versions prior to 19.x, when i updated a single show, the cleaning process was way less than 1 minute.

Now, it takes a full 10 minutes to clean a 6 episode show. That makes no sense.

My theory is that it is doing a full clean instead of just the specific show being updated. In the log, you can see it complaining about 2 show folders that haven't existed in months. Those are not folders for the one show i am updating so why is it trying to clean shows that don't even exist anymore?

The cleaning process goes to 99% in about 5 seconds and then stays at 99% for another almost 10 minutes.

My advancedsettings.xml:
xml:

    <videolibrary>
        <hideallitems>true</hideallitems>
        <recentlyaddeditems>300</recentlyaddeditems>
        <cleanonupdate>true</cleanonupdate>
        <dateadded>1</dateadded>
    </videolibrary>

Relevant parts of the log:
Code:

2022-04-15 14:31:57.628 T:20356    INFO <general>: CVideoDatabase::CleanDatabase: Starting videodatabase cleanup ..
2022-04-15 14:31:57.724 T:20356   ERROR <general>: XFILE::CDirectory::GetDirectory - Error getting smb://CNS-TVSERVER/TV Shows 02/30 Days/Season 1/
2022-04-15 14:31:57.731 T:20356   ERROR <general>: XFILE::CDirectory::GetDirectory - Error getting smb://CNS-TVSERVER/TV Shows 02/30 Days/Season 2/
2022-04-15 14:31:57.740 T:20356   ERROR <general>: XFILE::CDirectory::GetDirectory - Error getting smb://CNS-TVSERVER/TV Shows 02/30 Days/Season 3/
2022-04-15 14:31:57.744 T:20356   ERROR <general>: XFILE::CDirectory::GetDirectory - Error getting smb://CNS-TVSERVER/TV Shows 02/According to Jim/Season 1/
2022-04-15 14:35:46.059 T:12196    INFO <general>: TAG-GEN: Starting IMDb tag writing.
2022-04-15 14:35:46.069 T:11228    INFO <general>: CActiveAESink::OpenSink - initialize sink
2022-04-15 14:41:37.477 T:20356    INFO <general>: CVideoDatabase::CleanDatabase: Cleaning videodatabase done. Operation took 09:39
2022-04-15 14:41:37.483 T:20356    INFO <general>: VideoInfoScanner: Finished scan. Scanning for video info took 09:55

Full log: https://paste.kodi.tv/zapinozima.kodi

Before you ask for the full debug log, i don't see the point of it just listing all my TV Shows it's trying to clean. Clearly it is not cleaning that 1 show but all my shows. But if you absolutely need it, i will find a small show and rename 1 file then run the update...
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#2
You need to enable debug mode.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#3
(2022-04-15, 21:13)Pr.Sinister Wrote: Now, it takes a full 10 minutes to clean a 6 episode show. That makes no sense.

A cleanup always involves the full video library. For example, it needs to be determined if any videos have been deleted manually from the video source(s) without using Kodi, like plenty of people do.

FYI, a cleanonupdate is really not necessary. If you remove your videos properly via the Kodi GUI, then most of the cleaning for that movie/tvshow will already be done.
Reply
#4
(2022-04-15, 21:50)Klojum Wrote: A cleanup always involves the full video library.

It was not always like this. If i say just scan this 1 TV Show, it used to clean just that one show. If i tell Kodi to update the whole library, then of course you clean the whole library. If i say just update this one show, then what is the logic in cleaning the whole library?

If you look at the source code prior to 19.x, i am sure you will see it wasn't like that. For 17.x, i am 10000% it used to clean only that show. If you want to point me to the place in the source code, i will run a diff on 19.x and 17.x and i am sure it will show 17.x was doing just the show. I am pretty sure it was also the case for 18.x but i don't want to speak in absolutes if i am not sure. For 17.x, i am sure.
 
(2022-04-15, 21:50)Klojum Wrote: FYI, a cleanonupdate is really not necessary. If you remove your videos properly via the Kodi GUI, then most of the cleaning for that movie/tvshow will already be done.

That is not practical for people with huge libraries and replacing 3 seasons of 24 episodes for a show that has 8 seasons. I am using Sonarr to auto-download stuff as it comes out but once a season is complete or said season is available on Amazon Prime or HBO Max, i then use StreamFab to download the H.265 version directly from AP or HMAX. I then go to the folders and delete all the episodes from the disk, move the new ones to the folders, then have Sonarr regenerate the NFOs, rename the files and set the mtime to the aired date. Sonarr then sends Kodi a command to update that single show. It used to take less than a minute but now it takes 10 minutes. I have 1.5gbps connection so it very often happens that Sonarr will download an episode of show X, send the update command to Kodi, then 2 minutes later, downloads show Y and also sends the update command to Kodi but only show X appears in the library because Kodi was busy cleaning for 10 minutes and the Update commands are not enqueued by Kodi.

I have cleanonupdate set to true and Update library on startup because once i replace a bunch of Movies to the better version (in this case, H.265), i close Kodi and reopen it and it takes care of updating the associated movie file and not the artwork or anything else. 

It is also needed because of the issue above about 2 or more Update Commands being sent while Kodi is cleaning.
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#5
(2022-04-15, 21:24)Karellen Wrote: You need to enable debug mode.

https://paste.kodi.tv/efesebexuw.kodi

Scan of single show starts at line 2510 and takes 11 seconds.
Cleanup starts at line 2536 and takes 9 minutes and 48 seconds. Even if it is intended to be this way, it is woefully inefficient 

Code:

2022-04-15 16:51:49.536 T:10776    INFO <general>: VideoInfoScanner: Starting scan ..
<snip>
2022-04-15 16:52:00.031 T:10776    INFO <general>: CVideoDatabase::CleanDatabase: Starting videodatabase cleanup ..
<snip>
2022-04-15 17:01:48.246 T:10776    INFO <general>: CVideoDatabase::CleanDatabase: Cleaning videodatabase done. Operation took 09:48
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#6
I'm not a C programmer, but looking at the changes in PR 19565 it does seem like maybe the logic was changed on how video library is cleaned if JSON RPC VideoLibrary.Clean is called (assume that is what SONARR is doing?). The purpose of the PR was to move methods out of the main application code, but it seems like there was a logic change made as well in how a bool userInitiated is used in builtins/LibraryBuiltins CleanLibrary() If I get some time I might try hacking the code a bit to confirm or disprove my theory.

scott s.
.
maintainer of skin  Aeon MQ5 mods for post-Gotham Kodi releases:
Matrix see: Aeon MQ5 Mod Matrix release thread
Nexus see: Aeon MQ5 Mod Nexus release thread
Aeon MQ 5 skin and addon repo 11.1.0
Reply
#7
(2022-04-16, 00:57)scott967 Wrote: I'm not a C programmer, but looking at the changes in PR 19565 it does seem like maybe the logic was changed on how video library is cleaned if JSON RPC VideoLibrary.Clean is called (assume that is what SONARR is doing?). The purpose of the PR was to move methods out of the main application code, but it seems like there was a logic change made as well in how a bool userInitiated is used in builtins/LibraryBuiltins CleanLibrary() If I get some time I might try hacking the code a bit to confirm or disprove my theory.

scott s.
.

Thanks scott!

In the meantime, i just installed 18.9 and am loading the library to test. Because of the size, it will only be done by Sunday morning Sad
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#8
FWIW, cleaning libraries can take longer if inaccessible network/external sources are in play, and timeouts will add to the total cleanup time.

Also, 18.9 will not really improve on the actual cleaning part. Database triggers in Kodi will currently only remove a small portion of all the interlinked info in the database. I don't know how far things are in Kodi 19 or 20 (I'm not part of the dev team), but from a bit of reverse engineering in my own home-grown database maintenance, I know that removing an entire TV show can affect upto 18 tables in the video database. Something similar is the case with movies.

So it's not just the files, movie or tvshow database tables that require a scrub. I currently have a Kodi database with only some 200 tvshows (no movies), and the actor_link table holds almost 144,500 links in them, with every actor linked to every TV episode he/she is in. That table isn't touched by Kodi when doing a Clean operation. And overall, that's okay. All that (old) data is not interfering as database engines are very effective. But logic would suggest to remove all outdated data from all tables. The full logic in cleaning has not reached the Kodi core yet.

Triggering a full clean per TV show could be done when deleting a movie/tvshow from the Kodi GUI directly. When movies/tvshows suddenly disappear from the video sources 'by some external force', that will involve even a bit more effort to properly clean the video database.
Reply
#9
You can save a bit of time updating; individual source update is possible along with updating a single show that is in focus. Kodi cleaning for 10 minutes is not FOREVER. Large libraries are prone to corruption easier than a small one.
Reply
#10
(2022-04-16, 18:14)PatK Wrote: You can save a bit of time updating; individual source update is possible along with updating a single show that is in focus. Kodi cleaning for 10 minutes is not FOREVER. Large libraries are prone to corruption easier than a small one.

That is exactly what i said. And when i say forever, i mean for a SINGLE SHOW.

If i update 10,566 Movies and 1860 TV Shows (86,163 Episodes), cleaning takes 10 minutes
If i update a single show with 6 episodes, cleaning takes 10 minutes.

How does that make sense? I am not complaining about a full update/cleaning. Just the single show one.

And BTW, i renamed myvideos119 to myvideos107 and used Kodi 17 to clean it. Took 6 Minutes. Renamed it back to 119 and cleaned it, 10 minutes.

So Kodi 19 is 40% slower at cleaning. And it's actually no cleaning anything because it's already clean. I understand that Kodi 17 may have less to clean because it has less columns to worry about but 40%?

Anyway, another user has suggested i take a look Mezzmo as the backend server instead of Kodi.

I really love Kodi and been using it since back when it was called XMBP (before they merged into XBMC) so i don't really want to but if it does what i want, i guess i'll have to.

Or i could just go the extreme route, learn C++ and fork a version that rethinks the way the library works  Laugh
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#11
(2022-04-16, 20:21)Pr.Sinister Wrote: If i update a single show with 6 episodes, cleaning takes 10 minutes.
As per your log snippet in the opening post, 4 season folders that cannot be found via SMB for some reason, leading to in total 10 minutes of timeouts.

Try adding the following to your advancedsettings.xml file.
xml:

<samba>
<clienttimeout>10</clienttimeout>
</samba>
and see if it makes a difference.
Reply
#12
(2022-04-16, 20:34)Klojum Wrote:
(2022-04-16, 20:21)Pr.Sinister Wrote: If i update a single show with 6 episodes, cleaning takes 10 minutes.
As per your log snippet in the opening post, 4 season folders that cannot be found via SMB for some reason, leading to in total 10 minutes of timeouts.

Try adding the following to your advancedsettings.xml file.
xml:

<samba>
<clienttimeout>10</clienttimeout>
</samba>
and see if it makes a difference.

Thanks i'll try that. Can you tell me where it would be looking for these in the db so i can manually delete them? Those shows were deleted from the hard drive so i can understand it complaining the 1st time i run a clean but it's been months and i did a full refresh of the TV Shows library 3 weeks ago in a quest to speed up the update/cleaning process
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#13
I am kinda baffled...

sql:
SELECT * FROM myvideos119.tvshow_view where c00 = 'According%';

Returns nothing. Same with

sql:
SELECT * FROM myvideos119.tvshow_view where c00 = '30 Days%';
[4 Kodi Clients + 4 Norco RPC-4224 Media Servers w/376 TB HDD Space]
Reply
#14
Try searching in only the path table. A view is mostly a combination of fields from multiple tables which may obstruct the full outcome for these searches.
Also use % at both ends of the c00 field.

sql:
SELECT * FROM myvideos119.path where strPath LIKE '%30 Days%';
Reply
#15
https://forum.kodi.tv/showthread.php?tid=339156

https://github.com/the-black-eagle/scrip...se.cleaner
HOW TO - Kodi 2D - 3D - UHD (4k) HDR Guide Internal & External Players iso menus
DIY HOME THEATER WIND EFFECT

W11 Pro 24H2 MPC-BE\HC madVR KODI 22 GTX960-4GB/RGB 4:4:4/Desktop 60Hz 8bit Video Matched Refresh rates 23,24,50,60Hz 8/10/12bit/Samsung 82" Q90R Denon S720W
Reply

Logout Mark Read Team Forum Stats Members Help
Library cleaning takes FOREVER0