[Dudeman]

Does the Library Cleanup function only removes the references from the database, or does it also clear the data?

I recently installed a new Amazon Fire Cube, and was having issues.  Previously I had copied the org.xbmc.kodi folder so I can keep my settings and libraries, but this time I decided to start fresh and scraped all my movies and TV shows, and the total size data folder is now only 1.68GB.  Compare this to 5.4GB that my previous folder was, so I started wondering why this is.  Same number of movies and TV shows, same settings, yet one is nearly 3 times the size.

So after some tests I think I found the reason, and please feel free to correct me if I'm wrong.  I think that when you move some things around in your library, or maybe change source locations, when you go to clean up your library, the program only cleans up references about it, it doesn't physically remove any of the data it stored about those movies, so all the thumbnails, clear art, posters, fan art, etc is still there, it's just no longer showing in the library.

The when you add back the new source and re-scrape, it downloads all new copies of the same thing it did before and adds it to the library and the DB, literally doubling up the space used.

I tested this by removing one of the sources I had setup for 3D movies.  it contained 311 movies with poster, fan art and thumb nail, as well as actors.  Once the cleanup was done, it asked if I wanted to remove the media items, and I removed that..  Checking the library, all the 3D movie information was gone.

I then exited Kodi and started ES Explorer, to check if the data file got any smaller, but no, it actually got a little larger.

Previously the  folder had 32,151 files in 1185 folders, totaling 1,808,329,400 bytes (1.68GB)
After Cleanup the folder had 32,156 files in 1185 folders, totaling 1,808,736,867 bytes, so roughly 400K larger than before.

So there's something wrong with the cleanup routine, or I'm not understanding how this is supposed to work.

[Karellen]

So after some tests I think I found the reason, and please feel free to correct me if I'm wrong.  I think that when you move some things around in your library, or maybe change source locations, when you go to clean up your library, the program only cleans up references about it, it doesn't physically remove any of the data it stored about those movies, so all the thumbnails, clear art, posters, fan art, etc is still there, it's just no longer showing in the library.

The when you add back the new source and re-scrape, it downloads all new copies of the same thing it did before and adds it to the library and the DB, literally doubling up the space used.

That is correct.
It is not considered broken or a bug. It is not urgent or critical. But it is something that could do with some improvement, but no developer has volunteered to improve this area of Kodi.
I think this is one of those improvements that needs to come from the community.

[DarrenHill]

The database cleaner addon may also be worth a look, although it's still in beta and of course as it directly works with your database it comes with its own risks (backups are highly recommended!).

[Dudeman]

"It is not considered broken or a bug. It is not urgent or critical".

While I agree that this may not be a "critical" or "urgent" issue in the PC, Mac or Linux world, but when it comes to devices with limited storage like Android or FireCube or TV Sticks, it should be considered a critical issue. 

Unless you can, and are willing to move the data to an external storage and take the performance hit, you will be limited to the size of the database you can have on your device. And then one day you need to make chnages to your sources, and fid out that you don't have enough space to scrape your library, because the old data was never removed.

I wish someone in the Dev team would reconsider this and fix the database cleanup function.

[Dudeman]

The database cleaner addon may also be worth a look, although it's still in beta and of course as it directly works with your database it comes with its own risks (backups are highly recommended!).

Thank you I'll give this a try..

[Klojum]

you will be limited to the size of the database you can have on your device

The size of the database is really not the problem. It only contains data, it doesn't contain "blobs" with large images or fan art. A typical Kodi database would be upto 300-400 MB in size, which is smaller than a single 4K movie background poster. If you are talking storage capacity, you'll need to look at the thumbnail cache. That's the big one.

[Dudeman]

I knew this, but when i was referring to "database" IO meant the DATA folder (org.xbmc.kodi).

Anyway, I do have a question regarding thumbnails.  When you look at the folder, there are basically a bunch of folders named 0-9 and a-f, each one with roughly 2500 files.  How can you tell which file belongs to what.  Moreover, they are not grouped in anyway, actors, clear art, posters, fan art, they are all mixed randomly.  You would think that files belonging to the same movie would be named sequentially or something, but no such luck.

[Karellen]

You would think that files belonging to the same movie would be named sequentially or something, but no such luck.

https://medium.com/eonian-technologies/f...bb03aa4091

[Dudeman]

You would think that files belonging to the same movie would be named sequentially or something, but no such luck.

https://medium.com/eonian-technologies/f...bb03aa4091

OK, so for the sake of efficiency and performance, we sacrificed readability. 

Which makes the process of removal of unwanted or unused files all but impossible since it's impossible (at least to mere mortals like me) to determine what is used and what is redundant.  Because even when I find the same images repeated over and over, there's no way to tell which one is current and which is not.

[Karellen]

OK, so for the sake of efficiency and performance, we sacrificed readability. 

No, nothing is sacrificed. You are trying to do something that is not meant to be done.

If you want to clear out your thumbnails, then delete the entire \thumbnails\ folder and also delete the Textures database. Kodi will rebuild it. Be warned, if you do not have local artwork and rely on scraped links, many images may remain missing as they have been deleted from the scraper sites.

[Hitcher]

You could always use this -

Texture Cache Maintenance utility

[Klojum]

@Hitcher

I've updated your link, as the original TextureCache Tool is not Matrix/v19 compatible.
Also the Video Database Cleaner tool has an updated version onboard.

[Hitcher]

Thanks, I didn't realise.