[moviebuff1017]

Hello everyone,

First of all, sorry for the newbie question. My apologies if it has been asked before - I could find similar issues, but not quite identical, and no hints for a solution. I have a large collection of 22.000 rare movies on my NAS, mostly ripped by myself from my huge VHS collection. I was thinking of using Kodi for a nice GUI, both on my Windows 7 64-bit machine (and on my Raspberry Pi 3B+ - but this is not the subject of the present post). So I started experimenting, created almost 20.000 NFO files containing the imdb links to help the universal movie scraper and started scraping with Kodi v17.6 

Everything went smoothly, and it took about 3-4 days to scrape everything. I lost network connection a couple of times,  but scraping restarted normally.

For the first couple of days, everything went great. But now it has become completely unstable. I can start KODI but it usually crashes after about one minute, whatever I'm doing with KODI at the time (or even if I don't do anything). Once in a while, it starts correctly and can use it for a couple of hours.

The movie107.db database is about 320 MB. Textures13.db is 26 MB.  I disabled the library updates from automatically starting, but it still crashes.

Should it be stable with such a large collection? At the time of the crash, memory usage is about 1GB (but I still have plenty of RAM left on my 8 GB machine).

Here's a log when Kodi crashed after a minute without my doing anything with it - https://pastebin.com/HnTYY5sG

Here's a log when Kodi crashed after I used the gUI for a short while - https://pastebin.com/KPxRXW1W

It's clearly related to the content of the database - if I move the content of the database folder out of the way and restart a "clean" Kodi, it works perfectly again.

I made a couple of (newbie) hypotheses - for what they're worth: 
1. Memory allocation and out-of-memory errors
In the second log, there are memory allocation issues, but not in the first. I suspected a memory issue, but then I carefully inspected memory usage when starting Kodi and waiting for it to crash : memory footprint increases quickly up to 1.5 GB, then decreases to about 300 MB, then crashes.

2. Something wrong inside the database
I made a copy of the textures and movie databases and explored it more or less thoroughly with sqlitestudio - seems ok. Maybe something with the thumbnail cache, though. Maybe some art cannot be pulled from cache, or is corrupted. Should I try and clean the thumbnail cache completely and wait for it to be rebuilt?


Thanks for your help

[Karellen]

Hello @moviebuff1017

Interesting problem. Unfortunately you have not enabled debug mode, so there is no real useful information to look at. Your second log shows a few of the following lines...

21:10:05.268 T:1900 ERROR: CBaseTexture::Allocate - Could not allocate 16777216 bytes. Out of memory.

Can you...

1. recreate the logs using this guide https://forum.kodi.tv/showthread.php?tid=323719
2. provide details of your hardware running kodi, and details of your network and nas if you are using them.

[PatK]

"Should it be stable with such a large collection? " Ahm; There are larger.

" 2. Something wrong inside the database " Likely; proper debug log might offer a clue. Investigate https://kodi.wiki/view/Texture_Cache_Mai...ce_utility

[moviebuff1017]

Thank you very much for your help.
@Karellen : hardware is a Lenovo S540 i5 laptop with 8GB Ram running at 1.6 Ghz under windows 7 SP1, and some kind of Intel HD Graphics Family card (can't find any details on that one in device manager). Video files are accessed through SMB from a Synology 12-bay DS2415+ running DSM 6.1.4 through a gigabit ethernet network (using a mini-switch, no devolo or anything fishy between laptop and NAS) 

I tried fiddling with the SQLite database (after making a copy of it, with Kodi and sqlitestudio closed, obviously) , deleting all art and thumbs I could find in the video107.db database  (delete from art ; update movie set C08='', c20='' ; ... ) , then VACUUMing it... and after that, Kodi ran smoothly for two hours (without any pictures, obviously, except i few I probably left out or where in some other cache) before I exited it manually.

But when I restored back the database with the art, it first started ok, I could browse the movies somewhat. I closed Kodi, reopened it, then it crashed again. So, following your advice, I had set the loglevel to 2 in advancedsettings.xml. So here are the logs from that new crash :  https://pastebin.com/Msd7Zcia

@PatK , I will also try and check the texture cache maintenance utility that you pointed to ( https://kodi.wiki/view/Texture_Cache_Mai...ce_utility ) ... but it seems it requires Kodi to be running (since it requires Kodi's webserver to be enabled on port 8080, I guess it accesses the database and/or cache through Kodi itself). Moreover, I understand how that tool can be used to deal with one problematic thumbnail or art that you know is corrupted ; but I don't understand how it can be used to identify all corrupted art. Is it through the "C" option (automatically re-cache missing artwork, with option to force download of existing artwork (remove first, then re-cache)). Can you please help?

Thanks again

[Karellen]

@moviebuff1017

I was expecting to see the failure during a caching process, but I see nothing out of the ordinary in your log

If you want to clear cached images from your system, this is the method... https://kodi.wiki/view/Artwork#Delete_textures13.db

If you choose to clear cache using the above method, then please capture a debug log, when you restart Kodi and it re-caches images.

Might want to try PatK's suggestion first though.

[PatK]

I would try and use the 'P' function of the Texture Cache utility to clean out some of the cruff, and proceed to recache. But if Kodi will no longer hold for the time it takes, then we'll put that aside for now. I do note you have some skins available, are you using default? (if not, check that the skin is updated to work with 17.6) and the "Could not allocate 16777216 bytes. Out of memory" does seem to be ominous. You can definately try clear and rebuild on the chance there is a corrupt files. Do you export your custom .nfo, and do they reflect the file? This may all come down to baby sitting a scraper, and in that case have a look a TMM (Tiny Media Manager).

As an aside: You could entertain multiple sources each with their own Kodi in portable mode.

[afedchin]

My suggestion is install x64 version. Unfortunately there is only v18-x64 version of Kodi.

21:10:05.268 T:1900 ERROR: CBaseTexture::Allocate - Could not allocate 16777216 bytes. Out of memory.

means that system cannot allocate memory for Kodi process, it causes what some internals (DirectX and other system call) also may be affected by OOM and this may cause the crash.

Looks like there's the issue with memory management in Kodi with large libraries.

[Karellen]

Thanks @afedchin

[moviebuff1017]

@afedchin  :  I will install the 64-bit version of Kodi V18 asap.

@PatK  : Up to now, I've only used scraping NFOs because - for most of my files - I already have a db that links the filepath with the imdb-id. I don't understand what you mean by "do you export your custom .nfo, and do they reflect the file?" 
If upgrading to 64 bit doesnt fix the issue, I will first empty the art cache and db as you (and @Karellen ) suggested and wait for it to be rebuilt. If it still doesn't work, I'll just try and use other methods of scraping (like generating full movie nfos from my own database ; possibly tinymediamanager after I study it) to provide custom nfos with a smaller number of thumbs (for instance: a small number of posters per movie, just one picture per actor). I guess this might reduce the overall size (database, cache, ...)  enough so Kodi can reliably run, without running a different instance per source.

Many thanks for your replies.

[PatK]

It wouldn't be the first time that I've seen corrupt thumbs, gum up the works, at one point in the past, I had some issues with posters showing up and the system bogged down, after a visual inspection of the thumb folder, I found corrupt images and manaully deleted the offenders as the solve.  Since that time I found Texture Cache utility invaluable to keep a tight slim library and the gfx under control. Weighing your laptop down with multiple skins, extra art work, add-ons and fragmentizing lib might not be a good way to keep a sprightly set-up. From my experience Kodi set-up cleanly on a desktop with suffient ram and speed should be able to handle large libraries layed out with multiple sources ( I have seen a few examples ). I haven't explored the upper limits of the library yet, and there may indeed be issues in these uncharted waters, but the option of segregation into multiples, work-rounds such as TV shows on an Android device, and Movies on the Laptop come to mind.

[Edit] to add some personal investigation.

For my own interest, I created a library of 14,000 movies and an equal amout of TV episodes (mostly duplicates, created with chanigng drive letters, scraped with a local information scraper). I played out a film, and utilized the interface long enough to see stability, the skin was Transparency! and had numerous typical system add-ons. Test was done with a windows 7/64 o/s 12 gigs of memory, 10 available on boot, 9 available after Kodi load up, graphic engine had 1 gig available and on Kodi load up, still had the majority available (100 megs used while playing). The interface was spiffy, and didn't seem to slow down as I built the library from 2K to 14K.

Local scrapes took ~10 minutes for every 2 gigs. I suppose I should have built this library to fail point, but I ran out of drive letters!

[moviebuff1017]

Thanks... I'm unable to devote a lot of time to my Kodi issues now... on my windows install, I just removed all thumbs and it seems to be stable. Meanwhile, I'm rebuilding a library on my Raspberry Pi 3+ . At some point, I'll have to fine tune with local NFOs anyway.

What I'll do now over the next few months is build the "functionally largest" KODI which will be stable with my collection, using various scrapers and coding a little to build to best possible NFOs.... Thumbnails/art are not my priority, so if I have to cut down on that, it's ok for me (for instance, one thumb per actor, one art per movie). I'll keep you posted but it'll take some time

@PatK : if thumbnails are the issue, I'm not sure your simulation of a very large library is adequate - the number of movie items will be large, but the actors will be the same, and the thumbnails (for the actors and so forth) will point to the same URLs . I guess they'll then have the same hash in the textures db (if I check the "texture" table in the texture13db, key sames to be the url, so I guess two identical urls won't be repeated) -> if this is correct, the number of thumbs/art will be much lower with - say - 3500 movies each duplicated 4 times than with 14.000 actually different movies, won't it ?

[PatK]

Alas, I don't collect Actor images locally, I have other systems for looking into actors bio etc. I tried to replicate your issue to some degree, as well as looking at future issues of 'largess' collections. When I added movies at 2K each run, subsequent library grew in size and the thumbnail directory became larger to accomodate as they don't use the same hash. The URL links in the meta-data grows as needed (check out thumbnail folders in icon mode and you'll see what I'm talking about). I'm statisfied that Kodi can handle large 14K x 2 with TV shows included collections, and for me the 'large' part of the equation has yet to be reached on the system as posted previously, but as adfetchin ponited out there could be an issue with memory limitations, I beleive you're running out of a laptop?

Suggestion: Use a 3rd party media manager like TMM for organization, clean up using Pre-wash https://forum.kodi.tv/showthread.php?tid=272112  (use texturecache) scrape .nfo locally and sequentially. Don't put multiple sources to-gether for the scrape. My suspicion is that you have unorgainzed 'shotgun' library, with files scattered willy nilly and in that are some corruptions whihc are causing some run away memory issues.