[bzowk]

Hey Guys - 

I am a long time Kodi/XBMC user who has a couple of large issues maintaining a large library which includes almost 30,000 episodes (~900 shows), over 5,000 movies, and some other misc media.  My hope in posting is that someone may be able to examine the details of my environment and procedures then be able to suggest alternative tools, addons, or methods for how to maintain and update it which will potentially resolve the issues and ideally make the whole system more efficient.  Below are specifications for my environment's setup, procedures I use to add media and update my libraries, and details about the two issue I am having.  If you have any suggestions, I'd love to hear what they are!

Environmental Specs
Everything below is on the same LAN
Primary Workstation
  - Windows 10 x64 / i7-3770k / 32gb RAM
  - Use for everything you can think of including gaming, work, Oculus Rift, Plex Media Server, and file repository
  - It stores all media on its D: partition is 43tb consisting of 12 (11 platter + 1 SSD for caching) disks in a disk pool (Stablebit DrivePool)
  - On the disk pool, all media shares one of four common base paths:
      - D:\Media\Movies
      - D:\Media\Movies - New
      - D:\Media\Movies - Collections
      - D:\Media\TV Shows
      "D:\Media\Movies - New" is where newly acquired movies go.  They are moved out of this folder and into "D:\Media\Movies" after x days via a PowerShell script I have run nightly...
VMWare ESXi Host
  - Ubunutu 16.04 VM running Kodi 17.6 dedicated to updating Kodi Library as described above (x2 CPU / 4gb RAM)
     I soon plan to replace the above VM with a Docker container that runs Kodi headless (once I figure out how to schedule updating and add Trakt addon)
  - Atomic Docker Host VM (Fedora 28) which includes a mySQL Server 5.7 container.  The entire container / SQL Server is dedicated to the Kodi library
Players 
  - Dedicated PC connected to living room TV running Kodi 17.6 on Windows 10 x64
  - x3 Raspberry Pis running OSMC (version equivalent to 17.6)
 
How I Currently Implement New Media
Obviously TV Shows and Movies are different, but not much.
TV Episodes
1. Obtain or rip physical media resulting in a digital file
2. Sonarr imports it, tags it, renames it, then moves it into a library folder
3. Approximately every couple of days an episode level scan is performed on all shows with Media Center Master to add metadata Sonarr doesn't add
Movies
1. Obtain or rip physical media resulting in a digital file
2a. For about 70%, I manually move it into a library folder
2b. For about 30%, CouchPotato imports it, tags it, renames it, then moves it into a library folder
3. At least once a day (or after each manual move) I scan new titles with MCM to tag, rename, and clean them

My Main Issues
The two main issues I am currently experiencing are with Kodi Library updating and crashing of the tool I use to organize the media.  Details for each are below
Tagging & Renaming Tools
For the longest time, I've used Media Center Master to keep metadata and naming up to date for both TV and Movies.  Unfortunately, the bigger my libraries have gotten; the more MCM crashes - to the point where 90% of the time I invest updating metadata is ending task on MCM, restarting the app, and waiting for it to initialize (if it does properly).  I've posted many times on their forum, but answers are vague and don't help.  I recently downloaded the latest build of Ember Media Manager Alpha (1.4.8.0) which I configured and added my media to and am having a few minor issues with it that I'm trying to work out.  I've used TinyMediaManager in the past as well with unstable results.  What's the suggested one for libraries like mine?
Kodi Library Updating / Addons
As you can see below, I use a mySQL server for my Kodi Library and have a dedicated Linux VM just to update it hourly.  This is done by running Kodi in an Ubunutu 16.04 VM which has the Library Auto Updater and Trakt addons installed.  It's configured to update my video library hourly and once done clean it then sync with Trakt.  When everything works well, an update takes 20-25 minutes.  I'm sure it would be faster if the update and DB were on the same system as the media files, but updates kill my CPU when being performed which is why I have them separated.  I've tried Watchdog in the past when I still had a large library, but had major issues with it.  I don't recall the details.  What would be the suggested method to keep my type of library up to date?

Notes & Some Things I've Already Tried
- Soon I'll probably being use Kodi for only my dedicated Kodi system in the living room and covert the x3 RiPis to RasPlex.  I've considered going to Plex 100% but it's client isn't close to as powerful as Kodi is and on that living room PC I use a good amount of addons for things like Steam, automation, plus a theme that's easy for the wife to use it.  When doing this I could migrate from mySQL back to a standard, local DB on that single system however it's currently connected via wifi so updating would be slower unless I run cable
- I've tried the Plex addon for Kodi as you'd think that would be a great solution, but it caused havoc with my Kodi database.  I don't remember the details, but it would cause my Plex Server to trigger library rescanning nonstop thus using tons of resources.  After tons of troubleshooting, I was finally told in a forum that it was the blame one once I removed the addon and recreated my mySQL database, all was good.  I don't know if it would affect a standard Kodi db as I could go to that soon
- Instead of CouchPotato, I've tried both Radarr and Watcher.  I really wanted to use Radarr since it shares a lot of source code with Sonarr and like it so much.  However, they both had difficulties managing libraries of this size and in the end I reverted back to CouchPotato which works much better

Whew - I think that covers it!  Any suggestions?  Thanks, Guys!

[Karellen]

There seems to be a lot going on there, but some general comments...

Unfortunately, the bigger my libraries have gotten; the more MCM crashes

Not something we can help you with. If you ask pertinent questions in the Ember or TMM sub-forums, I am sure you will receive assistance.

Is it critical that you use a Media Manager? Why not let Kodi scrape the video files?

It's configured to update my video library hourly and once done clean it then sync with Trakt.

Hourly? Really? Are you running a Netflix type service? And if you are obtaining that many files in an hour, then there are obvious questions of legality and our stance on piracy.

It is a little bit worrisome that you have 30,000 episodes in one Source, and 5000 movies spread across two main sources. (I assume Movies-new is temporary holding). It would be prudent to split this into numerous Sources. One main reason is that you can then update the active Source individually instead of scanning the entire library each time. See this page for the various methods of updating the library... https://kodi.wiki/view/Updating_or_removing_videos

[fswings]

In terms of scraping files you could try filebot, especially in command line mode running off your Ubuntu VM. I do this myself because I like having the resultant metadata files for non-kodi uses or clients.

As the for hourly updates and it taking 25 mins, that in my experience is more to do with syncing to Trackt. How many items are we taking about being added? What is the run time for non-trackt usage? What are you using Trackt sync for? Why not do it once a day?

I haven't quite absorbed all of what you've said, but overall I found filebot in Linux quite effective.

[bpak]

Hello,

Thanks for bringing this interesting topic. Although my video library is not as large as yours, I may face similar issues.
I've run some tests and am currently working on a solution, below are my 2 cents:

Objective: a single + sustainable + solution-agnostic source of metadata:

• Whatever the end-user media center solution is (Kodi, Plex, Emby, ...), video information should be the same on any device, at any time
  
• Scrapping is a one-time operation, and retrieved information must be kept recorded once and for all (unless you decide to refresh it of course)
  
• Manual (e.g. tags, user rating, ...) and automatic (e.g. watched status) updates should be reflected on all HTPC devices
  

My current approach:

• Use .nfo files (one per video file):
  • As an alternative, you could rely on a main Kodi / Plex / Emby database as your single source of data, and use connectors (such as the ones for Emby) to keep your devices up-to-date. I must admit I have only given a little try on those, but I'm quite reluctant on using such solutions, as they will probably never prove to be 100% compatible. I anticipate much trouble with that approach...
    
  • Another alternative would be to use tags inside the video files themselves. That is a kind of Holy Grail for sure, but it does not seem to be very mature at this time of writing. And anyway not all types of video files will allow you to embed tags, so once again much trouble in there...
    
  • On the opposite, NFOs are simply XML files, that are compatible with any of the media center solutions. The XML structure used by Kodi / Plex / Emby seems to be approximately the same, so you wouldn't get into much trouble I think.
    
  • Obviously, importing .nfo files in kodi is a very trivial and fast approach, which does not require any configuration on client side
    
  • .nfo files are kept along with video ones. I like this, as it means I can move the files (video + nfo) without losing the link between my media and its related metadata
    
• Find a suitable supplementary tool to generate / update the .nfo files:
  • Haven't found any perfect solution for now... I've given a try with some of them, and currently using Media Companion. Not bad, but pretty complex, and I don't know how it will behave with a 1000+ video library...
    
  • Or you could simply use one Kodi instance to perform the initial scrapping, and export your library to multiple .nfo files, of course. Here my concern is that Kodi is not designed to manage further manual updates, for example adding tags manually is a pain...
    
  • Anyway I don't consider this a primary concern: of course I need a proper initial scrapping, and this tool should get me that. But once this is done, and I'm satisfied with content of the .nfo file, I don't need to modify it by myself further more. So even if the tool is not able to properly handle a large quantity of videos, I should be able to use it for only small chunks of my library
    
• Keep the .nfo files AND the kodi databases up-to-date:
  • That is the tricky part! By definition, Kodi will use .nfo files only for the initial import of information. Should the .nfo file be updated afterwards, Kodi will ignore it, unless you manually refresh the single library entry, or do a full regeneration of the library
    
  • I couldn't find any addon to serve this purpose, so I'm currently developing a service addon to do this. Basically it will automatically refresh the Kodi video library when .nfo files are updated, and will automatically update the .nfo file when a library entry is modified (e.g. user rating, watched status)
    
  • This will rely on the excellent texturecache.py utility for the first part, and on specific code triggered by Kodi events for the second.
    

I would be very interested to get your feedbacks on this guys, especially on the need to develop a new addon for this. I made some research, and could not find any kind of well-integrated + all-in-one addon to perform this kind of "nfo synchronization" activities.
Any suggestion someone?

[bobrap]

@bpak  Your idea for an addon sounds like a great idea.  There is one addon available that updates the nfo watched state, but, it is no longer being maintained.  Looking forward to seeing your addon when you release it.  Any time frame?

[bpak]

yes, I also took a look to this addon, too, and it seems fairly simple to integrate (or rather redevelop from scratch).
Basically, my idea is simply to merge the behaviour of this addon with the one of texturecache.py QA activities.
And maybe extend by keeping other tags up-to-date (user rating typically).

ETA should be 1 week or 2 I guess, for an alpha version covering only movies first.
I'll keep you updated if you're interested.

[bobrap]

yes, I also took a look to this addon, too, and it seems fairly simple to integrate (or rather redevelop from scratch).
Basically, my idea is simply to merge the behaviour of this addon with the one of texturecache.py QA activities.
And maybe extend by keeping other tags up-to-date (user rating typically).

ETA should be 1 week or 2 I guess, for an alpha version covering only movies first.
I'll keep you updated if you're interested.

 Absolutely, thanks.

[bry]

@bzowk you might want to look into using something like Plex or Emby as your back-end for a large library like that. I personally use Emby (Premier) to manage my back-end library (not nearly as large as yours). This basically replaces metadata management and MySQL db. also allows for syncing library and watched status across multiple devices/platforms. not the solution but an alternative.

[__B__]

That is the tricky part! By definition, Kodi will use .nfo files only for the initial import of information. Should the .nfo file be updated afterwards, Kodi will ignore it, unless you manually refresh the single library entry, or do a full regeneration of the library

@bpak 

This is really tricky because you need to define if this synchronization is two-way or one-way. Should the tags in nfo be appended to the tags in Kodi? Should everything in Kodi be purged in favor of the values in the nfo? Should the tags added in Kodi be exported to the nfo file? You ask the user first? You show a comparative on the interface and the user decides on the fly which way s/he wants?

I think that's why Kodi developers don't try to mess with this.

[Ruk12345]

“Keep the .nfo files AND the kodi databases up-to-date:
That is the tricky part! By definition, Kodi will use .nfo files only for the initial import of information. Should the .nfo file be updated afterwards, Kodi will ignore it, unless you manually refresh the single library entry, or do a full regeneration of the library
I couldn't find any addon to serve this purpose, so I'm currently developing a service addon to do this. Basically it will automatically refresh the Kodi video library when .nfo files are updated, and will automatically update the .nfo file when a library entry is modified (e.g. user rating, watched status)
This will rely on the excellent texturecache.py utility for the first part, and on specific code triggered by Kodi events for the second.”



I would love an easy to use addon for this!

[whitebelly]

I use MCM but have it scrape my new movies in a working folder, then I move the resulting folder to my movie directory. KODI set to local info only.

[Powerhouse]

The original poster never responded back, but for those others interested in this, here is what I do with my large library (3900+ Movies, 720+ TV Shows, containing over 50,000 episodes, a large Music collection with 2,000+ artists, and 4000+ Music Videos).

All files are stored on a NAS (FreeNAS in this case) that has a pool of 40TB. All Movies are stored in separate folders (Iron Man, Iron Man 2, etc). TV shows stored in a folder with the TV shows name (Firefly\season 1, etc), Music is stored by Artists name, and Music Videos also follow the Music folder by Artists name.

1. Ripped Movie (or TV Show, or Music), is placed in the appropriate folder. Then I run MediaElch (https://forum.kodi.tv/showthread.php?tid=136333) to get all the necessary info on said item, including all artwork (Poster, Banner, Logo, Fanart, Disc, etc), and ratings (from theTVDB, but can switch where ratings come from), Genre's (in which you can create custom Genre's if needed), whether it is a Set item or stand alone file, etc. Then when I am happy with the data, I just hit save, and MediaElch generates a .NFO file with all this info, and creates all the artwork in the same folder as well.

2. Update Kodi's Library (and do a database clean after), and all the new items I've added, are now displayed within Kodi. I have used the CRON add-on in Kodi to set a time at night (12 midnight, with 30 minute increments for each additional Kodi box I have), to perform these library updates and library clean, daily.

3. I use the Trak.TV add-on to maintain all my "Watched status" with all my boxes, and it works great (with the occasional issue, but that's rare). I considered using MySQL at one point, but seeing all the issues people have upgrading from one version of Kodi to the next, keeps me from attempting this. Which leads me to the next point...

Now I do know about the updated .NFO file issue with Kodi, but how often are you going back into a Movie or TV show, and changing around the data? My simple rule is, just do a complete "FRESH" install of the new version of Kodi (and re-import all your data) when it comes out (I NEVER, install one version of Kodi over another one, just too many issues to deal with, with add-on's needing to be updated, or Skin issues, etc).

Now, I could possibly automate step one, but really, it takes maybe a few minutes to load MediaElch, grab all the info for something new I've added, and save. So not worth all the hassle of setting up, and maintaining an automation process. Plus with MediaElch, I can create custom Genre's, so I have a Superheroes Genre, Martial Arts Genre, and NewTV show Genre (so I can have all the new TV's shows under a Main heading in Kodi). This comes in very handy for the rest of the family. Oh, and did I mention that MediaElch is the only Media Manager that handles Music. 8)

P.S. One last thing, the reason why I don't let Kodi scrap all my data, is I have several Kodi boxes, so Imagine with my collection, Kodi going to all the services (TheTVDB, Fanart.TV, IMDB, etc) for all my stuff, on each separate device. Talk about a ton of traffic those websites don't need to deal with.

[publicENEMY]

One of the problem you have is you are scraping too many media at one time. Even when the media is already scraped. I also experience the same problem. Most media scraper apps are not build for handling very large number of media efficiently. I experience crashes and slowdowns when I scrape.

I fix this problem by reducing the number of media to scrape. I do this by creating a folder and create symlinks to the media that i want to scrape. This way i can manage the media i want to scrape without having to make physical changes to the original structure. I use EMM and TMM. TMM for all, EMM for rar splitted media.

[newtester]

Hi to all. Have the structure like yours with many content for personal use. How I manage it:
I use one kodi box as admin box (planning to use a VM with Ubuntu and Kodi on it as Author uses) to scrape/update new content to sql base and others user's kodi boxes to just sync to sql base.
After downloading a new content a script is run with filebot which renames,clear,download arts,generate .nfo,chmod/chown,compare etc new folders and files inside. After that I copy manually new content to server. I use https with apache on the server to transmit a content to clients. After that I run updating of the library on the admin box and all other boxes see the updates without any movements.
The issues I have now:
1. I have movies with 1 or 2 audio language tracks. Filebot generating .nfo file put tracks language info inside correctly, but there're a few kodi skins which can display this info and they display it only when additional "information" button from context menu is pressed - not a good decision. I would like to find a way to dispay language tracks info when listing movies/twshows in kodi. Please tell me who knows the way. This info is present is sql in special table.
2. Remote control. Used different ones but still looking for ideal one with comfortable control and presence of all needed functions for kodi.
3. Have a slow displaying of the movies info like posters and fanarts when opening a movies/tvshows library. Experimenting with presence of .nfo files now if the speed depends on it.

[docwra]

My system is pretty automated from downloaders to Kodi but I do keep things in Archives (video sources with "Exclude path from library updates" selected). Stuff like my 500 Standup Comedy DVDs or Extreme Sports DVDs never really need to be updated.

I also use Amazon and Netflix Add-ons now quite a bit to make a lot of my media collection redundant.

Trakt and TheAudioDB Add-ons to sync playcounts and ratings for Movies, TVShows and Music.

For me, an HTPC should be about ease of use and automation, not spending hours re-tagging or sorting things.