Best Management Methods for Large Libraries
#16
I search for a way the user rating tag to be updated or created in the NFO file when one add its own rating in kodi. Did anyone make an extension for that purpose ?
  • Windows 10 x64
  • Kodi 17.6
Reply
#17
I've a large library.

I use a temp folder solution with sick gear to pull and update show details.

I use EMBY to update movies, by just putting the movie file into a movie directory.

Kodi updates the library from local files whenever it's been idle.
Reply
#18
I don't have nearly the size of library as you (I don't know if there are many at all to be honest lol), but i have about 4TB of media that's hosted on a Rpi3 over WiFi. WiFi isn't my bottleneck, but the USB 2.0 of raspberry Pis most definitely is. that being said, I do have a few hundred movie folders on one external drive connected to the pi. Eventually I started to notice how just entering the folder in a GUI would seem to almost hang just before finally loading every folder (this is without a thumbnail view). That's when I started having tinyMediaManager move each movie into it's sorted title, first letter folder. Now I've got a library of only 27 folders in the root (A-Z plus #), and it takes no time at all to browse or have kodi scrape for that matter. You can imagine how great a difference that made for few thousand artist folders I had for my music, which I now use picard (MusicBrainzdb) to manage in the same fashion.
I'll finish by mentioning that I don't use an SQL database, and that may be how one gets around the very issues I'm describing as having here. I've been looking more and more into how to implement and use a database as my library grows, but of course will have put it off until it's a mess xD.

TL;DR -- Nest your library as much as possible to cut down on the query time for the index/file list of such large directories. Movies/TV Shows might be great to use the first letter in the sorted title, followed by first genre, year, then finally the title, or however makes the most sense to whoever is managing it. With the roughly 500 movies I'm approaching, in addition to another drive, I'm thinking of a new pattern myself of how to best store things.
Reply
#19
@psyburr  , am I understanding the above post correctly , your saying you have all your movies in alphabetical folders on TMM ?
same with series ?
Reply
#20
(2018-06-01, 21:04)bzowk Wrote: Hey Guys - 

I am a long time Kodi/XBMC user who has a couple of large issues maintaining a large library which includes almost 30,000 episodes (~900 shows), over 5,000 movies, and some other misc media.  My hope in posting is that someone may be able to examine the details of my environment and procedures then be able to suggest alternative tools, addons, or methods for how to maintain and update it which will potentially resolve the issues and ideally make the whole system more efficient.  Below are specifications for my environment's setup, procedures I use to add media and update my libraries, and details about the two issue I am having.  If you have any suggestions, I'd love to hear what they are!

Environmental SpecsEverything below is on the same LANPrimary Workstation
  - Windows 10 x64 / i7-3770k / 32gb RAM
  - Use for everything you can think of including gaming, work, Oculus Rift, Plex Media Server, and file repository
  - It stores all media on its D: partition is 43tb consisting of 12 (11 platter + 1 SSD for caching) disks in a disk pool (Stablebit DrivePool)
  - On the disk pool, all media shares one of four common base paths:
      - D:\Media\Movies
      - D:\Media\Movies - New
      - D:\Media\Movies - Collections
      - D:\Media\TV Shows
      "D:\Media\Movies - New" is where newly acquired movies go.  They are moved out of this folder and into "D:\Media\Movies" after x days via a PowerShell script I have run nightly...VMWare ESXi Host
  - Ubunutu 16.04 VM running Kodi 17.6 dedicated to updating Kodi Library as described above (x2 CPU / 4gb RAM)
     I soon plan to replace the above VM with a Docker container that runs Kodi headless (once I figure out how to schedule updating and add Trakt addon)
  - Atomic Docker Host VM (Fedora 28) which includes a mySQL Server 5.7 container.  The entire container / SQL Server is dedicated to the Kodi library
Players 
  - Dedicated PC connected to living room TV running Kodi 17.6 on Windows 10 x64
  - x3 Raspberry Pis running OSMC (version equivalent to 17.6)How I Currently Implement New Media
Obviously TV Shows and Movies are different, but not much.
TV Episodes
1. Obtain or rip physical media resulting in a digital file
2. Sonarr imports it, tags it, renames it, then moves it into a library folder
3. Approximately every couple of days an episode level scan is performed on all shows with Media Center Master to add metadata Sonarr doesn't add
Movies
1. Obtain or rip physical media resulting in a digital file
2a. For about 70%, I manually move it into a library folder
2b. For about 30%, CouchPotato imports it, tags it, renames it, then moves it into a library folder
3. At least once a day (or after each manual move) I scan new titles with MCM to tag, rename, and clean them

My Main IssuesThe two main issues I am currently experiencing are with Kodi Library updating and crashing of the tool I use to organize the media.  Details for each are belowTagging & Renaming Tools
For the longest time, I've used Media Center Master to keep metadata and naming up to date for both TV and Movies.  Unfortunately, the bigger my libraries have gotten; the more MCM crashes - to the point where 90% of the time I invest updating metadata is ending task on MCM, restarting the app, and waiting for it to initialize (if it does properly).  I've posted many times on their forum, but answers are vague and don't help.  I recently downloaded the latest build of Ember Media Manager Alpha (1.4.8.0) which I configured and added my media to and am having a few minor issues with it that I'm trying to work out.  I've used TinyMediaManager in the past as well with unstable results.  What's the suggested one for libraries like mine?
Kodi Library Updating / Addons
As you can see below, I use a mySQL server for my Kodi Library and have a dedicated Linux VM just to update it hourly.  This is done by running Kodi in an Ubunutu 16.04 VM which has the Library Auto Updater and Trakt addons installed.  It's configured to update my video library hourly and once done clean it then sync with Trakt.  When everything works well, an update takes 20-25 minutes.  I'm sure it would be faster if the update and DB were on the same system as the media files, but updates kill my CPU when being performed which is why I have them separated.  I've tried Watchdog in the past when I still had a large library, but had major issues with it.  I don't recall the details.  What would be the suggested method to keep my type of library up to date?

Notes & Some Things I've Already Tried
- Soon I'll probably being use Kodi for only my dedicated Kodi system in the living room and covert the x3 RiPis to RasPlex.  I've considered going to Plex 100% but it's client isn't close to as powerful as Kodi is and on that living room PC I use a good amount of addons for things like Steam, automation, plus a theme that's easy for the wife to use it.  When doing this I could migrate from mySQL back to a standard, local DB on that single system however it's currently connected via wifi so updating would be slower unless I run cable
- I've tried the Plex addon for Kodi as you'd think that would be a great solution, but it caused havoc with my Kodi database.  I don't remember the details, but it would cause my Plex Server to trigger library rescanning nonstop thus using tons of resources.  After tons of troubleshooting, I was finally told in a forum that it was the blame one once I removed the addon and recreated my mySQL database, all was good.  I don't know if it would affect a standard Kodi db as I could go to that soon
- Instead of CouchPotato, I've tried both Radarr and Watcher.  I really wanted to use Radarr since it shares a lot of source code with Sonarr and like it so much.  However, they both had difficulties managing libraries of this size and in the end I reverted back to CouchPotato which works much better

Whew - I think that covers it!  Any suggestions?  Thanks, Guys!

IMO your #1 problem is that you're running it under MS Windows.  Windows is the worst resource hog ever and will consume every physical resource you have just to manage the OS.

I run a server with 87 remote, off-site users, 8 local in-network users, 3,200+ movies, 22,000 TV episodes in 600+ TV shows, 400+ audio books, and 18,200+ music tracks running on a Freenas Linux VM server running on a mere i5 4-core equivalent with 4GB of RAM and it's snappy and responsive as hell for all users, local and remote.  I promise.  Your efficiency problem is Windows.
Reply
#21
(2019-07-30, 00:43)ontap Wrote: @psyburr  , am I understanding the above post correctly , your saying you have all your movies in alphabetical folders on TMM ?
same with series ?

So this was neglected for far too long by me, I apologize. To answer your question, yes, I do have my movies set up like that and TMM can auto-rename the MOVIES for you like that just fine. As far as series go, I do not have series setup like that because previously TMM wouldn't be able to sort them with the renamer by itself if you used a single root source folder for series in the application. I would have had to manually move them all to their own root folder per letter manually, then enter them each in as  a source. I do think that they have changed that since last replying here, but be sure to read the documentation regarding that because I haven't changed my structure as I don't have too many unique series titles (at this point), just quite a few episodes which are all handled as they should in a Title -> Season_## -> file.ext tree.
Again, sorry this is late, but maybe it can help those stumbling upon it in the future if you've already solved your problem at-hand.
Reply
#22
Bottom line Just just Mediaportal
Reply



Logout Mark Read Team Forum Stats Members Help
Best Management Methods for Large Libraries0
This forum uses Lukasz Tkacz MyBB addons.