2018-06-01, 21:04
Hey Guys -
I am a long time Kodi/XBMC user who has a couple of large issues maintaining a large library which includes almost 30,000 episodes (~900 shows), over 5,000 movies, and some other misc media. My hope in posting is that someone may be able to examine the details of my environment and procedures then be able to suggest alternative tools, addons, or methods for how to maintain and update it which will potentially resolve the issues and ideally make the whole system more efficient. Below are specifications for my environment's setup, procedures I use to add media and update my libraries, and details about the two issue I am having. If you have any suggestions, I'd love to hear what they are!
Environmental Specs
Everything below is on the same LAN
Primary Workstation
- Windows 10 x64 / i7-3770k / 32gb RAM
- Use for everything you can think of including gaming, work, Oculus Rift, Plex Media Server, and file repository
- It stores all media on its D: partition is 43tb consisting of 12 (11 platter + 1 SSD for caching) disks in a disk pool (Stablebit DrivePool)
- On the disk pool, all media shares one of four common base paths:
- D:\Media\Movies
- D:\Media\Movies - New
- D:\Media\Movies - Collections
- D:\Media\TV Shows
"D:\Media\Movies - New" is where newly acquired movies go. They are moved out of this folder and into "D:\Media\Movies" after x days via a PowerShell script I have run nightly...
VMWare ESXi Host
- Ubunutu 16.04 VM running Kodi 17.6 dedicated to updating Kodi Library as described above (x2 CPU / 4gb RAM)
I soon plan to replace the above VM with a Docker container that runs Kodi headless (once I figure out how to schedule updating and add Trakt addon)
- Atomic Docker Host VM (Fedora 28) which includes a mySQL Server 5.7 container. The entire container / SQL Server is dedicated to the Kodi library
Players
- Dedicated PC connected to living room TV running Kodi 17.6 on Windows 10 x64
- x3 Raspberry Pis running OSMC (version equivalent to 17.6)
How I Currently Implement New Media
Obviously TV Shows and Movies are different, but not much.
TV Episodes
1. Obtain or rip physical media resulting in a digital file
2. Sonarr imports it, tags it, renames it, then moves it into a library folder
3. Approximately every couple of days an episode level scan is performed on all shows with Media Center Master to add metadata Sonarr doesn't add
Movies
1. Obtain or rip physical media resulting in a digital file
2a. For about 70%, I manually move it into a library folder
2b. For about 30%, CouchPotato imports it, tags it, renames it, then moves it into a library folder
3. At least once a day (or after each manual move) I scan new titles with MCM to tag, rename, and clean them
My Main Issues
The two main issues I am currently experiencing are with Kodi Library updating and crashing of the tool I use to organize the media. Details for each are below
Tagging & Renaming Tools
For the longest time, I've used Media Center Master to keep metadata and naming up to date for both TV and Movies. Unfortunately, the bigger my libraries have gotten; the more MCM crashes - to the point where 90% of the time I invest updating metadata is ending task on MCM, restarting the app, and waiting for it to initialize (if it does properly). I've posted many times on their forum, but answers are vague and don't help. I recently downloaded the latest build of Ember Media Manager Alpha (1.4.8.0) which I configured and added my media to and am having a few minor issues with it that I'm trying to work out. I've used TinyMediaManager in the past as well with unstable results. What's the suggested one for libraries like mine?
Kodi Library Updating / Addons
As you can see below, I use a mySQL server for my Kodi Library and have a dedicated Linux VM just to update it hourly. This is done by running Kodi in an Ubunutu 16.04 VM which has the Library Auto Updater and Trakt addons installed. It's configured to update my video library hourly and once done clean it then sync with Trakt. When everything works well, an update takes 20-25 minutes. I'm sure it would be faster if the update and DB were on the same system as the media files, but updates kill my CPU when being performed which is why I have them separated. I've tried Watchdog in the past when I still had a large library, but had major issues with it. I don't recall the details. What would be the suggested method to keep my type of library up to date?
Notes & Some Things I've Already Tried
- Soon I'll probably being use Kodi for only my dedicated Kodi system in the living room and covert the x3 RiPis to RasPlex. I've considered going to Plex 100% but it's client isn't close to as powerful as Kodi is and on that living room PC I use a good amount of addons for things like Steam, automation, plus a theme that's easy for the wife to use it. When doing this I could migrate from mySQL back to a standard, local DB on that single system however it's currently connected via wifi so updating would be slower unless I run cable
- I've tried the Plex addon for Kodi as you'd think that would be a great solution, but it caused havoc with my Kodi database. I don't remember the details, but it would cause my Plex Server to trigger library rescanning nonstop thus using tons of resources. After tons of troubleshooting, I was finally told in a forum that it was the blame one once I removed the addon and recreated my mySQL database, all was good. I don't know if it would affect a standard Kodi db as I could go to that soon
- Instead of CouchPotato, I've tried both Radarr and Watcher. I really wanted to use Radarr since it shares a lot of source code with Sonarr and like it so much. However, they both had difficulties managing libraries of this size and in the end I reverted back to CouchPotato which works much better
Whew - I think that covers it! Any suggestions? Thanks, Guys!
I am a long time Kodi/XBMC user who has a couple of large issues maintaining a large library which includes almost 30,000 episodes (~900 shows), over 5,000 movies, and some other misc media. My hope in posting is that someone may be able to examine the details of my environment and procedures then be able to suggest alternative tools, addons, or methods for how to maintain and update it which will potentially resolve the issues and ideally make the whole system more efficient. Below are specifications for my environment's setup, procedures I use to add media and update my libraries, and details about the two issue I am having. If you have any suggestions, I'd love to hear what they are!
Environmental Specs
Everything below is on the same LAN
Primary Workstation
- Windows 10 x64 / i7-3770k / 32gb RAM
- Use for everything you can think of including gaming, work, Oculus Rift, Plex Media Server, and file repository
- It stores all media on its D: partition is 43tb consisting of 12 (11 platter + 1 SSD for caching) disks in a disk pool (Stablebit DrivePool)
- On the disk pool, all media shares one of four common base paths:
- D:\Media\Movies
- D:\Media\Movies - New
- D:\Media\Movies - Collections
- D:\Media\TV Shows
"D:\Media\Movies - New" is where newly acquired movies go. They are moved out of this folder and into "D:\Media\Movies" after x days via a PowerShell script I have run nightly...
VMWare ESXi Host
- Ubunutu 16.04 VM running Kodi 17.6 dedicated to updating Kodi Library as described above (x2 CPU / 4gb RAM)
I soon plan to replace the above VM with a Docker container that runs Kodi headless (once I figure out how to schedule updating and add Trakt addon)
- Atomic Docker Host VM (Fedora 28) which includes a mySQL Server 5.7 container. The entire container / SQL Server is dedicated to the Kodi library
Players
- Dedicated PC connected to living room TV running Kodi 17.6 on Windows 10 x64
- x3 Raspberry Pis running OSMC (version equivalent to 17.6)
How I Currently Implement New Media
Obviously TV Shows and Movies are different, but not much.
TV Episodes
1. Obtain or rip physical media resulting in a digital file
2. Sonarr imports it, tags it, renames it, then moves it into a library folder
3. Approximately every couple of days an episode level scan is performed on all shows with Media Center Master to add metadata Sonarr doesn't add
Movies
1. Obtain or rip physical media resulting in a digital file
2a. For about 70%, I manually move it into a library folder
2b. For about 30%, CouchPotato imports it, tags it, renames it, then moves it into a library folder
3. At least once a day (or after each manual move) I scan new titles with MCM to tag, rename, and clean them
My Main Issues
The two main issues I am currently experiencing are with Kodi Library updating and crashing of the tool I use to organize the media. Details for each are below
Tagging & Renaming Tools
For the longest time, I've used Media Center Master to keep metadata and naming up to date for both TV and Movies. Unfortunately, the bigger my libraries have gotten; the more MCM crashes - to the point where 90% of the time I invest updating metadata is ending task on MCM, restarting the app, and waiting for it to initialize (if it does properly). I've posted many times on their forum, but answers are vague and don't help. I recently downloaded the latest build of Ember Media Manager Alpha (1.4.8.0) which I configured and added my media to and am having a few minor issues with it that I'm trying to work out. I've used TinyMediaManager in the past as well with unstable results. What's the suggested one for libraries like mine?
Kodi Library Updating / Addons
As you can see below, I use a mySQL server for my Kodi Library and have a dedicated Linux VM just to update it hourly. This is done by running Kodi in an Ubunutu 16.04 VM which has the Library Auto Updater and Trakt addons installed. It's configured to update my video library hourly and once done clean it then sync with Trakt. When everything works well, an update takes 20-25 minutes. I'm sure it would be faster if the update and DB were on the same system as the media files, but updates kill my CPU when being performed which is why I have them separated. I've tried Watchdog in the past when I still had a large library, but had major issues with it. I don't recall the details. What would be the suggested method to keep my type of library up to date?
Notes & Some Things I've Already Tried
- Soon I'll probably being use Kodi for only my dedicated Kodi system in the living room and covert the x3 RiPis to RasPlex. I've considered going to Plex 100% but it's client isn't close to as powerful as Kodi is and on that living room PC I use a good amount of addons for things like Steam, automation, plus a theme that's easy for the wife to use it. When doing this I could migrate from mySQL back to a standard, local DB on that single system however it's currently connected via wifi so updating would be slower unless I run cable
- I've tried the Plex addon for Kodi as you'd think that would be a great solution, but it caused havoc with my Kodi database. I don't remember the details, but it would cause my Plex Server to trigger library rescanning nonstop thus using tons of resources. After tons of troubleshooting, I was finally told in a forum that it was the blame one once I removed the addon and recreated my mySQL database, all was good. I don't know if it would affect a standard Kodi db as I could go to that soon
- Instead of CouchPotato, I've tried both Radarr and Watcher. I really wanted to use Radarr since it shares a lot of source code with Sonarr and like it so much. However, they both had difficulties managing libraries of this size and in the end I reverted back to CouchPotato which works much better
Whew - I think that covers it! Any suggestions? Thanks, Guys!
