Kodi Community Forum

Full Version: problem importing large library on Android
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
for the past few days I have been trying to import my library of TV Shows and Movies off of my NAS

I run Windows on my Desktop and have an
Android 6 box with 2gig ram 8gig primary and a 32gig microsd running Kodi 17 Fresh install from the Play Store

No additional addons are installed.

My Windows Computer seems to import most if not all of my shows its updating right now.

My Android Box seems to be failing at 286 TV Series I didn't count how many episodes probably thousands

My Movies seem to import without problem there are about 2000 of them

On the NAS I created an account just for the Recorded TV

The storage of files is like such

Movies

/Movies/Comedy/Ace Ventura Pet Detective-2015-09-06 1409.mp4

(this movie imports fine it is the DVR record date not the movie release date)


TVShows

/TVShows/Flying Wild Alaska/Flying Wild Alaska - 1x07 - Deep Freeze.mp4

The TV Show is following normal naming conventions and was renamed using FileBot


On the Computer I believe all of the directories are present

On the android box
if i click on TV Shows in the menu it shows 191
if I go to AtoZ it shows 259
if I go into Library / Manage Sources/ Videos and click TVShows share it shows 286


On the Windows Computer
if i click on TV Shows in the menu it shows 339
if I go to AtoZ it shows 445
if I go into Library / Manage Sources/ Videos and click TVShows share it shows 495


My Actual Directories for TV Shows is 495
so neither of them are picking up everything but the windows computer does better



Something seems to be choking

The windows computer does much better..
Both are on Gigabit Wired on the same switch as the NAS

Don't know if Android is timing out
Don't know if I am having a memory/storage problem (turned off fanart)


I have tried many times refreshing and searching for new content


I am still recording shows from my Ceton DVR so the Library is only going to get bigger
Probably not Twice as big as I have been recording for years to get to this point
but I expect to add a good amount in the future


Are there any limits on the Android distribution?

Where do I look to find out what is choking?

Also I know some of my folders and files are not renamed correctly ... but even some of the ones not named correctly will show up as series with 0/0

Thanks in advance
Quote:Don't know if Android is timing out
Don't know if I am having a memory/storage problem (turned off fanart)
Maybe, keep the posters up at least have something that looks good.

Some of the issues reading your post that stand out. Ace Venture Pet Detective-2015-09-06 1409.mp4 should be Ace Ventura: Pet Detective (1994) so if you expect the scraper to find it... time out. You didn't mention if you have your video files in folders of the same name, but it's smart thing to do. You didn't mention how large your collection is, so I can only make a few assumptions, and an Android box is going to have limited memory for posters/fanart etc.. some libraries can easily overwhelm the storage capacity (you might think in the future of using Path substitution (wiki)) and of course the CPU on most of these boxes are just enough.

I would recommend, that you export your library to separate files from the PC, each folder gets a poster/fanart etc... Then in the Android device, choose the 'local information scraper' and you will not have to be dependant on internet traffic, or response and the scraper will pick up exactly what your PC has. Should some title miss, then it's likely to have been a slight network timeout on your system, or some dedicated cache in the Android system is out of running room. Once yuou have them all scraped it.... run though the various sources giving the Android system time to set-up caches and hash# for the artwork. I find TV shows much more fussy when it comes to nomenclature Naming video files/TV shows (wiki). The numbers given in the totals for various folders/films can be a bit deceiving and would be dependant on the skin, sets on/off, and various videos missing or whatnot, not to mention versions of the software. Don't put more than 200 files in one source (less than 100 is even better) otherwise you start getting unbelievable slowdowns just listing the page.

Quote:Something seems to be choking
Nomenclature, use the local scraper after you have your collection looking good on one of the boxes.
As far as I can tell most if not all of my Movies import the date on the end doesn't seem to cause a problem.

My main problem is with my TV Series and episodes

Yes the reason I turned off fanart and trailers is to save space

My complete Archive is about 27000 videos with 2100 of them being Movies

Are you saying that I should add many sources for TV?
Make Sub Folders for Alphabetical Folders and store those shows in them to make the source contents smaller?

Add each one of the following directories on my NAS as a individual TVSource?
/TVShows/A-D
/TVShows/E-H
/TVShows/I-K
/TVShows/......

Having Many TVShow Sources won't cause problems? I have never tried that and thought the database would still be a single database.

Many of my individual TV Series exceed your recommendation (such as The Simpsons) and have over 200 episodes for the series

Timeout... I was thinking more that Android would just give up searching but the thing is
After reading about the way the Scrapers work
If an Item is in the database it won't do a lookup on it
So Searching for New Content over and over should populate everything as it would start where it left off.

But it is not picking up any additional Series or Shows.


I just tried the Missing Movies Plugin and it only showed 27 missing TV Series and there must be 200 missing.



I also saw that it can't do anything with some shows I record like This Old House
because my Meta and file name as originally recorded don't give me episode info
So it just saves all the files like ..

This Old House Ashburn Project.mp4

then when I copy them over I get a folder of files like this because I say.. Keep Both.. so it adds a number
This Old House Ashburn Project (01).mp4
This Old House Ashburn Project (02).mp4
This Old House Ashburn Project (03).mp4


I appreciate your reply

I am going to try to work on some of the renaming of files
but I have been using TVMaze as my scraper in FileBot because TheTVDB fails a lot and sometimes both of them fail

I read for ones that just completely fail I can make a .nfo file for the series..
and I can do that for old series I finished recording but not so much for new shows still playing on TV



Yes I was also considering exporting the DB from my computer and putting it on my NAS so I can import it into the android box.

The box is pretty good it has a 912 processor with 2gigs ram and 8gigs rom and a 32gb microsd
It can do h.265 well and thats why i bought it because I am converting my videos to that format to save space.
cost me $56 which is kinda high for me but not like some of the $300 boxes heh

I dumped all the apps that came on it and reinstalled Kodi17

I am hoping to get liveTV working on it maybe and pseudo if I can manage that but the fact I am working hard on this i might skip pseudo.

again thanks and if anything I replied with helps you understand I hope you let me know






(2017-02-27, 02:23)PatK Wrote: [ -> ]
Quote:Don't know if Android is timing out
Don't know if I am having a memory/storage problem (turned off fanart)
Maybe, keep the posters up at least have something that looks good.

Some of the issues reading your post that stand out. Ace Venture Pet Detective-2015-09-06 1409.mp4 should be Ace Ventura: Pet Detective (1994) so if you expect the scraper to find it... time out. You didn't mention if you have your video files in folders of the same name, but it's smart thing to do. You didn't mention how large your collection is, so I can only make a few assumptions, and an Android box is going to have limited memory for posters/fanart etc.. some libraries can easily overwhelm the storage capacity (you might think in the future of using Path substitution (wiki)) and of course the CPU on most of these boxes are just enough.

I would recommend, that you export your library to separate files from the PC, each folder gets a poster/fanart etc... Then in the Android device, choose the 'local information scraper' and you will not have to be dependant on internet traffic, or response and the scraper will pick up exactly what your PC has. Should some title miss, then it's likely to have been a slight network timeout on your system, or some dedicated cache in the Android system is out of running room. Once yuou have them all scraped it.... run though the various sources giving the Android system time to set-up caches and hash# for the artwork. I find TV shows much more fussy when it comes to nomenclature Naming video files/TV shows (wiki). The numbers given in the totals for various folders/films can be a bit deceiving and would be dependant on the skin, sets on/off, and various videos missing or whatnot, not to mention versions of the software. Don't put more than 200 files in one source (less than 100 is even better) otherwise you start getting unbelievable slowdowns just listing the page.

Quote:Something seems to be choking
Nomenclature, use the local scraper after you have your collection looking good on one of the boxes.
As far as I can tell most if not all of my Movies import the date on the end doesn't seem to cause a problem.

Follow proper nomenclature, Naming video files/Movies (wiki) This is why you have scraping issues. Movie_name(date).mkv


My main problem is with my TV Series and episodes

Yes the reason I turned off fanart and trailers is to save space

My complete Archive is about 27000 videos with 2100 of them being Movies


Noted, use TheRenamer (wiki) drag & drop simple, even full folders of seasons in one shot) saving a paltry 100 megs for a poster, isn't much help when we're talking gigs per movie, dress up your collection and you'll feel like maintaining it. Your numbers are typical.


Are you saying that I should add many sources for TV?
Make Sub Folders for Alphabetical Folders and store those shows in them to make the source contents smaller?

I don't know how you have your sources organized, but I have maybe 50 sources as genre and a few extras, then split up the directories when they get a bit too big, e.g. SiFi and Space, or Comedy and Laffs etc. My sets are handled as sub folders, no more than 30 - 50 films in each as James Bond or Woody Allen etc. Take any flat folder with 500 files and do a directory listing... takes time right? Try it with 20 files, bang it's there.... that time difference for the look-up is always going to be an issue regardless of the program that wants to list it, solution make your page lists smaller and tighter.

Add each one of the following directories on my NAS as a individual TVSource?
/TVShows/A-D
/TVShows/E-H
/TVShows/I-K
/TVShows/......

TV shows are a special case,TV sources are already in a hierarchy of folders, and sub folders.... let's say you have 200 TV shows in one drive, then seasons are sub folders and most don't go beyond 10-20 seasons, then within that each season has only 10-30 episodes... but loading them into a few segments would make them a bit quicker to display. e.g. move the documentary into a special folder, public domain in another, drama, etc...

Having Many TVShow Sources won't cause problems? I have never tried that and thought the database would still be a single database.
Many of my individual TV Series exceed your recommendation (such as The Simpsons) and have over 200 episodes for the series

The Simpsons would be only one listing in your TV drive with 25 sub-folders for seasons, then each seasons sub folder would be a limited episode count. Sure clicking on the TV button from the front just brings up the root folders. With a collection the size you have you would have already seen the impracticality of looking at your entire collection as a flat file in 'Movies' or 'TV', that's why 'Nodes, Tags, Smart-playlists, Favourites, Sets and all the other tools to slice & dice your collection into meaningful categories. yes you can have speed and eye candy together with a bit of careful planning.

Timeout... I was thinking more that Android would just give up searching but the thing is
After reading about the way the Scrapers work
If an Item is in the database it won't do a lookup on it
So Searching for New Content over and over should populate everything as it would start where it left off.


It'll just make the same mistakes over and over.... until you correct your nomenclature, or decide to pull them into your library one by one.... re-read the part I mentioned about local scraper..... which just pulls local data that's already scraped on your PC.


But it is not picking up any additional Series or Shows.


You are just not seeing that your bad naming convention are the root of this issue. I beg you to use TheRenamer, if this program see's the file, Kodi will se it.


I just tried the Missing Movies Plugin and it only showed 27 missing TV Series and there must be 200 missing.

I also saw that it can't do anything with some shows I record like This Old House
because my Meta and file name as originally recorded don't give me episode info
So it just saves all the files like ..

This Old House Ashburn Project.mp4

then when I copy them over I get a folder of files like this because I say.. Keep Both.. so it adds a number
This Old House Ashburn Project (01).mp4
This Old House Ashburn Project (02).mp4
This Old House Ashburn Project (03).mp4
Go here https://thetvdb.com/?tab=advancedsearch and get the proper name listed in this database, if named the same it will scrape 100%

I am going to try to work on some of the renaming of files
but I have been using TVMaze as my scraper in FileBot because TheTVDB fails a lot and sometimes both of them fail

I read for ones that just completely fail I can make a .nfo file for the series..
and I can do that for old series I finished recording but not so much for new shows still playing on TV


Get your information from the source,Use the link, the TVdb is where Kodi get's it's information.


Yes I was also considering exporting the DB from my computer and putting it on my NAS so I can import it into the android box.

The box is pretty good it has a 912 processor with 2gigs ram and 8gigs rom and a 32gb microsd
It can do h.265 well and thats why i bought it because I am converting my videos to that format to save space.
cost me $56 which is kinda high for me but not like some of the $300 boxes heh

I dumped all the apps that came on it and reinstalled Kodi17

I am hoping to get liveTV working on it maybe and pseudo if I can manage that but the fact I am working hard on this i might skip pseudo.

again thanks and if anything I replied with helps you understand I hope you let me know

Go with Lazy TV to start off with.
well .. this is so pitiful .. like I was saying there was a different number of TVShows being imported on the Android box only about half.

So I kept wondering what was different.....

Finally I logged out as admin on the windows box
and logged back in as the account the TV Box was using

BOOM 286 folders were visible not the 495 actual folders.

So it was folder permissions however I have no idea why.
When I created the TVBox Account I selected the top directory and
all of the permissions should have been sent down to lower folders
All of the Movies worked and most of the TVShow Directories did too
Stranger was that it wasn't alphabetical when permissions were not applied
to some of the folders it was random.
There were folders visible from the A's to the Z's so whatever order it used
when it must have timed out on the NAS when applying the permissions I don't know

I logged into the NAS which is a Netgear Ready NAS and reset the permissions
Right away all 495 folders were available when I browsed to add a video source.

I added the video source and it scraped for the rest of the day and now my
TVShows are showing 444 Series instead of 286 Series.
It is not the full 495 Series that are recorded but it gives me a lot of hope.

I have no idea why the user account TVBox wasn't given permission to all of the sub directories

I am glad I decided to login from my windows computer under that account to check

So right now I am rescraping my Movies and that shouldn't take long
Later I will check which TVShows are missing and unfortunately the ReadyNAS does not
give you a folder by folder chmod or properties list.. so i will have to make sure
that everything is available.. I forsee problems in the future... and this is slightly scary but

At least I know the android box can do pretty well and scrap the TVShows pretty well


Thanks again for your help...

Maybe the next person will find this thread useful.

If you find that your account for your Android box is not picking up all of your files or directories.
use a computer to login with that user's Name and Password
and Maybe you will find out that your permissions are not set correctly.

I was not really suspecting this as a permission problem but I knew there was something going on.

ALSO .. This is over SMB protocol so keep that in mind too


Thanks again for the company

I am still in the early stages of figuring this problem out.. i am a long time Kodi User since 11 I believe
thing is I never had this problem because I never tried using it in this way.
hopefully i figure it out.. if not .. i'll get in the choppa and i'll be back

>:o)