Scraping a BIG library?
#1
Hi!
I have a library of 950 movies & 600 TV Shows on my webserver (hosted at GoDaddy.com) and I want to add it to my XBMC on my RPi.
I've tried adding it as FTP Source but after 3 days of scraping I had only about 60 TVShows.
Can anyone suggest a solution for this?
I have several RPi's, some at my place, others at my friend's and I want that all of our RPi's will have the same movies and tvshows.
I thought about doing it with MySQL but then I noticed that when my friend finishes watching an episode it appears at my RPi too and that's a problem Sad

RPi = Raspberry Pi btw ^^
Reply
#2
Excuse me, what exactly do you mean by "scraping"?

Maybe a solution would be to "scrape" your lib with a MySQL solution directly on your webserver and after that, export the Mysql db and finally import it into a local one on your RPi? E.g. here http://arnaudbertrand.com/export-import-...rary-kodi/

#edit:
okay, i now know what's all about scraping: http://kodi.wiki/view/Scraper

#edit2:
After reading the above mentioned wiki page, I have the question: Do you have the default scraper enabled or installed any special plugin for that purpose?
Reply
#3
(2014-12-30, 14:32)Sagiigi340 Wrote: I have several RPi's, some at my place, others at my friend's and I want that all of our RPi's will have the same movies and tvshows.
I thought about doing it with MySQL but then I noticed that when my friend finishes watching an episode it appears at my RPi too and that's a problem Sad

Presumably for the Pi's at your house, having shared watched flags is useful?

I would suggest you set up a shared MySQL database for the Pi's at your house.
Your friend sets up a (separate) shared MySQL database for the Pi's at his house.

You both add a more powerful machine (e.g. a Windows PC) that connects to MySQL (possibly hosts it too) and does the initial scan.
For incremental changes, then scanning on the Pi should be fine.

So there are two unrelated MySQL databases, one in each house. Each house will get unique watched flags. The video content you watch is common.
Reply
#4
Scrape it on your computer (using the same source path that the Pi will use) and then copy the MyVideos90.db file off the userdata (wiki)/Database/ folder on the PC to the same folder on the Pi.
Reply
#5
(2014-12-30, 15:31)Strubbl Wrote: Excuse me, what exactly do you mean by "scraping"?

Maybe a solution would be to "scrape" your lib with a MySQL solution directly on your webserver and after that, export the Mysql db and finally import it into a local one on your RPi? E.g. here http://arnaudbertrand.com/export-import-...rary-kodi/

#edit:
okay, i now know what's all about scraping: http://kodi.wiki/view/Scraper

#edit2:
After reading the above mentioned wiki page, I have the question: Do you have the default scraper enabled or installed any special plugin for that purpose?
The default scraper

(2014-12-30, 15:48)popcornmix Wrote:
(2014-12-30, 14:32)Sagiigi340 Wrote: I have several RPi's, some at my place, others at my friend's and I want that all of our RPi's will have the same movies and tvshows.
I thought about doing it with MySQL but then I noticed that when my friend finishes watching an episode it appears at my RPi too and that's a problem Sad

Presumably for the Pi's at your house, having shared watched flags is useful?

I would suggest you set up a shared MySQL database for the Pi's at your house.
Your friend sets up a (separate) shared MySQL database for the Pi's at his house.

You both add a more powerful machine (e.g. a Windows PC) that connects to MySQL (possibly hosts it too) and does the initial scan.
For incremental changes, then scanning on the Pi should be fine.

So there are two unrelated MySQL databases, one in each house. Each house will get unique watched flags. The video content you watch is common.
I have 5 friends with RPi and we both share the same SeedBox so that would be not very efficient to create several MySQL databases Sad

(2014-12-30, 15:48)Ned Scott Wrote: Scrape it on your computer (using the same source path that the Pi will use) and then copy the MyVideos90.db file off the userdata (wiki)/Database/ folder on the PC to the same folder on the Pi.
But I want that when I download a new movie \ series to the SeedBox, both me and my friend will be able to have it
Reply
#6
That would just be for the initial scan. New additions can be individually scanned and won't take as long.
Reply
#7
(2014-12-30, 16:46)Ned Scott Wrote: That would just be for the initial scan. New additions can be individually scanned and won't take as long.

Can you please explain me how? As far as I know adding a new TV show requires scanning the whole thing all over again
Reply
#8
(2014-12-30, 17:18)Sagiigi340 Wrote: Can you please explain me how? As far as I know adding a new TV show requires scanning the whole thing all over again

If only a few videos have been added a library scan updates quickly even on a Pi (e.g. takes 1 minute).
Reply
#9
(2014-12-30, 18:06)popcornmix Wrote:
(2014-12-30, 17:18)Sagiigi340 Wrote: Can you please explain me how? As far as I know adding a new TV show requires scanning the whole thing all over again

If only a few videos have been added a library scan updates quickly even on a Pi (e.g. takes 1 minute).

How do you update your library then? When I update my library even on my I5 computer it takes several days to be completed
Reply
#10
Well, just to be honest - Kodi is meant to be used with a local library (local as in "in your House") - FTP is wildly inefficient for this task, especially if used over the Internet...
Reply
#11
(2014-12-30, 18:16)kuldan Wrote: Well, just to be honest - Kodi is meant to be used with a local library (local as in "in your House") - FTP is wildly inefficient for this task, especially if used over the Internet...

Do you have a recommendation for another protocol? I assume Samba would do better but I cannot use that since the RPi's are connected to different LANs
Reply
#12
*bump*
Reply
#13
WebDAV (wiki) seems to work fairly good for internet connections.
Reply
#14
RPi, owners listen up, the RPi is too slow to scape Metadata itself especially as a Video Library starts to get larger.
It does not have enough memory and the write speed to the SD card is far to slow.

Employ proper file management now before it gets out of control. I learnt my lesson this year.

Store all your Video Files in separate folders on an external drive or NAS. The folder name must match the Movie Title or TV Series Title.
Scape all Metadata required with a separate media manager external to the RPi, I use MediaElch and its fast.

I add the Video Shares, and set:
- the scraper to The Movie Data Base
- Movies are in separate folders that match the movie title - ON
- Scan recursively - OFF
- Selected folder contains a single video

This is a video library that contains over 3400 titles.
Library Import is reasonably quick, not as quick as a PC or Mac but far far quicker than a RPi using internal scraping.
Kodi library import speed is far quicker than Gotham.
Plus movie playback over ethernet using NFS is pretty much instantaneous.
Kodi Library navigation is fast with a large library.
The library then become extremely portable with no scraping on a new media box necessary when you move your media storage around.

The only downside is you must Export your Library data occasionally to update the watched status.

You really must Overclock a RPi with a large library or it will be far too slow.

Reply
#15
(2014-12-31, 13:54)wrxtasy Wrote: I add the Video Shares, and set:
- the scraper to The Movie Data Base
- Movies are in separate folders that match the movie title - ON
- Scan recursively - OFF
- Selected folder contains a single video

Doesn't setting the scrapper to TMDB trigger full scrapping again, ignoring the local metadata previewsly fetched from the external scrapper?
I think the Local Information Only "scrapper" must be set to take advantage of the external metadata scrapping.

Anyway in my case (about 1000 movies) I find the whole proccess unnecesarry, especially with helix.
Internal scrapping is fast enough, so is navigating.
But I do have a fast (the fastest!) usb stick for my userdata.
Reply

Logout Mark Read Team Forum Stats Members Help
Scraping a BIG library?0