Enable more than one scraper or not?
#1
Currenty i have TMDB scraper enabled

but should i also enable another scraper like IMDB etc..

What effect would this have on the scanning process?

Would it increase the time it takes to scan?
And how much time increase would occur?
Reply
#2
IMDB has no API and is very slow. We have to parse each website to get the informations. It's not to much time for movies, but it needs many time to parse all data if you use IMDB for episodes.

For movies I use TMDB as first scraper and IMDB as second. In TMDB I've disabled Rating and in IMDB I've enabled only Rating and Top250.
For tv shows I've only enabled TVDB for data. TMDB has an API that is not very good for scraping many episodes. TMDB has a limit a 30 API calls in 10 seconds, then you will be paused for 30 seconds. If you rescrape a tv show like The Simpsons with more than 500 episodes... never ending story :-(
Reply
#3
(2016-07-24, 12:11)DanCooper Wrote: IMDB has no API and is very slow. We have to parse each website to get the informations. It's not to much time for movies, but it needs many time to parse all data if you use IMDB for episodes.

For movies I use TMDB as first scraper and IMDB as second. In TMDB I've disabled Rating and in IMDB I've enabled only Rating and Top250.
For tv shows I've only enabled TVDB for data. TMDB has an API that is not very good for scraping many episodes. TMDB has a limit a 30 API calls in 10 seconds, then you will be paused for 30 seconds. If you rescrape a tv show like The Simpsons with more than 500 episodes... never ending story :-(

So setting the IMDB to scrape Rating and Top250 will mean Ember still needs to scan the entire page to find those words?

That's weird but why IMDB don't have API?

Thanks for the informative reply.
Reply
#4
Yes. But it does not make much difference if you get one or several data fields from IMDB. Most of all informations are on one webpage (e.g. http://www.imdb.com/title/tt0499549/combined ). Only for plot and trailers we have to download another page. The parsing itself is fast enough to neglect it.

But that only applies to movies. For episodes we have to call the tv show webpage, parse each season webpage link, download each season webpage to get each episode webpage link and then we have to download and parse each episode website to get the informations.

IMDB has no free API, only one for IMDB Pro members. $ 19.00 per month or $ 150.00 per year ;-)
Reply
#5
What do u enable on Scrapers - Data?

Can you post a screenshot
Reply
#6
Here's how I have them set, serves me well Smile

Image

Image
4x R-Pi4b LibreELEC v10 | Aeon Nox: SiLVO | Flirc cases
Storage Synology DS411 | 4 x WD RED 6TB
Software MariaDB 10.4.19 | Filebot | Ember Media Manager
wiki (wiki) | First time user (wiki) | Debug_Log (wiki) | mysql (wiki) | artwork (wiki)
Reply

Logout Mark Read Team Forum Stats Members Help
Enable more than one scraper or not?0