2015-03-28, 14:30
(2015-03-28, 13:49)richarddscott76 Wrote: Hi, I am having a bit of a problem, I needed to run a re-scrape on all of my .nfo files but for some reason the scraper for 1.4 seems to be incredibly slow and is taking around 20 seconds per movie.which may not sound all that bad but when you have 3000 movies to scrape thats over 16 hours of scraping. 1.3 was scraping much much faster than this. Am I doing something wrong or is there a setting I need to change to speed things up?
How you do scrape? If I want to only update NFO I do a NFO only scrape. I just did some test: I have TMDB Data scraper first then IMDB (all fields activated). I hit scrape while I was downloading ouside of Ember --> took over 30seconds! I then canceled my download outside Ember and started scraping process again: Now around 11s with all fields in IMDB/TMDB activated. Then I looked in the log and noticed that TMDB scraper only took 2s but IMDB around 9seconds! So now I deactivated IMDB and only TMDB DataScraper was activated -> Bingo! 3seconds per movie at NFO only! I can't disable IMDB though, because I need that scraper as a fallback source for missing fields on TMDB. So now I activated IMDB again and disabled some fields to check if theres a field that took long to scrape. In my tests it was the trailer field in IMDB Data scraper. So I have now all fields checked in IMDBData Scraper but "trailer" and now duration for a NFO scraper is around 4seconds - which is acceptable!
To sum it up, I would suggest to disable the trailer field in IMDB Datascraper and try to scrape 1,2 movies and check time. Maybe it also works for you?
