Bug Ember Automatic Scraping stops, doesn't work, needs to restart
#1
Next to the problem described here, I also run into a different problem when scraping my large list of movies (2500 movies). Whenever I do a full scrape with TMDB (I need that one because it has german descriptions, plots and outlines), it is not only extremely slow but also stops working after several entries. There is no error message or anything, it simply stops working. When I try to restart scraping (even with a single entry) it simply doesnt work, I don't get any content.

First of all I thought TMDB has a limit in its API so people can only scrape a certain amount of movies each hour. Though, when I close Ember, restart it and restart scraping, it works again. Now, the whole problem is, that only about 10-30 movies get scraped, this would make me restart Ember about 125 times until I am finished and that sounds like a big pain -_-

I am using BETA Ember 1.4 - 20140110 btw, I also uploaded the log file to my DropBox if this helps somehow. I really hope a solution can be found because I am very frustrated at the moment.

Image

Link: https://dl.dropboxusercontent.com/u/5462...-01-17.csv

I also noticed that if I do not close Ember, the CSV gets larger and larger and larger... something strange is going on...

Thanks for yout work and help,
best regards
Torm
Reply
#2
I think the problem is not due to the number of requests.

Quote:Request Rate Limiting

We do enforce a small amount of rate limiting. Please be aware that should you exceed these limits, you will receive a 503 error.

- 30 requests every 10 seconds per IP
- Maximum 20 simultaneous connections per IP

I have checked your log: it's always the same error:
Quote:Invalid id - The pre-requisite id is invalid or not found
6

I have no idea what that might be Confused

I will also test it tomorrow with my database (900 movies) and again report.
Reply
#3
Ok, i think i have fixed most of the issues while scraping. Only one last error while scraping my 900 movies.

Please try again with next beta (release in 1-2 days).
Reply
#4
Thanks alot, looking forward to this Smile
Reply

Logout Mark Read Team Forum Stats Members Help
Ember Automatic Scraping stops, doesn't work, needs to restart0