2012-06-23, 23:27
(2012-06-10, 16:24)night199uk Wrote: hey olympia.
this seems to over run musicbrainz query/sec limit for me on a semi-regular basis. the problem is, once the qps limit kicks in musicbrainz start serving up a really simple fast reject page, which means the queries go even faster and the rate-limit just stays in force. so i have to stop scanning and restart.
i think unfortunately the real solution is a rate-limit per domain, and nothing we can do in the scraper. :-(
I had a similar issue with other services that I run and the solution was to setup a local musicbrainz mirror in my network. That way you can make several thousand queries a minute without hitting any rate limit. It would be great if there was an easy way to set musicbrainz mirror settings, host and port are the usual options, in the Universal Scraper options?
