The logic and future of Music scrapers?
#26
(2017-02-13, 19:24)jjd-uk Wrote: Also while reading this https://wiki.musicbrainz.org/XML_Web_Ser...e_Limiting I saw

Quote:For "python-musicbrainz/0.7.3": we allow through (on average) 50 requests per second, and decline the rest (though recently this has not been hit).

Don't if this might be any use in developments https://github.com/alastair/python-musicbrainzngs

we don't use that particular user-agent.
the scraper addon sets a, what they call, 'meaningful User-Agent'.
they don't apply throttling based on user-agent in this case.

we'll still hit the 1 call per second based on ip address though.

(2017-02-13, 20:01)DaveBlake Wrote: Not sure when that Musicbrainz quote dates from, but there were major issues last year, and again around 17.0 release time, even 1 request per sec was getting timeouts (as well as TADB being off line completely).

if musicbrainz is under heavy load, they throttle regardless of what time-out value you use i think:
Quote:We allow through 300 requests each second (on average), and decline (http 503) the rest.


while testing things i found out there is a very popular skin helper addon in our repo that is also making a lot of calls to musicbrainz
when you are scraping your music collection.
if you have this addon installed (it's a dependency of many skins) you're very likely to end up with many items that failed to scrape (due to throttling),
as the addon is basically doubling up the number of calls to musicbrainz.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply


Messages In This Thread
RE: The logic of Music scrapers? - by ronie - 2017-02-08, 00:14
RE: The logic of Music scrapers? - by jjd-uk - 2017-02-08, 11:23
RE: The logic of Music scrapers? - by ronie - 2017-02-13, 03:04
RE: The logic of Music scrapers? - by ronie - 2017-02-13, 03:12
RE: The logic of Music scrapers? - by ronie - 2017-02-13, 03:28
RE: The logic and future of Music scrapers? - by ronie - 2017-02-14, 04:16
Logout Mark Read Team Forum Stats Members Help
The logic and future of Music scrapers?0