2017-06-29, 15:45
Musicbrainz server 503 errors (throttling issues)
Musicbrainz throttling (server 503 errors) does happen because of global server load, some other addons do make Musicbrainz requests, and during "Query info for all" Kodi does go rapidly through those albums that have already been scraped. But there is a longstanding flaw in the current scraper that means in some circumstances it also can make more than 1 request per sec to the Musicbrainz server.
One of those circumstances is having allmusic.com as one of the sources chosen in settings - an additional request is made of Musicbrainz to fetch the allmusic.com link for the album, and this request happens almost immediately after any lookup by album title and artist name, there is no delay. The current scraper 1s sleep only applies when doing a lookup by mbid (provided by music file tags).
Bottomline: using allmusic.com, or when music fies don't have Musicbrainz id tags, you will get IP adress related 503 errors when scraping either from "query info for all", or scanning many new items with "Fetch online info on update" enabled. It could take many repeat attempts to fill the resulting gaps in artwork and information. Kodi running on faster processors e.g. i7 rather than a RPi, will probably get more throttling (more 503 fails) because requests will be even closer together than on a slower processor..
It is something that I will attempt to fix as it involves core changes (not just an XML tweek) to catch every time the scraper is attempting to make a request to Musicbrainz.
(2017-02-24, 21:52)Merwenus Wrote: Musicbrainz:I replied
For example: if your requests are coming in at 4 requests per second, we don't honour 25% of them and decline the other 75% - we decline 100% of them, until the rate drops to 1 per second or lower.
And the scraper is an idiot, try to download 10/seconds instead of 1/1,5 seconds....
(2017-02-25, 00:12)DaveBlake Wrote: I don't think it is the scraper doing more than 1 per sec. Probably an addon that is hammering Musicbrainz.and
(2017-02-26, 10:02)DaveBlake Wrote: The scraper has a 1s sleep built into it, if you are sure there are more requests that that happening then it is from an addon.well I was wrong. Sorry about that Merwenus, and everyone else.
Musicbrainz throttling (server 503 errors) does happen because of global server load, some other addons do make Musicbrainz requests, and during "Query info for all" Kodi does go rapidly through those albums that have already been scraped. But there is a longstanding flaw in the current scraper that means in some circumstances it also can make more than 1 request per sec to the Musicbrainz server.
One of those circumstances is having allmusic.com as one of the sources chosen in settings - an additional request is made of Musicbrainz to fetch the allmusic.com link for the album, and this request happens almost immediately after any lookup by album title and artist name, there is no delay. The current scraper 1s sleep only applies when doing a lookup by mbid (provided by music file tags).
Bottomline: using allmusic.com, or when music fies don't have Musicbrainz id tags, you will get IP adress related 503 errors when scraping either from "query info for all", or scanning many new items with "Fetch online info on update" enabled. It could take many repeat attempts to fill the resulting gaps in artwork and information. Kodi running on faster processors e.g. i7 rather than a RPi, will probably get more throttling (more 503 fails) because requests will be even closer together than on a slower processor..
It is something that I will attempt to fix as it involves core changes (not just an XML tweek) to catch every time the scraper is attempting to make a request to Musicbrainz.
