Kodi Community Forum

Full Version: Universal Scraper for Music Albums
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
They are updated every day I believe. But there was a recent schema change at musicbrainz and none of them are currently replicating due to it.

The MusicBrainz-Mirror guy is going to look at it this weekend apparently to restore regular syncing.
Cool, this is great news. Thanks!
It looks like no success yet on the EU one. Any other mirror succeeded to address the schema change?
Nope still waiting. I've asked in the metabrainz channel on irc too, but no luck Sad
For someone that doesn't scrape much, what are the consequnces for the user of the schema changes/MB server mirrors issue? What does the user experience, what messages do we see in the debug log? In other words how can we spot a user is having this issue, not some other bug?

It would be useful to have a simple reply that explains what the problem is too
(2016-06-19, 09:47)DaveBlake Wrote: [ -> ]For someone that doesn't scrape much, what are the consequnces for the user of the schema changes/MB server mirrors issue? What does the user experience, what messages do we see in the debug log? In other words how can we spot a user is having this issue, not some other bug?

It would be useful to have a simple reply that explains what the problem is too

I have no idea about the discussion about the schema changes as schema it itself should NOT impact scraping. Certainly there can be (still) bugs after schema changes, but the scraper should be more protected with the latest updates and handle these changes more gracefully.

As for the mirror/ limiter issue you can see the issue easily using the following example url:
http://musicbrainz.org/ws/2/artist/5dcdb...c=url-rels

Just fire it up in a browser and do some frequent refreshes... you will be hit by the limiter issue quite quickly.

In such cases in Kodi the user get a message: "Download artist information failed" or "Download album information failed".

In the logs it shows:
Code:
09:59:25 T:1816   DEBUG: CurlFile::Open(0ACB0BA0) http://musicbrainz.org/ws/2/artist/72c536dc-7137-4477-a521-567eeb840fa8?inc=url-rels
09:59:26 T:1816   ERROR: CCurlFile::FillBuffer - Failed: HTTP returned error 503
09:59:26 T:1816   ERROR: CCurlFile::Open failed with code 503 for http://musicbrainz.org/ws/2/artist/72c536dc-7137-4477-a521-567eeb840fa8?inc=url-rels
09:59:26 T:1816   ERROR: ADDON::CScraper::Run: Unable to parse web site
Thanks Olympia. I'm picking up user support in "music corner" with complaints about album and artist art not appearing. This could be down to a number of things, and I just wanted it clear in my mind which things are server related rather than bugs I could chase.

No worries, the scraper is being graceful Smile But of course if there are problems at online sites it is unable to make the images magically appear, and users notice and complain.
As one of the first steps we should protect backends from being spammed by Kodi according to our recent discussion... Tongue Smile
Following that they might be able to have their limiter less restricted.
(2016-06-19, 10:25)olympia Wrote: [ -> ]As one of the first steps we should protect backends from being spammed by Kodi according to our recent discussion... Tongue Smile
Following that they might be able to have their limiter less restricted.

Yes, indeed Smile
I know that Kodi has made the situation worse, and it is on my work list. Meanwhile trying to separate the effects of server overload from the other things that users experience.
This is not a scraper issue, its a musicbrainz issue.

They insist on changing their schema then taking months to update their virtual image (which all mirrors use). The result is all the mirrors stopped replicating on the 22nd of last month.

Their main API is at 100% capacity and has been pretty useless for about 6 months. They are working on it, but I don't know the status, I even wrote them a blog here to try and help.

If they actually built a Virtual Image each time they updated their schema then I would run a Kodi mirror myself and we could move to it by default. I'm pretty sure that would go a long way to fixing their capacity issues.
Any idea whether there is any way to encourage the MBZ developers to update the virtual image or some other way to get the mirror working again?

It has been over a month now that music scrapers are not functinonal.

Would donations help?
Good question. Can someone that understands the schema issue start a dialog on the Musicbrainz forum. @zag perhaps?

Meanwhile trying to scrape even albums that have mbids in the album table results in 403 errors when using a mirror and 503 when using main server. Since the actual metadata comes from other databases than Musicbrainz could it be stepped around in some way? I don't understand the scraping process, so forgive me if that is a silly question.

From a user viewpoint music scraping is not functioning, and that does deminish the user experience.
You can ask in their irc channel. When they do I will look at running a mirror.
If any of you were using unRAID, you could easily run your own mirror Smile
Would this mirror work? The public mirrors stopped working a month ago since some change in the MBZ structure. To quote @zag: "They insist on changing their schema then taking months to update their virtual image (which all mirrors use). The result is all the mirrors stopped replicating on the 22nd of last month.".
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34