Kodi Community Forum
Release Universal Scraper for Music Albums - Printable Version

Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: Music Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=303)
----- Thread: Release Universal Scraper for Music Albums (/showthread.php?tid=133547)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34


RE: [Release] Universal Scraper for Music Albums - meowmoo - 2017-02-26

(2017-02-26, 13:45)Merwenus Wrote: So you are saying this thing in Music area is ONLY a single request? Because I can clearly see in the LOG file and with my eyes when I click on it that on top of the screen albums appear/disappear pretty fast while searching for info.

Image

Music Library mode with universal scrapper, that means it has nothing to do with this scrapper? And it is some kind of internal addon of KODI?
Offtopic....
Is this aeonmq?


RE: [Release] Universal Scraper for Music Albums - Merwenus - 2017-02-26

Amber


RE: [Release] Universal Scraper for Music Albums - docwra - 2017-02-28

(2017-02-26, 15:10)Merwenus Wrote: Some albums are missing the THUMB picture. So I checked TheAudioDB.com maybe they are missing it so I upload them. But it says:
{Please Login to see images}

For example
http://www.theaudiodb.com/album/2159639
Image



How can I "login" to the scrapper to allow audiodb.com to download EVERY pictures? Because with a freshly made library I have way too much missing thumbnails.

You need to register on the TADB site and then images will show. You can then also upload images to the website so everyone benefits.


RE: [Release] Universal Scraper for Music Albums - docwra - 2017-02-28

(2017-02-25, 00:12)DaveBlake Wrote: I don't think it is the scraper doing more than 1 per sec. Probably an addon that is hammering Musicbrainz.

1 second isn't enough, we will hit the rate limiter after about 10 hits with that delay. It needs to be longer.

My tests show 2 seconds should be just about enough, but I default to 5 seconds on my personal MusicBrainz scripts to be sure.


RE: [Release] Universal Scraper for Music Albums - Merwenus - 2017-03-01

(2017-02-28, 17:14)docwra Wrote: You need to register on the TADB site and then images will show. You can then also upload images to the website so everyone benefits.

You missread something or I wasn't clear enough. I know the site, I used to upload images and fill informations. Thank you.

My problem is that in my Kodi Music Library, using Universal Scrapper, with settings to download and USE thumbnails from audiodb is broken, because it does not download a few images (mostly the ones you need to login). That is why I asked for help, so I can login to the scrapper, and download automatically these missing thumbs with my account.


RE: [Release] Universal Scraper for Music Albums - docwra - 2017-03-01

(2017-03-01, 10:23)Merwenus Wrote: My problem is that in my Kodi Music Library, using Universal Scrapper, with settings to download and USE thumbnails from audiodb is broken, because it does not download a few images (mostly the ones you need to login). That is why I asked for help, so I can login to the scrapper, and download automatically these missing thumbs with my account.

Ahh well the album you linked to is AC/DC so its probably the the forward slash causing the issue.

Try ACDC as the artist. That should work.


RE: [Release] Universal Scraper for Music Albums - Merwenus - 2017-03-01

Well, it finds the album, but does not download the thubmnail (it does when I do 1by1, but it stops working when I try to do mass download)
Check timestamps, it tries to download thumbs too fast and I guess I get banned, because after this the 1by1 download does not work anymore.

Log file:
Code:
15:58:51.940 T:7272 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer
16:17:10.864 T:1632 WARNING: Previous line repeats 22 times.
16:17:10.864 T:1632   ERROR: XFILE::CDirectory::GetDirectory - Error getting
16:17:11.068 T:7272   ERROR: Previous line repeats 13 times.
16:17:11.068 T:7272 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer
16:17:26.800 T:3456 WARNING: Previous line repeats 52 times.
16:17:26.800 T:3456   ERROR: ADDON::CScraper::Run: Unable to parse web site
16:17:28.913 T:7272 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer
16:18:01.466 T:6832 WARNING: Previous line repeats 100 times.
16:18:01.466 T:6832 WARNING: CScraperUrl::Get: Can't find precise charset for HTML "http://www.theaudiodb.com/api/v1/json/58424d43204d6564696120/album-mb.php?i=", using "UTF-8" as fallback
16:18:01.466 T:6832   ERROR: ADDON::CScraper::Run: Unable to parse web site
16:18:01.507 T:6832 WARNING: CScraperUrl::Get: Can't find precise charset for HTML "http://www.theaudiodb.com/api/v1/json/58424d43204d6564696120/album-mb.php?i=", using "UTF-8" as fallback
16:18:01.507 T:6832   ERROR: ADDON::CScraper::Run: Unable to parse web site
16:18:01.547 T:6832 WARNING: CScraperUrl::Get: Can't find precise charset for HTML "http://www.theaudiodb.com/api/v1/json/58424d43204d6564696120/album-mb.php?i=", using "UTF-8" as fallback
16:18:01.547 T:6832   ERROR: ADDON::CScraper::Run: Unable to parse web site
16:18:01.587 T:6832 WARNING: CScraperUrl::Get: Can't find precise charset for HTML "http://www.theaudiodb.com/api/v1/json/58424d43204d6564696120/album-mb.php?i=", using "UTF-8" as fallback
16:18:01.587 T:6832   ERROR: ADDON::CScraper::Run: Unable to parse web site
16:18:01.629 T:7272 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer



RE: [Release] Universal Scraper for Music Albums - docwra - 2017-03-01

Yep that log shows that the scraper is trying to request the TADB api with a NULL musicbrainz Album ID.

That probably means the lookup on the musicbrainz web service failed due to the flood limit.


RE: [Release] Universal Scraper for Music Albums - DaveBlake - 2017-03-01

(2017-03-01, 18:36)docwra Wrote: Yep that log shows that the scraper is trying to request the TADB api with a NULL musicbrainz Album ID.

That probably means the lookup on the musicbrainz web service failed due to the flood limit.

And therefore if you tagged with Musicbrainz ID tags you would not have this issue.

The skin.helper.service also does it's own Muisicbrainz lookup calls, I don't think that has any wait in it, so that is more likely to have caused the flood limit to be hit as the Universal scraper. What skin do you use @Merwenus? Do you have the skin.helper.service enabled?


RE: [Release] Universal Scraper for Music Albums - docwra - 2017-03-01

(2017-03-01, 19:02)DaveBlake Wrote: And therefore if you tagged with Musicbrainz ID tags you would not have this issue.

Yes I would hope so Smile But is it confirmed the scraper uses the MBID tag if present?

I remember from a while back it only looked up the name whatever the tags on the MusicBrainz server.


RE: [Release] Universal Scraper for Music Albums - meowmoo - 2017-03-01

(2017-03-01, 20:19)docwra Wrote:
(2017-03-01, 19:02)DaveBlake Wrote: And therefore if you tagged with Musicbrainz ID tags you would not have this issue.

Yes I would hope so Smile But is it confirmed the scraper uses the MBID tag if present?

I remember from a while back it only looked up the name whatever the tags on the MusicBrainz server.

I also think it did not read the tag.
When I rescrape an artist/album with universal scraper I don't get any information or art, but when I switch to theaudiodb scraper art and info shows up.
All my mp3s have a MBID

Haven't take a look at the log, so just guessing Smile


RE: [Release] Universal Scraper for Music Albums - Merwenus - 2017-03-02

(2017-03-01, 19:02)DaveBlake Wrote: And therefore if you tagged with Musicbrainz ID tags you would not have this issue.

The skin.helper.service also does it's own Muisicbrainz lookup calls, I don't think that has any wait in it, so that is more likely to have caused the flood limit to be hit as the Universal scraper. What skin do you use @Merwenus? Do you have the skin.helper.service enabled?

Amber skin and yes it has skin.helper.service enabled. What can I do to avoid this error?


RE: [Release] Universal Scraper for Music Albums - docwra - 2017-03-02

You can use a musicbrainz mirror server in the scraper settings

http://musicbrainz-mirror.eu:5000/

Is the only one live at the moment but its a bit overloaded.


RE: [Release] Universal Scraper for Music Albums - cliffx - 2017-06-25

I don't know if this had been answered throughout the thread, I haven't read it all, but if MB is presenting issues, can we just use other sources likes TheAudioDB instead?

I reinstalled kodi and I'm missing a lot of artist thumbs from the library.


RE: [Release] Universal Scraper for Music Albums - DaveBlake - 2017-06-25

The Universal Scraper for Music Albums (the subject of this thread) is based upon identifying the album using Musicbrainz, and so access to their servers is essential.

To outline how it works: If music files are tagged with Musicbrainz Album Id (release id), then idenitifaction is obvious, otherwise it does a lookup on album and artist. If a match is found (and 1st release will be taken if there are many) then it fetches artist credits and track data, which if you have "Prefer online info" enabled could be used to sync your lib with MB. Then it uses the release group id (something we don't currently fetch from tag but could) returned by MB to access TADB, and fanart.tv for other info and art. If present it uses any allmusic url returned from MB (they have links) to access data from allmusic.com

However could there be other music scrapers, that do not use MB, just up to someone to write them (in Python now, the scraper API has been extened to allow that). I think there is a TADB sepcific one, but I don't know if that uses just album title and artist, or has a quiet MB lookup tucked inside. But TADB also runs out of server resources too.

I think we could improve things by reducing the requests Kodi makes to MB (or any other server). For example if it had been easy (i.e. automatically happens) to store links to those missing thumbnails of yours, Kodi would not be trying to find them all again. Likewise we could store the release and release-group IDs. I am working on both, slowly because the scraper "regexp parsing of xml" side of things is a mystery (@olympia can you help?)

Otherwise a donation of more server power to the global comminity would help I guess - throw more hardware at it - but demand can be a bottomless pit.

EDIT: Just to add that IMO using MB to identify albums is a good idea, album title and artist is often not enough but if that is all we have then MB is the place to find it accurately


This forum uses Lukasz Tkacz MyBB addons.