Kodi Community Forum
Release Universal Scraper for Music Albums - Printable Version

Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: Music Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=303)
----- Thread: Release Universal Scraper for Music Albums (/showthread.php?tid=133547)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34


RE: [Release] Universal Scraper for Music Albums - Martijn - 2012-06-21

(2012-06-21, 13:26)Midnight Tboy Wrote: sorry....just to answer my own question

"Your audio files MUST have a valid ID tag for them to work properly in the XBMC music library" from the wiki.

Damn....guess I better find a good id3 tag mp3 manager for the pc

I recommend:
MusicBrainz Picard
TagScanner


RE: [Release] Universal Scraper for Music Albums - Midnight Tboy - 2012-06-21

Thanks - trying out Picard right now - seems to mess some folders up, like trying to seperate soundtrack albums into about half a dozen different albums, but will play til get it right Smile

Will try TagScanner out next


RE: [Release] Universal Scraper for Music Albums - Martijn - 2012-06-21

(2012-06-21, 14:02)Midnight Tboy Wrote: Thanks - trying out Picard right now - seems to mess some folders up, like trying to seperate soundtrack albums into about half a dozen different albums, but will play til get it right Smile

Will try TagScanner out next

Drag one album in. Select the files and click on "cluster".
This will make sure it searches for an album that contains all of them


RE: [Release] Universal Scraper for Music Albums - Midnight Tboy - 2012-06-21

thanks - though when I select Cluster, it seems to do nothing.

example pain album

10 Things I Hate About You - OST

all files are stupidly called

10 Things I Hate About You - OST - t01
10 Things I Hate About You - OST - t02, etc
with no id3 tag info in.

I drag them all in, select Cluster but they still stay under "Unmatched Files", as if the button does nothing

Then when I "scan" It splits them into 8 albums, 6 of them being in the correct album. Fun and games Smile Not looking forward to sorting this for hundreds of albums lol.....argh

anyways, sorry for going off topic


RE: [Release] Universal Scraper for Music Albums - McB83 - 2012-06-22

I have a problem with scraping the album rating from allmusic.com. Every album gets a rating of 2 stars, however the album rating on allmusic.com is higher. For example this album has 3.5 stars, and this one has 4 stars.


RE: [Release] Universal Scraper for Music Albums - Plecebo - 2012-06-23

(2012-06-10, 16:24)night199uk Wrote: hey olympia.

this seems to over run musicbrainz query/sec limit for me on a semi-regular basis. the problem is, once the qps limit kicks in musicbrainz start serving up a really simple fast reject page, which means the queries go even faster and the rate-limit just stays in force. so i have to stop scanning and restart.

i think unfortunately the real solution is a rate-limit per domain, and nothing we can do in the scraper. :-(

I had a similar issue with other services that I run and the solution was to setup a local musicbrainz mirror in my network. That way you can make several thousand queries a minute without hitting any rate limit. It would be great if there was an easy way to set musicbrainz mirror settings, host and port are the usual options, in the Universal Scraper options?



RE: [Release] Universal Scraper for Music Albums - Martijn - 2012-06-24

(2012-06-21, 15:11)Midnight Tboy Wrote: I drag them all in, select Cluster but they still stay under "Unmatched Files", as if the button does nothing

Then when I "scan" It splits them into 8 albums, 6 of them being in the correct album. Fun and games Smile Not looking forward to sorting this for hundreds of albums lol.....argh

anyways, sorry for going off topic

You should hit "search" and not "scan".
Scan will use the audio fingerprint to match the audio files



RE: [Release] Universal Scraper for Music Albums - Zippolighter - 2012-06-25

Hi, I see a problem with the scraper.
I try to scrape Maroon 5 - Overexposed to the db, but the title list shows only one song Nr. 2 with titel and runtime. All other titles are missing.

If I try the url that use in the log, I get the complete title list, the runtime is missing on all songs.

[code]05:16:11 T:4964 DEBUG: ------ Window Init (DialogProgress.xml) ------
05:16:11 T:4816 DEBUG: Thread MUSIC_GRABBER::CMusicInfoScraper start, auto delete: 0
05:16:11 T:4816 DEBUG: ADDON::CScraper::FindAlbum: Searching for 'Maroon 5 - Overexposed' using Universal Album Scraper scraper (path: 'C:\Users\Andreas\AppData\Roaming\XBMC\addons\metadata.album.universal', content: 'albums', version: '1.1.1')
05:16:11 T:4816 DEBUG: scraper: CreateAlbumSearchUrl returned <url>http://search.musicbrainz.org/ws/2/release/?fmt=xml&query=release:"Overexposed"%20AND%20artist:"Maroon%205"</url>
05:16:11 T:4816 DEBUG: FileCurl::Open(002BCB30) http://search.musicbrainz.org/ws/2/release/?fmt=xml&query=release:"Overexposed"%20AND%20artist:"Maroon%205"
05:16:11 T:4816 INFO: XCURL:Big GrinllLibCurlGlobal::easy_aquire - Created session to http://search.musicbrainz.org
05:16:12 T:4816 ERROR: PCRE: missing ). Compilation failed at offset 933 in expression 'id="([^"]*)"><title>([^<]*)</title><status>Official</status><text-representation><language>[^<]*</language><script>[^<]*</script></text-representation><artist-credit><name-credit(?:>)*(?:\sjoinphrase="[^"]*">)*(?:<name>[^<]*</name)*(?:>)*<artist\sid="[^"]*"><name>([^<]*)</name>(?:<sort-name>[^<]*</sort-name>)*(?:<disambiguation>[^<]*</disambiguation>)*(?:<alias-list>.*?</alias-list>)*</artist></name-credit>(?:<name-credit(?:>)*(?:\sjoinphrase="[^"]*">)*(?:<name>[^<]*</name)*(?:>)*<artist\sid="[^"]*"><name>[^<]*</name>(?:<sort-name>[^<]*</sort-name>)*(?:<disambiguation>[^<]*</disambiguation>)*(?:<alias-list>.*?</alias-list>)*</artist></name-credit>)*</artist-credit><release-group(?:\stype="Album"\sid="[^"]*">(?:<primary-type>[^<]*</primary-type>)*(?:<secondary-type-list><secondary-type>[^<]*</secondary-type></secondary-type-list>)*</release-group><date>(\d{4})[^<]*</date>(?:<country>)*([^<]*)?.*?<track-list\scount="(\d+)'
05:16:12 T:4816 DEBUG: scraper: GetAlbumSearchResults returned <results><entity><year>XW-2012-T#17</year><artist>Maroon 5</artist><title>Overexposed</title><url cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></entity><entity><year>GB-2012-T#13</year><artist>Maroon 5</artist><title>Overexposed</title><url cache="mb-f9aaa857-7567-4dc1-afca-086601e1db95-album.xml">http://musicbrainz.org/ws/2/release/f9aaa857-7567-4dc1-afca-086601e1db95?inc=recordings+release-groups+artists+labels+ratings</url></entity></results>
05:16:12 T:4816 DEBUG: Thread MUSIC_GRABBER::CMusicInfoScraper 4816 terminating
05:16:12 T:4692 DEBUG: Thread MUSIC_GRABBER::CMusicInfoScraper start, auto delete: 0
05:16:12 T:4692 DEBUG: ADDON::CScraper::GetAlbumDetails: Reading 'http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings' using Universal Album Scraper scraper (file: 'C:\Users\Andreas\AppData\Roaming\XBMC\addons\metadata.album.universal', content: 'albums', version: '1.1.1')
05:16:12 T:4692 DEBUG: FileCurl::Open(002BCB30) http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings
05:16:13 T:4692 DEBUG: scraper: GetAlbumDetails returned <details><chain function="GetMBAlbumTitleByMBID">c870093e-5d68-4842-a663-291701678229</chain><chain function="GetMBAlbumArtistByMBID">c870093e-5d68-4842-a663-291701678229</chain><chain function="GetMBAlbumLabelByMBID">c870093e-5d68-4842-a663-291701678229</chain><chain function="GetMBAlbumDateByMBID">c870093e-5d68-4842-a663-291701678229</chain><chain function="GetMBAlbumTracksByMBID">c870093e-5d68-4842-a663-291701678229</chain><chain function="GetLastFMAlbumReviewByAlbumAndArtist">Overexposed::Maroon 5::Overexposed::Maroon 5</chain><chain function="GetMBAlbumRatingByMBID">c870093e-5d68-4842-a663-291701678229</chain><url function="GetAMGAlbumStyles" cache="mb-ed3c55b6-e49d-4d4e-a338-62b823b1e2d4-rg.xml">http://musicbrainz.org/ws/2/release-group/ed3c55b6-e49d-4d4e-a338-62b823b1e2d4?inc=url-rels</url><url function="GetAMGAlbumMoods" cache="mb-ed3c55b6-e49d-4d4e-a338-62b823b1e2d4-rg.xml">http://musicbrainz.org/ws/2/release-group/ed3c55b6-e49d-4d4e-a338-62b823b1e2d4?inc=url-rels</url><url function="GetAMGAlbumThemes" cache="mb-ed3c55b6-e49d-4d4e-a338-62b823b1e2d4-rg.xml">http://musicbrainz.org/ws/2/release-group/ed3c55b6-e49d-4d4e-a338-62b823b1e2d4?inc=url-rels</url><chain function="GetFanartTvAlbumThumbsByMBID">ed3c55b6-e49d-4d4e-a338-62b823b1e2d4</chain><url function="GetAMGAlbumThumbs" cache="mb-ed3c55b6-e49d-4d4e-a338-62b823b1e2d4-rg.xml">http://musicbrainz.org/ws/2/release-group/ed3c55b6-e49d-4d4e-a338-62b823b1e2d4?inc=url-rels</url><chain function="GetLastFMAlbumThumbs">Overexposed::Maroon 5::Overexposed::Maroon 5</chain></details>
05:16:13 T:4692 DEBUG: scraper: GetMBAlbumTitleByMBID returned <details><url function="ParseMBAlbumTitle" cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></details>
05:16:13 T:4692 DEBUG: scraper: ParseMBAlbumTitle returned <details><title>Overexposed</title></details>
05:16:13 T:4692 DEBUG: scraper: GetMBAlbumArtistByMBID returned <details><url function="ParseMBAlbumArtist" cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></details>
05:16:13 T:4692 DEBUG: scraper: ParseMBAlbumArtist returned <details><artist>Maroon 5</artist></details>
05:16:13 T:4692 DEBUG: scraper: GetMBAlbumLabelByMBID returned <details><url function="ParseMBAlbumLabel" cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></details>
05:16:13 T:4692 ERROR: ADDON::CScraper::Run: Unable to parse web site
05:16:13 T:4692 DEBUG: scraper: GetMBAlbumDateByMBID returned <details><url function="ParseMBAlbumDate" cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></details>
05:16:13 T:4692 DEBUG: scraper: ParseMBAlbumDate returned <details><releasedate>2012-06-25</releasedate><year>2012</year></details>
05:16:13 T:4692 DEBUG: scraper: GetMBAlbumTracksByMBID returned <details><url function="ParseMBAlbumTracks" cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></details>
05:16:13 T:4692 DEBUG: scraper: ParseMBAlbumTracks returned <details><track><position>2</position><title>Payphone</title><duration>231</duration></track></details>
05:16:13 T:4692 DEBUG: scraper: GetLastFMAlbumReviewByAlbumAndArtist returned <details><url function="ParseLastFMAlbumReview" cache="lastfmalbum-Maroon 5-Overexposed.xml">http://ws.audioscrobbler.com/2.0/?method=album.getinfo&artist=Maroon%205&amp;album=Overexposed&amp;api_key=71e468a84c1f40d4991ddccc46e40f1b&lang=en</url></details>
05:16:13 T:4692 DEBUG: FileCurl::Open(002BCB30) http://ws.audioscrobbler.com/2.0/?method=album.getinfo&artist=Maroon%205&album=Overexposed&api_key=71e468a84c1f40d4991ddccc46e40f1b&lang=en
05:16:13 T:4692 ERROR: ADDON::CScraper::Run: Unable to parse web site
05:16:13 T:4692 DEBUG: scraper: GetMBAlbumRatingByMBID returned <details><url function="ParseMBAlbumRating" cache="mb-c870093e-5d68-4842-a663-291701678229-album.xml">http://musicbrainz.org/ws/2/release/c870093e-5d68-4842-a663-291701678229?inc=recordings+release-groups+artists+labels+ratings</url></details>
05:16:13 T:4692 DEBUG: scraper: ParseMBAlbumRating returned <details><rating>5</rating></details>
05:16:13 T:4692 DEBUG: FileCurl::Open(002BCB30) http://musicbrainz.org/ws/2/release-group/ed3c55b6-e49d-4d4e-a338-62b823b1e2d4?inc=url-rels
05:16:14 T:4692 DEBUG: scraper: GetAMGAlbumStyles returned <details><chain function="GetAMGAlbumStylesByAMGID">mw0002358095</chain></details>
05:16:14 T:4692 DEBUG: scraper: GetAMGAlbumStylesByAMGID returned <details><url cache="am-mw0002358095-album.html" function="ParseAMGAlbumStyles">http://www.allmusic.com/album/mw0002358095</url></details>
05:16:14 T:4692 DEBUG: FileCurl::Open(002BCB30) http://www.allmusic.com/album/mw0002358095
05:16:16 T:4692 DEBUG: scraper: ParseAMGAlbumStyles returned <details><style>Adult Alternative Pop/Rock</style><style>Alternative/Indie Rock</style><style>Contemporary Pop/Rock</style><style>Pop</style></details>
05:16:16 T:4692 DEBUG: scraper: GetAMGAlbumMoods returned <details><chain function="GetAMGAlbumMoodsByAMGID">mw0002358095</chain></details>
05:16:16 T:4692 DEBUG: scraper: GetAMGAlbumMoodsByAMGID returned <details><url cache="am-mw0002358095-album.html" function="ParseAMGAlbumMoods">http://www.allmusic.com/album/mw0002358095</url></details>

Thanks
Zippolighter


RE: [Release] Universal Scraper for Music Albums - Zippolighter - 2012-06-25

I found another issue,
I scrape Die Toten Hosen - Ballast Der Republik.

The scraper add a 2Disc Variante to the db, that was fine. But of CD 2 the titles 3 and 8 are missing. Both are available on Musicbrainz Webside

Zippolighter


RE: [Release] Universal Scraper for Music Albums - solidsatras - 2012-06-25

@Zippolighter
(2012-06-24, 15:06)Martijn Wrote: How to post Debug Log?: Use XBMC log uploader



RE: [Release] Universal Scraper for Music Albums - olympia - 2012-06-25

(2012-06-25, 07:43)solidsatras Wrote: @Zippolighter
(2012-06-24, 15:06)Martijn Wrote: How to post debug log?: Use XBMC log uploader

Strange that he still does this given his 326 posts....


RE: [Release] Universal Scraper for Music Albums - olympia - 2012-06-25

(2012-06-25, 05:27)Zippolighter Wrote: Hi, I see a problem with the scraper.
I try to scrape Maroon 5 - Overexposed to the db, but the title list shows only one song Nr. 2 with titel and runtime. All other titles are missing.
Fixed in v1.1.2

(2012-06-25, 07:07)Zippolighter Wrote: I found another issue,
I scrape Die Toten Hosen - Ballast Der Republik.

The scraper add a 2Disc Variante to the db, that was fine. But of CD 2 the titles 3 and 8 are missing. Both are available on Musicbrainz Webside
Fixed in v1.1.3 (however please note, that track duration won't be scraped for such tracks)


RE: [Release] Universal Scraper for Music Albums - McB83 - 2012-06-25

(2012-06-22, 23:06)McB83 Wrote: I have a problem with scraping the album rating from allmusic.com. Every album gets a rating of 2 stars, however the album rating on allmusic.com is higher. For example this album has 3.5 stars, and this one has 4 stars.

I've inspected the debug logs and the rating is scraped correctly from allmusic.com. However when I export my music library I notice the following. Albums scraped with the deprecated allmusic.com scraper have a rating between 1-10. Albums scraped with the new universal scraper have a rating between 1-5. So my assumption is that XBMC works with a different rating then allmusic.com. The scraped rating from allmusic.com must be multiplied with 2 to get the correct rating in XBMC. I can provide debug logs if necessary.


RE: [Release] Universal Scraper for Music Albums - olympia - 2012-06-25

(2012-06-25, 11:14)McB83 Wrote:
(2012-06-22, 23:06)McB83 Wrote: I have a problem with scraping the album rating from allmusic.com. Every album gets a rating of 2 stars, however the album rating on allmusic.com is higher. For example this album has 3.5 stars, and this one has 4 stars.
So my assumption is that XBMC works with a different rating then allmusic.com. The scraped rating from allmusic.com must be multiplied with 2 to get the correct rating in XBMC. I can provide debug logs if necessary.

...or allmusic.com changed their rating system during the site re-design.
In any case, something is broken in XBMC yes: as you've noticed in the log yourself the scraper scrapes the [1-5] rating correctly, but XBMC even divide this by 2 (e.g a scraped 4 gets 2 at the end).


RE: [Release] Universal Scraper for Music Albums - Zippolighter - 2012-06-25

Thanks olympia for the very fast solution

Zippolighter


This forum uses Lukasz Tkacz MyBB addons.