Kodi Community Forum
Release Universal Scraper for Music Albums - Printable Version

Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: Music Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=303)
----- Thread: Release Universal Scraper for Music Albums (/showthread.php?tid=133547)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34


RE: [Release] Universal Scraper for Music Albums - DaveBlake - 2018-01-11

(2018-01-11, 16:28)Epg1 Wrote: What can I do, except doing a *.nfo file, to solve this problem?
 It is a good example of why using individual files per tack (not cuesheets) and fully tagging with Musicbrainz IDs is a good idea.

The cuesheet standard does not supply a means to specify all Musicbrainz IDs, and Kodi does not process partial Musicbrainz tags (just having the Musicbrainz album ID in the single file is not helping you).  Musicbrainz integration was simply not designed to work with single file and cuesheets. So the alternative to an album.NFO file is to split to individual tracks and tag.


RE: [Release] Universal Scraper for Music Albums - Boulder - 2018-01-12

(2018-01-10, 09:43)DaveBlake Wrote: 1) I think it is that the scraper needs to add the max rating from the data source e.g. for 5 for Allmusic to the interim xml it creates, otherwise Kodi assumes it is the default max of 10. Something that we need @olympia to deal with.

2) There is a bug in "Change all" that is fixed for Leia (actually the whole way music scraper settings are selectively applied has been reworked, and made it easier to set the default). But I would have expected changing the addon settings, adding source again etc. to have worked. I'll see if I can repeat this.

But also Boulder could you check what version of the addon you have installed.
 The version is 2.7.3. Do you need a Trac ticket for item 1?


RE: [Release] Universal Scraper for Music Albums - olympia - 2018-01-12

No


RE: [Release] Universal Scraper for Music Albums - DaveBlake - 2018-01-12

(2017-09-17, 18:59)Boulder Wrote: I have two issues with the addon:
1) If I use allmusic.com to fetch ratings, the points are between 1-5 and so the number of stars shown is max 2,5 in Krypton.
Having done some testing I can not repeat this issue. I set allmusic as the rating source and see ratings > 5 both in the DB and on screen. Sorry Olympia if I sent you off on a wild goose chase!

You @Boulder you mention "number of stars" not a number, what skin are you using? I am wondering if the issue is the skin translation of number to stars. Switch to say Confluence (something that shows a number on album info dialog, Estuary doesn't show much at all) are the numbers wrong?
 
(2017-09-17, 18:59)Boulder Wrote: 2) I'm unable to change the rating to be fetched from Musicbrainz. I tried removing my music source, cleaned the library, changed in the addon settings that Musicbrainz is used and readded the music source to scan. Allmusic.com was set for all albums Sad
I can't reproduce that either. Changing the addon settings, and scraping music that has not been scraped before (and does not have album.nfo files) picks up the rating from the source in the scraper settings.

I can only wonder if you have album.nfo files?
(2017-09-17, 18:59)Boulder Wrote: If I try to change the information provider in the album list, it only affects the selected album and not all of my albums (even though I confirmed that I want to change all).
To clarify, yes that part is the known  bug.


RE: [Release] Universal Scraper for Music Albums - Boulder - 2018-01-12

(2018-01-12, 11:00)DaveBlake Wrote:
(2017-09-17, 18:59)Boulder Wrote: I have two issues with the addon:
1) If I use allmusic.com to fetch ratings, the points are between 1-5 and so the number of stars shown is max 2,5 in Krypton.
Having done some testing I can not repeat this issue. I set allmusic as the rating source and see ratings > 5 both in the DB and on screen. Sorry Olympia if I sent you off on a wild goose chase!

You @Boulder you mention "number of stars" not a number, what skin are you using? I am wondering if the issue is the skin translation of number to stars. Switch to say Confluence (something that shows a number on album info dialog, Estuary doesn't show much at all) are the numbers wrong?
 
(2017-09-17, 18:59)Boulder Wrote: 2) I'm unable to change the rating to be fetched from Musicbrainz. I tried removing my music source, cleaned the library, changed in the addon settings that Musicbrainz is used and readded the music source to scan. Allmusic.com was set for all albums Sad
I can't reproduce that either. Changing the addon settings, and scraping music that has not been scraped before (and does not have album.nfo files) picks up the rating from the source in the scraper settings.

I can only wonder if you have album.nfo files?
(2017-09-17, 18:59)Boulder Wrote: If I try to change the information provider in the album list, it only affects the selected album and not all of my albums (even though I confirmed that I want to change all).
To clarify, yes that part is the known  bug.  
 There may have been changes in allmusic.com at some point because now it produces the correct rating (on a scale 1-10). Musicbrainz however seems to have only 1-5 (tested with Uriah Heep's "Very 'eavy...very 'umble"). Tested with both Titan and Confluence.

So after the known bug is fixed, is it possible to force refresh of data of all albums? I was thinking of a situation where there are some missing ratings or new albums which don't have a reliable rating yet.

EDIT: also, I wonder why that album doesn't get any rating from allmusic.com? I can find it using the web page but no rating. The catalog number I have tagged with Picard is CLACD 105.


RE: [Release] Universal Scraper for Music Albums - olympia - 2018-01-12

(2018-01-12, 12:01)Boulder Wrote: EDIT: also, I wonder why that album doesn't get any rating from allmusic.com? I can find it using the web page but no rating. The catalog number I have tagged with Picard is CLACD 105. 

Because it has no external link recorded to allmusic.com on MB.
By the way, what do you mean by doing a *.nfo file? You mean a full nfo?
 - if yes, I have a good news: you can create an album.nfo and throw only the MB release link of a specific album into it what will force the scraper to get info for that one
 - if even that is too much of a hassle, then I don't have any solution, other than the one mentioned by DaveBlake.

@DaveBlake no worries. Sorry for not having a response earlier. I remembered that the rating scale is included in the scraper, but wanted to do some tests before replying, just didn't have the time yet...


RE: [Release] Universal Scraper for Music Albums - Boulder - 2018-01-12

(2018-01-12, 15:32)olympia Wrote:
(2018-01-12, 12:01)Boulder Wrote: EDIT: also, I wonder why that album doesn't get any rating from allmusic.com? I can find it using the web page but no rating. The catalog number I have tagged with Picard is CLACD 105. 

Because it has no external link recorded to allmusic.com on MB.
By the way, what do you mean by doing a *.nfo file? You mean a full nfo?
 - if yes, I have a good news: you can create an album.nfo and throw only the MB release link of a specific album into it what will force the scraper to get info for that one
 - if even that is too much of a hassle, then I don't have any solution, other than the one mentioned by DaveBlake.

@DaveBlake no worries. Sorry for not having a response earlier. I remembered that the rating scale is included in the scraper, but wanted to do some tests before replying, just didn't have the time yet... 

Thanks, I'll try adjusting those troublesome cases with a manual .nfo file. Some albums have a zillion different releases so they don't hit AMG it seems.

One more question: does the AMG API return only the "official" AllMusic rating or is the user rating included as well?


RE: [Release] Universal Scraper for Music Albums - olympia - 2018-01-12

Please register on MB and add the allmusic link instead. This way others will benefit as well. AMG link is on release group level, not on release level, so only need to be added once, not for all the zillion releases.
If I were you, I would only add the link to album.nfo, instead of the full and making sure the sites to scrape from have the information I want (except AMG as you have no control over there).

There is no AMG API, the scraper scrapes the website.


RE: [Release] Universal Scraper for Music Albums - Boulder - 2018-01-12

Yes, I'll do that. It took a while to figure out what to do, but I think I got it. Apparently I need to query the music database to easily find out which albums don't have a rating, so time to get those tools installed Smile


RE: [Release] Universal Scraper for Music Albums - olympia - 2018-01-12

I could actually add an option to scrape the AMG user rating instead of the "official". Question is how much this AMG thing is actually futureproof. I (and I think we all) much more prefer, community driven, open databases.
However, for ratings it's always "tricky" as those popular websites both for video and music have much much more voter than the open databases.


RE: [Release] Universal Scraper for Music Albums - Boulder - 2018-01-12

Never mind, it seems you replied while I was writing and testing Blush But yes, user rating would be much more accurate, being an average opinion..

It seems to be working. It looks a bit funny though, that album gets a rating of 5/10 from AMG although visually 3/5 stars are shown. In that sense, it would be better to use the user ratings as they are the more average opinion and usually the albums are rated by people who know the artist better. Would it be possible to add that as an option?


RE: [Release] Universal Scraper for Music Albums - docwra - 2018-01-12

Just for info I am thinking about grabbing all the AllMusic website user and official ratings, then adding them to TheAudioDB.

I already did this for rateyourmusic.com and metacritic.com ratings.

You could then use the TADB rating sync Add-on.

I just thought I would mention it as those ratings would be there forever and not require any scraping of websites Wink


RE: [Release] Universal Scraper for Music Albums - olympia - 2018-01-12

hey zag, cannot follow. Those ratings would be _where_ forever? What do you mean?

btw, I just looked at allmusic to see how quickly I could add option to scrape user ratings, but actually this is not scrapeable at all. They don't have the user rating in the page source, so it's not something the scraper addon can deal with.

Not sure if actually you can grab them?! Can you check?


RE: [Release] Universal Scraper for Music Albums - docwra - 2018-01-12

(2018-01-12, 22:29)olympia Wrote: hey zag, cannot follow. Those ratings would be _where_ forever? What do you mean?

btw, I just looked at allmusic to see how quickly I could add option to scrape user ratings, but actually this is not scrapeable at all. They don't have the user rating in the page source, so it's not something the scraper addon can deal with.

Not sure if actually you can grab them?! Can you check?
 TheAudioDB can now support ratings from other sites Wink We keep them in our DB and are available from our API.

I just need to import them somehow, usually I write a scraper using PHP.

There is now a Kodi Add-on to sync those ratings from TADB to a local Kodi install (or the other way round).

I come up against the same problem as you though with allmusic, the HTML does not contain enough data to automatically scrape it.


RE: [Release] Universal Scraper for Music Albums - docwra - 2018-01-13

@olympia i'm just trying to scrape the allmusic album ratings but can't work out how to get the initial Allmusic Album ID, can you tell me how you get it?

Does the scraper get it from the Music Brainz Web Service? If so what is the web service URL?

EDIT: Nevermind, worked it out
Code:
http://musicbrainz.org/ws/2/release-group/9c5aa9d2-068b-4626-b352-ae6f4f4e0b15?inc=url-rels

I will add the AllMusicalbumID, DiscogsID, WikidataID to all TADB albums as a first step Wink


This forum uses Lukasz Tkacz MyBB addons.