Improved allmusic.com scraper (plus a few questions)

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
mkortstiege Offline
Team-Kodi Developer
Posts: 2,971
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #31
Should be fixed in r23269. No idea what went wrong on my first tests to reproduce this issue Wink

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #32
vdrfan Wrote:Should be fixed in r23269. No idea what went wrong on my first tests to reproduce this issue Wink

Awesome, thanks!! I kept testing it repeatedly and wasn't sure if I should keep pestering you about it or not Wink
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #33
Ok, now that the music scraper settings are working properly (thanks vdrfan!), I'm back to work on creating my ultimate music scraper. Wink

I've decided that Allmusic has the best text info, Last.fm has the best artist photos (after discovering that original/unresized photos are available through the API!), and Discogs has the best album artwork. So I've abstracted functions from all those sources, and have combined them into the Allmusic scraper as optional settings, much in the same way HTBackdrops is an option in all the scrapers.

I've also switched Discogs to use the API calls rather than HTML scraping, which I've found to be MUCH faster.

I've zipped up my music scraper folder for anyone who is interested in trying it (note: you NEED SVN r23269 or higher for this to work properly!):

EDIT: My changes are now in the latest SVN builds. Just download that instead!

Note that the Discogs functions use my own personal API key, not that it really matters, but maybe one of the XBMC devs could sign up for an "official" API key that I can add to the scrapers?

Edit: Also note, I've left the default settings to use Allmusic for everything and NOT use the other sources, so you'll have to go into the Scraper Settings if you want to enable scraping from Last.fm and/or Discogs.
(This post was last modified: 2009-11-03 01:37 by talisto.)
find quote
mkortstiege Offline
Team-Kodi Developer
Posts: 2,971
Joined: Jan 2008
Reputation: 8
Location: Germany
Post: #34
Sweet. Trac it please.

Btw, I've played with the discogs API myself some time ago but dropped it, as it seemed to be a PITA to identify the correct album. Searching for "Kelly Clarkson" (no, not my music taste, just an example) returned a Disney Single in first place eg.

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules
For troubleshooting and bug reporting please make sure you read this first.
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #35
vdrfan Wrote:Btw, I've played with the discogs API myself some time ago but dropped it, as it seemed to be a PITA to identify the correct album. Searching for "Kelly Clarkson" (no, not my music taste, just an example) returned a Disney Single in first place eg.

The artists search seems to work alright, as you can specify type=artists, e.g:

http://www.discogs.com/search?type=artis...7404bc20ec

The problematic part is the album search, you can specify type=releases, but you can't separate the artist and album into separate variables, so you need to mash them together like this:

http://www.discogs.com/search?type=relea...7404bc20ec

Which can result in theory some false positives, but in my testing I've found it to be pretty reliable.

The head-scratcher is that the API information doesn't seem to always match the HTML version. For instance, look at the HTML page for Beck:

http://www.discogs.com/artist/Beck

Then look at the API version:

http://www.discogs.com/artist/Beck?f=xml...7404bc20ec

Notice the API doesn't have the biography/profile? It DOES for some artists, but not for others. There's a lot of inconsistencies in that regard. So I think it's useful in the way I'm using it, since I'm only using it for images, and it's much faster than parsing the HTML. But I'm not so sure it's a good replacement for the stand-alone Discogs scraper.
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #36
vdrfan Wrote:Sweet. Trac it please.

Done!

http://trac.xbmc.org/ticket/7254
find quote
blacklist Offline
Posting Freak
Posts: 814
Joined: Jul 2009
Reputation: 1
Location: Atlanta, Ga, USA
Post: #37
Quote:Edit: Also note, I've left the default settings to use Allmusic for everything and NOT use the other sources, so you'll have to go into the Scraper Settings if you want to enable scraping from Last.fm and/or Discogs.

Forgive me if I'm being dense... but where are the music scraper settings? I can get them from all the video sources, but don't seem to have access to anything but simply picking the scraper in the music settings.

I know I'm missing something obvious...
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #38
blacklist Wrote:Forgive me if I'm being dense... but where are the music scraper settings? I can get them from all the video sources, but don't seem to have access to anything but simply picking the scraper in the music settings.

Well, first, you need to have a VERY recent SVN version of XBMC. The scraper settings were only added a couple weeks ago. If you do have a recent SVN version, you can access the scrapers by going to Settings > Music > Library > Scraper Settings. If you don't have that option, you need to update your version of XBMC. Smile
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #39
BTW, I nuked my whole music library and rescanned all my files (admittedly a meager collection of roughly 10,000 files; 800 albums and 1000 artists) with my new scraper update, using Last.fm for the artist thumbs and Discogs for the album thumbs (all other settings disabled except for HTbackdrops), and I must say, I'm REALLY happy with the results! Nearly every artist has a high-resolution thumbnail and a full bio, and nearly every album does as well, with no false positives that I can see (yet).

Not to toot my own horn, but I really do think this covers all the bases better than expected. Most of the "glory" of XBMC goes to it's video-handling capabilities, but for those of you out there who use it primarily for music like I do, I think you're going to be really pleased with this once it hits the SVN.
find quote
steve1977 Offline
Posting Freak
Posts: 1,981
Joined: Dec 2006
Reputation: 0
Post: #40
talisto Wrote:Ok, now that the music scraper settings are working properly (thanks vdrfan!), I'm back to work on creating my ultimate music scraper. Wink

I've decided that Allmusic has the best text info, Last.fm has the best artist photos (after discovering that original/unresized photos are available through the API!), and Discogs has the best album artwork. So I've abstracted functions from all those sources, and have combined them into the Allmusic scraper as optional settings, much in the same way HTBackdrops is an option in all the scrapers.

I've also switched Discogs to use the API calls rather than HTML scraping, which I've found to be MUCH faster.

I've zipped up my music scraper folder for anyone who is interested in trying it (note: you NEED SVN r23269 or higher for this to work properly!):

http://www.mediafire.com/download.php?bxdvj1jzntz

Note that the Discogs functions use my own personal API key, not that it really matters, but maybe one of the XBMC devs could sign up for an "official" API key that I can add to the scrapers?

Edit: Also note, I've left the default settings to use Allmusic for everything and NOT use the other sources, so you'll have to go into the Scraper Settings if you want to enable scraping from Last.fm and/or Discogs.

Thanks, this sounds very great. Three brief questions:

1) I am not sure whether I understand your "edit". Will it be sufficient just to copy the files you have uploaded somewhere and then everything is set or do I still need to play with the settings? Given your huge music database and success of identifying artist and album art, I would like to follow your settings.

2) I have actually pretty comprehensive ID-Tags including the album cover. Does this matter for the scraper settings?

3) Do I need to activate HTBackdrops in the settings for "artist thumbs" in order to get the fanart? In other words, if I deactivate "artist thumbs from HTBackdrops", will I still get fanart from HTBackdrops?


Cheers!!!

Server: Asus Sabertooth Z77 | Intel Core i5 3.4 GHz | 12 GB DDR3 | 256 GB SSD, 40 TB (1 x 4 TB, 6 x 3TB, 9 x 2 TB)
HTPC 1: Apple Mac Mini (mid-2011) | Intel Core i5 2.5 GHz | 4GB DDR3 | 500 GB
HTPC 2: Apple Mac Mini (mid-2007) | Intel Core 2 Duo 2.0 GHz | 4 GB DDR2 | 16 GB SSD
(This post was last modified: 2009-10-02 12:37 by steve1977.)
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #41
steve1977 Wrote:Thanks, this sounds very great. Three brief questions:

1) I am not sure whether I understand your "edit". Will it be sufficient just to copy the files you have uploaded somewhere and then everything is set or do I still need to play with the settings? Given your huge music database and success of identifying artist and album art, I would like to follow your settings.

2) I have actually pretty comprehensive ID-Tags including the album cover. Does this matter for the scraper settings?

3) Do I need to activate HTBackdrops in the settings for "artist thumbs" in order to get the fanart? In other words, if I deactivate "artist thumbs from HTBackdrops", will I still get fanart from HTBackdrops?

Hey Steve!

Sorry for the delayed response, I've been away for the weekend. Hopefully you got it working Smile In any case, I'll do my best to answer your questions:

1) The options I've added aren't enabled by default. I made that decision primarily because I thought it was best to keep the Allmusic scraper as, well, an Allmusic scraper by default, so it only scrapes Allmusic sources unless set otherwise. If you want to get better results, you need to go into the settings and disable thumb scraping from Allmusic, and enable it from Last.fm or Discogs. I'd recommend scraping artist thumbs from Last.fm and album thumbs from Discogs, though the artist thumbs from Discogs are good as well.

2) Having good ID3 tags are integral to getting good results from all the scrapers. However, as far as I can tell, XBMC prefers the album thumbs from scraper sources over the album thumbs in the ID3 tag. So if the scraper is able to fetch an album thumb from an online source, it will use that even if an album thumb exists in the ID3 tag. Maybe this isn't the intended behaviour, but it seems to be the case from my testing.

3) The "Get Fanart" is a separate setting, so as long as you keep that enabled, the scraper will still fetch fanart from htbackdrops regardless of what you set the other settings to. But FYI, I'd recommend keeping "Get artist thumbs from HTBackdrops" enabled AS WELL as getting artist thumbs from another source, because if it can't find a thumb from one source, it will automatically look to the other source. I've actually given HTBackdrops a higher priority so that it will look there first, as all the artist thumbs on that site are optimized for XBMC. But they just don't have a very large database, so it's best to have another source to fall back on.

Hope that answers your questions! Let me know how it works for you. Smile
find quote
paco Offline
Senior Member
Posts: 238
Joined: Mar 2008
Reputation: 0
Location: Toronto
Post: #42
Thanks for this awesome scraper improvement talisto! I downloaded your files and dumped them in a recent DX build. Once I enabled the Last.fm thumbs option, I rescanned most of my library. I love the much higher quality thumbs.

One question: have your patches been added to the SVN? When I downloaded the latest SVN, I went into the allmusic scraper and the scraper options were grayed out. Of course I just replaced the scraper with the most recent one you posted in this thread and all was good!

Not a fault with your kick-ass scraper tweak, but I notice HTBackdrops doesn't find any backdrops for U2. I browsed their site and noticed that they had more than 10 HD Backdrops for U2. However, even their own site search turns up 0 results. Maybe they have a minimum search string length?
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #43
paco Wrote:have your patches been added to the SVN? When I downloaded the latest SVN, I went into the allmusic scraper and the scraper options were grayed out.

Hey paco! Glad to hear the script is working for you Smile I've submitted a patch for my changes to be added to the SVN (nearly two weeks ago now) but so far it hasn't been approved yet. That might have something to do with the team's plan to re-organize XBMC's settings/options, I'm not sure. Hopefully it makes it in there eventually though! Smile

You're right about HTbackdrop's search, they must have a minimum character count. I played around with it a bit but couldn't find a workaround. It'd probably be best to contact the site owner; I noticed there's no contact link on the website, but here's his forum profile, try sending him a PM: http://forum.xbmc.org/member.php?action=...&uid=28343
find quote
Roborob Offline
Senior Member
Posts: 143
Joined: Jan 2009
Reputation: 0
Location: The Netherlands
Post: #44
Sounds very nice this script, but not working here. I can see the script in the settings and I can set the new options, problem is if i go back to the settings it is default again. If an Artist is not found It get's info from another artist (always the same). Checked all the security and write options in Windows 7 and that seems ok. I'll get this with the standard Allmusic also. Do you know what is happening ?? . Firewall Also ok. Confused Also use Latest SVN and Skin.. Happens with all the scrapers, if I search for an artist on Last FM on the internet there is lots of Info, but with the scraper it finds nothing
(This post was last modified: 2009-10-17 16:18 by Roborob.)
find quote
talisto Offline
Junior Member
Posts: 40
Joined: Jun 2005
Reputation: 0
Post: #45
Roborob Wrote:Sounds very nice this script, but not working here. I can see the script in the settings and I can set the new options, problem is if i go back to the settings it is default again.

Are you changing the scraper settings from the library with the context menu (pressing "C" from the music library screen), or are you changing the settings from (Main Menu) > Settings > Music > Library > Scraper Settings? I've noticed that when you change the settings from the context menu, it only remembers the settings for one lookup and then reverts back to your previous settings. Whether this is "by design" or a bug, I'm not sure. Going through the full settings menu should work fine though.
find quote
Post Reply