[Release] Universal Scraper for Music Artists

  Thread Rating:
  • 9 Vote(s) - 4.44 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
olympia Offline
Team-Kodi Member
Posts: 2,718
Joined: May 2008
Reputation: 39
Post: #16
Dear Andy,

I am absolutely not interested in overall comments unless those are positive.
In case you have a problem or complaint you need to be very specific! "Sometimes there, sometimes not" doesn't help at all.

I see that you are new to the forum and you mentioned your are brand new to xbmc, but this is not a good excuse to share your unqualified feedback which might mislead others. E.g. Musicbrainz scraper and Univeral Scraper can't give different results at all since the Universal Scraper is just an extended Musicbrainz scraper (which in fact doesn't provide a lot in its own).

There can be several reasons why last.fm and Universal scrapers give different results and I would be interested to look at this.
So let me re-articulate my request: _give me an artist to test_.
(This post was last modified: 2012-06-04 19:42 by olympia.)
find quote
andydufresne Offline
Member
Posts: 71
Joined: Jun 2012
Reputation: 0
Post: #17
(2012-06-04 19:40)olympia Wrote:  Dear Andy,

I am absolutely not interested in overall comments unless those are positive.
In case you have a problem or complaint you need to be very specific! "Sometimes there, sometimes not" doesn't help at all.

I see that you are new to the forum and you mentioned your are brand new to xbmc, but this is not a good excuse to share your unqualified feedback which might mislead others. E.g. Musicbrainz scraper and Univeral Scraper can't give different results at all since the Universal Scraper is just an extended Musicbrainz scraper (which in fact doesn't provide a lot in its own).

There can be several reasons why last.fm and Universal scrapers give different results and I would be interested to look at this.
So let me re-articulate my request: _give me an artist to test_.

I shared my experience and results with the Universal Scraper. I had poor results. Sorry it offended you but I thought it might be helpful to others to share what I experienced and the steps I took to get artist information somewhat properly populated.

For me there were many artists that had missing information and my focus was getting my large music collection in decent shape in XBMC and properly scraped. It took me running the universal scraper, last.fm, musicbrainz and then universal scraper again one last time to get it there.

I didn't document and track things artist by artist. There were many. To recreate the results for testing purposes I would have to completely wipe everything, including all cache if I knew where everything is stored, and re-run to see precisely all the artists where information was missed. It took so much time just to get things where they are right now I'm not going to retrace my steps. But I will focus artist by artist for future additions and provide you those names.
find quote
saladasalad Offline
Junior Member
Posts: 6
Joined: Jun 2012
Reputation: 0
Post: #18
Great job on the scraper, it works very nicely! A few points of discussion, I'm very new to XBMC so let me know if I've got the wrong end of a stick here...
  • IMHO,using MBID's is essential for fetching accurate data. There are many artists that have the same/very similar names.
  • Most artists (and albums) on Musicbrainz have a link to the Wikipedia page attached, this could be used to fetch bio information.
  • Google should also be added as a source for images, I understand that people should contribute images to fanart.tv but that is very time-consuming (plus I have contributed countless hours to editing MB) and finding images is something that google is already very good at.
  • Discogs is another very good source of music metadata that would make a good addition to this scraper.
  • I would also love to see a Universal Scraper for albums, many possibilities there.

Thanks, keep up the good work.
find quote
olympia Offline
Team-Kodi Member
Posts: 2,718
Joined: May 2008
Reputation: 39
Post: #19
(2012-06-06 07:29)saladasalad Wrote:  [*]IMHO,using MBID's is essential for fetching accurate data. There are many artists that have the same/very similar names.
This is not possible currently (MBID is not scanned in, stored and made available to the scrapers for lookup), but this is something actively worked on atm: http://forum.xbmc.org/showthread.php?tid=132573

(2012-06-06 07:29)saladasalad Wrote:  [*]Most artists (and albums) on Musicbrainz have a link to the Wikipedia page attached, this could be used to fetch bio information.
I will look at this, but ot high on my prio list.

(2012-06-06 07:29)saladasalad Wrote:  [*]Google should also be added as a source for images, I understand that people should contribute images to fanart.tv but that is very time-consuming (plus I have contributed countless hours to editing MB) and finding images is something that google is already very good at.
No way at all. This is a scraper for top quality.

(2012-06-06 07:29)saladasalad Wrote:  [*]Discogs is another very good source of music metadata that would make a good addition to this scraper.
What Discogs has which is not available on MusicBrainz? I am really not a fan of discogs, but I can be convinced.

(2012-06-06 07:29)saladasalad Wrote:  [*]I would also love to see a Universal Scraper for albums, many possibilities there.
This is in the works now.
However I don't see the many possibilities there. Hints are welcome. (Actually I don't see so much sense to be able to scrape tracklist from 5 different site if it will be the same). Having different sources IMO only make sense if the data we scrape is different (like review, ratings, artwork).
find quote
saladasalad Offline
Junior Member
Posts: 6
Joined: Jun 2012
Reputation: 0
Post: #20
(2012-06-06 09:22)olympia Wrote:  
(2012-06-06 07:29)saladasalad Wrote:  [*]IMHO,using MBID's is essential for fetching accurate data. There are many artists that have the same/very similar names.
This is not possible currently (MBID is not scanned in, stored and made available to the scrapers for lookup), but this is something actively worked on atm: http://forum.xbmc.org/showthread.php?tid=132573
I see. Thanks for the link, I'll have a good read through that later. Sounds promising.

(2012-06-06 09:22)olympia Wrote:  
(2012-06-06 07:29)saladasalad Wrote:  [*]Google should also be added as a source for images, I understand that people should contribute images to fanart.tv but that is very time-consuming (plus I have contributed countless hours to editing MB) and finding images is something that google is already very good at.
No way at all. This is a scraper for top quality.
Fair enough. I do think that with the right settings (high image quality, -covers, photos, etc.) you could get decent quality but it's probably something better suited to another addon.

(2012-06-06 09:22)olympia Wrote:  
(2012-06-06 07:29)saladasalad Wrote:  [*]Discogs is another very good source of music metadata that would make a good addition to this scraper.
What Discogs has which is not available on MusicBrainz? I am really not a fan of discogs, but I can be convinced.
Nothing particularly useful on an artist level now I think about it, other than some images (which are probably on last.fm anyway.) Discogs does usually have more album releases than MB though, unfortunately more people seem to contribute to Discogs than MB. I'm not a fan of the site either, TBH.

(2012-06-06 09:22)olympia Wrote:  
(2012-06-06 07:29)saladasalad Wrote:  [*]I would also love to see a Universal Scraper for albums, many possibilities there.
This is in the works now.
However I don't see the many possibilities there. Hints are welcome. (Actually I don't see so much sense to be able to scrape tracklist from 5 different site if it will be the same). Having different sources IMO only make sense if the data we scrape is different (like review, ratings, artwork).

Good to hear. Many possibilities was an exaggeration on my part but there are certainly possibilities Big Grin. Multiple reviews from metacritic for each album. Ratings from rateyourmusic, metacritic or MusicBrainz. You could also get release metadata like label, cat# and release date. Album specific Wikipedia links for album descriptions. That's all I've got right now but I'm sure I'll think of more, I've just discovered XBMC recently and I'm brimming with ideas but I need to write them down as I think of them or something Wink
find quote
curtis-r Offline
Donor
Posts: 487
Joined: Feb 2009
Reputation: 3
Location: San Francisco, USA
Post: #21
Yesterday I decided to rebuild my music db due to a corruption, and figured I'd try the Universal scrapers, since Allmusic doesn't seem to work properly. Now I'm finding the Library Update crashes a lot.

Once I determine the artist the crash occured on, I rename the folder with "zz-" in front of it, so it's at the end of the library for now. But why the crashes. I've check and redone some of the ID3 tags, but still crashes. No punctuation in the artist or song names either.

Win7x64. Intel Core2Duo E8500 3.26GHz on Gigabyte GA-EP43-UD3L. 4GB RAM. Nvidia GT240 512MB PCIx16. Pioneer 60" Kuro plasma + Epson Pro Cinema 7500UB projector + 100" Da-Lite screen.
find quote
transceiverfreq Offline
Junior Member
Posts: 3
Joined: Jun 2012
Reputation: 0
Post: #22
(2012-06-17 19:56)curtis-r Wrote:  Yesterday I decided to rebuild my music db due to a corruption, and figured I'd try the Universal scrapers, since Allmusic doesn't seem to work properly. Now I'm finding the Library Update crashes a lot.

Once I determine the artist the crash occured on, I rename the folder with "zz-" in front of it, so it's at the end of the library for now. But why the crashes. I've check and redone some of the ID3 tags, but still crashes. No punctuation in the artist or song names either.

What Skin are you using at the time of the crash?

I ask because when I updated in "PM.3 HD" skin scraping worked fine but when I tried doing it while in Aeon Nox it crashed repeatedly during update as well.

Each time XBMC crashed I removed the offending artist folder from the source folder and restarted XBMC. The updating seemed to continue until I came across a crash from another artist. I then repeated the removing and the updating continued until it happened again. I also checked and redid ID3 tags. I rebuilt my entire library from ground up before putting it back in.

Still crashed.

Then I switched to "PM.3 HD" skin to get my SVN and enable debugging (to share for this thread). Once debug logging was enabled I tried to reproduce the problem and it ran through perfectly. There is a chance that Universal Scraper was updated during that moment. If so, I'm sure Olympia will know.

I am going to:
- switch back to "Aeon Nox"
- remove the source folder
- re-add the source folder
- scan it into library
- upload my log here

Here's hoping I can reproduce the crash.

I will edit this post with a paste of my log momentarily.

EDIT: Sadly (actually happily), I was unable to reproduce the crash but i do have the older crash log uploaded (http://pastebin.ubuntu.com/1046032/). The good news however is that it's working just fine now.
(This post was last modified: 2012-06-17 21:20 by transceiverfreq.)
find quote
IzeQube Offline
Junior Member
Posts: 9
Joined: Jun 2011
Reputation: 0
Post: #23
I'm on a German Windows x64 and I have the scraper is crashing XBMC when specific Artists should be scraped.

I think it has something to do with special characters in either the Filename or in the Tags. I could reproduce the crash as soon as "a-ha" should be scraped. I then removed the files and it scraped up to the artist "Dan Bälan" and crashed again.
(This post was last modified: 2012-06-18 11:03 by IzeQube.)
find quote
curtis-r Offline
Donor
Posts: 487
Joined: Feb 2009
Reputation: 3
Location: San Francisco, USA
Post: #24
Definitely not the skin. Was using Aeon MQ3 but still crashes when I switch to Confluence. Still not sure what the common denominator is b/c one of the artist crashes is for "Band Aid", which clearly has no special characters.

Win7x64. Intel Core2Duo E8500 3.26GHz on Gigabyte GA-EP43-UD3L. 4GB RAM. Nvidia GT240 512MB PCIx16. Pioneer 60" Kuro plasma + Epson Pro Cinema 7500UB projector + 100" Da-Lite screen.
find quote
olympia Offline
Team-Kodi Member
Posts: 2,718
Joined: May 2008
Reputation: 39
Post: #25
Hopefully and potentially fixed in v2.0.1
find quote
Plecebo Offline
Junior Member
Posts: 16
Joined: Jun 2009
Reputation: 0
Post: #26
Any ideas when v2.0.1 will drop I'm currently without a functioning scrapper.

Here is debug crash log from my scraping: http://pastebin.ubuntu.com/1046682/
(This post was last modified: 2012-06-18 06:52 by Plecebo.)
find quote
IzeQube Offline
Junior Member
Posts: 9
Joined: Jun 2011
Reputation: 0
Post: #27
My XBMC just Updates the Scraper to v2.0.1.

It all seems to be working now.

Thanks for the quick fix.
find quote
nonob Offline
Senior Member
Posts: 147
Joined: Sep 2011
Reputation: 0
Post: #28
Hi,

SInce 2.0 version, there is "fallback" settings, what is it ?

Thanks
find quote
olympia Offline
Team-Kodi Member
Posts: 2,718
Joined: May 2008
Reputation: 39
Post: #29
http://www.usingenglish.com/reference/ph...ck+on.html
find quote
transceiverfreq Offline
Junior Member
Posts: 3
Joined: Jun 2012
Reputation: 0
Post: #30
Thank You!
find quote
Post Reply