Kodi Community Forum
Release Universal Scraper for Music Artists - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: Music Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=303)
+----- Thread: Release Universal Scraper for Music Artists (/showthread.php?tid=132623)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42


RE: [Release] Universal Scraper for Music Artists - PHS Philip - 2013-02-24

I am currently unable to scan artist information with this addon at all. I have tried upgrading from Eden, recompiling, reinstalling, deleting and regenerating databases, reinstalling the addon, etc. Not one artist scans in, whether I try query all or one by one. Every time I try to scan, the log file gets a 'failed to load scraper xml' warning. When I go to ~/.xbmc/addons/metadata.artists.universal/, the xml is there. Any suggestions?


RE: [Release] Universal Scraper for Music Artists - hoopsdavis - 2013-02-27

PHS Phillip I'm having the same issue, something has changed in the past few days.

Anyone know how to resolve this?


RE: [Release] Universal Scraper for Music Artists - Rififi77 - 2013-02-27

Hello,
I have the same problem, impossible to get information about the artist

see my logs

http://www.xbmclogs.com/show.php?id=1282

@+


RE: [Release] Universal Scraper for Music Artists - RiotGrrl - 2013-02-27

Me too, same thing - XBMC kept crashing. Rolled back to version 2.6.2 and everything's working fine again!


RE: [Release] Universal Scraper for Music Artists - RiotGrrl - 2013-02-28

(2013-02-23, 19:46)PatrickVogeli Wrote: Hi there,

just a very quick question: am I right that, currently, this addon only scrapes biography info from TheAudioDB??

I know that allmusic stopped working because of copyright issues, but what about last.fm? Has it stopped working too?

EDIT: I forgot, in the addon you can put allmusic as the source to scrape Styles and Moods info, by I don't get those neither. I suppose allmusic is blocking everything?

EDIT 2: ok, got it. Last+fm did some changes in the api, that's because you don't use it anymore. It's depressing to see how much XBMC has improved in 2 years and how the scraper thing has gone downwards because the sites not permiting the scraping anymore.

The scraper always checks MusicBrainz first. It can only grab the info from allmusic if there is an allmusic link on the MusicBrainz artist page. If the link isn't there, you can easily add it (need to log in). Read the add-on description, it's all there! Hope that helps!


RE: [Release] Universal Scraper for Music Artists - aptalca - 2013-03-04

(2012-11-14, 04:29)aptalca Wrote: Hi olympia,

I have a rather unusual issue. I am trying to add the artist Kitaro to the library. However, the dude is Japanese :-) and his name on MusicBrainz is a couple of Japanese characters. The sort name on MusicBrainz shows it as Kitaro, but the scraper doesn't retrieve anything.

Any ideas??

Thanks

EDIT: I also cannot enter the name manually as I don't have Japanese character input. Even if I did, I wouldn't even know which characters to put in really (other than maybe copy and paste :-)

Hi olympia,

I had asked this question a while back, but it got lost in the thread without an answer

I would like to bring it up again since I still don't have a solution for scraping Kitaro. I get a blank fanart, blank poster and no info

Here is the musicbrainz search, you can see that the actual name is japanese characters, but the sort name is "Kitaro": http://musicbrainz.org/search?query=kitaro&type=artist&method=indexed

The search/match score is 100%

Thanks


RE: [Release] Universal Scraper for Music Artists - n1md4 - 2013-03-05

(2013-03-04, 23:31)aptalca Wrote:
(2012-11-14, 04:29)aptalca Wrote: Hi olympia,

I have a rather unusual issue. I am trying to add the artist Kitaro to the library. However, the dude is Japanese :-) and his name on MusicBrainz is a couple of Japanese characters. The sort name on MusicBrainz shows it as Kitaro, but the scraper doesn't retrieve anything.

Any ideas??

Thanks

EDIT: I also cannot enter the name manually as I don't have Japanese character input. Even if I did, I wouldn't even know which characters to put in really (other than maybe copy and paste :-)

Hi olympia,

I had asked this question a while back, but it got lost in the thread without an answer

I would like to bring it up again since I still don't have a solution for scraping Kitaro. I get a blank fanart, blank poster and no info

Here is the musicbrainz search, you can see that the actual name is japanese characters, but the sort name is "Kitaro": http://musicbrainz.org/search?query=kitaro&type=artist&method=indexed

The search/match score is 100%

Thanks

I'm not 100% sure how the scraper works, I think the artist must exist on Musicbrainz, you found that - check! But the scraping happens (I think) against theaudiodb.com , the record for 喜多郎 did not exist. I have created this for you (I was feeling generous - it might have been beer induced ; ) it should now scrape correctly - touch wood!

99% of the time, should an artist not scrape, check first it exists on theaudiodb.com , using the name that matches musicbrainz. If you have more, you are welcome to create accounts on these sites, and get a nice community collection of metadata Smile

I hope that helps. Let me know if you need anything else.


RE: [Release] Universal Scraper for Music Artists - capfuturo - 2013-03-05

Hi there,

I hope all of you read the following and think seriously about the issue here:

"I have tested dozens of times the situation where, within XBMC library, artists are waiting for their bio and albums info to be scraped by Universal Scraper successfully. It happens ONLY if the artists and its albums appear to be in AudioDB, and not just mentioned but has to actually contain the information you want to be scraped under its listings. What is even worse, and contrary to what's been previously said by someone in this thread: No info is scrapped from MusicBrainz or Allmusic whatsoever. No link to Allmusic, BBC, Disclogs or Wikipedia, appearing as relationships in MusicBrainz or AudioDB, will be addressed by the Universal Scraper to retrieve the data. It is all useless. Only if you add the info manually to AudioDB it will be scraped and retrieved.

Now, this presents a very basic question: if organisations such as BBC (read this please) are scraping e.g., the data related to the artist from Wikepedia, why don't we just make the Universal Scraper pull the info we need from Wikepedia instead? This way we make sure that the info will be mostly reliable and, what is most important, always up to date. If we input the info by ourselves in AudioDB, I assure you that it will be mostly a copy and paste from Wikipedia or BBC or sites that license their info openly and free to anybody who wants to use it. The downside of this is that we then need to keep updating AudioDB database as months go by. Do you see the issue here? Wikipedia is being updated at a daily rate by hundreds of thousands of like-minded individuals, including me, around the world for free and to be shared for free, with the latest verifiable info. We do need a solution such as this to keep our front-end XBMC fed with this kind of updated data regularly.

Could Universal Scraper be configured to provide the option to scrape Wikipedia directly or to follow the links appearing under the listings for the artists or albums in AudioDB or MusicBrainz.

I strongly believe we should all invest time and resources in making XBMC better and better, but I do not share the vision of feeding and maintaining yet another parallel database such as AudioDB, when actually there is much better, complete and intelligently organised work out there, presented as a combination of MusicBrainz, Wikipedia and fanart.tv .

I truly hope my opinions are considered. CF


RE: [Release] Universal Scraper for Music Artists - n1md4 - 2013-03-05

(2013-03-05, 01:59)capfuturo Wrote: Hi there,

I hope all of you read the following and think seriously about the issue here:

"I have tested dozens of times the situation where, within XBMC library, artists are waiting for their bio and albums info to be scraped by Universal Scraper successfully. It happens ONLY if the artists and its albums appear to be in AudioDB, and not just mentioned but has to actually contain the information you want to be scraped under its listings. What is even worse, and contrary to what's been previously said by someone in this thread: No info is scrapped from MusicBrainz or Allmusic whatsoever. No link to Allmusic, BBC, Disclogs or Wikipedia, appearing as relationships in MusicBrainz or AudioDB, will be addressed by the Universal Scraper to retrieve the data. It is all useless. Only if you add the info manually to AudioDB it will be scraped and retrieved.

Now, this presents a very basic question: if organisations such as BBC (read this please) are scraping e.g., the data related to the artist from Wikepedia, why don't we just make the Universal Scraper pull the info we need from Wikepedia instead? This way we make sure that the info will be mostly reliable and, what is most important, always up to date. If we input the info by ourselves in AudioDB, I assure you that it will be mostly a copy and paste from Wikipedia or BBC or sites that license their info openly and free to anybody who wants to use it. The downside of this is that we then need to keep updating AudioDB database as months go by. Do you see the issue here? Wikipedia is being updated at a daily rate by hundreds of thousands of like-minded individuals, including me, around the world for free and to be shared for free, with the latest verifiable info. We do need a solution such as this to keep our front-end XBMC fed with this kind of updated data regularly.

Could Universal Scraper be configured to provide the option to scrape Wikipedia directly or to follow the links appearing under the listings for the artists or albums in AudioDB or MusicBrainz.

I strongly believe we should all invest time and resources in making XBMC better and better, but I do not share the vision of feeding and maintaining yet another parallel database such as AudioDB, when actually there is much better, complete and intelligently organised work out there, presented as a combination of MusicBrainz, Wikipedia and fanart.tv .

I truly hope my opinions are considered. CF

My assumption is that there have been sites that licensed their data to the public, but later pulled their agreements? Last.fm, allmusic.com, to name two since I've been playing with XBMC (only a few months). Is theaudiocdb.com not a reaction to these actions?

I can certainly see the logic of scraping dynamic content, if it's available from reliable sources, but these need to be reliably open sources. I'm no developer (no affiliation with this scraper), but I can't see why scraping from wikipedia shouldn't be possible; fanart.tv, and musicbrainz can already be scraped - in fact all 3 sites are closely knit, from what I can tell. The only reason I can think not to scrape from wikipedia would be the ambiguity of multiple searches .. not sure, think I read that somewhere, something like that any way.

Personally, my interest and investment is in community driven. open projects; wikipedia, fanart, the*db's, etc. I'm not entirely trusting in large corporations and how reliable they are at sharing their data.

Interesting posts raised, and I must admit doing something intelligent with wikipedia for biographies must be possible, no?


RE: [Release] Universal Scraper for Music Artists - olympia - 2013-03-05

(2013-03-04, 23:31)aptalca Wrote:
(2012-11-14, 04:29)aptalca Wrote: Hi olympia,

I have a rather unusual issue. I am trying to add the artist Kitaro to the library. However, the dude is Japanese :-) and his name on MusicBrainz is a couple of Japanese characters. The sort name on MusicBrainz shows it as Kitaro, but the scraper doesn't retrieve anything.

Any ideas??

Thanks

EDIT: I also cannot enter the name manually as I don't have Japanese character input. Even if I did, I wouldn't even know which characters to put in really (other than maybe copy and paste :-)

Hi olympia,

I had asked this question a while back, but it got lost in the thread without an answer

I would like to bring it up again since I still don't have a solution for scraping Kitaro. I get a blank fanart, blank poster and no info

Here is the musicbrainz search, you can see that the actual name is japanese characters, but the sort name is "Kitaro": http://musicbrainz.org/search?query=kitaro&type=artist&method=indexed

The search/match score is 100%

Thanks

you can always include the musicbrainz link to the artist in an artist.nfo to create a one to one link with the desired artist in case of such issues.


RE: [Release] Universal Scraper for Music Artists - zag - 2013-03-06

(2013-03-05, 01:59)capfuturo Wrote: I strongly believe we should all invest time and resources in making XBMC better and better, but I do not share the vision of feeding and maintaining yet another parallel database such as AudioDB, when actually there is much better, complete and intelligently organised work out there, presented as a combination of MusicBrainz, Wikipedia and fanart.tv .

Wikipedia is in no way "intelligently organised" Smile Every page is different and very hard to scrape.

I created theaudiodb.com for exactly the reason so we had a standard way of accessing music metadata that would never change, break a scraper or have to be kept updated. All of this as well as being community driven will hopefully mean soon it will be the best stable music resource out there. Just like TMDB and TheTVDB have been for the last 5+ years.


RE: [Release] Universal Scraper for Music Artists - capfuturo - 2013-03-07

(2013-03-06, 12:14)zag Wrote:
(2013-03-05, 01:59)capfuturo Wrote: I strongly believe we should all invest time and resources in making XBMC better and better, but I do not share the vision of feeding and maintaining yet another parallel database such as AudioDB, when actually there is much better, complete and intelligently organised work out there, presented as a combination of MusicBrainz, Wikipedia and fanart.tv .

Wikipedia is in no way "intelligently organised" Smile Every page is different and very hard to scrape.

I created theaudiodb.com for exactly the reason so we had a standard way of accessing music metadata that would never change, break a scraper or have to be kept updated. All of this as well as being community driven will hopefully mean soon it will be the best stable music resource out there. Just like TMDB and TheTVDB have been for the last 5+ years.

Good evening Zag,

I understand your point. I would like to ask you: regardless of its degree of difficulty, is there a way to automatically have at least the bios of the artists and descriptions of the albums updated in AudioDB by linking them to the Wikipedia pages that are being updated daily with the info we are after? What I am trying to get to here, is avoiding the copy/paste manual process or having to post my own writings/findings for a particular artist or work of art, not just on one website but on both, Wikipedia and AudioDB. Can we make the link to Wikipedia a more "useful link" which means a link that the AudioDB could feed from, just as BBC or MusicBrainz do?

I too understand your concerns about "what if the links break", for instance. However, wouldn't be easier for any of us to just relink when we find a broken page than manually updating the pages on AudioDB every time there are news with respect to an artist or a music release?

Tackling another matter, I notice there is a section in AudioDB that lists the upcoming shows for the artist. How can I get this data to show up in my XBMC and possibly follow its links and buy tickets for shows? Is this implemented yet?

Best, CF


RE: [Release] Universal Scraper for Music Artists - zag - 2013-03-08

(2013-03-05, 01:59)capfuturo Wrote: Good evening Zag,

I understand your point. I would like to ask you: regardless of its degree of difficulty, is there a way to automatically have at least the bios of the artists and descriptions of the albums updated in AudioDB by linking them to the Wikipedia pages that are being updated daily with the info we are after? What I am trying to get to here, is avoiding the copy/paste manual process or having to post my own writings/findings for a particular artist or work of art, not just on one website but on both, Wikipedia and AudioDB. Can we make the link to Wikipedia a more "useful link" which means a link that the AudioDB could feed from, just as BBC or MusicBrainz do?

I too understand your concerns about "what if the links break", for instance. However, wouldn't be easier for any of us to just relink when we find a broken page than manually updating the pages on AudioDB every time there are news with respect to an artist or a music release?

Yes that's certainly possible but you would loose the biography and album description from our API. Wikipedia is impossible to scrape in a standard way, that's why musicbrainz and other sites only show the first paragraph cut off. I really don't see the thing you are trying to achieve. The whole idea is to get people contributing to the site and keeping it up-to-date by our users, and NOT relying on 3rd party sites.

Your idea of having a real URL link to Wikipedia does have some merit though, I'll think about adding that.

(2013-03-05, 01:59)capfuturo Wrote: Tackling another matter, I notice there is a section in AudioDB that lists the upcoming shows for the artist. How can I get this data to show up in my XBMC and possibly follow its links and buy tickets for shows? Is this implemented yet?

Best, CF

Yep really good idea! I had thought about this myself. Songkick has an API for exactly this, it uses the musicbrainz ID for lookup so would be trivial to add to a scraper.

Unfortunately the problem is... where, and how, do we show this in XBMC? (for example there are currently no database fields for live concerts). Ideas on that one are welcome Smile


RE: [Release] Universal Scraper for Music Artists - PatrickVogeli - 2013-03-08

For the Live concerts, I would believe a new pluging would be the best... because this info would always habe to be newly fetched. To me, it makes no sense of saving this info into a DB, but makes much more sense to open a plugin an request the info, showing a list with the artist on tour and, if selected, showing the tour dates.


RE: [Release] Universal Scraper for Music Artists - olympia - 2013-03-08

(2013-03-07, 23:39)capfuturo Wrote: I understand your point. I would like to ask you: regardless of its degree of difficulty, is there a way to automatically have at least the bios of the artists and descriptions of the albums updated in AudioDB by linking them to the Wikipedia pages that are being updated daily with the info we are after? What I am trying to get to here, is avoiding the copy/paste manual process or having to post my own writings/findings for a particular artist or work of art, not just on one website but on both, Wikipedia and AudioDB. Can we make the link to Wikipedia a more "useful link" which means a link that the AudioDB could feed from, just as BBC or MusicBrainz do?

As a consequence of being TheAudioDb.com available now, people actually stop putting efforts into maintaning wiki pages because it is useless for sharing the data/information further on. It is the same thing that wikipedia is not the main source for movies. The primary source of such artist/album information should be a site like TheAudioDb what make it available for sharing and scraping easily for free.