Last.fm scraper in development - help wanted

  Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #16
that would be neat, and some system to cross reference several scrapers in order to make matches more reliable
find quote
kriziz Offline
Junior Member
Posts: 4
Joined: Jun 2008
Reputation: 0
Location: UT, Netherlands
Post: #17
AllMusic.com isn't updated on a regular basis and doesn't list many relatively unknown artists. I'd expect a boost in search results from almost any new scraper, if its last.fm or discogs.. These two sites are both as good, provide a decent API etc.. One might be better than the other when it comes to specific musical nices. Most people are better off to use their favorite scraper for regular searches. We might want to make it a bit easier to switch scrapers, from the Album/Movie info screen or the manual search screen maybe ?

Anyway, in meantime I did some testing on a discogs.com scraper. This scraper script is just a few lines, it has CreateAlbumSearchUrl and GetAlbumSearchResults nodes and thats about it. Where I get stuck is testing the thing, XBMC simply crashes if I use it (no messages in xbmc.log).
The 'Scrap' test tool looks promising, however it is built to handle video scrapers, not music. Tried to fix that as well, but got caught up in C++ syntax Confused Scraper development would be much easier if anyone could fix that Scrap tool. Doesn't look like an easy job, though ..
Am I missing a development tool here, should I be debugging from Visual C++ etc. ?
(This post was last modified: 2008-12-30 12:40 by kriziz.)
find quote
Aron Parsons Offline
Senior Member
Posts: 153
Joined: Oct 2003
Reputation: 0
Location: Virginia
Post: #18
kriziz Wrote:Where I get stuck is testing the thing, XBMC simply crashes if I use it (no messages in xbmc.log).
The 'Scrap' test tool looks promising, however it is built to handle video scrapers, not music. Tried to fix that as well, but got caught up in C++ syntax Confused Scraper development would be much easier if anyone could fix that Scrap tool. Doesn't look like an easy job, though ..
Am I missing a development tool here, should I be debugging from Visual C++ etc. ?

I was fighting with the same thing last night. The 'Scrap' tool doesn't work correctly. Getting it to compile and run on Linux will be the first step, so maybe I'll try to get that sorted out over the upcoming weekend. Trying to debug scrapers through XBMC's GUI doesn't sound like fun.
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #19
scrap is broken as we lost the source (blame donj for bad committing practices).

the way i do scrapers is using a regexp tool + printf. it works just fine for me, but an updated scrap tool would be invaluable for sure
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #20
oh damn. kriziz, i have a semi-done discogs scraper (only album part)! we should try not to dupe work Smile

if you want i'll gladly leave the artist part to you Smile
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #21
basic scraper added to svn
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #22
not even a 'it does not work you silly sod'?
find quote
succo Offline
Fan
Posts: 361
Joined: Mar 2008
Reputation: 0
Location: Taranto, Italy
Post: #23
eh Smile
i tried it but it didn't retrieve much... does it need a last.fm account set up?

(this was a politically correct version for 'it does not work you silly sod' Tongue )
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #24
shouldnt require anything liek that no.

last.fm offers very little info. just artist, album name, year, releasedate, review and track listing.

even less for artists, although some of the biography's are good.

you did not get any info at all?
find quote
succo Offline
Fan
Posts: 361
Joined: Mar 2008
Reputation: 0
Location: Taranto, Italy
Post: #25
really i didn't try it much... i'm going to try again and see, thanks Smile
find quote
bashflyng Offline
Junior Member
Posts: 3
Joined: Dec 2008
Reputation: 0
Post: #26
Thanks a lot spiff, I don't have the time to check it now, but this is one of the features I was really looking forward for XBMC to have Smile
find quote
Loto_Bak Offline
Fan
Posts: 476
Joined: Sep 2004
Reputation: 0
Post: #27
Would you be able to scrape each tracks 'listeners' and apply it to the 'ratings' field of the database?
each album might need to be calculated relative to itself

[Image: loafamotive.gif]
find quote
spiff Offline
Retired Developer
Posts: 12,386
Joined: Nov 2003
Post: #28
i have absolutely no idea what that last sentence is supposed to mean.

http://www.last.fm/api

as you can see there is no way to search for tracks from a specific album
find quote
TheNME123 Offline
Junior Member
Posts: 26
Joined: Sep 2004
Reputation: 0
Thumbs Up   
Post: #29
Hey spiff,

i tried out your last.fm scraper and must say it works brilliant. The only downside are the crappy artist/album pics from last.fm but that is not your fault.

It would be nice if some kind of a language selection would be possible. Perhaps with a fallback to english if no information for the selected language is availabe.

Thanks for your great work

TheNME123
find quote
Stranger Offline
Fan
Posts: 341
Joined: Oct 2009
Reputation: 1
Location: London, UK
Post: #30
TheNME123 Wrote:Hey spiff,

The only downside are the crappy artist/album pics from last.fm but that is not your fault.

I agree. I've just started using this scraper instead of allmusic because of Last.fm's superior artist thumbs, but the thumbs scraped are dramatically compressed. Is this currently an unavoidable limitation or something simple I could improve on my end? Tweaking the thumbsize tag in advancedsettings is not making any difference. Hopefully I haven't resurrected this thread out of ignorance since my searches have got me nowhere.

Scraped

[Image: screenshot215v.jpg]

Original (Last.fm)

[Image: Aaliyah+nAaliyazh.png]
(This post was last modified: 2011-01-21 03:19 by Stranger.)
find quote