2009-09-06, 08:03
Hey gang,
Longtime user, (almost) first-time poster.
Anyhoo, I've been searching for the best music scraper for a while now, and haven't really been happy with any of them.. allmusic.com has fantastic artist information/reviews but awful photos (and the scraper never really seemed to get all the proper info), discog has decent photos but really limited artist information, and last.fm has medicore everything. So I started mucking around with the existing allmusic scraper (from r22528) and I've fixed a few problems with it, and improved it a bit (for my own needs, anyhow). Here's what I've done:
- The artist information (aside from the bio) wasn't being parsed properly; the ParseAMGArtist function was being passed the value "test" instead of the actual URL. Fixed.
- The album information wasn't being parsed properly either; the ParseAMGAlbum function was being passed the value "placeholder" instead of the actual URL. Fixed.
- The caching was glitchy, it seems that the cache file should be unique to each artist, whereas it was set to use the same cache file for every artist, so subsequent lookups would often have duplicate/incorrect information from the previous artist. Fixed.
- The scraper was set to only get thumbs from htbackdrops (same with all the other music scrapers now?), which is fine and may be preferrable to some, but htbackdrops barely has any thumbnails for the artists in my library. I noticed that discogs generally has decent photos for their artists, and are quite extensive, so I've changed the scraper to also check discogs for thumbs (though it will still use the thumb from htbackdrops as the primary if one is available).
With these changes, pretty much every artist and album in my library has a proper thumb, as well as full bio/reviews/discography/etc. Definitely a HUGE improvement.
I've posted it here: EDIT: my changes are now in the latest SVN builds. Just download that instead!
Hopefully it helps someone else as well
Maybe someone can merge these fixes into SVN.
1st question for the scraper gurus: is it possible to nest URL fetches or functions? I couldn't figure out how to fetch a URL, parse it, and then use the resulting string to fetch another URL and parse that.
2nd question: Is there a way to ensure a variable/buffer is URL encoded properly for use in a GET string?
Thanks!
Longtime user, (almost) first-time poster.
Anyhoo, I've been searching for the best music scraper for a while now, and haven't really been happy with any of them.. allmusic.com has fantastic artist information/reviews but awful photos (and the scraper never really seemed to get all the proper info), discog has decent photos but really limited artist information, and last.fm has medicore everything. So I started mucking around with the existing allmusic scraper (from r22528) and I've fixed a few problems with it, and improved it a bit (for my own needs, anyhow). Here's what I've done:- The artist information (aside from the bio) wasn't being parsed properly; the ParseAMGArtist function was being passed the value "test" instead of the actual URL. Fixed.
- The album information wasn't being parsed properly either; the ParseAMGAlbum function was being passed the value "placeholder" instead of the actual URL. Fixed.
- The caching was glitchy, it seems that the cache file should be unique to each artist, whereas it was set to use the same cache file for every artist, so subsequent lookups would often have duplicate/incorrect information from the previous artist. Fixed.
- The scraper was set to only get thumbs from htbackdrops (same with all the other music scrapers now?), which is fine and may be preferrable to some, but htbackdrops barely has any thumbnails for the artists in my library. I noticed that discogs generally has decent photos for their artists, and are quite extensive, so I've changed the scraper to also check discogs for thumbs (though it will still use the thumb from htbackdrops as the primary if one is available).
With these changes, pretty much every artist and album in my library has a proper thumb, as well as full bio/reviews/discography/etc. Definitely a HUGE improvement.
I've posted it here: EDIT: my changes are now in the latest SVN builds. Just download that instead!
Hopefully it helps someone else as well
Maybe someone can merge these fixes into SVN.1st question for the scraper gurus: is it possible to nest URL fetches or functions? I couldn't figure out how to fetch a URL, parse it, and then use the resulting string to fetch another URL and parse that.
2nd question: Is there a way to ensure a variable/buffer is URL encoded properly for use in a GET string?
Thanks!
