Improved allmusic.com scraper (plus a few questions)
#61
talisto Wrote:Ok, so it seems that Discogs is checking the user agent of the request, and is rejecting XBMC's user agent. It seems that they're specifically targeting XBMC, since if I enter a random user agent, it works fine, yet any user agent starting with "XBMC" is rejected. This is fairly troubling since Discogs is the default scraper for XBMC, so probably 99% of the userbase now has a failing music scraper.

Clearly they don't want XBMC scraping their pages anymore. So I suppose the question now is, do we start spoofing a browser's user agent to get around the problem, or do we merely skip Discogs and move to other sources allowing more legitimate scraping (e.g. the last.fm API)?

(edit: the real unfortunate thing is that they're even blocking XBMC's usage of the API, so we can't even use that an alternative to the HTML scraping.)

That may address discogs but all of the scrapers are not working. I'll post a bug log but it seems odd that all of them would stop. Can anyone else test to see if this is a remote issue ?
Reply
#62
Ronald Pagan Wrote:That may address discogs but all of the scrapers are not working. I'll post a bug log but it seems odd that all of them would stop. Can anyone else test to see if this is a remote issue ?

Read my post previous to the one you replied to (#59); I've tested the other scrapers and they work fine. There's a new option you may have to enable.
Reply
#63
Well, i have updated to latest build, but i cant say, that all scapers work. So far only working scraper seems to be Allmusic, last.fm and maybe allmusic+polish sites. Freebase.org doesnt show up even when i try to look at it in firefox. Most importantly, when i try to scrap all artist at once (by pressing c on artists and using download artist infor for all) xbmc crashes after first atempt to download anything. It may be a problem with my system only, so I am asking if anyone else experiences such problems. Scraping from allmusic on individual artist works fine.

Another problem is with scraper settings. When you change default scraper in settings, it seems, that xbmc uses the previous one until restart.
Reply
#64
chumaj001 Wrote:Well, i have updated to latest build, but i cant say, that all scapers work. So far only working scraper seems to be Allmusic, last.fm and maybe allmusic+polish sites. Freebase.org doesnt show up even when i try to look at it in firefox.

Ok, maybe not ALL the scrapers are working, clearly the freebase scraper isn't going to work if the site is down, you've diagnosed that one yourself, that has nothing to do with XBMC. However this thread is about the allmusic scraper specifically, which DOES work, so lets try to keep this on-topic, ok? Smile

If you're having problems with XBMC crashing, try removing your userdata folder and starting with a fresh database. I know it sucks to nuke your database, but hey, that's all part of the fun of testing SVN builds.
Reply
#65
Please svn up and try again. There was a problem assigning the default album scraper and a possible crash (related to non-existent functions). Both issues should be fixed in current SVN.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#66
Using the latest SVN (24107-GL) and I think this is working great for me! Thanks so much Smile

Only issue is each time I go back to scaper settings, the default choices are selected again.

Anyone else experience this?

ie. I'm pretty sure it is obeying my choices, but if I go back to the scraper settings, my choices are gone.
Reply
#67
Please elaborate "each time I go back to scaper settings". Working fine for me.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#68
kiboy6 Wrote:Using the latest SVN (24107-GL) and I think this is working great for me! Thanks so much Smile

Only issue is each time I go back to scaper settings, the default choices are selected again.

Anyone else experience this?

ie. I'm pretty sure it is obeying my choices, but if I go back to the scraper settings, my choices are gone.

if you're talking about the music scrapers, is it possible you change those settings in the context menu and not in settings -> music -> scraper?

that would explain.
cheers,azido :;):
Reply
#69
since spiff's motto is 'test before commit' i'm sure there's nothing wrong,
but i find it a bit odd the discogs settings ended up in the video section. ;-)

http://trac.xbmc.org/changeset/24107
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#70
pfft. mere details
Reply
#71
ronie Wrote:since spiff's motto is 'test before commit' i'm sure there's nothing wrong,
but i find it a bit odd the discogs settings ended up in the video section. ;-)

http://trac.xbmc.org/changeset/24107

lol.. did they?
cheers,azido :;):
Reply
#72
azido Wrote:if you're talking about the music scrapers, is it possible you change those settings in the context menu and not in settings -> music -> scraper?

that would explain.

No, definitely in settings -> music -> scraper .... very strange Huh Like I said - the settings do seem to take effect, but are returned to default when i revisit settings -> music -> scraper.

Nobody else experiencing this? Would you suggest nuking my profile folder and doing a completely clean XBMC install?
Reply
#73
kiboy6 Wrote:No, definitely in settings -> music -> scraper .... very strange Huh Like I said - the settings do seem to take effect, but are returned to default when i revisit settings -> music -> scraper.

Nobody else experiencing this?

i can confirm.
looks like the settings are not being saved to guisettings.xml, so they're lost after restarting xbmc.

kiboy6 Wrote:Would you suggest nuking my profile folder and doing a completely clean XBMC install?

i tried with a fresh guisettings.xml file and still get the same issue, so i don't think reinstalling xbmc would do you any good.
Do not PM or e-mail Team-Kodi members directly asking for support.
Always read the Forum rules, Kodi online-manual, FAQ, Help and Search the forum before posting.
Reply
#74
Is there a ticket for this issue?
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
Reply
#75
svn revision?
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
Improved allmusic.com scraper (plus a few questions)00