Making Kodi bypass IP blocks when using scrapers
#1
Hi, I'm using Advanced Launcher and my favorite scraper from that addon is the GameFAQs one, because it returns the most game screens to use as fanart and also gives me better covers. However, GameFAQs is blocking the IP of people who try to scrape their site. It was said in the Advanced Launcher thread that I could use proxies or a VPN to get by GameFAQs' block, however I need help to set Kodi up to make use of them.

I have successfully set up a proxy within Kodi in the past but that doesn't work well since GameFAQs will just block the proxy instead. Are there better alternatives? If so, could someone point me towards a tutorial to help set everything up?

I'm on Windows BTW, and the computer in which Kodi is installed receives wifi signal from a router, so it has a fixed IP in my network.
Reply
#2
Working backwards through your post... it does not matter what happens inside your house - wifi, Ethernet, whatever - it is your external ISP supplied IP Address that GameFAQs are watching out for.

Trouble is, there is no real answer for you. If GameFAQs don't want you scraping their website then they don't want you to do it. Full stop. They are well within their rights to block you. And using a VPN isn't going to help as they will just block that too which will mean many more legitimate users are blocked because you want to break their rules.

If I was in your position I would adjust the scraper so it runs VERY slowly. Take a COPY of each game image one at a time. Spread the requests out over hours. Take local copies of all the images and future updates will trigger less problems.

If there is no way to slow down the scraper, then scrap one show per hour by hand. Something that will more politely follow the rules of GameFAQs

Scraping of websites for artwork causes a lot of financial costs for the companies who are hosting those websites. So it makes a LOT of sense for GameFAQ to put these kinds of restrictions into place.
Reply
#3
(2015-06-14, 18:31)BatterPudding Wrote: Working backwards through your post... it does not matter what happens inside your house - wifi, Ethernet, whatever - it is your external ISP supplied IP Address that GameFAQs are watching out for.

Trouble is, there is no real answer for you. If GameFAQs don't want you scraping their website then they don't want you to do it. Full stop. They are well within their rights to block you. And using a VPN isn't going to help as they will just block that too which will mean many more legitimate users are blocked because you want to break their rules.

If I was in your position I would adjust the scraper so it runs VERY slowly. Take a COPY of each game image one at a time. Spread the requests out over hours. Take local copies of all the images and future updates will trigger less problems.

If there is no way to slow down the scraper, then scrap one show per hour by hand. Something that will more politely follow the rules of GameFAQs

Scraping of websites for artwork causes a lot of financial costs for the companies who are hosting those websites. So it makes a LOT of sense for GameFAQ to put these kinds of restrictions into place.

Yeah, I'll resort to doing it manually. I hope they don't block manual downloads! Their site is a gold mine because they have lots of rare covers.
Reply
#4
Maybe you can kindly ask them to unblock your IP range? Couldn't hurt, worst case back to above.
Reply
#5
(2015-06-15, 04:34)k4sh1n Wrote: Maybe you can kindly ask them to unblock your IP range? Couldn't hurt, worst case back to above.

Well, according to BatterPudding, GameFAQs will block the IP provided by my ISP. My IP is dynamic, though, because the modem is in bridge mode and connected to a router which distributes the signal to the house. So they can't really "lift" the ban, because it'll be lifted once my IP changes.

The machines I access GameFAQs with each have their own IP as assigned by the router, but if GameFAQs goes through the router and blocks my modem's IP directly, then I can't really do anything except wait for an IP reshuffle.
Reply

Logout Mark Read Team Forum Stats Members Help
Making Kodi bypass IP blocks when using scrapers0