Heads Up on Failure with IMDB Scraping
#1
It appears IMDB has instituted some new measures to reduce the traffic from programatic scraping of their site.  I know both the Team Kodi supported scrapers for TMDb include the option to scraper IMDB ratings, and with this change from IMDB, that functionality is broken.  IMDB charges $50K a year plus additional for bandwidth for API access to the ratings, so it is unlikely that's going to be a viable path moving forward (although maybe IMDB has a program for open source projects - there are some conversations happening around that).

If you support a scraper addon that uses IMDB, please check to see if your scraper at least handles a 403 http reply (forbidden) gracefully and doesn't crash.

UPDATE: It appears IMDB has backtracked at least partially on some of it's changes.  The TMDb TV Show scraper is again able to get IMDB ratings, and based on what I'm seeing, I suspect that will be true for the TMDb Movies scraper as well (or any scraper that uses the IMDB json to get data).  It's hard to say how long this will last though, so the suggestion above remains valid.  If you are maintaining a scraper, make sure it gracefully handles IMDB if no data is returned.
Reply
#2
Do we have any updates on the situation, ie. what portions of using IMDB should currently work? Also, where can I find information about the situation from the IMDB side, at least their https://community-imdb.sprinklr.com/ discussion forum does not seem to have anything on this?
Reply
#3
I haven't heard anything about any new issues, so I guess it's still working.  This is the challenge when you scrape web sites instead of using an API.  It can break at anytime for any number of reasons.  There's no telling.  I'd use the IMDB API, but it's several hundred thousand dollars to get access.  That's not happening.
Reply
#4
There is a company IMDB-API which has some form of deal with IMDB.com and a full API for IMDB.   I did a 1 year trial with them for a reasonable price and up to 5K queries per day.  Overall I have been satisfied with this approach vs. $50K.  I use it for pulling actor / actress images for my Python artwork tool which checks TMDB first for actor / actress artwork and then goes to IMDB if it wasn't found.  Probably not an option for everyone but it is an alternative depending upon your needs.  Certainly not an answer for broader Kodi shared usage.  Their APi is fairly straightforward and has worked well.  It broke once when they made a change but it was an easy fix.  It isn't blazing fast like TMDB but it gets the job done.



Jeff
Running with the Mezzmo Kodi addon.  The easier way to share your media with multiple Kodi clients.
Service.autostop , CBC Sports, Kodi Selective Cleaner and Mezzmo Kodi addon author.
Reply

Logout Mark Read Team Forum Stats Members Help
Heads Up on Failure with IMDB Scraping0