TMDB scraper failures for previously working movies
#1
Hi,

I've a folder with a couple of hundred movies which - until today - have been quite happily retrieving data using the TMDB scraper. However, today I tried and failed to add a couple of new ones which I could clearly see were on TMDB. It further turns out that if I remove then re-add (and rescan) the source, many of the movies that used to work are no longer identified. Some 90 or so are now failing.

An extract from the kodi.log for one that works and one that doesn't shows:

Code:
18:04:55.801 T:12260   DEBUG: VideoInfoScanner: Scanning dir 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/' as not in the database
18:04:55.815 T:12260   DEBUG: VideoInfoScanner: No (new) information was found in dir nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/
18:04:55.925 T:12260   DEBUG: VideoInfoScanner: Scanning dir 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/' as not in the database
18:04:55.937 T:12260   DEBUG: CVideoDatabase::GetMovieId (nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/Pirates of the Caribbean At Worlds End (2007).mp4), query = select idMovie from movie where idFile=4391
18:04:55.944 T:12260   DEBUG: VideoInfoScanner: No NFO file found. Using title search for 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/Pirates of the Caribbean At Worlds End (2007).mp4'
18:04:55.944 T:12260   DEBUG: ADDON::CScraper::FindMovie: Searching for 'At World's End' using The Movie Database scraper (path: 'C:\Users\chris\AppData\Roaming\Kodi\addons\metadata.themoviedb.org', content: 'movies', version: '3.9.3')
18:04:55.959 T:12260   DEBUG: scraper: CreateSearchUrl returned <url>http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&amp;query=At%20World%27s%20End&amp;year=2007&amp;language=en</url>
18:04:55.959 T:12260   DEBUG: CurlFile::Open(14AF8558) http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=At%20World%27s%20End&year=2007&language=en
18:04:55.998 T:12260   DEBUG: CScraperUrl::Get: Using "UTF-8" charset for "http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=At%20World%27s%20End&year=2007&language=en"
18:04:55.998 T:12260   DEBUG: scraper: GetSearchResults returned <results></results>
18:04:55.998 T:12260   DEBUG: ADDON::CScraper::FindMovie: Searching for 'At World's End' using The Movie Database scraper (path: 'C:\Users\chris\AppData\Roaming\Kodi\addons\metadata.themoviedb.org', content: 'movies', version: '3.9.3')
18:04:55.999 T:12260   DEBUG: scraper: CreateSearchUrl returned <url>http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&amp;query=At%20World%27s%20End&amp;year=&amp;language=en</url>
18:04:55.999 T:12260   DEBUG: CurlFile::Open(14AF8558) http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=At%20World%27s%20End&year=&language=en
18:04:56.040 T:12260   DEBUG: CScraperUrl::Get: Using "UTF-8" charset for "http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=At%20World%27s%20End&year=&language=en"
18:04:56.041 T:12260   DEBUG: scraper: GetSearchResults returned <results></results>
18:04:56.041 T:12260 WARNING: No information found for item 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/Pirates of the Caribbean At Worlds End (2007).mp4', it won't be added to the library.
18:04:56.041 T:12260   DEBUG: VideoInfoScanner: No (new) information was found in dir nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/
18:04:56.080 T:12260   DEBUG: VideoInfoScanner: Scanning dir 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/extrafanart/' as not in the database
18:04:56.080 T:12260   DEBUG: VideoInfoScanner: No (new) information was found in dir nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/extrafanart/
18:04:56.109 T:12260   DEBUG: VideoInfoScanner: Scanning dir 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/extrathumbs/' as not in the database
18:04:56.109 T:12260   DEBUG: VideoInfoScanner: No (new) information was found in dir nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/At World's End (2007)/extrathumbs/
18:04:56.202 T:12260   DEBUG: VideoInfoScanner: Scanning dir 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/Dead Man's Chest (2006)/' as not in the database
18:04:56.208 T:12260   DEBUG: CVideoDatabase::GetMovieId (nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/Dead Man's Chest (2006)/Pirates of the Caribbean Dead Mans Chest (2006).mp4), query = select idMovie from movie where idFile=4392
18:04:56.214 T:12260   DEBUG: VideoInfoScanner: No NFO file found. Using title search for 'nfs://192.168.0.2/mnt/HD/HD_a2/Public/Shared Videos/Movies/Pirates Of The Caribbean/Dead Man's Chest (2006)/Pirates of the Caribbean Dead Mans Chest (2006).mp4'
18:04:56.214 T:12260   DEBUG: ADDON::CScraper::FindMovie: Searching for 'Dead Man's Chest' using The Movie Database scraper (path: 'C:\Users\chris\AppData\Roaming\Kodi\addons\metadata.themoviedb.org', content: 'movies', version: '3.9.3')
18:04:56.223 T:12260   DEBUG: scraper: CreateSearchUrl returned <url>http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&amp;query=Dead%20Man%27s%20Chest&amp;year=2006&amp;language=en</url>
18:04:56.223 T:12260   DEBUG: CurlFile::Open(14AFAC20) http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=Dead%20Man%27s%20Chest&year=2006&language=en
18:04:56.267 T:12260   DEBUG: CScraperUrl::Get: Using "UTF-8" charset for "http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=Dead%20Man%27s%20Chest&year=2006&language=en"
18:04:56.267 T:12260   DEBUG: scraper: GetSearchResults returned <results><entity><title>Pirates of the Caribbean: Dead Man's Chest</title><id>58</id><year>2006</year><url cache="tmdb-en-58.json">http://api.tmdb.org/3/movie/58?api_key=f7f51775877e0bb6703520952b3c7840&amp;language=en</url></entity><entity><title>Pirates of the Caribbean: Dead Man's Chest</title><id>58</id><year>2006</year><url cache="tmdb-en-58.json">http://api.tmdb.org/3/movie/58?api_key=f7f51775877e0bb6703520952b3c7840&amp;language=en</url></entity></results>

In both cases, if I call the API from a browser I get a valid response back from TMDB. Admittedly in this example one does return multiple results and the other does not, but this factor isn't consistent across all failures. However, what does seem to be consistent is the order in which the JSON data is return, and particularly whether the "total_results" field appears at the beginning or end.

E.g. for a failed result, the JSON is:

Code:
{"page":1,"total_results":4,"total_pages":1,"results":[{"vote_count":3612,"id":285,"video":false,"vote_average":6.9,"title":"Pirates of the Caribbean: At World's End","popularity":27.43969,"poster_path":"\/bXb00CkHqx7TPchTGG131sWV59y.jpg","original_language":"en","original_title":"Pirates of the Caribbean: At World's End","genre_ids":[12,14,28],"backdrop_path":"\/8ZgpAftUiYTU76IhUADITa3Ur9n.jpg","adult":false,"overview":"Captain Barbossa, long believed to be dead, has come back to life and is headed to the edge of the Earth with Will Turner and Elizabeth Swann. But nothing is quite as it seems.","release_date":"2007-05-19"},{"vote_count":0,"id":247646,"video":false,"vote_average":0,"title":"The Lion at World's End","popularity":1.065,"poster_path":"\/g03DkgBVjCw6V25m3elrw2NZkdI.jpg","original_language":"en","original_title":"The Lion at World's End","genre_ids":[99,12],"backdrop_path":"\/qe2532dMhyt1oJOOIxYtH7iS8tg.jpg","adult":false,"overview":"The astonishing true story of a zoo-born lion who found himself in the pet department of Harrods, then the cellar of an antique shop in London and, ultimately the wilds of Africa.","release_date":"1971-02-28"},{"vote_count":8,"id":19306,"video":false,"vote_average":5.4,"title":"At World's End","popularity":1.209159,"poster_path":"\/kyx08K6qBzXbepULUOu7EfXEOgO.jpg","original_language":"da","original_title":"Ved verdens ende","genre_ids":[12,28,35,53],"backdrop_path":"\/c1D9mKNYukz0q2ToifkaSor8KLJ.jpg","adult":false,"overview":"An english tv-team discovers a special white flower in the indonesian rainforest, but they never get to investegate it further, before the danish recluse Severin has shot them down. A danish special enforcement with a psychiatrist in front is sent off, and they find out, that Severin claims, that he is 129 years old, and it's the flower, which keeps him young. Soon all hunts the white flower, which apparently gives eternal life.","release_date":"2009-10-09"},{"vote_count":28,"id":203351,"video":false,"vote_average":2.1,"title":"Atlantic Rim","popularity":1.020435,"poster_path":"\/9kgK6GpzqreO8DUbLY1L4QD1nrC.jpg","original_language":"en","original_title":"Atlantic Rim","genre_ids":[28,878],"backdrop_path":"\/kX83lWzst7U7z4zRbykv1Zw3VVY.jpg","adult":false,"overview":"When monsters suddenly appear from the bottom of the Atlantic Ocean, a special team pilots giant robots to combat the new threat.","release_date":"2013-07-09"}]}

Compared to a successful result where the total_results appears at the end of the JSON:

Code:
{"page":1,"results":[{"poster_path":"\/waFr5RVKaQ9dzOt3nQuIVB1FiPu.jpg","adult":false,"overview":"Captain Jack Sparrow works his way out of a blood debt with the ghostly Davey Jones, he also attempts to avoid eternal damnation.","release_date":"2006-06-20","genre_ids":[12,14,28],"id":58,"original_title":"Pirates of the Caribbean: Dead Man's Chest","original_language":"en","title":"Pirates of the Caribbean: Dead Man's Chest","backdrop_path":"\/hdHgIcljPHli4xaJGt0INz8Gn3J.jpg","popularity":21.639369,"vote_count":4154,"video":false,"vote_average":6.9}],"total_results":1,"total_pages":1}

This behaviour seems consistent in all the results I've looked at - if the total_results comes at the end then Kodi can parse the response. If it comes at the start, Kodi cannot parse the response. The number of results returned seems to have no bearing on success or failure (or where the total_results field is placed).

Has anyone else seen this behaviour?

Chris
Reply
#2
I encountered the same behaviour on 3 different Kodi clients.
Reply
#3
Same issues. Current roughly 20% of my movies are no longer recognized. IMDB scraping is working fine. As I did not install any updates I would assu.e the problem lies within the movie database or their API has changed.
Reply
#4
I have the same symptoms, with kodi 17.3 and 17.1 on ubuntu and osmc.
Code:
21:11:54.479 T:139701176485632   DEBUG: Get: Using "UTF-8" charset for "http://api.tmdb.org/3/search/movie?api_key=f7f51775877e0bb6703520952b3c7840&query=Woodstock&year=1970&language=fr"
21:11:54.480 T:139701176485632   DEBUG: scraper: GetSearchResults returned <results></results>
Reply
#5
Just to confirm the scraper is working poorly. I added about 100 movie titles and only about 20 caught. A simple title like Gran Torino (2008) failed.

Looks like a Kodi interface issue to me.

Rgds.
Reply
#6
I have the same problem with TMDB (on 3 clients). Only about 10% of all movies will be recognized. The TMDB website in up and running. Everything worked fine yesterday.
Reply
#7
Kodi 14.2

I have the same issues. For example The feature Logan (2017) is not scraping anymore. For Some of the titles I was able to resolve The issue by changing the name of the folder from capitals to regular Lines.
Something is wrong with the themoviedb.
Reply
#8
I'm having the same problem. I was scraping movie titles yesterday with no issues but this morning, I keep getting "No information found" errors in my debug log, no matter what the movie is. I even tried renaming one of them to "Star Wars". Still got nothing. C'mon, no information found for Star Wars? I tried the Universal scraper configured for TMDB and got the same results. I think the problem is at TMDB. I tried to submit a trouble report on their forum but every time I tried, the submit failed with a very unhelpful error message.
Reply
#9
Layout of tmdb API returns has changed (or behaving differently due to other reasons). I pushed a scraper update to the repo, but it will take a while until it gets updated... Till then, please be patient.
Reply
#10
Same conversation happening here...

https://forum.kodi.tv/showthread.php?tid=315803
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#11
@olympia: thx for the enlightenment.

Just updated tmdb addon and everythings working again. Wink
Reply
#12
Thanks @olympia for the fast response and patch!
Reply
#13
Hello,
for me it still does not work. My tmdb addon is v 3.9.3 now but still get errors with most movies.
I tried Kodi 17.3 (LibreElec on Raspberry Pi) and Kodi 16.1.
As an example, it will not scan the movie "Moby Dick (1956)".
Reply
#14
@boh73,

I just tested your movie Moby Dick (1956) and it scraped first time for me.

I am also using v3.9.3.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#15
Hello,

On my Kodi 17.03 with TMDB v3.9.3 the scraper could not connect to Server. No Film Information is loading.
The Addon dose not find any Update. If I search on Kodi.tv for the Addon, I´ll find v3.9.4 but I can´t install it by Installing from Zip.
Update Error occours.

What going arround?
Reply

Logout Mark Read Team Forum Stats Members Help
TMDB scraper failures for previously working movies0