[olympia]

P.S.: Of course you are aware of that TMDB TV scraper is now the default TV scraper in the last Kodi 18 nightly builds............

 Yes, so we will now see if it is really only you or not

By the way I have a 1000Mbit optical connection here, so I presume my network is fast enough so I should see the issue arising if this was the issue.
...but to be honest, I am a bit clueless at the moment. I will need to find some time to look at your log. I was browsing it through in 1 min the other day and there were some suspicious lines suggesting that it is maybe about caching is not kicking in at your side as it should and Kodi is hammering more the provider than it should. But I wouldn't know at the moment how this can happen. So definitely some more digging is needed, but indeed I am wondering how many will be impacted by this now, that tmdb became the default scraper.

[axlt2002]

Ok, your connection is even faster than mine...hope you will have time to check the debug log in more detail...I took a look but it is really something that I can not decode...  I have just noticed that the issue seems to happen in relation to the download of the images (i.e. thumbs of episodes) that ara almost all at high resolution or not less than 1280x720...may be chacing is playing a role in it, but I'm not an expert on this...

@olympia do you think you can give a try with exactly the same TV Show I was trying? I can send you a zip file containing all the .nfo and .strm files. Would that be worth to be tried?

Well, let's see if someone else will experience the issue in the near future...

[olympia]

I can give it a try later, yes.

[axlt2002]

I can give it a try later, yes.

Thanks a lot!

I have uploaded a test package (please note that the .strm files are just samples that are not playable). I have just tried to scrape it and it exited three times; you can find the related log as well (not in debug mode).

I will wait for news (good or bad).

Thanks in advance as usual.

Alex

[docwra]

Yep just tested latest nightly and I also get a 429 error:

ERROR: CCurlFile::FillBuffer - Failed: HTTP returned error 429 
ERROR: CCurlFile::Open failed with code 429 for http://api.themoviedb.org/3/tv/1402/seas...en,en,null:
ERROR: ADDON::CScraper::Run: Unable to parse web site

Full Log: https://paste.kodi.tv/qovebodepa.kodi

From what I can tell, large TV series needs a lot of API hits that is hitting the 40 requests in 10 seconds limit.

My connection is also very fast 500mbit fibre.

[olympia]

@zag don't forget about caching, large TV series should not make a big difference...

...and I just ran this test folder and it has got scraped without any issue

[axlt2002]

Finally!    Thanks a lot for chacking @docwra! 

Looking at your log, the scraper is exited at the same point...so I have tried agian to check if the error is systematic but it happened on other episodes:

Code:

12:39:13.940 T:4152  NOTICE: VideoInfoScanner: Starting scan ..
12:39:33.087 T:4152   ERROR: CCurlFile::FillBuffer - Failed: HTTP returned error 429
12:39:33.087 T:4152   ERROR: CCurlFile::Open failed with code 429 for http://api.themoviedb.org/3/tv/1402/season/5/episode/10? [omitted by me]:
12:39:33.087 T:4152   ERROR: ADDON::CScraper::Run: Unable to parse web site
12:39:33.095 T:4152  NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:19
12:39:47.352 T:4152  NOTICE: VideoInfoScanner: Starting scan ..
12:39:55.430 T:4152   ERROR: CCurlFile::FillBuffer - Failed: HTTP returned error 429
12:39:55.430 T:4152   ERROR: CCurlFile::Open failed with code 429 for http://api.themoviedb.org/3/tv/1402/season/7/episode/6? [omitted by me]:
12:39:55.430 T:4152   ERROR: ADDON::CScraper::Run: Unable to parse web site
12:39:55.430 T:4152 WARNING: No information found for item 'C:\Users\00918203\Videos\Serie TV\The Walking Dead (2010)\', it won't be added to the library.
12:39:55.438 T:4152  NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:08
12:39:59.111 T:4152  NOTICE: VideoInfoScanner: Starting scan ..
12:40:07.556 T:4152   ERROR: CCurlFile::FillBuffer - Failed: HTTP returned error 429
12:40:07.556 T:4152   ERROR: CCurlFile::Open failed with code 429 for http://api.themoviedb.org/3/tv/1402/season/9/episode/2? [omitted by me]:
12:40:07.556 T:4152   ERROR: ADDON::CScraper::Run: Unable to parse web site
12:40:07.556 T:4152 WARNING: No information found for item 'C:\Users\00918203\Videos\Serie TV\The Walking Dead (2010)\', it won't be added to the library.
12:40:07.584 T:4152  NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:08
12:40:10.321 T:4152  NOTICE: VideoInfoScanner: Starting scan ..
12:40:13.460 T:4152   ERROR: XFILE::CDirectory::GetDirectory - Error getting
12:40:13.474 T:4152  NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 00:03

[axlt2002]

...and I just ran this test folder and it has got scraped without any issue

Ouch..........

Well, at least @docwra proofed that I'm not crazy...I usually thake the positive side of everything! 

So, any suggesiton now?

[olympia]

I am also puzzled by this and it is not possible to slow down the scraper to test.

[axlt2002]

Yes, it is definitely puzzling...do you know which is the code part of Kodi that triggers the requests? May be we can just add some kind of xbmc.sleep command to make 3 requests per second as a first try...but if the issue is related to chaching, I really don't know what suggest...is there some buffer size set somewhere? Maybe you know some developer dealing with it...

[docwra]

The music scraper already does this I believe, maybe @DaveBlake knows how its specifically done.

I have also asked on the TMDB forums about extending API the rate limit.

[axlt2002]

I appreciate your help in this @docwra! 

I don't think that increasing the rate can be an agreable solution for them...I think is duty of the programs/addons using the API to take care of not exceeding the limit. Anyway let's see their answer.

Keep us updated! 

[axlt2002]

Ok, I gave a look to my log posted here and made a very rough calculation based on the curl requests to http://api.themoviedb.org/..., for example:

Code:

CurlFile::Open(0x25ad7b446b0) http://api.themoviedb.org/3/tv/1402/season/2/episode/3

This are the relevant data:
 

• First curl at: 17:34:56
  
• Error 429 at: 17:35:06
  
• Time interval: around 10 seconds
  
• Number of curls in that interval: 41
  

So we are really playing around the TMDB API rate limit of 40 requests every 10 seconds...

[axlt2002]

I am also puzzled by this and it is not possible to slow down the scraper to test.

May be we could fork the github xbmc project and try some modifications...but I really don't know where to put the hands on...

Do you think we should open a related issue on github?

[scudlee]

Ok, I gave a look to my log posted here and made a very rough calculation based on the curl requests to http://api.themoviedb.org/..., for example:

Code:

CurlFile::Open(0x25ad7b446b0) http://api.themoviedb.org/3/tv/1402/season/2/episode/3

This are the relevant data:
 

• First curl at: 17:34:56
  
• Error 429 at: 17:35:06
  
• Time interval: around 10 seconds
  
• Number of curls in that interval: 41
  

So we are really playing around the TMDB API rate limit of 40 requests every 10 seconds...    

 If you turn on component-specific logging in the Kodi system settings, and select verbose logging for the libcURL library, you'll be able to see the Rate Limit in the returned headers in the debug log.

e.g.

Code:

16:13:18.911 T:8164 DEBUG: Curl::Debug - HEADER_IN: X-RateLimit-Limit: 40
16:13:18.912 T:8164 DEBUG: Curl::Debug - HEADER_IN: X-RateLimit-Remaining: 23
16:13:18.912 T:8164 DEBUG: Curl::Debug - HEADER_IN: X-RateLimit-Reset: 1542816811