[axlt2002]

That may sounds reasonable...but it happens almost every time to me...while, for example, Karellen didn't have this issue while scraping his TV show...may be @olympia can have some thoughts on this issue.

If too many requests (from the same IP?) is confirmed to be the issue, it would be nice, for example, to restart the scraping autonomously instead of exiting and oblige the user to search for new content (or restarting Kodi) in order to continue the scraping...

Anyway is strange that I'm the only one to have noticed this...

[Karellen]

olympia is looking into it but he is puzzled as nobody else has reported this problem and I have been scraping quite a lot over the last couple of weeks and have not encountered it.

Curious that it happens to you after only a few episodes. Are you using a VPN?

[Hitcher]

I've just rescraped my entire collection (1640 episodes) with no problems so it must have something to do with your connection.

[axlt2002]

Thanks guys for considering this issue that makes me puzzled as well! It is true that there is a problem (the log tells it) and it would be great to understand the reason why...

About the connection, I'm quite sure that I have tried among two different networks...one at my house that has a 200 Mbps optical fibre (the log is related to this one) and the second at my office with a high speed open connection (I can try on this again when I will be back to the office). Anyway, no VPN in both cases.

Just to understand: does the scraper connect to different servers according to the country in which you are? How is the number of requests to the server managed? For example is it based on the connection speed so that too fast connection produces too many requests? Does TMDB check the number of requests on IP basis?

It is defintely an interesting issue.

P.S.: Actually I was involved some time ago in a software development at my office...I had the "tag" of the guys that was unfortunately discovering hidden issues in a completely random manner!!!

[axlt2002]

Just to add that I have tried on another machine and I have experienced the same issue. I will try again to connet from another network and let you know...but I'm not expecting surprises...

- edit -

I have checked the TMDB api limitations; here an extract of what is reported:
 

We do enforce a small amount of rate limiting. Our current limits are 40 requests every 10 seconds and are limited by IP address, not API key. You can think of this is being burstable to 40 in a single second, or as an average of 4 requests/second. The timer will reset 10 seconds from your first request within the current 10 second "bucket". This means that if you trigger the limit you will have to wait up to 9 seconds before the timer resets but depending where you are within the 10 second window, it could be the very next second. You can use the X-RateLimit headers that get returned with every request to keep track of your current limits. If you exceed the limit, you will receive a 429 HTTP status with a Retry-After header. As soon your cool down period expires, you are free to continue making requests.

I think that this can explain my issue (including the 10 seconds window after which I can restart the scraping). May be is possible to manage this in the scraper.

@olympia, @Karellen what do you think?

[axlt2002]

Hi guys,

It was my intention to check the add-on behaviour expecting a .py module instead of an .xml...  So I really don't know where to start, and if it is possible, to add some kind of control about the number of requests in order to respect the limits imposed by the TMDB API.

An initial question is: is Kodi itself that dictates the timing of the requests? I was looking for some kind of "for cycle" in the add-on but I din't find nothing like that...

Thanks for any feedback.

Alex

[axlt2002]

Hi guys, I'm back to the office...

I have just tried to scrape Game of Thrones using the network here and I had the usual issue.

Just trust in you for a solution now...please tell me if you need some more information on my side... 

Waiting for a feedback.

Alex

- edit -

Just for a double check I have tried on Kodi 17.6 and no luck as well.

[olympia]

Apologies for not having the time to closely follow the troubleshooting of this, but it seems to be a specific issue on your side (doesn't mean something couldn't be tuned to avoid the issue).
Many are using the scraper, including team members, so if there was a general rate limiter issue, then it wouldn't been surely came onto the surface already. Just completed a quick test with my own library and it has been scraped without a hick.

So we would need to figure your specifics, but not sure when I will have time for that.

[axlt2002]

Thanks for your feedback olympia and no need to apologize of course

Well, actually I have tried on two different PCs and two different network connections with Kodi 17.6 and Kody 18 nighlty as well. So this makes me really puzzled and I can not understand why this issue can effect only me then...may be something related to my person?   

Do you have anyway any information to provide on how the requests to TMDB are performed? Is there a possiblity to manage the error 249 from the addon itself or everything is managed by Kodi? Looking at the code of the addon it seems that is really triggered by Kodi...please correct me if I'm wrong. The point is that I would love to look myself at the issue, but I think I'm missing some background here

Thanks for your effort, always!

[olympia]

No, that's indeed not possible with the XML scrapers, but would be really surprised it this was the underlining issue. For some reason Kodi on your site hitting the provider quicker than it should.
Are you using nfo files? Have you tried with different shows/ libraries?

[Martijn]

No, that's indeed not possible with the XML scrapers, but would be really surprised it this was the underlining issue. For some reason Kodi on your site hitting the provider quicker than it should.
Are you using nfo files? Have you tried with different shows/ libraries?

 Didn't have some rate limiter for musicbrainz in core code that if it receives a certain HTTP error it starts throttling?

[olympia]

This was developed specifically for music and for musicbrainz in core by @DaveBlake

...but again, would be good to see why this is happening here and if it happening under "normal circumstances" and not as a result and/ or together with another issue.

It's very suspicious why this is not occurring to any of us

[docwra]

@axlt2002 I think your issue is with the metadata provider not Kodi here. Its probably worth starting a thread on themoviedb forums with a log of a test scrape of your library.

Just let us know if there are any recommendations for API rate limiting that TMDB recommend as Kodi will be using this scraper a lot more in the future.

[docwra]

Actually I did a little digging and found out the limit:

The new system will share the state of an IP address across all 8 instances and provide proper balanced rate limiting. The rate limits themselves remain unchanged (max. 30 requests in a 10 second span). The key difference is in the response handling during your requests and when you trip the rate limits. I'll give you some examples so you can make changes to your code before we go live with this change.

SOURCE: https://www.themoviedb.org/talk/5317af69...05940352ae

I'm not sure how often the scraper requests things but this could be an issue depending on library content I guess...

EDIT: If we ever move to python scrapers then its possible to of course look out for the 429 code and wait. Something like this: https://gist.github.com/heyoni/7e7fab644...596b5b47ac

[axlt2002]

Hi guys,

Thanks for your feedbacks so far.

Again, what looks strange to me is that it really seams I'm the only one experiencing this issue...but I have tried over different machines and connections as well (the common thing is Windows 10, but I don't think it is the guilty...at least this time!)...sometimes the screaping reaches almost the end, sometimes is just a continuous exiting...hey, once I didn't have any issue! 

@olympia, yes, I have tried with and without nfo files as well as with different TV shows...the point is that the issue is just "random", that means is exactly related to the network connection status and speed...may be in this case a congested or low speed network helps in avoiding the problem. Of course I'm just brainstorming on this...anyway when the scraper exits, the log always reports error 429...

@docwra you exactly reported what I mentioned in post #155 above. Actually you referenced to something that was introduced in 2014 and that now foresees a limit of 40 requests every 10 seconds (take look to the page I linked). Of course, as you said, it would be great to have pyton scrapers to manage independently the requirements that the different sites impose; it is my understanding that Kodi use the same way to trigger the confirgured scraper so that we loose in same way the management of particular cases...Anyway, if from the debug log, the error 429 come out, that means Kodi is able to recognize it; would it then be possible to modify the code of Kodi to manage it? Another way could be just to set the scraping triggers in such a way that it would respect the most stringent requirements, i.e. to have no more than 40 requests every 10 seconds at a first rough instance (and have it refined in a second stage).

What do you think?

P.S.: Of course you are aware of that TMDB TV scraper is now the default TV scraper in the last Kodi 18 nightly builds............