[doglover]

We had to block the access of Kodi (IPTV) for their request of EPG.

At Rytec we are not against the Kodi community using our XMLTV EPG.
Only the grab plug-in seems to be badly programmed:

Some examples:

Code:

85.245.70.103 - - [01/Apr/2015:18:27:28 +0200] "GET /rytecxmltvportugal.gz HTTP/1.1" 404 306 "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:28:30 +0200] "HEAD /rytecxmltvportugal.gz HTTP/1.1" 404 - "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:28:30 +0200] "GET /rytecxmltvportugal.gz HTTP/1.1" 404 306 "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:28:53 +0200] "GET /rytecxmltvportugal.gz HTTP/1.1" 404 306 "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:29:13 +0200] "HEAD /rytecxmltvportugal.gz HTTP/1.1" 404 - "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:29:30 +0200] "GET /rytecxmltvportugal.gz HTTP/1.1" 404 306 "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:29:44 +0200] "GET /rytecxmltvportugal.gz HTTP/1.1" 404 306 "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:34:46 +0200] "HEAD /rytecxmltvportugal.gz HTTP/1.1" 404 - "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"
     85.245.70.103 - - [01/Apr/2015:18:34:47 +0200] "GET /rytecxmltvportugal.gz HTTP/1.1" 404 306 "-" "Kodi/15.0-ALPHA2 (X11; Linux armv6l) OpenELEC/5.0 HW_RaspberryPi/1.0 App_Bitness/32 Version/15.0-ALPHA2-Git:ada3a4b"

All from the same IP address, 10-15 seconds apart.
Which will lead to an enormous amount of requests. Here the file did not exist.

Also when the file did exist, requests came in form the same IP address 5 minutes apart, downloading the file.
We have collected now tens of thousand IP addresses doing the same.

This lead to overload of the servers, in so far I could not even update them anymore.

These examples are for the portugese EPG. But all countries are affected. Anoter really bad on is the german file (skyde)

jin at KODInerds (germany) did some testing and found this:

ok here is the result of some testing.

how does the iptv simple pvr addon work?
there is a setting for epg where you can insert an url to an xml file, e.g. the rytec xml url.
and there is a setting to cache this file. default is off i think, which is pretty bad in this case.
if cache is off, unfortunatly, for every single channel it is requesting the file .
if cache is on and the file is not an xml or corrupted, it ignores that and is also trying to download the url for every channel.

so it actually doesnt help if you create a dummy(corrupted) 1gb zipped file, because it is still downloading the small file for every channel, even if cache is on. and it doesn't look like its extracting the file.

pls get in contact with afedchin(author of iptv simple) at forum.kodi.tv and/or make a new thread here: forum.kodi.tv/forumdisplay.php?fid=215

you probably have to block kodi user somehow(user agent) if that helps - creating dummy files dont

Conditions at which the software should confirm too before access can be granted again:

The software may for example not contain continuous update checks (like every 5 minutes or so), it may only download once a day, it must not contain a retry loop, it should not download between 3 and 7 GMT, etc.



The XMLTV files are only updated once a day, so continuous checks on updates are not needed. They only cause an heavy load on the servers.
It should accept no for an answer, Not retrying continuously. At the moment there are 4 mirrors. If there is a negative answer from one mirror, you can try on the other mirrors, but leave the one which said no alone.
Between 3 and 7 GMT. the servers are updated. When there is heavy traffic going on, on these servers we are not even able to update the servers.
So one successful download a day should be sufficient. No more data is the gathered doing more downloads a day.
In case of emergency there is a manual update request allowed. (i.e. the users presses a button : Update XMLTV manually)

If the Kodi community, will donate an extra mirror, I will gladly stuff it with the XMLTV-epg files.

Willy

[Fry7]

Ahhh... at last I know now that it wasn't my config that's gone mad...

Since I used (half a year or so now) your epg data, I simply set up a local apache, gave it a cronjob to download your file ONCE a night (sadly during your update period as I just learned) and linked my local devices to my local webserver. Worked a treat.

Now that I learned why is doesn't anymore I sincerely hope that a soltution will be found, soon...

[doglover]

In order to illustrate further the problem, we have in april now for 7 days:

for rytecxmltvskyde.gz 6,094,291 refused requests.
or 87 061 refused requests per day.

or 61 refused requests/minute.
And this on one server.

Pls. remove the requests for rytecxmltvskyde.gz from your configuration.
The file does not exist anymore. You will not get any data. It only generates traffic on our servers.

Willy

[afedchin]

Hi,

there is a setting for epg where you can insert an url to an xml file, e.g. the rytec xml url.
and there is a setting to cache this file. default is off i think, which is pretty bad in this case.

It's not true. Caching is enabled by default. GitHub

if cache is off, unfortunatly, for every single channel it is requesting the file .

It's not true. The addon loads an EPG data (from cache if it is enabled and cache is not expired) only once for first channel GitHub. As you can see if EPG loaded and requested period is same the addon doesn't try to load EPG again. And it does not matter whether caching is enabled or not.

if cache is on and the file is not an xml or corrupted, it ignores that and is also trying to download the url for every channel.

It's not true. If the addon cannot load file it try to load it again two times and if the file still cannot be loaded the addon marks EPG as loaded GitHub. And it does not matter whether caching is enabled or not.

The caching will work correctly if a server supports and sends Last-Modified header. If the server doesn't support this header the addon cannot compare time of modification of the cached file and the remote file and will try to load file again.

The XMLTV files are only updated once a day, so continuous checks on updates are not needed. They only cause an heavy load on the servers.

Your servers is not alone. Another servers can update EPG a more often. The addon checks for updates only by HEAD request and if server replied correctly the addon will not load file again and again. Also users can configure update interval in Kodi settings TV -> Guide -> EPG update interval.

plug-in seems to be badly programmed:

Wellcome to the GitHub with fixes. The addon is open sourced.

BR,
Anton

[doglover]

OK.
You deny that there is a problem.
This will mean, that we will keep on blocking Kodi.
Too bad.

Willy

[afedchin]

The problem exists, but not in the addon. Users can disable cache, users can configure a small update interval, your servers can not send Last-Modified header. You can select any issue.

It is your right block or not.

BR,
Anton.

[Fry7]

This will mean, that we will keep on blocking Kodi.

[doglover]

The problem exists, but not in the addon. Users can disable cache, users can configure a small update interval, your servers can not send Last-Modified header. You can select any issue.

It is your right block or not.

BR,
Anton.

If you do not disable these option in your plug-in, we cannot handle the traffic.

Last Modified header method is not reliable.

Willy

[afedchin]

Which options I shoul disable? And why I should disable an option if your servers doesn't work correctly?

Last Modified header method is not reliable.

Which method is reliable?

http://www.w3.org/Protocols/rfc2616/rfc2...l#sec14.29

HTTP/1.1 servers SHOULD send Last-Modified whenever feasible.

BR,
Anton

[Ned Scott]

OK.
You deny that there is a problem.
This will mean, that we will keep on blocking Kodi.
Too bad.

Willy

As I mentioned to you in an e-mail, we believe the issue is coming from a modified version of the add-on that another group distributes. The traffic you see is not due to an add-on that Team Kodi has control over. While this problem is not caused by us, I am attempting to contact that other group so that they can correct their version. We are sorry that this has happened, and will be glad to help in any way we can, but we did not cause your high traffic. The IPTV Simple Client add-on included with Kodi doesn't even come pre-configured to use any EPG.

I hope this clears up any misunderstandings so that we can identify and contact whoever is actually causing you this issue, so that it may be resolved.

[JinJin]

hi everyone,
this is jin from kodinerds.net who did a few tests and im wondering if anybody even tried to reproduce them?

so this is what i get on kodi alpha 2 windows 8 - guess the versions doesnt matter.

if you turn off cache and using a fresh m3u with new channel names it is downloading the xml for every channel.
the same will happen when the link is not an xml or the xml is corrupted.

http://pastebin.com/v6ifDcA4 - cache off with legit url
http://pastebin.com/CVHHkJzh - cache on with google url

[Ned Scott]

hi everyone,
this is jin from kodinerds.net who did a few tests and im wondering if anybody even tried to reproduce them?

so this is what i get on kodi alpha 2 windows 8 - guess the versions doesnt matter.

if you turn off cache and using a fresh m3u with new channel names it is downloading the xml for every channel.
the same will happen when the link is not an xml or the xml is corrupted.

http://pastebin.com/v6ifDcA4 - cache off with legit url
http://pastebin.com/CVHHkJzh - cache on with google url

I will attempt to reproduce this, and that does sound like a legitimate bug, but the fact still remains that it is not our add-on that is defaulting to turning the cache off or even defaulting to using these EPG urls. Even if what you describe is a bug and we can fix it, that still won't prevent the real issue, and that is that another group is using a different version (that we don't control) that is causing massive traffic. We also won't prevent users from being able to change their settings, for those who do need to disable EPG cache.

[afedchin]

@ned Scott
The fix of issues described by JinJin already in master. Here is PR https://github.com/kodi-pvr/pvr.iptvsimple/pull/12

[doglover]

We also won't prevent users from being able to change their settings, for those who do need to disable EPG cache.

If it is possible to change the settings to a value, which creates massive problems in certain sources, then this is a real problem.
I cannot tell you how you have to fix this, because I do not know. But something should be done about this.

It looks like Jin will produce something which will work on the Rytec EPG sources. It is being tested at the moment.
I will provide Jin with the needed info where and how to access the sources, after we tested his solution for some time.

Willy

[JinJin]

hi again,

thank you afedchin for fixing this.
have you checked nor fixed the first issue? even though cache is off this should not happen.

and i might have found another major? 'bug' because i think this is really unnecessary for iptv simple xmltv epg :

if there is no epg found for a channel, the system is checking every 5 minutes for an update in the xml. (i have not seen a setting for it and this is standard)
http://pastebin.com/1UsKpv7Z - also read my notes
this is also unnecessary increasing the cpu on (lower) systems - the more channels the longer

note that i only want to help and make things better cuz doglover might not be the only one
and of course its up to the user not to turn off cache

EDIT: ok looks like its only happening when kodi is not in use or the player is not active - but still, these update checks are not need, are they?