[beredim]

Kodi 15.0 Alpha 1 (Isengard) WebDAV->Kodi->UPnP (With Subtitles) --> Unwanted traffic ( Is this a Bug or a feature? )

Setup is as follows:
A) WebDAV server (Remote)
B) Kodi (Local) with UPnP feature activated
C) TV (LG Webos 2014 model) (probably same behavior for any DLNA client)

So I have setup Kodi to access my remote webdav server and it has scraped it into the library.

When I'll try to play a movie on Kodi, it will do some requests to the server to try and locate subtitles (btw I only use moviename.srt at same dir as movie) and then will start playing
These requests look something like this:

Code:

PROPFIND /Example Movie (1998)/subtitles/ -> 404 Not Found
PROPFIND /Example Movie (1998)/Subtitles/ -> 404 Not Found
PROPFIND /Example Movie (1998)/subs/ -> 404 Not Found
PROPFIND /Example Movie (1998)/Subs/ -> 404 Not Found
PROPFIND /Example Movie (1998)/vobsubs/ -> 404 Not Found
PROPFIND /Example Movie (1998)/Vobsubs/ -> 404 Not Found
PROPFIND /Example Movie (1998)/sub/ -> 404 Not Found
PROPFIND /Example Movie (1998)/Sub/ -> 404 Not Found
PROPFIND /Example Movie (1998)/vobsub/ -> 404 Not Found
PROPFIND /Example Movie (1998)/Vobsub/ -> 404 Not Found
PROPFIND /Example Movie (1998)/subtitle/ -> 404 Not Found
PROPFIND /Example Movie (1998)/Subtitle/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd1/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd2/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd3/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd4/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd5/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd6/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd7/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd8/ -> 404 Not Found
PROPFIND /Example Movie (1998)/cd9/ -> 404 Not Found

Now I can live with that. It's only a handful of requests that take milliseconds to complete.

However the problem starts when I fire up my TV. It communicates with Kodi over UPnP and my WebDAV server starts getting flooded with requests like the above.
But this happens for every movie, and multiple times for every movie (because every movie is being offered on multiple paths (Files, Library, Genre, Sets, Year etc)
And this happens without any playback at all. It just starts the moment my TV finishes booting (probably because it starts adevrtising as DLNA client)

For a modest library of about 1000 movies, this results in a traffic of about 300-400kiloBytes/sec that continues for more than 1/2 hour.

So either Kodi starts offering, or my TV requests info, about every item in the UPnP path, even when nothing is being played.

1. Is this expected behavior (ie. Is there something in the spec about subtitle info that has to be included when adertising an item over UPnP?)
2. If not, shouldn't Kodi scrape the server for subtitles, only when an item is being requested for playback on UPnP, and not when advertising the library items (like when playing on Kodi)
2. Shouldn't/Couldn't Kodi cache info like this when accessing remote sources?

PS A quick test on a DLNA client on my Android phone, shows the same behavior. So it is definitely something on the Kodi side.

[Ned Scott]

I think recently a feature was merged for external subtitle support over UPnP. This might be the result from that.

[beredim]

Yes subtitles over UPnP is a new feature (Kodi 15.0+). And as any new feature understandably it has side effects.

In my opinion, the amount of traffic generated is a serious issue (even for local network sources, let alone remote ones). I'm avoiding to definitely call it a bug because I really don't know if this behavior (analytic information about every item offered over UPnP) is considered UPnP spec.

If it is not spec, then it's a bug that has to be addressed. Just look for subtitles only when playback over UPnP starts (the same as happens when playing back on Kodi's internal player)

If it is spec, then it should somehow be bypassed by having Kodi cache info like this, and only update it on demand.

[Ned Scott]

I left a note with the author of the feature on github that points to this thread.

[Montellese]

@beredim: Would you be able to run a test build to see if things improve if I provided you with a patched nightly build (depending on the OS you are running Kodi on)? I don't have any external subtitles nor do I have any knowledge about its implementation but I think I know how to fix the UPnP problem.

[beredim]

yes

I'm running on Windows, so either a patched .exe or a full installer for Windows will be ok.

[Montellese]

@beredim: This is a first attempt at a fix and is based on the latest nightly build. You can download it from http://mirrors.xbmc.org/test-builds/win3...titles.exe but you might have to give it an hour or so before it is available on your mirror.
Please make sure to backup your userdata before installing this since there may have been changes to the database layout etc since Alpha 1.

[beredim]

No go.

External Subtitles over UPnP stopped working completely with this patch (even for non-network/local folder sources). It's like those requests I posted above are simply never sent (or the local directory is never searched for subs in the case of local/non-network sources)

On the other hand, Kodi's internal player continues to display subs fine, with the handful of above requests (only about the specific movie being played) being sent when playback starts.

I'm attaching a debug log with UPnP and CURL logs enabled ( http://xbmclogs.com/pffrwycl4 (I've changed WEBdav server's ip to 111.111.111.111 and changed true username and password))
XBMC host's IP is 192.168.1.10
DLNA Client's (TV) IP is 192.168.1.15
(There appear to be also some dlna requests from my android phone in the log with ip 192.168.1.2)

PS. Thank you for the effort you put into this.

[Montellese]

Thanks for the fast feedback and the extensive debug log. Unfortunately I'm not sure how to solve this issue. Ideally we only want to look up and retrieve the subtitles when starting playback because otherwise they aren't of much interest. The changes I made probably work fine if you use "Play using.." i.e. you browse your library in Kodi and then you choose an item to play and tell Kodi to play it on another device. In that case I know exactly that the item will be played so I can retrieve the subtitles. But in the case where you browse your library on another device using UPnP there's no way of knowing which item will be played. The request to play an item is a simple HTTP request to the webserver and has nothing to do with UPnP anymore and at that point it's already too late because the client will use the information about the item it has previously received to start playback.

The best we can do for now is probably to make scanning for external subtitles optional through a setting so that people with external subtitles can turn it on and everyone else can turn it off. It doesn't improve the situation for people that have the setting turned on though as they will still see a lot of traffic.

[beredim]

The changes I made probably work fine if you use "Play using.." i.e. you browse your library in Kodi and then you choose an item to play and tell Kodi to play it on another device. In that case I know exactly that the item will be played so I can retrieve the subtitles.

It works this way. I hadn't thought of trying the "Play using..." option until you mentioned it. I can confirm that it works as you expect it. So using "Play using..." and targeting my TV, it will play the movie with external subtitles and without hammering my webdav server.

But in the case where you browse your library on another device using UPnP there's no way of knowing which item will be played. The request to play an item is a simple HTTP request to the webserver and has nothing to do with UPnP anymore and at that point it's already too late because the client will use the information about the item it has previously received to start playback.

If I understand correctly, what you're saying is that the TV (or UPnP/DLNA clients in general) when browsing the UPnP server, is essentialy browsing a listing of DIDL items. Before your patch these items where created on the fly, as the TV browsed the folders, (hence the hammering of the remote server) to include the subtitle streams. With your patch, the DIDL items available when browsing don't contain info about subtitles streams, therefore the TV doesn't know to request subtitles because it only sees DIDL items containing only one stream (the video file). Using the "Play Using..." however means we're essentially pushing a custom DIDL item (different than the ones the TV is browsing through) to the TV, this time containing an additional stream (the subtitles file).

The best we can do for now is probably to make scanning for external subtitles optional through a setting so that people with external subtitles can turn it on and everyone else can turn it off. It doesn't improve the situation for people that have the setting turned on though as they will still see a lot of traffic.

For a short-term solution this would appear the best. A setting for "External Subtitles over UPnP" with a default of ON, alongside a warning about the extra traffic generated for network sources (and a note that one can use the "Play Using..." option to avoid the extra network traffic)

I feel that a better, long-term solution would be to cache information about available subtitles when scraping network sources, but as I understand it that would mean re-engineering the network-sources portions of the code to include this additional functionality. Still, it could be worth pursuing as there are other problems that can be solved alongside it, in such an effort (for example the inability to use external subtitles altogether when using a HTTP (non webdav) source).

Unfortunately, I don't have any knowledge (let alone experience) in C++ (I mostly develop in Java) and can't be of any real assistance other than testing.

With that said, a big THANK YOU for devoting your time on this issue.

PS. While writing the response I came up with a flow for a workaround that might be of some value. Let me know if this sounds feasible:
1. TV requests a movie (over UPnP)
2. Before responding, Kodi probes relevant source (whether local or remote) for available subtitles (this would take 1-2 seconds probably for remote sources, but I don't think that's enough time for the TV/client to consider the request has timed-out)
3. If Kodi doesn't find any external subtitles it responds normally, sending back the video stream.
4. If, however, Kodi does find external subtitles then:
---4a. It denies the client's request (responding with something like a 404, 400, 500)
---4b. and then, in a new action, Kodi pushes a new custom DIDL (with the subttitle stream included) to the client.

4 (alternative). If UPnP clients support 301/302 redirects, Kodi creates (on the fly) a custom DIDL (containing the subtitles stream url) and responds with a redirect to that DIDL to the client.