Krypton changes to URL options
#1
with https://github.com/xbmc/xbmc/pull/9330 the usual way of setting http headers was removed and this will break all addons that rely on http streaming with header checks (referer=, cookies=, etc).
Reply
#2
Is there a proper way to get the same behaviour now?
Reply
#3
Nope. Some devs think it is progress to remove things peeps need, damn the consequences. I will try to find some time to fix this but i have little time these days Sad
Reply
#4
also it is worth noticing that this does not affect vanilla http(s) which is unchanged, it is only for protocols that runs on top like hls and mmsh.
Reply
#5
(2016-09-01, 18:25)ironic_monkey Wrote: Nope. Some devs think it is progress to remove things peeps need, damn the consequences. I will try to find some time to fix this but i have little time these days Sad

(2016-09-01, 18:35)ironic_monkey Wrote: also it is worth noticing that this does not affect vanilla http(s) which is unchanged, it is only for protocols that runs on top like hls and mmsh.

I don't mind them fixing things at all, after all it was bad that the HttpProxy internal URL command was send as HTTP header. But not allowing any headers except cookie and user-agent to ffmpeg does not make a lot of sense. The should have prevented the "special" Kodi internal headers to just be removed everywhere. Not all headers for ffmpeg only.

I would really prefer them to revert this change and add a proper fix in the next release. Hell, I will even try to help out. But putting this in Krypton really cripples many, many add-ons that rely on http headers.
Reply
#6
So glad somebody noticed these changes that started with Krypton. I don't think my actual problem is happening exactly because of them however it is related.

I'm using an addon that parses a website which provides streaming services for a bunch of different TV channels. One of its biggest advantages for me is that has a few days' worth of EPG archive which means I can watch, let's say, yesterday's news broadcast the following day. It's pretty neat and is free of charge. Videos on the site are encoded in x264 and streaming is done via HTTP and HLS. The only check they perform is for User Agent in the request headers. Even for the actual .ts chunks from m3u8.

Up until Krypton everything worked fine with user agent being appended to the URL in python script. Simply putting something in line of "|User-Agent=Mozilla/5.0..." made it work just fine. However v17 had some changes brought in (all alphas and now beta) that completely broke the functionality of it all and as a result it's no longer working.

I can reproduce this easily with a simple change in the addon so that it request files from my own webserver. See the headers:

Code:
- - [01/Sep/2016:21:21:46 +0300] "HEAD *****.mp4 HTTP/1.1" 200 - "-" "\"TEST\""
- - [01/Sep/2016:21:21:46 +0300] "HEAD *****.mp4 HTTP/1.1" 200 - "-" "\"TEST\""
- - [01/Sep/2016:21:21:46 +0300] "GET *****.mp4 HTTP/1.1" 206 497689 "-" "\"TEST\""
- - [01/Sep/2016:21:21:46 +0300] "GET *****.mp44 HTTP/1.1" 206 65536 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"
- - [01/Sep/2016:21:21:46 +0300] "GET *****.mp4 HTTP/1.1" 206 497689 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"

It's obvious User-Agent headers are no longer being passed to actually play the video. Only to request first 64KB to get its mime type etc.

What's even more frustrating is that there is no way to override default User-Agent even with advancedsettings.xml. It's hardcoded in xbmc/settings/AdvancedSettings.cpp:


Code:
m_userAgent = g_sysinfo.GetUserAgent();

Any suggestions without recompiling my own version (I would need both for Windows and RaspberryPi) or using a proxy with header rewriting?
Reply
#7
(2016-09-01, 21:01)FatalityX Wrote: So glad somebody noticed these changes that started with Krypton. I don't think my actual problem is happening exactly because of them however it is related.

I'm using an addon that parses a website which provides streaming services for a bunch of different TV channels. One of its biggest advantages for me is that has a few days' worth of EPG archive which means I can watch, let's say, yesterday's news broadcast the following day. It's pretty neat and is free of charge. Videos on the site are encoded in x264 and streaming is done via HTTP and HLS. The only check they perform is for User Agent in the request headers. Even for the actual .ts chunks from m3u8.

Up until Krypton everything worked fine with user agent being appended to the URL in python script. Simply putting something in line of "|User-Agent=Mozilla/5.0..." made it work just fine. However v17 had some changes brought in (all alphas and now beta) that completely broke the functionality of it all and as a result it's no longer working.

I can reproduce this easily with a simple change in the addon so that it request files from my own webserver. See the headers:

Code:
- - [01/Sep/2016:21:21:46 +0300] "HEAD *****.mp4 HTTP/1.1" 200 - "-" "\"TEST\""
- - [01/Sep/2016:21:21:46 +0300] "HEAD *****.mp4 HTTP/1.1" 200 - "-" "\"TEST\""
- - [01/Sep/2016:21:21:46 +0300] "GET *****.mp4 HTTP/1.1" 206 497689 "-" "\"TEST\""
- - [01/Sep/2016:21:21:46 +0300] "GET *****.mp44 HTTP/1.1" 206 65536 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"
- - [01/Sep/2016:21:21:46 +0300] "GET *****.mp4 HTTP/1.1" 206 497689 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"

It's obvious User-Agent headers are no longer being passed to actually play the video. Only to request first 64KB to get its mime type etc.

What's even more frustrating is that there is no way to override default User-Agent even with advancedsettings.xml. It's hardcoded in xbmc/settings/AdvancedSettings.cpp:


Code:
m_userAgent = g_sysinfo.GetUserAgent();

Any suggestions without recompiling my own version (I would need both for Windows and RaspberryPi) or using a proxy with header rewriting?
User-agent passes on just fine, as is the Referrer and Cookie header. All others fail. Just append "|http-headers="....." to the url. It's all the other headers that no longer work.

CORRECTION: not even Referrer is allowed Sad
Reply
#8
(2016-09-01, 21:53)Basje Wrote: User-agent passes on just fine, as is the Referrer and Cookie header. All others fail. Just append "|http-headers="....." to the url. It's all the other headers that no longer work.

I see. However I'm doing just that, well the addon does via appending "user-agent" to the URL just like in the old versions.

PHP Code:
uri 'http://my.own.server.tld/path/filename.mp4|user-agent=TEST' 

But... Even direct request for a simple mp4 file over HTTP isn't rewritten with a new header. Consider these log outputs from a webserver with an older Kodi:

Code:
- - [02/Sep/2016:06:11:14 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:11:14 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:11:14 +0300] "GET ****.mp4 HTTP/1.1" 206 65536 "-" "TEST"
- - [02/Sep/2016:06:11:14 +0300] "GET ****.mp4 HTTP/1.1" 206 497689 "-" "TEST"

and Krypton:

Code:
- - [02/Sep/2016:06:16:48 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:16:48 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:16:48 +0300] "GET ****.mp4 HTTP/1.1" 206 497689 "-" "TEST"
- - [02/Sep/2016:06:16:48 +0300] "GET ****.mp4 HTTP/1.1" 206 65536 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"
- - [02/Sep/2016:06:16:48 +0300] "GET ****.mp4 HTTP/1.1" 206 497689 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"

Krypton does one extra request it seems (4 vs 5), I can live with that. However, user-agent is back to original one in the headers when it starts actually playing the video.

Strange, isn't it?
Reply
#9
(2016-09-02, 05:26)FatalityX Wrote:
(2016-09-01, 21:53)Basje Wrote: User-agent passes on just fine, as is the Referrer and Cookie header. All others fail. Just append "|http-headers="....." to the url. It's all the other headers that no longer work.

I see. However I'm doing just that, well the addon does via appending "user-agent" to the URL just like in the old versions.

PHP Code:
uri 'http://my.own.server.tld/path/filename.mp4|user-agent=TEST' 

But... Even direct request for a simple mp4 file over HTTP isn't rewritten with a new header. Consider these log outputs from a webserver with an older Kodi:

Code:
- - [02/Sep/2016:06:11:14 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:11:14 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:11:14 +0300] "GET ****.mp4 HTTP/1.1" 206 65536 "-" "TEST"
- - [02/Sep/2016:06:11:14 +0300] "GET ****.mp4 HTTP/1.1" 206 497689 "-" "TEST"

and Krypton:

Code:
- - [02/Sep/2016:06:16:48 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:16:48 +0300] "HEAD ****.mp4 HTTP/1.1" 200 - "-" "TEST"
- - [02/Sep/2016:06:16:48 +0300] "GET ****.mp4 HTTP/1.1" 206 497689 "-" "TEST"
- - [02/Sep/2016:06:16:48 +0300] "GET ****.mp4 HTTP/1.1" 206 65536 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"
- - [02/Sep/2016:06:16:48 +0300] "GET ****.mp4 HTTP/1.1" 206 497689 "-" "Kodi/17.0-BETA2 (Windows NT 6.1; WOW64) App_Bitness/32 Version/17.0-BETA2-Git:20160831-f39d849"

Krypton does one extra request it seems (4 vs 5), I can live with that. However, user-agent is back to original one in the headers when it starts actually playing the video.

Strange, isn't it?
Indeed, something is very inconsistent here and is really causing issues with a lot of add-ons.
Reply
#10
Basje i totally agree we tried to make the same point. In norway we have a saying 'disposing of the baby and not just the bathing water'.
Reply
#11
are you sure im passing headers like

headers = {'User-Agent': UA,
'Referer': referrer,
'X-Requested-With': 'ShockwaveFlash/22.0.0.20'}
part = ""

for name, desc in headers.iteritems():
part = part + '&' + name + '=' + urllib.quote(desc)

xbmcgui.ListItem(path=stream + "|" + part[1:])

and i see in wireshark that ffmpeg is doing head and get request with all headers (krypton beta)
Reply
#12
What I see, is that ffmpeg does an initial GET of an M3u8 URL including the headers. But for the actual segments that are available in the M3u8 no headers are used.
Reply
#13
it's not ffmpeg that does the GET, that is CURL (it's pre-parsed to choose the correct stream according to configured bandwidth, as there is no adaptive streams yet). then the show is passed on to ffmpeg and it will not set those headers on the contents of the m3u8 (i.e. the actual streams).
Reply
#14
(2016-09-05, 11:43)ironic_monkey Wrote: it's not ffmpeg that does the GET, that is CURL (it's pre-parsed to choose the correct stream according to configured bandwidth, as there is no adaptive streams yet). then the show is passed on to ffmpeg and it will not set those headers on the contents of the m3u8 (i.e. the actual streams).

I am working on getting the basic headers to ffmpeg. Hopefully it will make it in the Krypton final.
Reply

Logout Mark Read Team Forum Stats Members Help
Krypton changes to URL options1