Kodi Community Forum

Kodi Community Forum > Support > General Support > Linux > XBMC and curl v7.38.0 do not work together - cannot scrape
Full Version: XBMC and curl v7.38.0 do not work together - cannot scrape
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5

graysky

2014-09-19, 23:33
Running xbmc 13.2 and just pulled down the latest curl version 7.38.0. I also rebuilt xbmc against it, but still, I cannot scrape anything. Known issue?

Relevant part of log:
Code:
17:28:41 T:140038428538624  NOTICE: script.grab.fanart: Grab Fanart Service Started
17:28:46 T:140039210633216   ERROR: CheckDisplayEvents - no display event after 3 seconds
17:28:51 T:140037610010368   ERROR: CCurlFile::FillBuffer - Failed: Timeout was reached(28)
17:28:51 T:140037610010368   ERROR: CCurlFile::CReadState::Connect, didn't get any data from stream.
17:28:51 T:140037610010368   ERROR: Open - failed to open source <http://mirrors.xbmc.org/addons/gotham/addons.xml.md5>
17:29:01 T:140037610010368   ERROR: CCurlFile::FillBuffer - Failed: Timeout was reached(28)
17:29:01 T:140037610010368   ERROR: CCurlFile::CReadState::Connect, didn't get any data from stream.
17:29:01 T:140037610010368   ERROR: Open - failed to open source <http://mirrors.xbmc.org/addons/frodo/addons.xml.md5>
17:29:11 T:140037610010368   ERROR: CCurlFile::FillBuffer - Failed: Timeout was reached(28)
17:29:11 T:140037610010368   ERROR: CCurlFile::CReadState::Connect, didn't get any data from stream.
17:29:11 T:140037610010368   ERROR: Open - failed to open source <http://mirrors.xbmc.org/addons/gotham/addons.xml|Encoding=gzip>
17:29:21 T:140037610010368   ERROR: CCurlFile::FillBuffer - Failed: Timeout was reached(28)
17:29:21 T:140037610010368   ERROR: CCurlFile::CReadState::Connect, didn't get any data from stream.
17:29:21 T:140037610010368   ERROR: Open - failed to open source <http://mirrors.xbmc.org/addons/frodo/addons.xml|Encoding=gzip>
17:29:21 T:140037610010368   ERROR: Repository XBMC.org Add-ons returned no add-ons, listing may have failed

Complete log.

Downgrading to curl v7.37.1 fixes the problem.

lanlocked

2014-09-20, 01:15
Thank you for pointing this out.. I have been screwing around with this for the last couple of days. A quick downgrade to 7.37 (ArchLinux) and XBMC was fixed!

EDIT: I should have mentioned that when I used curl from the CLI using URLs pulled out of the debug log.... it worked fine, thus I never suspected an issue with curl.

graysky

2014-09-20, 02:23
I opened a bug report against xbmc since, as you pointed out, curl from the shell seems to work fine.

http://trac.xbmc.org/ticket/15441

fritsch

2014-09-20, 11:31
Report that to curl upstream. They should not break compatibility.

kalpik

2014-09-20, 13:46
Same issue here. Subscribing to get updates.

graysky

2014-09-20, 17:19
Disabling IPV6 in xbmc fixes this issue but not consistently. As a test, I rebuilt my database and found that many entries refused to download prompting me to skip and abort. Downgrading to xbmc 13.2-3 and curl 3.37.1-1 causes a database rebuild to work consistently.
Reference: https://bugs.archlinux.org/task/42001

jonandermb

2014-09-20, 20:48
Same problem here on arch linux. downgraded to curl 3.37 and everything seems to work again now. waiting for a fix.

cheers

fritsch

2014-09-20, 22:05
We are roughly 7 active linux devs. If you are a using a roling release like arch linux - you will run into trouble everytime you "just bump another library" without having it tested properly. Most of our devs are running Ubuntu to make sure the ppa is working as expected. We simply don't have the manpower and also no interest to test new libraries two days after release.

So - if such issues happen to you - cause you want always latest and greatest and untested and just compile tested - prepare yourself to bisect upstream(!) in that case - curl - to find solutions yourself.

Edit: We can write that into the release announcement "xbmc is developed and tested under Ubuntu Linux Version X,Y - using version x,y of library a,b" We had the same trouble with distros bumping away ffmpeg or exchange it with libav ... we cannot forbid such things - but we don't need to support it either.

graysky

2014-09-20, 23:39
@fritsch. I understand and am very grateful for the hard work you guys put into this project. Is there a chance you can post a comment that would help curl upstream debug? I do not have the technical knowledge to intelligently answer them.

https://sourceforge.net/p/curl/bugs/1426

fritsch

2014-09-20, 23:49
As I cannot test myself with this new version of libcurl - I suggest you start git bisecting.

Use 7.37.1 as good and 7.38.0 as bad and see which commit breaks xbmc.

You always need to rebuild curl, install it and then build xbmc against it.

wsnipex

2014-09-21, 07:49
And just to be sure, try with a helix nightly first.

fritsch

2014-09-21, 07:55
Nice, Johne E has tracked it down: http://sourceforge.net/p/curl/bugs/1426/

From looking at the patch - it seems like the logic in: https://github.com/bagder/curl/commit/ca...18bc82ebfb , especially: https://github.com/bagder/curl/commit/ca...feb02R1572

Can you try to change that like:
Code:
- Curl_expire_latest(data->easy_conn->recv_pipe->head->ptr, 1);
+ Curl_expire(data->easy_conn->recv_pipe->head->ptr, 1);

I think a curl timesout before actually getting anything.

wsnipex

2014-09-21, 08:00
btw, you can enable curl trace logging in xbmc. Component-Specific logging under Debug logging.

wsnipex

2014-09-21, 08:14
relevant parts of xbmc code: https://github.com/xbmc/xbmc/blob/master...e.cpp#L429
https://github.com/xbmc/xbmc/blob/master....cpp#L1418

Curl timout can be overridden via advancedsettings.xml, default is 10s:
https://github.com/xbmc/xbmc/blob/master...s.cpp#L317

fritsch

2014-09-21, 08:16
That won't help in that case. If it really is the part I linked above. See the second paramter looks like it's hard coded to 1 ms ... which already expires in the computation - but we will see.
Pages: 1 2 3 4 5
Kodi Community Forum > Support > General Support > Linux > XBMC and curl v7.38.0 do not work together - cannot scrape