Problems downloading dependencies from rwth.aachen.de
#1
Sad 
I am trying to set up my MBP so I can build Kodi for OS X. Every download of a file > 7MB from rwth.aachen.de, the automatically selected mirror, has failed. Curl exits without error but the subsequent tar fails because the file is only partially downloaded.

I have worked around this by downloading in my web browser where I can restart the download, after the failure, to fetch the remaining part of the file.

I am posting here partly because I don't know if it is appropriate to file a bug for something like this and partly because I am currently unable to log into the bug tracker.
Reply
#2
Moving this to http://forum.kodi.tv/forumdisplay.php?fid=35
Reply
#3
That issue has to be on your side (either ISP or your network - check lower mtu values for example). Just tested a min ago:

http://pastebin.com/6vMvfHLc
AppleTV4/iPhone/iPod/iPad: HowTo find debug logs and everything else which the devs like so much: click here
HowTo setup NFS for Kodi: NFS (wiki)
HowTo configure avahi (zeroconf): Avahi_Zeroconf (wiki)
READ THE IOS FAQ!: iOS FAQ (wiki)
Reply
#4
(2016-06-08, 07:26)Memphiz Wrote: That issue has to be on your side (either ISP or your network - check lower mtu values for example).

Thank you for testing. If this is the case, why don't I see similar problems with any other download sites? I am using default TCP/IP settings on OS X.

There was a persistent issue downloading Kodi add-ons from the Far East, particularly Japan where I am now, throughout the life of Kodi 15. There is a long thread about it which contains many similar statements that the servers are working fine and it must be a client problem. But oops, it turned out to be a server related problem that wsnipex finally fixed. So I take such statements with a grain of salt.

Also, in your test you only downloaded fontconfig-2.11.1.tar.bz2. The problem I am reporting in this thread happens with files > 7MB. fontconfig-2.11.1.tar.bz2 is only 1.4M.
Reply
#5
I understand your grain of salt, so lets try to reproduce and get logs.

Aachen handles roughly half our downloads worldwide, so if this error is for everyone, again: we would have many more reports.
Reply
#6
It was reproducible over a period of 2 or 3 hours when I was attempting to download and build the Kodi dependencies. What happened with all the large files, e.g. gettext which is 15MB, is:
  • download took many many minutes, more than 10m in the case of gettext. My ISP provides a fairly reliable ~50Mbi/s service.
  • tar failed because the tar file was bad, though there was no error from curl.
  • I tried manual download in Firefox
  • download failed in the middle so I restarted the download which Firefox fortunately restarted from the point of failure
  • repeat the previous step 1 or 2 more times to get the whole file successfully downloaded

What standard logs might have been collected during this that I could provide?

If I choose to repeat this exercise, which I am loathe to do as it took such a long time the first time, what logs might I enable?

Do the aachen.de server logs show anything? The failures were in the hour(s) immediately preceding my starting this thread.
Reply
#7
there are no logs of those downloads besides the mirror server, which we don't have access to.
You could only change the curl settings in Makefile.include to make curl more verbose.
Reply
#8
Thanks @wsnipex. If I try again I'll change the curl settings. Mind you curl did not actually notice that there had been an error.
Reply
#9
Had issues downloading the latest dev builds, went looking here.
I can connect just fine to the aachen.de server. The problem looks to be on their side.

KodiSetup-20160610-536fa3c-gettext.pdb 2016-06-11 05:55 109M
That is the most recent file I see when looking at:
http://ftp.halifax.rwth-aachen.de/xbmc/n...2/?C=M;O=D

Edit: Found the reason;
http://forum.kodi.tv/showthread.php?tid=...pid2360129
No worries then. I'll wait.
Reply
#10
@RMike Glad you found a solution to your issue but it was totally different from this problem. In your case files were missing from the server.

I have just had an exact repeat of the issue I described here, this time when downloading Python-2.7.10.tar.xz which is 12.3 MB. I got only 7.5MB before the first download failed. Unfortunately I had not added --verbose to the curl arguments. I have now. Manual download in Firefox also failed at around 8.1MB. It completed after restart. So this issue is certainly not a one-time problem.

A consistent factor during these failures is how much the remaining time meter jumps around. For example it started by indicating the download would take 21 minutes. Then it dropped to 4 minutes. Then it climbed back up 5,6,7,9,11,15 minutes. Then it very slowly dropped until the failure.
Reply
#11
Just got another failure. Here is the verbose output from curl. There is no obvious cause for the failure as far as I can see.
Code:
cd /Users/Shared/xbmc-depends/tarballs; /usr/bin/curl --verbose -Ls --create-dirs -f -O http://mirrors.kodi.tv/build-deps/sources/swig-3.0.10.tar.gz
*   Trying 89.16.176.16...
* Connected to mirrors.kodi.tv (89.16.176.16) port 80 (#0)
> GET /build-deps/sources/swig-3.0.10.tar.gz HTTP/1.1
> Host: mirrors.kodi.tv
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: nginx
< Date: Mon, 20 Jun 2016 09:49:54 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 99
< Connection: close
< Cache-Control: private, no-cache
< Location: http://ftp.halifax.rwth-aachen.de/xbmc/build-deps/sources/swig-3.0.10.tar.gz
< X-Frame-Options: SAMEORIGIN
<
* Closing connection 0
* Issue another request to this URL: 'http://ftp.halifax.rwth-aachen.de/xbmc/build-deps/sources/swig-3.0.10.tar.gz'
*   Trying 137.226.34.46...
*   Trying 2a00:8a60:e012:a00::21...
* Immediate connect fail for 2a00:8a60:e012:a00::21: No route to host
* Connected to ftp.halifax.rwth-aachen.de (137.226.34.46) port 80 (#1)
> GET /xbmc/build-deps/sources/swig-3.0.10.tar.gz HTTP/1.1
> Host: ftp.halifax.rwth-aachen.de
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.6.2
< Date: Mon, 20 Jun 2016 09:50:03 GMT
< Content-Type: application/octet-stream
< Content-Length: 8029827
< Last-Modified: Thu, 16 Jun 2016 18:19:21 GMT
< Connection: keep-alive
< Accept-Ranges: bytes
<
{ [1214 bytes data]
* transfer closed with 723453 bytes remaining to read
* Closing connection 1
make[2]: *** [/Users/Shared/xbmc-depends/tarballs/swig-3.0.10.tar.gz] Error 18
make[1]: *** [swig-native] Error 2
make: *** [native/.installed-x86_64-osx-native] Error 2
Reply
#12
Tried again this morning, 16 hours later. Got the same failure, except that this time bytes remaining to read was 716213. Whatever the problem is, it appears to be persistent.
Reply
#13
are you sure your internet connection is stable?
Reply
#14
I tried again a couple of hours ago, around 30 hours since the last try and got the same failure. With memory of the Add-On download problem still fresh, I thought I'd try a VPN with an exit node in the US. Guess what? The download completed! Looks like your mirrors/servers still have geolocation issues. Of course, it could just be coincidence but given the persistence of the issue over the past few days and success on the first attempt via the VPN, that seems only a small probability. Given the time it takes to build the dependencies I am loathe to try again.

As to whether my connection is stable, I have not noticed issues with any other sites or downloads and speed test results from bandwidthplace.com show 70.40Mbps down, and 28.70Mbps up. testmy.net shows 76.3 down and 28.4 up. In both cases using servers in Japan. Using a server in Texas, testmy.net managed 40Mbps download.
Reply
#15
it has nothing to do with geolocation, since you can resolve, connect and download from the mirrors.
If a download terminates with curl error 18 it indicates one of 2 reasons:
1. server issues: e.g. too much load/traffic
2. network issues on the client side or in between
Its absolutely possible that your ISP has bad peering to europe.

As I already told you in your other thread: you CAN override the mirror used when building depends, so give this a try.

here is a list of mirrors to select from: http://mirrors.kodi.tv/nightlies/android...mirrorlist
Don't mind the android apk here, its just used as example to get the listing.
Reply

Logout Mark Read Team Forum Stats Members Help
Problems downloading dependencies from rwth.aachen.de0