Posts: 46
Joined: Feb 2012
I'm wondering why USA traffic is sent to EU mirrors?
I know there was drama in the past about the plugins but the leia exe is being redirected overseas?
Posts: 5,174
Joined: Jan 2010
Reputation:
97
Kib
Team-Kodi Member
Posts: 5,174
2019-08-19, 00:41
(This post was last modified: 2019-08-19, 00:41 by Kib.)
That can happen when the locations near you have limited bandwidth compared to the monster mirrors in the EU.
I tried with your IP and it gives OS6 mirror about 6% of the time which is right considering the amount of bandwidth each has available.
We could use a few really high-speed mirrors in the US !
Posts: 46
Joined: Feb 2012
(2019-08-19, 00:41)Kib Wrote: That can happen when the locations near you have limited bandwidth compared to the monster mirrors in the EU.
I tried with your IP and it gives OS6 mirror about 6% of the time which is right considering the amount of bandwidth each has available.
We could use a few really high-speed mirrors in the US !
http://mirrors.kodi.tv/releases/windows/...mirrorlist
there's plenty of 10gig plus servers in the US. The weights are set up funky. it's silly to go halfway around the world for a download when there's 10 other North American servers.
Oregon State University 50+gbit
OS6 is a 1 gigabit server that receives very little traffic from kodi.
Posts: 5,174
Joined: Jan 2010
Reputation:
97
Kib
Team-Kodi Member
Posts: 5,174
2019-08-19, 17:34
(This post was last modified: 2019-08-19, 17:35 by Kib.)
The weights are set up based on the bandwidth they have available for us.
It is possible something is set up wrong, but please do not make up data. You state OSU has 50+ gbit. Where do you get this info ?
Posts: 5,174
Joined: Jan 2010
Reputation:
97
Kib
Team-Kodi Member
Posts: 5,174
2019-08-19, 17:46
(This post was last modified: 2019-08-19, 18:04 by Kib.)
Uni's
Princeton: 10 Gbit
OSU: 1 Gbit
UMD: 1 Gbit
Companies
Limestone: 1 Gbit
Syringa: 1 Gbit
Xmission: 1 Gbit
Leaseweb (Dallas) : 10 Gbit
Leaseweb (San Fransisco): 10 Gbit
^^ That is pretty much all we have dedicated to the US where most of our users come from
The three 10 Gbit lines are not plenty.
Posts: 46
Joined: Feb 2012
2019-08-19, 21:54
(This post was last modified: 2019-08-19, 22:13 by User 117171.)
(2019-08-19, 17:46)Kib Wrote: Uni's
Princeton: 10 Gbit
OSU: 1 Gbit
UMD: 1 Gbit
Companies
Limestone: 1 Gbit
Syringa: 1 Gbit
Xmission: 1 Gbit
Leaseweb (Dallas) : 10 Gbit
Leaseweb (San Fransisco): 10 Gbit
^^ That is pretty much all we have dedicated to the US where most of our users come from
The three 10 Gbit lines are not plenty.
I think the USA mirrors are just under utilized. I manage os6.org and the only time I see spikes in traffic is when other mirrors are removed due to not having the latest files when mirrorbits checks. When mirrorbrain was
still being used I used to see about 20-30MB of sustained traffic ( ~2,000 concurrent clients ) now i see less than 5MB ( no more than 100 concurrent clients ). See the below screenshots from mirror.os6.org nginx traffic and network traffic.
I realize other mirrors might not want the full amount of USA Kodi traffic hitting their mirrors, but feel free to send 800mbit of traffic to os6. The server as a whole is underutilized.
http://mirrors.kodi.tv/releases/windows/...irrorstats Most of the USA mirrors are always on the bottom of the mirrorstats list.
http://osuosl.org/services/hosting/details/
FTP Mirroring
We have a cluster of three servers behind the
ftp.osuosl.org name with a total bandwidth capacity of more than 50 gigabits per second. These servers are hosted geographically across the United States. Instead of pushing files and releases out from your own server, let us take care of the dirty work for you. We currently host approximately 100 projects on our mirroring servers using around 11TB of disk space. We do our best to host as many projects as we can, however space is limited.
Posts: 46
Joined: Feb 2012
(2019-08-19, 17:34)Kib Wrote: The weights are set up based on the bandwidth they have available for us.
It is possible something is set up wrong, but please do not make up data. You state OSU has 50+ gbit. Where do you get this info ?
Not sure what data I'm making up? os6.org is the busiest server in USA per mirrorbits and i don't even see 100 mbit from Kodi yet we somehow we don't have enough USA mirrors? so far today for the busiest USA server, mirrorbits has clocked a total of 191GB downloaded in 20 hours.
Per the ubuntu mirrors, someone has to sign up and populate the data.
https://launchpad.net/ubuntu/+archivemirrors
OSU Open Source Lab not Oregon State University
https://ftp.osuosl.org/pub/xbmc/ 50+ GBbit according to website or 2gbit according to ubuntu mirrors
Oregon State University has their own mirror at
http://mirror.uoregon.edu but its not mirroring Kodi and its 1gbit
Princeton University 20 Gbps
Syringa Networks 10 Gbps
Limestone Networks 2 Gbps
Looks like leaseweb has dropped kodi from all its mirrors.
Posts: 5,174
Joined: Jan 2010
Reputation:
97
Kib
Team-Kodi Member
Posts: 5,174
You are taking all these speeds from their sites, not the speeds they asked us to use. OSUOSL is 1 Gbit for us (not 50).
I was taking my data from the mirrorbits config but I did ineed not look at the activity of servers. You are right about Leaseweb and there are therefore even less active servers in the US.
There are not many high bandwidth servers in the US compared to the enormous amount of high bandwidth servers in Europe.
EUROPE: We have a combined 158 Gbit of mirror bandwidth. There are 31 active servers - 2 servers that have 20 Gbit for us, 6 servers that have 10 Gbit for us, and the rest is 5, 4, 3, 2 or 1 Gbit.
NORTH AMERICA: We have a combined 10.1 Gbit of mirror bandwidth. There are 9 active servers in North America. One is set to max on 2 Gbit (yours), 7 are on 1 Gbit, and one is 100 Mbit.
Mirrorbits makes a weighted list of all the mirrors which have the requested file and are allowing the connection, using the weights of all mirrors. It than randomizes base on that weight.
We set the weights based on the available bandwidth of the mirror.
I'm not sure how we can change this without breaking evrrything since we have our bandwidth mostly in the EU, and our users mostly in the US.
On top of that not all US mirrors are synchronising against all the addons I believe, so they even have less files available.
If I set up restrictions based on continent, your server is immediately flooded and people in US can not download kodi anymore. Meanwhile the Euro servers are underutilized.
If I pretend you have more bandwidth, a spike will mean disconnected users and people can not download anymore.
If you have a brilliant idea on how to improve the weighting system of mirrorbits I am all ears, but for now I think it is just a consequence of the difference in users and servers between the two continents.
Posts: 46
Joined: Feb 2012
(2019-08-21, 22:09)Kib Wrote: You are taking all these speeds from their sites, not the speeds they asked us to use. OSUOSL is 1 Gbit for us (not 50).
I was taking my data from the mirrorbits config but I did ineed not look at the activity of servers. You are right about Leaseweb and there are therefore even less active servers in the US.
There are not many high bandwidth servers in the US compared to the enormous amount of high bandwidth servers in Europe.
EUROPE: We have a combined 158 Gbit of mirror bandwidth. There are 31 active servers - 2 servers that have 20 Gbit for us, 6 servers that have 10 Gbit for us, and the rest is 5, 4, 3, 2 or 1 Gbit.
NORTH AMERICA: We have a combined 10.1 Gbit of mirror bandwidth. There are 9 active servers in North America. One is set to max on 2 Gbit (yours), 7 are on 1 Gbit, and one is 100 Mbit.
Mirrorbits makes a weighted list of all the mirrors which have the requested file and are allowing the connection, using the weights of all mirrors. It than randomizes base on that weight.
We set the weights based on the available bandwidth of the mirror.
I'm not sure how we can change this without breaking evrrything since we have our bandwidth mostly in the EU, and our users mostly in the US.
On top of that not all US mirrors are synchronising against all the addons I believe, so they even have less files available.
If I set up restrictions based on continent, your server is immediately flooded and people in US can not download kodi anymore. Meanwhile the Euro servers are underutilized.
If I pretend you have more bandwidth, a spike will mean disconnected users and people can not download anymore.
If you have a brilliant idea on how to improve the weighting system of mirrorbits I am all ears, but for now I think it is just a consequence of the difference in users and servers between the two continents.
If you don't mind I can reach out to other mirrors and try to build a bigger US mirror pool for Kodi.
Looking at vlc's mirrorbits stats it seems they favor geoIP. I haven't setup Mirrorbits in a while so i can't recommended a setting but the dev that created it seems to be active. There's a setting that might need adjusting but the config doesn't explain the setting.
https://get.videolan.org/vlc/3.0.8/win32...mirrorlist
https://get.videolan.org/vlc/3.0.8/win32...irrorstats
## Adjust the weight/range of the geographic distribution
# WeightDistributionRange: 1.5
Posts: 5,174
Joined: Jan 2010
Reputation:
97
Kib
Team-Kodi Member
Posts: 5,174
2019-08-22, 08:48
(This post was last modified: 2019-08-22, 08:49 by Kib.)
I can take a look at changing the relative weight of the GeoIP nearness to the bandwidth score and I'll reach out to etix as well and ask him if he has any recommendations.
Seeing the results I get with your IP at VLC's mirrorstats it looks like his setup only selects 3 mirrors out of the pool they have and they are all US based.
Perhaps they set them up to be continent only or he has made another adjustment.
I will need to figure out what happens when a files is not available in your continent and mirrors are set to continent only though.
Any help in finding more US mirrors would be highly appreciated !
Posts: 46
Joined: Feb 2012
(2019-08-22, 08:48)Kib Wrote: I can take a look at changing the relative weight of the GeoIP nearness to the bandwidth score and I'll reach out to etix as well and ask him if he has any recommendations.
Seeing the results I get with your IP at VLC's mirrorstats it looks like his setup only selects 3 mirrors out of the pool they have and they are all US based.
Perhaps they set them up to be continent only or he has made another adjustment.
I will need to figure out what happens when a files is not available in your continent and mirrors are set to continent only though.
Any help in finding more US mirrors would be highly appreciated !
https://kodi.wiki/view/Mirrors . Are these instructions current?
Posts: 46
Joined: Feb 2012
(2019-08-22, 08:48)Kib Wrote: I can take a look at changing the relative weight of the GeoIP nearness to the bandwidth score and I'll reach out to etix as well and ask him if he has any recommendations.
Seeing the results I get with your IP at VLC's mirrorstats it looks like his setup only selects 3 mirrors out of the pool they have and they are all US based.
Perhaps they set them up to be continent only or he has made another adjustment.
I will need to figure out what happens when a files is not available in your continent and mirrors are set to continent only though.
Any help in finding more US mirrors would be highly appreciated !
Kib, I have decomissioned mirror.os6.org due to low utilization. Please remove it from the mirror lists.
