Why the Internet in Sub-Saharan Africa Sucks(facebook.com)
facebook.com
Why the Internet in Sub-Saharan Africa Sucks
https://www.facebook.com/dweekly/posts/10101729307500703
126 comments
https://angani.co/blog/its-not-how-much-fibre-you-have-its-w... a response to this post, highlighting our experience over the years in Africa
I think Ethiopia is the worst!
Summary: Nairobi needs a datacenter, with Akamai and Cloudflare.
A local Cloudflare would definitely help with the TLS setup since that's their magic (fast SSL setup without requiring your private key).
Akamai could help, depending on the use case. If the content isn't cacheable, there's not much you can do. They do have a service that keeps open TCP sessions from certain POPs back to your origin, thus cutting down on TCP 3-way handshake time. Clients connect quickly, locally and then send the request because the origin connection is already established. That service is pretty expensive compared to CDN and they don't push it very hard. I used it when I ran an ad network where latency mattered (and could pay for this feature in an explicit increase in conversions and impressions).
Akamai could help, depending on the use case. If the content isn't cacheable, there's not much you can do. They do have a service that keeps open TCP sessions from certain POPs back to your origin, thus cutting down on TCP 3-way handshake time. Clients connect quickly, locally and then send the request because the origin connection is already established. That service is pretty expensive compared to CDN and they don't push it very hard. I used it when I ran an ad network where latency mattered (and could pay for this feature in an explicit increase in conversions and impressions).
Both Cloudflare and Cloudfront (when used as a dynamic content CDN) offer persistent open connections to your origin, as a standard feature of their offering.
https://blog.cloudflare.com/keep-alive/
https://support.cloudflare.com/hc/en-us/articles/200168346-B...
https://aws.amazon.com/blogs/aws/amazon-cloudfront-support-f...
https://blog.cloudflare.com/keep-alive/
https://support.cloudflare.com/hc/en-us/articles/200168346-B...
https://aws.amazon.com/blogs/aws/amazon-cloudfront-support-f...
Also, QUIC helps a lot.
QUIC is OK, but I'm much more interested in 0-RTT TLS 1.3
Is QUIC protocol out of experimental phase?
It's only in Chrome and only supported by Google's web server. They are using it though. You can see the stats by going to chrome://net-internals/#quic
They are "working" to get it into Apache Traffic Server.
They are "working" to get it into Apache Traffic Server.
Yeah, its communicating over port 443. Apache Traffic Server would sure be a big step forward, if and when that happens.
Yeah all QUIC connections are on 443. Here's the most recent public talk from Google about QUIC, given at NANOG 64: https://www.youtube.com/watch?v=cSNT88_gedw
443 UDP though - different to anything else.
Yes and no. It is a ways off from being solidified as an RFC AFAIK, but it serves a healthy portion of Google's traffic. So, more than an experiment but less than a standard.
I was looking for a reliable audio streaming method for a project, I came across QUIC and was in process of studying it. Its definitely seems to be the future, but I am still unsure though about using it for the project.
You should take a look at the WebRTC stack. Opus => RTP => DTLS => SCTP => UDP.
For simpler applications you can drop parts of that, for example just RTP over UDP if you don't need retransmissions or encryption.
For simpler applications you can drop parts of that, for example just RTP over UDP if you don't need retransmissions or encryption.
Yes, WebRTC is also a contender, almost final before I came across QUIC. I was also studying it, the application has to be reliable, so except for the DTLS, I can't skip any other layer of stack. I just have to make a final test to compare the reliability and speed characteristics of TCP vs SCTP (and maybe QUIC) for the application. WebRTC also has excellent P2P features, that's again a big plus. Thanks for the help!
If it has to be reliable as in no loss is allowed, you're going to need slightly higher latency. If TCP-level latencies are acceptable, maybe you should look at icecast. It's super simple to set up, and works in most browsers directly.
Thanks for sharing, icecast looks nice. However, it doesn't relate to the type of application I was trying to build. I was trying to improve over TCP latencies, that's why I was looking over reliable but faster protocols at transport level.
I will probably benchmark everything later and share it with everyone.
I will probably benchmark everything later and share it with everyone.
We have contemplated a POP in Kenya. We will get there. Johannesburg should be less than 200ms, so let me see if I can improve that... I would expect about 100ms.
JHB to Europe is generally round the 190ms mark. 170 if you're closer to the undersea cables. No way you're going to drop that to 100ms.
JHB is much further away from Europe than Kenya, though.
JHB means usually going to Cape Town first, then up the west coast of the continent, wheras Mombassa uses SeaCom, going around the east of the continent. That saves a lot of time from a pure distance perspective alone.
JHB means usually going to Cape Town first, then up the west coast of the continent, wheras Mombassa uses SeaCom, going around the east of the continent. That saves a lot of time from a pure distance perspective alone.
>JHB is much further away from Europe than Kenya, though.
Indeed. I was responding to a comment that was discussing JHB though.
SA traffic is split - some of it goes round east (Seacom), some west (SAT3). Ends up being roughly 200ms east or west.
Source: Got my interwebs over said cables for years
Indeed. I was responding to a comment that was discussing JHB though.
SA traffic is split - some of it goes round east (Seacom), some west (SAT3). Ends up being roughly 200ms east or west.
Source: Got my interwebs over said cables for years
Who is talking about Europe? This is about reaching Kenya. It should actually be ~60-70ms.
Solution: let's replace the "Internet" with only 100 faster-loading websites that we (and the local governments) approve. Win-win, right?
I can't help but notice that the writer is only checking US-based websites. I'm sure that local alternatives actually hosted in-country are much faster. Why should users in Kenya care about loading some Silicon Valley dating app?
Without debating whether or not it's a good thing, it's inarguable that a healthy portion of the apps that are popular worldwide (outside China) are run by companies headquartered in the US: Facebook, WhatsApp, YouTube, Google, Twitter, Instagram. And now Uber.
Given that each one of these has the vigorous majority of their users outside the US and most have substantial non-US infrastructure build, it gets a little strange to label them "US-based".
But I think it's fair to point out that if performance lags to the extreme it may open market opportunities for local competitors to differentiate themselves with speed. Local hosting options are limited, however, making things more challenging for indigenous entrepreneurs. I was amused and appalled to find that many "Kampala web hosting" companies are actually themselves hosted in Los Angeles.
So: not much is hosted in-country. Very little traffic terminates domestically outside caches.
Given that each one of these has the vigorous majority of their users outside the US and most have substantial non-US infrastructure build, it gets a little strange to label them "US-based".
But I think it's fair to point out that if performance lags to the extreme it may open market opportunities for local competitors to differentiate themselves with speed. Local hosting options are limited, however, making things more challenging for indigenous entrepreneurs. I was amused and appalled to find that many "Kampala web hosting" companies are actually themselves hosted in Los Angeles.
So: not much is hosted in-country. Very little traffic terminates domestically outside caches.
Internet freedom for all!
Elephant in the room here -> There's no money to be made there (yet)
You're grossly mistaken here I'm afraid.
Check out the prices from vendors. Also, check out the profits. Also the margin! from the ISPs. Take into account the Capex too. Everything will seem odd, except the volume of data consumed per user [which is related, or better said limited to the maximum actual bandwidth]
Safaricom which is well known for its MPESA platform is the most profitable firm in sub saharan africa [not south africa or nearby] and it's all because of data.
Datacenters are difficult:
- Higher cost of electricity
- Unreliable electricity (blackouts and under-rated voltage)
- Costs of acquiring equipment/servers
- Heating/Cooling is on the harder side
- Lack of competition means pricing is farther up on the higher end
- Back down to cost of data, which is really expensive per GB or to lease the fiber required.
Another thing is there is a large latency chunk added as the vast majority of consumers use 3G which adds another chunk of latency in. This is slightly disincentivising to add in local datacenters.
It may be worth noting AWS is actually run in great part from Cape Town. Everyone uses their data centers in Virginia yet does anyone wonder why AWS doesn't offer anything in South Africa when it's being built here?
Check out the prices from vendors. Also, check out the profits. Also the margin! from the ISPs. Take into account the Capex too. Everything will seem odd, except the volume of data consumed per user [which is related, or better said limited to the maximum actual bandwidth]
Safaricom which is well known for its MPESA platform is the most profitable firm in sub saharan africa [not south africa or nearby] and it's all because of data.
Datacenters are difficult:
- Higher cost of electricity
- Unreliable electricity (blackouts and under-rated voltage)
- Costs of acquiring equipment/servers
- Heating/Cooling is on the harder side
- Lack of competition means pricing is farther up on the higher end
- Back down to cost of data, which is really expensive per GB or to lease the fiber required.
Another thing is there is a large latency chunk added as the vast majority of consumers use 3G which adds another chunk of latency in. This is slightly disincentivising to add in local datacenters.
It may be worth noting AWS is actually run in great part from Cape Town. Everyone uses their data centers in Virginia yet does anyone wonder why AWS doesn't offer anything in South Africa when it's being built here?
The question then: Is there sufficient first-mover advantage that a far-minded company setting up a datacenter there now could reap the benefits when there is money to be made?
I think this leaves out something critical: information infrastructure isn't merely something you deliver to a region to extract profits from it, but something that accelerates growth and helps build a civil society and flourishing middle class.
If I had a couple billion dollars to blow, I'd be spending it on that. Though, maybe that's why I don't have a couple billion to blow =)
If I had a couple billion dollars to blow, I'd be spending it on that. Though, maybe that's why I don't have a couple billion to blow =)
Judging by how quickly certain African countries have taken to mobile payments - in the cases of much of the population, skipping banking altogether - there's plenty of money for those with the right vision.
The profits from serving ads for consumer goods will be pretty slim though.
The profits from serving ads for consumer goods will be pretty slim though.
Far-minded and deep pocketed. Opening a POP in any new country requires a lot of capex, opex, and human capital (legal, coordination, ops, etc).
This glosses over the fact that your service even can geo-scale well over high latency and high jitter networks.
This glosses over the fact that your service even can geo-scale well over high latency and high jitter networks.
Continent with the fastest growing middle class, though.
I'd love to put a POP in joburg (and one in Sao Paulo), but the economics just don't work out for us there. The usual stuff, plus the incremental internationalization costs for the product (more translations, more payments, more support, etc).
Like any business, we'll get the low hanging fruit first, then move on to the higher, harder to reach branches. We'll improve Internet service in Africa, but not before the time is right.
Like any business, we'll get the low hanging fruit first, then move on to the higher, harder to reach branches. We'll improve Internet service in Africa, but not before the time is right.
In one of Google's lobby there is a 3-D animation of a spinning globe, with spikes that appear wherever and whenever searches are performed, with the height of the spikes being proportional to the rate of query submissions.
The recruiter who showed me this was very proud of google's accomplishment and indeed it was impressive to see the spikes erupting from the east and west coast of the US, Europe, Japan and India.
But I watched quietly, sadly and solemnly South America and Africa. There were modest spikes in south africa and brazil. Not many anywhere else.
Both continents are quite populous.
Among the reasons for the choices I've made in my life and my career the last few years is that I have come to regard the vast majority of the code I've written in my long career as a developer as being of no real lasting value to anyone.
Instead I wander around downtown and oldtown portland seeking out crazy people then finding some way to lift their spirits somehow. It can be a little frightening as I specialize on the particularly violent sort. Those are also the most cruelly isolated:
"LEAVE ME ALONE! LEAVE ME ALONE! WHY DON'T YOU LEAVE ME THE FUCK ALONE?"
The gent he was accusing had no clue.
"Hi! My name is Mike. What's yours?"
"Jerry! Pleased to meet you."
Does it have to be so hard?
My point is that those google search spikes would be erupting all over South America and Africa were someone to take responsibility to enable them.
I intend to contribute somehow, someday, in part by setting up some nameservers in cybercafes. There are actually lots of them, cybercafes are quite popular in the developing world.
The recruiter who showed me this was very proud of google's accomplishment and indeed it was impressive to see the spikes erupting from the east and west coast of the US, Europe, Japan and India.
But I watched quietly, sadly and solemnly South America and Africa. There were modest spikes in south africa and brazil. Not many anywhere else.
Both continents are quite populous.
Among the reasons for the choices I've made in my life and my career the last few years is that I have come to regard the vast majority of the code I've written in my long career as a developer as being of no real lasting value to anyone.
Instead I wander around downtown and oldtown portland seeking out crazy people then finding some way to lift their spirits somehow. It can be a little frightening as I specialize on the particularly violent sort. Those are also the most cruelly isolated:
"LEAVE ME ALONE! LEAVE ME ALONE! WHY DON'T YOU LEAVE ME THE FUCK ALONE?"
The gent he was accusing had no clue.
"Hi! My name is Mike. What's yours?"
"Jerry! Pleased to meet you."
Does it have to be so hard?
My point is that those google search spikes would be erupting all over South America and Africa were someone to take responsibility to enable them.
I intend to contribute somehow, someday, in part by setting up some nameservers in cybercafes. There are actually lots of them, cybercafes are quite popular in the developing world.
I'm in Africa and... I'm sorry, but we have nameservers already. That's not really a problem. Are you really talking about DNS servers here? I wasn't sure.
The issues which prevent those spikes you'd like to see on the maps at Google are education (primarily literacy, especially English literacy), access to devices, and culture. Access to devices is being solved by cheap smartphones becoming obtainable to some degree and constant improvements to data networks in African countries, but the other two have no easy solution.
EDIT: To maybe make it a bit easier - consider someone in a village (with a cellphone and access to a data network) who wants to find out why something happened to the carrot he was growing at home. All of us would pop open Google and search about some information about carrots. A guy in a village would head up the road to talk to his friend who's been farming for 50 years and knows about carrots. If you tried to tell him "hey, you could ask Google that question" the only reply you'd get would be "I don't really know about that".
I've had someone come to me to ask me to look up bus fares on the internet for them. They'd hand me a piece of paper with the URL for the bus companies tariff page. Not too interesting until that same person later pulled out his Windows Phone smartphone to make a phone call, and he had full internet access and a good mobile browser on that phone.
The issues which prevent those spikes you'd like to see on the maps at Google are education (primarily literacy, especially English literacy), access to devices, and culture. Access to devices is being solved by cheap smartphones becoming obtainable to some degree and constant improvements to data networks in African countries, but the other two have no easy solution.
EDIT: To maybe make it a bit easier - consider someone in a village (with a cellphone and access to a data network) who wants to find out why something happened to the carrot he was growing at home. All of us would pop open Google and search about some information about carrots. A guy in a village would head up the road to talk to his friend who's been farming for 50 years and knows about carrots. If you tried to tell him "hey, you could ask Google that question" the only reply you'd get would be "I don't really know about that".
I've had someone come to me to ask me to look up bus fares on the internet for them. They'd hand me a piece of paper with the URL for the bus companies tariff page. Not too interesting until that same person later pulled out his Windows Phone smartphone to make a phone call, and he had full internet access and a good mobile browser on that phone.
I'm not sure how related this is, but my parents are from a Pacific island, Palau (I've mentioned it before in other comment threads).
One of the things that hurts progress in my country isn't really a lack of education as much as it is ignorance in the strict sense. It's funny, because a couple of years ago, a trans-Pacific cable was laid close to the island, but the decision was made to not invest in tapping into it due to the cost--even nowadays, people are interested in improving connections, but it isn't a priority. They really haven't experienced the internet, at least not in it's full capacity, so they don't know what they are missing. They know about it from what they see on T.V. ("viral" review shows targeted at baby-boomers who still watch T.V., I suppose) and social networking, so simply, they don't know what they're missing. I recall when I finish high school and moved back here to the US for college--I basically missed the dotcom-boom days of the late 90's and early to mid 2000's, the internet was not just a convenience, it completely changed my living habits.
But the so-called "other three billion" really can't know this because they haven't experienced this, so not only do you face real obstacles, but you face lethargy as well. At least for Palau, I feel like that is one of the main reasons internet access is pretty subpar there today.
[*] I do admit that this might not be relevant in parts of Africa. For one, Palau is a pretty small place (no more than twenty-thousand people), so attitudes are probably much more homogeneous (with that attitude towards the urgency of improving internet infrastructure) compared to hosts of nations with millions of people in sub-Saharan Africa.
One of the things that hurts progress in my country isn't really a lack of education as much as it is ignorance in the strict sense. It's funny, because a couple of years ago, a trans-Pacific cable was laid close to the island, but the decision was made to not invest in tapping into it due to the cost--even nowadays, people are interested in improving connections, but it isn't a priority. They really haven't experienced the internet, at least not in it's full capacity, so they don't know what they are missing. They know about it from what they see on T.V. ("viral" review shows targeted at baby-boomers who still watch T.V., I suppose) and social networking, so simply, they don't know what they're missing. I recall when I finish high school and moved back here to the US for college--I basically missed the dotcom-boom days of the late 90's and early to mid 2000's, the internet was not just a convenience, it completely changed my living habits.
But the so-called "other three billion" really can't know this because they haven't experienced this, so not only do you face real obstacles, but you face lethargy as well. At least for Palau, I feel like that is one of the main reasons internet access is pretty subpar there today.
[*] I do admit that this might not be relevant in parts of Africa. For one, Palau is a pretty small place (no more than twenty-thousand people), so attitudes are probably much more homogeneous (with that attitude towards the urgency of improving internet infrastructure) compared to hosts of nations with millions of people in sub-Saharan Africa.
> consider someone in a village (with a cellphone and access to a data network) who wants to find out why something happened to the carrot he was growing at home. All of us would pop open Google and search about some information about carrots. A guy in a village would head up the road to talk to his friend who's been farming for 50 years and knows about carrots
Speaking as a US native, the reason we'd look for carrot information on the internet is that none of us know anybody with carrot experience. If I knew someone who knew about carrots, they'd see all of my carrot questions before I ever asked the internet.
Speaking as a US native, the reason we'd look for carrot information on the internet is that none of us know anybody with carrot experience. If I knew someone who knew about carrots, they'd see all of my carrot questions before I ever asked the internet.
Yes, but this is the central advantage of distributed information: while your carrot-farmer buddy is a higher-bandwidth and possibly deeper source of carrot information than google is, that guy doesn't know so much about goat husbandry. Or fishing. Or roof construction, or car repair, etc.
So if carrot info is all you ever need, great -- but otherwise, you are going to need nearby buddies who are experts in many, many other things. In fact, you're going to need far more buddies than you can ever actually have, due to the limits of the number of social connections any of us can have (not to mention the limits that time itself imposes on the conversations we'd have to have in order to maintain those relationships even if we could keep them all straight in our heads).
Google's primary utility isn't in making you a perfect expert on anything in particular -- it's that it allows you to become sufficiently expert at anything at all in a really short period of time. Even an army of friends, with the most varied experience you can imagine, cannot come close to matching the breadth of expertise available via the net.
So if carrot info is all you ever need, great -- but otherwise, you are going to need nearby buddies who are experts in many, many other things. In fact, you're going to need far more buddies than you can ever actually have, due to the limits of the number of social connections any of us can have (not to mention the limits that time itself imposes on the conversations we'd have to have in order to maintain those relationships even if we could keep them all straight in our heads).
Google's primary utility isn't in making you a perfect expert on anything in particular -- it's that it allows you to become sufficiently expert at anything at all in a really short period of time. Even an army of friends, with the most varied experience you can imagine, cannot come close to matching the breadth of expertise available via the net.
I plan to live out my whole life knowing nothing about carrot farming, goat husbandry, fishing, roof construction, or car repair.
What's going on here is that Africa isn't really reaping the benefits of specialization of labor in the way the civilized world is. The guy in the example isn't even growing multiple carrots! When I get a problem with my roof, I'm not going to try to fix it myself after a crash course in Google-based roof repair. I'm going to pay someone else to do it for me. I do the same when I want to eat fish or goat, or fix my car. The whole point of a developed economy is that you don't know how to produce the stuff you need. Instead, you produce what you're good at producing, and everyone reaps the gains from trade.
The title of this piece strikes me as a little over-specific. What in sub-Saharan Africa doesn't suck?
What's going on here is that Africa isn't really reaping the benefits of specialization of labor in the way the civilized world is. The guy in the example isn't even growing multiple carrots! When I get a problem with my roof, I'm not going to try to fix it myself after a crash course in Google-based roof repair. I'm going to pay someone else to do it for me. I do the same when I want to eat fish or goat, or fix my car. The whole point of a developed economy is that you don't know how to produce the stuff you need. Instead, you produce what you're good at producing, and everyone reaps the gains from trade.
The title of this piece strikes me as a little over-specific. What in sub-Saharan Africa doesn't suck?
Yes, the issues are substantially bigger. South Africa can't even provide reliable electricity in the cities, never mind elsewhere. People living there tell me it's not a problem. If they're not bothered by that, I doubt they give a damn about google searches. If memory serves me correctly that's been going on for about 10 years now.
South African here.
There's the electricity issue (we call the rolling blackouts "load-shedding") which affects the cellular base-stations and telco exchanges as much as anything, but also another more obvious issue for content delivery: the local loop.
For the past 15+ years (as long as I've had Internet access), the incumbent telco (TelkomSA) has not been forthcoming with upgrading or maintaining the ADSL infrastructure. Finally, there seems to be traction with FTTx providers, but this seems limited to affluent pockets of a few thousand people in the country. All the new fibre offerings serve the same locations.
While it's hopeful this will improve the situation in the medium-term, the backhaul from ISP to eyeball is oversubscribed at least 30:1 for ADSL, with equipment being replaced by (IMO) inferior network equipment. Additionally, there are reports of automated line-conditioning software which should improve local loop connectivity, but I've seen DSL sync issues only since the software has been implemented.
The (ex-public, privatised under Telkom) Exchanges have been neglected and have recently been divided into three groups: Earners, Maintainers and "Lost Causes". The latter two will not receive upgrades from ADSL1 and the Earners will likely migrate to newer technologies. This is largely due to copper-theft and vandalism of exchanges/cellular base-stations (which are oversubscribed enough that wireless technologies routinely sustain 3000-20000ms latencies). The oligopoly running cellular networks have no incentive to improve this, since the Regulatory Authorities don't really seem to push competition as well as we'd like.
TL;DR: There are issues, but rose-tinted glasses make me hopeful in the medium term. Claims are that wireless will fix everything seem bleak, since base stations become oversubscribed faster than they can be built/upgraded (and the backhaul from them is measured in megabits/s, rather than gigabits/s charged at ~US$0.15/MB).
There's the electricity issue (we call the rolling blackouts "load-shedding") which affects the cellular base-stations and telco exchanges as much as anything, but also another more obvious issue for content delivery: the local loop.
For the past 15+ years (as long as I've had Internet access), the incumbent telco (TelkomSA) has not been forthcoming with upgrading or maintaining the ADSL infrastructure. Finally, there seems to be traction with FTTx providers, but this seems limited to affluent pockets of a few thousand people in the country. All the new fibre offerings serve the same locations.
While it's hopeful this will improve the situation in the medium-term, the backhaul from ISP to eyeball is oversubscribed at least 30:1 for ADSL, with equipment being replaced by (IMO) inferior network equipment. Additionally, there are reports of automated line-conditioning software which should improve local loop connectivity, but I've seen DSL sync issues only since the software has been implemented.
The (ex-public, privatised under Telkom) Exchanges have been neglected and have recently been divided into three groups: Earners, Maintainers and "Lost Causes". The latter two will not receive upgrades from ADSL1 and the Earners will likely migrate to newer technologies. This is largely due to copper-theft and vandalism of exchanges/cellular base-stations (which are oversubscribed enough that wireless technologies routinely sustain 3000-20000ms latencies). The oligopoly running cellular networks have no incentive to improve this, since the Regulatory Authorities don't really seem to push competition as well as we'd like.
TL;DR: There are issues, but rose-tinted glasses make me hopeful in the medium term. Claims are that wireless will fix everything seem bleak, since base stations become oversubscribed faster than they can be built/upgraded (and the backhaul from them is measured in megabits/s, rather than gigabits/s charged at ~US$0.15/MB).
I also live in South Africa.
Our mobile networks are in pretty good shape:
* Anyone can buy a prepaid SIM from any supermarket and access the Internet within 10 or 20 minutes. Last time I visited France, it took me 4 days to achieve a similar result.
* Mobile data is cheap when buying in bulk. I pay USD2 per gigabyte.
* LTE coverage is growing fast and RTT is good. For example my ping time to google.com is 30ms.
Conclusion: Private ownership and free markets lead to good service and low cost. ESKOM proves that state ownership of companies and monopolies lead to bad service at high cost.
Our mobile networks are in pretty good shape:
* Anyone can buy a prepaid SIM from any supermarket and access the Internet within 10 or 20 minutes. Last time I visited France, it took me 4 days to achieve a similar result.
* Mobile data is cheap when buying in bulk. I pay USD2 per gigabyte.
* LTE coverage is growing fast and RTT is good. For example my ping time to google.com is 30ms.
Conclusion: Private ownership and free markets lead to good service and low cost. ESKOM proves that state ownership of companies and monopolies lead to bad service at high cost.
> South Africa can't even provide reliable electricity in the cities
Even India cannot provide reliable electricity to its cities. Rural India has scheduled blackouts for 6-7 hours a day.
Even India cannot provide reliable electricity to its cities. Rural India has scheduled blackouts for 6-7 hours a day.
My father was an electrical power engineer. It's quite a lot harder than one would think to distribute power with a nationwide grid.
It's never really made sense to me that we even have a nationwide grid. Why not local grids with local power plants? That's what islands use. While there is economy of scale in a big power plant then you need a bigger grid to consume its power.
There are many ways to generate electrical power. I expect many aren't used because power companies can't earn money from them.
I expect you'd have fewer scheduled blackouts if the rural areas had their own local generators.
It's never really made sense to me that we even have a nationwide grid. Why not local grids with local power plants? That's what islands use. While there is economy of scale in a big power plant then you need a bigger grid to consume its power.
There are many ways to generate electrical power. I expect many aren't used because power companies can't earn money from them.
I expect you'd have fewer scheduled blackouts if the rural areas had their own local generators.
> It's quite a lot harder than one would think to distribute power with a nationwide grid.
I am not an expert on this topic but I am Indian so I know that corruption and apathy play a big role too.
I am not an expert on this topic but I am Indian so I know that corruption and apathy play a big role too.
I've just gotten into the PDX Renters Unite! Facebook group. Rental prices in Portland are skyrocketing largely due to venture-backed companies hiring from out of state.
The solutions are largely straightforward, for example "inclusionary zoning" would require that new housing developments include a certain proportion of affordable housing. But inclusionary zoning is forbidden by Oregon State Law.
In what way does that law serves the needs of the people? Well that was not considered by the legislators, no the developers lobbied them, no doubt set them up with hookers and blow so as to maximize shareholder value while throwing lots of young people out on the street.
The solutions are largely straightforward, for example "inclusionary zoning" would require that new housing developments include a certain proportion of affordable housing. But inclusionary zoning is forbidden by Oregon State Law.
In what way does that law serves the needs of the people? Well that was not considered by the legislators, no the developers lobbied them, no doubt set them up with hookers and blow so as to maximize shareholder value while throwing lots of young people out on the street.
And instead of basic infrastructure the West (which milked colonial Africa for centuries, and continues to do so post-colonially, placing friendly lackeys in power to ensure underdevelopment and advantageous for them business climate), comes with feel-good schemes like "One Laptop Per Child" etc...
South Africa's electricity supply crisis has nothing whatsoever to do with the West or colonialism, it's entirely about bad governance and a lack of long-term planning.
The country has around 40GW of installed capacity, dating from over-investment before 1994, but the state opted to both not build new stations and not carry out any maintenance on the existing stations for years so about 10GW is now offline and unusable. Hence the frequent power outages.
The country has around 40GW of installed capacity, dating from over-investment before 1994, but the state opted to both not build new stations and not carry out any maintenance on the existing stations for years so about 10GW is now offline and unusable. Hence the frequent power outages.
Google isn't the leading search engine in all countries of the world, so they should not spike everywhere.
I'm surprised you didn't notice Russia and China not spiking as much either.
I'm surprised you didn't notice Russia and China not spiking as much either.
Google is the leading search engine in both Africa and South America though.
China here. Google isn't working, supposedly they have connection issues (but we all know the GFW is blocking it). Not surprising to not get traffic from a country not letting it in.
I once read that the GFW permits google from time to time so as to fool those who search with google into thinking that it is a technical fault on google's part, rather than censorship.
If you ask China Telecom why Google and Facebook aren't working, they'll tell you it "must be due to technical problems on their side" -- they do not officially block any websites (some WTO concern) and are incredibly opaque about what they do block.
It seems to me Google and Facebook would have standing to force the US government to tariff or even block chinese imports.
Google is second to Yandex in Russia, but still very, very popular.
> There were modest spikes in south africa and brazil. Not many anywhere else.
Huh? South America has good overall Internet penetration[1] and Chile is going to 70% right now.
[1] https://en.wikipedia.org/wiki/List_of_countries_by_number_of...
Huh? South America has good overall Internet penetration[1] and Chile is going to 70% right now.
[1] https://en.wikipedia.org/wiki/List_of_countries_by_number_of...
South America is the most sparsely populated continent except Australia and Antarctica. The Amazon and Patagonia regions host very little human population (thank goodness).
So South America would always show up as a few big coastal cities with vast emptiness inside, even with huge search penetration on Google's cool visualization. It really doesn't tell you much either way without raw data.
So South America would always show up as a few big coastal cities with vast emptiness inside, even with huge search penetration on Google's cool visualization. It really doesn't tell you much either way without raw data.
What is your requirement for something to be 'lasting'? I don't expect my code to last forever, but if the code I write manage to be part of something that is state of the art for a while in one particular niche, then that is more than enough for me. The only truly lasting pieces of code are the really basic building blocks, stuff like ntpd. Everything else is ephemeral but evolves over time. Eventually new basic building blocks will appear from that, but the process of getting those are mostly accidental.
As for Africa, it is probably a case of not having gained a critical momentum for the kind of infrastructure the article talks about. This doesn't mean it won't happen - while we don't build CDNs at the place I work, we do build basic infrastructure for access control in wireless and mobile networks, and we are definitely seeing more activity and business from African regions compared to how it was earlier.
As for Africa, it is probably a case of not having gained a critical momentum for the kind of infrastructure the article talks about. This doesn't mean it won't happen - while we don't build CDNs at the place I work, we do build basic infrastructure for access control in wireless and mobile networks, and we are definitely seeing more activity and business from African regions compared to how it was earlier.
I want to leave something behind for others when I'm gone.
Most of the code I've ever written, if it will build, won't run on today's computers. While it could be made to by porting a few years after that again it won't work.
Consider Homer's Iliad and Odyssey; you can find them in any library and most good bookstores despite having been composed almost 3,000 years ago.
Most of the code I've ever written, if it will build, won't run on today's computers. While it could be made to by porting a few years after that again it won't work.
Consider Homer's Iliad and Odyssey; you can find them in any library and most good bookstores despite having been composed almost 3,000 years ago.
Sounds like the spikes should be the higher depending on how slow they are, that would make people want to fix it.
Northern China talks to Southern China through Japan, to the extent that they talk at all. All of China at first had one state telecom monopoly, but then some party official set his son up with Northern China. What was left of the state monopoly won't route to it.
My take on what to do about that is to set up servers in both north and south that have website mirrors, as well as a geographically-sensitive nameserver. I don't have a clue how to do the GeoDNS but surely it is a solved problem as big sites require it for load balancing.
My take on what to do about that is to set up servers in both north and south that have website mirrors, as well as a geographically-sensitive nameserver. I don't have a clue how to do the GeoDNS but surely it is a solved problem as big sites require it for load balancing.
China Telecom also has a separate AS dedicated to high party officials, VIP's, etc. It still goes through GFW, but it goes through a dedicated, higher performant GFW.
The US Government, at least at one time had something called FTS, for the federal telecommunications system. It was a private phone system for Uncle Sam.
That is suprising to me.
I lived in Tanzania for a year, and while the internet connection there was not perfect by any stretch of immagination, I had way less ping than this (100 ms to London, where my company's servers are, 120-150 to Boston).
For reference, that is: aerial to my ISP two kilometers away, then fiber to Dar Es Salaam, then Seacom to Mombassa, then Seacom again to Marseille, then terrestrial fiber to London via the channel).
This smells a lot like contention on the seacom link (the big fiber optics carrier that goes up the east coast of Africa through the Suez canal and accross the mediterranean to Marseille). I would be interested to see a traceroute.
Most of the times ISPs in west Africa don't buy enough upstream traffic on undersea cables, creating contention. I can't blame them, it's expensive, but it's mostly an economic argument, not because the servers are too far away (also, most ISPs run pretty agressive caches of everything under the sun).
I lived in Tanzania for a year, and while the internet connection there was not perfect by any stretch of immagination, I had way less ping than this (100 ms to London, where my company's servers are, 120-150 to Boston).
For reference, that is: aerial to my ISP two kilometers away, then fiber to Dar Es Salaam, then Seacom to Mombassa, then Seacom again to Marseille, then terrestrial fiber to London via the channel).
This smells a lot like contention on the seacom link (the big fiber optics carrier that goes up the east coast of Africa through the Suez canal and accross the mediterranean to Marseille). I would be interested to see a traceroute.
Most of the times ISPs in west Africa don't buy enough upstream traffic on undersea cables, creating contention. I can't blame them, it's expensive, but it's mostly an economic argument, not because the servers are too far away (also, most ISPs run pretty agressive caches of everything under the sun).
Are you sure that's round trip? It's at the extreme physical limit of being possible, as the distance to Boston, for example, is greater than 12k km as the crow flies, and the speed of light in fiber is about 70% that of light.
Seacom ends in Egypt (not Marseille). Here's the link on a map:
http://www.submarinecablemap.com/#/submarine-cable/seacomtat...
the eassy link covers much of the same route: http://www.submarinecablemap.com/#/submarine-cable/eastern-a...
the eassy link covers much of the same route: http://www.submarinecablemap.com/#/submarine-cable/eastern-a...
Taking one second to establish the connection doesn't really explain a 12 second load time. TCP takes a little bit to ramp up, but not 30 round trips.
Plus you have TCP preconnect as a lightweight way to avoid that initial delay.
Plus you have TCP preconnect as a lightweight way to avoid that initial delay.
It's that, along with local-loop congestion (and eventually packet loss). Then the retransmits begin.
At least that's my experience with anything cellular-network based.
At least that's my experience with anything cellular-network based.
But the article is talking about plentiful bandwidth over wires. Not a terrible cell link.
However, looking at Twitters front page I see it establishing connections to 6 different domains, that's one TLS session for each one.
This is why CloudFlare is building out PoPs in Africa. Our goal is to be 10s of ms from people living there, not 100s, so that people who use CloudFlare for their sites provide a great experience in African countries.
We are already live in Johannesburg and there will be many more.
We are already live in Johannesburg and there will be many more.
Your Johannesburg PoP is always crashing and gives lots of error messages. Very noticeable every few days with Reddit and HN [clearly shows Johannesburg as the PoP on the error message].
That being said the speed boost is very obvious :)
That being said the speed boost is very obvious :)
We experience the same kind of thing to assembla.com. It often times out. Quite frustrating when in the middle of development.
What sort of error messages are you seeing?
I would have to screenshot it to you when it shows up next and email/tweet it to you. Something along the lines of overloaded, took too long to respond and then the Cloudflare location shows up as Johannesburg.
When it's with HN it shows something like the twitter feed of @HNStatus pointing out the CloudFlare server as Johannesburg with a request id (I think).
When it's with reddit it shows the reddit mascot crushed up with his orange beady eyes over a pile of orange stuff. Scary stuff.
When it's with HN it shows something like the twitter feed of @HNStatus pointing out the CloudFlare server as Johannesburg with a request id (I think).
When it's with reddit it shows the reddit mascot crushed up with his orange beady eyes over a pile of orange stuff. Scary stuff.
I can confirm this; I'm in Durban, South Africa. It says something about a gateway timeout and pops the HN error page with the twitter feed.
Next time it happens, will screenshot and send to cloudflare.
Next time it happens, will screenshot and send to cloudflare.
Lacking any further context i'm almost certain this isn't an issue with cloudflare, instead with the upstream web server.
I saw a cloudflare gateway timeout on HN within the last hour, and the twitter feed had a recent post excusing some technical issue.
I saw a cloudflare gateway timeout on HN within the last hour, and the twitter feed had a recent post excusing some technical issue.
These don't have the twitter feed excusing the issue, the last tweet was a days prior when it happened last. It's as if they only affect African users, the Johannesburg PoP is also used by Kenyan users.
At first, I wondered why the HN Feed never explained the downtime. Then it made sense, it must be cloudflare.
Given that CloudFlare is everywhere & on everyone's websites, it's a pain when it behaves this way affecting users all over a continent.
Uptime websites also report the site as up.
At first, I wondered why the HN Feed never explained the downtime. Then it made sense, it must be cloudflare.
Given that CloudFlare is everywhere & on everyone's websites, it's a pain when it behaves this way affecting users all over a continent.
Uptime websites also report the site as up.
Feel free to email jgc @ cloudflare DOT com.
Does your question mean you turned off logging on the servers?
What sorts of traffic are you seeing out of Africa? How does it compare to Europe and NA, and do you think it's worth for CDNs and large companies to host content there?
It's definitely worth it for us. We plan to be _everywhere_.
And sniff/proxy _everything_ for economic gain.
Your company's motives are not pure and altruistic. Don't pretend they are.
Your company's motives are not pure and altruistic. Don't pretend they are.
Your company's motives are not pure and altruistic.
Huh?
Let's be clear: CloudFlare is a for-profit company and hence wants to take in more money than it spends. We do that by getting people to pay us. Our revenue is from web site owners paying us. It's pretty clear what our business model is: pay is a little for bandwidth and hardware is possible, get web site owners to pay us.
Huh?
Let's be clear: CloudFlare is a for-profit company and hence wants to take in more money than it spends. We do that by getting people to pay us. Our revenue is from web site owners paying us. It's pretty clear what our business model is: pay is a little for bandwidth and hardware is possible, get web site owners to pay us.
That's a nice straight forward business model most people can get behind.
The problem comes later, as it does when any company gets big enough. Few years down the line, the board of directors has a look over the company and see's a big opportunity: all that data you're flowing around can be turned into money with some very clever thinking at 'no real loss to anyone'.
People don't like to leave money on the table if they can just find a clever enough way to put it in their pockets, and more than enough very smart people saying "hey, pay me a little bit and I'll get all that nice money off the table for you - nobody will mind, privacy is dead anyway!"
The problem comes later, as it does when any company gets big enough. Few years down the line, the board of directors has a look over the company and see's a big opportunity: all that data you're flowing around can be turned into money with some very clever thinking at 'no real loss to anyone'.
People don't like to leave money on the table if they can just find a clever enough way to put it in their pockets, and more than enough very smart people saying "hey, pay me a little bit and I'll get all that nice money off the table for you - nobody will mind, privacy is dead anyway!"
Sure, but you have to remember that CloudFlare's customers are web site owners.
When they pay for our service they are trusting us with their DNS and HTTP traffic. If CloudFlare were to start messing with their data it breaks that trust and we lose our customers. We'd end up biting the hand that feeds us.
Related: https://blog.cloudflare.com/what-cloudflare-logs/
When they pay for our service they are trusting us with their DNS and HTTP traffic. If CloudFlare were to start messing with their data it breaks that trust and we lose our customers. We'd end up biting the hand that feeds us.
Related: https://blog.cloudflare.com/what-cloudflare-logs/
The board of directors that would make this decision would have to be a pretty short-sighted one.
Most companies that see a lot of traffic, could decide to engage in shady side activities to monetize every bit of data they see. But no leadership in their right mind would want to take that sort of gamble.
It's interesting that the "evil eye of sauron" argument always comes up for discussion involving CloudFlare, but is never even mentioned for companies like Akamai. My guess is because of the freemium approach which "has to have a catch, right?". But CloudFlare is a proven business model with substantial revenues coming from people who like and trust the service. There is no catch.
ps. former employee here who joined on the idea, and still firmly believes of offering a quality service for a very affordable cost, without being deceitful.
Most companies that see a lot of traffic, could decide to engage in shady side activities to monetize every bit of data they see. But no leadership in their right mind would want to take that sort of gamble.
It's interesting that the "evil eye of sauron" argument always comes up for discussion involving CloudFlare, but is never even mentioned for companies like Akamai. My guess is because of the freemium approach which "has to have a catch, right?". But CloudFlare is a proven business model with substantial revenues coming from people who like and trust the service. There is no catch.
ps. former employee here who joined on the idea, and still firmly believes of offering a quality service for a very affordable cost, without being deceitful.
If CloudFlare even hinted that it was thinking of playing fast and loose with its customers' data, it would be out of business very quickly.
That's somewhat helpful but won't be useful for anything uncachable (e.g. Someone's Facebook wall, twitter feed, emails, etc).
Those things are extremely cacheable. The smart move here is to transport diff as much as possible, which is what Cloudflare is doing with railgun (https://www.cloudflare.com/railgun)
250ms to Facebook and that 'sucks', glad to see New Zealand's opinion on our internet connectivity is validated!
I routinely see 270-300ms pings to facebook. Anything on Amazon's CDN is at 150ms that as is CloudFlare. Akamai does have local nodes, as does Google but the lack of competition on the cable makes it slow.
Guess I should move to Kenya?
I routinely see 270-300ms pings to facebook. Anything on Amazon's CDN is at 150ms that as is CloudFlare. Akamai does have local nodes, as does Google but the lack of competition on the cable makes it slow.
Guess I should move to Kenya?
Same for Australia. (at least outside of state capitals) It's surprising how quickly you can get used to facts like github actually timing out on ssh connection...
I really hope that something like ppsp / swarm / ipfs gets integrated into browsers. I'm pretty sure almost every non-user-content file I download every day has been downloaded somewhere in my town. I shouldn't have to go all the way to the US to get it.
I really hope that something like ppsp / swarm / ipfs gets integrated into browsers. I'm pretty sure almost every non-user-content file I download every day has been downloaded somewhere in my town. I shouldn't have to go all the way to the US to get it.
https://mosh.mit.edu/ may be of use to you as an alternative to ssh
I don't think you're being completely fair. We have excellent infrastructure, especially compared to Australia. Cloudflare now has NZ caches (and they peer locally) so HN is 3-5ms for me. Facebook has strange routing which seems to go via Auckland -> Australia -> California -> North Carolina rather than directly to the USA, which explains the added latency.
HTTP 2.0 is going to be great for you Kiwis.
It is, so is QUIC. My employer hosts in the US but serves to NZ so I've been experimenting with new protocols to improve performance, it's always interesting.
Still won't shake the pain that is a single connection to the rest of the world (the AU links are only used for local traffic).
Still won't shake the pain that is a single connection to the rest of the world (the AU links are only used for local traffic).
I'm in Johannesburg, and my ping to facebook.com is 310ms. The thing is everyone seems to be used to the lag, so there isn't any complaints.
For comparison (Bristol, UK)…
10ms to hacker-news… 130ms to facebook. But only 12ms to a facebook CDN: scontent-lhr3-1.xx.fbcdn.net 10ms to twitter. 13ms to BBC.
10ms to hacker-news… 130ms to facebook. But only 12ms to a facebook CDN: scontent-lhr3-1.xx.fbcdn.net 10ms to twitter. 13ms to BBC.
We are currently building applications for our shops in Africa (Rwanda, Kenya and Uganda) at my current company.
Kenya is not even the worst of all three, Uganda is really a disaster in terms of connectivity. In Uganda it's even difficult to get actual phone signal in most of the country. And you never have reliable Internet anyway, it works like half of the time even if you are in the few right places in the country. Building applications for this kind of environment has been quite challenging.
In a few shops in Uganda, the only way to get data back is by USB keys, so you have an actual truck going there with data to send to the shop and data to get back from the shop. We are building connectivity out of trucks and USB keys.
Kenya is not even the worst of all three, Uganda is really a disaster in terms of connectivity. In Uganda it's even difficult to get actual phone signal in most of the country. And you never have reliable Internet anyway, it works like half of the time even if you are in the few right places in the country. Building applications for this kind of environment has been quite challenging.
In a few shops in Uganda, the only way to get data back is by USB keys, so you have an actual truck going there with data to send to the shop and data to get back from the shop. We are building connectivity out of trucks and USB keys.
That sounds like a challenge! What line of business are you in that can survive that bad connectivity?
We are selling small solar panel kits in small towns on those three countries. The goal is to have a few lights in places where there is no electricity grid at all (it's a for-profit company, not charity). We handle pretty much everything ourselves there since there is not a lot of services pre-built to help you. The shops are selling the kits on payplan since it's quite expensive for the local purchase power. We are the equivalent of a car dealer locally to give a comparison. And the shops are running some software to handle all of that, so this is where my problems are starting ! :)
Sounds like you could branch out and offer these infrastructure services to other businesses, too.
Bummer about all those IPv4 traceroutes. I'd like to see the output of IPv6 traceroutes or mtr output.
For .ZA, I've only really seen native IPv6 at jupiter.is and on the TEnet (NREN) network. My university campus doesn't even have native IPv6 available for servers I maintain.
From what I've seen running an HE tunnel through London, Youtube and Facebook make up the bulk of IPv6 traffic (as expected). At least those networks are the only thing routinely dedicating 10%+ of the total inbound traffic to IPv6 according to my home network flow analysis (I think it's about 18% IPv6 usage on a monthly basis).
From what I've seen running an HE tunnel through London, Youtube and Facebook make up the bulk of IPv6 traffic (as expected). At least those networks are the only thing routinely dedicating 10%+ of the total inbound traffic to IPv6 according to my home network flow analysis (I think it's about 18% IPv6 usage on a monthly basis).
Sadly the majority of consumer traffic is still on IPv4; so if you want to understand what regular people are experiencing - it will be v4 traffic. The network I was on in this case for doing the testing was v4-only. All that said, I'd be quite keen to see where v6 peering is happening.
Is there a way to read this posting without having to log into Facebook?
Type Nairobi in the search box, and you'll see that Akamai has a presence in Nairobi. By the way also in Kampala (Uganda) and Kigali (Rwanda): http://wwwnui.akamai.com/gnet/globe/index.html
>Apple Company
Huh? I thought they're Apple, Inc.
Huh? I thought they're Apple, Inc.
Hmmm, I live in India and I get ~280 ms ping to Facebook and 180 ms ping to Twitter. Seems mostly an issue of "not-living-in-the-US".
This is interesting! It makes me wonder if it could also explain why AJAX based google apps like Gmail totally don't work in Addis Abeba in neighboring Ethiopia, even on the speediest of hotel wifis (really, the download speed was not bad, except for the initial lag). And I mean totally don't work ... gmail.com just gives a blank page, while the plain html version works.
I have been wondering if it could be something with limits of the number of multiple connections, or even requests, affecting the AJAX requests in the background, or some kind of unfortunate timeout limit in the javascript layer.
Not sure, but with this info, I'm of course thinking more about the lag.
Would be interesting with more details on how it affects typical AJAX apps!
I have been wondering if it could be something with limits of the number of multiple connections, or even requests, affecting the AJAX requests in the background, or some kind of unfortunate timeout limit in the javascript layer.
Not sure, but with this info, I'm of course thinking more about the lag.
Would be interesting with more details on how it affects typical AJAX apps!
My company provides mobile app content delivery for a bunch of apps. We use a mobile specific protocol (and backend smarts) that measures and records the latencies for each of these connections. Unfortunately, due to a lack of reasonably easy hosting or cloud solution in Africa, all African traffic is currently served out of Europe. As for sub-saharan countries, Nigeria Kenya and South Africa are the only countries where we see any significant traffic at all. And yes, the latencies are horrible, but if we look at our worldwide traffic, there are bad mobile connections everywhere (even in US and Europe) and we have specifically designed the protocol to minimize roundtrips and handle losses more cleverly than TCP, so we do ok under these conditions (though clearly not as well as we would if we had extensive server presence in the continent).
There is some slight chance that Amazon or Google might introduce compute cloud nodes in Africa... as soon as that happens, we'd love to extend our presence there. And if that's not forthcoming in the near future, we'll definitely explore other hosting options (as we've done in India for now).
There is some slight chance that Amazon or Google might introduce compute cloud nodes in Africa... as soon as that happens, we'd love to extend our presence there. And if that's not forthcoming in the near future, we'll definitely explore other hosting options (as we've done in India for now).
What's the problem with Hetzner? http://www.hetzner.co.za/
Hmm... that's an intriguing possibility. I always thought of Hetzner as a Germany only hosting provider. Thanks for the pointer.
having spent a considerable amount of time in the Horn of Africa, the reason African internet connectivity "sucks" is mostly obvious: lack of local talent, substantially increased costs, graft/corruption, and minimal use of peering infrastructure (plenty of peering fabrics, little to no engagement or use)
while I certainly love the excitement this has generated, it's not as easy as "we'll PoP Kenya! We'll PoP Tanzania! We'll PoP all of Africa! 50ms for everyone!"
you would think a place like Djibout with five submarine cables and terrestrial routes into Ethiopia, Sudan, and others, would have a vibrant peering fabric or CDN nodes.
Level3 just put a CDN node in there, and Telecom Italia Sparkle has been there for a while. AFAIK the CDN node is completely under-utilized and Sparkle provides high cost MPLS services for private and governmental outfits; not as much general IP transit.
More work needs to be put in getting carriers to peer at existing points of presence, and costs have to be lowered any way they can.
while I certainly love the excitement this has generated, it's not as easy as "we'll PoP Kenya! We'll PoP Tanzania! We'll PoP all of Africa! 50ms for everyone!"
you would think a place like Djibout with five submarine cables and terrestrial routes into Ethiopia, Sudan, and others, would have a vibrant peering fabric or CDN nodes.
Level3 just put a CDN node in there, and Telecom Italia Sparkle has been there for a while. AFAIK the CDN node is completely under-utilized and Sparkle provides high cost MPLS services for private and governmental outfits; not as much general IP transit.
More work needs to be put in getting carriers to peer at existing points of presence, and costs have to be lowered any way they can.
instead of wires, even between citys the internet is transmitted by air. Im currently in southwest cameroon, and only means for access is over the air. At home, I use mobile network. But even in all the cyber cafes, offices, etc they use different over the air solutions. As far as I know, no one is wired directly to the network. And apparently even from this region, traffic goes to douala over the air, where it meets optical sea cable. Oh, and incoming connection is under monopoly. If it goes down, everyone go dark.
There are startups whose plan is to create local network coverin city for high bandwith content, without relaying to the outside connection as this is mostly bandwith limited. So called alternative network of catche servers and wifi, as often even watching youtube is pain depending on the time and location.
There are startups whose plan is to create local network coverin city for high bandwith content, without relaying to the outside connection as this is mostly bandwith limited. So called alternative network of catche servers and wifi, as often even watching youtube is pain depending on the time and location.