Starting an Internet Service Provider(chrishacken.com)
chrishacken.com
Starting an Internet Service Provider
http://chrishacken.com/starting-an-internet-service-provider/
247 comments
>A small new player can't compete in either of those markets.
We're Wifidabba.com and we're in the current batch of YC. We provide low-cost internet in India at tea-stalls and bakeries. Consumers buy tokens and get access to wifi. In effect, we take a broadband connection and slice it into tiny pieces and sell that as wifi.
I guess in some ways we're an ISP, the government certainly thinks so, we've had to apply for an ISP license.
We're only a few months old, but we're reasonably profitable and the future looks good.
I think there's room for new ISP's if you're willing to be creative about how to grow the number of customers you want to reach. For example, we're getting tons of inquiries about being an IOT platform amongst other things.
*edited for clarity
We're Wifidabba.com and we're in the current batch of YC. We provide low-cost internet in India at tea-stalls and bakeries. Consumers buy tokens and get access to wifi. In effect, we take a broadband connection and slice it into tiny pieces and sell that as wifi.
I guess in some ways we're an ISP, the government certainly thinks so, we've had to apply for an ISP license.
We're only a few months old, but we're reasonably profitable and the future looks good.
I think there's room for new ISP's if you're willing to be creative about how to grow the number of customers you want to reach. For example, we're getting tons of inquiries about being an IOT platform amongst other things.
*edited for clarity
Your business plan is a bit different though, you charge on data.
While you are apparently cheap you'll still make more money per GB than an isp that tries to compete on the "Unlimited" plans.
While you are apparently cheap you'll still make more money per GB than an isp that tries to compete on the "Unlimited" plans.
Bingo :)
Hmm, went through your website and i like your idea.
What do you use for backhaul? chaiwalas and paanwalas aren't going to have network access in form of fiber or even cable on the street so i guess 4G hotspot? Also, Is there a way people can pay the stall owner some money and get a token from him? People here generally dont prefer online transactions.
What do you use for backhaul? chaiwalas and paanwalas aren't going to have network access in form of fiber or even cable on the street so i guess 4G hotspot? Also, Is there a way people can pay the stall owner some money and get a token from him? People here generally dont prefer online transactions.
We use the local isp and get a fibre line to the chai wallah. We help do the kyc, installation and whatever else is necessary.
Tokens are physical and work exactly the way you describe. Pay cash, get wifi.
Tokens are physical and work exactly the way you describe. Pay cash, get wifi.
The pricing looks super cheap. All the best.
We believe in charging for a good service. We don't ever want to show ads or mess with your connection in any way. thanks !
this is mesh network, right? can you tell a bit about your technology?
It's a mesh built on low cost custom hardware. Each node is also super powerful with >128gb storage and some nice computation power, with a GPU to boot :) We're looking to be more than just an ISP in the long run.
For what value of "more than just an ISP"?
A sort of AWS style platform for retail. Eg. Are you building a POS or Inventory app or do you want to sell your online goods offline? You can access our merchants and authenticated users that already have a prepaid wallet and are at a retail location predisposed to buying. And because of our connectivity, you don't have to worry if your users have internet.
Or maybe an IOT platform of sorts, for example, you're an FMCG and you want to know that your shipments really did arrive at the location your distributor was supposed to deliver it.
India's a challenging country where > 80% of our retail is fragmented into millions of micro-stores.
Or maybe an IOT platform of sorts, for example, you're an FMCG and you want to know that your shipments really did arrive at the location your distributor was supposed to deliver it.
India's a challenging country where > 80% of our retail is fragmented into millions of micro-stores.
So using the user's phone just for auth, akin to cards? Nice idea!
What do you need the GPU for? The box does not appear to have a display.
A distributed machine-learning network for some whatever reason :)
If you buy a cheap ARM SoC, there's a decent chance that you get a GPU "for free" (in the sense that chips without a GPU turn out to be more expensive).
Reliance jio is providing 4g data almost for free in india why would i want to buy wifi as well ?
Jio's 4G data isn't going to last forever. They are going to charge for it someday. Maybe from april, maybe from next year. But one day they will. Also Jio at some places is too slow still.
So this is something that surprised me as well, especially given we don't offer free plans or a trial.
When we spoke to our users they said the following: - Way faster than Jio (We offer 100mbps) - The price was so low that they didn't mind paying - They don't trust reliance and the ID process of getting a jio connection - It's so cheap and fits in their daily routine of getting chai, cigarette, etc.
When we spoke to our users they said the following: - Way faster than Jio (We offer 100mbps) - The price was so low that they didn't mind paying - They don't trust reliance and the ID process of getting a jio connection - It's so cheap and fits in their daily routine of getting chai, cigarette, etc.
> No. And the reality is that IP connectivity is not much different to these other utilities.
I understand where you're coming from, but it is different. You have ample power to power your home; utilities can fulfill whatever your power requirements are.
The same can't be said about IP. I am literally incapable of live streaming from my parents house, no matter how much I'm willing to pay. There's no internet option, among the few available, that are capable of fulfilling my requirements.
I understand where you're coming from, but it is different. You have ample power to power your home; utilities can fulfill whatever your power requirements are.
The same can't be said about IP. I am literally incapable of live streaming from my parents house, no matter how much I'm willing to pay. There's no internet option, among the few available, that are capable of fulfilling my requirements.
Yeah no need to explain the situation because I started my own ISP 16 years ago because I bought a house in Montana that the telco assured me could get T1 service. But they lied, so I had to build my own wireless ISP 10 miles out from the nearest town to get service. And to justify the investment in upstream and equipment I decided to serve the "underserved" businesses in town. Been there, have the T-shirt.
Consider the analogy further : if you built a house in a place with no power line, you would expect to either not use much power, or to pay a hefty fee (10's of thousands) to get power brought in. But folks expect to get Internet capable of streaming Netflix 4K (on two screens concurrently) for $50/mo.
Consider the analogy further : if you built a house in a place with no power line, you would expect to either not use much power, or to pay a hefty fee (10's of thousands) to get power brought in. But folks expect to get Internet capable of streaming Netflix 4K (on two screens concurrently) for $50/mo.
I agree with you that rural-broadbandification will be as important as rural-electrification was in the '30s. That said, I think that most lucrative (read: capital efficient) business model here is low-bandwidth web sites, not high-bandwidth ISPs. Does a news site really need to ship 2Mb of JS, a 20Mb auto-play video, and 5Mb of ads to communicate 1.5Kb of plain-text? Maybe if your users have fast connections but not if they're in rural Idaho and it takes 2 minutes (that's 2,000 milliseconds) to load a single page.
Edit: numbers
Edit: numbers
Small correction 2 minutes is 120000 milliseconds
You could get fibre put in, if your budget is big enough, but I get what you mean.
Fantastic write up, Chris.
Chris has been hanging out with us (Treasure State Internet, our startup ISP in Montana) in a Slack channel called ISP School.
If you're interested in any aspect of starting and running an ISP, please join us. http://slack.tsi.io/#ispschool
Chris has been hanging out with us (Treasure State Internet, our startup ISP in Montana) in a Slack channel called ISP School.
If you're interested in any aspect of starting and running an ISP, please join us. http://slack.tsi.io/#ispschool
It says ura_limit_reached. I'm guessing you don't have enough paid accounts.
I'll look into it. We should have an unlimited account.
Edit: it appears there is a 5 guest limit per paid account.
Edit 2: I think I'm going to set up a Mattermost server and push people that way.
Edit: it appears there is a 5 guest limit per paid account.
Edit 2: I think I'm going to set up a Mattermost server and push people that way.
Or you could just use an IRC channel...
But that's old. I'm only interested in new new NEW!!
You have to admit, getting people on slack is much easier than getting them to join your irc channel.
Despite having to have Slack for work anyway, I find joining public projects' Slack channels to be too high-commitment, so I never do it. IRC? No problem. I also like that IRC somehow has both a less formal and a lower-BS feel/culture to it.
For non-tech people, yeah, it's probably easier.
For non-tech people, yeah, it's probably easier.
[deleted]
Hi, let us know when this is ready, i would love to join as well.
Consider matrix first.
You could also consider discord. It's popular in the gaming community but is capable of housing large discussions
That would work too.
Would definitely recommend Discord, although it's aimed at gamers, it seems to be an amazing all-round chat service.
Rooms are of an unlimited size and have all the core functionality like file & image sharing, code highlighting, "reactions", which we've come to expect but also focuses more on getting people to speak to each other. It's built on electron and is both really quick & also require almost no resources when running in the background.
Rooms are of an unlimited size and have all the core functionality like file & image sharing, code highlighting, "reactions", which we've come to expect but also focuses more on getting people to speak to each other. It's built on electron and is both really quick & also require almost no resources when running in the background.
Consider simply using matrix.org, no need for hosting anything yourself unless you want to
Might want to try Riot. It uses a protocol called Matrix, which is the successor to IRC. What this means is that you're not tied to them. Also the web, desktop, and mobile clients are pretty sweet.
Does Matrix really have the userbase in place to start calling it an IRC successor so definitively?
It may be aiming for it, and I do hope we see an open chat protocol succeed over all the Slacks and HipChats, but calling it the successor right now seems a little disingenuous!
It may be aiming for it, and I do hope we see an open chat protocol succeed over all the Slacks and HipChats, but calling it the successor right now seems a little disingenuous!
Don't see anything else around that fits that bill. They also have amazing IRC bridging - I have been using Matrix bridged into Freenode as my primary 'IRC Client' for almost a year now.
IRCv3?
It's more aimed to be a successor to XMPP/SMS - hence the focus on federation and mobile friendliness.
Edit: Note this statement is based on my talking to people actually at matrix, on the dev team.
Edit: Note this statement is based on my talking to people actually at matrix, on the dev team.
Calling Matrix the successor to IRC, when there's actually new IRC standards in development, is very misleading.
I'd really like to see usage of Matrix/Riot instead. A federated, open network and you can easily bridge your existing slack channels.
[deleted]
Trying to sign up as well to no avail.
looks like the channel is full.
I think if your slack instance is a "community/group" there is no limit on users. It is only when you jump over to the paid plans.
I've seen a possibly related issue with static electricity in a large building in PA that had 3 power feeds, after lightning we would have ethernet port problems.
I would suggest that you declare a 2am Thursday maintenance window and use a script to automatically reboot all the devices. This in combination with checking that you have a good separate ground wire connection at all antenna locations should pretty much fix it. The reboot cycles the power and should hopefully drain anything extraneous.
I would suggest that you declare a 2am Thursday maintenance window and use a script to automatically reboot all the devices. This in combination with checking that you have a good separate ground wire connection at all antenna locations should pretty much fix it. The reboot cycles the power and should hopefully drain anything extraneous.
That sounds like a great idea. I've actually debated doing this before, but I wasn't sure if the service disruption was worth it considering it happens maybe once every other month and sporadically at that.
I ran a bootstrapped ISP much the same way. I ran into the static issue as well did a few things to fix it. Ran a better ground on the roof. Isolated the tower from the roof surface with wood (that was done during tower erection). Lastly, I put a fiber link cut in-between the run of Ethernet from the data center, and the router box on the tower.
I did a bunch of ISP builds in my younger days. It is worth while to see if you can horse trade some internet for an electrician with tower experience (or a real RF engineer) to help you.
As other commenters have suggested, check your ground potential and make corrections as necessary.
As other commenters have suggested, check your ground potential and make corrections as necessary.
Unless I open up the device, there's nothing to attach ground wire to. The device is constructed primarily of plastic. Grounding the dish wouldn't do anything, would it?
I think you want to be grounding the shield on your Coax line. But I'm not an electrician or an RF engineer, so I might be completely wrong.
The last level3 contract I had had such a maintenance window declared in it. So I can't claim credit for it :)
You could make the maintenance once a month on days your monitoring shows already have about no traffic. If a HA setup, then fail-over, reboot main, fail-back-over, reboot other stuff. Minimal disruption there.
> The reboot cycles the power
Er, does it? Some of the components might go offline, but there's usually still a trickle-charge running to the components even when the computer is "off"—and especially the ethernet card, for Wake-on-LAN support (even if it's disabled.)
Er, does it? Some of the components might go offline, but there's usually still a trickle-charge running to the components even when the computer is "off"—and especially the ethernet card, for Wake-on-LAN support (even if it's disabled.)
[deleted]
... were the racks not properly grounded?
Godspeed man. I wish the regulatory landscape was more open to this kind of thing all over the country. The incumbent ISPs need some competition.
Thank you. I obviously completely agree; we tried relentlessly to contact our local utilities to gain access to their underground conduit and was never able to get someone on the phone or even much of a response.
I eventually got to the point where I was like, "Fk it, I'll do it on my own." Now we're looking at microtrenching, which is probably an even larger endeavor, but at least we'll own the conduit.
I eventually got to the point where I was like, "Fk it, I'll do it on my own." Now we're looking at microtrenching, which is probably an even larger endeavor, but at least we'll own the conduit.
You might get more response if you call them and say "Hey, since we haven't heard from you we're going to be trenching in our own conduit. This is a written request for up-to-date maps of where yours is so we can be sure we don't cut anything."
It might not be the response you want, but it may make it more likely that someone will respond.
It might not be the response you want, but it may make it more likely that someone will respond.
They won't send any maps, they will just come out and mark their underground utilities before you dig.
Perhaps try reaching out to the B4RN folks to learn some of their lessons. A lot won't apply because UK vs. US but I'm sure some of their experiences might help you avoid problems! https://b4rn.org.uk
Agreed 100% on the B4RN suggestion, they're doing some pretty amazing work on relatively tiny budgets.
There was a presentation at UKNOF34 from B4RN: https://youtu.be/C4Xm_-gt6mw?list=PLjzK5ZtLlc91r5YZVNiebhcq0...
> we tried relentlessly to contact our local utilities to gain access to their underground conduit and was never able to get someone on the phone or even much of a response.
I find that really hard to believe. Go to their offices and ask for the manager.
> I eventually got to the point where I was like, "Fk it, I'll do it on my own." Now we're looking at microtrenching, which is probably an even larger endeavor, but at least we'll own the conduit.
Building your own facilities is no joke. Unless you are made out of money, you'll be far better off putting more effort into renting existing ducts.
I find that really hard to believe. Go to their offices and ask for the manager.
> I eventually got to the point where I was like, "Fk it, I'll do it on my own." Now we're looking at microtrenching, which is probably an even larger endeavor, but at least we'll own the conduit.
Building your own facilities is no joke. Unless you are made out of money, you'll be far better off putting more effort into renting existing ducts.
Renting duct is $3/ft/year. That's $15,840 per mile per year. That's not including ~$30,000 to contract out the work (I was quoted $7/ft) and $5,000 that the utility charges to have someone on-site to oversee the operation. I've determined we can microtrench ourselves for between $30,000 and $40,000.
I've been on contact with another ISP, https://tsi.io who has successfully done micro-trenching with very minimal capital. We're doing all of the work ourselves to reduce overhead.
I've been on contact with another ISP, https://tsi.io who has successfully done micro-trenching with very minimal capital. We're doing all of the work ourselves to reduce overhead.
That website is very cool. It will be really interesting if this catches on. There are a ton of rural customers underserved by DSL or even wireless ISPs.
How do you know the duct rent if you haven't been able to contact the utility?
$7/ft is more than you should be paying. Get some more competing quotes.
I'd be very surprised if you can hit $30-40k per mile all in when microtrenching.
$7/ft is more than you should be paying. Get some more competing quotes.
I'd be very surprised if you can hit $30-40k per mile all in when microtrenching.
It's in their agreement, which is available publicly online. I managed to get 2 or 3 email responses, but they go dark for 2 months at a time.
https://www.dropbox.com/s/wcd5z1itzu141ky/Screenshot%202017-...
https://www.dropbox.com/s/sbkhujxkpepadjw/Screenshot%202017-...
https://www.dropbox.com/s/wcd5z1itzu141ky/Screenshot%202017-...
https://www.dropbox.com/s/sbkhujxkpepadjw/Screenshot%202017-...
You might check what the agreement between the utility and the public utility commission is to see if you can have the PUC lean on the utility to provide the data you need.
http://www.puc.pa.gov/
http://www.puc.pa.gov/
I've had several meetings with a District Rep, who contacted his government rep at the utility for the area, who leaned into them. They said they would contact me; 3 times, and never did.
What happens when you go to their offices?
Do you actually expect to turn a profit doing this? Seems like it will never pay for itself.
>Now we're looking at microtrenching, which is probably an even larger endeavor, but at least we'll own the conduit.
Hi Chris - good work, inspiring to read. I'm in Bournemouth, UK, and around 10 years ago FibreCity was a company created to attempt to connect all the houses in Bournemouth to FTTH - initially by using sewers, but eventually by microtrenching. I actually had a month's trial (1Gps, which was incredible), but the company folded, and from what I recall the costs were immensely greater than they even thought with the microtrenching; they also managed to P off nearly everyone in the area as they promised very little disruption, but actually led to no difference compared to normal trenching works. Ultimately large parts of Bournemouth now have FTTH which is dark, which is disappointing.
Not trying to put you off, but just mentioning it as a study or worthwhile research - you seem like a good guy doing a good job, so hoped I could contribute something.
Hi Chris - good work, inspiring to read. I'm in Bournemouth, UK, and around 10 years ago FibreCity was a company created to attempt to connect all the houses in Bournemouth to FTTH - initially by using sewers, but eventually by microtrenching. I actually had a month's trial (1Gps, which was incredible), but the company folded, and from what I recall the costs were immensely greater than they even thought with the microtrenching; they also managed to P off nearly everyone in the area as they promised very little disruption, but actually led to no difference compared to normal trenching works. Ultimately large parts of Bournemouth now have FTTH which is dark, which is disappointing.
Not trying to put you off, but just mentioning it as a study or worthwhile research - you seem like a good guy doing a good job, so hoped I could contribute something.
What're your thoughts on this idea: internet access is a right and should be treated like a utility. Operators like you and others create supply and then are guaranteed a price or rate as (probably) negotiated by the consortium of suppliers like yourself and others and the local government. And then in return businesses and consumers can rely on (fiber based) internet for a reasonable cost. I personally don't care for this because given a guaranteed stream of income I think incubanants would have little incentive to innovate. I'd rather there be an open and fair market for people like you to come into an underserved market and turn things on its head and lead to innovations like wireless gigabit and other tech. Curious on your thoughts though.
The biggest problem is: where is the money going to come from?
Even if you guarantee a rate of return, somebody still needs to finance the buildout.
Second issue is, how will infrastructure that is installed, but not used be paid for? Take rate is rarely 100%, so who is going to pay for that part and how?
Even if you guarantee a rate of return, somebody still needs to finance the buildout.
Second issue is, how will infrastructure that is installed, but not used be paid for? Take rate is rarely 100%, so who is going to pay for that part and how?
If you can guarantee a rate of return, getting money from any bank sounds easy. And if take rate is 60%, then those that take the offer have to pay 100%/60%=166% of what they would have payed with a 100% take rate. Of course that requires estimating the take rate, but that's an everyday buisiness problem.
> If you can guarantee a rate of return, getting money from any bank sounds easy.
Sadly this isn't true. Banks lend you money if you have collateral.
> And if take rate is 60%, then those that take the offer have to pay 100%/60%=166% of what they would have payed with a 100% take rate.
That only works if the customers are willing to pay 166%. The shit really hits the fan if you get the take rate wrong and your costs exceed what your customers have agreed to pay.
Sadly this isn't true. Banks lend you money if you have collateral.
> And if take rate is 60%, then those that take the offer have to pay 100%/60%=166% of what they would have payed with a 100% take rate.
That only works if the customers are willing to pay 166%. The shit really hits the fan if you get the take rate wrong and your costs exceed what your customers have agreed to pay.
Governments would pay for this via taxes. At least that's one way. So in the end all of us would be paying for internet, and I guess if there were a way to get internet at 1gig up/1 gig down with no data caps that was very reliable for under 100 bucks a month I would do it. But the build-out costs a lot as you've stated so the capital would have to come from somewhere and perhaps that'd come from infrastructure spending.
We can't convince voters to raise taxes/utility rates enough to fix crumbling dams, lead water pipes, and overflowing sewers.
That's very true. Unfortunately, if you threatened to take away their Facebook I think they'd suddenly reconsider.
Depends on the locality; my city just raised taxes to address infrastructure needs in schools and storm sewers. It's probably not a coincidence that they are also looking at municipal fiber.
Unfortunately I happen to agree. Until a disaster happens that the public can be convinced is linked directly to a lack of infrastructure spending I think nothing will happen.
Just set up a special district: https://www.youtube.com/watch?v=3saU5racsGE
This is (roughly) how the New Zealand telecommunications market works.
Find out as much as you can about your city's regulations. In Madison, WI if your conduit is 3" or larger the city can take ownership of it if they want with zero compensation for the cost you spent to build it out. They will reuse it for utilities, traffic lights, etc. You can still use it, but they own it.
Jeez, that's rough. Sounds like a great recipe to drive away investment and for only installing 2" conduit to the detriment of all.
This is why you get an agreement with the city to share their conduit and help each other out where possible. Not sure of the political climate there, but getting on their good side will save you money and time.
Become a CLEC. Or, more appropriately, I suppose: look into the benefits that being a CLEC would give you (in this regard) and consider if that's worth it for you.
That's in the works.
You don't have to be a CLEC to access right of way, including ducts and poles. It's enough to be a BIAS (Broadband Interner Access Provider), which you already are.
Becoming a CLEC does classify you as a "utility" though, which has other benefits.
Which benefits do you need that a CLEC has but a BIAS does not?
Chris - I might have an interesting low-cost microtrenching option for you. This company demo'd at a recent TechBreakfast: http://traxyl.com/. I know the founders and they are very motivated to make this work for early customers. Let me know if you want a connection.
Can you hook me up too? Email in profile.
France has historically been a country with some small, associative ISPs. The oldest ISP in France, FDN (French Data Network) is an association. In 2011 they created a federation of such ISPs, FFDN. Check out their home page if you are interested, they have some information in English: https://www.ffdn.org/en
While I don't wish you ill will, I do wish to remark that you have had it remarkably easy with this.
Source: 7 ISPs. In Africa. On unreliable power, broken fibre, people stealing your spectrum, etc.
Source: 7 ISPs. In Africa. On unreliable power, broken fibre, people stealing your spectrum, etc.
Do your customers tolerate all that? As in, can you ignore a good chunk of it because Internet issues are more acceptable in Africa? Or did you have to prevent, detect, and/or mitigate all of it because they expect nearly-perfect Internet in your area?
By way of answering, let me recount a short story for you.
It takes place during the very first time I visited Kampala (the capital of Uganda). It was about 3 years before I had started working in the ISP/telecom space in Africa. This particular trip was soccer-specific (I was working on the team that delivered one of the big trans-national African tournaments' games on television), thus the non-ISP nature of it.
We arrive at our hotel, having driven just under an hour from Entebbe to Kampala. It's fairly pleasant, albeit a tad humid (it's mid-day and near the equator after all; 36C and >=70% humidity - not the worst). There's a group of around 30 of us that now need to check in... this always takes a little while.
So, everything goes on. You loaf around the hotel reception until it's your point in the queue, grab some water now and then when you can. Finally, you're standing at the front desk, starting your process of checkin.
You read off your name... the person behind the counter finds you in the list (of fortunately mostly checked-off people - yay alphabet precedence!), and starts doing some stuff on the front desk computer. At which point you glance over to said desktop and notice the UPS jacked into the UPS jacked into the UPS (yes, three) jacked into the computer. You idly inquire about this. The response is "oh, yeah, the other ones died with all the power outages".
Not a joke, as it turns out. While you're doing your checking, the building experiences a brownout twice.
Years later, you get there for a totally different reason. Time (literal years) has passed. And during setup of some equipment (here, I'm skipping about two days' worth of time), you find that the reason your laptop screen kept dimming and your DC UPS' kept beeping.... is because you're getting such major voltage swings on the building feed that everything is going over-or-under-voltage.
This is daily life (in many parts of Africa). It isn't a thing to fight against. It just is. You take it, and try to do your utter best. You try to deliver your utter best.
And if you just push hard enough (and, imo, if you're really, really lucky) you maybe get somewhere with it!
It takes place during the very first time I visited Kampala (the capital of Uganda). It was about 3 years before I had started working in the ISP/telecom space in Africa. This particular trip was soccer-specific (I was working on the team that delivered one of the big trans-national African tournaments' games on television), thus the non-ISP nature of it.
We arrive at our hotel, having driven just under an hour from Entebbe to Kampala. It's fairly pleasant, albeit a tad humid (it's mid-day and near the equator after all; 36C and >=70% humidity - not the worst). There's a group of around 30 of us that now need to check in... this always takes a little while.
So, everything goes on. You loaf around the hotel reception until it's your point in the queue, grab some water now and then when you can. Finally, you're standing at the front desk, starting your process of checkin.
You read off your name... the person behind the counter finds you in the list (of fortunately mostly checked-off people - yay alphabet precedence!), and starts doing some stuff on the front desk computer. At which point you glance over to said desktop and notice the UPS jacked into the UPS jacked into the UPS (yes, three) jacked into the computer. You idly inquire about this. The response is "oh, yeah, the other ones died with all the power outages".
Not a joke, as it turns out. While you're doing your checking, the building experiences a brownout twice.
Years later, you get there for a totally different reason. Time (literal years) has passed. And during setup of some equipment (here, I'm skipping about two days' worth of time), you find that the reason your laptop screen kept dimming and your DC UPS' kept beeping.... is because you're getting such major voltage swings on the building feed that everything is going over-or-under-voltage.
This is daily life (in many parts of Africa). It isn't a thing to fight against. It just is. You take it, and try to do your utter best. You try to deliver your utter best.
And if you just push hard enough (and, imo, if you're really, really lucky) you maybe get somewhere with it!
I also work in tech in Africa (2nd largest population on the continent if that's any indication) and it is certainly challenging.
Love to chat/commiserate more about your experiences
Love to chat/commiserate more about your experiences
Interesting. So, you actually had it easier than Chris since the expectations of service are so low in Africa. You might have had it harder because of all the extra issues you deal with. As I figured, it's not as straight-forward as your original comment implied.
As the parent has said, the consumer are paying and often higher prices than in Europe so they complain. A LOT.
Ah, no no. The expectations are not low.
People are paying for a service, and they expect it to work.
I once (another country, another ISP) dealt with a query/complaint of "it doesn't work!" which turned out to be due to a snowstorm taking out _the entire area_. Which people knew about! And still complained!
The expectation for the service is really much of the same.
Almost ironically, the first marker that people phone about is "my email doesn't work!"
edit/addendum: your customers also have their own solutions for dealing with power outages - more UPSs, generators, laptops (with internal wimax or 3G chips), etc
People are paying for a service, and they expect it to work.
I once (another country, another ISP) dealt with a query/complaint of "it doesn't work!" which turned out to be due to a snowstorm taking out _the entire area_. Which people knew about! And still complained!
The expectation for the service is really much of the same.
Almost ironically, the first marker that people phone about is "my email doesn't work!"
edit/addendum: your customers also have their own solutions for dealing with power outages - more UPSs, generators, laptops (with internal wimax or 3G chips), etc
In that case, your comment stands and it seems like a hell of a tough market.
Yeah... it's tough..
Still, I'm sticking around. There's potential to do good here.
Still, I'm sticking around. There's potential to do good here.
Also interested in you doing a detailed writeup on your tactics, costs, etc as an African ISP.
Depends on many, many things. Most are fairly specific to the region in question (whichever it might be).
As a mini-followup: due to this kind of scenario, we ended up having to replace UPS battery packs in about 8 months (1/3 of their minimum intended lifecycle) due to how heavily they got load-rushed over the months.
The load cycle literally killed them in that time.
The load cycle literally killed them in that time.
Interesting - so I guess there is need for a solution which charges UPS-es in some smarter way?
Better smarts on discharge of the UPS can help. Most of them currently let the battery pack fully discharge, causing irreversible damage to the battery pack, especially if power isn't restored quickly. A setting for cutting out at 40% depth of charge would prevent damage to the pack.
Over provisioning the runtime on the UPS can help if you can afford the extra upfront cost.
What someone really needs to do is make a UPS that just attaches to a standard deep cycle battery, as those can be easily sourced locally pretty much anywhere and one can pick the battery to fit the application, rather than be stuck with the crap batteries the UPS manufacturer decides to use (they typically use high C batteries which do not like deep cycling).
Over provisioning the runtime on the UPS can help if you can afford the extra upfront cost.
What someone really needs to do is make a UPS that just attaches to a standard deep cycle battery, as those can be easily sourced locally pretty much anywhere and one can pick the battery to fit the application, rather than be stuck with the crap batteries the UPS manufacturer decides to use (they typically use high C batteries which do not like deep cycling).
I don't believe that in a situation with wildly fluctuating power there is any alternative to "grab all available energy from the grid when there is power", you never know how huge the next brown/blackout will be.
What could definitely help is putting good power conditioning equipment in front of the UPS. That at least will protect the AC-to-DC first stage from overvoltages and shorter brownouts. If the PCE comes with fat enough supercaps, it can also protect the main UPS by preventing it from draining the battery at all.
What could definitely help is putting good power conditioning equipment in front of the UPS. That at least will protect the AC-to-DC first stage from overvoltages and shorter brownouts. If the PCE comes with fat enough supercaps, it can also protect the main UPS by preventing it from draining the battery at all.
I subscribed to Chris' newsletter as I hope to hear more about the challenges involved in his endeavor. Starting a small ISP has been a vision of mine since my first after-school job at a ISP helpdesk 15 years ago. I'd really love to be involved in running a community ISP in an underserved area, so I'm interested in reading about the technical and regulatory challenges especially.
@froztbyte I'd be doubly interested in a writeup of your experience, have you blogged anywhere?
@froztbyte I'd be doubly interested in a writeup of your experience, have you blogged anywhere?
Nothing exciting published, no. Blogging wasn't particularly big around this part of the world/specific-weird-subset-of-the-market for most of that time. Should I consider writing up some of them?
Past aside, there are many things already underway, in many areas, that you could be involved in if you wished. Do some research around areas that you find yourself interested in (both technologically, and geographically), and see where that takes you.
Past aside, there are many things already underway, in many areas, that you could be involved in if you wished. Do some research around areas that you find yourself interested in (both technologically, and geographically), and see where that takes you.
I'm the type of person who can lose hours reading stories on sites like Jargon File, Folklore.org, Raymond Chen's blog about MS internals, and even more modern internet history stories. And as I wouldn't limit my search on areas to get involved in muni ISPs to my own country, a global perspective would be interesting to read.
Seconded. I spent a lot of time in my teenage years just reading plaintext HOWTOs and RFCs (metered dialup sucks).
Thirded.
It's fun, and educational. It empowers the ADHD in me.
It's fun, and educational. It empowers the ADHD in me.
Thanks for this article. I have been considering the feasibility of starting a WISP here in West Texas, so it's cool to see the beggining stages from the inside perspective.
I whish I knew more about the laws on common access to equipment/lines. For example, the only cable provider locally is Suddenlink, AT&T Uverse is still on copper phone lines, and there is only one local ISP bringing fiber to residential customers. The local ISP got a huge bunch of grants for the fiber rollout (of course someone embezzled a bunch of it (few mil) and they had to get bought out to get the new set of grants) but all I want to know is: if the government payed for the fiber rollout, shouldn't I be able to use it as well for a nominal fee of some sort?
What about ownership and fees on the last mile? How does all that work? I just wish there was a clear and simple place to find this stuff out. It feels like the ISP game is really geared towards big companies expanding or buying existing ISP's, and is not very encouraging of small local competition.
Of course, that's where WISPs come in, because it is so much simpler to not have to deal with all the trenching and agreements, and the equipment is getting better and better and cheaper and cheaper. (loving ubiquity airfiber products for example).
Anyway, more power to you, I hope things work out. I would like to see more posts about the business/political/technical struggles you run into.
I whish I knew more about the laws on common access to equipment/lines. For example, the only cable provider locally is Suddenlink, AT&T Uverse is still on copper phone lines, and there is only one local ISP bringing fiber to residential customers. The local ISP got a huge bunch of grants for the fiber rollout (of course someone embezzled a bunch of it (few mil) and they had to get bought out to get the new set of grants) but all I want to know is: if the government payed for the fiber rollout, shouldn't I be able to use it as well for a nominal fee of some sort?
What about ownership and fees on the last mile? How does all that work? I just wish there was a clear and simple place to find this stuff out. It feels like the ISP game is really geared towards big companies expanding or buying existing ISP's, and is not very encouraging of small local competition.
Of course, that's where WISPs come in, because it is so much simpler to not have to deal with all the trenching and agreements, and the equipment is getting better and better and cheaper and cheaper. (loving ubiquity airfiber products for example).
Anyway, more power to you, I hope things work out. I would like to see more posts about the business/political/technical struggles you run into.
I wonder when point-to-point radio network could be more cost effective solution than fiber network in someone else’s ground or trench. I assume that point-to-point radio has more expensive equipment, is that true? Probably it all depends on a scale. Curious how would curves look like - maybe population density on x and approximated cost of infrastructure on y.
Wireless point-to-point links are comparatively cheap, but require clear line of sight, uncontested spectrum and power. Sometimes you have none of those. Wireless also scales poorly past 1 Gbps.
Fiber is more expensive up front, but cheaper in the long run if your bandwidth requirements grow. Fiber also does not become obsolete, like wireless equipment.
Fiber is more expensive up front, but cheaper in the long run if your bandwidth requirements grow. Fiber also does not become obsolete, like wireless equipment.
(all below is in an American/US context)
Read about ILECs/CLECs. Understand what the cable offerings' contention model is.
Do the math.
Understand the support burdens of NAT-on-NAT.
Run the numbers of RF contention, and shared-medium bandwidth management.
Do the math.
Probably the best advice I could give here: It's very easy to do a mediocre job of all this. It's a fair bit harder to do a good job of it. And think of how much you dislike the mediocre ISPs.
Read about ILECs/CLECs. Understand what the cable offerings' contention model is.
Do the math.
Understand the support burdens of NAT-on-NAT.
Run the numbers of RF contention, and shared-medium bandwidth management.
Do the math.
Probably the best advice I could give here: It's very easy to do a mediocre job of all this. It's a fair bit harder to do a good job of it. And think of how much you dislike the mediocre ISPs.
I've been building WISPs in the US (including Texas) for about 14 years now. Happy to chat, email in my profile.
Another presentation at UKNOF 36 entitled "Starting a Wireless ISP by accident": https://youtu.be/-hF_UccnHxk?list=PLjzK5ZtLlc91r5YZVNiebhcq0...
this is awesome. I was in a similar situation several years ago using crap like Clearwire, Hughes Net, and some mifi hacks to get internet to a dislocated house. I ended up researching and finding out about ubiquiti radios and found a way to hook up comcast to a garage by the road. the distance is about quarter of a mile from the modem location to the house. I went from having the most miserable internet experience (SATELLITE INTERNET SUCKS), to practically being in internet heaven.
Here's the kicker. Comcast was willing to drop a line to the house for around $15,000. No thanks. With two m5 ubiquiti radios and 100ft of ethernet cable, I solved the problem spending less than $300. It would have been nice to have a direct connection, but I'd rather save some money.
The radio's have been up for 5+ years now. I had a problem with one of the POE adaptors once, but other than that I think these devices are cost-effective and robust. I believe these radios can help bring about internet connectivity for a large portion of the unserved population. Easy to setup. Cheap. Robust.
Here's the kicker. Comcast was willing to drop a line to the house for around $15,000. No thanks. With two m5 ubiquiti radios and 100ft of ethernet cable, I solved the problem spending less than $300. It would have been nice to have a direct connection, but I'd rather save some money.
The radio's have been up for 5+ years now. I had a problem with one of the POE adaptors once, but other than that I think these devices are cost-effective and robust. I believe these radios can help bring about internet connectivity for a large portion of the unserved population. Easy to setup. Cheap. Robust.
> Investment would be nice, but I have had zero luck with that to-date.
Hey Chris!
I'm interested in what sort of investment and the terms you'd be comfortable with (I've helped built and operate datacenters, a web hosting company, etc; I'm familiar with the capex and returns involved). Mind if I reach out via email?
Hey Chris!
I'm interested in what sort of investment and the terms you'd be comfortable with (I've helped built and operate datacenters, a web hosting company, etc; I'm familiar with the capex and returns involved). Mind if I reach out via email?
Of course; chris (at) nepafiber.com
Chris - great to hear your story. I'm the CEO of Pilot, we service a few hundred buildings in New York City. While the logistics and techniques may differ, building an ISP is, well, building an ISP.
Would be great to talk shop sometime.
Would be great to talk shop sometime.
I've been hearding that Pilot is starting to do buildings in Center City Philadelphia. Good to see you on HN, that certainly raises your cred.
My CEO would like to chat with you guys soon.
My CEO would like to chat with you guys soon.
That'd be great! Shoot me over an email: chris (at) nepafiber.com
Sorry in advance if I don't get back to you right away.
Sorry in advance if I don't get back to you right away.
If you haven't heard of it, you may be interested in WISPA [0] as it sounds like you're still doing some fixed wireless. There should be a PA state-level group/mailing list that you might be able to reach out to.
You should be able to find a list of other, similar ISPs in PA on their web site that you may be able to collaborate with.
Also, their bi-annual conference is in a few weeks in Memphis, TN. There will be "fiber workshops" that you may or may not find interesting.
[0]: http://www.wispa.org
You should be able to find a list of other, similar ISPs in PA on their web site that you may be able to collaborate with.
Also, their bi-annual conference is in a few weeks in Memphis, TN. There will be "fiber workshops" that you may or may not find interesting.
[0]: http://www.wispa.org
That's my hometown. Im 10-30 min away from about any part. Might try to go if it's open to anyone. Trying to collect info to help new ISP's get started.
I hadn't planned on going myself (I prefer the one in Vegas!) but there will certainly be plenty of ISP folks there for you to chat with.
I mean, can just anyone go there? And is it free cuz Im kind of broke?
You don't have to be a member of WISPA to attend, AFAIK. It is not free, however; cf. http://www.wispa.org/Events/WISPAmerica
Past the booking date and that's quite an entry fee. Oh well. Might hit it next time, though. Thanks for tipoff about it! :)
I think I asked you for the write-up before. Thanks for doing it. I'd like more details, though, on the costs of equipment, installation, and so on for wireless and fiber. I mean, outside of running fiber to houses or businesses, what's the cost of the switches, any boxes protecting them, and the bandwidth? And how much bandwidth can you get away with offering them or reserving for yourself?
The answer to some of those questions are complex. I was trying to keep the length of this post to a minimum, so I decided to keep that information out, but I'll try to provide some of the details you asked about.
The cost of equipment depends on the installation. If we use Ubiquiti sectors and CPE's, the sectors run about $500/each and each CPE is around $130. I don't think we have a sector with more than 6 clients on it and I don't think I'd be willing to put more than 10 on one, depending on the channel and signal, each antenna has around 250 Mbps of capacity. So if we have 6 clients on a sector, that's about $213/subscriber.
Most of our business customers are on PtP radios. I use Mimosa B5-lite's for those, which are $300/set.
Some of our customers are in properties in which we already have equipment, so the cost there depends on how many customers are in the same building. I'll typically run the CPE to a switch and then branch out to customers from there. Let's say that the property has 3 subscribers and I used a Mimosa B5-lite to the property. Add $300 for a switch and we're at around $200/subscriber.
I'd probably say that a client, on average, costs around $250 to get setup.
Every antenna and switch in our PoP is connected to a UPS system; I have one large 3U UPS and a few stand-alone desktop UPS's. They probably total around $2,500. We use a small fraction of our available bandwidth; that's not a concern. What is a concern is the spectrum available per antenna, which is why I said I would not put more than 10 clients on an antenna.
I'll write another post about the cost of fiber.
The cost of equipment depends on the installation. If we use Ubiquiti sectors and CPE's, the sectors run about $500/each and each CPE is around $130. I don't think we have a sector with more than 6 clients on it and I don't think I'd be willing to put more than 10 on one, depending on the channel and signal, each antenna has around 250 Mbps of capacity. So if we have 6 clients on a sector, that's about $213/subscriber.
Most of our business customers are on PtP radios. I use Mimosa B5-lite's for those, which are $300/set.
Some of our customers are in properties in which we already have equipment, so the cost there depends on how many customers are in the same building. I'll typically run the CPE to a switch and then branch out to customers from there. Let's say that the property has 3 subscribers and I used a Mimosa B5-lite to the property. Add $300 for a switch and we're at around $200/subscriber.
I'd probably say that a client, on average, costs around $250 to get setup.
Every antenna and switch in our PoP is connected to a UPS system; I have one large 3U UPS and a few stand-alone desktop UPS's. They probably total around $2,500. We use a small fraction of our available bandwidth; that's not a concern. What is a concern is the spectrum available per antenna, which is why I said I would not put more than 10 clients on an antenna.
I'll write another post about the cost of fiber.
That's great info that I'm saving. It's way cheaper than I thought it would be since the cost of everything has gone down much vs back when I started networking. I was also considering possibility of reaching out to people in Shenzhen finding and testing gear from various suppliers to see if that cost could be reduced. Curious if you've considered any kind of mesh networking that relays from one location to another?
I look forward to the fiber write-up.
I look forward to the fiber write-up.
Keep in mind that in 3-5 years or so (as bandwidth demand per-user doubles), you get to rip all that gear out and replace it with new gear that can provide more bandwidth.
What kind of latency does meshing add? I'd reckon a few more ms per hop?
For reference, in the unlicensed Ubiquiti gear that I've used, the point-to-multipoint latency is variable with the signal quality of the connected stations.
If all client stations have good, clear signals, the latency is around 1-2ms. But that can spike quickly if you're dealing with interference or other causes of low SNR.
If all client stations have good, clear signals, the latency is around 1-2ms. But that can spike quickly if you're dealing with interference or other causes of low SNR.
That's something I'm curious about, too. I haven't done much research on it except to say I've used it a few times with no noticeable impact and some others are doing mesh networks. I was just curious if OP had done any research or analysis of that option.
This was great to read. I started 32Waves in April of 2016 under the same premises you did here. We're now 176 customers strong and are getting into fiber as well (just lit up a 10gig local ring that runs to several tall structures). Today we're 95% wireless, but we've had amazing success with it. No issues at all with stability and reliability, we spent a metric shit ton of time engineering everything.
Our city is looking at doing a community fiber build, where the city owns the strands and the use is open access. I think it's the future and we're pushing for it as a company as well.
I've started another company recently called OpenOptic where we are working on SDN solutions that automate operations with virtualization and user control portals.
It's been a freakin fantastically fun first year so far.
Love to swap stories sometime.
Our city is looking at doing a community fiber build, where the city owns the strands and the use is open access. I think it's the future and we're pushing for it as a company as well.
I've started another company recently called OpenOptic where we are working on SDN solutions that automate operations with virtualization and user control portals.
It's been a freakin fantastically fun first year so far.
Love to swap stories sometime.
Trying to figure out how 50 customers make it profitable enough for him to be laying fiber etc. Even if it costs $100/mo that's only $5000, half of which just goes for his office rent.
Also WebPass in SF is very similar those guys are awesome, can't live without them, one of the reasons I always choose soma skyscrapers to live in.
Also WebPass in SF is very similar those guys are awesome, can't live without them, one of the reasons I always choose soma skyscrapers to live in.
The majority of our customers are businesses paying >$100/m. Aside from that, I still work full-time as a Systems Engineer at USPS +$3,400/m after taxes.
> I still work full-time as a Systems Engineer
I hope then that you're not the only person working at NEPA Fiber. How many employees do you have?
I hope then that you're not the only person working at NEPA Fiber. How many employees do you have?
He probably is running it all himself, it's only a 50 customer business. Combined with a day job that allows him to take the occasional personal call, it's very doable.
If you look on the DSLReports Wireless ISP forum, most of the smaller WISPs are doing exactly this, and eventually they may bring a person on to answer the phone, and an installer, but until you get a bit over 200 subs you can do this.
If you look on the DSLReports Wireless ISP forum, most of the smaller WISPs are doing exactly this, and eventually they may bring a person on to answer the phone, and an installer, but until you get a bit over 200 subs you can do this.
I guess so, just seemed a bit odd at face value. All this WISP stuff seems really interesting to me. I don't think I have what it takes to start one myself, but I wish I could find someone starting out in the business who needs a partner.
Chris, after starting an ISP[1] in the UK in 2014 I certainly feel you're pain around the difficulties in starting an ISP! Myself and my co-founders have often looked at the comparable easy of growth software based startups and wondered what on earth we were doing, but we soldier on and our looking at our reliable, contracted MRR now is certainly starting to make up for it :)
Good luck, I look forward to keeping up with how things are going your side of the Atlantic.
[1] https://telcom.io/
Good luck, I look forward to keeping up with how things are going your side of the Atlantic.
[1] https://telcom.io/
> Myself and my co-founders have often looked at the comparable easy growth software based startups and wondered what on earth we were doing ...
I've got an answer for that - "You're doing God's work". Sounds silly, sure, but you've decided to tackle a really hard problem that's ultimately much riskier than that which you've mentioned (higher capital, lower profit margins, smaller customer base). But speaking as someone with a lot of experience in communities poorly served for Internet service, it's a life changing thing when broadband suddenly becomes available where it once wasn't (or was so poorly served by one of "The Bigs" that it might as well not have been).
I'm only half-joking about God's work. I wouldn't have the stones to consider trying to solve the problem you and Chris are tackling and I greatly respect you for taking that risk. Many thanks, even though I'm not in your country and can't enjoy the service you provide!
I've got an answer for that - "You're doing God's work". Sounds silly, sure, but you've decided to tackle a really hard problem that's ultimately much riskier than that which you've mentioned (higher capital, lower profit margins, smaller customer base). But speaking as someone with a lot of experience in communities poorly served for Internet service, it's a life changing thing when broadband suddenly becomes available where it once wasn't (or was so poorly served by one of "The Bigs" that it might as well not have been).
I'm only half-joking about God's work. I wouldn't have the stones to consider trying to solve the problem you and Chris are tackling and I greatly respect you for taking that risk. Many thanks, even though I'm not in your country and can't enjoy the service you provide!
Your prices are amazing. I might leave a post-it with your web address on our office landlord's desk.
I'm fascinated by the economics of running fiber. A neighboring company from a few counties away has brought in a fiber backbone into the county. This was run via aerial fiber, they were pretty clear their objective was to hook up schools, government and businesses once in the county. After that then focus on residential. Our county is pretty rural, the largest town 5k people. The backbone ran past my parents house which is very rural farming area. I kept telling them they wouldn't be able to get internet by tapping into a point to point run. Apparently I was wrong the company sold them 100mb down for $80/m with a $700 install fee. They are tapping thier backbone at each house and coming off of the tap with a smaller fiber directly into the house.
My condolences running a wisp as well. I know some of the guys at our local wisp, they sell a lot of subscriptions but they really don't have the capacity to keep selling service. Really poor quality connections along with things the OP mentioned like interference and equipment issues.
My condolences running a wisp as well. I know some of the guys at our local wisp, they sell a lot of subscriptions but they really don't have the capacity to keep selling service. Really poor quality connections along with things the OP mentioned like interference and equipment issues.
They probably ran a 48-144 strand backbone so they'd be able to splice off a strand here and there. Another possibility is they reserved 5-10 strands for GPON; which basically allows you to use a single strand for up to 32 or 64 subscribers.
That is interesting, the later sounds like what they are doing, but not for sure. Either way it is working really well.
Wow...this brings back some memories. Back in the 90s, 2400 baud dial-up was still a normal thing, I was employee #1 at a bootstrapped ISP. It was started in some spare office space in the back of a warehouse by some college friends and financed mostly on credit cards.
The equipment was mostly home rolled, real MacGyver type stuff, but we targeted above average industry standards for service. With a little advertising, amazingly we grew very fast and hit several thousand subscribers in just a few months. We eventually outgrew our space and moved into another one just next door to our upstream provider who ran a cable through their ceiling/our floor so we could have have service.
I worked there as we slowly upgraded equipment until we were 56k on all lines and then we started hitting major capital equipment cost issues. Getting those phone company to get us the proper lines for 56k was hard enough (their standard of service was "if you can hear a voice it's fine"), but getting equipment that could support ISDN -- the next big thing -- was a huge transition point.
So we sunk virtually all of our money into the equipment, and then the industry rapidly moved on to rolling out DSL. Turns out customer weren't generally willing to pay the extra fees involved in having ISDN service and were willing to simply wait for the DSL rollout.
Effectively locked out of the new Internet access technology (you could only really offer it at that time if you owned the last mile lines and we could never get close to affording that) the company was sold and we all moved on -- a story that was repeated thousands of times across the country.
To put into perspective the magnitude of the change in the industry, it would be like somebody starting an ISP today, being able to buy and run fiber to the last mile and all that, and then next year a blimp fleet and satellite constellation started offering higher speed, more reliable internet for about the same price you were barely scraping by on. And now to compete you need to launch your own fleets and constellations.
The equipment was mostly home rolled, real MacGyver type stuff, but we targeted above average industry standards for service. With a little advertising, amazingly we grew very fast and hit several thousand subscribers in just a few months. We eventually outgrew our space and moved into another one just next door to our upstream provider who ran a cable through their ceiling/our floor so we could have have service.
I worked there as we slowly upgraded equipment until we were 56k on all lines and then we started hitting major capital equipment cost issues. Getting those phone company to get us the proper lines for 56k was hard enough (their standard of service was "if you can hear a voice it's fine"), but getting equipment that could support ISDN -- the next big thing -- was a huge transition point.
So we sunk virtually all of our money into the equipment, and then the industry rapidly moved on to rolling out DSL. Turns out customer weren't generally willing to pay the extra fees involved in having ISDN service and were willing to simply wait for the DSL rollout.
Effectively locked out of the new Internet access technology (you could only really offer it at that time if you owned the last mile lines and we could never get close to affording that) the company was sold and we all moved on -- a story that was repeated thousands of times across the country.
To put into perspective the magnitude of the change in the industry, it would be like somebody starting an ISP today, being able to buy and run fiber to the last mile and all that, and then next year a blimp fleet and satellite constellation started offering higher speed, more reliable internet for about the same price you were barely scraping by on. And now to compete you need to launch your own fleets and constellations.
This was a super interesting read. The one thing I never fail to get over though is how expensive internet is in the states. In the UK you can get 200/20 for about £58 p/m including TV & Phone. I couldn't imagine paying nearly $100 p/m for 100mbps.
I get symmetric 200/200 for about $70/month in Bangalore, India. It's just a dumb pipe though (no included TV/phone, though I could also go for those).
That's thanks to the BT/OpenReach split in the 90s forcing small ISPs to compete on service & cost (because they all have the same guaranteed cost to access the infrastructure).
Plus y'know you can blow fibre to 90% of the population in a reasonable amount of time.
Plus y'know you can blow fibre to 90% of the population in a reasonable amount of time.
Except it's not - for 200/20 at those prices the GP must be talking about a Virgin Media line, and they run their own cables. The increased competition between providers probably contributes to driving down prices to just above the point of profitability, but its not responsible for the infrastructure in this case.
It's not that the bandwidth is expensive; it's the cost to deploy everything. If there was preexisting conduit/fiber in the ground and all we had to do was plug two cables together, I'd be able to charge $25/m too.
I've never been to the UK, but I believe your population density is a lot higher than that of the US; which is where all of our costs originate (interconnecting the customer to our network).
I've never been to the UK, but I believe your population density is a lot higher than that of the US; which is where all of our costs originate (interconnecting the customer to our network).
I don't think that's the reason really.
The main reason I can see is that Comcast (or cable internet) has no real competition. I don't understand why that is. AT&T, Verizon etc seem to have given up on competing. Verizon stopped their rollout of FTTH a while back nearly entirely, though it has restarted very slowly.
AT&Ts VDSL based UVerse solution is way too sparse, with very long cable runs. Compare this to BT in the UK where most VDSL2 runs they are doing are <500m. They're now looking at GFast to push fibre even closer to customers, getting copper runs down to ~200m.
Ok, so this may be caused by low population density, but I'm not entirely convinced. It must be way cheaper to dig trenches in suburbia USA as many of the places don't even have sidewalks to dig up and then expensively refill, you could just trench along the side of the road.
The main reason I can see is that Comcast (or cable internet) has no real competition. I don't understand why that is. AT&T, Verizon etc seem to have given up on competing. Verizon stopped their rollout of FTTH a while back nearly entirely, though it has restarted very slowly.
AT&Ts VDSL based UVerse solution is way too sparse, with very long cable runs. Compare this to BT in the UK where most VDSL2 runs they are doing are <500m. They're now looking at GFast to push fibre even closer to customers, getting copper runs down to ~200m.
Ok, so this may be caused by low population density, but I'm not entirely convinced. It must be way cheaper to dig trenches in suburbia USA as many of the places don't even have sidewalks to dig up and then expensively refill, you could just trench along the side of the road.
One other reason is that counties have the land rights. They negotiate with the big Telco's like Comcast, Verizon, Cox, Charter, etc and then issue a monopoly or duopoly in the county for a fixed period of time - usually 10 years at once.
The Telco can offer a few things like fiber connectivity between county buildings, tax payments, etc. By doing that, they secure the rights to the area. Given the federal system of the United States, this is a difficult thing to stop.
This makes true market-based competition within most areas very difficult to achieve.
The Telco can offer a few things like fiber connectivity between county buildings, tax payments, etc. By doing that, they secure the rights to the area. Given the federal system of the United States, this is a difficult thing to stop.
This makes true market-based competition within most areas very difficult to achieve.
> One other reason is that counties have the land rights. They negotiate with the big Telco's like Comcast, Verizon, Cox, Charter, etc and then issue a monopoly or duopoly in the county for a fixed period of time - usually 10 years at once.
They do not do that. It's illegal under federal law. Pole and conduit owners are required to rent out access at non-discriminatory rates: https://www.law.cornell.edu/uscode/text/47/224.
The reason companies don't overbuild is because it's expensive and there isn't any return. FiOS came to my building in Baltimore. I was the only person on my floor to switch away from Comcast. Even these days, people choose their broadband provider based primarily on the TV package.
They do not do that. It's illegal under federal law. Pole and conduit owners are required to rent out access at non-discriminatory rates: https://www.law.cornell.edu/uscode/text/47/224.
The reason companies don't overbuild is because it's expensive and there isn't any return. FiOS came to my building in Baltimore. I was the only person on my floor to switch away from Comcast. Even these days, people choose their broadband provider based primarily on the TV package.
> They do not do that. It's illegal under federal law.
But they most certainly can impose enough bureaucracy and other hurdles to make renting factually impossible. Or simply both the incumbent provider and the county employ just a single FTE (or less!) to handle permits, and one can't do anything about it. All while following the letter of the law, because there's nothing in the law that says "county has X days to deal with the permit else it is being automatically granted".
But they most certainly can impose enough bureaucracy and other hurdles to make renting factually impossible. Or simply both the incumbent provider and the county employ just a single FTE (or less!) to handle permits, and one can't do anything about it. All while following the letter of the law, because there's nothing in the law that says "county has X days to deal with the permit else it is being automatically granted".
That feels easily litigated to me. The law probably makes some statement like 'will make available'. You then go to court showing that the opportunity to rent isn't available even though there technically is a service.
> easily litigated
Nothing in the US is "easily" litigated, especially not when you're up against a multimedia conglomerate and a portion of the government itself.
Nothing in the US is "easily" litigated, especially not when you're up against a multimedia conglomerate and a portion of the government itself.
"easily"
"litigated"
Those two words do not go together.
Those two words do not go together.
> They do not do that. It's illegal under federal law.
They definitely do. Our city has a franchise agreement with the incumbent cable provider; they're the only company allowed to provide cable television service. ~10 years ago a company called NuNet tried to come in and run fiber in a neighboring city, Hazleton, and both they and the city were sued by the incumbent for breaching the franchise.
They definitely do. Our city has a franchise agreement with the incumbent cable provider; they're the only company allowed to provide cable television service. ~10 years ago a company called NuNet tried to come in and run fiber in a neighboring city, Hazleton, and both they and the city were sued by the incumbent for breaching the franchise.
The 1992 Cable Act made exclusive franchises illegal, but does not retroactively apply to pre-1992 contracts. Almost all of those contracts have since expired and been renewed as non-exclusive contracts under the 1992 Act. Hazelton, for example, granted an exclusive franchise before 1992, which expired in 2005: http://articles.mcall.com/2004-03-25/business/3517245_1_fibe.... The suit in 2004 was over the last year of exclusivity.
I imagine there are some 30-year exclusive franchises granted in 1991 that are still in effect. I don't actually know of any.
I imagine there are some 30-year exclusive franchises granted in 1991 that are still in effect. I don't actually know of any.
Interesting; that clears things up. Thanks!
I don't think that's true with the massive switch to cord cutting - cable TV revenues are collapsing.
Was FiOS much cheaper/better than Comcast (apart from the underlying tech)?
Was FiOS much cheaper/better than Comcast (apart from the underlying tech)?
We do in theory have competition for DSL service but the telcos just conspired to let their cable plant rot to make that undesirable.
But why though? It seems really silly. Even in a hyper competitive market like the UK, the incumbent telco still makes a fortune off home broadband service.
In the US AT&T and Verizon seem to have given up and want to just do cellular to make money and let their copper plant rot. They are gifting the entire market to Comcast et al.
In the US AT&T and Verizon seem to have given up and want to just do cellular to make money and let their copper plant rot. They are gifting the entire market to Comcast et al.
No, we don't. The Brand X decision killed unbundling for DSL.
Verizon stopped competing because they are transitioning from a wired and wireless company to solely wireless. They have no interest in holding onto their old POTS, and FCC required them to service it.
Cost of the Telcom union, employers and pension adds up, not to mention infrastructure.
They avoid all this by wireless voice, cellular communications.
Still under the FCC though.
Cost of the Telcom union, employers and pension adds up, not to mention infrastructure.
They avoid all this by wireless voice, cellular communications.
Still under the FCC though.
They could just try not having jacked prices and offering good packages. Their customer-base would skyrocket and revenue would more than cover costs.
It's interesting that this is always the american response whenever the broadband comparisons are made. I don't mean this in a negative way, but it's a very persistent argument and I wonder if there's some external factor that pushes it (telcos/media/other).
I think this is definitely a factor - but it's vastly overstated. The USA is more urbanised than many European countries at 80%, so there's definitely quick wins for the vast majority of the population.
The big difference I see is the way these services are regulated. In the UK for example, the physical infrastructure associated with the last mile is owned by a company that just manages this, with regulated prices linked to asset investment required. Services are then resold to Internet Service Providers who provide the IP services and compete on price and other factors.
The capital costs of providing the infrastructure are restrictive and the advantage of competition does not outweigh the cost of having multiple last-mile infra.
Note - I've used some simplifications on the ownership and extents of infraco/telco ownership, but the concept holds.
I think this is definitely a factor - but it's vastly overstated. The USA is more urbanised than many European countries at 80%, so there's definitely quick wins for the vast majority of the population.
The big difference I see is the way these services are regulated. In the UK for example, the physical infrastructure associated with the last mile is owned by a company that just manages this, with regulated prices linked to asset investment required. Services are then resold to Internet Service Providers who provide the IP services and compete on price and other factors.
The capital costs of providing the infrastructure are restrictive and the advantage of competition does not outweigh the cost of having multiple last-mile infra.
Note - I've used some simplifications on the ownership and extents of infraco/telco ownership, but the concept holds.
The effect of density is not overstated. States' average bandwidth in the Akamai rankings line up closely with population density. I live in Maryland (one of the most densely populated states, though half the density of the U.K.) and most of the urban areas have fiber available (the exception being Baltimore city, for political reasons).
Even if you live in an urban area, the prevalence of rural areas (and poor urban areas), has an impact on you, because the US supports rural telecommunications through various internal cross-subsidies instead of direct support.
The regulatory regime plays a role too, but it's more complicated than your simplistic presentation. In the U.K., the last mile infrastructure is owned by a single company (BT Openreach), but equally importantly, the government ensures it is a quite profitable company. Much more so than American utilities. That was a conscious part of the BT privatization: designing a rate structure that would ensure prices high enough to create adequate incentives for investment.
Even if you live in an urban area, the prevalence of rural areas (and poor urban areas), has an impact on you, because the US supports rural telecommunications through various internal cross-subsidies instead of direct support.
The regulatory regime plays a role too, but it's more complicated than your simplistic presentation. In the U.K., the last mile infrastructure is owned by a single company (BT Openreach), but equally importantly, the government ensures it is a quite profitable company. Much more so than American utilities. That was a conscious part of the BT privatization: designing a rate structure that would ensure prices high enough to create adequate incentives for investment.
We do have some examples of this kind of structure in the US - for example, in Manhattan, NYC, a company called "Empire City Subway" has maintained general telecommunications conduit an a license from the state for over a hundred years. http://www.empirecitysubway.com/
Unfortunately, this kind of thing is rare here.
Unfortunately, this kind of thing is rare here.
Their rates are surprisingly reasonable. For example, I mentioned that leasing conduit from the utility here is $3/ft/yr; those rates are as low as $0.6909/ft/yr, significantly cheaper.
Same cost in the US but you get much less speed.
You can get cheap internet in the states in many urban areas.
Depends where you are. Before Brexit, 58 pounds was about $90. I pay about $110 for 150/150 symmetric fiber + TV + phone in Maryland.
The much slower upload speed means your download is bottlenecked (tcp ACKs). A lot of providers offer such asymmetry because it works well for video (udp) but otherwise it's not the great deal you might think it is.
As another point of reference, I live in semi-rural Canada about an hour from a major city and have 150/150 fibre for CDN$85/month.
As another point of reference, I live in semi-rural Canada about an hour from a major city and have 150/150 fibre for CDN$85/month.
[deleted]
> I was starting an ISP, perhaps the most capital intense business on the planet
Semiconductor manufacturing is literally a million times more expensive (~3B investment in a new fab). The sentence I'm quoting doesn't appear to be intended as a joke.
Semiconductor manufacturing is literally a million times more expensive (~3B investment in a new fab). The sentence I'm quoting doesn't appear to be intended as a joke.
"perhaps" intended to mean, among the top*
But that's still debatable depending on what level of scale you're talking.
Deploying fiber throughout the entire United States, to every single household, would be near or exceed $1 Trillion. We have semiconductors in likely 99.99% of households now; at what cost? (I don't know, but I'd assume it's probably close.)
But that's still debatable depending on what level of scale you're talking.
Deploying fiber throughout the entire United States, to every single household, would be near or exceed $1 Trillion. We have semiconductors in likely 99.99% of households now; at what cost? (I don't know, but I'd assume it's probably close.)
In semiconductors, you need to invest billions before you make and sell your first chip. With fiber, you can start with a dozen customers. There is no comparison.
Gotta disagree here, especially with the statement that there's "no comparison". Your comment is making it sound like the author is both over-claiming the difficulty in the business he's setup and over-stating the capital required. I read it as almost the opposite -- I can't figure out how he could have possibly gotten to where he is from the point he started.
Yes, you're talking apples and oranges, but both industries have a few of the same problems. On the "similarities side", deploying fiber at the scope and reach of deployed silicon would be comparatively expensive. On the growth similarities side, a company wishing to enter the "chip production" business could start comparatively small on something like a ARM or FPGA board targeted at a niche market like Bitcoin mining or specialty network equipment using off the shelf chips, custom software and custom printed boards in low quantities ranging from relatively cheap to somewhat expensive, but some would land in the range of "small, regional, ISP", feeding that money back into R&D toward designing a chip to be fabricated by an existing manufacturer (akin to "the national roll out of fiber). There are parts of that business that exist between "soldering a board of off-the-shelf parts" and "Intel" (ARM fits somewhere in that spectrum).
On the oranges vs. apples side, silicon has little/no geographical component outside of regulatory export restrictions. Short of the non-"first chip" related examples I listed above, there are comparatively fewer ways to start small and grow from profits. But expanding from a small regional fiber business to a national fiber roll-out, at this point, hasn't really been done successfully[0]. It's also been tried by several big players and ended in outcomes ranging from disaster (ref. any number of businesses from 1998-2002) to fade-out/quiet abandonment/refocus (ref. Google Fiber). It will take an incredibly long time to do it by feeding profits back into the business.
Also on the "easier to do silicon side" of things is that there are still boutique and niche market opportunities where, yes, a large capital outlay will still be required, but a healthy profit margin can be found due to being the only provider of a market need where the players are willing to spend the money. No such market exists in broadband. Even in locations where service is limited to dial-up, there's a pretty low upper limit on what a business or consumer will pay for that service whereas an FPGA board designed in such a way as to allow it to eek out 25% or greater efficiency doing something comparatively unimportant like mining Bitcoin would have a small, but healthy market with a higher upper limit
Then there's the regulatory overhead which is substantially costlier on the fiber side. On the "chip" side, if you're the actual manufacturer, you have environmental concerns around sourcing materials and disposal, which is no small cost, but see what it takes to just run an underground wire between your home and your neighbors, legally without crossing a street and you'll discover how much harder it is to roll fiber out in a city. Couple that with municipalities/states where large players (for all intents and purposes) own state and local governments and make them very interested in causing grief for a smaller player that could rob them of campaign contributions from the local mono-duopoly.
The worst part, though, is that there's a hard, upper limit on what consumers will pay for broadband even when the only alternative is dial-up. Businesses who require fiber-quality service simply won't locate themselves in places where adequate services aren't offered and consumers won't pay more than about three times the national average (and most will set that limit a lot lower due to lower incomes). It's a commodity service that an entrenched incumbent has every motivation to step in and suffocate a promising start-up early rather than see them succeed enough to cause prices to erode further. Similar problems exist on the "chip" side, but there are opportunities to target boutique/niche markets (bitcoin mining, again?) where the purchasers have specific needs not being met by existing solutions and would be willing to pay substantial sums for something that fits their needs (i.e. anywhere you can substantially reduce compute time while reducing consumption will have a market with players willing to spend comparatively larger amounts of money).
[0] Maybe it has, but I can't find an example of a micro-regional ISP expanding nationally and all of the large players I'm aware of started as large players in another business that they still serve -- DSL providers mostly were once local telcos, Cable Internet -- cable TV, "enterprise" providers/carriers like Level 3 are really a combination of several local telcos, long distance providers and other telecom service providers that leveraged/extended their existing networks to carry data and merged with their competitors to expand nationally. He mentions the SONET network issues and that just reminded me of Global Crossing's old SONET network (now owned by Level 3).
EDIT: Formatting and removal of a sentence I rewrote but failed to delete.
Yes, you're talking apples and oranges, but both industries have a few of the same problems. On the "similarities side", deploying fiber at the scope and reach of deployed silicon would be comparatively expensive. On the growth similarities side, a company wishing to enter the "chip production" business could start comparatively small on something like a ARM or FPGA board targeted at a niche market like Bitcoin mining or specialty network equipment using off the shelf chips, custom software and custom printed boards in low quantities ranging from relatively cheap to somewhat expensive, but some would land in the range of "small, regional, ISP", feeding that money back into R&D toward designing a chip to be fabricated by an existing manufacturer (akin to "the national roll out of fiber). There are parts of that business that exist between "soldering a board of off-the-shelf parts" and "Intel" (ARM fits somewhere in that spectrum).
On the oranges vs. apples side, silicon has little/no geographical component outside of regulatory export restrictions. Short of the non-"first chip" related examples I listed above, there are comparatively fewer ways to start small and grow from profits. But expanding from a small regional fiber business to a national fiber roll-out, at this point, hasn't really been done successfully[0]. It's also been tried by several big players and ended in outcomes ranging from disaster (ref. any number of businesses from 1998-2002) to fade-out/quiet abandonment/refocus (ref. Google Fiber). It will take an incredibly long time to do it by feeding profits back into the business.
Also on the "easier to do silicon side" of things is that there are still boutique and niche market opportunities where, yes, a large capital outlay will still be required, but a healthy profit margin can be found due to being the only provider of a market need where the players are willing to spend the money. No such market exists in broadband. Even in locations where service is limited to dial-up, there's a pretty low upper limit on what a business or consumer will pay for that service whereas an FPGA board designed in such a way as to allow it to eek out 25% or greater efficiency doing something comparatively unimportant like mining Bitcoin would have a small, but healthy market with a higher upper limit
Then there's the regulatory overhead which is substantially costlier on the fiber side. On the "chip" side, if you're the actual manufacturer, you have environmental concerns around sourcing materials and disposal, which is no small cost, but see what it takes to just run an underground wire between your home and your neighbors, legally without crossing a street and you'll discover how much harder it is to roll fiber out in a city. Couple that with municipalities/states where large players (for all intents and purposes) own state and local governments and make them very interested in causing grief for a smaller player that could rob them of campaign contributions from the local mono-duopoly.
The worst part, though, is that there's a hard, upper limit on what consumers will pay for broadband even when the only alternative is dial-up. Businesses who require fiber-quality service simply won't locate themselves in places where adequate services aren't offered and consumers won't pay more than about three times the national average (and most will set that limit a lot lower due to lower incomes). It's a commodity service that an entrenched incumbent has every motivation to step in and suffocate a promising start-up early rather than see them succeed enough to cause prices to erode further. Similar problems exist on the "chip" side, but there are opportunities to target boutique/niche markets (bitcoin mining, again?) where the purchasers have specific needs not being met by existing solutions and would be willing to pay substantial sums for something that fits their needs (i.e. anywhere you can substantially reduce compute time while reducing consumption will have a market with players willing to spend comparatively larger amounts of money).
[0] Maybe it has, but I can't find an example of a micro-regional ISP expanding nationally and all of the large players I'm aware of started as large players in another business that they still serve -- DSL providers mostly were once local telcos, Cable Internet -- cable TV, "enterprise" providers/carriers like Level 3 are really a combination of several local telcos, long distance providers and other telecom service providers that leveraged/extended their existing networks to carry data and merged with their competitors to expand nationally. He mentions the SONET network issues and that just reminded me of Global Crossing's old SONET network (now owned by Level 3).
EDIT: Formatting and removal of a sentence I rewrote but failed to delete.
Having grown up in Susquehanna county, it's really cool to see Hacker News content from the area!
Nice work.
It's pretty incredible the kind of issues you see when you're running a service your customers depend on - especially if that's potentially latency sensitive. I've seen packets between Chicago and rural Ohio getting routed through both San Jose and Florida. Random down links between major backbone providers and other crazy messes.
It's pretty incredible the kind of issues you see when you're running a service your customers depend on - especially if that's potentially latency sensitive. I've seen packets between Chicago and rural Ohio getting routed through both San Jose and Florida. Random down links between major backbone providers and other crazy messes.
I've been contemplating setting up a small [NW]ISP for our neighborhood (rural, wooded, private roads) but LOS issues limit my wireless options. One thought was to install a few airfibers and "demux" them with G.fast to each CPE via its service drop interconnect in the pedestal. This has the benefit of not having to trench the last 100 yards from the street to the residence. BUT I can find scant information on what authority one needs to do so. Do I need to be a CLEC/BIAS? Or is the local cable plant completely hands-off? Any pointers would be immensely appreciated (and thanks, Chris, for a great article. I look forward to reading more!)
The Brand X decision killed local loop unbundling in the US. So unless you own the copper plant this is a no-go. If you are outside the US, even then there are problems as access to subloops can be hard to get.
Yep, this would be in the US. I had hope that there would be, at a minimum, some exclusion for infrastructure on private property, but it appears that isn't the case. Thank you very much for pointing me in the right direction.
I noticed that the "subscribe" button is not leading to an https address. However, the link in the confirmation email does. So I guess, it would be an easy fix for you to just change it to https.
Thanks, fixed. I just got around to setting this up yesterday so the site itself isn't https yet and I just put // instead of the full path for the form POST.
This is a great read. I wish it was cheaper to run fiber.
I've been wondering lately if it's possible to have an Uber-style fiber service where people sign up to serve as nodes and then users pay a nearby node to connect. This would skirt what seems to be the major roadblock to installing fiber -- city regulations -- by having too many people running improvised fiber lines to enforce the rules. Is this just not possible?
You'd need some kind of mesh network, rather than a tree network to get anywhere near enough reliability. I'd also guess that passing through a node might cause some slowdowns that kinda stack up.
There still is a major privacy issue with peeps that are nodes. And maintenance is going to be a bitch.
That said, those are all hypotheticals. Don't let my imagination stop anyone trying this.
That said, those are all hypotheticals. Don't let my imagination stop anyone trying this.
> I'd also guess that passing through a node might cause some slowdowns that kinda stack up.
Not really. Fiber has bandwidth to spare. It might add some latency tho, if there are a lot of hops.
> There still is a major privacy issue with peeps that are nodes.
Encryption is a thing.
> And maintenance is going to be a bitch.
Yes.
Not really. Fiber has bandwidth to spare. It might add some latency tho, if there are a lot of hops.
> There still is a major privacy issue with peeps that are nodes.
Encryption is a thing.
> And maintenance is going to be a bitch.
Yes.
No, it is not possible.
Unless you plan to do this purely on private property, you are going to have to negotiate and pay for access to public right of way.
Also, even if you only did this on private property and did not cross any roads or other public spaces, you'd be SOL if your neighbor wasn't a node and they didn't feel like giving you permission to cross their property.
Unless you plan to do this purely on private property, you are going to have to negotiate and pay for access to public right of way.
Also, even if you only did this on private property and did not cross any roads or other public spaces, you'd be SOL if your neighbor wasn't a node and they didn't feel like giving you permission to cross their property.
Interesting points, but I just wonder if there is sufficient will to stop large numbers of determined "drivers" from ignoring the rules. Uber drivers could be said to be violating rules, but Uber is still successful.
Well, unless you bury your wires, sooner of later somebody is going to find out you did the dirty deed. Then all they have to do is follow the cable to your house and fine you.
I'm glad to no longer be in the ISP business (I did my years in the trenches, '95-02), but I sure enjoy trading "war stories" with other folks who started out their careers in a similar manner.
Makes me wonder just how many people are still using dialup these days, due to a lack of any other options (or lack of desire to change..)
Makes me wonder just how many people are still using dialup these days, due to a lack of any other options (or lack of desire to change..)
Wow. Starting an ISP on credit cards and $2,500 cash. Very resourceful. I guess it also helps also knowing this stuff since you worked at a datacenter. I wonder what kind of regulations there are for something like this being wireless. Sounds like it would be a bureaucratic mess to license frequencies. Really inspiring blog.
Not the OP, but doing something similar. For fixed wireless broadband in rural areas of the US, it's actually not too onerous. The FCC designates certain frequencies as licensed or unlicensed, and the hardware options for running an ISP in the unlicensed bands have increased dramatically in the last 10 years.
The process is basically:
[0]: https://en.wikipedia.org/wiki/Communications_Assistance_for_...
[1]: https://www.fcc.gov/general/form-477-resources-filers
The process is basically:
- The FCC designates certain rules, including power limits, for each band.
- Hardware manufacturers build to the rules, get their gear FCC-labeled.
- WISPs (Wireless ISPs) buy name-brand FCC-labeled gear.
- WISPs maintain CALEA[0] capabilities (so 3-letter agencies can spy on you).
- File form 477 with the FCC [1].
[0]: https://en.wikipedia.org/wiki/Communications_Assistance_for_...
[1]: https://www.fcc.gov/general/form-477-resources-filers
Interesting. I heard of CALEA in the past and wondered how CALEA is implemented.
>The IP-based "soft switches" typically do not contain a built-in CALEA intercept feature
Is there a specification you must follow? Like maybe you had a VOIP company with your own custom protocol or softphone, would you just code up some web interface where they have a login and then type in a number to see if any active calls, then click a "Listen" button then an audio stream starts playing. Or is there a well defined way for them to access the call, audio codec, etc has to be?
>The IP-based "soft switches" typically do not contain a built-in CALEA intercept feature
Is there a specification you must follow? Like maybe you had a VOIP company with your own custom protocol or softphone, would you just code up some web interface where they have a login and then type in a number to see if any active calls, then click a "Listen" button then an audio stream starts playing. Or is there a well defined way for them to access the call, audio codec, etc has to be?
Thanks for sharing your story! I noticed your avatar at the bottom of the post looks like you're in Kalalau valley
That's exactly where it's from! Amazing place.
Chris, sounds like your doing an awesome job! Perhaps, you could let the audience of hacker news know what some of your biggest challenges are. It may spark some ideas for start ups that could help solve those problems =)
Based on your article, it sounds like there's a lot of initial capital investment in the beginning.
Based on your article, it sounds like there's a lot of initial capital investment in the beginning.
Would love to see more details on what actually goes down on the nitty gritty level -- it sounds like the business hemorrhaged money for quite some time and given that you didn't have investment it'd be great to know how you got from there to having the capital on-hand to start running fiber.
Really nice writeup. I'm super curious the economics of how you can offer uncapped 1Gbps for only $130 a month? Don't you need edge routers (Cisco, Juniper), fiber cables, employees, insurance, bandwidth (peering), office space, etc.
If I buy a building that has the fiber installed what would be required to run the internet just for me or my company? How much (and what) in equipment am I looking at? I wouldn't want to be an ISP....just provide internet for my company.
That would depend entirely on who's fiber is installed in your building and where you're located. If it's Fios or Comcast in a major metro market, you could probably get a residential or business line for as low as a few hundred. If it's a major transit provider like Zayo or Level 3 or if you're not in a major market; expect to pay a few thousand dollars.
Any reason why you bought the van instead of leasing (or maybe you meant that)?
Great read! I've always wanted to start one too. How did you go about the addressing? Did you purchase IP blocks from Zayo? If you could go technical it would be great.
If he's using Level 3, he would have to have his own PI-IP block. I'd be interested to hear more about IPv4 and IPv6 in the small ISP world. With 50-100 clients, a single /24 block would serve his current client base.
Why is that? Level3 is one of my upstreams and I've had two /23s from them for a few years. I'm pretty sure they still have some addresses available, although they might be harder to get nowadays.
You can get provider-aggretable (PA) IP addresses from your ISP, but they can only be used with that ISP. You can also get provider-independent (PI) addresses, which you can use and move between ISPs with BGP. If they're using Zayo and Level 3 at the same time, you would need PI-IPs that you can move to the other ISP when one ISP is down.
Or, another option is that they're doing CGNAT on all of their client traffic.
Or, another option is that they're doing CGNAT on all of their client traffic.
> You can get provider-aggretable (PA) IP addresses from your ISP, but they can only be used with that ISP.
Sorry, this is incorrect -- I'm a network engineer and I've been doing exactly this for years. I advertise "Level3 IPs" to other ISPs/peers, I advertise "non-Level3 IPs" to Level3, hell, I even "re-advertise" some of my customers' PI addresses (that they advertise to me) up to Level3.
Actually, I can advertise any prefixes I want via Level3 -- even yours (assuming you had your own).
n.b.: Now, some ISPs may require an LOA (or similar) before they'll accept certain prefixes from you but that's a (easily solved) procedural issue -- not a technical issue.
Sorry, this is incorrect -- I'm a network engineer and I've been doing exactly this for years. I advertise "Level3 IPs" to other ISPs/peers, I advertise "non-Level3 IPs" to Level3, hell, I even "re-advertise" some of my customers' PI addresses (that they advertise to me) up to Level3.
Actually, I can advertise any prefixes I want via Level3 -- even yours (assuming you had your own).
n.b.: Now, some ISPs may require an LOA (or similar) before they'll accept certain prefixes from you but that's a (easily solved) procedural issue -- not a technical issue.
Sure, there's no technical issue since a prefix is a prefix is a prefix. It's all just bits. If you can convince your upstream/peers to accept a route, there's nobody to stop you. I haven't run an ISP, so maybe I haven't had to push hard enough, but my transit providers' willingness to accept someone else's PA block has been nil.
Not true. You do not need PI addresses. You can announce IPs owned by one ISP to another one. Many ISPs will allow you to do this.
I have a /24 from Zayo and a /23 from ARIN for IPv4. For IPv6 I have a /32 IPv6 from ARIN.
Are you going to use NG-PON2?
Living the dream.
Although this stuff can be great fun, you have to stop and ask yourself why the market is "under served" and why there isn't already some dude supplying these customers, driving around the city in his Lexus and waxing his new boat on the weekends.
The unfortunate truth is that there just isn't that great a business there, especially in light of its quite high capital costs and quite significant risks.
It is a business that suits larger players who can spread the fixed costs across many cities. It is a business that suits providers offering a basket of services, not just IP transit (TV, Phone, value-added network services). It is a business that demands either a) high price boutique offerings (e.g. the people you end up buying your upstream connectivity from) or b) rock-bottom/who-cares-about-quality, high-volume products (e.g. Charter/Comcast). A small new player can't compete in either of those markets.
I could go on, but really : would you be super-excited if you had the idea to build better electricity transmission lines to a bunch of folk; or a better sewer system? No. And the reality is that IP connectivity is not much different to these other utilities.