Belkin Explains Why Its Routers Stopped Working(techcrunch.com)
techcrunch.com
Belkin Explains Why Its Routers Stopped Working
http://techcrunch.com/2014/10/08/belkin-explains-why-its-routers-stopped-working/
69 comments
I don't get it... why would a router block access to the entire internet because it can't talk to the manufacturer's servers? Why is it talking to the manufacturer's servers at all?
It didn't shut off all internet access. The DNS resolver on the router checked for connectivity. If it couldn't resolve Belkin's domain, it disabled the DNS service because it assumed the Internet wasn't really connected.
Edit: see the workaround here https://belkininternationalinc.statuspage.io/
Edit: see the workaround here https://belkininternationalinc.statuspage.io/
OK, this is much less scary than I thought reading the article. Probably wouldn't hurt to diversify it a bit and use some major provider like Google or Amazon (or both?) for the check, but at least it doesn't look as bad as the vague description in the article that made me think some vital router functions and not simple DNS check depend on external infrastructure.
Running your own servers for Internet checks is what you're supposed to do -- when you don't, you cause problems for other people (e.g. if Google decided to sell an address block or change its DNS name or if someone decides to deprecate an NTP pool server).
For example, look at the document made by the NTP Pool: http://www.pool.ntp.org/en/vendors.html. Microsoft has time.windows.com, Apple has time.apple.com, and Canonical has ntp.ubuntu.com.
I recall reading a story where someone checked network connectivity by pinging a DNS name with a known but invalid IP address, and relying on the exact error message. When a sysadmin later removed the "invalid" DNS entry, ping would return a different error message, which caused the network connectivity check to fail. People do strange things for connectivity checks -- what Belkin did here seems pretty "by-the-book" by comparison.
For example, look at the document made by the NTP Pool: http://www.pool.ntp.org/en/vendors.html. Microsoft has time.windows.com, Apple has time.apple.com, and Canonical has ntp.ubuntu.com.
I recall reading a story where someone checked network connectivity by pinging a DNS name with a known but invalid IP address, and relying on the exact error message. When a sysadmin later removed the "invalid" DNS entry, ping would return a different error message, which caused the network connectivity check to fail. People do strange things for connectivity checks -- what Belkin did here seems pretty "by-the-book" by comparison.
This is why when I want to implement a negative connectivity check I always make a deal with admins to have a will-never-exist.whatever.com or similar record with known and documented behaviour.
Yes, that's because I've been caught out before :)
Yes, that's because I've been caught out before :)
Amazon/Google might be a little annoyed if every Belkin router on the planet started pinging them.
They probably get much more legit traffic than that. Router doesn't need to do it that frequently, only when it needs to find out if internet connection is alive. Which is usually on boot and maybe on infrequent occasions after that. So they wouldn't even notice.
No. Terrible idea: http://pages.cs.wisc.edu/~plonka/netgear-sntp/
They would love it. Why do you think they provide public DNS in the first place?
Google provide public DNS because they think they can make people's DNS faster and more reliable, which causes web pages to load faster and more reliably, causing Internet users to be happier when using the Internet, and therefore providing more analytics data and ad revenue to Google.
[deleted]
and, they know where people are looking/surfing/...
>some vital router functions and not simple DNS check
How do you expect 99.99% of the world to get to facebook.com without DNS? It's pretty damn vital if you ask me, especially for a consumer grade router.
How do you expect 99.99% of the world to get to facebook.com without DNS? It's pretty damn vital if you ask me, especially for a consumer grade router.
But their DNS doesn't actually depend on that server (at least according to the explanation above). Only the part of the code checking "do we have internet?" depends on it, and only because they use some internal company's address and apparently whoever is responsible for it doesn't know that if this host ever gets down it breaks the internet for all Belkin customers. So it's a simple case of miscommunication and failure to think systemically, not something fundamentally broken.
Do a dig of heartbeat.belkin.com and you get a bunch of 54.0.0.0/8 addresses that Amazon bought from Merck. It's using AWS.
Looks good, but how comes then it still didn't work properly? I see only two possibilities: somebody managed to mess up all the redundant servers at once, by either messing up firewall rules or replicated config or something like that, or somebody messed up heartbeat.belkin.com. DNS entry.
Looks like I was wrong cause I checked too late, Belkin moved to AWS as part of the fix, doh!
https://news.ycombinator.com/item?id=8429789
https://news.ycombinator.com/item?id=8429789
For most people, DNS on the router is vital because they don't know how to get online without it.
[deleted]
It shouldn't be less scary. Any time proprietary software calls home it could receive orders to open a backdoor or anything else. It's a command and control feature disguised as a DNS check, or at the very least it could be easily turned into one at any point without raising suspicion.
I think you're being too paranoid here (I don't say this often :). A ping to known good host is a benign thing. Can it be replaced with malicious thing? I'm sure it can, somebody could gain access to your router and reflash it with malware and you wouldn't even know. This is a real possibility. But it remains so regardless of presence or absence of DNS ping, so DNS ping by itself does not add any specific threat.
Calling it a "DNS ping" is just confusing or wrong. It's a DNS query.
It's dangerous because:
1. It allows them to create a database with the IP address of every internet-connected router.
2. It allows them to control routers via DNS packets, which are frequently permitted even behind firewalls.
3. It providers a cover for traffic to their servers on the DNS port.
This kind of thing is very common. It's also very dangerous and we should demand alternatives.
It's dangerous because:
1. It allows them to create a database with the IP address of every internet-connected router.
2. It allows them to control routers via DNS packets, which are frequently permitted even behind firewalls.
3. It providers a cover for traffic to their servers on the DNS port.
This kind of thing is very common. It's also very dangerous and we should demand alternatives.
I called it DNS ping because it essentially serves the same function as "ping" command but on slightly different level - by checking if DNS functions.
1. This is true, but most routers check for firmware updates anyway, so there you go. Not to mention online warranty registrations, etc. which would get you into their DB anyway.
2. Unlikely, unless you have a RCE bug in your DNS implementation, in which case the router is toast without the DNS querying.
3. One query is not "traffic". It is pretty easy to distinguish between one query on boot and constant communication. Again, it is only relevant if you already have malware in your router.
1. This is true, but most routers check for firmware updates anyway, so there you go. Not to mention online warranty registrations, etc. which would get you into their DB anyway.
2. Unlikely, unless you have a RCE bug in your DNS implementation, in which case the router is toast without the DNS querying.
3. One query is not "traffic". It is pretty easy to distinguish between one query on boot and constant communication. Again, it is only relevant if you already have malware in your router.
#1 and #3 are not even refutations and #2 you didn't even understand. You're absolutely wrong that this isn't a security issue. The simple proof is that no secure network would ever allow a device to call home like this.
Maybe you think that because it's common it's not a problem. That's not how security works.
Maybe you think that because it's common it's not a problem. That's not how security works.
Hanlon's Razor seems to fit this debacle fairly well:
"Never attribute to malice that which is adequately explained by stupidity."
"Never attribute to malice that which is adequately explained by stupidity."
Humans are rarely stupid, unfortunately.
Better to remain 'agnostic'
Better to remain 'agnostic'
Rarely?
Can I live in your reality? <3
Can I live in your reality? <3
There are two kinds of people in the world:
(1) Those who say 'this is stupid'; and
(2) Those who say "oh, i get it"
Rarely do they coincide empirically.
(1) Those who say 'this is stupid'; and
(2) Those who say "oh, i get it"
Rarely do they coincide empirically.
In any reality, humans are ingenious at things you don't want them to do.
[deleted]
[deleted]
A friend of mine has a newer Belkin router. It's actually kind of handy in the case that your cable/DSL modem is screwing up because the router will tell you that something is wrong with your connection. But I think it should probably have some "I'm sure my internet is fine, please continue" button or something.
Sure, on the router configuration page, I get that it might inform you that you don't have a connection. But (a) why only contact one company server? can't they ping anything else? and (b) why would you disable DNS if you suspect a problem with the connection? surely if there were a problem, the addresses wouldn't resolve and the result would be the same
Most routers monitor manufacturer servers for firmware updates, I believe.
I'm undecided whether this should be opt-in or opt-out, but it surely shouldn't break the router if something upstream breaks.
I hope the router implements TLS to transfer the firmware. Otherwise, that certainly isn't a massive attack vector.
http://www.amazon.com/TP-Link-TL-WR703N-Wireless-iphone4-and... .. This small little fella works perfectly for me with OpenWRT and even if you are not a techie you can use the chinese default firmware which works well as well .. More info on http://wiki.openwrt.org/toh/tp-link/tl-wr703n in case you want to install OpenWRT .. Plus this little fella just costs $30
Source might be this. TC adds nothing: http://community.belkin.com/t5/Wireless/Belkin-Routers-Inter...
> caused a false denial of service
Huh? Does this mean "the service thought it was DoS'd but it really wasn't"?
Huh? Does this mean "the service thought it was DoS'd but it really wasn't"?
Yeah, the article is extremely vague, not an explanation at all like the title says.
Another comment in this thread explains:
> It didn't shut off all internet access. The DNS resolver on the router checked for connectivity. If it couldn't resolve Belkin's domain, it disabled the DNS service because it assumed the Internet wasn't really connected.
By sp332, https://news.ycombinator.com/item?id=8429287
Another comment in this thread explains:
> It didn't shut off all internet access. The DNS resolver on the router checked for connectivity. If it couldn't resolve Belkin's domain, it disabled the DNS service because it assumed the Internet wasn't really connected.
By sp332, https://news.ycombinator.com/item?id=8429287
1) there is still no explanatation for the rational of doing so, and the origin of the black out !
2) there is still no valid reason to add a useless point of failure.
3) talking about security, the continuity of service is AS important as anything else (imagine you had two belkins set up for redundancy and multihoming, because internet is used to check credentials, you have a loss of a critical services. A useless SPOF IS a security risk! )
4) It is not Belkin role to make sure «internet» is available and take measure when not working. A router needs an IP connectivity including a DNS resolver, these are clearly checks that are the ISPs responsability.
5) there is no arm in letting a router not being able to resolve a DNS server (they can be statically configured on the hosts or by a DHCPd server, routers at best acts as a DNS cache and they are less efficient than tons of other solution even a stupid dnsmasq).
6) it means blackholing the IP to the cloud of Belkin is an efficient way to disconnect any user from the internet using their router. Is it a feature made for Russia, China and USA (I mean any place where privacy is considered a crime)?
Clearly Belkin has been bricking their gears when they are not connected to internet. Why? Why have a killswitch of the internet?
I cannot see any good reason for spending THAT much money in terms of investment and recurring cost. Someone planned this feature, and found smart to spend recuring money on obviously unneeded feature.
Facts don't add up. Companies are here to make money. They spent a hell lot of money to insert and plan and release this feature in their equipment. What do they gain of bricking any devices not able to contact their «cloud»?
Clearly Belkin has been bricking their gears when they are not connected to internet. Why? Why have a killswitch of the internet?
I cannot see any good reason for spending THAT much money in terms of investment and recurring cost. Someone planned this feature, and found smart to spend recuring money on obviously unneeded feature.
Facts don't add up. Companies are here to make money. They spent a hell lot of money to insert and plan and release this feature in their equipment. What do they gain of bricking any devices not able to contact their «cloud»?
I cannot fathom why this kind of "phone home" behavior is necessary to operate a router. Lets call it what it is, a kill switch.
iOS (and presumably Android) has a similar functionality, but at least in iOS's case it's to find out if the device should consider this wifi to be 'working' (and send traffic via WiFi) or 'blocked' (and continue using the mobile data network).
I can't think of any reason to have this failover behaviour on a consumer router that isn't multihomed. It's not the router's job to ensure that my internet is working according to some predefined rules, when there is no tangible benefit to doing so.
I can't think of any reason to have this failover behaviour on a consumer router that isn't multihomed. It's not the router's job to ensure that my internet is working according to some predefined rules, when there is no tangible benefit to doing so.
Can the feature be switched off? If not, that sounds pretty awful.
One thing I cannot tell is that Monday my Belkin n600 router started crapping out intermittently every few minutes. I would unplug and plug in the router again, and it would work until it went down next.
Was I actually affected by this issue?
The description of this issue makes it seem like the routers were completely unable to make a connection vs. the intermittent behavior I experienced so I'm really confused.
I reflashed by firmware at Belkin's advice later that day and it seemed to resolve the issue, but this can't just be coincidence...
Was I actually affected by this issue?
The description of this issue makes it seem like the routers were completely unable to make a connection vs. the intermittent behavior I experienced so I'm really confused.
I reflashed by firmware at Belkin's advice later that day and it seemed to resolve the issue, but this can't just be coincidence...
If your computer was using a combination of working DNS IPs and your Belkin router's IP, it's a plausible explanation for intermittent outages.
So if you had 8.8.8.8, 8.8.4.4, and 192.168.1.1, 1/3 DNS lookups would time out. Specifics are up to your operating system and lookup configuration, but you would almost certainly notice the problem, even if it was just minor annoyance. Lookups will usually be retried after a few seconds, so it might have just seemed like everything was extra laggy.
So if you had 8.8.8.8, 8.8.4.4, and 192.168.1.1, 1/3 DNS lookups would time out. Specifics are up to your operating system and lookup configuration, but you would almost certainly notice the problem, even if it was just minor annoyance. Lookups will usually be retried after a few seconds, so it might have just seemed like everything was extra laggy.
"“One of our cloud services associated with maintaining router operations was negatively impacted by a change made in our data center that caused a false denial of service."
Well, that's a non-explanation explanation. What "cloud services associated with maintaining router operations" are being performed? "Lawful interception", perhaps? Somebody needs to record the traffic between the router and Belkin HQ.
Well, that's a non-explanation explanation. What "cloud services associated with maintaining router operations" are being performed? "Lawful interception", perhaps? Somebody needs to record the traffic between the router and Belkin HQ.
While the conspiracy theories are fun, what actually was happening was that the router pinged a heartbeat server they had to detect whether it was connected to the internet, and when it wasn't, it would shut off the dns resolver so browsers would get the "internet is not connected" error messages, rather than some other random one.
this seems like fairly poor design. A router can't ping some heartbeat IP, therefore it shuts itself down?
It's not the heartbeat I take issue with - it's the dependency on the heartbeat for operation that doesn't seem to make sense.
It's not the heartbeat I take issue with - it's the dependency on the heartbeat for operation that doesn't seem to make sense.
That's what Reddit says. (http://www.reddit.com/r/technology/comments/2ik43h/belkin_fi...) Belkin, though, didn't say that. Some people thought that Belkin did a remote firmware update on the device. Belkin's own site indicates that firmware updates must be performed through the device's web interface, which, hopefully, can't be reached from the outside network.
Belkin routers apparently talk to "heartbeat.belkin.com", which is really an Amazon AWS instance.
Belkin routers apparently talk to "heartbeat.belkin.com", which is really an Amazon AWS instance.
Apparently they moved it to AWS in the past day, perhaps to recover from the outage.
During the outage heartbeat.belkin.com resolved to 67.20.176.130 which is held by Sungard.
We (and several other ISPs) got our customers back online by putting 67.20.176.130/32 on a loopback interface on one of our routers so it would always respond to pings. Other ISPs put overrides in their caching DNS servers so heartbeat.belkin.com always resolves to 127.0.0.1 (belkin.com doesn't roll DNSSEC).
During the outage heartbeat.belkin.com resolved to 67.20.176.130 which is held by Sungard.
We (and several other ISPs) got our customers back online by putting 67.20.176.130/32 on a loopback interface on one of our routers so it would always respond to pings. Other ISPs put overrides in their caching DNS servers so heartbeat.belkin.com always resolves to 127.0.0.1 (belkin.com doesn't roll DNSSEC).
http://www.dd-wrt.com/site/index
Problem solved.
Problem solved.
https://openwrt.org/
Though, dd-wrt's website looks a bit more user-friendly. Though, OpenWRT is where the most of the development and porting is happening.
Though, dd-wrt's website looks a bit more user-friendly. Though, OpenWRT is where the most of the development and porting is happening.
Totally creepy and unacceptable that a private piece of equipment can be affected remotely. What's not to say that many routers (and the devices attached to them) couldn't be attacked if (or now when with this information coming to light) Belkin was compromised by an external party?
Do you carry a cell phone? The baseband networking stacks deep in the phone (below even the operating system) can and have been remotely controlled by law enforcement on many occasions. The FBI used these attacks to turn on the microphone of cell phones and gather evidence on suspected mafia members: http://www.zdnet.com/news/fbi-taps-cell-phone-mic-as-eavesdr...
> Totally creepy and unacceptable that a private piece of equipment can be affected remotely.
Any modern mobile network client can be affected by the developer's servers going down, because that's the only way to do captive WiFi network detection.
Chrome, for instance: http://www.chromium.org/chromium-os/chromiumos-design-docs/n...
(I believe this kind of thing won't be necessary when Passpoint networks are deployed.)
Any modern mobile network client can be affected by the developer's servers going down, because that's the only way to do captive WiFi network detection.
Chrome, for instance: http://www.chromium.org/chromium-os/chromiumos-design-docs/n...
(I believe this kind of thing won't be necessary when Passpoint networks are deployed.)
They can, they are, and that is what you get for buying a proprietary black box and centering your network around it, in most cases.
openwrt / ddwrt / that fsf firmware project / etc exist for a reason, and nowdays hundereds of routers support open firmware to such a degree it is your fault to buy some backdoor infested unworkable disaster.
openwrt / ddwrt / that fsf firmware project / etc exist for a reason, and nowdays hundereds of routers support open firmware to such a degree it is your fault to buy some backdoor infested unworkable disaster.
I'd make that statement the other way around- open source firmware supports hundreds of routers. It doesn't look like the openwrt folks and others get much help from the router manufacturers.
Great. I thought my Belkin router croaked the other day, so I'm already out of $200+ for a new router plus overnight delivery. I was going to return it when I saw the old one was working again, but after reading this I'm getting rid of the Belkin on principle.
Did you remember to buy a router that can have open firmware?
This gave me a hell of a time the other day - I was perplexed as to what was happening and equally perplexed when it started to work again.
This has definitely spurred me to take another look at my home networking, this kind of thing simply should not happen.
This has definitely spurred me to take another look at my home networking, this kind of thing simply should not happen.
What the hell are they doing? How is this added value in any sense? I'm very glad that I run OpenWRT which avoids me a lot of the worry.
Hey... has everyone forgotten the Smart TV issues that were unable to connect internet due to the company service went down due to fire.
http://www.michaelhanscom.com/eclecticism/2003/11/07/belkin-...