.io domains down?
96 comments
There are some things you can do to protect yourself against these kinds of problem. About 92% of DNS resolvers honour the TTL setting for the NS records from the leaf-node child-zones (e.g. the NS records you control as the owner of a domain). If your domain is reasonably popular, then that caching can help you outlast a problem with your TLDs name-servers. 2 days is a pretty common number for a cache lifetime.
Unfortunately for the 8% of resolvers that honour the TTL in the parent domain, you have less control. In this case for .io, the parent-zone NS ttls are just one hour;
Another important thing to consider is that the TLD you choose for your domain itself isn't the only TLD that your domain resolution may depend upon. Resolution also depends on all of the domains of your nameservers resolving too. Route 53 assigns every hosted zone 4 nameservers from 4 different TLDs, e.g.;
Full-disclosure: Route 53 engineer here.
Unfortunately for the 8% of resolvers that honour the TTL in the parent domain, you have less control. In this case for .io, the parent-zone NS ttls are just one hour;
;; AUTHORITY SECTION:
nic.io. 3600 IN NS ns1.communitydns.net.
nic.io. 3600 IN NS b.nic.ac.
nic.io. 3600 IN NS a.nic.io.
nic.io. 3600 IN NS b.nic.sh.
nic.io. 3600 IN NS ns4.nic.io.
nic.io. 3600 IN NS dns01.cdns.net.
nic.io. 3600 IN NS b.nic.io.
Another important thing to consider is that the TLD you choose for your domain itself isn't the only TLD that your domain resolution may depend upon. Resolution also depends on all of the domains of your nameservers resolving too. Route 53 assigns every hosted zone 4 nameservers from 4 different TLDs, e.g.;
example.info. 172800 IN NS ns-1834.awsdns-37.co.uk.
example.info. 172800 IN NS ns-22.awsdns-02.com.
example.info. 172800 IN NS ns-912.awsdns-50.net.
example.info. 172800 IN NS ns-1233.awsdns-26.org.
Full-disclosure: Route 53 engineer here.
I apologize for bothering you with this but I was nerding out on DNS recently and I have learned just enough to get very confused. It seems that you (the royal you) are benefiting from glueless out of bailiwick entries?
Should I not have such a strict belief that thou shall not have glueless out of bailiwick entries?
I appreciate you indulging my curiosity/confusion.
Should I not have such a strict belief that thou shall not have glueless out of bailiwick entries?
I appreciate you indulging my curiosity/confusion.
I'd say that the advice is outdated, but I'll try my best to explain and you can decide.
"Bailiwick" is the term used for the namespace a nameserver is considered authoritative for. If example.com is delegated to ns1.example.com then all of the DNS names ending in example.com are in-bailiwick. foo.example.com, bar.example.com, foo.bar.example.com and so on. If example.org is also delegated to ns1.example.com, then names ending in example.org are also considered in-bailiwick for that name-server to return.
When example.com is delegated to ns1.example.com there's an obvious circular dependency; how can ns1.example.com be resolved without first resolving example.com? The way that this is solved is by using glue records. A glue record is when you tell the name-servers for the parent zone (.com in this case) what the IP address for ns1.example.com is. That way the com servers - when asked about www.example.com can say; "actually, you need to ask ns1.example.com, and by the way here's the IP address for ns1.example.com".
With an in-bailiwick delegation and glue record in place, a query can get all the way to the server it needs within 2 or 3 queries.
Now, if example.com is delegated to ns1.otherdomain.net, the delegation is out of bailiwick. Now when the resolver asks the com server for www.example.com it gets a response that says "you need to ask ns1.otherdomain.net". But the resolver doesn't know the IP address for ns1.otherdomain.net. So now it has to put the original query on-pause and go and resolve ns1.otherdomain.net. This is part of why resolvers are called "recursive resolvers" - they have to recursively resolve all dependencies.
Of course ns1.otherdomain.net could itself require out-of-bailiwick recursion, and the process could get quite lengthy. That's why some guides recommend against it - especially when managing you're own DNS; you may as well make everything in-bailiwick.
So how do we manage this? A few ways;
Firstly, there's no reason that managed DNS customers can't use in-bailiwick names anyway. Route 53 customers, for example, are free to create their own names that resolve to their Route 53 name-servers and to use those names. For a real world example, if you "dig amazonaws.com" you'll see that we have four names ; r[1234].amazonaws.com that actually point to regular Route 53 IP addresses.
Secondly, the nameserver names used by large providers are usually well cached. At Route 53 we host a lot of customers domains, and many many domains refer to each name like "ns-22.awsdns-02.com.", so for that reason the names are usually very well-cached, even if your own domain is low-traffic.
Thirdly, we chose TLDs that are large, popular and well-run. .com/.net are operated under a single contract by verisign, so if you do; "dig allcosts.net @g.gtld-servers.net" , you'll see that our .com and .net customers actually get the benefit of two in-bailiwick glue records (we have two nameservers that end in .com or .net). Customers with domains ending in .org or .co.uk get one each.
Fourthly, our nameserver domains are themselves exclusively delegated to in-bailiwick names, for example if you do; "dig awsdns-50.net. @g.gtld-servers.net", you'll see that all four of its nameservers are handled as glue-records. This limits the amount of recursion a resolver must do to just one level deep.
Our goal here is to be able provide a simple easy-to-use set of names that work well and are robust regardless of the customer's own domain of choice, without having to maintain hundreds of nameserver names in every single TLD.
"Bailiwick" is the term used for the namespace a nameserver is considered authoritative for. If example.com is delegated to ns1.example.com then all of the DNS names ending in example.com are in-bailiwick. foo.example.com, bar.example.com, foo.bar.example.com and so on. If example.org is also delegated to ns1.example.com, then names ending in example.org are also considered in-bailiwick for that name-server to return.
When example.com is delegated to ns1.example.com there's an obvious circular dependency; how can ns1.example.com be resolved without first resolving example.com? The way that this is solved is by using glue records. A glue record is when you tell the name-servers for the parent zone (.com in this case) what the IP address for ns1.example.com is. That way the com servers - when asked about www.example.com can say; "actually, you need to ask ns1.example.com, and by the way here's the IP address for ns1.example.com".
With an in-bailiwick delegation and glue record in place, a query can get all the way to the server it needs within 2 or 3 queries.
Now, if example.com is delegated to ns1.otherdomain.net, the delegation is out of bailiwick. Now when the resolver asks the com server for www.example.com it gets a response that says "you need to ask ns1.otherdomain.net". But the resolver doesn't know the IP address for ns1.otherdomain.net. So now it has to put the original query on-pause and go and resolve ns1.otherdomain.net. This is part of why resolvers are called "recursive resolvers" - they have to recursively resolve all dependencies.
Of course ns1.otherdomain.net could itself require out-of-bailiwick recursion, and the process could get quite lengthy. That's why some guides recommend against it - especially when managing you're own DNS; you may as well make everything in-bailiwick.
So how do we manage this? A few ways;
Firstly, there's no reason that managed DNS customers can't use in-bailiwick names anyway. Route 53 customers, for example, are free to create their own names that resolve to their Route 53 name-servers and to use those names. For a real world example, if you "dig amazonaws.com" you'll see that we have four names ; r[1234].amazonaws.com that actually point to regular Route 53 IP addresses.
Secondly, the nameserver names used by large providers are usually well cached. At Route 53 we host a lot of customers domains, and many many domains refer to each name like "ns-22.awsdns-02.com.", so for that reason the names are usually very well-cached, even if your own domain is low-traffic.
Thirdly, we chose TLDs that are large, popular and well-run. .com/.net are operated under a single contract by verisign, so if you do; "dig allcosts.net @g.gtld-servers.net" , you'll see that our .com and .net customers actually get the benefit of two in-bailiwick glue records (we have two nameservers that end in .com or .net). Customers with domains ending in .org or .co.uk get one each.
Fourthly, our nameserver domains are themselves exclusively delegated to in-bailiwick names, for example if you do; "dig awsdns-50.net. @g.gtld-servers.net", you'll see that all four of its nameservers are handled as glue-records. This limits the amount of recursion a resolver must do to just one level deep.
Our goal here is to be able provide a simple easy-to-use set of names that work well and are robust regardless of the customer's own domain of choice, without having to maintain hundreds of nameserver names in every single TLD.
Anthony from DNSimple here. I'll pass along what I know so far: some of nic.io's name servers appear to be no longer delegating .io domains out to authoritative name servers. Some of their name servers are still delegating properly, but some are not.
This issue also affects .sh and .ac TLDs I believe since they are the same registry. If I find out anything new I'll post on Twitter in the @dnsimple account and here.
This issue also affects .sh and .ac TLDs I believe since they are the same registry. If I find out anything new I'll post on Twitter in the @dnsimple account and here.
ooo, you do 301 redirects, I'll be over shortly...
To explain why that's useful, we have three domains, draw.io (mutter), diagramly.com and diagram.ly that all map to the same thing. The actual site runs on a Google App Engine appspot domain. On GAE, you have to buy a Google Apps for Business account to map a domain to an GAE appspot domain.
We have three Apps accounts because of this. We're not happy to do the 301 on a physical server, that loses the scalability advantage of GAE. This has always needed to be done at the DNS level ( I know it's not truely a DNS feature, but I'm assuming the scaling of the redirects at someone who does it as a service is better than mine ). This is the first time I've seen this service. That's all...
To explain why that's useful, we have three domains, draw.io (mutter), diagramly.com and diagram.ly that all map to the same thing. The actual site runs on a Google App Engine appspot domain. On GAE, you have to buy a Google Apps for Business account to map a domain to an GAE appspot domain.
We have three Apps accounts because of this. We're not happy to do the 301 on a physical server, that loses the scalability advantage of GAE. This has always needed to be done at the DNS level ( I know it's not truely a DNS feature, but I'm assuming the scaling of the redirects at someone who does it as a service is better than mine ). This is the first time I've seen this service. That's all...
For the lazy: https://twitter.com/dnsimple
That would explain why it's not consistent globally. I have an odd example here from my location:
https://gist.github.com/MiniCodeMonkey/5729249
Indeed, http://www.icb.co.uk seems to be responsible for all those TLDs.
OpenDNS has cleared our cache of all .io domains in case any NXDOMAINS are being cached.
I always love when the CEO of a biggish company is sitting in the comments
We're back online at http://candid.io. Someone on twitter posted "VCs should think twice before investing in startups without a .com" -> I think that's patently absurd. Azure, AWS, and the other 'core infrastructure' services suffer similar outages form time to time.
I hate genericized ccTLDs, but I don't think the VCs are the ones who should be avoiding them. People creating services should stick to .com, get the .net, and if they have to do something like "getdropbox.com" until they can buy dropbox.com, so be it.
Going from dropbox.io to dropbox.com is ok, but I'd probably go for getdropbox.com in preference to dropbox.io.
Going from dropbox.io to dropbox.com is ok, but I'd probably go for getdropbox.com in preference to dropbox.io.
Well as a person hosting a service under a .io domain I must say that it's next to impossible to find a decent domain under .com these days. They are all taken by domain sharks hoarding away and running link-farms. The .io landscape is a refreshing one, it's well recognized by it's typical audience (tech people), expensive enough so that it's not attractive for the domain-sharks, and there's plenty available.
Why exactly do you hate them?
Why exactly do you hate them?
1) Once you're successful you have to get the .com/.net/etc anyway. There's a lot of ambiguouty when you say "go to my site, de.lici.o.us" (which was probably the dumbest one ever), but even saying "referly" makes me think "referly.com" more than "refer.ly". If you're going to get the gccTLD, you absolutely must have sitenameio.com too. So, it doesn't really save you from "decent domain not available" -- prefixing with "get" or "my" in .com isn't any worse.
2) ccTLD has a defined meaning -- it's the country. I don't hate ccTLDs; e.g. Amazon.co.uk makes a lot of sense for the UK site of Amazon. But when people
3) They expose you to additional risk. The US can largely shut down most things, but .ly means you're exposed to Libya (JFC! an internet business voluntarily deciding to be exposed to Libya?)
4) If you really want something other than .com, you could use another TLD. Generally they all tend to be scams (I'm not aware of many .info or .biz domain names which are anything but scams, although I guess a few single-purpose/serving sites are in .info)
In fairness, IO is actually the least objectionable of all the gccTLDs, since "IO" is a pretty bogus ISO code in terms of representing a country.
ccTLDs make sense when you want to "fly the flag" of that country, too -- wikileaks.is or wikileaks.ch makes sense. All the gccTLDs have policies which are strictly worse than .com though.
And 2-letter domains (which are mostly ccTLDs) do make sense for link shorteners, particularly temporary/transient ones.
2) ccTLD has a defined meaning -- it's the country. I don't hate ccTLDs; e.g. Amazon.co.uk makes a lot of sense for the UK site of Amazon. But when people
3) They expose you to additional risk. The US can largely shut down most things, but .ly means you're exposed to Libya (JFC! an internet business voluntarily deciding to be exposed to Libya?)
4) If you really want something other than .com, you could use another TLD. Generally they all tend to be scams (I'm not aware of many .info or .biz domain names which are anything but scams, although I guess a few single-purpose/serving sites are in .info)
In fairness, IO is actually the least objectionable of all the gccTLDs, since "IO" is a pretty bogus ISO code in terms of representing a country.
ccTLDs make sense when you want to "fly the flag" of that country, too -- wikileaks.is or wikileaks.ch makes sense. All the gccTLDs have policies which are strictly worse than .com though.
And 2-letter domains (which are mostly ccTLDs) do make sense for link shorteners, particularly temporary/transient ones.
I like .info for personal point of presence sites.. I use tracker1.info as an example.
I also do like .io for previously stated reasons by others... It's techie/geeky (I/O), and it's expensive enough that the squatters have mostly stayed away.
I also do like .io for previously stated reasons by others... It's techie/geeky (I/O), and it's expensive enough that the squatters have mostly stayed away.
When is the last time you heard of the .com infrastructure failing across the board?
Just because you're exposed to some reliability risk doesn't mean that you can't expose yourself to more by using a smaller TLD.
Just because you're exposed to some reliability risk doesn't mean that you can't expose yourself to more by using a smaller TLD.
Yes, you're taking on a bit more risk in the short-term... but everything's a trade-off, you're getting much back for a slightly elevated risk in having a short, concise, very meaningful (in our case) name that influences adoption. There's no other alternative -> .COM is a limited resource, only reasonable assumption is that reliability of other TLDs must improve in the long run.
They have 7 name servers, 5 are broken and returning NXDOMAIN for all domains.
Working:
b.nic.io. ns1.communitydns.net.
Not working:
a.nic.io. a.ns13.net. b.nic.ac. b.ns13.net. ns3.icb.co.uk.
Working:
b.nic.io. ns1.communitydns.net.
Not working:
a.nic.io. a.ns13.net. b.nic.ac. b.ns13.net. ns3.icb.co.uk.
Some of those that were broken are starting to work again. However the NXDOMAIN responses will have been cached and take a little while to expire.
Wow. About a year ago the the entire .st registry went down for about 8 hours:
http://blorn.com/post/29851770158/beware-cutesy-two-letter-t...
I guess the title of the blog post pretty much says it all.
http://blorn.com/post/29851770158/beware-cutesy-two-letter-t...
I guess the title of the blog post pretty much says it all.
On April 29 I received a notice from [email protected] concerning user-generated content that I was to remove from an IO domain "by April 2" (ie a month earlier) - otherwise they would advise the complaintant to contact law enforcement and would suspend the domain "upon receipt of a formal request to do so".
When I attempted to reply, the email address was out of order ("Write error: Broken pipe") and remained so for several days.
It doesn't seem to be the most professionally run domain...
It doesn't seem to be the most professionally run domain...
Man, you think you plan for everything, and then your whole TLD just stops working. What's the best way to plan for something like this?
Have an alternate domain using completely different infrastructure.
Having a .com or .net as a backup domain, even if it isn't as sexy looking, doesn't hurt.
Having a .com or .net as a backup domain, even if it isn't as sexy looking, doesn't hurt.
This may work for API endpoints but there's little re-course for your primary domain. It makes more sense to have .COM/.NET as your primary domain and have the .IO as a convenient alternative than the other way around. But my opinion is still that this is a short-term issue: the reliability of other TLDs will improve.
That's what we do at Phaxio: api.phaxio.com api.phaxio.co. We added the 'co' ever since the Zerigo DNS fiasco last year.
I'd recommend not worrying about it. You can't plan for everything and the time you spend being redundant for this (very unlikely) case is probably better spent somewhere else.
Send an email to one user asking for feedback and you will get a much better ROI than trying to plan for TLD failure.
Send an email to one user asking for feedback and you will get a much better ROI than trying to plan for TLD failure.
My site http://gitignore.io was down and I just pinged my CDN and my domain host to see what the problem was. Everything seems to be back up and running now. Does anyone know what happened?
Check out http://www.whatsmydns.net pretty useful for seeing where issues are or tracking propagation.
Thanks, that's an excellent tool!
I experienced this as well at http://quantify.io - New Relic reported a 2 hour outage which just ended.
Yes! My site (robert.io) is down and I've been chasing my tail trying to figure out what's wrong. It's refreshing to see that I'm not the only one.
For me too.. well, sometimes it resolves, sometimes it doesn't. There seems to be an issue with .io globally: https://twitter.com/dnsimple/status/342991722365734912
I'm glad I'm not crazy.
$ host www.filepicker.io www.filepicker.io is an alias for anycast.filepicker.io. anycast.filepicker.io has address 174.129.254.207 Host anycast.filepicker.io not found: 3(NXDOMAIN) Host anycast.filepicker.io not found: 3(NXDOMAIN)
$ host filepicker.io Host filepicker.io not found: 3(NXDOMAIN)
$ host www.filepicker.io www.filepicker.io is an alias for anycast.filepicker.io. anycast.filepicker.io has address 174.129.254.207 Host anycast.filepicker.io not found: 3(NXDOMAIN) Host anycast.filepicker.io not found: 3(NXDOMAIN)
$ host filepicker.io Host filepicker.io not found: 3(NXDOMAIN)
We had this happen before to our .st domain. http://blorn.com/post/29851770158/beware-cutesy-two-letter-t...
Pingdom reckons https://www.photographer.io is still up, but I guess I really can't be certain. It hasn't been a great week for my domain, what with the DNSimple attack and this.
It looks like all of the nic.io root name servers are delegating again.
Intercom.io is down as well. Particularly frustrating since its preventing me from replying to customer emails.
Technical details of permanent delivery failure: DNS Error: Domain name not found
Technical details of permanent delivery failure: DNS Error: Domain name not found
In some sort of irony, statuspage.io is down. Was looking at their offering about an hour ago and thinking of taking it for a spin, but now I'm not so sure...
Isn't that a bit unfair to judge the quality of their service based on the DNS being down for every .io service?
The service is only as strong as the weakest link.. choose those links wisely.
Steve from StatusPage here -- an issue like this is extremely rare so we just haven't prioritized a solution for it. This is something we plan on being able to handle in the future.
Thanks Steve, agreed - just unfortunate that I only found your service earlier today, and when I went back the site was down :( Would having a custom domain have got around the problem, or is that a CNAME to your .io domain?
Hey Alex, Danny here from StatusPage. Agreed, not the best timing! We'll have a fix for this in the future and to answer your question, yes, a custom domain would get around the problem.
Hmm every .io domain on this page has worked for me even if I've never visited them before. Maybe my ISP (VirginMedia) has cached the records.
I had the same problem with notable.ac The problem appears to have been resolved. The latest DNS queries show that it is now delegating properly.
Yep, it seems the same registry runs .io, .sh and .ac.
Yeah, I was deploying to Heroku when this roots error occurred. This screwed up the deploy (as it crashed in the middle) and just 500'ed on every page load. And meant every DNS request in my deploy chain that relied on *.notable.ac resolving ended up caching negative results. Luckily it's back up now :)
It's like a million hipsters cried out in terror and were suddenly silenced.
I will pour out some PBR for good ol' .io.
I will pour out some PBR for good ol' .io.
http://pineapple.io has been up whole time
Unfortunately it may appear up for you and for monitoring systems yet still not resolve for others, depending on DNS cache freshness as well as which root name server you (randomly) fall on.
It is currently also impossible to register .io domains (I can confirm this since yesterday afternoon)
Fine for me in the UK:-
$ host algasm.io
algasm.io has address 176.58.123.62you can check whatsmydns.net, how some places are working, and some not, for example: http://www.whatsmydns.net/#A/www.works.io
My website, brand.io, seems to be up... I hope my email isn't down. :(
Even more self-marketing for you:
"Brandon Thomas is a triple major undergrad at Southern Polytechnic State University in the Computer Science, Biology, and Chemistry programs. Brand loves science, uses Linux and vim, and programs primarily in Python, Javascript, and C++. Computing interests include machine learning and computer graphics. After graduation Brand intends to study computational metabolomics or possibly found a startup company."
"Brandon Thomas is a triple major undergrad at Southern Polytechnic State University in the Computer Science, Biology, and Chemistry programs. Brand loves science, uses Linux and vim, and programs primarily in Python, Javascript, and C++. Computing interests include machine learning and computer graphics. After graduation Brand intends to study computational metabolomics or possibly found a startup company."
God dammit! You just made me realise my internet was down :-( I thought that I was just having a quiet day. So that's an entire days worth of business lost.
from Germany a.ns13.net and b.ns13.net return NXDOMAIN. Both seem to be located in JP. The other DNS are working for me.
They recently switched their backend to a new platform (May 29th) maybe this issue is related…
They recently switched their backend to a new platform (May 29th) maybe this issue is related…
My domain is down as well - words.io.
Good thing it's just some crappy side project.
Good thing it's just some crappy side project.
Oh my, I just announced bitnodes.io like a couple hours ago..
Nope: Our sites (fine.io, fry.io) are up
It would appear that fry.io is actually down. That's the one hosted with nic.io. This is why I wanted to move away from them. They are really incompetent.
Is it possible to move away with them? I mean, aren't they finally responsible for delegating all .io domains to the correct nameservers?
It is not possible to move away from them. nic.io is the registry for .io domains, so you can't move to anyone else. They also act as a registrar for their own TLD, so you could move to another registrar, but ultimately they're responsible for .io delegation.
I thought it was as I transferred fine.io over to someone else because I found out nic.io were storing passwords in plain text. Sadly, it seems not. It's quite frustrating to know your business is essentially at the mercy of people that don't know what they are doing.
I can't access fine.io or fry.io. I can access nic.io, stanford.io and www.filepicker.io (Note the www) so far.
Oh really? Fine.IO works perfect here. That's certainly annoying.
Mine seems to be working fine: sawt.io
My site at kun.io is working for me…
no downtime for http://read.io
That you are aware of. There was outages at the io registry nameservers, read.io was most certainly affected.
Our site focal.io is down as well.
My farmer.io domain is down also!
Works fine here in México.
[deleted]
nature.io up and running.
nitrous.io is working...
git.io is UP.
not mine.
Our site is not down: http://82.io
This story won't reach the front page. It's not mainstream enough.
Are all down for me.
$ host filepicker.io Host filepicker.io not found: 3(NXDOMAIN)