Tell HN: All .fj domains have gone offline
52 comments
University of the South Pacific, which runs the DNS servers, [1] is up.
Google's DNS server at 8.8.8.8 is having problems.
Lookup using DNS server of sonic.net, which is a normal DNS server.
The Fiji DNS servers are unreachable, but you can get as far as a router at the University of the South Pacific. So the problem is local to there.
Google's DNS server at 8.8.8.8 is having problems.
Lookup using DNS server of sonic.net, which is a normal DNS server.
nslookup usp.ac.fj
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: usp.ac.fj
Address: 144.120.141.80
nslookup
> server 8.8.8.8
Default server: 8.8.8.8
Address: 8.8.8.8#53
> usp.ac.fj
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find usp.ac.fj: SERVFAIL
> example.com
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: example.com
Address: 93.184.216.34
Name: example.com
Address: 2606:2800:220:1:248:1893:25c8:1946
The Fiji DNS servers are unreachable, but you can get as far as a router at the University of the South Pacific. So the problem is local to there.
traceroute 144.120.146.1
traceroute to 144.120.146.1 (144.120.146.1), 30 hops max, 60 byte packets
...
8 AARNET-PTY.edge5.London1.Level3.net (217.163.113.74) 274.795 ms 274.476 ms 273.781 ms
9 xe-0-0-0.pe1.a.suv.aarnet.net.au (113.197.15.213) 312.534 ms 311.764 ms 311.766 ms
10 fastethernet0-0.aarnie.usp.ac.fj (202.158.204.194) 313.356 ms 312.176 ms 312.363 ms
11 * * *
12 * * *
> University of the South Pacific, which runs the DNS servers, [1] is up.
Not up from within Fiji. All government websites (and email) are down. Banks, Universities, mobile money wallets, etc.
Not up from within Fiji. All government websites (and email) are down. Banks, Universities, mobile money wallets, etc.
Oddly enough, right now..:
> server 144.120.141.80 Default server: 144.120.141.80 Address: 144.120.141.80#53 > usp.ac.fj Server: 144.120.141.80 Address: 144.120.141.80#53
* server can't find usp.ac.fj: SERVFAIL
> server 144.120.141.80 Default server: 144.120.141.80 Address: 144.120.141.80#53 > usp.ac.fj Server: 144.120.141.80 Address: 144.120.141.80#53
* server can't find usp.ac.fj: SERVFAIL
Fijivillage.com reports: "All websites and apps hosted in Fiji with the dotcom.fj suffix are currently down and this has also affected Vodafone’s M-PAiSA services.
This is due to an outage in the University of the South Pacific hosted dotcom.fj domain." [1][2]
[1] https://www.fijivillage.com/news/All-websites--apps-in-Fiji-...
[2] https://twitter.com/fijivillage/status/1501070675691278339
This is due to an outage in the University of the South Pacific hosted dotcom.fj domain." [1][2]
[1] https://www.fijivillage.com/news/All-websites--apps-in-Fiji-...
[2] https://twitter.com/fijivillage/status/1501070675691278339
And this folks is one of the reasons you don't use a 1 minute TTL on your DNS responses...
I use 24 hours on anything I'm not planning on making changes to. High TTL is a better experience for customers because they'll probably have it in local-ish caches even if there is internet routing disruption.
And 10 mins on anything I'm about to make changes to. That means if I accidentally make the wrong change, the 'blast radius' is minimized.
Obviously, when changing 24h down to 10 mins, keep a close eye on DNS server load, packet loss on links close to it, etc. If in doubt, raise and lower ttl's slowly.
And 10 mins on anything I'm about to make changes to. That means if I accidentally make the wrong change, the 'blast radius' is minimized.
Obviously, when changing 24h down to 10 mins, keep a close eye on DNS server load, packet loss on links close to it, etc. If in doubt, raise and lower ttl's slowly.
shamefully opens domain registrar and changes TTL
15 min?
I would recommend an hour for almost everything except where very fast updates is expected, in which case 5m is my lowest number (I work at a registrar).
Obviously registrars allow 1m - is there a reason not too besides extra load on DNS servers?
There are rumors of DNS resolvers deciding that some TTLs are “too low” to be valid, and applying their own default TTL value instead, thereby negating any benefit a low TTL would have had.
If the DNS server is out for more than 1 minute you'll get an outage. You also increase latency by preventing anybody from caching the response for more than 1 minute.
3 of the 5 nameservers for fj are online and responding (they are all anycasted around the world). It looks like connectivity to Fiji itself is what is disrupted.
Two are operated by gransy.com and one by PCH.net (which you should consider donating to and helping random country codes stay online: https://www.pch.net/about/donate).
Two are operated by gransy.com and one by PCH.net (which you should consider donating to and helping random country codes stay online: https://www.pch.net/about/donate).
It's not connectivity to Fiji. The internet works just fine across the country. Only .fj domains (and email accounts) are affected.
This might be a DNSSEC issue rather than a nameserver reachability issue:
https://dnsviz.net/d/fj/dnssec/
https://dnsviz.net/d/fj/dnssec/
I guess Google's DNS rigidly adheres to DNSSEC.
https://developers.google.com/speed/public-dns/faq#dnssec
"Google Public DNS is a validating, security-aware resolver. All responses from DNSSEC signed zones are validated unless clients explicitly set the CD flag in DNS requests to disable the validation."
"Google Public DNS is a validating, security-aware resolver. All responses from DNSSEC signed zones are validated unless clients explicitly set the CD flag in DNS requests to disable the validation."
Looks like you were right: https://lists.dns-oarc.net/pipermail/dns-operations/2022-Mar...
It's true...
couldn't get address for 'ns4.fj': not found
couldn't get address for 'ns5.fj': not found
couldn't get address for 'ns2.fj': not found
couldn't get address for 'ns1.fj': not found
couldn't get address for 'ns3.fj': not found
dig: couldn't get address for 'ns4.fj': no moreUsing a Recursive DNS server its down for me. UK based.
I also setup a RIPE measurement for those interested.
It asks for the NS record of ns1.fj.
https://atlas.ripe.net/measurements/39252885/#general
https://atlas.ripe.net/measurements/39252885/#general
[deleted]
Reachable for me.
I'm late to the party, but https://fiji.gov.fj/ worked for me, although it was rather slow to load.
I'm in England, for what that's worth.
I'm in England, for what that's worth.
Working slowly from Germany as well.
Would you mind saying where in the world you are?
Not working for me, in New Zealand.
This is why I always say you should think twice when choosing a ccTLD if you don’t have a presence there. Not every registry has the same uptime as Verisign.
Given the amount of past outages of ccTLDs we've seen (almost none), i'd say this problem is negligible and nothing to worry about.
There is one very recent exception however, the russian plans to partly separate from the internet shortly (especially regarding DNS) could well be an issue.
There is one very recent exception however, the russian plans to partly separate from the internet shortly (especially regarding DNS) could well be an issue.
Technical and legal issues that cause issues for customers at small ccTLDs are not uncommon. It all depends on who is running the registry. Try "cctld hijacked" and "cctld downtime" in your favorite search engine for a dozen or so examples.
Registries range in resources from "highly experienced billion-dollar organizations", to literally "we have a guy who updates the zone file in notepad", and they range in legal environment from, "functional democratic state with well-established judiciary" to "our government was violently overthrown this week".
Whether or not the problem is negligible depends on your particular use-case. For some, it might be a good choice, for others, it might not be.
Registries range in resources from "highly experienced billion-dollar organizations", to literally "we have a guy who updates the zone file in notepad", and they range in legal environment from, "functional democratic state with well-established judiciary" to "our government was violently overthrown this week".
Whether or not the problem is negligible depends on your particular use-case. For some, it might be a good choice, for others, it might not be.
Like I have said before¹²: Neither your registrar nor your chosen TLD registry should be in the habit of suspending domains at the drop of a hat, or be at risk of going out of business suddenly. If you mostly trust your local government, your national ccTLD should suffice. In fact, it should be your default choice unless you have strong indications it does not fulfill the above criteria.
1. https://news.ycombinator.com/item?id=29108933
2. https://news.ycombinator.com/item?id=21614298
1. https://news.ycombinator.com/item?id=29108933
2. https://news.ycombinator.com/item?id=21614298
And yet to my knowledge this has never happened before.
I vividly remember .de being offline causing trouble for a couple of hours in 2010.
https://www.theregister.com/2010/05/12/germany_top_level_dom...
https://www.theregister.com/2010/05/12/germany_top_level_dom...
It's too late to edit my comment, but I was referring to .fj, not ccTLD's in general.
Didn't .io ( which is a crime against humanity to boot) go down for a few hours a few years ago?
Yes. https://hackernoon.com/stop-using-io-domain-names-for-produc...
Also there were some comments from the owner at the time that I thought were pretty unprofessional about it, too.
Also there were some comments from the owner at the time that I thought were pretty unprofessional about it, too.
Why do you say .io is a crime against humanity?
.io is the TLD for the British Indian Ocean Territory, which is associated with the forced displacement of the Chagos islander people to make room for the Diego Garcia military base, amongst others. The British unilaterally decided to detach the Chagos archipelago from Mauritius before it got independent, which the deported people say (not without reason) is a crime against their human rights.
To back that up - the Chagosian claim is backed by the UK and rulings at the International Court of Justice. The UN supports Mauritius' claim to the territory.
Even besides the literal crimes against humanity, the .io domain really doesn't need to exist - what with the British Indian Ocean Territory having no permanent population and really not being a country at all.
"Being a country" isn't the criteria for the ISO code or domain.
Consider AQ, TW, GG, SJ, EH, HK and many others.
Consider AQ, TW, GG, SJ, EH, HK and many others.
TW is a controversial inclusion in your list.
Several of them are controversial to someone.
AQ is controversial to countries who want to exploit Antarctica's natural resources.
TW is controversial to China.
GG is controversial where it's used for tax evasion etc.
EH to Morocco, HK to the UK/China.
AQ is controversial to countries who want to exploit Antarctica's natural resources.
TW is controversial to China.
GG is controversial where it's used for tax evasion etc.
EH to Morocco, HK to the UK/China.
[deleted]
No idea. I was referring to .fj
Maybe not, but it is predicable that the quality and quantity of resources and governance available to a registry has an impact on their operations.
fiji.gov.fj
usp.ac.fj
etc.
http://www.iana.org/domains/root/db/fj.html