Measuring open DNS resolver use(blog.apnic.net)
blog.apnic.net
Measuring open DNS resolver use
https://blog.apnic.net/2019/09/23/dns-resolver-centrality/
48 comments
Folks here might be interested in my own (more tl;dr) survey of public resolver usage:
https://twitter.com/mikedamm/status/1136409254305263616?s=20
Do you know how Cloudflare picked up so much market share even with 8.8.8.8 around?
Google doesn’t promote 8.8.x.x lately, so when there were some sites being blocked by Indian ISPs, I saw cloudflare DNS being recommended. Also I believe cloudflare doesn’t implement ban-lists provided by Indian courts. I have noticed sites blocked in google dns, working with cloudflare dns.
To extend this, I’ve never seen google promote 8.8 publicly. All recommendations I’ve seen have been from other users or forum members. It maybe that Google’s trust isn’t what it once was, causing a reduction in recommendation.
How about measuring zone file access programs?
How many users utilise the full number of existing domain names in the world today?
How many names do users realistically need to access in a lifetime?
What if we exclude ad servers and other domains that exist solely for marketing?
It depends on the user, but in some cases the majority of their non-commercial web^1 use can be accomplished without ever making remote DNS queries; the IP addresses can be stored and used on a long-term basis. That is because a user may only visit the same small number of websites. The foregoing is of course only an opinion based on testing conducted by yours truly. Every user is different.
Try measuring how many times the non-commercial websites you visit change IP addresses in a year.
You might find that DNS resolution is like that "definition of insanity" meme: making the same query day after day, expecting a different answer.
1 It makes a difference whether or not a user is using the web to make purchases. For making purchases online, DNS resolution is almost always required. Domains and IP addresses in that context are constantly changing. Go figure. OTOH, if you are visiting a website such as news.ycombinator.com on a frequent basis, is it really necessary to look up the IP address for news.ycombinator.com every time you visit the website? I have used the same IP address for years at a time.
How many users utilise the full number of existing domain names in the world today?
How many names do users realistically need to access in a lifetime?
What if we exclude ad servers and other domains that exist solely for marketing?
It depends on the user, but in some cases the majority of their non-commercial web^1 use can be accomplished without ever making remote DNS queries; the IP addresses can be stored and used on a long-term basis. That is because a user may only visit the same small number of websites. The foregoing is of course only an opinion based on testing conducted by yours truly. Every user is different.
Try measuring how many times the non-commercial websites you visit change IP addresses in a year.
You might find that DNS resolution is like that "definition of insanity" meme: making the same query day after day, expecting a different answer.
1 It makes a difference whether or not a user is using the web to make purchases. For making purchases online, DNS resolution is almost always required. Domains and IP addresses in that context are constantly changing. Go figure. OTOH, if you are visiting a website such as news.ycombinator.com on a frequent basis, is it really necessary to look up the IP address for news.ycombinator.com every time you visit the website? I have used the same IP address for years at a time.
I've worked on software that attempts to estimate the end-user demand represented by a given resolver. Most of the specifics are covered by NDA, but suffice to say that it's surprisingly difficult to do accurately. There is a lot of non-standard behavior in various resolver software, some of it avoidable, some of it not.
This trick of altering the hostname of a subresource to identify the client is a thing I long wished we could do, but sadly we didn't have enough control of the content to do that.
This trick of altering the hostname of a subresource to identify the client is a thing I long wished we could do, but sadly we didn't have enough control of the content to do that.
What this data shows is that concerns about centralization, especially in relation to DoH, are overblown. Only 1.15% of users in this dataset are using Cloudflare DNS, and APNIC is in a region riddled with government censorship and crappy ISPs -- two major incentives for people to try alternative resolvers. Without such incentives, nobody would even bother to change their devices' DNS settings.
I'm in a country with both of these problems, so every machine I set up gets Cloudflare as the primary resolver with Google as the secondary. Fix these problems first, and I won't have to do this anymore. Centralization? I dunno. Taking control of DNS away from my state and ISP would actually count as decentralization in my book.
I'm in a country with both of these problems, so every machine I set up gets Cloudflare as the primary resolver with Google as the secondary. Fix these problems first, and I won't have to do this anymore. Centralization? I dunno. Taking control of DNS away from my state and ISP would actually count as decentralization in my book.
Agree that shifting control from your state and ISP are beneficial, but centralizing DNS to one or two for-profit providers in the process is less than ideal IMO. I'm working on a project that's aiming to make DNS fully decentralized and wrote an explainer article in case you're interested https://www.namebase.io/blog/meet-handshake-decentralizing-d...
If it's not too revealing, can you share what country you're in?
If it's not too revealing, can you share what country you're in?
Sure, even more decentralization would be good. Nevertheless, I think the usual concerns about Cloudflare are massively overblown and misses a crucial role that they're playing in the fight against censorship and surveillance.
Cloudflare is the first well-known provider that decided to support a working protocol for encrypted DNS. DoH might not be the best possible protocol, but it's shipping now and others are not. (DNSCrypt is also shipping, but it has the weird property of speaking something that isn't HTTPS on port 443. That's too easy to censor.) There will be healthy competition if other people would please stop arguing and start shipping, too.
I'm in South Korea. The censorship regime here is more prudish than draconian, and I'm not in any danger of prosecution for criticizing it. The way it is implemented is extremely crappy, though. On top of DNS-based censorship, ISPs are doing DPI on TLS handshakes to sniff hostnames in the SNI extension. Lots of techies here are very interested in new technologies showcased by Cloudflare. That includes not only DoH but also their proposal to use keys stored in DNS to close the SNI loophole. Once again, what works in this situation is not a novel, theoretically perfect protocol but one that ships now and blends into the petabytes of HTTPS traffic that Cloudflare handles every day.
Cloudflare is the first well-known provider that decided to support a working protocol for encrypted DNS. DoH might not be the best possible protocol, but it's shipping now and others are not. (DNSCrypt is also shipping, but it has the weird property of speaking something that isn't HTTPS on port 443. That's too easy to censor.) There will be healthy competition if other people would please stop arguing and start shipping, too.
I'm in South Korea. The censorship regime here is more prudish than draconian, and I'm not in any danger of prosecution for criticizing it. The way it is implemented is extremely crappy, though. On top of DNS-based censorship, ISPs are doing DPI on TLS handshakes to sniff hostnames in the SNI extension. Lots of techies here are very interested in new technologies showcased by Cloudflare. That includes not only DoH but also their proposal to use keys stored in DNS to close the SNI loophole. Once again, what works in this situation is not a novel, theoretically perfect protocol but one that ships now and blends into the petabytes of HTTPS traffic that Cloudflare handles every day.
I think DNS-over-HTTPS is a much bigger cause for concern.
What is the cause for concern? I am trying to understand why DNS-over-HTTPS could be a bad thing from the user end.
It makes it very hard to control your network. I have a DNS setup at home that I want all my equipment using. It blocks ads and other sites I don't want accessed. With DoH, I can't really be sure that browsers, devices, etc aren't using an alternative DNS system.
You could null route or firewall all the open resolvers, or at least the most common ones.
If your router is linux, that might look like
[1] - https://github.com/bambenek/block-doh
[2] - https://github.com/curl/curl/wiki/DNS-over-HTTPS
If your router is linux, that might look like
/sbin/ip route add blackhole 9.9.9.9 2>/dev/null
/sbin/ip route add blackhole 1.1.1.1 2>/dev/null
/sbin/ip route add blackhole 1.0.0.1 2>/dev/null
/sbin/ip route add blackhole 8.8.8.8 2>/dev/null
/sbin/ip route add blackhole 8.8.4.4 2>/dev/null
[1] - https://github.com/bambenek/block-doh
[2] - https://github.com/curl/curl/wiki/DNS-over-HTTPS
You can't be sure anyway though? The only thing that router controls are standard DNS queries, probably sent through a standard port. If they're using anything different at all, the router won't catch it, even if it's not encrypted.
But the plain DNS makes you absolutely unable to control your DNS queries outside your network.
Nothing stops anyone in the middle intercepting your queries and returning whatever they want, including your provider.
Nothing stops anyone in the middle intercepting your queries and returning whatever they want, including your provider.
It's not. It's an unalloyed good thing. The concern is with configurations that make it hard to use anything but Cloud Flare. There are alternatives.
What do you think about Paul Vixie's views on DoH?
On the face of it, it seems like we're going to end up with a bunch of black box devices (from Google, Amazon etc) in our homes that are totally immune to most forms of network policing because between DoH, ESNI, TLS and CDN fronting, you can't see anything.
On the face of it, it seems like we're going to end up with a bunch of black box devices (from Google, Amazon etc) in our homes that are totally immune to most forms of network policing because between DoH, ESNI, TLS and CDN fronting, you can't see anything.
Paul Vixie has been one of the Internet's most dedicated proponents of DNSSEC, a technology that essentially escrows keys with governments. If DNSSEC and DANE had progressed according to Vixie's preferred schedule, Muammar Gaddafi would have owned BIT.LY's CA. Vixie operates a company that relies on passive DNS observation to generate telemetry for corporations. Smart dude. Would not weight his privacy opinions heavily.
DNSSEC doesn't "essentially escrow keys with governments".
It's exactly as true to say that DANE gave Gaddafi ownership of bit.ly's CA as to say that today Boris Johnson owns the CA for slither.io - and as ridiculous. Back when you first started claiming this the Ten Blessed Methods weren't even a thing. You're complaining about the inadequate back door lock on a house that has the front door propped open.
I work for an outfit that buys passive DNS data (among other things) and all our suppliers agree that DoH makes no difference to their roadmaps - because they don't care who asked, only what the query and answer are. Is Vixie doing something vastly different? Maybe so but you've offered no evidence what that could be.
It's exactly as true to say that DANE gave Gaddafi ownership of bit.ly's CA as to say that today Boris Johnson owns the CA for slither.io - and as ridiculous. Back when you first started claiming this the Ten Blessed Methods weren't even a thing. You're complaining about the inadequate back door lock on a house that has the front door propped open.
I work for an outfit that buys passive DNS data (among other things) and all our suppliers agree that DoH makes no difference to their roadmaps - because they don't care who asked, only what the query and answer are. Is Vixie doing something vastly different? Maybe so but you've offered no evidence what that could be.
You haven't offered a rebuttal other than saying this argument is "ridiculous". I'm obviously not coming out of nowhere with it. Can you do better than "nuh-uh"?
If you don't agree that it's ridiculous to say Boris Johnson controls the CA for slither.io then... I guess we just have a very different definition of what it means to be ridiculous.
I can live with that.
I can live with that.
If my iPhone honored my passphrase, but also a second secret passphrase held by Apple in iCloud that I had no control over, what would you call that scheme? I would say that Apple had an escrowed passphrase. Well, that's precisely how DNSSEC works. I control my own key, sure, and never have to reveal it, but that doesn't matter, because the roots can simply override that key with their own.
You keep making these emotional appeals, but if I'm wrong, I don't see you actually rebutting me. "Your argument is so bad I will not deign to engage with it" is not a rebuttal.
You keep making these emotional appeals, but if I'm wrong, I don't see you actually rebutting me. "Your argument is so bad I will not deign to engage with it" is not a rebuttal.
At the risk of wasting my time with Humpty Dumpty:
OK, so first up I guess the problem is you've no idea what "escrow" is and so you ended up confused as to what key escrow could be. So let's fix that first.
Escrow is a service people or companies offer, the idea goes like this. Alice wants to do a deal with Bob, and Bob wants to make a deal with Alice, they're going to swap some of Alice's baseball cards for Bob's antique vase, but they don't trust each other. Fortunately they both trust Trent, a Third Party. Trent offers an Escrow service, both Alice and Bob agree with Trent that the baseball cards and the antique vase go to Trent, and then when he's got both things he'll send them to their new owners. If anything goes wrong, Trent gives back anything he received to the person who sent it and the deal is off.
In the tech industry the most likely set up you'll see is that investors are worried all the stuff they're spending a lot of money on at a startup is not material that a bunch of burly guys can pick up and put in a truck if the firm fails - instead it's in Git repos or Google Drives and if it falls apart they know it has residual value that could be realised but they're not technical people, they're money people. So an escrow firm says fine, pay us a bunch of money and make the startup sign this data escrow agreement, and then if they fail you activate this clause and we give you a bunch of USB drives full of source code and whatever else. The escrow firm has IT people who can do stuff like send over keys for access to a GitHub, who know the difference between an S3 bucket and a SD card, which the investors don't want to have to learn about.
In key escrow _your_ keys are held by a third party on your behalf. You can do end-to-end encryption if you want, but you're obliged to use this escrowed key. If the third party releases the key (e.g. because of a warrant, or an NSL, or because they're corrupt) then whoever gets it can now decrypt everything you've sent, or impersonate you seamlessly.
If your iPhone honors a secret Apple passphrase that's a _backdoor_. If instead Apple insists on keeping a copy of your passphrase in a safe at Apple HQ that Tim Cook promises not to open _that_ would be escrow.
Your objection that in DNSSEC "the roots can simply override that key with their own" also applies to any PKI, the whole _point_ of a PKI is that the trusted third party binds identity to keys, and if trust is misplaced they might falsely bind an identity to the wrong key. So the problem is - as I illustrated - that objections on the basis of DNSSEC being a PKI work just as well against the Web PKI. As a reason to prefer the Web PKI over DNSSEC they're ineffective.
OK, so first up I guess the problem is you've no idea what "escrow" is and so you ended up confused as to what key escrow could be. So let's fix that first.
Escrow is a service people or companies offer, the idea goes like this. Alice wants to do a deal with Bob, and Bob wants to make a deal with Alice, they're going to swap some of Alice's baseball cards for Bob's antique vase, but they don't trust each other. Fortunately they both trust Trent, a Third Party. Trent offers an Escrow service, both Alice and Bob agree with Trent that the baseball cards and the antique vase go to Trent, and then when he's got both things he'll send them to their new owners. If anything goes wrong, Trent gives back anything he received to the person who sent it and the deal is off.
In the tech industry the most likely set up you'll see is that investors are worried all the stuff they're spending a lot of money on at a startup is not material that a bunch of burly guys can pick up and put in a truck if the firm fails - instead it's in Git repos or Google Drives and if it falls apart they know it has residual value that could be realised but they're not technical people, they're money people. So an escrow firm says fine, pay us a bunch of money and make the startup sign this data escrow agreement, and then if they fail you activate this clause and we give you a bunch of USB drives full of source code and whatever else. The escrow firm has IT people who can do stuff like send over keys for access to a GitHub, who know the difference between an S3 bucket and a SD card, which the investors don't want to have to learn about.
In key escrow _your_ keys are held by a third party on your behalf. You can do end-to-end encryption if you want, but you're obliged to use this escrowed key. If the third party releases the key (e.g. because of a warrant, or an NSL, or because they're corrupt) then whoever gets it can now decrypt everything you've sent, or impersonate you seamlessly.
If your iPhone honors a secret Apple passphrase that's a _backdoor_. If instead Apple insists on keeping a copy of your passphrase in a safe at Apple HQ that Tim Cook promises not to open _that_ would be escrow.
Your objection that in DNSSEC "the roots can simply override that key with their own" also applies to any PKI, the whole _point_ of a PKI is that the trusted third party binds identity to keys, and if trust is misplaced they might falsely bind an identity to the wrong key. So the problem is - as I illustrated - that objections on the basis of DNSSEC being a PKI work just as well against the Web PKI. As a reason to prefer the Web PKI over DNSSEC they're ineffective.
If you'd prefer to refer to DNSSEC as a "backdoored" crypto protocol, I'm happy to stipulate that here.
Saying that backdoors are "the whole _point_" of a cryptosystem is not the devastating refutation you appear to think it is.
Saying that backdoors are "the whole _point_" of a cryptosystem is not the devastating refutation you appear to think it is.
You got into this saying DNSSEC "essentially" does key escrow and you've now walked this back to a stipulation that the whole point of all PKI is a "backdoor".
But that goes from being a relevant point about Vixie (his support of a system with key escrow - a false claim) to the irrelevant and vague (trusted people might betray your trust) that I guess if I squint I could read as sort of vague anti-establishment sentiment and otherwise is just pointless.
Here's a much shorter way to write what you meant without needing to slip in a false claim about DNSSEC:
"Paul Vixie is a smart guy but he's wrong about this"
But that goes from being a relevant point about Vixie (his support of a system with key escrow - a false claim) to the irrelevant and vague (trusted people might betray your trust) that I guess if I squint I could read as sort of vague anti-establishment sentiment and otherwise is just pointless.
Here's a much shorter way to write what you meant without needing to slip in a false claim about DNSSEC:
"Paul Vixie is a smart guy but he's wrong about this"
my impression from his latest talk at eurobsdcon this weekend was that he promotes dns over tls first and foremost. you can see for yourself when the videos go up.
If the device is a black box, you don't even need key exchanges though? You can just send encrypted data with a pre-configured, unique, randomly generated (for the device) key to a hard-coded IP-address. And even if you don't want to use a hard-coded IP-address, there are about a million ways to have some sort of custom encrypted "address resolution" through IRC or whatever else. In fact you could also just send the data to some Tor node or even base64 encoded through IRC. And a million other ways. It's not like DoH is really going to make a difference here. I mean you can even just send pings with what seems like randomly generated payloads that actually contain encrypted data. Really not sure what DoH is supposed to change about that. All it does is make spying with zero effort very slightly more difficult to detect.
It's really common for "security" products to assume bad guys will meekly obey conventions like telling the truth. "Are you a bad guy?" "No". Ok then, test passed.
Plenty of anti-TLS 1.3 stuff for example says that they "need" the plaintext Certificate message which in TLS 1.3 is now encrypted. But that message is literally just some public data - an X.509 certificate, if you're using it to "verify" anything your security is broken because a bad guy can send someone else's cert. They can't send a working CertificateVerify for it, but you don't know that because that message was never plaintext.
Plenty of anti-TLS 1.3 stuff for example says that they "need" the plaintext Certificate message which in TLS 1.3 is now encrypted. But that message is literally just some public data - an X.509 certificate, if you're using it to "verify" anything your security is broken because a bad guy can send someone else's cert. They can't send a working CertificateVerify for it, but you don't know that because that message was never plaintext.
Will there be any alternatives to those black box devices from Google, Amazon, etc.?
From a security and privacy standpoint I think DoH is a good thing.
I wish it didn't have the overhead of TCP/HTTP (which is why I was a bigger fan of DNSCrypt). Anyone can stand up a resolver that can handle 50k UDP requests a second and operate a public resolver. It starts to get operationally dicey to stand up infrastructure that can do 50k HTTP requests a second. As a result you end up with a small handful of players who can operate medium to large scale public resolvers.
I wish it didn't have the overhead of TCP/HTTP (which is why I was a bigger fan of DNSCrypt). Anyone can stand up a resolver that can handle 50k UDP requests a second and operate a public resolver. It starts to get operationally dicey to stand up infrastructure that can do 50k HTTP requests a second. As a result you end up with a small handful of players who can operate medium to large scale public resolvers.
I don't buy it. At 50k requests per second, you're servicing a lot of users. Cost of customer acquisition dominates cost of serving the marginal 1000 HTTP requests. UDP doesn't make that business any more viable.
I don't think parent commenter is implying it has to be a commercial application. Imagine something released with little-to-no cost, with community-driven support and users that organically find the page. Not all services are commercial.
Why does it need to be a business? A network of decentralized recursive resolvers operated by a number of different parties is preferable in almost every way to CloudFlare running a single endpoint.
The NTP pool works quite well on this model. It would be a much larger commitment for participants had to manage stateful connections.
The NTP pool works quite well on this model. It would be a much larger commitment for participants had to manage stateful connections.
"Unalloyed" is a strong claim - you are saying effectively that it is pure, flawless gold.
I can't think of anything bad about it. One popular strain of criticism is clearly bogus, that it removes a measure of visibility from network operators (that's the point). Another is that it centralizes DNS services at companies like Cloud Flare, which, no, you can run your own DoH resolver server (and probably shouldn't use Cloud Flare regardless). Finally, people say that it's clunky compared to datagram-based alternatives; from my perspective, DNS has been hampered by the UDP service model for decades, and revisiting it so that, at least at the retail level, it's TCP/HTTPS like everything else is an opportunity, not a problem.
>you can run your own DoH resolver server
How does this work in your mind?
How does this work in your mind?
I don't understand your question. The first Google SERP will give you a multiplicity of tutorials on setting up a DoH server.
At the point when I would be "run[ning] my own DoH resolver", I'm already at the point of setting up my own resolver. Why wouldn't I just turn on unbound and call it a day?
The point of running your own DoH resolver is that you're doing it somewhere off-network, to tunnel DNS out of an ISP network you don't trust to somewhere else you trust more.
You can also just not run DoH if you trust your network. I'm saying DoH is a good thing, not that everyone has to use it.
You can also just not run DoH if you trust your network. I'm saying DoH is a good thing, not that everyone has to use it.
Your ISP knows where you go without looking at your DNS. Because after you do the lookup you go there.
Alright, but the reason for the complaint about centralization you're dismissing is that most people are encountering DoH via Firefox's plan to default resolve via DoH to Cloudflare, rather than in the context of being deft Unix hands searching for a way to tunnel their DNS out of an untrustworthy network.
I do not think a feature that delegates DNS to Cloud Flare is an unalloyed good. I think the protocol is. The distinction is important.
Oh. Well, I lean to the view that X-over-HTTP is often a bad model; partly because HTTP has been heavily-commercialized.
I can see that there are problems with DNS; there always have been, and it's always been obvious. But for now, I think I prefer running my own resolver. Unbound is really easy to set up.
I can see that there are problems with DNS; there always have been, and it's always been obvious. But for now, I think I prefer running my own resolver. Unbound is really easy to set up.
Serious question - why do you prefer DoH vs say, DNS over TLS? The former always seemed rather hacky to me.
I don't think there's anything wrong with DoT, but DoH is deliberately more difficult to block, which is a sensible design goal for a privacy and anti-censorship protocol. Both are better than directly using legacy plaintext DNS.
The argument is well known: the intertubes promises much equanimity but capitalism and stuff.
OK I am being a bit cruel but this is what we have. If you don't own up to intending to cuddle up to Amazon, Google, Apple, Microsoft in the next 30s then you are probably a liar or a bit deluded.
Thing is, I'm a bit of a fan of capitalism but perhaps some sort of light touch regulation is needed in the Wild West. A bloke with a big old star on the chest might be nice.
OK I am being a bit cruel but this is what we have. If you don't own up to intending to cuddle up to Amazon, Google, Apple, Microsoft in the next 30s then you are probably a liar or a bit deluded.
Thing is, I'm a bit of a fan of capitalism but perhaps some sort of light touch regulation is needed in the Wild West. A bloke with a big old star on the chest might be nice.
It's probably even worse than what you describe.
You'll be cuddling up to the Amazon, Google, Apple, Microsoft of whichever global power you're closest to.
I'm in Australia, so I don't know if in twenty years I'll still be connected to Westnet or Sinonet. I'll probably buy a black market connection to Westnet from a guy with a mohawk and implants in his head.
I'm in Australia, so I don't know if in twenty years I'll still be connected to Westnet or Sinonet. I'll probably buy a black market connection to Westnet from a guy with a mohawk and implants in his head.
Agreed. The world is trending towards an internet that's split up between the different countries. There are some technologies that can counter that (ie check my bio) but it'll be an uphill battle imo.