I have allergies. I remember walking home from work one day thinking "Ah, my sinuses have cleared, the pollen must have finally gone." only to wake up the next morning all stuffed up again. Then it dawned upon me that work has HEPA filtered air, but my bedroom does not despite spending ~⅓ of my life there. Having an air purifier in my bedroom means I've avoided almost all symptoms of allergies for the last 5+ years without needing to resort to medication.
> If you’re getting a new phone or are worried about losing your existing one, you can always choose to back up your data to the cloud so it doesn’t get lost. We’ll automatically encrypt your backed-up data so no one can read it, including Google.
Close. AIUI, Google was annoyed at people meddling with DNS results, and with ISPs providing terrible DNS performance for their customers.
Google wins when the Internet is fast. Google's goals here are that you can use Websearch quickly and without hotspots etc meddling with traffic. The story is simple: If companies meddle with DNS results, or provide poor performance, then users don't use the Internet as much, and Google doesn't get to sell ads. Google wants you to have the fastest, most reliable Internet it can, because that means you don't get frustrated and go do something else. Google doesn't need to scrape the logs for whatever people think google might be scraping the logs for.
Google _does_ log some data for dealing with abuse (as you can probably imagine 8.8.8.8 gets a _lot_ of intentional and unintentional abuse), but tries to be as clear as it can be about what gets logged, what it's used for, and how long it's kept. Google treats these logs very very carefully, carefully limiting who has access, and when you need to use the logs to debug something, it's very carefully audited.
Disclaimer: I used to be one of the SREs oncall for Google Public DNS (but not any more – I now work for a different SRE team at Google).
The janitor is trying to help make packaging less toilsome:
- The janitor is trying to automatically migrate people to newer versions of debhelper, reducing the number of different build system variants.
- The janitor attempts to perform validation of all the long, complex, changing rules for you, so if you meet their requirements then it will propose an upgrade. If the janitor can automatically bring you into compliance with the rules, it will, otherwise it'll leave you upgraded to the step that requires a human to intervene. You can manually upgrade it one step, and then the janitor will loop back around and do the rest for you.
- The janitor will also fix suboptimal, or unnecessary packaging changes and improve them for you too, hopefully leaving you with a much smaller, simpler packaging system that's easier to reason about.
One change the Janitor can try to make automatically is upgrading debhelper version. Newer debhelper versions include support for compiling code with additional security measures. So we end up with more secure packages.
If ~everyone migrates off an older debhelper version, then the debhelper developers don't need to keep maintaining that code and can delete it, leading to a more maintainable code base, less complex documentation that says "X, unless you're on an older version, when it's Y" and so on.
The Janitor is pretty good at trying something like upgrading debhelper, then doing a build, and running the package tests, performing a package diff to make sure nothing unexpected happened then proposing a merge. Doing this by hand is slow and laborious. If debhelper always tried to autoupgrade to the newest version when it was run, it would be frustratingly slow.
And once the merge proposal is accepted, then it's clear which packages need human attention, and which were able to be cleanly migrated by automation. We use this a lot in the janitor. Run a fixer over all eligible packages, and see which ones succeed. Then investigate a handful of packages that failed, see if there is a common pattern that can be extracted and fixed. Then we reapply the fixer to all the remaining packages, see how many were fixed, and repeat the process until there are only a very few remaining.
Hi, I've been working with Jelmer on the janitor and related infrastructure for over a year now and worked with him writing this blog post.
As others have pointed out, this is used for Debian metadata, so upstreaming doesn't really apply. While "priority extra" might seem trivial, there are other much more sophisticated examples of what the janitor can do. For example, applying multiarch fixes. These fixes are a lot more complicated, so we didn't want to show their code and have to explain everything that's going on during the blog post, as the blog post was already long enough. This was just to give people an idea that it's possible to make large scale, high quality fixes relatively easily and how they might go about it if they also have a fix they want to propose.
We've been using the lintian metadata fixes as a proving ground to learn how to make safe, high quality changes at scale. Fixing one package is trivial, doing even a mediocre job of fixing 60,000 packages is much much harder. Nobody wants crappy automated junk in the repository.
There are also other easy to overlook, massive non-technical problems in this space, like trying to convince people that the Janitor is doing worthwhile work and people should embrace it. Significantly more thought and effort has gone into making sure people are delighted (or at least not too annoyed) by the janitor than the actual changes themselves. There's things like making sure you can correctly identify and preserve a large number of idiosyncratic formatting styles. This is open source software, and everyone has their own distinct opinions as to what is "best". You also don't want to overwhelm people with fixes they're not interested in. We've already got sketches of blog posts coming up that discuss some of these issues and how the janitor handles them.
The janitor can do things other than just lintian fixes, such as multiarch improvements and merge in new upstream releases and even upstream snapshots. These aren't enabled yet, as there are still being polished up to the standards Debian demands. For some of them (like merging new upstreams) there a lot of very difficult non-technical questions that still need answering. (How do you make sure that a compromised upstream developer submitting a backdoor into a common library doesn't have that code automatically pushed out to everyone?)
The goal of the janitor is to help Debian move forward (more) rapidly, to increase the overall quality of packages in the archive, and free up the humans so they can concentrate on things that require human judgement (eg writing high quality descriptions).
If the janitor is successful, then you shouldn't have to spend much time working on packages. All of the drudge work that takes human time should be automated, leaving only the things that require actual human discretion. This helps alleviate your "I ain't got no time for that". We're not there yet, but we're working towards it.
We have discussed being able to apply fixes further upstream. It's an even harder problem to get right, and again, most of the complexity is in the non-technical side. So far, we've not tackled this, but we have deeply considered it. For the technical sided problems, before we can do any upstream fixes, we need a reliable way of being able to find the upstream repositories, we need to be working on the latest upstream snapshot, we need to know how to build the package, etc. These are all problems the Janitor is already working on solving cleanly and elegantly first.
Don't worry, we've got some awesome plans for the Janitor. Lintian fixes are just the very first step.
There's lots of "privacy" improving DNS servers, but none of them mention trying to remove unintentional DNS queries.
It turns out lots of things will resolve anything that looks vaguely like a hostname to see if, in fact, they are a hostname. eg, "untitled.pdf". These queries get passed to your ISP, and then on towards the root name servers. So if you run a large nameserver, you quickly find that most of your DNS queries are very obviously rubbish.
With DNSSEC there are two new records (NSEC, NSEC3), that let you say "between these two names, I guarantee there is no valid records". Thus if your nameserver supports this, it can say "there are no valid names between .pccw and .pe, and thus anything that ends with .pdf is invalid". NSEC and NSEC3 records can both be cached and your resolver can synthesise NXDOMAIN records for them. (See RFC8198 for details).
So, instead of spraying queries for "untitled.pdf" across the internet, you can quickly, and efficiently return NXDOMAIN.
Another cause of these is search paths, when you look up "news.ycombinator.net", if that resolution fails, it will try adding the search path, eg: "news.ycombinator.net.example.org", again, leaking typos, and filenames to everyone in your search path.
If you actually value your privacy, this is the first step that you should take.
My uninformed assumption is that chrome generally uses whatever is configured in the system resolver, otherwise things like captive portals, and split horizon DNS wouldn't work properly, but I don't specifically know.
tl;dr: These companies are successful when more people use the Internet. DNS is a major contributor to poor perceived internet performance. Thus providing better DNS services encourages people to use the Internet more.
tl;dr: Google succeeds when people use websearch. Cloudfare succeeds when people use the Internet in general. People use websearch more when the Internet is fast and reliable. ISPs resolvers are frequently not fast nor reliable. Thus if you can improve peoples Internet experience with a better resolver, you improve your core business.
No need to track anyone via DNS for this to be successful.
You'd have to change your nameserver manually unfortunately.
But you could build a good webpage that actually measures how well your current nameserver performs over a wide variety of different types of lookups, both warm and cold caches etc.
With QNAME minimalisation, RFC7129 (Authenticated denial of existence) and RFC8020 (NXDOMAIN: There really is nothing underneath), you should be sending almost nothing to the root servers of use.
QNAME minimalisation will only send <randomstring>.com to the root for them to give you the referral.
and RFC7129/RFC8020 mean that when you get a NXDOMAIN back from the root, you'll cache it and never try again for a large swath of possible names.
Things to look for in comparing recursive DNS servers performance:
The 95%ile DNS response time for cached/uncached names. The 95%ile DNS response when one/some of the authoritative nameservers is "lame" or not responding. (better yet, 99%ile, but that requires even more queries...)
The average packet loss to the nameserver. (As many resolvers use the default of a 5s timeout, better resolvers use a 1s timeout, the best stub resolvers would use a dynamic timeout, but afaik, none do...).
Do they implement RFC7129 (authenticated denial of existence)? This can be used to prevent your service being used to attack an authoritative nameserver, prevents leaks of useless domains (eg machines looking up untitled.pdf as a domain), and allows you to return NXDOMAIN with much lower latency, making DNS search paths faster. RFC8020 (NXDOMAIN: There is really nothing underneath) would be another example where you can prevent leaking names, and return faster responses from a smaller cache (although I admit I've never seen anyone implement RFC8020 yet).
Will they accept (signed) responses into their cache in the additional section? Again, this can significantly reduce the time for uncached responses.
[hint: These are good reasons you should sign your domain, it can make things faster and reduce load on your authoritative nameserver!]
What is their story for domains that need a cache flush?
Do they (correctly) implement IPv6 from the recursive to the authoritative nameservers? Do they (correctly) implement IPv6 from the stub to the recursive nameserver?
How big is their cache? How long do things stay in their cache? There's no point being close to a nameserver with an empty cache. Querying www.google.com isn't really going to tell you much about their cache depth, nor is the Alexa 1M. You need a very very wide variety of names.
Do they provide good GeoIP responses? There's no point in getting an answer for the middle of the US in <1ms if you happen to be 300ms away in Asia somewhere. The DNS response was fast, but the webserver it sent you to is going to give you abysmal performance. This is often done with EDNS0-Client-Subnet, but it can also be fudged by making the outbound IPs for the iterative requests being diverse enough for different localities.
Do they "lie" about names? In what circumstances do they lie? Do they NXDOMAIN malicious domains? adult websites? ad domains? random websites? Do they redirect ad websites to their own ad farm? How do their lies handle DNSSEC?
Do they perform QNAME minimalisation to help protect your queries from servers that don't need it?
What other features do they implement to make sure their cache is never poisoned?
What is their abuse plan? If I send them a vast number of queries what happens? Do they send back TrunCated responses and force me over TCP? Will they respond with SERVFAIL? Or will they drop the queries? Or will they pass them all through to the authoritative nameservers? Do I need to do anything (other than stop sending abusive amounts of load) to be unblocked? What if the reason I'm sending a large number of queries is because I'm a carrier grade NAT IP pool and I have one broken/bad user?
What is their reliability story? Is it expected that they will go down for 10 minutes every now and again?
What do they do about general Internet Hygiene? Do they have protects against being used for reflection attacks?
Do they do preemptive lookups to keep their cache warm or is someone always guaranteed to have to wait for the full resolution? How do they make sure they don't accidentally DoS authoritative nameservers with preemptive resolutions?
Things not to look for:
ICMP/mtr times are essentially meaningless, except as providing general information about routing decisions.
The mean response time, as it tends to be washed out by cached response times, and what you don't care about is if it takes 15ms or 17ms on average, as you can't perceive the difference. What you _do_ care about is if one nameserver has 1/5000 queries which take >1s as that will become a frequent noticeable problem when your surfing.
Just looking at a few common names that are likely to be in the cache. Yes, those are important, but as with anything at scale, it's the long tail that's actually interesting and will dominate your perception of performance. You can set up your own domain, and search for random strings and force the full end-to-end query flow. (Beware about wildcard domains for this, if your domain is signed, in theory the nameserver could synthesize responses without going back to your nameserver).
Where are your vantage points for measurements? Many people appear to measure from places like AWS zones, and then report spectacular performance for DNS servers also hosted in the same AWS zones – despite most of their users not being hosted there.
Hmm, I'm sure there's more, but that's off the top of my head.
(Disclaimer: Once upon a time, I was one of the engineers oncall for Google Public DNS, so I have Opinions)
- One for up to 48 hours which contains IP addresses, which is used for handling abuse.
- One is permanent which doesn't contain any personal identifying information (eg IP addresses), which is used for things like internal performance monitoring, load testing, and tracking frequency of longer term abuse etc.
Google provides Google public DNS because if you see the internet as being slow because of poor DNS performance, then you don't use websearch as much. Google doesn't need to use Google Public DNS to track users, and would rather not have the information (as it makes it available for government requests etc which are a major pain to deal with). But running a large scale recursive DNS server tends to attract a _lot_ of abuse, both intentional, and unintentional as I'm sure 9.9.9.9 and 1.1.1.1 are discovering.
Google provides Google Public DNS because a lot of ISPs provide extremely poor default recursive nameservers (having tiny caches, dropping queries due to overload, not implementing IPv6, DNSSEC validation, EDNS0 payload size, or other important modern DNS features. Some ISPs also hijack domains for their own purposes, or "stretching" DNS TTLs etc) so providing a better alternative to improve overall Internet use is clearly in Google's best interest.
Having other public resolvers, with different trade offs is clearly better for everyone, including Google as long as they are reliable, trustworthy, and provide low latency responses.
Good luck to everyone who's joining in the fun of running a planet scale recursive DNS server.
(Disclaimer: I have previously worked on Google Public DNS, but no longer do)