Stop using low DNS TTLs (2019)(blog.apnic.net)
blog.apnic.net
Stop using low DNS TTLs (2019)
https://blog.apnic.net/2019/11/12/stop-using-ridiculously-low-dns-ttls/
109 comments
We should do the right thing even if other people do the wrong thing.
Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit.
Fundamentally however, giving clients control over when failover occurs means giving up control over failover.
Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft.
The conversation went like this:
> "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem.
> "Client software often ignores DNS changes."
> "We designed Azure so that Traffic Manager does failover and load balancers are always active-active."
> "That's nice, but client software ignores DNS changes."
> "Umm..."
Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible.
The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-...
Traffic signs shouldn't be taken down just because some people speed regardless of the posted speed limit.
Fundamentally however, giving clients control over when failover occurs means giving up control over failover.
Literally just a couple of hours ago, I was patiently explaining this to some Azure techs from Microsoft.
The conversation went like this:
> "Just use Traffic Manager for active-passive fail over!" they piped up, helpfully trying to solve my problem.
> "Client software often ignores DNS changes."
> "We designed Azure so that Traffic Manager does failover and load balancers are always active-active."
> "That's nice, but client software ignores DNS changes."
> "Umm..."
Fundamentally, the DNS protocol is just too old and crusty for the modern Internet, much like SMTP and FTP. It's best avoided to the extent possible.
The big providers seem to be converging on Anycast IP routing, where a single address can be routed to multiple data centres dynamically. You're all probably familiar with 8.8.8.8 and 1.1.1.1, both of which are Anycast IPs, but anyone can create a global static IP like that in minutes via a service such as Azure's Cross-region load balancer: https://learn.microsoft.com/en-us/azure/load-balancer/cross-...
> Fundamentally however, giving clients control over when failover occurs means giving up control over failover.
Fundamentally the problem is that you want to direct a client to particular datacenter and it is not always shortest part (shortest != lowest latency) so anycast DNS isn't helping here.
There is reason biggest CDNs use it to direct to particular DC instead of " just anycasting everything" like you're proposing, it's suboptimal.
> You're all probably familiar with 8.8.8.8 and 1.1.1.1,
And both companies behind that still direct you to particular datacenter, not anycasted IP, for the actual content.
Anycast IP are also "expensive", you need to use entire /24 route to get one, and it takes routing table space in every router's memory that is limited, and requires costly replacement if it runs out everywhere, not just where you distribute it.
Fundamentally the problem is that you want to direct a client to particular datacenter and it is not always shortest part (shortest != lowest latency) so anycast DNS isn't helping here.
There is reason biggest CDNs use it to direct to particular DC instead of " just anycasting everything" like you're proposing, it's suboptimal.
> You're all probably familiar with 8.8.8.8 and 1.1.1.1,
And both companies behind that still direct you to particular datacenter, not anycasted IP, for the actual content.
Anycast IP are also "expensive", you need to use entire /24 route to get one, and it takes routing table space in every router's memory that is limited, and requires costly replacement if it runs out everywhere, not just where you distribute it.
Aren’t routing tables also cached (not by clients, but by ISPs)? What is their typical TTL?
It varies; for direct peering/connections where a link goes down (and fast failover protocol is used), usually 5 seconds or so.
Global propagation takes in the order of 5 minutes or less: https://labs.ripe.net/author/vastur/the-shape-of-a-bgp-updat...
They aren't really cached. Routes are kept in the routing table as long as peers are responsive, if they stop responding to hello's or a link goes down to directly connected peer the routes are removed (or replaced with a less preferred one). While its possible to configure a router to keep routes in the table for a peer that is down for some time its typically not used unless you have a good reason for it.
They are added and withdrawn on demand, or dropped all at once if router loses connectivity with another.
So really it's processing time of every node in the chain. Few seconds to few minutes, not really that different from DNS.
The advantage is that there is no updates if there is no changes, instead of update every time TTL expires but that's about it.
So really it's processing time of every node in the chain. Few seconds to few minutes, not really that different from DNS.
The advantage is that there is no updates if there is no changes, instead of update every time TTL expires but that's about it.
routing tables usually aren't "cached" in the same sense that DNS caches entries.
however. Inside ISP;s routes are usually distributed by an IBGP with a couple of route reflectors, and route reflectors will drop routes if no router within the IBGP domain is advertising the route.
Routes between ISP's are usually fairly static thanks to mechanisms like graceful restart, which makes sure routes do not get dropped, even if BGP has a restart between routers. THere are various other mechanism in which to achieve this. (look up route flap dampening if you want to learn more).
however. Inside ISP;s routes are usually distributed by an IBGP with a couple of route reflectors, and route reflectors will drop routes if no router within the IBGP domain is advertising the route.
Routes between ISP's are usually fairly static thanks to mechanisms like graceful restart, which makes sure routes do not get dropped, even if BGP has a restart between routers. THere are various other mechanism in which to achieve this. (look up route flap dampening if you want to learn more).
The service I worked for in AWS faced this as a routine pain. No matter what the TTL, stuff completely ignored it. In the end the only sensible thing we could do was accept that we were going to impact customers and that we just can't realistically account for misbehaving DNS infrastructure. We had something like a 5 minute TTL on our public record, and would wait 15 minutes after removing the IP from the DNS records before we would take action that would potentially break connectivity.
By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control.
What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.
By setting the DNS TTL, we had established the terms by which we offered the service. Failure to accept those terms is not something we can control.
What we generally found was that clients disconnected would reconnect almost immediately to one of the other IP addresses. Any disruption to the client appeared to be almost negligible.
It doesn’t help that Java’s default configuration is to cache DNS entries forever for “security”[0]. Any long running server that makes external requests will eventually hit an issue where one their dependencies no longer works because DNS has changed and the TTL is ignored. I’ve seen this bite teams for 15 years and it will probably keep doing so well into the future.
0 - https://docs.oracle.com/javase/7/docs/technotes/guides/net/p...
0 - https://docs.oracle.com/javase/7/docs/technotes/guides/net/p...
Only if you have a security manager enabled (=are running in an applet/webstart). The entire security manager concept has been deprecated, so things are looking better
.NET HTTP clients used to suffer from this, too, but they no longer do as of .Net Core 3, as long as they're used correctly.
I seem to recall that HAProxy and Squid used to do this too.
I'd say about 80% of traffic moves within 2, maybe 3 TTLs. That 20% of traffic though... I've never really seen traffic on old servers go to zero real looking traffic, even after waiting months. Sometimes there's a dropoff when you send RSTs to SYNs, but sometimes not.
If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?
If you're using a DNS service that charges per query, set the TTLs high! Otherwise, 60s is probably good enough for most failover. I wouldn't go too far under 30s, unless you have a good reason; some recursives do stupid things with low values, and why tempt fate. Otoh, a lot of high profile sites and services do have < 10s ttl, so it probably works?
From the CDN & authoritative name server side Ive mostly dealt with eyeballs and a smattering of devices. With ~1min TTLs itd be 80% in 5min, 95% in 20-30min, 99% at an hour, 99.9% 1day, the remainder entirely variable. Literally stopped announcing BGP prefixes for years, turn them back up, get immediate client traffic to 80 & 443. Have done down to 10s and lower and its fine (as far as we were concerned).
That's my experience exactly. About 80% move within an hour, and then the remaining 20% trickle in after that.
[deleted]
> "When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds"
Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes.
DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design.
Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.
Likely because the clients just weren't doing DNS lookups. I would often run into clients written to resolve a hostname and then query the resolved IP for the duration of the process (often this is done as part of a library or runtime). Our customers wouldn't restart their apps for weeks or months so there was always a long tail of clients which never picked up dns changes.
DNS wasn't designed for failovers. We can use it for that, but it's never going to be perfect by design.
Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.
> Because of this, IP level failovers are the way to go wherever possible. Routing is designed to handle this sort of thing in the best possible way.
But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?
But you can't just suddenly make one IP address refers to another datacenter with another ISP in another country, can you?
You sure can. There are many details and caveats and it takes a lot more infrastructure and technical sophistication. Many small sites don't control their own routing at all, for example.
DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.
DNS is very simple, easy, and mostly good enough. And DNS failover is now well-enough known that we can mostly expect clients to accept fault if they don't engineer to expect it.
Yes, with BGP and anycast.
What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.
What you can't do with the above is more fine grain load balancing. If your LAX node is getting overloaded, you can use a combination of geo and DNS to start splitting some amount of traffic away.
via the magic of Anycast :)
>It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.
IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconvenience for your customers.
IMO that is a little misguided. 60-second and higher TTL will be honored most of the time. And I don't really get what that ISP in Alaska was doing: DNS is about 0.1% of Internet traffic (according to some quick googling), so decreasing that will save you only a tiny tiny bit of bandwidth at disproportionate inconvenience for your customers.
Not sure about what kind of connection Alaska has but I imagine this is more about reducing latency for new connections than bandwidth.
When it comes to TTL I try to imagine the perspective of the person on the other side that implemented the resolver at a university (which copies might be anywhere) or isp a couple decades ago. What odd behavior did they have to manage and what quick hacks might they have done to fix them?
From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.
From this I have chosen a minimum of 5 minutes of TTL. Anything that might have caused a large amount of dns traffic in the early day of the internet to go off every minute, or worse ever few seconds, might killed their server and taught the young programmer to sanitize inputs.
>Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.
That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
That doesn't really matter. Once a server becomes saturated you just remove it from being returned. Having multiple records being returned just slows down how fast a server can become saturated. You don't need the multiple records to be saturated at the same rate.
I think you're misunderstanding. If I have a lookup for foo.com configured to return 1.2.3.4 half the time and 5.6.7.8 the other half of the time, I want half the traffic to go to each load balancer. Maybe I'm running different software on each, or behind each. Maybe I'm using different networking gear. There are a ton of cases where getting close to 50/50 is what I need to get good experimental data.
Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.
Why would you think that it doesn't matter that the traffic doesn't match the ratios I picked? I picked them for a reason.
I think charcircuit suggests returning the inverse of the actual measured traffic: if the target is 50/50 and backend A gets 70 while backend B gets 30, one solution could be to return A in only 30% of the requests and B in 70% of them, leading to an actual 50/50 distribution.
I am suggesting that the returned servers are the servers with the least load. Yes, there may be uneven load being assigned to these servers, but clients are being given the choice of selecting a server with low load.
I'm suggesting to use more horizontal scaling. So if you have 1.2.3.0/24 and you advertise 2 servers at a time you can start with 1.2.3.1 and 1.2.3.2 and then you would cycle out servers as they increase in load. If one server is getting less traffic then it just takes longer for it to cycle out.
People without cached records will always get the returned servers with the lowest load.
I was referring to load balancing and not splitting traffic for an experiment.
People without cached records will always get the returned servers with the lowest load.
I was referring to load balancing and not splitting traffic for an experiment.
You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results.
Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "closer" to their current IP. Because if you're in 1.0.0.0/8, it's certainly going to be better to connect to 1.2.3.4 instead of 5.6.7.8. Same thing happens in v6 land, of course.
Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "closer" to their current IP. Because if you're in 1.0.0.0/8, it's certainly going to be better to connect to 1.2.3.4 instead of 5.6.7.8. Same thing happens in v6 land, of course.
If get unlucky A's load will rise quickly and it will no longer have a record for it. As the TTL expires the resolvers will get a new set of servers with low load.
Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.
Lumpy results just means the times for an IP to be rotated out will be lumpy, and not that the server load will be lumpy.
You're suggesting that the DNS server(s) get realtime load feedback and adjust appropriately. That's possible, but not always available.
Yes, that is how you scale DNS load balancing. There is a pretty low limit for the number of records you are allowed to return in practice. Putting 100 A records works with 1.1.1.1, but will break other's resolvers on the internet.
>That's possible, but not always available.
If you build it, then it's always available.
>That's possible, but not always available.
If you build it, then it's always available.
Sure, but a lot of people use other people's DNS services. The number of times I've been allowed to build a DNS service for my employer is zero. I've used static DNS, where you just put in your X number of A records and hope; dynamic services where you put in X and have them serve only Y of them in any request and hope. I've used services with load feedback, but those are always a much higher tier.
>Sure, but a lot of people use other people's DNS services
There are many of DNS services which offer an API for querying and updating records.
It is very easy to write a service / script that just fines the X least loaded servers and then call an API to set those as the available records. In practice you will also want to implement some monitoring that it is actually working.
There are many of DNS services which offer an API for querying and updating records.
It is very easy to write a service / script that just fines the X least loaded servers and then call an API to set those as the available records. In practice you will also want to implement some monitoring that it is actually working.
> At every place I've worked, we've had trouble where ISPs just ignore TTLs and set them arbitrarily high.
DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.
DigitalOcean does this, or at least some of the collocation/hosting providers they are using for their infra. They appear to redirect :53 connections to their own resolvers that cache NXDOMAINs.
It's an ugly situation. DNS seems like the wrong way to do load balancing. Meanwhile web browsers seem to have decided that it's a great idea to send out DNS requests for every domain on a web page (of which there are dozens, thanks to adtech, CDNs and other stuff.)
If you're in a corporate environment with control over the infrastructure, low DNS TTLs allow you to mostly guarantee a failover. There will always be clients that ignore TTLs, but most of them are fixed by IT rebooting the box.
As the poster says, when it comes to the public internet DNS TTLs are more like guidelines.
Luckily most client DNS caches are cleared on a reboot.
FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".
As the poster says, when it comes to the public internet DNS TTLs are more like guidelines.
Luckily most client DNS caches are cleared on a reboot.
FYI, if I remember correctly Windows (and MacOS?) used to cache negative responses for "a while".
> It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake.
That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.
That's a weird conclusion. My conclusion would be that you can't force low TTLs for everyone or even most users to you don't can't use them to allow for quick changes and may as well let everyone else benefit from caching too.
[deleted]
The scenario I think about is, what happens if the DNS entry I just pushed had a typo in it? If my TTL is 60s, then I push a fixed entry and my site is back up quickly. If the TTL was 1hr, then anyone who cached that bad entry can't connect for about that long. 60s is better for disaster recovery, 1hr is better for performance.
I've seen this happen on internal networks where IT support tries to guide end users through flushing their local DNS cache so the users can get back to work before TTL expires on a broken entry.
The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address.
Most sites I've worked with change DNS <1 time per year, so outside infrequent, planned maintenance a long cache time is best for performance.
I've seen this happen on internal networks where IT support tries to guide end users through flushing their local DNS cache so the users can get back to work before TTL expires on a broken entry.
The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address.
Most sites I've worked with change DNS <1 time per year, so outside infrequent, planned maintenance a long cache time is best for performance.
If you have DNS typo for a fixed time, the longer the TTL the fewer clients will ever see the error. E.g. lets say your typo is up for 10 mins. With a 5 minute TTL everyone will be seeing the error. but with a 60 min TTL, 5/6 of your customers will never see the typo (assuming there's enough traffic for caching)
Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rather this is a trade-off and one might consider both directions).
Slow onset of impact is probably more useful in minimizing incidents than fast onset of fix (edit: or rather this is a trade-off and one might consider both directions).
I think the amount of people that would see the error either way would be the same, assuming constant fix time and traffic. This means lowering the TTL as much as possible should be the right solution, because those affected will only have the issue for as long as the fix takes (hopefully less than 1hr) whereas if it is kept at 1hr those affected by the error will have it for an entire hour.
Disclaimer I'm not OP
Disclaimer I'm not OP
Testing strategy is also pretty important here. If you can run integration tests that bypass DNS caches immediately after the change, then you should know quick if things are working. If both old and new DNS settings are valid, the slow onset works great and minimizes impact of an error.
I've handled DNS changes where the underlying host was changing IP address. In that instance the old DNS entry immediately became invalid, so a quick DNS cut-over was needed for all clients.
I've handled DNS changes where the underlying host was changing IP address. In that instance the old DNS entry immediately became invalid, so a quick DNS cut-over was needed for all clients.
> The model I've been trying recently is to make DNS changes in three parts. First: change existing DNS entry TTL from 1hr to 1s. Wait 1hr as the caches clear. Update entry to the new address, keeping 1s TTL. Wait, test, and monitor for a while. Finally, increase the TTL back to 1hr. It takes more planning, but I avoid a 1hr partial outage if I mess up the address.
This is my preferred way of doing it, especially for MX records.
This is my preferred way of doing it, especially for MX records.
Or you can: 1) Lower TTL before maintenance 2) change records 3) when all is good, increase TTL.
That’s exactly what was said in the above comments
Quoting from https://cr.yp.to/djbdns/tinydns-data.html :
Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.
Beware that cache times below 300 seconds will be treated as 300 by some clients, and NS cache times below 2 seconds can cause lookup failures.
I wonder what those clients are, none of these warnings ever give examples in my experience so I can't tell if this is old out-of-date information being repeated or it is an active modern problem.
I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs (<300s) as errors and "corrected" them by using its default (often 24 hours). I actually encountered it and its effect in the wild, but this was back in the very late 90s or very early 00s. It was a DNS cache daemon on an old Unix variant (I forget which one, my university had a few floating around at the time), so old that if someone is still using it they have far greater issues than my low TTL value!
My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short enough for anything I envisage doing ATM & 300s seems to be reliable) and if someone's DNS cache is broken that is their problem, much like I don't pay any attention to issues Internet Explorer users might have looking at some HTML+CSS+JS I've inexpertly strung together.
I know there was a DNS resolver that did worse than impose a minimum like that, it treat short TTLs (<300s) as errors and "corrected" them by using its default (often 24 hours). I actually encountered it and its effect in the wild, but this was back in the very late 90s or very early 00s. It was a DNS cache daemon on an old Unix variant (I forget which one, my university had a few floating around at the time), so old that if someone is still using it they have far greater issues than my low TTL value!
My current attitude is that if I have the need for short TTLs then I'm going to use them (though 300s is more than short enough for anything I envisage doing ATM & 300s seems to be reliable) and if someone's DNS cache is broken that is their problem, much like I don't pay any attention to issues Internet Explorer users might have looking at some HTML+CSS+JS I've inexpertly strung together.
HughesNet used to run a caching DNS resolver on their customer equipment circa 2016 (not sure if they still do) which overwrote TTLs to 12 hours.
Java/JVM historically ignored TTL and just did a single DNS lookup and cached the result unless configured otherwise.
PGBouncer ignores DNS TTL and has its own configuration.
I've seen lots of software that will create a "connection" object and re-use that with TCP reconnect handling. In practice, that means the DNS resolution happens when the connection is recreated but not necessarily when the TCP connection reconnects due to timeout/failure. I've also seen some issues with long lived connections (where the connection greatly outlives the TTL). Long lived connections aren't necessarily a DNS TTL problem but they mean DNS load balancing/redirection don't always work how you expect.
That said, the solution is usually to just fix the issue in the software and not ignore DNS TTLs.
As counter examples, AWS ALBs and Kubernetes service discovery both rely on short TTLs with fairly good success.
Java/JVM historically ignored TTL and just did a single DNS lookup and cached the result unless configured otherwise.
PGBouncer ignores DNS TTL and has its own configuration.
I've seen lots of software that will create a "connection" object and re-use that with TCP reconnect handling. In practice, that means the DNS resolution happens when the connection is recreated but not necessarily when the TCP connection reconnects due to timeout/failure. I've also seen some issues with long lived connections (where the connection greatly outlives the TTL). Long lived connections aren't necessarily a DNS TTL problem but they mean DNS load balancing/redirection don't always work how you expect.
That said, the solution is usually to just fix the issue in the software and not ignore DNS TTLs.
As counter examples, AWS ALBs and Kubernetes service discovery both rely on short TTLs with fairly good success.
I wonder what those clients are
Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.
Some heavily loaded ISP's have modified daemons that set a lower and upper TTL threshold. Anyone running Unbound DNS can also do this. Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's though I do not have a current table of those misbehaving. The same goes for some IoT's but I have no idea what resolver libraries they are using.
> Anyone running Unbound DNS can also do this
Yep. Just checked the docs for that and “cache-min-ttl” is a thing. For those unfamiliar, unbound is a pretty common resolver: it is the default for most BSDs and things based on them (such as pfSense) and used by the popular PiHole and its forks. How common it is to use this setting I can't comment on.
> Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's
This at least is less of an issue if all you are futzing with the DNS for is services intended to be consumed by a web browser: if common browsers and DNS resolvers behave OK then you are good, if someone consuming your stuff through their own code hits a problem they can work around it themselves :)
It goes without saying that only a fool would play trick like this for email as that is already a mountain of things that can be easily upset by anything off the beaten track.
EDIT:
Also, yep, people are using the setting in their normal environments: https://stackoverflow.com/questions/21799834/how-to-determin... (that person currently having it set to 30 minutes)
Yep. Just checked the docs for that and “cache-min-ttl” is a thing. For those unfamiliar, unbound is a pretty common resolver: it is the default for most BSDs and things based on them (such as pfSense) and used by the popular PiHole and its forks. How common it is to use this setting I can't comment on.
> Some versions and distributions of Java do all manor of odd things with DNS including ignoring TTL's
This at least is less of an issue if all you are futzing with the DNS for is services intended to be consumed by a web browser: if common browsers and DNS resolvers behave OK then you are good, if someone consuming your stuff through their own code hits a problem they can work around it themselves :)
It goes without saying that only a fool would play trick like this for email as that is already a mountain of things that can be easily upset by anything off the beaten track.
EDIT:
Also, yep, people are using the setting in their normal environments: https://stackoverflow.com/questions/21799834/how-to-determin... (that person currently having it set to 30 minutes)
The reason I mention Java is that when updating DNS in a business, there may be business to business flows that are critical. This is becoming more common than ever with cloud-to-cloud-to-cloud dependencies. Mishaps in cloud services can affect a service used by a large number of people. That is why I always tried to teach people to design apps so that TTL could in theory be 1 week and it would not matter. e.g. apex domain anycast IP used for authentication, then direct browsers, API clients and cloud daemons to a service dedicated domain name that is also anycast and uses health checks behind the scenes to take nodes out the picture rather than relying on DNS.
Getting DNS right is non trivial, and UDP will be dropped at times. If you have to lookup the name every two seconds means you have a high risk of getting failed lookups even if your infra team is competent. (10 minutes should be ok)
One cool thing about DJBs tinydns is that you can configure TTLs if you plan to roll over to another system at a particular point in time.
I.e. you can have multiple records each with its own time range where it is valid.
From the tinydns-data documentation:
--
You may include a timestamp on each line. If ttl is nonzero (or omitted), the timestamp is a starting time for the information in the line; the line will be ignored before that time. If ttl is zero, the timestamp is an ending time (``time to die'') for the information in the line; tinydns dynamically adjusts ttl so that the line's DNS records are not cached for more than a few seconds past the ending time. A timestamp is an external TAI64 timestamp, printed as 16 lowercase hexadecimal characters. For example, the lines
--
Do other authorative resolvers support this, too?
From the tinydns-data documentation:
--
You may include a timestamp on each line. If ttl is nonzero (or omitted), the timestamp is a starting time for the information in the line; the line will be ignored before that time. If ttl is zero, the timestamp is an ending time (``time to die'') for the information in the line; tinydns dynamically adjusts ttl so that the line's DNS records are not cached for more than a few seconds past the ending time. A timestamp is an external TAI64 timestamp, printed as 16 lowercase hexadecimal characters. For example, the lines
+www.heaven.af.mil:1.2.3.4:0:4000000038af1379
+www.heaven.af.mil:1.2.3.7::4000000038af1379
--
Do other authorative resolvers support this, too?
Yeah, this works if it is a planned cutover. If you are having to ditch your current IP network in a hurry, then this approach won't work.
Note this data many no longer be correct. I just checked and numbers for some of the websites are much longer now:
raw.githubusercontent.com has 1 hour TTL. github.map.fastly.net is 5 minutes TTL. detectportal.firefox.com is 60/120 seconds TTL.
Another thing missing from the post is exactly _how_ bad is it -- because latency matters. If there is a choice of 30mS extra latency on initial site visit vs a chance of 1 hour of downtime in case of hardware failure, it is not clear what the best choice is. Especially things like detectportal.firefox.com which are asynchronous and normally do not introduce extra user-visible latency at all.
raw.githubusercontent.com has 1 hour TTL. github.map.fastly.net is 5 minutes TTL. detectportal.firefox.com is 60/120 seconds TTL.
Another thing missing from the post is exactly _how_ bad is it -- because latency matters. If there is a choice of 30mS extra latency on initial site visit vs a chance of 1 hour of downtime in case of hardware failure, it is not clear what the best choice is. Especially things like detectportal.firefox.com which are asynchronous and normally do not introduce extra user-visible latency at all.
Fascinating read. Of course I'm no big business but for the last 20 years I've done it like this for non-critical stuff:
And every single time there was some problem, setting the TTL to < 1minute would not have helped, so I guess I'll stick to it.
- NS Records at 86400
- A and CNAME at 3600
- 1-2 days before moving stuff to other infra (ISP, DNS reseller, etc), switch to 600 or lower, then 1-2 days later switch back.
And every single time there was some problem, setting the TTL to < 1minute would not have helped, so I guess I'll stick to it.
The way of measuring this affects (and skews) the outcome. DNS queries with low TTLs are requested more frequently and because of that, the author seeing more of them pass through his patched DNS relay, which was only left running for a few hours instead of for at least the max TTL they wanted to measure.
That's what I thought of as well. Data can be presented in many ways to show many meanings at first sight. If charts would be generated in number of unique NAMEs in relation to TTL, readers would probably get a diffetent feeling. Or maybe not.
I don't know, but this issue probably has a name ... selection bias?
I don't know, but this issue probably has a name ... selection bias?
Yeah,that was one of my first thoughts as well. I guess the raw.github example gets not only more traffic (irrelevant here) but probably also many more requests than hundreds of other sites combined.
Also no clue how actual RR DNS load balancing (without anycast) is still widely practiced, because I guess that also influences it a lot, you may be able to fix a DC going down on a completely different level than DNS. And my experience with cheap home routers is basically that they might cache hostnames for a month (we had stuff with tracking pixels in 2013-2017, EOLing some stuff was wild, we switched DNS and days and weeksl later still got valid requests to the old IPs).
Also no clue how actual RR DNS load balancing (without anycast) is still widely practiced, because I guess that also influences it a lot, you may be able to fix a DC going down on a completely different level than DNS. And my experience with cheap home routers is basically that they might cache hostnames for a month (we had stuff with tracking pixels in 2013-2017, EOLing some stuff was wild, we switched DNS and days and weeksl later still got valid requests to the old IPs).
One reason to use low TTL’s is for a DNS-based load balancer that is typically used for DR such as Azure Traffic Manager. That and paranoia about incorrectly updating a record causing your service to break for a period of time with nothing you can do about it.
There are use cases for low TTLs and use cases for high TTLs
The statement here is misguided.
Use whatever TTL serves your purpose.
DNS was designed to be distributed and scalable. And in my opinion it is THE MOST scalable protocol as it basically allowed the entire internet to work that way. Any user in the world can get any public DNS record for a site.
The statement here is misguided.
Use whatever TTL serves your purpose.
DNS was designed to be distributed and scalable. And in my opinion it is THE MOST scalable protocol as it basically allowed the entire internet to work that way. Any user in the world can get any public DNS record for a site.
Now if only they could have put the components in the right order...
I interpret their data differently: only 47% of DNS lookups were for expired entries due to TTL. If some of those were for longish TTLs, some for actually changed records, then an even bigger majority of requests would be needed anyway, and longer TTLs would not have stopped them.
Is there a push mechanism in DNS?
My uninformed model is that it's all pull based, and for changes to propagate you basically have to wait for cached entries to expire. It seems to me that if there were a cache invalidation mechanism you could have fast updates, but long cache TTLs.
(I know that if this doesn't exist, adding it would be basically impossible, since some servers won't update to new protocols)
My uninformed model is that it's all pull based, and for changes to propagate you basically have to wait for cached entries to expire. It seems to me that if there were a cache invalidation mechanism you could have fast updates, but long cache TTLs.
(I know that if this doesn't exist, adding it would be basically impossible, since some servers won't update to new protocols)
There's DNS NOTIFY (RFC 1996), which is commonly used to trigger a pull. But to minimize malfeasance it's usually ignored unless authenticated (e.g. via DNS TSIG) or appears to originate from a known address. The typical scenario is a master nameserver notifying a list of [known] slaves about a zone update.
There's also DNS Push (RFC 8765), but it's a very recent mechanism. I'm not familiar, but I doubt it's widely supported.
There's also DNS Push (RFC 8765), but it's a very recent mechanism. I'm not familiar, but I doubt it's widely supported.
How would that even work, since you can’t in general know who has cached the entries you control? Broadcast updates to the whole internet?
There is the NOTIFY extension, but it's for master servers to notify secondary servers that they should do a pull. Not really usable for unknown general clients, of course.
An outage at a large (but not mega) company will likely have a 5-10 minute period to get looked at, a 5-10 minute period to understand the problem, and a 5-10 minute period for resolution. Any DNS related problem with a TTL higher than 5-10 minutes could greatly extend an outage.
>The urban legend that DNS-based load balancing depends on TTLs (it doesn’t).
It does. You need to get loaded servers out of what's being advertised. The only way to do this is with a TTL.
It does. You need to get loaded servers out of what's being advertised. The only way to do this is with a TTL.
> DNS caching has become next to useless.
Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.
Another example of the Internet routing around damage. The DNS caching design dates back to the 1980s when it made perfect sense. It no longer does.
It makes a lot of sense, if we think about how the protocol works. Without caching, I assume every client would send every query in it's entirety to the root server, right?
Since a single zone can have multiple labels ({abc.def}.ghi.example - abc.def is a NAME in ghi.example), the server needs to receive the entire requested name in order to properly respond (yeah, root queries usualy optimize this, because all NAMEs there are one label long).
Since a single zone can have multiple labels ({abc.def}.ghi.example - abc.def is a NAME in ghi.example), the server needs to receive the entire requested name in order to properly respond (yeah, root queries usualy optimize this, because all NAMEs there are one label long).
It made as much sense as it does now, but differently from back then, people today neglect the performance and reliability impact it has.
Not really, because infrastructure is much more dynamic today. Maybe if your IPs never changed long TTLs are OK, but when you're running behind a cloud load balancer, with changing IPs and failover to different regions/providers, low TTLs make more sense.
Its understandable that you need to change your routing dynamically, but DNS is not the right tool for that.
I use low TTLs because I don’t like waiting too long when I broke something or did a DNS configuration mistake.
I like changing my IP address from time to time. I host my own VPN and connect to it through a domain. When I update my IP, I have no internet access for whatever the TTL length is.
If you use a pihole, you can force the minimum ttl for downstream clients (upto a maximum value of 3600s) by setting, for example, min-cache-ttl=2400 in /etc/dnsmasq.d/ Never had a problem with this.
Of course, you can force a flush of the cache using `pihole restartdns` in case you need to.
Of course, you can force a flush of the cache using `pihole restartdns` in case you need to.
I'm using Netlify DNS for most of my websites. Usually I set TTL low (~600 seconds) in order to quickly change configuration in case I fuck up. When things are working though, I remember that Netlify doesn't have an edit function for DNS, so I just leave it as is.
IME DNS TTLs are low because debugging DNS during development is a pain.
So you set them low the first time you run into a DNS issue where you had to wait an hour or a day to resolve some bad configuration and your "fuck this" meter went from 0 to 100.
So you set them low the first time you run into a DNS issue where you had to wait an hour or a day to resolve some bad configuration and your "fuck this" meter went from 0 to 100.
Even if the DNS update is available to an end user, if you're using HTTP2 then web browsers will keep the connection to the old IP address until they receive a GOAWAY frame.
Anyone going “just use low TTLs all the time” doesn’t have a lot of traffic to deal with.
I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.
I have a commit of ~15bn queries/month with my DNS provider ATM, and if just one busy record has its TTL set erroneously low, it can cost an extra few thousand $ a month in overages. It’s not a great conversation to have with the CFO when that happens. Caching matters.
If your 15B queries are generated by 15B website visits monthly, your web properties should be generating tens of millions of dollars in monthly income. So a few thousand dollars is nothing.
Unless your traffic is non-revenue generating or somehow poorly monetizable...
Unless your traffic is non-revenue generating or somehow poorly monetizable...
> So a few thousand dollars is nothing.
In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.
In any business doing proper budgeting and trying to make a profit, a few thousands in UNEXPECTED costs, is huge. It can be the difference between your department having the money for new equipment, or begging for a budget increase because of unexpected costs to get new equipment.
> a few thousands in UNEXPECTED costs, is huge.
It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month?
It's a fixed _overhead_ issue, not a fixed _budget_ issue.
It seems like the problem is expecting a fixed infrastructure budget while trying to create ever larger customer engagement. Is it just modern companies that don't realize you can't count your profits until you've got your AWS invoices for that month?
It's a fixed _overhead_ issue, not a fixed _budget_ issue.
You should be reserving those instances so it IS a FIXed cost, with very few exceptions.
There are many AWS services for which you cannot make reservations. You can get better per unit rates by pre-purchasing, but even with instances, network transfer is a per unit factor.
Your only way out is the built in tools to limit service in these cases or to build your own circuit breakers and implement them, or make your usage so small as to not truly require the cloud in the first place.
Your only way out is the built in tools to limit service in these cases or to build your own circuit breakers and implement them, or make your usage so small as to not truly require the cloud in the first place.
Most clients ignore TTLs anyways, so your 1 day TTL barely is helping you're website's performance at all: https://www.ctrl.blog/entry/dns-client-ttl.html
And yet most of that doesn't matter, because you almost always have local resolvers that do cache properly.
Your average consumer router is a pile of garbage that barely works.
I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it.
Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.
I'll admit I don't have the stats in front of me, but I'd bet over half of them either don't attempt caching at all or do a horrible job at it.
Of course, your upstream resolver (hopefully) does do some caching, but as most of the latency is getting to this resolver in the first place that doesn't really matter.
Consumer router doesn't seem to need to given that they typically have such low client counts. Your computer is doing caching, so there's not much benefit for your home router to also do caching.
The latency to most ISP cores is low ms range. You really should just stop digging.
Maybe you live in a country where ISP routers are well designed and low latency, but where I live in the US the primary latency in DNS lookup is almost always the round trip to the horribly configured and slow ISP DNS.
It seems the only reason given is "some clients are broken"?
They mention latency but I wonder what the real world impact is. On a home network, most users are using ISP supplied DNS which /usually/ has pretty low latency and will have a huge cache already built up. Corporate users will have DNS cached to all the common host they're visiting.
Anecdotally, AWS ALBs (HTTP load balancers) have a CNAME with a 10 minute TTL (maybe it is 60 seconds) and can have their IPs change 100s of times a month and this works fine as a default configuration for many use cases.
They mention latency but I wonder what the real world impact is. On a home network, most users are using ISP supplied DNS which /usually/ has pretty low latency and will have a huge cache already built up. Corporate users will have DNS cached to all the common host they're visiting.
Anecdotally, AWS ALBs (HTTP load balancers) have a CNAME with a 10 minute TTL (maybe it is 60 seconds) and can have their IPs change 100s of times a month and this works fine as a default configuration for many use cases.
When I got most of my TLS certs from a commercial CA via OV or EV processes, I could have my cert in place before I repointed DNS.
Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before.
Also, DNS service is a lot cheaper than it was years ago, so it doesn’t hurt my budget to send more requests back to the name servers.
Now most of my certs come from Let’s Encrypt via DV, which checks DNS. So I have to repoint DNS first, and risk users seeing a cert error before certbot finishes getting the new cert. So I keep my DNS TTLs a lot lower than I did before.
Also, DNS service is a lot cheaper than it was years ago, so it doesn’t hurt my budget to send more requests back to the name servers.
That sounds like a server setup issue. Nothing about LE prevents you from getting a cert with the old server and moving it to the new server before switchover.
Use acme.sh, then all of this happens magically. Also, look at ZeroSSL.
Well if you don't like people using low TTLs maybe you should have put validations in the original spec. Oh you didn't write the spec? well sucks to be you then.
Another reason to set a low DNS TTL, is if you are running a service with very limited resources and still want some kind of HA / Disaster recovery. For instance you setup your web app in both Digital Ocean and Linode, with a system that takes the database (for instance mysql replication) and the web app synchronized in the "replica" VM. When something bad happens and you can't recover, you switch DNS and that's it.
As an admin, I use low TTLs when I do some major DNS maintenance. I wouldn't mind upping the TTLs after, but... what about if I need to do some emergency maintenance, like the IPs on a load balancer change and I can't get the old IPs back? I don't want to wait an hour for clients to pick up the new IP.
It would be great if there were some reverse notification to let revolvers know to refresh their caches.
It would be great if there were some reverse notification to let revolvers know to refresh their caches.
I almost always use 24 hours DNS TTL except for things like dynamic DNS. I never felt the need to urgently change DNS.
I've been bitten in the rear far too often with high TTLs. Especially stuff like AWS Beanstalk which tends to delete and recreate resources if you're not extremely careful... thanks but I'll stay with 60s or 300s TTLs.
[deleted]
How would a CDN work if TTL is large? Then you run the risk of using a cached IP that isn't close to you.
Ooops! good point, I should check the DNS TTL for my shelf-hosted email server.
When I worked at reddit, it took a month for traffic to shed off of the old load balancer, despite me lowering the TTL to 5 seconds. And even then there was still some traffic, so I just had to cut it off at that point. I've had similar problems when using DNS based traffic management. Sometimes when we say 50% to one IP and 50% to another, it can end up at 70/30 because of ISP and client caching.
And lastly, my friend who ran an ISP in Alaska told me that they just set all TTLs to 7 days minimum to avoid traffic back to the lower 48.
It doesn't matter what you set your DNS TTL to, so you might as well make it low for your own sake. It's a catch-22. No one plays nice with TTLs anymore because everyone sets them low, and everyone sets them low because no one plays nice.
[0] https://www.ctrl.blog/entry/dns-client-ttl.html