Dealing with IPv6 fragmentation in the DNS(blog.apnic.net)
blog.apnic.net
Dealing with IPv6 fragmentation in the DNS
https://blog.apnic.net/2017/08/22/dealing-ipv6-fragmentation-dns/
44 comments
I agree that switches shouldn't care about TCP or UDP ports. That is a job for a firewall.
However, that does not mean it is a good idea to have variable length headers in IPv6. In fact it seems like a singularly bad idea to me.
Regarding firewalls, I imagine it still sucks a firewall cannot judge each IP-packet separately but needs to deal with possible fragmented packets. Then again, that is the cost of being a firewall. You need to function at the application level and yet decide on the IP-packet level.
Regarding firewalls, I imagine it still sucks a firewall cannot judge each IP-packet separately but needs to deal with possible fragmented packets. Then again, that is the cost of being a firewall. You need to function at the application level and yet decide on the IP-packet level.
Switches use L4 ports to implement flow-switching. When multiple paths for a packet are available, it is convenient (it leads to less re-ordering) if the related packets from the same flow all choose the same path.
It's not just convenience for anycast addresses, with multiple pathes and anycast if your round robin packets across different routes, you may end up receiving them in two different locations (ok for UDP, but TCP broken completely).
Anycast is widely deployed now and imho ECMP support is now considered "standard".
I'm not sure what are other alternative solution to ECMP that don't require looking into TCP headers.
I don't see an issue with extension headers in IPv6. The RFCs have rules on who should look at what header and what to do when an unrecognized one is encountered.
Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.
Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.
> another painful issue in today’s IPv6 Internet, namely that of network filters discarding ICMPv6 Packet Too Big messages.
How about we just let the people who configure these filters solve their own problems?
How about we just let the people who configure these filters solve their own problems?
I've actually had this conversation before, and it get's a bit tricky.
I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not.
- It's not perceived to be a problem. Most services just work, so TCP/IP with mss clamping will avoid the fragmentation issue for many services. So the services that have an issue, might be something like 2%, and the question that always lingers is if everything else works, and this one thing doesn't, it's not the filters problem.
- ICMP doesn't necessarily work with multiple layers of encapsulation anyways I encountered this on mobile wireless networks, where doing multiple rounds of encapsulation caused multiple packet size reductions. Basically, if we encapsulate in tunnels used by the mobile network, then encapsulate again in say IPSec, the IPSec node will send it's icmp packet too big to the tunnel endpoint, not the originator of the traffic.
While I suppose it could be argued that the tunnel endpoint should maintain it's own PMTU, and pro-actively send it's own icmp packet too big based on this, it can be difficult to guarantee these components work.
I've seen lots of telco equipment struggle with handling this properly, and it's not just filtering, it's bugs.
So unfortunately, we're in a situation where many networks are likely miss-configured, but lots of other services for various reasons just work. So unless you can drive the networks with problems to actually change, realistically you just lose a portion of your own customer base if you don't handle the situation while your competitors just work.
I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not.
- It's not perceived to be a problem. Most services just work, so TCP/IP with mss clamping will avoid the fragmentation issue for many services. So the services that have an issue, might be something like 2%, and the question that always lingers is if everything else works, and this one thing doesn't, it's not the filters problem.
- ICMP doesn't necessarily work with multiple layers of encapsulation anyways I encountered this on mobile wireless networks, where doing multiple rounds of encapsulation caused multiple packet size reductions. Basically, if we encapsulate in tunnels used by the mobile network, then encapsulate again in say IPSec, the IPSec node will send it's icmp packet too big to the tunnel endpoint, not the originator of the traffic.
While I suppose it could be argued that the tunnel endpoint should maintain it's own PMTU, and pro-actively send it's own icmp packet too big based on this, it can be difficult to guarantee these components work.
I've seen lots of telco equipment struggle with handling this properly, and it's not just filtering, it's bugs.
So unfortunately, we're in a situation where many networks are likely miss-configured, but lots of other services for various reasons just work. So unless you can drive the networks with problems to actually change, realistically you just lose a portion of your own customer base if you don't handle the situation while your competitors just work.
> ICMP is dangerous, and needs to be blocked
I've fought this argument countless times and lost. Not being able to ping a server, check for "destination unreachable" messages, etc. leads to frustrating troubleshooting sessions.
I've fought this argument countless times and lost. Not being able to ping a server, check for "destination unreachable" messages, etc. leads to frustrating troubleshooting sessions.
> I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not.
It's actually quite easy to identify which should be allowed and which should not.
ICMP: 0,3,8,11 (echo reply, destination unreachable, echo, time exceeded)
ICMP6: 1,2,3,4,128,129,135,136 (unreachable, packet too big, time exceeded, parameter problem, echo request, echo reply, neighbor solicitation, neighbor advertisement)
See, that wasn't hard. These are the base requirements for you to have a reasonably functional IPv4 network, and the absolute minimum requirements for IPv6 to work properly.
It's actually quite easy to identify which should be allowed and which should not.
ICMP: 0,3,8,11 (echo reply, destination unreachable, echo, time exceeded)
ICMP6: 1,2,3,4,128,129,135,136 (unreachable, packet too big, time exceeded, parameter problem, echo request, echo reply, neighbor solicitation, neighbor advertisement)
See, that wasn't hard. These are the base requirements for you to have a reasonably functional IPv4 network, and the absolute minimum requirements for IPv6 to work properly.
But why filter ICMP at all? I can understand that ICMP allows for covert tunnelling, but by that logic any IP protocol number should be blocked.
I think it's more the incoming ICMP that is troublesome, particularly redirect, and to a lesser extent destination unreachable (DOS).
Under windows you cant whitelist applications allowed to use ICMP, all ICMP traffic originates from deep down ring 0 NT Kernel process.
because some ICMP types are actually dangerous and can be abused
Agreed.
It's just the argument that gets made for filtering, is we'd rather just filter it all instead of trying to understand what should and should not be filtered.
It's just the argument that gets made for filtering, is we'd rather just filter it all instead of trying to understand what should and should not be filtered.
That's fine enough when that only directly affects those people or their networks. The article mentions Google public dns having this problem. If your customers are using them, and you're returning large DNS responses over ipv6, your customers will have trouble with resolving you names, and blame you, and you may not even be able to tell it's Google's fault.
Having a variable header complicates Cut-Through switching[0], which forwards the entire frame just after the switch gets the destination field of the header.
This sounds like it would force a return to Store-And-Forward[1], where the switch waits for the entire frame to load into memory, before forwarding to its next hop. Waiting these few milliseconds doesn't sound bad, until you consider that's added to each packet, reducing (e.g.) application performance.
[0] https://en.wikipedia.org/wiki/Cut-through_switching
[1] https://en.wikipedia.org/wiki/Store_and_forward
This sounds like it would force a return to Store-And-Forward[1], where the switch waits for the entire frame to load into memory, before forwarding to its next hop. Waiting these few milliseconds doesn't sound bad, until you consider that's added to each packet, reducing (e.g.) application performance.
[0] https://en.wikipedia.org/wiki/Cut-through_switching
[1] https://en.wikipedia.org/wiki/Store_and_forward
The way cut-through is really implemented is that the first N bytes of the packet (usually 128 or 256) are treated as the "header" and are passed through the header pipeline regardless of how big the actual header is. So variable headers like VLAN tags, MPLS stacks, or IP extension headers shouldn't prevent cut-through.
Also, the difference between cut-through and store-and-forward is generally in single-digit microseconds, not milliseconds. A good rule of thumb is ~1 us/KB at 10G.
Also, the difference between cut-through and store-and-forward is generally in single-digit microseconds, not milliseconds. A good rule of thumb is ~1 us/KB at 10G.
So there was a huge argument of CT vs SF switching way back when. IIRC there was Kalpana (CT) which Cisco acquired and Granite (Cat4K, SF) and Crescendo (Cat5/6K, SF). Cisco killed cut-through. The 1st real ASIC that did cut-through since then was the FM4000 (Fulcum) which was used in the Arista 7124, BNT, etc. The FM4000 latency was 600ns vs. 6 microseconds for the standard Wall Street switch at the time (Cat4900). 600ns to 6 microseconds is huge.
The FM4000 looked at 64 bytes before forwarding (IIRC). The 1st Broadcom chips that did cut-through were the Trident line. They looked at something like ~200 or ~500 bytes before making a forwarding decision (sorry exact number is not in cache anymore). They cut that down with the T+ chips.
The funny issue about cut-through is that it will forward a corrupted packet. Trying to pinpoint the source can be fun.
With static webpages, etc, old school internet the latency was not a big deal. The improvements in switching with cut-through and low latency were a game changer for storage, video, etc.
The FM4000 looked at 64 bytes before forwarding (IIRC). The 1st Broadcom chips that did cut-through were the Trident line. They looked at something like ~200 or ~500 bytes before making a forwarding decision (sorry exact number is not in cache anymore). They cut that down with the T+ chips.
The funny issue about cut-through is that it will forward a corrupted packet. Trying to pinpoint the source can be fun.
With static webpages, etc, old school internet the latency was not a big deal. The improvements in switching with cut-through and low latency were a game changer for storage, video, etc.
You seem to be confusing IP headers with Ethernet headers. Cut-through switching only looks at the Ethernet header, which is not variable, and thus will still be perfectly viable.
Totally different layers. Switching occurs at layer 2 and isn't affected by whether the frame contains IPv4 or IPv6.
I was confused at first, because switches were mentioned in the article. But the Fates have allowed my mistake to be correct this one time, and provided a trove of education about the state of the art of network switching.
It would be nice if that was the case, but switches often look deeper for ECMP.
The real problem is that you can no longer look at a fixed offset for your parameters, which complicates filtering silicon.
In other words, if you want to have a firewall rule of "drop tcp 135", in ipv4 you can just look at bytes 0x24-25 (for Ethernet). IPv6's extension-header mechanism means that the header has a non-fixed length, so you need to do work for the same effect. Given that firewalls are critical in a world without NAT, this can be a scary prospect.
Part of the blame falls on vendors who go "well, IPv6 is fringe so we'll just do all firewalling in software." The protocol is 20 years old, it's time to design the silicon.
In other words, if you want to have a firewall rule of "drop tcp 135", in ipv4 you can just look at bytes 0x24-25 (for Ethernet). IPv6's extension-header mechanism means that the header has a non-fixed length, so you need to do work for the same effect. Given that firewalls are critical in a world without NAT, this can be a scary prospect.
Part of the blame falls on vendors who go "well, IPv6 is fringe so we'll just do all firewalling in software." The protocol is 20 years old, it's time to design the silicon.
But IPv4 headers could have a variable length too, it's just that we don't encounter that much in the wild. And what about the AH header, GRE and IPIP tunnel headers, 6rd/6in4 tunnel headers, etc. Filtering at fixed offsets sounds very brittle to me.
Isn't the real problem here that network providers and administrators appropriate the right to filter on OSI layers they shouldn't be touching? This problem sounds like the exact reason why Google insisted on having its QUIC headers ciphered: so network equipment can't pull of this kind of misbehaviour.
At least with IPv4 you can easily compute the start of layer 4, it's just ipv4[0]&0xf * 4 - trivially implemented with little real-estate in hardware.
IPv6 requires you to loop through all extension headers to reach layer 4, and you need to know about a handful of those extension headers as not all follow the same format.
IPv6 requires you to loop through all extension headers to reach layer 4, and you need to know about a handful of those extension headers as not all follow the same format.
Most enterprise firewalls these days do some level of layer 7 packet inspection which requires going beyond looking at fixed offsets. The silicon to support this already exists. Even the $50 Ubiquiti ER-X router I use at home can do deep packet inspection assisted by silicon.
It sounds like IPv6 acknowledges that most protocols try hard to avoid fragmentation anyway, so fragmented packets are the exception.
And it's not like fragmentation hasn't caused problems with firewalls in IPv4 either.
E.g. http://all.net/Analyst/netsec/1995-09.html
The fix is for equipment to handle the fragmented packets properly, just like for IPv4. How is forward fragmentation, which involves creating multiple new packets + headers with recomputed checksums, easier than simply parsing a header, which is apparently "anathema to a switch"?
And it's not like fragmentation hasn't caused problems with firewalls in IPv4 either.
E.g. http://all.net/Analyst/netsec/1995-09.html
The fix is for equipment to handle the fragmented packets properly, just like for IPv4. How is forward fragmentation, which involves creating multiple new packets + headers with recomputed checksums, easier than simply parsing a header, which is apparently "anathema to a switch"?
The problem is that the fragmentation header in IPv6 is not deterministic, so it can't be parsed in constant time, which is what makes it "anathema to a switch" which needs to be able to guarantee certain levels of packet throughput. IPv4-style fragmentation can be done in constant time.
It is still bounded time, which is still O(1).
Yes, the standard doesn't limit the number of extension headers, but looking at the fixed header, and at most 5-6 extenion headers is the maximum time required to process a packet. And that'd be enough for 100% of traffic.
The exact limit is the number of header types known to the processing node. Because encountering an unknown header should result in Parameter Error, and only one header can be used multiple times (the Destination Options), but that should be after the Frag one anyway.
Yes, the standard doesn't limit the number of extension headers, but looking at the fixed header, and at most 5-6 extenion headers is the maximum time required to process a packet. And that'd be enough for 100% of traffic.
The exact limit is the number of header types known to the processing node. Because encountering an unknown header should result in Parameter Error, and only one header can be used multiple times (the Destination Options), but that should be after the Frag one anyway.
[deleted]
> The initial connection between Cloudflare's network and the origin web server timed out. As a result, the web page can not be displayed.
Sad to see even a NIC letting itself be MITMed by Cloudflare :(
Sad to see even a NIC letting itself be MITMed by Cloudflare :(
Maybe we should bow to the inevitable and recognise that IPv6 is an unfixable problem.
[deleted]
if it's unfixable, why is 40% of our in-office internet traffic running over IPv6? Why is 100% of our LAN traffic over IPv6? Why is 30% of the traffic to our sites running over IPv6?
The article suggests "Fragmentation in IPv6 is unfixable", not ipv6 itself. Try running this to see if fragments get delivered to you:
http://icmpcheckv6.popcount.org
http://icmpcheck.popcount.org (IPv4 version)
via: https://blog.cloudflare.com/ip-fragmentation-is-broken/
http://icmpcheckv6.popcount.org
http://icmpcheck.popcount.org (IPv4 version)
via: https://blog.cloudflare.com/ip-fragmentation-is-broken/
The comment I was responding to asserted that IPv6 as a whole is unfixable which is not true.
With regards to fragmentation, I would say "fragmentation in general is unfixable", no matter the protocol. Thankfully with IPv6, the guaranteed minimal packet size is relatively reasonable (1280 bytes), so maybe it's now time to finally go there.
v6 already changed the rules with fragmentation a bit in that routers won't have to fragment any more but are allowed to just drop packets smaller than the MTU and sending back an ICMPv6 packet announcing that they have done so.
Unfortunately, ICMP is still dropped at many firewalls even though I would argue that it's absolutely essential for IPv6 functionality (when it was more optional in v4).
As it stands now, hosts are still allowed to fragment packets themselves (which is what's happening in this article), but the solution is probably to not do that either in the future.
DNSSec over UDP is a problem in that regards, but people aren't convinced that DNSSec is the way to go anyways (just ask tptacek here on this site).
(edit: with regards to the fragment checking site you pointed me to: In our case, all is well because I do recognise the necessity of ICMPv6 and thus have configured the firewalls accordingly)
With regards to fragmentation, I would say "fragmentation in general is unfixable", no matter the protocol. Thankfully with IPv6, the guaranteed minimal packet size is relatively reasonable (1280 bytes), so maybe it's now time to finally go there.
v6 already changed the rules with fragmentation a bit in that routers won't have to fragment any more but are allowed to just drop packets smaller than the MTU and sending back an ICMPv6 packet announcing that they have done so.
Unfortunately, ICMP is still dropped at many firewalls even though I would argue that it's absolutely essential for IPv6 functionality (when it was more optional in v4).
As it stands now, hosts are still allowed to fragment packets themselves (which is what's happening in this article), but the solution is probably to not do that either in the future.
DNSSec over UDP is a problem in that regards, but people aren't convinced that DNSSec is the way to go anyways (just ask tptacek here on this site).
(edit: with regards to the fragment checking site you pointed me to: In our case, all is well because I do recognise the necessity of ICMPv6 and thus have configured the firewalls accordingly)
Works great for me on my laptop. I don't doubt for a second that there are problems (my phone doesn't get ipv6 at all and one of the ipv4 tests failed... but mobile networks are notorious when it comes to filtering traffic), but "unfixable"?
Network I am on atm, IPv6 give me both green, IPv4 MTU fails ...
It means my IPv6 vpn is better than IPv4 delivered by some wannabe corpo network masters.
It means my IPv6 vpn is better than IPv4 delivered by some wannabe corpo network masters.
the firewall is probably blocking ICMP. For a long time, it was considered best-practice to blanket-block ICMP and for a long time you could mostly get away with it.
But over time some really useful stuff has been added to ICMPv4 (early congestion notification for example) and IPv6 mostly won't work at all without ICMP.
I would argue that it's time for firewall administrators to reconsider this decision. The time where ICMP packets could do harm are long past (remember the ping-of-death on Windows 98?) and I would argue that the time when a network could work at peak efficiency (or at all) without ICMP are also mostly past.
But over time some really useful stuff has been added to ICMPv4 (early congestion notification for example) and IPv6 mostly won't work at all without ICMP.
I would argue that it's time for firewall administrators to reconsider this decision. The time where ICMP packets could do harm are long past (remember the ping-of-death on Windows 98?) and I would argue that the time when a network could work at peak efficiency (or at all) without ICMP are also mostly past.
> For a long time, it was considered best-practice to blanket-block ICMP
I agree with your point, but think this needs some clarification about who considered it a best practice - among enterprise firewall admins and financial-industry compliance audits, perhaps.
Large scale network operators have seen the problems associated with entirely blocking ICMP for at least a decade. (v4 has blackhole problems too)
A lot of the hatred for ICMP came from simple things like easy endpoint discoverability. nmap blew that argument out of the water pretty quick.
I agree with your point, but think this needs some clarification about who considered it a best practice - among enterprise firewall admins and financial-industry compliance audits, perhaps.
Large scale network operators have seen the problems associated with entirely blocking ICMP for at least a decade. (v4 has blackhole problems too)
A lot of the hatred for ICMP came from simple things like easy endpoint discoverability. nmap blew that argument out of the water pretty quick.
Totally agree, ICMP is needed.
When it's not blocked, you can debug and have healthy network, instead of a frankensteins monster.
When it's not blocked, you can debug and have healthy network, instead of a frankensteins monster.
Seems to work for me (?).
For the reasons outlined in the article. Most DNS responses aren't fragmented, and IPv4 covers up failures where they are.
The comment I was responding to asserted that IPv6 in general is unfixable whereas the article was only talking about IPv6 fragmentation where I'd say it's totally correct.
But in general, fragmentation is a bad idea and we should not have gone there in the first place.
DNSSec should only ever have been specified over TCP (or not at all if you ask some people like tptacek here on this site)
But in general, fragmentation is a bad idea and we should not have gone there in the first place.
DNSSec should only ever have been specified over TCP (or not at all if you ask some people like tptacek here on this site)
If it is so much anathema to their nature then maybe they shouldn't insist on violating network layers?