DNS over IPv6: Lessons from the field
We have learned some lessons in the field about DNS over IPv6. The other day, one of our clients called us asking for help with their configurations. They were doing some lab testing while working on their annual upgrade of our software. To give you some background, our customers normally certify each upgrade they intend to use and they do extensive lab testing before they roll it out to their production DNS servers. They do this to make sure it works properly in their environment so that there are no unpleasant surprises. After that our customers typically don’t upgrade or install patches for a year or so. That is what carrier grade software is all about.
The customer’s objection was that their recursive DNS performance results were inconsistent with what they had seen historically, so they asked us for help. Specifically, they couldn’t reach more than 66% of the maximum queries per second (QPS) rate that they had seen a year ago. This was too big of a challenge to solve over the phone, so I visited with them for a few days to get to the bottom of it. The results of what we discovered are interesting so I thought that I would share them on our blog.
It turned out that the IPv6 network was broken in the lab that they were using. The cause of the problem was a few router hops upstream from the DNS server, so it was impossible for the DNS server to detect the issue by looking at the link status or any other indicator locally on the server. What happened was that each time a DNS server had an IPv6 address (such as nic.lagerholm.com), there was a possibility that it would try to use that address over the broken IPv6 interface. Our system is using an algorithm called RTT banding to select what authoritative servers to communicate to. The RTT banding algorithm does not differentiate between IPv4 and IPv6 addresses and uses them equally often if they have the same round trip time. Only after realizing that the server was unreachable over IPv6 does the algorithm take the IPv6 address “out of rotation”. But the damage was already done at that stage.
Once they corrected the IPv6 issue in their lab we could see a significant boost in performance. We immediately achieved performance numbers that were similar to what we had seen last year.
During our investigation we turned off IPv6 completely for a while and made an interesting discovery – running dual stack actually hurts performance a little bit. In our tests, the throughput of a recursive resolver is about 18% slower for a dual stacked server compared to an IPv4 only server. Why is this? Well, it turns out that, just like for clients or any other hosts, the DNS system will have to work harder when IPv6 is enabled. For each nameserver query there are now both A and AAAA addresses that must be considered as targets so additional queries are required to fetch all possible addresses.
What we can learn from the experience above is that you need to take into account a performance hit on the maximum QPS your DNS system can cope with when turning on IPv6. And don’t turn on IPv6 in your recursive DNS servers unless your IPv6 connectivity is working properly.
To learn more about DNS solutions Contact us.
