Absolutely correct on the privacy. We're actually dropping everything from Google / Facebook and anything third-party at the moment (since it's not technically legal in Europe anyway). It will be a bit of a process though.
We don't strictly use an IP DNS database to figure out the location, but a series of factors that make it much more accurate than standard GeoDNS. Something we've had to do to improve our own routing for the CDN. We're constantly looking for feedback and how to solve problems, so make sure to reach out. Your feedback is what allows you to improve.
That was actually the original idea, then for compatibility reasons, we decided to take this path initially. We are exploring adding fallback support as well, given that it greatly simplifies things and DNS can't really afford to do a whole lot of waiting anyway.
Our goal was not to use this to orchestrate a global CDN, it's just one of the many use-cases and can be combined with anycast easily. This is just one example and we're not really trying to push GeoDNS specifically. What we envision is smart load balancing, backups, dynamic routing for validation or service discovery and a bunch of other things that are currently very painful to do. GeoDNS is an old concept and can be solved much better without scripting anyway.
In terms of accuracy, it is generally very accurate on a country level. Anything else is pretty useless.
I see, my bad. I guess I was a bit hasty since the metrics kind of bothered me.
Still, regarding the other point. Any thoughts on that? We've been monitoring performance of various CDNs ourselves, mainly to improve our own performance through our own RUM metrics as well as platforms such Citrix and PerfOps, and while CloudFlare is definitely one of the fastest, it's definitely not the fastest as per your results. For example CacheFly and Limelight seem to be excelling in the US way ahead of others.
Limelight being a multi-hundred million dollar revenue CDN was somehow excluded entirely.
So the "All networks" TTLB metric for example says CloudFlare is #1 in 5000+ networks, other CDNs are faster in almost 15.000 other networks, meaning CloudFlare is only the fastest in roughly 30% of the networks.
Then the blog post ends by saying CloudFlare is not the fastest in 10% of cases which seems simply false.
The metric selection as well seems to be selected very much in a way where it's skewing the actual results into making them be much better than they are. A much better result would be to actually show the actual global median response times, which would likely be extremely close to each other between all major providers. Meanwhile this post attempts to make CloudFlare look multiple X better than Akamai for example.
Looking at the post in general it seems that it's very much aimed at being as misleading as possible.
I do! In fact, I work like 90 hours a week. I decided to go the bootstrap way (looking back not sure if that was the best idea, but we are where we are), but we're growing 3X year over year, so things are picking up. :)
Hey, glad to hear that and sorry again about any issues. If you're experiencing any ongoing problems, please message our support team. I'm not aware of anything actively broken, but if there's a problem I'm sure we'll be able to help.
For critical systems (or let's call them services) such as DNS, CDN, optimizer, storage, we usually deploy either on a server to server basis, regional basis or cluster basis before going live. What I mean here was that this was not really a critical service that nobody thought could actually cause any harm, so we didn't do canary testing there as it would add a very high level of complexity.
Right, this was our biggest failure (not the only one of course, but we are here to improve). Relying on our own systems to maintain our own systems.
We are dropping BinaryPack mainly because we're a small team, and it wasn't really a big benefit anyway, so spending more time than necessary to try and salvage that makes no sense. This was more of a hot-fix since we don't want the same thing repeating in a week.
Dejan here from bunny.net. I was reading some of the comments, but wasn't sure where to reply, so I guess I'll post some additional details here. I tried to keep the blog post somewhat technical, but not overwhelm non-technical readers.
So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructive. Unfortunately this deployment went to a system that we believed was not a critical part of infrastructure (oh look how wrong we were) and was not made redundant, since the rest of the code was supposed to handle it gracefully in case this whole system was offline or broken.
It was not my intention to blame the library, obviously this was our own fault, but I must admit we did not expect a stack overflow out of it, which completely obliterated all of the servers immediately when the "non-critical" component got corrupted.
This piece of data is highly dynamic and processes every 30 seconds or so based on hundreds of thousands of metrics. Running a checksum did nothing good here, because the distributed file was perfectly fine. The issue happened when it was being generated, not distributed.
Now for the DNS itself, which is a critical part of our infrastructure.
We of course operate a staging environment with both automated testing and manual testing before things go live.
We also operate multiple deployment groups so separate clusters are deployed first, before others go live, so we can catch issues.
We do the same for the CDN and always use canary testing if possible. We unfortunately never assumed this piece of software could cause all the DNS servers to stack overflow.
Obviously, I mentioned, we are not perfect, but we are trying to improve on what happened. The biggest flaws we discovered were the reliance on our own infrastructure to handle our own infrastructure deployments.
We have code versioning and CI in place as well as the options to do rollbacks as needed. If the issue happened under normal circumstances, we would have the ability to roll back all the software instantly, and maybe experience a 2-5 minute downtime. Instead, we brought down the whole system like dominos because it all relied on each other.
Migrating deployment services to third-party solutions is therefore our biggest fix at this point.
The reason we are moving away from BinaryPack is because it simply wasn't really providing that much benefit. It was helpful, but it wasn't having a significant impact on the overall behavior, so we would rather stick with something that worked fine for years without issues. As a small team, we don't have the time or resources to spend improving it at this point.
I'm somewhat exhausted after yesterday, so I hope this is not super unstructured, but I hope that answers some questions and doesn't create more of them :)
If I missed any suggestions or something that was unclear, please let me know. We're actively trying to improve all the processes to avoid similar situations in the future.