That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.
There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.
Team has a good sense, typically. In this case, the names of the columns in the Bot Management feature table seemed sensitive. The person who included that in the master document we were working from added a comment: “Should redact column names.” John and I usually catch anything the rest of the team may have missed. For me, pays to have gone to law school, but also pays to have studied Computer Science in college and be technical enough to still understand both the SQL and Rust code here.
We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.
Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
I have plenty of friends who went down that path. They’ve done very well as lawyers. But suffice it to say that if offered they’d readily trade places.
Those 8 years were painful. To make money, I worked as a bartender, an LSAT test prep instructor, as an adjunct law professor at a law school that was so bad it doesn’t exist anymore. I remember 4am at the bar in Chicago where I worked, cleaning up some patron’s puke off the floor, and thinking: I need to figure something else out.
All the time I was trying to find an idea for a startup. I still had the lawyer bit flipped on so lots of things I tried had a legal/regulatory bent. That was definitely a blind spot that held me back for a while.
The fun YC-related story on the founding of Cloudflare is that, before YC, Paul Graham used to host a conference called the “MIT Anti-Spam Conference.” He invited me the second year of the conference (2003, I think) to give a talk on how to write effective anti-spam laws. The very technical crowd was polite to the lawyer. I met a ton of interesting people, many of whom played outsized roles in machine learning over the next few years, including John Graham-Cumming, now Cloudflare’s CTO. Paul invited me back the following year saying I should do something similar.
I was pretty sure the audience wouldn’t tolerate the lawyer giving another talk about regulation, so I went to a young engineer on the team of the (bad) startup I was working on and suggested we build a system to track how spammers scrape your email addresses. He agreed to build the backend if I built the front end (which I largely stole from the hot startup of the time: LinkedIn). That turned into Project Honey Pot, which I gave a talk on at Paul’s conference. Project Honey Pot gave the initial seed of an idea that turned into Cloudflare. And the young engineer was Lee Holloway who cofounded Cloudflare with me and Michelle Zatlyn.
Lesson to me has always been even in times where you don’t feel like you’re making forward progress in your life and career, find ways to stay involved with interesting people and projects and chances are they’ll pay dividends in ways you don’t expect later in life.
I clearly remember walking back to Paul’s house in Cambridge after the 2004 conference where I’d presented Project Honey Pot. I believe he and Jessica had relatively recently started dating. They were talking about startups and how people didn’t understand how they worked. Paul suggested they should teach a class at MIT. And that, of course, is what later turned into YC.
There were other dramatic events that evening in Cambridge that I think sharpened all our minds and made us appreciate there’s no time like the present, but I’ll leave that story for another day.
Had the dotcom bubble not burst I’d likely be an attorney. I’d accepted an offer at a law firm in San Francisco to work in their Securities practice, largely taking companies public or doing M&A.
In March of 2000, the firm called and said: “Good news bad news. Good news: you still have a job [unlike a lot of my law school classmates]. Bad news: we don’t need any more Securities lawyers, but we have lots of room in our Bankruptcy practice.”
Being a Bankruptcy lawyer didn’t sound like fun. A law professor’s brother was starting a B2B startup. He offered me a job. The startup was a colossal failure, but I was hooked on the idea of a group of people starting something from nothing.
Next ~8 years were painful with lots of ideas that went no where, but it all worked out. So, in the end, always remember that but for the dotcom bubble bursting, I’d be keeping track of my time in six minute increments.
DNS either has integrity or it doesn’t. We get a response from an Authoritative server and, as a Resolver, we believe our responsibility is to return it. If we start making exceptions because of bad PR, how can you trust us to do the right thing when the stakes are even higher (e.g., nationstate pressure)?
As an aside, I used to think that when Emerson said that “a foolish consistency is the hobgoblin of little minds” he meant that we were foolish to try and be consistent. Increasingly I wonder if instead he meant that when you’re trying to reason with people who may not have the same detailed knowledge of a problem as you, there’s an enhanced importance to being consistent. Unfortunately, most policy makers globally don’t have a detailed understanding of how technical systems like DNS work, so we think it’s especially important we be consistent.