What is necessarily provocative about the term? Both the noun and verb forms of the word "desert" fit well here, especially in the context of the history that this article talks about.
"If you don't have a car ... you are going to have a significantly limited life in so many ways that having to drive to the supermarket just seems like a triviality in comparison." Except that everyone needs to eat, and we know that a poor diet can lead to many, many bad health outcomes. You're correct that there are other disadvantages, such as access to employment opportunities. But if you didn't live in a food desert, you could get a job at the grocery store, right?
"people just like, have cars, it's normal"
Except that owning (or leasing) a car is a significant, ongoing expense. I barely use mine (<1,000 miles per year) and it still costs me $2k in insurance, registration, and maintenance. Some people just can't afford that. And what if you are young, old, or are unable to drive for any reason?
Thank you! I think this is the first time I've logged in for 10 years.
I don't mind the current UI change, but I wish significant changes would come with a toggle button to let me look at the old and new renderings. I don't think it's possible to mentally "place" the improvements without side-by-side comparisons and affordances for finding edge cases where the new UI may be lacking.
- There's nothing scarier than "the automation had no rate-limiting or health-checking". Of course, what do we mean by automation? At some point it becomes impractical to slow every change down to a crawl, so some judgement is required. But "slow enough to mount a human response to stop it" is the standard I've applied to critical systems.
- Thankfully I've avoided having to support "real" storage systems. The challenges of "simple" distributed databases storing is enough for me. :-)
On the "pets vs. cattle" metaphor, I think most people fail to grok the second component of that. I don't think there are many successful cattle ranchers that will just "take the cattle out of line and shoot it in the head." The point of the metaphor is: When you have thousands of head of cattle, you need to think about the problems of feeding and caring for cattle operationally, not as a one-off.
Despite what https://xkcd.com/1737 might make one believe, people don't just throw out servers when one part goes bad, or (intentionally) burn down datacenters. What the "hyperscalers" do is decouple the operational challenges of running machines from the operational challenges of running services (or at least try to). Of course this results in a TON of work on both ends.
I'm going to guess somewhere between 50 and 400 people. Smaller than 50: this is probably an org where "we maintain the source code and documentation. Running, debugging, and scaling your cluster is YOUR problem". Larger than 400: the org probably fulfills other functions besides "running thousands of production Cassandra clusters."
This is a lot, and is one of the big things that surprised me when I first joined a large organization. But here's an example breakdown in no particular order:
- 10 to 30 people for managing, contributing, and maintaining the open source aspect. Just a handful of engineers could contribute features and fixes to the project, but once the open source project and community gets larger it becomes a full time job. System component maintainers, foundation boards, committees, conferences, etc. add up.
- 10 to 70 people for "operations". As you mentioned, the load here tends to scale with the number of clusters (customers). At the large end this is several teams, with say a team dedicated to (a) fleet-management of the individual machines in a cluster, (b) cluster lifecycle management, and (c) macro level operations above the cluster level. Alerts can't all go to this team, so some of the work is writing alerts that go to customers in a self-serve model.
- 10 - 40 people for "scale projects." At this scale you have 1 - 10 customers that are on the verge of toppling the system over. They've grown and hit various system bottlenecks that need to be addressed. And you'd be lucky if they're all hitting the same bottleneck. With this many customers, it's likely that they've all adopted orthogonal anti-patterns that you need to find a fix for: too many rows, too many columns, bad query optimization, too many schema changes, too many cluster creates and destroys, etc. So you probably have multiple projects ongoing for these.
- 10 - 30 people for "testing infrastructure". Everyone writes unit tests, but once you get to integration and scale testing, you need a team that writes and maintains fixtures to spin up a small cluster for some tests and a large cluster for scale tests (which your "scale projects" teams need, btw). And your customers probably need ways of getting access to small test Cassandra clusters (or mocks of the same) for THEIR integration and scale tests, since Cassandra is just a small part of their system.
- 10 - 30 people for automating resource scaling and handling cost attribution. These may not be one function, but I'm lumping them together. "Operations" might handle some of the resource scale problems, but at some point it's probably worth a team to continually look for ways to manage the multi-dimensional scaling problem that large cluster software systems inevitably create. (Is it better to have few large nodes, or many small nodes?) You need some way of attributing cost back to customer organizations, otherwise you're paying $50M because one engineer on the weather team forgot to tear down a test cluster in one automated test 6 months ago and... You need to make sure that growth projections for customers are defined and tracked so you have enough machines on hand.
- I'll add that it'll be worth adding whole teams for some of the more complex internal bits of this system, even if the actual rate of change in that sub-system is not very high. At this scale organizations need to optimize for stability, not efficiency. You don't want to be in the situation where the only person who understands the FizzBuzz system leaves and now dozens of people/projects are blocked because nobody understands how to safely integrate changes into FizzBuzz.
- Things not covered: security, auditing, internal documentation, machine provisioning, datacenter operations, operating system maintenance, firmware maintenance, new hardware qualification, etc. Maybe there's an entire organization dedicated to each of these, in which case you get it for free. If not, some of your time needs to be spent on these. (Even "free" might have a cost as you need to integrate with those services and update when those services change.)
"If you don't have a car ... you are going to have a significantly limited life in so many ways that having to drive to the supermarket just seems like a triviality in comparison." Except that everyone needs to eat, and we know that a poor diet can lead to many, many bad health outcomes. You're correct that there are other disadvantages, such as access to employment opportunities. But if you didn't live in a food desert, you could get a job at the grocery store, right?
"people just like, have cars, it's normal" Except that owning (or leasing) a car is a significant, ongoing expense. I barely use mine (<1,000 miles per year) and it still costs me $2k in insurance, registration, and maintenance. Some people just can't afford that. And what if you are young, old, or are unable to drive for any reason?