Translated: “I have over 64,000 karma and am somehow surprised I have an unfavorable reputation in a regular reader, and my reaction to that is to get feisty.”
You’re right though, I’ll drop that, that was unfair and unnecessary. I apologize.
I couldn’t imagine it, no, because I care more about the number of hairs on my scrotum than whether you consider me boring or, really, any larger opinion you’d have of me at all. HN opinions of my character are like fruit flies: annoying, tiny, and wholly irrelevant once disposed of appropriately.
“Those people up there sleep 15 to a room and have a sustenance existence. They aren’t doing it to be a cultural attraction, they actually have to fish to live. How sad. I immediately ran back to the other people like me to breaking news bulletin that there are other people in the world who don’t obsess about Javascript frameworks and convertible notes.”
“I spent some time with indigenous communities who still support themselves the same way they have even before my ancestors hit this continent, and I sat down with them, and listened to stories and partook of their hospitality. I still correspond with some of the friends I met, and I’m rounded out by the perspective I gained in people who are not like me.”
Think about why you typed the word “bad” behind conditions, or why aviators typed “sad”. This is the entire crux: it’s bad or sad in your culture and perspective. I dated a lovely woman who descended from the very Inuit tribes you two are hand waving at, and I have met no group of people more generous nor proud of their heritage, while acknowledging the difficulty of integrating into white people customs, than them. There is absolutely no consideration among either of you that you might be lacking perspective, might have something to learn, and instead sit back on your expensive computer judging a group of people you’ve never touched and explaining them. That is completely and irredeemably cancerous, and the continued disingenuous unwillingness of this community to even approach understanding that is making the entire fucking planet worse, especially now that it wields the megaphone of tech to enforce its culture worldwide.
That’s what you got? Calling out a very colonial, otherizing view of the world, and I’m upset he’s got money? The money is illustrative of the sentiment, not the call to arms. That’s twice now I’ve called out a serious cultural issue on HN and the reply has reinterpreted my meaning along an axis of wealth generation, which is a pattern I’m starting to notice (and it fits).
Literally, you replied the words “seriously poor” to me, while knocking me for calling him rich, in your interpretation. The best part of you saying that is that, like the guy you’re sticking up for, you fail to understand those cultures predate your opinion of economics. They’re not pursuing capitalism and dominating the world with tech, and instead fish to survive instead of $20 butter (horrors!) and somehow this is worth noting because the white people showed up, took everything, introduced currency, then started looking at them funny for not having currency. To buy the $20 butter THEY BROUGHT TO THEM. Pop quiz: do either of you, at this moment, understand the role of butter in Arctic First Nations cuisine? ARE THERE COWS IN THE ARCTIC? Christ, I’m almost emotional at the display of complete idiocy.
And now they’re cultural attractions for Cirrus owners on their way past the non-West where there are no Starbucks.
People who speak like him and you, and reinforce the idea that there’s weird shit outside of this bubble, are a significant contributor to the cancerous toxicity this community injects into the industry and the larger world. The problem is one you’ve spotted: it just sounds like a nice story. It isn’t. It reinforces a divisive view that a lot of wealthier white people have about people who aren’t them. And that comes across largely unconsciously in a lot of the economic output of this very community. So yeah, I call it out, and wait out the very people who need to understand that downvoting it to hide that unpleasantness rather than even acknowledge my side, and continue sighing.
There’s an imagery created by your story of you, standing next to your half million dollar airplane in your aviators and fancy khakis you bought just for this trip, annoyed you can’t get LTE for your next leg’s wx, and putting your hands on your hips and muttering to nobody in particular, wow, these native folk sure have it rough, huh? No Starbucks in Greenland, it turns out.
Please, enlighten me of the point of your story. You ferried across and stopped over in several less advantaged places, then felt compelled to explain to a community of mostly wealthy people that one time, while flying your airplane to Europe (and for some reason while thinking an iPhone has any relevance whatsoever to your situation in an RNAV emergency), you discovered the fabled “worse off” people that you knew existed but had never seen. What would you like me to feel in response to that story? Sad, like you? I am sad, but I’m sad that you’re describing a community in which you stopped over as if you’d discovered a species of three legged unicorn. Had you been in trouble, those sad, poor indigenous cultures likely would have given you everything off their back. Then you swing that brush everywhere: first it was “people up north” (of where? Atherton?) then for some reason instead of deleting that silly grouping, you decided to correct it in a parenthetical and swing right for the natives.
Did you take a bunch of pictures of the communities? When you got back, did you lay awake, troubled, wondering how you could write an app or raise some VC for a nonprofit to help them? When you told this story to your buddies at the FBO, did you include even worse observations that you chose not to type here? The weird irony of your story is you casually refer to “arctic Canada,” of course completely unaware that those First Nations cultures you were prodding and dissecting for a couple hours predate the word “Canada,” all the while lamenting a subtext of “why don’t we help them?” while discarding centuries of colonial attitudes. And then sharing one. What the hell is a “cultural attraction?” You think they’re hunting whale to amuse you?
Congratulations on your... enlightenment?... that there are people out there who can’t afford a SR22, I guess, and thanks for taking the time to call out the specific model so the rest of us pilots know you’re not screwing around. (If you don’t fly, specifying ‘my SR22’ instead of saying ‘my airplane’ or even ‘my Cirrus’ is in the same vein as saying ‘my Lamborghini,’ a phenomenon unique to SR22 owners.) I’m happy you got to experience that and teach a bunch of tech folks that the natives sure are a disadvantaged and lesser species somewhere up north, where the trolls live.
EDIT: Pump up the jam, it’s a using a flag as a super downvote party! Silence the uncomfortableness of the heathen! WTB a vouch
Correct. They had it first, which is why I lightly check the “they use $ too? That’s confusing.” sentiment.
The Dutch, ever innovative in trade, can lay claim as a bigger influence on the word “dollar” and the currency form itself, however, and colonial Americans traded regularly in Dutch daalders (we still pronounce it that way, unlike doh-LAHR/doh-LAHR-ehss for the Spanish varieties). Daalders themselves were descendants of Bohemian thalers, as were Spanish dollars. We just borrowed the neighboring dollars when the time came, probably due to our foreign policy environment at the time, trade with Florida, and so on.
Context. Canadian dollars use $ too, and you only see CAD near the border or when it isn’t clear. If I’m a Colombian using a Colombian site and pesos use $, I don’t need the context. Also, properly, you’d say 5 USD, not USD $5; the dollar sigil is then redundant.
There’s a bit of americentrism down the confusing line of thought, for what it’s worth.
The Y axis of the graph which actually has relevant information. You are looking at a campaign page, and assuming those Colombian pesos are available to the author’s team.
When you said “the image” I thought you were looking at the right one, and I thought it odd you were off several orders of magnitude from what I assumed to be your misunderstanding. That explains that. I had to go back and find your figure.
No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million.
The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a dot after it, because it is abbreviated there (mayo).
Every programmer should understand locales even if they do not speak the language.
That’s like saying “we hired the getaway driver, we can rob the bank now, right?” You’ve identified the first step of the plan. There are about nineteen more, and the theoretical network of conspirators required to accomplish this Oscar-winning screenplay would be quite large, which always spells trouble.
To that end, I’m amazed it took that long for the FBI to take down the network in the article. The more people who are read in to criminal activity, the risk exponentially increases, as anybody who has been on either end of investigative leverage can tell you. I’m stunned one person in the early days of this scam, particularly when it started involving colorful people, didn’t flip as a bargaining tool for other things they were into.
One of the largest and most secretive companies in the world, the same one obsessed with preventing all leaks from exiting the company, the same one who produces ubiquitous devices with occasional national security implications that interest foreign governments, the same one who deals with serious IP problems in the very example nation you just happened to choose, has no thinking or plans around the well-known threats of industrial espionage or sabotage, is what you’re essentially saying. Consider for a moment whether that could be remotely plausible, and I think you’ll see it isn’t.
Good thing it wasn’t my primary responsibility and I didn’t, you know, study the system Google expected me to administer for a living. That sort of knowledge might have shaped my career if I had spent two months studying a software system.
Oh wait, I did, and it did.
Come on, John, I’ve shipped entire products in less than two months. So have you, I’m sure. You really think I passed a SRE-SWE panel then couldn’t figure out a complex software system in eight weeks? Is this really the approach you’d want someone to remember you for? Butthurt and lashing out because I don’t agree with you on software you’ve convinced Stripe to pay you to fuck around with, and assailing credentials instead of listening to what might be a valuable point, or even disputing it?
I always think about the people who flip me off from the safety of halfway down the exit ramp when I’m in threads like these. Not that I’d do anything about it, because I have priors, but still, I smile.
I love when a thread has been dead for hours and the roaches come out of the woodwork to begin turning a conversation personal because it was linked somewhere, and I obviously and emphatically hit a nerve by criticizing something you hold dear. That’s the only reason I’m still refreshing it at this point.
Looking me up (Lord have mercy) and swiping at me as you have here tells me volumes more about you than the effect you intended. What are you expecting? It really says something about Valley culture that I take a swing at Kubernetes, of all bloody things, not gender equality, not politics, not the worth of your contributions to society such as they are, cluster orchestration software, and the absolute worst of you comes out. No factual content, just your smug satisfaction after trying to take an Internet username down a peg. You made the world a better place today. If I cared about you, I might find the whole thing rather sad. I suspect you already do, or will soon, so I need not expend the effort.
Don’t worry, I remember you. If you think you have a handle on my personality from reading disdain I vent on Hacker News, you’re much dumber than I remember, which is disappointing given the respect I have for what little I know of your work. Shame, really.
Indeed. The universe where you aren’t trying to replace Borg, and instead have other motives for conceiving of Kubernetes in the first place. That’s all I’m trying to say.
Hadoop is nowhere close to the original thinking behind MapReduce, and Kubernetes is nowhere close to the original thinking behind Borg. The difference is Yahoo! wrote one, and the other leads us to speculate about why they’d want to offer such a good concept to the world, then ship something that so spectacularly misses that mark, particularly scalewise, but get away with it because it’s Google doing it.
There are only a couple explanations, especially when you factor in that corporations always act in their interest. That’s not a moral dimension, that’s just how it works. What upside is there for Google having everyone do cluster orchestration the way they do? Really, I challenge you to think of one. When I was there, you couldn’t even say the word Borg externally (despite the original article not being the first security-based leak of it). Operations and engineering velocity was, and remains, a crucial competitive advantage for Google. Since I’ve left, there’s a clear shift toward teaching the industry how to do operations, from the Borg paper on through to the SRE book. Why? Why suddenly cede that competitive advantage, the ability to ship a whole production service in like, a weekend?
Because you do it poorly, drive people to Google Cloud because administering Kubernetes at any scale is like self-flagellating with a rake (don’t get me started and be satisfied that I’m speaking from ulcers here), and keep ahead of the competition who are forced to respond to GKE now that a whole community is clamoring for Docker in prod. It’s the only explanation that fits. Maybe it wasn’t even intentional, but it’s what has played out.
Another possible explanation is that Google wanted to end run Mesos, which was threatening to dominate until Kubernetes got a bunch of mindshare and venture capital. Mesos, with care and careful choice of schedulers, can be on the same field as Borg. Twitter is very happy with it at scales that would decimate Kubernetes, as well as many others I cannot name. Google’s VC endeavors have been dumping into Kubernetes startups, and now you don’t hear about Mesos any more. Just like Docker before it, we just can’t get enough bad operational systems as a profession, apparently.
My whole career I’ve been obsessed with high scale orchestration because it is a uniquely difficult problem. I saw Borg and went Jesus, we are all behind the curve as an industry. Then the first few releases of Kubernetes came out and I killed it dead with a few dozen nodes and moderate load. It’s gotten better, yes, but Borg has literally centuries of engineering in it, and a steering committee is going to push Kubernetes to OpenStack or Java status (I’d argue it’s already there, given that people are now shipping distributions of it). And now operations as a discipline suffers for it. I can’t get a SRE gig any more without a dev team who doesn’t need Kubernetes, at all, making it a showstopper to ship.
Kinda funny, by the way, that this thread has been off the front page for hours and is still active with new faces. I must have ruffled eng-misc. :)
Borg will remain orders of magnitude beyond Kubernetes until Kubernetes is completely rearchitected. It’s not scalability bugs. It’s decisions regarding how the cluster maintains state that hamstring it, and that’s so fundamental to everything it’s not a find/squish loop.
As I said in my comment, those major customers (one personal experience, three anecdotally, eight or nine I’ve consulted with) have quietly ruled out Kubernetes, either by trying it or prying it apart and deciding not to try it. That feedback isn’t coming. At Borg scale, Kubernetes is very much considered a nonstarter.
I’m talking about competitors to GKE, which is the entire thesis of that point and that you refuse to acknowledge, which is why you’re knee jerking FUD. I’m a solo developer. I make a Kubernetes app. Do I think Kubernetes is great if I have to build a CloudFormation thing to spin up all of its infrastructure and run it myself, or would I rather pay big company to do it?
The latter.
GKE. EKS/etc. They exist for that use case because that’s where the lock in happens. Now you’re a Kubernetes deployable and too dug in to get off. For a while, your only option was GKE or (God help you, at first) running it yourself. Now other vendors have stepped up to compete, which is the exact situation Google wanted. They shook out the Kubernetes issues with the early adopters with ops experience, then went after pure developers, and had first mover advantage on that customer until the others were forced to respond. I wish I could tell you how many teams I’ve consulted with who when I say “where are we deploying?” immediately say Kubernetes on GKE without a second thought. That is what Google wanted.
This doesn’t take a rocket scientist to figure out, but if you work on Kubernetes I wouldn’t expect you to know this strategy firsthand. It would come from Diane or her people. You’re just having a hootin’ and hollerin’ grand old time building some open source software with a foundation (ooo! So OpenStack!) and mindshare and all the cool kids surrendering their startup to go all in on Kubernetes. Meanwhile, you’re supported by Google, and in fact were conceptualized, to drive sales to Google Cloud. Sorry.
Did you ever ask yourself what the higher level point of Kubernetes is? Like, why throw an entire team on shipping an entire new infrastructure category just to open source it? To make operations better for the industry? Google competes on operations. To use public company capital to chase a hobby? Think about the leadership decision to initiate Kubernetes, and why it was made.
> but you said a whole lot of other crap that is very much misleading in my opinion
The entire comment was quite clearly in support of Google’s decision calculus to not replace Borg. Your rebuttals are honestly more misleading, in my opinion, than my points, because you’re personally wrapped up in it and that’s coming across.
> I don't disagree that Kubernetes is not positioned to be a replacement for Borg
Good. Because that was my point, but the verbiage “don’t disagree” says a lot. We agree, except for the timeline. We will all be dead before Borg is replaced with Kubernetes. You can take that to the bank.
Kinda weird to fire up a throwaway, presumably to conceal your Google credentials, then attack a Xoogler who used to work on Borg SRE (alongside sjh under Peter Dahl, and long enough my NDA has long since lapsed) and has run Kubernetes since it was able to OOM as I described, for spreading FUD. The term FUD implies that I don’t know what I’m talking about and I’m making shit up, while I’m one of the few people, including presumably you, who can actually coherently comment on both.
It can only span multiple clouds now because other clouds had to ship Kubernetes. Remember the timeline: hello world, we made a container thing. Now we offer it as a service. Now Amazon does too. Oh, we are now multicloud. Your rebuttals are quite disingenuous, and casting them with a mocking aspersion doesn’t sell your point. It makes you come across nothing like you intend.
Get back on your real username and stop being offended I criticized Kubernetes. I know I’m one of the few who does, but there are legitimate concerns, and sharing them toward a “why Borg isn’t going anywhere” point is a weird hill for you to die on.
So it descends from Borg, which is fine. It does not replace Borg or indicate a Google strategy to replace Borg with Kubernetes, which was my entire point with supporting points on why, and explaining why you made the choices in Kubernetes that were made does not dispute that at all.
I note you were careful to use the word descendant, instead of my successor.
What I mean is simple: Borg has borgmaster. Kubernetes approached the same concept like a Web application, and now Kubernetes has an entire SIG to play on the same field as Borg. It was a poor architectural decision, along with many others in Kubernetes, but I wasn’t discussing that. I was discussing why Google won’t replace Borg with it.
I suppose that’s fair, but I’d argue against switching even being a primary motivation for anyone at Google, which is why I don’t think of it that way. You do have a point, though.
Without intimate knowledge of Borg, I can understand the successor discussion. With knowledge of what changed (i.e., was getting rid of borgmaster really that important to sacrifice that much perfwise?) I can’t even remotely fathom any purpose for Kubernetes other than what I’ve described. You, however, know far better than me. :)
There’s a meme out there, helpfully nudged along by Google, that Kubernetes is Borg “done right” and the successor. It’s even mentioned in this article. Neither of those things are true. Not even remotely. Please pass along to everyone to stop repeating the meme, because it distracts from Kubernetes’ true purpose, which is to lock people into GKE and force competitors to ship a Kubernetes runtime to compete. It’s partly the OpenStack playbook: if everybody runs the same platform, competition inherently drifts toward other aspects of the businesses, such as customer support. Seriously, I’m the only one who noticed the timing of Kubernetes and Google Cloud? Really? But Google shipped it, so now it has an ecosystem, and zealots who force entire teams onto it for zero upside and nonzero overhead, while the system that actually looks like Borg, the far superior to k8s HashiCorp Nomad, twiddles its thumbs with pretty much no mindshare.
For one, if Kubernetes were the successor to Borg, they wouldn’t have hobbled it architecturally as much as they have by marrying it to both Docker (kinda) and etcd, and deciding in the beginning to do every cluster mutation via external consensus in etcd, because you know, that’s a great idea and a classic Google design. Remember when Kubernetes pushed all job state changes through consensus and a flapping job could OOM etcd? I do too. Someone cynical could argue its fundamental architectural limitations are intentional. (I would argue simply that it didn’t have Paul Menage and most of the other names on the Borg paper working on it, to my knowledge.) I hear keyboards getting angry to yammer about how it just works. Not at seven-digit machine scale, it doesn’t, and never will. I’m happy it works for you. It’s a toy for a large fleet, which I’ll revisit in another point. Everyone I am aware of at Borg or Mesos scale has ruled out or failed with Kubernetes. No, really.
Relatedly, if Kubernetes were the successor to Borg, it’d be in C++. It just would be, and that’s not a language flame war. Ever wonder what percentage of systems at Google are C++? Ever wonder what that number does when you qualify “infrastructural”?
For two, Google containers aren’t an entire operating system, unlike Docker without crazy gymnastics. Seriously, this paragraph could be an essay. To paraphrase Jeremy Clarkson, Docker looks like a proper container system described over a blurry fax. Maybe we do need seventy probably-not-deduplicated copies of getty on every machine, and I’m yelling at clouds. I doubt it, and it reeks of “disk is cheap, fuck it.”
For three, Kubernetes is several orders of magnitude behind Borg and Omega, if that’s still alive, in terms of scheduling performance and maximum “cluster” size (I quote cluster because Google identifies a Borg unit as a cell, and a cluster means something else). This is not fixable in Kubernetes, in my reasonably informed opinion, without doing consensus differently. To my point, Borg does consensus and cluster state much differently, and you know what? It’s fine. Anybody who has used fauxmaster will back me up on that, and John Wilkes even said yup, every time we hit a limit we manage to double it with no end in sight. Why would that suddenly change? etcd remains Kubernetes’ Achilles heel, and this is why messaging around Kubernetes has gravitated toward smaller, targeted clusters. Bonus: if you find a bug in etcd make sure it loudly affects Kubernetes so it gets properly prioritized. Double bonus: someone was brave enough to suggest Consul in #1957 and children. Go read and sigh.
For four, when’s the last time you ran a 10,000+ node MapReduce on Kubernetes? Surprise, the underpinnings of Borg handle both batch and interactive with the exact same control plane, which is where the billions of containers a day number they occasionally talk about comes from. I mean, several JARs of glue might get you to Hadoop scheduling via Kubernetes, but that’s a much different animal than the platform thinking in terms of jobs with different interactivity requirements.
For five, half of Borg is the shit around it. Borg works because everything behaves the same. Everything is a Web server. Everything exposes /statusz. Everything builds and monitors the same way. Everything speaks the same RPC to each other. All of this is implemented by forcing production systems at Google to choose from four (as of my tenure) languages which are well tended and manicured by hundreds of people. Google has a larger C++ standard library team than many startups have engineers. Borg works because apps let it work. They’re not black boxes. Unlike Kubernetes.
Which brings me to point the sixth, which is that the reason you haven’t seen open source Borg is (a) they’re not moving off it, like, ever (I’d bet my season tickets on it), because significant parts of every production system and tool would have to change and (b) they can’t unravel Borg and the rest of the google3 codebase, because it’s so fundamental to the Google ecosystem and half of Borg’s magic is wrapped up in other projects within Google which they aren’t keen to show you.
Link to this answer next time anyone gets tempted to relay what they’ve heard about Borg and Kubernetes, please. For years I’ve watched this tale evolve until it’s barely recognizable as factual. Saying Kubernetes is Borg’s proper successor not only drastically insults Borg and the hundreds (thousands?) who have worked on it, it also calls to mind thinking of a cotton candy machine as the successor to the automobile. They’re that different.
You’re right though, I’ll drop that, that was unfair and unnecessary. I apologize.