Etcd, or, why modern software makes me sad(roguelazer.com)
roguelazer.com
Etcd, or, why modern software makes me sad
https://www.roguelazer.com/2020/07/etcd-or-why-modern-software-makes-me-sad/
210 comments
I really dislike articles like these. I don't think the author is interested in having any kind of productive discussion or criticism.
There's a million reasons to hate Kubernetes, the author couldn't be bothered to venture beyond the lowest hanging fruit (yaml)?
If I could downvote this I would.
There's a million reasons to hate Kubernetes, the author couldn't be bothered to venture beyond the lowest hanging fruit (yaml)?
If I could downvote this I would.
But it's not about kubernetes. It's about simple API's becoming complicated for no reason (according to the author, I have no experience on etcd) using etcd as an example
The author doesn't detail _what_ is more complicated about the gRPC API, other than the fact that it's gRPC. One could implement the exact same API in HTTP or with gRPC; so without specific examples, it's kind of a meaningless critique.
Surely, most consumers of a database like etcd are using a client library, in which case why does it matter if the API is HTTP or gRPC?
Tangential: after reading some of the comments, I was surprised that the blog post was only like 250 words; the author really says very little.
Surely, most consumers of a database like etcd are using a client library, in which case why does it matter if the API is HTTP or gRPC?
Tangential: after reading some of the comments, I was surprised that the blog post was only like 250 words; the author really says very little.
"detail what is more complicated"...
why does everything with a negative valence attract this criticism? surely there are plenty of times in one's life where he or she says negative things about something (a coworker, a boss, a partner, an acquaintance) without any intent to foster a "productive discussion". as though that is something owed to products / services / people / experiences / whatever that we hate for whatever reason. as though those things would somehow magically improve with any amount of "productive discussion". how about not everyone take it personally when someone dislikes something purely because of the experience that they have had with it? given the comments section, the only way for this "discussion" (some call it a post) to have engendered this tone of discussion while still being on the front page is either (1) everyone disagrees with the post, but hate-upvotes it anyway or (2) the points being made strike a chord with people. and regardless of which it is, this whatever the opposite of saber rattling is that causes this kind of reaction clearly misses the point.
>The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 [appendex a] is just a list of the 61 most popular headers from Google properties.
I thought this was moderately-funny cheeky banter, but I wanted to see what implementation decision they were making fun of with this silly misrepresentain. Quote RFC 7541: "The static table was created from the most frequent header fields used by popular web sites." Oh.
I thought this was moderately-funny cheeky banter, but I wanted to see what implementation decision they were making fun of with this silly misrepresentain. Quote RFC 7541: "The static table was created from the most frequent header fields used by popular web sites." Oh.
Why do you think that is a bad header compression scheme for a static protocol where the majority of traffic contains those headers/values repeated over and over? That indexed table shaves about 30% off header size.
https://www.keycdn.com/blog/http2-hpack-compression
https://www.keycdn.com/blog/http2-hpack-compression
[deleted]
This just reads as a systems engineer who is angry about new tooling/processes replacing parts of their job.
I think he/she made it clear it is needlessly replacing parts of the job.
The question is, should one feel any one way about that. I’d imagine it would be the most annoying thing on earth, so it’s a valid emotion being expressed.
The question is, should one feel any one way about that. I’d imagine it would be the most annoying thing on earth, so it’s a valid emotion being expressed.
> I think he/she made it clear it is needlessly replacing parts of the job
Maybe? There weren't any actual reasons for why the author so vehemently disliked all of the technologies mentioned, and there was certainly no acknowledgement of the potential benefits of them.
Maybe? There weren't any actual reasons for why the author so vehemently disliked all of the technologies mentioned, and there was certainly no acknowledgement of the potential benefits of them.
"This is systems. You are to be trying to be clever. Please stop." is a damn good rule of thumb and I think lots of people tend to go all-in on k8s without considering whether their use of the features will actually justify the level of additional clever so introduced.
To me, that being the 'actual reasons' was implicit from the rant - I don't disagree that it was a rant rather than an argument.
To me, that being the 'actual reasons' was implicit from the rant - I don't disagree that it was a rant rather than an argument.
The author is arguing that simple, easy to use, and easy for hobbyists and upstarts to learn tools are being replaced by complex systems designed by and for the 1% of companies who will actually need the complexity.
They are arguing that "House Builders Inc" has invented an automated robotic nail gun and is encouraging standards industries to recommend the only things built with such a system are compliant.
They are arguing that "House Builders Inc" has invented an automated robotic nail gun and is encouraging standards industries to recommend the only things built with such a system are compliant.
Nice projection my dude.
It's a lamentation of the introduction of complexity into a formerly-simple set of APIs.
It's a lamentation of the introduction of complexity into a formerly-simple set of APIs.
> This just reads as a systems engineer who is angry about new tooling/processes replacing parts of their job.
but it doesn't its needs a bunch of work to upgrade/migrate replace. which is what they are complaining about. having to replace perfectly good stuff, with _new_ stuff.
Unless you are bringing teams together onto one platform, k8s is never going to save you cash/engineer hours.
but it doesn't its needs a bunch of work to upgrade/migrate replace. which is what they are complaining about. having to replace perfectly good stuff, with _new_ stuff.
Unless you are bringing teams together onto one platform, k8s is never going to save you cash/engineer hours.
This seems rude and also dishonest/revisionist, eg "In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers)".
What is dishonest/revisionist about the statement you have quoted?
How tedious to have to constantly refute FUD when it's easy to find the answers with Google.
Anyone involved in Kubernetes, near CoreOS at the time, or really anywhere in the space at the time (instead of looking back at it with anger), knows this all to be false. CoreOS was setting direction for etcd, and understandably adding features for one of its bigger users (and in fact, some of those features are used by things of larger scale than k8s).
Kubernetes itself was started by Googlers, many of whom are still there or left to go... do Kubernetes at Red Hat (IBM) or as a startup, or at Microsoft. But to act like it was an outside project started by people who had previously quit, or are somehow unqualified to work on an orchestrator, is just an an angry untruth. Every major committer to Kubernetes besides a handful of RH folks were at Google when Kubernetes 1.0 came out. I'm happy to be corrected but I know it's hip af to hate k8s (just like two days ago https://news.ycombinator.com/item?id=23807556)
Anyone involved in Kubernetes, near CoreOS at the time, or really anywhere in the space at the time (instead of looking back at it with anger), knows this all to be false. CoreOS was setting direction for etcd, and understandably adding features for one of its bigger users (and in fact, some of those features are used by things of larger scale than k8s).
Kubernetes itself was started by Googlers, many of whom are still there or left to go... do Kubernetes at Red Hat (IBM) or as a startup, or at Microsoft. But to act like it was an outside project started by people who had previously quit, or are somehow unqualified to work on an orchestrator, is just an an angry untruth. Every major committer to Kubernetes besides a handful of RH folks were at Google when Kubernetes 1.0 came out. I'm happy to be corrected but I know it's hip af to hate k8s (just like two days ago https://news.ycombinator.com/item?id=23807556)
Thanks. I just wanted to know what the author meant; it wasn't obvious that I needed to Google the quote to figure out the meaning. I certainly hope that I wasn't adding any FUD to the discussion.
Of the folks heralded as "founders": Brendan Burns is at Microsoft, Craig McLuckie and Joe Beda are at VMware via the Heptio acquisition.
Google is still the plurality contributor, followed by Red Hat. If you count Red Hat and IBM jointly, they overtake Google[0].
Disclosure: I work for VMware, which competes with Red Hat/IBM.
[0] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var...
Google is still the plurality contributor, followed by Red Hat. If you count Red Hat and IBM jointly, they overtake Google[0].
Disclosure: I work for VMware, which competes with Red Hat/IBM.
[0] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var...
How about that Kubernetes was released in 2014, not 2015? It's the first thing that shows up when you Google "Kubernetes" - Initial release: 7 June 2014; 6 years ago.
Evidently, OP couldn't even be bothered to gather basic facts about the tech he trashes.
Evidently, OP couldn't even be bothered to gather basic facts about the tech he trashes.
Thank you. I'm sorry that you feel like I was trashing the tech. I really didn't mean to come off that way by asking a question about a statement that the author made.
How many of the original committers still work at Google? I think it's clear that the author means to point out that the originators have left Google.
According to the others on this thread it's related to the date when Kubernetes was released. It wasn't clear to me, but thank you, I think you raise another good point.
it really feels like the author has some huge problem with ex-google employees. i mean, why even point out that "xooglers" created a project?
Agreed. The writing style is so full of bile that any actual point just drowns in it and makes the author seem petty and bitter about the world. Not a good read, even for those that agree with the author about unneeded complexity.
Should bile and bitterness and pettiness never ever be expressed in written form?
There is a higher bar. This post is full of actually false things, giving it no credibility. A high quality rant I can get behind, but not a bad one.
"Kubernetes is the worst thing to happen to system administration since systemd."
I'll take that quote into my fortune file.
I'll take that quote into my fortune file.
What are some of the arguments against `systemd`? I've used it in several production systems and don't really have an opinion on it.
Branding perhaps. If 50 tools had been released, called "newInit", "newDns", "newNTP" etc, with each tool succeeding or failing (upstart) on it's own merits, that would be one thing.
There is a perception that systemd has taken over the entire system as a cancer - replacing things that have worked for decades just because it's cool.
One annoyance I get is this: I've just taken an Ubuntu 1604 server running apache and b0rked the config file.
You run the first and it outputs the last few lines of log. Why not output that when I call restart in the first place?
Another one is taking over DNS from resolv.conf. Muscle memory from years of "cat /etc/resolv.conf" (with a tab somewhere in the middle) has been replaced with the ungainly "systemd-resolve --status".
Now all of this is great if you're running servers as cattle - if you have thousands of machines doing web-scale stuff you don't want to see what's happening on a single machine, your layers of automation abstract it all away, your system can cope with machines going down, you probably don't even have an ssh account.
If you are looking after individual machines though, systemd came in and turned everything upside down to presumably make things easier for cattle owners.
There is a perception that systemd has taken over the entire system as a cancer - replacing things that have worked for decades just because it's cool.
One annoyance I get is this: I've just taken an Ubuntu 1604 server running apache and b0rked the config file.
$ sudo systemctl restart apache2.service
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.
$
$ sudo service apache2 reload
* Reloading web server apache2
* The apache2 configtest failed. Not doing anything.
Output of config test was:
AH00526: Syntax error on line 1 of /etc/apache2/sites-enabled/sd.conf:
Invalid command 'blah', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
You run the first and it outputs the last few lines of log. Why not output that when I call restart in the first place?
Another one is taking over DNS from resolv.conf. Muscle memory from years of "cat /etc/resolv.conf" (with a tab somewhere in the middle) has been replaced with the ungainly "systemd-resolve --status".
Now all of this is great if you're running servers as cattle - if you have thousands of machines doing web-scale stuff you don't want to see what's happening on a single machine, your layers of automation abstract it all away, your system can cope with machines going down, you probably don't even have an ssh account.
If you are looking after individual machines though, systemd came in and turned everything upside down to presumably make things easier for cattle owners.
There is https://blog.darknedgy.net/technology/2020/05/02/0/ which puts systemd in its historical context, explains some of the criticism, and then goes into the technical details. It's a long post and the tone makes it difficult to continue reading at times, but I learnt a lot from reading it.
I for one think the team behind it and especially Poettering have incredibly bad taste in software, which bleeds into their designs, and possibly the worst attitudes and culture of any major OS initiative, making Linus look tame (hell, at least if he got bent out of shape he was usually right, and not just throwing a tantrum because he could get away with it). I don’t necessarily disagree with it’s goals, even, I just think the implementation’s not great and the team are, largely, self-righteous jerkasses. See links others have posted here, their attitude and demeanor’s pretty out-there.
You may not realize you're inciting a flame war. If you google for this there's no shortage of arguments going back 10 years and continuing to this day
"You can't discuss this topic because of a flame war" is ceding territory to the assholes. I'm willing to accept that when it comes to talking about political topics here, because this isn't a political forum, but I refuse to accept that we can't talk about technical topics here because some people refuse to keep a civil tongue.
I didn't realize it was a touchy subject. That's my bad. I'll do some research on this.
oh no worries! I figured as much, hence my comment :-)
FWIW I really like systemd, but I agree the scope creep is real and isn't all positive.
FWIW I really like systemd, but I agree the scope creep is real and isn't all positive.
One practical problem I've had is with journald. The log files are useless outside of the system they were generated on. They could have just used a simple text file and avoided major headaches.
It violates the old school UNIX Philosophy of (1) simple standalone tools (2) text as the universal interface.
(1) systemd is a monolith that aspires to do almost everything from booting to networking to sound. It's in many ways a Kubernetes sans distributed systems.
(2) systemd uses binary logs, abandoning "text is the universal interface"
systemd is also a giant single point of failure.
Personally I don't see the big deal. The actual reason people hate systemd is that it's the type of design a software company like Microsoft would come up with rather than one that a bunch of gritty basement hackers would create. They have found refuge in OSes like Gentoo and OpenBSD, so systemd served as a kind of explusion of this type of programmer/user from the Linux ecosystem.
(1) systemd is a monolith that aspires to do almost everything from booting to networking to sound. It's in many ways a Kubernetes sans distributed systems.
(2) systemd uses binary logs, abandoning "text is the universal interface"
systemd is also a giant single point of failure.
Personally I don't see the big deal. The actual reason people hate systemd is that it's the type of design a software company like Microsoft would come up with rather than one that a bunch of gritty basement hackers would create. They have found refuge in OSes like Gentoo and OpenBSD, so systemd served as a kind of explusion of this type of programmer/user from the Linux ecosystem.
i went trying to learn how to make my linux box do something at startup.
as somebody learning how to use my linux, systemd clouded the discovery process of learning how to linux in the intended way.
i ended up using systemd, but now that's the thing i'm familiar with if i want to add some service to my computer... so it has infected me with some non-linux-y knowledge that won't go away soon.
as somebody learning how to use my linux, systemd clouded the discovery process of learning how to linux in the intended way.
i ended up using systemd, but now that's the thing i'm familiar with if i want to add some service to my computer... so it has infected me with some non-linux-y knowledge that won't go away soon.
Also compare it to doing things the Windows way: Something to run on startup? Put it in the "Startup" (magic) folder. Want something to do the things a service does, where it gets auto-restarted and auto-run and even logs success and failure? Like three clicks can run arbitrary anything using the scheduled task GUI and things get auto-logged in the Windows Event Log
i guess if i wanted windows, i could use windows... right?
edit for tone:
sry, that sounded saltier than i wanted. i actually went to linux because i was convinced that was the better way to do things, and i wanted to do things the better way.
i trust neckbeards over most others.
edit for tone:
sry, that sounded saltier than i wanted. i actually went to linux because i was convinced that was the better way to do things, and i wanted to do things the better way.
i trust neckbeards over most others.
I feel like this page actually comes off as a pretty good recommendation for systemd. Like it's a piece of software with normal bugs and the usual crop of hard decisions.
Almost like with etcd, it started simple, but got expanded into a huge incomprehensible monster by the original developers. Politically and business-wise, systemd is a clear win for Lennaert and his clique, as well as the whole Redhat/IBM.
But I am sure that future historians will view the impact of systemd on computing as largely negative.
But I am sure that future historians will view the impact of systemd on computing as largely negative.
The systemd/RH relationship is complicated. Red Hat doesn't use most of systemd's features--they don't even ship systemd-networkd on RHEL8, preferring NetworkManager instead. On the flipside, I see more use of systemd's features on Arch or Debian.
I don't think you can frame systemd as some kind of RH trojan horse when so little of it makes it into RHEL/Fedora.
I don't think you can frame systemd as some kind of RH trojan horse when so little of it makes it into RHEL/Fedora.
So while Arch ships everything, and being Arch it doesn't really have a default, its recommended networking system isn't systemd-networkd but rather netctl, which is basically everything systemd-networkd should have been.
There is no recommended networkmanager.
`netctl` was explicitly written because systemd didn't have a network daemon. The past 3-4 years the community has in general recommended against using `netctl`.
It was never removed from the ISO because the releng maintainer didn't put that much thought into it, but I'm happy to tell you `netctl` was replaced with `iwd` on this month ISO release.
`netctl` was explicitly written because systemd didn't have a network daemon. The past 3-4 years the community has in general recommended against using `netctl`.
It was never removed from the ISO because the releng maintainer didn't put that much thought into it, but I'm happy to tell you `netctl` was replaced with `iwd` on this month ISO release.
Greeks probably would not pull a Trojan horse into their own city either, despite making one for Troy.
> But I am sure that future historians will view the impact of systemd on computing as largely negative.
I don't really think so. They probably will point out a lot problems with at some point somehow got fixed but it should be net-neutral or positive.
The thing is todays linux is running manny services and to do so nicely you want to have some form of service manager which does startup/shutdown/restart/circicute braking and helps with interconnect. Which is what systemd does and what mainly differentiates it from many previous systems which where mainly "just" start-up helpers. Through there are IMHO a large amount of problems with systemd the general approach is IMHO good, just the implementation isn't so much.
I don't really think so. They probably will point out a lot problems with at some point somehow got fixed but it should be net-neutral or positive.
The thing is todays linux is running manny services and to do so nicely you want to have some form of service manager which does startup/shutdown/restart/circicute braking and helps with interconnect. Which is what systemd does and what mainly differentiates it from many previous systems which where mainly "just" start-up helpers. Through there are IMHO a large amount of problems with systemd the general approach is IMHO good, just the implementation isn't so much.
Sorry, no. Systemd doesn't add anything for service management that didn't previously exist elsewhere other than the unit file syntax for service definition. What it did is pull as many disparate aspects of "how do you run a thing" under its umbrella as it could, so it controls as much of the environment as possible and doesn't have to care about playing nice with anyone else. That's the general approach.
It's a logical fallacy to assume, because systemd was the thing that came along and resulted in the problems in sysv being fixed, that systemd's approach or implementation were necessary, desirable, or the best option available.
It's a logical fallacy to assume, because systemd was the thing that came along and resulted in the problems in sysv being fixed, that systemd's approach or implementation were necessary, desirable, or the best option available.
Here's a good article that I saw on HN a few months ago:
https://blog.darknedgy.net/technology/2020/05/02/0/index.htm...
Also: https://suckless.org/sucks/systemd/
Personally I avoided it for a while because of the hate, but haven't really had issues with it since using it.
Also: https://suckless.org/sucks/systemd/
Personally I avoided it for a while because of the hate, but haven't really had issues with it since using it.
Ditto. my personal experience with systemd has been "wow, it's easier than ever to write an init script that runs when it's needed".
Looking at the components that make up a modern UNIX system, it's definitely time to think "how can we make a new operating system, evolved from UNIX, but with a more coherent core running the majority of run time orchestration".
Looking at the components that make up a modern UNIX system, it's definitely time to think "how can we make a new operating system, evolved from UNIX, but with a more coherent core running the majority of run time orchestration".
Similar opinion here. I put it off for when I could learn it better, and so far it's been fine... sometimes nice too when it has a feature I want that's already built-in (depending on version level and all that).
Thanks for the share from suckless.org. I'm interested to learn about init now :)
I've grown to think its a very vocal minority that objects simply ideological reasons, then finds techincal reason to back that up
Even https://nosystemd.org/ mostly just lists bugs
After moving all my services from varius init systems to systemd, I'd never go back
Even https://nosystemd.org/ mostly just lists bugs
After moving all my services from varius init systems to systemd, I'd never go back
I'm mostly a happy systemd user. Unit files are much easier to work with and more consistent than init scripts, the faster bootup (at least on my arch and Ubuntu systems, maybe you could make Gentoo even more minimal with initscripts), and the container/isolation features won me over.
That said, I can understand the objection to software that would previously support BSD or macOS now only providing systemd units or depending on other parts of the systemd project that in turn depend on the init making at a pain for portability across unices. Or software like gnome taking up huge amounts of time for the BSDs to support.
That said, I can understand the objection to software that would previously support BSD or macOS now only providing systemd units or depending on other parts of the systemd project that in turn depend on the init making at a pain for portability across unices. Or software like gnome taking up huge amounts of time for the BSDs to support.
Unit files are great. SystemD should have stayed there. But then it handles networking(including configuring network interfaces), and time synchronization, and name resolution, and the kitchen sink.
systemd even had a type of container system (I think predating docker by a few years) but no one used it
https://wiki.archlinux.org/index.php/Systemd-nspawn
https://wiki.archlinux.org/index.php/Systemd-nspawn
From an administrator's perspective it's the same problem with every declarative run control system: there is a layer of indirection between what's written in the configuration files and what the machine does (as opposed to an imperative system, where it's a bunch of shell commands that are literally the actions the machine takes).
With systemd in particular the indirection is notably (needlessly) opaque, with a big spaghetti mess of unit files symlinked in 18 different places, some of which have semantic meaning to the init system and some of which do not.
With systemd in particular the indirection is notably (needlessly) opaque, with a big spaghetti mess of unit files symlinked in 18 different places, some of which have semantic meaning to the init system and some of which do not.
I have used systemd successfully in production and have a positive opinion of it. I like that it is integrated, documented, and self consistent.
None of the negative ideological complaints against it resonate with me, as none of them seem to impact me (yet, anyways). My experience with past init systems is that they are cryptic and error prone.
None of the negative ideological complaints against it resonate with me, as none of them seem to impact me (yet, anyways). My experience with past init systems is that they are cryptic and error prone.
I don't know anything about what's happened with etcd in particular, but I think there is a genuine problem in this area with the model of "the people doing the work get to make the decisions".
If you have a piece of software which is basically finished, and a group of people come along who are interested in extending it far beyond its original purpose, then it's very easy for those people to end up as the official maintainers of the software simply because they're going to be the most active.
So when it comes to deciding whether expanding the software so much is a good idea (where the alternative is to make a fork to add all the new stuff in), the people who are happy with it as it stands don't get as much voice as perhaps they should, because for obvious reasons they aren't contributing many patches.
If you have a piece of software which is basically finished, and a group of people come along who are interested in extending it far beyond its original purpose, then it's very easy for those people to end up as the official maintainers of the software simply because they're going to be the most active.
So when it comes to deciding whether expanding the software so much is a good idea (where the alternative is to make a fork to add all the new stuff in), the people who are happy with it as it stands don't get as much voice as perhaps they should, because for obvious reasons they aren't contributing many patches.
Hi, thank you for admitting in your first sentence you're not qualified to comment on the subject at hand, yet felt compelled to share your two cents. Here's the context for your totally uninformed opinion:
I was a major contributor for etcd 2.3-3.2 at CoreOS. The people who extended etcd to support gRPC included the original etcd author (hi Xiang). gRPC support was necessary for good performance; we had benchmarks to justify the decision. Likewise, v3 brought about a key-value model change that was incompatible with v2 to better support binary data, ranging over the keyspace, transactions etc. A v2-style gateway for v3 with a pretty JSON API was planned but never completed due to lack of resources; the ugly gRPC json gateway turned out to be good enough for most people. Similarly I wrote a proxy to run v2 requests over v3 instances, which does support the v2 JSON API. This isn't as if a new group of people showed up and ruined the software without caring about existing users. None of us were "Xooglers".
It seems what you're proposing is what the author both argues against and wrongly believes is what happened. We constantly pushed back against k8s influence. If we didn't, etcd would be a k8s sub-project right now. I'd also like to point out that removing the people who do the difficult work of actually writing the software from the decision process is incredibly insulting and devalues their labor. How dare you.
I was a major contributor for etcd 2.3-3.2 at CoreOS. The people who extended etcd to support gRPC included the original etcd author (hi Xiang). gRPC support was necessary for good performance; we had benchmarks to justify the decision. Likewise, v3 brought about a key-value model change that was incompatible with v2 to better support binary data, ranging over the keyspace, transactions etc. A v2-style gateway for v3 with a pretty JSON API was planned but never completed due to lack of resources; the ugly gRPC json gateway turned out to be good enough for most people. Similarly I wrote a proxy to run v2 requests over v3 instances, which does support the v2 JSON API. This isn't as if a new group of people showed up and ruined the software without caring about existing users. None of us were "Xooglers".
It seems what you're proposing is what the author both argues against and wrongly believes is what happened. We constantly pushed back against k8s influence. If we didn't, etcd would be a k8s sub-project right now. I'd also like to point out that removing the people who do the difficult work of actually writing the software from the decision process is incredibly insulting and devalues their labor. How dare you.
I am saying:
- The problem the author describes can happen (I have seen it happen)
- I do not know whether it happened in the case of etcd
I am not saying:
- I have a proposed solution to this problem
- The people who do the difficult work of actually writing the software should be removed from the decision process
Apologies if that wasn't sufficiently clear.
- The problem the author describes can happen (I have seen it happen)
- I do not know whether it happened in the case of etcd
I am not saying:
- I have a proposed solution to this problem
- The people who do the difficult work of actually writing the software should be removed from the decision process
Apologies if that wasn't sufficiently clear.
I tried to find something valuable in this post, but this is just a few paragraphs of rant.
What a breath of fresh air. The author is spot on and cuts right through the bullshit.
When I see comments like this:
> cancerous user-hostile protocols of HTTP/2
I know not to take the article seriously. I'm simply not interested in hearing from someone with that kind of attitude.
> cancerous user-hostile protocols of HTTP/2
I know not to take the article seriously. I'm simply not interested in hearing from someone with that kind of attitude.
etcd switching to grpc makes complete sense. The author seems to have limited experience with the downsides of http long polling and all the implementation and operational complexities it comes with.
As a counter example of Xooglers making things more complicated, I wanted to bring forward the removal of the protobuf exposure format for prometheus 2.0 after it was discovered that it's actually more performant to parse prometheus' text format.
As a counter example of Xooglers making things more complicated, I wanted to bring forward the removal of the protobuf exposure format for prometheus 2.0 after it was discovered that it's actually more performant to parse prometheus' text format.
It's free open source software. Take it or leave it. Fork it if you want. Use an old version if you want. At the end of the day the maintainers (presumably) decided to do this to etcd. They could've rejected the changes but didn't
I think the takeaway here is that if you don't want a project co-opted by $MEGAHUGECORP, release (or fork) it under GPL 2/3.
This doesn't sound right:
> In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers)
1.0 was released in 2015, but seems to me the relevant year was 2014:
https://www.wired.com/2014/06/google-kubernetes/
And how was it Xooglers? Wikipedia certainly seems to be saying this is a Google-initiated product, through and through: https://en.wikipedia.org/wiki/Kubernetes
> In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers)
1.0 was released in 2015, but seems to me the relevant year was 2014:
https://www.wired.com/2014/06/google-kubernetes/
And how was it Xooglers? Wikipedia certainly seems to be saying this is a Google-initiated product, through and through: https://en.wikipedia.org/wiki/Kubernetes
bai
ad-hominem and hearsay.
>There were a lot of bad ones
Sounds like he had a point, then. Also, apparently HN would rather read your rather salty gossip, than comment on the article on its own merits. Nice!
>There were a lot of bad ones
Sounds like he had a point, then. Also, apparently HN would rather read your rather salty gossip, than comment on the article on its own merits. Nice!
well-put
> This guy used to run infra at uber and was incredibly salty about every single new technology. There were a lot of bad ones, but every conversation was about as constructive, free of evidence, and bitter as this blog post.
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
The apprehensiveness of sysadmins May have been justified in the world 5 years ago but today it sounds somewhat out of place. Note that critically evaluating new technologies is still an important skill and many infrastructure people I work with are extremely cautious about adopting new tech without spiking/getting to know it. But the sysadmins with the penchant for saying no is probably one of the reasons the devops movement actually kicked off: developers and infrastructure folks could see the immense productivity gains from being able to ship code quickly to production and got onboard the technologies that enabled this after being frustrated with the amount of time that traditional software deployment processes took.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
> the devops movement actually kicked off
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
Almost everything in your comment is wrong. Kubernetes has enabled a whole host of observability tooling (eg opentracing), promoted a culture where app logs are easily and always accessible, enabled 0 downtime deployments for teams without dedicate infra specialists and so much more. It has made deploying reliable applications a lot more easier than ever before.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
Kubernetes does indeed have advantages but also brings a whole layer of complexity, overhead and moving parts. From my experience, in many cases the theoretical advantages don't end up being worth the tradeoff and/or don't even end up being implemented. Furthermore the particular things you mention (tracing, centralized logging & no-downtime deploys) can be done just as easily without Kubernetes.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
Your answer to everything I said is “actually, it’s not that hard without kubernetes”. Maybe it’s not for you. For most developers, it absolutely is. And that’s why kubernetes is popular.
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
I am approaching this from a developer's perspective. Kubernetes and similar (even local Docker) introduces an extra layer of indirection that you often have to fight with. Sometimes it's worth it, sometimes it's not. For me, if my application is misbehaving in production, I prefer being able to just SSH into the machine and figure out what's wrong than fight with the container layer, its authentication system, command-line syntax just to obtain a shell inside the container. When I am developing locally, I prefer having all my files on the local filesystem instead of having to worry about volume redirection and "docker exec".
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
I'm an infrastructure type and everything you said rings true for me.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
Revisit this comment of yours in a decade.
A decade is a long time in software engineering. Kubernetes would be obsolete by then. What do I need to revisit?
It is and it isn't. I remember the first time I thought "this software is just X that everybody forgot about, reinvented again". That was 2005. Not coincidentally that was a decade after I started working as a sysadmin.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
I find it hard to parse anything meaningful from your comment, sorry. I don’t mean to be rude, perhaps blunt.
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
> A decade is a long time in software engineering.
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
That’s pretty cool, really. I can’t really find studies on average software life time, so all I really have is anecdotes. I can’t imagine software running for that long on eg public clouds without any kind of major refactor considering how quickly technology changes today, and how aggressively cloud costs push companies to change. Specifically, I can see something like “Serverless” coming to maturity and completely changing the paradigm (again!) on how software is architected. But it’s possible I’m wrong and like you say, k8s will be around for much longer than I anticipate.
[deleted]
I love that we're far enough in the future that there's such a thing as a classically trained sysadmin.
hardly new, Matt Simmons coined the term in 2014.
https://standalone-sysadmin.com/the-difference-between-site-...
https://standalone-sysadmin.com/the-difference-between-site-...
saying "no" does not a devops engineer make
What is a devops engineer? Devops is a culture as originally defined. With sysadmins and software engineers.
And, yeah, sysadmins used to say no a lot but younger ones tend not to because they haven't been rubbed wrong by the silo's of yore.
And, yeah, sysadmins used to say no a lot but younger ones tend not to because they haven't been rubbed wrong by the silo's of yore.
..yet
if your company has a position called 'devops engineer,' they've failed as an entire company to resemble anything called devops.
> I'm going to assume you're a developer?
No, but thanks for explaining my career to me and helping me empathize with toxic people.
No, but thanks for explaining my career to me and helping me empathize with toxic people.
> thanks for explaining my career to me
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
You're welcome. :)
[0]: https://books.google.se/books?id=0VRnDwAAQBAJ&pg=PA490&lpg=P...
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
You're welcome. :)
[0]: https://books.google.se/books?id=0VRnDwAAQBAJ&pg=PA490&lpg=P...
If you dislike the article, go ahead and say why. Leaving negative personal comments about the person who wrote it approximates your own comments: unconstructive, free of evidence and bitter
I haven't even read the article and hence don't really have an opinion either way, but "[not] constructive" and "free of evidence" seem like pretty specific criticisms to me.
Given that Uber is infamous for its extreme NIH syndrome, unnecessarily reinventing the wheel and spending enormous amounts of dev effort to create internal versions of products that already exist, I think it's relevant to the article to point out that the author almost probably had a key role in that culture, and what his philosophy leads to if taken unchecked.
Run infra? Not according to his LinkedIn.
Its safe to say the unix crowd recreated the Windows Registry. I guess it turns out the Microsoft guys weren't idiots its just a messy problem that requires a messy solution.
etcd is not a windows registry- it's more similar to active directory, but with consensus protocols to implement the multi-machine requirements for consistency.
As far as I can tell, the only thing this article is saying is that the author doesn't like gRPC, preferring hand-rolled APIs. (The rest of the rant doesn't really talk enough about the problems to respond to. The author doesn't like Kubernetes. The author doesn't like systemd. The author doesn't like software-defined networking. No reason is given as to why, so there is really no way to have a constructive conversation about it.)
Hand rolled APIs are easier to understand, but harder to maintain. It's great if you're only ever going to have one client, but once you need more than one, it's sure tedious to write and rewrite it for every language you want to support. Using gRPC means that you can auto-generate the client, and while they might not be as wonderful as writing each one of them by hand, at least you can get a client for whatever language you're using. And, the clients all behave the same way -- trying to figure out how to add interceptors to every bespoke client you need is quite tedious. (Look at how long it took AWS to get contexts in go, or how hard it is to add OpenTelemetry to the random hand-rolled HTTP client, etc. With gRPC, you just do those things once!)
Using protos as the transport layer lets you make backwards-compatible changes smoothly; adding fields is safe, renaming fields is safe, etc. The same is not true of using JSON -- if you call something "foo", you can't just one day rename it to "bar". Clients won't know what "bar" is. So you have to update clients and servers at the same time, and you can never "make before break". You see this all the time when someone rolls out a client/server update for a browser app -- your browser cached the Javascript, and it can't talk to the server anymore until that cache expires. It's nasty. I don't understand why people do that to themselves.
gRPC also adds defined semantics for TCP connection length; with HTTP/1.1, maybe you can reuse your connection, maybe you can't, it depends. You can't have multiple requests in flight on the same connection, even if you can reuse it. HTTP/2 fixes this, but gRPC has first-class channels and behavior is well-understood for request/response, streams, etc.
It is unfamiliar and not as easy to debug on the command-line as "curl http://api.example/foo", but once you get up and running, easy things are easy and hard things are possible.
As for Kubernetes, I dunno, it hasn't been bad for me. I tend to use the managed offerings, so I don't have to spare 5 machines for an etcd cluster / masters, or maintain them. I build a container and Kubernetes ensures that it runs forever. If it dies, it's restarted. If more replicas are added, they start receiving traffic. I can manage 100% of the configuration in Git, so if my cluster or cloud provider blows up, I can re-apply somewhere else and have a 99.9% chance of it all working within 15 minutes. Before containers and k8s, production felt very much like a "yolo" thing to me. Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release. You needed to distribute root credentials, hoping that you could fully trust everyone on your team to not mess anything up. It mostly worked, but through sheer brute force rather than any system working behind the scenes to make things run smoothly. With k8s, you can delegate this tedium to software. A developer can update a config file, have the PR approved, and software will ensure that the new version of the software starts, is assigned some load, and the old version is shut down. It's smooth, hard for someone to manually mess up, and quite productive.
I get that people have made their own ad-hoc orchestration and they like it. I guess that's OK. What I like about k8s is that it's a common language for this sort of thing -- I've used it for 3 projects, and they've all looked about the same. I've never had to learn anything company-specific; I can just run my software. I don't think that's a bad thing.
(The solid alternative to k8s, it seems, is just paying someone else to run everything but the one application that your company makes. Can't figure out how to run Prometheus? Just buy a Datadog subscription. Can't figure out how to run a frontend proxy? Just buy an Application Load Balancer. Can't figure out how to search and retain application logs? Just buy a Splunk subscription. Can't figure out how to install MySQL? Just buy it from your cloud provider. That is all great, but you end up spending a lot of money because you can't efficiently manage software and instead spend all your time writing rants about orchestration frameworks. Sometimes I wonder.)
Whenever I see articles like this, I have to ask what the transition from "big UNIX" to Linux would have looked like on HN. I am sure some people hated it, and I'm sure some of those people had good reasons. But things got smoothed out and it turned out that Linux and its ecosystem was pretty good. If you maintain a production application today, it probably runs on Linux, and that's good because that's one less thing you have to teach your team. I kind of see k8s in the same place. Lots of people have trouble running multiple pieces of software in production. k8s is a common language for doing those things. You can build it yourself or you can buy a managed service. You can extend it to do the crazy things you need it to do. It's not a bad thing, and I think it's pretty disingenuous to compare it to systemd.
Hand rolled APIs are easier to understand, but harder to maintain. It's great if you're only ever going to have one client, but once you need more than one, it's sure tedious to write and rewrite it for every language you want to support. Using gRPC means that you can auto-generate the client, and while they might not be as wonderful as writing each one of them by hand, at least you can get a client for whatever language you're using. And, the clients all behave the same way -- trying to figure out how to add interceptors to every bespoke client you need is quite tedious. (Look at how long it took AWS to get contexts in go, or how hard it is to add OpenTelemetry to the random hand-rolled HTTP client, etc. With gRPC, you just do those things once!)
Using protos as the transport layer lets you make backwards-compatible changes smoothly; adding fields is safe, renaming fields is safe, etc. The same is not true of using JSON -- if you call something "foo", you can't just one day rename it to "bar". Clients won't know what "bar" is. So you have to update clients and servers at the same time, and you can never "make before break". You see this all the time when someone rolls out a client/server update for a browser app -- your browser cached the Javascript, and it can't talk to the server anymore until that cache expires. It's nasty. I don't understand why people do that to themselves.
gRPC also adds defined semantics for TCP connection length; with HTTP/1.1, maybe you can reuse your connection, maybe you can't, it depends. You can't have multiple requests in flight on the same connection, even if you can reuse it. HTTP/2 fixes this, but gRPC has first-class channels and behavior is well-understood for request/response, streams, etc.
It is unfamiliar and not as easy to debug on the command-line as "curl http://api.example/foo", but once you get up and running, easy things are easy and hard things are possible.
As for Kubernetes, I dunno, it hasn't been bad for me. I tend to use the managed offerings, so I don't have to spare 5 machines for an etcd cluster / masters, or maintain them. I build a container and Kubernetes ensures that it runs forever. If it dies, it's restarted. If more replicas are added, they start receiving traffic. I can manage 100% of the configuration in Git, so if my cluster or cloud provider blows up, I can re-apply somewhere else and have a 99.9% chance of it all working within 15 minutes. Before containers and k8s, production felt very much like a "yolo" thing to me. Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release. You needed to distribute root credentials, hoping that you could fully trust everyone on your team to not mess anything up. It mostly worked, but through sheer brute force rather than any system working behind the scenes to make things run smoothly. With k8s, you can delegate this tedium to software. A developer can update a config file, have the PR approved, and software will ensure that the new version of the software starts, is assigned some load, and the old version is shut down. It's smooth, hard for someone to manually mess up, and quite productive.
I get that people have made their own ad-hoc orchestration and they like it. I guess that's OK. What I like about k8s is that it's a common language for this sort of thing -- I've used it for 3 projects, and they've all looked about the same. I've never had to learn anything company-specific; I can just run my software. I don't think that's a bad thing.
(The solid alternative to k8s, it seems, is just paying someone else to run everything but the one application that your company makes. Can't figure out how to run Prometheus? Just buy a Datadog subscription. Can't figure out how to run a frontend proxy? Just buy an Application Load Balancer. Can't figure out how to search and retain application logs? Just buy a Splunk subscription. Can't figure out how to install MySQL? Just buy it from your cloud provider. That is all great, but you end up spending a lot of money because you can't efficiently manage software and instead spend all your time writing rants about orchestration frameworks. Sometimes I wonder.)
Whenever I see articles like this, I have to ask what the transition from "big UNIX" to Linux would have looked like on HN. I am sure some people hated it, and I'm sure some of those people had good reasons. But things got smoothed out and it turned out that Linux and its ecosystem was pretty good. If you maintain a production application today, it probably runs on Linux, and that's good because that's one less thing you have to teach your team. I kind of see k8s in the same place. Lots of people have trouble running multiple pieces of software in production. k8s is a common language for doing those things. You can build it yourself or you can buy a managed service. You can extend it to do the crazy things you need it to do. It's not a bad thing, and I think it's pretty disingenuous to compare it to systemd.
thanks for the succinct explanation of gGRPC over HTTP / REST / whatever.
I've been trying to figure that out for a while.
Think your analysis is spot on overall.
The author strikes me as the person who was saying "why can't I just keep writing Assembly?" in the 90s .
The answer is you can keep using <X technology at a lower abstraction level>.
But don't be mad when people are using the new abstraction layer to build interesting stuff because they don't have to worry about the lower level as much.
I've been trying to figure that out for a while.
Think your analysis is spot on overall.
The author strikes me as the person who was saying "why can't I just keep writing Assembly?" in the 90s .
The answer is you can keep using <X technology at a lower abstraction level>.
But don't be mad when people are using the new abstraction layer to build interesting stuff because they don't have to worry about the lower level as much.
> Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release.
that's not really true, configuration management on mutable systems was pretty widespread before everyone started doing immutable infra.
that's not really true, configuration management on mutable systems was pretty widespread before everyone started doing immutable infra.
One should not forget that a big corporate is not made out of great people, but actually only of normal sized people... just, well, a lot of them!
If you hire a lot of people, they all will need to occupy a space each. To deserve that space, everybody needs to be either quite good... or good at faking being good.
Truly innovative things are done by great individuals. And corporates are not the natural habitat for great individuals.
If you hire a lot of people, they all will need to occupy a space each. To deserve that space, everybody needs to be either quite good... or good at faking being good.
Truly innovative things are done by great individuals. And corporates are not the natural habitat for great individuals.
Isn’t Etcd Apache licensed? Don’t like the direction it’s taking? Fork it.
Is your fork not getting much traction and community support? Maybe you’re the edge case.
Is your fork not getting much traction and community support? Maybe you’re the edge case.
People think it's easy to create a durable fork that will gain enough traction to live on many years (in a stable manner) and attract contributors. It's not. Just look at the tons of forks on github for abandonned opensource projects.
Let's say there are 10 people willing to fork it to keep it simple as it was previously. Where is the go-to place for that to happen in the open source world?
Let's say there are 10 people willing to fork it to keep it simple as it was previously. Where is the go-to place for that to happen in the open source world?
It's very hard, that's what I was trying to say.
In your 10 people example, it seems that there are a few problems to me: - finding like minded peers - getting critical mass (are 10 enough to maintain the fork?) - getting them all to agree on what subset of features to keep
If I were in the market for a fork, I would search Github first before forking my own, so I think the first item is basically solved. The second varies on a case by case basis, of course. The third seems more critical to me, probably dispersing the effort more than anything else.
In your 10 people example, it seems that there are a few problems to me: - finding like minded peers - getting critical mass (are 10 enough to maintain the fork?) - getting them all to agree on what subset of features to keep
If I were in the market for a fork, I would search Github first before forking my own, so I think the first item is basically solved. The second varies on a case by case basis, of course. The third seems more critical to me, probably dispersing the effort more than anything else.
[deleted]
This post sums up the last decade. Shouldn't tools and protocols become simpler and easier to use? If megacorps can use complexity to reduce democratization of the web, they will because of profit motive.
> Anything that has a simple and elegant feature-set ends up coöpted by people who just want to build big ungainly architecture and ends up inheriting features from whatever megacorp the coöpters came from
So true.
So true.
My favorite excerpt from this article and one that sums it up nicely can actually be found in one of its footnotes:
> I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol [HTTP/2] pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.
> I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol [HTTP/2] pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.
Don't like what etcd has become? Fork off an older version and help maintain it as a simpler product.
Open source is a market. If you stop building it, they might come too.
Open source is a market. If you stop building it, they might come too.
that's exactly what i wrote. Just fork it!
> That's it. That's the story. Popular modern technology is taken over by expats from a megacorp and made worse in the service of a hyper-specialized (and just plain over-hyped) orchestration platform
If you hate the direction the open source project has gone in, why not fork it and have the v2/HTTP version live on forever? The project does not owe it to you to stay crystallized at a moment in time.
If you hate the direction the open source project has gone in, why not fork it and have the v2/HTTP version live on forever? The project does not owe it to you to stay crystallized at a moment in time.
When v2 is removed, I'll be shocked if etcd isn't forked because it still has enough value to enough people to maintain it.
But it hasn't been removed yet. It might be cynical, but I think a v2 fork now won't accomplish anything because the people most interested in one would simply continue to use v2 upstream.
But it hasn't been removed yet. It might be cynical, but I think a v2 fork now won't accomplish anything because the people most interested in one would simply continue to use v2 upstream.
Unrelated note / self plug - your comment section is hard to read. I built a comment tool that will auto adjust to dark sites like yours, and it supports importing from your comment tool:
https://blog.fastcomments.com/(7-07-2020)-fastcomments-on-si...
https://blog.fastcomments.com/(7-07-2020)-fastcomments-on-si...
This a rant post that backs up very few of its assertions. Though, the author may not have been trying to write for serious consumption. Sometimes it's therapeutic to have a good rant.
I'm not sure what there is for us to discuss. Nice rant, I guess? The post does not attempt to persuade or change opinions (which, again: cool. sometimes it's nice to have a good rant).
I'm not sure what there is for us to discuss. Nice rant, I guess? The post does not attempt to persuade or change opinions (which, again: cool. sometimes it's nice to have a good rant).
> I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server,
Ahahahahahah, yes, anyone can implement a HTTP server, just badly. HTTP/1.1 is quite complex, the spec alone spans over eight RFCs: if you can implement all of that I doubt the HTTP/2 serialization is much of a concern. :P
Ahahahahahah, yes, anyone can implement a HTTP server, just badly. HTTP/1.1 is quite complex, the spec alone spans over eight RFCs: if you can implement all of that I doubt the HTTP/2 serialization is much of a concern. :P
I stopped reading at "Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd.". These are the best tools to happen for system administration in 20 years. Also, etcd is a terrible db. You cant back it up and can easily lose quorum (and all your data) with a network outage.
OP's piece is heat not much light except maybe replacing HTTP with gRPC (if true). Should have added second path and made it configurable. Storing Kubernetes stuff in ETC is the raison d'etre for multi PAXOS systems: it stores stuff. Also ETCD seems to be an innocent by-stander: the real whine seems to be about K8s.
The big missing piece in this article: if the author has a real need for the old simple etcd, he can always fork it. If nobody has done that, that means nobody has enough of a need for a different etcd from the one the Xooglers took over. In which case it's hard to work up a lot of concern.
[deleted]
Summarized by GPT-3 as "Etcd is a cool thing, but some bad people took it over and made it worse."
> If you are running a truly enormous system and want to have off-the-shelf orchestration for it, Kubernetes may be the tool for you. For 99.9% of people out there, it's just an extra layer of complexity that adds almost nothing of value.
This is probably even true for 99.9% of HN readers.
This is probably even true for 99.9% of HN readers.
Why on earth did this garbage make the front page? For shame!
The author is a total jerk trashing everything in his path with hyperbole rather than sound logical arguments. Read his rant on protobuf. Why would anyone waste their time tearing down the contributions of others?
The author is a total jerk trashing everything in his path with hyperbole rather than sound logical arguments. Read his rant on protobuf. Why would anyone waste their time tearing down the contributions of others?
At some point huge complexity are introduced to simplify a problem and solve for a gap or feature. Don't look at this as a burden, look it as a use case you don't require currently and this may not be the right tool for you to use.
Meh, author certainly has a point about K8s being overkill for 99.9% people out there.
Unless you need to operate at a certain scale (that most people never will) then Kubernetes is like taking an F1 car onto a regular road. Recipe for pain.
Unless you need to operate at a certain scale (that most people never will) then Kubernetes is like taking an F1 car onto a regular road. Recipe for pain.
Hot take, the original stuff using simple etcd had no reason to be decentralized in the first place other than more overengineering.
Everything post says is bad is bad, everthing the post says is good is also bad.
Everything post says is bad is bad, everthing the post says is good is also bad.
Well can't say I'm surprised at these reactions. Random internet dude releases opinion, followed by everyone else releasing their opinions all over the internet. Discord ensues.
I might not agree entirely, or even mostly, but I have seen how trying to emulate google can seriously harm applications or open source projects. Guess etcd is an example.
> The software development world would prefer to use their multi-gigabyte IDEs running on ElectronJS to build thousand-dependency Java applications targeting ungainly APIs on hard-to-operate systems than support something simpler and better.
I think that developer tools have always taken up a significant proportion of the resources available on a developer machine. Back in the 90's, Visual Basic was criticizes as being bloated for requiring a 4MB of RAM (the exact number may be off).
Now we have vastly more powerful computers. I think using those resources to have easier, more extensible, and more capable developer environments is a good tradeoff.
I think that developer tools have always taken up a significant proportion of the resources available on a developer machine. Back in the 90's, Visual Basic was criticizes as being bloated for requiring a 4MB of RAM (the exact number may be off).
Now we have vastly more powerful computers. I think using those resources to have easier, more extensible, and more capable developer environments is a good tradeoff.
First time I’m aware of that reading the footnotes provides at least as much meat and interest as the article itself. There’s even footnotes in the footnotes.
I think that author is more dev than ever ops, a year to try to support the project not on dev but on prod in highly available configuration could make eyes open
It all seems like a position of a dev not ops.
I suppose a year in supporting his projects in prod in highly available configuration can change his oppinions.
[deleted]
I am a Xoogler and I needlessly added protocol buffers to my project, at great expense. The author is correct on this point.
I have not found etcd to be 'an absolute pleasure to work with', to put it lightly. It has been a plague of stability issues and it sometimes seems better to roll one's own than continue tracking down issue after issue in etcd (yes I realize DIY has its own set of probably bigger problems :) ) I don't know if this experience is due to the changes on the original etcd implementation that the author is describing.
The Kubernetes podcast explained just last week how moving etcd from HTTP to gRPC allowed a large gain in scalability
The tone of this article is so unrelentingly snarky and negative, I think it's just flame bait. So I flagged it.
i always cringe when the tech megacorps buy out an up and coming start up that shows promise. good for the start up employees as they cash in, but i can’t help but think that overall these actions stiffle future innovation moments that could have been if the start ups matured more on their own.
There is an element of "craftsman blaming their tool" here. I don't mean that as a criticism of the author, who is clearly frustrated with some choices they didn't make but nonetheless have to suffer the consequences of, but rather of the larger context. Clearly, someone, somewhere is craftsman blaming their tools.
> to infect etcd with Google technologies, as is their way. Etcd's simple HTTP API was replaced by a "gRPC"
etcd needed a gatekeeper like Linus Torvalds [1]
[1] https://lkml.org/lkml/2020/6/1/1567
etcd needed a gatekeeper like Linus Torvalds [1]
[1] https://lkml.org/lkml/2020/6/1/1567
I thinkThe author Mahers some good points. Why not fork the old version?
I actually block all QUIC traffic, the only use I have seen for it so far from my OSX machines is chrome phone home traffic. At the weirdest moments, suddenly little snitch pops up, hey chrome wants to connect to ip x.y.z.1 443 QUIC
It's open source. You can always start your own fork, no?
He is partially right because i think containers or vagrant are good tools for development in 80% of all cases...but on the other hand i like having monolith's in testing and production...so there's that ;)
What happened to etcd v1 that it's no longer useable?
I enjoyed your rant James. Its like a walk to Hermanos.
Does one not simply make a fork of the older version?
Too much hate when the solution is quite easy: Fork it from the version that you like and maintain it.. I'm quite sure lot of people would love to support you on that.
It’s difficult to build simple (to use) things
gRPC is actually pretty nice. It's simpler than COM. More complex than passing JSON around but also less bug prone.
I found these footnote labyrinths¹ hard to follow.
¹See https://xkcd.com/1208/.
¹See https://xkcd.com/1208/.
I think I've just discovered one of my new favourite blogs, speaking unbirdled truth.
I liked this because the author advocates for simplicity.
There's a lot of justified hatred for kubernetes, but for fuck's sake why did anyone adopt it in the first place, if they didn't understand what they were getting into? It's like the startups from 10 years ago who absolutely had to use MongoDB because "scale", and then they never get off the ground because they're trying to implement ACID from first principles.
Hype train tickets are cheap to get on, but expensive to get off.
Since hackernews is mostly junior devs or senior principle architects (formerly junior devs), I expect there's a lot of stockholm syndrome going on in these comments.
Better get out of here before they notice us.
Better get out of here before they notice us.
It's possible that many people (like me) had been trying to solve a similar non-business problem for years, including rolling our own solutions and the like, and when an open source option widely backed emerged and looked like a possible standard, we accept some warts in exchange for a broad, general purpose, flexible, automatable, and well-thought out (yes) solution.
Of course K8s is not perfect, and it's overkill for small to medium apps (I think hype train convinced a lot of people they would need to scale to massive cloud levels when really they didn't), but if you have ever needed K8s (especially for a complex microservice system at big enterprise level) then you know the value and you remember the proprietary vendor-locked era of sadness before K8s emerged.
Of course K8s is not perfect, and it's overkill for small to medium apps (I think hype train convinced a lot of people they would need to scale to massive cloud levels when really they didn't), but if you have ever needed K8s (especially for a complex microservice system at big enterprise level) then you know the value and you remember the proprietary vendor-locked era of sadness before K8s emerged.
and me)
There is a cost for not getting on a hype train. CEOs rather waste a couple of billion dollars on that machine learning/AI hype train, because if they don't hop on that train, Wall street folks see these companies old-school, not innovating. So, they bring in Chief AI officer and write about about how machine learning gonna help their companies in the Quarterly reports. Even if they lose $2B in this AI venture, just treat as insurance premium to keep the stock price going up.
Same with all these technologies.
Same with all these technologies.
AI is cargo-culting from executives in publicly traded companies.
Kubernetes is a grassroots phenomenon of developers experimenting with self-immolation while padding the resume.
Major difference. Your example comes from the top. Kubernetes comes from the bottom, and is more insidious in my opinion.
Kubernetes is a grassroots phenomenon of developers experimenting with self-immolation while padding the resume.
Major difference. Your example comes from the top. Kubernetes comes from the bottom, and is more insidious in my opinion.
This. I cannot tell you the number of times where I've seen teams use Mongo and then reinvent a schema as part of the development contracts. It's enraging.
Engineering-driven-development, where the purpose of engineering is engineering and business problems take a backseat in favor of hiring lots of engineers, giving them toys to play with and having buzzwords to on your careers page. In this case, your objective is to fill up your stack with as many shiny things as possible regardless of whether they are necessary or appropriate to solve the business problem at hand.
This strongly correlates with bullshit VC-funded companies that wouldn't be profitable even on a good day so their only objective is to raise as much funding as possible (the business problem is secondary, because even solving the business problem perfectly wouldn't make them profitable) and chasing fads and buzzwords is sadly a valid strategy for raising capital.
This strongly correlates with bullshit VC-funded companies that wouldn't be profitable even on a good day so their only objective is to raise as much funding as possible (the business problem is secondary, because even solving the business problem perfectly wouldn't make them profitable) and chasing fads and buzzwords is sadly a valid strategy for raising capital.
Well, I am using it as my PO wants it, to serve only 4 apis.
Everything in AtOmXpLuS
ddevault(1)
Cool, I need to look into Kubernetes way more then! There hasn't been any single development in the Linux world that has made my job as a sysadmin easier than systemd. Services, Timers and Networking that just works, is concisely defined and can be used across all relevant distros? Hell yeah, I don't ever want to go back into init.d/ifupdown/crontab world filled with bad configuration files, strange footguns and bugs that won't ever get fixed because someone's legacy system might depend on them.