Minimal Viable Programs (2014)(joearms.github.io)
joearms.github.io
Minimal Viable Programs (2014)
https://joearms.github.io/published/2014-06-25-minimal-viable-program.html
72 comments
I really believe this is the only way you can write good software. It's mind blowing to me that most people try to "figure out" the problems before seeing what it actually is. I always feel strange when someone is writing a "study" by reading the documentation of some API or even worse, by reading someone's feature request. The writer of a feature request usually knows even less about the whole thing and really did not think things through. We need tools and programming languages where you can create really dirty but working solutions and make it iteratively better. You need to find all the edge cases and pitfalls and for that you need to fail. Of course you need to fail fast, this method does not work if the iteration is slow and the thing is in production when you find out that it barely works.
Where do I go to get paid to actually put in the TIME and effort required to produce something elegant and high quality?
As a contractor working on project basis, who can leverage this technique to produce code of such high quality that a commensurate price can be commanded.
You’d need to find a niche that values this level of quality, but if that exists, and if the theory is true, then there’s your gold.
You’d need to find a niche that values this level of quality, but if that exists, and if the theory is true, then there’s your gold.
Love to see that magical place.
I come to think that the whole concept of MVP/prototype became a bane to software development when the management/business side got aware of its existence. Architecture and design sessions can be skipped because we just build a prototype.
I have yet to see a prototype that did not end in production. It's "good enough software", let's move to the next feature.
When you are really lucky you can revisit your prototype a year or two later and try to improve it's design now that you got some data on its actual usage, but you have to figure out again, what the heck you actually did...
I come to think that the whole concept of MVP/prototype became a bane to software development when the management/business side got aware of its existence. Architecture and design sessions can be skipped because we just build a prototype.
I have yet to see a prototype that did not end in production. It's "good enough software", let's move to the next feature.
When you are really lucky you can revisit your prototype a year or two later and try to improve it's design now that you got some data on its actual usage, but you have to figure out again, what the heck you actually did...
"Plan to Throw One Away" is a classic bit of Fred Brooks advice.
https://course.ccs.neu.edu/cs5500f14/Notes/Prototyping1/plan...
https://course.ccs.neu.edu/cs5500f14/Notes/Prototyping1/plan...
Neat. That's essentially how I program, too. Early history in some of my git repos has the pattern
git add program
...
... #first implementation works; time to rewrite
git add newprogram
...
... #second implementation works; bury the first
git rm program; git mv newprogram program
...
Except that way there's basically zero semantic continuity between the commits of the old program and rewrite.
It's preferable to work in branches and rewrite and refactor incrementally whenever possible, merging the new code in as things progress at convenient milestones.
Then the history documents which functionalities in the old implementation became what files and lines in the new implementation.
That information can be very helpful when the rewrite breaks previously working functionality and customers are upset, especially if the rewrite authors aren't even around anymore.
It's preferable to work in branches and rewrite and refactor incrementally whenever possible, merging the new code in as things progress at convenient milestones.
Then the history documents which functionalities in the old implementation became what files and lines in the new implementation.
That information can be very helpful when the rewrite breaks previously working functionality and customers are upset, especially if the rewrite authors aren't even around anymore.
> Except that way there's basically zero semantic continuity between the commits of the old program and rewrite.
That's the goal of this practice. Throw it all out and rewrite from scratch.
> That information can be very helpful when the rewrite breaks previously working functionality and customers are upset, especially if the rewrite authors aren't even around anymore.
I'm talking about writing non-releases. The entire structure of the code, algorithms, data structures, API / CLI must not be preserved. Anything that stays constant between non-releases is either (a) a good idea or (b) a stone unturned. Rename variables, split and join modules, violently reorder function arguments... nothing is off-limits. After I've been through a few rounds of this, the first release should be entirely free of inessential warts (and maintenance is typically quite easy at this stage).
After the first release, I'm a stickler about backwards-compatibility and my development pattern is closer to what you're suggesting.
Among folks who know me, I've got a reputation for writing clean, optimized code. It isn't because I'm good at writing clean code; it's because I'm good at throwing away ugly code and I'm very reluctant to share code that doesn't meet the bar.
That's the goal of this practice. Throw it all out and rewrite from scratch.
> That information can be very helpful when the rewrite breaks previously working functionality and customers are upset, especially if the rewrite authors aren't even around anymore.
I'm talking about writing non-releases. The entire structure of the code, algorithms, data structures, API / CLI must not be preserved. Anything that stays constant between non-releases is either (a) a good idea or (b) a stone unturned. Rename variables, split and join modules, violently reorder function arguments... nothing is off-limits. After I've been through a few rounds of this, the first release should be entirely free of inessential warts (and maintenance is typically quite easy at this stage).
After the first release, I'm a stickler about backwards-compatibility and my development pattern is closer to what you're suggesting.
Among folks who know me, I've got a reputation for writing clean, optimized code. It isn't because I'm good at writing clean code; it's because I'm good at throwing away ugly code and I'm very reluctant to share code that doesn't meet the bar.
Sure, that's perfectly fine if it's for what amounts to entirely private development iterating on something nobody has ever seen.
In my experience my MVPs often end up in production and that utopia of iterating in isolation is fairly short-lived.
In my experience my MVPs often end up in production and that utopia of iterating in isolation is fairly short-lived.
That's true, but it's also something you have some degree of influence and control over. Don't allow prototypes or first awkward stabs to get into the hands of users without them being explicitly told this is a preview and their data will be destroyed.
Sorry to sound cynical but... have you ever been introduced to the concept of "deadline"?
I really want a way to filter out comments which boil down to a snarky “SOME people can’t do this”.
Yes yes, have points, well done, but it’s a boring low effort class of comments that applies to nigh every single idea ever, and skips the interesting or important bits - in this case whether it produces better software or helps understand things - in favour of a cheap vote grab. “It’s not universally applicable” - no, nothing is.
and you don’t sound cynical you sound like you’re trying to dismiss and status grab, like someone talking about the “out here in the real world”.
Yes yes, have points, well done, but it’s a boring low effort class of comments that applies to nigh every single idea ever, and skips the interesting or important bits - in this case whether it produces better software or helps understand things - in favour of a cheap vote grab. “It’s not universally applicable” - no, nothing is.
and you don’t sound cynical you sound like you’re trying to dismiss and status grab, like someone talking about the “out here in the real world”.
I know it sounds snarky, and I am sorry (I even tried to apologize preventively, but I was in a hurry and this probably was taken as extra sarcasm on my part).
Still, my point stands: I am not sure if this is really a matter of “SOME people can’t do this”, I suspect it is the other way around: “SOME people can do this... because they are working on free software, on their own personal project, on programs or systems that do not have any TTM aspect”.
And it is not just a matter of "deadlines" either: if your code is part of some bigger system you may not have the chance to reimplement things from scratch not because you don't have time, but because you could impact someone else using your library/module/API.
In other words, this approach is eminently laudable, but in my experience does not scale in most cases.
Still, my point stands: I am not sure if this is really a matter of “SOME people can’t do this”, I suspect it is the other way around: “SOME people can do this... because they are working on free software, on their own personal project, on programs or systems that do not have any TTM aspect”.
And it is not just a matter of "deadlines" either: if your code is part of some bigger system you may not have the chance to reimplement things from scratch not because you don't have time, but because you could impact someone else using your library/module/API.
In other words, this approach is eminently laudable, but in my experience does not scale in most cases.
It does work somewhat under light pressure. Under high deadline pressure it obviously can't, but I'd argue that if a company is slave-driving programmers, code is going to be garbage and development will start slowing down due to accumulated technical debt.
> if your code is part of some bigger system you may not have the chance to reimplement things from scratch not because you don't have time, but because you could impact someone else using your library/module/API.
In this case, if your code is used by someone else, you can't do that. The trick is not releasing your code "out there" until after you've done several rewriting passes on it. In the cases I've done that in professional context, I worked on a feature branch. I would occasionally rebase it on top of master, to stay current with rest of the product, but nobody ever looked at my feature branch until I've done a few rounds on the code, declared it's ready for integration, and issued a merge request.
> if your code is part of some bigger system you may not have the chance to reimplement things from scratch not because you don't have time, but because you could impact someone else using your library/module/API.
In this case, if your code is used by someone else, you can't do that. The trick is not releasing your code "out there" until after you've done several rewriting passes on it. In the cases I've done that in professional context, I worked on a feature branch. I would occasionally rebase it on top of master, to stay current with rest of the product, but nobody ever looked at my feature branch until I've done a few rounds on the code, declared it's ready for integration, and issued a merge request.
I got 3 downvotes to my original (terse) post and 3 upvotes to the more articulate answer to you.
I suppose this should teach me to wait until I have time enough to write a proper post, next time.
I suppose this should teach me to wait until I have time enough to write a proper post, next time.
Hahaha, nice one. Cynical answer: I wish we could ask Ken that.
My situation isn't typical. To put it in common parlance, this is how I nucleate my 20% projects, and personal projects. Which, once released, almost never require significant maintenance because of the care and attention spent upfront.
I'm more than capable of whipping out a prototype that will do the job and incur technical debt. And I'm upfront about tradeoffs in planning, and meticulous about the rare deadline I get saddled with.
My situation isn't typical. To put it in common parlance, this is how I nucleate my 20% projects, and personal projects. Which, once released, almost never require significant maintenance because of the care and attention spent upfront.
I'm more than capable of whipping out a prototype that will do the job and incur technical debt. And I'm upfront about tradeoffs in planning, and meticulous about the rare deadline I get saddled with.
> Not sure how to solve that particular problem though.
I think it is less a problem to be solved than a fundamental trade-off that comes with the scale of systems: the more components a system is going to have, the greater the cost of not minimizing complexity of individual components.
I think it is less a problem to be solved than a fundamental trade-off that comes with the scale of systems: the more components a system is going to have, the greater the cost of not minimizing complexity of individual components.
“I really like systems that do one essential thing and do it well. Good examples are Dropbox and Twitter. Dropbox just works. Twitter has a no fuss 140 character tweet box. Simple, easy to understand and minimalist.”
They used to... Maybe even when this was published in 2014.
They used to... Maybe even when this was published in 2014.
Yup. This remark made me search for the publication date. Yes, Dropbox did one thing and did it well... 5 years ago. Today, not so much.
And yet when I expressed a negative opinion on dropbox "Paper" I was downvoted into oblivion. Same happened when I was critical of antirez "disque" project, which is abandonware now. Part of HN seems breathless about new and shiny, and others seem to like this kind of "Unix philosophy" minimalism. I guess it's just a matter of taste at the end of the day?
Disque is not abandoned fortunately! I'm in the process of porting it as a Redis module. It was not viable to continue it as a fork.
Huh. I had my own definition of Minimal Viable Programs: code that demonstrate how to do X, and only X.
I found that tutorials often had unnecessary amount of complexity to show people how to do something, like encumbering an example with classes, title setting, etc, none of which is essential to the operation of the program and which makes it difficult to understand and learn.
So, if I found a working example, I would strip the example down to its most essential until I cannot take away anymore without it ceasing to function.
Maybe I should call it minimum viable examples instead.
I found that tutorials often had unnecessary amount of complexity to show people how to do something, like encumbering an example with classes, title setting, etc, none of which is essential to the operation of the program and which makes it difficult to understand and learn.
So, if I found a working example, I would strip the example down to its most essential until I cannot take away anymore without it ceasing to function.
Maybe I should call it minimum viable examples instead.
This always becomes a tricky debate. Take the `ls` program. MVP would just by `ls`. Or maybe `ls -la` to get all the details. You could theoretically pipe that into other programs to get more information.
But I'll be damned if I don't use `ls -lah` because I am a human, not a robot. `ls -1` is also super handy. I am sure there are lots of others that folks use.
So where do you draw the line? It makes way more sense to include a few extra options in your tool rather than requiring your users to jump through extra hoops to get something _they_ find useful.
edit: On the other hand, you don't need to include the kitchen sink, either. I have seen programs with literally hundreds of long-form command line arguments. It's a nightmare to wade through.
But I'll be damned if I don't use `ls -lah` because I am a human, not a robot. `ls -1` is also super handy. I am sure there are lots of others that folks use.
So where do you draw the line? It makes way more sense to include a few extra options in your tool rather than requiring your users to jump through extra hoops to get something _they_ find useful.
edit: On the other hand, you don't need to include the kitchen sink, either. I have seen programs with literally hundreds of long-form command line arguments. It's a nightmare to wade through.
The flags you talk about are mostly about the presentation of the output. The, shall we say, "business logic" of `ls` is still pretty bare-bones. That could be one way to draw a line.
Also, I think these presentation/formatting flags are a consequence of the Unix shell "everything-is-text" paradigm. In Powershell, where you pass around structured data rather than text, typically you handle presentation just by piping into the appropriate cmdlet.
Also, I think these presentation/formatting flags are a consequence of the Unix shell "everything-is-text" paradigm. In Powershell, where you pass around structured data rather than text, typically you handle presentation just by piping into the appropriate cmdlet.
> But I'll be damned if I don't use `ls -lah` because I am a human, not a robot.
I'd prefer that '-lah' be an external composable modifier. 'ls -la' looks like it is, I think, because it tries to create simpler common case, for the expense of that case be more specialized. Since, as somebody decided, 'ls' by default contain only some information available, one presumably needs 'ls -zz' (a made up option) which would produce all information available, and a composed to it filter which would select what is needed at the time.
If composability would be really cheap, we wouldn't need that special '-zz' option. We can even consider an option as a composition of additional operation over generic 'ls' behavior - in this case we wouldn't need the shell 'pipe' operator, and syntax of shell statements would become closer to Haskell's. For example, 'cat' could do something generic by itself, while 'cat note.txt' would be considered a composition of 'cat' and 'note.txt' which creates a specialized behavior (which sort of reminds Forth).
I'd prefer that '-lah' be an external composable modifier. 'ls -la' looks like it is, I think, because it tries to create simpler common case, for the expense of that case be more specialized. Since, as somebody decided, 'ls' by default contain only some information available, one presumably needs 'ls -zz' (a made up option) which would produce all information available, and a composed to it filter which would select what is needed at the time.
If composability would be really cheap, we wouldn't need that special '-zz' option. We can even consider an option as a composition of additional operation over generic 'ls' behavior - in this case we wouldn't need the shell 'pipe' operator, and syntax of shell statements would become closer to Haskell's. For example, 'cat' could do something generic by itself, while 'cat note.txt' would be considered a composition of 'cat' and 'note.txt' which creates a specialized behavior (which sort of reminds Forth).
I'm aware of "h" but never really had an issue determining that e.g. 1144623161 means 1.1G. I guess I recognize the actual length of the group of digits and then my brain splits it up backwards into 161 623 144 followed by a a 1.
Tangent: I miss Joe Armstrong. I never even programmed in Erlang but I just enjoyed watching his contalks and the interviews he held with other people so much.
> If you want a job done find the busiest person you know and give them an extra job. This is because the reason they are busy is that lot's of people want them to do things because they are good at doing things and that's why they are busy.
Oh man... that is so accurate it hurts.
Oh man... that is so accurate it hurts.
You know you became that guy when you get your own swimlane... bonus points if there's a list of items that last at least 3 months there.
I’ve been interested in shell scripting lately and I have to admit that I posted this not because of Joe’s high-level observations on minimalism of architecture, but because of his discussion of a tiny, shell-based system his team used for ticket management, which reminded me of this shell-based notepad posted a few days ago: https://news.ycombinator.com/item?id=21416338
TFA made me think of this recent similar post: A static site generator using make, shell script and pandas, in about 150 lines including templates.
http://stephenbalaban.com/static-site-generation-in-50-lines...
http://stephenbalaban.com/static-site-generation-in-50-lines...
I think you meant Pandoc rather than Pandas?
Yes I did. I wanna blame autocorrect but I’m afraid it was probably some sort of Freudian slip
Good software is written like this. Joe could not be more sharp on that.
I did the exact same thing at my current company. I created a ticket system integrated in Slack where employees can quickly open a new issue and put a tag on it.
A ticket is created in HelpScout where out IT support team follow up.
Works 100% of the time and had zero downtime so far (around 2 years on production).
No need for new features. It just works.
Things are becoming more like the unix philosophy, modular and shared. Even though feature creep is increasing on modular systems, as long as the core components are simple to understand and introduced first, extra features do not necessarily decrease utility.
For example tmux, if you look at a cheat sheet you might be overwhelmed, but when I teach people to use it they find it extremely simple. I just tell them ctrl-c to create a new window, and ctrl-w to switch windows. They only need 2 commands to get value, even though there are hundreds of features, and later they can incrementally learn about them.
This is the same way I learned programming, the people learn emacs and vim. You start with the MVP feature set that will make it functional and you learn about features from there.
Even AWS, Google Cloud, Azure, even though they're mega services of now insane size and feature breadth, they're creating single services that are easy to understand and compose together. You don't need to understand a lot to copy files in and out of AWS s3 for example, and you can compose many services together even after a basic understanding. You can slowly learn about more features as you work.
In the future, things like apache arrow will also help as you can retain services as modular but potentially share memory when composing services. This idea of service composition is also common in Haskell and other functional languages, where if it's possible to do lazy evaluation you can avoid redundant work. Now that the unix philosophy is being more widely adopted, although renamed to microservice fundamentals, these ideas of lazy evaluation and protected data sharing probably be the next level of services research.
For example tmux, if you look at a cheat sheet you might be overwhelmed, but when I teach people to use it they find it extremely simple. I just tell them ctrl-c to create a new window, and ctrl-w to switch windows. They only need 2 commands to get value, even though there are hundreds of features, and later they can incrementally learn about them.
This is the same way I learned programming, the people learn emacs and vim. You start with the MVP feature set that will make it functional and you learn about features from there.
Even AWS, Google Cloud, Azure, even though they're mega services of now insane size and feature breadth, they're creating single services that are easy to understand and compose together. You don't need to understand a lot to copy files in and out of AWS s3 for example, and you can compose many services together even after a basic understanding. You can slowly learn about more features as you work.
In the future, things like apache arrow will also help as you can retain services as modular but potentially share memory when composing services. This idea of service composition is also common in Haskell and other functional languages, where if it's possible to do lazy evaluation you can avoid redundant work. Now that the unix philosophy is being more widely adopted, although renamed to microservice fundamentals, these ideas of lazy evaluation and protected data sharing probably be the next level of services research.
Might you be looking at this through rose tinted glasses?
I'm still a VIM user to this day, but my learning experience was hardly a clean incremental process. VIM has no MVP if you're using it to program computers. Early on, I spent hours getting sucked into making the editor do what I wanted. I only stuck with it because I believed the investment of time was worth it (and it was).
> You don't need to understand a lot to copy files in and out of AWS s3 for example, and you can compose many services together even after a basic understanding. You can slowly learn about more features as you work.
This is dangerous though. You might unknowingly become a victim of vendor lock-in. Or you might have to throw out all of your work because you didn't understand the constraints of the system.
Subjectively, my evidence is how I (an experienced person) evaluates software and tools today. Step 1 is learning the architecture, Step 2 is finding the limits/constraints, etc (there's more but you get the idea). When I was evaluating Git I spent a couple hours researching the internals. This sold me on Git but it also made Git much easier to use down the road. It's not a very big time investment most of the time. This kind of evaluation has been so important when evaluating complex stuff like databases and SaaS products.
> Now that the unix philosophy is being more widely adopted, although renamed to microservice fundamentals, these ideas of lazy evaluation and protected data sharing probably be the next level of services research.
Microservices have been a disaster. The majority of implementations I've seen are brittle/buggy/convoluted messes. Complexity that was previously managed in code has been pushed to infrastructure -- and no one knows how to do infrastructure!
Most people that I talk to don't really understand what made the Unix philosophy work. Everyone cites "programs that do one thing and do it well". That's certainly the fun part, but it isn't what makes Unix programs so great. It's how programs do I/O (text and pipes).
In other words, it's not enough to build tiny modular single-use programs. You need an equally elegant system the combines the programs together. The programs need to strictly adhere to this system otherwise everything breaks down. Containers are not it. K8s is not it.
I'm still a VIM user to this day, but my learning experience was hardly a clean incremental process. VIM has no MVP if you're using it to program computers. Early on, I spent hours getting sucked into making the editor do what I wanted. I only stuck with it because I believed the investment of time was worth it (and it was).
> You don't need to understand a lot to copy files in and out of AWS s3 for example, and you can compose many services together even after a basic understanding. You can slowly learn about more features as you work.
This is dangerous though. You might unknowingly become a victim of vendor lock-in. Or you might have to throw out all of your work because you didn't understand the constraints of the system.
Subjectively, my evidence is how I (an experienced person) evaluates software and tools today. Step 1 is learning the architecture, Step 2 is finding the limits/constraints, etc (there's more but you get the idea). When I was evaluating Git I spent a couple hours researching the internals. This sold me on Git but it also made Git much easier to use down the road. It's not a very big time investment most of the time. This kind of evaluation has been so important when evaluating complex stuff like databases and SaaS products.
> Now that the unix philosophy is being more widely adopted, although renamed to microservice fundamentals, these ideas of lazy evaluation and protected data sharing probably be the next level of services research.
Microservices have been a disaster. The majority of implementations I've seen are brittle/buggy/convoluted messes. Complexity that was previously managed in code has been pushed to infrastructure -- and no one knows how to do infrastructure!
Most people that I talk to don't really understand what made the Unix philosophy work. Everyone cites "programs that do one thing and do it well". That's certainly the fun part, but it isn't what makes Unix programs so great. It's how programs do I/O (text and pipes).
In other words, it's not enough to build tiny modular single-use programs. You need an equally elegant system the combines the programs together. The programs need to strictly adhere to this system otherwise everything breaks down. Containers are not it. K8s is not it.
This is an important point, and I'll (re)state it: the pieces have to be modular, AND the "glue" has to afford that modularity.
> This is dangerous though. You might unknowingly become a victim of vendor lock-in.
The simplicity of the individual services means it's easy to compose within a cloud, but it also means (particularly when the fundamental problem concept each is directed at is similar) that it is easy to abstract from a particular cloud by composing the solutions from separate clouds for the same problem behind a shared abstraction layer.
The simplicity of the individual services means it's easy to compose within a cloud, but it also means (particularly when the fundamental problem concept each is directed at is similar) that it is easy to abstract from a particular cloud by composing the solutions from separate clouds for the same problem behind a shared abstraction layer.
The dirty reality is that software is tuned to constraints of a particular cloud provider. Talk to anyone who's turned a long-lived single-cloud production system into multi-cloud.
Example: How does AWS EKS provision control plane worker nodes? AWS CloudFormation. How do you manage security policies? AWS IAM. What kind of servers do you tune your software to run on? AWS EC2 of course.
Now let's say you wanna take advantage of this fancy "composing the solutions from separate clouds" thing. Let's say you like Azure's Blob Storage over AWS's S3. Guess what? You're now paying for cross-cloud latency! Both in time and in cost.
The devil is in the details.
Example: How does AWS EKS provision control plane worker nodes? AWS CloudFormation. How do you manage security policies? AWS IAM. What kind of servers do you tune your software to run on? AWS EC2 of course.
Now let's say you wanna take advantage of this fancy "composing the solutions from separate clouds" thing. Let's say you like Azure's Blob Storage over AWS's S3. Guess what? You're now paying for cross-cloud latency! Both in time and in cost.
The devil is in the details.
People evaluate things differently. Some people care a great deal about social proof, others make decisions based on the color palette of the default theme. Some care a great deal about the programs having features they will never use. Etc.
> Things are becoming more like the unix philosophy, modular and shared.
I sure hope you are right.
Just look at kubernetes which wraps UNIX in a big complicated opaque 'opinionated' wet blanket.
I sure hope you are right.
Just look at kubernetes which wraps UNIX in a big complicated opaque 'opinionated' wet blanket.
Kubernetes is the operating system in this analogy, and it’s quite a modular operating system at that. Specifically, it schedules the programs that themselves should adhere to the Unix philosophy. I don’t think the Unix philosophy ever extended to the OS (although microkernels seem to be driving in that direction).
If you say so. To me kubernetes wishes it was an operating sytem.
It just puts an opaque YML fronted on a bunch of stuff I already know how to install configure and troubleshoot myself: firewalls proxies, web servers etc...
It dawned on me that people probably use it because they haven't done much sysadmin, but they want to deploy production systems.
As a sysadmin/programmer of 20+ years (what full stack used to mean I guess) it just mystifies me why someone would want to layer a barely documented front end on top of their infrastructure.
To me k8 feels kind of like a monster hybridization of cpanel & webshpere.
It just puts an opaque YML fronted on a bunch of stuff I already know how to install configure and troubleshoot myself: firewalls proxies, web servers etc...
It dawned on me that people probably use it because they haven't done much sysadmin, but they want to deploy production systems.
As a sysadmin/programmer of 20+ years (what full stack used to mean I guess) it just mystifies me why someone would want to layer a barely documented front end on top of their infrastructure.
To me k8 feels kind of like a monster hybridization of cpanel & webshpere.
Yeah, veteran sysadmins probably aren't its target audience. Orgs probably want to be able to enable their dev teams to deploy/operate production infrastructure without hiring a legion of sysadmins with decades of experience. Kubernetes, like all technology, is meant to lower the bar.
> Specifically, it schedules the programs that themselves should adhere to the Unix philosophy
Containers are not programs. And even they were, they absolutely do not adhere to the Unix philosophy.
> Kubernetes is the operating system in this analogy
I've heard this enough times and seen enough K8s systems in production that I'm labeling this "fluff". K8s is an operating system in the same way that "everything is everything". K8s is software that manages containers and their resources. I think of it more like a virtual environment than an operating system.
From Wikipedia:
> An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs.
You can generalize "software resources" as container resource limits. You can generalize "common services" as the network and filesystem layers. And you can generalize "computer hardware" as kernel configuration on the control plane.
All of these are purpose built abstractions built on top of the real operating systems. By that logic you could call any virtual environment or virtual machine an operating system. And if that's the true then the comparison isn't useful at all.
Containers are not programs. And even they were, they absolutely do not adhere to the Unix philosophy.
> Kubernetes is the operating system in this analogy
I've heard this enough times and seen enough K8s systems in production that I'm labeling this "fluff". K8s is an operating system in the same way that "everything is everything". K8s is software that manages containers and their resources. I think of it more like a virtual environment than an operating system.
From Wikipedia:
> An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs.
You can generalize "software resources" as container resource limits. You can generalize "common services" as the network and filesystem layers. And you can generalize "computer hardware" as kernel configuration on the control plane.
All of these are purpose built abstractions built on top of the real operating systems. By that logic you could call any virtual environment or virtual machine an operating system. And if that's the true then the comparison isn't useful at all.
> Containers are not programs. And even they were, they absolutely do not adhere to the Unix philosophy.
They are programs (barring some semantic nitpickery about "programs run in containers"), and like programs, we can't make blanket statements about whether or not they implement the Unix philosophy. Some do and others don't.
> I've heard this enough times and seen enough K8s systems in production that I'm labeling this "fluff". K8s is an operating system in the same way that "everything is everything". K8s is software that manages containers and their resources.
Yep, like I said, an analogy: "An operating system is software that manages processes and their resources". By definition, the operands of an analogy are different, so if your thing is taking any analogy and calling "gotcha!" because you found some differences, then I'm afraid you're missing the point.
> From Wikipedia: >> An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs. > You can generalize "software resources" as container resource limits. You can generalize "common services" as the network and filesystem layers. And you can generalize "computer hardware" as kernel configuration on the control plane. > All of these are purpose built abstractions built on top of the real operating systems. By that logic you could call any virtual environment or virtual machine an operating system. And if that's the true then the comparison isn't useful at all.
That's quite the straw man.
They are programs (barring some semantic nitpickery about "programs run in containers"), and like programs, we can't make blanket statements about whether or not they implement the Unix philosophy. Some do and others don't.
> I've heard this enough times and seen enough K8s systems in production that I'm labeling this "fluff". K8s is an operating system in the same way that "everything is everything". K8s is software that manages containers and their resources.
Yep, like I said, an analogy: "An operating system is software that manages processes and their resources". By definition, the operands of an analogy are different, so if your thing is taking any analogy and calling "gotcha!" because you found some differences, then I'm afraid you're missing the point.
> From Wikipedia: >> An operating system (OS) is system software that manages computer hardware, software resources, and provides common services for computer programs. > You can generalize "software resources" as container resource limits. You can generalize "common services" as the network and filesystem layers. And you can generalize "computer hardware" as kernel configuration on the control plane. > All of these are purpose built abstractions built on top of the real operating systems. By that logic you could call any virtual environment or virtual machine an operating system. And if that's the true then the comparison isn't useful at all.
That's quite the straw man.
"programs run in containers" isn't nitpickery - it's a very important distinction. Containers are more like a runtime environment. There's a lot going on between a K8s pod, a container, and the actual program. It's just not Unix-y at all. You can't execute containers and string them together in the same way you do with Unix programs.
Again, you can generalize and call them "programs" but the generalization renders the meaning of "program" in this context useless.
I don't see the strawman here. K8s and containerization is frequently mischaracterized as the operating system of the future. The technical details matter and speaking in generalities is misleading (IMO).
Again, you can generalize and call them "programs" but the generalization renders the meaning of "program" in this context useless.
I don't see the strawman here. K8s and containerization is frequently mischaracterized as the operating system of the future. The technical details matter and speaking in generalities is misleading (IMO).
It's not an important distinction at all. A "program" isn't well-defined; it's inherently abstract. A container is a manifestation of a running program as is a Unix process; both are "runtime environments" differing in their isolation properties (Unix processes provide memory isolation, containers provide memory as well as other kinds of isolation, at least to the extent that the underlying operating system supports).
> There's a lot going on between a K8s pod, a container, and the actual program. It's just not Unix-y at all.
The Unix philosophy pertains to composition, not implementation.
> I don't see the strawman here.
You picked someone else's definition of "operating system", generalized it, and then attacked it for being too general.
> K8s and containerization is frequently mischaracterized as the operating system of the future. The technical details matter and speaking in generalities is misleading (IMO).
Yes, again, this is an analogy. If you interpret analogies as a guarantee that the operands are identical instead of simply "similar", you will be misled.
> There's a lot going on between a K8s pod, a container, and the actual program. It's just not Unix-y at all.
The Unix philosophy pertains to composition, not implementation.
> I don't see the strawman here.
You picked someone else's definition of "operating system", generalized it, and then attacked it for being too general.
> K8s and containerization is frequently mischaracterized as the operating system of the future. The technical details matter and speaking in generalities is misleading (IMO).
Yes, again, this is an analogy. If you interpret analogies as a guarantee that the operands are identical instead of simply "similar", you will be misled.
Quite the opposite. Modern security models (which are to an extent necessary on the hostile Internet) are reserving modular sharing to the elite professional developer class, and preventing users and aftermarket developers from remixing their systems. Things like Greasemonkey are being smothered and blocked by walled-garden native apps, signatures, and browser lockdowns.
This happened to work because all the users were programmers who were already familiar with the command line, text editing, and the CVS tool, and they had already been granted access to their source control system. It wouldn't work for the general public.
Maybe the lesson is to figure out your minimum viable audience and build on the tools they already know?
Maybe the lesson is to figure out your minimum viable audience and build on the tools they already know?
This is the real lesson. Programmers prefer simple, atomic programs that are short and digestible.
Most other users need safety restraints such as a GUI. Doesn't stop you from using minimum viable product as a philosophy.
Most other users need safety restraints such as a GUI. Doesn't stop you from using minimum viable product as a philosophy.
Great post. I think this also ties into configurations. It is so annoying when getting started with a new tool requires reading through a ton of documentation and then making a big configuration file just to get it to work. Adding features can be great but adding cognitive overhead isn’t
One opinion, possibly shared by others, is that small programs are actually more versatile than larger ones. The obvious example is working with text in UNIX versus in a large, graphical program with, e.g., cascading menus full of features chosen by a company, not by users. The text processing functionalty in the large program is usually pre-determined, not usually able to be modified by users. We can say what the program can and cannot do. Whereas it is difficult to say with any certainty what are the limits of a UNIX user who is creative in her use of a variety of small, single purpose programs. She may write her own, in addition to using the ones provided.
From the first sentence of the article:
> A minimal viable program is the smallest program that solves a particular problem. It is small and beautiful. It has no additional features.
From the FreeBSD source tree (/usr/src/usr.bin/tr):
> A minimal viable program is the smallest program that solves a particular problem. It is small and beautiful. It has no additional features.
From the FreeBSD source tree (/usr/src/usr.bin/tr):
$ svn log tr.c | tail -6 | grep -v '^---'
r1590 | rgrimes | 1994-05-27 08:33:43 -0400 (Fri, 27 May 1994) | 2 lines
BSD 4.4 Lite Usr.bin Sources
https://news.ycombinator.com/item?id=5920732 most minimal viable program
> A minimal viable program is the smallest program that solves a particular problem. It is small and beautiful. It has no additional features.
Yet, when a node developer creates left-pad...
Yet, when a node developer creates left-pad...
There's two problems with leftpad:
(1) Dependencies always add certain minimal level of complexity, so a too-trivial dependency can add more-than-warranted complexity as a dependency even when the implementation itself would not do so as an integrally-maintained component of the larger system.
(2) the nature of the npm distribution ecosystem (and this is by no means unique to npm) means that the complexity and risk added by dependencies managed through that ecosystem is more than it otherwise could be with immutable, append-only repositories of dependencies.
(1) Dependencies always add certain minimal level of complexity, so a too-trivial dependency can add more-than-warranted complexity as a dependency even when the implementation itself would not do so as an integrally-maintained component of the larger system.
(2) the nature of the npm distribution ecosystem (and this is by no means unique to npm) means that the complexity and risk added by dependencies managed through that ecosystem is more than it otherwise could be with immutable, append-only repositories of dependencies.
Joe Armstrong agrued vociferously for mny years for breaking up all packages into one universal database of importable functions.
http://erlang.org/pipermail/erlang-questions/2011-May/058768...
> - contribution to open source can be as simple as contributing a single function > - there are no "open source projects" - only "the open source > Key-Value database of all functions"
Look at the examples he gives:
https://news.ycombinator.com/item?id=8342755
https://joearms.github.io/published/2015-03-12-The_web_of_na...
http://erlang.org/pipermail/erlang-questions/2011-May/058768...
> - contribution to open source can be as simple as contributing a single function > - there are no "open source projects" - only "the open source > Key-Value database of all functions"
Look at the examples he gives:
is_blank_line
is_prefix
remove_leading_whitespace
remove_trailing_whitespace
remove_leading_and_trailing_whitespace
skip_blanks
trim
https://news.ycombinator.com/item?id=8342755
https://joearms.github.io/published/2015-03-12-The_web_of_na...
What's an example of an immutable, append-only repository of dependencies? That _sounds_ intriguing but I don't follow.
I don't know of any perfect examples, but the idea is one where once a version of a library is posted, that version can no longer be deleted or modified, only later versions added.
Legal and practical constraints would probably mean any real system would be at best an approximation of this ideal, but a big part of he leftpad problem is that NPM at the time didn't even try to approximate it (the response also underlined an NPM policy existing at the time that was useful in that context but dangerous in general, in that NPM generally allowed others to claim the name of abandoned projects.)
Legal and practical constraints would probably mean any real system would be at best an approximation of this ideal, but a big part of he leftpad problem is that NPM at the time didn't even try to approximate it (the response also underlined an NPM policy existing at the time that was useful in that context but dangerous in general, in that NPM generally allowed others to claim the name of abandoned projects.)
So like the first use of a shared ledger ever for binary packages?
Maybe like C's `include`s? Where (I think) the code is in-lined without modification?
The include is immutable (unless there are macros).
Append only sounds like the included code should be evaluated after other code in a given file (so main defs would take precedent over included defs? Or main defs would be over-written by included defs... I guess it depends on compiler/evaluator implementation.
The more I type here, the more I realize how much I don't know about this stuff...
The include is immutable (unless there are macros).
Append only sounds like the included code should be evaluated after other code in a given file (so main defs would take precedent over included defs? Or main defs would be over-written by included defs... I guess it depends on compiler/evaluator implementation.
The more I type here, the more I realize how much I don't know about this stuff...
nixpkgs works like this when you pin to a specific git commit. It's effectively like vendoring all of your dependencies into your source control system, but without the overhead of storing everything locally (instead just storing hashes of dependencies).
There is creating left-pad, and there is packaging it alone in a library distribution infrastructure to be manually managed as a dependency.
There is absolutely no problem with the first one. Nobody is complaining about the author writing left-pad, only about it not being available in the standard library or some larger package. Changing the entire infrastructure so code becomes immutable and requires no trust would work too.
There is absolutely no problem with the first one. Nobody is complaining about the author writing left-pad, only about it not being available in the standard library or some larger package. Changing the entire infrastructure so code becomes immutable and requires no trust would work too.
The OP Joe Armstrong was certainly advocating for it NOT available in the standard library or some larger package.
http://erlang.org/pipermail/erlang-questions/2011-May/058768...
http://erlang.org/pipermail/erlang-questions/2011-May/058768...
I don’t think this extends to libraries, which are units of composition and each unit comes with a more-or-less fixed amount of overhead for managing the trust relationship with the maintainer (as well as other things, like managing the version to pin against, etc).
New features mean new sales opportunities, good for the company but not good for the user.
The problem with adding features to MVP is that when we ship more complex products like complete operating systems that are packed with programs, the complexity of the individual programs contributes to the complexity of the whole.
Not sure how to solve that particular problem though.
This made me think of a blogpost linked on Hacker News when Joe passed away: https://github.com/lukego/blog/issues/32. This part seems relevant here:
Joe wrote amazingly simple programs and he did so in a peculiar way. First he wrote down the program any old way just to get it out of his head. Then once it worked he would then immediately create a new directory program2 and write it again. He would repeat this process five or six times (program5, program6, ...) and each time he would understand the problem a little better and sense which parts of the program were essential enough to re-type. He thought this was the most natural thing in the world: of course you throw away the first few implementations, you didn't understand the problem when you wrote those!