A Generation Lost in the Bazaar (2012)(queue.acm.org)
queue.acm.org
A Generation Lost in the Bazaar (2012)
https://queue.acm.org/detail.cfm?id=2349257
74 コメント
> Most developers and architects seem to have lower and lower confidence in what is technically achievable without some complex dependency web, and that becomes a self-fulfilling prophecy
One of the biggest problems I’ve had with hiring engineers out of big companies is that they’ve developed a sort of learned helplessness. The longer they spend in big, slow companies the more they’ve developed an idea that everything they do must adhere to mountains of process and approvals and documents.
It can be a challenge to un-teach these ideas from people after hiring them into a company where they don’t need to put together presentations and seek multiples levels of approval before doing anything.
One surprise that I didn’t account for is that many people come to like these process-heavy systems with approvals everywhere and no progress until the rules have been followed. It becomes a safety net to pull as many different people and process into a decision before action is taken. If something goes wrong, no one can blame you because you followed the process and got all of the approvals, right? I think this leads a lot of people down the road of preferring high-process, low-action environments because following the process is a type of low-effort, low-risk work whereas producing something new is high-effort, high-risk.
One of the biggest problems I’ve had with hiring engineers out of big companies is that they’ve developed a sort of learned helplessness. The longer they spend in big, slow companies the more they’ve developed an idea that everything they do must adhere to mountains of process and approvals and documents.
It can be a challenge to un-teach these ideas from people after hiring them into a company where they don’t need to put together presentations and seek multiples levels of approval before doing anything.
One surprise that I didn’t account for is that many people come to like these process-heavy systems with approvals everywhere and no progress until the rules have been followed. It becomes a safety net to pull as many different people and process into a decision before action is taken. If something goes wrong, no one can blame you because you followed the process and got all of the approvals, right? I think this leads a lot of people down the road of preferring high-process, low-action environments because following the process is a type of low-effort, low-risk work whereas producing something new is high-effort, high-risk.
I've worked in environments like that (in and out of software), and this is almost always driven in reaction to something, which is why it's hard to unteach. In my current office there is a culture of consensus building and feedback. If someone feels they were not given sufficient opportunity to participate in a decision, they will complain, resist or otherwise cause an issue in the future. Sometimes they are correct, which is why it's not as easy as simply firing them. Sometimes the red tape is in response to problems in the past. Policies that were poorly enforced that resulted in negative consequences for customers (support failures or discriminatory behaviors). Sometimes it's there to protect low level staff from accusations of the above or to provide clear guidance when dealing with a high stakes/high stress situation. Sometimes it's to protect customers, such as regular security reviews.
So no, I do not think this is about "low effort", though that can be a side-effect. No one sits in an office wanting to find ways to make their coworkers lives difficult. More often than not the policy is a response to pressure to do something to prevent whatever failure prompted the policy. The people who like the process heavy approach, in my experience, anyway, are often remembering a time when failure to follow a procedure resulted in negative consequences for someone.
So no, I do not think this is about "low effort", though that can be a side-effect. No one sits in an office wanting to find ways to make their coworkers lives difficult. More often than not the policy is a response to pressure to do something to prevent whatever failure prompted the policy. The people who like the process heavy approach, in my experience, anyway, are often remembering a time when failure to follow a procedure resulted in negative consequences for someone.
> No one sits in an office wanting to find ways to make their coworkers lives difficult.
That’s not exactly what’s happening. It’s not about making coworkers’ lives more difficult. It’s about preferring pseudo-work that looks like progress (meetings, process, consensus-building, planning sessions, endless documents) instead of work that delivers results.
Once a project ships, everyone can see whether it was a success or failure. However, if you can keep it in the planning limbo for as long as possible you can avoid that moment of truth. Another round of meetings and refactoring and consensus building is safe because it’s easy to say you’ve followed the process and therefore produce good process-work. Shipping to customers is risky because then the product can actually fail.
In companies where someone’s reputation and promotion prospects depend on avoiding project failures, it becomes attractive (and safer) to deal more in the pseudo-work than the actual work.
That’s not exactly what’s happening. It’s not about making coworkers’ lives more difficult. It’s about preferring pseudo-work that looks like progress (meetings, process, consensus-building, planning sessions, endless documents) instead of work that delivers results.
Once a project ships, everyone can see whether it was a success or failure. However, if you can keep it in the planning limbo for as long as possible you can avoid that moment of truth. Another round of meetings and refactoring and consensus building is safe because it’s easy to say you’ve followed the process and therefore produce good process-work. Shipping to customers is risky because then the product can actually fail.
In companies where someone’s reputation and promotion prospects depend on avoiding project failures, it becomes attractive (and safer) to deal more in the pseudo-work than the actual work.
> In companies where someone’s reputation and promotion prospects depend on avoiding project failures, it becomes attractive (and safer) to deal more in the pseudo-work than the actual work.
Isn't this almost all companies? Remembering one's success and other's failure better seems to be human.
"Jim really fucked the project up" vs. "Our project was a great success" (junior dev Jim getting no mention).
Isn't this almost all companies? Remembering one's success and other's failure better seems to be human.
"Jim really fucked the project up" vs. "Our project was a great success" (junior dev Jim getting no mention).
> it becomes attractive (and safer) to deal more in the pseudo-work than the actual work.
More charitably, they feel this work will make the product better. That may or may not be true, but if you've been involved in failures and part of the after action review was "this person should have been consulted because they are the expert in whatever failed", that becomes part of the process. Pretty soon, this refactoring and consultation becomes expected, and so on until we're at the point you are speaking of.
And I say this as someone who is part of the problem. I have unfinished projects right now that are stuck in meetings and consensus building. I want, at minimum, to be able to say, "you were given the opportunity to participate in the process, but you left it to me." but I've also made mistakes and poor decisions for lack of those steps more than once.
I have no doubt that there are people avoiding failure by not shipping, but I do not think it is as simple as that, there are a variety of cultural norms and experience colliding here.
More charitably, they feel this work will make the product better. That may or may not be true, but if you've been involved in failures and part of the after action review was "this person should have been consulted because they are the expert in whatever failed", that becomes part of the process. Pretty soon, this refactoring and consultation becomes expected, and so on until we're at the point you are speaking of.
And I say this as someone who is part of the problem. I have unfinished projects right now that are stuck in meetings and consensus building. I want, at minimum, to be able to say, "you were given the opportunity to participate in the process, but you left it to me." but I've also made mistakes and poor decisions for lack of those steps more than once.
I have no doubt that there are people avoiding failure by not shipping, but I do not think it is as simple as that, there are a variety of cultural norms and experience colliding here.
I agree that procedures to follow, meant as a way to benefit from past experiences are usually well intentioned. However, a common problem usually arises over time as the nature of the work evolves: is the procedure still relevant in the current context, still beneficial despite its overhead? Better have some system that can give an idea of the actual usefulness and current relevance of whatever procedures people have to follow.
AFAIK, most people mainly focus on what they do and deliver, as it's what matters the most, but many fail to consider that how they do it might matter as much. Yet, knowing how every aspect of the environment impacts the efficiency of the work would be priceless. Being able to continuously improve and adapt the work environment so that the work keeps being done in the best possible way? One can dream ;)
The technical debt concept somewhat applies to the organizational field: what may have worked well for the small startup is almost always re-worked once the company grows, and even when the company no longer grows in size, but still in age, frequent maintenance is required for the business to stay relevant in this changing world.
Obviously, how to design how good work is done is no silver bullet, but as someone said: “I would rather have questions that can't be answered than answers that can't be questioned.” Like the not-to-be-further-questioned answer I once got: "Look, that's just how things are done around here." ;)
AFAIK, most people mainly focus on what they do and deliver, as it's what matters the most, but many fail to consider that how they do it might matter as much. Yet, knowing how every aspect of the environment impacts the efficiency of the work would be priceless. Being able to continuously improve and adapt the work environment so that the work keeps being done in the best possible way? One can dream ;)
The technical debt concept somewhat applies to the organizational field: what may have worked well for the small startup is almost always re-worked once the company grows, and even when the company no longer grows in size, but still in age, frequent maintenance is required for the business to stay relevant in this changing world.
Obviously, how to design how good work is done is no silver bullet, but as someone said: “I would rather have questions that can't be answered than answers that can't be questioned.” Like the not-to-be-further-questioned answer I once got: "Look, that's just how things are done around here." ;)
Complex process creates more work, sets barriers to entry for competitor workers, and reduces individual responsibility for something that goes wrong.
So, from the employee perspective complex process means a lot of work to do (job security), higher pay (even if the domain problem is not too difficult process complication makes it difficult reducing the available supply of qualified labor) and less responsibility if something goes wrong.
From the business point of view compexity means, being current, keeping up with the latest tech, signaling you are not a dinosaur. Keeping up with security because the shoddy earlier layers had holes because software engineers do not output a finished work product like other engineers. Even if some dinosaurs may do things cheaper and better.
From the customer perspective the incentives are pretty often the same. Many customers want a touch screen instead of a knob or button because the latest tech all has touch screens. They want to use Slack or VS Code because that's what the cool kids do these days.
From the outer space perspective it looks like humans are building a new tower of babble with their tech stacks. Ever higher layers of complexity not actually going any place or doing much new.
Eventually it might all crumble into a pile of confusion.
So, from the employee perspective complex process means a lot of work to do (job security), higher pay (even if the domain problem is not too difficult process complication makes it difficult reducing the available supply of qualified labor) and less responsibility if something goes wrong.
From the business point of view compexity means, being current, keeping up with the latest tech, signaling you are not a dinosaur. Keeping up with security because the shoddy earlier layers had holes because software engineers do not output a finished work product like other engineers. Even if some dinosaurs may do things cheaper and better.
From the customer perspective the incentives are pretty often the same. Many customers want a touch screen instead of a knob or button because the latest tech all has touch screens. They want to use Slack or VS Code because that's what the cool kids do these days.
From the outer space perspective it looks like humans are building a new tower of babble with their tech stacks. Ever higher layers of complexity not actually going any place or doing much new.
Eventually it might all crumble into a pile of confusion.
I do agree with PHK on his feelings about autotools. It is a cargo cult dumpsterfire, unfortunately it mostly works and nobody wants to go in and fix it. Also suffers from changes breaking stuff for somebody somewhere who insists that modern software should run on his ancient OS running on even older hardware.
For me personally I have a bit of a dislike for GNU software in general because of the same type of things. It's all autotools and legacy code being dragged around. Don't get me wrong, I fully appreciate the GNU project and how much they have done; I just don't like their code very much.
On a larger scale the Bazaar suffers from not having managers paying people to do boring things. This leads to boring things generally not being done as much, people on their own wants to work on interesting things.
For me personally I have a bit of a dislike for GNU software in general because of the same type of things. It's all autotools and legacy code being dragged around. Don't get me wrong, I fully appreciate the GNU project and how much they have done; I just don't like their code very much.
On a larger scale the Bazaar suffers from not having managers paying people to do boring things. This leads to boring things generally not being done as much, people on their own wants to work on interesting things.
Yes, there's a lot to be said for blowing away the cruft every now and then. At least autotools has been in a steady retreat since this was written and I don't see it too often now. It's perhaps noteworthy that this article coincided with autotools development winding down, after 2012 it didn't have a release until 2020 (https://lwn.net/Articles/834682/).
Take a look at Meson[1]. A lot of GNU, Gtk, GNOME and Freedesktops related projects have migrated or are migrating. The C, C++ and also Rust people use it.
I've started reading the Autotools manual and turned away, to much bloat and hard to read code. Meson popped up and I immediately went that direction.
[1] https://mesonbuild.com/
I've started reading the Autotools manual and turned away, to much bloat and hard to read code. Meson popped up and I immediately went that direction.
[1] https://mesonbuild.com/
My favorite thing to see when I download a project is a build system that consists of a single 6 line Makefile. Granted this usually means the project only has a handful of files to manage and it does run out of headroom as the project becomes more complex, but those build systems always work on the first try and finish quickly.
Sure it means I would run into complications on a 30 year old AIX system with some old defunct C compiler, but that's a tradeoff I'm willing to make to simplify the build process.
Sure it means I would run into complications on a 30 year old AIX system with some old defunct C compiler, but that's a tradeoff I'm willing to make to simplify the build process.
I recently came across a project with no external build system at all. Just a perl script which mimicked enough of the functionality of a build tool to get the job done. Basically "which files need rebuilding, just rebuild those". It's actually rather trivial when you look into it. Plus Makefile syntax is so bad it actually makes perl look like a hot new language.
That works fine if you’re only building for Linux. If, however, you want to run on BSD, you need something like autotools.
A short Makefile means you aren't leaning on OS specific quirks and you have a short dependency list. It usually compiles on BSD just fine.
If you aren't including headers from the "linux" directory a BSD system is almost identical to a Linux system as far as C is concerned. Common libraries like OpenSSL, zlib, libpng, and so forth are identical as well. pkg-config exists on modern BSD systems and takes a lot of the guesswork out of dependencies.
If you aren't including headers from the "linux" directory a BSD system is almost identical to a Linux system as far as C is concerned. Common libraries like OpenSSL, zlib, libpng, and so forth are identical as well. pkg-config exists on modern BSD systems and takes a lot of the guesswork out of dependencies.
As someone who used FreeBSD and Interix/SFU/SUA for a while, those were always the projects I hated the most. Just get over yourself and use autotools. Sure, your build will be a little slower, but guess what: it works.
> the Bazaar suffers from not having managers paying people to do boring things
That does definitely not work, see enterprise software
That does definitely not work, see enterprise software
Nothing particularly against managers, but that idea hits me a bad way. Managers have a tendency to want more and more in their management pool. Either people or products, if they can't get clients. So... I don't see that being a good path.
Granted, developers often do this, too. If you can find a way to align incentives with users, I'm all for it.
Granted, developers often do this, too. If you can find a way to align incentives with users, I'm all for it.
Funny thing is that the cathedral was meant to represent GNU, vs the Bazaar that was Linux.
>The bazaar meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedrals of the pre-dot-com
The pre dot com days were not a golden age of software quality.
He also has this idea that Unix is a "beautiful cathedral". No, the descriptions of the original environment where Unix was designed resembles to a internal AT&T version of a bazaar. Multics was the well architected, beautiful cathedral.
The pre dot com days were not a golden age of software quality.
He also has this idea that Unix is a "beautiful cathedral". No, the descriptions of the original environment where Unix was designed resembles to a internal AT&T version of a bazaar. Multics was the well architected, beautiful cathedral.
Hard to take the analysis seriously when it (a) take UNIX as an example and (b) pretends Linux doesn't exist.
The Linux kernel looks to be a successful Bazaar project with a single person (Linus) having nominal responsibly.
I recently return to UNIX after a 20 year break. Linux in 2020 is much better than Solaris in 1998. However, Windows made giant leaps as well. Both still have massive amounts of deadwood but Windows does a better job of hiding it.
The Linux kernel looks to be a successful Bazaar project with a single person (Linus) having nominal responsibly.
I recently return to UNIX after a 20 year break. Linux in 2020 is much better than Solaris in 1998. However, Windows made giant leaps as well. Both still have massive amounts of deadwood but Windows does a better job of hiding it.
> Linux in 2020 is much better than Solaris in 1998
Depending on your purposes, Linux in 1998, was better than Solaris in 1998, even on Sun hardware.
Depending on your purposes, Linux in 1998, was better than Solaris in 1998, even on Sun hardware.
Well he's a BSD guy, so he refers to the entire bunch as UNIX.
"One of Brooks's many excellent points is that quality happens only if somebody has the responsibility for it"
Linux is a prime example of this with Linus. I fear the day he's not around as a guiding force.
"One of Brooks's many excellent points is that quality happens only if somebody has the responsibility for it"
Linux is a prime example of this with Linus. I fear the day he's not around as a guiding force.
Windows has the vested interest (customers and money) to make their deadwood compatible.
There are some great stories about the engineering they've done to continue to support legacy applications (because those applications matter to customers).
There are some great stories about the engineering they've done to continue to support legacy applications (because those applications matter to customers).
I don’t think the author says that a Bazaar project can’t be successful.
> That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it.
I think that all the article is self disqualifying. Maybe this was acceptable language in 2012, but it was still having a bad effect on people, arrogance cannot substitute argumentation.
A Bazaar in conjunction with regulations is more sensible. As an architect at a medium sized organization my job is not too build pretty cathedrals, but to allow the teams in my department to row in the same direction. I set non functional requirements, I set direction, but I cannot know the systems that each team is developing as well as they do. Bazaars require rules, but they evolve dynamically with the needs of the organization.
I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.
I think that all the article is self disqualifying. Maybe this was acceptable language in 2012, but it was still having a bad effect on people, arrogance cannot substitute argumentation.
A Bazaar in conjunction with regulations is more sensible. As an architect at a medium sized organization my job is not too build pretty cathedrals, but to allow the teams in my department to row in the same direction. I set non functional requirements, I set direction, but I cannot know the systems that each team is developing as well as they do. Bazaars require rules, but they evolve dynamically with the needs of the organization.
I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.
Barring apocalypse, Sagrada Familia will still be standing five hundred years from now. UNIX will be mostly forgotten by then - in the same way blacksmithing and carriage building are mostly forgotten today.
That's the whole point of cathedral projects - make it beautiful and build it right because you're building for the future. Not to scratch your own personal boredom itch, or to nail a bit of wood to another bit of wood because why not.
That's the whole point of cathedral projects - make it beautiful and build it right because you're building for the future. Not to scratch your own personal boredom itch, or to nail a bit of wood to another bit of wood because why not.
That has zero to do with development style, and everything to do with medium. There have been plenty of cathedral software projects that are already lost, much less lost 500 years from now.
btw, the (purported) world's oldest running continuous market -- happens to also be called a bazaar -- was constructed in 1455, so it's been running in excess of 500 years.
> UNIX will be mostly forgotten by then
UNIX may not be the best example for your argument: it's often criticized because of how long its legacy has endured, with some people arguing it's time to move on to more modern visions of what an operating system should be. So UNIX, for good or bad, is indeed the software equivalent of an ancient cathedral which endures from ages past.
UNIX may not be the best example for your argument: it's often criticized because of how long its legacy has endured, with some people arguing it's time to move on to more modern visions of what an operating system should be. So UNIX, for good or bad, is indeed the software equivalent of an ancient cathedral which endures from ages past.
Indeed, UNIX will definitely not be forgotten 5 centuries from now (if humanity still exists by then). It will at least be in the history books, taught about in schools, and some quirky people will for sure be running it as a "retro computing" hobby.
Still, looking at ancient constructions, 150 years is a huge amount of time. Most roman constructions that have survived to today were built at most in a single life time - at an extreme end, it seems that the Hagia Sophia was built in 5 years. The more than 4500 years old Great Pyramid of Giza was built in a single pharaoh's lifetime.
Which is to say, engineering culture and goals can absolutely degrade over time in terms of productivity, even with great increases in construction technology.
Which is to say, engineering culture and goals can absolutely degrade over time in terms of productivity, even with great increases in construction technology.
Agreed. People don't flourish inside cathedrals, no matter how attractive they look.
> Agreed. People don't flourish inside cathedrals, no matter how attractive they look.
Nor do they flourish inside supermarkets, these buildings are designed for something else. Walking in St. Peter's in Roma gives a sensation of awe even to devout atheists, something I can't imagine the local Walmart doing.
Nor do they flourish inside supermarkets, these buildings are designed for something else. Walking in St. Peter's in Roma gives a sensation of awe even to devout atheists, something I can't imagine the local Walmart doing.
Didn't Yeltsin experience something akin to a religious conversion on seeking an American supermarket? We're only numb to the miracle because we see it every day.
Off-topic, but all that the St Peter's Cathedral in Rome gave me was of obscene wealth and displays of wealth.
For awe, I would much more recommend the Pantheon, especially knowing that you're stepping into a building for which 500 years is barely more than a quarter of its history.
For awe, I would much more recommend the Pantheon, especially knowing that you're stepping into a building for which 500 years is barely more than a quarter of its history.
Of course, the Pantheon that stands today is not even the original, it was rebuilt at least twice, from what I can tell.
One of the emperors that rebuilt it still kept the "Agrippa did this" inscription, you don't mess with Augustus.
One of the emperors that rebuilt it still kept the "Agrippa did this" inscription, you don't mess with Augustus.
> Off-topic, but all that the St Peter's Cathedral in Rome gave me was of obscene wealth and displays of wealth.
Even further off-topic, but if you are referring to the big church in the Vatican adjacent to St. Peter’s Square, it is Saint Peter’s Basilica; it is the second-ranking of the four major basilicas but it is not (though many tourist-focussed lists of cathedrals include it) a cathedral.
The cathedral that is the seat of the Bishop of Rome (and also the first-ranking of the four major basilicas) is the Archbasilica of St. John Lateran.
Even further off-topic, but if you are referring to the big church in the Vatican adjacent to St. Peter’s Square, it is Saint Peter’s Basilica; it is the second-ranking of the four major basilicas but it is not (though many tourist-focussed lists of cathedrals include it) a cathedral.
The cathedral that is the seat of the Bishop of Rome (and also the first-ranking of the four major basilicas) is the Archbasilica of St. John Lateran.
Yes, you are right.
I think if everyone is simply rowing in one direction, you end up with a favela - functional, but not great in an earthquake or flood.
I once talked to a CEO who expected and wanted an important piece of software to be rewritten every few months so it could use the app framework du jour.
I wonder how many other shops there are where implementations aren't designed to last long, especially in web development.
I wonder how many other shops there are where implementations aren't designed to last long, especially in web development.
[deleted]
> I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.
Interestingly, this may also be a sign of declining engineering. The Hagia Sophia, the largest church in the world for 1000 years, was apparently built in 5 years (532- 537).
In general, it seems Rome was able to build fantastically persistent buildings in fractions of the time required later. Even today I wonder if we actually have the engineering and logistics practices required to build buildings that would survive thousands of years in a under a decade - of which the Romans had several.
Interestingly, this may also be a sign of declining engineering. The Hagia Sophia, the largest church in the world for 1000 years, was apparently built in 5 years (532- 537).
In general, it seems Rome was able to build fantastically persistent buildings in fractions of the time required later. Even today I wonder if we actually have the engineering and logistics practices required to build buildings that would survive thousands of years in a under a decade - of which the Romans had several.
>>Interestingly, this may also be a sign of declining engineering.
Or it could simply mean that not enough resources were committed to the project.
Or it could simply mean that not enough resources were committed to the project.
500 AD, that's about the time the Western Roman Empire collapsed, of course the Byzantines needed a gigantic project to show they were the real Roman empire now. It's probably more similar to the Manhattan Project.
> Maybe this was acceptable language in 2012, but it was still having a bad effect on people, arrogance cannot substitute argumentation.
I find it refreshingly honest, vivid, cathartic and spot on. Complaints from the peanut gallery just make me love it even more.
In 9 or 90 years, maybe they'll think all of our 2021 eggshell tiptoeing to be completely useless drivel devoid of insight and dump it down the memory hole. Sensibilities change!
I find it refreshingly honest, vivid, cathartic and spot on. Complaints from the peanut gallery just make me love it even more.
In 9 or 90 years, maybe they'll think all of our 2021 eggshell tiptoeing to be completely useless drivel devoid of insight and dump it down the memory hole. Sensibilities change!
>I love the sagrada familia un Barcelona, but it's still under construction after 150 years. Building cathedrals maybe not the way.
Well, they are not jus nailing two pieces of wood.
Also, if I were to build a cathedral today I would use 3d printing and CNC milling machines as much as possible.
Well, they are not jus nailing two pieces of wood.
Also, if I were to build a cathedral today I would use 3d printing and CNC milling machines as much as possible.
For something like a cathedral, having it take a long time is a feature. Of course you could slap something of the same size together from prefab concrete blocks and some 3d printed stuff, but you would "just" get a soulless building with no effort from the faithful invested in it.
A real cathedral is a monument for the ages, bigger than any individual who worked on it. Something for entire generations to pour their effort into so that at the end of construction you can say: "my grandfather started building this, my parents worked on it, I worked on it and during the time of my great-grandchildren this building will still be as magnificent as it is now".
Trying to build it as quick and cheap as possible is simply not the point for a cathedral.
A real cathedral is a monument for the ages, bigger than any individual who worked on it. Something for entire generations to pour their effort into so that at the end of construction you can say: "my grandfather started building this, my parents worked on it, I worked on it and during the time of my great-grandchildren this building will still be as magnificent as it is now".
Trying to build it as quick and cheap as possible is simply not the point for a cathedral.
Overly dramatic headline is overly dramatic, and sub-headline is so obvious that it should be self-evident.
I can't help but think all articles like this are fighting a very old and outdated strawman. I'd venture that literally no one is arguing that "Free Software is magic" anymore.
Look, sometimes Free Software is the thing that works well, sometimes non-free but open, sometimes proprietary is what you have to use. There are a lot of things we can work on and do better, and concrete solutions and experiments are important to consider: but this sort of big picture whining isn't helping anyone.
I can't help but think all articles like this are fighting a very old and outdated strawman. I'd venture that literally no one is arguing that "Free Software is magic" anymore.
Look, sometimes Free Software is the thing that works well, sometimes non-free but open, sometimes proprietary is what you have to use. There are a lot of things we can work on and do better, and concrete solutions and experiments are important to consider: but this sort of big picture whining isn't helping anyone.
I love that article, but it made me sad. It is just as relevant today, as it was, nine years ago.
I'm a huge proponent of improving software Quality. I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.
So, my approach, is to do the very best I can, in my own work, and limit the amount of other people's work on which I rely. When I do use it, I am quite careful, and vet the work. Even so, I have had to back out several highly-rated packages that I had thought were good, but my apps started crashing, as soon as I incorporated them.
Just mentioning that, has often proved incredibly unpopular. It seems to be considered blasphemy.
It does reduce the scope of what I can accomplish, but I am quite happy with what I do.
I'm a huge proponent of improving software Quality. I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.
So, my approach, is to do the very best I can, in my own work, and limit the amount of other people's work on which I rely. When I do use it, I am quite careful, and vet the work. Even so, I have had to back out several highly-rated packages that I had thought were good, but my apps started crashing, as soon as I incorporated them.
Just mentioning that, has often proved incredibly unpopular. It seems to be considered blasphemy.
It does reduce the scope of what I can accomplish, but I am quite happy with what I do.
Re:
> I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.
IMO creating real change in an organization is almost never done through complaining, it’s done through pushing practical solutions in a positive/constructive way. It might not be the content of your argument, but the way you present it. This comment, for example, instantly biased me against your argument, because the tone felt complain-y and self-righteous. I think regardless of what your argument is, if you present it with this tone, you won’t be persuasive.
> I have found that complaining about it doesn't actually do much good. My complaints tend to get downvoted into the Mantle, and I just get pissed off.
IMO creating real change in an organization is almost never done through complaining, it’s done through pushing practical solutions in a positive/constructive way. It might not be the content of your argument, but the way you present it. This comment, for example, instantly biased me against your argument, because the tone felt complain-y and self-righteous. I think regardless of what your argument is, if you present it with this tone, you won’t be persuasive.
Oh, I wasn't talking about in my organization (which was actually famous for Quality). I was talking about making posts in tech communities, like this one.
Software Quality is not a popular topic for discussion. HN isn't too bad, but some other communities react to Quality discussions, as if I was insulting God.
I've learned to just keep my opinions to myself. Even then, when I talk about my own experiences, and my own process, it is not received well.
Software Quality is not a popular topic for discussion. HN isn't too bad, but some other communities react to Quality discussions, as if I was insulting God.
I've learned to just keep my opinions to myself. Even then, when I talk about my own experiences, and my own process, it is not received well.
A touché, but regardless, I think tone and the way you make your argument matters a lot, vs. just the position you support. I personally see a tonne of people passionate about software quality on HN, and often their arguments are very well received.
If your argument comes off as “I don’t use libraries because everyone else’s software is crap, so I write everything myself,” people aren’t going to respond well to that. But if you can make a convincing argument for why it makes sense to avoid dependencies in specific situations, I’d imagine you’ll get a positive reaction.
If your argument comes off as “I don’t use libraries because everyone else’s software is crap, so I write everything myself,” people aren’t going to respond well to that. But if you can make a convincing argument for why it makes sense to avoid dependencies in specific situations, I’d imagine you’ll get a positive reaction.
I don't really get too bent out of shape about other folks' code. I've found that it doesn't really help my position to compare against others'. If the only way that I can justify my own worth, is to tear down others, then my own position isn't so good.
But I have had issues, when incorporating the work of others. I do use a couple of dependencies, in my work; but just a couple (besides the default ones, like standard libs, and platform SDKs). That's a big reason that I don't use cross-platform frameworks, like QT or React Native. They are fine, but I feel that they reduce the quality of the user experience. I've tried using them for decades (cross-platform frameworks are nothing new), and have never been happy with the result.
I am always nervous about running the critical path through dependencies. I worked on a project, a few years ago, that was completely based around a particular tool; assuming the tool would improve, and that our implementation of it would improve, as we learned more.
Neither happened. In fact, the more we learned about the tool, the more we realized how unsuitable it was. That ended up being a real disaster. Basically flushed a couple of years (and several million dollars) down the drain.
So I tend to stick with native, and I also write my own dependencies. I believe in modular architecture. It does have a few issues, but these can be greatly mitigated, if I have control of the whole stack.
I come from an embedded background. I have worked with computers, right down to the FET junctions, and I like to know what's going on, everywhere. I'm uncomfortable relying on stuff I don't understand. I've learned to relax, over the years. Compiler toolchains are really quite good, and I don't understand them (in detail). I don't understand AI/ML, and I'm not quite to the point, where I incorporate them without question, but I think we'll be at that point, fairly soon.
I'm kind of amazed that a lot of folks google a dependency, then add it as a critical path component, without a second thought. I guess it has positive results (meaning that money is made) often enough, but I consider that living a bit too dangerously, for my comfort.
But I have had issues, when incorporating the work of others. I do use a couple of dependencies, in my work; but just a couple (besides the default ones, like standard libs, and platform SDKs). That's a big reason that I don't use cross-platform frameworks, like QT or React Native. They are fine, but I feel that they reduce the quality of the user experience. I've tried using them for decades (cross-platform frameworks are nothing new), and have never been happy with the result.
I am always nervous about running the critical path through dependencies. I worked on a project, a few years ago, that was completely based around a particular tool; assuming the tool would improve, and that our implementation of it would improve, as we learned more.
Neither happened. In fact, the more we learned about the tool, the more we realized how unsuitable it was. That ended up being a real disaster. Basically flushed a couple of years (and several million dollars) down the drain.
So I tend to stick with native, and I also write my own dependencies. I believe in modular architecture. It does have a few issues, but these can be greatly mitigated, if I have control of the whole stack.
I come from an embedded background. I have worked with computers, right down to the FET junctions, and I like to know what's going on, everywhere. I'm uncomfortable relying on stuff I don't understand. I've learned to relax, over the years. Compiler toolchains are really quite good, and I don't understand them (in detail). I don't understand AI/ML, and I'm not quite to the point, where I incorporate them without question, but I think we'll be at that point, fairly soon.
I'm kind of amazed that a lot of folks google a dependency, then add it as a critical path component, without a second thought. I guess it has positive results (meaning that money is made) often enough, but I consider that living a bit too dangerously, for my comfort.
Yeah, I get the “I like to understand how everything works” angle, and I like implementing things from scratch in personal projects. When working with a team, I feel it really depends on the team and the business. I think if you’re extremely focused on quality, and don’t need a really rapid pace of development (especially in the earlier days), then going low/no dependency can make sense. It’s certainly working for SQLite!
On the flip side … some of the worst software mistakes at places I’ve worked at have been of the “let’s build some common thing ourselves” variety. For example, for some crazy reason, someone implemented their own HTTP-esque protocol on top of ZeroMQ, complete with servers and clients in multiple languages, that was used in a massive number of services. A total disaster, insane amounts of dev work invested in it, tonnes of bugs and performance issues, eventually we ripped it out and replaced it with HTTP, but it took literally years of work to do so. Less extreme were in-house service discovery tooling, and an in-house version of something like Redux. Tonnes of investment to build them, poorly documented with poor test coverage and lots of bugs. Then the main author/advocate leaves the company, it becomes a huge maintenance burden, and eventually it’s ripped out and replaced with a popular open source version at great cost/effort. In all cases, the in-house version was massively lower quality than the top open source alternatives, in basically every conceivable way.
I agree that introducing a dependency is a significant decision, you should research the dependency well and not add it lightly. But building everything in-house can be a major mistake with massive unnecessary costs, you need pretty specific use cases and the right company culture for it to work out.
On the flip side … some of the worst software mistakes at places I’ve worked at have been of the “let’s build some common thing ourselves” variety. For example, for some crazy reason, someone implemented their own HTTP-esque protocol on top of ZeroMQ, complete with servers and clients in multiple languages, that was used in a massive number of services. A total disaster, insane amounts of dev work invested in it, tonnes of bugs and performance issues, eventually we ripped it out and replaced it with HTTP, but it took literally years of work to do so. Less extreme were in-house service discovery tooling, and an in-house version of something like Redux. Tonnes of investment to build them, poorly documented with poor test coverage and lots of bugs. Then the main author/advocate leaves the company, it becomes a huge maintenance burden, and eventually it’s ripped out and replaced with a popular open source version at great cost/effort. In all cases, the in-house version was massively lower quality than the top open source alternatives, in basically every conceivable way.
I agree that introducing a dependency is a significant decision, you should research the dependency well and not add it lightly. But building everything in-house can be a major mistake with massive unnecessary costs, you need pretty specific use cases and the right company culture for it to work out.
Well, the classic example of “don’t roll your own,” is encryption. But choosing an encryption system is one of the more important decisions a developer makes, as an error can be an “E.L.E,” so to speak.
https://m.youtube.com/watch?v=j0silSyYFPM
https://m.youtube.com/watch?v=j0silSyYFPM
The problem is, we get less of you and more of them, it damages us all. As long as you keep writing good software though, that's what really matters.
We used to stand in a field where everywhere we looked we could see and hear another of us, now there's 10,000 fools between each of us, and it's rare for us to hear or see another over the crowd.
We used to stand in a field where everywhere we looked we could see and hear another of us, now there's 10,000 fools between each of us, and it's rare for us to hear or see another over the crowd.
Well, the issue with discussing Quality, is that it inevitably turns to dependencies, and that's where the "My Precious" stuff starts to happen.
> The bazaar meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedrals of the pre-dot-com years, unfortunately did, not die with the dot-com madness, and today Unix is rapidly sinking under its weight.
A level-headed comparison of actual historic Unix sources, which are publicly available for study, reveals this to be false. Unix was always "just hack it", and later Unix is, overall, an improvement over earlier Unix.
It seems I have not come across test cases in old Unix code bases.
Can someone point to some circa 1980 Unix code base where there test cases? For anything: system calls, shell, yacc, cc, ..
I think the idea may have been that if Unix rebuilds from scratch and you get to a system prompt where you can type "ls", the toolchain has been validated.
A level-headed comparison of actual historic Unix sources, which are publicly available for study, reveals this to be false. Unix was always "just hack it", and later Unix is, overall, an improvement over earlier Unix.
It seems I have not come across test cases in old Unix code bases.
Can someone point to some circa 1980 Unix code base where there test cases? For anything: system calls, shell, yacc, cc, ..
I think the idea may have been that if Unix rebuilds from scratch and you get to a system prompt where you can type "ls", the toolchain has been validated.
If I could up vote this 100 times I would. I love open source. But 40 years in computing (hardware to apps) has left me with questions. Like, Why could I write an entire Point of Sale system in 64K in 1987 (Apple - The Store Manager) and now you couldn't do it in 64G.
as I see it, it comes down to ownership.
Something about treating something which ain't yours (such as a software system with millions of lines of code) as if it were.
Something about maintaining a complex software system which you are responsible for, but somehow are never allowed to make simpler for your own sake. You are only paid for work which increase the "value" (money, profit, whatever) of the real owners of the business. It's not really worth their money to pay you to make your own job easier.
> Quality happens only when someone is responsible for it.
The easy way for this to happen is to be able to own, really own what you do. This just doesn't happen in the 'software industry'.
Something about treating something which ain't yours (such as a software system with millions of lines of code) as if it were.
Something about maintaining a complex software system which you are responsible for, but somehow are never allowed to make simpler for your own sake. You are only paid for work which increase the "value" (money, profit, whatever) of the real owners of the business. It's not really worth their money to pay you to make your own job easier.
> Quality happens only when someone is responsible for it.
The easy way for this to happen is to be able to own, really own what you do. This just doesn't happen in the 'software industry'.
Fully agree. This is one of the implicit advantages of most corporate structures; the ownership question has default answers.
Who owns the product? The company.
Who maintains the product? Whoever the company pays to maintain it. Usually, companies will build out trees of responsibility so some sub-group is responsible for smaller pieces of large products (with ultimate responsibility "rolling up" back to company ownership).
How can I contribute? Your boss will tell you what needs maintenance or improvement. Do it or you're fired. If you have something else you want to work on, a good company will respond to your feedback and shuffle responsibilities around (you get better work when employees are satisfied), but ultimately, someone needs to maintain the build tools and if that's what they hired you to do, that's what you're doing until you decide to stop taking their money for it.
There are lots of disadvantages to the corporate model, but (like the old chestnut from Thomas Paine's writings), it has the advantage of simple ownership.
Who owns the product? The company.
Who maintains the product? Whoever the company pays to maintain it. Usually, companies will build out trees of responsibility so some sub-group is responsible for smaller pieces of large products (with ultimate responsibility "rolling up" back to company ownership).
How can I contribute? Your boss will tell you what needs maintenance or improvement. Do it or you're fired. If you have something else you want to work on, a good company will respond to your feedback and shuffle responsibilities around (you get better work when employees are satisfied), but ultimately, someone needs to maintain the build tools and if that's what they hired you to do, that's what you're doing until you decide to stop taking their money for it.
There are lots of disadvantages to the corporate model, but (like the old chestnut from Thomas Paine's writings), it has the advantage of simple ownership.
Yeah, yeah, autoconf sucks, so does maven (IMHO), software is complex, and, yes, we could do better... But, to reason from these examples about thousands of projects is disingenuous.
The rumors of open source's demise have been greatly exaggerated.
I use open source both privately (Linux, some open source games, FreeBSD a long time ago), and professionally (mostly Java and database stuff), and while sometimes I can't help rolling my eyes, on the whole open source works incredibly and surprisingly well; especially compared to the alternative of closed-only corporate software.
The rumors of open source's demise have been greatly exaggerated.
I use open source both privately (Linux, some open source games, FreeBSD a long time ago), and professionally (mostly Java and database stuff), and while sometimes I can't help rolling my eyes, on the whole open source works incredibly and surprisingly well; especially compared to the alternative of closed-only corporate software.
Unix was invented long before modern dynamic linking schemes were. PHK seems to think it was possible, around 1990 or before, to wave a magic wand and have all vendors agree on a way to build and link dynamic libraries. There are a number of decisions to be made before you even get there. First, which binary format -- COFF? ELF? Something else? Secondly, how do you account for the fact that the library will be loaded into different programs' memory spaces at run time? Do you:
1) insist on PIC (what Linux does)?
2) replace all pointers with dummies and fix them up at load time (what Windows does)?
3) always load a given library to the same place in virtual memory (what Linux did in the a.out days)?
Before you have a single compiler flag for dynamic library generation, you have to answer questions like this. And no single answer existed in the Unix environment of 1990, which wasn't a bazaar -- it was more like a struggle among competing cathedrals! Hence libtool, whose messiness reflects the messiness of the Unix scene at the time. GNU actually made monumental efforts to clean up the different Unixen and make them more consistent.
Fortunately, today, all widely used Unixen these days seem to have settled on ELF format and option one -- all, that is, except macOS!
1) insist on PIC (what Linux does)?
2) replace all pointers with dummies and fix them up at load time (what Windows does)?
3) always load a given library to the same place in virtual memory (what Linux did in the a.out days)?
Before you have a single compiler flag for dynamic library generation, you have to answer questions like this. And no single answer existed in the Unix environment of 1990, which wasn't a bazaar -- it was more like a struggle among competing cathedrals! Hence libtool, whose messiness reflects the messiness of the Unix scene at the time. GNU actually made monumental efforts to clean up the different Unixen and make them more consistent.
Fortunately, today, all widely used Unixen these days seem to have settled on ELF format and option one -- all, that is, except macOS!
This is a great article that I find myself re-reading and referencing all the time.
I think it's unfortunate that people only skim over it and then use it to complain/defend Linux/windows/m4/autoconf/etc. More important is the main point that "quality only happens when someone is responsible for it". This explains a mistake I see time and time again in all manners of software projects: anything by committee, anything overly democratic, can only aim to be mediocre at best. You need an owner/architect/champion/tyrant if you hope to achieve anything above average (not guaranteed though).
I think it's unfortunate that people only skim over it and then use it to complain/defend Linux/windows/m4/autoconf/etc. More important is the main point that "quality only happens when someone is responsible for it". This explains a mistake I see time and time again in all manners of software projects: anything by committee, anything overly democratic, can only aim to be mediocre at best. You need an owner/architect/champion/tyrant if you hope to achieve anything above average (not guaranteed though).
I occasionally contemplate this article (and am glad I found it again!), mostly because of this quote:
> there is no standardized way to build a shared library in Unix. Instead of standardizing how to do that across all Unixen—something that would take just a single flag to the ld(1) command
I've never looked into it, but I'd love to know what ld flag PHK was thinking about. Could the entirety of libtool still be replaced by that flag?
> there is no standardized way to build a shared library in Unix. Instead of standardizing how to do that across all Unixen—something that would take just a single flag to the ld(1) command
I've never looked into it, but I'd love to know what ld flag PHK was thinking about. Could the entirety of libtool still be replaced by that flag?
[deleted]
There is truth in this article, but I do not feel as pessimistic as the author seems to feel.
I think we need to push for more quality in Open Source software, and in commercial software. That quality can come from a team, though it seems more likely to be driven by a charismatic BDFL. The quality though comes from tests, attention to detail, QA, etc.
I think we need to push for more quality in Open Source software, and in commercial software. That quality can come from a team, though it seems more likely to be driven by a charismatic BDFL. The quality though comes from tests, attention to detail, QA, etc.
Is the book mentioned in this article worth reading?
fddddd(1)
Commenters here I think are missing what’s most interesting about this essay - it’s a sort of paean to an age we’ve now forgotten, an age that had ended in 2012, written by a high priest (published in the ACM!)
Poul Henning-Kamp has been around long enough to see the end of his system and the rise of what Raymond talked about; high priests like Henning-Kamp just don’t matter as much as they used to, compared to the insane productivity of the entire world pumping out stuff whenever and however they want, gluing together whatever they need.
There is a lot of shit being made, sold and given away in the Bazaar, but there are a lot of gems. A tech startup cost millions of dollars to start in say, 2008, the height of the dotcom one boom. For reference, discount hosting providers charged on the order of $30,000 a month for a single Sun server to serve your website.
Today, a solid startup can glue together hundreds or thousands of open source libraries and launch product for free. Even if you ignore free time from the AWSes of the world, many a startup can structure their infra around serverless functions and be paying in the 10s of dollars a month. That’s the power of the bazaar, or whatever system we have now, which is more complex than Raymond described.
I think the vast majority of users of this site do not want to return to a world where releasing products takes someone of Henning-Kamp’s intelligence and skill - there just aren’t that many of him in the world, and we would have far fewer awesome things in that world.
Poul Henning-Kamp has been around long enough to see the end of his system and the rise of what Raymond talked about; high priests like Henning-Kamp just don’t matter as much as they used to, compared to the insane productivity of the entire world pumping out stuff whenever and however they want, gluing together whatever they need.
There is a lot of shit being made, sold and given away in the Bazaar, but there are a lot of gems. A tech startup cost millions of dollars to start in say, 2008, the height of the dotcom one boom. For reference, discount hosting providers charged on the order of $30,000 a month for a single Sun server to serve your website.
Today, a solid startup can glue together hundreds or thousands of open source libraries and launch product for free. Even if you ignore free time from the AWSes of the world, many a startup can structure their infra around serverless functions and be paying in the 10s of dollars a month. That’s the power of the bazaar, or whatever system we have now, which is more complex than Raymond described.
I think the vast majority of users of this site do not want to return to a world where releasing products takes someone of Henning-Kamp’s intelligence and skill - there just aren’t that many of him in the world, and we would have far fewer awesome things in that world.
As believing something is possible is a pre-requisite to actually approaching it, I expect things to continue getting worse.