Why internal SW is almost invariably sh_t?
31 comments
I'm going to use an analogy from my hobby: because internal software is like shop furniture in a woodworking shop. Those things are mostly used in the production of other things that make the company revenue. It does its job, has room for many improvements, but isn't something you'd go out and sell as a product. It might be built from scraps, using crappy lumber and pieced together with whatever fasteners were around or cheap.
Open Source tooling on the other hand is like independent woodworker sharing plans with their community. They want it to be the best they can produce because they know their peers are going to see it, possibly use it, possibly mention it to others if its really good.
Open Source tooling on the other hand is like independent woodworker sharing plans with their community. They want it to be the best they can produce because they know their peers are going to see it, possibly use it, possibly mention it to others if its really good.
There are other reasons too:
* Time pressure - OSS is rarely under deadline pressure so the temptation to half ass it is mitigated.
* Survivorship bias - there's actually a lot of OSS shit out there too but it is much easier to ignore coz nobody is gonna force you to use it.
* Time pressure - OSS is rarely under deadline pressure so the temptation to half ass it is mitigated.
* Survivorship bias - there's actually a lot of OSS shit out there too but it is much easier to ignore coz nobody is gonna force you to use it.
Similar to your survivorship bias, just because something is done for free doesn't mean it's an insignificant "nothing" project, and just because you sell something doesn't mean much effort was put into it.
You could be comparing the work of hundreds of dedicated engineers for over a decade to something someone threw together in an afternoon, that was never intended for mainstream use... sure they're charging $20k for it, but that doesn't make it "good".
You could be comparing the work of hundreds of dedicated engineers for over a decade to something someone threw together in an afternoon, that was never intended for mainstream use... sure they're charging $20k for it, but that doesn't make it "good".
There are so many reasons for this. Here are some of them that I can write without much thinking or checking my notes.
1. Enterprise problem domains are usually much more complex than the ones which are targeted by the OSS.
2. Enterprise problem domains are full of unclear, wrong and even contradicting requirements.
3. Enterprises usually have lots of legacy code/systems.
4. Enterprises usually have lots of glue code duct-taping various internal systems.
5. If nobody will see your code publicly, then you have less incentive to invest in its UX/DX/documentation/code quality/etc. You incentivized to take shortcuts and use quick-and-dirty solutions.
6. Even though usually enterprises have internal SQA teams, it's no match to many testers in the open-source world. Internal SW is only deployed in a single environment/platform and tested with the happy path only. OSS is tested in lots of different environments/platforms with lots of edge cases.
7. Unlike the OSS world, the developers working in enterprises are usually 9-5ers, working-to-live, not living-to-work.
8. The quality/skill level of developers is usually lower. They're usually become experts in their business domain, but their coding skills may remain on the "advanced beginner" level.
9. They usually working under micromanagement-like SDLCs which impede their productivity and code quality: "Agile"/SAFe/Scrum/daily standups.
Some of the things I wrote might be anecdotes or stereotypes, so certainly not all enterprise developers are like this.
1. Enterprise problem domains are usually much more complex than the ones which are targeted by the OSS.
2. Enterprise problem domains are full of unclear, wrong and even contradicting requirements.
3. Enterprises usually have lots of legacy code/systems.
4. Enterprises usually have lots of glue code duct-taping various internal systems.
5. If nobody will see your code publicly, then you have less incentive to invest in its UX/DX/documentation/code quality/etc. You incentivized to take shortcuts and use quick-and-dirty solutions.
6. Even though usually enterprises have internal SQA teams, it's no match to many testers in the open-source world. Internal SW is only deployed in a single environment/platform and tested with the happy path only. OSS is tested in lots of different environments/platforms with lots of edge cases.
7. Unlike the OSS world, the developers working in enterprises are usually 9-5ers, working-to-live, not living-to-work.
8. The quality/skill level of developers is usually lower. They're usually become experts in their business domain, but their coding skills may remain on the "advanced beginner" level.
9. They usually working under micromanagement-like SDLCs which impede their productivity and code quality: "Agile"/SAFe/Scrum/daily standups.
Some of the things I wrote might be anecdotes or stereotypes, so certainly not all enterprise developers are like this.
This list seems to carry a heap of bias with it. Enterprise cares about what works for them, and quick-and-dirty is just fine. Startups care about what works for a broader customer base and need higher quality. Different needs, and different styles. Neither approach is wrong... but they serve different purposes and will appear like cruddy work if you put an enterprise approach into the startup world. And vice versa.
The one point which seems completely wrong is #9 - I've never seen agile-hamster-wheel-rituals bog down an enterprise shop. I've seen the other extreme, where in an enterprise, platforms are dealt out to the devs like a deck of cards, and as long as you keep your platforms running and working for the business, your manager barely even checks in with you. I'd easily go days, often weeks, and sometimes even months without any discussions at all with my boss.
The one point which seems completely wrong is #9 - I've never seen agile-hamster-wheel-rituals bog down an enterprise shop. I've seen the other extreme, where in an enterprise, platforms are dealt out to the devs like a deck of cards, and as long as you keep your platforms running and working for the business, your manager barely even checks in with you. I'd easily go days, often weeks, and sometimes even months without any discussions at all with my boss.
Worth re-iterating that OP was just explaining the various bias and stereotypes. I think the comment speaks more to perception than reality. There really isn't any good reason to think open source good, closed source bad.
I will say that source control is one area where enterprise is almost total garbage, probably for some of the reasons listed (as well as a lot of unlisted reasons like customers being scared of change). There's probably many other examples where this holds true, and some where it doesn't.
I will say that source control is one area where enterprise is almost total garbage, probably for some of the reasons listed (as well as a lot of unlisted reasons like customers being scared of change). There's probably many other examples where this holds true, and some where it doesn't.
> The one point which seems completely wrong is #9 - I've never seen agile-hamster-wheel-rituals bog down an enterprise shop. I've seen the other extreme, where in an enterprise, platforms are dealt out to the devs like a deck of cards, and as long as you keep your platforms running and working for the business, your manager barely even checks in with you. I'd easily go days, often weeks, and sometimes even months without any discussions at all with my boss.
I've seen it all. At one place it went from Cowboy Coding where you were left to your own devices for months, straight to SAFe/Scrum/standups;) Both are equally bad and no match to RUP, informal mini-waterfall (think->design->build(iterate)->ship) or Shape Up method.
I've seen it all. At one place it went from Cowboy Coding where you were left to your own devices for months, straight to SAFe/Scrum/standups;) Both are equally bad and no match to RUP, informal mini-waterfall (think->design->build(iterate)->ship) or Shape Up method.
But how are you defining "equally bad"?
Lets ask what it means for tech work to be "bad". If I kept the business working, the tech evolving, while saving the business money and meeting every need it asked, including migration to new tech as the systems aged... which part is bad?
Lets ask what it means for tech work to be "bad". If I kept the business working, the tech evolving, while saving the business money and meeting every need it asked, including migration to new tech as the systems aged... which part is bad?
1. Cowboy Coding - means no knowledge transfer, bus factor of 1 - bad for business.
2. Agile Manifesto says "working software over comprehensive documentation", which also leads to information silos and no knowledge transfer, and thus bad for business.
Did you ever inherited an enterprise project without any documentation, written in an esoteric language on a proprietary platform, with one-letter identifiers and without comments, and then the new management asking you to reverse engineer it and to explain to them what it does?
3. "Agile"/SAFe/Scrum are incentivizing short-term over long-term thinking. It's like picking up small-value coins on the street in front of a steamroller: you make small profits until you get wiped out. The Demo meetings are like Social Media dopamine injections - you feel good, but it's impossible to build something meaningful in a 1-3 weeks sprint.
2. Agile Manifesto says "working software over comprehensive documentation", which also leads to information silos and no knowledge transfer, and thus bad for business.
Did you ever inherited an enterprise project without any documentation, written in an esoteric language on a proprietary platform, with one-letter identifiers and without comments, and then the new management asking you to reverse engineer it and to explain to them what it does?
3. "Agile"/SAFe/Scrum are incentivizing short-term over long-term thinking. It's like picking up small-value coins on the street in front of a steamroller: you make small profits until you get wiped out. The Demo meetings are like Social Media dopamine injections - you feel good, but it's impossible to build something meaningful in a 1-3 weeks sprint.
I think we agree on many things, but your comment makes me feel like you jumped from me being independent to assuming Cowboy Coding. You are assuming that because some enterprise systems are undocumented, that is what every enterprise coder does. The Agile Manifesto also is clear that putting one side above the other does not mean the other is not important.
FWIW, yes, I've taken over horrid undocumented systems... and figured them out, added documentation, and gotten them handed over into sustainable models. On the Agile side, I've been on startup teams that delivered amazing feature sets in a 3 week sprint, and did so sprint after sprint... but I've also been on teams who churned through sprints with nothing to show for it.
I don't think we can boil down a good vs. bad environment, or good vs. bad code, to SDLC processes or enterprise vs. startup. There are good teams, there are bad teams. And matching up the right environment and process to the right team can work wonders. But to be fair, most teams haven't found the right match.
FWIW, yes, I've taken over horrid undocumented systems... and figured them out, added documentation, and gotten them handed over into sustainable models. On the Agile side, I've been on startup teams that delivered amazing feature sets in a 3 week sprint, and did so sprint after sprint... but I've also been on teams who churned through sprints with nothing to show for it.
I don't think we can boil down a good vs. bad environment, or good vs. bad code, to SDLC processes or enterprise vs. startup. There are good teams, there are bad teams. And matching up the right environment and process to the right team can work wonders. But to be fair, most teams haven't found the right match.
There is often no time pressure, and no commercial pressure.
This means that polishing around the edges, e.g. documentation can have more time applied.
There is also different context - for internal tools, the scope is often well understood. The team who are using it are long term colleagues. There is no need for a logo, flashy landing page etc to convince people to use the tool. There is no need to sell it to stakeholders - they asked for it in the first place.
We can also make really good assumptions about what features can be cut. In an internal tool we know our users, we know which teams need the tooling etc. We can do training. So we can very deliberately ignore certain use cases etc.
In OSS we don't know any of that, so often have to cater to everyone, hence building a much more robust tool.
It's also some survivorship bias - the incredible OSS projects rise to the top and do have all this stuff. There are 10,000 internal tools built for every 1 popular OSS tool. The 'bad' OSS projects aren't being seen or used, so we don't notice how bad quality those ones are.
This means that polishing around the edges, e.g. documentation can have more time applied.
There is also different context - for internal tools, the scope is often well understood. The team who are using it are long term colleagues. There is no need for a logo, flashy landing page etc to convince people to use the tool. There is no need to sell it to stakeholders - they asked for it in the first place.
We can also make really good assumptions about what features can be cut. In an internal tool we know our users, we know which teams need the tooling etc. We can do training. So we can very deliberately ignore certain use cases etc.
In OSS we don't know any of that, so often have to cater to everyone, hence building a much more robust tool.
It's also some survivorship bias - the incredible OSS projects rise to the top and do have all this stuff. There are 10,000 internal tools built for every 1 popular OSS tool. The 'bad' OSS projects aren't being seen or used, so we don't notice how bad quality those ones are.
All good points.
Thanks
You answered your own question:" The internal messes are always being built by people on a steady payroll."
The internal software is a direct money sink to a company, it is _VERY_ visible to the organization that they're pouring money into a product that's only tangential to making money. So internal software is allowed to be developed until the point where it becomes useful, but not more. You can argue how wise it is to use internal software, or to prioritize it in this way,but the incentive for not pouring more cash at it (wrong as it may be) is clear enough.
Disclaimer: Author of many awful internal tools.
The internal software is a direct money sink to a company, it is _VERY_ visible to the organization that they're pouring money into a product that's only tangential to making money. So internal software is allowed to be developed until the point where it becomes useful, but not more. You can argue how wise it is to use internal software, or to prioritize it in this way,but the incentive for not pouring more cash at it (wrong as it may be) is clear enough.
Disclaimer: Author of many awful internal tools.
Agile or not software projects are normally .. projects, with an expected budget. Much of that budget gets 'used' figuring out what the point of the project is, requirements, managing the project deliver, stakeholder management and reporting, working with other teams to access, integrate and understand systems ... then the building of the system is built with what left, which leaves no room for your 'gold plating' ...
Also the means of the many internal tools that I've been using are mostly short sighted and built with minimal expenses.
https://en.m.wikipedia.org/wiki/Invented_here
I have encountered Invented Here Syndrome when I joined my current employer about a popular open source application I wrote. They didn’t realize I wrote the application they were advocating.
https://news.ycombinator.com/item?id=30004405
Selection bias in hiring is also a factor. As an open source author I have noticed the biggest difference between populations is a preference for criticality away from accepted norms that does not so much exist on internal teams or may even be despised. It’s the difference between doing what’s popular because that appears to work versus doing something that becomes popular because it works better.
Another factor is hard work. Professionally hard work is often rewarded and encouraged when it should be discouraged. As software developers we should be focused on automating away the hard work, which is often the motivation behind most open source.
I have encountered Invented Here Syndrome when I joined my current employer about a popular open source application I wrote. They didn’t realize I wrote the application they were advocating.
https://news.ycombinator.com/item?id=30004405
Selection bias in hiring is also a factor. As an open source author I have noticed the biggest difference between populations is a preference for criticality away from accepted norms that does not so much exist on internal teams or may even be despised. It’s the difference between doing what’s popular because that appears to work versus doing something that becomes popular because it works better.
Another factor is hard work. Professionally hard work is often rewarded and encouraged when it should be discouraged. As software developers we should be focused on automating away the hard work, which is often the motivation behind most open source.
Love.
How much love does one invest in an internal product, only to have it rebuffed, denigrated or simply undervalued or taken for granted? After a few times the talented person finds some other avenue for expression.
Because if open source products are bad, they won't get used, but you don't have a choice but to use your company's internal stuff.
Competition works, at least to some extent. The median open source project is worthless, but the ones people choose to use are pretty good. There is usually no competition within the company.
The sunk cost fallacy also tends to kick in. If the company spent $2 million on a project, it must be worthwhile. Admitting the money was at least partially wasted is hard to swallow.
The sunk cost fallacy also tends to kick in. If the company spent $2 million on a project, it must be worthwhile. Admitting the money was at least partially wasted is hard to swallow.
Software polishing is usually tied to the number of users/customers. In my work, I use industrial software. It is expensive, and unpolished. Users are expected to train on it, learning it's unique quirks as a skill. The more software requires skill/knowing it, usually the less polished it is. Obviously, not programs such as Photoshop, but merely to run a machine, definitely.
It's intentionally built to be crap. If your requirements are constantly changing, you want to be able to turn on a dime. If you have "nicely architected" software, that's going to be a lot of work. If the "software" is just a folder of Python files and some shell scripts, adding in some hack to meet some business need is very quick work.
In mathematical terms, the ratio of A - the amount of hours spent writing the software to B - the amount of hours spent running the software is going to be very high. It doesn't make sense to spend two hours to fix a bug, if you can just spend ten seconds applying a workaround, unless you'll run it more than 720 times within its intended design lifespan.
In mathematical terms, the ratio of A - the amount of hours spent writing the software to B - the amount of hours spent running the software is going to be very high. It doesn't make sense to spend two hours to fix a bug, if you can just spend ten seconds applying a workaround, unless you'll run it more than 720 times within its intended design lifespan.
> the amount of hours spent writing the software to B - the amount of hours spent running the software is going to be very low
I agree on this point, the amount of time writing software is almost always going to be dwarfed by the maintenance period for software that makes it to production.
I've also seen businesses pay a lot of money software maintenance, but you seem to be implying that under-investment is a conscious choice. I would suggest that's not the main driver for software chaos.
The most common reason I've seen good software turn into an unmaintainable mess is knowledge loss / employee turnover over time. This usually results in a culture of fear around making changes to something that is poorly understood. When changes are made, they are done with incomplete understanding, and are consequently not in line with the original design choices.
Over time, even the maintainers move on. This continues until the whole thing is a complicated mess that nobody can understand, even the original author. That's when The Rewrite is suggested.
I agree on this point, the amount of time writing software is almost always going to be dwarfed by the maintenance period for software that makes it to production.
I've also seen businesses pay a lot of money software maintenance, but you seem to be implying that under-investment is a conscious choice. I would suggest that's not the main driver for software chaos.
The most common reason I've seen good software turn into an unmaintainable mess is knowledge loss / employee turnover over time. This usually results in a culture of fear around making changes to something that is poorly understood. When changes are made, they are done with incomplete understanding, and are consequently not in line with the original design choices.
Over time, even the maintainers move on. This continues until the whole thing is a complicated mess that nobody can understand, even the original author. That's when The Rewrite is suggested.
> the amount of time writing software is almost always going to be dwarfed by the maintenance period for software that makes it to production.
Oh sorry, I meant it the other way around!
So, I mean, I've seen this in scripts that I wrote to "quickly do some stuff".
1. Write the easiest possible implementation you can think of, just string together code in the REPL until it works, then put it all into a .py file
2. Some new need arises. Make the easiest possible change that causes it to work, like adding a new switch that completely changes the control flow.
3. Some new bug arises. Make the quickest possible change that causes it to work, like running the entire program in a while-true loop in case it crashes.
4. Continue along this manner until half of the code is commented-out garbage.
5. Rewrite.
There's no need to make it a thing of beauty, because it's easier to just scrap it and rewrite it. For the short term, making a "quick fix" is nearly always better, and for the long term, running a buggy piece of shit might still be preferable to spending good money to rewrite it.
(Also, the good stuff often already exists as open-source software, so it's only the hyper-specific stuff that's proprietary, and that will be closely tied to business needs.)
Oh sorry, I meant it the other way around!
So, I mean, I've seen this in scripts that I wrote to "quickly do some stuff".
1. Write the easiest possible implementation you can think of, just string together code in the REPL until it works, then put it all into a .py file
2. Some new need arises. Make the easiest possible change that causes it to work, like adding a new switch that completely changes the control flow.
3. Some new bug arises. Make the quickest possible change that causes it to work, like running the entire program in a while-true loop in case it crashes.
4. Continue along this manner until half of the code is commented-out garbage.
5. Rewrite.
There's no need to make it a thing of beauty, because it's easier to just scrap it and rewrite it. For the short term, making a "quick fix" is nearly always better, and for the long term, running a buggy piece of shit might still be preferable to spending good money to rewrite it.
(Also, the good stuff often already exists as open-source software, so it's only the hyper-specific stuff that's proprietary, and that will be closely tied to business needs.)
I get what you were saying now... you were talking tooling, not product software.
I agree with your take on the tooling side. No need to gold-plate something that works in the narrow scope it was created for.
I agree with your take on the tooling side. No need to gold-plate something that works in the narrow scope it was created for.
There are a lot of good points here, but I think the most important reason is being overlooked: not grokking the needs of the users. In my Fortune 150, all of the internal software is developed by teams of outsourced developers. Whether it's over the wall, or halfway around the world, the effect is the same: the developers don't truly understand how the users will use the software. So it winds up being crap, regardless of how carefully some middle manager -- who also will not use the software -- writes the specs.
I've made a career out of being a mechanical engineer who writes software for other engineers, while being embedded in the group, and not spirited away in some dusty development group in another building or another continent. As an engineer, I understand precisely how the other engineers will be trying to use my software, and it makes for a better overall product. Of course, I'm biased, but I think my software has had much more success than competing projects, because of this. If I had the time, I could name at least 2 large, direct examples where this has proven true, and several more smaller ones.
TL;DR: Programmers need domain-specific understanding of the problem they are trying to solve with the software.
I've made a career out of being a mechanical engineer who writes software for other engineers, while being embedded in the group, and not spirited away in some dusty development group in another building or another continent. As an engineer, I understand precisely how the other engineers will be trying to use my software, and it makes for a better overall product. Of course, I'm biased, but I think my software has had much more success than competing projects, because of this. If I had the time, I could name at least 2 large, direct examples where this has proven true, and several more smaller ones.
TL;DR: Programmers need domain-specific understanding of the problem they are trying to solve with the software.
The best software organization I ever worked for had internal tools that still blow my mind in both their features and reliability. Corners were cut in documentation certainly, but the main factors are the quality of the developer teams involved and the nature of their mission. Lots of developers nowadays are under pressure to ship features as quickly as possible for money, and if that is the whole job then of course the results will be spotty and modestly reliable.
Internal users are a captive audience. One of the key functions of middle management in a tech company is to enforce that internal platforms are used wherever applicable, no matter how bad they are or how much internal customers don’t want to, to promote consistency and synergy across the org. Open source is a more competitive landscape.
This at least has applied everywhere I've worked. If people choose to use it, it's often a good tool, if people are forced to use it, there's no incentive to deliver.
It's simply making a good general solution, vs the tyranical decision making that goes on internally at companies, where they tweak little details to custom workflows bc stakeholders said so.
Very simple — because economy.
Is there a competition for the internal software? No.
Is it going to be directly monetized? No.
So why bother going that extra mile to make it beautiful if users will use it anyway.
Is there a competition for the internal software? No.
Is it going to be directly monetized? No.
So why bother going that extra mile to make it beautiful if users will use it anyway.
Internal software? That's what the interns do, right?
Having worked with a good few Open Source tooling/libraries/frameworks etc and with a lot of similar internal ones I have come to the conclusion that almost invariably the quality of the later is far below the quality of the former. Being documentation, ergonomics, bugs, you name it.
Open source alternatives are much better than proprietary closed source ones. Mind you that I'm not talking about code quality, I'm talking about mere usability and dev experience.
This is consistent across quite a few medium, large and very large companies (enterprises) I've been. Worst case (and quite common) is when a company "builds" upon OS (e.g. a framework based on Flask). The end result is a painful to work with half-baked mess.
What I don't get though is that OS is built (in many cases) by hobbyists for free. The internal messes are always being built by people on a steady payroll. I just don't get it...