Drowning in code: The ever-growing problem of ever-growing codebases(theregister.com)
theregister.com
Drowning in code: The ever-growing problem of ever-growing codebases
https://www.theregister.com/2024/02/12/drowning_in_code/
73 comments
Basically, "divide and conquer." Big things are composed of small things.
I don't think this is the controversial part or the difficult conclusion, the challenge is how exactly to do that.
I don't think this is the controversial part or the difficult conclusion, the challenge is how exactly to do that.
Which is what I mentioned. I think the “controversial” part, is probably the care taken, in developing the small things. The dependencies on which I rely (operating systems, compilers, SDKs, etc.) are all very well-designed, solid, and supported. I’m not a huge fan of surfing the bleeding edge. I like to use mature tools (usually. I’ve been writing Swift, since the day it was announced).
Like I said, I spend a great deal of time, on each of my modules. Often, the testing code far outweighs the product code. Documentation is also a big deal.
I write code and documentation for an audience of one (Yours Truly), but I publish it for everyone. It’s not my concern, whether or not the projects get any GitHub stars. In fact, I prefer it, if they don’t.
Here’s an example. I’m currently writing an SDK to provide cleaned data for ML, so I can play around with ML, but it will be quite some time, before I get to that point. I’m still working on making the SDK rock-solid. I have spent the last couple of days, just working on the parser. I still need to write the query/importer. Yesterday, was all about unit tests for the parser. The next part, will require a test harness[0] (a SwiftUI app). Unit tests won’t be useful for that. It will also be my platform for the ML application. Also, code coverage for the parser tests is only about 85%. I plan to use the test harness to test the remaining 15%.
Many folks would simply rush through this part, but they would end up shipping (or throwing out) the sloppy code they wrote as a lash-up. Even if I decide that I don’t have a current application for ML, I’ll still have a top-Quality SDK, to use as a component for future work.
The app I just released, has a backend server, that I originally designed as a general-purpose application server. It took me seven months to write, as it was written in layers, and each layer was rigorously tested, before moving to the next. The purpose of that project, was as a teaching exercise. I didn’t actually have an application in mind, when I wrote it.
The server is a lot more complex than it needs to be, because it was designed as a general-purpose server, and the current implementation uses a fraction of its capacity. But the server was available, when I started work on the app, and I already knew it was extremely high-Quality, so using it wasn’t a big deal. It probably saved me a couple of months, and I really have a difficult time, trusting backend dependencies.
[0] https://littlegreenviper.com/miscellany/testing-harness-vs-u...
Like I said, I spend a great deal of time, on each of my modules. Often, the testing code far outweighs the product code. Documentation is also a big deal.
I write code and documentation for an audience of one (Yours Truly), but I publish it for everyone. It’s not my concern, whether or not the projects get any GitHub stars. In fact, I prefer it, if they don’t.
Here’s an example. I’m currently writing an SDK to provide cleaned data for ML, so I can play around with ML, but it will be quite some time, before I get to that point. I’m still working on making the SDK rock-solid. I have spent the last couple of days, just working on the parser. I still need to write the query/importer. Yesterday, was all about unit tests for the parser. The next part, will require a test harness[0] (a SwiftUI app). Unit tests won’t be useful for that. It will also be my platform for the ML application. Also, code coverage for the parser tests is only about 85%. I plan to use the test harness to test the remaining 15%.
Many folks would simply rush through this part, but they would end up shipping (or throwing out) the sloppy code they wrote as a lash-up. Even if I decide that I don’t have a current application for ML, I’ll still have a top-Quality SDK, to use as a component for future work.
The app I just released, has a backend server, that I originally designed as a general-purpose application server. It took me seven months to write, as it was written in layers, and each layer was rigorously tested, before moving to the next. The purpose of that project, was as a teaching exercise. I didn’t actually have an application in mind, when I wrote it.
The server is a lot more complex than it needs to be, because it was designed as a general-purpose server, and the current implementation uses a fraction of its capacity. But the server was available, when I started work on the app, and I already knew it was extremely high-Quality, so using it wasn’t a big deal. It probably saved me a couple of months, and I really have a difficult time, trusting backend dependencies.
[0] https://littlegreenviper.com/miscellany/testing-harness-vs-u...
This seems like you're saying 'I do the same thing as everyone else, but I'm really good and I do it better somehow'
Not really. I’ve been trained to take care on the small stuff. I’m not sure that “everyone else” spends that kind of time on the small stuff; or even cares about it, so maybe it is a unique approach. I spent most of my career at hardware manufacturers, and hardware screwups are very costly, compared to software. A product recall is a bit more involved than a software patch.
There’s a lot of arguments against that kind of care, as it can be difficult and expensive. There’s a reason that a Mercedes costs a lot more than a Toyota, even though they may seem to be the same car, superficially, but Toyota is probably a more profitable corporation.
It’s just the way I work. I’m not particularly interested in slagging anyone else’s work. I’m just talking about the way I do things.
There’s a lot of arguments against that kind of care, as it can be difficult and expensive. There’s a reason that a Mercedes costs a lot more than a Toyota, even though they may seem to be the same car, superficially, but Toyota is probably a more profitable corporation.
It’s just the way I work. I’m not particularly interested in slagging anyone else’s work. I’m just talking about the way I do things.
How is this not exactly what I just said? There aren't any details here, you're just saying "I've been trained to take care of the small stuff" and your last posts were just you saying you're careful and you spend time on modules and time on SDKs. There is nothing specific.
Why are you arguing? I'm not interested in arguing about it. My work is out there (I was actually quite specific in my examples). Feel free to peruse it. I don’t particularly care whether or not I’m “better than” anyone else. I’m not competing with anyone, and there’s a ton of folks out there, that can easily outshine me, in many ways.
I think that’s pretty cool, as it means there’s a lot to learn, and I love learning new stuff[0].
[0] https://littlegreenviper.com/miscellany/thats-not-what-ships...
I think that’s pretty cool, as it means there’s a lot to learn, and I love learning new stuff[0].
[0] https://littlegreenviper.com/miscellany/thats-not-what-ships...
I'm not arguing and I don't care about competition of some sort, it just seems like you posted on how you solve problems or complex programs but didn't give any details and just made long posts about how you 'spend time on modules' and 'care about the details'. I don't see the point without technical information or examples.
The original post was about codebase complexity.
There’s nothing unique about how to handle it. We’ve been doing it for thousands of years, and that’s pretty much exactly what I said. If you want to look at how I do stuff, I posted several articles that I wrote, explaining things in great detail, and here’s my GH page, with links to more detail than you could possibly want[0]. It’s not my responsibility to cut it into bite-size pieces for others.
Complexity is not the problem. In fact, it’s the solution. It’s just how we implement it, that is important, and that’s not new at all. I'm surprised that it is even an issue.
[0] https://github.com/ChrisMarshallNY
There’s nothing unique about how to handle it. We’ve been doing it for thousands of years, and that’s pretty much exactly what I said. If you want to look at how I do stuff, I posted several articles that I wrote, explaining things in great detail, and here’s my GH page, with links to more detail than you could possibly want[0]. It’s not my responsibility to cut it into bite-size pieces for others.
Complexity is not the problem. In fact, it’s the solution. It’s just how we implement it, that is important, and that’s not new at all. I'm surprised that it is even an issue.
[0] https://github.com/ChrisMarshallNY
I don't really understand what point you're trying to make with all these posts. Now you're saying Complexity is not the problem. In fact, it’s the solution. but you're not explaining anything, backing up anything you are saying, giving any details, it's just bold claims on top of bold claims. When someone says there isn't any substance to your claims and instead of giving a single line of detail you link an entire github repo and tell them to look through it, it just seems pointless.
You’re right.
It’s pointless.
Have a great day!
It’s pointless.
Have a great day!
> fact, Jared Diamond posited that we're actually a bit stupider
Please don't enable this man when there are literally millions of trained historians you can talk to.
Please don't enable this man when there are literally millions of trained historians you can talk to.
He isn't a historian, though.
It does seem plausible. We are descended from Cro-magnon man; Neanderthals had bigger brains than us.
https://www.archaeology.org/issues/103-1309/trenches/1138-ne...
And we have smaller brains than our Cro-magnon ancestors.
https://phys.org/news/2011-06-farming-blame-size-brains.html
It does seem plausible. We are descended from Cro-magnon man; Neanderthals had bigger brains than us.
https://www.archaeology.org/issues/103-1309/trenches/1138-ne...
And we have smaller brains than our Cro-magnon ancestors.
https://phys.org/news/2011-06-farming-blame-size-brains.html
> He isn't a historian, though.
Doesn't stop him from pushing his own grand narratives of history, apparently.
The claim isn't particularly interesting given the nebulous and culture-bound definition of intelligence, but he's not a person I would tap into for a sense of expertise backed by consensus.
Doesn't stop him from pushing his own grand narratives of history, apparently.
The claim isn't particularly interesting given the nebulous and culture-bound definition of intelligence, but he's not a person I would tap into for a sense of expertise backed by consensus.
It is my understanding this is how Apple does their development. Everything is home grown and not a hodge podge of dependencies from outside the company.
I do like Mac products but just can't get passed their walled garden. However, to have the quality they do I guess it kind of makes sense to have the garden.
I personally like the idea of few-to-none outside dependencies, solid testing and solid, up-to-date documentation. I need to work on being more disciplined and motivated to do it.
I do like Mac products but just can't get passed their walled garden. However, to have the quality they do I guess it kind of makes sense to have the garden.
I personally like the idea of few-to-none outside dependencies, solid testing and solid, up-to-date documentation. I need to work on being more disciplined and motivated to do it.
I think that Facebook is similar, but more of their stuff comes from acquisitions, so the conglomerate is quite a bit more heterogenous, than a company like Apple.
I think that most corporations don't like to cede control. I have been told that one of the big reasons for Boeing's troubles, is that they have pushed a lot of important functionality to external contractors.
I think that most corporations don't like to cede control. I have been told that one of the big reasons for Boeing's troubles, is that they have pushed a lot of important functionality to external contractors.
> Large codebases are inevitable.
I vehemently disagree. New codebases might be built on top of large stacks of framework code and infrastructure code, but source repositories ought to actually be tighter and more business-focused than they used to be.
But they're often bloated monstrosities, filled with resume-padding architecture astronaut bullshit, or contorted to fit the latest hip framework's requirements.
Apologies ahead for this long-winded rant, but this exact issue has been boiling my blood for weeks now because I've been wading through the neck-deep mess that a customer has made, such as writing an astonishing three megabytes of source (typed by hand, not generated!) for a single web form. One!
Especially in the era of modern web frameworks, we can avoid most of the "ceremony" of older systems like C or C++ where just making a call to a database was pages of fiddly boilerplate. Instead of embracing this, I see developers reintroducing unnecessary code because they must have gotten so used to the verbosity that they felt exposed without it.
Let me give a concrete example from a codebase I've just spent a week reviewing. I've read through hundreds of pages of code, found dozens of critical resource leaks and security issues, and I'm yet to encounter any actual business logic.
The first mistake that happened was that someone decided to use Angular. Why? It's hip! It's cool! It's what all the kids are doing!
Except now, you have to have an API instead of just web pages. That API will need to serialize things to JSON, so you need to data transfer objects and Newtonsoft code. Suddenly, you need to deal with JWT, and make sure the untrusted client doesn't receive internal (secret) fields from the database. The client is no longer strongly typed or closely tied to the C# code, so you need a full megabyte of test code that you never would have needed before to validate all the Typescript.
This code does nothing of business value. It's a form. You fill it out, and hit submit. There's no "interactivity", there's no single-page-app features used, it's just... waste.
Then, then, on top of that, someone that really likes "grand cathedral" style code architecture decided that this whole thing would be more secure if there was a "middle tier". You know, an API. No, not the Angular API facing the Internet! A back-end API. For no reason. For an app written by two developers that has its own, dedicated database, and will never interact with any other app, tool, or system.
So now you have to write server-to-server API call code including the DTOs and related definitions, the server code, the client code, the service authentication code, the error handling, the distributed tracing, the HTTP header forwarding (securely!), and on, and on.
All typed out by hand, because: "REST is easy!"
No wait, there's more: The original architect also wanted to make the system "database agnostic", so the mid-tier repeats every abstraction. It has interfaces, types, etc... so that you can "swap in" another database. This never happened, and will never happen. This is purely wasted effort.
Then someone put API Management in front of this. Why? Because the word "API" was used. Remember: Angular uses JSON APIs, not HTML, so it's a "thing" that "needs" management. Then that API management -- a cloud SaaS service -- has a reverse proxy in front of it operating in API management mode for "security".
Meanwhile all these layers are riddled with URL injection, SQL injection, and similar basic security errors precisely because it's too hard to add a new field to the form, so developers get lazy and simply add a generic "ExecuteQuery(string sql)" mid-tier method and move on with their lives.
This is why we have "too much code". Because architects can't justify their positions if there's only two icons with a single arrow between them in their designs. Because "hip new thing" is a shiny we all love to chase. Because of magical thinking that assumes that more abstractions means the code is more good.
We have nobody to blame for this but ourselves.
I vehemently disagree. New codebases might be built on top of large stacks of framework code and infrastructure code, but source repositories ought to actually be tighter and more business-focused than they used to be.
But they're often bloated monstrosities, filled with resume-padding architecture astronaut bullshit, or contorted to fit the latest hip framework's requirements.
Apologies ahead for this long-winded rant, but this exact issue has been boiling my blood for weeks now because I've been wading through the neck-deep mess that a customer has made, such as writing an astonishing three megabytes of source (typed by hand, not generated!) for a single web form. One!
Especially in the era of modern web frameworks, we can avoid most of the "ceremony" of older systems like C or C++ where just making a call to a database was pages of fiddly boilerplate. Instead of embracing this, I see developers reintroducing unnecessary code because they must have gotten so used to the verbosity that they felt exposed without it.
Let me give a concrete example from a codebase I've just spent a week reviewing. I've read through hundreds of pages of code, found dozens of critical resource leaks and security issues, and I'm yet to encounter any actual business logic.
The first mistake that happened was that someone decided to use Angular. Why? It's hip! It's cool! It's what all the kids are doing!
Except now, you have to have an API instead of just web pages. That API will need to serialize things to JSON, so you need to data transfer objects and Newtonsoft code. Suddenly, you need to deal with JWT, and make sure the untrusted client doesn't receive internal (secret) fields from the database. The client is no longer strongly typed or closely tied to the C# code, so you need a full megabyte of test code that you never would have needed before to validate all the Typescript.
This code does nothing of business value. It's a form. You fill it out, and hit submit. There's no "interactivity", there's no single-page-app features used, it's just... waste.
Then, then, on top of that, someone that really likes "grand cathedral" style code architecture decided that this whole thing would be more secure if there was a "middle tier". You know, an API. No, not the Angular API facing the Internet! A back-end API. For no reason. For an app written by two developers that has its own, dedicated database, and will never interact with any other app, tool, or system.
So now you have to write server-to-server API call code including the DTOs and related definitions, the server code, the client code, the service authentication code, the error handling, the distributed tracing, the HTTP header forwarding (securely!), and on, and on.
All typed out by hand, because: "REST is easy!"
No wait, there's more: The original architect also wanted to make the system "database agnostic", so the mid-tier repeats every abstraction. It has interfaces, types, etc... so that you can "swap in" another database. This never happened, and will never happen. This is purely wasted effort.
Then someone put API Management in front of this. Why? Because the word "API" was used. Remember: Angular uses JSON APIs, not HTML, so it's a "thing" that "needs" management. Then that API management -- a cloud SaaS service -- has a reverse proxy in front of it operating in API management mode for "security".
Meanwhile all these layers are riddled with URL injection, SQL injection, and similar basic security errors precisely because it's too hard to add a new field to the form, so developers get lazy and simply add a generic "ExecuteQuery(string sql)" mid-tier method and move on with their lives.
This is why we have "too much code". Because architects can't justify their positions if there's only two icons with a single arrow between them in their designs. Because "hip new thing" is a shiny we all love to chase. Because of magical thinking that assumes that more abstractions means the code is more good.
We have nobody to blame for this but ourselves.
I empathize with this so much.
Burned out a few years back writing an admin panel mobile app for an embedded device. The actual panel and the interaction with the device was fine, it was the insanity of the customer architect who only had time to design the system but not build it. You'd think that would have informed them on their excessive creativity, but alas.
One use case was that an operator working with a device could at times need external support. One feature was to be able to dial support from the phone whilst using the app such that the operator could receive instructions, or have support remotely change settings for them.
At least, that's what I thought. That would make sense. But no.
They wanted a full real time socket connection between the hardware, the mobile app, a server, another server and the support app. Important detail is that operators often worked in areas with bad reception.
This meant ensuring that all these connections were up at all times, with error handling, ensuring that messages were delivered and the gist, whilst simultaneously serializing and deserializing these messages and running whatever algorithms over them, often calling other services written in node that would take ages to respond if they didn't just timeout completely (30s!).
Oh, and they wanted to sell this forward so everything, everything, needed to be customizable.
Every meeting with said architect and he had new ideas, new protocols, new solutions.
Eventually I needed to fix a bug in their code, which turned out to be a 15Gig monorepo where 1/3-2/3 of all code was commented out.
You can imagine what the final product was. Which was rolled out to everyone on day one without any testing in live environments.
I'm tired just talking about it.
Burned out a few years back writing an admin panel mobile app for an embedded device. The actual panel and the interaction with the device was fine, it was the insanity of the customer architect who only had time to design the system but not build it. You'd think that would have informed them on their excessive creativity, but alas.
One use case was that an operator working with a device could at times need external support. One feature was to be able to dial support from the phone whilst using the app such that the operator could receive instructions, or have support remotely change settings for them.
At least, that's what I thought. That would make sense. But no.
They wanted a full real time socket connection between the hardware, the mobile app, a server, another server and the support app. Important detail is that operators often worked in areas with bad reception.
This meant ensuring that all these connections were up at all times, with error handling, ensuring that messages were delivered and the gist, whilst simultaneously serializing and deserializing these messages and running whatever algorithms over them, often calling other services written in node that would take ages to respond if they didn't just timeout completely (30s!).
Oh, and they wanted to sell this forward so everything, everything, needed to be customizable.
Every meeting with said architect and he had new ideas, new protocols, new solutions.
Eventually I needed to fix a bug in their code, which turned out to be a 15Gig monorepo where 1/3-2/3 of all code was commented out.
You can imagine what the final product was. Which was rolled out to everyone on day one without any testing in live environments.
I'm tired just talking about it.
That’s a fair assessment, but I guess by “large,” I really mean “composite,” Every large human artifact tends to be quite complex, but is composed of smaller, less complex parts.
I don’t see any way to avoid it. In fact, every discipline, other than software development, seems to be managing the complexity (albeit, not always well).
I don’t see any way to avoid it. In fact, every discipline, other than software development, seems to be managing the complexity (albeit, not always well).
I fully agree that composing modules written by independent teams is necessary and good for productivity. We don’t want to reinvent wheels, and layering on top of standard abstractions is generally better even when the fit isn’t 100% perfect.
My problem is that if each of those layers is bloated to 5-10x the size and complexity they should be, then we have a problem. This can compound, not just additively but multiplicatively. Eventually you end up in a quagmire where nothing works right and you can’t fix anything because that would just reveal more breakage at layers you don’t control.
One thing I noticed over this last week is how much nicer it is to work with something like ASP.NET Core 8.0 instead of the old ASP.NET MVC 5. It’s leaner and meaner, but has enough “batteries included” that I don’t have to drag hundreds of megabytes of slightly incompatible third-party modules in just to get essentials such as JSON and Dependency Injection.
As an industry we need to acknowledge the need for cohesive and consistent platforms, not just components.
We need to occasionally clean house and consolidate, not just keep throwing packages over the fence and calling it a day.
My problem is that if each of those layers is bloated to 5-10x the size and complexity they should be, then we have a problem. This can compound, not just additively but multiplicatively. Eventually you end up in a quagmire where nothing works right and you can’t fix anything because that would just reveal more breakage at layers you don’t control.
One thing I noticed over this last week is how much nicer it is to work with something like ASP.NET Core 8.0 instead of the old ASP.NET MVC 5. It’s leaner and meaner, but has enough “batteries included” that I don’t have to drag hundreds of megabytes of slightly incompatible third-party modules in just to get essentials such as JSON and Dependency Injection.
As an industry we need to acknowledge the need for cohesive and consistent platforms, not just components.
We need to occasionally clean house and consolidate, not just keep throwing packages over the fence and calling it a day.
> As an industry we need to acknowledge the need for cohesive and consistent platforms, not just components.
Agreed, but it's also, apparently, a difficult bar to clear.
Apple manages to do it, by having iron-fisted control of the hardware, the operating system, and the SDKs.
But even they are starting to fray. I've been writing Apple code since 1986, and I'm getting familiar with some of the newer stuff, like CreateML and SwiftUI.
There's a ... difference ... in the way that documentation is being handled. In The Days of Yore, we had Inside Macintosh, which was a multi-volume Encyclopedia Britannica-like set of tomes, delving deep into the operating system and SDKs. It was written by really skilled people; often with advanced and esoteric degrees, and with great writing chops.
These days, not so much.
I'm glad for bloggers and online enthusiasts.
Agreed, but it's also, apparently, a difficult bar to clear.
Apple manages to do it, by having iron-fisted control of the hardware, the operating system, and the SDKs.
But even they are starting to fray. I've been writing Apple code since 1986, and I'm getting familiar with some of the newer stuff, like CreateML and SwiftUI.
There's a ... difference ... in the way that documentation is being handled. In The Days of Yore, we had Inside Macintosh, which was a multi-volume Encyclopedia Britannica-like set of tomes, delving deep into the operating system and SDKs. It was written by really skilled people; often with advanced and esoteric degrees, and with great writing chops.
These days, not so much.
I'm glad for bloggers and online enthusiasts.
Microsoft land is the same. The first few versions of C# had amazing documentation and the libraries were very consistent and cohesive. GUI programming especially was a breeze.
Now Microsoft can’t even make a GUI framework that is actually used by anyone, including themselves!
Now Microsoft can’t even make a GUI framework that is actually used by anyone, including themselves!
This is part 1 of my Reg adaptation of my FOSDEM 2024 talk:
https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-...
I plan to return to the rest. Thanks for sharing it.
https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-...
I plan to return to the rest. Thanks for sharing it.
Thank you for the talk, it was very interesting!
But, a very pertinent question to everyone: is there some magic ffmpeg incantation or filter to remove mic noise from a video, or otherwise improve the audio of spoken word? Some of the FOSDEM talks, this one included, are very hard to listen to for someone with sensitive hearing issues.
But, a very pertinent question to everyone: is there some magic ffmpeg incantation or filter to remove mic noise from a video, or otherwise improve the audio of spoken word? Some of the FOSDEM talks, this one included, are very hard to listen to for someone with sensitive hearing issues.
I've uploaded the script I wrote for it to the FOSDEM site, and it's on my personal blog as well.
https://liam-on-linux.dreamwidth.org/90185.html
I also also shared both the slideshow, and the slideshow annotated with speaker's notes -- basically the script, cut into bits and added to each slide.
Either should help. That's about all I can offer; I'm not a video person and I know nothing of sound processing or anything.
https://liam-on-linux.dreamwidth.org/90185.html
I also also shared both the slideshow, and the slideshow annotated with speaker's notes -- basically the script, cut into bits and added to each slide.
Either should help. That's about all I can offer; I'm not a video person and I know nothing of sound processing or anything.
I think the main reason codebases are ever-growing is that we never ask, in a serious way, the question "could this be much smaller". I can't recall a company where there was ever a concerted effort to consistently refactor a codebase for conciseness, nor where the inevitable breaking changes from that activity would be tolerated.
It is difficult to find an economic argument for such activity but it bears a significant resemblance to a "tragedy of the commons" scenario.
By contrast, if I look at my large private codebases, which I religiously refactor and modernize every few years if I think I could improve them even on elegance grounds, have shrunk in size and complexity by a significant integer factor over the many years. And this observable shrinkage happened while adding significant features and materially improving scalability, performance, parallelism, and robustness. The difference is that I can break my APIs and architectures anytime I want and the only person who is mildly inconvenienced is me, and I don't have to justify the time spent doing so.
If I look at the mechanics of how the shrinkage happens, I invest a lot in building highly refined code tools that replace recurring development patterns seen across codebases, and then deploy them everywhere that pattern occurs. It is a kind of inductive library building, done to high standard. Again, not a pattern I see much in other large codebases. If the code works, we rarely change it, and it is difficult to justify this kind of tool building when we could be shipping features instead.
I understand why the incentives in engineering organizations of any size will never encourage working this way, except outside perhaps the occasional small team. There is a strong bias toward maximizing convenience and minimizing transaction costs in the moment, which are felt keenly. Whereas, the significant drag on productivity and maintainability is highly diffuse and only felt over a long time.
It is difficult to find an economic argument for such activity but it bears a significant resemblance to a "tragedy of the commons" scenario.
By contrast, if I look at my large private codebases, which I religiously refactor and modernize every few years if I think I could improve them even on elegance grounds, have shrunk in size and complexity by a significant integer factor over the many years. And this observable shrinkage happened while adding significant features and materially improving scalability, performance, parallelism, and robustness. The difference is that I can break my APIs and architectures anytime I want and the only person who is mildly inconvenienced is me, and I don't have to justify the time spent doing so.
If I look at the mechanics of how the shrinkage happens, I invest a lot in building highly refined code tools that replace recurring development patterns seen across codebases, and then deploy them everywhere that pattern occurs. It is a kind of inductive library building, done to high standard. Again, not a pattern I see much in other large codebases. If the code works, we rarely change it, and it is difficult to justify this kind of tool building when we could be shipping features instead.
I understand why the incentives in engineering organizations of any size will never encourage working this way, except outside perhaps the occasional small team. There is a strong bias toward maximizing convenience and minimizing transaction costs in the moment, which are felt keenly. Whereas, the significant drag on productivity and maintainability is highly diffuse and only felt over a long time.
That is exactly what we do in our codebase for Nyxt, and I assure you, it is not cheap. However, the benefits are very tangible.
It depends, are you making a premium engineering product, or something quick?
It depends, are you making a premium engineering product, or something quick?
> I think the main reason codebases are ever-growing is that we never ask, in a serious way, the question "could this be much smaller"
I think we do ask this question, but code bases grow just like cities grow. We have more stuff and do more things and have more people.
I think there is a curation and pruning process but still gets bigger over time.
Even with serious refactoring, the codebase still grows as the deleted portions are still available in the codebase for reference.
I think we do ask this question, but code bases grow just like cities grow. We have more stuff and do more things and have more people.
I think there is a curation and pruning process but still gets bigger over time.
Even with serious refactoring, the codebase still grows as the deleted portions are still available in the codebase for reference.
For me it seems like I spend at least 10x as much time making something lean vs bloated and ugly. Unfortunately I am not measured by lean code but by output of other things, so in the end the code base reflects these incentives.
> It is difficult to find an economic argument for such activity but it bears a significant resemblance to a "tragedy of the commons" scenario.
To me the canonical example of this is base-10 numbering system. If we were base 12 (and it's not unheard of), we'd have the best properties of both the metric system (being based in the radix of the number system) and imperial (half units that are easily divisible into common integer fractions, halves, thirds, quarters, etc).
Blank state, no doubt, base-12 is better.
But now imagine the costs of switching our society to base-12 when the US cannot even get with Celsius which is clearly the most intuitive of the metric measures.
Some things are truly moot and require seismic events to ever be possible to change so they are effectively set in stoneas far as mortals are concerned.
PS- for those interested: https://en.wikipedia.org/wiki/Duodecimal PPS- base-16 is not covenient for thirds, and really only makes sense for a society anticipating binary logic
To me the canonical example of this is base-10 numbering system. If we were base 12 (and it's not unheard of), we'd have the best properties of both the metric system (being based in the radix of the number system) and imperial (half units that are easily divisible into common integer fractions, halves, thirds, quarters, etc).
Blank state, no doubt, base-12 is better.
But now imagine the costs of switching our society to base-12 when the US cannot even get with Celsius which is clearly the most intuitive of the metric measures.
Some things are truly moot and require seismic events to ever be possible to change so they are effectively set in stoneas far as mortals are concerned.
PS- for those interested: https://en.wikipedia.org/wiki/Duodecimal PPS- base-16 is not covenient for thirds, and really only makes sense for a society anticipating binary logic
Thanks for the article.
Regarding the article which talks about multicore. Multicore and distributed systems are hard. Behaviour implementation, and transforming event sequences across multicore and multimachine consistently is the hard part of computing for me (distributed systems). And understanding large existing systems such as Chromium, Firefox or GCC. Any enterprise K8s cluster with operators everywhere. That's a lot of code.
This is just my thoughts, I find it easier to write code than read large code bases because like trying to understand a very large mathematical formulae in one go. You cannot even get the entire system into a screenful to understand it, like you could potentially a small formulae. I have to break it out into ordered steps to understand mathematics. They're like an object in the mind that you interact with intuitively. My mental model is the key driver to the act of programming. I sometimes get the mental model right and serialise it into code. Writing is an expression of thoughts and there is a thought that gets turned into notation. Do either of them (thought,notation) represent the other losslessly?
I once used OpenGrok, a web based sourcecode repository viewer to tell myself I would try reading and understanding a large code base.
The semantics of programming languages have an impedance mismatch to how my mental model works.... I have mental models about relational databases and what is fast on a computer. And then there is behaviour, state and IO to work with.
I could change my mental model to match the programming language semantics on the computer but I would probably be less happy that way.
I feel programming languages such as C++ and Rust and Haskell introduce abstractions that work with a certain mental model of solving problems, but not everyone has the same mental model. They work great for the compiler codegen writer, but not for implementing behaviours.
If you've ever worked in a enterprise Java codebase or a frontend software React codebase, from my perspective they're a mess because of the imposed semantics of the language and false problems of trying to get the computer to have the right sequence of actions or behaviour sequence.
Regarding the article which talks about multicore. Multicore and distributed systems are hard. Behaviour implementation, and transforming event sequences across multicore and multimachine consistently is the hard part of computing for me (distributed systems). And understanding large existing systems such as Chromium, Firefox or GCC. Any enterprise K8s cluster with operators everywhere. That's a lot of code.
This is just my thoughts, I find it easier to write code than read large code bases because like trying to understand a very large mathematical formulae in one go. You cannot even get the entire system into a screenful to understand it, like you could potentially a small formulae. I have to break it out into ordered steps to understand mathematics. They're like an object in the mind that you interact with intuitively. My mental model is the key driver to the act of programming. I sometimes get the mental model right and serialise it into code. Writing is an expression of thoughts and there is a thought that gets turned into notation. Do either of them (thought,notation) represent the other losslessly?
I once used OpenGrok, a web based sourcecode repository viewer to tell myself I would try reading and understanding a large code base.
The semantics of programming languages have an impedance mismatch to how my mental model works.... I have mental models about relational databases and what is fast on a computer. And then there is behaviour, state and IO to work with.
I could change my mental model to match the programming language semantics on the computer but I would probably be less happy that way.
I feel programming languages such as C++ and Rust and Haskell introduce abstractions that work with a certain mental model of solving problems, but not everyone has the same mental model. They work great for the compiler codegen writer, but not for implementing behaviours.
If you've ever worked in a enterprise Java codebase or a frontend software React codebase, from my perspective they're a mess because of the imposed semantics of the language and false problems of trying to get the computer to have the right sequence of actions or behaviour sequence.
Trying to figure out someone elses code is usually like being told to go into a strangers house and find where the hammer is. And your boss expects you to know where exactly it is as if there’s any standard to how different people organize their lives or thoughts.
I thought the reason was pretty dang simple: Every sprint I get assigned tickets to add new features and code and functionality and edge case handling, and very rarely do I get a sprint where I get a ticket to attempt to remove a minimal dependency.
I can't take the time to do that on my own, because then the much vaunted "metrics" will go down.
I can't take the time to do that on my own, because then the much vaunted "metrics" will go down.
Isn’t part of the bloat that the instruction sets of these platforms are also bloated?
The Intel manual is quite large. Are we close to compiling towards the optimal code for Intel chips? If we could achieve that then we’d at least be in lock step with the hardware…
But syntactically and semantically we’re kind of stuck in the stone ages. We’ve had languages that can compress a ton of information into a single combinator for some time but the dominant mode of thinking in industry is procedural.
I wonder if Eliot is right and the way forward at a certain scale is through correct software: designed and compiled from dependently typed languages with the aid of theorem provers.
The Intel manual is quite large. Are we close to compiling towards the optimal code for Intel chips? If we could achieve that then we’d at least be in lock step with the hardware…
But syntactically and semantically we’re kind of stuck in the stone ages. We’ve had languages that can compress a ton of information into a single combinator for some time but the dominant mode of thinking in industry is procedural.
I wonder if Eliot is right and the way forward at a certain scale is through correct software: designed and compiled from dependently typed languages with the aid of theorem provers.
I don't believe that's the problem, and in any case, 99% of software developers never have to worry about the underlying instruction set. That said, ARM is already a reduced instruction set, so if someone happens to work in that area, focusing on ARM is an option.
The article intersects very well with AI tools poised to analyze the code and try to understand it for us. Software tools already exist to deal with large code bases, for example, right now, we use software tools (valgrind, coverity) to help us understand the code. Lead architects use analysis tools as well as developers eyeballs to vet the code. Soon enough an AI tool will guide developers on understanding their own codebase and manage it. The same goes along with infrastructure, ChatGPT can guide you right now with examples on how to create a AWS service from scratch.
So yes, the codebase is growing fast and with it the tools to manage it.
So yes, the codebase is growing fast and with it the tools to manage it.
We may have the tools to manage it, but we are losing the ability to understand it.
AI writing software will be a exponential explosion in software complexity.
AI would very well create its own programming language to be more efficient for the AI to code in that we have no hopes of understanding. Imagine that AI started to output a large SaaS app written today in Python in Assembly because for the AI the extra cognitive overhead of using Assembly doesn't exist. At first we might resist and tell it to use a language we understand, but then as time goes on we grow more comfortable that it does the "right" thing and down the road people are just generated raw Assembly using AI without really understanding what the code is doing, only looking to see if the code behaves the way they expect.
Imagine entire codebases spun up in seconds with so many lines of code, a single person would never have hopes of understanding everything, needing to rely on AI to summarize and explain the code for them. Now imagine that massive code base being iterated and worked on for a decade over the life of a company.
AI could bring Terabyte sized code bases in a decade or so.
AI writing software will be a exponential explosion in software complexity.
AI would very well create its own programming language to be more efficient for the AI to code in that we have no hopes of understanding. Imagine that AI started to output a large SaaS app written today in Python in Assembly because for the AI the extra cognitive overhead of using Assembly doesn't exist. At first we might resist and tell it to use a language we understand, but then as time goes on we grow more comfortable that it does the "right" thing and down the road people are just generated raw Assembly using AI without really understanding what the code is doing, only looking to see if the code behaves the way they expect.
Imagine entire codebases spun up in seconds with so many lines of code, a single person would never have hopes of understanding everything, needing to rely on AI to summarize and explain the code for them. Now imagine that massive code base being iterated and worked on for a decade over the life of a company.
AI could bring Terabyte sized code bases in a decade or so.
This already happens when you compile code for a specific architecture. The thing you fear will happen has already happened and we’re fine.
I’d worry more about generated code from an AI that doesn’t fully understand the codebase. That would be as bad as letting the junior devs run loose.
I’d worry more about generated code from an AI that doesn’t fully understand the codebase. That would be as bad as letting the junior devs run loose.
Not really at all.
What you are describing is an traceable transformation of code in which there are several intermediate layers that people can inspect and understand. They can inspect the exact rules for that transformation. The process is repeatable and verifiable.
What I am describing is a black-box stochastic generation of low level code in which there is no higher level representation anymore. AI generating Assembly not by a set of rules, but using statistics. There will be no individual layers to unwind or inspect, because for AI it doesn't need them. Our separation of concerns was built for our human brains and limiting complexity of projects to our understanding.
What you are describing is an traceable transformation of code in which there are several intermediate layers that people can inspect and understand. They can inspect the exact rules for that transformation. The process is repeatable and verifiable.
What I am describing is a black-box stochastic generation of low level code in which there is no higher level representation anymore. AI generating Assembly not by a set of rules, but using statistics. There will be no individual layers to unwind or inspect, because for AI it doesn't need them. Our separation of concerns was built for our human brains and limiting complexity of projects to our understanding.
If you have an AI capable of writing machine code based on natural language you likely also have an AI that can translate that machine code to any other language you would like. You could then use a normal compiler to verify if it is correct and then read the code yourself. Or you could just get good at reading and writing assembly.
Yes, that is exactly my point. You will have to rely on AI to translate it back out, but that translation is built on probabilities not machine rule-based translation. So you can ask and have the AI explain everything to you, but you are still trusting the "black box" to tell you what is happening. Very different from today.
Also, you can "get good" at reading assembly, but that doesn't matter if the AI can output a custom OS from scratch and a custom VM to execute the program it wrote to solve your use case. It will be so impossibly complex that it would be the equivalent studying protein folding.
Instead people will just trust the AI.
It also won't help you if the code base the AI produces for a SaaS app is a million lines of assembly.
Instead of having different layers of OS, compiler, high level language, an AI will just be able to produce one layer. because after decades of trusting the AI to write our code, why wouldn't it?
The current gen of AI outputting code that in human-centric programming languages will be a blip in the history of AI. As it advances, it can just skip that step.
Its will be orders of magnitude more complex and opaque than anything we have today.
Also, you can "get good" at reading assembly, but that doesn't matter if the AI can output a custom OS from scratch and a custom VM to execute the program it wrote to solve your use case. It will be so impossibly complex that it would be the equivalent studying protein folding.
Instead people will just trust the AI.
It also won't help you if the code base the AI produces for a SaaS app is a million lines of assembly.
Instead of having different layers of OS, compiler, high level language, an AI will just be able to produce one layer. because after decades of trusting the AI to write our code, why wouldn't it?
The current gen of AI outputting code that in human-centric programming languages will be a blip in the history of AI. As it advances, it can just skip that step.
Its will be orders of magnitude more complex and opaque than anything we have today.
AI tools will make the problem immensely worse, not better.
No, this is not merely anti-AI sentiment or a complaint about how the AIs of today aren't really as good as they are being presented. For the purposes of this argument I'm assuming AIs will work exactly as advertised some point in the relatively near future and you really can just AI-prompt your way through a non-trivial problem.
The reason why we have way too much code isn't that code is too hard, it is that it is too easy. Modern cars are complicated, but their complication is restricted by the limitations of being in three-dimensional space, being something that has to be physically manufactured where each individual part is itself something that a person may have spent a person-month or three on, the failure rates of physical parts and how those rates multiply if you keep piling on the complexity, and so on and so forth. The result is that a car has tens of thousands of parts, but it can't have trillions of parts. It doesn't matter how many engineers you throw at the problem, nothing like a modern car could be that complicated. It would simply be guaranteed that too many parts would fail. (To be that complicated it would have to be a different kind of thing, like some sort of biological organism, which are themselves rather complicated. But we humans don't make those.)
We have monstrous code bases because they are easy to add to. We can add a new module and test that module in a degree of isolation no physical engineer could dream of. We have analysis tools with no physical equivalent. We don't even design the raw parts, which would be the compiled assembly, we have piles and piles of tools to take things much simpler to understand ("source code") and turn them into the final parts rather than working with the final parts directly. It is easy to be cynical about things like browser quality and I see a lot of fashionable poses struck about this on HN, but the reality is there is nothing like Chrome in the physical world. It is the software world whose artifacts are approaching biological levels of complexity. This isn't because we software engineers are special as people. Engineering tasks by their nature tend to rise to the maximum level of complexity the engineers can deal with, and we all have basically the same intellectual prowess. But we software engineers have unparalleled tools to deal with our complexity, so we get orders of magnitude farther down the path of creating it, and then having to deal with it.
Thus, when you understand that our code bases are complex because creating more code is easy, not because it is hard, you can then easily see that any paradigm shift that makes it even easier to create code means that we will have even more of it, not less. If I can type "Dear GPT-12, I don't like my web browser. Please write a standards-compliant web browser except the tabs stack in 3D in my VR environment instead.", and a few hours later the world actually has another 150 millions lines of code and a new browser (I'm assuming more standards the browser will have to implement in the meantime) to deal with, that's not going to contribute to simplifying the code world.
It will not be a situation where you "can" use AI tools. It will become a situation where you will have no ability to do anything else, and the code will naturally grow to the limits of what AI can deal with, until it then exceeds even those tools as much as our current code bases are exceeding ours.
Of course, then the question is, what will we get for that complexity? I don't know. But you'd better hope it doesn't break because your only option to fix it will basically be "Oh AI genie, please fix this code." And I do think I can guarantee a lot of quirkiness in the result, where things just randomly don't work and it becomes even beyond the AI's abilities to understand why.
No, this is not merely anti-AI sentiment or a complaint about how the AIs of today aren't really as good as they are being presented. For the purposes of this argument I'm assuming AIs will work exactly as advertised some point in the relatively near future and you really can just AI-prompt your way through a non-trivial problem.
The reason why we have way too much code isn't that code is too hard, it is that it is too easy. Modern cars are complicated, but their complication is restricted by the limitations of being in three-dimensional space, being something that has to be physically manufactured where each individual part is itself something that a person may have spent a person-month or three on, the failure rates of physical parts and how those rates multiply if you keep piling on the complexity, and so on and so forth. The result is that a car has tens of thousands of parts, but it can't have trillions of parts. It doesn't matter how many engineers you throw at the problem, nothing like a modern car could be that complicated. It would simply be guaranteed that too many parts would fail. (To be that complicated it would have to be a different kind of thing, like some sort of biological organism, which are themselves rather complicated. But we humans don't make those.)
We have monstrous code bases because they are easy to add to. We can add a new module and test that module in a degree of isolation no physical engineer could dream of. We have analysis tools with no physical equivalent. We don't even design the raw parts, which would be the compiled assembly, we have piles and piles of tools to take things much simpler to understand ("source code") and turn them into the final parts rather than working with the final parts directly. It is easy to be cynical about things like browser quality and I see a lot of fashionable poses struck about this on HN, but the reality is there is nothing like Chrome in the physical world. It is the software world whose artifacts are approaching biological levels of complexity. This isn't because we software engineers are special as people. Engineering tasks by their nature tend to rise to the maximum level of complexity the engineers can deal with, and we all have basically the same intellectual prowess. But we software engineers have unparalleled tools to deal with our complexity, so we get orders of magnitude farther down the path of creating it, and then having to deal with it.
Thus, when you understand that our code bases are complex because creating more code is easy, not because it is hard, you can then easily see that any paradigm shift that makes it even easier to create code means that we will have even more of it, not less. If I can type "Dear GPT-12, I don't like my web browser. Please write a standards-compliant web browser except the tabs stack in 3D in my VR environment instead.", and a few hours later the world actually has another 150 millions lines of code and a new browser (I'm assuming more standards the browser will have to implement in the meantime) to deal with, that's not going to contribute to simplifying the code world.
It will not be a situation where you "can" use AI tools. It will become a situation where you will have no ability to do anything else, and the code will naturally grow to the limits of what AI can deal with, until it then exceeds even those tools as much as our current code bases are exceeding ours.
Of course, then the question is, what will we get for that complexity? I don't know. But you'd better hope it doesn't break because your only option to fix it will basically be "Oh AI genie, please fix this code." And I do think I can guarantee a lot of quirkiness in the result, where things just randomly don't work and it becomes even beyond the AI's abilities to understand why.
While I do think we’ll go through a phase where AI writes a significant amount of code I think it will eventually replace the need for most software. To illustrate what I mean, take your browser example. Why would you even want a browser when you can just make a request to the AI and it can give you exactly the information or images you are looking for formatted in your preferred style and edited and annotated to perfectly match your ability to comprehend it? Honestly, I think it will go beyond even that. Our current mode of consuming information may be entirely replaced by simply conversing with personalized AI. We won’t search for information, our assistant will spontaneously tell us what we need to know and we will interrogate it about things that spark our interest. There will be no UI, only an active agent that you speak with and write to who answers your questions, acts on your behalf and delivers the information you want as a document on a tablet, a video on a screen, an immersive VR experience or performed as ballad if the mood strikes you.
There’s no chance they give you full reigns of output considering present business paradigms at least. There’d be no differentiating your product from competitors or justifying upgrades if it was always how you wanted it to work.
> It is the software world whose artifacts are approaching biological levels of complexity.
This is an interesting analogy to make because that complexity, much like the complexity in large software systems, is emergent from many agents interacting in small ways without top down coordination. So perhaps the right low level behaviors from software engineers would allow for monstrous complexity whilst still being effective.
This is an interesting analogy to make because that complexity, much like the complexity in large software systems, is emergent from many agents interacting in small ways without top down coordination. So perhaps the right low level behaviors from software engineers would allow for monstrous complexity whilst still being effective.
Unlike biological complexity where the creator remains elusive, I would expect AI to travel along with its creations and explain everything in excruciating detail at ELI4 level.
The AWS documentation can also guide you right now how to make an AWS service from scratch. At the end of the day you might as well read the scripture instead of playing telephone with it.
The problem is managing complexity. The way we typically do this is to summarize things so that we can get a good idea of the whole, while focusing on the points we need to at the time.
I believe that dynamic documentation is the way forward when it comes to this. Auto Wiki by Mutable AI looks very promising.
https://wiki.mutable.ai/
https://news.ycombinator.com/item?id=38915999
I believe that dynamic documentation is the way forward when it comes to this. Auto Wiki by Mutable AI looks very promising.
https://wiki.mutable.ai/
https://news.ycombinator.com/item?id=38915999
> If you’re okay waiting a bit, and we’ll generate it for free.
Does this mean it will generate one now automatically, and I can check back after some time, or will I need to contact them for this?
Does this mean it will generate one now automatically, and I can check back after some time, or will I need to contact them for this?
I'm not affiliated with them, you'd have to ask them.
It's not just software, it's all technology. Here's a question: how big would a factory (building) need to be to build everything needed for a "personal computer" AND build everything inside the factory? I'm trying to answer this at https://civboot.org
We need to try to simplicity and document the technology our civilization depends on or it will get away from us.
We need to try to simplicity and document the technology our civilization depends on or it will get away from us.
Saying that's a monumental task would be the understatement of the week. You're compiling Hari Seldon's Encyclopedia Galactica!
Would be interested to check out your site, but https://civboot.org seems to be down.
Would be interested to check out your site, but https://civboot.org seems to be down.
http://civboot.org (http) works, and is a redirect to https://github.com/civboot/civboot .
I'm trying. It's good to have goals in life. Also, I think it's more possible than you think it is (although clearly not achievable by one person).
Oberon sounds interesting. How refreshing it could be to have a simple computer system. Even my little IoT project has grown over the years to a rather ungainly proportion, requiring 20min in CI to build many targets. Matter is really the last straw… so bloated!
[Article author here]
It is!
A potted summary:
https://ignorethecode.net/blog/2009/04/22/oberon/
A wonderful academic look, short and readable: "Oberon -- the overlooked jewel"
https://people.cis.ksu.edu/~danielwang/Investigation/System_...
(PDF, over HTTP, safe)
It is!
A potted summary:
https://ignorethecode.net/blog/2009/04/22/oberon/
A wonderful academic look, short and readable: "Oberon -- the overlooked jewel"
https://people.cis.ksu.edu/~danielwang/Investigation/System_...
(PDF, over HTTP, safe)
That's exactly why we need to modularize and componentize. We need to reduce dependency, loose the coupling and eliminate side effects. Monolithic is a dead end. Microservices when things get too complex.
Microservices to reduce complexity and coupling is a fallacy. https://riak.com/posts/technical/microservices-please-dont/ is a great and succinct counter to all the advantages that microservices claim to have. In summary:
> You don’t need to introduce a network boundary as an excuse to write better code
> Distributed Transactions are never easy
> A bunch of engineers working in isolated codebases leads to ‘Not my problem’ syndrome
> You don’t need to introduce a network boundary as an excuse to write better code
> Distributed Transactions are never easy
> A bunch of engineers working in isolated codebases leads to ‘Not my problem’ syndrome
https://grugbrain.dev/ has the following to say about microservices:
"grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug"
"grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug"
> Microservices when things get too complex.
Microservices are a huge step up in complexity. I'm not convinced monoliths can't be implemented well.
Microservices are a huge step up in complexity. I'm not convinced monoliths can't be implemented well.
Doesn't modularization and compartmentalization increase dependency? I'm curious on how you propose we do both at the same time.
On the other hand, now there are a couple hundred cryptically named processes running at all times. How many services does Adobe need to run on my computer when I am not actually running their programs? Quite a few as it were.
I don't understand the singling out of Debian. Debian includes an unbelievable number of software packages, a vanishingly small minority being the core OS.
[Article author here]
> I don't understand the singling out of Debian.
To counter: I don't understand why it would matter which distro.
It is simply because Debian published these statistics. I couldn't find any other distro vendor that did.
I used those numbers because I could get those particular numbers.
At this scale, it doesn't matter which distro. What do you estimate the scale of the size difference is between general-purpose distros? An order of magnitude? Surely not as much as 2 OoM, and probably less than 1 OoM. Maybe ITRO of 25-50% for most non-enterprise ones.
> I don't understand the singling out of Debian.
To counter: I don't understand why it would matter which distro.
It is simply because Debian published these statistics. I couldn't find any other distro vendor that did.
I used those numbers because I could get those particular numbers.
At this scale, it doesn't matter which distro. What do you estimate the scale of the size difference is between general-purpose distros? An order of magnitude? Surely not as much as 2 OoM, and probably less than 1 OoM. Maybe ITRO of 25-50% for most non-enterprise ones.
I think my comment was probably unclear. My problem is the comparison of the reported lines of code between the Debian Project and Oberon OS. The reported lines of code for debian include everything in the repos. The OS, compilers, interpreters, DBMSs, Document managers (Libreoffice, Texlive are probably massive), games, etc. To make it like for like, you would need to compare Debian and Oberon + all the software written for Oberon.
I was simply picking 2 extremes.
The real interest here is the midpoint, which is Plan 9. The 9front image is half a gig, and that too includes everything: apps, demos, games, the entire thing. As I said:
« “The kernel is 5,119,091 bytes - 5MB or so.
“The entire distribution including all sources, documents, local Git repository and binaries is circa 530MB for amd64.”
The Plan 9 kernel has 38 system calls. I’ve not been able to find a firm number for Linux, but I have some estimates.
As of 2016, kernel 4.7 had about 335. By 2017 that was up to 341. I’ve tried to estimate the number for kernel 6.8 and I think there are 520 across all architectures. »
ISTM that you're obsessing over the exact position of one end, whereas in fact, given that we are talking of a difference of roughly 5 orders of magnitude -- five zeroes -- it would make no difference at all if we included every program ever written for every variant of Oberon that ever existed, and the most minimal possible Debian system that can run two shells at once.
Debian 12 is on the order of 150,000 lines of code for every single line of A2.
The real interest here is the midpoint, which is Plan 9. The 9front image is half a gig, and that too includes everything: apps, demos, games, the entire thing. As I said:
« “The kernel is 5,119,091 bytes - 5MB or so.
“The entire distribution including all sources, documents, local Git repository and binaries is circa 530MB for amd64.”
The Plan 9 kernel has 38 system calls. I’ve not been able to find a firm number for Linux, but I have some estimates.
As of 2016, kernel 4.7 had about 335. By 2017 that was up to 341. I’ve tried to estimate the number for kernel 6.8 and I think there are 520 across all architectures. »
ISTM that you're obsessing over the exact position of one end, whereas in fact, given that we are talking of a difference of roughly 5 orders of magnitude -- five zeroes -- it would make no difference at all if we included every program ever written for every variant of Oberon that ever existed, and the most minimal possible Debian system that can run two shells at once.
Debian 12 is on the order of 150,000 lines of code for every single line of A2.
Code should be booked in books like a assets service costs. The maintenance of your garden is not free even though it's nice 2 have.
Also, @dang: this is a dupe of:
https://news.ycombinator.com/item?id=39346402
https://news.ycombinator.com/item?id=39346402
> There is an urgent need for smaller, simpler software.
No. There is not. The evidence proves the opposite.
Users have problems and they pay good money to have them solved. They care not one bit how you achieve that.
They neither care nor see that you installed 10,000 npm modules to make the text in your app also work for right-to-left arabic users. They don't care whether your app is 50MB or 150MB. You ship Electron and they see that their beloved app works on their iPad exactly as on the web browser your work PC has installed.
In any case, most of the assumptions in the article are wrong anyway.
> they can struggle to recognize our faces
Hundreds of millions of iPhones are unlocked dozens of times per day, probably with face recognition.
No. There is not. The evidence proves the opposite.
Users have problems and they pay good money to have them solved. They care not one bit how you achieve that.
They neither care nor see that you installed 10,000 npm modules to make the text in your app also work for right-to-left arabic users. They don't care whether your app is 50MB or 150MB. You ship Electron and they see that their beloved app works on their iPad exactly as on the web browser your work PC has installed.
In any case, most of the assumptions in the article are wrong anyway.
> they can struggle to recognize our faces
Hundreds of millions of iPhones are unlocked dozens of times per day, probably with face recognition.
> They neither care nor see that you installed 10,000 npm modules to make the text in your app also work for right-to-left arabic users.
They do care about being hacked, short load times and some guarantee of support in the future.
> They don't care whether your app is 50MB or 150MB.
Sysadmins, people with poor connections and people with cheap hardware do.
> You ship Electron and they see that their beloved app works on their iPad exactly as on the web browser your work PC has installed.
As in slow as balls. Sorry, that last one was a layup.
They do care about being hacked, short load times and some guarantee of support in the future.
> They don't care whether your app is 50MB or 150MB.
Sysadmins, people with poor connections and people with cheap hardware do.
> You ship Electron and they see that their beloved app works on their iPad exactly as on the web browser your work PC has installed.
As in slow as balls. Sorry, that last one was a layup.
> They do care about being hacked, short load times and some guarantee of support in the future.
Yeah, I do care about my integrity and support in the future (as a customer) but that’s not my responsibility, it’s yours as developer.
Yeah, I do care about my integrity and support in the future (as a customer) but that’s not my responsibility, it’s yours as developer.
Yeah, we all care about good, fast software. But I don't think we (as in users) "need" small and fast software.
Reality is often disappointing, but I don't see people choosing one music streaming service over the other because their desktop client is written in a more concise programming language.
Reality is often disappointing, but I don't see people choosing one music streaming service over the other because their desktop client is written in a more concise programming language.
[Article author here]
> Hundreds of millions of iPhones are unlocked dozens of times per day, probably with face recognition.
You miss the point.
https://www.asiaone.com/digital/woman-discovers-colleague-ca...
https://www.quora.com/My-Apple-face-ID-opened-with-someone-e...
Etc. etc.
In fact, you seem to miss most of my points.
> Hundreds of millions of iPhones are unlocked dozens of times per day, probably with face recognition.
You miss the point.
https://www.asiaone.com/digital/woman-discovers-colleague-ca...
https://www.quora.com/My-Apple-face-ID-opened-with-someone-e...
Etc. etc.
In fact, you seem to miss most of my points.
We stand on the shoulders of giants (or at least geeks with good posture).
One of the things that folks don't like to think about, is that people, these days, aren't really any smarter than they were, five thousand years ago (in fact, Jared Diamond posited that we're actually a bit stupider).
We just have a baseline social infrastructure that allows our "starting point" to be that much more advanced. We are the equivalent of privileged offspring, inheriting family wealth. If we are trained to manage and utilize that wealth, it grows. If we aren't, it will likely get squandered.
I started off, writing my own operating systems in assembly. I even have the manual, for my very first engineering project, in 1987 (including the OS code)[0]. It worked, but it was very primitive. I can't imagine writing an OS in assembler, these days. I'm glad someone else did, in C (or whatever).
So my "starting point," these days, is having a high-quality, parallel operating system, with a developer SDK (again, someone else that knows what they are doing, did the hard work for me), a compiler and linker (thanks, Chris Lattner, et al), etc.
For my own projects, I use a lot of dependencies. The difference is, is that I write most of my own dependencies. These are standalone modules, with each one having its own project lifecycle, documentation, testing, etc. Even if no one else will use them, I still publish them on GitHub, as supported open-source projects, because that means that I don't need to worry about them, when I use them in my own aggregate projects.
Basically, "divide and conquer." Big things are composed of small things. It's important for me to do a good job on the small things, so the big things turn out better.
But that's just me, and the way I work. YMMV.
[0] https://littlegreenviper.com/wp-content/uploads/2022/07/TF30... (Downloads a PDF)