Build It Yourself(lucumr.pocoo.org)
lucumr.pocoo.org
Build It Yourself
https://lucumr.pocoo.org/2025/1/24/build-it-yourself/
262 comments
How is Debian / Ubuntu secure?
It's signed by a maintainer. And maintainers are vetted. You trust Debian/Ubuntu to only allow trustworthy people to sign packages.
How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
Yes.
We're basically back to sending around EXE and DLL files in a ZIP. It's just that now we call it a container and proudly start it as root.
BTW, I agree with the author of the article: Sometimes you're best off just merging dependency source code. It used to be called "vendoring" and was a big thing in Rails / Ruby. The big advantage is that you're not affected by future malicious package takeovers. But you can still merge security patches from upstream, if you choose to do that.
It's signed by a maintainer. And maintainers are vetted. You trust Debian/Ubuntu to only allow trustworthy people to sign packages.
How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
Yes.
We're basically back to sending around EXE and DLL files in a ZIP. It's just that now we call it a container and proudly start it as root.
BTW, I agree with the author of the article: Sometimes you're best off just merging dependency source code. It used to be called "vendoring" and was a big thing in Rails / Ruby. The big advantage is that you're not affected by future malicious package takeovers. But you can still merge security patches from upstream, if you choose to do that.
> How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
I know who created the Docker images, because I'm the person who built them!
A lot of the time you can build your images either from scratch, or based on the official base images like Alpine or Ubuntu/Debian or some of the RPM base images, not that much different than downloading an ISO for a VM. With a base, you can use apk/apt/dnf to get whatever packages you want if you trust that more, just remember to clean up the package cache so it's not persisted in the layers (arguably wastes space). For most software, it actually isn't as difficult as it might have initially seemed.
As an alternative, you can also look for vaguely trustworthy parties that have a variety of prepackaged images available and you can either borrow their Dockerfiles or just trust their images, for example, https://bitnami.com/stacks/containers and https://hub.docker.com/u/bitnami
Most likely you have to have trust somewhere in the mix, for example, I'm probably installing the Debian/Ubuntu packaged JDK instead of compiling mine in most cases, just because that's more convenient.
Also, rootless containers are pretty cool! People do like Podman and other solutions a lot, you can even do some remapping with Docker if there are issues with how the containers expect to be run https://docs.docker.com/engine/security/userns-remap/ or if you have to use Docker and need something a bit more serious you can try this https://docs.docker.com/engine/security/rootless/
I know who created the Docker images, because I'm the person who built them!
A lot of the time you can build your images either from scratch, or based on the official base images like Alpine or Ubuntu/Debian or some of the RPM base images, not that much different than downloading an ISO for a VM. With a base, you can use apk/apt/dnf to get whatever packages you want if you trust that more, just remember to clean up the package cache so it's not persisted in the layers (arguably wastes space). For most software, it actually isn't as difficult as it might have initially seemed.
As an alternative, you can also look for vaguely trustworthy parties that have a variety of prepackaged images available and you can either borrow their Dockerfiles or just trust their images, for example, https://bitnami.com/stacks/containers and https://hub.docker.com/u/bitnami
Most likely you have to have trust somewhere in the mix, for example, I'm probably installing the Debian/Ubuntu packaged JDK instead of compiling mine in most cases, just because that's more convenient.
Also, rootless containers are pretty cool! People do like Podman and other solutions a lot, you can even do some remapping with Docker if there are issues with how the containers expect to be run https://docs.docker.com/engine/security/userns-remap/ or if you have to use Docker and need something a bit more serious you can try this https://docs.docker.com/engine/security/rootless/
I don’t follow this reasoning: you might trust this distribution packager to be honest, but this doesn’t stop them from honestly packaging malicious code. It’s unlikely that the typical distribution packager is reviewing more than a minority of the code in the packages they’re accepting, especially between updates.
There are significant advantages to the distribution model, including exact provenance at the point of delivery. But I think it’s an error to treat it as uniquely trustworthy: it’s the same code either way.
There are significant advantages to the distribution model, including exact provenance at the point of delivery. But I think it’s an error to treat it as uniquely trustworthy: it’s the same code either way.
"Trust" as two meanings in English:
- You can trust someone as in "think they're honest and won't betray me".
- You can trust someone as in "think they are competent and won't screw-up".
In this case, we trust distributions packagers in both ways. Not only do we trust that they are non-malicious, we also trust that they won't clumsily package code without even skimming through it.
- You can trust someone as in "think they're honest and won't betray me".
- You can trust someone as in "think they are competent and won't screw-up".
In this case, we trust distributions packagers in both ways. Not only do we trust that they are non-malicious, we also trust that they won't clumsily package code without even skimming through it.
The point was not about competency. I think typical distribution packagers are very competent.
"They won't screw up" is the wrong way to look at security: humans always screw up. We're essentially fallible. We would all do well to remember that the xz backdoor was not caught by distribution code review: it was caught due to a performance side effect that the next adversary will be more careful to avoid.
Software stacks are getting bigger, largely due to engineering pressures that are outside of the control of distributions. It's a basic fact that that means work for the same (or fewer) people, which in turn means less time spent on more complicated packages. Under those conditions, competency is not the deciding factor.
This is, separately, why having lots of little packages can be (but isn't necessarily) a good thing: small packages that decompose into well-understood units of work can be reviewed more quickly and comprehensively, and can be assigned capabilities that can be tracked and audited with static analysis. You can't do that as easily (or at all) when every package decides to hand-roll its own subcomponents.
"They won't screw up" is the wrong way to look at security: humans always screw up. We're essentially fallible. We would all do well to remember that the xz backdoor was not caught by distribution code review: it was caught due to a performance side effect that the next adversary will be more careful to avoid.
Software stacks are getting bigger, largely due to engineering pressures that are outside of the control of distributions. It's a basic fact that that means work for the same (or fewer) people, which in turn means less time spent on more complicated packages. Under those conditions, competency is not the deciding factor.
This is, separately, why having lots of little packages can be (but isn't necessarily) a good thing: small packages that decompose into well-understood units of work can be reviewed more quickly and comprehensively, and can be assigned capabilities that can be tracked and audited with static analysis. You can't do that as easily (or at all) when every package decides to hand-roll its own subcomponents.
> "They won't screw up" is the wrong way to look at security: humans always screw up. We're essentially fallible.
Sure. So pick a system that includes at least one extra possible barrier to them screwing up than the system that has none.
Sure. So pick a system that includes at least one extra possible barrier to them screwing up than the system that has none.
It’s not “either or.” It’s been decades since a “package the world” approach has been practical at human scales.
(I think it’s great that distributions provide one extra set of eyes. But we’re deluding ourselves if we think that can scale to the actual realized demand for both new software and continuous change to software. It’s also not very fair to the distributions.)
(I think it’s great that distributions provide one extra set of eyes. But we’re deluding ourselves if we think that can scale to the actual realized demand for both new software and continuous change to software. It’s also not very fair to the distributions.)
The average distro package maintainer is not reviewing the underlying code changes in packages to determine if they are malicious, nor is the average disto package maintainer qualified to spot intentionally obfuscated malicious code.
The the average distro package maintainers is “person who cares that package repos stay up to date and is willing to commit their time to that work”
The the average distro package maintainers is “person who cares that package repos stay up to date and is willing to commit their time to that work”
You are making a sweeping assumption that people who create container images are incompetent.
It's not only the packager. Some distros have an actual security team. That does not mean they audit everything, but it's vastly better than getting random code from PyPi.
Some companies providing container images also have security teams. Some companies providing pypi packages also have security teams. Your point is?
The point is that if you pull 100 dependencies from PyPi, you have absolutely no idea who you are trusting unless you put the effort to go check those 100 dependencies.
If you link to dependencies that you get from Canonical, then you know that a security team is caring about the packages that Canonical distributes and actually patches them. Who patches security issues in PyPi? The individual package maintainers, which may well be pushing a malware leveraging typo-squatting. Typo-squatting is not really a thing in the Ubuntu package manager, is it?
Not saying it's perfect. Just that it's better than the far west. PyPi is pretty much like downloading a .exe from a webpage without even opening the webpage (because back in the days, opening the webpage could sometimes - not always - tell you that it was clearly shady).
If you link to dependencies that you get from Canonical, then you know that a security team is caring about the packages that Canonical distributes and actually patches them. Who patches security issues in PyPi? The individual package maintainers, which may well be pushing a malware leveraging typo-squatting. Typo-squatting is not really a thing in the Ubuntu package manager, is it?
Not saying it's perfect. Just that it's better than the far west. PyPi is pretty much like downloading a .exe from a webpage without even opening the webpage (because back in the days, opening the webpage could sometimes - not always - tell you that it was clearly shady).
All I hear is “here’s a business opportunity, go figure out if you can sell a python registry with libraries vetted by a security team”. Maybe there’s money in it.
Canonical doesn’t do this as a good will. They make money on it, right? Who vets packages for fedora? alpine? arch?
Canonical doesn’t do this as a good will. They make money on it, right? Who vets packages for fedora? alpine? arch?
The money is in the service that flags stuff in the registry, which you can sell N times to N companies that now have both embedded and vendor security teams. The companies now need to self host the registry and the scanner to see what’s out of compliance, otherwise they can’t be a vendor for anyone else. And their embedded security team basically siphons cash away from corporate, functioning as an engine of harassment for devs trying to get work done, while working as middlemen for the upstream security vendor.
Since none of this is particularly effective or efficient, money is made from someone’s perspective, but mostly it functions like a “cost of doing business” tax that you can’t avoid
Since none of this is particularly effective or efficient, money is made from someone’s perspective, but mostly it functions like a “cost of doing business” tax that you can’t avoid
Isn't that what Anaconda does? As opposed to say conda-forge? Anaconda tries hard to get orgs to pay money when they could just use conda-forge which has more packages, more up to date etc.
> How is Debian / Ubuntu secure?
You’re also forgetting process isolation and roles which provide strong invariants to control what a given package can do.
No such guarantees exist for code you load into your process.
You’re also forgetting process isolation and roles which provide strong invariants to control what a given package can do.
No such guarantees exist for code you load into your process.
> It's signed by a maintainer. And maintainers are vetted. You trust Debian/Ubuntu to only allow trustworthy people to sign packages.
> How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
Me neither. But the same goes for those from Debian/Ubuntu. In fact, neither I know anyone who vets those who sign and publish packages. What I know is that I can build my own images from container files and then I’m back to installing those apparently trusted packages from Debian/Ubuntu.
> We're basically back to sending around EXE and DLL files in a ZIP. It's just that now we call it a container and proudly start it as root.
I don’t get your point. And what’s an rpm or deb? You also potentially run stuff as root… sudo apt install -y… post install scripts…
> How are Docker / Python / Rust secure? I don't know any of the people who created my docker images, PyPi packages, or Rust crates.
Me neither. But the same goes for those from Debian/Ubuntu. In fact, neither I know anyone who vets those who sign and publish packages. What I know is that I can build my own images from container files and then I’m back to installing those apparently trusted packages from Debian/Ubuntu.
> We're basically back to sending around EXE and DLL files in a ZIP. It's just that now we call it a container and proudly start it as root.
I don’t get your point. And what’s an rpm or deb? You also potentially run stuff as root… sudo apt install -y… post install scripts…
You use in house solution??? any of these guys that want build something minimal dependency free is valid concern, I agree with that
but but what if people just don't want that??? it given people jobs and things to do (lol, this is serious)
also there are certain things better to use third party library than develop in house like crypto
but but what if people just don't want that??? it given people jobs and things to do (lol, this is serious)
also there are certain things better to use third party library than develop in house like crypto
The vetting process for open source maintainers has very little overlap with the vetting process for “is this person trustworthy”.
This is true for individual libraries and also for Linux distros.
This is true for individual libraries and also for Linux distros.
I genuinely don't know what people with this opinion work on that they can so easily choose to completely re-invent the wheel in every project, just because they're afraid of hypothetical dangers. Most of which are still present for private re-implementations (like bugs).
I cannot think of a single project where NIH syndrome would've been a net positive. Even dependencies that aren't "essential" are super helpful in saving time.
When you recreate parts of an "optional" dependency for every single project, how do you find the time to fix all the extra bugs, the edge cases, the support for different versions of lower level dependencies, of different platforms?
I cannot think of a single project where NIH syndrome would've been a net positive. Even dependencies that aren't "essential" are super helpful in saving time.
When you recreate parts of an "optional" dependency for every single project, how do you find the time to fix all the extra bugs, the edge cases, the support for different versions of lower level dependencies, of different platforms?
> that they can so easily choose to completely re-invent the wheel in every project
Nobody, and I mean 0, chooses to completely re-invent the wheel in every project. I understand why you would find this weird. I don't start every project with nand gates.
But the tendency is to use a library to check if a number is even or odd. It is a gradient, and you have to choose what you write and when you rely on a dependency. If you need to parse an XML file, or run HTTP requests, most likely you should find a library for it. But personally, if I can write it in 1-2 days, I always do it instead of adding a dependency. And I'm pretty sure it's worth it.
> how do you find the time to fix all the extra bugs, the edge cases, the support for different versions of lower level dependencies, of different platforms
If you take e.g. C++, maintaining your dependencies takes time. If you support multiple platforms, probably you need to (cross-)compile the dependencies. You should probably update them too (or at least check for security vulnerabilities), and updating dependencies brings its lot of issues (also in Rust: most libraries are 0.x.x and can break the API whenever they please).
Then, if you miss a feature in the library, you're pretty much screwed: learning the codebase to contribute your feature may be a lot of work, and it may not be merged. Same for a bug (debugging a codebase that is not yours is harder). If you end up forking the dependency and having to live with a codebase you did not write, most likely it's better to write it yourself.
The people advocating the "not re-inventing the wheel because NIH is really bad" philosophy seem to assume that libraries are always good. Now if you have a team of good developers, it may well be that the code they write is better than the average library out there. So if they write their features themselves, you end up with less but better code to maintain, and the people who understand that code work for you. Doesn't that sound good?
Nobody, and I mean 0, chooses to completely re-invent the wheel in every project. I understand why you would find this weird. I don't start every project with nand gates.
But the tendency is to use a library to check if a number is even or odd. It is a gradient, and you have to choose what you write and when you rely on a dependency. If you need to parse an XML file, or run HTTP requests, most likely you should find a library for it. But personally, if I can write it in 1-2 days, I always do it instead of adding a dependency. And I'm pretty sure it's worth it.
> how do you find the time to fix all the extra bugs, the edge cases, the support for different versions of lower level dependencies, of different platforms
If you take e.g. C++, maintaining your dependencies takes time. If you support multiple platforms, probably you need to (cross-)compile the dependencies. You should probably update them too (or at least check for security vulnerabilities), and updating dependencies brings its lot of issues (also in Rust: most libraries are 0.x.x and can break the API whenever they please).
Then, if you miss a feature in the library, you're pretty much screwed: learning the codebase to contribute your feature may be a lot of work, and it may not be merged. Same for a bug (debugging a codebase that is not yours is harder). If you end up forking the dependency and having to live with a codebase you did not write, most likely it's better to write it yourself.
The people advocating the "not re-inventing the wheel because NIH is really bad" philosophy seem to assume that libraries are always good. Now if you have a team of good developers, it may well be that the code they write is better than the average library out there. So if they write their features themselves, you end up with less but better code to maintain, and the people who understand that code work for you. Doesn't that sound good?
> Even dependencies that aren't "essential" are super helpful in saving time.
They might save time up front, but over the lifetime of a long-lived project, my experience is dependencies end up costing more time. Dependencies are a quintessential but often overlooked form of tech debt.
If you only work on short-lived projects or build-it-and-move-on type contract work where speed matters more than quality, sure, go nuts with dependencies. But if you care about the long term maintainability of a project, actively reducing dependencies as close to zero as possible is one of the best things you can do.
They might save time up front, but over the lifetime of a long-lived project, my experience is dependencies end up costing more time. Dependencies are a quintessential but often overlooked form of tech debt.
If you only work on short-lived projects or build-it-and-move-on type contract work where speed matters more than quality, sure, go nuts with dependencies. But if you care about the long term maintainability of a project, actively reducing dependencies as close to zero as possible is one of the best things you can do.
I think it probably depends heavily on the dependency (heh).
Would you reimplement a regex engine yourself? I hope not. Left-pad? Obviously yes. I don't think you can have blanket advice that dependencies should be avoided.
I suspect even quite simple dependencies are still a time saver in the long run. Probably people prefer reimplementing them because writing code is much more fun than updating dependencies, even if it actually takes longer and isn't as good.
Would you reimplement a regex engine yourself? I hope not. Left-pad? Obviously yes. I don't think you can have blanket advice that dependencies should be avoided.
I suspect even quite simple dependencies are still a time saver in the long run. Probably people prefer reimplementing them because writing code is much more fun than updating dependencies, even if it actually takes longer and isn't as good.
> Would you reimplement a regex engine yourself? I hope not.
I've implemented parallel-parsing-processes-style [1] parser combinators in at least three languages, though.
[1] https://www.cambridge.org/core/services/aop-cambridge-core/c...
That's what I say when I converse about my colleagues: having a package manager from the start that greatly lowers the friction of adding or publishing yet another package deprives a language's ecosystem from something very useful: a good measure of natural selection.
Add to that a free-for-all/no curation repository situation like pypi, npm or cargo together with a too small standard library and prepare to suffer.
Add to that a free-for-all/no curation repository situation like pypi, npm or cargo together with a too small standard library and prepare to suffer.
I wonder how much of this is just the move away from shared libraries.
In the .NET space nuget certainly makes it easy to add dependencies, but dependencies do seem to be overall fewer and the primary interesting difference I'd note is that a dependency is in fact it's own DLL file - to the extent that it's a feature that you can upgrade and replace them by dropping in a new file or changing configuration.
It strikes me that we'd perhaps see far less churn like this if more languages were back to having shared libraries and ABI compatibility as a first class priority. Because then the value of stable ABIs and more limited selections of upgrades would be much higher.
In the .NET space nuget certainly makes it easy to add dependencies, but dependencies do seem to be overall fewer and the primary interesting difference I'd note is that a dependency is in fact it's own DLL file - to the extent that it's a feature that you can upgrade and replace them by dropping in a new file or changing configuration.
It strikes me that we'd perhaps see far less churn like this if more languages were back to having shared libraries and ABI compatibility as a first class priority. Because then the value of stable ABIs and more limited selections of upgrades would be much higher.
The quest for performance makes macros, monomorphisation/specialization and LTO too attractive for simple dynamic linking to remain the norm, unfortunately. And in a way, I understand, a Stalin/MLton style whole-program optimizing compiler certainly is worth it when you have today's computing power.
Unless you have a very good JIT compiler, like JVMs. Than you can defer LTO and whole program optimization to runtime.
So, the feature of "Here's a set of easily pulled libraries" is an anti-feature because it makes it easy to pull supporting libraries? I suspect this is actually about developers and not Rust or JS. Developers choose what dependencies to pull. If nobody pulled a dependency it would wither and die. There are a lot of dependencies for most libraries because most developers prefer to use dependencies to build something.
But I digress. If we're talking build system, nobody is forcing you to use Crates.io with cargo, they just make it easy to. You can use path-based dependencies just like CMake/VCPkg,Conan, or you can DIY the library.
Even with crates.io, nobody is forcing you to not use version pinning if you want to avoid churn, but they just make it easy to get latest.
It's easy to build software on existing software in Rust. If you don't like the existing software or the rate it changes don't blame Cargo. Just do it the way you like it.
But I digress. If we're talking build system, nobody is forcing you to use Crates.io with cargo, they just make it easy to. You can use path-based dependencies just like CMake/VCPkg,Conan, or you can DIY the library.
Even with crates.io, nobody is forcing you to not use version pinning if you want to avoid churn, but they just make it easy to get latest.
It's easy to build software on existing software in Rust. If you don't like the existing software or the rate it changes don't blame Cargo. Just do it the way you like it.
> because it makes it easy to pull supporting libraries?
No, because it's used as an excuse for a lack of large(r) standard library. It's the equivalent of "the bazaar/free market will solve it!".
You basically end up with a R5RS level of fragmentation and cruft outside your pristine small language; something that the Scheme community decided is not fun and prompted the idea of R6RS/R7RS-large. Yeah, it's hard to make a good, large and useful stdlib, but punting it to the outside world isn't a proper long-term solution either.
It's really a combination of factors.
No, because it's used as an excuse for a lack of large(r) standard library. It's the equivalent of "the bazaar/free market will solve it!".
You basically end up with a R5RS level of fragmentation and cruft outside your pristine small language; something that the Scheme community decided is not fun and prompted the idea of R6RS/R7RS-large. Yeah, it's hard to make a good, large and useful stdlib, but punting it to the outside world isn't a proper long-term solution either.
It's really a combination of factors.
Standard library omissions aren’t there just because.
For almost any functionality missing in the standard library, you could point to 2-3 popular crates that solve the problem in mutually exclusive ways, for different use cases.
Higher level languages like Go or Python can create “good enough” standard libraries, that are correct for 99% of users.
Rust is really no different than C or C++ in this regard. Sure, C++ has a bigger standard library. But half of it is “oh don’t use that because it has foot guns for this use case, everyone uses this other external library anyways”.
The one big exception here is probably async. The language needs a better way for library writers to code against a generic runtime implementation without forcing a specific runtime onto the consumer.
For almost any functionality missing in the standard library, you could point to 2-3 popular crates that solve the problem in mutually exclusive ways, for different use cases.
Higher level languages like Go or Python can create “good enough” standard libraries, that are correct for 99% of users.
Rust is really no different than C or C++ in this regard. Sure, C++ has a bigger standard library. But half of it is “oh don’t use that because it has foot guns for this use case, everyone uses this other external library anyways”.
The one big exception here is probably async. The language needs a better way for library writers to code against a generic runtime implementation without forcing a specific runtime onto the consumer.
You're just elaborating on my use of the word "hard". Yes it is, and C++ is a good example of what not to do. Also, "good" is a spectrum, which means something a bit less shiny but standard is worth existing.
What R7RS-large is doing by standardizing stuff that was already discussed at length through SRFI seems like a good way.
From an external PoV, Clojure seem to be doing okay too.
What R7RS-large is doing by standardizing stuff that was already discussed at length through SRFI seems like a good way.
From an external PoV, Clojure seem to be doing okay too.
And, what, because it's named `std` it'll be magically better? Languages with giant stdlibs routinely have modules rot away because code maintenance doesn't get any easier, like Python. There are plenty of crates on crates.io made by the Rust project developers, I trust the RustCrypto guys pretty much the same amount, and merging the two teams together wouldn't solve any problems.
> And, what, because it's named `std` it'll be magically better?
Who said that? It won't be better, but it'll be always here, stable, conservative and won't depend on something outside std. And it won't be stored on an uncurated supply attack vector like crates.io.
If you want to reach a Common Lisp level of stability, there's no 101 ways to go about it.
Who said that? It won't be better, but it'll be always here, stable, conservative and won't depend on something outside std. And it won't be stored on an uncurated supply attack vector like crates.io.
If you want to reach a Common Lisp level of stability, there's no 101 ways to go about it.
> If we're talking build system, nobody is forcing you to use Crates.io with cargo, they just make it easy to.
Using cargo with distributed dependencies (e.g.: using git repositories) has several missing features, like resolving the latest semver-compatible version, etc. No only is it _easier_ to use cargo with crates.io, it's harder to use with anything else because of missing or incomplete features.
> You can use path-based dependencies just like CMake/VCPkg,Conan, or you can DIY the library.
Have you tried to do this? Cargo is a "many things in one" kind of tool, compiling a Rust library (e.g.: dependency) without it is a pain. If cargo had instead been multiple programs that each on one thing, it might be easier to opt out of it for regular projects.
Using cargo with distributed dependencies (e.g.: using git repositories) has several missing features, like resolving the latest semver-compatible version, etc. No only is it _easier_ to use cargo with crates.io, it's harder to use with anything else because of missing or incomplete features.
> You can use path-based dependencies just like CMake/VCPkg,Conan, or you can DIY the library.
Have you tried to do this? Cargo is a "many things in one" kind of tool, compiling a Rust library (e.g.: dependency) without it is a pain. If cargo had instead been multiple programs that each on one thing, it might be easier to opt out of it for regular projects.
Compared to ... cmake? vckpg? conan?
I have never had a good experience with those. However, using
mydep.path = <path>
in cargo has never been an issue.
And I hate to say it, path-based deps are much easier in C++ / C than other "make the build system do a coherent checkout" options like those mentioned above. So we're at worst parity for this use case, IMHO and subject to my own subjectivity, of course
I have never had a good experience with those. However, using
mydep.path = <path>
in cargo has never been an issue.
And I hate to say it, path-based deps are much easier in C++ / C than other "make the build system do a coherent checkout" options like those mentioned above. So we're at worst parity for this use case, IMHO and subject to my own subjectivity, of course
Speaking of Conan.
Something I don't see anyone talking about is, with such systems you have not one but at least two dependencies to audit per any library you want to include. One is the library itself, but the other one is the recipe for the package manager. At least with Conan, the recipes are usually written by third parties that are not affiliated with the dependency they're packaging. Now, Conan recipes are usually full-blown Python scripts that not only decide where to pull the target sources from themselves, they also tend to ship patches, which they apply to the target source code before building. That is, even if package source is specified directly and fixed on specific release, you still can't assume you're actually building the exact source of that release, because the recipe itself modifies the code of the release it pulled.
IMHO, trying to do OSS clearance on dependencies pulled via Conan makes very little sense if you don't also treat each Conan recipe as a separate library. But no one does.
Something I don't see anyone talking about is, with such systems you have not one but at least two dependencies to audit per any library you want to include. One is the library itself, but the other one is the recipe for the package manager. At least with Conan, the recipes are usually written by third parties that are not affiliated with the dependency they're packaging. Now, Conan recipes are usually full-blown Python scripts that not only decide where to pull the target sources from themselves, they also tend to ship patches, which they apply to the target source code before building. That is, even if package source is specified directly and fixed on specific release, you still can't assume you're actually building the exact source of that release, because the recipe itself modifies the code of the release it pulled.
IMHO, trying to do OSS clearance on dependencies pulled via Conan makes very little sense if you don't also treat each Conan recipe as a separate library. But no one does.
Why would you want to automatically resolve the latest semver compatible version from git if you care about security? That's worse than cargo.io where tags are stable, whereas git allows tags to be edited.
> No only is it _easier_ to use cargo with crates.io, it's harder to use with anything else because of missing or incomplete features.
You're saying the same thing twice here: yes it's easier to use cargo with crates.io. It follows immediately that it's harder to use without crates.io. That doesn't make your argument stronger.
Here's an article on how to use cargo without crates: https://thomask.sdf.org/blog/2023/11/14/rust-without-crates-...
> No only is it _easier_ to use cargo with crates.io, it's harder to use with anything else because of missing or incomplete features.
You're saying the same thing twice here: yes it's easier to use cargo with crates.io. It follows immediately that it's harder to use without crates.io. That doesn't make your argument stronger.
Here's an article on how to use cargo without crates: https://thomask.sdf.org/blog/2023/11/14/rust-without-crates-...
yes a large component of it is about developers. if developers were perfect beings, we wouldn't need rust in the first place.
Rust may not be what you want to write, but it's what you want your coworkers to write.
chrisco255(2)
> People will complain that "it's hard to add a dependency to a C++ project"
The way I see it the issue is that it's hard to add a dependency _in such a way that no people will have issues building your project with it_. This is problematic because even if you manage to make it work on your machine it may not work on some potential user or contributor's.
> But that just destroys the OS package distribution model: when I choose a Linux distribution, I choose to trust those who build it.
Distros still build Rust packages from sources and vendor crate dependencies in their repos. It's more painful because there are usually more dependencies with more updates, but this has nothing to do with shared libraries.
The way I see it the issue is that it's hard to add a dependency _in such a way that no people will have issues building your project with it_. This is problematic because even if you manage to make it work on your machine it may not work on some potential user or contributor's.
> But that just destroys the OS package distribution model: when I choose a Linux distribution, I choose to trust those who build it.
Distros still build Rust packages from sources and vendor crate dependencies in their repos. It's more painful because there are usually more dependencies with more updates, but this has nothing to do with shared libraries.
> The way I see it the issue is that it's hard to add a dependency _in such a way that no people will have issues building your project with it_.
From my point of view, if it's done properly I can just build/install the dependency and use pkgconfig. Whenever I have a problem, it's because it was done wrong. Because many (most?) developers can't be arsed to learn how to do it properly; it's easier to just say that dependency management in C++ sucks.
From my point of view, if it's done properly I can just build/install the dependency and use pkgconfig. Whenever I have a problem, it's because it was done wrong. Because many (most?) developers can't be arsed to learn how to do it properly; it's easier to just say that dependency management in C++ sucks.
Taking pride in being willing to take the longer and more error-prone and tedious path makes me wonder if you're not just flexing here.
There are thousands of tools and methodologies screaming for our attention to "be arsed to learn to do them properly".
They're not owed that attention, they must deserve it. And statistically + historically speaking, most have failed to do so.
You may choose to interpret this as you belonging to a small elite group of intellectuals who "are arsed to learn to do stuff properly" -- that's your right.
I choose to interpret it as "Cargo solves a real problem that has wasted numerous hours of my time in the past". And it wasn't because, for the third time, "I wasn't arsed to learn to do it properly", it's because nobody followed basic protocol and good practices to make the legendary "proper way" work reliably.
You're likely living and working in a bubble. It's much worse than the Wild West out there, man. Any tool that reduces the drudgery and allows for quicker doing of the annoying parts is a net positive.
There are thousands of tools and methodologies screaming for our attention to "be arsed to learn to do them properly".
They're not owed that attention, they must deserve it. And statistically + historically speaking, most have failed to do so.
You may choose to interpret this as you belonging to a small elite group of intellectuals who "are arsed to learn to do stuff properly" -- that's your right.
I choose to interpret it as "Cargo solves a real problem that has wasted numerous hours of my time in the past". And it wasn't because, for the third time, "I wasn't arsed to learn to do it properly", it's because nobody followed basic protocol and good practices to make the legendary "proper way" work reliably.
You're likely living and working in a bubble. It's much worse than the Wild West out there, man. Any tool that reduces the drudgery and allows for quicker doing of the annoying parts is a net positive.
> being willing to take the longer and more error-prone and tedious path
I disagree with the fact that it is all that. Usually, what I see from people doing CMake wrong is downright malpractice. Something like "I can't be arsed to read about it for 30min, so I'll just trial-and-error and copy-paste stuff without thinking for 4h until it works, and then I'll say it's not my fault if it sucks.
> There are thousands of tools and methodologies screaming for our attention to "be arsed to learn to do them properly".
That's bad faith. I learn to use all my tools properly. The more tools you know, the easier it gets to learn new ones. Also if you learn how they work, you can usually operate at a lower level, which means you need fewer tools (because lower level tools are more powerful). And there aren't new tools coming out every month. I mostly never have to learn new tools nowadays (and no, I'm not working in a super niche domain in C with one old framework).
> You're likely living and working in a bubble.
No. I feel the pain of most developers making a mess. The fact is that I can take a messy CMake setup, fix it, and then it's easy to use and it works.
But most developers don't actually know the two commands needed to build a CMake project. Tell them to set an option with `-D` and they will start crying.
I disagree with the fact that it is all that. Usually, what I see from people doing CMake wrong is downright malpractice. Something like "I can't be arsed to read about it for 30min, so I'll just trial-and-error and copy-paste stuff without thinking for 4h until it works, and then I'll say it's not my fault if it sucks.
> There are thousands of tools and methodologies screaming for our attention to "be arsed to learn to do them properly".
That's bad faith. I learn to use all my tools properly. The more tools you know, the easier it gets to learn new ones. Also if you learn how they work, you can usually operate at a lower level, which means you need fewer tools (because lower level tools are more powerful). And there aren't new tools coming out every month. I mostly never have to learn new tools nowadays (and no, I'm not working in a super niche domain in C with one old framework).
> You're likely living and working in a bubble.
No. I feel the pain of most developers making a mess. The fact is that I can take a messy CMake setup, fix it, and then it's easy to use and it works.
But most developers don't actually know the two commands needed to build a CMake project. Tell them to set an option with `-D` and they will start crying.
> I disagree with the fact that it is all that. Usually, what I see from people doing CMake wrong is downright malpractice. Something like "I can't be arsed to read about it for 30min, so I'll just trial-and-error and copy-paste stuff without thinking for 4h until it works, and then I'll say it's not my fault if it sucks.
I'll equate this to bash. I really wanted to learn it well and properly. I did, and I'd venture to say that happened at least 7 times (that I can remember of) and I forgot it each time because I don't use it often. I need it once or twice a month for a useful script and then it fades from memory. Sadface but those are the realities of the human brain (and I never wanted to resort to spaced repetition for bash in particular, though thinking of it now it was probably a much better idea than the other methods I tried for remembering all its quirks).
Ultimately I concluded that if something is quirky and needs you to put a special effort to remember how not to shoot yourself in the foot, then it's a bad tool. That's why I also abandoned `make`, learned `just` in literal 15 minutes and never looked back since.
After 23 years of a professional career and about 30 in total with computers I simply demand more quality of my tools:
- Have one and only one way to do thing X (also goes for programming languages though that's more difficult to achieve and not such a deal-breaker as it is with CLI / GUI tools);
- Make your most-commonly-used scenarios the easiest to do and the most trivial to remember;
- Make your tool and its most desired features easy to discover (via detailed help commands is one way, but there is also the "did you mean X?" etc.)
I am not taking jabs at CMake btw, I know exactly nothing about it. I am telling you that if I had to regularly do what I did with bash and make I'd ultimately conclude that it's a bad tool and will look to work around it... or I'll make a ton of shell aliases doing what I need from it and be very grumpy in the process (I mean: if I can't escape from CMake).
> That's bad faith. I learn to use all my tools properly.
OK, you got me, you are correct here. As admitted in the other comment, I am getting agitated by some of your comments because they seem to show lack of nuanced thinking. If I am wrong then I'd be happy.
> The more tools you know, the easier it gets to learn new ones.
...up to a point, after which you just get annoyed and want to scream "CAN'T YOU ALL JUST TALK TO EACH OTHER AND STANDARDIZE SOME STUFF ALREADY!".
I don't exaggerate when I say that I have learned no less than 200 CLI tools in the last 10-12 years, for many types of automating tasks (and even distributing them to clusters of machines), to wipe the arse of half-arsed niche databases some overly enthusiastic startup founder figured he needs to use as a competitive advantage (while he was on speed and LSD), to do data analysis for my own goals, to experiment with various task runners and build systems, and... yeah, maybe I should write some blog posts about it, it'll become too big.
My point is that what you say has an easily achievable peak after which you get just sick of everyone's crap. Just saying. Maybe you haven't reached that point yet.
> But most developers don't actually know the two commands needed to build a CMake project. Tell them to set an option with `-D` and they will start crying.
Now you are arguing in bad faith. :P I don't think you or I here should criticize those mystical "almost not devs" people just so we can feel smart. Using `-D` is something that every normal junior that I've coached will use and be grateful that I have showed them this "clever hack" (their words and they honestly made me laugh).
And if we try to be fair to all sides, many programmers want to, you know, only do programming, not learn a plethora of other tools that shouldn't have relevance to their work as much as they do. That's one of the reasons I gave up on C/C++ almost 20 years ago; I was doing much more effort making sure stuff compiles on several UNIX variants than actually solving any problems with code; the ratio was so bad that it was something like 70% to 30% on a good day; on some days it was 95% to 5% even.
It's all about striking a balance. Your comments come across as "git gud scrub!" to me and I can't ever let that go. It's demeaning and it hints at suffering from the main character syndrome (where you believe all others are stupid NPCs). And I am here to tell you that there are VERY GOOD REASONS why are you observing what you are observing and they are very rarely what you think they are.
I'll equate this to bash. I really wanted to learn it well and properly. I did, and I'd venture to say that happened at least 7 times (that I can remember of) and I forgot it each time because I don't use it often. I need it once or twice a month for a useful script and then it fades from memory. Sadface but those are the realities of the human brain (and I never wanted to resort to spaced repetition for bash in particular, though thinking of it now it was probably a much better idea than the other methods I tried for remembering all its quirks).
Ultimately I concluded that if something is quirky and needs you to put a special effort to remember how not to shoot yourself in the foot, then it's a bad tool. That's why I also abandoned `make`, learned `just` in literal 15 minutes and never looked back since.
After 23 years of a professional career and about 30 in total with computers I simply demand more quality of my tools:
- Have one and only one way to do thing X (also goes for programming languages though that's more difficult to achieve and not such a deal-breaker as it is with CLI / GUI tools);
- Make your most-commonly-used scenarios the easiest to do and the most trivial to remember;
- Make your tool and its most desired features easy to discover (via detailed help commands is one way, but there is also the "did you mean X?" etc.)
I am not taking jabs at CMake btw, I know exactly nothing about it. I am telling you that if I had to regularly do what I did with bash and make I'd ultimately conclude that it's a bad tool and will look to work around it... or I'll make a ton of shell aliases doing what I need from it and be very grumpy in the process (I mean: if I can't escape from CMake).
> That's bad faith. I learn to use all my tools properly.
OK, you got me, you are correct here. As admitted in the other comment, I am getting agitated by some of your comments because they seem to show lack of nuanced thinking. If I am wrong then I'd be happy.
> The more tools you know, the easier it gets to learn new ones.
...up to a point, after which you just get annoyed and want to scream "CAN'T YOU ALL JUST TALK TO EACH OTHER AND STANDARDIZE SOME STUFF ALREADY!".
I don't exaggerate when I say that I have learned no less than 200 CLI tools in the last 10-12 years, for many types of automating tasks (and even distributing them to clusters of machines), to wipe the arse of half-arsed niche databases some overly enthusiastic startup founder figured he needs to use as a competitive advantage (while he was on speed and LSD), to do data analysis for my own goals, to experiment with various task runners and build systems, and... yeah, maybe I should write some blog posts about it, it'll become too big.
My point is that what you say has an easily achievable peak after which you get just sick of everyone's crap. Just saying. Maybe you haven't reached that point yet.
> But most developers don't actually know the two commands needed to build a CMake project. Tell them to set an option with `-D` and they will start crying.
Now you are arguing in bad faith. :P I don't think you or I here should criticize those mystical "almost not devs" people just so we can feel smart. Using `-D` is something that every normal junior that I've coached will use and be grateful that I have showed them this "clever hack" (their words and they honestly made me laugh).
And if we try to be fair to all sides, many programmers want to, you know, only do programming, not learn a plethora of other tools that shouldn't have relevance to their work as much as they do. That's one of the reasons I gave up on C/C++ almost 20 years ago; I was doing much more effort making sure stuff compiles on several UNIX variants than actually solving any problems with code; the ratio was so bad that it was something like 70% to 30% on a good day; on some days it was 95% to 5% even.
It's all about striking a balance. Your comments come across as "git gud scrub!" to me and I can't ever let that go. It's demeaning and it hints at suffering from the main character syndrome (where you believe all others are stupid NPCs). And I am here to tell you that there are VERY GOOD REASONS why are you observing what you are observing and they are very rarely what you think they are.
I think we tend to agree, but you're more frustrated by the tools and I am more frustrated by the rants from people who clearly haven't put the effort (apparently not you).
Because it's easy to criticise CMake/Automake: we love to hate them. But in my experience, even though there are tons of valid points to make, many people who say "it's just because CMake sucks" are actually just making a mess and blaming CMake.
> Have one and only one way to do thing X
Agreed. To add some nuance though, most projects are perfectly fine by just using a minimal set of features in CMake: just use `find_package` and install the dependency on the system, or locally and learn how to find it with CMAKE_PREFIX_PATH and pkgconfig. I would even say that writing a custom CMake command is almost always a bad idea.
But to me it doesn't mean that you need to have one tool that does everything (and that also makes coffee). To me it's a bad idea to configure an entire CMake project to be run by gradle for Android. Mixing build systems makes matter worse, not better.
> Make your most-commonly-used scenarios the easiest to do and the most trivial to remember;
Sure, but the risk there is that many people hide the lower level because they try to "keep it simple stupid". Which IMO is wrong. Expose the lower level, but clearly do it in... a lower-level abstraction. So that advanced users can use the low-level, and others can use the higher-level. And usually it means: start with the low-level API. If it's good, maybe someone else will write an abstraction on top of it!
I am often frustrated because projects put a lot of effort into hiding lower-level APIs and then I'm stuck (and need to e.g. fork).
> That's why I also abandoned `make`
Note that I don't write projects with make. But it has merit, and I learned to understand/maintain projects based on it. I don't learn all the build systems under the sun, but make, CMake and meson (in the C++ ecosystem) seem valuable to at least understand.
The risk with introducing yet another tool like "just" is that you make "your" problem worse: it introduces a new tool. I am usually fine learning new tools, but in my projects I try not to introduce them: I use CMake or Meson because it makes sense to expect that C++ devs should understand them.
Because it's easy to criticise CMake/Automake: we love to hate them. But in my experience, even though there are tons of valid points to make, many people who say "it's just because CMake sucks" are actually just making a mess and blaming CMake.
> Have one and only one way to do thing X
Agreed. To add some nuance though, most projects are perfectly fine by just using a minimal set of features in CMake: just use `find_package` and install the dependency on the system, or locally and learn how to find it with CMAKE_PREFIX_PATH and pkgconfig. I would even say that writing a custom CMake command is almost always a bad idea.
But to me it doesn't mean that you need to have one tool that does everything (and that also makes coffee). To me it's a bad idea to configure an entire CMake project to be run by gradle for Android. Mixing build systems makes matter worse, not better.
> Make your most-commonly-used scenarios the easiest to do and the most trivial to remember;
Sure, but the risk there is that many people hide the lower level because they try to "keep it simple stupid". Which IMO is wrong. Expose the lower level, but clearly do it in... a lower-level abstraction. So that advanced users can use the low-level, and others can use the higher-level. And usually it means: start with the low-level API. If it's good, maybe someone else will write an abstraction on top of it!
I am often frustrated because projects put a lot of effort into hiding lower-level APIs and then I'm stuck (and need to e.g. fork).
> That's why I also abandoned `make`
Note that I don't write projects with make. But it has merit, and I learned to understand/maintain projects based on it. I don't learn all the build systems under the sun, but make, CMake and meson (in the C++ ecosystem) seem valuable to at least understand.
The risk with introducing yet another tool like "just" is that you make "your" problem worse: it introduces a new tool. I am usually fine learning new tools, but in my projects I try not to introduce them: I use CMake or Meson because it makes sense to expect that C++ devs should understand them.
There's a corollary here to "build it yourself", which is "vet it yourself". Cargo, npm, and pip all default to a central registry which you're encouraged to trust implicitly, but we've seen time and time again that central registries do not adequately protect against broken or malicious code that causes major security flaws downstream. Each of these ecosystems trains its developers to use hundreds of dependencies—far more than they can personally vet—with the understanding that someone else must surely have done so, even though we've seen over and over again that the staff of these registries can't actually keep up and that even long-running and popular projects can suddenly become insecure.
I'd like to see an ecosystem develop that provides a small amount of convenience on top of plain old vendoring. No central repository to create a false sense of security. No overly clever version resolution scheme to hide the consequences of libraries depending on dozens of transitive dependencies. Just a minimal version resolution algorithm and a decentralized registry system—give each developer an index of their own libraries which they maintain and make downstream developers pick and choose which developers they actually trust.
Maybe a bit like Maven if Maven Central didn't exist?
I'd like to see an ecosystem develop that provides a small amount of convenience on top of plain old vendoring. No central repository to create a false sense of security. No overly clever version resolution scheme to hide the consequences of libraries depending on dozens of transitive dependencies. Just a minimal version resolution algorithm and a decentralized registry system—give each developer an index of their own libraries which they maintain and make downstream developers pick and choose which developers they actually trust.
Maybe a bit like Maven if Maven Central didn't exist?
Thought about it many times but never had the time or energy to tackle something as big and new. You?
I agree. It's really nice how few dependencies get pulled into a typical c++ project. When I started playing with rust, I was shocked at at how many dependencies got pulled in to just build hello world. I'm just not interested in adopting the npm left-pad model.
While Rust projects unquestionably pull more dependency, there's zero of the left-pad culture in the community. It's simply pragmatism. And the OP is wrong about the terminal functionality by the way; stuff does actually change there still (sadly, though not very often).
No point reacting to something that is almost disinformation.
No point reacting to something that is almost disinformation.
> In C++, I control my dependencies. In Rust I get above 100 so quickly I just
Just don't add dependencies, it's that simple. If you have enough time to control C++ dependencies, you can control them in Rust as well.
Just don't add dependencies, it's that simple. If you have enough time to control C++ dependencies, you can control them in Rust as well.
That's not an answer when the entire ecosystem is built around the idea of adding lots of dependencies to do stuff. I don't need no dependencies, I'd like to live in a world where I can add two or three. But if the culture is so far gone that those two or three transitively import 20 each, I don't have that as an option—it's all or nothing.
[deleted]
> That's not an answer when the entire ecosystem is built around the idea of adding lots of dependencies to do stuff.
Again. Don't add dependencies. Just don't. Vendor it yourself. Or write it yourself. Absolutely nothing is forcing you to use the dependencies except your own desire to save time.
Cargo is giving you the option to A) save your own time B) minimize dependencies. You choose A) and blame Cargo.
Again. Don't add dependencies. Just don't. Vendor it yourself. Or write it yourself. Absolutely nothing is forcing you to use the dependencies except your own desire to save time.
Cargo is giving you the option to A) save your own time B) minimize dependencies. You choose A) and blame Cargo.
I don't think it's entirely fair. You're telling me "don't use any dependency at all". Vendoring is not a solution, because it does not remove the transitive dependencies. So I have to essentially rewrite it myself.
I find many libraries in C++ that don't pull 20 dependencies. It's common for me to have a C++ project that has ~6-8 dependencies. In Rust I have never seen that. Either I have no dependency at all, or I have 100+.
I actually had this toy project that I wrote both in C++ and in Rust, for the learning experience. Both did exactly the same thing, with a very similar design. In C++ I had 8 dependencies. In Rust, 186. The thing is, 186 is enough for me to not even read all their names in the lock file. As soon as 25 dependencies magically appear in my lock file, I stop caring and start YOLOing it.
I find many libraries in C++ that don't pull 20 dependencies. It's common for me to have a C++ project that has ~6-8 dependencies. In Rust I have never seen that. Either I have no dependency at all, or I have 100+.
I actually had this toy project that I wrote both in C++ and in Rust, for the learning experience. Both did exactly the same thing, with a very similar design. In C++ I had 8 dependencies. In Rust, 186. The thing is, 186 is enough for me to not even read all their names in the lock file. As soon as 25 dependencies magically appear in my lock file, I stop caring and start YOLOing it.
I think that with C++ there’s a much bigger culture of one big library that does many things.
It’s been a few years but most projects I worked on in C++ pulled in Boost which is more akin to 50 libraries all together in one.
It’s been a few years but most projects I worked on in C++ pulled in Boost which is more akin to 50 libraries all together in one.
> Vendoring is not a solution, because it does not remove the transitive dependencies. So I have to essentially rewrite it myself.
Vendoring means, you fork it, make any modification you want, such as reducing the number of dependencies, but keeping it up to date with upstream is on you.
Rewriting it for yourself is what happens in C++. C++ didn't get flatter by waving a magic wand, it just moved the slider. It's just that code reuse is hard, so you rewrite things that would be a shared dependency in Rust into a header file and call it a day.
> I find many libraries in C++ that don't pull 20 dependencies.
I can find many libraries in Rust that don't pull 20 dependencies (mostly by searching crates written by people like the_mitsuhiko). Some people care, but they are a minority. If you truly care about this, like the_mitsuhiko does, write it with no dependencies.
I, personally, try to minimize my dependencies, but there is a reason why most people don't care - it has little to no impact on them and saving time by NIH is better than not learning from history and reinventing wheel only square.
---
Problem is: There are infinitely many of properties to improve (number of dependencies, byte size, compile time, run time, alignment, crates not made by X, time to market, reproducibility, compliance with law XYZKM, safety etc.) and not enough time or possibility space to do them all.
Vendoring means, you fork it, make any modification you want, such as reducing the number of dependencies, but keeping it up to date with upstream is on you.
Rewriting it for yourself is what happens in C++. C++ didn't get flatter by waving a magic wand, it just moved the slider. It's just that code reuse is hard, so you rewrite things that would be a shared dependency in Rust into a header file and call it a day.
> I find many libraries in C++ that don't pull 20 dependencies.
I can find many libraries in Rust that don't pull 20 dependencies (mostly by searching crates written by people like the_mitsuhiko). Some people care, but they are a minority. If you truly care about this, like the_mitsuhiko does, write it with no dependencies.
I, personally, try to minimize my dependencies, but there is a reason why most people don't care - it has little to no impact on them and saving time by NIH is better than not learning from history and reinventing wheel only square.
---
Problem is: There are infinitely many of properties to improve (number of dependencies, byte size, compile time, run time, alignment, crates not made by X, time to market, reproducibility, compliance with law XYZKM, safety etc.) and not enough time or possibility space to do them all.
> you fork it, make any modification you want, such as reducing the number of dependencies, but keeping it up to date with upstream is on you.
Say a pretty complex library has 30 dependencies. I fork it, I remove the dependencies, and now it doesn't build anymore. What do I do?
I feel like you're saying "You should not write the small feature you need yourself. Instead, you should heavily modify a big dependency that contains the feature you need, because that's certainly easier and better".
If the library did not rely on those 30 dependencies, it would not depend on them, would it?
> If you truly care about this, like the_mitsuhiko does, write it with no dependencies.
So you start by telling me that I should just remove the dependencies from my dependencies, and then you tell me that you don't do that and instead choose dependencies that don't have many dependencies themselves? It feels like we agree...
> not learning from history and reinventing wheel only square
This is the thing: "I learned from history so I don't reinvent the wheel" should not be treated like a religious belief. It's probably not a good idea to write your own kernel. But I often spend a day or two implementing a feature that I would have gotten from a dependency in 2h. And it's almost always worth it! And sometimes I depend on libraries (e.g. I usually don't rewrite an HTTP library).
Say a pretty complex library has 30 dependencies. I fork it, I remove the dependencies, and now it doesn't build anymore. What do I do?
I feel like you're saying "You should not write the small feature you need yourself. Instead, you should heavily modify a big dependency that contains the feature you need, because that's certainly easier and better".
If the library did not rely on those 30 dependencies, it would not depend on them, would it?
> If you truly care about this, like the_mitsuhiko does, write it with no dependencies.
So you start by telling me that I should just remove the dependencies from my dependencies, and then you tell me that you don't do that and instead choose dependencies that don't have many dependencies themselves? It feels like we agree...
> not learning from history and reinventing wheel only square
This is the thing: "I learned from history so I don't reinvent the wheel" should not be treated like a religious belief. It's probably not a good idea to write your own kernel. But I often spend a day or two implementing a feature that I would have gotten from a dependency in 2h. And it's almost always worth it! And sometimes I depend on libraries (e.g. I usually don't rewrite an HTTP library).
> So you start by telling me that I should just remove the dependencies from my dependencies, and then you tell me that you don't do that and instead choose dependencies that don't have many dependencies themselves? It feels like we agree...
There are two cases: 1. You have 100 dependencies. 2. You just started writing your code.
If you are doing 1) then you have to vendor/fork till it's acceptable.
If you are starting from 2) just don't add them.
> This is the thing: "I learned from history so I don't reinvent the wheel" should not be treated like a religious belief.
Nothing in programming should be treated as religious belief. But duplicating code can be just as deadly as dependencies glut, taken to it's logical extreme.
A golden middle approach is best, sadly no one can agree on it.
There are two cases: 1. You have 100 dependencies. 2. You just started writing your code.
If you are doing 1) then you have to vendor/fork till it's acceptable.
If you are starting from 2) just don't add them.
> This is the thing: "I learned from history so I don't reinvent the wheel" should not be treated like a religious belief.
Nothing in programming should be treated as religious belief. But duplicating code can be just as deadly as dependencies glut, taken to it's logical extreme.
A golden middle approach is best, sadly no one can agree on it.
> A golden middle approach is best, sadly no one can agree on it.
That's what I was saying: I think that we agree there. My initial point was that I don't think that the package management culture in Rust is a middle approach: it feels like most crates don't care about how many random dependencies they pull.
That's what I was saying: I think that we agree there. My initial point was that I don't think that the package management culture in Rust is a middle approach: it feels like most crates don't care about how many random dependencies they pull.
> My initial point was that I don't think that the package management culture in Rust is a middle approach
And I don't care about "cultures". They come and go, and change with time. What I care about are unchanging things - the programs.
Sane package managers allow you to make your code however you want to. OSS allows you to tweak the code in whatever way you need, arguing for CMake over npm is frankly baffling.
And I don't care about "cultures". They come and go, and change with time. What I care about are unchanging things - the programs.
Sane package managers allow you to make your code however you want to. OSS allows you to tweak the code in whatever way you need, arguing for CMake over npm is frankly baffling.
Well cargo works well, I can't deny it.
I was really talking about culture, which is important to me. The fact that many devs assume that everybody using Linux is on Ubuntu is culture, right? In Rust, it is often assumed that Rust was installed with rustup. It's nice to say "you are not forced to use rustup", but if it doesn't work when you don't, then you are kind of forced :-).
I was really talking about culture, which is important to me. The fact that many devs assume that everybody using Linux is on Ubuntu is culture, right? In Rust, it is often assumed that Rust was installed with rustup. It's nice to say "you are not forced to use rustup", but if it doesn't work when you don't, then you are kind of forced :-).
> Say a pretty complex library has 30 dependencies. I fork it, I remove the dependencies, and now it doesn't build anymore. What do I do?
If you don't know what to do in this situation it means you don't know how your dependency works, which means you've abdicated responsibility for your own code's runtime behavior. Please don't take this as accusatory, I do this almost every single day at work because it's unavoidable--I'd be fired if I didn't. But it makes me feel dirty.
If you don't know what to do in this situation it means you don't know how your dependency works, which means you've abdicated responsibility for your own code's runtime behavior. Please don't take this as accusatory, I do this almost every single day at work because it's unavoidable--I'd be fired if I didn't. But it makes me feel dirty.
The alternative is just as grim. I heard[1] that Excel went with the slogan "kill your dependencies" at least during Joel Spolsky's time. This ended with Excel team building and shipping their own C compiler.
Imagine NIH so hard, you start rewriting the compiler.
[1]https://blog.codinghorror.com/dependency-avoidance/
Imagine NIH so hard, you start rewriting the compiler.
[1]https://blog.codinghorror.com/dependency-avoidance/
> Vendoring means, you fork it, make any modification you want, such as reducing the number of dependencies, but keeping it up to date with upstream is on you.
This is not what vendoring means, this is called forking. Vendoring is just the copy+commit phase, not the modification part.
This is not what vendoring means, this is called forking. Vendoring is just the copy+commit phase, not the modification part.
[deleted]
Dunno. That's what one project I worked on called it. Forking code on GH, pulling as submodules and maintaining it (most repos were unmaintained for years).
If you can implement it in C++ with no dependencies, you can also just, do the same with rust. Like just don't add dependencies.
Obviously. But that's not what we are talking about here. I am saying that if I can implement it in C++ with few dependencies, I get tons of dependencies when doing the same in Rust.
It's different.
It's different.
This frankly sounds like a rationalization for an aesthetic preference. It is undeniable that being able to easily build on top of others' hard work is an enormous advantage in any domain.
Duplicate work should only happen if you are confident that your requirements are significantly different, and that you can deliver an implementation as good as a team that has likely focused on the problem for much longer and has acquired a lot more know-how.
It is true that such an attitude might be justifiable for certain security or performance critical applications, you might need end-to-end control. But even in those cases, the argument for trusting yourself by default over focused and experienced library authors is dubious.
Either way, a good dependency manager opens the door for better auditing, analysis and tagging of dependencies, so that such critical requirements can be properly guaranteed, again probably better than you can do yourself.
Duplicate work should only happen if you are confident that your requirements are significantly different, and that you can deliver an implementation as good as a team that has likely focused on the problem for much longer and has acquired a lot more know-how.
It is true that such an attitude might be justifiable for certain security or performance critical applications, you might need end-to-end control. But even in those cases, the argument for trusting yourself by default over focused and experienced library authors is dubious.
Either way, a good dependency manager opens the door for better auditing, analysis and tagging of dependencies, so that such critical requirements can be properly guaranteed, again probably better than you can do yourself.
> Duplicate work should only happen if you are confident that [...]
You are making a ton of assumptions on the quality of the average library you can depend on: that a whole team worked on it, that it worked on it a long time and that it produced good code. That's really not a given.
You are also assuming that writing a feature instead of using a library is "duplicate work". It's not necessarily the case. Maybe the library approaches the problem in a much more generic way than you need. Your requirements are not "significantly different" then, just a subset of the library.
> so that such critical requirements can be properly guaranteed, again probably better than you can do yourself.
Are you saying that the packages available in cargo are systematically audited? I believe that the vast majority of them is not audited at all.
You are making a ton of assumptions on the quality of the average library you can depend on: that a whole team worked on it, that it worked on it a long time and that it produced good code. That's really not a given.
You are also assuming that writing a feature instead of using a library is "duplicate work". It's not necessarily the case. Maybe the library approaches the problem in a much more generic way than you need. Your requirements are not "significantly different" then, just a subset of the library.
> so that such critical requirements can be properly guaranteed, again probably better than you can do yourself.
Are you saying that the packages available in cargo are systematically audited? I believe that the vast majority of them is not audited at all.
> it figures out your terminal dimensions. The underlying APIs it uses have effectively been stable since the earliest days of computing terminals—what, 50 years or so?
No, they haven't been stable, not really. The TIOCGWINSZ ioctl has never been standardized to my knowledge, and it has many different names on different Unixes and BSDs. The tcgetwinsize() function only got in POSIX in 2024, and this whole thing has really sad history, honestly [0], and that's before we even get to the Windows side of things.
[0] https://news.ycombinator.com/item?id=42039401
No, they haven't been stable, not really. The TIOCGWINSZ ioctl has never been standardized to my knowledge, and it has many different names on different Unixes and BSDs. The tcgetwinsize() function only got in POSIX in 2024, and this whole thing has really sad history, honestly [0], and that's before we even get to the Windows side of things.
[0] https://news.ycombinator.com/item?id=42039401
This was vaguely my take away from the article: It's not that his replacements are simpler because they're better or made by him. They're simpler because they're only handling his use-cases.
Sometimes - that's fine.
Sometimes - that's making his software worse for folks who have different use-cases, or are running on systems he doesn't understand or use himself.
The real value of a library, even with all those dependencies (and to be clear, I disagree that 3 or 4 dependencies for a library that runs across windows/linux is "all that many", esp when his platform specific implementation still uses at least 1), is that it turns out even relatively simple problems have a wealth of complexity to them. The person who's job it is to write that library is going to be more experienced in the subject domain than you (at least in the good cases) and they can deal with it. Most importantly - they can deal with your unknown, unknowns. The places you don't even have the experience to know you're missing information.
Sometimes - that's fine.
Sometimes - that's making his software worse for folks who have different use-cases, or are running on systems he doesn't understand or use himself.
The real value of a library, even with all those dependencies (and to be clear, I disagree that 3 or 4 dependencies for a library that runs across windows/linux is "all that many", esp when his platform specific implementation still uses at least 1), is that it turns out even relatively simple problems have a wealth of complexity to them. The person who's job it is to write that library is going to be more experienced in the subject domain than you (at least in the good cases) and they can deal with it. Most importantly - they can deal with your unknown, unknowns. The places you don't even have the experience to know you're missing information.
> They're simpler because they're only handling his use-cases.
This is a major part of the thesis of no dependencies. General code is bad code. It’s slow, branchy, complex, filled with mutexes, nan checks, etc. Read “the old new thing”, to see the extreme
When you have a concrete goal you can apply assumptions that simplify the problem space.
A good example of this was Casey’s work on a fast terminal. At first all the naysaying was “production terminals are really hard because you have to handle fonts and internationalization, accessibility, etc”. Indeed those problems suck, but he used a general windows API to render a concrete representation of the char set on demand, and the the rest was simple.
This is a major part of the thesis of no dependencies. General code is bad code. It’s slow, branchy, complex, filled with mutexes, nan checks, etc. Read “the old new thing”, to see the extreme
When you have a concrete goal you can apply assumptions that simplify the problem space.
A good example of this was Casey’s work on a fast terminal. At first all the naysaying was “production terminals are really hard because you have to handle fonts and internationalization, accessibility, etc”. Indeed those problems suck, but he used a general windows API to render a concrete representation of the char set on demand, and the the rest was simple.
> General code is bad code.
For whom?
I think most times, as a user of software, I almost always prefer to have something that solves my problem, even if it's got some rough edges or warts. That's what general code is - stuff that solves a problem for lots of people.
Would I prefer a tool that solves exactly my problem in the best way possible? Yeah, sure. Do I want to pay what that costs in money, time or attention? Usually no. The general purpose tool is plenty good enough to solve the problem now and let me move on.
The value I get from solving the problem isn't really tied to how optimally I solve the problem. If I can buy a single hammer that drives all the nails I need today - that's a BETTER solution for me than spending 10 hours speccing out the ideal hammer for each nail and each hand that might hold it, much less paying for them all.
I'll have already finished if I just pick the general purpose hammer, getting my job done and providing value.
---
So to your terminal example - I think you're genuinely arguing for more general code here.
There's performance in making a terminal run at 6k fps. It's an art. It's clearly a skill and I can respect it. Sounds like it's an edge case that dude wants, so I'm in favor of trying to make the terminal faster (and more general).
But... I also don't give a flying fuck for anything I do. Printing 1gb of text to the terminal is useless to me from a value perspective (it's nearly 1000 full length novels of text, I can't read that much in a year if it was all I did, so going from 5 minutes to 5 seconds is almost meaningless to me).
The sum total of the value I see from that change is "maybe once or twice a year when I cat a long file by mistake, I don't have to hit ctrl-c".
I also genuinely fail to understand how this guy gets meaningful value from printing 1gb of text to a terminal that quickly either... even the fastest of speed readers are still going to be SO MANY orders of magnitude slower to process that, and anything else he might want to do with that text is already plenty fast - copying it to a new file? already fast. Searching it? fast. Deleting it? fast. Editing it? fast.
So... I won't make any comment on why this case is slow or the discussion around it (I haven't read it, it sounds like it could be faster, and they made a lot of excuses not to solve his specific edge case). All I'll say is your argument sure sounds like adding an edge case that nearly no one has, there-by making the terminal more general.
Any terminal I wrote for myself sure as fuck wouldn't be as fast as that because I don't have the rendering experience he has, and my use case doesn't need it at all.
For whom?
I think most times, as a user of software, I almost always prefer to have something that solves my problem, even if it's got some rough edges or warts. That's what general code is - stuff that solves a problem for lots of people.
Would I prefer a tool that solves exactly my problem in the best way possible? Yeah, sure. Do I want to pay what that costs in money, time or attention? Usually no. The general purpose tool is plenty good enough to solve the problem now and let me move on.
The value I get from solving the problem isn't really tied to how optimally I solve the problem. If I can buy a single hammer that drives all the nails I need today - that's a BETTER solution for me than spending 10 hours speccing out the ideal hammer for each nail and each hand that might hold it, much less paying for them all.
I'll have already finished if I just pick the general purpose hammer, getting my job done and providing value.
---
So to your terminal example - I think you're genuinely arguing for more general code here.
There's performance in making a terminal run at 6k fps. It's an art. It's clearly a skill and I can respect it. Sounds like it's an edge case that dude wants, so I'm in favor of trying to make the terminal faster (and more general).
But... I also don't give a flying fuck for anything I do. Printing 1gb of text to the terminal is useless to me from a value perspective (it's nearly 1000 full length novels of text, I can't read that much in a year if it was all I did, so going from 5 minutes to 5 seconds is almost meaningless to me).
The sum total of the value I see from that change is "maybe once or twice a year when I cat a long file by mistake, I don't have to hit ctrl-c".
I also genuinely fail to understand how this guy gets meaningful value from printing 1gb of text to a terminal that quickly either... even the fastest of speed readers are still going to be SO MANY orders of magnitude slower to process that, and anything else he might want to do with that text is already plenty fast - copying it to a new file? already fast. Searching it? fast. Deleting it? fast. Editing it? fast.
So... I won't make any comment on why this case is slow or the discussion around it (I haven't read it, it sounds like it could be faster, and they made a lot of excuses not to solve his specific edge case). All I'll say is your argument sure sounds like adding an edge case that nearly no one has, there-by making the terminal more general.
Any terminal I wrote for myself sure as fuck wouldn't be as fast as that because I don't have the rendering experience he has, and my use case doesn't need it at all.
> For whom?
For users and programmers. Slower to use. Harder to read and maintain. More code to do the same tasks.
> I almost always prefer to have something that solves my problem
I didn’t say anything about losing features. I don’t read Arabic. I’m glad the OS supports it. But it would be bad if Arabic complexity had to leak into every line of code. Limit the complexity scope. Casey’s terminal supports Arabic by following this principle.
> The value I get from solving the problem isn't really tied to how optimally I solve the problem.
Yes it is. There are ok products and there are great products.
> text still uses general solution
Operating systems have to solve general problems which is why they are expensive and brittle. Every program is not an operating system.
> There's performance in making a terminal run at 6k fps. It's an art.
I think you are missing context about the issue that led to this. And yes, as soon as there are no spinning wheels while doing normal tasks, then we can stop complaining about performance. But that’s not the situation.
> and my use case doesn't need it at all.
Would the world be better or worse if your operating system respected your time more?
For users and programmers. Slower to use. Harder to read and maintain. More code to do the same tasks.
> I almost always prefer to have something that solves my problem
I didn’t say anything about losing features. I don’t read Arabic. I’m glad the OS supports it. But it would be bad if Arabic complexity had to leak into every line of code. Limit the complexity scope. Casey’s terminal supports Arabic by following this principle.
> The value I get from solving the problem isn't really tied to how optimally I solve the problem.
Yes it is. There are ok products and there are great products.
> text still uses general solution
Operating systems have to solve general problems which is why they are expensive and brittle. Every program is not an operating system.
> There's performance in making a terminal run at 6k fps. It's an art.
I think you are missing context about the issue that led to this. And yes, as soon as there are no spinning wheels while doing normal tasks, then we can stop complaining about performance. But that’s not the situation.
> and my use case doesn't need it at all.
Would the world be better or worse if your operating system respected your time more?
> Would the world be better or worse if your operating system respected your time more?
This is a non-sequitur.
It's like saying that my Honda CRV is disrespecting my time because it has a top speed of 110, when it could be a 230 mph F1 racecar - it's a bad argument. I drive on roads with a max speed limit of 75... and most are down near 25mph.
> There are ok products and there are great products.
And users don't give a flying fuck about which yours is if they can't use it in the first place. That's why general things exist.
This is a non-sequitur.
It's like saying that my Honda CRV is disrespecting my time because it has a top speed of 110, when it could be a 230 mph F1 racecar - it's a bad argument. I drive on roads with a max speed limit of 75... and most are down near 25mph.
> There are ok products and there are great products.
And users don't give a flying fuck about which yours is if they can't use it in the first place. That's why general things exist.
[deleted]
100%. If OP is willing to maintain a rust crate that takes in no dependencies and can determine terminal size on any platform I choose to build for, then I will gladly use your crate.
OTOH, if minimizing dependencies is important for a very specific project, then the extra work of implementing the functionality falls on that project. It will be simpler because it must only support one project. It may not receive critical compatibility updates or security updates also.
It does not fall on the community to go and remove dependencies from all their battle tested crates that are in common use. I think anyone and everyone would choose a crate with fewer over more dependencies. So, go make them?
OTOH, if minimizing dependencies is important for a very specific project, then the extra work of implementing the functionality falls on that project. It will be simpler because it must only support one project. It may not receive critical compatibility updates or security updates also.
It does not fall on the community to go and remove dependencies from all their battle tested crates that are in common use. I think anyone and everyone would choose a crate with fewer over more dependencies. So, go make them?
> If OP is willing to maintain a rust crate that takes in no dependencies and can determine terminal size on any platform I choose to build for, then I will gladly use your crate.
I already mentioned this on twitter but not a lot of people work this way. I don't have to point you farther than my sha1-smol crate. It was originally published under the sha1 name and was the one that the entire ecosystem used. As rust-crypto became more popular there were demands that the name was used for rust-crypto instead.
I have given up the name, moved the crate to sha1-smol. It has decent downloads, but it only has 40 dependents vs. >600 for sha1. Data would indicate that people don't really care all that much about it.
(Or the sha1 crate is that much better than sha1-smol, but I'm not sure if people actually end up noticing the minor performance improvements in practice)
I already mentioned this on twitter but not a lot of people work this way. I don't have to point you farther than my sha1-smol crate. It was originally published under the sha1 name and was the one that the entire ecosystem used. As rust-crypto became more popular there were demands that the name was used for rust-crypto instead.
I have given up the name, moved the crate to sha1-smol. It has decent downloads, but it only has 40 dependents vs. >600 for sha1. Data would indicate that people don't really care all that much about it.
(Or the sha1 crate is that much better than sha1-smol, but I'm not sure if people actually end up noticing the minor performance improvements in practice)
Not trying to be perverse - but why does sha1-smol need any dependencies, let alone 40? Isn’t it a single public pure function from a slice of bytes to a u128? (Implemented with some temporary state and a handful of private functions.)
(I am likely being completely naive, but I am well satisfied by Julia’s stdlib function for this, so wondering what makes Rust different since I’ve been doing more and more rust lately and am trying to understand what the difference might be).
(I am likely being completely naive, but I am well satisfied by Julia’s stdlib function for this, so wondering what makes Rust different since I’ve been doing more and more rust lately and am trying to understand what the difference might be).
> Not trying to be perverse - but why does sha1-smol need any dependencies, let alone 40?
It has zero dependencies, but only 40 crates on the eco system depend on it. Compared to sha1 which has 10 dependencies, but > 660 crates depend on it.
It has zero dependencies, but only 40 crates on the eco system depend on it. Compared to sha1 which has 10 dependencies, but > 660 crates depend on it.
Oh right! Gotcha, I had it backwards.
Ha! Hi, I think we're having parallel conversations on there as well. Well, good to make a second loop closure. I won't copy my reply here.
> but it only has 40 dependents
Isn’t sha1 a several hundred line function? What’s going on?
Isn’t sha1 a several hundred line function? What’s going on?
The rust-crypto sha1 crate has four direct dependencies
cfg-if - a tiny library making it easier to use different implementations on different platforms. Maintained by the official rust-lang libs team. Also used by the rust std library. No recursive dependencies.
cpufeatures - a tiny library for detecting CPU features maintained by the rust-crypto people, with a recursive dependency only on libc (which is maintained by rust-lang and used by the standard library).
digest - a library providing traits abstracting over different hash functions, maintained by the rust-crypto-people. In turn digest depends blockbuffer (some code to minimize boundchecks) and cryptocommon (more traits for abstracting), both maintained by the rust-crypto people. Both in turn depend on
- typenum, a third party library for compile time math, no dependencies.
- generic-array, a third library for fixed length arrays with lengths computed by typenum, typenum is its only dependency.
- version_check, a third party library for checking what features the rustc being used supports (only block-buffer depends on this one).
sha1-asm - A library with a faster assembly implementation of sha1, maintained by the rust-crypto people. Dependent only on cc (c compiler support), maintained by rust-lang, used in building the rust compiler. cc is itself in turn dependent only on
- shlex, a third party library for splitting things the same way a shell does. So this tree ends in an actual third party dependency, but one you're already indirectly dependent on by simply using the rust compiler at all.
Oh, and in the next release
sha1-asm is being removed
The crypto common dependencies are being replaced with hybrid-array, which is a replacement for generic-array maintained by the rust-crypto people, and is only dependent on typenum.
The version_check dependency is being dropped.
---
So what's going on is
1. This is a faster implementation of sha1, using all the tricks to have different implementations on different platforms. This resulted in 2 third party dependencies, down to 1 in the next release.
2. This is fitting into a trait ecosystem, and is using tricks to abstract over them without degrading performance. This resulted in 2 third party dependencies, down to 1 in the next release.
3. The count of non-third party dependencies is inflated because rust-crypto has split up its code into multiple crates, and the rust-lang team has split its code up into multiple crates.
4. It's not 40, it's 9.
Is it worth the extra code? I guess that depends on how many times you are hashing things and how much you care about the performance and environmental impact of extra code. Or if you're doing something where you want to be generic over hash functions.
cfg-if - a tiny library making it easier to use different implementations on different platforms. Maintained by the official rust-lang libs team. Also used by the rust std library. No recursive dependencies.
cpufeatures - a tiny library for detecting CPU features maintained by the rust-crypto people, with a recursive dependency only on libc (which is maintained by rust-lang and used by the standard library).
digest - a library providing traits abstracting over different hash functions, maintained by the rust-crypto-people. In turn digest depends blockbuffer (some code to minimize boundchecks) and cryptocommon (more traits for abstracting), both maintained by the rust-crypto people. Both in turn depend on
- typenum, a third party library for compile time math, no dependencies.
- generic-array, a third library for fixed length arrays with lengths computed by typenum, typenum is its only dependency.
- version_check, a third party library for checking what features the rustc being used supports (only block-buffer depends on this one).
sha1-asm - A library with a faster assembly implementation of sha1, maintained by the rust-crypto people. Dependent only on cc (c compiler support), maintained by rust-lang, used in building the rust compiler. cc is itself in turn dependent only on
- shlex, a third party library for splitting things the same way a shell does. So this tree ends in an actual third party dependency, but one you're already indirectly dependent on by simply using the rust compiler at all.
Oh, and in the next release
sha1-asm is being removed
The crypto common dependencies are being replaced with hybrid-array, which is a replacement for generic-array maintained by the rust-crypto people, and is only dependent on typenum.
The version_check dependency is being dropped.
---
So what's going on is
1. This is a faster implementation of sha1, using all the tricks to have different implementations on different platforms. This resulted in 2 third party dependencies, down to 1 in the next release.
2. This is fitting into a trait ecosystem, and is using tricks to abstract over them without degrading performance. This resulted in 2 third party dependencies, down to 1 in the next release.
3. The count of non-third party dependencies is inflated because rust-crypto has split up its code into multiple crates, and the rust-lang team has split its code up into multiple crates.
4. It's not 40, it's 9.
Is it worth the extra code? I guess that depends on how many times you are hashing things and how much you care about the performance and environmental impact of extra code. Or if you're doing something where you want to be generic over hash functions.
> 4. It's not 40, it's 9.
Just like a bunch of others, you understood their comment backwards. Dependents, not dependencies. 40 other things rely on it.
Just like a bunch of others, you understood their comment backwards. Dependents, not dependencies. 40 other things rely on it.
Uh, no, I understand the comment I was replying to forwards.
I simply decided not to point out the source of their misunderstanding since it had already been pointed out by the time I replied, and this comment is already way too long.
I simply decided not to point out the source of their misunderstanding since it had already been pointed out by the time I replied, and this comment is already way too long.
In case I was unclear it has 0 dependencies, but 40 crates in the ecosystem depend on it.
My mistake.
Do you still stand by your decision to give over the `sha1` name?
I don't care that much about the name. It was already clear that even ignoring the name, that the rust-crypto ecosystem became the mainstream choice.
It's not standardized but those calls do not change. The windows calls in particular are guaranteed ABI stable since they are compiled into a lot of binaries. There are definitely issues with ioctl but the changes landing in terminal-size or any of the dependencies that caused all these releases, are entirely unrelated to ioctl/TIOCGWINSZ constants/winsize struct. That code hasn't changed.
In this case the terminal-size crate just calls Rustix tcgetwinsize, which in turn just calls the libc tcgetwinsize. So I suppose you could save yourself a whole bunch of dependencies by just doing the same yourself. The only cost is Windows support.
If this particular API has been stable, or at least reasonably defined for 50 or 25 years is a detail, because the dependency doesn't even pretend to deal with that and the function is unlikely to change or be removed in the near future.
If this particular API has been stable, or at least reasonably defined for 50 or 25 years is a detail, because the dependency doesn't even pretend to deal with that and the function is unlikely to change or be removed in the near future.
> If this particular API has been stable
Well, it hasn't. The tcgetwinsize() was proposed (under this name) in 2017 and was standardized only in 2024. So it's less than a 10 year old API, which is missing from lots of libc implementations, see e.g. [0]. Before its appearance, you had to mess with doing ioctl's and hoping your libc has exposed the TIOCGWINSZ constant (which glibc by default didn't).
[0] https://www.gnu.org/software/gnulib/manual/html_node/tcgetwi...
Well, it hasn't. The tcgetwinsize() was proposed (under this name) in 2017 and was standardized only in 2024. So it's less than a 10 year old API, which is missing from lots of libc implementations, see e.g. [0]. Before its appearance, you had to mess with doing ioctl's and hoping your libc has exposed the TIOCGWINSZ constant (which glibc by default didn't).
[0] https://www.gnu.org/software/gnulib/manual/html_node/tcgetwi...
I had to check the Rustix implementation again, because that would indicate that that terminal-size wouldn't work on a number of operating systems. However Rustix also uses TIOCGWINSZ in it's tcgetwinsize implementation.
Terminals are a good example of something that seems really simple but is a major PITA because of too many different vendors in the early days, and no industry standard emerged. What is the closest thing? VT100? VT102? I mostly write raw to those, but stuff like terminal size and various other features like raw (non-cooked) mode are crappy and require ioctl's and such. Frankly, it sucks.
...but the libraries suck even more! If you don't want to link against ncurses then may God have mercy on your soul.
...but the libraries suck even more! If you don't want to link against ncurses then may God have mercy on your soul.
Previous summer I've toyed with trying to write an "async prompt" a-la Erlang's shell with output scrolling and line-editing (see e.g. [0] for example of what I am talking about), but it is so bloody difficult to do correctly, especially when the input spans several lines and there are some full-width characters on the screen, that I've abandoned it.
[0] https://asciinema.org/a/s2vmkOfj6XtJkQDzeM6g2RbPZ
[0] https://asciinema.org/a/s2vmkOfj6XtJkQDzeM6g2RbPZ
I recently revived a web app I wrote in 2006, my first startup. It was a social media site focused on media sharing. A pretty simple LAMP stack for the time. PHP 5, MySQL 3.2, but it has all of your typical (for the time) social media features. I revived this app because I wanted some hands-on time with new CI/CD tech that I don't get to use at my day job, so I'm working to extremely over-engineer the app's deployment process as a learning project. I could have used Wordpress or some other Hello World app, but this is a lot more fun.
I had written nearly all of the PHP from scratch. I wrote libraries for authentication/authorization, templating, form processing etc. I used one PEAR library for sending email. The frontend was vanilla HTML and there was barely any JavaScript to speak of. We used Flash for media playback. In other words, myself and my small team built nearly all of it ourselves. This was just how you did most things in 2006.
It only took me about an hour to get the 19-year old app up and running. I had to update the old PHP mysql drivers to mysqli, and update the database schema and some queries to work in MySQL 8 (mostly wrapping now-reserved words with backticks and adjusting column defaults which are now more strict). The only thing that didn't work was the Flash.
An hour to revive an app from 2006. Contrast this with my day job, wherein we run scores of Spring Boot apps written in Java 8 that have pages of vulnerabilities from tens of dozens of dependencies, which are not easy to update because updating one library necessitates updating many other libraries, and oh my goodness, the transitive dependencies. It's a nightmare, and because of this we only do the bare minimum of work to update the most critical vulnerabilities. There's no real plan to update everything because it's just too tall of an order.
And the funny thing is, if you compare what this PHP app from 2006 did, which had truly, barely any dependencies, to what these Spring Boot apps do, there is not a lot of difference. At the end of the day, it's all CRUD, with a lot more enterprise dressing and tooling around it.
I had written nearly all of the PHP from scratch. I wrote libraries for authentication/authorization, templating, form processing etc. I used one PEAR library for sending email. The frontend was vanilla HTML and there was barely any JavaScript to speak of. We used Flash for media playback. In other words, myself and my small team built nearly all of it ourselves. This was just how you did most things in 2006.
It only took me about an hour to get the 19-year old app up and running. I had to update the old PHP mysql drivers to mysqli, and update the database schema and some queries to work in MySQL 8 (mostly wrapping now-reserved words with backticks and adjusting column defaults which are now more strict). The only thing that didn't work was the Flash.
An hour to revive an app from 2006. Contrast this with my day job, wherein we run scores of Spring Boot apps written in Java 8 that have pages of vulnerabilities from tens of dozens of dependencies, which are not easy to update because updating one library necessitates updating many other libraries, and oh my goodness, the transitive dependencies. It's a nightmare, and because of this we only do the bare minimum of work to update the most critical vulnerabilities. There's no real plan to update everything because it's just too tall of an order.
And the funny thing is, if you compare what this PHP app from 2006 did, which had truly, barely any dependencies, to what these Spring Boot apps do, there is not a lot of difference. At the end of the day, it's all CRUD, with a lot more enterprise dressing and tooling around it.
Go and the C linux world have sold me on the fat library philosophy. You brought a library to solve a problem in a domain, then add your specific bits. You don't go and bring a dependency for each item in your check list. Yes there may be duplicate effort, but the upgrade path is way easier.
That works until you’re sitting between two fat libraries with overlapping, but incompatible concerns. It’s tradeoffs all the way down.
> That works until you’re sitting between two fat libraries with overlapping, but incompatible concerns. It’s tradeoffs all the way down.
This reads like the dev equivalent of manager speak.
This reads like the dev equivalent of manager speak.
Most of that new CI/CD tech is standard now precisely because of all the complexity added by maintaining third-party dependencies and constant changes in the runtime environment. This isn't a problem for the most part for an old LAMP application deployed by scp.
While I might agree with you on some points, I don't want to spend time reinventing the wheel and introduce bugs into it.
Take for example standard communication message formats like FHIR or HL7. You definitely don't want to implement the whole definitions for the standard, which is already complicated.
Writing Cryptographic functions by yourself is also typically a shot in your foot, has proved in all these years of found critical security issues.
We live in a time where you want to actual solve a business problem, by focusing on the problem and not on how the solution is built properly. With the advent of AI this is even more critical, since all the code feels like stitched together blindly.
Spending time on developing all by yourself might give you a good shot in the long run, but first you need to survive the competition, who maybe has already caught the market, by using fast and throw-away code at the beginning.
Take for example standard communication message formats like FHIR or HL7. You definitely don't want to implement the whole definitions for the standard, which is already complicated.
Writing Cryptographic functions by yourself is also typically a shot in your foot, has proved in all these years of found critical security issues.
We live in a time where you want to actual solve a business problem, by focusing on the problem and not on how the solution is built properly. With the advent of AI this is even more critical, since all the code feels like stitched together blindly.
Spending time on developing all by yourself might give you a good shot in the long run, but first you need to survive the competition, who maybe has already caught the market, by using fast and throw-away code at the beginning.
> my day job, wherein we run scores of Spring Boot apps written in Java 8 that have pages of vulnerabilities from tens of dozens of dependencies, which are not easy to update because updating one library necessitates updating many other libraries, and oh my goodness, the transitive dependencies.
At my job we have a fairly strict static analysis policy and starting in April it is going to get even more strict.
Have you looked at https://docs.openrewrite.org/ to automatically upgrade your dependencies?
I just migrated from Java 8, Spring Boot2 and Swagger to Java 17, Spring Boot 3.3 and OpenApi 3. It was pretty painless.
Now, I still have update some dependencies and transient dependencies but the biggest hurdles were taken care of by the migrations.
At my job we have a fairly strict static analysis policy and starting in April it is going to get even more strict.
Have you looked at https://docs.openrewrite.org/ to automatically upgrade your dependencies?
I just migrated from Java 8, Spring Boot2 and Swagger to Java 17, Spring Boot 3.3 and OpenApi 3. It was pretty painless.
Now, I still have update some dependencies and transient dependencies but the biggest hurdles were taken care of by the migrations.
An important point here is that the transitive dependency issue completely does not exist in Rust. If you upgrade a crate to a version which upgrades its public dependency, i.e. it uses it in its APIs and you need to interact with it to interact with those APIs, then you obviously need to upgrade your copy of the subdependency at the same time. But private transitive dependencies are totally irrelevant unless they link to C libraries. You can have as many semver-incompatible versions of a crate in the same dependency tree as you want, and even depend on multiple versions directly if you need to. No Java-style sweeping upgrades are ever needed, just upgrade the one thing with the vulnerability. (I believe C# has the same feature, though it's a little more baroque about it.)
I’m curious if the OpenRewrite project has any value to you in keeping your Java stuff up to date?
(I’m not affiliated with it; just curious about strategies for upgrading and maintaining apps that use big frameworks.)
(I’m not affiliated with it; just curious about strategies for upgrading and maintaining apps that use big frameworks.)
I agree 100% .
Even though NodeJS is largely responsible for my career, NPM has given me more trauma than my messed up childhood.
Imagine you're a new programmer, you're working on a brand new app to show to all your friends. But you want to add a new dependency, it doesn't like all the other dependencies, cool you say I'll just update them. Next thing you know absolutely nothing works, Babel is screaming at you.
No worries, you'll figure something out. Next thing you know you're staring at open git issues where basic things literally don't work. Expo for example has an open issue where a default new react native project just won't build for Android .
It's like no one cares half the time, and in that case the solution isn't even in the node ecosystem, it's somewhere in the Android ecosystem. It's duct tape all the way down. But this can also inspire confidence if a billion dollar project can ship non-functional templates, then why do I have imposter syndrome when my side projects don't work half the time!
Even though NodeJS is largely responsible for my career, NPM has given me more trauma than my messed up childhood.
Imagine you're a new programmer, you're working on a brand new app to show to all your friends. But you want to add a new dependency, it doesn't like all the other dependencies, cool you say I'll just update them. Next thing you know absolutely nothing works, Babel is screaming at you.
No worries, you'll figure something out. Next thing you know you're staring at open git issues where basic things literally don't work. Expo for example has an open issue where a default new react native project just won't build for Android .
It's like no one cares half the time, and in that case the solution isn't even in the node ecosystem, it's somewhere in the Android ecosystem. It's duct tape all the way down. But this can also inspire confidence if a billion dollar project can ship non-functional templates, then why do I have imposter syndrome when my side projects don't work half the time!
This is a problem for Node, but it isn't a problem for Rust. You don't need any dependencies to 'like' other dependencies. You can have all the versions at the same time and nothing breaks.
Certainly nodejs and npm getting out of hand was a wakeup call for me, about 15 years ago.
I started to view it has a personal failure when I brought in that first dependency, which meant needing all of the packaging and organization just to use that first package.json, rather than just having plain js loaded from a script tag.
I started to view it has a personal failure when I brought in that first dependency, which meant needing all of the packaging and organization just to use that first package.json, rather than just having plain js loaded from a script tag.
I have a side project right now where I tried my hardest to use anything aside from NodeJS.
Even though it's just for my personal consumption, and I doubt more than five or six people will ever see it, I ended up having to use NodeJS and HTML because it's simply the best solution.
Imagine an alternate timeline where Google decided to offer other languages as first class citizens in the browser. Could you imagine Golang or Dart. No compiling down to JS, just running natively.
Even though it's just for my personal consumption, and I doubt more than five or six people will ever see it, I ended up having to use NodeJS and HTML because it's simply the best solution.
Imagine an alternate timeline where Google decided to offer other languages as first class citizens in the browser. Could you imagine Golang or Dart. No compiling down to JS, just running natively.
Mozilla (Eich) wrote JS, and specifically Microsoft wrote the XMLHttpRequest, which allowed Google to exist.
And supposedly WASM is to be that thing, so any language can compile down to it. Hasn't totally worked out I guess. A browser would have to take the leap to be WASM-only, and compile even JS to it.
I'm doing backend work mostly right now. If I start some other project, I'll probably try using htmx and tailwindcss as my two js script tags, and the rest will be controlled from the backend.
I'm doing backend work mostly right now. If I start some other project, I'll probably try using htmx and tailwindcss as my two js script tags, and the rest will be controlled from the backend.
I'm more thinking Chrome could ship with a Golang/Dart VM where apps could run.
WASM is neat though. Tends to ship with mega blobs for now
WASM is neat though. Tends to ship with mega blobs for now
See https://x.com/BrendanEich/status/1260733952160985089. Google, especially with the antitrust trouble they are in, is not going to try for an "ActiveX redux" act here. They definitively renounced trying to ram Dart into the Web standards here:
https://web.archive.org/web/20150328124454/https://news.dart... (yes, they took it down so Wayback to the rescue)
The GC barrier overhead is too damn high!
https://web.archive.org/web/20150328124454/https://news.dart... (yes, they took it down so Wayback to the rescue)
The GC barrier overhead is too damn high!
See https://news.ycombinator.com/item?id=42838110 (downthread). Polyglot VMs are hard to pull off and always have a _primum inter pares_ (C# for .NET, Java for the JVM, JS for the JS+wasm "Web VM").
Thank you for JavaScript!
Your simple to learn programming language is what took me from a broke college dropout to six figures in about 3 years.
It's the first programming language that actually stuck with me, even though one of the senior developers at my first salaried job actively hated it. He was an old Java guy though.
Do you feel that JavaScript is always a capable tool for the job?
At a lot of companies, at least when I started programming 10 or 12 years ago, we'd have a bunch of front-end people who are nifty with JavaScript and just tell them to write NodeJS server code.
JavaScript is still my go to language when I just need to get something done. With Python a close second. C# has it's place for larger projects ( as well as stable employment).
I'm actually a bit excited for WASM. For a good while you had companies shipping plugins to extend the browser, first Adobe with flash, and then unity with their Unity Web plugin. Which lasted about 2 years before they started building native web builds.
Now everything is compiling straight to JS or JS+WASM.
Your simple to learn programming language is what took me from a broke college dropout to six figures in about 3 years.
It's the first programming language that actually stuck with me, even though one of the senior developers at my first salaried job actively hated it. He was an old Java guy though.
Do you feel that JavaScript is always a capable tool for the job?
At a lot of companies, at least when I started programming 10 or 12 years ago, we'd have a bunch of front-end people who are nifty with JavaScript and just tell them to write NodeJS server code.
JavaScript is still my go to language when I just need to get something done. With Python a close second. C# has it's place for larger projects ( as well as stable employment).
I'm actually a bit excited for WASM. For a good while you had companies shipping plugins to extend the browser, first Adobe with flash, and then unity with their Unity Web plugin. Which lasted about 2 years before they started building native web builds.
Now everything is compiling straight to JS or JS+WASM.
Thanks for the kind words.
JS is not the hammer for all nails, but ubiquity means it will be used to pound in screws and bolts more than it ought to be.
I fire up Node but also Python from command line; Wolfram Alpha in browser.
Wasm was the victor in old plugin vs. integrated PNaCl style VM vs. polyglot VM battles. JS VMs (ubiquitous again, and see write barrier cost of poly-GC vs. mono-GC down the page in my linked comment) rather more easily became polyglot via WebAssembly as second syntax low enough to target from other languages' compilers.
Still more to do on the Wasm roadmap. Always more to do...
JS is not the hammer for all nails, but ubiquity means it will be used to pound in screws and bolts more than it ought to be.
I fire up Node but also Python from command line; Wolfram Alpha in browser.
Wasm was the victor in old plugin vs. integrated PNaCl style VM vs. polyglot VM battles. JS VMs (ubiquitous again, and see write barrier cost of poly-GC vs. mono-GC down the page in my linked comment) rather more easily became polyglot via WebAssembly as second syntax low enough to target from other languages' compilers.
Still more to do on the Wasm roadmap. Always more to do...
Hypothetically if Microsoft threw enough money at it, could they get C# to work natively in the browser ?
Or does WASM make this unnecessary?
I actually really like Flutter Web, Google has done an amazing job with it. I built a small game for a friend, but it's not as customizable as an actual website coded in JavaScript.
It appears they've taken a hybrid approach where you get both a JavaScript and a Wasm build.
https://docs.flutter.dev/platform-integration/web/wasm
I feel like Unity in particular has really pushed the limits of what you can do in a browser. For example, I have a simple 3D game that performs just as well as the android build.
As far as making a small game, it's very easy to get someone to open a link. The browser sandboxes every and protects you.
It's near impossible to convince someone to download a binary, with good reason.
One last question. Do you consider yourself to be a Computer Science pioneer ? Your literally responsible for the foundations of the modern web!
Or does WASM make this unnecessary?
I actually really like Flutter Web, Google has done an amazing job with it. I built a small game for a friend, but it's not as customizable as an actual website coded in JavaScript.
It appears they've taken a hybrid approach where you get both a JavaScript and a Wasm build.
https://docs.flutter.dev/platform-integration/web/wasm
I feel like Unity in particular has really pushed the limits of what you can do in a browser. For example, I have a simple 3D game that performs just as well as the android build.
As far as making a small game, it's very easy to get someone to open a link. The browser sandboxes every and protects you.
It's near impossible to convince someone to download a binary, with good reason.
One last question. Do you consider yourself to be a Computer Science pioneer ? Your literally responsible for the foundations of the modern web!
It's unlikely to be a money problem (nor Microsoft cares that much about .NET to do so). WASM also makes it pretty much irrelevant. Current Mono-WASM solution works decently enough but is far from perfect. Hopefully NativeAOT-LLVM project gets released in the next couple of versions as it improves the performance and bundle size massively.
Right, wasm is the low energy path. IMO you should think about this problem-space of the Web as an evolutionary system, with costs weighting all the paths through a graph from current state to future state.
MS throwing money at C# native VM (the CLR? Silverlight's DLR? lol) as second VM in browsers, with separate GCs bridged, incurring the write barrier overhead that I cited already? In all browsers, not just the ones MS can bribe to do the considerable work? No.
C# compiled to WebAssembly? One GC, all good, perhaps some warm evolution is on order, but it is an incremental cost.
MS throwing money at C# native VM (the CLR? Silverlight's DLR? lol) as second VM in browsers, with separate GCs bridged, incurring the write barrier overhead that I cited already? In all browsers, not just the ones MS can bribe to do the considerable work? No.
C# compiled to WebAssembly? One GC, all good, perhaps some warm evolution is on order, but it is an incremental cost.
Friend shared https://maxima-on-wasm.pages.dev/, very cool to see old Lisps live again thanks to Wasm.
"Dedicated to the memory of William Schelter."
"Dedicated to the memory of William Schelter."
This was something that surprised me about the Rust ecosystem, coming from Go. Even a mature Go project (e.g. some business' production web backend) may only have 10-20 dependencies including the transitive ones.
As noted in this post, even a small Rust project will likely have many more than that, and it's virtually guaranteed if you're doing async stuff.
No idea how much of it is cultural versus based on the language features, e.g. in Go interfaces are implicitly satisfied, no need to import anything to say you implement it.
As noted in this post, even a small Rust project will likely have many more than that, and it's virtually guaranteed if you're doing async stuff.
No idea how much of it is cultural versus based on the language features, e.g. in Go interfaces are implicitly satisfied, no need to import anything to say you implement it.
One big reason is because Go has a very nice complete standard library, and Rust really does not.
Things you can find in go’s built in to the language or in standard libraries that need a dependency in Rust:
- green threads
- channels
- regular expressions
- http client
- http server
- time
- command line flags
- a logger
- read and write animated GIFs
I don’t love the Go language, but it’s the leader for tooling and standard library, definitely the best I’ve used.
Things you can find in go’s built in to the language or in standard libraries that need a dependency in Rust:
- green threads
- channels
- regular expressions
- http client
- http server
- time
- command line flags
- a logger
- read and write animated GIFs
I don’t love the Go language, but it’s the leader for tooling and standard library, definitely the best I’ve used.
I hate that. I don't want dependencies for serialization or logging. But you do and now you have to choose which of the dozen logging crates you need.
As a beginner this is horrible, because everybody knows serde, but I have to learn that serde is the defacto, and that is not easy because when coming from other languages, it sounds like the second best choice. And that is with most rust crates.
As a beginner this is horrible, because everybody knows serde, but I have to learn that serde is the defacto, and that is not easy because when coming from other languages, it sounds like the second best choice. And that is with most rust crates.
This is why things like https://blessed.rs exist. Although since they're unofficial, their discoverability is also likely a problem.
Thanks for that. Didn’t know about the site
Which may or may not be fine in a Go binary that runs on a modern desktop CPU, but what if your code is supposed to run on say an ESP32-C3 with a whopping 160 MHz RISC-V core, 400 KB of RAM and maybe 2 MB of XIP flash storage?
You could of course argue that that's why no-std exists in Rust, or that your compiler might optimize out the animated GIF routines, but personally, I'd argue that in this context, it is bloat, that - while it could occasionally be useful - it could just as easily be a third party library.
You could of course argue that that's why no-std exists in Rust, or that your compiler might optimize out the animated GIF routines, but personally, I'd argue that in this context, it is bloat, that - while it could occasionally be useful - it could just as easily be a third party library.
It’s the same as in C, Rust, or any other programming language I’ve ever used. If you don’t use a library, it doesn’t end up linked in your executable. Don’t want to animate GIFs on your microcontroller, then you don’t write `import “image/gif”` in your source file.
For a microcontroller sized runtime, there’s https://tinygo.org/
I think the lack of strong standard library actually leads to more bloat in your program in the long run. Bloat is needing to deal with an ecosystem that has 4 competing packages for time, ending up with all 4 installed because other libraries you need didn’t agree, and then you need ancillary compatibility packages for converting between the different time packages.
For a microcontroller sized runtime, there’s https://tinygo.org/
I think the lack of strong standard library actually leads to more bloat in your program in the long run. Bloat is needing to deal with an ecosystem that has 4 competing packages for time, ending up with all 4 installed because other libraries you need didn’t agree, and then you need ancillary compatibility packages for converting between the different time packages.
> It’s the same as in C, Rust, or any other programming language I’ve ever used. If you don’t use a library, it doesn’t end up linked in your executable.
I don't think that's true. If the standard library is pre-compiled, and it doesn't use `-ffunction-sections` etc. then I'm pretty sure you'll just get the whole thing.
There is experimental support for building Rust's standard library from source, but by default it is pre-compiled.
I don't think that's true. If the standard library is pre-compiled, and it doesn't use `-ffunction-sections` etc. then I'm pretty sure you'll just get the whole thing.
There is experimental support for building Rust's standard library from source, but by default it is pre-compiled.
Does anyone use the full standard library for embedded targets? I've not seen it done in C, java has a special embedded edition, python has micro-python, rust seems to usually use no-std, but I might be wrong there.
It seems like a bad reason to constrain the regular standard library
It seems like a bad reason to constrain the regular standard library
I have in the past, but most people don't.
E.g. for esp32 see https://docs.esp-rs.org/book/overview/using-the-standard-lib...
E.g. for esp32 see https://docs.esp-rs.org/book/overview/using-the-standard-lib...
For Rust and Go in particular, the difference is in the standard library. The Rust stdlib is (intentionally) small.
Agreed, and the small stdlib is one of the main reasons for this problem. I understand the reasoning why it's small, but I wish more people would acknowledge the (IMHO at least) rather large downside this brings, rather than just painting it as a strictly positive thing. The pain of dealing with a huge tree of dependencies, all of different qualities and moving at different trajectories, is very real. I've spent a large part of the last couple of days fighting exactly this in a Rust codebase, which is hugely frustrating.
What is the reason to keep it small? Genuinely interested, I actually don't understand.
Embedded systems maybe?
Embedded systems maybe?
(I've been on libs-api, and libs before that, for 10 years now.)
API Stability. When the standard library APIs were initially designed, "the Python standard library is where packages go to die" was very much on our minds. We specifically saw ourselves as enabled to have a small standard library because of tooling like Cargo.
There are no plans for Rust 2.0. So any change we merge into std is, effectively, something we have to live with for approximately forever. (With some pedantic exceptions over edition boundaries that don't change my overall point.)
Nuance is nearly dead on the Internet, but I'll say that I think designing robust and lasting APIs is a lot harder in Rust than it is in Go. Rust has a lot more expressiveness (which I do not cite as an unmitigated good), and the culture is more heavily focused on zero-overhead abstractions (which I similarly do not cite as an unmitigated good). That means the "right" API can be very difficult to find without evolution via breaking changes. But the standard library cannot, generally speaking, make breaking changes. Crates can.
I would suggest not reading the OP as a black-and-white position. But rather, a plea to change how we balance the pros and cons of dependencies in the Rust ecosystem.
API Stability. When the standard library APIs were initially designed, "the Python standard library is where packages go to die" was very much on our minds. We specifically saw ourselves as enabled to have a small standard library because of tooling like Cargo.
There are no plans for Rust 2.0. So any change we merge into std is, effectively, something we have to live with for approximately forever. (With some pedantic exceptions over edition boundaries that don't change my overall point.)
Nuance is nearly dead on the Internet, but I'll say that I think designing robust and lasting APIs is a lot harder in Rust than it is in Go. Rust has a lot more expressiveness (which I do not cite as an unmitigated good), and the culture is more heavily focused on zero-overhead abstractions (which I similarly do not cite as an unmitigated good). That means the "right" API can be very difficult to find without evolution via breaking changes. But the standard library cannot, generally speaking, make breaking changes. Crates can.
I would suggest not reading the OP as a black-and-white position. But rather, a plea to change how we balance the pros and cons of dependencies in the Rust ecosystem.
I can understand not wanting to add SMTP or CGI to the stdlib. But a lot of common POSIX functionality (which is sometimes a single syscall away) is missing too.
A lot of common POSIX functionality is not missing though. I was able to write ripgrep, for example, by almost entirely sticking to the standard library. (I think the only place I reach out to `libc` directly is to get the hostname of the current system for rendering hyperlinks in your terminal.)
We also came at the standard library with a cross platform mentality that included non-POSIX platforms like Windows. You want to be careful not to design APIs that are too specific to POSIX. So it falls under the same reasoning I explained above: everything that goes into std is treated as if it will be there forever. So when we add things to std, we absolutely consider whether the risk of us getting the API wrong is outweighed by the benefit of the API being in std in the first place. And we absolutely factor "the ease of using crates via Cargo" into this calculus.
We also came at the standard library with a cross platform mentality that included non-POSIX platforms like Windows. You want to be careful not to design APIs that are too specific to POSIX. So it falls under the same reasoning I explained above: everything that goes into std is treated as if it will be there forever. So when we add things to std, we absolutely consider whether the risk of us getting the API wrong is outweighed by the benefit of the API being in std in the first place. And we absolutely factor "the ease of using crates via Cargo" into this calculus.
I peeked at the code for gethostname in ripgrep, and it's nice and straightforward.
Much like op said here; we have a culture of "don't write unsafe code under any circumstance", and we then pull in a dependency tree for a single function that's relatively safe to contain. It solves the problem quickly, but at a higher price.
BTW, thanks for ripgrep. I don't actually use it, but I've read through different portions of the code over recent months and it's some very clean and easy to understand code. Definitely a good influence.
Much like op said here; we have a culture of "don't write unsafe code under any circumstance", and we then pull in a dependency tree for a single function that's relatively safe to contain. It solves the problem quickly, but at a higher price.
BTW, thanks for ripgrep. I don't actually use it, but I've read through different portions of the code over recent months and it's some very clean and easy to understand code. Definitely a good influence.
I don't think you should treat unsafe code as that level of toxic. It's necessary when interfacing with with system APIs. The important part is that you try to have safe wrappers around the unsafe calls and that you document why the way you're using them is safe.
I've been using ripgrep for years! Thanks a lot for that!
In addition to the points burntsushi gave in the sibling comment, I'd also add that keeping the standard library small and putting other well-scoped stuff like tegex, rand, etc. in dependencies also can reduce the burden of releases a lot. If some a bug gets found in a library that's not std, a new release can get pushed out pretty quickly. If a bug gets found in std, an entire new toolchain version needs to be published. That's not to say that this wouldn't be done for critical bugs, but when Rust already has releases on a six-week cadence, it's not crazy to try to reduce the need for additional releases on top of that.
This probably isn't as important as the stability concerns, but I think it still helps tilt the argument in favor of a small std at least a little.
This probably isn't as important as the stability concerns, but I think it still helps tilt the argument in favor of a small std at least a little.
One risk with a bigger standard library is that you'll do an imperfect job of it, then you'll be stuck maintaining it forever for compatibility reasons.
For example, Java developers can choose to represent time with Unix milliseconds, java.util.Date, java.util.Calendar, Joda-Time or java.time.Instant
For example, Java developers can choose to represent time with Unix milliseconds, java.util.Date, java.util.Calendar, Joda-Time or java.time.Instant
It’s really just Date and Instant. Joda-Time isn’t part of the standard library. And if you’re listing Calendar, you might as well also list ZonedDateTime, OffsetDateTime, and LocalDateTime, not to mention stuff like java.sql.Date.
In reality, there’s just one old API and one new API, similar to the old collection classes (HashTable, Vector, etc.) and the newer JCF ones.
In reality, there’s just one old API and one new API, similar to the old collection classes (HashTable, Vector, etc.) and the newer JCF ones.
The three main reasons I see being given are:
- backward compatbility: a big std lib increases the risk of incompatible changes and the cost of long term support
- pushing developers to be mindful of minimal systems: a sort of unrelated example is how a lot of node library use the 'fs' module just because it is there creating a huge pain point for browser bundling. If the stdlib did not have a fs module this would happen a lot less
- a desire to let the community work it out and decide the best API/implementations before blessing a specific library as Standard.
In my opinion a dynamic set of curated library with significantly shorted backward compatibility guarantees is the best of both worlds.
- backward compatbility: a big std lib increases the risk of incompatible changes and the cost of long term support
- pushing developers to be mindful of minimal systems: a sort of unrelated example is how a lot of node library use the 'fs' module just because it is there creating a huge pain point for browser bundling. If the stdlib did not have a fs module this would happen a lot less
- a desire to let the community work it out and decide the best API/implementations before blessing a specific library as Standard.
In my opinion a dynamic set of curated library with significantly shorted backward compatibility guarantees is the best of both worlds.
Other reasons also include:
- less burden on the stdlib maintainers (which are already overworked!)
- faster iteration on those libraries, since you don't need to wait a new release of the compiler to get updates for those libraries (which would take at least 12-16 weeks depending on when the PR is merged)
- less burden on the stdlib maintainers (which are already overworked!)
- faster iteration on those libraries, since you don't need to wait a new release of the compiler to get updates for those libraries (which would take at least 12-16 weeks depending on when the PR is merged)
AFAIK: Rust compiles to machine code. Even if the stdlib would be 600 mb, If you have 3 lines of Code your programm would be microscopically small.
It might be a bad choice on rust's part.
IMO they should over time fold whatever ends up being the de-facto choice for things into the standard library. Otherwise this will forever be a barrier to entry, and a constant churn as ever new fashionable libraries to do the same basic thing pops up.
You don't need a dozen regex libraries, you just need one that's stable, widely used and likely to remain so.
IMO they should over time fold whatever ends up being the de-facto choice for things into the standard library. Otherwise this will forever be a barrier to entry, and a constant churn as ever new fashionable libraries to do the same basic thing pops up.
You don't need a dozen regex libraries, you just need one that's stable, widely used and likely to remain so.
> You don't need a dozen regex libraries, you just need one that's stable, widely used and likely to remain so.
That is the case today. Virtually everyone uses `regex`.
There are others, like `fancy-regex`. But those would still exist even if `regex` was in std. But then actually it would suck, because then `fancy-regex` can't share dependencies with `regex`, which it does today. And because of that, you get a much smoother migration experience where you know that if your regexes are valid with `regex`, they'll work the same way in `fancy-regex`.
A better example might be datetime handling, of which there are now 3 general purpose libraries one can reasonably choose. But it would have been an unmitigated disaster if we (I am on libs-api) had just added the first datetime library to std that arose in the ecosystem.
That is the case today. Virtually everyone uses `regex`.
There are others, like `fancy-regex`. But those would still exist even if `regex` was in std. But then actually it would suck, because then `fancy-regex` can't share dependencies with `regex`, which it does today. And because of that, you get a much smoother migration experience where you know that if your regexes are valid with `regex`, they'll work the same way in `fancy-regex`.
A better example might be datetime handling, of which there are now 3 general purpose libraries one can reasonably choose. But it would have been an unmitigated disaster if we (I am on libs-api) had just added the first datetime library to std that arose in the ecosystem.
Agreed.
> and a constant churn as ever new fashionable libraries
Isn't that the situation in Javascript? I don't work in Javascript but to me it feels like people migrate to a new cool framework every 2 months.
> and a constant churn as ever new fashionable libraries
Isn't that the situation in Javascript? I don't work in Javascript but to me it feels like people migrate to a new cool framework every 2 months.
Less so than it once was, but more so than other languages/ecosystems.
To be fair though, I'd argue the environment Javascript lives in also changes faster than any other.
To be fair though, I'd argue the environment Javascript lives in also changes faster than any other.
I would expect crates like `stdlib-terminal` and `stdlib-web-api` in that case.
Honestly, something feels off with Rust trying to advertise itself for embedded: no stdlib and encourage stack allocation, but then married to Clang (which doesn't have a good embedded target support) and have panic in the language.
Building a C++ replacement for a browser engine rewrite and building a C replacement for embedded have different and often conflicting design constraints. It seems like Rust is a C++ replacement with extra unnecessary constraints of a C replacement.
Honestly, something feels off with Rust trying to advertise itself for embedded: no stdlib and encourage stack allocation, but then married to Clang (which doesn't have a good embedded target support) and have panic in the language.
Building a C++ replacement for a browser engine rewrite and building a C replacement for embedded have different and often conflicting design constraints. It seems like Rust is a C++ replacement with extra unnecessary constraints of a C replacement.
I often wonder about this: obviously Rust is fashionable, and many people push to use it everywhere. But in a ton of situations, there are modern memory-safe languages (Go, Swift, Kotlin, Scala, Java, ...) that are better suited.
To me Rust is good when you need the performance (e.g. computer vision) and when you don't want a garbage collector (e.g. embedded). So really, a replacement for C/C++. Even though it takes time because C/C++ have a ton of libraries that may not have been ported to Rust (yet).
Anyway, I guess my point is that Rust should focus on the problem it solves. Sometimes I feel like people try to make it sound like a competitor to those other memory-safe languages and... even though I like Rust as a language, it's much easier to write Go, Swift or Kotlin than Rust (IMHO).
To me Rust is good when you need the performance (e.g. computer vision) and when you don't want a garbage collector (e.g. embedded). So really, a replacement for C/C++. Even though it takes time because C/C++ have a ton of libraries that may not have been ported to Rust (yet).
Anyway, I guess my point is that Rust should focus on the problem it solves. Sometimes I feel like people try to make it sound like a competitor to those other memory-safe languages and... even though I like Rust as a language, it's much easier to write Go, Swift or Kotlin than Rust (IMHO).
> it's much easier to write Go, Swift or Kotlin than Rust (IMHO).
While I respect your opinion, I must say I find it much more difficult to get work done in golang than rust. Mainly due to stringly typed errors and nonexistent documentation culture. Trying to figure out what error states my program might get itself into is essentially a futile exercise in golang whereas in rust it's generally annotated right there in the source code by lsp.
While I respect your opinion, I must say I find it much more difficult to get work done in golang than rust. Mainly due to stringly typed errors and nonexistent documentation culture. Trying to figure out what error states my program might get itself into is essentially a futile exercise in golang whereas in rust it's generally annotated right there in the source code by lsp.
Go's vast standard library helps a lot with keeping dependency numbers down
Exactly this. You want logging in Rust? You will need at least `log` and another logger crate, for example `env_logger`, maybe the `dotenvy` crate to read `.env` files automatically, you already have 3 direct dependencies + all the transitive ones.
In Go: https://pkg.go.dev/log
In Go: https://pkg.go.dev/log
The built-in "logging" in Go is barely more than a fancy Println. For example, where are the levels, like DEBUG and WARN?
The slog (structured logging) package of the stdlib has those, I believe. As well as support for custom levels.
Love the thesis statement. There is a lot of hidden cost in allowing abstractions from other libraries to be exposed over your own. I'm not here to say that should never be done but the future costs really need to be balanced at the decision point.
If the encapsulating package churns over its design, or changes it's goals it creates instability. Functionality is lost or broken apart when previously it was previously the simplest form of guarantee in software engineering in existence. It also deters niche but expert owners who aren't career OSS contributors from taking part in an ecosystem."I made a clean way to do X!" being followed up by weeks of discussion, negotiation, politics so that "X fits under Y because maybe people like Z" is inefficient and wasteful of everyones time.
If there's one thing I've learned in my life it's that the simplest things survive the longest. Miniliths, and monoliths should be celebrated way more often. Rust isn't alone in this by the way, I've seen this across languages. I've often seen OSS communities, drive hard for atomistic size packages, and I often wonder if it's mostly for flag planting and ownership transfer purposes than it is to benefit the community that actually uses these things.
If the encapsulating package churns over its design, or changes it's goals it creates instability. Functionality is lost or broken apart when previously it was previously the simplest form of guarantee in software engineering in existence. It also deters niche but expert owners who aren't career OSS contributors from taking part in an ecosystem."I made a clean way to do X!" being followed up by weeks of discussion, negotiation, politics so that "X fits under Y because maybe people like Z" is inefficient and wasteful of everyones time.
If there's one thing I've learned in my life it's that the simplest things survive the longest. Miniliths, and monoliths should be celebrated way more often. Rust isn't alone in this by the way, I've seen this across languages. I've often seen OSS communities, drive hard for atomistic size packages, and I often wonder if it's mostly for flag planting and ownership transfer purposes than it is to benefit the community that actually uses these things.
> It's 2025 and it's faster for me to have ChatGPT or Cursor whip up a dependency free implementation of these common functions
I sort of stumbled upon this myself and am coming around to this viewpoint. Especially after dependency hell of a big react app.
And there's also the saas/3rd party services dependencies to consider. Many of them are common patterns and already solved problems that LLMs can clone quickly.
I sort of stumbled upon this myself and am coming around to this viewpoint. Especially after dependency hell of a big react app.
And there's also the saas/3rd party services dependencies to consider. Many of them are common patterns and already solved problems that LLMs can clone quickly.
I’ve definitely become much more likely to start with small internal utility functions implemented by AI before adding a library than I would have been in the past—it’s quite effective for contained problems, I can have the AI write much more complete/robust implementations than I would myself, and if I do eventually decide to add a library I have a natural encapsulation: I can change the implementation of my own functions to use the library without necessarily having to touch everywhere it’s being used. Makes it easy to test a couple of different libraries as well, when the time comes.
This is deliciously ironic for me. For those who aren't aware, Armin is the original author of Flask, the Python web framework. Around the same time, there was a very similar library called Bottle. They were almkst identical in functionality, but while Flask became very popular, I always preferred to stick with Bottle because it was a single file with no dependencies, which meant it was very easy to just copy into your project.
It also made it very easy to hack around with, and I got to the point where I understood the entire thing. I did couple it with Gevent for the server and websockets, but I was able to put together some heavy-lifting projects that way.
I still feel a strong impulse to use it for small web projects. Sadly, it didn't keep up with a lot of the more modern practices that Python has introduced over the years, so it does feel a bit dated now.
It also made it very easy to hack around with, and I got to the point where I understood the entire thing. I did couple it with Gevent for the server and websockets, but I was able to put together some heavy-lifting projects that way.
I still feel a strong impulse to use it for small web projects. Sadly, it didn't keep up with a lot of the more modern practices that Python has introduced over the years, so it does feel a bit dated now.
You need capable engineering for building it yourself. If you only got engineers, who only ever reached for libraries, in ecosystems like NPM or PyPI, you will find them hard-pressed to develop solutions for many things themselves, especially so, if they are supposed to be solutions, that stand the test of time, and have the flexibility they need. It takes a lot of practice to "avoid programming yourself into a corner".
Another thing I noticed is, that one can often easily do better than existing libraries. In one project I implemented a parser for a markdown variant, that has some metadata at the top of the file. Of course I wrote a little grammar, not even a screen of code, and just like that, I had a parser. But I did not expect the badness of the frontend library parsing the same file. That one broke, when you had hyphens in metadata identifiers. At first I was confused, why it could not deal with that. Then it turned out, that it directly used the metadata identifiers as object member names ... Instead of using a simple JSON object, they had knowingly or unknowingly chosen to artificially limit the choice of names and to break things, when there are hyphens like in "something-something". In the end my parser was abandoned, people arguing, that they would have to "maintain" it. Well, it just worked and could easily be adapted for grammar changes. There was nothing difficult to understand about it either, if you had just a little knowledge about parsers. Sounds incredible, but apparently no one except me on the team had written a parser by using a parser generator library before.
And like that, there are many other examples.
Another thing I noticed is, that one can often easily do better than existing libraries. In one project I implemented a parser for a markdown variant, that has some metadata at the top of the file. Of course I wrote a little grammar, not even a screen of code, and just like that, I had a parser. But I did not expect the badness of the frontend library parsing the same file. That one broke, when you had hyphens in metadata identifiers. At first I was confused, why it could not deal with that. Then it turned out, that it directly used the metadata identifiers as object member names ... Instead of using a simple JSON object, they had knowingly or unknowingly chosen to artificially limit the choice of names and to break things, when there are hyphens like in "something-something". In the end my parser was abandoned, people arguing, that they would have to "maintain" it. Well, it just worked and could easily be adapted for grammar changes. There was nothing difficult to understand about it either, if you had just a little knowledge about parsers. Sounds incredible, but apparently no one except me on the team had written a parser by using a parser generator library before.
And like that, there are many other examples.
> But when you end up using one function, but you compile hundreds, some alarm bell should go off.
About a year ago I ran a project to update 3rd party dependencies.
One of the dependencies was a rich math library, full of all kinds of mathematical functions.
I did a little bit of digging, and we were only using one single method, to find the median of a list.
I pointed the engineer to the Wikipedia page and told him to eliminate the dependency and write a single method to perform the mathematical operation.
---
But, IMO, the real issue isn't using 3rd party dependencies: It's that we need a concept of pulling in a narrow slice of a library. If I just need a small part of a giant library, why do I have to pull in the whole thing? I think I've heard someone propose "microframeworks" as a way to do this.
About a year ago I ran a project to update 3rd party dependencies.
One of the dependencies was a rich math library, full of all kinds of mathematical functions.
I did a little bit of digging, and we were only using one single method, to find the median of a list.
I pointed the engineer to the Wikipedia page and told him to eliminate the dependency and write a single method to perform the mathematical operation.
---
But, IMO, the real issue isn't using 3rd party dependencies: It's that we need a concept of pulling in a narrow slice of a library. If I just need a small part of a giant library, why do I have to pull in the whole thing? I think I've heard someone propose "microframeworks" as a way to do this.
In Rust, a library can define "features" that can be conditionally enabled or disabled when depending on it, which give a a built-in way to customize how much of the library is actually included. Tokio is a great example of this; people might be surprised to learn that the total number of direct dependencies that are required by tokio is only two[1]; everything else is optional!
Unfortunately, it doesn't seem like people are super diligent about looking into the default feature set that's used by their dependencies and proactively trimming that down. It doesn't help that the syntax for pulling in extra features is less verbose than removing optional but default ones (which requires both specifying "no-default-features" and then manually adding every one of the default features that you do still want back to the list of ones you pull in), and it _really_ doesn't help that the only way for libraries to expose the ability to prune unneeded features from their own dependencies to the users who inherit them is by manually making their own feature that maps to the features of every single one of their own dependencies. For example, if you're writing a library with five dependencies, and every one of them has one required dependency and four optional ones, giving the users of your library full control over what transitive features they pull in would mean making 20 features in your own library mapping to each of those transitive features, and that's not even counting the ones that you'd want to make for your own code in order to be a good citizen and not force downstream users to include all of your own code.
More and more I'm coming to the opinion that the ergonomics around features being so much worse for trying to cut down on the bloat is actually the catalyst for a lot of the issues around compile times in Rust. It doesn't seem to be super widely discussed when things like this come up though, so maybe it's time that I try to write a blog post or something with my strong feelings on this so at least I'll have something to point to assuming the status quo continues indefinitely.
[1]: https://github.com/tokio-rs/tokio/blob/ee19b0ed7371b069112b9...
Unfortunately, it doesn't seem like people are super diligent about looking into the default feature set that's used by their dependencies and proactively trimming that down. It doesn't help that the syntax for pulling in extra features is less verbose than removing optional but default ones (which requires both specifying "no-default-features" and then manually adding every one of the default features that you do still want back to the list of ones you pull in), and it _really_ doesn't help that the only way for libraries to expose the ability to prune unneeded features from their own dependencies to the users who inherit them is by manually making their own feature that maps to the features of every single one of their own dependencies. For example, if you're writing a library with five dependencies, and every one of them has one required dependency and four optional ones, giving the users of your library full control over what transitive features they pull in would mean making 20 features in your own library mapping to each of those transitive features, and that's not even counting the ones that you'd want to make for your own code in order to be a good citizen and not force downstream users to include all of your own code.
More and more I'm coming to the opinion that the ergonomics around features being so much worse for trying to cut down on the bloat is actually the catalyst for a lot of the issues around compile times in Rust. It doesn't seem to be super widely discussed when things like this come up though, so maybe it's time that I try to write a blog post or something with my strong feelings on this so at least I'll have something to point to assuming the status quo continues indefinitely.
[1]: https://github.com/tokio-rs/tokio/blob/ee19b0ed7371b069112b9...
That's how we get things like leftpad in the JS ecosystem.
On one side, I think if we had a good system of trust, that's not a problem.
And part of me likes the idea of something like Shadcn - you like a component? Copy it into your library. However, if there ends up being a vulnerability, you have no idea if you are affected.
For some code, that's not a problem. For other code, we truly depend on having as many eyes as possible on it.
On one side, I think if we had a good system of trust, that's not a problem.
And part of me likes the idea of something like Shadcn - you like a component? Copy it into your library. However, if there ends up being a vulnerability, you have no idea if you are affected.
For some code, that's not a problem. For other code, we truly depend on having as many eyes as possible on it.
Wow. For those who don’t know, here’s a pseudocode implementation:
Managed to resist calling is_odd there!
Median(list) {
let len = length(list)
if len % 2 == 0 {
let x = floor(len/2)
return (list[x] + list[x+1]) / 2
}
return list[len/2]
}
Managed to resist calling is_odd there!
In most cases I'd probably use nearly this. I note that it contains a bug due to integer overflow if naively translated to most languages.
But if I have a big enough list that I care about space usage (I don't want to make a copy that I sort and then throw away), or speed (I care about O(n) vs O(n log(n))) I'd be looking for a library before implementing my own.
Here are the relevant algorithms if you really want to implement your own fast median code though: https://cs.stackexchange.com/questions/1914/find-median-of-u...
But if I have a big enough list that I care about space usage (I don't want to make a copy that I sort and then throw away), or speed (I care about O(n) vs O(n log(n))) I'd be looking for a library before implementing my own.
Here are the relevant algorithms if you really want to implement your own fast median code though: https://cs.stackexchange.com/questions/1914/find-median-of-u...
I use that math textbook algorithm in production to produce a median from a list which has a bounded size and is already sorted by the db, though that bound could technically grow to INT_MAX if someone managed to make that many requests in five minutes. Not very likely. :-)
> and is already sorted by the db,
Right, if it's already sorted just taking the midpoint is the obviously correct algorithm (and O(1) time/space). It's only in the unsorted cases where with giant lists you should start thinking about alternatives.
If I'm working with gigabytes of photon counts (each element representing the number photons detected in a time interval) I don't want to sort my gigabyte long list before getting the median - sorting would destroy the very important structure of the data so I'd just have to throw away the copy afterwards. This is referencing some code I worked on a long time ago. I'm not sure I had to calculate a median specifically, but similar enough statistics. It's a simple function, but not a one size fits all algorithm.
Right, if it's already sorted just taking the midpoint is the obviously correct algorithm (and O(1) time/space). It's only in the unsorted cases where with giant lists you should start thinking about alternatives.
If I'm working with gigabytes of photon counts (each element representing the number photons detected in a time interval) I don't want to sort my gigabyte long list before getting the median - sorting would destroy the very important structure of the data so I'd just have to throw away the copy afterwards. This is referencing some code I worked on a long time ago. I'm not sure I had to calculate a median specifically, but similar enough statistics. It's a simple function, but not a one size fits all algorithm.
A metric that I would like to focus in is dependency depth. We had this with the OSI model way back, but at this point it seems that anything beyond layer 7 just gets bucketed into 8+.
We need to know if a dependency is level 1 or level 2 or level 3 or 45. And we need to know what the deepest dependency on our project is. I might be naive, but I think we should strive to reduce the depth of the dependency graph and maybe aim for like 4 or 5 layers deep for a web app, tops.
We need to know if a dependency is level 1 or level 2 or level 3 or 45. And we need to know what the deepest dependency on our project is. I might be naive, but I think we should strive to reduce the depth of the dependency graph and maybe aim for like 4 or 5 layers deep for a web app, tops.
I've often thought the same. I would love a depedency manager that not only surfaced this information, but required you to declare upfront what level your library is.
I think it would reign in the bloat.
I think it would reign in the bloat.
Worse, sometimes the upstream is complex enough that you don't want to do it yourself - then the upstream quits maintaining their project. I have in my company some open source projects that we still use that haven't been touched upstream since 2012, but either there is no replacement or the replacement is so different it isn't worth the effort to upgrade. Fortunately none of these are projects where I worry about security issues, I'm just annoyed by the lack of support, but if they faced the network I'd be concerned (and we do have security people who would force a change)
Software that hasn’t been touched in ten years and still does the job is about as ideal as a dependency can be.
I tend to agree, but it may have downsides to: it may do the job and have serious security issues. If you don't know what it does, no reason to know about the security issues.
In theory yes.
Although a ten year old dependency is written in Python, it's likely not going to work any more due to changes in the build system, stdlib, etc.
Although a ten year old dependency is written in Python, it's likely not going to work any more due to changes in the build system, stdlib, etc.
Maybe it's even actively maintained!
Or they bait-and-switch freedom.
So they start off with an open source Free solution, and then later switch to a paid for model, and abandon the Free version. This is particularly painful when it's a part of your system that's key to security.
You're left between wondering if you should just pay the ransom or switching to a different solution entirely, or gamble and leaving it on an old unpatched version.
( Looking at you, IdentityServer )
Either way you regret ever going with them.
So they start off with an open source Free solution, and then later switch to a paid for model, and abandon the Free version. This is particularly painful when it's a part of your system that's key to security.
You're left between wondering if you should just pay the ransom or switching to a different solution entirely, or gamble and leaving it on an old unpatched version.
( Looking at you, IdentityServer )
Either way you regret ever going with them.
Or fork it and maintain yourself. Wanted to write it yourself? Now is the perfect opportunity.
> I'm just annoyed by the lack of support
You know you can hire someone to support them right?
You know you can hire someone to support them right?
The issue is, imo, rust is a big pain to write.
It’s really nice when someone else has created all the abstractions for you. It’s fairly easy and fun to discover features of crates thanks to the type system.
But to actually build out those low level abstractions yourself is cumbersome and not very fun. It’s easy to type system yourself in a corner.
You even lose the whole big selling point of rust when you use `unsafe`.
There was an article a while ago comparing zig and rust and it had a good quote that went something like
“Rust is for programming in the large, and zig is for programming in the small”
And I think what you’re looking for is programming in the small.
Small, focused, programs with minimal dependencies.
It’s really nice when someone else has created all the abstractions for you. It’s fairly easy and fun to discover features of crates thanks to the type system.
But to actually build out those low level abstractions yourself is cumbersome and not very fun. It’s easy to type system yourself in a corner.
You even lose the whole big selling point of rust when you use `unsafe`.
There was an article a while ago comparing zig and rust and it had a good quote that went something like
“Rust is for programming in the large, and zig is for programming in the small”
And I think what you’re looking for is programming in the small.
Small, focused, programs with minimal dependencies.
IMHO, Rust is a replacement for C/C++, which are not so easy to write correctly.
Rust should not be a replacement for e.g. Go, Swift or Java, which are a lot easier to write and have memory safety.
I wonder who actually needs Rust (as opposed to using e.g. Go) and finds it a lot harder to write than C/C++?
Rust should not be a replacement for e.g. Go, Swift or Java, which are a lot easier to write and have memory safety.
I wonder who actually needs Rust (as opposed to using e.g. Go) and finds it a lot harder to write than C/C++?
The other extreme of this is:
* Bad abstractions which just stick around forever. There are some examples of this in UNIX which would never be invented in the way they are today but nonetheless aren't going anywhere (e.g. signal handling). This isn't good.
* Invent all of your own wheels. This isn't good either.
There's a balance that needs to be struck between all of these 3 extremes.
* Bad abstractions which just stick around forever. There are some examples of this in UNIX which would never be invented in the way they are today but nonetheless aren't going anywhere (e.g. signal handling). This isn't good.
* Invent all of your own wheels. This isn't good either.
There's a balance that needs to be struck between all of these 3 extremes.
I know it's just an example, but if you're on linux there's signalfd() which makes signals into IO so you can handle it in an epoll()-loop or whatever way you like doing IO
We can't remove the old way of course, as that would break things, but that doesn't stop improvements
We can't remove the old way of course, as that would break things, but that doesn't stop improvements
Sometimes removing the old way is the improvement though. E.g. adding an alternative to symlinks doesn't help if symlinks are still allowed.
Why?
That way you break a lot of things.
That way you break a lot of things.
You have to start somewhere. Strongly specialized programs that f.ex. never access disk and only access network are a good candidate to be tested in either restricted containers or brand new OS-es that carry legacy baggage.
It's doable, but nobody wants to put in the money, time and energy into pioneering it.
It's doable, but nobody wants to put in the money, time and energy into pioneering it.
I'm actually working on a linter for dependencies that checks all your dependencies on 15+ rules. https://github.com/DepshubHQ/depshub
It's true that dependency-free software is very rare these days. The most obvious reason is that people don't want to "reinvent the wheel" when doing something. While this is a 100% valid reason, sometimes people simply forget what they are building and for whom. Extensive usage of dependencies is just one of the forms of overengineering. Some engineering teams even do their planning and features because of the new shiny thing.
The problem of dependencies is massive these days, and most companies are focusing on producing more and more code instead of helping people manage what they already have.
It's true that dependency-free software is very rare these days. The most obvious reason is that people don't want to "reinvent the wheel" when doing something. While this is a 100% valid reason, sometimes people simply forget what they are building and for whom. Extensive usage of dependencies is just one of the forms of overengineering. Some engineering teams even do their planning and features because of the new shiny thing.
The problem of dependencies is massive these days, and most companies are focusing on producing more and more code instead of helping people manage what they already have.
Your thing looks promising. Do you plan to introduce GitHub releases? I would subscribe to those and read the CHANGELOG.
I do use GitHub releases already https://github.com/DepsHubHQ/depshub/releases
You can subscribe by clicking on Watch -> Custom -> Releases.
You can subscribe by clicking on Watch -> Custom -> Releases.
I used to think the same way seeing cargo pull so many LOCs, but let's be real, I am not going to write my HTTP server by hand, nor my postgresql driver. And those are the ones that pull a lot of dependencies.
As for smaller QOL dependencies, I might as well use them directly as I need them, instead of spending 3h building something similar but half as good and robust, just to decrement my dependencies counter by 1.
I've had very few issues with Rust where an old project can't be built anymore because of rotting dependencies, when it happens almost every time with JS or Python. This is my largest gripe with dependencies.
I do still think Rust would have been better off with a larger stdlib, but ultimately it's really not that big of a deal and isn't worth it rewriting code for the sake of reducing a meter.
As for smaller QOL dependencies, I might as well use them directly as I need them, instead of spending 3h building something similar but half as good and robust, just to decrement my dependencies counter by 1.
I've had very few issues with Rust where an old project can't be built anymore because of rotting dependencies, when it happens almost every time with JS or Python. This is my largest gripe with dependencies.
I do still think Rust would have been better off with a larger stdlib, but ultimately it's really not that big of a deal and isn't worth it rewriting code for the sake of reducing a meter.
Speaking only for myself, I avoid [other people's] dependencies like the plague.
But I use a lot of them; just ones that I wrote.
Almost all of my publicly-available work consists of Swift modules (SPM), with very few stars or forks. Not very popular.
Which suits me just fine. I write for an audience of 1. I publish my work, because I believe that this forces me to do a really good job on it. I don't like to worry about my dependencies. If I wrote them, and they follow my Quality bar, then they're fine.
I think, in all the work I do, I have maybe only two or three dependencies, outside of ones that I wrote, or that are provided by the development system (I write native, so there's no giant ball of mud hybrid library).
It does limit the scope of what I can do, but not really that much.
But I use a lot of them; just ones that I wrote.
Almost all of my publicly-available work consists of Swift modules (SPM), with very few stars or forks. Not very popular.
Which suits me just fine. I write for an audience of 1. I publish my work, because I believe that this forces me to do a really good job on it. I don't like to worry about my dependencies. If I wrote them, and they follow my Quality bar, then they're fine.
I think, in all the work I do, I have maybe only two or three dependencies, outside of ones that I wrote, or that are provided by the development system (I write native, so there's no giant ball of mud hybrid library).
It does limit the scope of what I can do, but not really that much.
Agreed with the sentiment of the author, but are teams really arguing for pulling in dependencies in the name of security?
I’ve seen the exact opposite — every dependency introduces a new opportunity for vulnerabilities, and to be judicious in pulling in new ones due to the potential security risk.
I’ve seen the exact opposite — every dependency introduces a new opportunity for vulnerabilities, and to be judicious in pulling in new ones due to the potential security risk.
I sympathize with the author, it's a tricky balance to strike and writing it yourself because how hard could it be can be a trap, but at the same time babelian towers of abstractions can and often do make for a great waste of mental resources.
One of the reasons I wrote https://crates.io/crates/self_cell/ was to avoid the dependency trees pulled in by proc macros like ouroboros.
One of the reasons I wrote https://crates.io/crates/self_cell/ was to avoid the dependency trees pulled in by proc macros like ouroboros.
It is really hard! And thank you for self_cell. It's one of the crates I'm happily pulling into projects because of having such a great balance between size and utility and also making it a goal to not have dependencies.
“When you’re working on a really, really good team with great programmers, everybody else’s code, frankly, is bug-infested garbage, and nobody else knows how to ship on time.”
— Joel Spolsky, In Defense of Not-Invented-Here Syndrome: <https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...>
— Joel Spolsky, In Defense of Not-Invented-Here Syndrome: <https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...>
I agree. Here is a similar post from another amazing developer.
https://www.mikeperham.com/2016/02/09/kill-your-dependencies...
https://www.mikeperham.com/2016/02/09/kill-your-dependencies...
The sentiment is good but our OP here is not making their argument well (and is incorrect about terminal development btw).
Mike Perham also talks about a dynamic language in an ecosystem that was just as obsessed with left-pad-like lunacies like the Node.JS ecosystem still is.
The sentiment is not wrong but doesn't apply to Rust. Be honest: would you roll your own HTTP and WebSockets server from scratch? Also how likely do you think it is that a business will allow you this huge downtime during which no moneymaking activities will be performed?
Mike Perham also talks about a dynamic language in an ecosystem that was just as obsessed with left-pad-like lunacies like the Node.JS ecosystem still is.
The sentiment is not wrong but doesn't apply to Rust. Be honest: would you roll your own HTTP and WebSockets server from scratch? Also how likely do you think it is that a business will allow you this huge downtime during which no moneymaking activities will be performed?
2025: Mitsuhiko discovers ::Tiny modules :) https://metacpan.org/search?size=500&q=%3A%3ATiny
The article shows it's a cultural problem, not a technical. The normal case should be that implementations come in all sizes of complexity on the market, and the end user can choose freely; and I consider this a healthy state of affairs. Rust and JS are a monoculture that has fallen prey to some mind virus where only big supply chain is valued, and so the market becomes distorted. Tell me whether I'm wrong in my assessment.
The article shows it's a cultural problem, not a technical. The normal case should be that implementations come in all sizes of complexity on the market, and the end user can choose freely; and I consider this a healthy state of affairs. Rust and JS are a monoculture that has fallen prey to some mind virus where only big supply chain is valued, and so the market becomes distorted. Tell me whether I'm wrong in my assessment.
As the author says, it's a balance. There are many, many problems we deal with that are relatively self-contained, whose solution is going to be stable over long swaths of time. For those types, I hereby grant everyone permission to craft/extract/copy the smallest most straightforward implementation directly into your codebase, and enjoy a life free of deprecation warnings and bogus security alerts and endless dependency churn. If you want some guidelines -- these sorts of cases should be at most one or two hundred lines of code, be free of any dependencies, and have a small api surface area.
It's important to remember that the most powerful form of code reuse is copy and paste.
Mutability for the win!
I think that although there are rational tradeoffs in bringing in too many dependencies, a lot of these arguments are just rationalizing a personal preference and a tendency of engineers to want to build more and more.
Is managing your software supply chain more work than writing it yourself? Probably not if there are thousands or millions of LOC involved. But if you have infinite money, go ahead and build everything from scratch..
Is managing your software supply chain more work than writing it yourself? Probably not if there are thousands or millions of LOC involved. But if you have infinite money, go ahead and build everything from scratch..
The issue is precisely that many people underestimate the cost of pulling someone else's code as a dependency.
Pulling a dependency with 1M LoC is probably more costly than writing 1000 LoC yourself.
Pulling a dependency with 1M LoC is probably more costly than writing 1000 LoC yourself.
Rather rich of OP (he's the author of flask and the rye package manager). Rust adopted the NPM style dependency resolution because it scales. The traditional Python school of dependency resolution doesn't scale. Sure you get a nice clean transitive graph, but you will also spend many hours wrangling upstream diamond dependency conflicts.
I feel strongly about this but figured it would be a decent exercise to challenge my assumptions? I've been doing this for a while and with the exception of a limited number of high quality packages I generally write a lot of my own utilities because in my opinion "the juice isn't worth the squeeze" from a lot of open source options. Obv The time trade off is different now that I have decent experience and with the advent of ChatGPT
From what I can tell the research tends to agree with me:
I would love to see other research if anyone has some... the quick ~5 studies I checked didn't seem to answer the question 100% but are good enough to confirm my thoughts.
From what I can tell the research tends to agree with me:
-https://www.researchgate.net/publication/221555430_An_empirical_study_of_build_maintenance_effort
- https://arxiv.org/abs/1710.04936
I would love to see other research if anyone has some... the quick ~5 studies I checked didn't seem to answer the question 100% but are good enough to confirm my thoughts.
One day I needed to parse and generate RISC-V instructions. I wrote a custom bitfield implementation, nothing fancy: no derive macros or any crap like that, only u32 bitfields specified by an array — [(RangeInclusive<u32> /src/, usize /length/, RangeInclusive<u32> /dst/]. One function + a bunch of tests. It worked great. Then I tried the `bitfield` crate, which claims to be as fast as code I would've written by hand. It had dependencies and was two orders of magnitude slower. Basically, it was as fast as code that I would've written by hand in Python.
I have a little JS library that has an analogue within cargo which really helps with this: https://www.npmjs.com/package/downgrade-build
It means I can keep my dependency spec broad, because I actually test my releases with minimum supported versions. Cargo has (or maybe had -- I can't find it any more) a similar feature built in.
Renovate will even broaden (rather than replace) dependency specifications, if asked, so if my package works with both the old and the new version then I can make a patch release with the wider dependency spec and not break anything.
(On which note: I can really highly recommend using Mend Renovate to keep dependencies up to date: https://www.mend.io/renovate/ )
It means I can keep my dependency spec broad, because I actually test my releases with minimum supported versions. Cargo has (or maybe had -- I can't find it any more) a similar feature built in.
Renovate will even broaden (rather than replace) dependency specifications, if asked, so if my package works with both the old and the new version then I can make a patch release with the wider dependency spec and not break anything.
(On which note: I can really highly recommend using Mend Renovate to keep dependencies up to date: https://www.mend.io/renovate/ )
Functions that fit on a page should usually be published as sample code rather than gems / crates / whatever.
> But there is a simpler path. You write code yourself. Sure, it's more work up front, but once it's written, it's done
Well, assuming you wrote it perfectly and didn't introduce any security vulnerabilities... that you will never be alerted to, because no one else is reviewing your code.
Well, assuming you wrote it perfectly and didn't introduce any security vulnerabilities... that you will never be alerted to, because no one else is reviewing your code.
This line of reasoning can’t get you far. Where do dependencies come from?
> security vulnerabilities
Does this code parse untrusted data? Does your process have unlimited access to sensitive resources?
> security vulnerabilities
Does this code parse untrusted data? Does your process have unlimited access to sensitive resources?
> Does this code parse untrusted data?
I don't understand the question. Yes, I trust the global community of software developers to write a parsing library with sanitization more than I trust myself + my one or two work colleagues.
I don't understand the question. Yes, I trust the global community of software developers to write a parsing library with sanitization more than I trust myself + my one or two work colleagues.
My observation with the Scala and Haskell ecosystems has been very similar. One primary source of churn is changing interfaces. And very advanced static type systems offer a large design space how to model interfaces. Which leads to lots of incremental improvements. But lots of interface changes together with deep dependency trees lead to exponential explosion of changes.
Rich Hickey has been discussing this very early, but almost nobody got it when he said it, myself included. Striving for obviously simple interfaces, that avoid nominal static types like the plague, is the way to avoid this often pointless waves of changes through language ecosystem, just because someone designed a better option monad or whatever.
Rich Hickey has been discussing this very early, but almost nobody got it when he said it, myself included. Striving for obviously simple interfaces, that avoid nominal static types like the plague, is the way to avoid this often pointless waves of changes through language ecosystem, just because someone designed a better option monad or whatever.
> Sure, it's more work up front, but once it's written, it's done.
I don’t think that’s entirely true. Unless you’re a star programmer you’ll find bugs and edge cases in your own code. So I end up updating it a lot. Not a dependency but also not write once
I don’t think that’s entirely true. Unless you’re a star programmer you’ll find bugs and edge cases in your own code. So I end up updating it a lot. Not a dependency but also not write once
At least in the JavaScript/NPM ecosystem, “if you want it done right, you have to do it yourself”.
Often the most popular package for X is actually low quality and/or obviously incorrect, especially for small-to-medium complexity topics. There may be better alternatives but they’re hard to find due to low usage numbers or poor SEO, so I usually need to build stuff myself out of necessity.
For example, the most popular sql`…` tagged template literal package available on NPM isn’t type safe, doesn’t understand differences in SQL dialects, can’t escape queries for debugging or manual execution, doesn’t have console.log formatter, lacks utilities like .join or binding helpers, etc. it’s an anemic package yet everyone uses it. Maybe there are better ones that are specific to a database like Postgres, but it’s a simple enough thing to build, plus if you build it yourself in the monorepo it’s super easy to improve things as they come up.
So, I stated with a little 300 line one for SQLite that’s grown over the years to have a nice Postgres dialect as well, an ecosystem of helpers and lint rules, and it seems far ahead of whatever in NPM.
Another area is ESlint rules. Often there’s an open source rule that does like 80% of the job, but the maintainers seem hostile or dislike typescript or something. So much easier to copy the rule into our repo and improve it there rather than trying to contribute back. Or, it’s the usual quality issue - original rule is doing some crazy slow shit and it’s much easier to write from scratch a simpler more correct version.
Often the most popular package for X is actually low quality and/or obviously incorrect, especially for small-to-medium complexity topics. There may be better alternatives but they’re hard to find due to low usage numbers or poor SEO, so I usually need to build stuff myself out of necessity.
For example, the most popular sql`…` tagged template literal package available on NPM isn’t type safe, doesn’t understand differences in SQL dialects, can’t escape queries for debugging or manual execution, doesn’t have console.log formatter, lacks utilities like .join or binding helpers, etc. it’s an anemic package yet everyone uses it. Maybe there are better ones that are specific to a database like Postgres, but it’s a simple enough thing to build, plus if you build it yourself in the monorepo it’s super easy to improve things as they come up.
So, I stated with a little 300 line one for SQLite that’s grown over the years to have a nice Postgres dialect as well, an ecosystem of helpers and lint rules, and it seems far ahead of whatever in NPM.
Another area is ESlint rules. Often there’s an open source rule that does like 80% of the job, but the maintainers seem hostile or dislike typescript or something. So much easier to copy the rule into our repo and improve it there rather than trying to contribute back. Or, it’s the usual quality issue - original rule is doing some crazy slow shit and it’s much easier to write from scratch a simpler more correct version.
I'll sound like a grumpy old man, but: that's one of the big advantages of using Java. The ecosystem is _super stable_ if you go with the usual stack (Spring Boot, etc.), you generally get support for all the modern cloud stuff, and it breaks very rarely.
Good news for you: Rust is already quite mature and is mostly the same as Java in this regard. OP is exaggerating, if not misinforming.
There was the "not-invented-here" syndrome, I think that Rust (cargo), Python (pypi) and Javascript (npm) push to the other extreme.
If you ship a library, you are responsible for its security. If you dynamically link to a library you don't ship, then it's not your problem anymore (someone else is shipping it). I think we tend to forget that: shared libraries have advantages in terms of security.
And whether you ship the dependency or not, you should be ready to replace it or maintain it yourself. Too many projects depend on a project they don't understand, and get screwed down the line because it becomes unmaintained. Many times it's actually better to write the code you need than to depend on a third-party you don't understand.
If you ship a library, you are responsible for its security. If you dynamically link to a library you don't ship, then it's not your problem anymore (someone else is shipping it). I think we tend to forget that: shared libraries have advantages in terms of security.
And whether you ship the dependency or not, you should be ready to replace it or maintain it yourself. Too many projects depend on a project they don't understand, and get screwed down the line because it becomes unmaintained. Many times it's actually better to write the code you need than to depend on a third-party you don't understand.
> If you ship a library, you are responsible for its security. If you dynamically link to a library you don't ship, then it's not your problem anymore
In what universe is that true? In this universe, you are now on the hook for the security of both libraries, and the extra one you don't actually have any influence over.
In what universe is that true? In this universe, you are now on the hook for the security of both libraries, and the extra one you don't actually have any influence over.
Well to be fair, in practice in this universe, nobody gives a damn about security. I meant it more in a normative way.
Nobody cares much about the security of their own code, and even less for the security of their dependencies. At least if you link dynamically, you give an opportunity to your distribution to have someone care about it.
Nobody cares much about the security of their own code, and even less for the security of their dependencies. At least if you link dynamically, you give an opportunity to your distribution to have someone care about it.
Rust is not taking this into the extreme at all. You're taking a ranty ill-informed article way too seriously. Maybe it confirms your bias?
I'm sharing my opinion on a related thread. Sure, I agree with the article. What makes you say that I am ill-informed, except that you disagree with me? Is it a rule that anyone disagreing with you is ill-informed?
Because you and others draw parallels to left-pad and that is very far from what is actually happening in the Rust ecosystem.
> Is it a rule that anyone disagreing with you is ill-informed?
Obviously not, I am getting a bit agitated that nuance is lost during such rants, and I get even more annoyed when people are like "yeah, I agree!" with zero attention to nuance.
Again, the article is ranty and it is not even factually correct -- the terminal functionality does not have such huge amounts of churn as many other modern tech stacks have, happily, but it's not frozen in time as the OP makes it sound. And it's not "change for the sake of change" either. OP even recognizes this part (by saying that other pieces are moving around this mostly static piece) later on which makes his earlier rant even worse IMO.
Hence my annoyance with your and others "+1" comments. Again, any parallels to others, much more churn-ey, ecosystems like Node.JS, are arbitrary and unfair.
> Is it a rule that anyone disagreing with you is ill-informed?
Obviously not, I am getting a bit agitated that nuance is lost during such rants, and I get even more annoyed when people are like "yeah, I agree!" with zero attention to nuance.
Again, the article is ranty and it is not even factually correct -- the terminal functionality does not have such huge amounts of churn as many other modern tech stacks have, happily, but it's not frozen in time as the OP makes it sound. And it's not "change for the sake of change" either. OP even recognizes this part (by saying that other pieces are moving around this mostly static piece) later on which makes his earlier rant even worse IMO.
Hence my annoyance with your and others "+1" comments. Again, any parallels to others, much more churn-ey, ecosystems like Node.JS, are arbitrary and unfair.
> Because you and others draw parallels to left-pad and that is very far from what is actually happening in the Rust ecosystem.
Sorry, where did I mention left-pad?
The only thing I say is: writing a project in C++, I can keep control and have a few dependencies. Doing the same thing in Rust explodes almost immediately: as soon as I add one dependency, the lock file is so big I stop giving a shit.
You can say I am wrong, but I'm just sharing a feeling.
Sorry, where did I mention left-pad?
The only thing I say is: writing a project in C++, I can keep control and have a few dependencies. Doing the same thing in Rust explodes almost immediately: as soon as I add one dependency, the lock file is so big I stop giving a shit.
You can say I am wrong, but I'm just sharing a feeling.
Correct, you're sharing a feeling. And feelings don't matter in our line of work.
Also I wasn't aware you should be "giving a shit" about a lock file. Nor why would that matter for anything at all.
The final program works and you didn't have to reinvent complex wheels (like terminal sizing in OP). I don't see the problem. I only see wins.
Also I wasn't aware you should be "giving a shit" about a lock file. Nor why would that matter for anything at all.
The final program works and you didn't have to reinvent complex wheels (like terminal sizing in OP). I don't see the problem. I only see wins.
Well the problem is exactly what I wrote in my original post: if you pull malware into your program as a transitive dependency, then you ship malware to your users.
You should care about not shipping malware to your users, I don't know how to put it differently.
You should care about not shipping malware to your users, I don't know how to put it differently.
We all know the dangers, dude.
It's always a ROI and other analyses: stakeholders don't care much about malware in transitive dependencies, they want feature X within timeline Y.
Do I like it? No, I hate it with all my heart, but that's the reality we live in. I am not paid to just tinker with the computer; I am paid to deliver stuff, within various deadlines, and of certain quality and that's not the highest possible, otherwise I'd thoroughly use fuzzers and fiddle with property tests for months, if not years.
It's always a ROI and other analyses: stakeholders don't care much about malware in transitive dependencies, they want feature X within timeline Y.
Do I like it? No, I hate it with all my heart, but that's the reality we live in. I am not paid to just tinker with the computer; I am paid to deliver stuff, within various deadlines, and of certain quality and that's not the highest possible, otherwise I'd thoroughly use fuzzers and fiddle with property tests for months, if not years.
Companies are more likely to reward engineers than scold them for pulling in that new “shiny library” that solves the problem they never actually had.
Perhaps I’m sheltered: is this really true in people’s experience?
Perhaps I’m sheltered: is this really true in people’s experience?
Depends on the company. Amazon had a pretty bad case of NIH, Meta tended to embrace open-source (and as often as not, transitioned to embrace-extend-extinguish if they liked it enough)
Yes, it's extremely common, think at least 95% of the time.
Time to delivery is the main metric for almost any company that ships software.
Time to delivery is the main metric for almost any company that ships software.
> when you end up using one function, but you compile hundreds, some alarm bell should go off.
That seems feasible to test automatically. Shouldn't an intelligent compiler automatically trim these things out?
That seems feasible to test automatically. Shouldn't an intelligent compiler automatically trim these things out?
[deleted]
It's all about straddling the line of 'do you want to know how it works', and 'do you want to get your task done'.
The second developer gets all the glory.
The second developer gets all the glory.
a.k.a. quality vs productivity. And as you said, the second gets all the glory.
Reinventing wheels does not always get you better wheels...
Often, with the type of developer who insists on reinventing wheels because it's "easy", you get a pretty bad wheel, as it's the first such wheel the developer has built and their estimation of "easy" was built on a foundation of ignorance.
Often, with the type of developer who insists on reinventing wheels because it's "easy", you get a pretty bad wheel, as it's the first such wheel the developer has built and their estimation of "easy" was built on a foundation of ignorance.
To be needlessly pedantic, reinventing wheels never gets you better wheels. The entire point of the idiom is that the wheel is already a perfect machine and any effort made trying to improve upon it is wasted. At best, you just expend time and effort to get the wheel again.
Which is why it doesn't belong in software - there is no software version of the wheel.
Which is why it doesn't belong in software - there is no software version of the wheel.
Yeah. Many times in software, I feel like I'm told: "Creating your own skateboard is reinventing the wheel, you don't want to do that. Instead use this train because it was written by Google and you won't do better".
Sure, but I need a skateboard, not a train.
Sure, but I need a skateboard, not a train.
[deleted]
Related, but not exactly the same: https://blog.gopheracademy.com/advent-2014/case-against-3pl/
I believe there's a balance to be struck between writing your own code and relying on dependencies. If Armin (the author of this post) is the one writing the code, I would highly recommend using that dependency!
"You write code yourself."
I don't see myself writing wgpu or tokio. And wgpu is a hefty price to pay in dependencies.
So yeah, dep's are a plague, but hey, we are writing super complex stuff nowadays.
And I don't care if my project needs 464 (yeah, you read it well) dependencies. I do my share by making sure I don't have 2 versions of the same lib and that's about it.
I don't see myself writing wgpu or tokio. And wgpu is a hefty price to pay in dependencies.
So yeah, dep's are a plague, but hey, we are writing super complex stuff nowadays.
And I don't care if my project needs 464 (yeah, you read it well) dependencies. I do my share by making sure I don't have 2 versions of the same lib and that's about it.
I'm so tired of rants like these. Both the pro and anti-dependency versions. It's just impossible to express an opinion on the issue without looking at specific applications.
What I want to see is someone actually break down a real application and its dependencies and argue through what's included and what's omitted.
At least there's one example. I just think it's wrong (and I'm not the only one...https://news.ycombinator.com/item?id=42813003). The terminalsize crate has special behavior for illumos. The average developer should never have to think about illumos.
(Note in case Bryan Cantrill walks by: no slight towards illumos or you--you seem great, loved the Oracle rant. Illumos seems cool. It's just that most developers should not think about it).
What I want to see is someone actually break down a real application and its dependencies and argue through what's included and what's omitted.
At least there's one example. I just think it's wrong (and I'm not the only one...https://news.ycombinator.com/item?id=42813003). The terminalsize crate has special behavior for illumos. The average developer should never have to think about illumos.
(Note in case Bryan Cantrill walks by: no slight towards illumos or you--you seem great, loved the Oracle rant. Illumos seems cool. It's just that most developers should not think about it).
Ha ha -- just out here on my daily walk through HN and this comment made my day...
Working with Go moved my mindset from use libraries for everything I had with JS/TS to writing things myself helped by stdlib
This article says that big number is bad but sometimes you can make number bigger. Big number is bad because of bad experience with big number.
If a project has 0 dependencies but uses the exact same code as a project with 100 dependencies which project is better?
They are the same. So dependencies are not inherently bad.
What if the dependencies also have another 100 lines of code that are not used in the project?
I would say that doesn't really matter. In some cases it might lead to a slower build time or even a slightly bigger binary.
The actual situation you will end up with is that you write your own code and it's nothing like the dependencies. This is great because you don't have to worry about the maintainer of the dependency wanting to add things you don't need or making mistakes. You only have to worry about you making mistakes or having made mistakes. Mistakes you will maybe never encounter and mistakes you will never find because most of your code will never be looked at again. The dependency you could have used did so you would have to deal with pesky updates that might not affect you.
You could also choose not to do any of the updates of the dependency. You just pin all dependencies to a specific version and never update. Whenever there is an issue you just pull in the dependency and fix the issue yourself. Now you've got the best of both worlds, you didn't have to do the work upfront of generating the code with AI and you still get to maintain everything yourself.
But we also know that using dependencies can be bad, experience has shown that. But why were those experiences bad. Leftpad was a problem of a package disappearing because it was not version pinned and the package manager allowed it to dissappear. This problem has been solved.
Then there is the issue of abandoned packages, version incompatibility or packages that move away from your use case. But how different is this from pinning all dependencies and maintaining them from that point.
The point that open source contributers and dependency users have made before is that once you download the code it's yours. You are always the maintainer of whatever code your project uses. This means you are responsible for maintaining the dependencies you use.
In the most basic sense you should start a fork for each dependency and start to maintain it. But that is an enormous task so not a solution. You can hope you picked correctly and the package will be maintained forever but you will surely pick some that will end up abandoned. Another solution could be to pay someone else to maintain the dependency or a fork of the dependency. If there are more projects that do this you can actually build on a stable basis where you can trust the dependencies are good. The number can be big but not bad. Otherwise you will have issues with whatever path you choose. Because fundamentally as a user of open source you are not a customer but a raccoon looking in a dumpster for code.
If a project has 0 dependencies but uses the exact same code as a project with 100 dependencies which project is better?
They are the same. So dependencies are not inherently bad.
What if the dependencies also have another 100 lines of code that are not used in the project?
I would say that doesn't really matter. In some cases it might lead to a slower build time or even a slightly bigger binary.
The actual situation you will end up with is that you write your own code and it's nothing like the dependencies. This is great because you don't have to worry about the maintainer of the dependency wanting to add things you don't need or making mistakes. You only have to worry about you making mistakes or having made mistakes. Mistakes you will maybe never encounter and mistakes you will never find because most of your code will never be looked at again. The dependency you could have used did so you would have to deal with pesky updates that might not affect you.
You could also choose not to do any of the updates of the dependency. You just pin all dependencies to a specific version and never update. Whenever there is an issue you just pull in the dependency and fix the issue yourself. Now you've got the best of both worlds, you didn't have to do the work upfront of generating the code with AI and you still get to maintain everything yourself.
But we also know that using dependencies can be bad, experience has shown that. But why were those experiences bad. Leftpad was a problem of a package disappearing because it was not version pinned and the package manager allowed it to dissappear. This problem has been solved.
Then there is the issue of abandoned packages, version incompatibility or packages that move away from your use case. But how different is this from pinning all dependencies and maintaining them from that point.
The point that open source contributers and dependency users have made before is that once you download the code it's yours. You are always the maintainer of whatever code your project uses. This means you are responsible for maintaining the dependencies you use.
In the most basic sense you should start a fork for each dependency and start to maintain it. But that is an enormous task so not a solution. You can hope you picked correctly and the package will be maintained forever but you will surely pick some that will end up abandoned. Another solution could be to pay someone else to maintain the dependency or a fork of the dependency. If there are more projects that do this you can actually build on a stable basis where you can trust the dependencies are good. The number can be big but not bad. Otherwise you will have issues with whatever path you choose. Because fundamentally as a user of open source you are not a customer but a raccoon looking in a dumpster for code.
Yet another security issue that can be fixed by slapping a nominal cost.
https://en.wikipedia.org/wiki/Sybil_attack#Economic_costs
It can also allow for maintainers of the RUSTSEC rating agency to perform some kind of basic check to see if there's any abuse.
https://en.wikipedia.org/wiki/Sybil_attack#Economic_costs
It can also allow for maintainers of the RUSTSEC rating agency to perform some kind of basic check to see if there's any abuse.
[deleted]
In C++, I control my dependencies. In Rust I get above 100 so quickly I just give up. In terms of security, let's be honest: I have no clue what I am shipping.
Also Rust does not have ABI compatibility and no culture of shared libraries (I guess it wouldn't be practical anyway given the number of libraries one needs). But that just destroys the OS package distribution model: when I choose a Linux distribution, I choose to trust those who build it. Say Canonical has a security team that tries to minimize the security issues in the packages that Ubuntu provides. Rust feels a lot more like Python in that sense, where anyone could push anything to PyPi.