Is my package reproducible yet?(ismypackagereproducibleyet.org)
ismypackagereproducibleyet.org
Is my package reproducible yet?
https://ismypackagereproducibleyet.org/
72 comments
>Are there any other languages that focus on "reproducible builds" for packages?
I think you misunderstand what "reproducible" means in the context of OP, but I will try to answer your question in the context of Rust.
When you upload a crate to crates.io it gets uniquely identified by its cryptographic hash. It's not possible to replace code for an already uploaded crate version. Now, when you get dependencies for your build you may get newer (semver-compatible) version, but you always can retrieve the older version. To prevent such automatic updates, Rust applications often commit Cargo.lock to their repos, which contains hashes of all dependencies. So unless you manually run `cargo update` you will get exactly the same dependency tree, which with some minor build tweaks makes properly reproducible builds possible.
Arguably, it's a much better system than relying on VCS as Go does.
I think you misunderstand what "reproducible" means in the context of OP, but I will try to answer your question in the context of Rust.
When you upload a crate to crates.io it gets uniquely identified by its cryptographic hash. It's not possible to replace code for an already uploaded crate version. Now, when you get dependencies for your build you may get newer (semver-compatible) version, but you always can retrieve the older version. To prevent such automatic updates, Rust applications often commit Cargo.lock to their repos, which contains hashes of all dependencies. So unless you manually run `cargo update` you will get exactly the same dependency tree, which with some minor build tweaks makes properly reproducible builds possible.
Arguably, it's a much better system than relying on VCS as Go does.
I've seen Rust crates where the code was generated from a Python script and some other data and neither of those things are in the crate, which is potentially a license violation for copyleft crates. Or crates where the tests were in VCS but not in the crate. Or there were prebuilt files in the crate. I think always going with the VCS is a better option.
>code was generated from a Python script and some other data and neither of those things are in the crate
Yes, it's (IMO unfortunately) possible to have non-reproducible code generation during compile time. The most prominent example is code generation based on DB schema. But such cases are quite rare and "going with the VCS" does not help here in any way.
If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here.
>Or crates where the tests were in VCS but not in the crate
Tests are not used in building the final application, so they are irrelevant for reproducibility.
>prebuilt files in the crate
While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
Yes, it's (IMO unfortunately) possible to have non-reproducible code generation during compile time. The most prominent example is code generation based on DB schema. But such cases are quite rare and "going with the VCS" does not help here in any way.
If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here.
>Or crates where the tests were in VCS but not in the crate
Tests are not used in building the final application, so they are irrelevant for reproducibility.
>prebuilt files in the crate
While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
> "going with the VCS" does not help here in any way.
It does, because then you can run the build process yourself from the real source files and check if it is reproducible.
> If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here.
Right. The problem is there is no source in the crate, just generated Rust code, so you can't know if the build process to create that Rust code is reproducible or not.
> While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
The problem is that these are prebuilt files and so your build process is just copying them not building them, so you can't know if the build of them is reproducible.
It does, because then you can run the build process yourself from the real source files and check if it is reproducible.
> If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here.
Right. The problem is there is no source in the crate, just generated Rust code, so you can't know if the build process to create that Rust code is reproducible or not.
> While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
The problem is that these are prebuilt files and so your build process is just copying them not building them, so you can't know if the build of them is reproducible.
I think your definition of "reproducible" is somewhat different from the one usually used. Reproducibility means that by running build process you always get the same binary, nothing more, nothing less. Having a bunch of generated Rust code or blobs in a published crate makes it harder to review (which is why I consider it a bad practice), but such crate is still completely reproducible.
The crate might be reproducible, but the build process used to create the crate is not reproducible. Every build process should be reproducible, even the ones that are used to create the crates themselves.
There is no build system that can guarantee that though - Go's certainly doesn't, with its use of go:generate tags that can easily do system-dependent things at build time and ability to link C lib blobs.
> One of my biggest pet peeves with NPM is that the packages aren't reproducible[...] Are there any other languages that focus on "reproducible builds" for packages?
NPM isn't a language. (Nor is NodeJS, for that matter.) It doesn't even occupy some semi-official position in its relationship to the actual language. Often, the NPM/NodeJS way even stands in stark contrast—working against the spirit and sometimes even the letter of the relevant standards and language features.
(It is pretty remarkable, I admit, how the opinionated set of practices associated with NPM/NodeJS have managed to reach the place it has in the associated cultural dominance hierarchy, to the point that people often forget about this stuff.)
> (Ideally via Git)
Yeah, and it's super easy: you can make your builds reproducible by committing your source code to Git. All of it. (I.e., the libraries you're using, too.) People get irrationally upset when you suggest it, though; it really feels like it shouldn't be that easy, so they don't let it.
NPM isn't a language. (Nor is NodeJS, for that matter.) It doesn't even occupy some semi-official position in its relationship to the actual language. Often, the NPM/NodeJS way even stands in stark contrast—working against the spirit and sometimes even the letter of the relevant standards and language features.
(It is pretty remarkable, I admit, how the opinionated set of practices associated with NPM/NodeJS have managed to reach the place it has in the associated cultural dominance hierarchy, to the point that people often forget about this stuff.)
> (Ideally via Git)
Yeah, and it's super easy: you can make your builds reproducible by committing your source code to Git. All of it. (I.e., the libraries you're using, too.) People get irrationally upset when you suggest it, though; it really feels like it shouldn't be that easy, so they don't let it.
> it's super easy
I have a suspicion that you're not putting the source for the specific versions of glibc and Linux you used into every one of your projects.
I have a suspicion that you're not putting the source for the specific versions of glibc and Linux you used into every one of your projects.
This is not nearly as clever a response as it sounds. It's moving the goalposts.
I don't produce anything where Linux's relationship to the application is as something that lives inside the build (unlike the types of third-party NPM dependencies we're talking about here) and I have a suspicion that there isn't any Linux inside the artifacts that your build step produces, either.
If that's not the case for you, then yeah, I'll bite the bullet. You absolutely can either: (a) include that source code in your repo or (b) stop building things like that. If you find that going with the former option isn't easy, then that's because it never was—whether you're checking in all the source code or not.
I don't produce anything where Linux's relationship to the application is as something that lives inside the build (unlike the types of third-party NPM dependencies we're talking about here) and I have a suspicion that there isn't any Linux inside the artifacts that your build step produces, either.
If that's not the case for you, then yeah, I'll bite the bullet. You absolutely can either: (a) include that source code in your repo or (b) stop building things like that. If you find that going with the former option isn't easy, then that's because it never was—whether you're checking in all the source code or not.
Pin a nixpkgs version and you're done.
A Nix package can do this easily!
> NPM isn't a language.
Fair point! I should have said "Are there any other package managers that focus on reproducible builds?" (Or maybe "ecosystem" would be a better term?)
> you can make your builds reproducible by committing your source code to Git.
That doesn't help when, for example, a TypeScript NPM module is compiled and "released" with only the JavaScript code + `*.d.ts` files. Or, worse, sometimes the release is entirely minified. (I feel like Deno might be tackling this problem, but I'm not sure.)
I agree that checking in the dependencies helps (we do that via Yarn PnP, which has it's own pain points too). It's still not the "end game" solution though like what Nix is targeting.
Fair point! I should have said "Are there any other package managers that focus on reproducible builds?" (Or maybe "ecosystem" would be a better term?)
> you can make your builds reproducible by committing your source code to Git.
That doesn't help when, for example, a TypeScript NPM module is compiled and "released" with only the JavaScript code + `*.d.ts` files. Or, worse, sometimes the release is entirely minified. (I feel like Deno might be tackling this problem, but I'm not sure.)
I agree that checking in the dependencies helps (we do that via Yarn PnP, which has it's own pain points too). It's still not the "end game" solution though like what Nix is targeting.
> That doesn't help when, for example, a TypeScript NPM module is compiled and "released" with only the JavaScript code + `*.d.ts` files
I don't understand. How does it not help in that scenario (or the minification one)?
I don't understand. How does it not help in that scenario (or the minification one)?
> Even if somebody links a package to their Git repo, who knows what commit correlates to a given release? Even if somebody pushes a "tag" for the release, there is still no guarantee that it's actually the same code
That's not what reproducible means in this context. Being able to go from a given release artifact to a git repo is not what the "reproducible builds" project/movement/whatever is about.
Reproducible builds are more about the packaging of something than about specific language ecosystems, so asking about languages that focus on reproducible builds makes less sense than asking about package ecosystems, build systems, and linux distros that do.
Let's look at an example: docker is written in go, and is marked as reproducible in debian, but unreproducible in archlinux: https://ismypackagereproducibleyet.org/?pkg=docker&query=que... (named docker.io on debian https://ismypackagereproducibleyet.org/?pkg=docker.io&query=...)
Why is that?
Well, debian sets the "version.BuildTime" option when it compiles it to a constant value each time it compiles it: https://salsa.debian.org/go-team/packages/docker/-/blob/4f1f...
Archlinux's PKGBUILD doesn't https://github.com/archlinux/svntogit-community/blob/cf04bef...
Because of that, "docker version" on arch linux outputs that it was built on the actual day it was built, whenever that was, and each time it's built that string changes. Hence, not reproducible.
Reproducible builds are about capturing enough information about how something is built, and specific enough build instructions, that two people can produce the same bit-for-bit identical output, and therefore verify that someone didn't compile a malicious backdoored binary, and also increase the chance that each build consistently works or doesn't work, regardless of who compiles it when.
That's not what reproducible means in this context. Being able to go from a given release artifact to a git repo is not what the "reproducible builds" project/movement/whatever is about.
Reproducible builds are more about the packaging of something than about specific language ecosystems, so asking about languages that focus on reproducible builds makes less sense than asking about package ecosystems, build systems, and linux distros that do.
Let's look at an example: docker is written in go, and is marked as reproducible in debian, but unreproducible in archlinux: https://ismypackagereproducibleyet.org/?pkg=docker&query=que... (named docker.io on debian https://ismypackagereproducibleyet.org/?pkg=docker.io&query=...)
Why is that?
Well, debian sets the "version.BuildTime" option when it compiles it to a constant value each time it compiles it: https://salsa.debian.org/go-team/packages/docker/-/blob/4f1f...
Archlinux's PKGBUILD doesn't https://github.com/archlinux/svntogit-community/blob/cf04bef...
Because of that, "docker version" on arch linux outputs that it was built on the actual day it was built, whenever that was, and each time it's built that string changes. Hence, not reproducible.
Reproducible builds are about capturing enough information about how something is built, and specific enough build instructions, that two people can produce the same bit-for-bit identical output, and therefore verify that someone didn't compile a malicious backdoored binary, and also increase the chance that each build consistently works or doesn't work, regardless of who compiles it when.
> Archlinux's PKGBUILD doesn't https://github.com/archlinux/svntogit-community/blob/cf04bef...
> Because of that, "docker version" on arch linux outputs that it was built on the actual day it was built, whenever that was, and each time it's built that string changes. Hence, not reproducible.
Fwiw, I'm the maintainer of Docker on Arch and this is a bit more complicated.
Arch doesn't need to care about build time because `pacman` sets `SOURCE_DATE_EPOCH` during package building which is then utilized by the docker build system.
https://github.com/moby/moby/commit/760763e9957840f1983a5006...
If you look at the diff from an actual reproduction of the `docker` package (the Debian Reproducible Builds CI is a fuzzer), you will see there is no issues around build time.
https://reproducible.archlinux.org/api/v0/builds/359851/diff...
What you do see is some weird differences around `NT_GNU_BUILD_ID` and `GO BUILDID`.
This is because of `lto`. In Arch we try to build most Go packages with cgo for the purpose of utilizing hardening flags. The issue is that the C code generation in Golang isn't actually reproducible with `lto` enabled.
One issue is what I submitted here; https://github.com/golang/go/pull/53528
Another issue which I have been trying to debug is why the code snippet located here doesn't reproduce.
https://pub.linderud.dev/cg/
This results in the BUILDID generated and sat by the cgo build process isn't reproducible.
I could disable lto and all the binary hardening and docker would probably be reproducible, but where is the fun in that :)?
Fwiw, I'm the maintainer of Docker on Arch and this is a bit more complicated.
Arch doesn't need to care about build time because `pacman` sets `SOURCE_DATE_EPOCH` during package building which is then utilized by the docker build system.
https://github.com/moby/moby/commit/760763e9957840f1983a5006...
If you look at the diff from an actual reproduction of the `docker` package (the Debian Reproducible Builds CI is a fuzzer), you will see there is no issues around build time.
https://reproducible.archlinux.org/api/v0/builds/359851/diff...
What you do see is some weird differences around `NT_GNU_BUILD_ID` and `GO BUILDID`.
This is because of `lto`. In Arch we try to build most Go packages with cgo for the purpose of utilizing hardening flags. The issue is that the C code generation in Golang isn't actually reproducible with `lto` enabled.
One issue is what I submitted here; https://github.com/golang/go/pull/53528
Another issue which I have been trying to debug is why the code snippet located here doesn't reproduce.
https://pub.linderud.dev/cg/
This results in the BUILDID generated and sat by the cgo build process isn't reproducible.
I could disable lto and all the binary hardening and docker would probably be reproducible, but where is the fun in that :)?
Neat, thanks for the explanation and links!
Sorry for being so off the mark there for your package, though my incorrect understanding I think did allow for a good explanation of what reproducibility is about and one thing that commonly fails!
Sorry for being so off the mark there for your package, though my incorrect understanding I think did allow for a good explanation of what reproducibility is about and one thing that commonly fails!
No worries :) I should blog more about these things so people do get better insight into obscure issues like the one above.
> Well, debian sets the "version.BuildTime" option when it compiles it to a constant value each time it compiles it: https://salsa.debian.org/go-team/packages/docker/-/blob/4f1f...
I feel like I understand why this is done, but it still feels extremely odd. We're making tools and such lie to us in the pursuit of reproducibility and oftentimes this means erasing certain information, that we might be interested in actually keeping!
Surely, we should be able to do better than that, something like:
That said, it's absolutely annoying to build a container image with Jib (just an example from when I last tried) and then see in Docker that its build time was supposedly decades ago. While having incorrect timestamps isn't the worst thing ever, I can imagine other pieces of information mattering more.
I feel like I understand why this is done, but it still feels extremely odd. We're making tools and such lie to us in the pursuit of reproducibility and oftentimes this means erasing certain information, that we might be interested in actually keeping!
Surely, we should be able to do better than that, something like:
/my-package
/reproducible
[CONSISTENT_BUILD_OUTPUT_FOR_CODE_INPUT]
/non-reproducible
build-time.txt
build-system-information.txt
build-log-with-timestamps.txt
build-package-mirrors-that-were-contacted.txt
build-step-timing-information-and-traces.txt
[OTHER_STUFF_GOES_HERE]
That said, it's absolutely annoying to build a container image with Jib (just an example from when I last tried) and then see in Docker that its build time was supposedly decades ago. While having incorrect timestamps isn't the worst thing ever, I can imagine other pieces of information mattering more.
If you have reproducible binaries, then you don't care about the build time, because all builds of the same version will be identical, regardless of when they were made. The simpler solution is to leave the build time out of the binary/package/whatever entirely.
Your last paragraph sounds like something that the GP poster seems to want from NPM. The NPM packages are downstream pre-built packages created (hopefully) from upstream git tags, but currently nothing verifies that and few people seem to care.
It's pretty doable to create reproducible builds with gradle or maven: https://reproducible-builds.org/docs/jvm/
Also, releases on maven central typically include a sources.jar along with the binaries. It's required to provide that as well as javadoc to publish on maven central, I think. They also require jars to be signed and there are some manual checks before they activate your account to verify e.g. domain ownership and metadata of your project. It's not perfect but it's better than what many other packagemanagers do. It's actually a bit of a PITA to setup; I've wasted quite a bit of time getting some stuff published there just trying to figure out their convoluted processes and tools and error messages. This stuff is way too hard in it's current form.
Not all projects that use these build tools are fully reproducible but that is probably pretty easy to fix if people raise awareness of issues around this topic.
There's even a website that tracks reproducibility for some common libraries on maven central: https://github.com/jvm-repo-rebuild/reproducible-central#rea...
If you want to use artififacts produced straight from git hashes, tags, or branches, jitpack.io is pretty neat.
Also, releases on maven central typically include a sources.jar along with the binaries. It's required to provide that as well as javadoc to publish on maven central, I think. They also require jars to be signed and there are some manual checks before they activate your account to verify e.g. domain ownership and metadata of your project. It's not perfect but it's better than what many other packagemanagers do. It's actually a bit of a PITA to setup; I've wasted quite a bit of time getting some stuff published there just trying to figure out their convoluted processes and tools and error messages. This stuff is way too hard in it's current form.
Not all projects that use these build tools are fully reproducible but that is probably pretty easy to fix if people raise awareness of issues around this topic.
There's even a website that tracks reproducibility for some common libraries on maven central: https://github.com/jvm-repo-rebuild/reproducible-central#rea...
If you want to use artififacts produced straight from git hashes, tags, or branches, jitpack.io is pretty neat.
Maybe something like SLSA could solve your issue https://slsa.dev/ the idea is to create a metadata standard so that packages can provide build providence information. So public build systems like github actions can sign a manifest explaining what build actions were executed and what the input and output of those actions are. That way you could verify a package was built from a specific set of sources with a specific set of build actions without requiring reproducibility. This assumes you trust the public build system not to lie though.
Composer (PHP) also does this. There is a package lock file that directly binds to a specific git commit, and all "composer install" builds are guaranteed to be identical for the same lock file (unless someone pushes something else under the same sha1, but that's Git's problem).
Packages are downloaded directly from GitHub or whatever repo configured.
Packages are downloaded directly from GitHub or whatever repo configured.
Thats the same with Rust/Cargo, Python/PyPI, Ruby/Gems and every other language packaging ecosystem too. Even old autotools tarballs are an example of this, who knows what gets into the tarball.
Doesn't Cargo always build from source? It still doesn't mean it's reproducible, but at least you can see what you're getting.
No, the Rust crate doesn't have to include the source. I've seen crates where the source is just files generated by a Python script and some other data, and the Python and data weren't in the crate, just in the upstream VCS.
But generated Rust code itself was included in the crate, correct? Otherwise, it wouldn't be possible to use such crate. If so, I don't see any problems for reproducibility.
There is no source in the crate, just generated Rust code, so you can't know if the build process to create that Rust code is reproducible or not.
But you don't need to know that for the reproducibility of your product. That's a problem for the maintainer of that crate, but you know that you're getting the same Rust code every time.
> you know that you're getting the same Rust code every time
Since code is generated, that could be considered the same as saying "you know you're getting the same binary every time" for pre-compiled binaries. The key argument is whether "generated code" can be considered "source code" or not.
Personally, I'd say no - "source code" almost always refers to the preferred representation that humans are editing. If code is generated, then the true source code is the code of the generator and its input. A similar distinction is declared in the GNU General Public License, to prevent companies from circumventing the spirit of the license by releasing obfuscated source code.
Rule of thumb: If the original author is not editing it directly, it's not source code.
Since code is generated, that could be considered the same as saying "you know you're getting the same binary every time" for pre-compiled binaries. The key argument is whether "generated code" can be considered "source code" or not.
Personally, I'd say no - "source code" almost always refers to the preferred representation that humans are editing. If code is generated, then the true source code is the code of the generator and its input. A similar distinction is declared in the GNU General Public License, to prevent companies from circumventing the spirit of the license by releasing obfuscated source code.
Rule of thumb: If the original author is not editing it directly, it's not source code.
Whether this distinction matters entirely depends on what your end goal is for which you want reproducible builds.
Besides, it's not a problem that Cargo alone can solve. At the end of the day it's picking up code from a git repo, and how you produced the code in that repo is not something it could control.
Besides, it's not a problem that Cargo alone can solve. At the end of the day it's picking up code from a git repo, and how you produced the code in that repo is not something it could control.
The website being discussed in this thread is about the reproducibility of the crates (or other packages) themselves.
It doesn't make a difference, even if you look at that particular crate itself. Every time you build version X of that crate, you will build the same Rust code. Cargo actually gets this part right. (Of course it doesn't mean building the same Rust code gives you the same binary every time - that's a separate problem)
Sure, _updating_ the crate might be a nightmare if the generation part is not stable - but that's unrelated to reproducibility.
Sure, _updating_ the crate might be a nightmare if the generation part is not stable - but that's unrelated to reproducibility.
No, it's not about the reproducibility of crates or language-package-manager-packages, it's only about the reproducibility of packages in specific linux distributions.
Those linux distributions often include their own custom build scripts and patches to make things reproducible. Debian isn't just execing "npm install" or "cargo install" when you "apt install <package>", so "reproducibility" in the context of this webpage doesn't necessarily mean anything at all about specific crates/packages, other than that the debian/arch/opensuse maintainers were able to make their build process for that package, possibly with patches, reproducible.
Those linux distributions often include their own custom build scripts and patches to make things reproducible. Debian isn't just execing "npm install" or "cargo install" when you "apt install <package>", so "reproducibility" in the context of this webpage doesn't necessarily mean anything at all about specific crates/packages, other than that the debian/arch/opensuse maintainers were able to make their build process for that package, possibly with patches, reproducible.
Right now there are indeed zero people working on the reproducibility of crates and other language-package-manager packages, but there is no reason why the site would be restricted to distro packages once the language-package-manager communities start working on the reproducibility of their packages.
Some distros actually do run `cargo install` or similar during build, although Debian does not indeed.
Some distros actually do run `cargo install` or similar during build, although Debian does not indeed.
What's wrong with maven? You have version in pom.xml, you can retrieve corresponding git tag and audit it.
Git tags can be changed, i.e. which commits they point to may change over time.
Version numbers are just a form of documentation, giving the developer's best guess of whether the API broke or not.
In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g.
- Making changes without updating the version number (bad practice, but easily done)
- Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit.
- Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.).
- Building with different versions of Maven, Java, etc.
- The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache.
Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both.
Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything.
FYI at my workplace we avoid some of these issues by:
- Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc.
- All builds are performed without network access. This prevents Maven resolving version numbers differently.
- The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other.
- Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).
In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g.
- Making changes without updating the version number (bad practice, but easily done)
- Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit.
- Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.).
- Building with different versions of Maven, Java, etc.
- The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache.
Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both.
Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything.
FYI at my workplace we avoid some of these issues by:
- Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc.
- All builds are performed without network access. This prevents Maven resolving version numbers differently.
- The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other.
- Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).
Those issues are common for almost all package management system. Asking user to use hashes instead of versions is absurd. Those who need more control can always build packages on their environment.
> Those issues are common for almost all package management system
Yes, the situation's pretty bad. On the plus side, I'm increasingly of the opinion that "package management" is unnecessary; and usually just a symptom of having a poor build system.
Yes, the situation's pretty bad. On the plus side, I'm increasingly of the opinion that "package management" is unnecessary; and usually just a symptom of having a poor build system.
Using Nix[0] should solve the reproducibility of a package. We use its package manager for bob[1] to achieve reproducible builds for projects.
[0] https://nixos.org/guides/how-nix-works.html
[1] https://bob.build/
[0] https://nixos.org/guides/how-nix-works.html
[1] https://bob.build/
Unfortunately, not 100% of Nix packages are reproducible, so using it doesn't guarantee reproducibility. Since the tests don't currently use disorderfs, there could be more issues lurking due to filesystem ordering issues.
https://r13y.com/
https://r13y.com/
Lets not forget that simply using any given build system doesn't magically solve all the issues.
If a `make install` invocation gzips manpages instead of the package manager or distro hooks, then you won't have reproducible manpages unless the upstream uses the `-n` switch as well. This is just an example of many small and obscure issues that hide inn different upstream projects.
If a `make install` invocation gzips manpages instead of the package manager or distro hooks, then you won't have reproducible manpages unless the upstream uses the `-n` switch as well. This is just an example of many small and obscure issues that hide inn different upstream projects.
I'm loving the related diffoscope utility (new to me)
[1] https://diffoscope.org/
It did a fantastic job of highlighting the few changes in a tree of XML "one liner" files (each file one ~ 25 Kb text line).
[1] https://diffoscope.org/
It did a fantastic job of highlighting the few changes in a tree of XML "one liner" files (each file one ~ 25 Kb text line).
(doesn't list or use the package manager most likely to already be reproducibile)
... Huh. Yeah, that's a really short list of distros, and omitting nix and guix seems like a weird thing in this space.
Seems Nix/Guix are tracked outside of the Reproducible Builds infrastructure, but the site only lists those within that infra.
https://reproducible-builds.org/citests/ https://r13y.com/ https://data.guix.gnu.org/repository/1/branch/master/latest-...
https://reproducible-builds.org/citests/ https://r13y.com/ https://data.guix.gnu.org/repository/1/branch/master/latest-...
Which package manager would be more reproducible than pacman and Arch?
We hover around 85% and 89% reproducible packages across all our packages and no distro, to my knowledge, is close to anything like this.
https://reproducible.archlinux.org/
We hover around 85% and 89% reproducible packages across all our packages and no distro, to my knowledge, is close to anything like this.
https://reproducible.archlinux.org/
Nix?
It's not. It's marketing. The focus of Nix is reproducability in the same wain as docker: Same dependencies for the same behavior. They can't guarantee bit-for-bit reproducability any more then your average distribution.
Arch Linux has a rebuilder setup and you can safely reproduce between 85% and 89% of the repository packages.
Nix has never done such a thing, and would be far away from any comparable numbers.
Arch Linux has a rebuilder setup and you can safely reproduce between 85% and 89% of the repository packages.
Nix has never done such a thing, and would be far away from any comparable numbers.
This is literally linked in a reply to you: https://r13y.com/
Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever.
Guix is another example, it is more into “bootstrappability” though, so packages can be built from source all the way down to a few kilobyte C-subset compiler written in assembly. Very cool!
Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever.
Guix is another example, it is more into “bootstrappability” though, so packages can be built from source all the way down to a few kilobyte C-subset compiler written in assembly. Very cool!
>Nix is 99.77% reproducible, so come on. It’s not marketing, nix just properly solves the problem for the first time ever.
You haven't read r13y properly.
>1733 out of 1737 (99.77%) paths in the nixos.iso_minimal.x86_64-linux installation image are reproducible!
600 packages becomes 1737 paths which are the inputs used to generate the image. Several of the compared paths are just static files which doesn't change, so the "path to package" comparison doesn't work either.
EDIT: An example set of stuff it attempts to compare can be found here:
https://paste.xinu.at/TmVM0y5FBDWX/
Taken from the build here: https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813... They have not solved the problem.
You haven't read r13y properly.
>1733 out of 1737 (99.77%) paths in the nixos.iso_minimal.x86_64-linux installation image are reproducible!
600 packages becomes 1737 paths which are the inputs used to generate the image. Several of the compared paths are just static files which doesn't change, so the "path to package" comparison doesn't work either.
EDIT: An example set of stuff it attempts to compare can be found here:
https://paste.xinu.at/TmVM0y5FBDWX/
Taken from the build here: https://buildkite.com/grahamc/r13y-dot-com/builds/1149#01813... They have not solved the problem.
It’s almost as if no matter how reproducible your dependencies are, if the given build process/tool wants to play dirty it will. I really don’t see how arch fairs any better here, care to elaborate? While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here.
> I really don’t see how arch fairs any better here, care to elaborate?
Arch has spent a considerably amount of time getting to the figure it has and is in the same position as NixOS to maintain reproducible builds. The claim that NixOS solves this, or is uniquely dispositioned to solve this, is the false. That is what I'm effectively trying to point out here.
>While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here.
That is not what Reproducible Builds is about.
Arch has spent a considerably amount of time getting to the figure it has and is in the same position as NixOS to maintain reproducible builds. The claim that NixOS solves this, or is uniquely dispositioned to solve this, is the false. That is what I'm effectively trying to point out here.
>While it is a different problem, my litmus test for package managers is installing gnome a (the whole group) and then seeing whether I can transition to kde or vice versa without a bunch of dependencies lingering around forever. Pacman, apt, etc. all fail terribly here.
That is not what Reproducible Builds is about.
Well, that’s fair, though with flakes it will further improve in case of nix.
I do know it is not about reproducible builds, I wrote as much, but I still feel that dismissing nix as you did above is not professional.
I do know it is not about reproducible builds, I wrote as much, but I still feel that dismissing nix as you did above is not professional.
I'm sorry I don't meet your expected level of professionalism on a random internet forum. Rest assured I have filed a deviation report to my manager.
[deleted]
The search-engine is case-sensitive.
The "weather" icons sunny vs raining vs clock is a blast from the (Jenkins?) past :)
For example: https://tests.reproducible-builds.org/debian/rb-pkg/unstable...
For example: https://tests.reproducible-builds.org/debian/rb-pkg/unstable...
The Reproducible Builds test infra indeed runs on Jenkins, specifically the jenkins.debian.net service.
I'm using a hook for pacman on Arch to verify reproducible builds but one issue I've found is it slows down package management substantially. Has anyone else encountered this and is there a solution for verification without slowing down the package manager?
What are you describing here, compile time for a local verification?
If so, how would you imagine that could be sped up?
- If the distro is working on reproducibility, they should have a dashboard somewhere with reproducibility status.
For arch that would be https://reproducible.archlinux.org/
It would be done in their CI and there should be no reason for you do also manually verify this, unless you are a very targeted individual and you think its reasonable that arch linux infrastructure is hacked to target you.
Basically, either you trust your distro or you don't. If you are running local verifications of reproducibility, I am guessing you are not.
So consider switching to a distro that you do trust.
If so, how would you imagine that could be sped up?
- If the distro is working on reproducibility, they should have a dashboard somewhere with reproducibility status.
For arch that would be https://reproducible.archlinux.org/
It would be done in their CI and there should be no reason for you do also manually verify this, unless you are a very targeted individual and you think its reasonable that arch linux infrastructure is hacked to target you.
Basically, either you trust your distro or you don't. If you are running local verifications of reproducibility, I am guessing you are not.
So consider switching to a distro that you do trust.
No, not compile time... just verifying binary transparency:
https://github.com/kpcyrd/pacman-bintrans
https://github.com/kpcyrd/pacman-bintrans
Would help if it explains which packages? NPM? Deb? Rpm?
The page mentions openSUSE, ArchLinux, Debian.
I'm not 100% certain, but I believe this is an area where Golang is doing better (code references a Git repo directly).
Are there any other languages that focus on "reproducible builds" for packages? (Ideally via Git)
Edit: I just want to add that this is a major frustration for me with Java too. Everything on Maven barely has any metadata to go back and audit the source code!