Python Cryptography, Rust, and Gentoo(lwn.net)
lwn.net
Python Cryptography, Rust, and Gentoo
https://lwn.net/SubscriberLink/845535/b994663dd32cf51c/
55 comments
I was wondering the same thing. It's not as though LLVM is some niche project that only Rust is using; it's the backend for a large number of compilers, and I imagine this will continue to be the case moving forward. There are now even languages that are written in languages that use LLVM for their backend, like gleam [1].
I think, ideally, there would be a concerted effort to extend LLVM compatibility to those platforms. But I don't think it's unreasonable to assume that there simply won't be--I sometimes run into problems on aarch64, and most of these are far more niche than that. As the article mentioned, it seems likely that eventually the ability for certain platforms for get support on new compilers will be a make-or-break for their ability to stay current with upgrades for a lot of the more fundamental packages.
Alternatively, I guess, there could be effort to get a new gcc-backend for more langauges; I believe there's an ongoing effort for Rust[2], but who knows down the line what that might look like. Eventually, this will probably need to come to a head.
[1] https://gleam.run/
[2] https://github.com/Rust-GCC/gccrs
I think, ideally, there would be a concerted effort to extend LLVM compatibility to those platforms. But I don't think it's unreasonable to assume that there simply won't be--I sometimes run into problems on aarch64, and most of these are far more niche than that. As the article mentioned, it seems likely that eventually the ability for certain platforms for get support on new compilers will be a make-or-break for their ability to stay current with upgrades for a lot of the more fundamental packages.
Alternatively, I guess, there could be effort to get a new gcc-backend for more langauges; I believe there's an ongoing effort for Rust[2], but who knows down the line what that might look like. Eventually, this will probably need to come to a head.
[1] https://gleam.run/
[2] https://github.com/Rust-GCC/gccrs
If the Linux kernel also adds Rust as a dependency to build modules, gccrs could actually be a good solution for the platforms still supported by gcc.
It's ok for Gentoo to drop support for these platforms IMO, the least used architectures need to be relegated to the museum sometime.
It's ok for Gentoo to drop support for these platforms IMO, the least used architectures need to be relegated to the museum sometime.
> Surely, if people actually care about these platforms there would be people willing to commit to this long term?
In some cases more than others. I feel the pain of people who want to keep running private unique architectures. I wish it would be easier for them.
But mentioning s390? Companies that bought mainframes from IBM don't get to complain about compatibility with open source projects in my opinion. They can deal with that issue themselves and contribute back. (Or fund a solution) Show me how much extra you spent on wages and contracts related to that choice and I'll show you my tiniest violin.
In some cases more than others. I feel the pain of people who want to keep running private unique architectures. I wish it would be easier for them.
But mentioning s390? Companies that bought mainframes from IBM don't get to complain about compatibility with open source projects in my opinion. They can deal with that issue themselves and contribute back. (Or fund a solution) Show me how much extra you spent on wages and contracts related to that choice and I'll show you my tiniest violin.
And bear in mind that this is s390, not s390x. s390 is a 31-bit architecture that was superseded in 2000 by the 64-bit Z series, and support in the Linux kernel was dropped in 2015.
I had this with one of my own projects - a complaint that it didn't support an s390. I proposed a patch based on the described issue and stated it would be merged the moment someone who actually had an s390 tested it. Noone came forward. I'm certain a lot of these compatibility complaints fall into a similar area.
Yeah, the tradeoffs seem to be "add various gentoo/arch support to llvm" vs "nobody should use llvm for anything popular, because gentoo can't use it on abnormal architectures"
Put some of this upset-energy into building out gentoo+llvm. There's no way you're going to convince the entire programming world to stop using llvm at this point, even if you claim it's reasonable.
Put some of this upset-energy into building out gentoo+llvm. There's no way you're going to convince the entire programming world to stop using llvm at this point, even if you claim it's reasonable.
FWIW, an M68k backend for LLVM is on its way upstream (see reviews.llvm.org and search for "M68k").
I think it would also take some effort to add support for new platforms in rust, even if LLVM already supports the platform. The `std` library is pretty large, and it would take effort porting it. (nvm if their code is `nostd`)
Why doesn't rust team add support for GCC instead? Seems like that would be a big win for them.
Alternately, python cryptography could have chosen a different memory safe language, like Go, which is supported by GCC. Why did they intentionally choose a poorly fit language like Rust? Use the best tool for the job, yes?
Alternately, python cryptography could have chosen a different memory safe language, like Go, which is supported by GCC. Why did they intentionally choose a poorly fit language like Rust? Use the best tool for the job, yes?
GCC support would only add backends for very obscure or borderline abandoned platforms, because everything else is either in the process of being merged into LLVM (i.e. ESP32 Xtensa) or it's already supported.
The main benefit of a GCC-based Rust compiler would be disentangling Rust from rustc, specification from implementation.
The main benefit of a GCC-based Rust compiler would be disentangling Rust from rustc, specification from implementation.
Go is effective for whole-system reimplementations, but it's not a good fit for 1:1 C library reimplementations; that's a place where Rust shines.
For historic reasons GCC is architected to be much more difficult to use as a backend.
More ideological reasons than historic ones. As far as I know, they're still concerned with the whole "someone might steal our frontend or backend!" thing, unless I missed something and this attitude changed?
AFAIK, the fear was that people would implement proprietary plugins for GCC and not upstream their changes and improvements. The classic Embrace, Extend, and Extinguish. Emacs modules were delayed for a long time for the same reason, RMS only approved them on the condition that they export GPL symbols.
LLVM has changed the situation quite a bit though. GCC could make itself easier to hack on but it might be too late.
LLVM has changed the situation quite a bit though. GCC could make itself easier to hack on but it might be too late.
The irony of it is that no one was really interested in making a new frontend. LLVM was meant to be submitted to GNU, but RMS didn't really show interest in the 00's and instead it rebuffed attempts to disentangle GCC's frontend from the backend, forcing the creation of Clang.
It's arguably true that RMS's zealotry actually backfired spectacularly.
It's arguably true that RMS's zealotry actually backfired spectacularly.
rms actually missed the offer to donate LLVM to GNU because of the idiosyncratic way he uses email. See: https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00...
I think it's probably for the best; having two competing solutions with different priorities is probably a win for everyone, gcc and clang users alike. gcc has improved much in the last few years in part due to LLVM/clang's pressure.
I think it's probably for the best; having two competing solutions with different priorities is probably a win for everyone, gcc and clang users alike. gcc has improved much in the last few years in part due to LLVM/clang's pressure.
> LLVM has changed the situation quite a bit though.
I've seen people bring this up a few years ago, but there wasn't much change in attitude on rms' end. I don't keep up on the day-to-day of gcc, so perhaps things changed in the last few years(?) I would be surprised if this would happen under rms' tenure though.
I've seen people bring this up a few years ago, but there wasn't much change in attitude on rms' end. I don't keep up on the day-to-day of gcc, so perhaps things changed in the last few years(?) I would be surprised if this would happen under rms' tenure though.
Sir, you deserve a honorary MBA.
I would not use any language with a GC for cryptography core. The GC pause would kill performance. No rust either. Just raw C and a brain.
The Rust portability issue is the reason I have been pushing forward with the new M68k backend for LLVM which is due to be merged shortly:
> https://github.com/M680x0/M680x0-mono-repo
Patches for the M68k backend are discussed on reviews.llvm.org.
I have also started a Rust port for m68k already that depends on the LLVM backend work above:
> https://github.com/glaubitz/rust/tree/m68k-linux
For other architectures, the goal is to get gccrs moving forward and merged upstream:
> https://github.com/philberty/gccrs/
For anyone interested in helping, there are Bountysource campaigns I started to support both efforts:
> https://www.bountysource.com/issues/90829856-llvm-complete-t...
> https://www.bountysource.com/issues/86138921-rfe-add-a-front...
> https://github.com/M680x0/M680x0-mono-repo
Patches for the M68k backend are discussed on reviews.llvm.org.
I have also started a Rust port for m68k already that depends on the LLVM backend work above:
> https://github.com/glaubitz/rust/tree/m68k-linux
For other architectures, the goal is to get gccrs moving forward and merged upstream:
> https://github.com/philberty/gccrs/
For anyone interested in helping, there are Bountysource campaigns I started to support both efforts:
> https://www.bountysource.com/issues/90829856-llvm-complete-t...
> https://www.bountysource.com/issues/86138921-rfe-add-a-front...
> For other architectures, the goal is to get gccrs moving forward and merged upstream:
I would much rather see rustc_codegen_gcc upstreamed: https://github.com/antoyo/rustc_codegen_gcc
A GCC backend for rustc would provide all the code generation of GCC without duplicating the Rust frontend.
I would much rather see rustc_codegen_gcc upstreamed: https://github.com/antoyo/rustc_codegen_gcc
A GCC backend for rustc would provide all the code generation of GCC without duplicating the Rust frontend.
I have no particular preference for either solution on the GCC side and this particular project is new to me. Will add it to my list of projects I keep an eye on.
FWIW, I think gccrs also tries to use the Rust frontend if I remember correctly.
FWIW, I think gccrs also tries to use the Rust frontend if I remember correctly.
If Gentoo cared so much about portability, why didn't they write portage in C instead of using Python and bitching about other people's language choices?
FOSS isn't about providing other people whatever code they need for free.
If someone decides to do stuff in Rust they have every right to do so, and there's nothing their users can do except switch to something else or fork and maintain the code themselves.
The thing that also annoys me is that the platforms that are going to lose support are either so old or so obscure I sincerely doubt there are more than a handful Gentoo installations running on those.
Damn Alpha has been dead for more than 25 years, if you care about it so much you don't want to switch to a still supported architecture you should start invest real money on supporting the software stack yourself.
FOSS isn't about providing other people whatever code they need for free.
If someone decides to do stuff in Rust they have every right to do so, and there's nothing their users can do except switch to something else or fork and maintain the code themselves.
The thing that also annoys me is that the platforms that are going to lose support are either so old or so obscure I sincerely doubt there are more than a handful Gentoo installations running on those.
Damn Alpha has been dead for more than 25 years, if you care about it so much you don't want to switch to a still supported architecture you should start invest real money on supporting the software stack yourself.
> FOSS isn't about providing other people whatever code they need for free.
That's true. But FOSS is also about cooperation and Rust in particular is marketing themselves as open and inclusive.
> If someone decides to do stuff in Rust they have every right to do so, and there's nothing their users can do except switch to something else or fork and maintain the code themselves.
And what would Rust developers think if LLVM decided to make such a radical change to their codebase that Rust would no longer be able to use LLVM as a backend? Would also be in favor of an LLVM fork?
> The thing that also annoys me is that the platforms that are going to lose support are either so old or so obscure I sincerely doubt there are more than a handful Gentoo installations running on those.
There is also just a handful of users running Haiku, Hurd, NetBSD as compared to Linux. Yet I have not seen that argument made anywhere.
The nice thing about FOSS is that also smaller voices and groups are being heard. And I think that's great.
> Damn Alpha has been dead for more than 25 years, if you care about it so much you don't want to switch to a still supported architecture you should start invest real money on supporting the software stack yourself.
The nice thing about these old architectures is that they often help expose bugs that remain hidden on main targets. We have fixed a lot of bugs that only became visible on architectures like SPARC and M68k.
That's true. But FOSS is also about cooperation and Rust in particular is marketing themselves as open and inclusive.
> If someone decides to do stuff in Rust they have every right to do so, and there's nothing their users can do except switch to something else or fork and maintain the code themselves.
And what would Rust developers think if LLVM decided to make such a radical change to their codebase that Rust would no longer be able to use LLVM as a backend? Would also be in favor of an LLVM fork?
> The thing that also annoys me is that the platforms that are going to lose support are either so old or so obscure I sincerely doubt there are more than a handful Gentoo installations running on those.
There is also just a handful of users running Haiku, Hurd, NetBSD as compared to Linux. Yet I have not seen that argument made anywhere.
The nice thing about FOSS is that also smaller voices and groups are being heard. And I think that's great.
> Damn Alpha has been dead for more than 25 years, if you care about it so much you don't want to switch to a still supported architecture you should start invest real money on supporting the software stack yourself.
The nice thing about these old architectures is that they often help expose bugs that remain hidden on main targets. We have fixed a lot of bugs that only became visible on architectures like SPARC and M68k.
> But FOSS is also about cooperation and Rust in particular is marketing themselves as open and inclusive.
I dont see your point. Both Rust and LLVM are open for anyone that wants to implement and support new platforms. Espressif is submitting a new Xtensa backend for their architecture to LLVM, and the community has been using this to port Rust to the ESP32. If anyone wanted to step in and write a backend for Alpha or whatever, nobody would be there to stop them.
The real issue is that people expect everything to stay the same and don't change so to avoid them any extra work. If you care so much about a dead architecture you should be aware that things may break or stop working, or that someone could stop supporting it, especially if you got everything for free.
> users running Haiku, Hurd, NetBSD These platforms are run by determined people that often are ready to shoulder on themselves the task of porting stuff to their OS. I don't think they expect anyone to work and do stuff in their place. These also mostly run on platforms supported by LLVM (except netbsd, which has billions of ports), thus only requiring to adapt the OS part of the compiler (which is arguably simpler than writing an LLVM backend)
I dont see your point. Both Rust and LLVM are open for anyone that wants to implement and support new platforms. Espressif is submitting a new Xtensa backend for their architecture to LLVM, and the community has been using this to port Rust to the ESP32. If anyone wanted to step in and write a backend for Alpha or whatever, nobody would be there to stop them.
The real issue is that people expect everything to stay the same and don't change so to avoid them any extra work. If you care so much about a dead architecture you should be aware that things may break or stop working, or that someone could stop supporting it, especially if you got everything for free.
> users running Haiku, Hurd, NetBSD These platforms are run by determined people that often are ready to shoulder on themselves the task of porting stuff to their OS. I don't think they expect anyone to work and do stuff in their place. These also mostly run on platforms supported by LLVM (except netbsd, which has billions of ports), thus only requiring to adapt the OS part of the compiler (which is arguably simpler than writing an LLVM backend)
How is this Rust not being open and inclusive? Someone who has access to those old architectures can contribute support for them.
I don't feel bad for these people at all. C has held the throne for decades, and while I'm not a Rust fan, C is crap in terms of being a modern language.
No package system, no module system, no memory safety. It only enjoys current popularity because of inertia.
No package system, no module system, no memory safety. It only enjoys current popularity because of inertia.
Still, there are a lot of practical reasons why C still works better in a lot of less common cases. The thing is, if you add up the long tail of all the less common cases then you'll find that combined they're actually quite a substantial fraction of all cases.
That doesn't mean this isn't the right direction to go in, but some friction is to be expected, and ideally also reduced whenever possible in this "transitional period". I don't think this can just be dismissed with "C sucks".
That doesn't mean this isn't the right direction to go in, but some friction is to be expected, and ideally also reduced whenever possible in this "transitional period". I don't think this can just be dismissed with "C sucks".
I don't see how any of your points applies to this situation?
C is the undefeated queen of compiler availability. This is a discussion very much about availability.
C is the undefeated queen of compiler availability. This is a discussion very much about availability.
That was already true in the 1990's, but UNIX.
This does seem quite worrying. I have always loved the Gentoo project because it lets you try to install anything on any architecture (not that it's guaranteed to compile though). In an age where a good amount of our software is theoretically cross-platform, Gentoo serves as a great proof of concept (whereas other distributions just seem to drop support of architectures over time).
Just as an example, I came across a good blog of someone installing Gentoo on an Alpha workstation [0], which I find to be great, seeing as it is officially supported.
Maybe it's just my dogmatic, impractical preference that the software we write should try to work on older hardware, but seeing the entire LLVM ecosystem in this sort of situation makes me sad.
[0] https://cyborgyn.blogspot.com/2019/
Just as an example, I came across a good blog of someone installing Gentoo on an Alpha workstation [0], which I find to be great, seeing as it is officially supported.
Maybe it's just my dogmatic, impractical preference that the software we write should try to work on older hardware, but seeing the entire LLVM ecosystem in this sort of situation makes me sad.
[0] https://cyborgyn.blogspot.com/2019/
Virtually everything has been dropping support for Alpha, HP/PA, IA64, and the m68k (if it ever had it) for the last few years. I'm glad too, because it's quite hard to test your patches on architectures which are only available on eBay.
Portability is nice but at some point it has to stop. If literally no one is working on LLVM support for an architecture, it's dead.
Portability is nice but at some point it has to stop. If literally no one is working on LLVM support for an architecture, it's dead.
> Virtually everything has been dropping support for Alpha, HP/PA, IA64, and the m68k (if it ever had it) for the last few years.
Nah, m68k is one of the most actively developed architectures. It has very active kernel developers and we're soon seeing an official LLVM backend for the architecture coming (search reviews.llvm.org).
Alpha and HPPA are also actively maintained. IA64 is currently hanging a bit on a string, but we're most likely going to save it for the near future.
Nah, m68k is one of the most actively developed architectures. It has very active kernel developers and we're soon seeing an official LLVM backend for the architecture coming (search reviews.llvm.org).
Alpha and HPPA are also actively maintained. IA64 is currently hanging a bit on a string, but we're most likely going to save it for the near future.
I've had my own run-ins with issues surrounding trying to use Cryptography and hitting compile troubles. A few months back I made a source distribution (sdist) of an internal Python module using Poetry and tried building an Alpine-based container that used it. Because I made a sdist, installing it required Poetry to "compile", and Poetry has a dependency chain down to Cryptography. Because Python/PyPi doesn't support musl-compiled native libraries, I found myself having to install a several hundred MB toolchain to basically copy a Python script to the correct location. I think Poetry now ships a stripped down version of its build engine that doesn't have this dependency so problem solved-ish.
Admittedly I haven't looked at the Cryptography code base to see how much it depends on native code but I wish people made a conscious effort to consider the downstream ramifications of including a particular dependency. I am the maintainer of a quasi-popular JavaScript library and I deliberately wrote my own helper functions for things like base64/UTF-8 conversion just to avoid potential issues (it's been around since 2008 so JavaScript was also a hot mess at the time).
Admittedly I haven't looked at the Cryptography code base to see how much it depends on native code but I wish people made a conscious effort to consider the downstream ramifications of including a particular dependency. I am the maintainer of a quasi-popular JavaScript library and I deliberately wrote my own helper functions for things like base64/UTF-8 conversion just to avoid potential issues (it's been around since 2008 so JavaScript was also a hot mess at the time).
Is there any reason why you use Alpine for Python app containers?
https://pythonspeed.com/articles/alpine-docker-python/
https://pythonspeed.com/articles/alpine-docker-python/
This entitlement is why people are deathly afraid of breaking backwards compatibility. Disgraceful IMO.
I'd be interested to know from someone wiser than me, their thoughts on this library and the impact vs the Cryptodome python library (if any at all).
Having once upon a time preferred pycryptodome for some reason or other (various additional support to the standard at the time), perhaps I'm out of date or confused as to what the best standard python cryptography library should be for someone who might occasionally play in the "hazmat" space (to use the terms of the article's library).
Having once upon a time preferred pycryptodome for some reason or other (various additional support to the standard at the time), perhaps I'm out of date or confused as to what the best standard python cryptography library should be for someone who might occasionally play in the "hazmat" space (to use the terms of the article's library).
Cryptodome is a replacement or continuation of PyCrypto, a library that has had a large amount of issues, and while it provides the hazmat primitives it makes it easy to misuse them.
Cryptography on the other hand while providing the low-level functionality if required, provides a higher level interface on top of those primitives that makes it hard to misuse and thereby potentially make cryptographic mistakes.
Cryptography on the other hand while providing the low-level functionality if required, provides a higher level interface on top of those primitives that makes it hard to misuse and thereby potentially make cryptographic mistakes.
I wonder if this could eventually be addressed with the tactical deployment of QEMU images as part of builds. It would add significant complexity but it might just work.
It seems LLVM is holding back Rust than helping it
That not true. These architectures are not supported by LLVM only because it was started in the 2000s and they were borderline insignificant even back then. The GCC backend for most of those architectures only exists because GCC is much older, so it made no sense to throw it into the rubbish bin while there were still people around to maintain it.
We got a RISC-V and WASM backend into LLVM very quickly because people gave a f*ck about these. Stuff like IA64 or Alpha have been dead for so long I'm surprised there's still support in the kernel for those.
We got a RISC-V and WASM backend into LLVM very quickly because people gave a f*ck about these. Stuff like IA64 or Alpha have been dead for so long I'm surprised there's still support in the kernel for those.
Fun fact: GCC will drop IA64 in a few years. The Linux kernel has been shedding away support for IA64 in the last few releases. I don't see Gentoo complaining about these though.
> The Linux kernel has been shedding away support for IA64 in the last few releases.
It just has no maintainer at the moment. But that has been true for other architectures such as SuperH which have gained new maintainers in the meantime.
It just has no maintainer at the moment. But that has been true for other architectures such as SuperH which have gained new maintainers in the meantime.
> Stuff like IA64 or Alpha have been dead for so long I'm surprised there's still support in the kernel for those.
Alpha is still well maintained in the kernel and in Debian (and Gentoo) which is why it's still being around.
The same is true for HPPA, M68k and SuperH. IA64 has been a little dormant but we're trying to keep it alive although its chances are worse than for other architectures.
Alpha is still well maintained in the kernel and in Debian (and Gentoo) which is why it's still being around.
The same is true for HPPA, M68k and SuperH. IA64 has been a little dormant but we're trying to keep it alive although its chances are worse than for other architectures.
It is not reasonable to demand that every important project donate work to maintain peoples basement museums of defunct architectures.
tl;dr Gentoo requires users to build packages locally meaning they are dependent on the build software for a library to have Gentoo support for their architecture. Rust has limited support for some of the architectures Gentoo otherwise supports. Since this is an important crypto library adopting Rust Gentoo may have to drop support for certain currently supported architectures.
Seems like Gentoo found an alternative package,
"Back in Gentoo-land, it turned out that the cryptography dependency for Portage came because it was using urllib3 and requests. Those two packages in Gentoo are dependent on cryptography, but it turns out that they do not actually need it. A pull request to fix that was merged, so the problem for Portage, which is pretty fundamental to the operation of a Gentoo system, was averted."
"Back in Gentoo-land, it turned out that the cryptography dependency for Portage came because it was using urllib3 and requests. Those two packages in Gentoo are dependent on cryptography, but it turns out that they do not actually need it. A pull request to fix that was merged, so the problem for Portage, which is pretty fundamental to the operation of a Gentoo system, was averted."
The issue is kinda more complex: cryptography is also a hard dependency of pip, the official Python package installer.
And CPython itself seem to support PPC64 and s390x — which are Tier 2 targets for rust.
Hence, pip now depends on software that is not tested on some platforms officially supported by Python…
Sources: - https://doc.rust-lang.org/nightly/rustc/platform-support.htm... - https://buildbot.python.org/all/#/grid
Hence, pip now depends on software that is not tested on some platforms officially supported by Python…
Sources: - https://doc.rust-lang.org/nightly/rustc/platform-support.htm... - https://buildbot.python.org/all/#/grid
I never liked python-cryptography in the first place, from the beginning, the hazmat thing let me think that it is an 'asshole' directed project like go.
No offense to be taken, but by that I mean the kind of software where they have some personal principes that they impose on everyone as being the one truth and considering users as kids that need to be framed.
I liked so much more the pycryptodome project that took care of covering the most important things for users like retro-compatibility.
I have worked in the embedded world and there, that kind of sudden change, with extra heavy dependency at compile time is a real pain in the ass!
So, the conclusion is probably like for Gentoo to avoid using the cryptography lib, that would certainly avoid further other unilateral hard sudden changes in the future...
No offense to be taken, but by that I mean the kind of software where they have some personal principes that they impose on everyone as being the one truth and considering users as kids that need to be framed.
I liked so much more the pycryptodome project that took care of covering the most important things for users like retro-compatibility.
I have worked in the embedded world and there, that kind of sudden change, with extra heavy dependency at compile time is a real pain in the ass!
So, the conclusion is probably like for Gentoo to avoid using the cryptography lib, that would certainly avoid further other unilateral hard sudden changes in the future...
Surely, if people actually care about these platforms there would be people willing to commit to this long term? If not, are people saying that everything needs to support all the GCC-supported platforms forever?