Resistance to Rust abstractions for DMA mapping(lwn.net)
lwn.net
Resistance to Rust abstractions for DMA mapping
https://lwn.net/SubscriberLink/1006805/f75d238e25728afe/
97 comments
Additionally regardless of what upstream decides to do, Microsoft and Google are pretty much down on having Rust on their Linux forks, e.g. Sphere OS, Azure Linux, Android, ChromeOS.
I wonder why that C file which maps a more abstract Rust-friendly C-API on top of the existing API can't live inside the Rust directory and build structure. If I would be one of the Rust maintainers, I would try to be as non-intrusive to the established project structure as possible, ideally completely invisible. It's also cleaner to keep all the Rust-related changes on the 'Rust side' of the project, even if they happen to be written in C (I'm also still wondering why Rust-for-Linux cannot live completely downstream in a temporary fork).
I also can fully understand that a C programmer doesn't want to deal with such a 'Rust idiomatic' C API.
I also can fully understand that a C programmer doesn't want to deal with such a 'Rust idiomatic' C API.
> I wonder why that C file which maps a more abstract Rust-friendly C-API on top of the existing API can't live inside the Rust directory and build structure
This is more or less what the RfL folks are asking for - they have a Rust API to be used by other Rust code, which uses the existing C API, and are promising to maintain that API themselves. It lives in the Rust "directory".
The C maintainer is rejecting this, seemingly because his goal isn't to find a compromise that works but to completely block the project.
This is more or less what the RfL folks are asking for - they have a Rust API to be used by other Rust code, which uses the existing C API, and are promising to maintain that API themselves. It lives in the Rust "directory".
The C maintainer is rejecting this, seemingly because his goal isn't to find a compromise that works but to completely block the project.
>I wonder why that C file which maps a more abstract Rust-friendly C-API on top of the existing API can't live inside the Rust directory and build structure.
That's what this patch does. It's what the patch _always_ did. Christoph seems not to have actually looked at the patch before rejecting it.
When it was pointed out to him that his initial complaints were in fact unfounded, he didn't say "oh well, I guess it's OK then", he came up with more unfounded reasons to reject it. And when those points were addressed, he basically said "nah, I don't want to".
I should note that he does not actually have any authority to reject the patches, since they're not in his subsystem. They are bindings to the DMA subsystem, he was CC'd as an advisory, but he has no more right to reject the patches than he would to reject a GPU driver that used DMA.
This is a waste of everyone's time.
That's what this patch does. It's what the patch _always_ did. Christoph seems not to have actually looked at the patch before rejecting it.
When it was pointed out to him that his initial complaints were in fact unfounded, he didn't say "oh well, I guess it's OK then", he came up with more unfounded reasons to reject it. And when those points were addressed, he basically said "nah, I don't want to".
I should note that he does not actually have any authority to reject the patches, since they're not in his subsystem. They are bindings to the DMA subsystem, he was CC'd as an advisory, but he has no more right to reject the patches than he would to reject a GPU driver that used DMA.
This is a waste of everyone's time.
> Christoph seems not to have actually looked at the patch before rejecting it.
I had the same impression as well, in particular due to his wording:
> "No rust code in kernel/dma, please"
When, in fact, the code is in "rust/kernel/dma" not "kernel/dma".
It seems like he missed this and then doubled down on his stance when questioned.
I had the same impression as well, in particular due to his wording:
> "No rust code in kernel/dma, please"
When, in fact, the code is in "rust/kernel/dma" not "kernel/dma".
It seems like he missed this and then doubled down on his stance when questioned.
AFAIK, the merge was for the sake of change, not like he fully agreed with the adoption of Rust nor prioritization of Rust. The point is to have actual discussions, rather than pushing things with 10-feet poles just for being new and different. Anything that make sense will become the outcome of those discussions.
> that ship has already sailed when Linus decided to merge R4L into the kernel
Of course not. Merging the basic infra for Rust didn't come with "it can be used in every single part of the kernel, regardless of what the people maintaining that think"-kind of guarantee. One way to use Rust is to only use it in drivers for example, and it seems Hellwig is not against that: "do that in your driver so that you have to do it".
Of course not. Merging the basic infra for Rust didn't come with "it can be used in every single part of the kernel, regardless of what the people maintaining that think"-kind of guarantee. One way to use Rust is to only use it in drivers for example, and it seems Hellwig is not against that: "do that in your driver so that you have to do it".
That's exactly the point. The patch in question is not to be used in any part of the kernel. It is a wrapper around the C API. It has no impact on current code and it will only be used by Rust drivers using DMA, the people of Rust for Linux were clear about it.
Duplicating this wrapper in every driver would just make painful the usage of DMA for Rust drivers for not benefit.
Duplicating this wrapper in every driver would just make painful the usage of DMA for Rust drivers for not benefit.
And Hellwig feels that's better done by "keep[ing] the wrappers in your code instead of making life painful for others".
I can see where he's coming from. I can also see why doing that is painful.
I don't really know what the best possible solution is. I do know that "Linus decided on Rust4Linux, therefore, this must be merged" is not a good argument. "Linus decided «moduke» is merged, therefore, accept my patches that are useful to «module» on «some other subsystem»". That doesn't really work, obviously.
I can see where he's coming from. I can also see why doing that is painful.
I don't really know what the best possible solution is. I do know that "Linus decided on Rust4Linux, therefore, this must be merged" is not a good argument. "Linus decided «moduke» is merged, therefore, accept my patches that are useful to «module» on «some other subsystem»". That doesn't really work, obviously.
Yeah pretty much this
And I think Linus should be very clear that rust is there to stay
And I think Linus should be very clear that rust is there to stay
That still doesn't mean that 'Rust idiomatics' have to creep into every Linux subsystem which the Rust code needs to talk to. It just feels like a poor technical design to not keep both sides entirely separated.
> It just feels like a poor technical design to not keep both sides entirely separated.
Maybe you should read the mail thread if the article was not clear enough. The PR was inside Rust folder.
Maybe you should read the mail thread if the article was not clear enough. The PR was inside Rust folder.
As I understand it, the proposed patches do not "creep" into any other subsystem, but act as another user of the existing APIs, just like any driver using the C API.
My daily soap opera fix
Seems like one of those situations that the "BDFL" will need to (temporarily) drop the "B" to get things moving.
Joking aside, Rust for Linux seems to be entering a phase where rubber is hitting the road and some maintainers who thought that Rust would just go away are having to express their opposition more honestly. There are definitely reasons for and against Rust in the Linux kernel and I feel for them.
Actually this pattern happens in all spheres of life. Some change is introduced. First, people ignore/deny it thinking it won't affect them or they feel that the system is too strong and the change won't really happen. They are (probabilistically) right because successful change requires really superhuman drive which the Rust for Linux maintainers by luck/unluck (depending on who you are) happen to have.
Now real change is coming to the door of some maintainers. Some of them hate the melange of Rust and C the Linux kernel is likely to become in the coming years. So they are hoping that their subsystem can escape. But it can't if Rust is really going to be a viable rather than toy option.
I think Rust has been baking in the kernel for a long time -- The BDFL/DFL needs to informally poll the maintainers and take a definitive decision to cast Linux's lot with Rust (or not!?). The Experimental tag in Rust gives some people the hope that this Rust experiment could be ended. This experimental tag should be replaced with in-progress/not-mature tag to further send the message down the line.
Joking aside, Rust for Linux seems to be entering a phase where rubber is hitting the road and some maintainers who thought that Rust would just go away are having to express their opposition more honestly. There are definitely reasons for and against Rust in the Linux kernel and I feel for them.
Actually this pattern happens in all spheres of life. Some change is introduced. First, people ignore/deny it thinking it won't affect them or they feel that the system is too strong and the change won't really happen. They are (probabilistically) right because successful change requires really superhuman drive which the Rust for Linux maintainers by luck/unluck (depending on who you are) happen to have.
Now real change is coming to the door of some maintainers. Some of them hate the melange of Rust and C the Linux kernel is likely to become in the coming years. So they are hoping that their subsystem can escape. But it can't if Rust is really going to be a viable rather than toy option.
I think Rust has been baking in the kernel for a long time -- The BDFL/DFL needs to informally poll the maintainers and take a definitive decision to cast Linux's lot with Rust (or not!?). The Experimental tag in Rust gives some people the hope that this Rust experiment could be ended. This experimental tag should be replaced with in-progress/not-mature tag to further send the message down the line.
Nobody is asking Christoph to maintain Rust code. Just work with the R4L developers occasionally when a breaking change happens, the same way he likely would need to do so with every C-based consumer of those APIs. But he doesn't want to do that, and he also doesn't want to let anyone co-maintain the DMA subsystem. At some point this does need to be called out as gatekeepy and counterproductive behavior by a maintainer who doesn't want to work with others.
But working with others is kind of, like, necessary, in a project as large and critical as the Linux project. Developing new kernel talent was one of Linus' express reasons for doing the project in the first place.
https://www.youtube.com/watch?v=OvuEYtkOH88&t=6m07s
But working with others is kind of, like, necessary, in a project as large and critical as the Linux project. Developing new kernel talent was one of Linus' express reasons for doing the project in the first place.
https://www.youtube.com/watch?v=OvuEYtkOH88&t=6m07s
> But Christoph Hellwig, who does a lot of work with the DMA-mapping layer, turned this submission away with a message reading, in its entirety: "No rust code in kernel/dma, please" (despite the fact that the patch did not put any code in that directory). When pressed, he added that developers should keep these abstractions in their own code and said that he had no interest in maintaining multi-language code. Rust developers should keep their wrapping code to themselves, he concluded.
> Danilo Krummrich pointed out that the proposed abstractions were doing exactly that — keeping the Rust code separate from the rest: "We wrote a single piece of Rust code that abstracts the C API for all Rust drivers, which we offer to maintain ourselves".
Please make it make sense. One of these folks is very wrong, but I can’t tell who?
> Danilo Krummrich pointed out that the proposed abstractions were doing exactly that — keeping the Rust code separate from the rest: "We wrote a single piece of Rust code that abstracts the C API for all Rust drivers, which we offer to maintain ourselves".
Please make it make sense. One of these folks is very wrong, but I can’t tell who?
IIUC, I think it is kernel policy that if you change some kernel API and break its consumers, you are responsible for fixing those consumers. (correct me if this is wrong.) So in theory DMA devs may need to touch Rust code if they make changes to the DMA subsystem.
Edit: this is wrong/inaccurate, see comments below.
Edit: this is wrong/inaccurate, see comments below.
That's the general policy, but Rust is currently an exception to that policy: maintainers are not required to keep Rust bindings working when they change C code. The Rust-for-Linux folks will come along and fix them later.
C maintainers are not required to consider RfL currently as a compromise to minimize the burden.
how does this ecosystem accept any patches if any possible changes could make the system worse? I understand the implication about languages, I'm just struggling to understand how any changes are possible at all.
Not a kernel developer, but I assume that it's done by having relatively stable/long lived APIs, and some form of deprecation process where both a new and old version coexist for a relatively long time.
Internal APIs break all the time, which is why out of tree drivers are so annoying
Well linux has obviously failed this—their abi is notoriously unreliable and relative to the distro you're using. There's a reason why game development is so reliant on wine.... microsoft actually has a stable ABI.
Linux: permanently stuck in the shittiest part of the nineties.
Linux: permanently stuck in the shittiest part of the nineties.
This is entirely a glibc and desktop Linux problem, the Linux kernel itself has a a very stable public syscall API (e.g. the mantra "don't break userspace" unfortunately only applies to Linux itself, but not to 'GNU/Linux' or desktop distros in general).
> This is entirely a glibc and desktop Linux problem,
Ah, so just everything developers interact with and need to deal with. God forbid glibc develop a stable ABI and globally lock development? Thank god their lives are easier, nobody else matters.
Ah, so just everything developers interact with and need to deal with. God forbid glibc develop a stable ABI and globally lock development? Thank god their lives are easier, nobody else matters.
It's offtopic for the discussion, since this is strictly about the Linux kernel.
Also, you can just ignore the glibc and link statically with MUSL instead - this is useful for distro-agnostic cmdline tools but won't help much for UI programs, since all desktop functionality is inside DLLs (the X11 and GL DLLs might actually be more stable than the glibc though).
FWIW Windows had that same problem with the MSVC runtime DLLs until not too long ago. Only the traditional Win32 DLLs are guaranteed to exist on each Windows installation and with a stable API.
Also, you can just ignore the glibc and link statically with MUSL instead - this is useful for distro-agnostic cmdline tools but won't help much for UI programs, since all desktop functionality is inside DLLs (the X11 and GL DLLs might actually be more stable than the glibc though).
FWIW Windows had that same problem with the MSVC runtime DLLs until not too long ago. Only the traditional Win32 DLLs are guaranteed to exist on each Windows installation and with a stable API.
[deleted]
I like Rust, but my worst experiences with Rust are interop with C. From this LWN article, some people seem very very stubborn against including Rust in Linux. As someone who really likes Rust, I must say its borrow-checker design, and other things like derive and proc-macros. Personally, if Rust people are willing to undergo the full burden of maintaining Rust in the kernel and not place undue burden on the C developers, I feel that it could be a good idea to include Rust. It seems a little petty to just block Rust as a blanket statement.
However, I must ask: is there a world in which interop didn't have to be so bad without hugely sacrificing memory safety? Is this a problem that Zig solves (I must be honest, I don't really know much about Zig)? What _is_ the better solution?
However, I must ask: is there a world in which interop didn't have to be so bad without hugely sacrificing memory safety? Is this a problem that Zig solves (I must be honest, I don't really know much about Zig)? What _is_ the better solution?
In Rust, unsafe (unchecked) C interop is almost automatic.
The hard part is in translating C code's high-level safety requirements into Rust APIs that enforce them. I'm talking about requirements that aren't expressed in the C syntax, but are arbitrary domain-specific rules defined in English by the C code's authors ("this function can be called only on Wednesdays"). These turn out to be hard, because they may not be precisely defined, and/or the conditions are complex and implementation-specific. That's less of language inteop problem, and more in creating formal definitions of informal documentation.
Having said that, there are a couple of things that Rust made harder for itself:
• Rust allows moving objects to a different address safely. C code often assumes objects never move and their addresses are unique and meaningful. If Rust had built-in support for immovable types, it wouldn't need Pin and macro hacks.
• Rust's reference types require strict immutability or strict pointer aliasing (exclusive access), while C allows memory to be mutated from anywhere, and pointers to const don't mean the data is immutable. This prevents Rust from using its nice safe reference and slice syntax on memory externally mutated in surprising/clever ways (by memory-mapped hardware, shared mem, MMU trickery).
The hard part is in translating C code's high-level safety requirements into Rust APIs that enforce them. I'm talking about requirements that aren't expressed in the C syntax, but are arbitrary domain-specific rules defined in English by the C code's authors ("this function can be called only on Wednesdays"). These turn out to be hard, because they may not be precisely defined, and/or the conditions are complex and implementation-specific. That's less of language inteop problem, and more in creating formal definitions of informal documentation.
Having said that, there are a couple of things that Rust made harder for itself:
• Rust allows moving objects to a different address safely. C code often assumes objects never move and their addresses are unique and meaningful. If Rust had built-in support for immovable types, it wouldn't need Pin and macro hacks.
• Rust's reference types require strict immutability or strict pointer aliasing (exclusive access), while C allows memory to be mutated from anywhere, and pointers to const don't mean the data is immutable. This prevents Rust from using its nice safe reference and slice syntax on memory externally mutated in surprising/clever ways (by memory-mapped hardware, shared mem, MMU trickery).
> However, I must ask: is there a world in which interop didn't have to be so bad without hugely sacrificing memory safety?
You can't really make interop nicer than the lowest common denominator, and when one of the sides is C then that lowest common denominator is very low. Rust is one of very few languages that can potentially interop with memory-safe languages without having to completely buy into a common runtime - e.g. it's increasingly popular to write Python extensions in Rust and I believe there are now libraries for doing that without having to go via C - and when you have richer abstractions available on both sides of the line then you can make use of them. But to work with C you have to work like C, at least at the boundary.
You can't really make interop nicer than the lowest common denominator, and when one of the sides is C then that lowest common denominator is very low. Rust is one of very few languages that can potentially interop with memory-safe languages without having to completely buy into a common runtime - e.g. it's increasingly popular to write Python extensions in Rust and I believe there are now libraries for doing that without having to go via C - and when you have richer abstractions available on both sides of the line then you can make use of them. But to work with C you have to work like C, at least at the boundary.
It's not "C - the language" but "C - the calling convention" that gets in the way of higher level languages. And it's actually also wrong to blame C alone, ABIs are mostly a convention between operating system and CPU vendors, C compilers have to play by those rules just as any other compiler that wants to output code that can call (efficiently) into operating system APIs (and we could just as well be stuck with Pascal calling conventions instead, which was used by 16-bit Windows versions).
> ABIs are mostly a convention between operating system and CPU vendors, C compilers have to play by those rules just as any other compiler that wants to output code that can call (efficiently) into operating system APIs
Yes and no. Unix has a very C-oriented ABI because it's written in and tied to C, and since Unix and C won other operating systems have felt pressure to make their ABI C-friendly.
> we could just as well be stuck with Pascal calling conventions instead, which was used by 16-bit Windows versions
If we'd gotten Pascal strings in the ABI that alone would have eliminated one major class of bugs.
Yes and no. Unix has a very C-oriented ABI because it's written in and tied to C, and since Unix and C won other operating systems have felt pressure to make their ABI C-friendly.
> we could just as well be stuck with Pascal calling conventions instead, which was used by 16-bit Windows versions
If we'd gotten Pascal strings in the ABI that alone would have eliminated one major class of bugs.
Zig has some features to make C interop easier (for instance string literals are both ptr/length-slices and zero-terminated), but in general the problems are quite similar to Rust's.
There is a subset of Zig types that is compatible with C APIs, but most higher-level Zig types (like slices, odd-width integer types, optionals, error unions...) can't be tunneled through or mapped to a C API...
The main thing may be that the Zig compiler doesn't track ownership/lifetimes like the Rust compiler does, so it doesn't matter that this information would be lost at the C-API barrier.
There is a subset of Zig types that is compatible with C APIs, but most higher-level Zig types (like slices, odd-width integer types, optionals, error unions...) can't be tunneled through or mapped to a C API...
The main thing may be that the Zig compiler doesn't track ownership/lifetimes like the Rust compiler does, so it doesn't matter that this information would be lost at the C-API barrier.
> I like Rust, but my worst experiences with Rust are interop with C.
Could you elaborate? I find the interop to be quite nice. I mean, obviously, interop between any two languages cannot ever be "better" than the least good language at any one thing.
Could you elaborate? I find the interop to be quite nice. I mean, obviously, interop between any two languages cannot ever be "better" than the least good language at any one thing.
Zug does not attempt to solve memory safety. It aims to be a better C
A bit more nuance please, Zig doesn't attempt to solve compile-time memory safety like Rust does, but at least it provides spatial runtime memory safety (but also doesn't have a builtin solution for temporal memory safety - except a debug allocator which catches most use-after-free attempts on the heap) - so tl;dr: Zig is much better than C or C++ when it comes to memory safety, but isn't watertight like Rust.
So similar in that regard to Herb Sutter's experimental Cpp2 perhaps?
Actually the interoperability is very good. C can directly call into Rust and Rust can directly call into C. Rust can construct any C type and can expose opaque pointers to C world.
That's... not what the issue is, that is the bare minimum expectation.
But then again: for a language as bare-bones as C, what more is there really to a good FFI system?
The issue isn't just FFI - for example calling directly from Rust into C without any expression of the semantics means you have no mechanism for reasoning about ownership, and that removes many of the security benefits of writing code in Rust in the first place. If you have a strong description of the semantics of the C code then you can write Rust bindings that expose that, but you're then constraining the maintainers of the C codebase in terms of their ability to make arbitrary changes without considering the impact that has on the Rust consumers. Doing this badly is fairly easy, doing this well is hard.
You are obviously a far more knowledgeable person than I am, so please don't take the following questions as dismissive of your expertise. I'm genuinely trying to learn.
> for example calling directly from Rust into C without any expression of the semantics means you have no mechanism for reasoning about ownership
Right. But since C cannot express Rust's ownership semantics, could one really do better? Does that mean that as long as there's significant amounts of C in Linux, then Linux should never introduce any language with stronger semantics than C? That seems unfortunate.
> and that removes many of the security benefits of writing code in Rust in the first place.
I've always thought of the idea being that _as long as the necessary manual reasoning is correct_, downstream Rust users _do_ get those benefits. But of course interfacing with C, lacking the necessary semantics, requires the interface writer to manually assure them. Wouldn't anything more demand more than C can deliver?
> If you have a strong description of the semantics of the C code then you can write Rust bindings that expose that, but you're then constraining the maintainers of the C codebase in terms of their ability to make arbitrary changes without considering the impact that has on the Rust consumers.
OK, but that seems like a political/social/organizational aspect far more than a technical problem with C-Rust-interop?
> for example calling directly from Rust into C without any expression of the semantics means you have no mechanism for reasoning about ownership
Right. But since C cannot express Rust's ownership semantics, could one really do better? Does that mean that as long as there's significant amounts of C in Linux, then Linux should never introduce any language with stronger semantics than C? That seems unfortunate.
> and that removes many of the security benefits of writing code in Rust in the first place.
I've always thought of the idea being that _as long as the necessary manual reasoning is correct_, downstream Rust users _do_ get those benefits. But of course interfacing with C, lacking the necessary semantics, requires the interface writer to manually assure them. Wouldn't anything more demand more than C can deliver?
> If you have a strong description of the semantics of the C code then you can write Rust bindings that expose that, but you're then constraining the maintainers of the C codebase in terms of their ability to make arbitrary changes without considering the impact that has on the Rust consumers.
OK, but that seems like a political/social/organizational aspect far more than a technical problem with C-Rust-interop?
I think your understanding is correct - for Rust to work well in Linux, the C maintainers need to maintain more discipline than has been absolutely required before (in that if they've changed those semantics then any breakage has been largely silent and any bugs created have potentially been subtle), and changing that is a social problem more than a technical one because there's no technical mechanism to enforce the same rigor on C. The attempted tradeoff of "The bindings are up to the Rust developers" is something that only really works if the C developers are willing to accept that additional constraint.
(To be clear I have spent many nights drinking with Christoph and think he's entirely in the wrong here)
(To be clear I have spent many nights drinking with Christoph and think he's entirely in the wrong here)
As a non-Rust dev, what is the issue with C interop?
Roughly the same problems as C++ interop, you can't map common Rust concepts (like error unions, optionals, tagged-union enums, ptr/length-strings, traits) to C APIs without an intermediate shim which does potentially expensive and brittle marshalling, and the C-API is also a 'visibility barrier' for the Rust compiler when it comes to Rust concepts like RAII and ownership tracking.
I think C++ is worse actually due to the lack of move constructors in Rust (move is always memcpy). It means you can't e.g. return std::string by value.
The issues with C FFI are comparatively minor and not at all unique to Rust. Hell, even C++ has those issues when interacting with C APIs. It's not like a std::variant magically becomes a tagged union.
The issues with C FFI are comparatively minor and not at all unique to Rust. Hell, even C++ has those issues when interacting with C APIs. It's not like a std::variant magically becomes a tagged union.
> Hell, even C++ has those issues when interacting with C APIs
That's what I meant but poorly expressed (e.g. C++ <=> C interop has roughly similar problems as Rust <=> C interop).
That's what I meant but poorly expressed (e.g. C++ <=> C interop has roughly similar problems as Rust <=> C interop).
> Already overworked kernel maintainers will have to find time to learn Rust well enough to manage it within their subsystems.
Well sure, but only if you explicitly reject any help and make up empty claims about impossibility of maintenance like Hellwig does
Well sure, but only if you explicitly reject any help and make up empty claims about impossibility of maintenance like Hellwig does
Seems like rfl people should do the work downstream like the rt folks did for a while. Maybe be an upstream for Android and the various corporate kernels.
Linus can decide whether to pull from their versions or not but basically maybe they should run a bit freer than having to deal with these maintainers.
A soft fork might clarify things like the old gcc fork did.
Linus can decide whether to pull from their versions or not but basically maybe they should run a bit freer than having to deal with these maintainers.
A soft fork might clarify things like the old gcc fork did.
> Maybe be an upstream for Android and the various corporate kernels.
> A soft fork might clarify things like the old gcc fork did.
Nice idea but in practice there is absolutely no chance that this would be a successful strategy for those people in favor of Rust.
Being out of mainline tree would perpetually put Rust in the "future" (like Nuclear Fusion -- always just "20 years away"). Meanwhile Linux would continue to evolve in ways that could be very disruptive for Rust integration. For example, Rust interop needs to think very carefully about locks, memory order etc. Any change in the mainline kernel could really mess the assumptions made in the Rust fork. A year or two of evolution of the slightly independent evolution of the two branches may make it almost impossible to merge the two.
Only companies as large as Google could maintain the Android fork. The RFL team is small and I don't think large vendors would like to invest their time and energy on forks. Who would pick up and package the fork ? Red Hat ? Debian ? Arch ? They are already complaining about compilation and maintainability of Rust when it is in the mainline kernel. No chance that they start using the down stream rust kernel in their main kernels.
The only way Rust for Linux succeeds is if it has the blessing and push from Greg/Linus.
> A soft fork might clarify things like the old gcc fork did.
Nice idea but in practice there is absolutely no chance that this would be a successful strategy for those people in favor of Rust.
Being out of mainline tree would perpetually put Rust in the "future" (like Nuclear Fusion -- always just "20 years away"). Meanwhile Linux would continue to evolve in ways that could be very disruptive for Rust integration. For example, Rust interop needs to think very carefully about locks, memory order etc. Any change in the mainline kernel could really mess the assumptions made in the Rust fork. A year or two of evolution of the slightly independent evolution of the two branches may make it almost impossible to merge the two.
Only companies as large as Google could maintain the Android fork. The RFL team is small and I don't think large vendors would like to invest their time and energy on forks. Who would pick up and package the fork ? Red Hat ? Debian ? Arch ? They are already complaining about compilation and maintainability of Rust when it is in the mainline kernel. No chance that they start using the down stream rust kernel in their main kernels.
The only way Rust for Linux succeeds is if it has the blessing and push from Greg/Linus.
Sounds like this needs a higher level decision to overrule that resistance. Those who refuse it are too late though, it was already decided to accept Rust in the kernel in general, so they should accommodate it otherwise such kind of attitude will only slow down the progress.
The scenario where someone wants to drive by dump some code, then you reject it, then they offer to maintain it, kind of sucks. Ive had this happen in the past, at least in my experience you can't force someone to keep maintaining it. When they offer to maintain it, they look like such a great collaborator or team mate, you look like shit for saying no still. What happens is someone quits, gets reorged, your oncall and they aren't so your fixing and supporting it anyway. If they do stick around, now there is communication overhead from a part time maintainer
I get kernel development is a different environment then corporate programming. Im not 100% what the deal is with the Rust for Linux people. but in a vacuum, I am sympathetic to reasons like maintenance complexity for rejecting changes
I get kernel development is a different environment then corporate programming. Im not 100% what the deal is with the Rust for Linux people. but in a vacuum, I am sympathetic to reasons like maintenance complexity for rejecting changes
> The scenario where someone wants to drive by dump some code, then you reject it, then they offer to maintain it, kind of sucks.
That's not what happened here.
This was a maintainer that was CCd on a Rust wrapper to a subsystem he maintains as a courtesy, immediately replying that that code shouldn't live in that subsystem and blocking the patch set from being merged by NAking, but the change was already not in the subsystem but rather the rust subdirectory, which gives the impression he didn't read the patch to begin with. After he was told this, he said that having multiple languages is a cancer that would kill the maintainability of Linux and he would block it any way he could.
That's a very different situation to the one you describe.
That's not what happened here.
This was a maintainer that was CCd on a Rust wrapper to a subsystem he maintains as a courtesy, immediately replying that that code shouldn't live in that subsystem and blocking the patch set from being merged by NAking, but the change was already not in the subsystem but rather the rust subdirectory, which gives the impression he didn't read the patch to begin with. After he was told this, he said that having multiple languages is a cancer that would kill the maintainability of Linux and he would block it any way he could.
That's a very different situation to the one you describe.
The caller takes ownership of the returned resources, i.e., will have the responsibility in calling `bindings::dma_free_attrs`.
> doesn't this violate RAII?
It's C, you don't get any RAII.
It's C, you don't get any RAII.
Not the Rust abstraction. If mutation is needed, you'd write closures into the interface, or provider borrows. It's definitely weird to barf out ownership of half of the fields.
E: I guess it sort of depends whether the handle is Copy semantics or not.
E2: I guess they're correctly leaking the resource, but I question whether they should, protections like that are kind of the whole point of using Rust to begin with.
E: I guess it sort of depends whether the handle is Copy semantics or not.
E2: I guess they're correctly leaking the resource, but I question whether they should, protections like that are kind of the whole point of using Rust to begin with.
No, but you get defer via compiler extensions, and maybe in C2y as standard language feature.
This is similar to `Box::into_raw` in Rust `std`. It destructs the RAII struct into a raw pointer. After that it's pretty much expected that you are responsible for cleanup.
Seems like the preferred interface would be a purely functional block that provides the access.
Obviously forcing callers to free means adding lints and docs. Returning borrows leaves the likelihood of entering into explicit lifetimes hell and such in calling code.
A functional interface like (paraphrasing)
Obviously forcing callers to free means adding lints and docs. Returning borrows leaves the likelihood of entering into explicit lifetimes hell and such in calling code.
A functional interface like (paraphrasing)
// pub fn apply_op(&mut self, FnMut(&mut T, &bindings::dma_addr_t... ))
As most of the kernel is in C, they probably need a raw pointer quite regularly to pass it into existing C APIs.
Just like `Box::into_raw`, this method shouldn't be used in pure Rust code. It's only available for C FFI.
Just like `Box::into_raw`, this method shouldn't be used in pure Rust code. It's only available for C FFI.
Right but, this isn't merely FFI code, those are via Bindgen, this is the abstraction intended to be provided to all DMA driver authors as the API, as I understand it.
The method is called `CoherentAllocation::into_parts`. And it takes self by value, which means the struct is consumed (this is implied by the into_ prefix). Of course you would either reconstruct the allocation or free the allocation yourself after this.
See Vec::into_raw_parts or Box::into_raw for an stdlib analogy.
See Vec::into_raw_parts or Box::into_raw for an stdlib analogy.
Can someone explain what NAking means in this context ?
I didn't find any explanation online.
I didn't find any explanation online.
This reporting feels a bit slanted. The author implies the big blockers are human beings; specific people & interactions he's called out for seemingly stalling the project.
Personally, I'm curious if there's hard technical blockers. Like, if there's features that Linux (or Rust) needs which could incur a serious burden that no team can sustainably maintain. That kind of reporting— deep insights into technical trade-offs— would be much more interesting than a play-by-play of drama.
On a side note, there's one paper on Rust in Linux that's been recommended but I've not seen it deeply discussed yet https://www.usenix.org/conference/atc24/presentation/li-hong...
Personally, I'm curious if there's hard technical blockers. Like, if there's features that Linux (or Rust) needs which could incur a serious burden that no team can sustainably maintain. That kind of reporting— deep insights into technical trade-offs— would be much more interesting than a play-by-play of drama.
On a side note, there's one paper on Rust in Linux that's been recommended but I've not seen it deeply discussed yet https://www.usenix.org/conference/atc24/presentation/li-hong...
The uncharitable take is that it's just job protection. That is current maintainer doesn't want to be replaced with some new Rust-writing folks.
A more charitable take is that the current maintainer is worried not about extra work now, but later when more things use Rust.
For example, say the Rust changes were merged and maintained by RfL folks. Then say NVIDIA replaces their current GPU drivers with a new Rust-based drivers.
Later a change to the DMA layer kernel code breaks the Rust code in a way that's not trivially fixable, and with that no NVIDIA GPU drivers.
Would the DMA changes still be allowed to be merged without any additional work by the current DMA layer maintainer?
This seems like something that could happen down the line, if the RfL project continues.
A more charitable take is that the current maintainer is worried not about extra work now, but later when more things use Rust.
For example, say the Rust changes were merged and maintained by RfL folks. Then say NVIDIA replaces their current GPU drivers with a new Rust-based drivers.
Later a change to the DMA layer kernel code breaks the Rust code in a way that's not trivially fixable, and with that no NVIDIA GPU drivers.
Would the DMA changes still be allowed to be merged without any additional work by the current DMA layer maintainer?
This seems like something that could happen down the line, if the RfL project continues.
> That is current maintainer doesn't want to be replaced with some new Rust-writing folks.
Almost anyone paying Christoph is not doing so because of his involvement in the DMA subsystem - he's got a sufficiently strong involvement in any number of core parts of the kernel that if he dropped all involvement in DMA he'd still have enough employment opportunities to spend as much of his time skiing as he wanted.
Almost anyone paying Christoph is not doing so because of his involvement in the DMA subsystem - he's got a sufficiently strong involvement in any number of core parts of the kernel that if he dropped all involvement in DMA he'd still have enough employment opportunities to spend as much of his time skiing as he wanted.
> Personally, I'm curious if there's hard technical blockers.
That LWN or the kernel devs are keeping secret? I would be amazed why, if Christoph Hellwig had technical blockers, he wouldn't raise them in the LKML, right now, of all times.
That LWN or the kernel devs are keeping secret? I would be amazed why, if Christoph Hellwig had technical blockers, he wouldn't raise them in the LKML, right now, of all times.
The author has been professionally writing about the Linux kernel for close to 30 years. He's also maintaining the Linux kernel documentation.
> Personally, I'm curious if there's hard technical blockers.
Not really, apart from the sheer size of Linux.
> Personally, I'm curious if there's hard technical blockers.
Not really, apart from the sheer size of Linux.
[deleted]
Putting the mappings in the driver that needs them has no effect on the maintainability of the kernel.
Putting them in the kernel has an effect if the maintainability of the kernel.
Putting them in the kernel has an effect if the maintainability of the kernel.
The kernel maintains in-tree drivers that break, so it does impact kernel maintenance. This isn't currently true for RfL, but in the case of a C driver anyone who wanted to go and make a breaking change would have to go and fix up breakages to merge.
Nitpick: There are actually a handful Rust drivers in-tree, e.g. https://github.com/torvalds/linux/blob/master/drivers/net/ph...
Though, I think all of them are direct ports of existing drivers.
Though, I think all of them are direct ports of existing drivers.
If it’s a common abstraction then wouldn’t it be more work to maintain the same piece of code in many different drivers?
matt3210(1)
Linux is a unix-like kernel written in C, with no stable internal APIs.
Introducing another programming language was never going to work. Not without ample support within the existing developer base.
The kind thing would have been to reject Rust from the start.
We're too late now. The next best option would be to remove it from the kernel entirely.
Remarkably, nothing stops Rust developers from forking the kernel or, even better, writing their own system from scratch.
Introducing another programming language was never going to work. Not without ample support within the existing developer base.
The kind thing would have been to reject Rust from the start.
We're too late now. The next best option would be to remove it from the kernel entirely.
Remarkably, nothing stops Rust developers from forking the kernel or, even better, writing their own system from scratch.
So far, there's no proof that it "was never going to work", and ample circumstantial evidence that it does, in fact, work.
There have been no serious technical obstacles, and no actual incidents where maintenance has become harder because of it.
This looks 100% like personal resistance, rather than technical.
There have been no serious technical obstacles, and no actual incidents where maintenance has become harder because of it.
This looks 100% like personal resistance, rather than technical.
>This looks 100% like personal resistance, rather than technical.
>there's no proof that it "was never going to work"
Did you really just skim the post you replied to? It said:
>Not without ample support within the existing developer base.
Should be clear enough, but I will expand: This is about pre-existing Linux developers and maintainers not willing to commit to the new language, and thus it has nothing to do with anything technical at all.
It won't work because it absolutely requires ample support within Linux's developer base to succeed, and such support is not there.
Nearly nobody, even within the core (which support is needed the most!), is willing to put any effort into this.
I will re-state what I already said:
The best and kindest path Linus could have taken would have been to firmly reject the new language from the start.
As that is no longer possible, the second best still is: To declare the experiment a failure and conclusively remove the new language from the kernel.
The resources freed in Linux's and Rust's communities could then be put to better use, and after a while everybody would be a little less miserable.
>there's no proof that it "was never going to work"
Did you really just skim the post you replied to? It said:
>Not without ample support within the existing developer base.
Should be clear enough, but I will expand: This is about pre-existing Linux developers and maintainers not willing to commit to the new language, and thus it has nothing to do with anything technical at all.
It won't work because it absolutely requires ample support within Linux's developer base to succeed, and such support is not there.
Nearly nobody, even within the core (which support is needed the most!), is willing to put any effort into this.
I will re-state what I already said:
The best and kindest path Linus could have taken would have been to firmly reject the new language from the start.
As that is no longer possible, the second best still is: To declare the experiment a failure and conclusively remove the new language from the kernel.
The resources freed in Linux's and Rust's communities could then be put to better use, and after a while everybody would be a little less miserable.
So far, the only actual support that has been required is to not actively sabotage it. That’s not a high bar.
There does seem to be very substantial interest in RfL in the kernel, with the exception of a few maintainers generating a lot of personal friction.
Frankly, it’s exactly the kind of behavior you would be very worried to see from key people maintaining the most important operating system on the planet.
My understanding is that part of the reason Linus approved the project is that the current state of Linux is unsustainable. I agree with him. Linux will evolve, or die. Gradually moving on from C will probably be part of that journey, sooner or later.
There does seem to be very substantial interest in RfL in the kernel, with the exception of a few maintainers generating a lot of personal friction.
Frankly, it’s exactly the kind of behavior you would be very worried to see from key people maintaining the most important operating system on the planet.
My understanding is that part of the reason Linus approved the project is that the current state of Linux is unsustainable. I agree with him. Linux will evolve, or die. Gradually moving on from C will probably be part of that journey, sooner or later.
>There does seem to be very substantial interest in RfL in the kernel
Citation needed.
>the current state of Linux is unsustainable.
I agree. And the issues are fundamental, to do with lack of structure. Doing anything in Linux is harder (in man hours) than doing it on a well-structured system. By orders of magnitude.
But I do not see how to "evolve" a project that big (MLoCs) from chaos towards structure.
Rather, I feel that a rewrite is a much better path.
It could be a fundamentally much better design. A microkernel, multiserver design.
Such efforts include:
seL4 (a formally verified microkernel that also happens to be the fastest available) and multiserver efforts on top of it such as LionsOS or Genode.
Redox (a multiserver OS with its own microkernel, written in Rust).
Citation needed.
>the current state of Linux is unsustainable.
I agree. And the issues are fundamental, to do with lack of structure. Doing anything in Linux is harder (in man hours) than doing it on a well-structured system. By orders of magnitude.
But I do not see how to "evolve" a project that big (MLoCs) from chaos towards structure.
Rather, I feel that a rewrite is a much better path.
It could be a fundamentally much better design. A microkernel, multiserver design.
Such efforts include:
seL4 (a formally verified microkernel that also happens to be the fastest available) and multiserver efforts on top of it such as LionsOS or Genode.
Redox (a multiserver OS with its own microkernel, written in Rust).
They could also revise how their C interop works such that the kernel internal interfaces can continue to be specified in C and just used easily from Rust. This is the Swift approach: Consistent, predictable interop that doesn’t insist or presume that the new way is the preferred way.
The kernel internal interfaces have no information about the expected semantics of the pointers passed in (i.e. for how much are they valid, who's responsible for "free"ing them, with what, etc etc). You have to specify those informations somewhere, and this is currently done when writing Rust bindings. If you want to avoid doing that then you'll just end up writing C in Rust with unsafe everywhere.
Yes, and that’s just what has to be done. Welcome to interfacing with C, it’s been like that for decades.
There are ways these semantics can be annotated via the attribute mechanism in many compilers; this, like nullability, should be added just to improve the ability to statically analyze the existing code and avoid bugs. It can also happen to improve importability of C APIs in other languages, but the primary reason is and should be to make the C code better.
So you write unsafe-to-safe wrapper code for anything you need to interact with—once—so as to encode the interface semantics by hand. Eventually it’s no longer needed because you’ve added annotations to the headers instead. And the kernel gets _better_ because these things can be statically checked and thus have bugs avoided, _regardless of_ the use of Rust.
There are ways these semantics can be annotated via the attribute mechanism in many compilers; this, like nullability, should be added just to improve the ability to statically analyze the existing code and avoid bugs. It can also happen to improve importability of C APIs in other languages, but the primary reason is and should be to make the C code better.
So you write unsafe-to-safe wrapper code for anything you need to interact with—once—so as to encode the interface semantics by hand. Eventually it’s no longer needed because you’ve added annotations to the headers instead. And the kernel gets _better_ because these things can be statically checked and thus have bugs avoided, _regardless of_ the use of Rust.
C doesn't have syntax/typesystem to express ownership, lifetimes and thread safety, but Rust does.
Personally I'd love if C was extended to be able define APIs more precisely, but that's not a realistic option at the moment. There isn't even any vendor-specific solution to this in C (the hodgepodge of attributes barely scratches the surface). Linux would have to invent its own interface definition language and make C and Rust adopt it. That means more changes for the C maintainers.
The C standard moves very slowly, and avoids making non-trivial changes. I'm still waiting for slices (pointer+length type). Rust is here today, and won't stop for a dream of C that may or may not happen in 2050.
Personally I'd love if C was extended to be able define APIs more precisely, but that's not a realistic option at the moment. There isn't even any vendor-specific solution to this in C (the hodgepodge of attributes barely scratches the surface). Linux would have to invent its own interface definition language and make C and Rust adopt it. That means more changes for the C maintainers.
The C standard moves very slowly, and avoids making non-trivial changes. I'm still waiting for slices (pointer+length type). Rust is here today, and won't stop for a dream of C that may or may not happen in 2050.
C has been extended in *exactly* that way using the `__((attribute))__` mechanism in GCC and clang. One does not need to wait for a standards body to do this.
This is one of the ways in which the static analyzer in clang works and they’re very open to further such extensions, which can be easily wrapped in macros so as to not affect compilers that don’t support them.
Do not let the perfect be the enemy of the good. It’s absolutely a realistic option, and it’s one that should be pursued if kernel code quality checking via static analysis is important.
This is one of the ways in which the static analyzer in clang works and they’re very open to further such extensions, which can be easily wrapped in macros so as to not affect compilers that don’t support them.
Do not let the perfect be the enemy of the good. It’s absolutely a realistic option, and it’s one that should be pursued if kernel code quality checking via static analysis is important.
Nothing in GCC or clang that exists today is anywhere near the level of expressiveness that Rust uses in its APIs (e.g. lifetimebound in [1]).
The current problem is not technical, it's maintainers not wanting to even think about another language in their C codebase. Adding non-standard foreign language's semantics on top of C, not supported natively by C compilers, and forced to have an even uglier syntax than Rust's own, would not make C maintainers happy.
You could add an __((attribute))__ that's equivalent to a code comment containing a Rust type, but that also would be strictly worse: less useful to Rust devs, still annoying to C maintainers who wouldn't want to maintain it nor worry about the attributes breaking another compiler for a different language.
I don't think it's feasible to use attributes to add functionality that actually does something meaningful, and doesn't look comically bad at the same time. Look at the zoo of annotations that clang had to add to work around lack of a built-in slice type in C, and that's just a simple thing that's written `[T]` in Rust: https://clang.llvm.org/docs/BoundsSafety.html
Lifetime annotations are annoyingly noisy and viral even in Rust itself, which has a first-class syntax for them, and generics and type inference to hide the noise. In C they'd be even noisier, and had to be added in more places.
You can't really take safety annotations that are for Rust and reuse them for static analysis of C code itself. They're for a boundary between languages, not for the C implementation. Rust itself is essentially a static analyzer, with everything in the language designed from the ground up for the static analyzer. C wasn't designed for that. If you transplant enough of Rust's requirements to C, you'll make C maintainers write Rust, but using a botched C syntax and a C compiler that doesn't understand the actual language they're writing.
Static analysis is fundamentally limited by C's semantics (limited by undecidability, not mere engineering challenges of implementing a Sufficiently Smart static analyzer). It's not something that can be easily solved with an attribute here and there, because that will be brittle and incomplete[2][3]. Static analysis at that level requires removing a lot of flexibility from C and adding new formalisms, which again would not make C maintainers happy who already resist even tiniest changes of their code to align better with non-C's restrictions.
[1]: https://docs.google.com/document/d/e/2PACX-1vRZr-HJcYmf2Y76D...
[2]: https://www.circle-lang.org/draft-profiles.html [3]: https://safecpp.org/draft-lifetimes.html
The current problem is not technical, it's maintainers not wanting to even think about another language in their C codebase. Adding non-standard foreign language's semantics on top of C, not supported natively by C compilers, and forced to have an even uglier syntax than Rust's own, would not make C maintainers happy.
You could add an __((attribute))__ that's equivalent to a code comment containing a Rust type, but that also would be strictly worse: less useful to Rust devs, still annoying to C maintainers who wouldn't want to maintain it nor worry about the attributes breaking another compiler for a different language.
I don't think it's feasible to use attributes to add functionality that actually does something meaningful, and doesn't look comically bad at the same time. Look at the zoo of annotations that clang had to add to work around lack of a built-in slice type in C, and that's just a simple thing that's written `[T]` in Rust: https://clang.llvm.org/docs/BoundsSafety.html
Lifetime annotations are annoyingly noisy and viral even in Rust itself, which has a first-class syntax for them, and generics and type inference to hide the noise. In C they'd be even noisier, and had to be added in more places.
You can't really take safety annotations that are for Rust and reuse them for static analysis of C code itself. They're for a boundary between languages, not for the C implementation. Rust itself is essentially a static analyzer, with everything in the language designed from the ground up for the static analyzer. C wasn't designed for that. If you transplant enough of Rust's requirements to C, you'll make C maintainers write Rust, but using a botched C syntax and a C compiler that doesn't understand the actual language they're writing.
Static analysis is fundamentally limited by C's semantics (limited by undecidability, not mere engineering challenges of implementing a Sufficiently Smart static analyzer). It's not something that can be easily solved with an attribute here and there, because that will be brittle and incomplete[2][3]. Static analysis at that level requires removing a lot of flexibility from C and adding new formalisms, which again would not make C maintainers happy who already resist even tiniest changes of their code to align better with non-C's restrictions.
[1]: https://docs.google.com/document/d/e/2PACX-1vRZr-HJcYmf2Y76D...
[2]: https://www.circle-lang.org/draft-profiles.html [3]: https://safecpp.org/draft-lifetimes.html
The same people that are against documenting their APIs when requested by the RfL project would likely reject the idea of adding annotations to C code for (in their eyes) only benefiting RfL.
Nevertheless, it should be pursued because it also benefits the C code when combined with static analysis.
Maybe I'm missing something, but I have yet to see concrete proposals for annotations that rival what Rust can express.
Maybe you’re too focused on “rival[ing] what Rust can express” and aren’t looking for “better than plain C in a way that improves security and analyzability and would also improve Rust import of C APIs.”
As an example, there are annotation schemes for pairing a size with a buffer pointer, and for indicating a pointer’s ownership transfer. These can then be statically checked, reducing bug risk.
These can also _improve_ interoperability with higher level languages that have such semantics, even if they don’t offer the _complete_ range of semantics possible in those higher level languages.
I don’t have links to share at the moment, but this is ongoing work in places like the clang community.
As an example, there are annotation schemes for pairing a size with a buffer pointer, and for indicating a pointer’s ownership transfer. These can then be statically checked, reducing bug risk.
These can also _improve_ interoperability with higher level languages that have such semantics, even if they don’t offer the _complete_ range of semantics possible in those higher level languages.
I don’t have links to share at the moment, but this is ongoing work in places like the clang community.
If these annotations are so powerful why haven't they been adopted in all these years then? Why mention them now as an alternative to Rust? And why not try both and see what sticks?
[deleted]
This might have some technical merits, but that ship has already sailed when Linus decided to merge R4L into the kernel. If Hellwig wants to reverse that decision, he'll have to bring it to Linus, instead he's trying to block R4L patches and waste everyone's time and energy.