Why Rust helps even if you have to use a lot of `unsafe`(v5.chriskrycho.com)
v5.chriskrycho.com
Why Rust helps even if you have to use a lot of `unsafe`
https://v5.chriskrycho.com/journal/unsafe/
178 comments
Even of the error originates in unsafe code, the UB can manifest elsewhere. The optimizer makes assumptions that you can easily invalidate in an unsafe block leading to UB. In many ways it is mich easier to introduce UB in an umsafe block than it is in C++. But at the very least you know the UB originated in one of your unsafe blocks. Its just figuring out which one. I very rarely use unsafe outside bindings though, so that helps too.
The nice thing is that you can check the invariants at some reasonable module boundaries. The bugs in unsafe code can manifest elsewhere, but at least, when wrapped properly, someone _else's_ bugs won't cause your unsafe code to have UB.
The problem I personally have is the "when wrapped properly". Functionally it's the same issue as "just write correct C code", but the invariants that have to hold in Rust aren't as well defined (because there's more of them). Writing "correct" `unsafe` code is a lot harder than it seems on the surface.
There's also the issue that I've seen it commonly advocated to keep `unsafe` blocks small - which makes intuitive sense, but brings into question whether `unsafe` is supposed to _just_ mark the unsafe operations being done, or mark all the code under which certain assumptions have to hold for UB to be avoided and it be 'safe'. It's been a little while since I looked into it, but there wasn't much agreement on this the last time I looked and I've seen people do it both ways.
There's also the issue that I've seen it commonly advocated to keep `unsafe` blocks small - which makes intuitive sense, but brings into question whether `unsafe` is supposed to _just_ mark the unsafe operations being done, or mark all the code under which certain assumptions have to hold for UB to be avoided and it be 'safe'. It's been a little while since I looked into it, but there wasn't much agreement on this the last time I looked and I've seen people do it both ways.
Have you written unsafe code in Rust? I have some experience writing unsafe code, and most of the time you can do it without problems, wrapped in sensible safe APIs that don't allow unsafety to leak. Of course, this needs experience and knowledge about API design and the invariants that must be followed... and preferably another set of eyes.
The thing is that once you've managed to do that, the _consumers_ of your API are free from worries. The situation becomes a whole lot better than the equivalent C situation, where both sides of the API need to be careful about what funny abominations the another side might devise.
The thing is that once you've managed to do that, the _consumers_ of your API are free from worries. The situation becomes a whole lot better than the equivalent C situation, where both sides of the API need to be careful about what funny abominations the another side might devise.
> Have you written unsafe code in Rust? I have some experience writing unsafe code, and most of the time you can do it without problems
Yes, I've written a decent amount of `unsafe` code. And I would also point out, C programmers say the same thing about their code :P The problem comes when an assumption you thought was ok turns out to be something the compiler writers thought you shouldn't do, and then your code is broken. Rust doesn't have a defined list of invariants you can't break in `unsafe`, which makes it pretty hard to be confident you didn't break one of them. I've loosely followed the work to pin down some of these things[0], but I don't personally feel like tons of progress has happened and they've been working on it for years.
That said don't get me wrong, I like Rust, I just wish the `unsafe` situation was improved. I also agree that creating safe APIs around `unsafe` code is the way to go, but I've seen a whole bunch of people get it wrong due to bad advice given around `unsafe` or lack of clarity around what `unsafe` is actually supposed to mean.
`unsafe` has also IMO been overloaded to mean both "this code does one of the unsafe operations" or "this code has some invariant you must keep". The latter tends to be the better way to go for creating APIs, but is also harder to do and ends up causing you to mark completely safe code with an `unsafe`.
[0]: https://github.com/rust-lang/unsafe-code-guidelines
Yes, I've written a decent amount of `unsafe` code. And I would also point out, C programmers say the same thing about their code :P The problem comes when an assumption you thought was ok turns out to be something the compiler writers thought you shouldn't do, and then your code is broken. Rust doesn't have a defined list of invariants you can't break in `unsafe`, which makes it pretty hard to be confident you didn't break one of them. I've loosely followed the work to pin down some of these things[0], but I don't personally feel like tons of progress has happened and they've been working on it for years.
That said don't get me wrong, I like Rust, I just wish the `unsafe` situation was improved. I also agree that creating safe APIs around `unsafe` code is the way to go, but I've seen a whole bunch of people get it wrong due to bad advice given around `unsafe` or lack of clarity around what `unsafe` is actually supposed to mean.
`unsafe` has also IMO been overloaded to mean both "this code does one of the unsafe operations" or "this code has some invariant you must keep". The latter tends to be the better way to go for creating APIs, but is also harder to do and ends up causing you to mark completely safe code with an `unsafe`.
[0]: https://github.com/rust-lang/unsafe-code-guidelines
"When wrapped properly" is a lot more likely to happen in Rust because idiomatic Rust involves things like proper error handling and returning Result for fallible operations. I use libraries that have gobs of unsafe all the time. Wgpu is basically an unsafe code factory because it glues Rust to C and C++ but all of its unsafe code returns Options and Results because Rust programmers get them all drilled into them by the community and all the learning resources said community provides.
The general rule of thumb is to ensure you've returned to safety guarantees before exiting an unsafe block. Obviously it's too much to expect humans to be perfect at this, but having that mentality makes it orders of magnitude harder to leak UB into safe code. The overwhelming majority of violations I've seen are where engineers dip in and out of unsafe code without upholding safety guarantees inbetween.
The one I usually see is an unsafe method leaving memory aliased.
Can you share an example so I know what to look for?
If an unsafe function borrows an input make sure it doesn't return an aliased reference to the same data.
I'm curious if a linter + the new C23 annotation syntax could be used for a similar purpose in C?
EDIT: I'm rate limited. If we assume an `[[unsafe]]` attribute would only apply to things like pointer arithmetic and unchecked array access (not lifetimes), could such a linter exist?
EDIT: I'm rate limited. If we assume an `[[unsafe]]` attribute would only apply to things like pointer arithmetic and unchecked array access (not lifetimes), could such a linter exist?
Unfortunately the attributes are not nearly rich enough. There are many proposals for massively increasing the information that you can give C compilers, such as the lifetime safety profile and n3005, but it's a very difficult problem to do this in a language that makes pointer aliasing and pointer arithmetic very easy.
n3005: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3005.pdf
n3005: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3005.pdf
A point in support of this stance is that the entire standard library is a safe wrapper over unsafe code. Vec does fiddly things with pointers and memory management so that you don't have to!
This article could be 1 sentence: “it has sum types” :)
I find programming in a language without sum types is like rock climbing in ankle weights. I just can’t model many types of data correctly, and I hate using product types for something that is clearly a sum type.
Even if Rust didn’t make any safety guarantees over other languages (although sum types provide safety guarantees through exhaustiveness!) I would still think Rust helps a lot because of that.
I guess Swift and Kotlin are basically “it has sum types” languages too, although they both come with baggage (Apple-only in practice, JVM).
I find programming in a language without sum types is like rock climbing in ankle weights. I just can’t model many types of data correctly, and I hate using product types for something that is clearly a sum type.
Even if Rust didn’t make any safety guarantees over other languages (although sum types provide safety guarantees through exhaustiveness!) I would still think Rust helps a lot because of that.
I guess Swift and Kotlin are basically “it has sum types” languages too, although they both come with baggage (Apple-only in practice, JVM).
Sum types are fantastic and I’ll join you singing their praises all day long. But this article is about rust’s memory model & memory safety - which is independent of sum types.
Unsafe blocks in rust are the only place you can dereference pointers. There are whole classes of memory bugs which are impossible to code up in safe rust code because of this.
As an example of how this helps, a couple years ago I was working on a complex & clever rope data structure. My code segfaulted - and I can count on one hand the number of times my rust code has ever done that. A segfault like that can only come from memory corruption or dereferencing invalid memory - both of which are only possible in unsafe code. And the bottom of the stack trace was a safe function, so I knew I’d corrupted memory somehow. These bugs are really hard to find in C - basically the worst case for debugging because the problem could be literally anywhere. But in rust I knew the bug was in one of only two unsafe blocks in the ~thousand or so lines of my library. And one of those blocks was executed not long before the crash.
5 minutes later I had found and fixed the bug. In C, I’ve had similar problems take hours to days to fix.
Bugs like this aren’t possible at all in Go or C# because of their memory model. But in those languages you pay for safety by sacrificing performance. The rust I’ve written & benchmarked so far somehow faster than the equivalent C code. Which is pretty wild!
Unsafe blocks in rust are the only place you can dereference pointers. There are whole classes of memory bugs which are impossible to code up in safe rust code because of this.
As an example of how this helps, a couple years ago I was working on a complex & clever rope data structure. My code segfaulted - and I can count on one hand the number of times my rust code has ever done that. A segfault like that can only come from memory corruption or dereferencing invalid memory - both of which are only possible in unsafe code. And the bottom of the stack trace was a safe function, so I knew I’d corrupted memory somehow. These bugs are really hard to find in C - basically the worst case for debugging because the problem could be literally anywhere. But in rust I knew the bug was in one of only two unsafe blocks in the ~thousand or so lines of my library. And one of those blocks was executed not long before the crash.
5 minutes later I had found and fixed the bug. In C, I’ve had similar problems take hours to days to fix.
Bugs like this aren’t possible at all in Go or C# because of their memory model. But in those languages you pay for safety by sacrificing performance. The rust I’ve written & benchmarked so far somehow faster than the equivalent C code. Which is pretty wild!
> These bugs are really hard to find in C - basically the worst case for debugging because the problem could be literally anywhere.
> 5 minutes later I had found and fixed the bug. In C, I’ve had similar problems take hours to days to fix.
Modern tooling like a memory synthesizer and good debugger makes this kind of debugging relatively easy.
In some case, the bug would have been detected even before you run with proper warnings.
> Bugs like this aren’t possible at all in Go or C# because of their memory model.
Except in their respective FFI or unsafe block of code.
> But in those languages you pay for safety by sacrificing performance.
That's a bit of a strong statement. Even if we accept (i don't) the performance aspect, GC languages give more safety,convenience and dev prod. than rust and at a significantly less complexity overload.
> As an example of how this helps, a couple years ago I was working on a complex & clever rope data structure. My code segfaulted - and I can count on one hand the number of times my rust code has ever done that.
This might simply be an indication that you were using the wrong tools in the first language. Most native languages (D, C++ etc...) have facilities to avoid unsafe behavior. Use them.
> 5 minutes later I had found and fixed the bug. In C, I’ve had similar problems take hours to days to fix.
Modern tooling like a memory synthesizer and good debugger makes this kind of debugging relatively easy.
In some case, the bug would have been detected even before you run with proper warnings.
> Bugs like this aren’t possible at all in Go or C# because of their memory model.
Except in their respective FFI or unsafe block of code.
> But in those languages you pay for safety by sacrificing performance.
That's a bit of a strong statement. Even if we accept (i don't) the performance aspect, GC languages give more safety,convenience and dev prod. than rust and at a significantly less complexity overload.
> As an example of how this helps, a couple years ago I was working on a complex & clever rope data structure. My code segfaulted - and I can count on one hand the number of times my rust code has ever done that.
This might simply be an indication that you were using the wrong tools in the first language. Most native languages (D, C++ etc...) have facilities to avoid unsafe behavior. Use them.
I've never heard of memory synthesizers.
I'll cede the point that some of Rust's advantages are merely psychological. But I would argue that most of us are tired, frustrated, and working on teams where we have neither the energy nor authority nor teaching experience to make everyone learn the right tools. I've had to teach some coworkers how to use a profiler. And then remind them to use it.
Therefore mere psychological benefits still improve outcomes.
It's easier to memorize "cargo run" than to memorize the vocabulary quizzes for Python or Kotlin tooling. (A gradle is used to serve soup. A wheel is part of a car.) Rust produces one binary by default, and I don't even have to think about how to bundle a runtime. That means I also don't need to teach anyone what Docker is.
I have a hunch this actually matters. Everyone who's learning to program right now will be deep in their work in 20 years. So every software that's succeeded in my lifetime - JavaScript, Python, Unity 3D, etc., has succeeded by appealing to noobies and then lasting long enough that everyone likes it. In the very long term, a low barrier to entry might matter as much as a high skill ceiling.
I'll cede the point that some of Rust's advantages are merely psychological. But I would argue that most of us are tired, frustrated, and working on teams where we have neither the energy nor authority nor teaching experience to make everyone learn the right tools. I've had to teach some coworkers how to use a profiler. And then remind them to use it.
Therefore mere psychological benefits still improve outcomes.
It's easier to memorize "cargo run" than to memorize the vocabulary quizzes for Python or Kotlin tooling. (A gradle is used to serve soup. A wheel is part of a car.) Rust produces one binary by default, and I don't even have to think about how to bundle a runtime. That means I also don't need to teach anyone what Docker is.
I have a hunch this actually matters. Everyone who's learning to program right now will be deep in their work in 20 years. So every software that's succeeded in my lifetime - JavaScript, Python, Unity 3D, etc., has succeeded by appealing to noobies and then lasting long enough that everyone likes it. In the very long term, a low barrier to entry might matter as much as a high skill ceiling.
> I've never heard of memory synthesizers.
I assume it's a typo for memory sanitizer (i.e., ASAN).
I assume it's a typo for memory sanitizer (i.e., ASAN).
> Even if we accept (i don't) the performance aspect, GC languages give more safety,convenience and dev prod. than rust and at a significantly less complexity overload
At the limit, I think the performance benefits of rust are pretty clear. The performance ceiling of rust seems to be pretty unambiguously higher than that of Go, Python, C#, javascript, etc. But yeah - no contest on rust being way harder to learn and use. I’ve been programming rust for years now, but I still reach for typescript when prototyping because I still find it much faster when making prototypes. Especially when async is involved.
> This might simply be an indication that you were using the wrong tools in the first language. Most native languages (D, C++ etc...) have facilities to avoid unsafe behavior. Use them.
Can you give some examples of those tools? The “first language” I mentioned above was C. What equivalent does C have to rust’s memory safety? Or C++ for that matter? Are smart pointers in C++ that ubiquitous and good that memory corruption is a thing of the past?
At the limit, I think the performance benefits of rust are pretty clear. The performance ceiling of rust seems to be pretty unambiguously higher than that of Go, Python, C#, javascript, etc. But yeah - no contest on rust being way harder to learn and use. I’ve been programming rust for years now, but I still reach for typescript when prototyping because I still find it much faster when making prototypes. Especially when async is involved.
> This might simply be an indication that you were using the wrong tools in the first language. Most native languages (D, C++ etc...) have facilities to avoid unsafe behavior. Use them.
Can you give some examples of those tools? The “first language” I mentioned above was C. What equivalent does C have to rust’s memory safety? Or C++ for that matter? Are smart pointers in C++ that ubiquitous and good that memory corruption is a thing of the past?
> Can you give some examples of those tools?
ASAN (address sanitizer), UBSAN (undefined behaviour sanitizer), Valgrind.
All of these only detect runtime violations, so they can find bugs but don't prove their absence. On top of that I got a sufficient number of false negative or even bugs that wouldn't reproduce/wouldn't give interesting information with these tools that I can confidently tell you they don't even approach the guarantees provided by Rust.
There are also static analyzer like cppcheck and clang tidy. They do find some useful issues, but they:
1. Have false positives 2. Are a hassle to add on top of existing projects 3. Have false negatives, so a code that passes is not necessarily devoid of UB.
> Are smart pointers in C++ that ubiquitous and good that memory corruption is a thing of the past?
Smart pointers in C++ effectively prevent some classes of double free errors and some classes of leaks (like all RAII, I might add). They do very little against use-after-free in the general sense, because as soon as you take an observer pointer/reference (the equivalent of a Rust borrow), you have to either:
1. Forfeit safety by handling its lifetime on your own 2. Forfeit performance by unnecessarily cloning data (and also forfeit correctness in some cases, as the "sea of shared pointers" approach of some codebases create object identity issues and memory cycles).
The standard library is also full of footguns that can cause UB, such as calling `vector::front()` on an empty vector, that come up a lot in practice. I think cppcheck was able to flag some of these but I'm unsure of this off the top of my head.
Lastly C++ has a slew of very ugly errors like ODR violations that do happen in practice (albeit rarely) and that are completely outside the purview of tools (the last one a former colleague wrote would generate completely garbage information in asan, and would cause valgrind itself to segfault. The error cost the afternoon of two engineers for what was a typo causing a namespace to be early closed in a file... The kind of abhorrent error that causes to avoid these languages as much as I can).
I developed in C++11 for almost 10 years, and I am now a professional Rust developer. Productivity, developer experience, and correctness are night and day, Rust's advantage.
ASAN (address sanitizer), UBSAN (undefined behaviour sanitizer), Valgrind.
All of these only detect runtime violations, so they can find bugs but don't prove their absence. On top of that I got a sufficient number of false negative or even bugs that wouldn't reproduce/wouldn't give interesting information with these tools that I can confidently tell you they don't even approach the guarantees provided by Rust.
There are also static analyzer like cppcheck and clang tidy. They do find some useful issues, but they:
1. Have false positives 2. Are a hassle to add on top of existing projects 3. Have false negatives, so a code that passes is not necessarily devoid of UB.
> Are smart pointers in C++ that ubiquitous and good that memory corruption is a thing of the past?
Smart pointers in C++ effectively prevent some classes of double free errors and some classes of leaks (like all RAII, I might add). They do very little against use-after-free in the general sense, because as soon as you take an observer pointer/reference (the equivalent of a Rust borrow), you have to either:
1. Forfeit safety by handling its lifetime on your own 2. Forfeit performance by unnecessarily cloning data (and also forfeit correctness in some cases, as the "sea of shared pointers" approach of some codebases create object identity issues and memory cycles).
The standard library is also full of footguns that can cause UB, such as calling `vector::front()` on an empty vector, that come up a lot in practice. I think cppcheck was able to flag some of these but I'm unsure of this off the top of my head.
Lastly C++ has a slew of very ugly errors like ODR violations that do happen in practice (albeit rarely) and that are completely outside the purview of tools (the last one a former colleague wrote would generate completely garbage information in asan, and would cause valgrind itself to segfault. The error cost the afternoon of two engineers for what was a typo causing a namespace to be early closed in a file... The kind of abhorrent error that causes to avoid these languages as much as I can).
I developed in C++11 for almost 10 years, and I am now a professional Rust developer. Productivity, developer experience, and correctness are night and day, Rust's advantage.
> At the limit, I think the performance benefits of rust are pretty clear. The performance ceiling of rust seems to be pretty unambiguously higher than that of Go, Python, C#, javascript, etc.
Not to be pedantic, but "at the limit" performance is not a very good metric. Programming is an engineering exercise, so IMO performance delta per unit of development work is better characterization if we are comparing programming language.
At the limit C# supports inline assembly, so at the limit it could be match rust pounds for pounds with enough efforts, but that's hardly a useful comparison.
In general, a lot of the modern performance differences between programming languages is caused (from the programming languages perspective) by the object model each programming language use. Python/Javascript are in a class of their own and i don't think belong in performance conversation against statically type languages. Can't comment on go cuz i don't know much it.
On C#, the memory model can be surprisingly close to rust. The combination of typed IR for efficient monomorphization + the the ability to use stack allocated struct, mean that C# code can be quite close to rust code. And then the differences are mainly a consequence of LLVM backend vs ruygenJIT or AOT.
To say nothing of the advantage that a GC can have by having a global view of memory allocation/deallocations.
Added to that the fact that the time a rust eng. is spending modeling his ownership relationships and converting then into something the borrow checker can/accept could be spend improving C# perf further.
> Can you give some examples of those tools? The “first language” I mentioned above was C. What equivalent does C have to rust’s memory safety? Or C++ for that matter? Are smart pointers in C++ that ubiquitous and good that memory corruption is a thing of the past?
I am not too familiar with C, but i don't a think any new tool is needed : wrap all the unsafe operation (naked array accesses, allocation etc...) in a safe API and only use the safe API...
In C++, its trivial to avoid memory unsafe operation between smart_pointer, &reference, for_each for iteration etc... etc... Manipulating raw memory in "modern" C++ is mostly a choice than a necessity these days.
Not to be pedantic, but "at the limit" performance is not a very good metric. Programming is an engineering exercise, so IMO performance delta per unit of development work is better characterization if we are comparing programming language.
At the limit C# supports inline assembly, so at the limit it could be match rust pounds for pounds with enough efforts, but that's hardly a useful comparison.
In general, a lot of the modern performance differences between programming languages is caused (from the programming languages perspective) by the object model each programming language use. Python/Javascript are in a class of their own and i don't think belong in performance conversation against statically type languages. Can't comment on go cuz i don't know much it.
On C#, the memory model can be surprisingly close to rust. The combination of typed IR for efficient monomorphization + the the ability to use stack allocated struct, mean that C# code can be quite close to rust code. And then the differences are mainly a consequence of LLVM backend vs ruygenJIT or AOT.
To say nothing of the advantage that a GC can have by having a global view of memory allocation/deallocations.
Added to that the fact that the time a rust eng. is spending modeling his ownership relationships and converting then into something the borrow checker can/accept could be spend improving C# perf further.
> Can you give some examples of those tools? The “first language” I mentioned above was C. What equivalent does C have to rust’s memory safety? Or C++ for that matter? Are smart pointers in C++ that ubiquitous and good that memory corruption is a thing of the past?
I am not too familiar with C, but i don't a think any new tool is needed : wrap all the unsafe operation (naked array accesses, allocation etc...) in a safe API and only use the safe API...
In C++, its trivial to avoid memory unsafe operation between smart_pointer, &reference, for_each for iteration etc... etc... Manipulating raw memory in "modern" C++ is mostly a choice than a necessity these days.
It is trivial yet there are plenty of CVE exploiting the failure to do so, as people being people, always take shortcuts in the name of performace, without bothering to use a profiler.
The problem is getting people to use that modern tooling, even static analysis is hardly used and lint was created in 1979.
This is what memory safe systems languages offer, there is no way around avoiding such tooling, it is part of the language and needs to be explicitly disabled.
This is what memory safe systems languages offer, there is no way around avoiding such tooling, it is part of the language and needs to be explicitly disabled.
> The problem is getting people to use that modern tooling, even static analysis is hardly used and lint was created in 1979
Maybe; rest to see if using modern tooling is easier or harder on the average programmer than learning a new language.
> This is what memory safe systems languages offer, there is no way around avoiding such tooling, it is part of the language and needs to be explicitly disabled.
The programming language is just one part of the chain. The goal is to detect/prevent bad code pattern and address them. There is no rules about this happening at the language level.
Maybe; rest to see if using modern tooling is easier or harder on the average programmer than learning a new language.
> This is what memory safe systems languages offer, there is no way around avoiding such tooling, it is part of the language and needs to be explicitly disabled.
The programming language is just one part of the chain. The goal is to detect/prevent bad code pattern and address them. There is no rules about this happening at the language level.
> Bugs like this aren’t possible at all in Go or C#
As a point of order, such bugs are possible for Go. Eg, hashmap's behaviour becomes undefined if you access from multiple threads at the same time.
Admittedly, that's a Go peculiarity. Making the programmer deal with concurrency on collections undoubtedly gets it a speed boost, but it's not the tradeoff most gc languages take.
The beauty of Rust, is you don't need to trade speed or memory for concurrency.
As a point of order, such bugs are possible for Go. Eg, hashmap's behaviour becomes undefined if you access from multiple threads at the same time.
Admittedly, that's a Go peculiarity. Making the programmer deal with concurrency on collections undoubtedly gets it a speed boost, but it's not the tradeoff most gc languages take.
The beauty of Rust, is you don't need to trade speed or memory for concurrency.
Oh I understand how safe helps and all, just wanted to sing Rust’s praises as a less low level memory-focused language too.
> hey both come with baggage (JVM).
I'm always surprised to hear people say this, when my perspective on the JVM is that it's a modern technical marvel several decades in the making with advantages like several unparalleled GCs and an ecosystem with extensive off-the-shelf & highly performant code available for any domain.
I'm always surprised to hear people say this, when my perspective on the JVM is that it's a modern technical marvel several decades in the making with advantages like several unparalleled GCs and an ecosystem with extensive off-the-shelf & highly performant code available for any domain.
I don’t work on server code, and I think a lot of discussions are really focused on server-side work. I have done desktop, mobile, and developer tools (CLIs and “scripts”).
If I was working on a server thing Kotlin would seem really good. In my work it is obviously applicable to Android, but not so much elsewhere.
I think Rust is nearly perfect for developer CLIs, you don’t really need to deal with the thorny memory and lifetime stuff because most things are allocated on the stack and the entire program is synchronous. clap is very good. Performance, especially startup performance, is incredible.
If I was working on a server thing Kotlin would seem really good. In my work it is obviously applicable to Android, but not so much elsewhere.
I think Rust is nearly perfect for developer CLIs, you don’t really need to deal with the thorny memory and lifetime stuff because most things are allocated on the stack and the entire program is synchronous. clap is very good. Performance, especially startup performance, is incredible.
You can use GraalVM Native Image and Kotlin/Native to compile Kotlin to a binary executable, which is fairly fast to start up but does not have all of the high-performance JVM features.
JVM is a marvel at making OOP code merely 10% or 50% slower than C, but I find procedural/data-oriented programming to be quite a lot clearer, faster, and more predictable (with respect to performance) and I don't need nor a huge runtime nor an expert to tune it nor a complicated toolchain to bundle it in with my application (or otherwise ensure that the properly tuned runtime is installed on my target).
To get it to that level of performance, you need to have expert java programmers, and the result will be hard to maintain.
For instance, you can’t use String on hot paths. Also, NPEs can happen almost anywhere, and the compiler doesn’t enforce thread safely (by which I mean “statically checked to be data race free”)
Also, the 10-50% slower rule of thumb only applies to throughput. Tail latency is generally much worse than that.
For instance, you can’t use String on hot paths. Also, NPEs can happen almost anywhere, and the compiler doesn’t enforce thread safely (by which I mean “statically checked to be data race free”)
Also, the 10-50% slower rule of thumb only applies to throughput. Tail latency is generally much worse than that.
It depends on what you're doing. In C you're going to be hampered by, again, the ecosystem.
There is nothing comparable in C as performant as all of the streaming tooling built up around Kafka for example.
So your can benchmark percentages all you want, but there are a lot of things you just can't do in C that you can on the JVM (without 100 developers and three years)
There is nothing comparable in C as performant as all of the streaming tooling built up around Kafka for example.
So your can benchmark percentages all you want, but there are a lot of things you just can't do in C that you can on the JVM (without 100 developers and three years)
> There is nothing comparable in C as performant as all of the streaming tooling built up around Kafka for example.
This is an ill-chosen example.
librdkafka is as or more performant than the Java stuff.
As a server, Redpanda blows Kafka out of the water (but this is rarely any bottleneck with brokers).
This is an ill-chosen example.
librdkafka is as or more performant than the Java stuff.
As a server, Redpanda blows Kafka out of the water (but this is rarely any bottleneck with brokers).
Yes, if you take the time to hand roll a replacement like Redpanda did it will be faster (though to your point that seems fairly pointless given the constraints imposed by IO)
I was referring more to the general ecosystem of Flink/Spark/KStreams/KTable etc.
I was referring more to the general ecosystem of Flink/Spark/KStreams/KTable etc.
> Flink/Spark/KStreams/KTable etc.
Then it's still a bit of an untruth since all of those use RocksDB for their high-performance storage layer, which isn't in Java. You could maybe (maybe!) argue for some ergonomics (though especially hard to argue for Spark!) but performance still goes plainly to C (/ C++).
Then it's still a bit of an untruth since all of those use RocksDB for their high-performance storage layer, which isn't in Java. You could maybe (maybe!) argue for some ergonomics (though especially hard to argue for Spark!) but performance still goes plainly to C (/ C++).
We're not talking about the systems you build on top of here. We're talking about the code. You write yourself. The fact that rocks is written in c is irrelevant to the fact that in order to use the ecosystem I'm describing you need to be on the JVM or your experience will be at best subpar.
Sure, if you ignore offloading all the hard work to a C library, we can pretend any language is as fast as any other. It's a boring conversation then though - might as well use Python...
Not to mention you still pay a real performance penalty for being unable to make direct use of RocksDB features, because Kafka is shoving a million "abstraction" and cache layers in there and you just want your your damned merge operator to apply.
I don't agree Kafka off the JVM is subpar. Kafka Streams has a lot of sharp edges from "abstractions" that aren't, Spark is a godawful developer experience and tuning it for performance is black magic for 99% of data engineers, and Flink keeps dithering about how consistent/stable baseline features like queries will be. You can stand up very good apps for a huge set of common cases just as quickly in Go or C++, and not have to deal with Kafka Streams's funny opinions about threads/tasks/consumers.
Not to mention you still pay a real performance penalty for being unable to make direct use of RocksDB features, because Kafka is shoving a million "abstraction" and cache layers in there and you just want your your damned merge operator to apply.
I don't agree Kafka off the JVM is subpar. Kafka Streams has a lot of sharp edges from "abstractions" that aren't, Spark is a godawful developer experience and tuning it for performance is black magic for 99% of data engineers, and Flink keeps dithering about how consistent/stable baseline features like queries will be. You can stand up very good apps for a huge set of common cases just as quickly in Go or C++, and not have to deal with Kafka Streams's funny opinions about threads/tasks/consumers.
this seems truthy but isn't in practice. a lot of work, my perf optimization team @ redpanda does (yes we have a full team chasing tail latencies) is spent on CPU optimization, debouncing, amortizing costs, metadata lookups, hash tables, profilers, etc. so there is a lot of additional work after the IO layer which a decent async eventing thing can get you to get good perf.
What I mean to say is that in my experience the time spent round-tripping to Kafka is more than the time it takes Kafka to do whatever I'm asking it to do. So at least for my use-cases a faster Kafka would be of no benefit.
Now if it means I can run fewer smaller brokers that's awesome.
Now if it means I can run fewer smaller brokers that's awesome.
def. that should be the case, we have iot companies pushing us on a single pthread a a few megs of ram.
Always funny to see OOP being thrown around like it's some sort of bogeyman. C++ is OOP, so is Rust (minus inheritance), yet both are as fast or faster than C.
You can write procedural/data-oriented code in Java.
Sort of (Java still doesn't have value types), and even then you still have most of the cons of a big runtime.
As long as you hire C/C++/Rust programmers to write it.
JVM memory consumption is prohibitive for small applications and overall the JVM is very memory inefficient. If you can, avoid Java for anything that isn't a long running server process
Have your worked with the JVM since Java 8?
Is memory util on servers still really a consideration in 2023 for most workloads if performance is otherwise good?
> If you can, avoid Java for anything that isn't a long running server process
Isn't that the JVM's niche?
Is memory util on servers still really a consideration in 2023 for most workloads if performance is otherwise good?
> If you can, avoid Java for anything that isn't a long running server process
Isn't that the JVM's niche?
> Is memory util on servers still really a consideration in 2023 for most workloads if performance is otherwise good?
The way that cloud scaling works is that as you increase RAM you also increase CPU, which costs a fortune. You can go for 'memory optimized' instances but even still, there are costs associated with memory that aren't just the RAM modules, unfortunately.
Plus with container orchestration it's increasingly important to pack your services into a single instance, and memory can veryyyy quickly become the constraint - in fact, it might be the one I've seen most often.
The way that cloud scaling works is that as you increase RAM you also increase CPU, which costs a fortune. You can go for 'memory optimized' instances but even still, there are costs associated with memory that aren't just the RAM modules, unfortunately.
Plus with container orchestration it's increasingly important to pack your services into a single instance, and memory can veryyyy quickly become the constraint - in fact, it might be the one I've seen most often.
I don't find this, but maybe it's something K8's helps us get away from?
For our workloads we seem to need more CPU when we need more ram anyways.
For our workloads we seem to need more CPU when we need more ram anyways.
No, you'd run into it with k8s more, if anything. But it is workload dependent.
Memory utilization on servers is absolutely a consideration in 2023. At work, we're looking seriously at migrating some key parts of our infrastructure from Java to Rust precisely because of factor (as well as startup time, throughput).
As someone working on systems that handle trillions of message per day in Java, I find this hard to relate to. Is it bad code? Some critical infrastructure flaw? Misunderstanding of memory options settings or how they work in containers?
Startup time is definitely a concern. Good luck on the throughput front though. Java libs (like Netty) still outpace what's available in Rust. You may end up with your own code being more performant, but the things you rely on externally being slower. This was our experience when trialing re-writes in Rust.
Startup time is definitely a concern. Good luck on the throughput front though. Java libs (like Netty) still outpace what's available in Rust. You may end up with your own code being more performant, but the things you rely on externally being slower. This was our experience when trialing re-writes in Rust.
It's probably RDD (resume driven development). It's always fun to rewrite in the latest new hype, and I don't have anything against Rust - I think it's great in the domains it is targeting.
I don't want several GCs, though.
Yeah, I've always derided the having of options as well. I find it best when you cannot toggle between various options to find the best set of tradeoffs for your use case.
Yes, for example, it's much better when the language completely takes away your ability to specify memory layouts or allocation patterns because it will never pick stupid ones.
As compared to what...
JVM is neither particularly advance nor particularly performant.
Can't quite comment on the GCs though
Can't quite comment on the GCs though
> JVM is neither particularly advance nor particularly performant
I'm not exactly a fan of Java, to put it mildly, but it is pretty efficient these days. To the point that it sometimes amazes me.
I'm not exactly a fan of Java, to put it mildly, but it is pretty efficient these days. To the point that it sometimes amazes me.
I don't want to sound like i am downplaying you understanding of the JVM. But i still have to ask : efficient as compare to what ?
From having played around the JVM, and spent quite a bite of time optimizer large spark workload , from the limited Monomorphization, to memory and cache inefficient object model, to the limited inlining depth, absence of code compaction ( for i-cache performance), the poor registor allocator etc... the JVM is really not up there.
I think it only shines when compare to things like go/python/javascript that the JVM can appear to be great.
From having played around the JVM, and spent quite a bite of time optimizer large spark workload , from the limited Monomorphization, to memory and cache inefficient object model, to the limited inlining depth, absence of code compaction ( for i-cache performance), the poor registor allocator etc... the JVM is really not up there.
I think it only shines when compare to things like go/python/javascript that the JVM can appear to be great.
Compared to doing distributed systems in DCOM/CORBA using C and C++ for example, that is one of the reasons it took off.
Many that like to bash Java EE, have no clue how bad it was before it came to be, it wasn't only Sun's marketing pushing it.
It isn't by accident that RMI had interoperability with CORBA.
Many that like to bash Java EE, have no clue how bad it was before it came to be, it wasn't only Sun's marketing pushing it.
It isn't by accident that RMI had interoperability with CORBA.
I am not sure how this is an answer the my argument. However saying that distributed system in JAVA EE are more performant that in C++ ( i am no familliar with DCOM/CORBA but i do know other libraries and system ) is news to me.
And again, the success of JAVA in the 90 and 00 can also be explained by better tooling, safer language,better x-platform stories,better libraries etc... etc...
Saying that java/jvm are not particularly performant is not bashing ... java/jvm have other great things going on... performance just isnt one of them
And again, the success of JAVA in the 90 and 00 can also be explained by better tooling, safer language,better x-platform stories,better libraries etc... etc...
Saying that java/jvm are not particularly performant is not bashing ... java/jvm have other great things going on... performance just isnt one of them
I guess if what one is measuring FPS for AAA, then surely not.
Naturally not having a clue about how to write proper Java code (for loops instead of System.arraycopy()), use profilers, use JIT caches with PGO feedback, not understanding how to allocate off heap, not using JNI to avoid writing everything in C or C++ with their own set of issues, doing background stuff on main threads,...also helps some Java applications not being performant.
Naturally not having a clue about how to write proper Java code (for loops instead of System.arraycopy()), use profilers, use JIT caches with PGO feedback, not understanding how to allocate off heap, not using JNI to avoid writing everything in C or C++ with their own set of issues, doing background stuff on main threads,...also helps some Java applications not being performant.
Have you used it since Java 8?
There's a reason the entire streaming ecosystem (Kafka/Flink/Spark/Etc.) has built up around the JVM.
There's a reason the entire streaming ecosystem (Kafka/Flink/Spark/Etc.) has built up around the JVM.
This is a huge deal, I find it so difficult to choose a language that doesn't have sum types and destructuring via pattern matching for the kinds of projects I like to work on (compiler/interpreter implementation).
IMHO the one downside with sum types in a performance-oriented systems programming language (or as I prefer to call them 'tagged unions' because that makes it clearer how they actually look like in memory) is that they are inherently cache-unfriendly, because they always take up as much space as the largest wrapped type (plus 8 bytes for the tag). When you have an array of those you need to be aware of the potential 'waste' between array items.
(now of course an automatic transformation into a structure-of-arrays would be pretty cool, I actually wonder what Zig's MultiArrayList does with tagged unions).
PS: in general I agree though, for instance in Typescript they are fantastic, but there I usually don't care much about memory layout and performance.
(now of course an automatic transformation into a structure-of-arrays would be pretty cool, I actually wonder what Zig's MultiArrayList does with tagged unions).
PS: in general I agree though, for instance in Typescript they are fantastic, but there I usually don't care much about memory layout and performance.
The overhead isn't always 8 bytes. When the type uses the naive representation, the overhead will generally match the alignment. For example an Option<u8> occupies 2 bytes, 1 of which is for the tag.
And if the type has some illegal values, these can sometimes be used to discriminate the type. When that's possible, the union will have the same size as the largest underlying type with no overhead for a tag. For example Option<&T> matches the size of &T, since it uses the illegal null value to represent None.
And if the type has some illegal values, these can sometimes be used to discriminate the type. When that's possible, the union will have the same size as the largest underlying type with no overhead for a tag. For example Option<&T> matches the size of &T, since it uses the illegal null value to represent None.
That depends on the sizes of the objects. If the objects in the union are generally small, cache coherence is better than if you have polymorphic objects allocated on the heap.
I think parent meant transforming a vector<A+T> into vector<A>*vector<T>
a good allocator could mitigate most of the allocation effects
Unlikely, because the allocator would need to know things only the compilers knows, and the compiler in turn needs to know how the allocator lays out the data in memory?
Some of that could probably be solved with comptime code (like Zig's MultiArrayList container I mentioned above).
Some of that could probably be solved with comptime code (like Zig's MultiArrayList container I mentioned above).
> Unlikely, because the allocator would need to know things only the compilers knows
Such as ?
Such as ?
Such a 'smart allocator' can't just work with a size and alignment, it also needs to know the internal structure of objects so that it can arrange them differently (maybe to remove padding bytes, maybe to turn arrays-of-structs into structs-of-arrays. And in turn the compiler needs to know how the 'allocator' has rearranged things in order to generate the code which accesses the data.
I also wasn't thinking about moving parts of a struct out into heap allocations, I think in most situations that would be worse than having to deal with padding. Instead at least have the programmer think about how the internal memory layout of sum types might affect performance in specific situations instead of just using them everywhere and in mass because they are convenient.
I also wasn't thinking about moving parts of a struct out into heap allocations, I think in most situations that would be worse than having to deal with padding. Instead at least have the programmer think about how the internal memory layout of sum types might affect performance in specific situations instead of just using them everywhere and in mass because they are convenient.
Rust sum types do not take up that much extra space, in fact often the overhead is completely compiled away. Lots of hard work to keep the wasted memory to a minimum, and they are not pointers, so there isn't an extra memory hop or anything.
If you are dealing with always-same-type arrays then you can express so as a sum type of arrays, not an array of sum types.
> plus 8 bytes for the tag
The tag can often be stored inline in the data, so doesn't necessarily require extra space.
> because they always take up as much space as the largest wrapped type
Yeah... but what's the alternative?
The tag can often be stored inline in the data, so doesn't necessarily require extra space.
> because they always take up as much space as the largest wrapped type
Yeah... but what's the alternative?
The alternative would be for Rust to generate machine code for the following
Today, clippy does what I think is the right thing: it teaches the user
enum Foo {
Bar,
Qux([i32; 1024]),
}
enum Foo {
Bar,
Qux(Box<[i32; 1024]>),
}
Today, clippy does what I think is the right thing: it teaches the user
warning: large size difference between variants
--> src/main.rs:1:1
|
1 | / enum Foo {
2 | | Bar,
| | --- the second-largest variant carries no data at all
3 | | Qux([i32; 1024]),
| | ---------------- the largest variant contains at least 4096 bytes
4 | | }
| |_^ the entire enum is at least 4100 bytes
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#large_enum_variant
= note: `#[warn(clippy::large_enum_variant)]` on by default
help: consider boxing the large fields to reduce the total size of the enum
|
3 | Qux(Box<[i32; 1024]>),
| ~~~~~~~~~~~~~~~~When you combine this with rust-analyzer's hints for the sizes of items, it makes it much easier to spot this kind of stuff and resolve it automatically while developing, too.
What's the alternative to sum types? What does a "structure of arrays" look like as it pertains to sum types? Are the arrays sparse or dense (and how is either going to be better for cache coherence)? Where does the tag live? In its own array?
> What does a "structure of arrays" look like as it pertains to sum types?
https://news.ycombinator.com/item?id=36195425
https://news.ycombinator.com/item?id=36195425
Unfortunately, that isn't what cache unfriendly means. You're concerned about wasting memory bandwidth. Your array is trivial to prefetch.
AFAIK the prefetcher doesn't help much if it's mainly busy with prefetching padding bytes of the 'inactive' unions (and I think even a smart prefetcher would at least always load whole cache lines which could be packed with active data instead). Of course that's also a bandwidth problem, you're wasting bandwidth loading data into the cache which then is ignored.
[deleted]
This is why I wish there was more momentum behind Hare[1]. It's basically C with sum types, namespaces, and other conservative improvements on C.
[1] https://harelang.org/
[1] https://harelang.org/
Eh, Hare expects you to write and rewrite collections like hashmaps yourself. It just looks like a less flexibe and less useful Zig.
And more complex, in implementation and usage. I like Zig too but I don't think there will ever be very good IDE integration (navigation, find usages, refactoring) given its flexibility with regard to the comptime stuff.
As a C++ dev I really don't follow. I don't remember any times where I thought something like a Union would help much.
Where have you found so much use for them?
Where have you found so much use for them?
I'm pretty fluent in C++ and Rust. You don't find much use for them in C++ because they're a relative pain in the ass without language support. std::variant is a sum type, but when's the last time you used it? I bet you had lots of fun with visitors and explicit type checks.
The most motivating example is Rust's Result<T, E>. Either you have a "good" type T, or some error E occurred. Want to bubble this error up? Just do `my_result?` to get the underlying T or return E early. This idiom is pervasive and makes error handling via return codes very easy to use.
It also makes it feasible to have types that have no "default" value. In C++ you have std::unique_ptr<T> which, for various reasons, has to have a nullptr state, or else it's frustrating to use. In Rust, owned pointers Box<T> don't have an empty state. If you need a nullable pointer, you use Option<Box<T>>, which is either Some(my_box) or None. The fact that Box<T> is always pointing to a valid object is a stronger guarantee than C++ smart pointers, and is only ergonomically possible because of the Option<T> sum type.
The most motivating example is Rust's Result<T, E>. Either you have a "good" type T, or some error E occurred. Want to bubble this error up? Just do `my_result?` to get the underlying T or return E early. This idiom is pervasive and makes error handling via return codes very easy to use.
It also makes it feasible to have types that have no "default" value. In C++ you have std::unique_ptr<T> which, for various reasons, has to have a nullptr state, or else it's frustrating to use. In Rust, owned pointers Box<T> don't have an empty state. If you need a nullable pointer, you use Option<Box<T>>, which is either Some(my_box) or None. The fact that Box<T> is always pointing to a valid object is a stronger guarantee than C++ smart pointers, and is only ergonomically possible because of the Option<T> sum type.
I love sum types, and i find it very hard to work without them when i am in C++.
However your examples i don't think are particularly convincing.
The examples you gave can mostly be done with std::variant<> and are simply examples are unions. (Result == std::expected)
In term of non nullable smart pointer, it would be easy to enforce the the non nullability in the constructor, and have the same ergonomy as Option<Box<T>>. In fact you can do something like std::expected<std::reference<T>> but why would you instead of just std::unique_ptr and handle the nullable case.
What makes sum type powerfull is to have full Algebraic DataType (tags + sum types + product types) + pattern matching + good compiler errors.
However your examples i don't think are particularly convincing.
The examples you gave can mostly be done with std::variant<> and are simply examples are unions. (Result == std::expected)
In term of non nullable smart pointer, it would be easy to enforce the the non nullability in the constructor, and have the same ergonomy as Option<Box<T>>. In fact you can do something like std::expected<std::reference<T>> but why would you instead of just std::unique_ptr and handle the nullable case.
What makes sum type powerfull is to have full Algebraic DataType (tags + sum types + product types) + pattern matching + good compiler errors.
optional<non_null_unique_ptr<T>> would be completely feasible in C++.
It is just that std::optional and unique_ptr do not have the right semantics out of the box, but it is not hard to roll your own.
But yes, I wish C++ has good builtin sum types and proper destructuring via pattern patching. Maybe one day....
It is just that std::optional and unique_ptr do not have the right semantics out of the box, but it is not hard to roll your own.
But yes, I wish C++ has good builtin sum types and proper destructuring via pattern patching. Maybe one day....
The problem with adding a non_null_unique_ptr<T> to C++ is that moving needs to leave behind a valid value.
You are right, but you could make it not movable (and add destructive mode hooks for optional and containers).
It is still possible to return non-movable, non-copyable types from functions.
But I guess ergonomics would not be good.
It is still possible to return non-movable, non-copyable types from functions.
But I guess ergonomics would not be good.
They're extremely useful in capturing the "business rules" for a given domain, particularly when it's a restricted domain: if something can only be `A | B | C`, being able to make that explicit, have it checked exhaustively everywhere by the compiler, and be able to dispatch on it easily using `match` is really powerful.
That's because it comes with a statically-tracked "tag" (so: sometimes called a "tagged union") that the compiler can use to check you. You can write the same thing with a `union` in C or C++, with an `enum` to tag it, but you're doing that by hand rather than having it built in.
Also: Where a struct or class's relationship over its members is the product of all of the types which make it up, a tagged union/enum is the sum of them. A common example: instead of a bunch of optional fields on a single class to model the different possible combinations, modeling something as only the legal set of combinations of fields. I recommend Scott Wlaschin's 'designing with types' series (uses F♯, but applies to any languages with 'sum types') for a great intro and then deep dive on the ideas here: https://fsharpforfunandprofit.com/series/designing-with-type...
That's because it comes with a statically-tracked "tag" (so: sometimes called a "tagged union") that the compiler can use to check you. You can write the same thing with a `union` in C or C++, with an `enum` to tag it, but you're doing that by hand rather than having it built in.
Also: Where a struct or class's relationship over its members is the product of all of the types which make it up, a tagged union/enum is the sum of them. A common example: instead of a bunch of optional fields on a single class to model the different possible combinations, modeling something as only the legal set of combinations of fields. I recommend Scott Wlaschin's 'designing with types' series (uses F♯, but applies to any languages with 'sum types') for a great intro and then deep dive on the ideas here: https://fsharpforfunandprofit.com/series/designing-with-type...
A Union in itself is not a sum type. Rust even has unions:
https://doc.rust-lang.org/reference/items/unions.html
The unions of Rust are tagged unions that can hold data:
https://doc.rust-lang.org/book/ch06-01-defining-an-enum.html
which is immensely powerful in data modeling.
https://doc.rust-lang.org/reference/items/unions.html
The unions of Rust are tagged unions that can hold data:
https://doc.rust-lang.org/book/ch06-01-defining-an-enum.html
which is immensely powerful in data modeling.
We do not use unions very often in C/C++ because they are not safe. You have to know how to interpret it; the compiler can't and won't save you if you read the memory as the wrong type. You usually keep a variable set to the type the union is, and check it at run time.
Some languages do that wrapping and checking for you, automatically. Everything from Ada (variant records) to Haskell has them. You cannot mishandle such objects by misinterpreting them as the wrong type; it's a compile-time error in those languages to do so.
Imagine you are parsing something. In my opinion, the most natural way to express this, is a function that takes an array of characters and returns an array of parsed objects or tokens, where each object may be of a different possible type. In C, you'd probably use a union of structs, and set which type of struct as you parse each object. This is error prone and verbose, and the compiler can't double-check it for you. It's all automatic in a language with sum types. Quite efficiently too; most of the runtime checks can be optimized away.
Some languages do that wrapping and checking for you, automatically. Everything from Ada (variant records) to Haskell has them. You cannot mishandle such objects by misinterpreting them as the wrong type; it's a compile-time error in those languages to do so.
Imagine you are parsing something. In my opinion, the most natural way to express this, is a function that takes an array of characters and returns an array of parsed objects or tokens, where each object may be of a different possible type. In C, you'd probably use a union of structs, and set which type of struct as you parse each object. This is error prone and verbose, and the compiler can't double-check it for you. It's all automatic in a language with sum types. Quite efficiently too; most of the runtime checks can be optimized away.
The main difference comes with the syntax sugar that's built into the language (and the additional information the compiler has).
Sum types are also quite nice to bridge the gap from a 'dynamically typed mindset' into a 'statically typed mindset'. Sum types basically provide the ability that a variable can assume different types during its lifetime (? although I don't know if such 'shape-shifting' is allowed in Rust), so it almost feels like Python or Javascript, but since the compiler 'knows' sum types it's all still completely typesafe.
In Typescript it's convenient and frankly quite nice, in a programming language like Rust where performance matters I'd be less enthusiastic though.
Sum types are also quite nice to bridge the gap from a 'dynamically typed mindset' into a 'statically typed mindset'. Sum types basically provide the ability that a variable can assume different types during its lifetime (? although I don't know if such 'shape-shifting' is allowed in Rust), so it almost feels like Python or Javascript, but since the compiler 'knows' sum types it's all still completely typesafe.
In Typescript it's convenient and frankly quite nice, in a programming language like Rust where performance matters I'd be less enthusiastic though.
> I don't know if such shape-shifting is allowed in Rust
I think so! You mean this?
I think so! You mean this?
let mut x: Result <bool, String> = Ok (false);
dbg! (&x);
// Get a reference to the bool and mutate it
if let Ok (b) = &mut x {
*b = true;
}
dbg! (&x);
// "shape-shift" to the other enum variant
x = Err ("No boolean :(".to_string ());
dbg! (&x);
I flipped through one of my Rust projects and the best use case I saw was for serializing messages for disk / network / IPC:
I haven't seen anything in C++ that's both as easy and as airtight as a Rust enum with serde's automatic serialization macro on the top.
// This derive macro expands to a visitor pattern during build
#[derive (serde::Serialize)]
enum Message {
QueryFoo (QueryFoo),
MutateBar (MutateBar),
}
struct QueryFoo {
foo_id: i64,
}
// If we receive a Message::MutateBar from outside our process,
// it's a compile error for us to accidentally access the field `QueryFoo::foo_id`.
struct MutateBar {
bar_id: String,
baz_value: u32,
}
fn send_msg_as_json (tx: &mut Sender, msg: &Message) -> anyhow::Result <()> {
// The JSON serialization / deserialization is generated, too
tx.send (serde_json::to_string (msg))?;
Ok (())
}
I haven't seen anything in C++ that's both as easy and as airtight as a Rust enum with serde's automatic serialization macro on the top.
FWIW using std::variant (or whatever is your in-house variant) to store message types is very common.
Also in well designed low level binary protocols message types have a discriminant header already so often common to wrap them in an ad hoc-variant which doesn't have a separate discriminant.
Also in well designed low level binary protocols message types have a discriminant header already so often common to wrap them in an ad hoc-variant which doesn't have a separate discriminant.
Option/Optional/Maybe is the most obvious one, as a basic example have you interacted with a API that sends some JSON where a field can be null or not? How would you represent that and distinguish it from a field that is always a string?
That's just std::optional<> vs std::string, isn't it?
Yes, and std::optional is a janky implementation of a sum type in a language that doesn’t have them. The ergonomics are bad, and it is easy to make your program blow up if you forget a check.
In a language with sum types like Rust, Kotlin, and Swift, if you’re following the default “happy path” of coding, you are not allowed to forget a check. The compiler will not let you.
In a language with sum types like Rust, Kotlin, and Swift, if you’re following the default “happy path” of coding, you are not allowed to forget a check. The compiler will not let you.
Not many options, pointers most likely.
You can make a tagged union in any language, and that can often be useful, but when it is a standard part of the language you prevent any accidental cases of:
1. failing to check for sigils, because nobody uses a sigil when you have sum types. This would have prevented a famous exploit in SUDO a couple years back where someone forgot to check for the special negative value on an input parameter.
2. Eliminating null pointer exceptions or anything similar. You can't forget to check, you can still choose to have this kind of error, but you have to do it explicitly.
Consider something like a function to open and give you back a file handle. In many languages such a function will return a pointer to a file, or null if there is a failure, or perhaps throw an exception if there is a failure. If you neglect to check for null, or forget to handle the exception, you have a problem. With sum types, the natural thing is to return something like a Result type, which is EITHER a file, or an error. And you have to check which it is, to get the File out.
1. failing to check for sigils, because nobody uses a sigil when you have sum types. This would have prevented a famous exploit in SUDO a couple years back where someone forgot to check for the special negative value on an input parameter.
2. Eliminating null pointer exceptions or anything similar. You can't forget to check, you can still choose to have this kind of error, but you have to do it explicitly.
Consider something like a function to open and give you back a file handle. In many languages such a function will return a pointer to a file, or null if there is a failure, or perhaps throw an exception if there is a failure. If you neglect to check for null, or forget to handle the exception, you have a problem. With sum types, the natural thing is to return something like a Result type, which is EITHER a file, or an error. And you have to check which it is, to get the File out.
This is one of these things that you don't know you're missing until you actually start actively using. Back in the day when I was writing a lot of C++ I was saying the same thing as you.... until I wrote a significant program in Haskell, where sum types are also very common. And once it clicks it's hard to go back.
Try representing JSON in C++. Sure, you can figure out a way that works, but the C++ code for printing out the JSON is much uglier than a language with tagged unions.
That is a consequence of lack of reflection, not sum types.
sum types can often eliminate the need for reflection, and can have lower runtime cost.
Compile time reflection doesn't have runtime cost.
Compile time reflection doesn't work for this case.
Template metaprogramming techniques as poor man's reflection prove otherwise.
Additionally that is one of the examples on the ongoing C++ reflection papers.
Then there are a couple of examples in D, Common Lisp, Scheme, C# code generators.
Additionally that is one of the examples on the ongoing C++ reflection papers.
Then there are a couple of examples in D, Common Lisp, Scheme, C# code generators.
No? A JSON object could have any type at run time. You cannot do anything at compile time to figure out the right type.
Please point to a specific library if you disagree.
Please point to a specific library if you disagree.
Java has sum types for a while now, and quite ergonomic with records.
Kotlin also runs on JS and Wasm these days, not just the JVM.
provided the language later lets you split them apart and declare a branch of code as only dealing with one subtype
> I do not take Rust to be remotely the final word in this space. It is the first memory safe systems language to be successful at industrial scale — not the last!
I think this side note is the most interesting tidbit, because I agree. I think Rust is a beautiful thing, but it's starting to get more competition now, and the "memory safe systems language" space is really starting to heat up. I'm particularly interested in seeing whether Val becomes its own thing, or if Swift and Rust (or another language) just absorb it's good ideas.
Rust is amazing and the community is full of great people, but there are a lot of rough edges and places that feel awkward for new folks. I appreciate seeing all of the constructive criticisms that have come out against Rust lately, and for folks like Tim McNamara who are pointing out all of Rust's rough edges in hopes of improving the language. If I were able to contribute more to the project, that is where I would focus, personally.
Also, Rust needs to get it's leadership issues sorted out. More and more projects are starting to sneak up on Rust, and if the folks in charge of the project keep fumbling the bag and causing rifts in the community (like with the Rust licensing, and driving phantomderp away), it's going to keep folks from wanting to do the hard work to improve the language.
I think this side note is the most interesting tidbit, because I agree. I think Rust is a beautiful thing, but it's starting to get more competition now, and the "memory safe systems language" space is really starting to heat up. I'm particularly interested in seeing whether Val becomes its own thing, or if Swift and Rust (or another language) just absorb it's good ideas.
Rust is amazing and the community is full of great people, but there are a lot of rough edges and places that feel awkward for new folks. I appreciate seeing all of the constructive criticisms that have come out against Rust lately, and for folks like Tim McNamara who are pointing out all of Rust's rough edges in hopes of improving the language. If I were able to contribute more to the project, that is where I would focus, personally.
Also, Rust needs to get it's leadership issues sorted out. More and more projects are starting to sneak up on Rust, and if the folks in charge of the project keep fumbling the bag and causing rifts in the community (like with the Rust licensing, and driving phantomderp away), it's going to keep folks from wanting to do the hard work to improve the language.
Ignoring the long usage of Ada in the (military) industrial scale, or predecessors.
Ada still has killer features that haven't made it into C-like languages.
One of my favourite is the split between the abstract/programmer's view of a type, and the machine implementation/representation of the type.
There's no bit twiddling, no masking with 0xFFFF..., no worries about order in a struct or packing. If you have a 4 bit value, and want to store it in bits 27 ... 31 of a word, Ada lets you specify that. From the programmer's perspective, you work on the abstract type, perhaps defined as range 0 .. 15 in this case. (It can be any type that fits into the number of bits available, by the way.) It packs/unpacks transparently behind the scenes. And if you overflow that 0 .. 15 range it will raise an overflow/underflow exception. If you were trying to cram that 0 .. 15 range into only 3 bits, that'll raise a compile time error.
Ever since I've encountered that its absence from the C family seems like a very obvious oversight.
One of my favourite is the split between the abstract/programmer's view of a type, and the machine implementation/representation of the type.
There's no bit twiddling, no masking with 0xFFFF..., no worries about order in a struct or packing. If you have a 4 bit value, and want to store it in bits 27 ... 31 of a word, Ada lets you specify that. From the programmer's perspective, you work on the abstract type, perhaps defined as range 0 .. 15 in this case. (It can be any type that fits into the number of bits available, by the way.) It packs/unpacks transparently behind the scenes. And if you overflow that 0 .. 15 range it will raise an overflow/underflow exception. If you were trying to cram that 0 .. 15 range into only 3 bits, that'll raise a compile time error.
Ever since I've encountered that its absence from the C family seems like a very obvious oversight.
A bit late reply, but that is the core problem for adaption. It is very commercial language after all. There is not big open source community. On military and aerospace level there is a lot of money to throw for language tooling. For indiciduals, not much is possible.
Do people generally say they need a lot of unsafe? Hasn't come up much for me.
Also I'm not sure the cognitive overhead is really that big. The compiler helps a lot with its useful error messages.
Also I'm not sure the cognitive overhead is really that big. The compiler helps a lot with its useful error messages.
Yes.
It’s pretty common to need to interact with existing APIs or libraries that only have a c interface.
Fortunately, rust is good at that.
Unfortunately, it’s impossible to do without unsafe code.
You can wrap libraries in safe wrappers, or if you’re lucky someone has a wrapper you can use…
…but I’ve heard it several times for this reason, especially in shops that have existing internal code bases (Ie. 0% chance of an existing safe wrapper) and want to incrementally adopt rust.
Never heard someone say it’s because of perf tho; that sounds like a strawman argument to me; if your code is seriously too slow because of the bounds checking you’re doing something crazy and probably want to be using a gpu instead.
It’s usually io bound, and unsafe code doesn’t fix that.
It’s pretty common to need to interact with existing APIs or libraries that only have a c interface.
Fortunately, rust is good at that.
Unfortunately, it’s impossible to do without unsafe code.
You can wrap libraries in safe wrappers, or if you’re lucky someone has a wrapper you can use…
…but I’ve heard it several times for this reason, especially in shops that have existing internal code bases (Ie. 0% chance of an existing safe wrapper) and want to incrementally adopt rust.
Never heard someone say it’s because of perf tho; that sounds like a strawman argument to me; if your code is seriously too slow because of the bounds checking you’re doing something crazy and probably want to be using a gpu instead.
It’s usually io bound, and unsafe code doesn’t fix that.
> The compiler helps a lot with its useful error messages.
As someone trying very hard to love Rust this is just simply not the case. It takes the shortest path to most answers. The most trivial example where the compiler becomes less helpful is a self-balancing binary tree. It will constantly suggest explicit lifetimes and other magic when in reality you probably want `Box<Rc<..>>`. It will suggest Boxing for recursive types...but that's about the end of it. In many cases you're better off knowing what to do. But the problem is knowing what to do requires you to have done what you need...and hence the Rust Ouroboros is born.
The nice thing about unsafe code (C, for example) is the cognitive overhead is initially very low. That allows you to get an idea down and then worry about stuff later. Of course, if you have a gun to your head from management you never get to the "worry about stuff later" stage and introduce all sorts of nasty bugs. Rust frontloads all of this making it sometimes very difficult to use without having everything already thought out way ahead of time. Both a benefit and a curse.
As someone trying very hard to love Rust this is just simply not the case. It takes the shortest path to most answers. The most trivial example where the compiler becomes less helpful is a self-balancing binary tree. It will constantly suggest explicit lifetimes and other magic when in reality you probably want `Box<Rc<..>>`. It will suggest Boxing for recursive types...but that's about the end of it. In many cases you're better off knowing what to do. But the problem is knowing what to do requires you to have done what you need...and hence the Rust Ouroboros is born.
The nice thing about unsafe code (C, for example) is the cognitive overhead is initially very low. That allows you to get an idea down and then worry about stuff later. Of course, if you have a gun to your head from management you never get to the "worry about stuff later" stage and introduce all sorts of nasty bugs. Rust frontloads all of this making it sometimes very difficult to use without having everything already thought out way ahead of time. Both a benefit and a curse.
It’s just the compiler assuming you know what you’re doing. You’re building this data structure on the stack, so it’ll point out where you’re not sufficiently explaining to it how the lifetimes of your code are safe. And even if you build it on the heap (using Box) you’ll still need to get the ownership right (using e.g. Rc).
There’s only so many variations that work for something like trees (especially with doubly linked nodes) in any language, without either inherently unsafe lifetimes (because the particular language can’t express them as part of the API) or some kind of indirection (such as ECS style indices). Or garbage collection or ref counting, but one probably uses Rust to be able to control that kind of overhead instead of having to use whatever the language creators picked.
If you want to learn about this, instead of using a ‘blessed’ implementation, read “too many linked lists” https://rust-unofficial.github.io/too-many-lists/.
As for the front-loading comment: yes, Rust requires you to get some things right from the start, particularly ownership and lifetimes. If you want to work in Rust like in a scripting language, you can just use Box, Rc and clone() everywhere. But getting it right without those might require a significantly different API. Which isn’t that much different with your rough C implementation. To get any kind of safe API there will require just as many changes (if not more).
There’s only so many variations that work for something like trees (especially with doubly linked nodes) in any language, without either inherently unsafe lifetimes (because the particular language can’t express them as part of the API) or some kind of indirection (such as ECS style indices). Or garbage collection or ref counting, but one probably uses Rust to be able to control that kind of overhead instead of having to use whatever the language creators picked.
If you want to learn about this, instead of using a ‘blessed’ implementation, read “too many linked lists” https://rust-unofficial.github.io/too-many-lists/.
As for the front-loading comment: yes, Rust requires you to get some things right from the start, particularly ownership and lifetimes. If you want to work in Rust like in a scripting language, you can just use Box, Rc and clone() everywhere. But getting it right without those might require a significantly different API. Which isn’t that much different with your rough C implementation. To get any kind of safe API there will require just as many changes (if not more).
> It will constantly suggest explicit lifetimes and other magic when in reality you probably want `Box<Rc<..>>`.
If you can provide examples of cases like these, please file a ticket in the issue tracker. Some suggestions have to be overly cautious in order to not lead people astray, so having examples of common (or uncommon!) things people try that could use with some "human touch" in what direction they should take instead is very useful and welcome.
If you can provide examples of cases like these, please file a ticket in the issue tracker. Some suggestions have to be overly cautious in order to not lead people astray, so having examples of common (or uncommon!) things people try that could use with some "human touch" in what direction they should take instead is very useful and welcome.
I wonder how tractable it would be for the compiler to suggest when your API needs Rc or boxing without being just as wrong in other cases.
I think now that too many lists is part of the book, maybe the compiler can link to that in some cases, hah.
I think now that too many lists is part of the book, maybe the compiler can link to that in some cases, hah.
> I wonder how tractable it would be for the compiler to suggest when your API needs Rc or boxing without being just as wrong in other cases.
A lot of the cases I've seen so far require some kind of global analysis, that we don't have today. But if we can have some kind of local analysis that will be reliable for a subset of cases, I'd jump at the opportunity to add it.
> I think now that too many lists is part of the book, maybe the compiler can link to that in some cases, hah.
That's not a bad idea, we already link to the book when it seems appropriate.
A lot of the cases I've seen so far require some kind of global analysis, that we don't have today. But if we can have some kind of local analysis that will be reliable for a subset of cases, I'd jump at the opportunity to add it.
> I think now that too many lists is part of the book, maybe the compiler can link to that in some cases, hah.
That's not a bad idea, we already link to the book when it seems appropriate.
I've used some unsafe, but only when using Rust for embedded programming. The amount of unsafe I have to use is surprisingly low, since a lot of what I'd have to use it for is wrapped up into libraries like RTIC or the various hal/pac board-specific libraries.
If you chose Rust because you want speed that comes with natively compiled code, and you want optimal performance, in general you will probably come across a place where you need to dereference pointers without anything extra running.
For what it's worth: references in Rust (both `&` and `&mut`) are just pointers with stronger static (compile time only) analysis. They have zero runtime impact vs raw pointers.
Even heap allocations like `Box` are still just pointers, that have destructors that run when they fall out of scope, but access is just a pointer dereference, the same as you could do with unsafe.
Even heap allocations like `Box` are still just pointers, that have destructors that run when they fall out of scope, but access is just a pointer dereference, the same as you could do with unsafe.
If you are just working with data, you can avoid unsafe completely. The issue is that for performant code, you often have to tap into linux libc stuff which doesn't guarantee safety. For example, if you try to to write something with packet_mmap for fast processing of traffic, you will have to use unsafes.
Look at the Firecracker codebase. There are a lot of unsafe calls into libc.
There is also a runtime reference counter which takes up cycles since it has to operate at run time.
Look at the Firecracker codebase. There are a lot of unsafe calls into libc.
There is also a runtime reference counter which takes up cycles since it has to operate at run time.
Okay, but using unsafe for FFI is entirely different than the claim that you can only achieve performance optimizations through "dereferencing pointers without anything extra running." The use of unsafe for FFI is unavoidable for obvious reasons.
Indeed. Raw pointers are mostly when either crossing FFI barriers (say, calling or being called by C) or when what you want to do can’t be expressed within Rust’s ownership rules without taking on overhead (Rc, cloning, indirections, etc) that’s unacceptable to your specific use case.
I haven't really had this experience. Usually a subtle reorg of how I iterate, for example, allows the compiler to eliminate bounds checks, effectively giving me full speed without unsafe in tight loops.
There's nothing "extra running" when you dereference a rust reference either, its just a pointer that follows the semantics that can be checked at compile time.
A lot of people say that they need it because they are trying to code C in rust. Often I see it as analogous to trying to learn an SOV spoken language but insisting on using SVO grammar still - sure you can muddle through to a mutual understanding with a conversant, but it's high friction and has a lot of confusion along the way.
I'm not trying to invalidate all claims that unsafe is necessary, nor even that there are cases where unsafe is quite helpful for performance. It's just that a little unsafe used deliberately and thoughtfully can go a long way, and accomplish those goals in a largely safe codebase.
I'm not trying to invalidate all claims that unsafe is necessary, nor even that there are cases where unsafe is quite helpful for performance. It's just that a little unsafe used deliberately and thoughtfully can go a long way, and accomplish those goals in a largely safe codebase.
I mostly hear it from folks who are explaining why they use Zig or Odin instead, or aren't why they are not interested in switching from C or C++.
Re: overhead—I think the compiler helps a ton here, but I do think that it makes a lot of memory-related concerns explicit where they are usually implicit, and dealing with lifetimes can be quite burdensome. Now: I also think that those concerns still exist when you're using other systems-level languages, and that the tradeoff therefore often ends up looking like "Deal with it up front, or deal with it on hard mode later."
Re: overhead—I think the compiler helps a ton here, but I do think that it makes a lot of memory-related concerns explicit where they are usually implicit, and dealing with lifetimes can be quite burdensome. Now: I also think that those concerns still exist when you're using other systems-level languages, and that the tradeoff therefore often ends up looking like "Deal with it up front, or deal with it on hard mode later."
I’ve spent a fair bit of time in rust and written more than my fair share of unsafe code. Unsafe is super necessary sometimes. But there are definitely reasons I’d consider using Zig or C or something if I needed a lot of unsafe in my project.
- From a language point of view, the syntax for dereferencing pointers in rust is awful. I don’t mind the unsafe keyword. But in rust you can’t do a->b. You need to write (*a).b. And that gets increasingly awful as you have more complex pointer expressions. Unsafe code in rust looks significantly uglier than the equivalent C.
- The Rust community - particularly on Reddit - seems to get very snide annd condescending about programmers using any amount of unsafe in their rust code. Even when it’s totally necessary. I’m not convinced the community will is there to fix rust’s bad pointer syntax, or other missteps with unsafe. Apparently having unreadable unsafe code is my atonement for wanting high performance code. Never mind that the harder my code is to read, the more likely I am to make mistakes. And in exactly the part of my codebase where bugs would be the most damaging to safety! Argh.
I love rust. I love the fresh ideas it’s brought to programming as a whole. But I can’t wait for some other, newer languages to iterate on these features & the language syntax some more. It’s not the top of the mountain.
- From a language point of view, the syntax for dereferencing pointers in rust is awful. I don’t mind the unsafe keyword. But in rust you can’t do a->b. You need to write (*a).b. And that gets increasingly awful as you have more complex pointer expressions. Unsafe code in rust looks significantly uglier than the equivalent C.
- The Rust community - particularly on Reddit - seems to get very snide annd condescending about programmers using any amount of unsafe in their rust code. Even when it’s totally necessary. I’m not convinced the community will is there to fix rust’s bad pointer syntax, or other missteps with unsafe. Apparently having unreadable unsafe code is my atonement for wanting high performance code. Never mind that the harder my code is to read, the more likely I am to make mistakes. And in exactly the part of my codebase where bugs would be the most damaging to safety! Argh.
I love rust. I love the fresh ideas it’s brought to programming as a whole. But I can’t wait for some other, newer languages to iterate on these features & the language syntax some more. It’s not the top of the mountain.
I think some of the snideness is probably due to different senses of what amount of unsafe is actually needed. There’s so much premature optimization in many cases that wouldn’t even come up if the language were Java, Go or whatever. It’s totally fine to initially accept some known overhead from cloning or Rc if you don’t know yet it it’s going to significantly impact performance.
I think the tricky thing here is that unsafe blocks pit safety and performance against each other at the limit. There’s essentially two languages both called rust: one which prioritises performance over safety. (“Make it safe - but only when safety doesn’t negatively impact performance”) And another which prioritises safety over performance (“Raw pointers and unsafe code should only be used as a last resort”).
Cargo stores packages from both sub-communities and we fight over the heart of rust. - And sometimes to our collective detriment, like what happened to actix.
Cargo stores packages from both sub-communities and we fight over the heart of rust. - And sometimes to our collective detriment, like what happened to actix.
> But in rust you can’t do a->b. You need to write (*a).b.
For what is worth, I think that we'll add that at some point.
For what is worth, I think that we'll add that at some point.
Pointer dereferencing is a missed opportunity to use a postfix operator. If a* was the way to dereference, then a*.foo() could have been the way to call a function on a pointer.
I definitely know what you mean re: snideness about `unsafe`, and I wish that were quite different. I think it's (a) good to have a default of "avoid unsafe, you probably need it less than you think" and (b) good to remember that the tool is there for a reason!
Also: agreed re: syntax.
> But I can’t wait for some other, newer languages to iterate on these features & the language syntax some more. It’s not the top of the mountain.
YEP. That's exactly what I was getting at in my callout at the top of the piece. First word, not the last.
Also: agreed re: syntax.
> But I can’t wait for some other, newer languages to iterate on these features & the language syntax some more. It’s not the top of the mountain.
YEP. That's exactly what I was getting at in my callout at the top of the piece. First word, not the last.
I wonder how much they would be interested, when we finally catch up with the other industries, and liability becomes a real thing.
I don't really need unsafe but I do enjoy using it. In particular, `unwrap_unchecked` is nice for when I'm 100% sure that an unwrap can't fail. I don't see any reason to leave that to the compiler to figure out.
Depends what you are doing, you might have 0.0 lines of unsafe, or you might be doing some custom SIMD string parsing code and its 100% unsafe. Or anywhere in between.
Been using ash for vulkan so lots of unsafe here
Most software does not need a lot of unsafe. There are a handful of software domains where you do need a lot of unsafe if you care about performance, but in these cases languages like C++ are often more ergonomic anyway.
I've put a bunch of rust code into production and I've never written unsafe code.
It primarily comes up when you're 1) interfacing with hardware 2) interfacing with C APIs 3) doing something weird with pointers.
It primarily comes up when you're 1) interfacing with hardware 2) interfacing with C APIs 3) doing something weird with pointers.
Depends how much performance matters and what the problem domain is, and how low level it is. It can vary a lot.
System languages with unsafe code blocks exist for decades, with ESPOL in 1961 being one of the first ones.
In fact, had languages like Modula-2 and Ada not lost to UNIX/C/C++ hegemony, this would already be quite common.
It is kind of tragic that we still need to keep discussing the value of this.
In fact, had languages like Modula-2 and Ada not lost to UNIX/C/C++ hegemony, this would already be quite common.
It is kind of tragic that we still need to keep discussing the value of this.
One misconception is also that in unsafe block everything goes and all checks for memory are disabled. This just isn't true.
Unsafe rust allows you to make operations which could invalidate all of rusts memory guarantees by granting you access to raw pointers, but that is (almost) it. It doesn't mean that rust suddenly does not enforce any of its rules.
I also can not believe that that much code actually needs raw pointer operations if the developer considers alternatives.
I also can not believe that that much code actually needs raw pointer operations if the developer considers alternatives.
Having `derive(Debug)` is the dream of all C devs.
Not having to deal with a completely asinine build chain is another.
Apart from "safe" and "unsafe" we should also have "trusted" and "untrusted".
Software is getting complicated and supply chain attacks are real.
Software is getting complicated and supply chain attacks are real.
Agreed. 5 years ago I thought that Haskell's IO monad was genius.
I hope it trickles into a mainstream language before I retire. Sure, it's an extra arg to every non-pure function.
So what? Write more pure functions! If a function shouldn't use the network, pass it an IO that can't do so. If it shouldn't touch disk, pass it an IO that can't do so.
This may complicated printf logging... Or maybe there is an IO that only allows passing messages to a logging module, and the logging module can write to disk. So the logging module gets more auditing attention than pure functions.
I hope it trickles into a mainstream language before I retire. Sure, it's an extra arg to every non-pure function.
So what? Write more pure functions! If a function shouldn't use the network, pass it an IO that can't do so. If it shouldn't touch disk, pass it an IO that can't do so.
This may complicated printf logging... Or maybe there is an IO that only allows passing messages to a logging module, and the logging module can write to disk. So the logging module gets more auditing attention than pure functions.
I think where this is headed is algebraic effects, not using an io monad.
Safe Haskell sort of addressed this, though I didn’t find it useful in practice.
https://downloads.haskell.org/~ghc/7.8.4/docs/html/users_gui...
https://downloads.haskell.org/~ghc/7.8.4/docs/html/users_gui...
All of the hardware your machine runs on would be 'untrusted'. We sit on a swiss cheese palace. Might as well get used to it.
[deleted]
Isn't it what "git blame" is for?
Basically, if you use a library that you don't trust, you have to run it inside a sandbox so that it cannot do stuff that you don't want it to do, e.g. access the entire filesystem or peek/poke in another library's memory.
The sandbox should be part of the language.
The sandbox should be part of the language.
Looking up links from the poster, Chris.
I didn't realize languages should have theme songs until now.
Think all languages should have a theme.
https://soundcloud.com/chriskrycho/new-rustacean-theme
Other Languages:
C++, some classic rock?
C#, some re-cover of a classic rock song by a corporate group?
Python, dubstep?
I didn't realize languages should have theme songs until now.
Think all languages should have a theme.
https://soundcloud.com/chriskrycho/new-rustacean-theme
Other Languages:
C++, some classic rock?
C#, some re-cover of a classic rock song by a corporate group?
Python, dubstep?
PHP: A one-man band trying to play a song while being attacked by an angry goose.
Babe, wake up, a new Rust post from Chris Krycho just dropped.
[deleted]
>When dealing in unsafe code — whether in Rust unsafe blocks or in all the code in languages like C and C++3 — the responsibility falls to the programmer to write code which is still safe. This is possible! It is simply very difficult.
This is an interesting Catch 22.
If you accept this premise as true, then Rust becomes ultimately pointless. Because the prescription would be to become a very good programmer and learn how to write memory safe code, at which point you really don't need any other language except C. You will likely write memorypools that have mechanisms to make sure that you never access memory you are not supposed to.
An alternative prescription is to dedicate effort to making higher level language compilers/transpilers better. That way everyone codes in Python, the compiler recognizes common access patterns, and does iterative optimization (probably ML powered) to get you the most optimal machine code.
If you accept the premise as false, i.e writing memory safe code is easy, then Rust doesn't really offer any benefits.
This is an interesting Catch 22.
If you accept this premise as true, then Rust becomes ultimately pointless. Because the prescription would be to become a very good programmer and learn how to write memory safe code, at which point you really don't need any other language except C. You will likely write memorypools that have mechanisms to make sure that you never access memory you are not supposed to.
An alternative prescription is to dedicate effort to making higher level language compilers/transpilers better. That way everyone codes in Python, the compiler recognizes common access patterns, and does iterative optimization (probably ML powered) to get you the most optimal machine code.
If you accept the premise as false, i.e writing memory safe code is easy, then Rust doesn't really offer any benefits.
>Because the prescription would be to become a very good programmer and learn how to write memory safe code, at which point you really don't need any other language except C.
Yes. Indeed you don't need any other language than C. Every program could be written in it.
The point of rust is to enforce strict rules on code that guarantees certain behaviour, but still allow operations which could violate those behaviours.
>Because the prescription would be to become a very good programmer and learn how to write memory safe code
No. The prescription is to be careful when dealing with memory. This is true for literally any language, the goal of rust is that the compiler can take away some of that workload.
Yes. Indeed you don't need any other language than C. Every program could be written in it.
The point of rust is to enforce strict rules on code that guarantees certain behaviour, but still allow operations which could violate those behaviours.
>Because the prescription would be to become a very good programmer and learn how to write memory safe code
No. The prescription is to be careful when dealing with memory. This is true for literally any language, the goal of rust is that the compiler can take away some of that workload.
> Because the prescription would be to become a very good programmer and learn how to write memory safe code
The problem isn't becoming a very good programmer. That's not difficult. The problem is becoming a very good programmer who is never tired or rushed. Any correct code in C will have implicit rules about how the memory is managed that are either written in comments or reside in the programmer's head. This doesn't work because eventually some will make a code change without understanding fully the implicit rule (often this person is the same person who figured out the rule).
The problem isn't becoming a very good programmer. That's not difficult. The problem is becoming a very good programmer who is never tired or rushed. Any correct code in C will have implicit rules about how the memory is managed that are either written in comments or reside in the programmer's head. This doesn't work because eventually some will make a code change without understanding fully the implicit rule (often this person is the same person who figured out the rule).
>The problem is becoming a very good programmer who is never tired or rushed.
So what stops that programmer from being lazy and wrapping all the blocks in unsafe in a Rust code base?
So what stops that programmer from being lazy and wrapping all the blocks in unsafe in a Rust code base?
nothing, but a programmer who does so knows they are creating an abomination while in C, it's just a normal day. (and for anyone looking at the code, the rust version will look like a buggy mess, while the C version will look like correct C code).
Yeah this doesn't play out in real world.
Look at any large enterprise Java project. People take shortcuts and break things all the time, even with fairly strict typing and memory safety.
Of course you can enforce things like "no unsafes ever in codebase" on a project, but if you have to reply on guidance for development, then you can simply say "first we write a complete set of tests that are a contract on correct operation, then we develop code", at which point the language choice becomes irrelevant.
Look at any large enterprise Java project. People take shortcuts and break things all the time, even with fairly strict typing and memory safety.
Of course you can enforce things like "no unsafes ever in codebase" on a project, but if you have to reply on guidance for development, then you can simply say "first we write a complete set of tests that are a contract on correct operation, then we develop code", at which point the language choice becomes irrelevant.
Banning unsafe is actually really easy. Just one line `#![forbid(unsafe_code)]` at the start of your code.
bottom line is: this doesn't happen. Feel free to study why not.
Well, in the real world, the premise is false, but not because writing memory safe code is in fact easy, but because in aggregate in large code bases it's impossible.
The world would have looked a lot better if there had been a safe dialect of C that we could have used for 95% of the stack. Instead we wrote everything in the chainsaw juggling dialect.
Ability to isolate high-risk "hold my beer" code from boring glue code has many nice side effects.
In a team, you can agree who can write and review `unsafe` code, and not be afraid that your junior team members will cause some horror heisenbug (there can always be other types of higher-level bugs in safe code, but safe code doesn't cause them, you'd have them anyway, on top of memory issues).
People often move abstractions built from unsafe code (like clever data structures, concurrency primitives, OS or C library wrappers) into separate crates. This makes it easier to thoroughly test and fuzz just the unsafe code, without having to test a larger project as a whole (once you ensure your safe API is indeed upholding safety invariants, the compiler will prove all uses of that code are safe too).