Memory safety is necessary, not sufficient(steveklabnik.com)
steveklabnik.com
Memory safety is necessary, not sufficient
https://steveklabnik.com/writing/memory-safety-is-a-red-herring
161 comments
> If you read the essay, it says “worse is better” means (paraphrasing) it’s more important that C compilers be easy to implement than easy to use. Also, it is more important that the implementation—or the design of the implementation—of a piece of software be simple than that it be correct. It is more important that it be simple than that it be consistent. It is more important that it be simple than that it be “complete” (for example, handle edge cases; it just needs to work in “most situations”).
This is really ironic considering C++, whose compilers may literally be the most complicated compilers to ever exist, and none of them implement the full (C++23) language spec.
What kind of maxim does C++ go by?
This is really ironic considering C++, whose compilers may literally be the most complicated compilers to ever exist, and none of them implement the full (C++23) language spec.
What kind of maxim does C++ go by?
1. If you don’t use a language feature in your code, your code shouldn’t pay the cost—no matter what else might be helped, enabled or prevented by that. To this day compilers let entire features like exceptions and RTTI be disabled entirely rather than accept that there might possibly be some costs worth imposing.
2. Binary compatibility and dynamic linking don’t matter at all, and shouldn’t ever be considered in design or revision of anything language-related.
3. All areas of computer science and engineering are open to being a part of the standard library and existing language and standard library authors are the best judges of what should be in the standard and how it should be implemented than anyone else, including platform vendors or subject matter experts.
2. Binary compatibility and dynamic linking don’t matter at all, and shouldn’t ever be considered in design or revision of anything language-related.
3. All areas of computer science and engineering are open to being a part of the standard library and existing language and standard library authors are the best judges of what should be in the standard and how it should be implemented than anyone else, including platform vendors or subject matter experts.
Zero cost abstractions & backward compatibility over implementation complexity, user friendliness or safety.
It's not like PL/I didn't allow for similar flaws, so "Worse is better" is neither here nor there. At least C was simple enough to learn for most programmers. It's still generally the case that a simple approach is more likely to be correct than an overly complicated one for which it's not even clear what "correct" means.
> It's not like PL/I didn't allow for similar flaws, so "Worse is better" is neither here nor there.
PL/I is one of the reasons that Multics was not plagued by buffer overflows:
"The net result is that a PL/I programmer would have to work very hard to program a buffer overflow error, while a C programmer has to work very hard to avoid programming a buffer overflow error." [1]
[1] P. Karger, R. Schell, "Thirty Years Later: Lessons from the Multics Security Evaluation"
PL/I is one of the reasons that Multics was not plagued by buffer overflows:
"The net result is that a PL/I programmer would have to work very hard to program a buffer overflow error, while a C programmer has to work very hard to avoid programming a buffer overflow error." [1]
[1] P. Karger, R. Schell, "Thirty Years Later: Lessons from the Multics Security Evaluation"
Software was significantly simpler in those days. I don't find it strange that they took a simplified view of software engineering. Specifically because those exact same simplified views still exist today - talk to people who've never worked on large complex systems, and you will usually encounter similar "anti-perfectionism".
People adopt simplified engineering practices when working on simple software by themselves, compared to when working on complex software within a large team.
People adopt simplified engineering practices when working on simple software by themselves, compared to when working on complex software within a large team.
One of the reasons C became popular because the compiler was actually reasonable to use on the hardware of the time.
Yeah it's hard for a lot of programmers to context switch back to when a megabyte a HUGE amount of memory (or even 640kB). I think everyone should play around on arduino uno or even smaller memory cpus to get some perspective. You didn't have gigabytes of rams to analyze all corner cases, data races, etc. during compilations.
>it is more important that the implementation—or the design of the implementation—of a piece of software be simple than that it be correct.
This is true and when your compiler actually abides by these values it is shocking how many issues just go away. The problem is that gcc and clang are nowhere near a simple implementation, asking the question of how exactly gcc or clang arrived at some given assembly for some given input is a really complicated answer. So much so that it makes me say that the relationship between the programmer and the compiler becomes adversarial. It would be one thing to just have very complicated optimizations but the whole "undefined behavior lets us to anything" approach is what makes it unbearable.
People can (and do) point at the C spec for fault of this and it is true that if the C spec was more strict then these compilers would not have the free pass to do these crazy miscompilations. However there is nothing stopping these compilers from just not doing that, there is nothing stopping them from just defining their own sane behavior for what the C spec defines as undefined behavior. It is no longer the case where we have a dozen or so C compilers that people need to target with their programs, the spec is not the bottom line anymore.
The Plan 9 compilers are the perfect example of getting this right. They define sane behavior that a reasonable programmer would expect for what the C spec calls undefined behavior. They are not gargantuan, their optimizations are not crazy. It is generally easy to understand how the compiler ended up with the assembly you see in the binary. Yet they are competent enough to selfhost an entire OS. The insane complexity of these other C compilers is simply not mandatory. They are not perfect of course it is still possible to write bad code but the result is no longer pathological, which is a giant help when you're actually trying to figure out what is going wrong.
This is true and when your compiler actually abides by these values it is shocking how many issues just go away. The problem is that gcc and clang are nowhere near a simple implementation, asking the question of how exactly gcc or clang arrived at some given assembly for some given input is a really complicated answer. So much so that it makes me say that the relationship between the programmer and the compiler becomes adversarial. It would be one thing to just have very complicated optimizations but the whole "undefined behavior lets us to anything" approach is what makes it unbearable.
People can (and do) point at the C spec for fault of this and it is true that if the C spec was more strict then these compilers would not have the free pass to do these crazy miscompilations. However there is nothing stopping these compilers from just not doing that, there is nothing stopping them from just defining their own sane behavior for what the C spec defines as undefined behavior. It is no longer the case where we have a dozen or so C compilers that people need to target with their programs, the spec is not the bottom line anymore.
The Plan 9 compilers are the perfect example of getting this right. They define sane behavior that a reasonable programmer would expect for what the C spec calls undefined behavior. They are not gargantuan, their optimizations are not crazy. It is generally easy to understand how the compiler ended up with the assembly you see in the binary. Yet they are competent enough to selfhost an entire OS. The insane complexity of these other C compilers is simply not mandatory. They are not perfect of course it is still possible to write bad code but the result is no longer pathological, which is a giant help when you're actually trying to figure out what is going wrong.
> People can (and do) point at the C spec for fault of this and it is true that if the C spec was more strict then these compilers would not have the free pass to do these crazy miscompilations. However there is nothing stopping these compilers from just not doing that, there is nothing stopping them from just defining their own sane behavior for what the C spec defines as undefined behavior. It is no longer the case where we have a dozen or so C compilers that people need to target with their programs, the spec is not the bottom line anymore.
Some compilers (like gcc and clang) do give you the option to make most undefined behaviors well-defined, using flags like -fwrapv, -fno-delete-null-pointer-checks, -fno-strict-aliasing, etc.
The key thing to understand is that if you use these flags, you opt-in to a non-standard C dialect. Going forward, you are no longer writing C code. Your code is now non-portable. Your code can no longer be compiled by a compiler that does not support these special C dialects.
Using these C dialects is a perfectly sensible thing to do, as long as you understand that this is what you are doing.
Some compilers (like gcc and clang) do give you the option to make most undefined behaviors well-defined, using flags like -fwrapv, -fno-delete-null-pointer-checks, -fno-strict-aliasing, etc.
The key thing to understand is that if you use these flags, you opt-in to a non-standard C dialect. Going forward, you are no longer writing C code. Your code is now non-portable. Your code can no longer be compiled by a compiler that does not support these special C dialects.
Using these C dialects is a perfectly sensible thing to do, as long as you understand that this is what you are doing.
> Your code can no longer be compiled by a compiler that does not support these special C dialects.
What compiler might that be?
It's true that MSVC doesn't have an equivalent of -fno-strict-aliasing, but that's because it just doesn't apply optimizations that assume strict aliasing in the first place. Admittedly the picture around signed overflow is more complicated, but it's essentially the same story.
> Using these C dialects is a perfectly sensible thing to do, as long as you understand that this is what you are doing.
I suppose that they technically are C dialects, but it seem more than a bit absurd to put it like that -- at least to me. By that standard the Linux kernel isn't written in C. And Firefox isn't written in C++. Postgres also wouldn't count as according-to-hoyle C under your definition. Even though Postgres compiles when -fwrapv and -fno-strict-aliasing are removed, and still passes all tests.
The implication of what you're saying seems to be that all of these open source projects each independently decided to "go there own way". I find it far more natural to explain the situation as one of GCC diverging when it decided to make -fstrict-aliasing the default around 15 years ago.
What compiler might that be?
It's true that MSVC doesn't have an equivalent of -fno-strict-aliasing, but that's because it just doesn't apply optimizations that assume strict aliasing in the first place. Admittedly the picture around signed overflow is more complicated, but it's essentially the same story.
> Using these C dialects is a perfectly sensible thing to do, as long as you understand that this is what you are doing.
I suppose that they technically are C dialects, but it seem more than a bit absurd to put it like that -- at least to me. By that standard the Linux kernel isn't written in C. And Firefox isn't written in C++. Postgres also wouldn't count as according-to-hoyle C under your definition. Even though Postgres compiles when -fwrapv and -fno-strict-aliasing are removed, and still passes all tests.
The implication of what you're saying seems to be that all of these open source projects each independently decided to "go there own way". I find it far more natural to explain the situation as one of GCC diverging when it decided to make -fstrict-aliasing the default around 15 years ago.
> It's true that MSVC doesn't have an equivalent of -fno-strict-aliasing, but that's because it just doesn't apply optimizations that assume strict aliasing in the first place. Admittedly the picture around signed overflow is more complicated, but it's essentially the same story.
Of course it's fine if the compiler doesn't implement this as an option, but rather as the default behavior -- as long as this is actually a documented guarantee, rather than "we just haven't implemented those optimizations yet, we might start exploiting this UB at any point in the future". I'm not familiar with what guarantees MSVC documents in this area.
> By that standard the Linux kernel isn't written in C.
I think that in a very real sense, it isn't. The kernel is written in standard C plus a few hundred GCC extensions, of which additional compiler options are but a small part. It took very extensive effort to make the kernel compile with clang (which already was mostly gcc-compatible, but nowhere near enough for the kernel's level of extension use).
> Even though Postgres compiles when -fwrapv and -fno-strict-aliasing are removed, and still passes all tests.
I think there is a lot of difference between "requires signed integer overflow to be well-defined because we explicitly rely on it" and "disable optimization based on signed integer overflow as a hardening measure, in case we got this wrong somewhere".
> The implication of what you're saying seems to be that all of these open source projects each independently decided to "go there own way".
You kind of make it sound like use of these options is common. To the best of my knowledge, this is not the case, and these projects are rare exceptions, not the rule. So, yes, they decided to go their own way (which is perfectly fine.)
Of course it's fine if the compiler doesn't implement this as an option, but rather as the default behavior -- as long as this is actually a documented guarantee, rather than "we just haven't implemented those optimizations yet, we might start exploiting this UB at any point in the future". I'm not familiar with what guarantees MSVC documents in this area.
> By that standard the Linux kernel isn't written in C.
I think that in a very real sense, it isn't. The kernel is written in standard C plus a few hundred GCC extensions, of which additional compiler options are but a small part. It took very extensive effort to make the kernel compile with clang (which already was mostly gcc-compatible, but nowhere near enough for the kernel's level of extension use).
> Even though Postgres compiles when -fwrapv and -fno-strict-aliasing are removed, and still passes all tests.
I think there is a lot of difference between "requires signed integer overflow to be well-defined because we explicitly rely on it" and "disable optimization based on signed integer overflow as a hardening measure, in case we got this wrong somewhere".
> The implication of what you're saying seems to be that all of these open source projects each independently decided to "go there own way".
You kind of make it sound like use of these options is common. To the best of my knowledge, this is not the case, and these projects are rare exceptions, not the rule. So, yes, they decided to go their own way (which is perfectly fine.)
>The key thing to understand is that if you use these flags, you opt-in to a non-standard C dialect.
This is not true. The C Standard explicitly states that a conforming implementation is welcome to provide well defined semantics to behavior that the standard states is undefined.
The whole point of undefined behavior is that the C Standard imposes no requirement on implementations about the semantics of that program, so if a specific implementation adds some kind of checks or provides some sort of deterministic behavior to something that is otherwise undefined, the program itself is still C code and adheres to the C Standard.
>Your code can no longer be compiled by a compiler that does not support these special C dialects.
Undefined behavior is a semantic property of a program, not a syntactic property, so any other conforming C compiler will have no problem compiling it.
This is not true. The C Standard explicitly states that a conforming implementation is welcome to provide well defined semantics to behavior that the standard states is undefined.
The whole point of undefined behavior is that the C Standard imposes no requirement on implementations about the semantics of that program, so if a specific implementation adds some kind of checks or provides some sort of deterministic behavior to something that is otherwise undefined, the program itself is still C code and adheres to the C Standard.
>Your code can no longer be compiled by a compiler that does not support these special C dialects.
Undefined behavior is a semantic property of a program, not a syntactic property, so any other conforming C compiler will have no problem compiling it.
For example Linux uses -fno-delete-null-pointer-checks
> The key thing to understand is that if you use these flags, you opt-in to a non-standard C dialect.
Isn't all C with undefined behavior non-standard. Or is there a standard for undefined behavior (obviously not, I would think)?
I don't understand why those flags wouldn't be turned on by default. Or do they affect more than just undefined behavior?
Isn't all C with undefined behavior non-standard. Or is there a standard for undefined behavior (obviously not, I would think)?
I don't understand why those flags wouldn't be turned on by default. Or do they affect more than just undefined behavior?
They affect performance, forcing the dev to insure they aren't doing unsafe/ill-defined things which is almost impossible on huge code bases in c/c++
They are unhappy being forced to write code that is free of undefined behavior?
It reminds me of the joke: Alice claims she is very fast at mental math. Bob asks "what's 4821 times 5997?" Alice replies "ten thousand." Bob says, "What, no, that's wrong, very wrong." Alice says, "But it was fast!"
Are you telling me C / C++ developers are like Alice? When given the choice between fast undefined behavior and slower but more likely to be correct undefined behavior, developers will choose the faster option that is more likely to be incorrect?
It reminds me of the joke: Alice claims she is very fast at mental math. Bob asks "what's 4821 times 5997?" Alice replies "ten thousand." Bob says, "What, no, that's wrong, very wrong." Alice says, "But it was fast!"
Are you telling me C / C++ developers are like Alice? When given the choice between fast undefined behavior and slower but more likely to be correct undefined behavior, developers will choose the faster option that is more likely to be incorrect?
You seem to be assuming that on a large code base that you have top 1% programmers. I suppose that is the case for some places that are paying $500k average pay, but most businesses don't pay that and have average developers and a few top 5% if big enough. That's why tools that check for issues like linters and sanitizers help, especially if you make them part of "the process" for checkin
> piece of software be simple than that it be correct
Here is the thing: which of these is more plausible or at least less far-fetched?
A. Write a program that is not correct now, but will eventually be.
B. Write a program that is complicated now, but will eventually be simpler.
:)
Simplicity isn't something you can leave out now and add later, yet correctness can often be treated that way. Even a shop that values correctness above all else still does debugging. (If not code, then debugging their proofs, and debugging whether their formal specification actually implement the functional requirements).
Here is the thing: which of these is more plausible or at least less far-fetched?
A. Write a program that is not correct now, but will eventually be.
B. Write a program that is complicated now, but will eventually be simpler.
:)
Simplicity isn't something you can leave out now and add later, yet correctness can often be treated that way. Even a shop that values correctness above all else still does debugging. (If not code, then debugging their proofs, and debugging whether their formal specification actually implement the functional requirements).
If you simplify a correct program, it will most likely still be correct. If you "correct" a simple program, it probably becomes complicated.
If you algebraically simplify a correct program it will be correct.
But the simplicity which is at play in the context of "worse is better" is simplicity of requirement specifications themselves, before any code is written. (I think I'm the one who muddied the waters here by insinuating that correctness is a matter of debugging an incorrect program against a correct specification.)
In projects that value simplicity over correctness, what that means is that what is considered correct (as in the requirement we shall implement) is the simpler requirements, which are regarded as incorrect by other projects.
Programs that implement the complex requirements are vanishingly improbable to be simplified into programs that implement simple requirements, simply because those are breaking changes.
E.g. you can't take a database engine that provides certain consistency guarantees and make it have weaker guarantees in the next version (for the sake of simplicity), without breaking all the applications that depend on the current guarantees.
Correctness can be added --- including at the requirements level, not only debugging. It can be because it's often backwards compatible. E.g. adding handling for cases that were previously ignored.
But the simplicity which is at play in the context of "worse is better" is simplicity of requirement specifications themselves, before any code is written. (I think I'm the one who muddied the waters here by insinuating that correctness is a matter of debugging an incorrect program against a correct specification.)
In projects that value simplicity over correctness, what that means is that what is considered correct (as in the requirement we shall implement) is the simpler requirements, which are regarded as incorrect by other projects.
Programs that implement the complex requirements are vanishingly improbable to be simplified into programs that implement simple requirements, simply because those are breaking changes.
E.g. you can't take a database engine that provides certain consistency guarantees and make it have weaker guarantees in the next version (for the sake of simplicity), without breaking all the applications that depend on the current guarantees.
Correctness can be added --- including at the requirements level, not only debugging. It can be because it's often backwards compatible. E.g. adding handling for cases that were previously ignored.
You add in simplicity by taking out code. It is very possible (and common) to find better abstractions or methods to approach a problem that reduce the complexity of the code.
It is vanishingly uncommon to simplify an entire application, so that it goes from something complex to something whose number one value is simplicity, such that the simplicity is reflected in the actual functional specification.
How do the Plan 9 compilers compare to gcc/clang when it comes to performance or portability?
The main problem is this: the undefined status of one construct A in the program changes the behavior of a different, independent construct B in the program, even in cases when, say, B executes first and is correct. Everything is jumbled together in the optimizer, which establishes logical ties between the pieces that are unrelated to the network of intent.
If the undefined behavior of construct A causes just A to misbehave, we are still in sane waters.
If the undefined behavior of construct A causes just A to misbehave, we are still in sane waters.
With all due respect, these “adversial compiler” expression just makes zero sense, and takes a lot away from your comment.
Guess what, the world is complex, and software has no bound for complexity. Which is better, a multi-million lines compiler that hundreds of people worked on for decades, or a toy one in a couple thousand lines written by a single programmer? What if the former can create 2-10x faster code than the latter (I probably even underestimate it, loop unswitching, vectorization, etc. can account for even more differences).
It turns out that we can build abstractions on top of abstractions, and if it’s designed well, it will scale with complexity (which we require). Would you change back to an OS that didn’t handle multithreading as it’s too complicated? Or that wouldn’t use GPUs?
Guess what, the world is complex, and software has no bound for complexity. Which is better, a multi-million lines compiler that hundreds of people worked on for decades, or a toy one in a couple thousand lines written by a single programmer? What if the former can create 2-10x faster code than the latter (I probably even underestimate it, loop unswitching, vectorization, etc. can account for even more differences).
It turns out that we can build abstractions on top of abstractions, and if it’s designed well, it will scale with complexity (which we require). Would you change back to an OS that didn’t handle multithreading as it’s too complicated? Or that wouldn’t use GPUs?
>With all due respect, these “adversial compiler” expression just makes zero sense, and takes a lot away from your comment.
I was describing how it seems these complex compilers look for excuses to give you miscompilations, this "looking for gotcha's" makes the relationship appear adversarial to me.
>Guess what, the world is complex, and software has no bound for complexity.
My argument is that it really ought to. I think there is a diminishing return.
>What if the former can create 2-10x faster code than the latter (I probably even underestimate it, loop unswitching, vectorization, etc. can account for even more differences).
In code I've used with both compilers this has not been the case. Good profiling tools and manual hotspot optimization go a really long way.
>It turns out that we can build abstractions on top of abstractions, and if it’s designed well, it will scale with complexity
GCC has not scaled with complexity well enough, that's why we have people who lament its behavior.
>Would you change back to an OS that didn’t handle multithreading as it’s too complicated?
Concurrent programming makes problems easier, not harder in my experience. For what it's worth Plan 9 has excellent concurrent programming facilities.
>Or that wouldn’t use GPUs?
GPUs these days only work on systems in large part due to graces of the vendors themselves. Would nvidia or AMD be usable (enough) on Linux if there was not dedicated people from those companies working on drivers?
I was describing how it seems these complex compilers look for excuses to give you miscompilations, this "looking for gotcha's" makes the relationship appear adversarial to me.
>Guess what, the world is complex, and software has no bound for complexity.
My argument is that it really ought to. I think there is a diminishing return.
>What if the former can create 2-10x faster code than the latter (I probably even underestimate it, loop unswitching, vectorization, etc. can account for even more differences).
In code I've used with both compilers this has not been the case. Good profiling tools and manual hotspot optimization go a really long way.
>It turns out that we can build abstractions on top of abstractions, and if it’s designed well, it will scale with complexity
GCC has not scaled with complexity well enough, that's why we have people who lament its behavior.
>Would you change back to an OS that didn’t handle multithreading as it’s too complicated?
Concurrent programming makes problems easier, not harder in my experience. For what it's worth Plan 9 has excellent concurrent programming facilities.
>Or that wouldn’t use GPUs?
GPUs these days only work on systems in large part due to graces of the vendors themselves. Would nvidia or AMD be usable (enough) on Linux if there was not dedicated people from those companies working on drivers?
> It was a very different world, and just being able to write in a higher-level language than assembly on a particular computer was a big deal.
It was different then, but not by too much. Our tooling, languages, etc, has improved yes, but the underlying hardware still exists and occasionally the general solution provided by our improved tooling is sub optimal for our use case.
I actually _like_ where C has wound up. I have way better static analysis tooling at my disposal than I had 15 years ago and I can use a "memory safe" language like python or OCaml and say "hey, I know better here", think carefully, and push the memory safe compiler out of my way.
It was different then, but not by too much. Our tooling, languages, etc, has improved yes, but the underlying hardware still exists and occasionally the general solution provided by our improved tooling is sub optimal for our use case.
I actually _like_ where C has wound up. I have way better static analysis tooling at my disposal than I had 15 years ago and I can use a "memory safe" language like python or OCaml and say "hey, I know better here", think carefully, and push the memory safe compiler out of my way.
One of the reasons C became popular because the compiler was actually reasonable to use on the hardware of the time.
"Worse is better" is much older then "undefined behaviour". Undefined behaviour was invented for C standardization, when C was already mature and had been out of the Unix childhood home for a long time.
For example, many UB semantics in the standard come from allowing for C ports for strange non-unix hardware, and making bold decisions when developments in compiler optimization state of the art ran into underspecified corners of C semantics.
For example, many UB semantics in the standard come from allowing for C ports for strange non-unix hardware, and making bold decisions when developments in compiler optimization state of the art ran into underspecified corners of C semantics.
"Worse Is Better", as a description of a set of engineering values embodied by C, is from 1989.
I'm taking your word for it that UB was invented for C standardization (I have no knowledge of that history, and your claim seems plausible), and I'm going to say that probably they didn't invent it in the last year of the seven-year standardization effort, so probably UB is older than Worse Is Better.
I'm taking your word for it that UB was invented for C standardization (I have no knowledge of that history, and your claim seems plausible), and I'm going to say that probably they didn't invent it in the last year of the seven-year standardization effort, so probably UB is older than Worse Is Better.
uh. No. Rust unsafe gives rust behavior a lot like C. If you at all break the rather subtle rules, then essentially anything can and will happen.
So for example, there was recently a thread where someone had code that checked if a value was in range to safely coerce it directly to an enum then did so. But because of eager evaluation of an argument the unsafe cast happened first. From this the compiler reasoned that the variable was preconditionally range constrained to always be in range and it optimized out the in-range test (which itself was not unsafe code).
This is a classic C bug where someone implements an overflow check that itself can overflow, causing the branch for overflow to get optimized out. But at least in C the simpler syntax at least made it clear that the triggering code got executed first. The more complex rust syntax obscured that.
Rust has improved the situation by narrowing the cases where you can get into this trouble, but on the other hand it adds a lot of other complexity that contributes to faulty code (and a nearly mandatory packaging ecosystem which is a security nightmare-- it's the norm for even simple rust utilities to pull in a million lines of unauditable (just by bulk) third party code, including multiple HTTPS libraries).
As a result, I don't think it can be taken for granted that rust as a whole is an advancement in software integrity-- it may be, but it's something that ought to be formally studied. In some cases rust might be replacing memory safety bugs with an even greater number of other defects which, depending on the application, may be worse. (not everything is an internet exposed service where hacks are the only failure of consequence and where input really should be assumed to be intelligently adversarial.)
In any case, "break the rules and all bets are off" is an issue that likely will continue to exist in any performant language. Automatic code generation will generate stuff with awful performance unless an optimizer goes through and eliminates 'impossible cases', but optimization isn't possible unless the compiler can assume the rules are followed.
So for example, there was recently a thread where someone had code that checked if a value was in range to safely coerce it directly to an enum then did so. But because of eager evaluation of an argument the unsafe cast happened first. From this the compiler reasoned that the variable was preconditionally range constrained to always be in range and it optimized out the in-range test (which itself was not unsafe code).
This is a classic C bug where someone implements an overflow check that itself can overflow, causing the branch for overflow to get optimized out. But at least in C the simpler syntax at least made it clear that the triggering code got executed first. The more complex rust syntax obscured that.
Rust has improved the situation by narrowing the cases where you can get into this trouble, but on the other hand it adds a lot of other complexity that contributes to faulty code (and a nearly mandatory packaging ecosystem which is a security nightmare-- it's the norm for even simple rust utilities to pull in a million lines of unauditable (just by bulk) third party code, including multiple HTTPS libraries).
As a result, I don't think it can be taken for granted that rust as a whole is an advancement in software integrity-- it may be, but it's something that ought to be formally studied. In some cases rust might be replacing memory safety bugs with an even greater number of other defects which, depending on the application, may be worse. (not everything is an internet exposed service where hacks are the only failure of consequence and where input really should be assumed to be intelligently adversarial.)
In any case, "break the rules and all bets are off" is an issue that likely will continue to exist in any performant language. Automatic code generation will generate stuff with awful performance unless an optimizer goes through and eliminates 'impossible cases', but optimization isn't possible unless the compiler can assume the rules are followed.
> If you at all break the rather subtle rules, then essentially anything can and will happen.
If by subtle rules you mean your invariants, that is missing fundamental assumptions.
It's akin to making a building without foundation and load bearing structures.
> So for example, there was recently a thread where someone had code that checked if a value was in range to safely coerce it directly to an enum then did so. But because of eager evaluation of an argument the unsafe cast happened first
You mean this: https://notgull.net/cautionary-unsafe-tale/
However note the UB goes away if you never use any unsafe code. Or if you expand your unsafe to encompas some safe code.
Issue it had was safe code was invalidating the invariants unsafe code was relying on. Iirc alignment.
If by subtle rules you mean your invariants, that is missing fundamental assumptions.
It's akin to making a building without foundation and load bearing structures.
> So for example, there was recently a thread where someone had code that checked if a value was in range to safely coerce it directly to an enum then did so. But because of eager evaluation of an argument the unsafe cast happened first
You mean this: https://notgull.net/cautionary-unsafe-tale/
However note the UB goes away if you never use any unsafe code. Or if you expand your unsafe to encompas some safe code.
Issue it had was safe code was invalidating the invariants unsafe code was relying on. Iirc alignment.
> If by subtle rules you mean your invariants, that is missing fundamental assumptions.
> It's akin to making a building without foundation and load bearing structures.
That's exactly how C approaches UB too.
> However note the UB goes away if you never use any unsafe code. Or if you expand your unsafe to encompas some safe code.
Right. The problem is that's untenable; any nontrivial program will have unsafe somewhere, and unsafe can cause failures arbitrarily far away from the incorrect code. The whole point of the article is that you need to be able to draw an actual boundary between the unsafe parts and the safe parts of your program and review the unsafe parts in isolation. If Rust doesn't give you more and better support in doing that than C does, then it's not really making a difference.
> It's akin to making a building without foundation and load bearing structures.
That's exactly how C approaches UB too.
> However note the UB goes away if you never use any unsafe code. Or if you expand your unsafe to encompas some safe code.
Right. The problem is that's untenable; any nontrivial program will have unsafe somewhere, and unsafe can cause failures arbitrarily far away from the incorrect code. The whole point of the article is that you need to be able to draw an actual boundary between the unsafe parts and the safe parts of your program and review the unsafe parts in isolation. If Rust doesn't give you more and better support in doing that than C does, then it's not really making a difference.
The problem is that's untenable; any nontrivial program will have unsafe somewhere
I'm up to 47,000 lines of Rust with no "unsafe". The main program starts with
#![forbid(unsafe_code)]
and that applies to the entire program, although not external crates. If you're not using foreign functions, you don't need "unsafe".
Some published crates I use do have "unsafe", more than they should. This is annoying.
I'm up to 47,000 lines of Rust with no "unsafe". The main program starts with
#![forbid(unsafe_code)]
and that applies to the entire program, although not external crates. If you're not using foreign functions, you don't need "unsafe".
Some published crates I use do have "unsafe", more than they should. This is annoying.
> If Rust doesn't give you more and better support in doing that than C does, then it's not really making a difference.
It does and if you've been missing that you've misunderstood this whole discussion. Rust allows you to wrap up some unsafe code in a safe abstraction. You use the type checker to enforce your invariants, such that the unsafe code can be reviewed in isolation.
Rust gives you exactly what you're saying you want. Steve is describing Rust as it is.
It does and if you've been missing that you've misunderstood this whole discussion. Rust allows you to wrap up some unsafe code in a safe abstraction. You use the type checker to enforce your invariants, such that the unsafe code can be reviewed in isolation.
Rust gives you exactly what you're saying you want. Steve is describing Rust as it is.
> It does and if you've been missing that you've misunderstood this whole discussion.
Don't tell it to me, tell it to the person I was replying to.
Don't tell it to me, tell it to the person I was replying to.
I believe you said
> any nontrivial program will have unsafe somewhere, and unsafe can cause failures arbitrarily far away from the incorrect code
But the point I was trying to make is that this claim is facially true but misses the critical context: in idiomatic Rust, the unsafe keyword is not the barrier that enforces integrity of the system.
The integrity of the system is enforced by the type checker, the same way it always is. The unsafe annotation alerts the reader that there exists an invariant the compiler can't check. Idiomatic code will have a SAFETY comment above describing the invariant. It should be locally possible to reason about how the type abstraction used to encapaulate truly enforces that invariant.
This is what Rust people are talking about when we say wrap unsafe code in a safe abstraction. If we do it right, it's no longer possible for spooky action at a distance.
Now you might go and point out the fallibility of humans and all that, and you'd be right, but that's in fact what makes it so valuable to try.
> any nontrivial program will have unsafe somewhere, and unsafe can cause failures arbitrarily far away from the incorrect code
But the point I was trying to make is that this claim is facially true but misses the critical context: in idiomatic Rust, the unsafe keyword is not the barrier that enforces integrity of the system.
The integrity of the system is enforced by the type checker, the same way it always is. The unsafe annotation alerts the reader that there exists an invariant the compiler can't check. Idiomatic code will have a SAFETY comment above describing the invariant. It should be locally possible to reason about how the type abstraction used to encapaulate truly enforces that invariant.
This is what Rust people are talking about when we say wrap unsafe code in a safe abstraction. If we do it right, it's no longer possible for spooky action at a distance.
Now you might go and point out the fallibility of humans and all that, and you'd be right, but that's in fact what makes it so valuable to try.
Exactly. All-hope-lost behavior is possible in rust code unless there is no unsafe anywhere (and no compiler bugs, but I think its fair to ignore those when discussing the language in the abstract).
Rust potentially benefits from fewer opportunities to footgun yourself, but rust also comes with other costs (including a more complex syntax, a bad dependency culture, a lot more front-loaded cognitive load around lifetime management, etc.) which might offset those benefits. Some of those extra complexity costs seem hard (or impossible) to avoid when trying to keep memory safety from having a disproportional runtime cost, so I'm not necessarily faulting rust. But some of the sources of defects in rust code may also be entirely avoidable, which is why I think it's important to actually study it rather than axiomatically assume its behavior makes programs correct. It doesn't, even absent unsafe.
Rust potentially benefits from fewer opportunities to footgun yourself, but rust also comes with other costs (including a more complex syntax, a bad dependency culture, a lot more front-loaded cognitive load around lifetime management, etc.) which might offset those benefits. Some of those extra complexity costs seem hard (or impossible) to avoid when trying to keep memory safety from having a disproportional runtime cost, so I'm not necessarily faulting rust. But some of the sources of defects in rust code may also be entirely avoidable, which is why I think it's important to actually study it rather than axiomatically assume its behavior makes programs correct. It doesn't, even absent unsafe.
> compiler bugs... its fair to ignore those when discussing the language in the abstract
When the language is defined as whatever the compiler does, as it is in the case of Rust, I'm not so sure.
If there were a Rust standard with multiple compliant compilers, I'd be more convinced, but Rust isn't there yet.
(And TRF's trademark policy may well prevent it from ever getting there.)
When the language is defined as whatever the compiler does, as it is in the case of Rust, I'm not so sure.
If there were a Rust standard with multiple compliant compilers, I'd be more convinced, but Rust isn't there yet.
(And TRF's trademark policy may well prevent it from ever getting there.)
Having more implementations doesn’t really help you avoid bugs in the one you’re using.
Not entirely true. When there are multiple implementations disagreements between them in normative behavior prove one or the other (or the language itself) has a bug. This means you can run randomized tests in your software, hash the results, and the hash should be the same between compilers (and across platforms).
That's an option that doesn't exist when there aren't multiple, and even if you won't use the potential personally other people using it will eliminate bugs in the compiler. The csmith program was used to find an incredible amount of bugs in gcc and clang. This approach can also be applied to many pieces of ordinary code too.
Prior poster's point is also that when there is only one the language is the compiler and it's not useful to talk about them in isolation. I don't fully agree with that in theory since the language is supposed to be stable-ish while bugs in the compiler will get fixed. So to the extent that something in rust is a footgun due to the language we might be stuck with it, but if its due to a compiler bug it will probably be fixed.
That's an option that doesn't exist when there aren't multiple, and even if you won't use the potential personally other people using it will eliminate bugs in the compiler. The csmith program was used to find an incredible amount of bugs in gcc and clang. This approach can also be applied to many pieces of ordinary code too.
Prior poster's point is also that when there is only one the language is the compiler and it's not useful to talk about them in isolation. I don't fully agree with that in theory since the language is supposed to be stable-ish while bugs in the compiler will get fixed. So to the extent that something in rust is a footgun due to the language we might be stuck with it, but if its due to a compiler bug it will probably be fixed.
[deleted]
> That's exactly how C approaches UB too.
> The problem is that's untenable; any nontrivial program will have unsafe somewhere, and unsafe can cause failures arbitrarily far away from the incorrect code.
Safe code needs to be safe for any possible value of arguments, fields, variables. Doing otherwise is unsoundness bug.
In C, the code you need to look is essentially ALL of your code + C std lib. In Rust, you can focus on places where safe and unsafe code mingle. And IFF (if and only iff) your code contains 0 unsafe, can you be sure there is a Rust bug.
This however assumes any code.
Idiomatic code in Rust will treat unsafe with big large friendly letters `SAFETY: this works while X and Y hold`. How do you do this is C? Since most lines are possible source of UB do you annotate every line of code? I assume not. How do you ensure something will not be mutated? How do you ensure something is thread-safe? Do you use type bounds? And if not how?
> If Rust doesn't give you more and better support in doing that than C does, then it's not really making a difference.
Like you don't go around and say, "Well seatbelts don't prevent head injuries when you hit a wall head-on. So remove seatbelts!", you add airbags to cushion the blow.
And spoiler alert: It's giving you the airbags as well. The normal thing when encountering a UB isn't to try to track it down it's to run `miri`, it found UB as soon as it ran.
Particularly notable: due to the restrictions that Rust chooses to enforce on safe code, `unsafe` is forced in a lot of situations where other languages maintain full runtime safety. I'm not saying going full-Java is the only answer, but a language that is safer than Rust is certainly possible.
Like what exactly?
> Issue it had was safe code was invalidating the invariants unsafe code was relying on. Iirc alignment.
You do recall correctly. The safe code was producing a pointer with alignment that was off, and the unsafe code dereferenced it without checking. I felt that it was kind of a bad take when several people said that it was UB caused by safe code, because really the issue was that the unsafe code wasn't doing its job. The Rust unsafe model is that you can't trust safe code when inside unsafe code. It's on the unsafe code to uphold invariants which it requires, not the safe code which calls it
You do recall correctly. The safe code was producing a pointer with alignment that was off, and the unsafe code dereferenced it without checking. I felt that it was kind of a bad take when several people said that it was UB caused by safe code, because really the issue was that the unsafe code wasn't doing its job. The Rust unsafe model is that you can't trust safe code when inside unsafe code. It's on the unsafe code to uphold invariants which it requires, not the safe code which calls it
> As a result, I don't think it can be taken for granted that rust as a whole is an advancement in software integrity-- it may be, but it's something that ought to be formally studied. In some cases rust might be replacing memory safety bugs with an even greater number of other defects which, depending on the application, may be worse.
I’m sorry, but without any supporting evidence for this claim, this is just FUD. Everything that we’ve seen in case studies of people reimplementing stuff in Rust indicates that memory safety and logic bugs are derived compared to something like C.
I’m sorry, but without any supporting evidence for this claim, this is just FUD. Everything that we’ve seen in case studies of people reimplementing stuff in Rust indicates that memory safety and logic bugs are derived compared to something like C.
Citation welcome to those case studies, because I've not seen them. It's on the advocates of rust to establish that it makes things better because it absolutely isn't unambiguous.
We really seem to be in the stone age in terms of what practices lead to higher quality software. We still have people who chant "goto harmful" against one simply forward jumps to on-error-cleanup code, yet still litter their C++ and java with exceptions which are a less safe and less clear version of the same thing.
I've personally found the rate of embarrassing errors in simple rust software is increased over comparable C code, but I freely admit that this experience is far from a formal study and may well be due to the lack of problem-domain competence or general haste in people participating in culty "re-implement in rust" exercises (and where the rust code is far more often someone's "learn rust" project). And at least where security w/ untrusted input is a concern the nature of the rust bugs is preferable to the bugs in comparable C code, but as mentioned that doesn't apply to a lot of software.
Another data-point is that the vast majority of firefox crashes I experience now are rust panics, even though the amount of rust code is small compared to C++ code. It's hard to reason from that however, since it can be said that the rust code is more complex and more heavily used than the bulk of the rest.
We really seem to be in the stone age in terms of what practices lead to higher quality software. We still have people who chant "goto harmful" against one simply forward jumps to on-error-cleanup code, yet still litter their C++ and java with exceptions which are a less safe and less clear version of the same thing.
I've personally found the rate of embarrassing errors in simple rust software is increased over comparable C code, but I freely admit that this experience is far from a formal study and may well be due to the lack of problem-domain competence or general haste in people participating in culty "re-implement in rust" exercises (and where the rust code is far more often someone's "learn rust" project). And at least where security w/ untrusted input is a concern the nature of the rust bugs is preferable to the bugs in comparable C code, but as mentioned that doesn't apply to a lot of software.
Another data-point is that the vast majority of firefox crashes I experience now are rust panics, even though the amount of rust code is small compared to C++ code. It's hard to reason from that however, since it can be said that the rust code is more complex and more heavily used than the bulk of the rest.
I appreciate your comments in this thread. They are a healthy skeptical technology-neutral take unaffected by novel-paradigm-dogma – in this case about Rust, but it really applies to any software practice. I have nothing against Rust and I believe it has advanced the space of mainstream imperative languages significantly, particular in high performance low-level coding. But you have to always keep an eye on the horizon.
> We really seem to be in the stone age in terms of what practices lead to higher quality software.
I agree. What I always come back to is simplicity, or reducing total cognitive complexity. The only way I know of that actually consistently works, is modularization (in the most liberal sense of the word), in order for our human brains to work in a reduced problem domain at any given point in time. So whatever languages, protocols, tooling and design patterns help with that, I lean into, although this is (currently) a subjective measure.
On some problems, it’s clear Rust is amazing in this respect, taking away certain worries so that you can focus on “what matters”. But this is not a truism in all domains, because the issues that Rust addresses are merely a good list, not an exhaustive one.
> We really seem to be in the stone age in terms of what practices lead to higher quality software.
I agree. What I always come back to is simplicity, or reducing total cognitive complexity. The only way I know of that actually consistently works, is modularization (in the most liberal sense of the word), in order for our human brains to work in a reduced problem domain at any given point in time. So whatever languages, protocols, tooling and design patterns help with that, I lean into, although this is (currently) a subjective measure.
On some problems, it’s clear Rust is amazing in this respect, taking away certain worries so that you can focus on “what matters”. But this is not a truism in all domains, because the issues that Rust addresses are merely a good list, not an exhaustive one.
> the vast majority of firefox crashes I experience now are rust panics
Have you considered that this might be precisely because Rust is catching and stopping something bad with a safe panic? And that the same programmer making the same mistake in C/C++ would have resulted in silent data corruption, a security vulnerability, and possibly a portal into the space between dimensions from which optimising compilers will happily allow the Others into our world because -- and I quote -- "that's allowed by the spec"?
I often see in the news some panic about a sudden rise in some rare disease, but it almost always turns out that the increase is just due to an improvement in detection, not a change in the actual prevalence.
I suspect that silent data corruption vs liberal use of asserts that trigger visible panics is in the same category.
Someone in this thread was complaining that they think exceptions are "dangerous" and gotos are "safe". Their thinking is probably coloured by endless stack traces from exceptions in managed languages, comparing that to the "oops I stepped over a mandatory cleanup using goto" in their own C code, which they probably won't even notice... because it's probably just silently leaking memory. Or file handles. Or threads. But not loudly and in your face!
Have you considered that this might be precisely because Rust is catching and stopping something bad with a safe panic? And that the same programmer making the same mistake in C/C++ would have resulted in silent data corruption, a security vulnerability, and possibly a portal into the space between dimensions from which optimising compilers will happily allow the Others into our world because -- and I quote -- "that's allowed by the spec"?
I often see in the news some panic about a sudden rise in some rare disease, but it almost always turns out that the increase is just due to an improvement in detection, not a change in the actual prevalence.
I suspect that silent data corruption vs liberal use of asserts that trigger visible panics is in the same category.
Someone in this thread was complaining that they think exceptions are "dangerous" and gotos are "safe". Their thinking is probably coloured by endless stack traces from exceptions in managed languages, comparing that to the "oops I stepped over a mandatory cleanup using goto" in their own C code, which they probably won't even notice... because it's probably just silently leaking memory. Or file handles. Or threads. But not loudly and in your face!
> Have you considered that this might be precisely because Rust is catching and stopping something bad with a safe panic
Absolutely! But I counter: How do you know that all of them are? Or that even many of them are?
The developers that work on it may well know, but it isn't something we can axiomatically assume. It is not a true statement that any rust panic would have been an error in C(++) code written by the same (competent in both) developer.
Increased error visibility can be either more errors or more error sensitivity. It's worth being mindful of this because otherwise we may adopt programming practices that increase the overall defect rate, and think we're making things better.
Perhaps it's easier to see with asserts. If your defect rate is 1 defect per 1000 lines of code, and you go add 1000 asserts you'll add a bug (actually I think you'll add 10: the boundary conditions that asserts test are far more likely than average code to be implemented wrong). Like any other code they could also start off correct but then become desynced with the rest of the code. If you go and ship those asserts in production and the effect of asserting is important in your application, then unless the asserts prevent more bugs than they created you made the program worse off. Even if the asserts don't cause bugs they may make the program harder to maintain or extend. Therefore in any codebase there must be some optimal level of asserts, more or less and the program is worse off.
Is the implicit assertion of rust's behavior optimal? Probably not for all codes, because different codes call for different tradeoffs. Or even for every code there might be language changes that could make things better. But we'll never find out this stuff if we can't admit that there is the potential for uncertainty or improvement and confuse memory safety for program correctness. It's one of the more important aspects of correctness but the world is full of serious bugs in software written in inherently memory safe languages.
And after all, a program consisting of nothing but exit(1) (or perhaps while(;;){}) is the most perfectly memory safe program possible. :P If memory safety were the only goal programming would be much simpler.
> and possibly a portal into the space between dimensions from which optimising compilers will happily allow the Others into our world
As mentioned up thread, rust also can create portals into the hell dimension if there is unsafe code (including in the standard library) that fails to obey all the not-very-simple contract requirements. Also the number of times errant C code has actually opened a portal in to the hell dimension thereby destroying the universe is greatly overstated, that's only happened twice at most, usually undefined behavior is just a crash or an exploitable misbehavior that lets some haxor steal your data and those can also happen from pure logic errors (or a panic, in the case of a crash). I do fully agree that narrowing the potential for UB misbehavior is very important and rust is an important advance there. But Rust doesn't just do that. It also has its own costs and quirks.
Absolutely! But I counter: How do you know that all of them are? Or that even many of them are?
The developers that work on it may well know, but it isn't something we can axiomatically assume. It is not a true statement that any rust panic would have been an error in C(++) code written by the same (competent in both) developer.
Increased error visibility can be either more errors or more error sensitivity. It's worth being mindful of this because otherwise we may adopt programming practices that increase the overall defect rate, and think we're making things better.
Perhaps it's easier to see with asserts. If your defect rate is 1 defect per 1000 lines of code, and you go add 1000 asserts you'll add a bug (actually I think you'll add 10: the boundary conditions that asserts test are far more likely than average code to be implemented wrong). Like any other code they could also start off correct but then become desynced with the rest of the code. If you go and ship those asserts in production and the effect of asserting is important in your application, then unless the asserts prevent more bugs than they created you made the program worse off. Even if the asserts don't cause bugs they may make the program harder to maintain or extend. Therefore in any codebase there must be some optimal level of asserts, more or less and the program is worse off.
Is the implicit assertion of rust's behavior optimal? Probably not for all codes, because different codes call for different tradeoffs. Or even for every code there might be language changes that could make things better. But we'll never find out this stuff if we can't admit that there is the potential for uncertainty or improvement and confuse memory safety for program correctness. It's one of the more important aspects of correctness but the world is full of serious bugs in software written in inherently memory safe languages.
And after all, a program consisting of nothing but exit(1) (or perhaps while(;;){}) is the most perfectly memory safe program possible. :P If memory safety were the only goal programming would be much simpler.
> and possibly a portal into the space between dimensions from which optimising compilers will happily allow the Others into our world
As mentioned up thread, rust also can create portals into the hell dimension if there is unsafe code (including in the standard library) that fails to obey all the not-very-simple contract requirements. Also the number of times errant C code has actually opened a portal in to the hell dimension thereby destroying the universe is greatly overstated, that's only happened twice at most, usually undefined behavior is just a crash or an exploitable misbehavior that lets some haxor steal your data and those can also happen from pure logic errors (or a panic, in the case of a crash). I do fully agree that narrowing the potential for UB misbehavior is very important and rust is an important advance there. But Rust doesn't just do that. It also has its own costs and quirks.
How are exceptions less safe or less clear?
Also, crashing fast is the best way to deal with unforeseen events one can’t recover from. Your anecdotal experience with regards to firefox and rust just shows that they put more assertions into the code (good!), which makes it easier to notice and later fix bugs, as rust makes it mandatory to handle error cases to some degree. This is also a plus for rust.
Also, crashing fast is the best way to deal with unforeseen events one can’t recover from. Your anecdotal experience with regards to firefox and rust just shows that they put more assertions into the code (good!), which makes it easier to notice and later fix bugs, as rust makes it mandatory to handle error cases to some degree. This is also a plus for rust.
If you can't recover, perhaps. But when it causes more unrecoverable states (e.g. because handling the case requires a massive refactor to satisfy the borrow checker, so you just panic...) it's another issue.
Even ignoring that, not everything is a security sensitive internet application. In plenty of cases trundling on with a corrupt state in production is superior, particularly since many corrupt states are benign (particularly randomly occurring ones, rather than attacker produced ones). E.g. is it better for the software driving your Christmas lights to potentially glitch for a moment and display the wrong colors or to just shut down (/get stuck, depending on the design)?
> as rust makes it mandatory to handle error cases to some degree
"handle" often just means panic, and in cases where security isn't a concern and there isn't any persistent data to corrupt crashing may already be the worst thing that could happen. So you might have that the C-written program would have correctly handled the case but due to rust's effort front loading it just doesn't get handled. ... but even if the C-written program handled it wrong it would be worse than the rust code that panicked (for some applications) and quite possibly better.
I've experienced rust sometimes normalizes intentionally writing code that is effectively if()elif()elif()else{abort();} as a result of the a culture of panic being 'safe' and making other choices require more upfront effort.
For Firefox which is generally security critical I fully agree that panicing is better than being corrupt. But we'd be assuming facts not in evidence if we assume that the panics would actually be bad states if that code were written in C++, it's possible that some of them would have been correctly handled but for the extra effort rust required up front. In firefox the tradeoff for more defects in exchange for being safer is still probably a good one. But one can't be in denial of this possibility if one is to minimize the cost of rust, apply it to where it's most applicable, etc.
> How are exceptions less safe or less clear?
Potentially! It depends on the culture of their use. They have a highly non-local effect, so some particular weird exception happens in library of library code of a sort that is completely incomprehensible to the code 20 steps back up the stack. And not just incomprehensible, but unknowable since the called code could be changed later and add exceptions that the caller couldn't have known of. The control flow diverts away for an invisible and potentially unknowable reason. For that reason many places have varrious rules about exceptions, including sometimes prohibiting them entirely.
Of course it can be used well and safely too with care.
But so can "goto fail; ... fail:...". And code that could locally handle an error case by unwidindign itself and e.g. giving an empty result probably ought to do so rather than bubble up any one of thirteen different exception states that may be differently mishandled. (or at least the better choice between the approaches isn't something that can be correctly answered by a maxim or anything less than case specific judgement)
Even ignoring that, not everything is a security sensitive internet application. In plenty of cases trundling on with a corrupt state in production is superior, particularly since many corrupt states are benign (particularly randomly occurring ones, rather than attacker produced ones). E.g. is it better for the software driving your Christmas lights to potentially glitch for a moment and display the wrong colors or to just shut down (/get stuck, depending on the design)?
> as rust makes it mandatory to handle error cases to some degree
"handle" often just means panic, and in cases where security isn't a concern and there isn't any persistent data to corrupt crashing may already be the worst thing that could happen. So you might have that the C-written program would have correctly handled the case but due to rust's effort front loading it just doesn't get handled. ... but even if the C-written program handled it wrong it would be worse than the rust code that panicked (for some applications) and quite possibly better.
I've experienced rust sometimes normalizes intentionally writing code that is effectively if()elif()elif()else{abort();} as a result of the a culture of panic being 'safe' and making other choices require more upfront effort.
For Firefox which is generally security critical I fully agree that panicing is better than being corrupt. But we'd be assuming facts not in evidence if we assume that the panics would actually be bad states if that code were written in C++, it's possible that some of them would have been correctly handled but for the extra effort rust required up front. In firefox the tradeoff for more defects in exchange for being safer is still probably a good one. But one can't be in denial of this possibility if one is to minimize the cost of rust, apply it to where it's most applicable, etc.
> How are exceptions less safe or less clear?
Potentially! It depends on the culture of their use. They have a highly non-local effect, so some particular weird exception happens in library of library code of a sort that is completely incomprehensible to the code 20 steps back up the stack. And not just incomprehensible, but unknowable since the called code could be changed later and add exceptions that the caller couldn't have known of. The control flow diverts away for an invisible and potentially unknowable reason. For that reason many places have varrious rules about exceptions, including sometimes prohibiting them entirely.
Of course it can be used well and safely too with care.
But so can "goto fail; ... fail:...". And code that could locally handle an error case by unwidindign itself and e.g. giving an empty result probably ought to do so rather than bubble up any one of thirteen different exception states that may be differently mishandled. (or at least the better choice between the approaches isn't something that can be correctly answered by a maxim or anything less than case specific judgement)
Undefined behavior is critical for performance. Without undefined behavior, C compilers would not be able to optimize at all. You'd be running everything at -O0 or worse.
> Undefined behavior is critical for performance
Not only is this not true, it's trivially easy to prove it's not true. Both rustc and clang generate LLVM IR and use LLVM to optimise and generate machine code. The code that's generated is equally performant, as you'd expect since most of the optimisation is being done by LLVM, not the front end.
The difference between the two frontends is that rustc is stricter, rejecting programs where UB may arise.
Not only is this not true, it's trivially easy to prove it's not true. Both rustc and clang generate LLVM IR and use LLVM to optimise and generate machine code. The code that's generated is equally performant, as you'd expect since most of the optimisation is being done by LLVM, not the front end.
The difference between the two frontends is that rustc is stricter, rejecting programs where UB may arise.
I do agree that if you remove `unsafe` blocks from Rust, then you have a performant language without UB. However: (1) We are talking about C here, not Rust (2) Rust has UB due to `unsafe` (3) LLVM IR has UB.
To play devil's advocate, that doesn't negate the claim that undefined behavior is critical for performance. Rust (as well as LLVM IR itself) also have a concept of undefined behavior.
The main performance-critical undefined behavior in C is provenance. The rest can be removed without major performance impact. (Which is not to say they can't give you 10% on specific workloads, just they aren't what is taking you from -O0 to -O3.)
A related student poster from EuroLLVM 2023: https://llvm.org/devmtg/2023-05/slides/Posters/05-Popescu-Pe... It tests the performance impact of some of the secondary undefined behaviors, and the result is basically what you'd expect. They do have impact, but if you average over all benchmarks the improvement is, at best, in the low single digits.
A related student poster from EuroLLVM 2023: https://llvm.org/devmtg/2023-05/slides/Posters/05-Popescu-Pe... It tests the performance impact of some of the secondary undefined behaviors, and the result is basically what you'd expect. They do have impact, but if you average over all benchmarks the improvement is, at best, in the low single digits.
Yes, I do agree that you basically only need provenance, and that C has more UB than necessary. You can indeed reduce UB without any major performance impact (e.g., shifts by a large n, signed overflow). I think that would be a good idea.
Even if that's true (which is not given, as others have said), it would be a worthy trade to make. Software needs to do what it's meant to do first and foremost. Speed doesn't mean shit if you can't trust that the software actually works.
But that is simply not true, and can be proved by looking at the world we live in.
C became the dominant language precisely due to hardware constraints, and the ability to extract every last drop from limited hardware was back in the day more important than software working perfectly always. If this wasn't the case, other safer alternatives would have been preferred.
Unless in very specific domains, hardware advances have outpaced the software needs (eg. there is only so much compute power a spreadsheet user will need). That is why today we allow ourselves to think about "luxuries of the past" such as corectness, safety, ergonomics, composability etc.
C became the dominant language precisely due to hardware constraints, and the ability to extract every last drop from limited hardware was back in the day more important than software working perfectly always. If this wasn't the case, other safer alternatives would have been preferred.
Unless in very specific domains, hardware advances have outpaced the software needs (eg. there is only so much compute power a spreadsheet user will need). That is why today we allow ourselves to think about "luxuries of the past" such as corectness, safety, ergonomics, composability etc.
C became dominat, because UNIX was a free beer OS, with source tapes and a book to come along for the ride (Lion's commentary).
Had UNIX been as expensive as VMS, or System/370, with a commercial license, no university would have cared to port UNIX, and focus on the systems language used to develop it (post-UNIX V5).
As for its performance myth, regarding 1980's C compilers.
"Oh, it was quite a while ago. I kind of stopped when C came out. That was a big blow. We were making so much good progress on optimizations and transformations. We were getting rid of just one nice problem after another. When C came out, at one of the SIGPLAN compiler conferences, there was a debate between Steve Johnson from Bell Labs, who was supporting C, and one of our people, Bill Harrison, who was working on a project that I had at that time supporting automatic optimization...The nubbin of the debate was Steve's defense of not having to build optimizers anymore because the programmer would take care of it. That it was really a programmer's issue.... Seibel: Do you think C is a reasonable language if they had restricted its use to operating-system kernels? Allen: Oh, yeah. That would have been fine. And, in fact, you need to have something like that, something where experts can really fine-tune without big bottlenecks because those are key problems to solve. By 1960, we had a long list of amazing languages: Lisp, APL, Fortran, COBOL, Algol 60. These are higher-level than C. We have seriously regressed, since C developed. C has destroyed our ability to advance the state of the art in automatic optimization, automatic parallelization, automatic mapping of a high-level language to the machine. This is one of the reasons compilers are ... basically not taught much anymore in the colleges and universities."
-- Fran Allen interview, Excerpted from: Peter Seibel. Coders at Work: Reflections on the Craft of Programming
Had UNIX been as expensive as VMS, or System/370, with a commercial license, no university would have cared to port UNIX, and focus on the systems language used to develop it (post-UNIX V5).
As for its performance myth, regarding 1980's C compilers.
"Oh, it was quite a while ago. I kind of stopped when C came out. That was a big blow. We were making so much good progress on optimizations and transformations. We were getting rid of just one nice problem after another. When C came out, at one of the SIGPLAN compiler conferences, there was a debate between Steve Johnson from Bell Labs, who was supporting C, and one of our people, Bill Harrison, who was working on a project that I had at that time supporting automatic optimization...The nubbin of the debate was Steve's defense of not having to build optimizers anymore because the programmer would take care of it. That it was really a programmer's issue.... Seibel: Do you think C is a reasonable language if they had restricted its use to operating-system kernels? Allen: Oh, yeah. That would have been fine. And, in fact, you need to have something like that, something where experts can really fine-tune without big bottlenecks because those are key problems to solve. By 1960, we had a long list of amazing languages: Lisp, APL, Fortran, COBOL, Algol 60. These are higher-level than C. We have seriously regressed, since C developed. C has destroyed our ability to advance the state of the art in automatic optimization, automatic parallelization, automatic mapping of a high-level language to the machine. This is one of the reasons compilers are ... basically not taught much anymore in the colleges and universities."
-- Fran Allen interview, Excerpted from: Peter Seibel. Coders at Work: Reflections on the Craft of Programming
> [..] and the ability to extract every last drop from limited hardware was back in the day more important than software working perfectly always
That seems to contradict the “very simple to implement compiler”
That seems to contradict the “very simple to implement compiler”
What behavior would you define for
*((int*)rand()) = 42Undefined behaviour enables only a fairly small set of optimisations. There's a large set of optimisations that can be implemented completely safely without having to make such dangerous assumptions. Other programming languages do this all the time, it's not just C/C++ that have optimisers!
This is simply not true. Almost all optimisations rely on undefined behavior, see https://news.ycombinator.com/item?id=38760475
Note that I am not talking about UB like signed integer overflow. Removing that would slow down programs by a couple of percent. The important type of UB is pointer provenance. This ensures that e.g., writing to a random memory address is UB.
Note that I am not talking about UB like signed integer overflow. Removing that would slow down programs by a couple of percent. The important type of UB is pointer provenance. This ensures that e.g., writing to a random memory address is UB.
Almost all optimisations in C/C++ compilers depend on undefined behaviour, because practically no behaviour is defined!
The trick is to define behaviour, which is what other programming languages do.
E.g.: in both C# and Rust, integers have fixed sizes. A C# Int32 is equivalent to a Rust i32. Only God knows what a C/C++ "int" is. It could have 17 bits and use ternary.
The trick is to define behaviour, which is what other programming languages do.
E.g.: in both C# and Rust, integers have fixed sizes. A C# Int32 is equivalent to a Rust i32. Only God knows what a C/C++ "int" is. It could have 17 bits and use ternary.
As I said, this isn't about ints, it's about pointers. What behavior will you define in C for writing to a random memory address?
Where's your proof?
Rust has far fewer UB than C yet its performance is comparable to C.
Rust has far fewer UB than C yet its performance is comparable to C.
Just to add, Safe Rust should have zero UB, modulo bugs in the compiler. And they're very serious about fixing such bugs. They won't dismiss it with "just be careful while programming".
Sure, safe Rust has no UB. The main reason why that doesn't apply to C is that if you wanted to make every C program have a defined behavior, then you also need to define behavior for out of bounds memory writes, including writes from other threads. This means that the compiler basically cannot apply any optimisations, because another thread could be overwriting your data structures at any moment.
Then fence it in and have it be as little of it as possible and as obvious as possible when it can happen.
C gives you a level of control and responsibility not found in other languages. That's a choice, not something that is inherently worse. It may be worse for what you are doing. Most people don't value the level of control that C gives you and would rather chose another language and that is fine. But having a language available with this level of control is valuable, even if few people chose to use it. Most UB in the C standard is there for a very good reason.
C is not a particularly low-level. It has no (standard) way to control vectorization, stack usage, calling conventions, etc.
It just had an insane about of money spent on making its compilers optimize better.
It just had an insane about of money spent on making its compilers optimize better.
You are right, but no language has tried to claim the space of giving the user more control, so C remains the lowest level language we have that is portable. I think there are lots of opportunities in this space, but i don't know of anyone working on it.
C++, Rust, Zig all are lower level, due to having control over vectors.
Whatever people it is C for low level coding, in reality are compiler specific language extensions, not available on ISO C.
All languages can have such specific extensions and many do.
All languages can have such specific extensions and many do.
I think it's underappreciated that Rust's `unsafe{}` doesn't exist in isolation. Rust has facilities for building safe abstractions on top of it, and has a culture of taking this abstraction layer seriously.
Danger of unsafe features and FFI is usually conditional — you can use a pointer only until some point, or only on a single thread, etc. A use of unsafe in Rust doesn't become "be careful!" kryptonite spreading around the program. It's possible to build walls around it to contain it completely.
In Swift + unsafe or Java + JNI I've struggled building equally solid abstractions around FFI. They don't have "you can call it only once" or "you can't call go() after finish()" as a compile-time check. They don't have "you can use it only within this scope" (they may use a closure/callback to give access to an unsafe object, but these aren't hermetic, so that's a convention not a guarantee). Exposing objects to Swift or Java requires the higher level language to be in charge of their lifetime.
Danger of unsafe features and FFI is usually conditional — you can use a pointer only until some point, or only on a single thread, etc. A use of unsafe in Rust doesn't become "be careful!" kryptonite spreading around the program. It's possible to build walls around it to contain it completely.
In Swift + unsafe or Java + JNI I've struggled building equally solid abstractions around FFI. They don't have "you can call it only once" or "you can't call go() after finish()" as a compile-time check. They don't have "you can use it only within this scope" (they may use a closure/callback to give access to an unsafe object, but these aren't hermetic, so that's a convention not a guarantee). Exposing objects to Swift or Java requires the higher level language to be in charge of their lifetime.
> They don't have "you can use it only within this scope" (they may use a closure/callback to give access to an unsafe object, but these aren't hermetic, so that's a convention not a guarantee).
You can use the same trick as Haskell's STRef to prevent reusing a "leaked" unsafe object, although it's cumbersome enough in Java that you may not want to.
You can use the same trick as Haskell's STRef to prevent reusing a "leaked" unsafe object, although it's cumbersome enough in Java that you may not want to.
The ST monad trick requires second-order polymorphism, though [to type runST :: forall a. (forall s. ST s a) -> a]. Are any of those languages capable of it?
Java certainly is; you might have to encode it as having your callback be an object with a (polymorphic) method, but for a long time that was something you had to do to express any callback in Java, so it's well-supported.
I haven’t had a chance to fully explore the new features and there are probably still some sharp edges, but the addition of non-copyable types and borrowing/consuming bindings in Swift 5.9 should bring it a lot closer to Rust in those respects, especially the hermeticity aspect. If you haven’t experimented recently, might be worth doing - this is also one of the big focuses of the language in the near term, so there should be lots more progress coming too.
Java’s recent Foreign Function & Memory API gives you temporal, spatial and thread-safety.
In short, foreign memory is represented as a Segment, with known bounds (it also has a possibility for unbounded “pointers”, for e.g. c strings), associated to a scope. It can only be used within that scope, and will get automatically closed at the end of it.
In short, foreign memory is represented as a Segment, with known bounds (it also has a possibility for unbounded “pointers”, for e.g. c strings), associated to a scope. It can only be used within that scope, and will get automatically closed at the end of it.
[deleted]
I'm not sure I understand what this piece is trying to say about Python memory safety. Conventionally, in software security, Python is considered a memory-safe language. The piece makes the case that Python isn't memory safe when you FFI into a C library. But neither is Rust, nor is it when you use `unsafe`. What matters in both case is how little unsafe code you end up writing.
Memory safety is a software security concern. You can squint and make it about reliability or resiliency, but the reason we talk about memory safety is (to a first approximation) browser vulnerabilities.
The piece goes on to discuss data races. I'm a little keyed up on software security essays that bring data race safety into the discussion. I have a hard time not reading them as shibboleths for "Rust is the only safe language", which is manifestly false.
The vulnerabilities endemic to memory-safe languages (logic and higher-level vulnerabilities like SQLI, metacharacter quoting, filesystem traversal, and cryptography bugs) are common both to languages like Python and Java and also to Rust --- the only super-common class of vulnerability endemic to languages like Python and Java that Rust avoids is deserialization (you avoid deserialization vulnerabilities by not building hypercapable serialization formats).
Data races are a common source of reliability bugs. They're a meaningful software engineering concern. In exotic scenarios (userspace-sandboxed attacker-controlled code), they can constitute practical security vulnerabilities. But in the main, data races have not empirically proved out as a source of exploited vulnerabilities. If you have a fixed budget to transition from a C codebase that would allow you to migrate to Python now, or if you saved up, to Rust next year, and all you care about is security, then ceteris paribus you should do the Python thing. The data races aren't going to burn you.
Memory safety is a software security concern. You can squint and make it about reliability or resiliency, but the reason we talk about memory safety is (to a first approximation) browser vulnerabilities.
The piece goes on to discuss data races. I'm a little keyed up on software security essays that bring data race safety into the discussion. I have a hard time not reading them as shibboleths for "Rust is the only safe language", which is manifestly false.
The vulnerabilities endemic to memory-safe languages (logic and higher-level vulnerabilities like SQLI, metacharacter quoting, filesystem traversal, and cryptography bugs) are common both to languages like Python and Java and also to Rust --- the only super-common class of vulnerability endemic to languages like Python and Java that Rust avoids is deserialization (you avoid deserialization vulnerabilities by not building hypercapable serialization formats).
Data races are a common source of reliability bugs. They're a meaningful software engineering concern. In exotic scenarios (userspace-sandboxed attacker-controlled code), they can constitute practical security vulnerabilities. But in the main, data races have not empirically proved out as a source of exploited vulnerabilities. If you have a fixed budget to transition from a C codebase that would allow you to migrate to Python now, or if you saved up, to Rust next year, and all you care about is security, then ceteris paribus you should do the Python thing. The data races aren't going to burn you.
Are you grouping kernel exploits in with user space sandboxes? Lots of local roots come from data races which I would not call exotic.
And there's always https://portswigger.net/research/smashing-the-state-machine for web stuff.
And there's always https://portswigger.net/research/smashing-the-state-machine for web stuff.
Right, these aren't data races; they're distributed systems races, more akin to tempfile races from the 1990s than to memory corruption.
Okay, fair.
To me it’s a deep philosophical post in the vein of “what even is memory safety anyway?”
> The piece makes the case that Python isn't memory safe
It’s a philosophy argument tactic. Take something everyone considers to be true “Python is memory safe” then push it to logical extremes. The purpose of this isn’t to learn anything about Python, the purpose is to learn about the extremes. In this case about memory safety.
I think the overall point is that “memory safe languages don’t truly exist” in the purist sense, since every lang must touch unsafe code at some point. However some languages and tools do a better job is isolating these interactions. We call these tools “memory safe”.
> The piece makes the case that Python isn't memory safe
It’s a philosophy argument tactic. Take something everyone considers to be true “Python is memory safe” then push it to logical extremes. The purpose of this isn’t to learn anything about Python, the purpose is to learn about the extremes. In this case about memory safety.
I think the overall point is that “memory safe languages don’t truly exist” in the purist sense, since every lang must touch unsafe code at some point. However some languages and tools do a better job is isolating these interactions. We call these tools “memory safe”.
> Conventionally, in software security, Python is considered a memory-safe language. The piece makes the case that Python isn't memory safe when you FFI into a C library.
Interesting and largely unknown trivia: it's possible to invoke memory errors in the underlying C interpreter from pure Python code — no libraries and no imports needed!
One way of doing this is by creating new `code` objects with crafted bytecode. There is no bytecode verifier in Python to make sure, say, referenced stack variables in the VM are valid...
Interesting and largely unknown trivia: it's possible to invoke memory errors in the underlying C interpreter from pure Python code — no libraries and no imports needed!
One way of doing this is by creating new `code` objects with crafted bytecode. There is no bytecode verifier in Python to make sure, say, referenced stack variables in the VM are valid...
Is this because of a bug and might be fixed in the future or is it considered an unavoidable consequence of some design decision and will stay that way for the foreseeable future?
From what I understand about Rust, if something similar was possible in safe Rust it would be considered a bug and eventually fixed.
From what I understand about Rust, if something similar was possible in safe Rust it would be considered a bug and eventually fixed.
I think it will stay that way for the foreseeable future (but who can say). Ways to fix the particular hole:
(1) disable creating new `code` objects directly from Python. This probably would break lots of things.
(2) Add a bytecode verification mechanism that would reject `code` objects whose bytecode would result in memory errors when executed. This could be a lot of implementation work; I'm not sure.
(1) disable creating new `code` objects directly from Python. This probably would break lots of things.
(2) Add a bytecode verification mechanism that would reject `code` objects whose bytecode would result in memory errors when executed. This could be a lot of implementation work; I'm not sure.
You also don't need to FFI into some buggy C library to violate memory safety with ctypes. It's trivial to produce a segfault with it without using anything but ctypes itself, which is part of the standard library. I doubt I'd have much trouble finding other ways to make a segfault with pure python and the standard library (struct springs to mind).
CPython really isn't very safe at all. Its focus has always been on being a convenient, dynamic scripting language with minimal-fuss access to native code. It has never been hard to violate its internal assumptions and it probably never will be.
And I'm pretty comfortable with that, FWIW.
CPython really isn't very safe at all. Its focus has always been on being a convenient, dynamic scripting language with minimal-fuss access to native code. It has never been hard to violate its internal assumptions and it probably never will be.
And I'm pretty comfortable with that, FWIW.
> logic bugs, SQL injection, quoting, filesystem traversal...
Actually Rust does go quite far in reducing the probability of these bugs, even if it doesn't have specific features for it. This is through a combination of:
* really strong type system ("if it compiles it works")
* Better ergonomics, e.g. using prepared queries is much easier than in C.
* Library code being generally very high quality, and easy to obtain.
Actually Rust does go quite far in reducing the probability of these bugs, even if it doesn't have specific features for it. This is through a combination of:
* really strong type system ("if it compiles it works")
* Better ergonomics, e.g. using prepared queries is much easier than in C.
* Library code being generally very high quality, and easy to obtain.
Data races are definitely exploited! If we are considering TOCTOU issues then this is a very easy way to get fairly reliable and simple exploits. If we are talking about races of the “two threads access the same value” kind then it’s easy (well, assuming reliability is an exercise for the reader) to turn this into a UAF or OOB access by having one thread work with a stale version of an object that has been modified elsewhere.
TOCTOU isn't a data race. It's a race condition, but a "data race" is something much more specific. I think the terminology is confusing to be honest.
Right, my understanding is that a data race is the second thing I mentioned. I was just so surprised to hear this viewpoint that I figured I’d throw it in just in case we were talking about different things.
While data races may not be a top category empirically, they are undefined behavior, which means that (a future version of) the compiler is allowed to make your program do anything at all after a data race happens. We are setting the bar incredibly low for ourselves if we just accept that things like that happen on the regular.
> But in the main, data races have not empirically proved out as a source of exploited vulnerabilities.
Say what? Data races, otherwise lumped under the bucket “timing attacks”, are a common source of security exploits. A basic example is racing with code that is creating a file and applying an ACL in two steps. If I can “time” things right from a concurrent thread/process, I can get into this file before the ACL prevents me.
There are countless scenarios where multi-step operations that need to be treated atomically can be exploited by racing.
Say what? Data races, otherwise lumped under the bucket “timing attacks”, are a common source of security exploits. A basic example is racing with code that is creating a file and applying an ACL in two steps. If I can “time” things right from a concurrent thread/process, I can get into this file before the ACL prevents me.
There are countless scenarios where multi-step operations that need to be treated atomically can be exploited by racing.
That's not a vulnerability Rust prevents; it's an interaction between multiple competing runtimes. I'm not denying that race conditions (or timing attacks, another bug class Rust doesn't prevent) exist and are exploited! I'm denying that in-process data races that corrupt memory are a meaningful source of exploitable vulnerabilities.
For background, I've spent most of my career doing vulnerability research. I'm by no means a world expert on memory corruption vulnerabilities (I'm still impressed that I got my imapd shellcode to work with no uppercase ASCII characters), but you can safely assume I'm not just completely blowing off huge classes of exploitable vulnerabilities because I've never heard of them. Doesn't mean I'm right! But like, if you're going "say what", you're probably misconstruing me.
For background, I've spent most of my career doing vulnerability research. I'm by no means a world expert on memory corruption vulnerabilities (I'm still impressed that I got my imapd shellcode to work with no uppercase ASCII characters), but you can safely assume I'm not just completely blowing off huge classes of exploitable vulnerabilities because I've never heard of them. Doesn't mean I'm right! But like, if you're going "say what", you're probably misconstruing me.
OK, but I think you are moving the goal posts. You referred to “data races” and “security exploits” and suggested the two were not related. Memory corruption is only one (small) class of security exploits. Data races cause just as many in process, in memory, exploits as multi-step file operations (we are talking breaking application security models). Perhaps Rust can prevent most of these! (I don’t know rust).
What are they? Show me the vulnerabilities you're talking about. I don't think I'm moving the goalposts here. The major distinction between Rust and (say) Java is Rust's type system formalisms to prevent in-process data race memory corruption. Those are real features, but they don't mitigate a major class of vulnerabilities.
Any multi-step code, e.g. AddUser(); SetPermissions();
But, fair enough, this is not what you were talking about, and I reacted to something you weren’t intending to convey.
But, fair enough, this is not what you were talking about, and I reacted to something you weren’t intending to convey.
Isn't it rather trivial to prevent this by creating a file with a random filename, applying the ACL, then renaming the file to the correct name?
> I have a hard time not reading them as shibboleths for "Rust is the only safe language", which is manifestly false.
Given that quite simple classes of vulnerabilities are endemic to all other major languages, no, it's not "manifestly false". The state of software safety really is bad enough that "all major languages that aren't Rust are unsafe" is plausible.
> The vulnerabilities endemic to memory-safe languages (logic and higher-level vulnerabilities like SQLI, metacharacter quoting, filesystem traversal, and cryptography bugs) are common both to languages like Python and Java and also to Rust --- the only super-common class of vulnerability endemic to languages like Python and Java that Rust avoids is deserialization (you avoid deserialization vulnerabilities by not building hypercapable serialization formats).
SQLI at least should be a lot less common in ML-family languages like Rust where manipulating structured data is relatively easy (or at least, the ease advantage of string manipulation over structured data is smaller). Carefully distinguishing between character strings, file paths, and byte sequences, as Rust does, should also eliminate at least some common kinds of vulnerabilities.
> The data races aren't going to burn you.
Eh maybe. All we can really say so far is that they haven't reached low-hanging fruit level yet. There have been plenty of similarly unsafe things that weren't thought to be exploitable that have turned out to be major sources of vulnerabilities as the bar gets raised and more effort gets put in, e.g. there was a time when the conventional wisdom was that double-free() was only a reliability/resiliency concern and not a security issue.
Given that quite simple classes of vulnerabilities are endemic to all other major languages, no, it's not "manifestly false". The state of software safety really is bad enough that "all major languages that aren't Rust are unsafe" is plausible.
> The vulnerabilities endemic to memory-safe languages (logic and higher-level vulnerabilities like SQLI, metacharacter quoting, filesystem traversal, and cryptography bugs) are common both to languages like Python and Java and also to Rust --- the only super-common class of vulnerability endemic to languages like Python and Java that Rust avoids is deserialization (you avoid deserialization vulnerabilities by not building hypercapable serialization formats).
SQLI at least should be a lot less common in ML-family languages like Rust where manipulating structured data is relatively easy (or at least, the ease advantage of string manipulation over structured data is smaller). Carefully distinguishing between character strings, file paths, and byte sequences, as Rust does, should also eliminate at least some common kinds of vulnerabilities.
> The data races aren't going to burn you.
Eh maybe. All we can really say so far is that they haven't reached low-hanging fruit level yet. There have been plenty of similarly unsafe things that weren't thought to be exploitable that have turned out to be major sources of vulnerabilities as the bar gets raised and more effort gets put in, e.g. there was a time when the conventional wisdom was that double-free() was only a reliability/resiliency concern and not a security issue.
Given that quite simple classes of vulnerabilities are endemic to all other major languages, no, it's not "manifestly false". The state of software safety really is bad enough that "all major languages that aren't Rust are unsafe" is plausible.
We're really very good at documenting vulnerabilities; the mere documentation of vulnerabilities is itself a 9-figure industry. So: cough up the examples. I can't think of any, so that's where I'm setting the bar for you.
A reminder that memory corruption bugs in FFI-bound libraries doesn't count --- Rust has plenty of those --- and neither do deserialization vulnerabilities, which were discussed upthread. It also doesn't matter if a condition makes it unsafe to run attacker-controlled code in a shared runtime; nobody does that (with native languages; they try, with Javascript, and it has been a disaster). You're looking for vulnerabilities that are widely exploited and intrinsic to a memory-safe language that isn't Rust. Not to a library, but to the language.
We're really very good at documenting vulnerabilities; the mere documentation of vulnerabilities is itself a 9-figure industry. So: cough up the examples. I can't think of any, so that's where I'm setting the bar for you.
A reminder that memory corruption bugs in FFI-bound libraries doesn't count --- Rust has plenty of those --- and neither do deserialization vulnerabilities, which were discussed upthread. It also doesn't matter if a condition makes it unsafe to run attacker-controlled code in a shared runtime; nobody does that (with native languages; they try, with Javascript, and it has been a disaster). You're looking for vulnerabilities that are widely exploited and intrinsic to a memory-safe language that isn't Rust. Not to a library, but to the language.
> We're really very good at documenting vulnerabilities; the mere documentation of vulnerabilities is itself a 9-figure industry. So: cough up the examples. I can't think of any, so that's where I'm setting the bar for you.
Your own post listed a bunch of vulnerability classes that happen in those languages ("logic and higher-level vulnerabilities like SQLI, metacharacter quoting, filesystem traversal, and cryptography bugs").
Your own post listed a bunch of vulnerability classes that happen in those languages ("logic and higher-level vulnerabilities like SQLI, metacharacter quoting, filesystem traversal, and cryptography bugs").
He says explicitly that these are endemic to memory-safe languages, including Rust. They aren't something that Rust handles better than Python or Java.
I wonder what the author thinks about Swift’s new C++ interop story? Since the Swift compiler includes Clang, and can thus compile both your C++ and Swift into LLVM IR, without the need for an FFI later between the two, couldn’t this be the “Typescript for C++” that the author points out a space for? The Swift folks are very much thinking about Swift as a C++ successor that can be incrementally migrated to so I’m a bit surprised the author didn’t discuss it further — especially given Swift’s spiritual similarities to Rust.
There’s a couple great talks on this by folks on the Swift team.
John McCall at CppNow https://m.youtube.com/watch?v=lgivCGdmFrw
Konrad Malawski at StrangeLoop 2023 https://m.youtube.com/watch?v=ZQc9-seU-5k
There’s a couple great talks on this by folks on the Swift team.
John McCall at CppNow https://m.youtube.com/watch?v=lgivCGdmFrw
Konrad Malawski at StrangeLoop 2023 https://m.youtube.com/watch?v=ZQc9-seU-5k
I do not know a ton about it. Thanks for the pointers.
I kept up with Swift more in the old days, but it doesn't seem to have gained a ton of relevance outside of Apple platforms, which I don't develop for. Doesn't mean that I think that it's bad, just that that's why I haven't spent a lot of time with it yet.
I kept up with Swift more in the old days, but it doesn't seem to have gained a ton of relevance outside of Apple platforms, which I don't develop for. Doesn't mean that I think that it's bad, just that that's why I haven't spent a lot of time with it yet.
For sure! Yeah I think a lot of people don’t realize that there’s actually a ton of really interesting work going on in Swift that is very relevant to the current conversation around languages these days (i.e. memory safety, data race safety, performance, development experience, etc.)
The cross-platform story is getting quite good with Swift, but I think a lot of people just aren’t aware of it. Linux support is good and Windows is getting better and better.
For example, The Browser Company is betting big on using Swift for its Windows app and talks about why they are doing it https://open.substack.com/pub/speakinginswift/p/interoperabi...
The cross-platform story is getting quite good with Swift, but I think a lot of people just aren’t aware of it. Linux support is good and Windows is getting better and better.
For example, The Browser Company is betting big on using Swift for its Windows app and talks about why they are doing it https://open.substack.com/pub/speakinginswift/p/interoperabi...
Swift is an interesting option, though that interop requires dropping support for non-clang toolchains, and I don't see that happening any time soon. I mean, it makes sense for code that exclusively targets Apple platforms, and maybe that's all that Apple and the Swift team care about?
I agree with the sentiment. Consider a hypothetical variant of Python that requires using eval() (with the same behavior as in Python, so full Python expression support) for converting from strings to integers or floats. Or that implicitly calls eval() on list subscripts, to turn strings into integers. None of these changes impact memory safety, but it still makes it much more likely that common code has security vulnerabilities. (There is actually a widely used programming language with the eval-on-subscript feature …)
> While a Go program may exhibit what a Rust or C++ program would consider undefined behavior, and it does also consider it an error, the consequences are very different. You don’t get time travel. You get 998 instead of 1,000.
This isn't correct. Data races on multi-word objects of built-in type, such as slices and interfaces, actually have undefined behavior, in the sense that array bounds checking and type checking may break down.
Russ Cox's old example still works if you disable optimizations: https://research.swtch.com/gorace
It looks like some form of dead store elimination happens to eliminate the data race with current compilers. For now, it's possible to bring it back by adding a pointless atomic operation, like this:
> While a Go program may exhibit what a Rust or C++ program would consider undefined behavior, and it does also consider it an error, the consequences are very different. You don’t get time travel. You get 998 instead of 1,000.
This isn't correct. Data races on multi-word objects of built-in type, such as slices and interfaces, actually have undefined behavior, in the sense that array bounds checking and type checking may break down.
Russ Cox's old example still works if you disable optimizations: https://research.swtch.com/gorace
It looks like some form of dead store elimination happens to eliminate the data race with current compilers. For now, it's possible to bring it back by adding a pointless atomic operation, like this:
go func(){
for !done {
k = i
atomic.AddUint32(&global, 1)
k = j
}
}()Regarding "Now, I am not a politican, and these bills are huge, so I wasn’t able to figure out how these bills do this specifically [..]", I think the relevant reference is on page 864 of [1]:
"SEC. 1613. POLICY AND GUIDANCE ON MEMORY-SAFE SOFT- WARE PROGRAMMING. (a) POLICY AND GUIDANCE.—Not later than 270 days after the date of the enactment of this Act, the Secretary of Defense shall develop a Department of Defense wide policy and guidance in the form of a directive memorandum to implement the recommendations of the National Security Agency contained in the Software Memory Safety Cybersecurity Information Sheet published by the Agency in November, 2022, regarding memory-safe software programming languages and testing to identify memory-related vulnerabilities in software developed, acquired by, and used by the Department of Defense."
The mentioned "Software Memory Safety Cybersecurity Information Sheet" is probably [2] which explicitly lists "C#, Go, Java®, Ruby™, Rust®, and Swift" as examples for memory safe languages.
I'm still looking for the equivalent EU document and would be grateful for any hints.
EDIT: I could not find any reference to memory safety in any of the EU documents but interestingly the "Impact Assessment Report" [3] mentions Rust and Go specifically.
[1] https://www.armed-services.senate.gov/imo/media/doc/fy24_nda...
[2] https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI...
[3] https://ec.europa.eu/newsroom/dae/redirection/document/89545
"SEC. 1613. POLICY AND GUIDANCE ON MEMORY-SAFE SOFT- WARE PROGRAMMING. (a) POLICY AND GUIDANCE.—Not later than 270 days after the date of the enactment of this Act, the Secretary of Defense shall develop a Department of Defense wide policy and guidance in the form of a directive memorandum to implement the recommendations of the National Security Agency contained in the Software Memory Safety Cybersecurity Information Sheet published by the Agency in November, 2022, regarding memory-safe software programming languages and testing to identify memory-related vulnerabilities in software developed, acquired by, and used by the Department of Defense."
The mentioned "Software Memory Safety Cybersecurity Information Sheet" is probably [2] which explicitly lists "C#, Go, Java®, Ruby™, Rust®, and Swift" as examples for memory safe languages.
I'm still looking for the equivalent EU document and would be grateful for any hints.
EDIT: I could not find any reference to memory safety in any of the EU documents but interestingly the "Impact Assessment Report" [3] mentions Rust and Go specifically.
[1] https://www.armed-services.senate.gov/imo/media/doc/fy24_nda...
[2] https://media.defense.gov/2022/Nov/10/2003112742/-1/-1/0/CSI...
[3] https://ec.europa.eu/newsroom/dae/redirection/document/89545
Thank you! I really appreciate it.
Our programs are growing so big by having so many (indirect) dependencies that we need a way to sandbox the libraries that we include from our main programs. This is the type of safety that I'm looking for, really.
Object-capability models really shine there: https://en.m.wikipedia.org/wiki/Object-capability_model
What if the only way to access the file system was to call methods on an object that was provided to main()? Then libraries would only be able to access the file system if they got a reference to that object.
What if the only way to access the file system was to call methods on an object that was provided to main()? Then libraries would only be able to access the file system if they got a reference to that object.
The OP does mention Austal as one of a few languages putting their own twist on Rust learnings, though the most interesting thing to me about Austral is indeed https://borretti.me/article/how-capabilities-work-austral rather than simply memory safety (though that's essential too).
Address spaces are typically difficult to mix things in, unfortunately. If you sandbox harder then it’s not actually very useful to have them in your process anymore.
Some people are looking at Wasm as a cross language way of doing this. Some work in language specific ways is also going on. Definitely an interesting area.
I've been idly wondering, what is the relationship between memory unsafety, or more generally, undefined behavior, and ambient authority? For security purposes, is the former a special, unintended (for the most part, it's a bug) form of the latter?
Java can use SecurityManager to do this.
It's deprecated for removal.[0]
[0] https://openjdk.org/jeps/411
[0] https://openjdk.org/jeps/411
GraalVM has Isolates that can do this on a much more fine-grained level and with multiple languages (it can also run LLVM-languages now).
I'd seen that https://www.graalvm.org/latest/security-guide/polyglot-sandb... is JavaScript-only at this point, I guess it must build on https://www.graalvm.org/latest/reference-manual/embed-langua... and one could roll their own for any language?
Is there a good example, doc, or is it even a thing, to use GraalVM Isolates to defend against Java supply chain attacks, nevermind other languages? I guess it might be possible, going by a comment on another thread https://news.ycombinator.com/item?id=38278131 but require careful construction of anything you'd want to have only the capabilities you pass to it?
(Naive questions, apologies, I should really learn by trying it out instead!)
Is there a good example, doc, or is it even a thing, to use GraalVM Isolates to defend against Java supply chain attacks, nevermind other languages? I guess it might be possible, going by a comment on another thread https://news.ycombinator.com/item?id=38278131 but require careful construction of anything you'd want to have only the capabilities you pass to it?
(Naive questions, apologies, I should really learn by trying it out instead!)
I believe it should work with any other Graal language (currently, JS, Python are the bigger ones and Sulong can run LLVM bytecode. There is also Espresso, which runs “java on top of java”, making it also eligible for these security boundaries).
I don’t think it is too commonly used yet, but yeah, it can even do stuff like limit CPU usage within an isolate, so it should be more than possible to limit the scope of such an attack.
I don’t think it is too commonly used yet, but yeah, it can even do stuff like limit CPU usage within an isolate, so it should be more than possible to limit the scope of such an attack.
Enforcing memory safety is good thing, even if it's not perfect; it's the first stage in the long-needed move from throw-it-in-a-bucket-and-hope-it-works "software engineering" toward proper formal-methods-driven actual software engineering.
I feel complaining about it as insufficient is not the ideal way to push things forward. Instead, let's treat the progress on memory safety policy as a first victory in that process, and build on it.
I feel complaining about it as insufficient is not the ideal way to push things forward. Instead, let's treat the progress on memory safety policy as a first victory in that process, and build on it.
To be clear, I absolutely do as well. I'm wondering about other possibilities, but I am ecstatic that this all is going as well as it is.
Thanks for all your work on Rust! If I had the option to choose just one small thing to go after next, it would be well-defined behavior and error handling for integer overflow and underflow in languages without bignums.
I don't think the government's goal in improving memory safety is because of Rust or any other particular technology.
The government buys technology from wherever, and until recently they never really cared where they got it from. If they need a USB Emulator, they buy the same Chinese Gotek from Ebay that you or I buy, and they get the same "Driver CD" full of buggy, broken English software, written by one person for $27 over the course of about 7 hours. Same thing with DVR systems, access controls, network appliances, access points. I've worked on government projects that were using 20 year old, broken, encryption libraries on active web servers. And when brought to their attention the people who use it don't care, and the people who are capable of doing something about it will never find out because it's too much work to replace and nobody wants to take that Zoom call.
I think the author is getting a little over-excited about Rust. Granted, they are a Rust developer. I'd love it if we could build everything in my favorite language. But the government doesn't want to replace everything with Rust. The government wants results. Period. It is the DOD, NSA, and CISA who need to become experts on how to realize those results. And they will probably release guidance that says "if you can't program securely in a memory unsafe language, don't use a memory unsafe language". The thing I think the author is missing is that nobody is going to mistaken "Made with RUST!" for "Impossible to cause undefined behavior!"
You can be a terrible programmer and still create undefined behavior, insecure code, data leaks, or any number of other problems in a memory safe language.
What the government is going to do is implement strict code import controls, similar to the export controls we already have. Purchasing departments will have to get software purchases signed off multiple times by qualified parties, and department leaders will have to prove that the technology they procure is adequately safe. In some situations I anticipate that means re-writing a bunch of code in Rust, and in other situations I'm sure that means hardening existing C. The majority of the government's security holes that need to be patched are coming from low quality unvetted imported technology, low quality self-written code, and code that should have been replaced 20 years ago. While it's exciting to think that this could mean a renaissance for the author's technology of choice, I just don't see it working out that way. If the government could write memory safe code, it would have. Rust isn't safe ENOUGH to save the government from itself alone.
The government buys technology from wherever, and until recently they never really cared where they got it from. If they need a USB Emulator, they buy the same Chinese Gotek from Ebay that you or I buy, and they get the same "Driver CD" full of buggy, broken English software, written by one person for $27 over the course of about 7 hours. Same thing with DVR systems, access controls, network appliances, access points. I've worked on government projects that were using 20 year old, broken, encryption libraries on active web servers. And when brought to their attention the people who use it don't care, and the people who are capable of doing something about it will never find out because it's too much work to replace and nobody wants to take that Zoom call.
I think the author is getting a little over-excited about Rust. Granted, they are a Rust developer. I'd love it if we could build everything in my favorite language. But the government doesn't want to replace everything with Rust. The government wants results. Period. It is the DOD, NSA, and CISA who need to become experts on how to realize those results. And they will probably release guidance that says "if you can't program securely in a memory unsafe language, don't use a memory unsafe language". The thing I think the author is missing is that nobody is going to mistaken "Made with RUST!" for "Impossible to cause undefined behavior!"
You can be a terrible programmer and still create undefined behavior, insecure code, data leaks, or any number of other problems in a memory safe language.
What the government is going to do is implement strict code import controls, similar to the export controls we already have. Purchasing departments will have to get software purchases signed off multiple times by qualified parties, and department leaders will have to prove that the technology they procure is adequately safe. In some situations I anticipate that means re-writing a bunch of code in Rust, and in other situations I'm sure that means hardening existing C. The majority of the government's security holes that need to be patched are coming from low quality unvetted imported technology, low quality self-written code, and code that should have been replaced 20 years ago. While it's exciting to think that this could mean a renaissance for the author's technology of choice, I just don't see it working out that way. If the government could write memory safe code, it would have. Rust isn't safe ENOUGH to save the government from itself alone.
> But the government doesn't want to replace everything with Rust.
To be clear, I don't think that they do. To be honest, I am mostly confused by your post, not because I disagree, but because I am unsure how you came to the conclusion that I believe these things.
To be clear, I don't think that they do. To be honest, I am mostly confused by your post, not because I disagree, but because I am unsure how you came to the conclusion that I believe these things.
Funny, because I came out with the almost opposite impression: that perhaps the legislators will require something stronger than even Rust to consider something "safe" because Rust still allows easy access to things like `unsafe` and FFI like most other languages - and true safety may require more than that.
I think our biases, together with the lack of a firm conclusion in this post, leaves the door open to vastly different interpretations.
I think our biases, together with the lack of a firm conclusion in this post, leaves the door open to vastly different interpretations.
Just like in every other kind of engineering, for software capability pulls the requirements.
The only reason everybody is complaining about lack of memory safety now is because there are alternatives for every use case. Before Rust existed, it was seen as an inevitable issue that one must work with, not as a problem to solve.
The only reason everybody is complaining about lack of memory safety now is because there are alternatives for every use case. Before Rust existed, it was seen as an inevitable issue that one must work with, not as a problem to solve.
>Before Rust existed, it was seen as an inevitable issue that one must work with, not as a problem to solve.
There were many memory safe languages before Rust, Java and C# evangelized this a lot, though at that time the accent was on memory leaks not on safety. Rust offers an alternative for low level programming and might be faster in soem scenarios then managed languages.
There were many memory safe languages before Rust, Java and C# evangelized this a lot, though at that time the accent was on memory leaks not on safety. Rust offers an alternative for low level programming and might be faster in soem scenarios then managed languages.
> might be faster in soem scenarios
in which scenarios would it not be faster?
in which scenarios would it not be faster?
My knowledge is probably outdated so I suggest to research this yourself, but mostly is about the maturity of JAva compilers that can perform advanced optimizations and the ability to optimize at runtime.
If you are interested into debating this with me and proving that Rust is the best I am not interested in a language war.
If you are interested into debating this with me and proving that Rust is the best I am not interested in a language war.
Just a small clarification here. The author isn’t just a fan of Rust. Steve was a member of the Rust Core Team for years and was co-author of the book “The Rust Programming Language,” which is the main recommended introductory text for the language.
I saw the authors credentials and I do respect them a lot. But to be fair, I'm sure the person who wrote the Go manual could have written the same blog post with the same outlook for the future only with Go in place of Rust. I'm trying to broaden the scope of conversation to a more holistic one, rather than just "this is our chance to take over the world!"
Like my Gotek USB emulator reference. The device costs $50, and it's pretty much the ONLY option you have for emulating a floppy drive with a USB stick in a bunch of obsolete hardware. The software that it comes with was written by a Chinese high school student in C++ during a study break and it is about as insecure and sketchy as you would expect it to be.
If you're the government looking to buy this, your choices are;
1. Buy this sketchy retroft device that is insecure and may be backdoored for a cost of $60.
2. Replace whatever needs the retrofit for a cost of $2m.
3. Write your own drivers for $100k.
Currently they just use the $60 device. The upcoming policy changes will take that option off the table for a lot of agencies, forcing them to make wiser purchasing decisions. It doesn't automatically mean Rust wins the day, or that rust deserves to win the day. It means intelligent conversations must be had and difficult decisions have to be made that used to get avoided.
Like my Gotek USB emulator reference. The device costs $50, and it's pretty much the ONLY option you have for emulating a floppy drive with a USB stick in a bunch of obsolete hardware. The software that it comes with was written by a Chinese high school student in C++ during a study break and it is about as insecure and sketchy as you would expect it to be.
If you're the government looking to buy this, your choices are;
1. Buy this sketchy retroft device that is insecure and may be backdoored for a cost of $60.
2. Replace whatever needs the retrofit for a cost of $2m.
3. Write your own drivers for $100k.
Currently they just use the $60 device. The upcoming policy changes will take that option off the table for a lot of agencies, forcing them to make wiser purchasing decisions. It doesn't automatically mean Rust wins the day, or that rust deserves to win the day. It means intelligent conversations must be had and difficult decisions have to be made that used to get avoided.
Nobody is claiming that rust wins the day. Memory-safe languages, of which rust is one of them, will get a boost. That's it.
Go has absolutely zero interesting properties here. It has a shitty, inexpressive type system, dangerous concurrency, brain-dead error handling. It is a managed language, which provided memory-safety for many many decades now.
> which provided memory-safety for many many decades now.
That is a plus, not a minus.
That is a plus, not a minus.
lol. The government doesn’t want results, unless the result is siphoning as much public money into private pockets as possible.
Nobody has been able explain to me what would be lost if we defined data races to yield one of the values that had been written to the memory in the past, instead of being undefined. It is not as if any optimizer can see that you are racing and delete the code path that has it.
Let me _try_ to explain it.
One reason is that non-atomic writes can be torn. So if you have a value like 0x00000000 and over write it with 0xFFFFFFFF, some hardware may do it as two separate writes. This means that another thread can read the data when its half way written and get 0xFFFF0000. I'm using a 32bit value here to illustrate In reality modern hardware is unlikely to tear it, but in other cases it may.
Another issue is that what you are writing may depend on something else. Consider this code:
x = 42; p = &x;
these two operations are independent, and a compiler, CPU or memory system could chose to execute them in any order. This means that even if the second operation is "atomic" and another thread can only read the before or after value of p, another thread could read p and access it, before 42 is written to x. This is why the p = &x; needs a "release" barrier that guarantees that everything before it is completed before the change to p happens.
Atomics are complex due to the way modern computers optimize and this is just a very surface level explanation. Still, I hope this helps.
One reason is that non-atomic writes can be torn. So if you have a value like 0x00000000 and over write it with 0xFFFFFFFF, some hardware may do it as two separate writes. This means that another thread can read the data when its half way written and get 0xFFFF0000. I'm using a 32bit value here to illustrate In reality modern hardware is unlikely to tear it, but in other cases it may.
Another issue is that what you are writing may depend on something else. Consider this code:
x = 42; p = &x;
these two operations are independent, and a compiler, CPU or memory system could chose to execute them in any order. This means that even if the second operation is "atomic" and another thread can only read the before or after value of p, another thread could read p and access it, before 42 is written to x. This is why the p = &x; needs a "release" barrier that guarantees that everything before it is completed before the change to p happens.
Atomics are complex due to the way modern computers optimize and this is just a very surface level explanation. Still, I hope this helps.
“Fun” stuff: Rust also doesn’t prevent this for more complicated objects. The ‘Sync’ trait will get applied to any struct that has only ‘Sync’ constituents, but that means that such an object can be observable in inconsistent states (e.g. a strange Date object where year, month and day are all atomic numbers can be 2023-02-31).
"One of the values written to memory" isn't really a thing when memory accesses can tear (e.g. because the values are 16 bytes large and are accessed using 8 byte memory ops). So it's not even necessarily about what would be lost, it's about what can be reasonably defined in the first place.
If you restrict yourself to relaxed atomic loads and stores -- i.e., memory accesses that are atomic in the sense that they cannot tear, but don't have much in the way of ordering guarantees -- then you do get "one of the values that had been written to the memory in the past".
Aside from memory tearing, one other issue is that you generally want to be able to rematerialize loads (i.e., in the face of register pressure, you may want to turn one load of a value into multiple loads instead of loading it once, then spilling to the stack and reloading from the stack). But when the compiler rematerializes a load, then you don't get "one of the values written to memory". From the perspective of the original program it looks like you get some weird superposition of values.
If you restrict yourself to relaxed atomic loads and stores -- i.e., memory accesses that are atomic in the sense that they cannot tear, but don't have much in the way of ordering guarantees -- then you do get "one of the values that had been written to the memory in the past".
Aside from memory tearing, one other issue is that you generally want to be able to rematerialize loads (i.e., in the face of register pressure, you may want to turn one load of a value into multiple loads instead of loading it once, then spilling to the stack and reloading from the stack). But when the compiler rematerializes a load, then you don't get "one of the values written to memory". From the perspective of the original program it looks like you get some weird superposition of values.
The optimizer assumes that a non-atomic non-volatile value written to memory stays the same, until some code that could modify it is executed. This allows a lot of obvious optimizations, like hoisting needlessly repeated computation out of loops, removal of redundant loads, and optimizations of common subexpressions and arithmetic.
If a value in memory could suddenly revert to something else, then
If a value in memory could suddenly revert to something else, then
if obj.x == 1 {
print(obj.x)
}
if obj.x < array.len {
array[obj.x]
}
I'm pretty sure that you have either an undecidable problem or a non deterministic piece of code that sometimes computes some value, other times a different value, depending on how the threads are scheduled.
Neither is good.
A lot of extra CPU time wasted while caches synchronize even though the other CPU isn't running code that uses it. Most of the time the typical racey code works without locks, making everything not race means tode without a potential race still gets locks.
Above I'm treating atomics and mutexs above.
Above I'm treating atomics and mutexs above.
This is what Java does, so in a way, data races are safe.
> coming legislation around MSLs for government procuremen
What’s he talking about here?
What’s he talking about here?
I link to them in the post, as well as discuss exactly how I came across them.
> threads.into_iter().for_each(|t| t.join().unwrap());
What a great title for the almost-last section, it made me laugh. When I saw the title of the final section, I fell from my chair:
> int atexit(void (*func)(void))
What a great title for the almost-last section, it made me laugh. When I saw the title of the final section, I fell from my chair:
> int atexit(void (*func)(void))
Thank you!
The part about potential C++ "successor" candidate languages had this interesting tidbit:
> This leads to four approaches:
> * We do not break backwards compatibility, and we ignore the memory safety issue (this is the ‘do nothing’ option). > * We do not break backwards compatibility, but we try to make things as memory safe as possible given the constraints we have. > * We break backwards compatibility, and we try to make things as memory safe as possible but still remain unsafe. > * We break backwards compatibility, and we become memory safe by default.
> The first option is seemingly untenable, but is also where I see C heading....
> The second option is how I view at least one of the possible C++ successor languages, Circle.....
> The third option is where the most famous successor languages live: Carbon and cpp2....
> But what about a fourth option? What if someone did a TypeScript for C++, but one that was closer to what Rust does? You might argue that this is basically just Rust? (Interestingly, it is also sort of what Zig is with C: you can use the same compiler to compile a mixed codebase, which is close enough in my opinion.) What makes me worry about option #3 is that it doesn’t pass the seeming test that is coming: memory safe by default. I still think moving to a #3 style solution is better than staying with #1, but is that enough? I don’t know, time will tell.
When put that way, I can't help but wonder about approach #3; if you're already going to break compatibility, why not go the whole way and be memory safe by default? Even as someone who really likes most of the design decisions that Rust has made that are orthogonal to memory safety, and there's a lot of room for differentiation here. Offhand, some of the features of Rust (and its ecosystem) that I've seen people express a desire for alternatives on include:
> If we think about all of these designs, they all are very similar conceptually: safe code is the default, but you can call into some sort of unsafe facility. And everyone is very clear on the relationship between the two: while the unsafe facility exists to be used, it must uphold the rules that the safe world relies on. And that means that safety and unsafety have a super/sub-set relationship. In the core is unsafety. But at some point, we draw a line, and on top of that line, safety exists.
> This leads to four approaches:
> * We do not break backwards compatibility, and we ignore the memory safety issue (this is the ‘do nothing’ option). > * We do not break backwards compatibility, but we try to make things as memory safe as possible given the constraints we have. > * We break backwards compatibility, and we try to make things as memory safe as possible but still remain unsafe. > * We break backwards compatibility, and we become memory safe by default.
> The first option is seemingly untenable, but is also where I see C heading....
> The second option is how I view at least one of the possible C++ successor languages, Circle.....
> The third option is where the most famous successor languages live: Carbon and cpp2....
> But what about a fourth option? What if someone did a TypeScript for C++, but one that was closer to what Rust does? You might argue that this is basically just Rust? (Interestingly, it is also sort of what Zig is with C: you can use the same compiler to compile a mixed codebase, which is close enough in my opinion.) What makes me worry about option #3 is that it doesn’t pass the seeming test that is coming: memory safe by default. I still think moving to a #3 style solution is better than staying with #1, but is that enough? I don’t know, time will tell.
When put that way, I can't help but wonder about approach #3; if you're already going to break compatibility, why not go the whole way and be memory safe by default? Even as someone who really likes most of the design decisions that Rust has made that are orthogonal to memory safety, and there's a lot of room for differentiation here. Offhand, some of the features of Rust (and its ecosystem) that I've seen people express a desire for alternatives on include:
* package management (centralized repos, lack of namespaces)
* scope of stdlib (relatively batteries-not-included e.g. regex, randomness APIs, http clients, async runtimes all being third-party)
* error handling (lack of exceptions, special operator to early return, somewhat prone to boilerplate like Ok-wrapping without resorting to heavy use of macros)
* integer semantics (no variable width or decimal type in std, overflow checked in debug but allowed in release mode, no implicit casting but lossy explicit casting allowed)
* type definitions (structs/enums with traits, no inheritance, reliance on the "newtype" pattern for certain things)
* function definitions (required type annotations for parameters and return values no overloading, no optional parameters, no variadic arguments)
* syntax/readability (ML-style type annotations, immutable by default, postfix operators like ? and .await, use of macros instead of functions for printing/formatting strings)
* concurrency design (built-in support for locks and 1:1 OS threading, de facto need for third-party runtime to use async, need for 3rd-party libraries for channels other than the deprecated ones in std, no "structured concurrency")
* opinionated APIs for allocations (heap pointers constructed via generic type wrappers, panicking by default if allocation fails, non-trivial boilerplate required to swap out allocator)
* linking (statically linked dependencies that don't use FFI, dynamically link to system's libc by default)
> If we think about all of these designs, they all are very similar conceptually: safe code is the default, but you can call into some sort of unsafe facility. And everyone is very clear on the relationship between the two: while the unsafe facility exists to be used, it must uphold the rules that the safe world relies on. And that means that safety and unsafety have a super/sub-set relationship. In the core is unsafety. But at some point, we draw a line, and on top of that line, safety exists.
The deep, idiosyncratic flaws of C trace back to “worse is better.” Few people remember or look up what “worse is better” actually meant. Wikipedia wrongly says it’s a “less is more” sort of thing, and some people think it’s about not being a perfectionist.
But actually… actually actually, if you read the essay, it says “worse is better” means (paraphrasing) it’s more important that C compilers be easy to implement than easy to use. Also, it is more important that the implementation—or the design of the implementation—of a piece of software be simple than that it be correct. It is more important that it be simple than that it be consistent. It is more important that it be simple than that it be “complete” (for example, handle edge cases; it just needs to work in “most situations”). This is not just anti-perfectionism, it is an objectively terrible set of engineering values. But there were so many different kinds of computers and operating systems back then—you didn’t even know if a byte was 8 bits—that it helped a lot that C compilers could do whatever they wanted in many situations. And making it easy for C compiler implementers enabled C to spread far and wide. It was a very different world, and just being able to write in a higher-level language than assembly on a particular computer was a big deal.