A Note on Fil-C(graydon2.dreamwidth.org)
graydon2.dreamwidth.org
A Note on Fil-C
https://graydon2.dreamwidth.org/320265.html
213 comments
One thing I've been wondering recently about Fil-C - why now? And I don't mean that in a dismissive way at all, I'm genuinely curious about the history. Was there some relatively recent fundamental breakthrough or other change that prevented a Fil-C-like approach from being viable before? Was it a matter of finding the right approach/implementation (i.e., a "software" problem), or is there something about modern hardware which makes the approach impractical otherwise? Something else?
I wrote a bounds checking patch to GCC (mentioned in a link from the article) back in 1995. It did full bounds checking of C & C++ while being compatible with existing libraries and ABIs, making it a bit more practical than Fil-C to deploy in the real world. You only had to recompile your application, if you trusted the libraries (although the bounds checking obviously didn't extend into the libraries unless you recompiled them). It didn't do the GC thing, but instead detected use after free at the point of use.
https://www.doc.ic.ac.uk/~phjk/BoundsChecking.html
https://www.doc.ic.ac.uk/~phjk/BoundsChecking.html
Interesting! How much interest did your work attract at the time?
My supervisor got a few papers out of it and they are fairly widely cited even today, and as academics that was (unfortunately) the best outcome for us.
The patch itself was maintained for many years, well into the mid 2000s, out of tree (actually by another person in the end), but as it never went upstream it was hard to keep doing that maintenance.
There were several problems in hindsight: C programmers at the time absolutely weren't willing to accept a large slow-down in order to get bounds checking. But also we didn't optimize our changes well (or very much at all) and I'm sure we could have got the delta down a bit if we'd put the work in. The main work that dominated performance was the lookup that you have to do from the raw pointer to the fat struct that stores the pointer bounds (and you have to do this on every pointer operation). We used a splay tree for this which was clever but not very fast. A plain hash or some other data structure could have been much faster.
The patch itself was maintained for many years, well into the mid 2000s, out of tree (actually by another person in the end), but as it never went upstream it was hard to keep doing that maintenance.
There were several problems in hindsight: C programmers at the time absolutely weren't willing to accept a large slow-down in order to get bounds checking. But also we didn't optimize our changes well (or very much at all) and I'm sure we could have got the delta down a bit if we'd put the work in. The main work that dominated performance was the lookup that you have to do from the raw pointer to the fat struct that stores the pointer bounds (and you have to do this on every pointer operation). We used a splay tree for this which was clever but not very fast. A plain hash or some other data structure could have been much faster.
Thanks for the insight! Curious what might have been if you had implemented the improvements you say, but alas, the past is the past :(
> Was there some relatively recent fundamental breakthrough or other change that prevented a Fil-C-like approach from being viable before?
The provenance model for C is very recent (and still a TS, not part of the standard). Prior to that, there was a vague notion that the C abstract machine has quasi-segmented memory (you aren't really allowed to do arithmetic on a pointer to an "object" to reach a different "object") but this was not clearly stated in usable terms.
The provenance model for C is very recent (and still a TS, not part of the standard). Prior to that, there was a vague notion that the C abstract machine has quasi-segmented memory (you aren't really allowed to do arithmetic on a pointer to an "object" to reach a different "object") but this was not clearly stated in usable terms.
Also in practical terms, you have a lot more address space to "waste" in 64 bit. It would have been frivolous in 32 and downright offending in 16 bit code.
The memory model always had segmented memory in mind and safe C approaches are not new. The provenance model makes this more precise, but the need for this was to deal with corner cases such as pointer-to-integer roundtrips or access to the representation bytes of a pointer. Of course, neither GCC nor clang get this right, to the extend that those compiler are internally inconsistent and miscompile even code that did not any clarification to be considered correct.
I’ve been thinking about this problem since 2004.
Here’s a rough timeline:
- 2004-2018: I had ideas of how to do it but I thought the whole premise (memory safe C) was idiotic.
- 2018-2023: I no longer thought the premise was idiotic but I couldn’t find a way to do it that would result in fanatical compatibility.
- 2023-2024: early Fil-C versions that were much less compatible and much less performant
- end of 2024: InvisiCaps breakthrough that gives current fanatical compatibility and “ok” performance.
It’s a hard problem. Lots of folks have tried to find a way to do it. I’ve tried many approaches before finding the current one.
Here’s a rough timeline:
- 2004-2018: I had ideas of how to do it but I thought the whole premise (memory safe C) was idiotic.
- 2018-2023: I no longer thought the premise was idiotic but I couldn’t find a way to do it that would result in fanatical compatibility.
- 2023-2024: early Fil-C versions that were much less compatible and much less performant
- end of 2024: InvisiCaps breakthrough that gives current fanatical compatibility and “ok” performance.
It’s a hard problem. Lots of folks have tried to find a way to do it. I’ve tried many approaches before finding the current one.
Beyond the Git history, is there any write-up of the different capability designs you've gone with?
I'm interested in implementing a safe low-level language with less static information around than C has (e.g. no static pointer-int distinction), but I'd rather keep around the ability to restrict capabilities to only refer to subobjects than have the same compatibility guarantees Invisicaps provide, so I was hoping to look into Monocaps (or maybe another design, if there's one that might fit better).
I'm interested in implementing a safe low-level language with less static information around than C has (e.g. no static pointer-int distinction), but I'd rather keep around the ability to restrict capabilities to only refer to subobjects than have the same compatibility guarantees Invisicaps provide, so I was hoping to look into Monocaps (or maybe another design, if there's one that might fit better).
I summarize past attempts in https://fil-c.org/invisicaps
That's a really interesting timeline! Sounds like it's been stewing for a lot longer than I expected. Was there anything in particular around 2018 that changed your opinion on the idiotic-ness of the premise?
If a hypothetical time machine allowed you to send the InvisiCaps idea back to your 2004-era self, do you think the approach would have been feasible back then as well?
If a hypothetical time machine allowed you to send the InvisiCaps idea back to your 2004-era self, do you think the approach would have been feasible back then as well?
> Was there anything in particular around 2018 that changed your opinion on the idiotic-ness of the premise?
The observation that the C variants used on GPUs are simplistic takes on memory safe C
The observation that the C variants used on GPUs are simplistic takes on memory safe C
This is super kind and awesome, I'm seriously flattered!
Did you know each other at Apple?
Long long ago, in 2009, Graydon was my official on-boarding mentor when I joined the Mozilla Javascript team. Rust already existed then but, as he notes, was quite different then. For one thing, it was GC'd, like Fil-C. Which I like -- I write a lot of my C/C++ code using Boehm GC, have my own libraries designed knowing GC is there, etc.
Long long ago, in 2009, Graydon was my official on-boarding mentor when I joined the Mozilla Javascript team. Rust already existed then but, as he notes, was quite different then. For one thing, it was GC'd, like Fil-C. Which I like -- I write a lot of my C/C++ code using Boehm GC, have my own libraries designed knowing GC is there, etc.
Yeah we interacted at Apple
Great work for the community.
This has obviously been 'rust'ling some feathers, as it challenges some of the arguments laid past; but once the dust settles, it is a major net benefit to the community.
I hope you get financed and can support other platforms than linux again.
This has obviously been 'rust'ling some feathers, as it challenges some of the arguments laid past; but once the dust settles, it is a major net benefit to the community.
I hope you get financed and can support other platforms than linux again.
> This has obviously been 'rust'ling some feathers,
I'm a Rust user and a fan. But memory safe C is actually an exciting prospect. I was hoping that the rise of Rust would encourage others to prioritize memory safety and come up with approaches that are much more ergonomic to the developers.
> as it challenges some of the arguments laid past
Genuinely curious. What are the assumptions you have in mind that Fil-C challenges? (This isn't a rhetorical question. I'm just trying to understand memory safety concepts better.)
> but once the dust settles, it is a major net benefit to the community.
Agreed, this is big! If Fil-C can fulfill its promise to make old C code memory safe, it will be a massive benefit to the world. God knows how many high-consequnce bugs and vulnerabilities hide in those.
I'm a Rust user and a fan. But memory safe C is actually an exciting prospect. I was hoping that the rise of Rust would encourage others to prioritize memory safety and come up with approaches that are much more ergonomic to the developers.
> as it challenges some of the arguments laid past
Genuinely curious. What are the assumptions you have in mind that Fil-C challenges? (This isn't a rhetorical question. I'm just trying to understand memory safety concepts better.)
> but once the dust settles, it is a major net benefit to the community.
Agreed, this is big! If Fil-C can fulfill its promise to make old C code memory safe, it will be a massive benefit to the world. God knows how many high-consequnce bugs and vulnerabilities hide in those.
Same here, I don't have any use for Rust, and am perfectly fine with automatic resource management languages (regardless of the approach).
However, Rust has been quite successful making more developers think about less known type systems, besides affine types, there is also linear types, effects, dependent types, prof systems.
And we as industry aren't going to throw away the millions and millions of stuff that was written in C, C++ and less extent Objective-C, thus efforts like Fil-C are quite welcomed.
However, Rust has been quite successful making more developers think about less known type systems, besides affine types, there is also linear types, effects, dependent types, prof systems.
And we as industry aren't going to throw away the millions and millions of stuff that was written in C, C++ and less extent Objective-C, thus efforts like Fil-C are quite welcomed.
Now that suggests an idea: Objective-Fil-C !
> I was hoping that the rise of Rust would encourage others to prioritize memory safety and come up with approaches that are much more ergonomic to the developers.
That's the end-goal right? I don't write Rust code myself, but I'm glad its existence means there's safer code out there now, and like you I have been looking forward to seeing shifts in safety expectations. I'm not surprised that it's happening so slowly though.
That's the end-goal right? I don't write Rust code myself, but I'm glad its existence means there's safer code out there now, and like you I have been looking forward to seeing shifts in safety expectations. I'm not surprised that it's happening so slowly though.
"But almost all programs have paths that crash, and perhaps the density of crashes will be tolerable."
This is a very odd statement. Mature C programs written by professional coders (Redis is a good example) basically never crash in the experience of users. Crashing, in such programs, is a rare occurrence mostly obtained by attackers on purpose, looking for code paths that generate a memory error that - if the program is used as it should - are never reached.
This does not mean that C code never segfaults: it happens, especially when developed without care and the right amount of testing. But the code that is the most security sensitive, like C Unix servers, is high quality and crashes are mostly a security problem and a lot less a stability problem.
This is a very odd statement. Mature C programs written by professional coders (Redis is a good example) basically never crash in the experience of users. Crashing, in such programs, is a rare occurrence mostly obtained by attackers on purpose, looking for code paths that generate a memory error that - if the program is used as it should - are never reached.
This does not mean that C code never segfaults: it happens, especially when developed without care and the right amount of testing. But the code that is the most security sensitive, like C Unix servers, is high quality and crashes are mostly a security problem and a lot less a stability problem.
Notice that it says "almost all programs" and not "almost all _C_ programs".
I think if you understand the meaning of "crash" to include any kind of unhandled state that causes the program to terminate execution then it includes things like unwrapping a None value in Rust or any kind of uncaught exception in Python.
That interpretation makes sense to me in terms of the point he's making: Fil-C replaces memory unsafety with program termination, which is strictly worse than e.g. (safe) Rust which replaces memory unsafety with a compile error. But it's also true that most programs (irrespective of language, and including Rust) have some codepaths in which programs can terminate where the assumed variants aren't upheld, so in practice that's often an acceptable behaviour, as long as the defect rate is low enough.
Of course there is also a class of programs for which that behaviour is not acceptable, and in those cases Fil-C (along with most other languages, including Rust absent significant additional tooling) isn't appropriate.
I think if you understand the meaning of "crash" to include any kind of unhandled state that causes the program to terminate execution then it includes things like unwrapping a None value in Rust or any kind of uncaught exception in Python.
That interpretation makes sense to me in terms of the point he's making: Fil-C replaces memory unsafety with program termination, which is strictly worse than e.g. (safe) Rust which replaces memory unsafety with a compile error. But it's also true that most programs (irrespective of language, and including Rust) have some codepaths in which programs can terminate where the assumed variants aren't upheld, so in practice that's often an acceptable behaviour, as long as the defect rate is low enough.
Of course there is also a class of programs for which that behaviour is not acceptable, and in those cases Fil-C (along with most other languages, including Rust absent significant additional tooling) isn't appropriate.
> Rust which replaces memory unsafety with a compile error
Rust uses panics for out-of-bounds access protection.
The benefit of dynamic safety checking is that it's more precise. There's a large class of valid programs that are not unsafe that will run fine in Fil-C but won't compile in Rust.
Rust uses panics for out-of-bounds access protection.
The benefit of dynamic safety checking is that it's more precise. There's a large class of valid programs that are not unsafe that will run fine in Fil-C but won't compile in Rust.
I don't think it's odd statement. It's not about segfaults, but use-after-free (and similar) bugs, which don't crash in C, but do crash in Fil-C. With Fil-C, if there is such a bug, it will crash, but if the density of such bugs is low enough, it is tolerable: it will just crash the program, but will not cause an expensive and urgent CVE ticket. The bug itself may still need to be fixed.
The paragraph refers to detecting such bugs during compilation versus crashing at runtime. The "almost all programs have paths that crash" means all programs have a few bugs that can cause crashes, and that's true. Professional coders do not attempt to write 100% bug-free code, as that wouldn't be efficient use of the time. Now the question is, should professional coders convert the (existing) C code to eg. Rust (where likely the compiler detects the bug), or should he use Fil-C, and so safe the time to convert the code?
The paragraph refers to detecting such bugs during compilation versus crashing at runtime. The "almost all programs have paths that crash" means all programs have a few bugs that can cause crashes, and that's true. Professional coders do not attempt to write 100% bug-free code, as that wouldn't be efficient use of the time. Now the question is, should professional coders convert the (existing) C code to eg. Rust (where likely the compiler detects the bug), or should he use Fil-C, and so safe the time to convert the code?
Doesn't Fil-C use a garbage collector to address use-after-free? For a real use-after-free to be possible there must be some valid pointer to the freed allocation, in which case the GC just keeps it around and there's no overt crash.
Yes, Fil-C uses some kind of garbage collector. But it can still detect use-after-free: In the 'free' call, the object is marked as free. In the garbage collection (in the mark phase), if a reference is detected to an object that was freed, then the program panics. Sure, it is also possible to simply ignore the 'free' call - in which case you "just" have a memory leak. I don't think that's what Fil-C does by default however. (This would be more like the behavior of the Boehm GC library for C, if I understand correctly.)
A free()-d object that is NOT garbage-collected during the next collection is a bug in itself.
The Fil-C GC will only GC a free'd object if it succeeds at repointing all capabilities to it to point at the free singleton instead.
Don't worry, it's totally sound.
Don't worry, it's totally sound.
I'm not sure what you mean. Do you mean there is a bug _in the garbage collection algorithm_, if the object is not freed in the very next garbage collection cycle? Well, it depends: the garbage collection could defers collection of some objects until memory is low. Multi-generation garbage collection algorithm often do this.
You can defer the actual freeing of the object until at least one GC pass finishes. Then alert if any of them are still reachable.
> it will just crash the program, but will not cause an expensive and urgent CVE ticket.
Unfortunately, security hysteria also treats any crash as "an expensive and urgent CVE ticket". See, for instance, ReDoS, where auditors will force you to update a dependency even if there's no way for a user to provide the vulnerable input (for instance, it's fixed in the configuration file).
Unfortunately, security hysteria also treats any crash as "an expensive and urgent CVE ticket". See, for instance, ReDoS, where auditors will force you to update a dependency even if there's no way for a user to provide the vulnerable input (for instance, it's fixed in the configuration file).
I agree security issues are often hyped nowadays. I think this is often due to two factors: (A) security researches get more money if they can convince people a CVE is worse. So of course they make it sound extremely bad. (B) security "review" teams in software companies do the least amount of work, and so it's just a binary "is a dependency with a vulnerability used yes/no" and then force the engineering team to update the dependency, even thought its useless. I have seen (was involved) in a number of such cases. This is wasting a lot of time. Long term, this can mean the engineering team will try to reduce the dependencies, which is not the worst of outcomes.
I think what you've written is pretty much what the "almost all programs have paths that crash" was intended to convey.
I think "perhaps the density of crashes will be tolerable" means something like "we can reasonably hope that the crashes from Fil-C's memory checks will only be of the same sort, that aren't reached when the program is used as it should be".
I think "perhaps the density of crashes will be tolerable" means something like "we can reasonably hope that the crashes from Fil-C's memory checks will only be of the same sort, that aren't reached when the program is used as it should be".
I think the point is that Fil-C makes programs crash which didn't crash before because use-after-free didn't trigger a segfault. If anything, I'd cite Redis as an example that you can build a safe C program if you go above and beyond in engineering effort... most software doesn't, sadly.
Redis uses a whole lot of fiddly data structures that turn out to involve massive amounts of unsafe code even in Rust. You'd need to use something like Frama-C to really prove it safe beyond reasonable doubt. (Or the Rust equivalents that are currently in the works, and being used in an Amazon-funded effort to meticulously prove soundness of the unsafe code in libstd.) Compiling it using Fil-C is a nice academic exercise but not really helpful, since the whole point of those custom data structures is peak performance.
sel4 is the example of building a safe C program if you go above and beyond in effort.
It's provably safer than rust, e.g.
It's provably safer than rust, e.g.
There are obviously multiple levels of correctness. Formal verification is just the very top of that spectrum, but it does comes at extraordinary effort.
did i read "above and beyond"
It is a question of probability and effort. My personal estimation rule for my type of projects is it takes 3 times longer from my prototype to something I‘m comfortable having others use it and another factor to get to an early resemblance of a product. A recent interview I read an AI expert said each 9 in terms of error probability is the same effort.
Most software written does not serve a serious nation level user base but caters to so a relatively small set of users. The effort spent eradicating errors needs to be justified by the effort of workarounds, remediation work and customer impact. Will not be fixed can a rationale decision.
Most software written does not serve a serious nation level user base but caters to so a relatively small set of users. The effort spent eradicating errors needs to be justified by the effort of workarounds, remediation work and customer impact. Will not be fixed can a rationale decision.
I think the focus should be on tools with high surface area that enforce security bounadaries. Especially those where performance is not so important. Like sudo, openssh, polkit, PAM modules. It would be make a lot more sense than these half-baked rust rewrites that just take away features. (I'm biased I personally had a backup script broken by uutils) I think rewrites in rust need 100% bit for bit feature parity before replacing the battletested existing tools in c userland. I say this as someone who writes rust security tools for linux.
A lot of my programs crash, and that’s a deliberate choice. If you call one of them like “./myprog.py foo.txt”, and foo.txt doesn’t exist, it’ll raise a FileNotFound exception and fail with a traceback. Thing is, that’s desirable here. If could wrap that in a try/except block, but I’d either be adding extraneous info (“print(‘the file does not exist’); raise”) or throwing away valuable info by swallowing the traceback so the user doesn’t see the context of what failed.
My programs can’t do anything about that situation, so let it crash.
Same logic for:
* The server in the config file doesn’t exist.
* The given output file has bad permissions.
* The hard drive is full.
Etc. And again, that’s completely deliberate. There’s nothing I can do in code to fix those issues, so it’s better to fail with enough info that the user can diagnose and fix the problem.
That was in Python. I do the same in Rust, again, deliberately. While of course we all handle the weird cases we’re prepared to handle, I definitely write most database calls like “foo = db.exec(query)?” because if PostgreSQL can’t execute the query, the safest option is to panic instead of trying foolhardily to get back the last known safe state.
And of course that’s different for different use cases. If you’re writing a GUI app, it makes much more sense to pop up a dialog and make the user go fix the issue before retrying.
My programs can’t do anything about that situation, so let it crash.
Same logic for:
* The server in the config file doesn’t exist.
* The given output file has bad permissions.
* The hard drive is full.
Etc. And again, that’s completely deliberate. There’s nothing I can do in code to fix those issues, so it’s better to fail with enough info that the user can diagnose and fix the problem.
That was in Python. I do the same in Rust, again, deliberately. While of course we all handle the weird cases we’re prepared to handle, I definitely write most database calls like “foo = db.exec(query)?” because if PostgreSQL can’t execute the query, the safest option is to panic instead of trying foolhardily to get back the last known safe state.
And of course that’s different for different use cases. If you’re writing a GUI app, it makes much more sense to pop up a dialog and make the user go fix the issue before retrying.
That strategy is ok if the expected user is a fellow developer. For anyone else, a back trace is spilling your guts on the floor and expecting the user to clean up. A tool for wider usage should absolutely detect the error condition and explain the problem as succinctly as possible with all the information necessary for a human to be able to solve the issue. Back trace details are extraneous information only useful for a software developer familiar with the codebase. There's of course a difference when talking about unexpected incorrect state, something like a fatal filesystem or allocation error that shouldn't happen unless the environment is in an invalid state, a nice human error then is not as necessary.
Hard disagree, esp re "only useful for a software developer familiar with the codebase". Too many "user-friendly" apps create "helpful" error messages that hide essential information.
"Cannot open data file: Not found" and that's it - no more context (such as a filename). Even for a user with no coding experience, this is absolutely useless, you cannot find good explanation for it on the Google. A backtrace might look ugly, but at least would have a much higher chance to point to a useful forum post. And now with AI advances, AI can analyze backtraces and sometimes give an explanation (not very often, but there is no alternatives...)
So by all means, add a nice, human-readable error message for a few common cases that user likely to encounter, such as "internet down" or "wrong type of input file"... but leave backtraces on for all other unexpected cases, like "server returned nonsense", "out of disk space", etc....
"Cannot open data file: Not found" and that's it - no more context (such as a filename). Even for a user with no coding experience, this is absolutely useless, you cannot find good explanation for it on the Google. A backtrace might look ugly, but at least would have a much higher chance to point to a useful forum post. And now with AI advances, AI can analyze backtraces and sometimes give an explanation (not very often, but there is no alternatives...)
So by all means, add a nice, human-readable error message for a few common cases that user likely to encounter, such as "internet down" or "wrong type of input file"... but leave backtraces on for all other unexpected cases, like "server returned nonsense", "out of disk space", etc....
A backtrace won't give you the values of variables needed to identify that.
I did specify "succinctly as possible with all the information necessary for a human to be able to solve the issue". An error that doesn't have "all the information necessary" is a bad error. It can be worse than a backtrace. That doesn't mean a backtrace is good.
I did specify "succinctly as possible with all the information necessary for a human to be able to solve the issue". An error that doesn't have "all the information necessary" is a bad error. It can be worse than a backtrace. That doesn't mean a backtrace is good.
> Mature C programs written by professional coders (Redis is a good example) basically never crash in the experience of users
That is a very difficult assertion to validate. It might well be true! But so many conversations about memory safety and C/C++ devolve to assertions with “get gud” at one extreme and “change platforms to one that avoids certain errors” at the other.
Without data, even iffy data, those groups talk past each other. Are memory-error CVE counts on C projects the data we need here? Is there some other quantitative measure of real world failures that occur due to memory unsafety?
This is all by way of saying that I’d love to see some numbers there. That’s not on you, or meant to question your claim. As you implied, errors in code don’t always translate to errors in behavior for users.
It just always sucks to talk about this because broad-spectrum quantitative data on software error rates and their causes is lacking.
That is a very difficult assertion to validate. It might well be true! But so many conversations about memory safety and C/C++ devolve to assertions with “get gud” at one extreme and “change platforms to one that avoids certain errors” at the other.
Without data, even iffy data, those groups talk past each other. Are memory-error CVE counts on C projects the data we need here? Is there some other quantitative measure of real world failures that occur due to memory unsafety?
This is all by way of saying that I’d love to see some numbers there. That’s not on you, or meant to question your claim. As you implied, errors in code don’t always translate to errors in behavior for users.
It just always sucks to talk about this because broad-spectrum quantitative data on software error rates and their causes is lacking.
> That is a very difficult assertion to validate.
Keep in mind he's limited his assertion to UX. That narrow point is almost certainly true in the case of his C codebase.
But read the rest-- he literally wrote how security researchers find memory safety errors in C codebases!
Dollars to donuts he came up with this UX-on-accident vs. security-researcher-on-purpose bug dichotomy in his head as a response to some actual CVE in his own C codebase.
In short, he's agreeing with the research that led to programming languages like Rust in the first place. And even though he's found an odd way to agree, there's no assertion to validate here (at least wrt security).
Edit: clarifications
Keep in mind he's limited his assertion to UX. That narrow point is almost certainly true in the case of his C codebase.
But read the rest-- he literally wrote how security researchers find memory safety errors in C codebases!
Dollars to donuts he came up with this UX-on-accident vs. security-researcher-on-purpose bug dichotomy in his head as a response to some actual CVE in his own C codebase.
In short, he's agreeing with the research that led to programming languages like Rust in the first place. And even though he's found an odd way to agree, there's no assertion to validate here (at least wrt security).
Edit: clarifications
I heard this argument about Rust vs. C that Rust might be memory safe, but the reason why memory safety issues are so prominent in C programs, is that basically every other kind of problem has been fixed throughout its lifetime, so these are the only kind of issues that remain. Both in terms of security and stability.
This is very much not the case for programs that are much newer, even if they are written in Rust they still need years of maturation before they reach the quality of older C programs, as Rust programs suffer from non-memory safety issues just as much. That's why just rewriting things in Rust isn't a panacea.
The perfect example of this the Rust coreutils drama that has been going on.
This is very much not the case for programs that are much newer, even if they are written in Rust they still need years of maturation before they reach the quality of older C programs, as Rust programs suffer from non-memory safety issues just as much. That's why just rewriting things in Rust isn't a panacea.
The perfect example of this the Rust coreutils drama that has been going on.
I don't agree with that assessment at all. The reason memory safety issues are so prominent is that they are extremely likely to be exploitable. Of course you can write exploitable bugs in any language, but most bug classes are unlikely to be exploitable. A bug that always crashes is about a trillion times less severe than a bug that allows someone else to take control of your computer.
I can only quote (from the top of my head) the Android team's findings, that having a C++ codebase extended with Rust cut down significantly on the number of memory safety-related issues. The reasoning was that since the stable C++ codebase was no longer actively changed, only patched, and new features were implemented in Rust, the C++ codebase could go through this stabilization phase where almost all safety issues are found.
How many "mature C programs" try to recover in a usable way when malloc() returns NULL? That's a crash - a well-behaved one (no UB involved) hence not one that would be sought by most attackers other than a mere denial of service - but still a crash.
On 64-bit systems (esp Linux ones) malloc almost never returns NULL but keeps overallocating (aka overcommiting). You don't get out of memory errors / kills until you access it.
Exactly. Also, it is extremely rare.
> when malloc() returns NULL? That's a crash - a well-behaved one (no UB involved)
Wrong, dereferencing a NULL pointer is UB.
Wrong, dereferencing a NULL pointer is UB.
Which on UNIXes is a crash because the zero page is unmapped so you get a SIGSEGV
Unless the compiler optimized the access away, or replaced it with a different address.
One divide when it comes to using Fil-C is C as an application (git) vs C as a library from another language (libgit2).
Suppose we assume that many C applications aren’t performance sensitive and can easily take a 2-4x performance hit without noticing. Browsers and OS internals being obvious exceptions. The ideal candidates are like the ones djb writes, and he’s already a convert to Fil-C. sudo, sshd, curl - all seem like promising candidates.
But as far as I can tell, Fil-C doesn’t work for C libraries that can be called from elsewhere. Even if it could be made to work, the reason other languages like Python or Node use C libraries is for speed. If they were ok with it being 2-4x slower, they would just write ordinary Python or Javascript.
C (and C++) are fundamentally important because of their use in performance sensitive contexts like operating systems, browsers and libraries. If we’re restricting Fil-C to pure C/C++ applications that aren’t performance sensitive, that might still be very important and useful, but it’s a small slice of the large C/C++ pie.
Also, it’s a great tool for an existing C application, certainly. A performance hit in exchange for security is a reasonable trade off while making a battle hardened application work. But for a new application, would someone choose Fil-C over other performant GC languages like Go or Java or C#? I’d be keen to hear why.
Still, I want to stress - this is a great project and it’ll generate a lot of value.
Suppose we assume that many C applications aren’t performance sensitive and can easily take a 2-4x performance hit without noticing. Browsers and OS internals being obvious exceptions. The ideal candidates are like the ones djb writes, and he’s already a convert to Fil-C. sudo, sshd, curl - all seem like promising candidates.
But as far as I can tell, Fil-C doesn’t work for C libraries that can be called from elsewhere. Even if it could be made to work, the reason other languages like Python or Node use C libraries is for speed. If they were ok with it being 2-4x slower, they would just write ordinary Python or Javascript.
C (and C++) are fundamentally important because of their use in performance sensitive contexts like operating systems, browsers and libraries. If we’re restricting Fil-C to pure C/C++ applications that aren’t performance sensitive, that might still be very important and useful, but it’s a small slice of the large C/C++ pie.
Also, it’s a great tool for an existing C application, certainly. A performance hit in exchange for security is a reasonable trade off while making a battle hardened application work. But for a new application, would someone choose Fil-C over other performant GC languages like Go or Java or C#? I’d be keen to hear why.
Still, I want to stress - this is a great project and it’ll generate a lot of value.
> If they were ok with it being 2-4x slower, they would just write ordinary Python or Javascript.
Python and JavaScript are much more than 4x slower than C/C++ for workloads that are git-like (significant amount of compute, not just I/O bound)
> C (and C++) are fundamentally important because of their use in performance sensitive contexts like operating systems, browsers and libraries
That's a fun thing to say but it isn't really true. C/C++ are fundamentally important for lots of reasons. In many cases, folks choose C and C++ because that's the only way to get access to the APIs you need to get the job done.
Python and JavaScript are much more than 4x slower than C/C++ for workloads that are git-like (significant amount of compute, not just I/O bound)
> C (and C++) are fundamentally important because of their use in performance sensitive contexts like operating systems, browsers and libraries
That's a fun thing to say but it isn't really true. C/C++ are fundamentally important for lots of reasons. In many cases, folks choose C and C++ because that's the only way to get access to the APIs you need to get the job done.
Yeah I suppose that’s the real question - what is the size of the target market of Fil-C? I believe it’s a small % because many C applications are performance sensitive. You say performance is only one consideration among many so the % of C applications that would benefit from your compiler are much higher.
You’re the guy actually building this, so you would have talked to potential customers more than me. You’re more likely to be correct, but I would be interested to know what these applications are.
Also, do you hope people pick Fil-C for a new code base over other performant GC languages like Go, Java and C#? These have less than a 2x overhead for CPU bound tasks.
You’re the guy actually building this, so you would have talked to potential customers more than me. You’re more likely to be correct, but I would be interested to know what these applications are.
Also, do you hope people pick Fil-C for a new code base over other performant GC languages like Go, Java and C#? These have less than a 2x overhead for CPU bound tasks.
Why can't it work? You need to assume that the C library is only ever passed well-behaved pointers and callbacks in order to avoid invoking UB that it can't know about - but other than that it's just a matter of marshaling from the usual C ABI to the Fil-C ABI, which should be doable.
I’m assuming the calling program is a GC language like Python or Node (the most popular run times by far), but the same holds with other popular languages like Ruby. Why would a GC language call out to slow code, that runs its own separate GC. Now you have two GCs running, neither of which knows about the other. I’m not declaring it’s impossible, I’m asking why someone would want to do this.
An example: GitHub’s entire business revolves around calling libgit2 (C) from Ruby. Are they more likely to slow down libgit2 and make it substantially more complex by running 2 GCs side by side, or are they going to risk accept any potential unsafety in regular C? It’s 100% the latter, I’ll bet on that.
An example: GitHub’s entire business revolves around calling libgit2 (C) from Ruby. Are they more likely to slow down libgit2 and make it substantially more complex by running 2 GCs side by side, or are they going to risk accept any potential unsafety in regular C? It’s 100% the latter, I’ll bet on that.
> ...Now you have two GCs running, neither of which knows about the other. ...
For a strictly time-limited interaction (like what's involved in a FFI call) it's not that bad. Everything that GC2 might directly access is temporarily promoted to a root for GC1, and vice versa.
For a strictly time-limited interaction (like what's involved in a FFI call) it's not that bad. Everything that GC2 might directly access is temporarily promoted to a root for GC1, and vice versa.
A native library is an obvious memory safety hole so I don't see why would it be that controversial to want to fill it even if it introduces another GC (but working on an independent heap, so the slowdown is not necessarily multiplicative)
The former could still be cheaper than stop using libgit altogether.
No one is asking them to stop using libgit2 though. They’re going to continue using it. If they find a serious bug, they’ll fix it and continue using it.
The cost of all the additional hardware is just not worth it. If it was a choice between higher hardware costs, higher request latency, greater operational complexity of a new technology and rewriting libgit2 in a different language without all those tradeoffs, GitHub definitely chooses the latter.
But it’s never going to reach that point because they’ll continue using libgit2 compiled by clang forever.
The cost of all the additional hardware is just not worth it. If it was a choice between higher hardware costs, higher request latency, greater operational complexity of a new technology and rewriting libgit2 in a different language without all those tradeoffs, GitHub definitely chooses the latter.
But it’s never going to reach that point because they’ll continue using libgit2 compiled by clang forever.
That's very interesting to learn.
Yes, safety got more important, and it's great to support old C code in a safe way. The performance drop and specially the GC of Fil-C do limit the usage however. I read there are some ideas for Fil-C without GC; I would love to hear more about that!
But all existing programming languages seem to have some disadvange: C is fast but unsafe. Fil-C is C compatible but requires GC, more memory, and is slower. Rust is fast, uses little memory, but us verbose and hard to use (borrow checker). Python, Java, C# etc are easy to use, concise, but, like Fil-C, require tracing GC and so more memory, and are slow.
I think the 'perfect' language would be as concise as Python, statically typed, not require tracing GC like Swift (use reference counting), support some kind of borrow checker like Rust (for the most performance critical sections). And leverage the C ecosystem, by transpiling to C. And so would run on almost all existing hardware, and could even be used in the kernel.
But all existing programming languages seem to have some disadvange: C is fast but unsafe. Fil-C is C compatible but requires GC, more memory, and is slower. Rust is fast, uses little memory, but us verbose and hard to use (borrow checker). Python, Java, C# etc are easy to use, concise, but, like Fil-C, require tracing GC and so more memory, and are slow.
I think the 'perfect' language would be as concise as Python, statically typed, not require tracing GC like Swift (use reference counting), support some kind of borrow checker like Rust (for the most performance critical sections). And leverage the C ecosystem, by transpiling to C. And so would run on almost all existing hardware, and could even be used in the kernel.
> Python, Java, C# [...] are slow
These might all be slower than well written C or rust, but they're not nearly the same magnitude of slow. Java is often within a magnitude of C/C++ in practice, and threading is less of a pain. Python can easily be 100x slower, and until very recently, threading wasn't even an option for more CPU due to the GIL so you needed extra complexity to deal with that
There's also Golang, which is in the same ballpark as java and c
These might all be slower than well written C or rust, but they're not nearly the same magnitude of slow. Java is often within a magnitude of C/C++ in practice, and threading is less of a pain. Python can easily be 100x slower, and until very recently, threading wasn't even an option for more CPU due to the GIL so you needed extra complexity to deal with that
There's also Golang, which is in the same ballpark as java and c
You are right, languages with tracing GC are fast. Often, they are faster than C or Rust, if you measure peak performance of a micro-benchmark that does a lot of memory management. But that is only true if you just measure the speed of the main thread :-) Tracing garbage collection does most of the work in separate threads, and so is often not visible in benchmarks. Memory usage is also not easily visible, but languages with tracing GC need about twice the amount of memory than eg. C or Rust. (When using an area allocator in C, you can get faster, at the cost of memory usage.)
Yes, Python is specially slow, but I think it's probably more because it's dynamically typed, and not not compiled. I found PyPy is quite fast.
Yes, Python is specially slow, but I think it's probably more because it's dynamically typed, and not not compiled. I found PyPy is quite fast.
I've built high load services in Java. GC can be an issue if it gets bad enough to have to pause, but it's in no way a big performance drain regularly.
pypy is fast compared to plain python, but it's not remotely in the same ballpark as C, Java, Golang
pypy is fast compared to plain python, but it's not remotely in the same ballpark as C, Java, Golang
Sure, it's not a big performance drain. For the vast majority of software, it is fine. Usually, the ability to write programs more quickly in eg. Java (not having to care about memory management) outweighs the possible gain of Rust that can reduce memory usage, and total energy usage (because no background thread are needed for GC). I also write most software in Java. Right now, the ergonomics of languages that don't require tracing GC is just too high. But I don't think this is a law of nature; it's just that there a now better languages yet that don't require a tracing GC. The closest is probably Swift, from a memory / energy usage perspective, but it has other issues.
> and total energy usage
Surprisingly, Java is right behind manual memory managed languages in terms of energy use, due to its GC being so efficient. It turns out that if your GC can "sprint very fast", you can postpone running it till the last second, and memory drains the same amount no matter what kind of garbage it holds. Also, just "booking" that this region is now garbage without doing any work is also cheaper than calling potentially a chain of destructors or incrementing/decrementing counters.
Surprisingly, Java is right behind manual memory managed languages in terms of energy use, due to its GC being so efficient. It turns out that if your GC can "sprint very fast", you can postpone running it till the last second, and memory drains the same amount no matter what kind of garbage it holds. Also, just "booking" that this region is now garbage without doing any work is also cheaper than calling potentially a chain of destructors or incrementing/decrementing counters.
> not visible in benchmarks
fwiw benchmarksgame uses benchexec
https://github.com/sosy-lab/benchexec
fwiw benchmarksgame uses benchexec
https://github.com/sosy-lab/benchexec
Of these languages, C# may actually be the fastest.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
In most cases the later entries in a language for the benchmark game are increasingly hyper-optimized and non-idiomatic for that language, which is exactly where C# will say "Here's some dangerous features, be careful" and the other languages are likely to suggest you use a bare metal language instead.
Presumably the benchmark game doesn't allow "I wrote this code in C" as a Python submission, but it would allow unsafe C# tricks ?
Presumably the benchmark game doesn't allow "I wrote this code in C" as a Python submission, but it would allow unsafe C# tricks ?
Note: 'possible hand-written vector instructions or "unsafe" or naked ffi' are flagged by *
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Note: Here are naive un-optimised single-thread programs transliterated line-by-line literal style into different programming languages from the same original.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Note: Here are naive un-optimised single-thread programs transliterated line-by-line literal style into different programming languages from the same original.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Unsafe C# is still C# though. Also C# has a lot more control over memory than Java for example, so you don't actually need to use unsafe to be fast. Or are you trying to say that C# is only fast when using unsafe?
Likely just that the fastest implementations in the benchmarks game are using those features and so aren't really a good reflection of the language as it is normally used. This is a problem for any language on the list, really; the fastest implementations are probably not going to reflect idiomatic coding practices.
Here are naive un-optimised single-thread programs transliterated line-by-line literal style into different programming languages from the same original.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
> The performance drop and specially the GC of Fil-C do limit the usage however. I read there are some ideas for Fil-C without GC; I would love to hear more about that!
I love how people assume that the GC is the reason for Fil-C being slower than C and that somehow, if it didn't have a GC, it wouldn't be slower.
Fil-C is slower than C because of InvisiCaps. https://fil-c.org/invisicaps
The GC is is crazy fast and fully concurrent/parallel. https://fil-c.org/fugc
Removing the GC is likely to make Fil-C slower, not faster.
I love how people assume that the GC is the reason for Fil-C being slower than C and that somehow, if it didn't have a GC, it wouldn't be slower.
Fil-C is slower than C because of InvisiCaps. https://fil-c.org/invisicaps
The GC is is crazy fast and fully concurrent/parallel. https://fil-c.org/fugc
Removing the GC is likely to make Fil-C slower, not faster.
Well I didn't mean GC is the reason for Fil-C being slower. I mean the performance drop of Fil-C (as described in the article) limits the usage, and the GC (independently) limits the usage.
I understand raw speed (of the main thread) of Fil-C can be faster with tracing GC than Fil-C without. But I think there's a limit on how fast and memory efficient Fil-C can get, given it necessarily has to do a lot of things at runtime, versus compile time. Energy usage, and memory usage or a programming language that uses a tracing GC is higher than one without. At least, if memory management logic can be done at compile time.
For Fil-C, a lot of the memory management logic, and checks, necessarily needs to happen at runtime. Unless if the code is annotated somehow, but then it wouldn't be pure C any longer.
I understand raw speed (of the main thread) of Fil-C can be faster with tracing GC than Fil-C without. But I think there's a limit on how fast and memory efficient Fil-C can get, given it necessarily has to do a lot of things at runtime, versus compile time. Energy usage, and memory usage or a programming language that uses a tracing GC is higher than one without. At least, if memory management logic can be done at compile time.
For Fil-C, a lot of the memory management logic, and checks, necessarily needs to happen at runtime. Unless if the code is annotated somehow, but then it wouldn't be pure C any longer.
I wonder if some of the Apple provided Clang annotations for bounds checking can be combined with Fil-C?
That then may allow for some of the uses to be statically optimised away, i.e. by annotating pointers upon which arithmetic is not allowed.
The Fil-C capability mechanisms for trapping double-free, and use-after free would probably have to be retained, but maybe it could optimise some uses?
That then may allow for some of the uses to be statically optimised away, i.e. by annotating pointers upon which arithmetic is not allowed.
The Fil-C capability mechanisms for trapping double-free, and use-after free would probably have to be retained, but maybe it could optimise some uses?
Nim fits most of those descriptors, and it’s become my favorite language to use. Like any language, it’s still a compromise, but it sits in a really nice spot in terms of compromises, at least IMO. Its biggest downsides are all related to its relative “obscurity” (compared to the other mentioned languages) and resulting small ecosystem.
The advantage of Fil-C is that it's C, not some other language. For the problem domain it's most suited to, you'd do C/C++, some other ultra-modern memory-safe C/C++ system, or Rust.
I agree. Nim is memory safe, concise, and fast. In my view, Nim lacks a very clear memory management strategy: it supports ARC, ORC, manual (unsafe) allocation, move semantics. Maybe supporting viewer options would be better? Usually, adding things that are lacking is easier than removing features, specially if the community is small and if you don't want to alienate too many people.
Slow to whom, though?
Yes, they might lose the meaningless benchmarks game that gets thrown around, what matters is are they fast enough for the problem that is being solved.
If everyone actually cared about performance above anything else, we wouldn't have an Electron crap crisis.
Yes, they might lose the meaningless benchmarks game that gets thrown around, what matters is are they fast enough for the problem that is being solved.
If everyone actually cared about performance above anything else, we wouldn't have an Electron crap crisis.
Seems like Windows is trying to address the Electron problem by adopting React Native for their WinAppSDK. RN is not just a cross-platform solution, but a framework that allows Windows to finally tap into the pool of devs used to that declarative UI paradigm. They appear to be standardizing on TypeScript, with C++ for the performance-critical native parts. They leverage the scene graph directly from WinAppSDK. By prioritizing C++ over C# for extensions and TS for the render code, they might actually hit the sweet spot.
https://microsoft.github.io/react-native-windows/docs/new-ar...
https://microsoft.github.io/react-native-windows/docs/new-ar...
Anything related to WinUI is a bad joke.
Have fun following the discussions and amount of bugs,
https://github.com/microsoft/microsoft-ui-xaml
That C++ support that WinUI team marketing keeps talking about relies on a framework that is no longer being developed.
> The reason the issues page only lets you create a bug report is because cppwinrt is in maintenance mode and no longer receiving new feature work. cppwinrt serves an important and specific role, but further feature development risks destabilizing the project. Additional helpers are regularly contributed to complimentary projects such as https://github.com/microsoft/wil/.
From https://github.com/microsoft/cppwinrt/issues/1289#issuecomme...
Have fun following the discussions and amount of bugs,
https://github.com/microsoft/microsoft-ui-xaml
That C++ support that WinUI team marketing keeps talking about relies on a framework that is no longer being developed.
> The reason the issues page only lets you create a bug report is because cppwinrt is in maintenance mode and no longer receiving new feature work. cppwinrt serves an important and specific role, but further feature development risks destabilizing the project. Additional helpers are regularly contributed to complimentary projects such as https://github.com/microsoft/wil/.
From https://github.com/microsoft/cppwinrt/issues/1289#issuecomme...
I don't know I think what matters is that performance is close to the best you can reasonably get in any other language.
People don't like leaving performance on the table. It feels stupid and it lets competitors have an easy advantage.
The Electron situation is not because people don't care about performance; it's because they care more about some other things (e.g. not having to do 4x the work to get native apps).
People don't like leaving performance on the table. It feels stupid and it lets competitors have an easy advantage.
The Electron situation is not because people don't care about performance; it's because they care more about some other things (e.g. not having to do 4x the work to get native apps).
Your second paragraph kind of contradicts the last one.
And yes, caring more about other things is why performance isn't the top number one item, and most applications have long stopped being written in pure C or C++ since the early 2000's.
We go even further in several abstraction layers, nowadays with the ongoing uptake of LLMs and agentic workflows in iPaaS low code tools.
Personally at work I haven't written a pure 100% C or C++ application since 1999, always a mix of Tcl, Perl, Python, C# alongside C or C++, private projects is another matter.
And yes, caring more about other things is why performance isn't the top number one item, and most applications have long stopped being written in pure C or C++ since the early 2000's.
We go even further in several abstraction layers, nowadays with the ongoing uptake of LLMs and agentic workflows in iPaaS low code tools.
Personally at work I haven't written a pure 100% C or C++ application since 1999, always a mix of Tcl, Perl, Python, C# alongside C or C++, private projects is another matter.
Most applications stopped being written in C/C++ when Java first came out - the first memory safe language with mass enterprise adoption. Java was the Rust of the mid-1990s, even though it used a GC which made it a lot slower and clunkier than actual Rust.
I would say that the "first" belongs to Smalltalk, Visual Basic and Delphi.
What Java had going for it was the massive scale of Sun's marketing, and the JDK being available as free beer, however until Eclipse came to be, all IDEs were commercial, and everyone was coding in Emacs, vi (no vim yet), nano, and so on.
However it only became viable after Java 1.3, when Hotspot became part of Java's runtime.
I agree with the spirit of your comment though, and I also think that the blow given by Java to C and C++ wasn't bigger, only because AOT tools were only available under high commercial prices.
Many folks use C and C++, not due to their systems programming features, rather they are the only AOT compiled languages that they know.
What Java had going for it was the massive scale of Sun's marketing, and the JDK being available as free beer, however until Eclipse came to be, all IDEs were commercial, and everyone was coding in Emacs, vi (no vim yet), nano, and so on.
However it only became viable after Java 1.3, when Hotspot became part of Java's runtime.
I agree with the spirit of your comment though, and I also think that the blow given by Java to C and C++ wasn't bigger, only because AOT tools were only available under high commercial prices.
Many folks use C and C++, not due to their systems programming features, rather they are the only AOT compiled languages that they know.
When performance doesn't matter, it doesn't matter.
> And leverage the C ecosystem, by transpiling to C
I heavily doubt that this would work on arbitrary C compilers reliably as the interpretation of the standard gets really wonky and certain constructs that should work might not even compile. Typically such things target GCC because it has such a large backend of supported architectures. But LLVM supports a large overlapping number too - thats why it’s supported to build the Linux kernel under clang and why Rust can support so many microcontrollers. For Rust, that’s why there’s the rust codegen gcc effort which uses GCC as the backend instead of LLVM to flush out the supported architectures further. But generally transpiration is used as a stopgap for anything in this space, not an ultimate target for lots of reasons, not least of which that there’s optimizations that aren’t legal in C that are in another language that transpilation would inhibit.
> Rust is fast, uses little memory, but us verbose and hard to use (borrow checker).
It’s weird to me that my experience is that it was as hard to pick up the borrow checker as the first time I came upon list comprehension. In essence it’s something new I’d never seen before but once I got it it went into the background noise and is trivial to do most of the time, especially since the compiler infers most lifetimes anyway. Resistance to learning is different than being difficult to learn.
I heavily doubt that this would work on arbitrary C compilers reliably as the interpretation of the standard gets really wonky and certain constructs that should work might not even compile. Typically such things target GCC because it has such a large backend of supported architectures. But LLVM supports a large overlapping number too - thats why it’s supported to build the Linux kernel under clang and why Rust can support so many microcontrollers. For Rust, that’s why there’s the rust codegen gcc effort which uses GCC as the backend instead of LLVM to flush out the supported architectures further. But generally transpiration is used as a stopgap for anything in this space, not an ultimate target for lots of reasons, not least of which that there’s optimizations that aren’t legal in C that are in another language that transpilation would inhibit.
> Rust is fast, uses little memory, but us verbose and hard to use (borrow checker).
It’s weird to me that my experience is that it was as hard to pick up the borrow checker as the first time I came upon list comprehension. In essence it’s something new I’d never seen before but once I got it it went into the background noise and is trivial to do most of the time, especially since the compiler infers most lifetimes anyway. Resistance to learning is different than being difficult to learn.
Well "transpiling to C" does include GCC and clang, right? Sure, trying to support _all_ C compilers is nearly impossible, and not what I mean. Quite many languages support transpiling to C (even Go and Lua), but in my view that alone is not sufficient for a C replacement in places like the Linux kernel: for this to work, tracing GC can not be used. And this is what prevents Fil-C and many other languages to be used in that area.
Rust borrow checker: the problem I see is not so much that it's hard to learn, but requires constant effort. In Rust, you are basically forced to use it, even if the code is not performance critical. Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python. The main disadvantage of Rust, in my view, is that it's verbose. (Also, there is a tendency to add too many features, similar to C++, but that's a secondary concern).
Rust borrow checker: the problem I see is not so much that it's hard to learn, but requires constant effort. In Rust, you are basically forced to use it, even if the code is not performance critical. Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python. The main disadvantage of Rust, in my view, is that it's verbose. (Also, there is a tendency to add too many features, similar to C++, but that's a secondary concern).
> Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python. The main disadvantage of Rust, in my view, is that it's verbose.
I think there's space for Rust to become more ergonomic, but its goals limit just how far it can go. At the same time I think there's space to take Rust and make a Rust# that goes further on the Swift/Scala end of the spectrum, where things like auto-cloning of references are implemented first, that can consume Rust libraries. From the organizational point of you, you can see it as a mix between nightly and editions. From a user's point of view you can look at it as a mode to make refactoring faster, onbiarding easier and a test bed for language evolution. Not being Rust itself it would also allow for different stability guarantees (you can have breaking changes every year), which also means you can be holder on tryin things out knowing you're not permanently stuck with them. People who care about performance, correctness and reuse can still use Rust. People who would be well served by Swift/Scala, have access to Rust's libraries and toolchain.
> (Also, there is a tendency to add too many features, similar to C++, but that's a secondary concern).
These two quoted sentiments seem contradictory: making Rust less verbose to interact with reference counted values would indeed be adding a feature.
I think there's space for Rust to become more ergonomic, but its goals limit just how far it can go. At the same time I think there's space to take Rust and make a Rust# that goes further on the Swift/Scala end of the spectrum, where things like auto-cloning of references are implemented first, that can consume Rust libraries. From the organizational point of you, you can see it as a mix between nightly and editions. From a user's point of view you can look at it as a mode to make refactoring faster, onbiarding easier and a test bed for language evolution. Not being Rust itself it would also allow for different stability guarantees (you can have breaking changes every year), which also means you can be holder on tryin things out knowing you're not permanently stuck with them. People who care about performance, correctness and reuse can still use Rust. People who would be well served by Swift/Scala, have access to Rust's libraries and toolchain.
> (Also, there is a tendency to add too many features, similar to C++, but that's a secondary concern).
These two quoted sentiments seem contradictory: making Rust less verbose to interact with reference counted values would indeed be adding a feature.
> Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python.
If that's what you're looking for, you can use Swift. The latest release has memory safety by default, just like Rust.
If that's what you're looking for, you can use Swift. The latest release has memory safety by default, just like Rust.
Someone, maybe Tolnay?, recently posted a short Go snippet that segfaults because the virtual function table pointer and data pointer aren't copied atomically or mutexed. The same thing works in swift, because neither is thread safe. Swift is also slower than go unless you pass unchecked making it even less safe than go. C#/f# are safer from that particular problem and more performant than either go or swift, but have suffered from the same deserialization attacks that java does. Right now if you want true memory and thread safety, you need to limit a GC language to zero concurrency, use a borrow checker, i.e. rust, or be purely functional which in production would mean haskell. None of those are effortless, and which is easiest depends on you and your problem. Rust is easiest for me, but I keep thinking if I justvwrite enough haskell it will all click. I'm worried if my brain starts working that way about the impacts on things other than writing Haskell.
Replying to myself because a vouch wasn't enough to bring the post back from the dead. They were partially right and educated me. The downvotes were unnecessary. MS did start advising against dangerous deserializers 8yrs ago. They were only deprecated three years ago though, and only removed last year. Some of the remaining are only mostly safe and then only if you follow best practice. So it isn't a problem entirely of the past, but it has gotten a lot better.
Unless you are writing formal proofs nothing is completely safe, GC languages had found a sweet spot until increased concurrency started uncovering thread safety problems. Rust seems to have found a sweet spot that is usable despite the grumbling. It could probably be made a bit easier. The compiler already knows when something needs to be send or synch, and it could just do that invisibly, but that would lead people to code in a way that had lots of locking which is slow and generates deadlocks too often. This way the wordiness of shared mutable state steers you towards avoiding it except when a functional design pattern wouldn't be performant. If you have to use mutex a lot in rust stop fighting the borrow checker and listen to what it is saying.
Unless you are writing formal proofs nothing is completely safe, GC languages had found a sweet spot until increased concurrency started uncovering thread safety problems. Rust seems to have found a sweet spot that is usable despite the grumbling. It could probably be made a bit easier. The compiler already knows when something needs to be send or synch, and it could just do that invisibly, but that would lead people to code in a way that had lots of locking which is slow and generates deadlocks too often. This way the wordiness of shared mutable state steers you towards avoiding it except when a functional design pattern wouldn't be performant. If you have to use mutex a lot in rust stop fighting the borrow checker and listen to what it is saying.
> C#/f# are safer from that particular problem and more performant than either go or swift, but have suffered from the same deserialization attacks that java does.
They have not in the past 10 years.
They have not in the past 10 years.
Yes. I do like Swift as a language. The main disadvantages of Swift, in my view, are: (A) The lack of an (optional) "ownership" model for memory management. So you _have_ to use reference counting everywhere. That limits the performance. This is measurable: I converted some micro-benchmarks to various languages, and Swift does suffer for the memory managment intensive tasks [1]. (B) Swift is too Apple-centric currently. Sure, this might be become a non-issue over time.
[1] https://github.com/thomasmueller/bau-lang/blob/main/doc/perf...
[1] https://github.com/thomasmueller/bau-lang/blob/main/doc/perf...
Re: borrow checker
Isn't it just enforcing something you should be doing in every language anyway, i.e. thinking about ownership of data.
Isn't it just enforcing something you should be doing in every language anyway, i.e. thinking about ownership of data.
The borrow checker involves documenting the ownership of data throughout the program. That's what people are calling "overly verbose" and saying it "makes comprehensive large-scale refactoring impractical" as an argument against Rust. (And no it doesn't, it's just keeping you honest about what the refactor truly involves.)
The annoying experience with the borrow checker is when following the compiler errors after making a change until you hit a fundamental ownership problem a few levels away from the original change that precludes the change (like ending up with a self referencial borrow). This can bite even experienced developers, depending on how many layers of indirection there are (and sometimes the change that would be adding a single Rc or Cell in a field isn't applicable because it happens in a library you don't control). I do still prefer hitting that wall than having it compile and end up with rare incorrect runtime behaviour (with any luck, a segfault), but it is more annoying than "it just works because the GC dealt with it for me".
There are also limits to what the borrow checker is capable of verifying. There will always be programs which are valid under the rules the borrow checker is enforcing, but the borrow checker rejects.
It's kinda annoying when you run into those. I think I've also ran into a situation where the borrow checker itself wasn't the issue, but rather the way references were created in a pattern match causing the borrow checker to reject the program. That was also annoying.
It's kinda annoying when you run into those. I think I've also ran into a situation where the borrow checker itself wasn't the issue, but rather the way references were created in a pattern match causing the borrow checker to reject the program. That was also annoying.
Polonius hopefully arrives next year and reduces the burden here further. Partial field borrows would be huge so that something like obj.set_bar(obj.foo()) would work.
Given the troubles with shipping Polonius, I imagine that there isn't much more room for improvements in "pure borrow checking" after Polonius, though more precise ways to borrow should improve ergonomics a lot more. You mentioned borrowing just the field; I think self-referential borrows are another.
The borrow checker is an approximation of an ideal model of managing things. In the general case, the guidelines that the borrow checker establishes are a useful way to structure code (though not necessarily the only way), but sometimes the borrow checker simply doesn't accept code that is logically sound. Rust is statically analyzed with an emphasis on safety, so that is the tradeoff made for Rust.
> Quite many languages support transpiling to C (even Go and Lua)
Source? I’m not familiar with official efforts here. I see one in the community for Lua but nothing for Go. It’s rare for languages to use this as anything other than a stopgap or a neat community poc. But my point was precisely this - if you’re only targeting GCC/LLVM, you can just use their backend directly rather than transpiling to C which only buys you some development velocity at the beginning (as in easier to generate that from your frontend vs the intermediate representation) at the cost of a worse binary output (since you have to encode the language semantics on top of the C virtual machine which isn’t necessarily free). Specifically this is why transpile to C makes no sense for Rust - it’s already got all the infrastructure to call the compiler internals directly without having to go through the C frontend.
> Rust borrow checker: the problem I see is not so much that it's hard to learn, but requires constant effort. In Rust, you are basically forced to use it, even if the code is not performance critical
Your only forced to use it when you’re storing references within a struct. In like 99% of all other cases the compiler will correctly infer the lifetimes for you. Not sure when the last time was you tried to write rust code.
> Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python.
Any language targeting the performance envelope rust does needs GC to be opt in. And I’m not sure how much extra verbosity there is to wrap the type with RC/Arc unless you’re referring to the need to throw in a RefCell/Mutex to support in place mutation as well, but that goes back to there not being an alternative easy way to simultaneously have safety and no runtime overhead.
> The main disadvantage of Rust, in my view, is that it's verbose.
Sure, but compared to what? It’s actually a lot more concise than C/C++ if you consider how much boilerplate dancing there is with header files and compilation units. And if you start factoring in that few people actually seem to actually know what the rule of 0 is and how to write exception safe code, there’s drastically less verbosity and the verbosity is impossible to use incorrectly. Compared to Python sure, but then go use something like otterlang [1] which gives you close to Rust performance with a syntax closer to Python. But again, it’s a different point on the Pareto frontier - there’s no one language that could rule them all because they’re orthogonal design criteria that conflict with each other. And no one has figured out how to have a cohesive GC that transparently and progressively lets you go between no GC, ref GC and tracing GC despite foundational research a few years back showing that ref GC and tracing GC are part of the same spectrum and high performing implementations in both the to converge on the same set of techniques.
[1] https://github.com/jonathanmagambo/otterlang
Source? I’m not familiar with official efforts here. I see one in the community for Lua but nothing for Go. It’s rare for languages to use this as anything other than a stopgap or a neat community poc. But my point was precisely this - if you’re only targeting GCC/LLVM, you can just use their backend directly rather than transpiling to C which only buys you some development velocity at the beginning (as in easier to generate that from your frontend vs the intermediate representation) at the cost of a worse binary output (since you have to encode the language semantics on top of the C virtual machine which isn’t necessarily free). Specifically this is why transpile to C makes no sense for Rust - it’s already got all the infrastructure to call the compiler internals directly without having to go through the C frontend.
> Rust borrow checker: the problem I see is not so much that it's hard to learn, but requires constant effort. In Rust, you are basically forced to use it, even if the code is not performance critical
Your only forced to use it when you’re storing references within a struct. In like 99% of all other cases the compiler will correctly infer the lifetimes for you. Not sure when the last time was you tried to write rust code.
> Sure, Rust also supports reference counting GC, but that is more _verbose_ to use... It should be _simpler_ to use in my view, similar to Python.
Any language targeting the performance envelope rust does needs GC to be opt in. And I’m not sure how much extra verbosity there is to wrap the type with RC/Arc unless you’re referring to the need to throw in a RefCell/Mutex to support in place mutation as well, but that goes back to there not being an alternative easy way to simultaneously have safety and no runtime overhead.
> The main disadvantage of Rust, in my view, is that it's verbose.
Sure, but compared to what? It’s actually a lot more concise than C/C++ if you consider how much boilerplate dancing there is with header files and compilation units. And if you start factoring in that few people actually seem to actually know what the rule of 0 is and how to write exception safe code, there’s drastically less verbosity and the verbosity is impossible to use incorrectly. Compared to Python sure, but then go use something like otterlang [1] which gives you close to Rust performance with a syntax closer to Python. But again, it’s a different point on the Pareto frontier - there’s no one language that could rule them all because they’re orthogonal design criteria that conflict with each other. And no one has figured out how to have a cohesive GC that transparently and progressively lets you go between no GC, ref GC and tracing GC despite foundational research a few years back showing that ref GC and tracing GC are part of the same spectrum and high performing implementations in both the to converge on the same set of techniques.
[1] https://github.com/jonathanmagambo/otterlang
I agree transpile to C will not result in the fastest code (and of course not the fastest toolchain), but having the ability to convert to C does help in some cases. Besides the ability to support some more obscure targets, I found it's useful for building a language, for unit tests [1]. One of the targets, in my case, is the XCC C compiler, which can run in WASM and convert to WASM, and so I built the playground for my language using that.
> transpiling to C (even Go and Lua)
Go: I'm sorry, I thought TinyGo internally converts to C, but it turns out that's not true (any more?). That leaves https://github.com/opd-ai/go2c which uses TinyGo and then converts the LLVM IR to C. So, I'm mistaken, sorry.
Lua: One is https://github.com/davidm/lua2c but I thought eLua also converts to C.
> Your only forced to use it when you’re storing references within a struct.
Well, that's quite often, in my view.
> Not sure when the last time was you tried to write rust code.
I'm not a regular user, that's true [2]. But I do have some knowledge in quite many languages now [3] and so I think I have a reasonable understanding of the advantages and disadvantages of Rust as well.
> Any language targeting the performance envelope rust does needs GC to be opt in.
Yes, I fully agree. I just think that Rust has the wrong default: it uses single ownership / borrowing by _default_, and RC/Arc is more like an exception. I think most programs could use RC/Arc by default, and only use ownership / borrowing where performance is critical.
> The main disadvantage of Rust, in my view, is that it's verbose. >> Sure, but compared to what?
Compared to most languages, actually [4]. Rust is similar to Java and Zig in this regard. Sure, we can argue the use case of Rust is different than eg. Python.
[1] https://github.com/thomasmueller/bau-lang [2] https://github.com/thomasmueller/lz4_simple [3] https://github.com/thomasmueller/bau-lang/tree/main/src/test... [4] https://github.com/thomasmueller/bau-lang/blob/main/doc/conc...
> transpiling to C (even Go and Lua)
Go: I'm sorry, I thought TinyGo internally converts to C, but it turns out that's not true (any more?). That leaves https://github.com/opd-ai/go2c which uses TinyGo and then converts the LLVM IR to C. So, I'm mistaken, sorry.
Lua: One is https://github.com/davidm/lua2c but I thought eLua also converts to C.
> Your only forced to use it when you’re storing references within a struct.
Well, that's quite often, in my view.
> Not sure when the last time was you tried to write rust code.
I'm not a regular user, that's true [2]. But I do have some knowledge in quite many languages now [3] and so I think I have a reasonable understanding of the advantages and disadvantages of Rust as well.
> Any language targeting the performance envelope rust does needs GC to be opt in.
Yes, I fully agree. I just think that Rust has the wrong default: it uses single ownership / borrowing by _default_, and RC/Arc is more like an exception. I think most programs could use RC/Arc by default, and only use ownership / borrowing where performance is critical.
> The main disadvantage of Rust, in my view, is that it's verbose. >> Sure, but compared to what?
Compared to most languages, actually [4]. Rust is similar to Java and Zig in this regard. Sure, we can argue the use case of Rust is different than eg. Python.
[1] https://github.com/thomasmueller/bau-lang [2] https://github.com/thomasmueller/lz4_simple [3] https://github.com/thomasmueller/bau-lang/tree/main/src/test... [4] https://github.com/thomasmueller/bau-lang/blob/main/doc/conc...
> I'm not a regular user, that's true [2]. But I do have some knowledge in quite many languages now [3] and so I think I have a reasonable understanding of the advantages and disadvantages of Rust as well.
That is skewing your perception. The problem is that how you write code just changes after a while and both things happen: you know how to write things to leverage the compiler inferred lifetimes better and the lifetimes fade into the noise. It only seems really annoying, difficult and verbose at first which is what can skew your perception if you don’t actually commit to writing a lot of code and reading others’ code so that you become familiar with it better.
> Compared to most languages, actually [4]. Rust is similar to Java and Zig in this regard. Sure, we can argue the use case of Rust is different than eg. Python.
That these are the languages you’re comparing of is a point in Rust’s favor - it’s targeting a significantly lower level and higher performance of language. So Java is not comparable at all. Zig however nice is fundamentally not a safe language (more like C with fewer razor blades) and is inappropriate from that perspective. Like I said - it fits a completely different Pareto frontier - it’s strictly better than C/C++ on every front (even with the borrow checker it’s faster and less painful development) and people are considering it in the same breath as Go (also unsafe and not as fast), Java (safe but not as fast) and Python (very concise but super slow and code is often low quality historically).
That is skewing your perception. The problem is that how you write code just changes after a while and both things happen: you know how to write things to leverage the compiler inferred lifetimes better and the lifetimes fade into the noise. It only seems really annoying, difficult and verbose at first which is what can skew your perception if you don’t actually commit to writing a lot of code and reading others’ code so that you become familiar with it better.
> Compared to most languages, actually [4]. Rust is similar to Java and Zig in this regard. Sure, we can argue the use case of Rust is different than eg. Python.
That these are the languages you’re comparing of is a point in Rust’s favor - it’s targeting a significantly lower level and higher performance of language. So Java is not comparable at all. Zig however nice is fundamentally not a safe language (more like C with fewer razor blades) and is inappropriate from that perspective. Like I said - it fits a completely different Pareto frontier - it’s strictly better than C/C++ on every front (even with the borrow checker it’s faster and less painful development) and people are considering it in the same breath as Go (also unsafe and not as fast), Java (safe but not as fast) and Python (very concise but super slow and code is often low quality historically).
I think transpiling to C is probably the least interesting way to tap into C. FFI is a lot valuable (and doable).
There are surprisingly many languages that support transpiling to C: Python (via Cython), Go (via TinyGo), Lua (via eLua), Nim, Zig, Vlang. The main advantage (in my view) is to support embedded systems, which might not match your use case.
Eiffel, that is how it always worked, a VM based workflow for development (Melt VM), compilation via C or C++ for release builds.
I suppose /some/ performance loss is inevitable. But this could be quite a game changer. As more folks play with it, performing benchmarks, etc -- it should reveal which C idioms incur the most/least performance hits under Fil-C. So with some targetted patching of C code, we may end up with a rather modest price for the memory safety
And I'm not done optimizing. The perf will get better. Rust and Yolo-C will always be faster, but right now we can't know what the difference will be.
Top optimization opportunities:
- InvisiCaps 2.0. While implementing the current capability model, when I was about 3/4 of the way done with the rewrite, I realized that if I had done it differently I would have avoided two branch+compares on every pointer load. That's huge! I just haven't had the appetite for doing yet another rewrite recently. But I'll do it eventually.
- ABI. Right now, Fil-C uses a binary interface that relies on lowering to what ELF is capable of. This introduces a bunch of overhead on every global variable access and every function call. All of this goes away if Fil-C gets its own object file format. That's a lot of work, but it will happen in Fil-C gets more adoption.
- Better abstract interpreter. Fil-C already has an abstract interpreter in the compiler, but it's not nearly as smart as it could be. For example, it doesn't have octagon domain yet. Giving it octagon domain will dramatically improve the performance of loops.
- More intrinsics. Right now, a lot of libc functions that are totally memory safe but are implemented in assembly are implemented in plain Fil-C instead right now, just because of how the libc ports happened to work out. Like, say you call some <math.h> function that takes doubles and returns doubles - it's going to be slower in Fil-C today because you'll end up in the generic C code version compiled with Fil-C. No good reason for this! It's just grunt work to fix!
- The calling convention itself is trash right now - it involves passing things through a thread-local buffer. It's less trashy than the calling convention I started out with (that allocated everything in the heap lmao), but still. There's nothing fundamentally preventing a Fil-C register-based calling convention, but it would take a decent amount of work to implement.
There are probably other perf optimization opportunities that I'm either forgetting right now or that haven't been found yet. It's still early days!
Top optimization opportunities:
- InvisiCaps 2.0. While implementing the current capability model, when I was about 3/4 of the way done with the rewrite, I realized that if I had done it differently I would have avoided two branch+compares on every pointer load. That's huge! I just haven't had the appetite for doing yet another rewrite recently. But I'll do it eventually.
- ABI. Right now, Fil-C uses a binary interface that relies on lowering to what ELF is capable of. This introduces a bunch of overhead on every global variable access and every function call. All of this goes away if Fil-C gets its own object file format. That's a lot of work, but it will happen in Fil-C gets more adoption.
- Better abstract interpreter. Fil-C already has an abstract interpreter in the compiler, but it's not nearly as smart as it could be. For example, it doesn't have octagon domain yet. Giving it octagon domain will dramatically improve the performance of loops.
- More intrinsics. Right now, a lot of libc functions that are totally memory safe but are implemented in assembly are implemented in plain Fil-C instead right now, just because of how the libc ports happened to work out. Like, say you call some <math.h> function that takes doubles and returns doubles - it's going to be slower in Fil-C today because you'll end up in the generic C code version compiled with Fil-C. No good reason for this! It's just grunt work to fix!
- The calling convention itself is trash right now - it involves passing things through a thread-local buffer. It's less trashy than the calling convention I started out with (that allocated everything in the heap lmao), but still. There's nothing fundamentally preventing a Fil-C register-based calling convention, but it would take a decent amount of work to implement.
There are probably other perf optimization opportunities that I'm either forgetting right now or that haven't been found yet. It's still early days!
This is such an interesting project.
I've always been firmly in the 'let it crash' camp for bugs, the sooner and the closer to the offending piece of code you can generate a crash the better. Maybe it would be possible to embed Fil-C in a test-suite combined with a fuzzing like tool that varies input to try really hard to get a program to trigger an abend. As long as it is possible to fuzz your way to a crash in Fil-C that would be a sign that there is more work to do.
That way 'passes Fil-C' would be a bit like running code under valgrind and move the penalty to the development phase rather than the runtime. Is this feasible or am I woolgathering, and is Fil-C only ever going to work by using it to compile the production code?
I've always been firmly in the 'let it crash' camp for bugs, the sooner and the closer to the offending piece of code you can generate a crash the better. Maybe it would be possible to embed Fil-C in a test-suite combined with a fuzzing like tool that varies input to try really hard to get a program to trigger an abend. As long as it is possible to fuzz your way to a crash in Fil-C that would be a sign that there is more work to do.
That way 'passes Fil-C' would be a bit like running code under valgrind and move the penalty to the development phase rather than the runtime. Is this feasible or am I woolgathering, and is Fil-C only ever going to work by using it to compile the production code?
From what I understand some things in Fil-C work "as expected" instead of crashing (e.g. dereferencing a pointer to an out of scope variable will give you the old value of that variable), so it won't work as a sanitizer.
You can use the built-in sanitizer from your compiler though.
At that point why use Fil-C for this though?
Because you don't want to let it crash in production? Sanitizer for testing Fil-C for shipping.
Fil-C will crash on memory corruption too. In fact, its main advantage is crashing sooner.
All the quick fixes for C that don't require code rewrites boil down to crashing. They don't make your C code less reliable, they just make the unreliability more visible.
To me, Fil-C is most suited to be used during development and testing. In production you can use other sandboxing/hardening solutions that have lower overhead, after hopefully shaking out most of the bugs with Fil-C.
All the quick fixes for C that don't require code rewrites boil down to crashing. They don't make your C code less reliable, they just make the unreliability more visible.
To me, Fil-C is most suited to be used during development and testing. In production you can use other sandboxing/hardening solutions that have lower overhead, after hopefully shaking out most of the bugs with Fil-C.
The great thing about such crashes is if you have coredumps enabled that you can just load the crashed binary into GDB and type 'where' and you most likely can immediately figure out from inspecting the call stack what the actual problem is. This was/is my go-to method to find really hard to reproduce bugs.
I think the issue with this approach is it’s perfectly reasonable in Fil-C to never call `free` because the GC will GC. So if you develop on Fil-C, you may be leaking memory if you run in production with Yolo-C.
Fil-C uses `free()` to mark memory as no longer valid, so it is important to keep using manual memory management to let Fil-C catch UAF bugs (which are likely symptoms of logic bugs, so you'd want to catch them anyway).
The whole point of Fil-C is having C compatibility. If you're going to treat it as a deployment target on its own, it's a waste: you get overhead of a GC language, but with clunkiness and tedium of C, instead of nicer language features that ground-up GC languages have.
The whole point of Fil-C is having C compatibility. If you're going to treat it as a deployment target on its own, it's a waste: you get overhead of a GC language, but with clunkiness and tedium of C, instead of nicer language features that ground-up GC languages have.
I think you're missing a bit of context from the parent comments:
> Maybe it would be possible to embed Fil-C in a test-suite
> Maybe it would be possible to embed Fil-C in a test-suite
Can you elaborate on what makes ELF (potentially with custom sections/extension and maybe custom ld.so plugin) insufficient?
A lot of remarkably unusual stuff has been shoved into the format without breaking the tooling, so wondering what the restrictions are.
A lot of remarkably unusual stuff has been shoved into the format without breaking the tooling, so wondering what the restrictions are.
> Rust and Yolo-C will always be faster
graydon points in that direction, but since you're here: how feasible is a hypothetical Fil-Unsafe-Rust? would you need to compile the whole program in Fil-Rust to get the benefits of Fil-Unsafe-Rust?
graydon points in that direction, but since you're here: how feasible is a hypothetical Fil-Unsafe-Rust? would you need to compile the whole program in Fil-Rust to get the benefits of Fil-Unsafe-Rust?
It's reasonably easy if you can treat the Safe Rust and Fil-Unsafe-Rust code as accessing different address spaces (in the C programming sense of "a broad subset of memory that a pointer is limited to", not the general OS/hardware sense), since that's essentially what the bespoke Fil-C ABI amounts to in the first place. Which of course is not really a good fit for every use of Unsafe Rust, but might suffice for some of them.
What is Fil-Rust and Fil-Unsafe-Rust?
in my mind it would be doing what fil-c does for c to unsafe rust: a hypothetical memory safe implementation of unsafe rust using the same methods fil-c does e.g. gc
Can't do GC unless you go all-in.
So that implies just running all of Rust through the Fil-C transformation
So that implies just running all of Rust through the Fil-C transformation
> Can't do GC unless you go all-in.
It can be done, especially with a safe non-GC language that can meaningfully guarantee it won't corrupt GC metadata or break its invariants. You only have real issues (and then only wrt. excess overhead, not unsoundness) with pervasive mutual references between the GC and non-GC parts of the program. You do need to promote GC pointers to a root anytime that non-GC code has direct access to them, and add finalizers to GC objects that may need to drop/run destructors on non-GC data.
It can be done, especially with a safe non-GC language that can meaningfully guarantee it won't corrupt GC metadata or break its invariants. You only have real issues (and then only wrt. excess overhead, not unsoundness) with pervasive mutual references between the GC and non-GC parts of the program. You do need to promote GC pointers to a root anytime that non-GC code has direct access to them, and add finalizers to GC objects that may need to drop/run destructors on non-GC data.
that's what I expected, thanks for making this clear!
what would fil-rust do that miri doesn't?
e.g. validate safety across safe/unsafe boundaries
Miri does do that? It is not aware of the distinction to begin with (which is one of the use cases of the tool: it lets us exercise safe code to ensure there aren't memory violations caused by incorrect MIR lowering). I might be mistaking what you mean. Miri's big limitation is not being able to interface with FFI.
hmmm I thought miri was used in the compiler for static analysis, wasn't aware it's a runtime interpreter.
I guess the primary reason would be running hardened code in production without compromising performance too much, same as you would run Fil-C compiled software instead of the usual way. I've no idea if it's feasible to run miri in prod.
I guess the primary reason would be running hardened code in production without compromising performance too much, same as you would run Fil-C compiled software instead of the usual way. I've no idea if it's feasible to run miri in prod.
I guess the confusion happens because MIR: the representation, mir: the stage, stable MIR: the potential future API interface for hooking into the compiler stage, and miri: the MIR interpreter all share pretty much the same name. Const evaluation uses MIR and that's the most likely culprit. Miri is an interpreter (as you found out on your own now), and it is not meant for use in production workloads due to the slowdown it introduces, so it limits its use to use in test suites and during debugging.
From my understanding Fil-C is an LLVM operation, so it should be possible to build integration to have a Fil-Rust binary that is slower but gives you some of the benefits of miri. I see value in doing something like that. There are plenty of other languages that would be well served by this too!
From my understanding Fil-C is an LLVM operation, so it should be possible to build integration to have a Fil-Rust binary that is slower but gives you some of the benefits of miri. I see value in doing something like that. There are plenty of other languages that would be well served by this too!
The savings of two conditional branches sounds interesting; what would the change be?
- Don’t put flags in the high bits of the aux pointer. Instead if an object has flags, it’ll have a fatter header. Most objects don’t have flags.
- Give up on lock freedom of atomic pointers. This is a fun one because theoretically, it’s worse. But it comes with a net perf improvement because there’s no need to check the low bit of lowers.
- Give up on lock freedom of atomic pointers. This is a fun one because theoretically, it’s worse. But it comes with a net perf improvement because there’s no need to check the low bit of lowers.
Scary! I'm excited to see how it turns out.
Love your Yolo-C remark. :)
So you'd have to implement binfmt_misc for the new binary format? Will you need to write your own ld.so?
Yes and yes
If you are not writing anything performance sensitive, you shouldn't be using C in the first place. Even if Fil-C greatly reduces its overhead, I can't see it ever being a good idea for actual release builds.
As a Linux user of two decades, memory safety has never been a major issues that I would be willing to trade performance for. It doesn't magically make my application work it just panics instead of crashes, same end result for me. It just makes it so the issue can not be exploited by an attacker. Which is good but like Linux has been already safe enough to be the main choice to run on servers so meh. The whole memory safety cult is weird.
I guess Fil-C could have a place in the testing pipeline. Run some integration tests on builds made with it and see if stuff panics.
That said, Fil-C is a super cool projects. I don't mean to throw any shades at it.
As a Linux user of two decades, memory safety has never been a major issues that I would be willing to trade performance for. It doesn't magically make my application work it just panics instead of crashes, same end result for me. It just makes it so the issue can not be exploited by an attacker. Which is good but like Linux has been already safe enough to be the main choice to run on servers so meh. The whole memory safety cult is weird.
I guess Fil-C could have a place in the testing pipeline. Run some integration tests on builds made with it and see if stuff panics.
That said, Fil-C is a super cool projects. I don't mean to throw any shades at it.
> If you are not writing anything performance sensitive, you shouldn't be using C in the first place.
Then why are all of the IO-bound low level pieces of Linux userland written in C?
Take just one example: udevd. I have a Fil-C version. There is zero observable difference in performance.
Then why are all of the IO-bound low level pieces of Linux userland written in C?
Take just one example: udevd. I have a Fil-C version. There is zero observable difference in performance.
udevd might actually be a good use for Fil-C. Good point.
My fear is that the performance difference might add up once use it on more and more part. I imagine it uses a lot more memory. Plus once Fil-C gets adopted in the mainstream it might lower the need for devs to actually fix the code and they might start just relying on Fil-C.
To be fair, systemd itself is corporate shite to begin with and I wouldn't mind seeing it being replaced with something written in a language with memory safety.
My fear is that the performance difference might add up once use it on more and more part. I imagine it uses a lot more memory. Plus once Fil-C gets adopted in the mainstream it might lower the need for devs to actually fix the code and they might start just relying on Fil-C.
To be fair, systemd itself is corporate shite to begin with and I wouldn't mind seeing it being replaced with something written in a language with memory safety.
> My fear is that the performance difference might add up once use it on more and more part
If that argument is valid, then why hasn't it stopped adoption of slow languages?
Like, Python is waaay slower than Fil-C. And so much of Linux userland is written in shell, which is slower still.
If that argument is valid, then why hasn't it stopped adoption of slow languages?
Like, Python is waaay slower than Fil-C. And so much of Linux userland is written in shell, which is slower still.
> it might lower the need for devs to actually fix the code and they might start just relying on Fil-C.
Well, the program would still halt upon memory flaw, so there would still be a need to fix it
Well, the program would still halt upon memory flaw, so there would still be a need to fix it
People with Linux servers keep getting hacked so idk if I buy the argument “if it’s in use it’s good enough”. That’s like saying “everyone else runs Pentium 2, why would I upgrade to Pentium 3?”
While memory safety can help reduce many security vulnerabilities it is not the only source of vulnerabilities. Furthermore as for getting hacked I would suspect the main problems to be social engineering, bad configuration and lack of maintenance and not really the software itself being insecure.
> That’s like saying “everyone else runs Pentium 2, why would I upgrade to Pentium 3?”
No one should blindly upgrade because bigger number is better. If I look into new hardware I research benchmarks and figure out if it would enable me to (better) run the software/games I care about it and if the improvement is worth my money.
Same with security. You need to read actual studies and figure out what the cost/benefit of certain measures is.
There are safer alternatives to Linux but apparently the situation isn't bad enough for people to switch to them.
And I am not saying you should create new projects in C or C++. Most people should not. But there is a lot of battle tested C and C++ code out there and to act as if we suddenly have this big problem with memory safety is a weird narrative to push. And if you discover a vulnerability, well fix it instead of wrapping it Fil-C and making the whole thing slower.
> That’s like saying “everyone else runs Pentium 2, why would I upgrade to Pentium 3?”
No one should blindly upgrade because bigger number is better. If I look into new hardware I research benchmarks and figure out if it would enable me to (better) run the software/games I care about it and if the improvement is worth my money.
Same with security. You need to read actual studies and figure out what the cost/benefit of certain measures is.
There are safer alternatives to Linux but apparently the situation isn't bad enough for people to switch to them.
And I am not saying you should create new projects in C or C++. Most people should not. But there is a lot of battle tested C and C++ code out there and to act as if we suddenly have this big problem with memory safety is a weird narrative to push. And if you discover a vulnerability, well fix it instead of wrapping it Fil-C and making the whole thing slower.
403's until you go to https://graydon2.dreamwidth.org/ first
I only got a captcha prompt on the direct link. Perhaps you have something that disables the captcha, and thus got 403'd?
403 until you go to https://graydon2.dreamwidth.org/ first with JavaScript enabled and temporarily allow third-party scripts from awswaf.com.
Getting a "not available in your state" page, does anyone have an archive? I've only recently tried out fil-c and hope to use it in some work projects.
I can see the value this brings vs regular C, but I’m less clear on what this brings on top of -fbounds-safety
-fbounds-safety is awesome!
Here’s what Fil-C gives you that -fbounds-safety doesn’t:
- Fil-C gives you comprehensive memory safety while -fbounds-safety just covers bounds. For example, Fil-C panics on use after free and has well defined semantics on ptr-int type confusion.
- -fbounds-safety requires you to modify your code. Fil-C makes unmodified C/C++ code memory safe.
FWIW, I worked on -fbounds-safety and I still think it’s a good idea. :-)
Here’s what Fil-C gives you that -fbounds-safety doesn’t:
- Fil-C gives you comprehensive memory safety while -fbounds-safety just covers bounds. For example, Fil-C panics on use after free and has well defined semantics on ptr-int type confusion.
- -fbounds-safety requires you to modify your code. Fil-C makes unmodified C/C++ code memory safe.
FWIW, I worked on -fbounds-safety and I still think it’s a good idea. :-)
It's always seemed obvious to me that it would be better to make C safer than it would be to rewrite the billions of lines of C that run all our digital infrastructure. Of course that will get pushback from people who care more about rewriting it in a specific language, but pragmatically it's the obvious solution. Nice to see stuff like Fil-C proving it's possible, and if the performance gap can get within 10% (which seems very possible) it would be a no-brainer.
It depends how much the C software is "done" vs being updated and extended. Some legacy projects need a rewrite/rearchitecting anyway (even well-written battle-tested code may stop meeting requirements simply due to the world changing around it).
It also doesn't have to be a complete all-at-once rewrite. Plain C can easily co-exist with other languages, and you can gradually replace it by only writing new code in another language.
It also doesn't have to be a complete all-at-once rewrite. Plain C can easily co-exist with other languages, and you can gradually replace it by only writing new code in another language.
Memory safety isn't the only benefit of rewriting C code in Rust. IMO it's maybe not even the biggest.
For example you also get a far stronger type system (leading to fewer logic bugs) and modern tooling.
For example you also get a far stronger type system (leading to fewer logic bugs) and modern tooling.
I am very excited about this. Thanks to HN I see such things. To bad normal media is not any longer interested in anything not AI. On Topic: I am quite sceptical about all this rusting only because we can. Going rust makes the amount of programmers willing to look at the code quite small. A way to add this static testing to c will on the other hand open up the whole c community to a needed thing: memory
As article points out, this does not solve all the things Rust does (apart for memory/performance, things like point 3.). So new code would be preferable in something like Rust (and some other PLs). However lot of existing code is in C and most of it will stay in C. So Fil-C seems to be really useful here.
Garbage collection is GOFAI (the LISP folks came up with it) and of course GOFAI is AI.
wakawaka28(5)
There's something off-putting about how Rust people communicate. Like you're talking to people selling something, or people with ulterior motives. They love talikng about memory safety as that's the only thing standing between humanity having secure and reliable software. They love external audiences, as their brand of fearmongering, virtue signaling and offering of panacea-like solutions resonates with a certain kind of risk-averse decision maker, who tend to in power in uncomfortable numbers.
First of all, Rust's 'fearless concurrency' largely boils down to 'no concurrency' - Rust has about as much concurrency as Javascript - you can copy objects between threads, but not share memory beyond that, with certain libraries allowing some escape hatches in Rust's case.
Additionally the case for aliasing control leading to better and safer code that's easier to reason about just isn't really true in practice - it's so rare that, for example your function can accidentally can alias memory in strictly typed languages - and when it does it's kind of intentional. The concern pretty much only manifests in memcpy, as the ugly hack of 'strict aliasing' - assuming 2 pointers of incompatible types pointing to different bits of memory - works very well in practice.
It even helps with situations people complain about in Rust, like when object A and B (both mutably borrowed) take a reference to a some common service object (which is super common) - but that sort of code simply doesn't compile in Rust.
All in all, I don't dislike Rust as it is but the project's tendency to do activism trying to bully its technical skeptics into submission (which is unfortunately what all activism really is - when you lost the argument, try to louder than the other guy and paint him as reprehensible) - they focused on fixing the technical issues. There has been research into ownership schemes, some exist that are less restrictive than Rust's while offering the same safety guarantees.
In my personal opinion Rust is not done on the conceptual level - by 'done' I mean Rust serving its purpose as the language it claims to be. Maybe there will be Rust 2.0 which will overhaul the ownership system completely or maybe there'll be another language that will do what Rust does but better.
Edit: I wish I could claim I'm some sort of tin-foil conspiracy theorist, but I'm commenting under an article written by one of the key people behind Rust, and it reeks of this attitude.
First of all, Rust's 'fearless concurrency' largely boils down to 'no concurrency' - Rust has about as much concurrency as Javascript - you can copy objects between threads, but not share memory beyond that, with certain libraries allowing some escape hatches in Rust's case.
Additionally the case for aliasing control leading to better and safer code that's easier to reason about just isn't really true in practice - it's so rare that, for example your function can accidentally can alias memory in strictly typed languages - and when it does it's kind of intentional. The concern pretty much only manifests in memcpy, as the ugly hack of 'strict aliasing' - assuming 2 pointers of incompatible types pointing to different bits of memory - works very well in practice.
It even helps with situations people complain about in Rust, like when object A and B (both mutably borrowed) take a reference to a some common service object (which is super common) - but that sort of code simply doesn't compile in Rust.
All in all, I don't dislike Rust as it is but the project's tendency to do activism trying to bully its technical skeptics into submission (which is unfortunately what all activism really is - when you lost the argument, try to louder than the other guy and paint him as reprehensible) - they focused on fixing the technical issues. There has been research into ownership schemes, some exist that are less restrictive than Rust's while offering the same safety guarantees.
In my personal opinion Rust is not done on the conceptual level - by 'done' I mean Rust serving its purpose as the language it claims to be. Maybe there will be Rust 2.0 which will overhaul the ownership system completely or maybe there'll be another language that will do what Rust does but better.
Edit: I wish I could claim I'm some sort of tin-foil conspiracy theorist, but I'm commenting under an article written by one of the key people behind Rust, and it reeks of this attitude.
> First of all, Rust's 'fearless concurrency' largely boils down to 'no concurrency' - Rust has about as much concurrency as Javascript - you can copy objects between threads, but not share memory beyond that
...this is just blatantly false? Like it is false to the extent that I am confused as to what you could even possibly be talking about - I don't know how anyone who has actually written non-trivial programs in both languages could come to the conclusion that they have the same memory model (or much of anything in common when it comes to threads, really).
...this is just blatantly false? Like it is false to the extent that I am confused as to what you could even possibly be talking about - I don't know how anyone who has actually written non-trivial programs in both languages could come to the conclusion that they have the same memory model (or much of anything in common when it comes to threads, really).
Everything but the most rudimentary threading architectures in Rust programs basically mean that you have to design your whole program around it - annotating everything with Send and Sync, which might be sometimes impossible or impractical because you don't control the modified code, using Arc<> etc.
In practice Rust doesn't make concurrency easier - it forces extremely conservative architectures or basically constantly copying everything between threads (hence my example of Js workers).
Sure it's safe, but it's the nanny kind of safety, where you are basically disallowed to do something remotely challenging or interesting, instead of the bungee cord kind of safety where the framework allows you to go crazy and if you obey a few simple rules you can make sure your code will still work.
People who do not work on nuclear reactors don't seek out this kind of safety by themselves - it is forced on them.
The other funny thing about the Rust memory model is that it kind of assumes an environment where there's a malloc/free function pair which is stateless and global, and every allocation needs to be freed.
This is absolutely not the case in low-level code, with memory pools, arenas etc. In these cases Rust either enforces unnecessary restrictions or can't really keep track of the lifetime of objects.
Rust is seriously undercooked.
In practice Rust doesn't make concurrency easier - it forces extremely conservative architectures or basically constantly copying everything between threads (hence my example of Js workers).
Sure it's safe, but it's the nanny kind of safety, where you are basically disallowed to do something remotely challenging or interesting, instead of the bungee cord kind of safety where the framework allows you to go crazy and if you obey a few simple rules you can make sure your code will still work.
People who do not work on nuclear reactors don't seek out this kind of safety by themselves - it is forced on them.
The other funny thing about the Rust memory model is that it kind of assumes an environment where there's a malloc/free function pair which is stateless and global, and every allocation needs to be freed.
This is absolutely not the case in low-level code, with memory pools, arenas etc. In these cases Rust either enforces unnecessary restrictions or can't really keep track of the lifetime of objects.
Rust is seriously undercooked.
I could argue on technical aspects, but firstly, how do you square what you're saying with Rust code used in production codebases? Do they not do multithreading, or shared memory, or complex allocation patterns? Sure, Rust is not a silver bullet (nothing is), and it has many pain points, but the picture you're presenting is entirely incongruous with the reality of programs written in Rust.
Ah, I see. So by "you can copy objects between threads, but not share memory beyond that" you actually meant "you can obviously share memory beyond copying objects between threads using primitives and constructs that the language gives you but I find them bad for vaguely specified reasons so I don't want to use them and this is somehow the same thing as 'you cannot share memory beyond that'".
This is of course not disingenuous in the slightest (especially the part where you bring up "using Arc", i.e. the equivalent of e.g. C++'s shared_ptr, as an example of the sort of "extremely conservative architecture" you are forced to use when writing multithreaded Rust). Anyone who deems it so is obviously just a Rust fanatic, probably. I assume mutexes are also a Rust invention to make concurrency difficult.
I've never quite understood the complaints about "the nanny kind of safety", either. To put it very bluntly, they just come across to me as wanting to be nannied. I am supposed to entertain the concept of a developer so competent with ideas so clever that the rules of safe Rust simply cannot express them, and yet so nervous and unsure of their own brilliance that they cannot just...write out their case in unsafe Rust and move on. It's as if people want to be smarter than the compiler and yet also never actually have to demonstrate that they are, and it just doesn't make sense.
This is of course not disingenuous in the slightest (especially the part where you bring up "using Arc", i.e. the equivalent of e.g. C++'s shared_ptr, as an example of the sort of "extremely conservative architecture" you are forced to use when writing multithreaded Rust). Anyone who deems it so is obviously just a Rust fanatic, probably. I assume mutexes are also a Rust invention to make concurrency difficult.
I've never quite understood the complaints about "the nanny kind of safety", either. To put it very bluntly, they just come across to me as wanting to be nannied. I am supposed to entertain the concept of a developer so competent with ideas so clever that the rules of safe Rust simply cannot express them, and yet so nervous and unsure of their own brilliance that they cannot just...write out their case in unsafe Rust and move on. It's as if people want to be smarter than the compiler and yet also never actually have to demonstrate that they are, and it just doesn't make sense.
That's a rather mean spirited interpretation of my comment and I debated whether to respond to it, but I guess here goes:
In general Rust claims to provide you with a set of safety rules that are a more comprehensive extension of C++ unique_ptr and co. so you can do write the same code, have like 10% more mental overhead because of additional safety, but in exchange you can fearlessly do anything, and know your program doesn't have memory leaks or race conditions.
However that's not how it works in practice. Rust rejects all non-trivial naive multi-threaded programs. There's a zero percent chance that you can map a non-trivial C++ program to Rust without major rewrites, even though said program is 100% safe from aliasing bugs.
How multi-threading actually works in Rust:
You copy everything between threads, or try to adapt every structure that needs to be threaded with Sync, Send, Arc etc. Not very efficient use of ones time.
Or you rely on one of the threading libraries/frameworks which makes certain styles of parallelism easier to express. These libraries usually contain giant blobs of unsafe Rust, which isn't exactly a win for the language.
The TLDR version of my post is Rust still rejects a significant percentage of non-trivial but correct programs. If we were anal about it and cared about the soundness of the language, and disallowed libraries with unsafe, that number would jump even higher.
In general Rust claims to provide you with a set of safety rules that are a more comprehensive extension of C++ unique_ptr and co. so you can do write the same code, have like 10% more mental overhead because of additional safety, but in exchange you can fearlessly do anything, and know your program doesn't have memory leaks or race conditions.
However that's not how it works in practice. Rust rejects all non-trivial naive multi-threaded programs. There's a zero percent chance that you can map a non-trivial C++ program to Rust without major rewrites, even though said program is 100% safe from aliasing bugs.
How multi-threading actually works in Rust:
You copy everything between threads, or try to adapt every structure that needs to be threaded with Sync, Send, Arc etc. Not very efficient use of ones time.
Or you rely on one of the threading libraries/frameworks which makes certain styles of parallelism easier to express. These libraries usually contain giant blobs of unsafe Rust, which isn't exactly a win for the language.
The TLDR version of my post is Rust still rejects a significant percentage of non-trivial but correct programs. If we were anal about it and cared about the soundness of the language, and disallowed libraries with unsafe, that number would jump even higher.
For mean spirited I would consider
> There's something off-putting about how Rust people communicate. Like you're talking to people selling something, or people with ulterior motives.
Which is a radically wrong description of the tone of the OP that is purportedly the reason for the comment. And for the most impressive bit of un-self-reflective projection I've seen in a long time, there's
> And the way you write is extremely rude and condescending, if you do decide to phrase your comments in such a demeaning way (which I don't ever recommend), please at least be right next time.
> There's something off-putting about how Rust people communicate. Like you're talking to people selling something, or people with ulterior motives.
Which is a radically wrong description of the tone of the OP that is purportedly the reason for the comment. And for the most impressive bit of un-self-reflective projection I've seen in a long time, there's
> And the way you write is extremely rude and condescending, if you do decide to phrase your comments in such a demeaning way (which I don't ever recommend), please at least be right next time.
Do you use Rust in a professional capacity? Because I do, and I do because I chose it for a project over C++. Rust is much nicer in certain ways than C++ (like cargo or the whole ecosystem of packages), but it's core idea and assumption (which imo can equivalently stated as 'the borrow checker' or 'exact aliasing control') is flawed as it is, and programmers of all skill levels struggle with it for various reasons when writing programs of real-world complexity.
There are 3 kinds of ways people percieve Rust - those who see it as a cool piece of tech (which I agree with), those who see it as a practical tool, and those who see liking or not liking Rust as an ideological statement. I see it as a tool.
There's a lot of statements around Rust that are true in the strict factual sense, but clearly meant to be read differently.
'Free beer at the pub tonight' is a true statement even though the full truth would be 'Free beer at the bar tonight (entry fee $200)'.
'Fearless concurrency' is among these statements meant to be interpreted as making concurrent programming easier. The true statement would be 'fearless concurrency (but you might need to rewrite your program)'.
There are 3 kinds of ways people percieve Rust - those who see it as a cool piece of tech (which I agree with), those who see it as a practical tool, and those who see liking or not liking Rust as an ideological statement. I see it as a tool.
There's a lot of statements around Rust that are true in the strict factual sense, but clearly meant to be read differently.
'Free beer at the pub tonight' is a true statement even though the full truth would be 'Free beer at the bar tonight (entry fee $200)'.
'Fearless concurrency' is among these statements meant to be interpreted as making concurrent programming easier. The true statement would be 'fearless concurrency (but you might need to rewrite your program)'.
> And for the most impressive bit of un-self-reflective projection
bit shitty of you given youre having him over his tone, no?
bit shitty of you given youre having him over his tone, no?
-6 karma says what?
You have no understanding of the context of this old conversation.
Goodbye.
You have no understanding of the context of this old conversation.
Goodbye.
"Up" to -15 now ... and of course the "unreflective self projection" claim is utter nonsense.
thats a bit of unreflective self projection there, mate
[deleted]
You're exaggerating heavily, which undermines your claims. While Rust claims to allow fearless concurrency, there is plenty of official documentation to show that the limitations are acknowledged. Anyone telling you Rust prevents memory leaks and race conditions is lying, straight up, and this is trivial to verify. You shouldn't be repeating it. Your comments on Send/Sync and unsafe are similarly exaggerated. I have to say, you must be interacting with Rust zealots beyond what I've ever seen, if this is truly your belief.
?? Now I have to come to the defense of Rust - it does prevent memory leaks (but not space leaks where you leak memory, but it's all being referenced from somewhere). The borrow checker should also prevent race conditions (as not doing so would violate the aliasing rule)
This is all provided we stay in safe Rust land, and don't do concurrency or call external APIs, but that's not a claim I'm holding against the language.
This isn't the point I'm making - the issue I have is that the Rust borrow checker is not powerful enough to accept most correct programs (of real-world complexity) for various reasons. This does not mean that you can't write complex programs in Rust, it only means that most complex programs which do not have the problems Rust claims to address (successfully), have basically zero chance compiling, unleast they're written the way Rust expects them to.
This is all provided we stay in safe Rust land, and don't do concurrency or call external APIs, but that's not a claim I'm holding against the language.
This isn't the point I'm making - the issue I have is that the Rust borrow checker is not powerful enough to accept most correct programs (of real-world complexity) for various reasons. This does not mean that you can't write complex programs in Rust, it only means that most complex programs which do not have the problems Rust claims to address (successfully), have basically zero chance compiling, unleast they're written the way Rust expects them to.
> Now I have to come to the defense of Rust - it does prevent memory leaks (but not space leaks where you leak memory, but it's all being referenced from somewhere). The borrow checker should also prevent race conditions (as not doing so would violate the aliasing rule)
Rc<> and Arc<> can create true memory leaks where a cycle of Rc<> or Arc<> referencing one another can stay allocated in memory even when all outside references to the objects have disappeared, so the leak cannot even be collected. This is the one thing that is not allowed to occur with a tracing garbage collector as in Fil-C. (Rust also allows leaking memory explicitly, e.g. for the purpose of passing a reference to that memory to foreign code where Rust cannot observe its lifecycle. Then in order to properly drop that memory in Rust, you have to essentially recreate that allocation from scratch.)
Race conditions are a logical error that has nothing to do with the borrow checker. The borrow checker can only prevent properly defined data races, which are entirely unrelated to logical race conditions.
Rc<> and Arc<> can create true memory leaks where a cycle of Rc<> or Arc<> referencing one another can stay allocated in memory even when all outside references to the objects have disappeared, so the leak cannot even be collected. This is the one thing that is not allowed to occur with a tracing garbage collector as in Fil-C. (Rust also allows leaking memory explicitly, e.g. for the purpose of passing a reference to that memory to foreign code where Rust cannot observe its lifecycle. Then in order to properly drop that memory in Rust, you have to essentially recreate that allocation from scratch.)
Race conditions are a logical error that has nothing to do with the borrow checker. The borrow checker can only prevent properly defined data races, which are entirely unrelated to logical race conditions.
You are right, you can absolutely cause memory leaks with Rc, I haven't considered that. However that kind of raises an interesting point - Rc<T>::clone clearly mutates something under the hood as the reference counter is shared, however its not mut. Having something in the core language that breaks the rules and causes actual problems in practice is kind of an ill omen.
And I meant data races, I have stated in my previous post, that race conditions due to issues existing outside of the language, like external libraries or network requests are not the fault of Rust. Neither are logic mistakes a programmer makes.
And I meant data races, I have stated in my previous post, that race conditions due to issues existing outside of the language, like external libraries or network requests are not the fault of Rust. Neither are logic mistakes a programmer makes.
Interior mutability is explicitly provided for in the language, and consistently marked under the hood via UnsafeCell<>. Thus, the compiler can always check whether interior mutability is possible, including wrt. reference counts. In practice, the &mut modifier has more to do with the absence of aliasing (much like *restrict in C) than mutability itself.
Again to be maybe overly blunt, it's rather strange to call my comment mean-spirited and then simply repeat the exact argument I called out as being wishy-washy.
The crux of what you're saying seems to be
> You copy everything between threads, or try to adapt every structure that needs to be threaded with Sync, Send, Arc etc. Not very efficient use of ones time.
You have not actually articulated any real problem with "Sync, Send, Arc, etc" beyond what just sounds like "but I don't wanna". Even leaving aside Sync and Send, which are constructs fairly unique to Rust, the fact that your other example of a construct you have to "try to adapt" your multithreaded code to use is Arc - quite literally the equivalent of std::shared_ptr - on its own casts heavy doubt on the seriousness of your complaint. Like, "oh no! I have to use an atomic reference counted smart pointer in my multithreaded program" is quite simply unserious as a complaint about a programming language.
How exactly you think pointers should be shared between threads, then? Just passing raw pointers around, and taking responsibility any use-after-free bugs that arise because you know what you're doing and definitely aren't going to write one? Well guess what, you can do that in Rust, too (and I have done that in a few non-trivial projects), but you have to mark that obviously unsafe operation with `unsafe`. For some reason, this six-letter keyword is apparently enough to get people who allegedly know what they're doing to break out in hives, and I just do not understand it.
And coming back to Send and Sync, if you understand what those traits are (versus just seeing them as something to slap on code to make the compiler stop screaming), then I'm sure you know that they are automatically derived traits. If you are having to explicitly implement them, it is because something in your structure is dubiously thread-safe - for instance, a raw pointer or a type that has been marked as thread-local. If you are that confident that you are handling whatever it is in a thread-safe manner, then what exactly is the big adaptation in the one-line implementation (each) of Send and Sync?
If anything, this whole "I have to write new, explicit syntax expressing my intentions and this is The Worst and the same thing as not being to do anything at all" business just reminds me of dynamic typing diehards' complaints about static typing; just replace "using Sync, Send, Arc, etc" with "writing out type annotations". They also complain about compilers rejecting "non-trivial but correct programs" for the minor sin of not being properly annotated.
> These libraries usually contain giant blobs of unsafe Rust, which isn't exactly a win for the language.
You simply cannot put up this cowboy coder "I don't want to be nannied" argument and then start calling for a nanny because the keyword for "I have information that the compiler doesn't, so I'm taking the reins here" is `unsafe`. It is profoundly unserious.
Same thing with this supposed overwhelming complexity of writing regular/safe multithreaded Rust, it really just sounds like wanting the language to pretend that multithreaded execution isn't complex and just handle everything about it for you as you write code as if it is single-threaded and have it just work, without having to think about the semantics. If you don't want to be nannied in that manner, then I don't understand how one can be so dissatisfied with that the language actually does (i.e. provides "certified safe - terms and conditions apply" constructs to express semantics the user is supposed to understand) that they resort to just copying everything instead and genuinely hold the opinion that the language is no different from JavaScript in that regard (a language that notoriously doesn't even have actual threads). And the comparison is plain dishonest no matter how you spin it.
The crux of what you're saying seems to be
> You copy everything between threads, or try to adapt every structure that needs to be threaded with Sync, Send, Arc etc. Not very efficient use of ones time.
You have not actually articulated any real problem with "Sync, Send, Arc, etc" beyond what just sounds like "but I don't wanna". Even leaving aside Sync and Send, which are constructs fairly unique to Rust, the fact that your other example of a construct you have to "try to adapt" your multithreaded code to use is Arc - quite literally the equivalent of std::shared_ptr - on its own casts heavy doubt on the seriousness of your complaint. Like, "oh no! I have to use an atomic reference counted smart pointer in my multithreaded program" is quite simply unserious as a complaint about a programming language.
How exactly you think pointers should be shared between threads, then? Just passing raw pointers around, and taking responsibility any use-after-free bugs that arise because you know what you're doing and definitely aren't going to write one? Well guess what, you can do that in Rust, too (and I have done that in a few non-trivial projects), but you have to mark that obviously unsafe operation with `unsafe`. For some reason, this six-letter keyword is apparently enough to get people who allegedly know what they're doing to break out in hives, and I just do not understand it.
And coming back to Send and Sync, if you understand what those traits are (versus just seeing them as something to slap on code to make the compiler stop screaming), then I'm sure you know that they are automatically derived traits. If you are having to explicitly implement them, it is because something in your structure is dubiously thread-safe - for instance, a raw pointer or a type that has been marked as thread-local. If you are that confident that you are handling whatever it is in a thread-safe manner, then what exactly is the big adaptation in the one-line implementation (each) of Send and Sync?
If anything, this whole "I have to write new, explicit syntax expressing my intentions and this is The Worst and the same thing as not being to do anything at all" business just reminds me of dynamic typing diehards' complaints about static typing; just replace "using Sync, Send, Arc, etc" with "writing out type annotations". They also complain about compilers rejecting "non-trivial but correct programs" for the minor sin of not being properly annotated.
> These libraries usually contain giant blobs of unsafe Rust, which isn't exactly a win for the language.
You simply cannot put up this cowboy coder "I don't want to be nannied" argument and then start calling for a nanny because the keyword for "I have information that the compiler doesn't, so I'm taking the reins here" is `unsafe`. It is profoundly unserious.
Same thing with this supposed overwhelming complexity of writing regular/safe multithreaded Rust, it really just sounds like wanting the language to pretend that multithreaded execution isn't complex and just handle everything about it for you as you write code as if it is single-threaded and have it just work, without having to think about the semantics. If you don't want to be nannied in that manner, then I don't understand how one can be so dissatisfied with that the language actually does (i.e. provides "certified safe - terms and conditions apply" constructs to express semantics the user is supposed to understand) that they resort to just copying everything instead and genuinely hold the opinion that the language is no different from JavaScript in that regard (a language that notoriously doesn't even have actual threads). And the comparison is plain dishonest no matter how you spin it.
Ill be extremely terse. Real world world programs usually have states where you can reach a large amount of objects. This runs into problems since you can borrow too much, and the borrow checker will push you into a corner. The solutions Rust uses to deal with this are Rc, Cell, RefCell which turn compile time borrow checks into runtime ones. However using any of these (either by you or a library you use as a dependency) disqualifies your code from being multithreaded (other than the Js worker way of running multiple copies of it on separate threads).
Anyone who has written nontrival applications in Rust has encountered this.
This is an issue as threading a program will require a huge rewrite. And I concede the point, multithreading is difficult - but Rust doesn't make it easier, it merely disallows most correct programs and forces you to write code in a very particular way.
Thus fearless concurrency is a disingenious statement I don't fear concurrency, I fear the borrow checker.
As for the cowboy/nanny argument I think this is not a good way to represent safety. Freedom comes from safety - for example, you can make pretty much anything nowadays in HTML/JS without thinking about limitations, and the end user can still be confident that your website won't compromise his system.
Nannying is when you are constantly being told what you cannot do. It might not even ensure safety, but definitely restricts your freedom.
And the way you write is extremely rude and condescending, if you do decide to phrase your comments in such a demeaning way (which I don't ever recommend), please at least be right next time.
Anyone who has written nontrival applications in Rust has encountered this.
This is an issue as threading a program will require a huge rewrite. And I concede the point, multithreading is difficult - but Rust doesn't make it easier, it merely disallows most correct programs and forces you to write code in a very particular way.
Thus fearless concurrency is a disingenious statement I don't fear concurrency, I fear the borrow checker.
As for the cowboy/nanny argument I think this is not a good way to represent safety. Freedom comes from safety - for example, you can make pretty much anything nowadays in HTML/JS without thinking about limitations, and the end user can still be confident that your website won't compromise his system.
Nannying is when you are constantly being told what you cannot do. It might not even ensure safety, but definitely restricts your freedom.
And the way you write is extremely rude and condescending, if you do decide to phrase your comments in such a demeaning way (which I don't ever recommend), please at least be right next time.
The very first line of the documentation of Rc (https://doc.rust-lang.org/std/rc/index.html) describes it as a single-threaded reference counting pointer. That alone should be enough indication to any dev that can actually be trusted with multithreading that it isn't going to be fit for purpose for sharing data between threads, whether the compiler yells at you about it or not. They should not need it explained to them that a program that shares non-atomically reference counted pointers between threads (while relying on their reference counting behaviour) is not a "correct program".
What actually happens in practice, as you've alluded to, is that people who simply want to continue writing software the way they always have (but in Rust, whether due to FOMO or necessity) hear "use Rc/RefCell/Cell if the compiler is yelling at you" and then start slapping those constructs everywhere to avoid having to read too many of those pesky compiler errors, without ever considering or understanding what they actually are. And then when an architectural change is needed - in this case, rewriting a single-threaded program to be multithreaded - suddenly it is also the compiler's fault because the original program was not written with a single thought towards multithreading.
Whether you are using a compiler that yells at you about it or not, properly threading any single-threaded program that was similarly written without concern for parallelism (say, a C or C++ program that also liberally used non-thread-safe reference counted pointers) would take a non-trivial rewrite. For some reason, the fact that Rust - a language that explicitly sells itself on catching such problems at compile time - actively surfaces the poor architecture as errors as opposed to leaving the user to hunt them down themselves is supposed to be considered Bad™.
The funny thing is that there actually are real examples of problems with the borrow checker which need to be fixed by the language team (for example async closures and async Fn traits are very much a work in progress). But "I used explicitly thread-unsafe constructs and now I have to refactor my code to run properly on multiple threads" is simply not one of those problems no matter how you spin it.
> Nannying is when you are constantly being told what you cannot do. It might not even ensure safety, but definitely restricts your freedom.
Nannying is also literally the job of looking after a child [that isn't yours], which is a definition that in my opinion applies with users who don't want to learn how to use a tool as prescribed (this part makes perfect sense to me and is completely fair, nobody HAS to use Rust or any other language or tool regardless of what its diehard fans think), yet also want it to protect them from their own mistakes and lack of foresight when they make a mess of using it (this part makes no sense to me).
To put it very bluntly (or rudely, or however else you want to read it): *all* languages (natural and programming alike) have constraints, that is what makes them languages. It's perfectly fine for the conversation to begin and end at "I don't like Rust's borrow checking rules". Sure, then don't use Rust! There are several languages that I simply don't use because I don't like something or other about them, even things as trivial as significant whitespace in Python. I don't want to deal with it so I simply...don't. But approaching a language in oppositional defiance and then complaining that you got the fight you were looking for makes little sense to me.
What actually happens in practice, as you've alluded to, is that people who simply want to continue writing software the way they always have (but in Rust, whether due to FOMO or necessity) hear "use Rc/RefCell/Cell if the compiler is yelling at you" and then start slapping those constructs everywhere to avoid having to read too many of those pesky compiler errors, without ever considering or understanding what they actually are. And then when an architectural change is needed - in this case, rewriting a single-threaded program to be multithreaded - suddenly it is also the compiler's fault because the original program was not written with a single thought towards multithreading.
Whether you are using a compiler that yells at you about it or not, properly threading any single-threaded program that was similarly written without concern for parallelism (say, a C or C++ program that also liberally used non-thread-safe reference counted pointers) would take a non-trivial rewrite. For some reason, the fact that Rust - a language that explicitly sells itself on catching such problems at compile time - actively surfaces the poor architecture as errors as opposed to leaving the user to hunt them down themselves is supposed to be considered Bad™.
The funny thing is that there actually are real examples of problems with the borrow checker which need to be fixed by the language team (for example async closures and async Fn traits are very much a work in progress). But "I used explicitly thread-unsafe constructs and now I have to refactor my code to run properly on multiple threads" is simply not one of those problems no matter how you spin it.
> Nannying is when you are constantly being told what you cannot do. It might not even ensure safety, but definitely restricts your freedom.
Nannying is also literally the job of looking after a child [that isn't yours], which is a definition that in my opinion applies with users who don't want to learn how to use a tool as prescribed (this part makes perfect sense to me and is completely fair, nobody HAS to use Rust or any other language or tool regardless of what its diehard fans think), yet also want it to protect them from their own mistakes and lack of foresight when they make a mess of using it (this part makes no sense to me).
To put it very bluntly (or rudely, or however else you want to read it): *all* languages (natural and programming alike) have constraints, that is what makes them languages. It's perfectly fine for the conversation to begin and end at "I don't like Rust's borrow checking rules". Sure, then don't use Rust! There are several languages that I simply don't use because I don't like something or other about them, even things as trivial as significant whitespace in Python. I don't want to deal with it so I simply...don't. But approaching a language in oppositional defiance and then complaining that you got the fight you were looking for makes little sense to me.
THERE IS NO ALTERNATIVE TO ADOPTION OF THOUGHTFUL DESING AND PRINCIPLES.
Downvotes to my comment proves Truth is offendig most of the time :)
cheers, anyway
cheers, anyway