Rust in Large Organizations(gist.github.com)
gist.github.com
Rust in Large Organizations
https://gist.github.com/rylev/0e3c3895dcb40b6a1c1cf8c427c01b5e
50 comments
It's interesting how many organizations had problems with build.rs. Other build systems like Make, scons, or autotools have the ability to run arbitrary things in any step in the process compared to Rust which has a clear separation.
There is a proposed RFC to reduce the need for build.rs scripts in many cases [1], will likely be merged within the next 2 weeks.
[1]: https://github.com/rust-lang/rfcs/pull/2523
There is a proposed RFC to reduce the need for build.rs scripts in many cases [1], will likely be merged within the next 2 weeks.
[1]: https://github.com/rust-lang/rfcs/pull/2523
If people are worried about build.rs files running arbitrary things, you can isolate cargo inside docker, using a project I recently released: https://gitlab.com/mikecardwell/saferrust
For code-gen, I'm experimenting with checking in the results and having a "is the code-gen correct" check in the CI
See https://github.com/crate-ci/imperative/pull/4 for an example
I hadn't even considered the benefit of reducing build.rs usage. I was focused on speeding up people's builds by reducing the number of dependencies involved and making my crate do less work when being built.
Tempted to turn this pattern into a crate to reduce the hump to overcome to adopt it.
See https://github.com/crate-ci/imperative/pull/4 for an example
I hadn't even considered the benefit of reducing build.rs usage. I was focused on speeding up people's builds by reducing the number of dependencies involved and making my crate do less work when being built.
Tempted to turn this pattern into a crate to reduce the hump to overcome to adopt it.
> It's interesting how many organizations had problems with build.rs. Other build systems like Make, scons, or autotools have the ability to run arbitrary things in any step in the process compared to Rust which has a clear separation.
I think the history behind this is that Cargo was built with only Rust in mind; when your world is 100% Rust, you generally don't need much extensibility in your build process. Make, scons, and autotools all seem like they were designed to be more language agnostic from the start. It'll be interesting to see how cargo needs to evolve to be able to support interior with other ecosystems at the level that larger companies need.
I think the history behind this is that Cargo was built with only Rust in mind; when your world is 100% Rust, you generally don't need much extensibility in your build process. Make, scons, and autotools all seem like they were designed to be more language agnostic from the start. It'll be interesting to see how cargo needs to evolve to be able to support interior with other ecosystems at the level that larger companies need.
The way I heard it was that Cargo was intentionally avoiding being a complete build system. The idea being that Make et al would still be used when necessary so build.rs (and cargo in general) could be free to concentrate on managing the Rust parts exclusively.
For a modern make, I recommend checking out https://bazel.build/ (or its offspring Buck and Pants).
It's mentioned in the writeup.
It's mentioned in the writeup.
[deleted]
Glad to see some thinking going on in this area. We're using Rust at FullStory and it's about 1/3 source-wise for one of our projects. A few of these concerns echo conversations that teammates have had on Slack.
FullStory itself is somewhat polyglot (Go on the server, TypeScript/react in the browser, Java/ObjJ/Rust on native mobile) and we've felt some of the pain around having a consistent and reliable build environment. We've also had internal discussions along these lines - ie: which deps can we trust from various package managers (npm has interesting arbitrary code execution as well!).
Building tooling to manage all these languages in a common system is a _huge_ commitment. We're actively hiring for the role of productivity engineers which is something that I see becoming more important in orgs as they scale up, the same way that devops/SRE split out in the 2000s.
FullStory itself is somewhat polyglot (Go on the server, TypeScript/react in the browser, Java/ObjJ/Rust on native mobile) and we've felt some of the pain around having a consistent and reliable build environment. We've also had internal discussions along these lines - ie: which deps can we trust from various package managers (npm has interesting arbitrary code execution as well!).
Building tooling to manage all these languages in a common system is a _huge_ commitment. We're actively hiring for the role of productivity engineers which is something that I see becoming more important in orgs as they scale up, the same way that devops/SRE split out in the 2000s.
Super interesting read.
The geometric growth of deps problem seems like a natural consequence of having a good package manager. In languages like C/C++, libraries often include all their code - there are no transitive dependencies.
My reading of that isn't that the problem is the large set of transitive deps but that it's opaque to the native build system at these companies (Buck/blaze) which means rebuilding one requires rebuilding all which is time-consuming. I infer this from the discussion about being able to query inputs/outputs of the build so that integration can be done more meaningfully into these systems to rebuild just changes.
I was entertained by this exchange:
ms: would like to know how to control use of unsafe in codebase
google: grep
ms: would like to know how to control use of unsafe in codebase
google: grep
They obviously meant ripgrep
You can totally hook into the linting system to index all usages of unsafe, even (and especially) in dependencies.
As I got to that point I started laughing and then explained it to a non-programmer that was with me.
My fumbled metaphor of "Find in page across a lot of pages" didn't really land...
My fumbled metaphor of "Find in page across a lot of pages" didn't really land...
So what do you do if “unsafe” is in a string or a comment? How do you automate the linting of it?
I believe that’s what MS is speaking about. Google probably simply puts this on the shoulders of the developer.
I believe that’s what MS is speaking about. Google probably simply puts this on the shoulders of the developer.
The build system issue isn't inherent to Rust or to large organizations. Even in a startup, we have a polyglot development with a separate build tool in addition to the language-specific build tool. We ended up doing much of the same things: use the language-specific build tool to generate artifacts which are then consumed by the normal build tool.
A slight difference is that Rust really wants to be the top level compiler. More than a lot of languages, even C++, it really relies on link time optimization to get the perf you'd expect.
In that case just print the command line cargo would've used to do the linking and let a different consume that and make changes. That's an easy case.
How this is more true of Rust than modern C++?
Most "modern" C++ involves template metaprogramming of a sort that is in header file, so the modern aspect doesn't stand to increase the need for LTO that much.
I would imagine that Rust programs benefit from LTO by removing static assertions necessary for memory safety that C++ programs simply omit, and by unraveling functional style code that Rust programs often get shoehorned into. Or maybe not, but that's one non-LTO that's needed.
I would imagine that Rust programs benefit from LTO by removing static assertions necessary for memory safety that C++ programs simply omit, and by unraveling functional style code that Rust programs often get shoehorned into. Or maybe not, but that's one non-LTO that's needed.
> The build system issue isn't inherent to Rust or to large organizations.
That may be true, but even the big boys can't converge, and they all have their own build systems.
Attempting to add/remove features from cargo/rust when even the big companies can't agree as to what is most important in a build system seems like a path to disaster.
The big boys aren't always right (see: Gradle/Groovy).
That may be true, but even the big boys can't converge, and they all have their own build systems.
Attempting to add/remove features from cargo/rust when even the big companies can't agree as to what is most important in a build system seems like a path to disaster.
The big boys aren't always right (see: Gradle/Groovy).
Can any rust experts tell me that unsafe rust is powerful and expressive enough to let me do anything?
I have a use case where I have two iterators on a linked list in nested loops and the second iterators deletes some elements as it goes. Is this something I can express in unsafe rust?
I have a use case where I have two iterators on a linked list in nested loops and the second iterators deletes some elements as it goes. Is this something I can express in unsafe rust?
I’m no expert but yes, you can do anything in unsafe as in C can do anything. Unsafe Rust provides you access to raw pointers.
I'll add more details to see if it helps-
C:
``` node prime= list.head;
while () { node probe = prime.next;
C:
``` node prime= list.head;
while () { node probe = prime.next;
while()
{
if () //condition on prime.data and probe.data {
probe.next = probe.next.next; //& drop stuff
}You need to use four space indent to create code blocks, triple-ticks don't work here.
I tried to learn those languages, my IQ level can handle java/golang/python/c just fine, but c++ and rust is beyond my comfort zone for sure, for c++, getting familiar with it is OK, but getting good at it is hard, really hard, mastering it is impossible for me. similar can be said to rust as far as I can tell.
even python has some "tricks", try "x=256; y=256; x is y" and "x=257, y=257, x is y" under python, or 'x=[], y=[], x is y; x=y=[], x is y' you will know what I mean, you actually need know some internal designs to use python properly.
At the moment I feel golang has the right trade off for programmers, be it beginners or senior developers.
even python has some "tricks", try "x=256; y=256; x is y" and "x=257, y=257, x is y" under python, or 'x=[], y=[], x is y; x=y=[], x is y' you will know what I mean, you actually need know some internal designs to use python properly.
At the moment I feel golang has the right trade off for programmers, be it beginners or senior developers.
I love high-level languages and think they've had a great democratizing effect that lets people across a wide array of backgrounds be reasonably productive without a ton of background knowledge. But, the underlying computer isn't magic. In many ways, it's pretty dumb. I suspect if you spent a bit of time looking at a system architecture, a lot of things would click into place.
Early in studies, I couldn't really wrap my head around pointers. I mean, I knew what they were and how I needed to work with them, but they just seemed like a weird abstraction. A short intro to assembly and later on, a computer architecture class, cleared that all up.
Everyone learns differently, so my experience likely isn't applicable. But, for me, no longer treating the CPU as a black box helped me get a much better mental model of lower-level languages. And I think I'm a better high-level developer for it.
Early in studies, I couldn't really wrap my head around pointers. I mean, I knew what they were and how I needed to work with them, but they just seemed like a weird abstraction. A short intro to assembly and later on, a computer architecture class, cleared that all up.
Everyone learns differently, so my experience likely isn't applicable. But, for me, no longer treating the CPU as a black box helped me get a much better mental model of lower-level languages. And I think I'm a better high-level developer for it.
not the case for me, my background is in EE and cpu architecture, and I know hardware better than software actually...
Relating to the "Python Tricks" you mention, remember that "is" means "these two pointers (x & y) point to the same place".
You should never be to surprised when you create a bunch of new values and a language like python isn't "smart" enough to say "these are all the same value, so I'll create one of those and have x, y, etc all point to the same thing."
You should never be to surprised when you create a bunch of new values and a language like python isn't "smart" enough to say "these are all the same value, so I'll create one of those and have x, y, etc all point to the same thing."
[deleted]
I think you just need to give yourself some time. C++ is a large enough language now that "mastering" it will likely take several years of daily/professional use. I used to be decent at C++ (in the early/mid 00s), but given that I haven't touched it since then, I'm completely lost as to current best practice. If I wanted to, I'd have to spend many months catching up to all the changes in the language since I last used it.
Rust has a much smaller surface area, but does have some unique things to understand like the borrowing and lifetimes. The good thing is that as the borrow and lifetime checkers in the compiler get smarter, the less you have to know about those things while writing plain-jane code.
You mention downthread that you have an EE/comp-arch background. I do as well, and have found that useful on occasion when working on embedded systems, but perhaps not a huge asset when dealing with language esoterica. Sure, perhaps understanding pointers and memory access is easier when you understand how memory actually works, but it'll only get you so far.
Rust has a much smaller surface area, but does have some unique things to understand like the borrowing and lifetimes. The good thing is that as the borrow and lifetime checkers in the compiler get smarter, the less you have to know about those things while writing plain-jane code.
You mention downthread that you have an EE/comp-arch background. I do as well, and have found that useful on occasion when working on embedded systems, but perhaps not a huge asset when dealing with language esoterica. Sure, perhaps understanding pointers and memory access is easier when you understand how memory actually works, but it'll only get you so far.
I don't think that's a great example for Python; that's just knowing what "is" is for. The fact that it works the same as == for small integers in CPython is an optimisation showing through, but only in a place where it doesn't really matter.
You still need to have a grasp on the difference between reference equality and value equality without getting into anything anyone would call tricks or implementation details (eg, after `x = []; y = []; x.append(1)`, how many elements does y have?).
You still need to have a grasp on the difference between reference equality and value equality without getting into anything anyone would call tricks or implementation details (eg, after `x = []; y = []; x.append(1)`, how many elements does y have?).
x=x.append(1) and guess what, now x is "None", yes it makes perfect sense for those who coded python for a while, but it's a disaster for newcomers.
too many similar demos
by the way I actually like python and use it often, just saying it can surprise you when you start, not intuitive per se until you become good at it.
too many similar demos
by the way I actually like python and use it often, just saying it can surprise you when you start, not intuitive per se until you become good at it.
Sure, x.append doesn't return a value. So x=x.append is None. You learn that one pretty quick.
Perhaps use x = x + [1] instead?
But yeah, surprises like this (unexpected return values) are actually a pretty strong case for using a language like Rust or C++ where the compiler will tell you this before you run the program.
Perhaps use x = x + [1] instead?
But yeah, surprises like this (unexpected return values) are actually a pretty strong case for using a language like Rust or C++ where the compiler will tell you this before you run the program.
x=1; id(x); x+=1; id(x); # different id, means x points to new location
x=[1,2],id(x), x+=[3], id(x) ; # same id, means x stay at the original location
there should be a blog page to list those surprises for beginners :)
x=[1,2],id(x), x+=[3], id(x) ; # same id, means x stay at the original location
there should be a blog page to list those surprises for beginners :)
>> yes it makes perfect sense for those who coded python for a while, but it's a disaster for newcomers.
It’s a disaster for newcomers to programming in general. I am pretty sure a seasoned developer in any other language would understand why that returns None, too.
It’s a disaster for newcomers to programming in general. I am pretty sure a seasoned developer in any other language would understand why that returns None, too.
>I don't think that's a great example for Python; that's just knowing what "is" is for. The fact that it works the same as == for small integers in CPython is an optimisation showing through, but only in a place where it doesn't really matter.
Python should have prevented checking primitives with is though -- only let object pointers...
Python should have prevented checking primitives with is though -- only let object pointers...
I believe they're still object pointers in CPython, they're just ordinary singletons; in other implementations, they aren't even singletons. Python's object model doesn't have primitives. Adding special rules around them would significantly complicate the language as well as limit the implementors' ability to fiddle with the ranges of pre-allocated instances, for minimal benefit.
>in other implementations, they aren't even singletons
That would have been less surprising (since for < 256 the singleton-ness means all same valued instances have the same "address" (id), so behave in equality check as if they were plain numbers).
That would have been less surprising (since for < 256 the singleton-ness means all same valued instances have the same "address" (id), so behave in equality check as if they were plain numbers).
Integers are object pointers, too. CPython just so happens to preallocate a range of small integers[0].
[0]: https://github.com/python/cpython/blob/master/Objects/longob...
[0]: https://github.com/python/cpython/blob/master/Objects/longob...
>even python has some "tricks", try "x=256; y=256; x is y" and "x=257, y=257, x is y" under python, or 'x=[], y=[], x is y; x=y=[], x is y' you will know what I mean, you actually need know some internal designs to use python properly.
Or don't compare numbers with "is", since you're not supposed to? Then you can remain oblivious to the underlying mechanics that break "is" comparison for larger integers...
Or don't compare numbers with "is", since you're not supposed to? Then you can remain oblivious to the underlying mechanics that break "is" comparison for larger integers...
Yep, this is totally implementation-defined behavior, so it's actually kind of harmful to think that you should "know some internal designs" here.
> I feel golang has the right trade off for programmers
Have you tried Clojure?
Have you tried Clojure?
Some sort of "how-to" for Rust in large organizations would be very nice.
Oh good lord. Reading that made me happy that (1) I am no longer in a large organisation, and (2) I program in Go. Good luck guys, you'll need it. You've got interesting problems.
https://users.rust-lang.org/t/rust-in-large-organizations-me...
This was a meeting between a few employees of large organizations using Rust (Facebook, Google, Microsoft, Mozilla) at RustConf two weeks ago to discuss the common needs of enterprise users and how to collaborate effectively on the language and ecosystem going forward.