How Turborepo is porting from Go to Rust(vercel.com)
vercel.com
How Turborepo is porting from Go to Rust
https://vercel.com/blog/how-turborepo-is-porting-from-go-to-rust?nxtPslug=how-turborepo-is-porting-from-go-to-rust
123 comments
I have a love/hate relationship with AWS's Kinesis Client Library.
In one of our implementations, we use the `amazon_kcl_helper.py` script (python) to download and configure the KCL Multilang Daemon (a java application) which calls out to our application (a golang binary) via STDIN/STDOUT.
And honestly, it's fine. The Multilang Daemon manages shard ownership, spins up one binary per shard, fetches records, passes them to the binary, updates checkpoints, etc. Our binary just focuses on our actual application logic. And the python script does a good job of setting everything up.
It just feels wild to involve all three.
In one of our implementations, we use the `amazon_kcl_helper.py` script (python) to download and configure the KCL Multilang Daemon (a java application) which calls out to our application (a golang binary) via STDIN/STDOUT.
And honestly, it's fine. The Multilang Daemon manages shard ownership, spins up one binary per shard, fetches records, passes them to the binary, updates checkpoints, etc. Our binary just focuses on our actual application logic. And the python script does a good job of setting everything up.
It just feels wild to involve all three.
This was referenced, which I thought was interesting:
https://github.com/golang/go/issues/13492
golang cannot be compiled as a C static library with musl.
https://github.com/golang/go/issues/13492
golang cannot be compiled as a C static library with musl.
[deleted]
Ah, another tale that includes getting bit by static linking and alpine.
Maybe I've just had a bad sample size, but I just haven't experienced a big enough win by using alpine to justify the weird scenarios that come up on occasion.
The idea is nice. A small, stripped down container that will load quickly and have very few maintenance issues (due to basically zero dependencies). But my debian-slim images work well enough and when I do hit a problem there's more community around it and it's more straightforward to fix.
Maybe I've just had a bad sample size, but I just haven't experienced a big enough win by using alpine to justify the weird scenarios that come up on occasion.
The idea is nice. A small, stripped down container that will load quickly and have very few maintenance issues (due to basically zero dependencies). But my debian-slim images work well enough and when I do hit a problem there's more community around it and it's more straightforward to fix.
I am concerned about the Linux/glibc monoculture, and its effect on the broader ecosystem.
Whatever breaks on Linux/musl, is also likely to break on other operating systems.
Whatever breaks loudly elsewhere, is also quite likely silently broken on Linux/glibc.
If our default response is to double down on barely patching it enough to limp along, no wonder it keeps breaking.
Whatever breaks on Linux/musl, is also likely to break on other operating systems.
Whatever breaks loudly elsewhere, is also quite likely silently broken on Linux/glibc.
If our default response is to double down on barely patching it enough to limp along, no wonder it keeps breaking.
Most the issues with alpine Linux people face are down to package maintainers breaking backwards compatibility constantly. Package renames are a regular problem.
Alpine has at least the following deficiencies:
DNS TCP resolution doesn't work (https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-o...). It's a feature that it doesn't retry truncated lookups or obey the relevant RFCs (https://twitter.com/RichFelker/status/994629795551031296).
Alpine has a 128k stack size (https://ariadne.space/2021/06/25/understanding-thread-stack-...), while most other OSes have at least a 512k stack.
DNS TCP resolution doesn't work (https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-o...). It's a feature that it doesn't retry truncated lookups or obey the relevant RFCs (https://twitter.com/RichFelker/status/994629795551031296).
Alpine has a 128k stack size (https://ariadne.space/2021/06/25/understanding-thread-stack-...), while most other OSes have at least a 512k stack.
Just to be clear, I'm not blindly defending musl/Alpine, rather noting the detrimental effects of monoculture.
Quite often fixing a build or a package on another system or architecture is a matter of a 3-5 line diff, and I've contributed quite a few over the years. It usually boils down to an incorrect assumption by the author/maintainer, stuff like "#ifdef __linux__" (when what you actually mean is: "any UNIX-like system with X11"), or hardcoding CC=gcc (where CC=cc just works).
Quite often fixing a build or a package on another system or architecture is a matter of a 3-5 line diff, and I've contributed quite a few over the years. It usually boils down to an incorrect assumption by the author/maintainer, stuff like "#ifdef __linux__" (when what you actually mean is: "any UNIX-like system with X11"), or hardcoding CC=gcc (where CC=cc just works).
DNS TCP resolution has been fixed this year
https://www.theregister.com/2023/05/16/alpine_linux_318/
https://www.theregister.com/2023/05/16/alpine_linux_318/
Thanks for letting me know!
One thing I love about NixOS so far is that when they do a rename, they can print a deprecation warning every time you use the old name while evaluating your config. When I get a moment, I go through and fix them.
You could probably do something similar in a traditional distro by creating an alias package that prints a warning on install, but then you actually have to watch the install logs to see that.
You could probably do something similar in a traditional distro by creating an alias package that prints a warning on install, but then you actually have to watch the install logs to see that.
People keep saying they need a light and fast distro - no systemd, no glibc - but it breaks a lot of other software. So, you need to be prepared to patch a lot of stuff, and build packages from source.
Personally, I would love more of them to consider FreeBSD. It's got all of those features, a linux compatibility layer if necessary, a more permissive license, etc. I'd just love to see a lot more developers helping over there.
Personally, I would love more of them to consider FreeBSD. It's got all of those features, a linux compatibility layer if necessary, a more permissive license, etc. I'd just love to see a lot more developers helping over there.
Except you can't run FreeBSD in Kubernetes really. Or really anywhere besides in VMs in the Linux monoculture.
> a more permissive license
I see this as a net disadvantage. The great insight of the GPL licences is forcing changes to be contributed back to the project, companies can't easily privatize a public effort.
I see this as a net disadvantage. The great insight of the GPL licences is forcing changes to be contributed back to the project, companies can't easily privatize a public effort.
And this makes GPL based software like Linux stronger, more popular, more community to help
Moving to FreeBSD would require patching even more things.
It's also much easier to set up debugging tools when necessary.
Glibc is very good. They should get plaudits somewhere by someone occasionally.
Alpine is a trap! For a Python product, we tried to make it work for about a year before throwing our hands up and switching to a plain Ubuntu image. We’ve had no regrets.
Small Docker image or small binary size is this decade's canonical premature optimization.
Especially base images. The size of your own layers might be worth worrying about, but not the shared base that should only be stored once for arbitrarily many containers
Often people complain how shells in Unix/POSIX communicate via strings and praise PowerShell for communication with objects. Yet stringification still is the most simplistic means for communication between multiple domains as shown in this article.
Yes, its interesting to compare the de-facto standards around input and output parsing of CLI tools in Unix land. Input has been a solved problem since the 70s for simple use cases, it's output that was always a pain.
Inputs are arbitrary string STDIN and tokenized command line args (argv/argv). Outputs are arbitrary string STDOUT/STDERR and well-formed simple uint8_t return codes.
For input, most tools use the command line args for specifying tunable parameters. These are easy to adjust on the shell or in shell scripts, as opposed to deserializing & modifying & reserializing input files or STDIN. Most use the getopt convention (`myprog --input myfile -b 123`), which makes them well-formed enough to be a stable API and describable by simple grammars. As they are (usually) position independent key-value pairs (with some value arrays for multiple files etc), it makes it easy to add options without breaking existing callers. Look at the mountain of shell scripts out there that continue to run even as tools are updated.
For output, nothing similar exists. It would be interesting if UNIX tools could also output argc/argv and shells had builtin functions to easily parse (getopt style) and index into those. Or even just have a flat key-value envvar style return list. (I guess you could have a convention of the last line of STDOUT being dedicated to that and piping it into some tool that sets a bunch of prefixed env vars). Would have made everyday output parsing from random tools a whole lot easier instead of grep/sed/awk and the dealing with changing output as people updated the tools ("ifconfig" versus "ip" well-formed grammar).
Arrays and nested data structures are where everything gets complicated, and you need something like Powershell (powerful but clunky IMHO) or JSON and full programming languages. jshn is cool but verbose as its necessarily just a POSIX shell "extension".
Inputs are arbitrary string STDIN and tokenized command line args (argv/argv). Outputs are arbitrary string STDOUT/STDERR and well-formed simple uint8_t return codes.
For input, most tools use the command line args for specifying tunable parameters. These are easy to adjust on the shell or in shell scripts, as opposed to deserializing & modifying & reserializing input files or STDIN. Most use the getopt convention (`myprog --input myfile -b 123`), which makes them well-formed enough to be a stable API and describable by simple grammars. As they are (usually) position independent key-value pairs (with some value arrays for multiple files etc), it makes it easy to add options without breaking existing callers. Look at the mountain of shell scripts out there that continue to run even as tools are updated.
For output, nothing similar exists. It would be interesting if UNIX tools could also output argc/argv and shells had builtin functions to easily parse (getopt style) and index into those. Or even just have a flat key-value envvar style return list. (I guess you could have a convention of the last line of STDOUT being dedicated to that and piping it into some tool that sets a bunch of prefixed env vars). Would have made everyday output parsing from random tools a whole lot easier instead of grep/sed/awk and the dealing with changing output as people updated the tools ("ifconfig" versus "ip" well-formed grammar).
Arrays and nested data structures are where everything gets complicated, and you need something like Powershell (powerful but clunky IMHO) or JSON and full programming languages. jshn is cool but verbose as its necessarily just a POSIX shell "extension".
In this case, the relevant aspect was not stringification, but serialization. It would have worked just as fine had they used a binary encoding like CBOR instead of JSON.
Coming soon: How Turborepo is porting from Rust to Zig/whatever hottest language is to have on resume next year
These blog posts make me laugh. Pretty sure they could achieve whatever they are doing with any language, heck even PHP would probably work. Just not cool though is it.
These blog posts make me laugh. Pretty sure they could achieve whatever they are doing with any language, heck even PHP would probably work. Just not cool though is it.
They explain their motivation here: https://vercel.com/blog/turborepo-migration-go-rust
uhtred(1)
They had a pretty balanced take in the article.
Their previous blog post (linked in the first sentence) about why the switch from Go to Rust is insightful.
I'm currently learning Go but have had my eye on Rust as well and have been wondering about the strengths and weaknesses of each language and how they compare and where each language is best applied.
The linked blog confirms my first impression that Go is well suited for networking applications and simplicity whereas Rust is well suited for OS/low-level applications.
Does anyone here have any experience using Go and Rust? What are your thoughts on the two languages?
I'm currently learning Go but have had my eye on Rust as well and have been wondering about the strengths and weaknesses of each language and how they compare and where each language is best applied.
The linked blog confirms my first impression that Go is well suited for networking applications and simplicity whereas Rust is well suited for OS/low-level applications.
Does anyone here have any experience using Go and Rust? What are your thoughts on the two languages?
I have no experience with Rust, but have been using Go professionally for ~5 years now. For a lot of the flak that Go gets (and praise that Rust gets), the Go standard library (and supporting ecosystem) indeed is fantastic for anything relating to a web server.
When I analyzed Rust around the same time, I noted the Rust standard library [1] did not have HTTP support, and it wasn't a first class consideration. I think Hyper [2] was around, but I've never analyzed it deeply (though based on GitHub stars it seems to be popular). Protobuf [3] is also extremely easy to work with in Go.
Given the differences in standard library and applications I see created in both ecosystems, your analysis seems right, though you can probably develop network(ed) applications in both fairly well at this point.
[0]: https://pkg.go.dev/std
[1]: https://doc.rust-lang.org/std/
[2]: https://github.com/hyperium/hyper/
[3]: https://github.com/golang/protobuf
When I analyzed Rust around the same time, I noted the Rust standard library [1] did not have HTTP support, and it wasn't a first class consideration. I think Hyper [2] was around, but I've never analyzed it deeply (though based on GitHub stars it seems to be popular). Protobuf [3] is also extremely easy to work with in Go.
Given the differences in standard library and applications I see created in both ecosystems, your analysis seems right, though you can probably develop network(ed) applications in both fairly well at this point.
[0]: https://pkg.go.dev/std
[1]: https://doc.rust-lang.org/std/
[2]: https://github.com/hyperium/hyper/
[3]: https://github.com/golang/protobuf
These days Axum is the most common way of building an HTTP server. It sits atop Hyper and it’s very composable thanks to Tower. It supports Websockets out of the box too.
Not more than a couple of months ago people would recommend actix. There is a lot of enthusiasm and excitement behind the language and the ecosystem reflects that, for better or worse.
People will still recommend actix. That doesn't mean that axum wouldn't be recommended either. It was there a couple of months ago too and doing extremely well.
Yeah, and I genuinely want to like Rust and I pick it up a few times every year (and have done for the last decade), but each time I get burned by trying to do something that would be trivial (and safe!) in Go or C.
Most recently, I'm trying to build a non-allocating, minimally copying `Lines` iterator which reads from an internal `Reader` into an internal buffer and then yields slices of the internal buffer on each call to `next` (each slice represents a single line, and it's an error if a line exceeds the capacity of the internal buffer); however, as far as I can tell, this is unworkable without some unsafety because the signature is `fn next(self: &mut Lines<'a>) -> Result<&'a [u8]>` which doesn't work because Rust thinks the mutable reference to self must outlive 'a, and explicitly setting the mutable self reference to 'a violates the trait. If I forego the trait and just make a thing with a `next()` method, then I can't use it in a loop without triggering some multiple mutable borrows error (each loop iteration constitutes a mutable borrow and for some reason these borrows are considered to be concurrent). The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer.
I don't run into this in C or Go, and for all of the difficulty of battling the borrow checker, I'm not getting any extra safety (in this case).
Most recently, I'm trying to build a non-allocating, minimally copying `Lines` iterator which reads from an internal `Reader` into an internal buffer and then yields slices of the internal buffer on each call to `next` (each slice represents a single line, and it's an error if a line exceeds the capacity of the internal buffer); however, as far as I can tell, this is unworkable without some unsafety because the signature is `fn next(self: &mut Lines<'a>) -> Result<&'a [u8]>` which doesn't work because Rust thinks the mutable reference to self must outlive 'a, and explicitly setting the mutable self reference to 'a violates the trait. If I forego the trait and just make a thing with a `next()` method, then I can't use it in a loop without triggering some multiple mutable borrows error (each loop iteration constitutes a mutable borrow and for some reason these borrows are considered to be concurrent). The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer.
I don't run into this in C or Go, and for all of the difficulty of battling the borrow checker, I'm not getting any extra safety (in this case).
Alternatively, use a closure: https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#me...
There are downsides to this approach because it uses internal iteration while most things in Rust use external iteration. But shit happens. Go doesn't even have a first class concept of iterators as an abstraction (yet), and its standard library contains patterns for both internal (sync.Map) and external (bufio.Scanner) iteration.
> The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer.
Yup that works too. That's basically the design used by the `streaming-iterator` crate: https://docs.rs/streaming-iterator/latest/streaming_iterator...
> I don't run into this in C or Go
Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. And Go lets you iterate over a fixed set of built-in types. (Currently. Maybe it's changing: https://research.swtch.com/coro)
Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.
There are downsides to this approach because it uses internal iteration while most things in Rust use external iteration. But shit happens. Go doesn't even have a first class concept of iterators as an abstraction (yet), and its standard library contains patterns for both internal (sync.Map) and external (bufio.Scanner) iteration.
> The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer.
Yup that works too. That's basically the design used by the `streaming-iterator` crate: https://docs.rs/streaming-iterator/latest/streaming_iterator...
> I don't run into this in C or Go
Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. And Go lets you iterate over a fixed set of built-in types. (Currently. Maybe it's changing: https://research.swtch.com/coro)
Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.
> Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them.
My goal isn’t “implement an Iterator trait”, I just want to iterate over lines in a file, so I don’t especially care that Go and C don’t have an iterator type.
> Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.
I’m not allocating, so the GC doesn’t matter. The Go and C versions look essentially the same—it’s only the Rust version that I had a hard time with because of the borrow checker.
My goal isn’t “implement an Iterator trait”, I just want to iterate over lines in a file, so I don’t especially care that Go and C don’t have an iterator type.
> Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.
I’m not allocating, so the GC doesn’t matter. The Go and C versions look essentially the same—it’s only the Rust version that I had a hard time with because of the borrow checker.
Great, then the closure approach should work perfectly!
> I just want to iterate over lines in a file
No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file."
> I’m not allocating, so the GC doesn’t matter.
Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.
> I just want to iterate over lines in a file
No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file."
> I’m not allocating, so the GC doesn’t matter.
Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.
> No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file."
Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial.
> Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.
GC only manages allocations. There are no allocations here to manage.
Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial.
> Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.
GC only manages allocations. There are no allocations here to manage.
> GC only manages allocations. There are no allocations here to manage.
You're missing the forest for the trees. The GC is integrally tied to lifetimes. If you have a `*Foo` in Go, that may or may not be on the stack. It might be on the stack, thus no allocation, if the Go compiler can prove something about its lifetime and usage. Same deal with `[]byte`. Maybe that's tied to an array on a stack somewhere. Maybe not. AFAIK, in order to get guarantees about this in Go you need to drop down into `unsafe`.
> Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial.
That's why the very first thing I said to you was to point out the closure approach. I even linked you to real code (that I've written) that does it. Yet 'round and 'round we go.
You're missing the forest for the trees. The GC is integrally tied to lifetimes. If you have a `*Foo` in Go, that may or may not be on the stack. It might be on the stack, thus no allocation, if the Go compiler can prove something about its lifetime and usage. Same deal with `[]byte`. Maybe that's tied to an array on a stack somewhere. Maybe not. AFAIK, in order to get guarantees about this in Go you need to drop down into `unsafe`.
> Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial.
That's why the very first thing I said to you was to point out the closure approach. I even linked you to real code (that I've written) that does it. Yet 'round and 'round we go.
TBF you could do it in most languages which do have iteration abstraction.
However assuming trying to keep to 0-alloc odds are good you'd probably just be telling the user to git gud as the slices would become nonsensical on every refill of the buffer.
However assuming trying to keep to 0-alloc odds are good you'd probably just be telling the user to git gud as the slices would become nonsensical on every refill of the buffer.
Yes, 0-alloc and minimal copying is a key constraint here. :-) If you take that away, then there's no real problem here.
I could easily imagine trying to do it because I can rather than because I actually need to.
Also because of the missing middle: in Go (or Java, or C#, or even python) you could hand out slices to a large buffer, but discard the buffer rather than refill it in place, relying on the GC to clean things up (and possibly give you the same buffer on the next allocation). So you get an efficiency middle ground where you allocate more than strictly necessary but not for every line, and maintain correctness.
In Rust that’d require some sort of Rc projection which I’m not sure even exists?
Also because of the missing middle: in Go (or Java, or C#, or even python) you could hand out slices to a large buffer, but discard the buffer rather than refill it in place, relying on the GC to clean things up (and possibly give you the same buffer on the next allocation). So you get an efficiency middle ground where you allocate more than strictly necessary but not for every line, and maintain correctness.
In Rust that’d require some sort of Rc projection which I’m not sure even exists?
What you're talking about is the canonical example for generic associated types, which were stabilised late 2022: https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#gener...
Ah, good to know. Thanks!
Yes, experience in both. I'm considerably more inclined to use Rust for any use case, but I'll freely admit that I'm biased because it appeals to my sensibilities more.
IMO the most compelling use case to use Go over Rust is if you're writing a networked service or networking code.
IMO the most compelling use case to use Go over Rust is if you're writing a networked service or networking code.
Having used both Go and Rust for networked and networking code, I'd say it's a pretty nuanced set of tradeoffs between the two.
I've written a couple protocol handlers in rust (that is parse the packet and handle the logic before giving the data to some other bit of code) and I found Rust really nice for that - real enums, pattern matching, and the use of traits can make interacting with binary protocols (particularly of the state machine variety) really nice with ergonomic interfaces. Meanwhile go's net/IP type (etc) often leave me frustrated.
On the other hand, goroutines and channels are such a nice way to handle a lot of message passing around a server application that I find myself reaching for mpsc channels and using tokio tasks like clunkier goroutines in rust often.
I think I'd draw the line more along "how often I need to work with byte arrays that I want to turn into semantic types" - if I'm going down to the protocol level and not doing a ton of high level logic I'll reach for rust, if it's more high level handling of requests/responses (http for example) I'll often reach for go.
(like the parent, I'm more inclined to reach for Rust over go if all else is equal, but I enjoy go too).
I've written a couple protocol handlers in rust (that is parse the packet and handle the logic before giving the data to some other bit of code) and I found Rust really nice for that - real enums, pattern matching, and the use of traits can make interacting with binary protocols (particularly of the state machine variety) really nice with ergonomic interfaces. Meanwhile go's net/IP type (etc) often leave me frustrated.
On the other hand, goroutines and channels are such a nice way to handle a lot of message passing around a server application that I find myself reaching for mpsc channels and using tokio tasks like clunkier goroutines in rust often.
I think I'd draw the line more along "how often I need to work with byte arrays that I want to turn into semantic types" - if I'm going down to the protocol level and not doing a ton of high level logic I'll reach for rust, if it's more high level handling of requests/responses (http for example) I'll often reach for go.
(like the parent, I'm more inclined to reach for Rust over go if all else is equal, but I enjoy go too).
> find myself reaching for mpsc channels and using tokio tasks like clunkier goroutines in rust often
My experience so far with mpsc/tokio tasks is that Fearless Concurrency TM is pretty solid. There is some clunkiness (mostly inconsistency around async styles), but I've been bit by bad channel usage in golang enough times that rust felt refreshing.
My experience so far with mpsc/tokio tasks is that Fearless Concurrency TM is pretty solid. There is some clunkiness (mostly inconsistency around async styles), but I've been bit by bad channel usage in golang enough times that rust felt refreshing.
Totally agree with this nuance. Really depends on what layer you’re working at.
I have done both. Currently moving a number of microservices from node/ruby/golang to a rust monolith. Slow and steady progress.
Our biggest struggle with golang was that our Ruby/FP backgrounds just clashed with golang's style. It wasn't terrible, but we always felt it was more verbose and clunkier to compose and reuse things than we expected. I'm sure generics are making this better, but even with generics the general feel of the type system just felt off to us.
Our monolith is http/graphql + postgres in Rust and we like it. Some learning curve for the team, but most of our "application code" is pretty easy Rust and people can jump in and work on it quickly enough. Most work is defining in/out types and implementing a few key traits for them. Our client engineers are comfortable doing it. Our "app infra code" is a little more technical, but we've appreciated the strong type system as we've built out pubsub systems and other important bits of infra.
Ultimately though, it's just a culture thing. I'd recommend one of jvm/rust/golang to anyone trying to build apps that handle significant traffic. Which of those 3 is just whichever matches up to your team and their personalities.
Our biggest struggle with golang was that our Ruby/FP backgrounds just clashed with golang's style. It wasn't terrible, but we always felt it was more verbose and clunkier to compose and reuse things than we expected. I'm sure generics are making this better, but even with generics the general feel of the type system just felt off to us.
Our monolith is http/graphql + postgres in Rust and we like it. Some learning curve for the team, but most of our "application code" is pretty easy Rust and people can jump in and work on it quickly enough. Most work is defining in/out types and implementing a few key traits for them. Our client engineers are comfortable doing it. Our "app infra code" is a little more technical, but we've appreciated the strong type system as we've built out pubsub systems and other important bits of infra.
Ultimately though, it's just a culture thing. I'd recommend one of jvm/rust/golang to anyone trying to build apps that handle significant traffic. Which of those 3 is just whichever matches up to your team and their personalities.
I have used Go for my day job, I'd say you can get the job done in Go. But the resulting code won't be as elegant as you could do in Rust. I'd chose Rust just because how good the type system is: sum type, pattern matching, iterators, typestate pattern, etc makes programming a pleasure. Navigating async code takes little getting used to though. And be ready for reading many new symbols i.e. ', ~, etc.
I've been programming for 12+ years and it gets real boring when you've to type/copy same thing again and again, and Go doesn't want to help you here.
I've been programming for 12+ years and it gets real boring when you've to type/copy same thing again and again, and Go doesn't want to help you here.
I don’t think comparing Go and Rust is particularly useful. The languages have different goals. I don’t want to start a flame war, but I prefer Rust to Go for a lot of reasons, mainly the lack of runtime, the type system, as well as more compile time safety guarantees.
> Does anyone here have any experience using Go and Rust? What are your thoughts on the two languages?
Rust has a much better type system, Go has a much better standard library. The problems with Rust are that the async solutions are shifting sands and that most people "cheat" by just shoving everything into a Box on the heap instead of properly using lifetimes on the stack. Both of those lead to a lot of inconsistency and sort of defeat the purpose of using Rust in the first place.
The main problem with Go is verbose error handling, it may also be too high level for some systems level tasks Rust may be better suited to.
Rust has a much better type system, Go has a much better standard library. The problems with Rust are that the async solutions are shifting sands and that most people "cheat" by just shoving everything into a Box on the heap instead of properly using lifetimes on the stack. Both of those lead to a lot of inconsistency and sort of defeat the purpose of using Rust in the first place.
The main problem with Go is verbose error handling, it may also be too high level for some systems level tasks Rust may be better suited to.
Theres lots of purposes of using rust in the first place.
Some people want the performance.
Some people want the safety guarantees and the checking the compiler does for correctness.
Some people want the expressive type system (and combined with the above, the ability to encode constraints into data types).
I don't think that people who choose based on the third but don't need super tight performance are using it wrong, they just have a different set of priorities than those that are using it for very high performance.
Some people want the performance.
Some people want the safety guarantees and the checking the compiler does for correctness.
Some people want the expressive type system (and combined with the above, the ability to encode constraints into data types).
I don't think that people who choose based on the third but don't need super tight performance are using it wrong, they just have a different set of priorities than those that are using it for very high performance.
"Their previous blog post (linked in the first sentence) about why the switch from Go to Rust is insightful."
I was actually a bit astonished by it. The most trenchant concrete example they could give is... Go's standard library abstraction around file permissions is a bit inconvenient for their use case? This is easily fixed in Go in about a day without a complete rewrite; you just write some new stuff directly against the relevant syscall libraries and use build tags to control the different OS builds.
Just because an abstraction exists doesn't mean you have to use it! It is not a requirement of Go that you must use that particular abstraction, it's just something provided by the standard library. We once wrote a library for Perl using the openat and other *at functions because we needed the security guarantees, and it worked fine, despite the "standard library" not working that way.
My read is that they're basically doing this because they want to, not because Go forced them to. That's fine. There's nothing wrong with that, if they want to pay the price for migration. (Were someone proposing it to me in real life I'd expect a better reason than "I don't like how it handles file permissions in the standard library", though.) However, as an external observer using it as a grounds to decide it's much weaker than may meet the eye; I wouldn't overprivilege it.
The real reason to prefer Rust here would be something on the lines of "We're doing so much crazy concurrent stuff that we need the guarantees that Rust provides via compiler but Go only provides via common practices." The latter can carry you a long ways but it does eventually give out and become insufficient for a codebase, and when that's the case Rust becomes one of the short list of options. "Go common practices" is actually pretty high up on the set of "ways to do reasonable concurrency", reaching above that requires a pretty significant shift to Erlang/Elixir, Haskell, or Rust, and given the goals of the project that would basically leave Rust as the only acceptable performance choice. It's possible that this would qualify for them, though I'm not sure... from what it sounds like they're doing, a task management system that ensures that only one worker is doing a given task and everyone else waits on the relevant output without redoing it would be the core of their architecture, and even if you use that thousands of times per execution that doesn't necessarily mean the code is complex internally.
I was actually a bit astonished by it. The most trenchant concrete example they could give is... Go's standard library abstraction around file permissions is a bit inconvenient for their use case? This is easily fixed in Go in about a day without a complete rewrite; you just write some new stuff directly against the relevant syscall libraries and use build tags to control the different OS builds.
Just because an abstraction exists doesn't mean you have to use it! It is not a requirement of Go that you must use that particular abstraction, it's just something provided by the standard library. We once wrote a library for Perl using the openat and other *at functions because we needed the security guarantees, and it worked fine, despite the "standard library" not working that way.
My read is that they're basically doing this because they want to, not because Go forced them to. That's fine. There's nothing wrong with that, if they want to pay the price for migration. (Were someone proposing it to me in real life I'd expect a better reason than "I don't like how it handles file permissions in the standard library", though.) However, as an external observer using it as a grounds to decide it's much weaker than may meet the eye; I wouldn't overprivilege it.
The real reason to prefer Rust here would be something on the lines of "We're doing so much crazy concurrent stuff that we need the guarantees that Rust provides via compiler but Go only provides via common practices." The latter can carry you a long ways but it does eventually give out and become insufficient for a codebase, and when that's the case Rust becomes one of the short list of options. "Go common practices" is actually pretty high up on the set of "ways to do reasonable concurrency", reaching above that requires a pretty significant shift to Erlang/Elixir, Haskell, or Rust, and given the goals of the project that would basically leave Rust as the only acceptable performance choice. It's possible that this would qualify for them, though I'm not sure... from what it sounds like they're doing, a task management system that ensures that only one worker is doing a given task and everyone else waits on the relevant output without redoing it would be the core of their architecture, and even if you use that thousands of times per execution that doesn't necessarily mean the code is complex internally.
That article definitely buried the lede. A fair reading of the article is that the only reason they are switching from Go to Rust is because their dev team wants to write in Rust. Which is fine! But I think they could have been more open about that rather than inventing a strange complaint about how Go handles file permissions.
> ... because their dev team wants to write in Rust. Which is fine! But I think they could have been more open about that ...
Not when pitching the project to hard-nosed non-technical managers burdened with personal responsibility for P&L results.
Not when pitching the project to hard-nosed non-technical managers burdened with personal responsibility for P&L results.
Agree.
I have a mental category for these posts:
We saved some of cloud / server side cost with Rust, so you user can spend more on local desktop /cloud cost with Javascript
Sometimes cost savings can be imaginary but since Rust is cool it is all fine.
I have a mental category for these posts:
We saved some of cloud / server side cost with Rust, so you user can spend more on local desktop /cloud cost with Javascript
Sometimes cost savings can be imaginary but since Rust is cool it is all fine.
Their technical justification is so weak that I think it was more like, 'We are bored, look at this new shiny toy called Rust. Let's think about reasons why we need to use it (to justify using it) so we can have some fun'.
Yes. To be honest I keep learning, writing bits n pieces of tools using cool new language/framework etc for my personal pleasure. But to read these kind of official announcement blog posts feels so fake. I don't know, but I can't take anything seriously from people/companies writing these articles.
Depending on your background, learning either will be beneficial in that you'll learn about a C-like memory model. My experience is that Go is way easier to get started with (both in terms of the language and the libraries). It has a certain concurrency model baked into the language (which in Rust is available in the standard library). It's kind of a lower-level Python. Also I'd say that if you're not familiiar with either language, Go programs are probably easier to read, which can make a difference if you're not working alone on a project.
Rust takes a bit more effort to initially get results, but once you get there, you work in a language that gives you expressive ways to create abstraction (this is a matter of taste maybe), generates generally faster code, and does not incur overhead by garbage collection. Also, there are ways to write asynchronous code in an elegant way. The price is that you spend more time learning about structuring programs in a way that the compiler accepts. Once you are past that point, you'll be at least as productive as in Go.
I used to dabble with Go and like it, but once I had to write more code, I found it more tedious compared to Rust. But as I said, more readable to the uninitiated, that was one of the Go design goals.
Rust takes a bit more effort to initially get results, but once you get there, you work in a language that gives you expressive ways to create abstraction (this is a matter of taste maybe), generates generally faster code, and does not incur overhead by garbage collection. Also, there are ways to write asynchronous code in an elegant way. The price is that you spend more time learning about structuring programs in a way that the compiler accepts. Once you are past that point, you'll be at least as productive as in Go.
I used to dabble with Go and like it, but once I had to write more code, I found it more tedious compared to Rust. But as I said, more readable to the uninitiated, that was one of the Go design goals.
Rust is also poison for the mind. Sweet, sweet poison.
I’m currently back to Python and losing my mind about doing error handling, as well as enforcing correct usage of my library API on the call site. Doing these things well (best?) in Rust is baked into that language’s DNA (Result, Option, newtype pattern, type state pattern, …). It’s painful to go without once you’ve seen the light.
I’m currently back to Python and losing my mind about doing error handling, as well as enforcing correct usage of my library API on the call site. Doing these things well (best?) in Rust is baked into that language’s DNA (Result, Option, newtype pattern, type state pattern, …). It’s painful to go without once you’ve seen the light.
Using Pydantic and the `validate_arguments` decorator can get you a lot more safety. Not as safe as Rust, but more safe than vanilla Python.
I love pydantic and use it whenever I can, but that decorator seems a bit misguided, no? It requires type hints anyway, by which point mypy would catch issues with types mismatches statically, before runtime. That would seem strictly more useful. Maybe I’m missing an aspect of what the decorator solves though.
Mypy is an amazing, should have mentioned it.
Pydantic enforces types when you're working with a Pydantic objects. However, it doesn't help with functions that accept vanilla types as parameters. validate arguments checks that the callers parameters matches the function's type hints.
This is helpful because a static analyzer like mypy won't catch the wrong type being passed in all situations.
Pydantic enforces types when you're working with a Pydantic objects. However, it doesn't help with functions that accept vanilla types as parameters. validate arguments checks that the callers parameters matches the function's type hints.
This is helpful because a static analyzer like mypy won't catch the wrong type being passed in all situations.
> Does anyone here have any experience using Go and Rust? What are your thoughts on the two languages?
I've worked on an extremely large golang monorepo, but I don't have production Rust experience. They're vastly different languages that they shouldn't really be compared. The only commonality they have is that they both compile to native code, that's about it. Other than that, golang is basically a python/ruby/perl/php competitor, whereas Rust is a C++ competitor.
I've worked on an extremely large golang monorepo, but I don't have production Rust experience. They're vastly different languages that they shouldn't really be compared. The only commonality they have is that they both compile to native code, that's about it. Other than that, golang is basically a python/ruby/perl/php competitor, whereas Rust is a C++ competitor.
Yeah I'm quite confused at how obsessed people are with comparing these two languages specifically. IMO it would make most sense to compare Go to C# or a JVM language. People seem to be thinking that the (good) idea of having it compile to a single executable out of the box means it's in the same class of languages as C, when it's still a managed language - it just bundles the runtime
It's a mix of historical happenstance, confusion and historical impression.
First, the two languages were initially revealed around the same time, and they were both new AOT-compiled languages targeting a bit downstack from your Java and C#, which was in opposition of the trend since the 90s.
Second, Rust was specifically targeted at "system programming" from the start, as from the start Mozilla desired using it to displace C++ in the browser; Go was also initially billed as a "system programming" language, though for a completely different interpretation of the term.
Finally, not only did the initial rust announcements very much emphasise concurrency (something Go also does), the initial rust was a much higher-level language, with a mandatory runtime, green threads, and (plans for) a GC.
Even though Rust changed drastically between the initial announcement and the 1.0 release, and went significantly downstack (aiming much more squarely at C and C++), those initial impressions have left lasting traces in the global consciousness.
And while the two have generally different niches, they very much do compete when it comes to... CLI utilities.
First, the two languages were initially revealed around the same time, and they were both new AOT-compiled languages targeting a bit downstack from your Java and C#, which was in opposition of the trend since the 90s.
Second, Rust was specifically targeted at "system programming" from the start, as from the start Mozilla desired using it to displace C++ in the browser; Go was also initially billed as a "system programming" language, though for a completely different interpretation of the term.
Finally, not only did the initial rust announcements very much emphasise concurrency (something Go also does), the initial rust was a much higher-level language, with a mandatory runtime, green threads, and (plans for) a GC.
Even though Rust changed drastically between the initial announcement and the 1.0 release, and went significantly downstack (aiming much more squarely at C and C++), those initial impressions have left lasting traces in the global consciousness.
And while the two have generally different niches, they very much do compete when it comes to... CLI utilities.
[deleted]
You should learn both and be able to use both comfortably.
> By serializing to JSON, a format with robust support in both Go and Rust, we were able to minimize our FFI surface area, and avoid a whole class of cross-platform, cross-language bugs.
There’s been a lot of discussion of the marshaling and unmarshaling costs of Json (and really any other text format). Should we be looking more strongly at binary formats, and if those have better outcomes than Json?
I’m guessing they didn’t want to introduce more change to the Go code than necessary, but I’m wondering if people have any horror stories to share about any of the binary formats available?
There’s been a lot of discussion of the marshaling and unmarshaling costs of Json (and really any other text format). Should we be looking more strongly at binary formats, and if those have better outcomes than Json?
I’m guessing they didn’t want to introduce more change to the Go code than necessary, but I’m wondering if people have any horror stories to share about any of the binary formats available?
Protobuf is incredibly annoying to use with rust (my usecase was specifically rust to-rust) because of how limited protobuf's type system is. For instance there is no way to represent an `Option` type because protobuf has decided that if something is no present, then it is really some default value instead. The only workaround I found was to have both `foo` and `is_fo_set` fields, which required having duplicate copies of every datatype one for (de)serialization and one for actually using in my codebase, manually implementing changes between them. Don't get me wrong, there's nothing show-stopping, but it was definitely a death by a thousand cuts situation for me.
Though, I know that another team at the same company loved it, but they were all in on java (and groovy and kotlin). As I understand it protobuf meshes with Java's type system much better (and probably Go's too?).
---
I did some benchmarking on various formats/rust libraries when we were deciding on what the protocol should be used for ^ and, IIRC, JSON fared better than you'd have expected. My assumption is that JSON just has had way more eyes/hands on the implementation than anything else. The things that I can remember beating it were (1) bincode, (2) cbor, and (3) protobuf, then JSON was #4.
In retrospect I wish we'd have gone with bincode, but we weren't sure if we were going to need some java code to interact with this service at the time and, at least at the time, bincode was rust-only (and possibly not even stable between compiler releases?). It would have been much faster to develop with and there was a whole bunch of overhead from protobuf that didn't really do anything for us since we were talking over a unix domain socket within a single system.
Though, I know that another team at the same company loved it, but they were all in on java (and groovy and kotlin). As I understand it protobuf meshes with Java's type system much better (and probably Go's too?).
---
I did some benchmarking on various formats/rust libraries when we were deciding on what the protocol should be used for ^ and, IIRC, JSON fared better than you'd have expected. My assumption is that JSON just has had way more eyes/hands on the implementation than anything else. The things that I can remember beating it were (1) bincode, (2) cbor, and (3) protobuf, then JSON was #4.
In retrospect I wish we'd have gone with bincode, but we weren't sure if we were going to need some java code to interact with this service at the time and, at least at the time, bincode was rust-only (and possibly not even stable between compiler releases?). It would have been much faster to develop with and there was a whole bunch of overhead from protobuf that didn't really do anything for us since we were talking over a unix domain socket within a single system.
Yes it is unsurprising that an IDL designed to be used as an interchange format between different languages (and other requirements such as backward compatibility etc) doesn't perfectly match your specific language type system.
An idiomatic Go structure does not map well to an idiomatic rust structure, and similarly other languages.
An idiomatic Go structure does not map well to an idiomatic rust structure, and similarly other languages.
To be fair, protobuf is a pretty poor/annoying match to many languages that aren't Go or Java, including TS. Optional types, for example, are very much not unique to Rust.
Don't get me wrong, it is a PITA.
But it's good enough and has very wide and mature support for many languages and a good story for backward compatibility which is important when you have files you want to read 2 years from now or when you have different version of your clients and servers co-existing.
There _could_ be a better interchange format in theory (and many have been proposed), but a combination of maturity and network effect (and the fact that many of the alternatives to protobuf do improve some things but do other things worse than protobuf) make it very hard to chose an alternative.
EDIT: ah, about discriminating zero value from unset value (aka options), they're back in proto3 (experimental) https://github.com/protocolbuffers/protobuf/blob/main/docs/i...
But it's good enough and has very wide and mature support for many languages and a good story for backward compatibility which is important when you have files you want to read 2 years from now or when you have different version of your clients and servers co-existing.
There _could_ be a better interchange format in theory (and many have been proposed), but a combination of maturity and network effect (and the fact that many of the alternatives to protobuf do improve some things but do other things worse than protobuf) make it very hard to chose an alternative.
EDIT: ah, about discriminating zero value from unset value (aka options), they're back in proto3 (experimental) https://github.com/protocolbuffers/protobuf/blob/main/docs/i...
iirc https://github.com/tokio-rs/prost has the proper proto3 'optional' -> Option<T> support
Have you tried FlatBuffers? And if you did, did it perform worse than JSON? The idea of deserialize-on-read is tempting to me, but I'd like to see real-world cases of it working out.
FlatBuffers should be faster than protobuff.
proto3 messsage fields allow for detecting set vs. not set; other field types (repeated, map, int32, string, bool, enum, etc.) have this "default value if not set" issue. The canonical way of handling this is to use wrapper messages (because one can detect if the wrapper is not set), and there are "well-known" canned messages/protos one can import and use without writing their own: https://protobuf.dev/reference/protobuf/google.protobuf/
Whether the codegen/libraries for a particular language provides a more idiomatic binding for these well-known wrappers is up to the implementation - for example, golang libraries have conveniences added for well known libraries: https://pkg.go.dev/google.golang.org/protobuf/types/known. Rust libraries may have the same; I'm not as familiar with the ecosystem there.
Whether the codegen/libraries for a particular language provides a more idiomatic binding for these well-known wrappers is up to the implementation - for example, golang libraries have conveniences added for well known libraries: https://pkg.go.dev/google.golang.org/protobuf/types/known. Rust libraries may have the same; I'm not as familiar with the ecosystem there.
recent versions of proto3 have added back the “optional” keyword that can be used on any field. see: https://github.com/protocolbuffers/protobuf/blob/main/docs/f...
On optional.. this was a regression in proto that is somewhat helped by https://github.com/protocolbuffers/protobuf/blob/main/docs/f... ; I have no idea whether protobuf for rust has started taking advantage of this.
JSON is awful in every way.
JSON is awful in every way.
It depends on what you’re marshaling, and how much. Anyway, in my experience, the biggest practical disadvantage of JSON is not performance, it’s that people tend not to put a schema validation/migration layer on top of it, which leads over time to chaos. Many of the popular binary formats have schema validation built in.
Text formats have innate issues when it comes to deserialization: you don't know the size of each field of your data until you scan the entire field. This makes deserializers generally more prone to DOS attacks. Many binary formats must necessarily store the size of fields as metadata, which means you might not need to deserialize the whole message to interpret parts of it. Further, JSON doesn't support streaming natively. These are mostly unique use-cases with marginal improvements, but it can matter.
Totally insane that people don't put validation since jsonschema is quite nice
A few months ago, I did some analysis on JSON data in our platform, and discovered that more than two-thirds of bytes were field names. Two thirds! That’s a lot of potentially unnecessary extra bytes.
In my case, I was thinking about storage costs for JSON data; but thinking about this use case: Isn’t it true that the CPU would have to spend basically three times as much time making FFI calls? Assuming that practically speaking, field names make up 2/3rds of the bytes in their real-world JSON data.
In my case, I was thinking about storage costs for JSON data; but thinking about this use case: Isn’t it true that the CPU would have to spend basically three times as much time making FFI calls? Assuming that practically speaking, field names make up 2/3rds of the bytes in their real-world JSON data.
JSON compresses very well, fortunately
Isn't this the point of things like protobuf and thrift? It seems like everyone beyond a certain scale sends data around in some binary format rather than JSON.
Yes, I’m wondering if people have bad experiences with CBOR, thrift, flatbuffers, protobuf, messagepack, etc. There are a huge number. I think Protobuf might be the best choice across all languages at this point, maybe?
But that’s my question, if you choose a binary format, what are the issues to look out for?
Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.
But that’s my question, if you choose a binary format, what are the issues to look out for?
Example, I work with Java, Rust, Go, and Python at work. I have specifically had issues with Avro (not my choice) which is really well supported in Java, but not much else.
I dislike protobuf for the reason that the generated code is often quite bad (in terms of the interface in the language you are using) and in external projects it's often a manual step (sometimes with additional headache of protoc2 vs protoc3) to just build the project. In terms of message specification it's quite good though.
This is the reason we opted for building our own code gen system instead of Thrift or Protobuf: they both generate really ugly code in the languages we wanted to use
Still early days, but we've been using CBOR instead of JSON lately at work for interfaces that have "settled" and it's been great. Means that you can shake out the early integration issues using human readable JSON, then just switch the ser/de once it's all playing nice.
Binary data support is pretty nice too for avoiding multipart request bodies.
Binary data support is pretty nice too for avoiding multipart request bodies.
Ah, I see what you were saying now. Yeah, I don't have direct experience with those binary formats so I can't say. I'm not an Avro fan either but there are other binary formats like parquet that are easier to work with and at least in my corner of the tech world seems to have become the de facto standard.
I would have suggested ProtoBuf too.
That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs.
I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal.
That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.
That said, there was a comment in their post about not wanting to tightly couple the Go and Rust since they weren't sure they could accurately represent their structures in both langs.
I don't know anything about this turbo tool, but it seems relatively low-throughput, and they intend to migrate it all to Rust anyway, so it's probably not a big deal.
That said; I'd have done this with binary comms and not JSON, ProtoBuf or not.
I'm using protobuf for exactly this use-case, but I'd say that it's clearly designed as a network format first, and makes trade-offs as such.
For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc...
MessagePack seems to make a better set of tradeoffs by comparison.
For example, it prioritizes backwards and forwards compatibility -- which is not a concern for IPC where you control both ends. So no real optional or required fields. Comparing structures (Go) is awkward and uses reflection. Structs embed a mutex and preserve bytes for unknown fields etc...
MessagePack seems to make a better set of tradeoffs by comparison.
They don't specify what types of issues they had with FFI which makes me somewhat dubious of the outcome here. There are tools to help paper over the rough edges of calling code through FFI, at least with Rust. Maybe the reasoning is more to do with the Go side.
From my experience, the issue is generally related to passing complex data structures across FFI boundaries. For this, adopting a serialization format is nicer than say deconstructing each object into proper parameters, which really end up being very similar code to that which is generated with the serialization libraries.
> since you can't statically link glibc due to licensing issues
Sure I can. Maybe YOU can't, due to the license you have chosen, but just say so.
Sure I can. Maybe YOU can't, due to the license you have chosen, but just say so.
That's pedantic. That's exactly what they wrote: due to licensing issues.
"You" here is a generic pronoun, not "you, reader"
"You" here is a generic pronoun, not "you, reader"
Not only that, but beyond the licensing issues it's really not supported by glibc itself because glibc will dlopen NSS and iconv, which link to libc, which leads to glibc being dynamically linked anyway except now you have two copies of it, which can play havoc on shared resources (like standard streams).
It's an issue from a licensing point of view, but it's also a rather poor idea from a technical point of view.
It's an issue from a licensing point of view, but it's also a rather poor idea from a technical point of view.
> but it's also a rather poor idea from a technical point of view
on glibc's part.
on glibc's part.
Why is Vercel investing so much into their own thing instead of a tool like EsBuild that already has widespread usage with Vite?
Isn't this because the vast majority of existing codebases and know-how are in webpack? They're trying to supplant webpack with a simple --turbo and keep existing momentum.
AFAIK the JS ecosystem is moving onto Vite which uses esbuild and Rollup under the hood.
Moving on because webpack is bad and turbo repo is alpha, so yes Vite is the best of the bunch right now. But you asked what they were trying to do. Most codebases are still webpack, and only some new ones are using Vite. Significantly upgrading to a drop in webpack replacement, with performance and usability of Vite is more ideal, and that’s what they’re trying to do. Just migrate to Vite is like a version upgrade without backwards compatibility.
I recommend using buildpacks.io or gcr.io/distroless or scratch or others of the ilk.
Using alpine for achieving slim containers is a waste of time and frankly stupid.
Using alpine for achieving slim containers is a waste of time and frankly stupid.
Can someone help me understand if I missed something here. Did they negate all the hypothetical performance gains made from Rust by crippling it with JSON IPC? If so, the mental overhead of Rust memory management doesn't seem worth it.
1. increasing performances is not one of the reasons they give for switching, although
2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow
Also JSON is used when delegating to Go because the Rust binary does not handle the command anyway.
2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow
Also JSON is used when delegating to Go because the Rust binary does not handle the command anyway.
> 2. ultimately getting rid of cgo would probably help a lot anyway, cgo calls are pretty slow
It’s slow compared to a function call in C, but it’s still measured in nanoseconds. If you’re not calling C functions from a tight loop in Go it’s unlikely to be noticeable.
It’s slow compared to a function call in C, but it’s still measured in nanoseconds. If you’re not calling C functions from a tight loop in Go it’s unlikely to be noticeable.
Yes, there's overhead in starting a new process to "just call a function", but I think this approach is still underutilized.
[0]: https://github.com/vercel/turbo/blob/c0ee0dea7388d1081512c93...