Why Discord is switching from Go to Rust (2020)(blog.discord.com)
blog.discord.com
Why Discord is switching from Go to Rust (2020)
https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f
281 comments
+1 going from meme language to meme language seems highly suspicious. Go and Rust are fine languages, but given these are relatively new and immature, I find it startling that Discord had to do writes/rewrites so often. I'd wager that if Discord had started sooner than 2015 (say, 2013ish) - we'd have seen Scala thrown into the fray as well.
In Discord’s defense, their product’s scope and userbase has increased orders of magnitude over just a few years. Setting language choice aside, one would expect several rewrites of core code and infra over this time.
My intention was to not set the language choice aside, especially where rewrites are squarely targeted at migrating to a different language.
Rewrites of services to decomp into different services, migrate to different datastores to scale up etc. is completely justified and is a good problem to have.
Rewrites of services to decomp into different services, migrate to different datastores to scale up etc. is completely justified and is a good problem to have.
Yup this makes me wonder why do companies go for relatively newer languages compared to old ones such as Java,C++ or C as they have been matured and has been widely used
Go was announced in 2009, and went 1.0 in 2012. Sure, it's "newer" than languages from the 90s, but calling it "new" when it's nearly a decade old doesn't pass the smell test. Serious software has been written in both Go and Rust, and real issues with Java, C++, and C that these newer languages address are well known.
Compare Go and C++, and I bet C++ has had more language changes since Go's 1.0 than Go has. That makes large portions of C++ "relatively new and immature" compared to Go.
Compare Go and C++, and I bet C++ has had more language changes since Go's 1.0 than Go has. That makes large portions of C++ "relatively new and immature" compared to Go.
I agree with you on that but this still makes me question the problem they are trying to solve can't they solve in Go instead of switching to Rust.
Furthermore, A lot of companies to this date have there performance critical code written in C++ , C or Java
Such as you can take HFT for example they use C++ for most of their trading applications instead of newer languages
I'm definitely not saying C++,C or Java is better than Rust or Golang each language has its own pros and cons
Furthermore, A lot of companies to this date have there performance critical code written in C++ , C or Java
Such as you can take HFT for example they use C++ for most of their trading applications instead of newer languages
I'm definitely not saying C++,C or Java is better than Rust or Golang each language has its own pros and cons
> I agree with you on that but this still makes me question the problem they are trying to solve can't they solve in Go instead of switching to Rust.
High performance systems in GC languages often preallocate or use object pools to reduce time spent in the garbage collector. LMAX's Disruptor, for example, allocates all memory at startup; no additional memory is allocated so the garbage collector never runs. Generational GCs make this strategy more effective because long-lived or large objects are traced rarely or not at all. Another option for trading systems is to run garbage collection after hours when the market has closed.
Go's GC is conservative, not generational, and has a hard coded collection interval of 2 minutes; you pay for a GC even if no memory has been allocated. There are only a few configuration options for Go's GC; if it doesn't work for your use case then you can either fork or use a different language.
High performance systems in GC languages often preallocate or use object pools to reduce time spent in the garbage collector. LMAX's Disruptor, for example, allocates all memory at startup; no additional memory is allocated so the garbage collector never runs. Generational GCs make this strategy more effective because long-lived or large objects are traced rarely or not at all. Another option for trading systems is to run garbage collection after hours when the market has closed.
Go's GC is conservative, not generational, and has a hard coded collection interval of 2 minutes; you pay for a GC even if no memory has been allocated. There are only a few configuration options for Go's GC; if it doesn't work for your use case then you can either fork or use a different language.
They did solve it in Go, then got performance improvements re-implementing a small piece in Rust.
This is more about scale and saving computing costs, than proving out a business idea or implementing new features.
This is more about scale and saving computing costs, than proving out a business idea or implementing new features.
Not all problems are specifically technical in nature.
One of the big issues with Go is that it's got a steep learning curve. (There was an article about this posted on HN yesterday, in fact)
Rust, by contrast, is an extremely easy language to learn. Of all the languages I've picked up at least partially in my ~12 years of professional programming, Rust was probably second only to to PHP in "get up and start running with it".
If Discord is trying to pick up new developers, or cross discipline seniors to improve their talent pool, Rust is definitely a better choice than Go.
One of the big issues with Go is that it's got a steep learning curve. (There was an article about this posted on HN yesterday, in fact)
Rust, by contrast, is an extremely easy language to learn. Of all the languages I've picked up at least partially in my ~12 years of professional programming, Rust was probably second only to to PHP in "get up and start running with it".
If Discord is trying to pick up new developers, or cross discipline seniors to improve their talent pool, Rust is definitely a better choice than Go.
Not to discount your experience, because I think that this topic has a lot of nuance and is very different on an individual level, but most people regard Go as easy to learn, and Rust as difficult to learn.
I wish we had a good way to evaluate this question in a quantitative way, but I haven't seen one yet. So reasonable people are going to disagree on the answers here.
I wish we had a good way to evaluate this question in a quantitative way, but I haven't seen one yet. So reasonable people are going to disagree on the answers here.
This is the reverse of the characterization of the two languages I usually see. I usually see "Go is easy to learn and use" and "Rust is hard to learn but has performance advantages". I often see comments about how you "really need to go read the Rust book" and not just try to pick up Rust on the fly.
This is the first time i have seen this argument. Just understanding ownership seems more complex to me than writing a simple web app in go or PHP for that matter.
I am sure rust is great but it is at the very least more difficult to learn then go.
I am sure rust is great but it is at the very least more difficult to learn then go.
to this day I can't wrap my head around the need to unwrap almost every single variable/function/whatever.
I don't get Rust. I got Go very easily. Rust just doesn't make sense for me.
In Go I can just write down my thoughts and they translate to code easy. I don't even know which mental game I have to play in Rust. And all the conventions in Rust are another step. The final nail is the un-community of Rust. I still have no idea how to write a simple RESTful http server with a mongodb backend. I've asked only to have been met with downvotes on Reddit. The official MongoDB discourse didn't answer either.
Time and again I want to do something with it but every time I just quit frustrated and irritated of the complexity of everything in Rust land.
I don't get Rust. I got Go very easily. Rust just doesn't make sense for me.
In Go I can just write down my thoughts and they translate to code easy. I don't even know which mental game I have to play in Rust. And all the conventions in Rust are another step. The final nail is the un-community of Rust. I still have no idea how to write a simple RESTful http server with a mongodb backend. I've asked only to have been met with downvotes on Reddit. The official MongoDB discourse didn't answer either.
Time and again I want to do something with it but every time I just quit frustrated and irritated of the complexity of everything in Rust land.
> to this day I can't wrap my head around the need to unwrap almost every single variable/function/whatever.
Think of Result as (foo, err) bundled into one thing. result.unwrap() is then equivalent to:
Don't get me wrong, there are complicated things in Rust, I just don't see how Result-based error handling is one of them. This is also not a new idea, many languages have something similar.
Think of Result as (foo, err) bundled into one thing. result.unwrap() is then equivalent to:
if err != nil {
panic(err)
}
if err != nil {
return ..., err
}
Don't get me wrong, there are complicated things in Rust, I just don't see how Result-based error handling is one of them. This is also not a new idea, many languages have something similar.
Because excited developers generally means more satisfaction from work. And one way to keep developers excited is to give them new toys to play with.
Also mature languages, even though they're more stable, are just more of a pain to work with since they don't have a lot of the new shortcuts.
For something like Discord, Rust is perfect.
Also mature languages, even though they're more stable, are just more of a pain to work with since they don't have a lot of the new shortcuts.
For something like Discord, Rust is perfect.
Crap developers get excited by new tech. Good devs want to get problems solved. If a new tech is the best solution then use shiny new tech. Usually it´s not the best choice.
What a ridiculous statement. You can be excited by new tech and realize how it can solve your problems.
And you can be maintaining a system built by developers who got excited by shiny new tech and made a load of inappropriate technology choices because they were flavor of the month and looked good on their resumes.
It sounds like you've had bad experiences with some devs and you're applying that to everyone.
Maintaining a codebase written in C is far far harder than one written in Java and that's far harder than one written in Go.
Maintaining a codebase written in C is far far harder than one written in Java and that's far harder than one written in Go.
I have worked in tech long enough to see some common patterns. It's usually intermediate devs (though not always) who get excited by new tech. The same people are usually pushing to use shiny new tech in places where it isn't needed. You don't get cynical old school devs trying to replace working parts of the tech stack with tech less than 5 years old.
Personally i think developer skill has far more to do with maintainability than language choice. And I have worked with a lot of shit code.
Personally i think developer skill has far more to do with maintainability than language choice. And I have worked with a lot of shit code.
New languages lack most of what is needed to be highly productive. Tooling is minimal to none. Libraries are sparse and still catching up, this takes many years. Many gotchas remain undiscovered and will bite you in production.
To get actual product work done with optimal efficiency, a mature ecosystem will always win.
To get actual product work done with optimal efficiency, a mature ecosystem will always win.
I agree with you, developers do get excited when are using newer technologies but the question is what if a new problem arises and they might not be able to find a to it solution.
Just like how they did in this case completely switching from Go to Rust
Just like how they did in this case completely switching from Go to Rust
Then they'll have to cross that bridge when they come to it. Nothing of this article gave me the impression that the second they saw this bug they're like WELL TIME TO SWITCH.
I'm sure we're glossing over probably hundreds of hours of investigations by dozens of people.
I'm sure we're glossing over probably hundreds of hours of investigations by dozens of people.
We kept dealing with GC pauses in go, and eventually after investing a bunch of time trying to fix this - we decided to research what it'd be like to rewrite it in another language.
The service itself is pretty simple, so it wasn't a gargantuan task. So it was either, rewrite this again in Go (using off-heap data structures to avoid GC pauses, which makes the software both harder to maintain, and also less idiomatic) or try a new language where we won't have these issues - where we don't have to resort to clever hacks to work around the runtime system.
Since then we've continued our investment in Rust, after success of this project (and many others we've yet to blog about as well.)
The service itself is pretty simple, so it wasn't a gargantuan task. So it was either, rewrite this again in Go (using off-heap data structures to avoid GC pauses, which makes the software both harder to maintain, and also less idiomatic) or try a new language where we won't have these issues - where we don't have to resort to clever hacks to work around the runtime system.
Since then we've continued our investment in Rust, after success of this project (and many others we've yet to blog about as well.)
[deleted]
Excitement < the countless hours required to rewrite a complex system in a non-trivial language.
And do you have numbers to back up excitement of devs vs their productivity?
It seems like everyone is just applying their subjective view. Which is fine. Just don't assume your equation works for everyone and every business.
It seems like everyone is just applying their subjective view. Which is fine. Just don't assume your equation works for everyone and every business.
The simplest answer is that these newer languages were created specifically to address problems with existing languages. If a new language has no compelling reason for anyone to switch to it, why bother to create it in the first place?
It's not that easy. A new language starts because a set of developers has a problem, however that doesn't mean that others share that problem or agree to the choices and solutions made. Rust seems to be in a good situation where this worked for large parts, but many languages came and got no or only short attention.
There are actually very good reasons to go with Rust over those languages, which were the motivating reasons Rust was developed in the first place. Rust supports lower-level development than Java, offers much stronger safety and correctness guarantees than C or C++. This is why Google's security team also supports reimplementing C libraries in Rust (https://security.googleblog.com/2021/02/mitigating-memory-sa...).
Have you ever considered that programming language design is an evolving field, and programming languages directly influence programmer productivity?
C++, C and Java are old, so some mistakes can be forgiven. However there are problems with these languages that new languages have addressed.
Do you think a car from 1985 is better than one from 2015? The same applies to programming languages. Given the explosion of programming, it is absurd that the best ideas from PLT are not in mainstream languages.
C++, C and Java are old, so some mistakes can be forgiven. However there are problems with these languages that new languages have addressed.
Do you think a car from 1985 is better than one from 2015? The same applies to programming languages. Given the explosion of programming, it is absurd that the best ideas from PLT are not in mainstream languages.
> programming languages directly influence programmer productivity
A complete re-write (preceded by a very deep analysis) is exactly the opposite of productivity.
> problems with these languages that new languages have addressed.
And new problems were introduced... case in point: this very article.
> Do you think a car from 1985 is better than one from 2015?
Depends on the car...
I think this analogy would be better: "Do you think a hammer and screwdriver from 1985 are better than the ones from 2015?"
A complete re-write (preceded by a very deep analysis) is exactly the opposite of productivity.
> problems with these languages that new languages have addressed.
And new problems were introduced... case in point: this very article.
> Do you think a car from 1985 is better than one from 2015?
Depends on the car...
I think this analogy would be better: "Do you think a hammer and screwdriver from 1985 are better than the ones from 2015?"
I definitely see the appeal of Java (used to be my main lang), but I would never in a million years base a company's flagship product on C or C++ unless it was a domain requirement (embedded, kernel, etc.). It's just not safe.
There's a difference between starting new projects and rewriting old ones. I have 7+ years of professional C++ under my belt and probably wouldn't start a new project in it. Same with Java. Sometimes new things are better.
Easier to hire people working in the new hot languages, and get younger programmers out of college.
At the same time, in college we are teaching our students Java and C++ because that's what companies expect our students to know. Half our CS curriculum is in Java. The students I teach loathe C and C++ and want to learn Rust, but we won't teach it to them except in an experimental elective that won't count toward their graduation. So it's hard to say how many young programmers out of college actually know these "new hot languages".
I totally agree with you on this comment since as I college graduate I learned Java , C++ ,C and python but I was exposed to Golang and Rust through online discussion forums.
Plus most of the organizations still write code in matured languages such as C,C++ and Java.
Plus most of the organizations still write code in matured languages such as C,C++ and Java.
I don't think so. There is a full load of people with experience in C++ or Java, but people with experience in performance critical (whatever that means) Rust code is more limited for being a newer language. You however win on the hipness scale and attention in places like hacker news.
As others have said, however, a big factor certainly is that an internal lead developer lobbied for Rust, which they experimented with.
As others have said, however, a big factor certainly is that an internal lead developer lobbied for Rust, which they experimented with.
> we'd have seen Scala thrown into the fray as well.
Scala has already found its niche in Big Data, streaming and distributed systems. Usually, the stack that you can find in my area, consist mostly of Java codebases, sprinkled with some Scala services that are purely data-oriented. There's a high probability that Discord is already using Scala.
Scala has already found its niche in Big Data, streaming and distributed systems. Usually, the stack that you can find in my area, consist mostly of Java codebases, sprinkled with some Scala services that are purely data-oriented. There's a high probability that Discord is already using Scala.
> I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language.
I would say it's cool _because_ of the technical reasons, though! You make it seem like people are inventing technical reasons to make rust seem cool. I could make a long "technical reasons" list why I would personally prefer to work in Rust, but it boils down that it makes me more productive, and the code more performant with fewer gotchas and less sacrifices/compromises (compared to say C++ or C). Do I think Rust is cool? I guess... But that which attracted me to Rust was there already; I didn't have to make it up[1].
I understand not every language is for everyone. I've had colleagues dismiss rust simply because they prefer C++ (and we work with C++). Some will even agree Rust is better when presented with technical reasons, but still not like Rust "just because."
> I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?
I write Go intermittently since I participate a little in the Kubernetes ecosystem, and haven't dwelled that deeply into it. How would this be done without reaching for C, though? Or would it be fitting to swap languages in this "subset" of the implementation?
[1]: I feel like "make up" is the wrong phrase here, but I can't think of a better one right now. I guess what I want to say that Rust's "cool" reputation is well deserved. I don't think the technical reasons are made-up to serve as office-politics cannon fodder.
I would say it's cool _because_ of the technical reasons, though! You make it seem like people are inventing technical reasons to make rust seem cool. I could make a long "technical reasons" list why I would personally prefer to work in Rust, but it boils down that it makes me more productive, and the code more performant with fewer gotchas and less sacrifices/compromises (compared to say C++ or C). Do I think Rust is cool? I guess... But that which attracted me to Rust was there already; I didn't have to make it up[1].
I understand not every language is for everyone. I've had colleagues dismiss rust simply because they prefer C++ (and we work with C++). Some will even agree Rust is better when presented with technical reasons, but still not like Rust "just because."
> I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?
I write Go intermittently since I participate a little in the Kubernetes ecosystem, and haven't dwelled that deeply into it. How would this be done without reaching for C, though? Or would it be fitting to swap languages in this "subset" of the implementation?
[1]: I feel like "make up" is the wrong phrase here, but I can't think of a better one right now. I guess what I want to say that Rust's "cool" reputation is well deserved. I don't think the technical reasons are made-up to serve as office-politics cannon fodder.
Agree, Rust seems like a good choice but I'm curious why they needed to replace Go. GC pause time is mostly due to the amount of pointer-containing heap memory. Moving the the cache storage to slabs of pointer-free "dumb" slices should pretty much avoid the problem so long as you force GC frequently (large heap will confuse the heuristic). Shard the slabs across a handful of goroutines, one channel each to handle locking, and that should perform fine compared to an equivalently complex Rust implementation.
I skimmed through this https://blog.kraken.com/post/7964/oxidizing-kraken/ yesterday
now its php->rust so a widely different migration but rust was measured for its latency cutting value, not just fad factor
I'd believe discord and others are on rust for this and not just the new shiney (that would be to their loss)
now its php->rust so a widely different migration but rust was measured for its latency cutting value, not just fad factor
I'd believe discord and others are on rust for this and not just the new shiney (that would be to their loss)
This is my experience too. I've seen a lot of decisions made due to personal preference, at considerable technical and financial expense.
The most recent example was a replacement for a complex YAML DSL, served by spring config sever and interpreted by a python program on every instantiation. This thing was literally using 20% of our CPU cycles over the whole compute cluster and estimated to cost around €6k per month. Long story short I provided a fully working pre-processing solution with effectively no runtime cost, no dependency on the spring config server and a more powerful & type-safe DSL. It was dhall with a bit of bash and python, although you could also have used jsonnet or any configuration language. Another developer with more clout pushed me out in favour of their own solution, which I believe was to keep me away from his political pedestal. A few months later I left and from what I've heard the company is still burning 20% of it's compute budget on this...
The most recent example was a replacement for a complex YAML DSL, served by spring config sever and interpreted by a python program on every instantiation. This thing was literally using 20% of our CPU cycles over the whole compute cluster and estimated to cost around €6k per month. Long story short I provided a fully working pre-processing solution with effectively no runtime cost, no dependency on the spring config server and a more powerful & type-safe DSL. It was dhall with a bit of bash and python, although you could also have used jsonnet or any configuration language. Another developer with more clout pushed me out in favour of their own solution, which I believe was to keep me away from his political pedestal. A few months later I left and from what I've heard the company is still burning 20% of it's compute budget on this...
Well, on the positive side a company like Discord moving to Rust means better tooling than what is currently available. So that's exciting.
Better tooling for the programmers you mean? I don’t regularly program in either language (Go or Rust), but I thought Google ran a pretty tight ship on good Go tooling, except with a bit of module SNAFU in its past.
Go's tooling is pretty nice. I was talking about better tooling for Rust. Maybe JetBrains will make a Rust-specific IDE.
So far though rust-analyzer >>>>> JetBrains' plugin
So far though rust-analyzer >>>>> JetBrains' plugin
i believe this also often happens when original designers of the systems are out and not involved anymore, the new team wants to rewrite everything from scratch but that's hard to sell. So they find few technical reasons and pick some latest fun tech to play with.
Yeah, unfortunately in some cases is easier to sell rewriting to a new language, than just refactoring/rewriting in the current stack.
Right, specially because last thing I read Discord was using Elixir: https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
They are using both https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
[deleted]
Ham Sandwich Development
> I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it
Ah yes, because what I really want out of an ostensibly memory-safe language like go is to have to resort to C, or what amounts to C, to get anything done. Or I could use a language which doesn't have this garbage issue. But that would just be political nonsense apparently.
Ah yes, because what I really want out of an ostensibly memory-safe language like go is to have to resort to C, or what amounts to C, to get anything done. Or I could use a language which doesn't have this garbage issue. But that would just be political nonsense apparently.
I’m not saying that it’s a perfect solution, and yes, using C memory defeats the purpose of memory safety. But it’s a trade-off, as most engineering things are. Rewriting in Rust is also a gigantic trade-off along many many more axes, and Rust isn’t immune to people using `unsafe` in practice either. In both Go and Rust, it’s possible to carefully engineer so as to isolate where sources of memory unsafety are, either through the language or through principled policy and software design.
You make an excellent point i just wanted to correct your usage of "unsafe" in rust: It does not signify "im writing c" now as many invariants are still upheld in Rust even with the usage of unsafe. Id say its still much "safer" than using unmanaged memory in go as you have far more facilities you may use and more invariants are upheld.
Also, they didn't have to do this.
I really am glad Discord did not go with this route. It's a chatting application, which means you're dealing with a TON of user generated content. Unmanaged memory and user generated content is an absolutely terrible idea.
...?
The article is about a very specific service handling state for if you've seen a message in a channel. It has nothing to do with processing user content.
The article is about a very specific service handling state for if you've seen a message in a channel. It has nothing to do with processing user content.
The problem with go here is (a) garbage collection, such that rust likely wouldn't have needed any unsafe outside of well-known libraries; and worse (b) when you need to do stupid C shit in go, there's no `unsafe` to contain it; it just spews over the rest of your code.
"Principled policy and software design" is exactly what's needed to get rid of memory bugs in C too! So we can conclude that these things are impossible, or else we would not need memory safe languages in the first place. Since it's increasingly apparent that go requires these mythical safeguards after all, it's safe to say that it is not a memory safe language in the useful sense.
"Principled policy and software design" is exactly what's needed to get rid of memory bugs in C too! So we can conclude that these things are impossible, or else we would not need memory safe languages in the first place. Since it's increasingly apparent that go requires these mythical safeguards after all, it's safe to say that it is not a memory safe language in the useful sense.
In general, I think this all-or-nothing attitude paired with FUD language is detrimental to good engineering. Memory unsafety is generally bad, I get it. But it’s exaggerating to say that these things “spew” all over codebases, or that you’re entering some inescapable memory hell that has no way to be treated properly. The original proposal was to manage some memory yourself within Go to solve one very specific problem with a very specific module of code. That’s not a lot of surface area and has a clear API. It doesn’t seem unfathomable that qualified, highly paid engineers can’t do a good job solving that problem comprehensively without rewriting everything else.
I think the question is why not rewrite just the LRU cache in C/C++/Rust (whatever floats your boat), as opposed to rewriting everything in Rust.
One reason would be that doing that would require switching over to cgo, at which point you deal with the PITA that is an FFI, and the PITA that is cgo (it's easy to enable but it drastically alters your "game plan").
And here the cache is essentially the entirety of the micro-service:
> [The service's] sole purpose is to keep track of which channels and messages you have read.
so it's a giant map of (user, object) -> read flag or something along those lines. The DB commit and service input would be very light, and not really your main concern.
And here the cache is essentially the entirety of the micro-service:
> [The service's] sole purpose is to keep track of which channels and messages you have read.
so it's a giant map of (user, object) -> read flag or something along those lines. The DB commit and service input would be very light, and not really your main concern.
The question I’d have is “why use a custom cache at all”, this sounds like a problem that has a whole pile of battle-tested off-the-shelf solutions.
The original implementation of this "service" was in Redis using lua scripts.
We hit the performance ceiling of redis, and rather than deciding to go with redis-cluster, and dealing with the headache of managing persistent redis, we went this way, so we could consolidate data storage to scylladb/cassandra (it was cassandra at the time of rewrite, scylla now since we've migrated our data stores over).
We hit the performance ceiling of redis, and rather than deciding to go with redis-cluster, and dealing with the headache of managing persistent redis, we went this way, so we could consolidate data storage to scylladb/cassandra (it was cassandra at the time of rewrite, scylla now since we've migrated our data stores over).
any reason to not consider Aerospike for Cache?
We don't have any operational experience with Aerospike.
We have a bunch of operational experience with C*/ScyllaDB.
We have a bunch of operational experience with C*/ScyllaDB.
That sounds like basically what they did; they rewrote the ReadStates service in Rust and it sounds like that's basically a wrapper around an LRU cache. The rest of Discord is still Go/Elixer/whatever else they use.
This is a fine suspicion to have. It’s a fine theory, even. But why raise this suspicion for this particular neo-language case? Why not raise it across the board?
Such a theory is more robust if it predicts how the organization will work in general. Thus one would also be partial to suspecting that neo-languages are declined for political reasons as well -- that a few powerful voices simply prefer to find other solutions, and then find post-hoc technical arguments.
Such a theory works less well if it is only used to explain some particular (ostensibly) technical decisions.
Such a theory is more robust if it predicts how the organization will work in general. Thus one would also be partial to suspecting that neo-languages are declined for political reasons as well -- that a few powerful voices simply prefer to find other solutions, and then find post-hoc technical arguments.
Such a theory works less well if it is only used to explain some particular (ostensibly) technical decisions.
GO promised to be very fast, but if you check the actual benchmarks:
https://www.techempower.com/benchmarks/#section=test&runid=8...
High-level C# (asp.net) is almost twice as fast as GO in the benchmark... Rust is also in the top 3.
So, why should I use GO instead of C#, if ASP.NET/C# is so blazing fast and requires a lot less lines of code to achieve the same? -> It makes absolutely sense to switch to C# or Rust.
https://www.techempower.com/benchmarks/#section=test&runid=8...
High-level C# (asp.net) is almost twice as fast as GO in the benchmark... Rust is also in the top 3.
So, why should I use GO instead of C#, if ASP.NET/C# is so blazing fast and requires a lot less lines of code to achieve the same? -> It makes absolutely sense to switch to C# or Rust.
Don't look at that kind of benchmark, most cheat, and notice how latency is bad, because C# is not native code, so the JIT needs warmup just like java, wich introduces hicups and non-deterministic performance
So these perfect situations where you only do the same thing and call same code over and over never happen; never!
Anyways, GO isn't popular for its RAW perf, GO is popular because:
- Simple language
- Native code
- GC
- Cross compilation
- Fast iteration (build/rebuild/deploy)
- Relatively tiny, statically compiled executable
So these perfect situations where you only do the same thing and call same code over and over never happen; never!
Anyways, GO isn't popular for its RAW perf, GO is popular because:
- Simple language
- Native code
- GC
- Cross compilation
- Fast iteration (build/rebuild/deploy)
- Relatively tiny, statically compiled executable
> - Simple language
So simple it can't do Object.keys()
Don't get me wrong, I enjoy writing code in Go, Rust, and TS. But my god, Go literally is a pain in the ass every time I want to do something that isn't a map reduce problem. And even for map reduce problems it's annoying due to lack of generics.
Maybe generics will finally give Go something that every other language has. A strong, useful, standard library for collections.
So simple it can't do Object.keys()
Don't get me wrong, I enjoy writing code in Go, Rust, and TS. But my god, Go literally is a pain in the ass every time I want to do something that isn't a map reduce problem. And even for map reduce problems it's annoying due to lack of generics.
Maybe generics will finally give Go something that every other language has. A strong, useful, standard library for collections.
I agree with you, i'm not using Go myself, i tried it many times but i always hit a roadblock, it's interesting because i hit the same kind of issues with ZIG
Simple on paper, but what's the experience as a result? some are okay with it, some love the restrictions and limitations, but i personally don't enjoy them
But again, i can understand the choices and the benefits
Simple on paper, but what's the experience as a result? some are okay with it, some love the restrictions and limitations, but i personally don't enjoy them
But again, i can understand the choices and the benefits
Neither Go nor Zig are spelled ALLCAPS
I mean those benchmarks are mostly web server related. And it's pretty common in web servers to trigger the same code path over and over again all the time. There's a reason why people are not rewriting their java web server application in something native to get more performance out of it.
These benchmarks also in no way represent actual webapps. Have you seen this code? Have you ever written a webapp where not using hardcoded, static headers like Content-Length was a bottleneck?
the plaintext c# one actually only sets a content length manually since without it it would be a streaming response (Transfer-Encoding: chunked), which would not make the server slower, but the client, because of buffers (also as far as I know the Content-Length needs to be set in this benchmark). btw. go does not do that when calling w.Write without wrapping the responsewriter inside a buffer (which is probably also the case why it is slower, since it needs to know the size of the written response before it goes over the network and btw. only go-std is slow because it's written for simple use cases, there are others like fasthttp who do stuff a little bit more like c# and thus are faster (even inside ;-) go-std never cared about being the fastest.)
> the JIT needs warmup just like java
You can use ReadyToRun[0] to pre-JIT portions of the code for faster startup. Full AOT compilation is in the works too[1] albeit it's still experimental.
[0]: https://docs.microsoft.com/en-us/dotnet/core/deploying/ready...
[1]: https://github.com/dotnet/runtimelab/tree/feature/NativeAOT
You can use ReadyToRun[0] to pre-JIT portions of the code for faster startup. Full AOT compilation is in the works too[1] albeit it's still experimental.
[0]: https://docs.microsoft.com/en-us/dotnet/core/deploying/ready...
[1]: https://github.com/dotnet/runtimelab/tree/feature/NativeAOT
I think you're missing the point. The point is that you can get all those properties in Rust without the performance hit. (Except for, well, GC, obviously)
Rust does not build fast. It’s more comparable to C++ if not worse.
The problem with rust is the language is kind of scary to learn, once you get past the learning curve, it is okay i guess
And on top of that iteration time is hurt because of both the language, the compiler perf, and the lifetime problems that you need to constantly think about
The benefits of using rust on other hand is non-negligible, i wish the team would focus on solving that kind of immediate problems
And on top of that iteration time is hurt because of both the language, the compiler perf, and the lifetime problems that you need to constantly think about
The benefits of using rust on other hand is non-negligible, i wish the team would focus on solving that kind of immediate problems
Lack of garbage is a pro, not a con.
> the JIT needs warmup just like java
That doesn't sound right. .Net has good support for caching of generated native code. It's well ahead of Java in this regard.
That doesn't sound right. .Net has good support for caching of generated native code. It's well ahead of Java in this regard.
Did you even read the article? That's exactly what their code is doing. It's a giant LRU cache on a very hot path.
Performance is rarely the top priority for a software engineering project. If you task a whole team to switch from Go to Rust or C# purely because “Rust goes brrr” you’re going to have a bad time.
I use Go because its decently fast, I like the syntax, has a great stdlib, and I feel like it’s very easy to share a Go codebase with other engineers.
I use Go because its decently fast, I like the syntax, has a great stdlib, and I feel like it’s very easy to share a Go codebase with other engineers.
One of the key things I like about Go, despite shortcomings, is your last point. Our team effortlessly shares Go code, and its tooling makes it so any of us at any time can easily compile and run projects without hassles or hang ups.
The other day we reviewed some oldish and complex code and it was so plain to see what it was doing and get back into the flow of that logic. I don’t find this happens with other languages as much.
I’m not saying Go is the right tool everywhere for everyone. I do love that about it though.
The other day we reviewed some oldish and complex code and it was so plain to see what it was doing and get back into the flow of that logic. I don’t find this happens with other languages as much.
I’m not saying Go is the right tool everywhere for everyone. I do love that about it though.
Definitely agree with this. Go is one of the only languages I've worked in where I can jump into _any_ project and it feels familiar, I just have to dig around to get into the specifics.
I have never had that same familiarity from Python, Ruby, JavaScript. I think there's something to be said about that in Go's favor.
I have never had that same familiarity from Python, Ruby, JavaScript. I think there's something to be said about that in Go's favor.
While I agree with you on the familiarity across different projects (and I applaud ideas like gofmt) - it's also very verbose code (due to the simplicity of the language).
That makes it very hard to read go code and build a mental model in your head.
That makes it very hard to read go code and build a mental model in your head.
> it's also very verbose code (due to the simplicity of the language).
That makes it very hard to read go code and build a mental model in your head.
Verbose does not equal complex usually.
Something that's a single method call on a built-in type may be 3 lines in Go. This doesn't make it hard to understand or keep in your head. "This iterates over a map" is the same regardless of the lines of code. You can very quickly skim verbose code, especially in a language like Go, where idioms and conventions are identical across many code bases.
Verbose does not equal complex usually.
Something that's a single method call on a built-in type may be 3 lines in Go. This doesn't make it hard to understand or keep in your head. "This iterates over a map" is the same regardless of the lines of code. You can very quickly skim verbose code, especially in a language like Go, where idioms and conventions are identical across many code bases.
Agreed, somewhere else in this thread someone complained that Go doesn't have an object iterator function and I really don't understand how k, v := range <map> isn't an intuitive approach, especially when range is so common in Go. Sure, it might be two more lines of code but it's not like it any more difficult to reason about or mentally track.
I agree that Go is a little bit more verbose, but I usually consider that an example of "explicit is better than implicit" and I like that Go shows me the internals a lot more than other languages. The one thing that really drew me to Golang over any other was that it was extremely _easy_ for me to build a mental model of fairly complex code.
Go's interfaces in particular feel like a really easy way to build clean, composable abstractions that make sense and hide complexity in the right way. The single-function-interface pattern is really one of my favorite things about Go lately.
Go's interfaces in particular feel like a really easy way to build clean, composable abstractions that make sense and hide complexity in the right way. The single-function-interface pattern is really one of my favorite things about Go lately.
We might want to highlight this more often that not.
Code legibility is starting to become a primary if not the primary factor for a lot of software these days, because that's what allows it to be maintained.
I think more analysis should be done on why this is for go. Java and C# should be similarly legible, they are not complex languages either. It's possible that go encourages certain paradigms.
Code legibility is starting to become a primary if not the primary factor for a lot of software these days, because that's what allows it to be maintained.
I think more analysis should be done on why this is for go. Java and C# should be similarly legible, they are not complex languages either. It's possible that go encourages certain paradigms.
This sounds great but thinking of my system at work, it´s not the language that is the problem. It´s all the moving parts that you need to get to play together nicely before you can even start to develop. Front end, back end, database, nginx proxies, email platforms.
Go has won here for us as well because it forces you to write very dumb code, resulting in exposing the logic of how systems interface in a very dumb but easy to understand way.
It may take some scanning to go through all the dumb code, but you typically won't get fooled by abstractions. Go offers so little in terms of abstractions that you can usually see right through them.
This won't be everyone's experience, but it has been mine very consistently across internal and 3rd party code.
It may take some scanning to go through all the dumb code, but you typically won't get fooled by abstractions. Go offers so little in terms of abstractions that you can usually see right through them.
This won't be everyone's experience, but it has been mine very consistently across internal and 3rd party code.
Performance is a cost concern; if I can pay for less server resources then I can either pay my programmers more or hire more programmers.
Databases and traffic is going to have your lunch anyway. Good developers cost a good deal more than extra infrastructure. Adding people to help you cut cloud cost is easier and more predictable than re-implementing.
Bad developers will cost you more.
Discord also runs in cloud. They could save $$$ by just going to the colo instead
I doubt that, with that many users they would probably need to build their own stuff, around the globe. their servers have such a low latency, that this would be impossible by just going "colo".
peope who say that they can just save money in their colo are probably never even working in the scale of discord, which is fine.
peope who say that they can just save money in their colo are probably never even working in the scale of discord, which is fine.
Then you’d probably be shocked to find out that they already do that for voice servers. I’m talking about the other part which doesn’t have to have many geo distributed locations only a couple
[deleted]
I think that the argument here is that the c# environment is much bigger and that the programming level is at a higher level. That these are things that are often made at a sacrifice for performance but that that's not the case here, so why use Go at all
I like Go for all these reasons too, and the fact that vim and a Makefile are all we need to be very productive.
I am not saying Go is faster or anything I am just saying that take the grain of salt with reading the benchmarks as they are on same level as googling for "best microwave oven in 2021" where you know it is better not to even start reading the results as they will be full of paid "fake" reviews. Not to even start about microwave oven brand fanboys.
You have to understand that the programming language is a product in same way as anything else and there are influencers, marketing bribes, paid advertisements, forum/hn/fb/... paid spammers, etc.
Just remember what spam spike the Rust (I am again not talking about rust as a language... just about the noise everyone made) had here when it emerged, they threw the shadow on the Nigerian spam, oh the amount of noise from Rust-Distributed-Evangelist-Task-Force (they will probably downvote me to hell for mentioning this :D), I had a lot of fun reading HN at that time, the most used word was 'and', the second was 'rust', it was so exaggerated that it was like watching Monty Python.
In today world the only review, or if you want, benchmark, is the one you make. Nothing else counts and if you feel that C# is faster then enjoy using it.
And quite frankly, who cares about speed today (except some very strange people like me), people are using very strange languages and want to run them on backend. And they actually do. Ignore the benchmarks, no one cares (except those that would like to advertise themself like "faster than C" :D). Just threw in some more vapor on the cloud and you have "fixed" the speed issues. /s
You have to understand that the programming language is a product in same way as anything else and there are influencers, marketing bribes, paid advertisements, forum/hn/fb/... paid spammers, etc.
Just remember what spam spike the Rust (I am again not talking about rust as a language... just about the noise everyone made) had here when it emerged, they threw the shadow on the Nigerian spam, oh the amount of noise from Rust-Distributed-Evangelist-Task-Force (they will probably downvote me to hell for mentioning this :D), I had a lot of fun reading HN at that time, the most used word was 'and', the second was 'rust', it was so exaggerated that it was like watching Monty Python.
In today world the only review, or if you want, benchmark, is the one you make. Nothing else counts and if you feel that C# is faster then enjoy using it.
And quite frankly, who cares about speed today (except some very strange people like me), people are using very strange languages and want to run them on backend. And they actually do. Ignore the benchmarks, no one cares (except those that would like to advertise themself like "faster than C" :D). Just threw in some more vapor on the cloud and you have "fixed" the speed issues. /s
Have you seen the kind of C# code they write to compete in Tech Empower? It's nowhere near what one would write on a daily basis: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
Perhaps Go folks should just use inline assembly to demonstrate how useless these benchmarks are.
Perhaps Go folks should just use inline assembly to demonstrate how useless these benchmarks are.
This 1000x times. Should be posted every time someone links to TechEmpower benchmark results. And especially for the most contrived benchmarks like "Plaintext" and "JSON serialization", which are basically echo server competitions that have nothing to do with real applications.
That's nowhere near as bad as inline assembly. It might not be the usual way to write a web app, but it's normal C# with common libraries. Any C# dev could use that as an example if they ran into a major performance issue.
These benchmarks aren't intended to show the performance of everyday code. They show what experts can achieve on each platform. That's why they take pull requests instead of writing the code themselves from tutorials.
These benchmarks aren't intended to show the performance of everyday code. They show what experts can achieve on each platform. That's why they take pull requests instead of writing the code themselves from tutorials.
That's true. .NET Core high performance optimization looks more approachable.
I'd still rather see the popular way of doing things be the benchmark.
I'd still rather see the popular way of doing things be the benchmark.
Most code is not CPU-bound, and for code that is, a hot spot is all the usually needs to be optimized. Today .Net Core is remarkably performant, and the fact that just by writing code carefully, or using Span<> etc., you can get close to bare-metal perf with w/o resorting to FFI to unsafe code is a good thing. This is much easier than trying to keep your friendly Rust borrow-checker happy even for moderately complex data-structures with varying lifetimes.
When you put it that way .NET Core runtime does seem to offer great performance at low cost.
btw. these are the platform benchmarks of c# (only kestrel + manually writing the response, basically no framework code) the real ones are in benchmarks (aspcore-mw), which just use a middleware and are also really really fast. btw. golang's std http server is just really really slow. fasthttp solves that and is basically 6x faster in some real code cases.
If performance is the top priority and time to market can take a back seat, I skip Go and choose Rust (or other known-fast languages).
However, time to market is almost always a priority. Unless my margins are razor-thin, spending twice as much on servers for shorter development cycles is a net win in most cases.
However, time to market is almost always a priority. Unless my margins are razor-thin, spending twice as much on servers for shorter development cycles is a net win in most cases.
Go is as fast as dot net in the same benchmark you posted: https://www.techempower.com/benchmarks/#hw=ph&test=plaintext
aspcore 7,016,966
gnet 7,010,982
People share results of benchmark where some languages are missing.
aspcore 7,016,966
gnet 7,010,982
People share results of benchmark where some languages are missing.
It wasn't that long ago that compariing .NET to Go was insane: Go was far faster but now it's a toss-up in some cases. Go still wins on small size of binary but in cloud workloads does it really matter if my lambda in go is half the memory and half the execution time when the "bigger, slower" C# version is still under 128MB and executing only a few milliseconds slower? In cloud cost terms the difference is small enough not to matter.
It tooks years and a lot of work for them to make it on part or faster, Go did not focus on performance recently. Also you should take benchmark with a grain of salt, every recent language can be fast nowdays.
Why? Well, for one, Go produces a statically linked binary and C# requires a runtime be installed.
If we’ve learned nothing from Java and Python, it’s that managing runtimes and environment can be a huge pain point.
EDIT: swipe keyboard put ringtone instead of runtime and I missed it. Corrected.
If we’ve learned nothing from Java and Python, it’s that managing runtimes and environment can be a huge pain point.
EDIT: swipe keyboard put ringtone instead of runtime and I missed it. Corrected.
>C# requires a ringtone be installed.
This confused the hell out of me until I realized it's supposed to be runtime.
This confused the hell out of me until I realized it's supposed to be runtime.
Thanks. I assumed it was an allusion to the .Net runtime having a bunch of unnecessary crap built into it, including a ringtone for some arcane reason.
Autocorrect: turning simple, obvious mistakes into opaque, irrecoverable mistakes.
It's never been clear to me why someone would want this.
It's never been clear to me why someone would want this.
> Go produces a statically linked binary and C# requires a runtime be installed
Not true since .NET Core 3.
$ dotnet publish --self-contained --runtime osx.11.0-x64 -p:PublishSingleFile=true
Not true since .NET Core 3.
$ dotnet publish --self-contained --runtime osx.11.0-x64 -p:PublishSingleFile=true
Not disagreeing with you, but just to be pedantic:
In .NET Core 3, you can build and distribute a single-file executable which, when run, will actually unpack an executable and a ton of DLLs into a temp directory, and run it from there.
In .NET 5 they changed it, so that when you build a "single-file" executable, you actually get an executable plus a few DLLs, which must be distributed together.
In .NET Core 3, you can build and distribute a single-file executable which, when run, will actually unpack an executable and a ton of DLLs into a temp directory, and run it from there.
In .NET 5 they changed it, so that when you build a "single-file" executable, you actually get an executable plus a few DLLs, which must be distributed together.
You can do that with C#: https://docs.microsoft.com/en-us/dotnet/core/deploying/singl...
With C# and Java, we have learned that time and battle tested VMs can remove terrible pain points in proper hands.
To be fair, I have found Java’s JRE to be relatively painless. It’s fairly simple to have multiple installed and choose which one to use as a command line option.
Python OTOH...
Python OTOH...
One of the things to consider when comparing C# and Go is how it does parallelism. In C#, the new async/await mechanism can go horribly wrong if the programmer doesn't use it correctly. This can cause threads to start up, which is in fact slower than older .NET tech that had larger starting thread pools. You can view this with PerfView to see if it's your load issue.
In Go, the scheduler is smart enough to stop something even when it's not blocking on IO. This makes it more robust.
Of course, use what you need for when you need it. Rust definitely shines in some circumstances.
In Go, the scheduler is smart enough to stop something even when it's not blocking on IO. This makes it more robust.
Of course, use what you need for when you need it. Rust definitely shines in some circumstances.
> In C#, the new async/await mechanism […]
C# grew the `await` operator in C# 5.0, released in August of 2012. Go altogether hit 1.0 on 2012-03-28. If the one is new, the other is new.
C# grew the `await` operator in C# 5.0, released in August of 2012. Go altogether hit 1.0 on 2012-03-28. If the one is new, the other is new.
They key takeaway really is that for high performance, the way you think about memory layout, data structures, memory management, etc. all start to really matter. It is possible to write programs with memory pools to reuse data structures as much as possible without having to defer the work to the GC, which can then introduce performance spikes.
That said, C# and JVM are both mature and have decades of GC improvements, which could explain why golang is falling behind. Time will tell.
That said, C# and JVM are both mature and have decades of GC improvements, which could explain why golang is falling behind. Time will tell.
[deleted]
This was covered a year ago, and is literally the most popular post for "Rust" ever on this site https://news.ycombinator.com/item?id=22238335
This, just realized that post was dated Feb 2020 not Feb 2021.
[deleted]
Original author here. Cool to see it posted again.
We're still using and investing in rust at Discord. For example, in the past year we successfully pulled part of the core messaging system out of python and into a rust service. Rust has been a boon for developer productivity and correctness of code.
We've also been very happy with the way the rust ecosystem is evolving and the amount of adoption the language has seen.
I'm happy to answer questions here.
P.S. We're still growing and hiring (including remotely). You can find my email in my profile.
We're still using and investing in rust at Discord. For example, in the past year we successfully pulled part of the core messaging system out of python and into a rust service. Rust has been a boon for developer productivity and correctness of code.
We've also been very happy with the way the rust ecosystem is evolving and the amount of adoption the language has seen.
I'm happy to answer questions here.
P.S. We're still growing and hiring (including remotely). You can find my email in my profile.
Something I rarely see asked would be: what's the maintenance story with rust?
Do you have metrics about the number of bugs compared to other languages, e.g. go, python?
When you have to add a new feature to an existing rust system, is it easier/harder/the same than with another language?
How easy is it to read old rust code? Obviously, you won't have 10 year old rust codebases, but even reading something I wrote a year ago can be difficult in, e.g. C++.
What happened with your Elixir stuff? ( https://blog.discord.com/scaling-elixir-f9b8e1e7c29b ). Is it still there or are you using something else now?
Elixir is without a doubt still one of the core languages. We try to use the right tool for the job and Elixir is the right tool for parts of our system.
We also combo Elixir NIFs and rust to speed up parts of the Elixir services https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
We also combo Elixir NIFs and rust to speed up parts of the Elixir services https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
Interesting. We are researching Elixir for an online transaction ledger (thousands of payments per second) along with kafka. So far thats the technology that seems more suitable. That discord article was really appreciated to understand some of the intricacies of using Elixir.
I certainly know of a few Adtech companies using Erlang to handle > 1M bid requests per seconds :). OTP is pretty great for that.
I'm happy to answer questions here.
I'm really glad this blog post is being discussed on HN. I've learned a lot. But there are also a few foolishly critical comments. You and jhgg should keep in mind: Illegitimi non carborundum.
I'm really glad this blog post is being discussed on HN. I've learned a lot. But there are also a few foolishly critical comments. You and jhgg should keep in mind: Illegitimi non carborundum.
Yeah, maybe if Discord didn't gather all user data and metadata it can on its users, just like its grandfather, OpenFeint.
is it true the GC problems in GO have been fixed since then?
rsc said they made some big improvements for this particular type of problem. I'm not sure which version of go the improvements landed in.
2021 will be about them switching back. All jokes aside, it's really common for engineers to randomly switch between solutions incurring huge costs for the companies they work for. Often as it is in this case with no gain for the company, other than some resume building for the involved team members. We use Go here at Stream and power activity feeds and chat for a billion end users. Works perfectly.
Cockroach also wrote about GC optimization: https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c...
For caching specifically there is this library as well: https://dgraph.io/blog/post/introducing-ristretto-high-perf-...
Nothing wrong with Rust, exciting project, but personally think Go is better for building performant micro services like this. Rust seems more suited for building browsers, drivers etc. Rust's performance is slightly better, but it's not as refined around go routines and developer productivity.
Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.
Cockroach also wrote about GC optimization: https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c...
For caching specifically there is this library as well: https://dgraph.io/blog/post/introducing-ristretto-high-perf-...
Nothing wrong with Rust, exciting project, but personally think Go is better for building performant micro services like this. Rust seems more suited for building browsers, drivers etc. Rust's performance is slightly better, but it's not as refined around go routines and developer productivity.
Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.
Not a chance. We are going hard on rust in 2021 after some very successful projects in 2019 and 2020. our engineers have ramped up on the language - and we have good support internally (both in terms of tools, but also knowledge) to see its success.
Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go.
Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands.
And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go.
Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands.
And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
I agree, rust is so much easier once you've passed that learning curve.
Not having to worry if this random reference you're using has been overwritten somewhere else in the code is absolutely refreshing.
It allows me to bring back the mental map of my code and the variables to the exact scope I'm writing in.
This is an underrated feature of rust. Especially for newer devs who do not have that mental map of the code yet, and are worried that any small change they make might cause cascading failures. That's just something you don't think about in Rust.
Not having to worry if this random reference you're using has been overwritten somewhere else in the code is absolutely refreshing.
It allows me to bring back the mental map of my code and the variables to the exact scope I'm writing in.
This is an underrated feature of rust. Especially for newer devs who do not have that mental map of the code yet, and are worried that any small change they make might cause cascading failures. That's just something you don't think about in Rust.
> rust is far easier
That's an unusual claim :) Could you elaborate? How is the learning curve, especially for fresh graduates who have been trained only on languages like C, Java and Python?
> The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
Agreed. I like Go, but that's big drawback of goroutines and threads.
That's an unusual claim :) Could you elaborate? How is the learning curve, especially for fresh graduates who have been trained only on languages like C, Java and Python?
> The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
Agreed. I like Go, but that's big drawback of goroutines and threads.
In my experience, Rust puts many costs upfront, which makes it harder to do stuff when you aren't used to doing it the Rust way, but makes it easier to get it right if you are used to Rust. It basically gives you a relatively strong guarantee that if your code compiles and has no warnings, it's mostly collect - yes, that includes logic errors to some extent thanks to the type system. That also makes refactoring easier - chances are, the bugs you possibly added by refactoring will get caught by the compiler. Obviously, that's not a complete replacement for testing, but it does reduce the amount of tests you need to write to ensure correctness.
Frankly, I don't consider myself a particularly good programmer - but when I use Rust, I feel confident because of being backed by the compiler. That's something I can't say about any other language, and I only have less than a year of Rust experience compared to 5 years of total experience programming.
Imo Rust is easier to master than other languages, but it's definitely harder to do basic stuff in it because of the upfront costs I mentioned.
Frankly, I don't consider myself a particularly good programmer - but when I use Rust, I feel confident because of being backed by the compiler. That's something I can't say about any other language, and I only have less than a year of Rust experience compared to 5 years of total experience programming.
Imo Rust is easier to master than other languages, but it's definitely harder to do basic stuff in it because of the upfront costs I mentioned.
> That's an unusual claim :) Could you elaborate? How is the learning curve, especially for fresh graduates who have been trained only on languages like C, Java and Python?
One way to look at it would be that Rust forces you to conform to a specific ownership model, and makes you be explicit about resource management and how things are shared and accessed. Having to do all that can make certain patterns awkward or hard to write, especially for those new to Rust who haven't internalized what the compiler wants yet.
But on the other hand because so much is explicitly spelled out it can end up reducing the amount of things you need to consider or keep track of while reasoning about code, making it easier. Especially for code which is essentially bolting libraries together with a bit of data transformation.
One way to look at it would be that Rust forces you to conform to a specific ownership model, and makes you be explicit about resource management and how things are shared and accessed. Having to do all that can make certain patterns awkward or hard to write, especially for those new to Rust who haven't internalized what the compiler wants yet.
But on the other hand because so much is explicitly spelled out it can end up reducing the amount of things you need to consider or keep track of while reasoning about code, making it easier. Especially for code which is essentially bolting libraries together with a bit of data transformation.
I've noticed this kind of reply on rust threads often. Someone says something like:
> Rust moves many potential program problems from runtime to compile time; it has a high bar to meet, but once you meet it the program is rock solid
And then somebody replies with:
> Oh my favorite language doesn't make me do all that work up front, therefore easier (TM). Why Rust so hard?
But the reply misses the point: Make no mistake, you're still paying the cost (with FASTCASHNOW-interest rates), but the cost is just smeared out over days or weeks as you fix, debug, and tune it during use. You may choose whether to do that work up front or while the program is in active use, but avoiding the work is not a choice.
> Rust moves many potential program problems from runtime to compile time; it has a high bar to meet, but once you meet it the program is rock solid
And then somebody replies with:
> Oh my favorite language doesn't make me do all that work up front, therefore easier (TM). Why Rust so hard?
But the reply misses the point: Make no mistake, you're still paying the cost (with FASTCASHNOW-interest rates), but the cost is just smeared out over days or weeks as you fix, debug, and tune it during use. You may choose whether to do that work up front or while the program is in active use, but avoiding the work is not a choice.
> That's an unusual claim
It's really... not. For people that have actually written big things in Rust. We keep repeating it, but we try to cautiously, tread carefully, lest we be accused of being part of this dreaded Evangelism Strike Force we keep hearing so much about.
Ease of use != ease of learning.
Rust is hard to learn (unless you have a deep ML/Haskell + C background). But once you're up the curve, it's a very expressive + productive language. Refactoring and maintaining large projects confidently is a breeze, and creating powerful abstractions that can be safely leveraged by large teams of programmers... just works.
It's really... not. For people that have actually written big things in Rust. We keep repeating it, but we try to cautiously, tread carefully, lest we be accused of being part of this dreaded Evangelism Strike Force we keep hearing so much about.
Ease of use != ease of learning.
Rust is hard to learn (unless you have a deep ML/Haskell + C background). But once you're up the curve, it's a very expressive + productive language. Refactoring and maintaining large projects confidently is a breeze, and creating powerful abstractions that can be safely leveraged by large teams of programmers... just works.
A language being easy to write has no relation at all with it being easy to learn.
How would a fresh graduate be able to write safe C code when even experienced people are unable to?
I have worked with multi-threaded C++ code long enough to see that it's impossible to work in a team while keeping the code safe. Usually servers crash and canaries catch it in production, but if we have type system improvements that solve these problem, why not take advantage of it?
I have worked with multi-threaded C++ code long enough to see that it's impossible to work in a team while keeping the code safe. Usually servers crash and canaries catch it in production, but if we have type system improvements that solve these problem, why not take advantage of it?
Those reasons make much more sense than the GC focus of the blogpost. Definitely a lot of nice features in Rust.
The GC was apparently the reason behind rewriting that particular service. No wonder there are more reasons when you look at all the services across the company.
But you couldn’t help yourself even after being corrected by someone who knows what they’re talking about.
But you couldn’t help yourself even after being corrected by someone who knows what they’re talking about.
When I read descriptions like yours, Rust sounds like the language delivering on the promises of Haskell in the Real World.
Some promises of Haskell.
Rust is still a very low level language, but if you like Haskell, you won't go into anything else when writing some low level program.
Rust is still a very low level language, but if you like Haskell, you won't go into anything else when writing some low level program.
Haskell overpromised a lot.
When I looked at it, the favourite quicksort example was so elegant, I wanted to learn the language. Then after I started looking at the run time (after wasting a lot of time going through learning Haskell), I realized that it was all a lie, ant that sort wasn't quick at all.
I don't see the same overpromising problems with Rust: you usually get the performance that you think you will get when you're writing the code.
When I looked at it, the favourite quicksort example was so elegant, I wanted to learn the language. Then after I started looking at the run time (after wasting a lot of time going through learning Haskell), I realized that it was all a lie, ant that sort wasn't quick at all.
I don't see the same overpromising problems with Rust: you usually get the performance that you think you will get when you're writing the code.
So let me get this straight: You don't like Haskell because someone used it to write an inefficient quicksort example online?
Not somebody, it was on the home page as a use case of how much better Haskell is than other languages.
The default libraries are extremely inefficient.
Generally if I use Python/Ruby I can get a 50x slow down that is accepted, but with Haskell it can be more, as it's very hard to think about what the compiler does (memory usage can blow up easily).
The default libraries are extremely inefficient.
Generally if I use Python/Ruby I can get a 50x slow down that is accepted, but with Haskell it can be more, as it's very hard to think about what the compiler does (memory usage can blow up easily).
To be fair, that code is to illustrate the beauty of the language rather than efficiency.
(Also the fibonacci sequence using 'zip'. Blew my mind.)
Haskell folks are fairly upfront about possible performance issues. Lazy evaluation comes with its own set of tradeoffs.
You can write very inefficient code in C++ if your algorithm isn't efficient.
(Also the fibonacci sequence using 'zip'. Blew my mind.)
Haskell folks are fairly upfront about possible performance issues. Lazy evaluation comes with its own set of tradeoffs.
You can write very inefficient code in C++ if your algorithm isn't efficient.
I still don’t see any tradeoffs explained, quite the opposite is on the home page:
,,The purity of Haskell code makes it easy to fuse chains of functions together, allowing for performance benefits.’’
Having something ,,beautiful’’ and inefficient is the problem I’m writing about. The language itself is guiding people to inefficiency.
I haven’t felt this with C++, so I’m not sure what you are talking about: usually simple / pretty code runs fast there, and more complex code runs slower.
,,The purity of Haskell code makes it easy to fuse chains of functions together, allowing for performance benefits.’’
Having something ,,beautiful’’ and inefficient is the problem I’m writing about. The language itself is guiding people to inefficiency.
I haven’t felt this with C++, so I’m not sure what you are talking about: usually simple / pretty code runs fast there, and more complex code runs slower.
Yeah, Haskell won't give you Rust-like performance. Very few languages will and, except for C, you won't even see them here on the comments.
Haskell can have a reasonable high performance, but if it's you main concern, or if predictability is important, you need a low level language.
Haskell can have a reasonable high performance, but if it's you main concern, or if predictability is important, you need a low level language.
Right now I'm using Julia for high performance math heavy modelling with lots of experimentation, and it's great for this purpose. I can get most of C++'s advantages without giving up on flexibility. My code base wouldn't scale to team work / production software
though as well as C++.
[deleted]
So there is also no point keep using Elixir then.
We will continue to use beam/otp where it makes sense. In fact some rust runs in BEAM now thanks to Rustler. We wrote a blog about this!
https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
You said that Go doesn't have: algebraic data types, enums, borrow checker, and compile time memory management/safety which Erlang / Elixir is even worse in some of those because it's a dynamic typed language so it's a bit confusing.
I think you're missing the point. We use Elixir to solve one class of problems which its excellent in doing (notably: real time distributed systems.) In this area, there is nothing even remotely close in terms of capability. So, we choose the runtime over the language (elixir is a pretty cool dynamic language though!). Also, due to the unique nature of Erlang/Elixir, a lot of problems that exist in C/C++/Rust/Go don't exist in that language. Of course this has meant writing a lot more tests to test things that a typechecker should be able to catch. But it is what it is. We're following Gleam though to see where that ends up.
BTW, here's some places I've spoken about Discord's Elixir usage:
- a podcast: https://smartlogic.io/podcast/elixir-wizards/s5e8-nowack-hei...
- interview with José Valim (elixir's creator): https://elixir-lang.org/blog/2020/10/08/real-time-communicat...
BTW, here's some places I've spoken about Discord's Elixir usage:
- a podcast: https://smartlogic.io/podcast/elixir-wizards/s5e8-nowack-hei...
- interview with José Valim (elixir's creator): https://elixir-lang.org/blog/2020/10/08/real-time-communicat...
Gleam looks like a very promising language! I hadn't seen it before.
https://gleam.run/book/tour/try.html
It also looks like there are currently some serious limitations:
https://github.com/gleam-lang/otp
https://gleam.run/book/tour/try.html
It also looks like there are currently some serious limitations:
https://github.com/gleam-lang/otp
I think a difference is that Elixir is deliberately dynamically typed, so you don't expect type safety. You write the happy path and rely on "fail fast" and the actor hierarchies to write robust code.
Besides, Elixir/Erlang has the best dynamically-typed equivalent of algebraic types (tuples + atoms + pattern matching) :-)
Besides, Elixir/Erlang has the best dynamically-typed equivalent of algebraic types (tuples + atoms + pattern matching) :-)
> Often as it is in this case with no gain for the company, other than some resume building for the involved team members
This is too cynical. A large re-write will not be approved at any large companies without millions of meetings and documentations filled with justifications.
This is too cynical. A large re-write will not be approved at any large companies without millions of meetings and documentations filled with justifications.
... All of which are researched with proof of concepts. Followed by some small trials. And finalized with one or two actual rewrites of non-critical software.
I'm not saying that all companies do it this way, and that there is no "resume building" or "magpie-ing" by some engineers in many companies. But it is perfectly possible to state a thesis "Rust is better for our X and Y and probably for Z". and to then research and prove that thesis.
I'm not saying that all companies do it this way, and that there is no "resume building" or "magpie-ing" by some engineers in many companies. But it is perfectly possible to state a thesis "Rust is better for our X and Y and probably for Z". and to then research and prove that thesis.
> resume building
We call this project farming where I am. I'm not a dev and don't work in IT industry, but I do work for a large company and every few months some new project is rolled out to 'improve' things and most of the time we all collectively groan and wait for the person to get promoted.
The most frustrating part is when once in a blue moon the project turns out to be useful, and the person gets promoted off of it and abandons the project. Nobody then wants to pick it up since they won't get any credit for running it and it ends up getting abandoned.
We call this project farming where I am. I'm not a dev and don't work in IT industry, but I do work for a large company and every few months some new project is rolled out to 'improve' things and most of the time we all collectively groan and wait for the person to get promoted.
The most frustrating part is when once in a blue moon the project turns out to be useful, and the person gets promoted off of it and abandons the project. Nobody then wants to pick it up since they won't get any credit for running it and it ends up getting abandoned.
[deleted]
> Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.
In other words, discard the technical arguments laid out in the article and just speculate that it was an HR/management decision.
In other words, discard the technical arguments laid out in the article and just speculate that it was an HR/management decision.
Sounds like Discord was running gigantic in memory Go caches.
Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense.
Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite.
[0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense.
Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite.
[0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
> I'm curious if they tried the Ballast[0] approach
From the essay, the issue discord had was that they were not triggering enough GCs. The ballast approach serves to tune the GC frequency down.
In Discord's case they were getting screwed by a periodic mandatory GC trigger.
> along with smaller cache's per service
They did use a smaller cache, but that increased their 99% since… there was now less stuff in the cache.
From the essay, the issue discord had was that they were not triggering enough GCs. The ballast approach serves to tune the GC frequency down.
In Discord's case they were getting screwed by a periodic mandatory GC trigger.
> along with smaller cache's per service
They did use a smaller cache, but that increased their 99% since… there was now less stuff in the cache.
Makes me wonder what their partitioning strategy was. I can't imagine a cache that is over 4GB+ that couldn't be partitioned at least a bit further. But good call out, thank you for the correction.
There's a proposal to put in official support for minimum heap size: https://github.com/golang/go/issues/44309
[deleted]
In its early days, this forum was full of excited young devs who had read PG's old essays and we're actively looking for technologies that would give them a competitive edge.
Now, it's completely the opposite.
Now, when a company successfully leverages technologies as a competitive edge and becomes a multi-billion dollar success, the thread is dominated by stodgy criticism for not picking the absolute most mainstream technologies possible.
Now, it's completely the opposite.
Now, when a company successfully leverages technologies as a competitive edge and becomes a multi-billion dollar success, the thread is dominated by stodgy criticism for not picking the absolute most mainstream technologies possible.
Saying the right thing has always been easier than saying the correct thing.
> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.
Big takeaway here for those who are married to their tools. Rust happens to be just another tool for this company, and it works well for them; where it makes sense of course.
Big takeaway here for those who are married to their tools. Rust happens to be just another tool for this company, and it works well for them; where it makes sense of course.
@bobthebuilders, it is one thing to give productive, insightful criticism of a technology, even if it is scathing. It is another thing to insult the creators of a technology. Please adjust your tone.
Pretty sure this comment has wandered astray?
bobthebuilders’ comment was marked dead, GP responded to the same parent comment as the dead comment. If you turn on showdead you'll be able to see what they're talking about.
Exactly. As my name implies, I'm not a fan of shadowbanning unless the account is clearly vicious or spam.
The impression I get when I hear they use Go, Elixir and Rust on the backend is that they are chasing every shiny trend. I doubt it makes sense from a purely tecnical perspective and is more a way of indulging and investing in the skills of their engineers.
Elixir was not us chasing a trend - but rather betting on a language and runtime system we knew would be amazing if it worked.
Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users?
Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!
Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users?
Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!
It's not really the subject of the discussion, but since you're here and interacting with people: would it be possible to ask the UI/UX folks if they could finally make the built-in shortcuts configurable (and deactivable)?
It's really weird that such a game adjacent piece of software completely hardcodes shortcuts, especially when so many of them either conflict with or are near-misses for common chords, especially for very disruptive actions like starting voice calls. Even more so as it already has configurable keychords, just not for most of the useful stuff.
Also they should fix the labelling around these, either it's just not working or it doesn't work reliably, but trying to override built-in shortcuts via the configuration (in order to disable them) has never worked for me.
It's really weird that such a game adjacent piece of software completely hardcodes shortcuts, especially when so many of them either conflict with or are near-misses for common chords, especially for very disruptive actions like starting voice calls. Even more so as it already has configurable keychords, just not for most of the useful stuff.
Also they should fix the labelling around these, either it's just not working or it doesn't work reliably, but trying to override built-in shortcuts via the configuration (in order to disable them) has never worked for me.
I don't work at Discord, but Elixir seems like a perfect fit for the sort of infrastructure they build. BEAM is a marvel and there are huge success stories for it when it comes to chat apps (e.g., WhatsApp). Elixir is not necessarily a fast language, though, so it's easy to see how they'd run into a need for something like Go or Rust for the occasional service where you want to push single-core performance to the limit.
Agreed. It still sounds like the Elixi/Erlang & Rust combo is incredibly powerful.
[deleted]
Off-topic: who is in charge of choosing colors for graphs? Being red-green colorblind, choosing blue and purple is makes it impossible for me to read the charts. I know its stylish, but every time i make a graph i choose very high contrast colors. Another point: is it just being color blind or do other people struggle in charts/maps that have shades of a color? like different shades of blue represent different rates of covid deaths across a map. just curious if anyone else struggles with this
I am not colorblind, and the chosen colors for this graph are similar enough to make it difficult to read.
High-contrast helps the rest of us, too.
High-contrast helps the rest of us, too.
These are just the default colors for Datadog graphs, so I guess the answer to your question is Datadog.
Brewer colour schemes ftw.
i'm curious if allocating a memory ballast would have helped discord in this case?
a one line patch in main() with something like
ballast := make([]byte, 10<<30)
might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so.
https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
a one line patch in main() with something like
ballast := make([]byte, 10<<30)
might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so.
https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
The GC cycles were triggered not due to heap size but due to the forced 2 minute max time between GC cycles [0], so a ballast wouldn't have helped here.
[0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
[0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
This is really interesting. Are there not alternative runtimes, or configurable gc periods?
My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
The Go core team intensely dislikes tunables, so Go has very few of these (really the opposite of the JVM).
There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430
There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430
> i'm curious if allocating a memory ballast would have helped discord in this case?
No. Twitch's ballast is because
> At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute).
Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below.
But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.
No. Twitch's ballast is because
> At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute).
Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below.
But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.
Switching languages is THE most efficient way to refactor a messy codebase.
The past color the idea that a language switch is a waste of time, because in the past the choice of languages was from worse to bad or for same to same-ish.
But MODERN languages have something that make this compelling: Actual good practiques baked in the language, as defaults.
---
So, if a codebase is in trouble you can:
- Try to refactor it with "better" idioms...
but if this is not the norm in the language/ecosystem you depend in discipline, and that have proved to fail at scale.
- Change to "shinny lang" where do the right thing is the default...
And now if the developers try to do wrong, them will fight against the current.
---
Switch to Go is a great way to solve a project with concurrency issues: Go is good at that, and make a compelling case of build good concurrency apps. Similar if switch to Elixir/Erlang.
The case of Rust is similar: it make the "best" practiques the default, and very painful to do wrong (and sometimes to do right, but this is not the focus here :) ).
So, a language switch is the ultimate clean up for a project.
I have done several in the past (I'm contractor) and let me tell you: Any codebase I get is in deeeeeep trouble.
The best results have been a language switch, when I move from "bad" to "better". Just the fact the new language have something that is truly better make the move great.
Now, the opposite is true: If you are switching and NOT because the new language have that something that is truly in need or worse, is a language with the need of a LOT of discipline (like C, PHP, JS,...) then yeah, is doomed...
The past color the idea that a language switch is a waste of time, because in the past the choice of languages was from worse to bad or for same to same-ish.
But MODERN languages have something that make this compelling: Actual good practiques baked in the language, as defaults.
---
So, if a codebase is in trouble you can:
- Try to refactor it with "better" idioms...
but if this is not the norm in the language/ecosystem you depend in discipline, and that have proved to fail at scale.
- Change to "shinny lang" where do the right thing is the default...
And now if the developers try to do wrong, them will fight against the current.
---
Switch to Go is a great way to solve a project with concurrency issues: Go is good at that, and make a compelling case of build good concurrency apps. Similar if switch to Elixir/Erlang.
The case of Rust is similar: it make the "best" practiques the default, and very painful to do wrong (and sometimes to do right, but this is not the focus here :) ).
So, a language switch is the ultimate clean up for a project.
I have done several in the past (I'm contractor) and let me tell you: Any codebase I get is in deeeeeep trouble.
The best results have been a language switch, when I move from "bad" to "better". Just the fact the new language have something that is truly better make the move great.
Now, the opposite is true: If you are switching and NOT because the new language have that something that is truly in need or worse, is a language with the need of a LOT of discipline (like C, PHP, JS,...) then yeah, is doomed...
I've never seen anyone look into just why Go forces a GC on 2 minute intervals. The code originally appeared in https://codereview.appspot.com/5451057 and there's a bit of discussion on https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E.
After reading this, I still don't understand. But I find https://codereview.appspot.com/5451057#msg38 particularly entertaining: there was going to be an environment variable to control the interval, which might have allowed Discord to keep using Go, but Russ Cox shot it down.
After reading this, I still don't understand. But I find https://codereview.appspot.com/5451057#msg38 particularly entertaining: there was going to be an environment variable to control the interval, which might have allowed Discord to keep using Go, but Russ Cox shot it down.
Ah Discord, that provides electron based chat client. I am sure they can write efficient software when its their money.
To be fair, they probably ship one of the best and most optimized Electron app out there.
I still don’t like Electron, but Discord is doing a good job with it. Even their React-Native mobile apps feel decently snappy (where usually large React-Native apps are buggy and slow as fuck).
I still don’t like Electron, but Discord is doing a good job with it. Even their React-Native mobile apps feel decently snappy (where usually large React-Native apps are buggy and slow as fuck).
That's pretty debatable.
I wouldn't mind looking into Rust to eventually replace our Java and Python based stacks, but every time I look at it, I tend to give some thought to the difficulty of writing Rust as opposed to reading/writing/maintaining Java for an entire team. Frankly I'm worried that not all team members will become good Rust developers, and the codebase will develop serious rot over time. Any experiences with this?
> our Java and Python based stacks
I always find it bizarre when python (and to some extent java) are lumped in as languages that you found Go or Rust a suitable replacement for. They're just such different ends of the spectrum - Go/Rust will just never be able to match the expressivity and rapid development ease of python because they have such different design goals.
If you find porting a production application from python to Go/Rust a satisfactory outcome, then I suspect you should never have been using python for it in the first place or you've been using python All Wrong.
I always find it bizarre when python (and to some extent java) are lumped in as languages that you found Go or Rust a suitable replacement for. They're just such different ends of the spectrum - Go/Rust will just never be able to match the expressivity and rapid development ease of python because they have such different design goals.
If you find porting a production application from python to Go/Rust a satisfactory outcome, then I suspect you should never have been using python for it in the first place or you've been using python All Wrong.
I have used Go on a project that could easily have been done in Python, though. So they are not that far apart in terms of suitability.
We switched Ruby -> Scala, and it took considerable training effort and some years before everyone was comfortable.
I would estimate 15% of the developers still can't write moderately good Scala code.
I think you should be worried instead of the amount of libraries that you give up from the JVM and Python ecosystems before anything else. Also, the applications of Rust and Java/Python generally do not overlap, I would be cautious in approaching this transition. Scala would be a better choice since you have expertise on the JVM and it has a familiar syntax to Python devs.
Feels like there might have been a middle ground short of a scratch rewrite. LRU caches are always suspect. LRU is pretty much the worst replacement policy you can choose. I wonder if a smaller cache with a better replacement policy (2Q, ARC, ...) would have solved this specific problem.
Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.
Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.
> Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.
Just to check that I understand what you're suggesting here, this is comparable to how we embedded systems folks might pre-allocate (either statically or at the very start of a program execution) a large chunk of memory and use it instead of using traditional malloc/free (or language equivalent), correct? So you bypass the GC and avoid any potential performance penalties introduced by having a smaller number of massive allocations that never get moved, freed, or reallocated.
Just to check that I understand what you're suggesting here, this is comparable to how we embedded systems folks might pre-allocate (either statically or at the very start of a program execution) a large chunk of memory and use it instead of using traditional malloc/free (or language equivalent), correct? So you bypass the GC and avoid any potential performance penalties introduced by having a smaller number of massive allocations that never get moved, freed, or reallocated.
Yes, the idea is similar. If you make a huge map in Go with millions of items, it will stress the GC. If you take the same information and just put it into a big byte slice, and index into it in a way that the GC doesn't think you're pointing to addresses, then it will be much faster. Of course now you are writing a little more code.
> Discord has never been afraid of embracing new technologies that look promising. For example, we were early adopters of Elixir, React, React Native, and Scylla. If a piece of technology is promising and gives us an advantage, we do not mind dealing with the inherent difficulties and instability of the bleeding edge. This is one of the ways we’ve quickly reached 250+ million users with less than 50 engineers.
this sounds like a recipe for disaster in every other company i have worked with. are Discord's engineers just THAT good or is there something else going on here?
this sounds like a recipe for disaster in every other company i have worked with. are Discord's engineers just THAT good or is there something else going on here?
These threads are always full of people criticizing these moves by companies. But nothing is gained of nothing is risked, right? Sometimes these things pay off, and we would never know without early adopters.
The reason why a lot of people criticize moves to rewrite large codebases is because the industry has collected a lot of knowledge and experience with these types of moves and it rarely works out. This is why it's considered a bad practice. We are not babes in the woods, walking around with our eyes newly opened, exploring this strange, wonderous land. The lay of the land is already known in some areas, and this is one such area.
Here I would recommend the following twenty year old article by Joel Spolsky, and Joel was merely reciting knowledge that was already well understood, but in an entertaining way.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Here I would recommend the following twenty year old article by Joel Spolsky, and Joel was merely reciting knowledge that was already well understood, but in an entertaining way.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Discussed at the time:
Why Discord is switching from Go to Rust - https://news.ycombinator.com/item?id=22238335 - Feb 2020 (640 comments)
also https://news.ycombinator.com/item?id=23874336 - July 2020 (7 comments)
Why Discord is switching from Go to Rust - https://news.ycombinator.com/item?id=22238335 - Feb 2020 (640 comments)
also https://news.ycombinator.com/item?id=23874336 - July 2020 (7 comments)
I really enjoyed reading this article last time it was posted. I particularly like it because it shows evidence that current tooling wasn't doing the job. I believe I remember seeing some comments on how Discord was on an old version of go that wasn't receiving the benefit around gc collections, but we'll never know if that would've been a solution.
That all said, I've noticed that people here are really off golang in favor of rust. I believe I sort of understand. I've recently switched from python to go at work, and I frequently bring up why I like it so much (caveat I'm overly enthusiastic about the things I like related to my work). But again, that all said, at the end of the day, both Python and golang will definitely get the job done.
That all said, I've noticed that people here are really off golang in favor of rust. I believe I sort of understand. I've recently switched from python to go at work, and I frequently bring up why I like it so much (caveat I'm overly enthusiastic about the things I like related to my work). But again, that all said, at the end of the day, both Python and golang will definitely get the job done.
Incredibly brave of them to go with nightly build of Rust for async functionality. Not as in the weeds, but I would have been afraid of release being delayed forever and being stuck with the nightly release for a long time.
This post is getting a bit long in the tooth, at least regarding relatively new and fast-moving ecosystems like Go and Rust. I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published.
I also wonder how well Rust has continued to work for Discord, if it's been a problem to keep up with the rapid movement particularly in the async ecosystem in the last year or so.
I also wonder how well Rust has continued to work for Discord, if it's been a problem to keep up with the rapid movement particularly in the async ecosystem in the last year or so.
> I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published.
Go 1.12 might have addressed this particular issue [0, 1], but it's possible that that came out too late for Discord to try before deciding to rewrite the service.
[0]: From Go 1.12 patchnotes: "Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live."
[1]: https://old.reddit.com/r/golang/comments/eywx4q/why_discord_...
Go 1.12 might have addressed this particular issue [0, 1], but it's possible that that came out too late for Discord to try before deciding to rewrite the service.
[0]: From Go 1.12 patchnotes: "Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live."
[1]: https://old.reddit.com/r/golang/comments/eywx4q/why_discord_...
(Feb 4, 2020)
Note: this is an account of using Rust in a single service
> The service we switched from Go to Rust is the “Read States” service.
And the conclusion doesn't indicate "switching", but rather Rust is an "option" for Discord, and already used in the stack:
> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.
> The service we switched from Go to Rust is the “Read States” service.
And the conclusion doesn't indicate "switching", but rather Rust is an "option" for Discord, and already used in the stack:
> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.
I didn't know Discord was written in Go to begin with. I thought it was written using electron or something like that.
We were experimenting with Go as a "fast" backend language to implement stuff like data services and high throughput systems in.
Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.
Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.
How Go disappointed you beyond the case in the linked article? Is there a few cons in using Rust compared to Go? In particular, what about language complexity/learning curve and compilation times?
The only major cons to rust imo are the slow compile times, the steep learning curve and the immature ecosystem.
You can solve the first hurdle with better hardware (for the same price as our usual specced macbook pro, I opted to custom build a beefy workstation with a 3790X).
The second, you just solve with time. I remember when I first started with rust I hated it. I was working on a native project, and felt like it would have been easier to do in C++. But internally now we have more mindshare and knowledgeable folk who understand Rust - this has helped tremendously in onboarding engineers to the language. Also, having a working editor (via rust-analyzer) and the quality of compiler error messages have improved significantly in the past few years.
In terms of immature ecosystem, I think the rate at which the ecosystem is advancing is incredibly promising. Tokio hit 1.0 in Dec 22 of last year, which will finally put an end to the dependency upgrade thrash we've had to deal with. Going from tokio 0.1 -> 0.2 -> 0.3 -> 1.0 has been a journey. But with 1.0, finally, libraries will have a stable API to target, so we won't have to worry about this much for a while. Additionally, in the following months we intend to start funding critical open source rust projects that we use to ensure their success.
You can solve the first hurdle with better hardware (for the same price as our usual specced macbook pro, I opted to custom build a beefy workstation with a 3790X).
The second, you just solve with time. I remember when I first started with rust I hated it. I was working on a native project, and felt like it would have been easier to do in C++. But internally now we have more mindshare and knowledgeable folk who understand Rust - this has helped tremendously in onboarding engineers to the language. Also, having a working editor (via rust-analyzer) and the quality of compiler error messages have improved significantly in the past few years.
In terms of immature ecosystem, I think the rate at which the ecosystem is advancing is incredibly promising. Tokio hit 1.0 in Dec 22 of last year, which will finally put an end to the dependency upgrade thrash we've had to deal with. Going from tokio 0.1 -> 0.2 -> 0.3 -> 1.0 has been a journey. But with 1.0, finally, libraries will have a stable API to target, so we won't have to worry about this much for a while. Additionally, in the following months we intend to start funding critical open source rust projects that we use to ensure their success.
Amazing feedback. Very much appreciated.
Does Discord use the cloud for infrastructure or are they big enough that they are going their own way?
We are hybrid. We use bare metal for our RTC workload (voice video streaming) and GCP for the rest.
I'm curious if you have any thoughts on power savings/sustainability from using Rust, or from using Rust with ARM-based processors on bare metal, for example. Is it meaningfully different at your scale?
Have you considered also switching the client to Rust?
No. React/TypeScript is too productive for our project engineers.
But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)
But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)
Are there any apps that you know of that use Servo as a rendering engine? I'm curious and would like to look at some examples; I thought Servo was intrinsically tied to Firefox.
No, not yet. But Servo can be used headless:
https://github.com/paulrouget/servo-embedding-example
https://github.com/paulrouget/servo-embedding-example
The client is electron, I believe Go/Rust is used in the backend systems.
The backend is still quite a bit of elixir (I imagine, I don't know as I don't work there) - as of October 2020 blog post
I think the response time / latency to use Cassandra on the "read state" will be higher, and mind you cassandra is written in java (JVM) and has more garbage collection magic and latency than Go. From my point of view, it's not the language, it's your data store.
How timely!
https://stackoverflow.blog/2021/02/22/choosing-java-instead-...
I do believe that there is nothing better tuned for performance vs developer productivity than the JVM.
The language semantics may be a question for personal distaste..but then you do have Kotlin (https://jooby.io/ ) which ranks fairly near the top on techempower benchmarks https://www.techempower.com/benchmarks/#section=data-r20&hw=...
https://stackoverflow.blog/2021/02/22/choosing-java-instead-...
I do believe that there is nothing better tuned for performance vs developer productivity than the JVM.
The language semantics may be a question for personal distaste..but then you do have Kotlin (https://jooby.io/ ) which ranks fairly near the top on techempower benchmarks https://www.techempower.com/benchmarks/#section=data-r20&hw=...
[deleted]
Very misleading title makes it sound like they are rewriting all code wholesale - in fact it's a targeted performance rewrite of a smallish component.
previous discussion: https://news.ycombinator.com/item?id=22238335
Could something like Discord be written in Swift?
The main thing that bothers me about rust is that there is one complete implementation of the language (and one or two very incomplete implementations) and the singular complete one isn't even self hosting.
Coming from scheme where everyone and their dog has an R5RS compatible REPL that's pretty disturbing.
Coming from scheme where everyone and their dog has an R5RS compatible REPL that's pretty disturbing.
> The main thing that bothers me about rust is that there is one complete implementation of the language
The essay is about Go which is pretty much in the same position: gccgo is a non-entity.
> and the singular complete one isn't even self hosting.
What are you talking about? Rust was self-hosting before it was even stable (a common source of problems as the language can then get optimised for implementing the language, thankfully it doesn't seem to have harmed it too badly).
The essay is about Go which is pretty much in the same position: gccgo is a non-entity.
> and the singular complete one isn't even self hosting.
What are you talking about? Rust was self-hosting before it was even stable (a common source of problems as the language can then get optimised for implementing the language, thankfully it doesn't seem to have harmed it too badly).
I think they don't consider it self-hosted because only the front-end is written in rust, rustc uses llvm as the back-end, which is not written in rust.
By that argument gcc isn't self hosting either because it uses various assembly syntaxes in its code that requires an assembler.
What makes rustc not self hosting?
[deleted]
I'm just curious why can't they use C or C++ instead of Rust
I don't think C++ is a compelling choice over Rust if you don't already have a lot of C++ experience in your organization. C++ is way harder to learn than Rust, IMO, not because of any set of language features or footguns, but because the ecosystem, tooling, etc... is just so different from what most "modern" devs have come to expect.
C just doesn't offer the same feature set at all.
C just doesn't offer the same feature set at all.
I see but C++ is widely used and you can easily find C++ devs compared to Rust.
This brings me to my next questions C++ has been there for a long time compared to Rust but still people decide to go with Rust
Just curious to know
This brings me to my next questions C++ has been there for a long time compared to Rust but still people decide to go with Rust
Just curious to know
It seems like Discord wasn't about to hire new developers for this, they wanted to use their existing talent. In that case, what matters is how easy a new language is to learn, not hire for.
Also, good C++ devs are decidedly not easy to come by. Especially when you get into performance sensitive work, they can command huge salaries; and a bad C++ dev can do more damage to a codebase than any other language I know of.
Also, good C++ devs are decidedly not easy to come by. Especially when you get into performance sensitive work, they can command huge salaries; and a bad C++ dev can do more damage to a codebase than any other language I know of.
Got it,
You explained it really well !!
Introducing C or C++ into an org with no expertise in them feels much riskier than introducing Rust. All three are difficult languages but at least with Rust you have Cargo and memory safety by default. I’ve seen smart folks burn days just setting up dependency resolution and builds for C++ stacks. Experts don’t have that problem but a team that has never used it professionally very well might.
Compile time memory safety?
Unless you have an existing code base or a necessary integration or requirement that demands it, I honestly wouldn’t start a C/C++ project in 2021 all other things being equal.
Unless you have an existing code base or a necessary integration or requirement that demands it, I honestly wouldn’t start a C/C++ project in 2021 all other things being equal.
It doesn’t look as good on a resume. There’s thousands of people with decades of experience in C or C++ that you need to compete with for clout. It’s easier to invent your own problems so you get to be the best man for the job.
I think you're being downvoted for pushing a narrative (somehow C langs are better than Rust) without providing your opinion or evidence. It baits people into defending Rust without you doing any work to defend your position.
Because nobody would upvote that on HN. The HN zeitgeist of programming languages is absurdly biased. In actual industrial practice the dominant languages are Java, C++, C, C#, Python, Javascript, dozens of things people rarely discuss like matlab, VB, delphi, a bunch of other shit, and way, way down at the bottom, Rust, and then about fifty other languages, and finally Erlang. It's ten times easier to find a job writing MIPS assembly code than to find one programming in Rust, but nobody wants to blog about that.
In actual industrial practice, most YOUNG C++ developers I know would love to be able to work with Rust at work.
There's a lot of territory between what one wishes to happen at work, and what actually happens at work.
Right, I'm just saying that the HN zeitgeist is -- at least in this case -- reflecting a real belief. It's what people want to work with, not what currently is run everywhere in production; and what developers want to work with definitely matters.
Discord rewrote this service in Rust because they had already had success using Rust for some performance critical services; it seems likely that they chose to invest in Rust and not C++ for those services because their developers pushed harder for or were more interested in Rust.
Discord rewrote this service in Rust because they had already had success using Rust for some performance critical services; it seems likely that they chose to invest in Rust and not C++ for those services because their developers pushed harder for or were more interested in Rust.
Exactly, which is why my original responses to this year-old blog post was "Why did we switch to Rust: because we wanted to". The blog post should focus on those social aspects, not invent tenuous technical back-justifications.
In other words, they solved the problem they wouldn't have if they had stuck to C in the first place.
We may never know for sure, but I bet they avoided a whole bunch of problems by using Go instead of C. Probably wise to go lower only when needs must.
[deleted]
Every time I see a company post about how they’re changing programming languages, I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. But, of course, you’d be lambasted by your colleagues if you didn’t provide technical reasons, so you go and hunt for technical reasons that seem insurmountable and build a case for it, turning a blind eye to perhaps less exciting solutions that don’t require a programming language change.
(It’s sort of like how a cop in America can find you doing something illegal in any circumstance if they look hard enough. Except in this case, it’s a matter of finding technical deficiencies.)
This strategy highly depends on clout though, which is why I think technical arguments are frequently really a façade. If, say, a junior programmer doesn’t have clout, even presenting an equivalent technical argument won’t lead to a change in programming language, especially if it’s the senior’s favorite. I’m sorry to say, but most tech organizations aren’t meritocratic or egalitarian in this way (and perhaps they shouldn’t be).
As it pertains to the article—which again I don’t claim is all “fake” politicking (I don’t know anything about or anyone at Discord)—they mostly ripped on garbage collection. They specifically talked about how GC was bad because their LRU cache had to be scanned, which was part of why they observed spikes in their metrics. I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?
I think using any language with any serious and large project will find deficiencies in the language. At some point in a product lifecycle it’s more profitable to just work with and ameliorate those deficiencies than do rewrites where the new deficiencies will be completely unknown.