Rust Is the Future of JavaScript Infrastructure(leerob.io)
leerob.io
Rust Is the Future of JavaScript Infrastructure
https://leerob.io/blog/rust
169 comments
None of what you said is unique to JavaScript. Take a look at any language that has a similarly large adoption and you'll see similar frustrations. Half-baked frameworks and libraries, largely undocumented popular projects and build tools with steep learning curves.
Why we have so many of the same thing that seemingly solves the same issues? Could it be because people have different needs and requirements that aren't already solved by these tools? Perhaps the tools over complicate things that could be simplified? From Grunt and Gulp, onto Webpack, Parcel and Rollup, and now Snowpack, ESBuild and WMR – Each solving a similar problem much differently from each other due to the new kind of applications and the problems that come with it. To provide a good user experience for a majority of people, there are a lot of things that has to be done to make that possible here and now. Browsers, unfortunately, don't update quick enough to provide the UX until the problem is big enough to warrant a built-in solution. That is how the web has been able to move forward; by peoples innovation in the space and the approaches they took to solve them.
Why we have so many of the same thing that seemingly solves the same issues? Could it be because people have different needs and requirements that aren't already solved by these tools? Perhaps the tools over complicate things that could be simplified? From Grunt and Gulp, onto Webpack, Parcel and Rollup, and now Snowpack, ESBuild and WMR – Each solving a similar problem much differently from each other due to the new kind of applications and the problems that come with it. To provide a good user experience for a majority of people, there are a lot of things that has to be done to make that possible here and now. Browsers, unfortunately, don't update quick enough to provide the UX until the problem is big enough to warrant a built-in solution. That is how the web has been able to move forward; by peoples innovation in the space and the approaches they took to solve them.
Most of my career in the mega Corp world has enjoyed watching people struggle to really put the Java in JavaScript. Epic fail. There are many valid reasons to hate JavaScript but ignorant Java developers is not one of them.
Going from Java to JavaScript really does seem to be painful for developers who don't know other languages as well, I know what you mean... "A closure?" I'm sure it's an easier switch today but I've read a lot of awkward JS written by people who came from Java 5 and earlier.
Have you considered a career change? I'm sure there are jobs that don't require JavaScript.
Your frustrations are felt by many.
Every tool is attempting to dethrone the competition, coming up with yet another way to do the same simple yet infinitely elusive fucking goal: render a website.
The whole ecosystem is fucked
Every tool is attempting to dethrone the competition, coming up with yet another way to do the same simple yet infinitely elusive fucking goal: render a website.
The whole ecosystem is fucked
JavaScript was made in like ten days just to add missing interactivity to HTML. If Brendan Eich had known how it was going to be used today, the language would have been very different, if at all.
Edit: it seems I don't have to wonder. Type coercion is one of the worst features IMO, and coincidentally one of his regrets:
> "One of the notorious ones was, ‘I’d like to compare a number to a string that contains that numeral. And I don’t want to have to change my code to convert the string to a number, or the number to a string. I just want it to work. Can you please make the equals operator just say, Oh this looks like a two, and this looks like a string number two. They’re equal enough.’"
> "And I did it. And that’s a big regret, because that breaks an important mathematical property, the equivalence relation property… It led to the addition of a second kind of equality operator when we standardized JavaScript.” One of the people who helped standardize JavaScript was Guy Steele, one of the co-creators of Scheme. “Guy said, ‘Don’t worry about it. There are Lisps that have five kinds of equals operators. We’ll just add another one.'"
https://thenewstack.io/brendan-eich-on-creating-javascript-i...
Edit: it seems I don't have to wonder. Type coercion is one of the worst features IMO, and coincidentally one of his regrets:
> "One of the notorious ones was, ‘I’d like to compare a number to a string that contains that numeral. And I don’t want to have to change my code to convert the string to a number, or the number to a string. I just want it to work. Can you please make the equals operator just say, Oh this looks like a two, and this looks like a string number two. They’re equal enough.’"
> "And I did it. And that’s a big regret, because that breaks an important mathematical property, the equivalence relation property… It led to the addition of a second kind of equality operator when we standardized JavaScript.” One of the people who helped standardize JavaScript was Guy Steele, one of the co-creators of Scheme. “Guy said, ‘Don’t worry about it. There are Lisps that have five kinds of equals operators. We’ll just add another one.'"
https://thenewstack.io/brendan-eich-on-creating-javascript-i...
The problem with equality and dynamic typing is rather deep and intractable. Kent Pitman (of Lisp fame) has an entire essay about it: https://www.nhplace.com/kent/PS/EQUAL.html
> If COPY can't be done properly, then neither can EQUAL. And, in fact, that's the case. There is no uniquely determined equality function for complex structures--there are only arbitrary ones.
What Steele says about equality is true for Scheme. Everyone has seen the famous "Wat" video of adding "{} + {}" in Ruby or JS. But no one ever mentions Scheme or Lisp:
> breaks an important mathematical property
Even mathematicians get lazy and create new syntax and short cuts to express their ideas. Programming is ultimately a human UI. It's easy to see why automatic type coercion was once popular (and probably will be again, I can almost guarantee this).
> If COPY can't be done properly, then neither can EQUAL. And, in fact, that's the case. There is no uniquely determined equality function for complex structures--there are only arbitrary ones.
What Steele says about equality is true for Scheme. Everyone has seen the famous "Wat" video of adding "{} + {}" in Ruby or JS. But no one ever mentions Scheme or Lisp:
scheme@(guile-user)> (eq? "test" "test")
$1 = #t
scheme@(guile-user)> (eqv? "test" "test")
$2 = #t
scheme@(guile-user)> (define x "test")
scheme@(guile-user)> (define y "test")
scheme@(guile-user)> (eq? x y)
$3 = #f
scheme@(guile-user)> (eqv? x y)
$4 = #f
> breaks an important mathematical property
Even mathematicians get lazy and create new syntax and short cuts to express their ideas. Programming is ultimately a human UI. It's easy to see why automatic type coercion was once popular (and probably will be again, I can almost guarantee this).
I was surprised that I was OK with that fact that in Rust a vector and an array with the same elements compare equal. It felt completely reasonable, and I definitely didn't feel that way about it in some other languages where that's true.
Erm, no, that doesn't work. I created a quick check in the Rust Playground: https://play.rust-lang.org/?version=stable&mode=debug&editio...
Do you have some more context? I suspect somehow things are getting coerced to slices and then compared.
Do you have some more context? I suspect somehow things are getting coerced to slices and then compared.
The implementation isn't symmetric, it'll work if you compare `b == a`. Apparently[0] it's something to do with avoiding code bloat.
[0] https://github.com/rust-lang/rust/blob/e4c23daeb461ac02413eb...
[0] https://github.com/rust-lang/rust/blob/e4c23daeb461ac02413eb...
[deleted]
This does work in Clojure. Edit: and Clojurescript, which is more relevant.
(let [x “test” y “test”]
(= x y))
; returns true with both (let) and individual (def)
I like Javascript (mainly Typescript) and use it quite a lot. If you aren't a fan, I completely understand. However I don't understand why people feel the need to discredit something that they don't like.
It honestly makes me not want to open HN threads about JS as I know I will be met with a barrage of negative comments about something that I enjoy. I'm all for open discussion of the downsides of the JS ecosystem but that is rarely what these comments are.
It honestly makes me not want to open HN threads about JS as I know I will be met with a barrage of negative comments about something that I enjoy. I'm all for open discussion of the downsides of the JS ecosystem but that is rarely what these comments are.
There is a special issue for JavaScript, and that's monopoly. If you want internet service and don't like your ISP, you'll be a lot more critical if you can't realistically switch to a different ISP. If you want the world-changing reach of the web and don't like JavaScript, you may be a lot more critical of it than you are of some server-side language that you don't like either (as long as you get to choose your server-side stack).
Same applies to any platform.
Using languages that don't come with the platform SDK, or aren't used to build the underlying platform, always add development costs with additional FFI, debugging tools, binding libraries,....
Using languages that don't come with the platform SDK, or aren't used to build the underlying platform, always add development costs with additional FFI, debugging tools, binding libraries,....
> There is a special issue for JavaScript, and that's monopoly
There are languages that treat javascript as a compilation target. Purescript, Elm, Closurescript, Rescript, Scalajs, etc. So if you really dislike javascript, you are still free to pick something else.
There are languages that treat javascript as a compilation target. Purescript, Elm, Closurescript, Rescript, Scalajs, etc. So if you really dislike javascript, you are still free to pick something else.
That just hides the lack of choice with another layer of complexity. A script language is not the best compilation target, but Web Assembly probably would be.
> A script language is not the best compilation target
Performance-wise, sure. But the parent commenters were not complaining about performance; they were upset about javascript's syntax, or lack of type guarantees, or typecasting, and so on. Well, all of this can be taken care of by the compiler, if the developer dislikes javascript itself so much.
Performance-wise, sure. But the parent commenters were not complaining about performance; they were upset about javascript's syntax, or lack of type guarantees, or typecasting, and so on. Well, all of this can be taken care of by the compiler, if the developer dislikes javascript itself so much.
Why are you so emotionally invested in whether other people like JS? Languages are tools -- not part of your identity.
Also what languages do you like less than JS? A lot of people think working with a certain tool is "as good as it's going to get" until they try something better.
Also what languages do you like less than JS? A lot of people think working with a certain tool is "as good as it's going to get" until they try something better.
Of course! People are free to dislike languages as much as they want. But is that something that is worthwhile discussing on HN?
I don't dislike any language in that sense, every language has pros and cons. Different tools for different jobs, some I enjoy more than others. But that wasn't the point.
I don't dislike any language in that sense, every language has pros and cons. Different tools for different jobs, some I enjoy more than others. But that wasn't the point.
> I don't dislike any language in that sense, every language has pros and cons. Different tools for different jobs, some I enjoy more than others. But that wasn't the point.
I dislike this overly positive attitude: responding to criticism of a language being worse than others by saying "every language has pros and cons" avoids addressing the criticisms and seems to imply that there are no bad languages.
It is possible to design a bad language for complex projects.
I dislike this overly positive attitude: responding to criticism of a language being worse than others by saying "every language has pros and cons" avoids addressing the criticisms and seems to imply that there are no bad languages.
It is possible to design a bad language for complex projects.
> But is that something that is worthwhile discussing on HN?
This thread is about improving JavaScript tooling. It seems like a very appropriate place to discuss things you dislike about JavaScript.
> I don't dislike any language in that sense, every language has pros and cons. Different tools for different jobs, some I enjoy more than others.
The problem with JavaScript is that it wasn't designed for the job it's doing and (until recently) we really had no other options. We still don't have another language that is browser-native, unless you count Web Assembly.
So some of the JavaScript hatred is due to the fact that 100% of web developers have to use it whether they like it or not.
This thread is about improving JavaScript tooling. It seems like a very appropriate place to discuss things you dislike about JavaScript.
> I don't dislike any language in that sense, every language has pros and cons. Different tools for different jobs, some I enjoy more than others.
The problem with JavaScript is that it wasn't designed for the job it's doing and (until recently) we really had no other options. We still don't have another language that is browser-native, unless you count Web Assembly.
So some of the JavaScript hatred is due to the fact that 100% of web developers have to use it whether they like it or not.
It is very well possible to like working in a language and to have a sincere dislike for its shortcomings. Language shortcomings are those things that repeatedly bite you in the backside or require work-arounds when there would have been alternatives known at the time of language design that would have avoided those particular pitfalls.
I completely agree with that, there are some things that would be nice to change that are unlikely ever to be changed due to the goal of being backwards compatible. But a lot of the OP's comment was unrelated to the language itself.
> browser incompatibilities are still prevalent
It's true, browser difference are a pain to work with. But what is the alternative, to only support a single browser engine? Note that this has improved a lot in recent years. As long as you don't need to support IE11 or below then you shouldn't come across many issues. You can find browser support using caniuse.com or looking at MDN docs.
> even popular libraries are barely documented and you regularly have to wade through hundreds of wannabe tutorials on Medium/Hackernoon/...
I don't think that incomplete documentation is is exclusive to Javascript by any means. It is quite possible that the standard for packages and articles is lower due to Javascript having a lower entry level. I think we saw a similar thing with PHP.
> one compiler/transpiler, no, there are several...
I personally don't see a problem with this? Usually they have different goals, pros and cons. Do we think that it would be beneficial to not create competing solutions? For example would it be beneficial to only have React and not have Angular, Vue, Svelte? Agreed that multiple solutions can be confusing for new users and generally I would direct new developers to all-in-one solutions so they don't need to deal with these things.
> some things like imports might work in completely different ways
I'm not quite sure what is being referred to here, maybe the difference between commonjs and es imports? I personally haven't encountered any issues but they are completely different. I might be able to point someone in the right direction if they clarify.
Again I understand if you don't like having to deal with these things, but there are reasons for them that can't just be glossed over. Discussions are great! But I find it hard to have them when developers ignore the nuances and insist on approaching them from a negative angle.
> browser incompatibilities are still prevalent
It's true, browser difference are a pain to work with. But what is the alternative, to only support a single browser engine? Note that this has improved a lot in recent years. As long as you don't need to support IE11 or below then you shouldn't come across many issues. You can find browser support using caniuse.com or looking at MDN docs.
> even popular libraries are barely documented and you regularly have to wade through hundreds of wannabe tutorials on Medium/Hackernoon/...
I don't think that incomplete documentation is is exclusive to Javascript by any means. It is quite possible that the standard for packages and articles is lower due to Javascript having a lower entry level. I think we saw a similar thing with PHP.
> one compiler/transpiler, no, there are several...
I personally don't see a problem with this? Usually they have different goals, pros and cons. Do we think that it would be beneficial to not create competing solutions? For example would it be beneficial to only have React and not have Angular, Vue, Svelte? Agreed that multiple solutions can be confusing for new users and generally I would direct new developers to all-in-one solutions so they don't need to deal with these things.
> some things like imports might work in completely different ways
I'm not quite sure what is being referred to here, maybe the difference between commonjs and es imports? I personally haven't encountered any issues but they are completely different. I might be able to point someone in the right direction if they clarify.
Again I understand if you don't like having to deal with these things, but there are reasons for them that can't just be glossed over. Discussions are great! But I find it hard to have them when developers ignore the nuances and insist on approaching them from a negative angle.
JavaScript is like lisp without the parentheses, and I didn't even like the parentheses. -Bryan Cantrill
They gave a laundry list of reason why it sucks so its a not a dislike based on personal preference. JS actually does suck.
I really like how javascript tooling appears to have followed the make it work -> make it good -> make it fast pattern, which is pretty much always the correct way to build software.
It is reassuring that we have actually got the feature sets and interfaces correct, since we are now, at last, turning to making stuff really fast. If we had started with speed being the goal of tools, I'd wager they would have been nowhere near the standard they are today in those regards.
It is reassuring that we have actually got the feature sets and interfaces correct, since we are now, at last, turning to making stuff really fast. If we had started with speed being the goal of tools, I'd wager they would have been nowhere near the standard they are today in those regards.
I've discovered a similar pattern, when I look at successful projects in my past. I prioritize "working", then "solid", then "fast" where it's needed.
Ironically, it's this principle that makes me look to non-Rust languages; I find that Rust's borrow checker forces us into faster but less flexible patterns up-front, and incurs a lot of refactoring friction later.
It's because of this that I'm interested in the PL space, and excited about upcoming languages like Lobster and Cone.
Ironically, it's this principle that makes me look to non-Rust languages; I find that Rust's borrow checker forces us into faster but less flexible patterns up-front, and incurs a lot of refactoring friction later.
It's because of this that I'm interested in the PL space, and excited about upcoming languages like Lobster and Cone.
That's just.. not how Rust works. I assume that after being frustrated by the Rust borrow checker, you chose to use easy to write wrappers like `Arc` and `RefCell`. I think it is something that beginners often do.
I wouldn't rewrite my project to some other language before I know how it is going to look like in the language. I had to be somewhat confident. If you find yourself frustrated by Rust's borrow checker, that tells me that you are not yet ready to port a project to Rust. You don't have the "Rust mindset" yet.
I wouldn't rewrite my project to some other language before I know how it is going to look like in the language. I had to be somewhat confident. If you find yourself frustrated by Rust's borrow checker, that tells me that you are not yet ready to port a project to Rust. You don't have the "Rust mindset" yet.
You make a lot of assumptions, and dismiss my valid opinion as uninformed. I would say that's arrogant and offensive, but I'm sure you meant well.
More to the point, in my many years of experience with Rust, I've found that if you're working on a stateless project, or a small project, or have a very clear picture up-front, you can know what data you will need to centralize and split ahead of time, and things will go smoothly. However, that's rarely the case in the parts of industry I've been in, where we need to prototype, refactor, and build out our programs constantly. In these cases, Rust is slower to work with compared to other languages, because the borrow checker makes our APIs brittle, causing refactors' blast radius to increase.
I'd be curious to hear if your architecture experience with large Rust projects might be different, I'm always open to more data points.
More to the point, in my many years of experience with Rust, I've found that if you're working on a stateless project, or a small project, or have a very clear picture up-front, you can know what data you will need to centralize and split ahead of time, and things will go smoothly. However, that's rarely the case in the parts of industry I've been in, where we need to prototype, refactor, and build out our programs constantly. In these cases, Rust is slower to work with compared to other languages, because the borrow checker makes our APIs brittle, causing refactors' blast radius to increase.
I'd be curious to hear if your architecture experience with large Rust projects might be different, I'm always open to more data points.
>make it good
Wait, when did that happen?
Wait, when did that happen?
Even "make it work" is debatable.
I'd say when ES6 landed. I used to hate JS, but since ES6, I think it's a pretty good language for building frontends.
ES6 is indeed pretty good, all things considered, but GP specified javascript tooling, which, uh, could still use some work before it's considered truly "good".
I'd say ES5 made it work and ES6+ made it good. I think by the time the Pattern Matching, and Record & Tuple proposals are completed and implemented it'll go to fast.
to be fair they didn't say it has completed the progression through that program.
This got a genuine laugh from me. Stockholm syndrome is a prerequisite for writing JS
I like how you've followed the delusional -> delusional pattern.
Author here. A few other Rust projects to note that I didn't mention in the original post I've since found:
- Boa (JS engine in Rust) – https://github.com/boa-dev/boa
- RSLint (JS/TS linter in Rust) – https://github.com/rslint/rslint
- Node version manager in Rust – https://github.com/Schniz/fnm
If you know of any other popular ones, let me know. I'm keeping a list :)
- Boa (JS engine in Rust) – https://github.com/boa-dev/boa
- RSLint (JS/TS linter in Rust) – https://github.com/rslint/rslint
- Node version manager in Rust – https://github.com/Schniz/fnm
If you know of any other popular ones, let me know. I'm keeping a list :)
I've been using NAPI-RS to bridge from Electron <-> Rust and it's been nice - https://napi.rs/ I believe SWC uses it behind the scenes.
They just put out an alpha release that makes it way more ergonomic to write Rust functions that interop with JS, too. There's a small example of it in the README: https://github.com/napi-rs/napi-rs#taste
NAPI isn't as well-known in the JS-to-Rust bindings world (Neon is the more well-known library, https://neon-bindings.com/) but it seems to be gaining steam.
They just put out an alpha release that makes it way more ergonomic to write Rust functions that interop with JS, too. There's a small example of it in the README: https://github.com/napi-rs/napi-rs#taste
NAPI isn't as well-known in the JS-to-Rust bindings world (Neon is the more well-known library, https://neon-bindings.com/) but it seems to be gaining steam.
Ah yes, how could I forget! NAPI-RS is very, very slick.
Volta is also written in Rust. I've only used it for managing Node and npm versions, but it can do more than that and describes itself as a "tool manager" rather than a node version manager.
https://docs.volta.sh/guide/
https://docs.volta.sh/guide/
Another one: Electron alternative Tauri written in Rust – https://news.ycombinator.com/item?id=26194990
I was interested to see that bun (https://bun.sh) is being created in Zig. "bun is like postcss, babel, node & webpack in one 100x faster tool for building modern web frontends."
People tend to write dev tools in the languages they develop in. C compilers written in C, JavaScript minifiers written in JavaScript, etc. There's often no reason to use the same language for the tooling, but you use what you know and like to work with, and if you've got a need for a language-specific tool (like Webpack) you're probably going to start out writing it in the same language.
If Rust becomes popular for JavaScript tooling, how long until those developers abandon JavaScript entirely? This isn't necessarily a problem, but it could lead to some, if you can't fix your own tools because you're not conversant in the language. More importantly, if the key toolmakers all switch to Rust and lose interest in JavaScript, the tool authors might lose touch with (or interest in) the problems of the JavaScript community. We all talk about dogfooding, and this is the opposite.
Rust seems to have this cult-like power. Rust developers are definitely drinking a lot of Kool-Aid. Maybe if all the JavaScript tools are written in Rust, we'll start seeing Rustscript in the browser.
If Rust becomes popular for JavaScript tooling, how long until those developers abandon JavaScript entirely? This isn't necessarily a problem, but it could lead to some, if you can't fix your own tools because you're not conversant in the language. More importantly, if the key toolmakers all switch to Rust and lose interest in JavaScript, the tool authors might lose touch with (or interest in) the problems of the JavaScript community. We all talk about dogfooding, and this is the opposite.
Rust seems to have this cult-like power. Rust developers are definitely drinking a lot of Kool-Aid. Maybe if all the JavaScript tools are written in Rust, we'll start seeing Rustscript in the browser.
> Rust seems to have this cult-like power. Rust developers are definitely drinking a lot of Kool-Aid.
The community is pretty exciting, new things are being written every day with the language, people are approaching it with optimism. Sure you can call it cult like and take the cynical route, I'm just going to be happy that I can write safe and performant code using a language with modern ergonomic features. I imagine this must have been what it was like in the early days of python and ruby.
The community is pretty exciting, new things are being written every day with the language, people are approaching it with optimism. Sure you can call it cult like and take the cynical route, I'm just going to be happy that I can write safe and performant code using a language with modern ergonomic features. I imagine this must have been what it was like in the early days of python and ruby.
It's an annoying cult, same as the Tesla Crypto NFT fan boys.
who are you and why do you matter again?
I'm not dissing Rust, I think it's great. I was just snarking about the rabid enthusiasm to emphasize that tool authors who switch to Rust might get so into the scene that they stop caring about JavaScript.
> Rust seems to have this cult-like power. Rust developers are definitely drinking a lot of Kool-Aid. Maybe if all the JavaScript tools are written in Rust, we'll start seeing Rustscript in the browser.
Rust has tremendous popularity because it solves an enormous class of problems for system programmers and require very few compromises compared to any alternative.
The hype is real, but it is very unlikely that another language will compete for this space.
Rust doesn't really work well in the niches most impacted by fad-driven-development. Nobody is claiming you should do web dev in Rust nor consider it comparable to much better languages for that space.
Rust has tremendous popularity because it solves an enormous class of problems for system programmers and require very few compromises compared to any alternative.
The hype is real, but it is very unlikely that another language will compete for this space.
Rust doesn't really work well in the niches most impacted by fad-driven-development. Nobody is claiming you should do web dev in Rust nor consider it comparable to much better languages for that space.
> Rust seems to have this cult-like power.
Rust is to lazy memory allocation as static types are to lazy typing.
A lot of people are content with dynamic typing until they see the other side. These people claim types make you think more. Ultimately, they reduce the maintenance burden and get rid of an entire class of bugs.
The same can be said of GC tuning.
Rust is a game changer, and it certainly doesn't hurt that the tooling and language ergonomics are also the best in class. They hit a home run with all of the bases loaded with this one.
Rust is to lazy memory allocation as static types are to lazy typing.
A lot of people are content with dynamic typing until they see the other side. These people claim types make you think more. Ultimately, they reduce the maintenance burden and get rid of an entire class of bugs.
The same can be said of GC tuning.
Rust is a game changer, and it certainly doesn't hurt that the tooling and language ergonomics are also the best in class. They hit a home run with all of the bases loaded with this one.
Go is active in this space too, esbuild is a very fast JavaScript bundler
- https://github.com/evanw/esbuild
- https://github.com/evanw/esbuild
Recently I've used ReScript + esbuild for a completely webpack-free experience with just a small node_modules folder
The more popular this kind of setup gets, the better. I've also vastly simplified a lot of my projects with just esbuild as the sole dev dependency.
our webpack based build took 70000ms (70s) to finished with a cached built. it only took 5s in esbuild (esbuild has no cache... or at least i don't know it yet).
> With Rust, developers have more control over memory allocation, without it being as painful as C++ or Go.
Memory allocation is painful in Go?
Memory allocation is painful in Go?
That was my takeaway from folks who had moved from Go to Rust. I'm not a Go expert though, so feel free to correct me.
https://blog.discord.com/why-discord-is-switching-from-go-to...
https://blog.discord.com/why-discord-is-switching-from-go-to...
Storing billions of states in garbage-collected memory sounds like using a wrong tool for the job (by the way, I wonder why they didn't use something like Redis? didn't scale well?). I think it's an edge case they ran into -- which doesn't mean Go's memory management is painful per se. It's not secret that large heaps with many old generation objects are problematic in presence of GC. And few projects have requirements of the scale of Discord. For stateless servers, Go is more than enough in my experience; in our services we do allocate garbage like there's no tomorrow and we're yet to run into GC problems, because all caching is delegated to a redis cluster and all garbage we produce during request processing is ephemeral so we don't have large object graphs that would slow down the GC. Rust is perfect for this kind of "core infrastructure" projects where you need maximum performance (Redis itself is written in C), but actual business logic on top of such infrastructure can easily be written in a simpler language like Go and GC is rarely a bottleneck there.
According to Redis themselves, a high end and well tuned redis server will get you ~200k requests per second[1]. This is if all your application does is send requests in a loop.
Am I correct that redis does not offer a native API and so forces you to craft commands as strings? Just having to run snprintf 500k per second wastes a considerable amount of CPU. Combined with the normal overhead of communicating over a socket, using redis is going to be considerably slower than just keeping that memory in your own process. That's assuming the overhead of the redis internals is 0.
[1] https://redis.io/topics/benchmarks
Am I correct that redis does not offer a native API and so forces you to craft commands as strings? Just having to run snprintf 500k per second wastes a considerable amount of CPU. Combined with the normal overhead of communicating over a socket, using redis is going to be considerably slower than just keeping that memory in your own process. That's assuming the overhead of the redis internals is 0.
[1] https://redis.io/topics/benchmarks
The trick is to precompute the Redis command strings whenever possible. We (not going to name names here) use Redis at above that scale. Our keys all have a fixed size, so we preallocate a buffer that can hold the command and the key, clear (so move the write pointer back to the beginning of the key) it on every new request, write directly into the buffer, and then pass a pointer to the buffer when writing to the socket. This is a similar trick we use whenever we're constructing load tests as a lot of the time can be spent creating request bodies instead of just pushing bytes onto the socket as fast as possible.
That sounds like it is written in highly optimized C, not in Rust?
You can do this in Rust with a mut pointer into the string. You can even do it in Go with a byte buffer. Lots of languages give you the tools to do stuff like this if you look. I hear Python's StringIO does similar things but I'm not as familiar with Pythons runtime in these situations.
In the article, they say:
>The service we switched from Go to Rust is the “Read States” service. Its sole purpose is to keep track of which channels and messages you have read
>On cache key eviction, we commit your Read States to the database. We also schedule a database commit for 30 seconds in the future whenever a Read State is updated.
So it's a separate service whose only purpose is to store data in-memory with occasional commits to persisted DB, which sounded a lot like Redis. But of course, a special optimized service is always going to outperform a general purpose tool such as Redis.
They also replaced HashMaps with BTreeMaps, reduced the number of memory copies they were doing, etc. -- which has nothing to do with Go, but also contributed to a performance increase
>The service we switched from Go to Rust is the “Read States” service. Its sole purpose is to keep track of which channels and messages you have read
>On cache key eviction, we commit your Read States to the database. We also schedule a database commit for 30 seconds in the future whenever a Read State is updated.
So it's a separate service whose only purpose is to store data in-memory with occasional commits to persisted DB, which sounded a lot like Redis. But of course, a special optimized service is always going to outperform a general purpose tool such as Redis.
They also replaced HashMaps with BTreeMaps, reduced the number of memory copies they were doing, etc. -- which has nothing to do with Go, but also contributed to a performance increase
What is a native API? Sending string commands and parsing the response is as much of an API as a HTTP API is.
It's not a library interface if you mean that - but that always applies for distributed systems.
It's not a library interface if you mean that - but that always applies for distributed systems.
Yeah, by native API I meant a native library that would allow you to call into redis directly. Native being the qualifier - as opposed to an HTTP API, which is not native.
The takeaways in this article are much more why Go's memory model becomes difficult at scale. When you're dealing with memory at Discord scale garbage collection is hard, for example on the mentioned Discord microservice:
> There are millions of Users in each cache. There are tens of millions of Read States in each cache. There are hundreds of thousands of cache updates per second.
This is something that is probably never going to be hit locally in the development toolchain. You can certainly prefer the Rust memory system to Go and have that be a valid reason to use Rust over Go for something like dev toolchains, but you're not going to hit scale problems like those mentioned in the article.
> There are millions of Users in each cache. There are tens of millions of Read States in each cache. There are hundreds of thousands of cache updates per second.
This is something that is probably never going to be hit locally in the development toolchain. You can certainly prefer the Rust memory system to Go and have that be a valid reason to use Rust over Go for something like dev toolchains, but you're not going to hit scale problems like those mentioned in the article.
Not just at scale, Go doesn't make it obvious when an allocation is on the heap or stack. The escape analysis can change (hopefully without regressions) from version to version of Go, and that can mean performance regressions can happen. It also means small refactors risk de-optimizing code without an obvious reason why that would be the case.
I think more engineers are becoming skeptical of optimizing compilers with that sort of implicit, potential "spooky action at a distance" where nonlocal code can cause performance regressions.
In Haskell, laziness and iterator fusion can do the same thing. Nonlocal code can cause local expressions to compile differently, resulting in - from an engineer's POV - nondeterministic performance, memory usage, GC pressure. "Space leaks" can also occur, again, due to nonlocal effects.
In JavaScript, the JIT can similarly cause local regressions due to nonlocal code. A single expression deep in a call stack mutating an object can result in it no longer fitting an optimized "shape", resulting in function deoptimization.
Just say no to spooky action at a distance in programming languages, in my opinion. It leads to tremendously difficult to debug regressions.
I think more engineers are becoming skeptical of optimizing compilers with that sort of implicit, potential "spooky action at a distance" where nonlocal code can cause performance regressions.
In Haskell, laziness and iterator fusion can do the same thing. Nonlocal code can cause local expressions to compile differently, resulting in - from an engineer's POV - nondeterministic performance, memory usage, GC pressure. "Space leaks" can also occur, again, due to nonlocal effects.
In JavaScript, the JIT can similarly cause local regressions due to nonlocal code. A single expression deep in a call stack mutating an object can result in it no longer fitting an optimized "shape", resulting in function deoptimization.
Just say no to spooky action at a distance in programming languages, in my opinion. It leads to tremendously difficult to debug regressions.
> the escape analysis can change (hopefully without regressions) from version to version of Go, and that can mean performance regressions can happen.
Because it's not the case with Rust or any compiled language? LLVM has shown regressions wich is what Rust uses to compile code.
2sec search on google: https://github.com/rust-lang/rust/issues/24194
Because it's not the case with Rust or any compiled language? LLVM has shown regressions wich is what Rust uses to compile code.
2sec search on google: https://github.com/rust-lang/rust/issues/24194
Of course there have been regressions, but the semantics of the code don't typically change. A "dead heap allocation", or removing an unused call to malloc, is an interesting example but I think you'd be hard pressed to find a case where it caused a massive performance regression.
I think there may be some edge cases (constant folding, dead code removal) of course, but a change to LLVM cannot cause the 3 sorts of changes I identified in Go, Haskell, and JavaScript.
I think there may be some edge cases (constant folding, dead code removal) of course, but a change to LLVM cannot cause the 3 sorts of changes I identified in Go, Haskell, and JavaScript.
>Not just at scale, Go doesn't make it obvious when an allocation is on the heap or stack. The escape analysis can change
I thought heap allocations are only triggered by taking pointers, i.e. using reference semantics (including casting to interface and calling a method whose receiver is by-ref)? If I'm careful to only use variables by-value (for types that can afford it, i.e. excluding map/array), what else can trigger heap allocation?
I thought heap allocations are only triggered by taking pointers, i.e. using reference semantics (including casting to interface and calling a method whose receiver is by-ref)? If I'm careful to only use variables by-value (for types that can afford it, i.e. excluding map/array), what else can trigger heap allocation?
Making a closure usually requires allocation. For example:
package main
import "fmt"
func main() {
closure := make_closure()
fmt.Println("closure():", closure())
}
func make_closure() func() int {
x := 1
return func() int { return x }
}
$ go run main.go
closure(): 1
Oh yes, closures too. Coming from C++ background, you kinda intuitively understand when things escape and when they don't, because you assume that Go would implement it in the most straightforward way (like it usually does), and in the case of closures, heap-allocating a captured variable is the simplest implementation (just let GC handle it!) considering the implicit reference semantics (that you can change "x" anywhere outside of the closure and it should be immediately visible in the closure, too). I.e. assume the worst (and don't expect Go to do some clever special-casing similar to move semantics in C++/Rust), and it's often the right answer :)
I don't know, building JavaScript things involves similar orders of magnitude.
Two things about this pain:
* They incur the pain even when nothing needs doing. This is really terrible. If you spend ten minutes furiously reading data from a network device into a fixed RAM buffer, twiddling it, and writing it out again, in Rust (or C, C++ and various other explicit allocation languages) you neither allocated nor de-allocated anything, therefore you pay exactly zero for this work you didn't do. In Go you will incur the cost of the GC verifying that yup, everything is as it was, there's nothing to do, apparently five times.
* The pain is swallowed in lumps, you can't spread it out. So you must either size hardware for lump swallowing, even though as we saw above we don't even want to swallow any lumps, or accept poor performance each time a lump is swallowed. With explicit allocation you can choose to swallow bigger lumps (e.g. arena allocation) to reduce overheads elsewhere, but you decide on the size of lumps you want for your allocation.
* They incur the pain even when nothing needs doing. This is really terrible. If you spend ten minutes furiously reading data from a network device into a fixed RAM buffer, twiddling it, and writing it out again, in Rust (or C, C++ and various other explicit allocation languages) you neither allocated nor de-allocated anything, therefore you pay exactly zero for this work you didn't do. In Go you will incur the cost of the GC verifying that yup, everything is as it was, there's nothing to do, apparently five times.
* The pain is swallowed in lumps, you can't spread it out. So you must either size hardware for lump swallowing, even though as we saw above we don't even want to swallow any lumps, or accept poor performance each time a lump is swallowed. With explicit allocation you can choose to swallow bigger lumps (e.g. arena allocation) to reduce overheads elsewhere, but you decide on the size of lumps you want for your allocation.
Sure when you disable the GC =P
Perhaps memory allocation is not, but control over memory allocation is. Like most other managed languages, Go trades less control for more ease of use.
There is a lot of pain over garbage collection and tuning that in really high pref cases
Definitely not painful for simple operations (obviously).
But every large Go project I've worked on has eventually invested a lot of time into minimizing allocations, rewriting critical path sections to use pools, and various other techniques to improve performance by taking more control over memory allocation. Go's GC is very good, but it still has limitations.
Honestly it's not a bad tradeoff most of the time. Write code that works as quickly as possible, profile it as load scales, and apply further optimization as hot paths are identified. On the other hand, I've been working with Rust long enough that I can almost write the same thing in Rust as fast as I could in Go (Rust is inherently more verbose and requires more cognitive overhead). I've been writing prototypes in Rust more and more lately and haven't really been missing Go as much as I did when I first started with Rust.
But every large Go project I've worked on has eventually invested a lot of time into minimizing allocations, rewriting critical path sections to use pools, and various other techniques to improve performance by taking more control over memory allocation. Go's GC is very good, but it still has limitations.
Honestly it's not a bad tradeoff most of the time. Write code that works as quickly as possible, profile it as load scales, and apply further optimization as hot paths are identified. On the other hand, I've been working with Rust long enough that I can almost write the same thing in Rust as fast as I could in Go (Rust is inherently more verbose and requires more cognitive overhead). I've been writing prototypes in Rust more and more lately and haven't really been missing Go as much as I did when I first started with Rust.
Sure maybe, but here we are talking about short running CLI applications not long running processes. IMO, Go is better suited at this 'js infrastructure' stuff.
When you work at certain scale with C++ and Rust you ends rewriting a GC inside those language because default allocator are not good enough. That's why GC languages are faster than C++/Rust in some scenarios.
>Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
- Greenspun's tenth rule
Joking aside, Java VM can be as fast as C++, not C, by using profiling and JIT optimization if tuned to a specific server and only measuring after running for a while. If you are really thinking about your allocators and not just calling malloc, nothing is faster. Half of this is due to less allocator calls and half is due to laying out the memory in a cache-friendly manner.
- Greenspun's tenth rule
Joking aside, Java VM can be as fast as C++, not C, by using profiling and JIT optimization if tuned to a specific server and only measuring after running for a while. If you are really thinking about your allocators and not just calling malloc, nothing is faster. Half of this is due to less allocator calls and half is due to laying out the memory in a cache-friendly manner.
I doubt that; for the foreseeable future, the future of JavaScript infrastructure will likely continue to be TypeScript and C++.
deno is too opinionated to absorb a significant number of node.js developers that just want to get stuff done.
Sure, there will be tools written in Rust (because it's the new sexy thing) just like there were tools written in Go (when it was the "new sexy thing"), but it's not going to take over the ecosystem any time soon.
deno is too opinionated to absorb a significant number of node.js developers that just want to get stuff done.
Sure, there will be tools written in Rust (because it's the new sexy thing) just like there were tools written in Go (when it was the "new sexy thing"), but it's not going to take over the ecosystem any time soon.
Agreed, many people enjoy writing bindings and creating tools, however the large majority will just pick what is supported out of the box and carry on with their actual work.
Nice writeup, Lee.
Zig is also pretty interesting for this type of tooling, see https://bun.sh/
Thank you! Jarred is doing incredible work, worth following on Twitter too: https://twitter.com/jarredsumner
The use of the word 'modern' is inappropriate in these sorts of articles.
> It's a modern replacement for languages like C++ or C with a focus on code safety and concise syntax.
Instead: "It's a replacement..."
> Babel: developers wanted to write modern JavaScript while supporting older browsers.
Instead: "developers want to write with new versions of Javascript..."
> Deno, created in 2018, is a simple, modern, and secure runtime for JavaScript and TypeScript that uses V8 and is built with Rust.
Instead: "Deno, created in 2018, is a runtime for JavaScript and TypeScript that uses V8 and is built with Rust, with a focus on Security."
> WebAssembly (WASM) is a portable low-level language that Rust can compile to. It runs in the browser, is interoperable with JavaScript, and is supported in all major modern browsers.
Instead: "all major browsers."
> It's a modern replacement for languages like C++ or C with a focus on code safety and concise syntax.
Instead: "It's a replacement..."
> Babel: developers wanted to write modern JavaScript while supporting older browsers.
Instead: "developers want to write with new versions of Javascript..."
> Deno, created in 2018, is a simple, modern, and secure runtime for JavaScript and TypeScript that uses V8 and is built with Rust.
Instead: "Deno, created in 2018, is a runtime for JavaScript and TypeScript that uses V8 and is built with Rust, with a focus on Security."
> WebAssembly (WASM) is a portable low-level language that Rust can compile to. It runs in the browser, is interoperable with JavaScript, and is supported in all major modern browsers.
Instead: "all major browsers."
It's futile, from the history we can expect that the next major generation of browsers would be called postmodern browsers, and next one would be post-postmodern browsers.
I only use Pre-Raphaelite browsers.
now this is the kind of pedantry i come to hn for
Front end engineer, here. Not very experienced, less than 10 years exp.
Dumb question - why do we still need compilers like Babel and SWC to compile our es6 (excluding TS) to...whatever version of js the compiler is targeting? Are browsers still so out of sync with each other that we still need to compile to es5? If yes, wouldn't it be more worthwhile for the front end community, including TC39 and every major browser engine developer (google, apple, mozilla, etc), to focus on the base implementation (engine) so they are no longer so out of sync? The most popular browser are based on open source engines anyway.
Don't get me wrong, I am sure there is a good reason for all this rust tooling - faster compilation does not hurt, but it seems inefficient to have an intermediate compiler (Babel / swc/ TS/ etc) that creates deployable code that will eventually be interpreted by the user's JS engine (V8 / webkit / gecko) anyway.
A standardized engine seems the way to go and it seems like we are going in that direction with only a handful of JS engines in use today.
Dumb question - why do we still need compilers like Babel and SWC to compile our es6 (excluding TS) to...whatever version of js the compiler is targeting? Are browsers still so out of sync with each other that we still need to compile to es5? If yes, wouldn't it be more worthwhile for the front end community, including TC39 and every major browser engine developer (google, apple, mozilla, etc), to focus on the base implementation (engine) so they are no longer so out of sync? The most popular browser are based on open source engines anyway.
Don't get me wrong, I am sure there is a good reason for all this rust tooling - faster compilation does not hurt, but it seems inefficient to have an intermediate compiler (Babel / swc/ TS/ etc) that creates deployable code that will eventually be interpreted by the user's JS engine (V8 / webkit / gecko) anyway.
A standardized engine seems the way to go and it seems like we are going in that direction with only a handful of JS engines in use today.
In a perfect world, things would be like you describe.
Unfortunately we live in a diverse landscape of browsers and devices, many of which have not been updated in quite some time. Factors ranging from IT policy at older companies, to geographical limitations, may prevent many users from upgrading to the latest and greatest.
Some companies choose to cut their losses, and support only certain browsers - these can get away with less compilation and polyfills. Other companies may be targeting these specific segments (such as B2B), and can't get away with such a luxury.
Unfortunately we live in a diverse landscape of browsers and devices, many of which have not been updated in quite some time. Factors ranging from IT policy at older companies, to geographical limitations, may prevent many users from upgrading to the latest and greatest.
Some companies choose to cut their losses, and support only certain browsers - these can get away with less compilation and polyfills. Other companies may be targeting these specific segments (such as B2B), and can't get away with such a luxury.
But we are so close - the wound has almost healed, and instead of taking off the band aid, we are putting on another band aid.
How are we 'so close' though? Browser fragmentation hasn't really decreased at all, perhaps even the opposite.
Browser competition is one that sparks inovation. Each vendor wants to be the leader. So they compete.
And also anyone want to browse web using any browser they have at the time, it may be outdated.
And also anyone want to browse web using any browser they have at the time, it may be outdated.
While a compiler-free world sounds ideal, my take is there will always be different, newer, syntax developers will want to write that isn't natively supported by the browser. For example, JavaScript language proposals that are maybe in stage 1 instead of the later stages.
Overall, browsers have gotten a lot better at implementing and supporting a common set of web APIs. There's still work to be done, but we're in a far better place than we were five years ago (in my opinion).
100% agree with your sentiment though.
Overall, browsers have gotten a lot better at implementing and supporting a common set of web APIs. There's still work to be done, but we're in a far better place than we were five years ago (in my opinion).
100% agree with your sentiment though.
Because there are old computers and phones with old browsers that are still running today and will not be upgraded. Many companies cannot risk upgrading and many consumers have not upgraded or bought a newer device.
The old browsers are on ES5.
The old browsers are on ES5.
There's many reasons to have a JS compilation step:
- Still targeting legacy browsers like IE11
- Using React / JSX syntax
- Using TypeScript
- Using other brand-new JS syntax that hasn't trickled out to wider support yet
- Using custom JS transforms for libraries like `styled-components` (see this Next.js discussion thread for examples of the many different non-standard Babel transformations that people are using: https://github.com/vercel/next.js/discussions/30174 )
That said, the ideal target language version _is_ getting better. Jason Miller ( creator of Preact) has done a lot of research on both browser support for ES2017, as well as tooling to reverse-compile ES5-targeted published libraries upwards to ES2017 syntax again for improved bundle size:
- https://jasonformat.com/enabling-modern-js-on-npm/
- https://web.dev/publish-modern-javascript/
- Still targeting legacy browsers like IE11
- Using React / JSX syntax
- Using TypeScript
- Using other brand-new JS syntax that hasn't trickled out to wider support yet
- Using custom JS transforms for libraries like `styled-components` (see this Next.js discussion thread for examples of the many different non-standard Babel transformations that people are using: https://github.com/vercel/next.js/discussions/30174 )
That said, the ideal target language version _is_ getting better. Jason Miller ( creator of Preact) has done a lot of research on both browser support for ES2017, as well as tooling to reverse-compile ES5-targeted published libraries upwards to ES2017 syntax again for improved bundle size:
- https://jasonformat.com/enabling-modern-js-on-npm/
- https://web.dev/publish-modern-javascript/
Excellent summary. Thank you!
It seems to me the future is rather in gradually replacing JS engines with a WASM back-end. The whole webpack/babel ecosystem could be replaced by a toolchain which targets WASM, and browsers could ship with a JS engine which just translates es5 to WASM for legacy support.
I guess the WASM api is lacking a lot of features which would be required to make this work, but I don't see why it shouldn't be the goal.
I guess the WASM api is lacking a lot of features which would be required to make this work, but I don't see why it shouldn't be the goal.
JS is too complex for that. But you can have a look at AssemblyScript.
It's more of an issue on the end user's side. All the main browsers have had support for ES6 and newer features for years. But that doesn't mean end users are running the latest versions of those browsers.
So when you transpile/polyfill a feature, you're aiming to support that small number of users who haven't updated their browser in 5+ years. If you don't care about those people, you have no need for that intermediate step. Or if you want to be really sophisticated, you can look at the user agent for an incoming request and serve a response that's transpiled/polyfilled exactly as much as is needed.
At this point, if you are only targeting modern browsers, it's more for the bundler than the transpiler. The module loader standard is still a mess, mostly because there is little incentive to fix it. Unless you are using HTTP/2 or another streaming connection, loading the hundreds of project files and thousands of dependencies, is just not feasible. The compilers also do a bunch of tree shaking to remove dead code from your dependency graph.
It will be interesting to see a tool that does dead code elimination (tree shaking) and minimization, then writing back individual es6 module files for http2. Maybe one of tools can be configured to do so now?
The Rollup tool supports that in conjunction with the `preserveModules` option. I've never used it outside of playing around, so can't vouch for it from experience, but yeah it exists. It can also do more of course, like convert to older module formats or the like, but obviously you don't need to use that.
The biggest downside is that even in http2 or http3, bundles can still perform better in terms of lower overall loading times, since individual modules you need to download the first one, and perform at least a bit of parsing to find the other modules statically imported before you can start downloading those, repeat.
The biggest downside is that even in http2 or http3, bundles can still perform better in terms of lower overall loading times, since individual modules you need to download the first one, and perform at least a bit of parsing to find the other modules statically imported before you can start downloading those, repeat.
Right, though you could use HTTP2 push, but to do that right… you would need a tool to know which files are bundled and their order and… it looks like today.
With one exception though… cache invalidation of a single source file would not invalidate an entire bundle or chunk.
With one exception though… cache invalidation of a single source file would not invalidate an entire bundle or chunk.
The important part is to know your target. I'm currently working on a project that has a complex webpack/babel setup. The reason for the complicated toolchain? I have no idea. Our target (Electron) supports ES6+ syntax yet we're still compiling down to ES5. The only other reason I can think is that the developers wanted ES2020+ features, but this, in my mind, is a bad tradeoff. The toolchain has broken more than once for me. Realistically, all we need is `tsc` and `esbuild`.
I've been a JS developer for almost 10 years. I'm convinced young hip developers are just addicted to tooling.
I've been a JS developer for almost 10 years. I'm convinced young hip developers are just addicted to tooling.
The answer is usually that whoever set up webpack grabbed the configuration from another project, and nobody wanted to touch it since then because it worked and it looks scary.
Young developers don’t know what they don’t know. If they don’t see editing the webpack configuration as part of their job, they’ll probably just never touch it.
And that means it’s your job to go fix it! Compiling down to es5 can have terrible performance implications for the resulting application - particularly if any of the bundled code touches typed arrays. There’s some free performance wins sitting on the table. If your coworkers are too junior, it falls to you to fix the build.
Young developers don’t know what they don’t know. If they don’t see editing the webpack configuration as part of their job, they’ll probably just never touch it.
And that means it’s your job to go fix it! Compiling down to es5 can have terrible performance implications for the resulting application - particularly if any of the bundled code touches typed arrays. There’s some free performance wins sitting on the table. If your coworkers are too junior, it falls to you to fix the build.
Electron uses node and node support for es6 modules is new, and electron itself is just putting its toes in the water. Same with Typescript. Are you using .mts .mtsx files?
Node support for ES6 modules has been out for years...
Es modules are one of hundreds of features that have been added to javascript since es5. Others include typed arrays, promises, async/await, and so on. Babel configured to recompile code to es5 will polyfill all of those JS features. So the highly efficient V8 implementation of Uint8Array is replaced with slow javascript. Code using async await will be rewritten to use a promise polyfill and callbacks. And so on. The resulting code will be bigger and slower. Sometimes much slower depending on what it’s doing.
List of features in each version of JS: https://kangax.github.io/compat-table/es2016plus/ (also check the ES6 tab)
List of features in each version of JS: https://kangax.github.io/compat-table/es2016plus/ (also check the ES6 tab)
> Realistically, all we need is `tsc` and `esbuild`.
esbuild and babel/webpack do basically the same thing. esbuild can also target older versions of JavaScript (just not es5).
esbuild and babel/webpack do basically the same thing. esbuild can also target older versions of JavaScript (just not es5).
Which is great! And it even takes care of JSX/TS transpilation.
I'm confused about what your point is then since either way you are using some system in between your code and the browser or Node.
You'd likely still need a bundler for vendor packages. esbuild ships as a static binary, is a single dependency, and is one of the fastest bundlers around. It's also pretty much zero-config. That other stuff is just icing on top. I don't know why someone would pick webpack + babel + the endless plugins and config involved over that.
The best way to drive a wedge between young and old developers is to negatively attribute something to a specific age group.
So keep going, you're on the right track.
So keep going, you're on the right track.
Point taken.
There might be one reason: compiling from ES6+ to ES5 transforms for of loops in regular for loops, which are faster.
Also, if we are going through a compile step anyway, why not a future for more support for WebAssembly and then we can use other languages instead of JavaScript.
You have to realize that it is not at all in the interest of most commercial web applications to be transparently delivered as ES modules, all neatly split up and perhaps lightly minified. No, the fact that your 1000+ constantly version churning NPM dependencies and your proprietary business code are all bundled up in a big undecipherable blob is seen as a feature. The only thing that I can see that is going to replace it, is probably going to be WASM or a binary format blob, or the application code executing entirely on the backend and the frontend streamed like Netflix.
You don't need Babel anymore. I recently updated a project of mine to only use Babel on jsx files only for jsx transform. webpack handles everything
Let's say we had no need for babel or similar level transpilers.
There would still be benefit to webpack-like bundlers, if for no other reason then for performance/obfuscation reasons we would probably want the number of modules as seen by the broswer to be smaller than the number of modules that developers like to use for development purposes.
Many would still want minification for both performance and obfuscation reasons.
Many would still want to tree shake out any unused code from the libraries used (again for performance reasons).
Once one have all of that tooling, adding a babel like transpiler to the stack is not that big a deal.
There would still be benefit to webpack-like bundlers, if for no other reason then for performance/obfuscation reasons we would probably want the number of modules as seen by the broswer to be smaller than the number of modules that developers like to use for development purposes.
Many would still want minification for both performance and obfuscation reasons.
Many would still want to tree shake out any unused code from the libraries used (again for performance reasons).
Once one have all of that tooling, adding a babel like transpiler to the stack is not that big a deal.
Rome is also being rewritten into Rust.
https://rome.tools/blog/2021/09/21/rome-will-be-rewritten-in...
https://rome.tools/blog/2021/09/21/rome-will-be-rewritten-in...
Good luck on your rewrite!
Rust is good for systems programming. For everything else, use a language with GC so you don't have to reinvent the wheel for every memory allocation and your brain is free to think about functionality, architecture, security, UX, ...
Indeed, something like having a micro-kernel or composition engine written in Rust, everything else on top should just use a language with any kind of automatic memory management and value types support.
I agree that wasm should be the future of web, but to be honest I'm not sure rust is the best candidate to replace JS from the developer perspective.
For one thing, Rust is slow to compile. I built a simple one-page app in rust/wasm to kick the tires, and it was much slower from a developer perspective than for instance a react/ts app. Even with hot reloading set up, it took noticeably longer for the rust app to refresh after hitting the save button.
Also rust is fairly demanding on the programmer. Right now there are an army of js programmers keeping the web running. To be perfectly honest, I don't think there is sufficient supply of programmers in the world with the aptitude to replace all those js programmers with rust programmers. Due to the sheer scale of the web, front-end programming has to be fairly easy for things to keep moving at the pace which they are.
I hope wasm does supplant js as the main deployment target for web development. But I think it will have to be a much simpler language than rust which is the one targeting wasm.
For one thing, Rust is slow to compile. I built a simple one-page app in rust/wasm to kick the tires, and it was much slower from a developer perspective than for instance a react/ts app. Even with hot reloading set up, it took noticeably longer for the rust app to refresh after hitting the save button.
Also rust is fairly demanding on the programmer. Right now there are an army of js programmers keeping the web running. To be perfectly honest, I don't think there is sufficient supply of programmers in the world with the aptitude to replace all those js programmers with rust programmers. Due to the sheer scale of the web, front-end programming has to be fairly easy for things to keep moving at the pace which they are.
I hope wasm does supplant js as the main deployment target for web development. But I think it will have to be a much simpler language than rust which is the one targeting wasm.
> I agree that wasm should be the future of web, but to be honest I'm not sure rust is the best candidate to replace JS from the developer perspective.
Rust & wasm don't replace JS, in my view. They're there to create good libraries & frameworks for the scripting languages to consume & use, at many times.
There will definitely be some Rust entering the scene, but I tend to think Rust & other wasms will not be the entire web app, but be used for engineering the better crafted, more re-usable bits.
Wasms success, imo, is ultimately gated on how well it's host-object bindings work. Wasm doesn't really yet play well with JS. Rust has a fairly unique edge right now in that Rust pioneered their own wasm<->js foreign interfacing, via wasm-bindgen, then generated a web-sys crate with all the web foreign objects already defined in it, allowing Rust far better inter-operation.
Because host-object binding/ interface types/ &c are still in-progress, there's much less ability to experiment with what pieces of the webapp make sense to keep in JS, and what pieces make sense to go engineer in wasm. We'll see more interesting blends, with more webdevs writing JS, but consuming/authoring against Wasm tools, as we start to emerge a real wasm ABI.
Rust & wasm don't replace JS, in my view. They're there to create good libraries & frameworks for the scripting languages to consume & use, at many times.
There will definitely be some Rust entering the scene, but I tend to think Rust & other wasms will not be the entire web app, but be used for engineering the better crafted, more re-usable bits.
Wasms success, imo, is ultimately gated on how well it's host-object bindings work. Wasm doesn't really yet play well with JS. Rust has a fairly unique edge right now in that Rust pioneered their own wasm<->js foreign interfacing, via wasm-bindgen, then generated a web-sys crate with all the web foreign objects already defined in it, allowing Rust far better inter-operation.
Because host-object binding/ interface types/ &c are still in-progress, there's much less ability to experiment with what pieces of the webapp make sense to keep in JS, and what pieces make sense to go engineer in wasm. We'll see more interesting blends, with more webdevs writing JS, but consuming/authoring against Wasm tools, as we start to emerge a real wasm ABI.
I've worked in JS for around 12 years now and welcome this change of keeping JS as a scripting language but incrementally retiring it for performance-intensive general-purpose applications.
To my eye, JavaScript has three applications:
* A GUI-building language: A language and ecosystem we use to build UIs, where code runs on the end user's machine, served over the Web or as desktop applications using a compatible API (e.g. Electron).
* A scripting language: An approachable high-level C-like language.
* A general purpose programming language: A server-side or developer machine-side programming language for computational tasks like linking dependencies, transpiling code, type checking, orchestrating tasks.
At the first it's unbeatable, and the ergonomics and ecosystem are phenomenal. This is why it "won" as a language to write even desktop apps like VSCode and Discord.
At the second, performance doesn't matter anyway, so making JS capable of this just makes it approachable for folks who learned the language for the GUI-building purpose.
It's at the third application that the fundamental limitations of JavaScript---the programming language---really show. These are things like the lack of true multithreading (technically possible with significant limitations around shared memory and messaging and high overhead with Web Workers), lack of low-level primitives to fine-tune performance, and the ease with which authors can accidentally write extremely slow code. So much of this (TypeScript compilation, NPM package resolution/linking, Webpack bundling) in the modern Web ecosystem is regrettably still done in mostly JS.
I hope the JS community rallies around one general-purpose language to offload these tasks to as well as a uniform shim layer like Ruby's C extensions so we don't end up with a chaotic mishmash of different technologies. If nothing else, it would be nice to have a good excuse to write not-JS every once in a while!
To my eye, JavaScript has three applications:
* A GUI-building language: A language and ecosystem we use to build UIs, where code runs on the end user's machine, served over the Web or as desktop applications using a compatible API (e.g. Electron).
* A scripting language: An approachable high-level C-like language.
* A general purpose programming language: A server-side or developer machine-side programming language for computational tasks like linking dependencies, transpiling code, type checking, orchestrating tasks.
At the first it's unbeatable, and the ergonomics and ecosystem are phenomenal. This is why it "won" as a language to write even desktop apps like VSCode and Discord.
At the second, performance doesn't matter anyway, so making JS capable of this just makes it approachable for folks who learned the language for the GUI-building purpose.
It's at the third application that the fundamental limitations of JavaScript---the programming language---really show. These are things like the lack of true multithreading (technically possible with significant limitations around shared memory and messaging and high overhead with Web Workers), lack of low-level primitives to fine-tune performance, and the ease with which authors can accidentally write extremely slow code. So much of this (TypeScript compilation, NPM package resolution/linking, Webpack bundling) in the modern Web ecosystem is regrettably still done in mostly JS.
I hope the JS community rallies around one general-purpose language to offload these tasks to as well as a uniform shim layer like Ruby's C extensions so we don't end up with a chaotic mishmash of different technologies. If nothing else, it would be nice to have a good excuse to write not-JS every once in a while!
> At the first it's unbeatable, and the ergonomics and ecosystem are phenomenal. This is why it "won" as a language to write even desktop apps like VSCode and Discord.
I don't really think this is attributed to JS in any way besides it being the only browser language. Desktop apps use JS because they are browsers, not necessarily because JS was the best language option. An ecosystem has developed around it not because it's good, but because there is no other option for working in a browser.
I don't really think this is attributed to JS in any way besides it being the only browser language. Desktop apps use JS because they are browsers, not necessarily because JS was the best language option. An ecosystem has developed around it not because it's good, but because there is no other option for working in a browser.
It's both, deal with it.
The state of Rust debugging is atrocious. No expression evaluators work with traits, and their implementations are the majority of all Rust code.
Until somebody fixes this, I don't see Rust taking on any mainstream languages.
Until somebody fixes this, I don't see Rust taking on any mainstream languages.
Ha. I hadn't even thought about this. In several months of writing Rust I have never once reached for a debugger, but you're right, the debuggers, at least the ones I have, don't understand how to evaluate Traits.
Maybe some day I'll write a Rust program that has a bug worth using a debugger on and I'll remember this.
Maybe some day I'll write a Rust program that has a bug worth using a debugger on and I'll remember this.
Well I hit this on my 2nd week of Rust trying to debug GPT-1 implementation in tch-rs. Sprinkling code with extra variables holding debug values I'd normally put as expressions in IDE Watches and restarting only works until assert violation happens after 5+ mins of program execution.
> The Rust version probably could be made to work at an equivalent speed with enough effort. But at a high level, Go was much more enjoyable to work with.
Wow. I always thought Go was a language for depressed (or depressing? :)) devs ;) . I mean, it's boring _by design_. On the other hand Rust is really a cool language, with many things that are designed cleverly, and really enjoyable to use. Well, to me, at least, as it seems some people like boring more :)
Wow. I always thought Go was a language for depressed (or depressing? :)) devs ;) . I mean, it's boring _by design_. On the other hand Rust is really a cool language, with many things that are designed cleverly, and really enjoyable to use. Well, to me, at least, as it seems some people like boring more :)
It probably isn't. Simply because JavaScript has a low barrier to entry and Rust has a high barrier to entry. It would be a bit like saying LISP or Erlang are the future of BASIC or PHP.
Reading some of the comments here has really made me feel better. I felt like I was the only one that felt JS ecosystem is just bad and painful to work with. Thanks everyone.
Rust probably will have its place, but it is useless to assume that it will oust code that is already used and trusted. Even the people at Firefox figured out that it would be near impossible to replace their C++ infrastructure to use something unproven. I guess rust advocates would have an easier time creating something new and successful rather than preaching that people should throw their code in the garbage just to rewrite it in the language of the day.
I guess reading the article isn't necessary to make comments anymore!
yesterday, i recommended "the c programming language" here on HN. and i wanted to see if it's still a number one best seller on amazon. i was quite surprised.
"the rust programming language" has taken over "the c programming language" in the "c programming" best sellers mind you!
i am getting more and more curious about this language. i have avoided it for a long time because of the inconsistent syntax i have seen in the wild.
"the rust programming language" has taken over "the c programming language" in the "c programming" best sellers mind you!
i am getting more and more curious about this language. i have avoided it for a long time because of the inconsistent syntax i have seen in the wild.
The syntax feels more unusual than some others but it isnt so bad once you use it.
It still baffles me when i try to understand parts of it in isolation. As with all things it normalizes with exposure.
It still baffles me when i try to understand parts of it in isolation. As with all things it normalizes with exposure.
> i have avoided it for a long time because of the inconsistent syntax i have seen in the wild.
What do you mean?
What do you mean?
I'm intrigued too. One obvious cause of "inconsistent syntax" would be if you saw any pre-1.0 Rust. Early Rust is a very different animal, it has a different concurrency model, it has a garbage collector, and the syntax is not at all like it is today. Such radical changes would of course be completely unacceptable for an industrial language, and sure enough 1.0 is the end of it.
There have been some changes since too.
I am hazy, not done as much Rust as I would like
I am interested in what the issues are people have had
I am hazy, not done as much Rust as I would like
I am interested in what the issues are people have had
I'm wondering about this too.
It seems as though the C Programming category on Amazon really just means System Programming. I've seen several different Rust books show up high on that list in the past several months (normally by looking at the book and seeing it listed as rank #x on that list, I don't tend to look at the C language list).
I'd love to build my stuff faster (of course), I tried SWC and it seems impressive. But as long as it doesn't include a TS compiler that check types, I'm not exactly sure where/how I would use it. Both locally and in my CI I want my types checked, otherwise I wouldn't bother writing them.
[deleted]
Nitpick: There are a ton of programming languages in use at Facebook, Amazon, Apple, Microsoft and Google. Maybe a hundred amongst them! The fact that Rust is even being considered by as many large slow-moving companies is, indeed, impressive. But the way it’s being phrased makes it sound like it is in production everywhere, and I think the truth is the most substantial amount of Rust code in production is probably from Mozilla and maybe Amazon. In many cases, the amount of Rust code is likely very small, isolated for now, and not necessarily in production code yet. Go has also gained an impressive portfolio, too, so Rust doesn’t stand alone. Personally, I suspect Zig will be following on some day.
My personal opinion is that all we can conclude today is that JavaScript is not the future of JavaScript infrastructure. However, I am somewhat unconvinced that Rust’s strengths are conclusively important enough to offset the added complexity in all use cases. In production where you might have cases where every cycle and byte counts and where a race condition or memory bug could come at a dire cost, Rust is an obvious winner. On a developer’s computer that is running mostly idle and running dev tools that are mostly embarrassingly parallel with little shared state, it’s just not so clear. Which is why I feel putting the Discord pull quote about their Rust migration underneath the testimony of evan regarding esbuild is a little bit misleading.
I am still very bullish on Rust, but when honeymoon periods end, the truth comes out, and in Rust my opinion is that the language is pretty complex, the build times need a lot of work, and the ecosystem has some pretty difficult to ignore weaknesses (I reasonably frequently run into packages that are incomplete, and the dependency trees in Rust projects seem to get as unwieldy as smaller JS/npm projects!)
Don’t get me wrong. Go and Zig, as two promising other languages, are obviously not competing with Rust directly. What Rust offers is fairly unique even for a memory safe language. Also, Go’s lack of sum types and pattern matching hits me harder than its verbose error handling or lack of generics. But many people, seeing the borrow checker in action, seem to conclude that it is the future, and that everything else is obsoleted by its existence. Disagree. I think all of these languages are formidable opponents and worthwhile improvements over older languages that people are migrating from. But the thing about zero cost abstractions is that they are a misnomer. They have zero cost at runtime. They have non-zero cost at compile time, they have non-zero costs on cognitive overhead, they have non-zero costs on language design… and once you acknowledge this, it’s only fair to recognize that there are cases where it’s not the trade-off you want.
Still, I hope to see more adoption of what I’ll call “next generation” programming languages. Be it Rust, Zig, Go, Pony, Nim… there really is a lot promising on the horizon, stuff that will be well worth the “RiiR” costs in the long-term. I have a feeling many of them will find a place and wind up complementing each other as options you can choose for differing tastes and goals. I am very interested to see which programming language ultimately wins for JS toolchains because it might tell us a lot about just how versatile Rust’s tradeoffs are or aren’t. For now, I am happy enough with esbuild, and looking out into the future to see what’s next.
My personal opinion is that all we can conclude today is that JavaScript is not the future of JavaScript infrastructure. However, I am somewhat unconvinced that Rust’s strengths are conclusively important enough to offset the added complexity in all use cases. In production where you might have cases where every cycle and byte counts and where a race condition or memory bug could come at a dire cost, Rust is an obvious winner. On a developer’s computer that is running mostly idle and running dev tools that are mostly embarrassingly parallel with little shared state, it’s just not so clear. Which is why I feel putting the Discord pull quote about their Rust migration underneath the testimony of evan regarding esbuild is a little bit misleading.
I am still very bullish on Rust, but when honeymoon periods end, the truth comes out, and in Rust my opinion is that the language is pretty complex, the build times need a lot of work, and the ecosystem has some pretty difficult to ignore weaknesses (I reasonably frequently run into packages that are incomplete, and the dependency trees in Rust projects seem to get as unwieldy as smaller JS/npm projects!)
Don’t get me wrong. Go and Zig, as two promising other languages, are obviously not competing with Rust directly. What Rust offers is fairly unique even for a memory safe language. Also, Go’s lack of sum types and pattern matching hits me harder than its verbose error handling or lack of generics. But many people, seeing the borrow checker in action, seem to conclude that it is the future, and that everything else is obsoleted by its existence. Disagree. I think all of these languages are formidable opponents and worthwhile improvements over older languages that people are migrating from. But the thing about zero cost abstractions is that they are a misnomer. They have zero cost at runtime. They have non-zero cost at compile time, they have non-zero costs on cognitive overhead, they have non-zero costs on language design… and once you acknowledge this, it’s only fair to recognize that there are cases where it’s not the trade-off you want.
Still, I hope to see more adoption of what I’ll call “next generation” programming languages. Be it Rust, Zig, Go, Pony, Nim… there really is a lot promising on the horizon, stuff that will be well worth the “RiiR” costs in the long-term. I have a feeling many of them will find a place and wind up complementing each other as options you can choose for differing tastes and goals. I am very interested to see which programming language ultimately wins for JS toolchains because it might tell us a lot about just how versatile Rust’s tradeoffs are or aren’t. For now, I am happy enough with esbuild, and looking out into the future to see what’s next.
Picking a random example from the article: "Deno’s linter, code formatter, and docs generator". Why you need to write a tool like this in low-level system language like Rust? And please, do not answer "performance" because: a) it is not critical for tool like this b) many other high-level modern languages for this type of project will give performance only marginally inferior to low-lewel implementation in Rust or C.
I'll go ahead and answer "performance". People are switching to these faster tools because speeding up the edit-compile-run cycle speeds up their work.
Yes, it's theoretically possible to write fast tools in JS, but somehow after all these years and millions/billions of dollars invested in the V8 JIT, they are all still slow enough that people will switch to newer tools, with fewer features, purely for the performance win. And these new tools end up faster with little developer time spent on profiling/optimization, because performant code is in the pit of success.
I also wouldn't think of Rust as a low-level language. It has plenty of high-level features like macros, typeclasses, iterators that are nicer than in JS, etc. Speaking as a developer, these make Rust much nicer/faster to use than other languages that lack these things.
Yes, it's theoretically possible to write fast tools in JS, but somehow after all these years and millions/billions of dollars invested in the V8 JIT, they are all still slow enough that people will switch to newer tools, with fewer features, purely for the performance win. And these new tools end up faster with little developer time spent on profiling/optimization, because performant code is in the pit of success.
I also wouldn't think of Rust as a low-level language. It has plenty of high-level features like macros, typeclasses, iterators that are nicer than in JS, etc. Speaking as a developer, these make Rust much nicer/faster to use than other languages that lack these things.
It is more about fashion and being afraid to have the right words on the CV than anything else really.
Any AOT compiled language would produce the same performace results.
Any AOT compiled language would produce the same performace results.
It's nice how you mentioned billions of dollars in the same sentence as v8.
As if your assembly-run CPU would be magically faster running three damn gigabytes of code in another language.
JS isn't the problem.
Babel is the problem. And every library using it, instead of using ES2020+ directly because every Browser supports it anyways.
NPM needs an ESM modules based header files system, and packages should be not binary, not uglified, but maintained by a standard of expectations what a library can and cannot do. Split up header files, namespaces for definitions, and expected types for all functions.
"Still using phantom? Sorry we block your package because it's outdated."
"Compiling to es5 commonjs with ie6 setting? Sorry, your package is outdated."
The ecosystem in JS has become a joke not because of the language, but because of the tooling. And now writing tooling in another language to replace that tooling will just end up in the exact same state all over again.
For the love of Rust, please don't fuck up rust crates the way the Babel fatigued ecosystem did. Don't go the micropackage way, it didn't work and it's time to reflect on that, and build up a standard library in a more centralized way.
As if your assembly-run CPU would be magically faster running three damn gigabytes of code in another language.
JS isn't the problem.
Babel is the problem. And every library using it, instead of using ES2020+ directly because every Browser supports it anyways.
NPM needs an ESM modules based header files system, and packages should be not binary, not uglified, but maintained by a standard of expectations what a library can and cannot do. Split up header files, namespaces for definitions, and expected types for all functions.
"Still using phantom? Sorry we block your package because it's outdated."
"Compiling to es5 commonjs with ie6 setting? Sorry, your package is outdated."
The ecosystem in JS has become a joke not because of the language, but because of the tooling. And now writing tooling in another language to replace that tooling will just end up in the exact same state all over again.
For the love of Rust, please don't fuck up rust crates the way the Babel fatigued ecosystem did. Don't go the micropackage way, it didn't work and it's time to reflect on that, and build up a standard library in a more centralized way.
what should it be written in
Any AOT compiled language would do, nothing special about Rust magic dust.
JavaScript linters and more specifically the TypeScript type checker are one of things that slow my computer down the most at work. Probably doesn’t help that I have 7 projects open at once, but that’s the nature of working a micro services based code based. Performance absolutely matters for this kind of code.
Many other high levels languages might give you the performance you need, but JavaScript won’t, and not many languages are as nice to use as Rust. So if you have to do it in another language then it’s a good pick.
Many other high levels languages might give you the performance you need, but JavaScript won’t, and not many languages are as nice to use as Rust. So if you have to do it in another language then it’s a good pick.
I use Rust for a lot of things where performance is completely irrelevant. It has a lot of nice features and benefits other than memory safety and performance.
as someone who has decent experience with frontend, I believe the future we will be treating JS as a assembly language. However, for that to happen we need a lua like language for js with inferred types. and for apps the will be rendered by wasm in a canvas like interface.
Bullish on Rust
I'm not sure what the point of my post is, other than to vent. I guess sometimes I just wish we would get rid of JavaScript entirely and switch to a sane language with sane build tools.