ESbuild – A fast JavaScript bundler and minifier in Go(github.com)
github.com
ESbuild – A fast JavaScript bundler and minifier in Go
https://github.com/evanw/esbuild/
138 comments
This is a cool project and I like your point about resetting expectations wrt performance. I work in an all-Python shop and it’s maddening that people are relatively okay with builds (mostly installing dependencies) in the tens of minutes, CLIs that take ten seconds to print —help, and requests that take tens of seconds at the median and timeout at p95, not to mention at least 5x more spend on our cloud bill.
With something like Go, it would take us a little up front effort to learn the language and we would easily get a 1-2 order of magnitude improvement in most of these metrics (and any productivity difference is well within the margin for error). This is our shop, but it’s also a microcosm of the broader Python community (e.g., takes half an hour to resolve dependencies for medium sized projects). It would be really cool if the broader community would rally around PyPy and improve its support or if CPython would move be willing to concede some backwards compatibility on its extensions interface in order to allow for more optimizations.
With something like Go, it would take us a little up front effort to learn the language and we would easily get a 1-2 order of magnitude improvement in most of these metrics (and any productivity difference is well within the margin for error). This is our shop, but it’s also a microcosm of the broader Python community (e.g., takes half an hour to resolve dependencies for medium sized projects). It would be really cool if the broader community would rally around PyPy and improve its support or if CPython would move be willing to concede some backwards compatibility on its extensions interface in order to allow for more optimizations.
constexpr - Cool project!
For those of us with slow internet connections please change your Makefile to use a shallow clone:
``` github/three: mkdir -p github git clone --depth 1 https://github.com/mrdoob/three.js.git github/three -b r108 cd github/three && git checkout r108 ```
This would save a couple of gigs and hours in download.
Just skimming your code I see variable mangling but no AST optimizations, is that correct? Which is odd because you show similar output sizes to the JS bundlers.
For those of us with slow internet connections please change your Makefile to use a shallow clone:
``` github/three: mkdir -p github git clone --depth 1 https://github.com/mrdoob/three.js.git github/three -b r108 cd github/three && git checkout r108 ```
This would save a couple of gigs and hours in download.
Just skimming your code I see variable mangling but no AST optimizations, is that correct? Which is odd because you show similar output sizes to the JS bundlers.
Thanks! Will do. No wonder the three.js repo takes so long to download :)
You probably aren't seeing the AST optimizations because they are actually in the parser. Search for "mangleSyntax" to find the optimization code.
This would normally be done in a separate pass but I'm trying to minimize the number of passes done over the full AST for speed. I figured it'd be faster to optimize the AST as it's parsed because the newly-created AST nodes are likely to still be in the CPU cache. That said, I haven't tested the performance impact of doing this. I just did this based on intuitions from performance work on previous compiler projects.
You probably aren't seeing the AST optimizations because they are actually in the parser. Search for "mangleSyntax" to find the optimization code.
This would normally be done in a separate pass but I'm trying to minimize the number of passes done over the full AST for speed. I figured it'd be faster to optimize the AST as it's parsed because the newly-created AST nodes are likely to still be in the CPU cache. That said, I haven't tested the performance impact of doing this. I just did this based on intuitions from performance work on previous compiler projects.
Thanks - I never thought to look in the parser for AST optimization.
By the way, it doesn't affect timings by more than 20%, but that repo has locked in rather old versions of rollup, terser, and other packages. You can disable tree shaking in rollup with --no-treeshake and still get comparable bundle sizes after using terser. That would save 3 seconds. There's no tree shaking opportunity in re-exporting all the symbols from the same library 10 times - all code must be retained. Using terser({compress: false}) would save an additional 17 seconds at the expense of making the final bundle 3% larger - 6.01mb total. Mind you even with upgraded packages and these config tweaks rollup+terser would still be 30x slower than esbuild - compiled languages will always win in speed.
Bundling and minifying a real app rather than a library might yield more interesting bundle size numbers due to dead code elimination. Providing gzip sizes would be interesting too - they aren't always proportional to the original ungzipped sizes.
By the way, it doesn't affect timings by more than 20%, but that repo has locked in rather old versions of rollup, terser, and other packages. You can disable tree shaking in rollup with --no-treeshake and still get comparable bundle sizes after using terser. That would save 3 seconds. There's no tree shaking opportunity in re-exporting all the symbols from the same library 10 times - all code must be retained. Using terser({compress: false}) would save an additional 17 seconds at the expense of making the final bundle 3% larger - 6.01mb total. Mind you even with upgraded packages and these config tweaks rollup+terser would still be 30x slower than esbuild - compiled languages will always win in speed.
Bundling and minifying a real app rather than a library might yield more interesting bundle size numbers due to dead code elimination. Providing gzip sizes would be interesting too - they aren't always proportional to the original ungzipped sizes.
Yeah benchmarks are tricky. I'd rather not tweak the parameters for Rollup in this case because then it wouldn't be a fair comparison. Rollup would then be cheating because it has to do less work.
It could definitely be useful to have additional benchmarks that test other configurations though. For example, development build speed is also relevant and would make a good benchmark. I'll think about adding more benchmarks if I have time.
I also want a benchmark that tests a real app. I just haven't yet found a large-enough open source app to use as a benchmark. Do you know of one? I work at Figma and Figma's main app makes a great test case, but obviously using that as benchmark wouldn't be reproducible by others outside of Figma. Figma's code base is around the same size as the benchmark I'm using (so 10x the size of three.js).
Ultimately I'm planning on implementing a TypeScript parser in esbuild, at which point I'd like to switch to a benchmark of a TypeScript code base. I also want to include multiple entry points in the benchmark because that's something Figma's main app uses, and esbuild has specific optimizations for that case.
Long story short the benchmark I'm currently using was helpful for me during the initial development but it's not the final one I plan to use, and I will likely completely change it as the project evolves.
Edit: Oh yeah and gzip sizes are great to track. I've tracked these for previous compiler projects I've done and you're right, sometimes you need to make the output larger to make the gzip output smaller. I haven't looked at the gzip sizes at all yet and there is room for improvement. Check out https://github.com/evanw/esbuild/issues/6#issuecomment-58667... for an idea of what specific numbers might currently look like for esbuild compared to other minifiers.
It could definitely be useful to have additional benchmarks that test other configurations though. For example, development build speed is also relevant and would make a good benchmark. I'll think about adding more benchmarks if I have time.
I also want a benchmark that tests a real app. I just haven't yet found a large-enough open source app to use as a benchmark. Do you know of one? I work at Figma and Figma's main app makes a great test case, but obviously using that as benchmark wouldn't be reproducible by others outside of Figma. Figma's code base is around the same size as the benchmark I'm using (so 10x the size of three.js).
Ultimately I'm planning on implementing a TypeScript parser in esbuild, at which point I'd like to switch to a benchmark of a TypeScript code base. I also want to include multiple entry points in the benchmark because that's something Figma's main app uses, and esbuild has specific optimizations for that case.
Long story short the benchmark I'm currently using was helpful for me during the initial development but it's not the final one I plan to use, and I will likely completely change it as the project evolves.
Edit: Oh yeah and gzip sizes are great to track. I've tracked these for previous compiler projects I've done and you're right, sometimes you need to make the output larger to make the gzip output smaller. I haven't looked at the gzip sizes at all yet and there is room for improvement. Check out https://github.com/evanw/esbuild/issues/6#issuecomment-58667... for an idea of what specific numbers might currently look like for esbuild compared to other minifiers.
Fair enough you'd want a like-for-like comparison. Even without changing the default configuration, upgrading the JS packages would improve their speed somewhat.
Unfortunately I am not aware of a large public ES codebase to test against. Most projects use Typescript as you mentioned. You could grab the ES bundle made by several projects and test against that, but that also wouldn't be a fair comparison.
The results in https://user-images.githubusercontent.com/406394/74600139-d1... are very informative. It matches my expectations.
I've been studying the Go code in esbuild - very clean and extensible. I think this project has a lot of potential. Thanks for sharing it with the world.
Unfortunately I am not aware of a large public ES codebase to test against. Most projects use Typescript as you mentioned. You could grab the ES bundle made by several projects and test against that, but that also wouldn't be a fair comparison.
The results in https://user-images.githubusercontent.com/406394/74600139-d1... are very informative. It matches my expectations.
I've been studying the Go code in esbuild - very clean and extensible. I think this project has a lot of potential. Thanks for sharing it with the world.
--depth 1 should be known by more people. I have seen many people cloning entire repos or downloading entire .zip file from GitHub.
Since it already has JSX support, adding CSS will cater for 80% of use cases.
Very cool project, do you know how this compares with Hugo's javascript minifier / bundler?
Hugo uses https://github.com/tdewolff/minify internally, so that’s the relevant comparison.
This supports ES6 modules, which I believe are totally unsupported by Hugo Pipes. I remember being annoyed at having to throw node and webpack into the pipeline just because of that (which slowed everything to a crawl).
Hugo doesn't have a JavaScript bundler, it just lets you concatenate JavaScript files which is different from a JS bundler which needs to support JS modules.
I’d like to know this as well.
[deleted]
Thank you for making it!
I believe something like this project is inevitable. You're probably thinking "Where's the Rust version of this?". I'll save you a roundtrip to Google that'll find you SWC. [1]
SWC is already more mature, at least more so than this project. Using rust does also seem to have some advantages compared to Go.
Still, nothing wrong with some competition, just think it'll be pretty hard to replace the entire ecosystem around Typescript/Babel/Webpack in one go. Not going to work for existing project - probably way too high of a risk for new ones, sadly.
[1] https://github.com/swc-project/swc
SWC is already more mature, at least more so than this project. Using rust does also seem to have some advantages compared to Go.
Still, nothing wrong with some competition, just think it'll be pretty hard to replace the entire ecosystem around Typescript/Babel/Webpack in one go. Not going to work for existing project - probably way too high of a risk for new ones, sadly.
[1] https://github.com/swc-project/swc
Author here. I think the Rust vs. Go question is interesting. I actually originally wrote esbuild in Rust and Go, and Go was the clear winner.
The parser written in Go was both faster to compile and faster to execute than the parser in Rust. The Go version compiled something like 100x faster than Rust and ran at something around 10% faster (I forget the exact numbers, sorry). Based on a profile, it looked like the Go version was faster because GC happened on another thread while Rust had to run destructors on the same thread.
The Rust version also had other problems. Many places in my code had switch statements that branched over all AST nodes and in Rust that compiles to code which uses stack space proportional to the total stack space used by all branches instead of just the maximum stack space used by any one branch: https://github.com/rust-lang/rust/issues/34283. I believe the issue still isn't fixed. That meant that the Rust version quickly overflowed the stack if you had many nested JavaScript syntax constructs, which was easy to hit in large JavaScript files. There were also random other issues such as Rust's floating-point number parser not actually working in all cases: https://github.com/rust-lang/rust/issues/31407. I also had to spend a lot of time getting multi-threading to work in Rust with all of the lifetime stuff. Go had none of these issues.
The Rust version probably could be made to work at an equivalent speed with enough effort. But at a high-level, Go was much more enjoyable to work with. This is a side project and it has to be fun for me to work on it. The Rust version was actively un-fun for me, both because of all of the workarounds that got in the way and because of the extremely slow compile times. Obviously you can tell from the nature of this project that I value fast build times :)
The parser written in Go was both faster to compile and faster to execute than the parser in Rust. The Go version compiled something like 100x faster than Rust and ran at something around 10% faster (I forget the exact numbers, sorry). Based on a profile, it looked like the Go version was faster because GC happened on another thread while Rust had to run destructors on the same thread.
The Rust version also had other problems. Many places in my code had switch statements that branched over all AST nodes and in Rust that compiles to code which uses stack space proportional to the total stack space used by all branches instead of just the maximum stack space used by any one branch: https://github.com/rust-lang/rust/issues/34283. I believe the issue still isn't fixed. That meant that the Rust version quickly overflowed the stack if you had many nested JavaScript syntax constructs, which was easy to hit in large JavaScript files. There were also random other issues such as Rust's floating-point number parser not actually working in all cases: https://github.com/rust-lang/rust/issues/31407. I also had to spend a lot of time getting multi-threading to work in Rust with all of the lifetime stuff. Go had none of these issues.
The Rust version probably could be made to work at an equivalent speed with enough effort. But at a high-level, Go was much more enjoyable to work with. This is a side project and it has to be fun for me to work on it. The Rust version was actively un-fun for me, both because of all of the workarounds that got in the way and because of the extremely slow compile times. Obviously you can tell from the nature of this project that I value fast build times :)
> Many places in my code had switch statements that branched over all AST nodes and in Rust that compiles to code which uses stack space proportional to the total stack space used by all branches instead of just the maximum stack space used by any one branch: https://github.com/rust-lang/rust/issues/34283.
Can you work around this by using separate functions for the branches? This will have other benefits for compile time as well. Generally, small functions are better for compile time, because some compiler passes are not O(n), including very basic ones like register allocation.
For large switch statements I do this for readability reasons, because I try to keep my functions small.
> Based on a profile, it looked like the Go version was faster because GC happened on another thread while Rust had to run destructors on the same thread.
Have you tried using jemalloc? It can help a lot for situations like this.
Can you work around this by using separate functions for the branches? This will have other benefits for compile time as well. Generally, small functions are better for compile time, because some compiler passes are not O(n), including very basic ones like register allocation.
For large switch statements I do this for readability reasons, because I try to keep my functions small.
> Based on a profile, it looked like the Go version was faster because GC happened on another thread while Rust had to run destructors on the same thread.
Have you tried using jemalloc? It can help a lot for situations like this.
> Have you tried using jemalloc? It can help a lot for situations like this.
Huh, I thought Rust already used jemalloc by default. I looked it up and it looks like it was removed relatively recently. When I was doing this experiment, I was using a version of Rust that included jemalloc by default so that 10% number already uses jemalloc. I remember this because I also thought of trying to speed up the allocator.
Like I said above, I profiled both Go and Rust and the 10% slowdown with Rust appeared to be running destructors for the AST. I think the appropriate solution to this would be some form of arena allocator instead of changing the system allocator. But that gets even more complicated with lifetimes and stuff.
> Can you work around this by using separate functions for the branches?
Yeah, I could have tried restructuring my code to try to avoid compiler issues. But this would have been even more time spent working around issues with Rust. Go was better than Rust by pretty much every metric that mattered for me, so I went with Go instead.
It's too bad because I was initially super excited about the promise of Rust. Being able to avoid the overhead of GC while keeping memory safety and performance is really appealing. But Rust turned out to be not a productive enough language for me.
Huh, I thought Rust already used jemalloc by default. I looked it up and it looks like it was removed relatively recently. When I was doing this experiment, I was using a version of Rust that included jemalloc by default so that 10% number already uses jemalloc. I remember this because I also thought of trying to speed up the allocator.
Like I said above, I profiled both Go and Rust and the 10% slowdown with Rust appeared to be running destructors for the AST. I think the appropriate solution to this would be some form of arena allocator instead of changing the system allocator. But that gets even more complicated with lifetimes and stuff.
> Can you work around this by using separate functions for the branches?
Yeah, I could have tried restructuring my code to try to avoid compiler issues. But this would have been even more time spent working around issues with Rust. Go was better than Rust by pretty much every metric that mattered for me, so I went with Go instead.
It's too bad because I was initially super excited about the promise of Rust. Being able to avoid the overhead of GC while keeping memory safety and performance is really appealing. But Rust turned out to be not a productive enough language for me.
> I think the appropriate solution to this would be some form of arena allocator instead of changing the system allocator. But that gets even more complicated with lifetimes and stuff.
If you're doing arena allocation in a compiler you might as well just leak all your allocations (which you could get with bumpalo with a 'static lifetime); then you won't have to deal with lifetimes at all.
> Yeah, I could have tried restructuring my code to try to avoid compiler issues.
Well, my point is that it would be good for readability to restructure your code in that way even in Go. 500-line functions are hard to read.
> But this would have been even more time spent working around issues with Rust. Go was better than Rust by pretty much every metric that mattered for me, so I went with Go instead.
I find the opposite to be true, especially for compilers. It's hard for me to go back to a language without pattern matching and enums (much less generics, iterators, a package ecosystem, etc.). The gain of productivity from GC and compile times is not worth Go's loss in productivity in other areas for me. But reasonable people can disagree here.
If you're doing arena allocation in a compiler you might as well just leak all your allocations (which you could get with bumpalo with a 'static lifetime); then you won't have to deal with lifetimes at all.
> Yeah, I could have tried restructuring my code to try to avoid compiler issues.
Well, my point is that it would be good for readability to restructure your code in that way even in Go. 500-line functions are hard to read.
> But this would have been even more time spent working around issues with Rust. Go was better than Rust by pretty much every metric that mattered for me, so I went with Go instead.
I find the opposite to be true, especially for compilers. It's hard for me to go back to a language without pattern matching and enums (much less generics, iterators, a package ecosystem, etc.). The gain of productivity from GC and compile times is not worth Go's loss in productivity in other areas for me. But reasonable people can disagree here.
> If you're doing arena allocation in a compiler you might as well just leak all your allocations (which you could get with bumpalo with a 'static lifetime); then you won't have to deal with lifetimes at all.
I considered this but it's a very limiting hack. Ideally esbuild could be run in watch mode to do incremental builds where only the changed files are rebuilt and most of the previous compilation is reused. Memory leaks aren't an acceptable workaround to memory allocation issues in that case. While I don't have a watch mode yet, all of esbuild was architected with incremental builds in mind and the fact that Go has a GC makes this very easy.
> 500-line functions are hard to read.
I totally recognize that this is completely subjective, but I've written a lot of compiler code and I actually find that co-locating related branches together is easier for me to work with than separating the contents of a branch far away from the branch itself, at least in the AST pattern-matching context.
> It's hard for me to go back to a language without pattern matching and enums
I also really like these features of Rust, and miss them when I'm using other languages without them. However, if you look at the way I've used Go in esbuild, interfaces and switch statements give you a way to implement enums and pattern matching that has been surprisingly ergonomic for me.
The biggest thing I miss in Go from Rust is actually the lack of immutability in the type system. To support incremental builds, each build must not mutate the data structures that live across builds. There's currently no way to have the Go compiler enforce this. I just have to be careful. In my case I think it's not enough of a problem to offset the other benefits of Go, but it's definitely not a trade-off everyone would be comfortable making.
I considered this but it's a very limiting hack. Ideally esbuild could be run in watch mode to do incremental builds where only the changed files are rebuilt and most of the previous compilation is reused. Memory leaks aren't an acceptable workaround to memory allocation issues in that case. While I don't have a watch mode yet, all of esbuild was architected with incremental builds in mind and the fact that Go has a GC makes this very easy.
> 500-line functions are hard to read.
I totally recognize that this is completely subjective, but I've written a lot of compiler code and I actually find that co-locating related branches together is easier for me to work with than separating the contents of a branch far away from the branch itself, at least in the AST pattern-matching context.
> It's hard for me to go back to a language without pattern matching and enums
I also really like these features of Rust, and miss them when I'm using other languages without them. However, if you look at the way I've used Go in esbuild, interfaces and switch statements give you a way to implement enums and pattern matching that has been surprisingly ergonomic for me.
The biggest thing I miss in Go from Rust is actually the lack of immutability in the type system. To support incremental builds, each build must not mutate the data structures that live across builds. There's currently no way to have the Go compiler enforce this. I just have to be careful. In my case I think it's not enough of a problem to offset the other benefits of Go, but it's definitely not a trade-off everyone would be comfortable making.
> interfaces and switch statements give you a way to implement enums and pattern matching that has been surprisingly ergonomic for me.
With no exhaustiveness checking (also no destructuring, etc.)
I should also note that you can change the default stack size in Rust to avoid overflows, though there should be a bug filed to get LLVM upstream stack coloring working. It's also possibly worth rerunning the benchmark again, as Rust has upgraded to newer versions of LLVM in the meantime.
With no exhaustiveness checking (also no destructuring, etc.)
I should also note that you can change the default stack size in Rust to avoid overflows, though there should be a bug filed to get LLVM upstream stack coloring working. It's also possibly worth rerunning the benchmark again, as Rust has upgraded to newer versions of LLVM in the meantime.
This really reeks to me of trying to shoehorn Rust into the solution rather than it being an organic fit to the problem space.
I don’t see the value in this level of thinking here. Why should any developer go through this much hassle when they have a perfectly good solution that, really, I’m not seeing any discussion about this that actually highlights issues in the approach to using ago for this sort of thing
I don’t see the value in this level of thinking here. Why should any developer go through this much hassle when they have a perfectly good solution that, really, I’m not seeing any discussion about this that actually highlights issues in the approach to using ago for this sort of thing
Generally, Rust "should" be faster, because it spends a lot of time on optimizations that Go doesn't do. That's what you're paying for in compile times. If Go is faster on some CPU bound workload despite doing a lot less optimization, that's interesting. (I should note that this is not the norm.)
Or, written another way: In theory, practise is like theory. In practise, it isn't :)
I'm not sure that this is a CPU bound workload. You're right that if this is CPU bound, LLVM's code generation should come out on top, even if only slightly, but that's not the case. Perhaps writing a JavaScript bundler is more of a memory bound or I/O bound task than a CPU bound task?
What optimizations is Go not doing that Rust is that makes Rust vastly superior to Go? (Or even superior at all)
I don’t think it’s that simple. I understand Go uses garbage collection but that doesn’t automatically mean Go doesn’t do compile time optimizations or is poor at CPU bound work.
While I understand GCs add overhead I don’t think that in and of itself means much here
This seems like yak shaving to me
I don’t think it’s that simple. I understand Go uses garbage collection but that doesn’t automatically mean Go doesn’t do compile time optimizations or is poor at CPU bound work.
While I understand GCs add overhead I don’t think that in and of itself means much here
This seems like yak shaving to me
[deleted]
I remember an article from Figma or even possible from you, where you mentioned that you have rewritten in Rust a tool originally written in node.js. And I remember, back at that time (I think it was 2 years ago) you were very exited about the language. How does it compare to the current situation? What made you to think that Go is a better and enjoyable language than Rust? Is it the faster GC?
A recently published article from a guy working at Discord throw a real flameware, because he was arguing about the opposite: Rust is a better and faster language then Go, but he based his assumption on very old version of Go (1.9) where the GC was way slower then in the current versions.
A recently published article from a guy working at Discord throw a real flameware, because he was arguing about the opposite: Rust is a better and faster language then Go, but he based his assumption on very old version of Go (1.9) where the GC was way slower then in the current versions.
Why work around it when the Go version worked as is?
I’m not sure I really see the benefit of working around it when Go fit the use case as is
I’m not sure I really see the benefit of working around it when Go fit the use case as is
Well, I'd like to know if there are things we can do to improve things in Rust.
Yes, fix this open issue
https://github.com/rust-lang/rust/issues/34283
That would help immensely.
https://github.com/rust-lang/rust/issues/34283
That would help immensely.
Do we know that Go uses stack coloring, actually? It's a relatively uncommon optimization.
The Rust version probably could be made to work at an equivalent speed with enough effort. But at a high-level, Go was much more enjoyable to work with. This is a side project and it has to be fun for me to work on it. The Rust version was actively un-fun for me, both because of all of the workarounds that got in the way and because of the extremely slow compile times. Obviously you can tell from the nature of this project that I value fast build times :)
Was the Rust parser written by hand or did you use one of the parser frameworks (e.g. nom or pest) out there? nom, for instance, goes to great lengths to be zero-copy which would probably be a big benefit here.
Was the Rust parser written by hand or did you use one of the parser frameworks (e.g. nom or pest) out there? nom, for instance, goes to great lengths to be zero-copy which would probably be a big benefit here.
Both the Rust and Go parsers were written by hand. They are also very similar (basically the Go version was a direct port of the Rust version) so the performance should be very comparable.
I assume by zero-copy you mean that identifiers in the AST are slices of the input file instead of copies? I was also careful to do this in both the Go and Rust versions. It's somewhat complicated because some JavaScript identifiers can technically have escape sequences (e.g. "\u0061bc" is the identifier "abc"), which require dynamic memory allocation anyway. See "allocatedNames" in the current parser for how this is handled.
Note that strings aren't slices of the input file because JavaScript strings are UTF-16, not UTF-8, and can have unpaired surrogates. So I represent string contents as arrays of 16-bit integers instead of 8-bit slices (in both Go and Rust).
In the past I tried using WTF-8 encoding (https://simonsapin.github.io/wtf-8/) for string contents, since that can both represent slices of the input file while also handling unpaired surrogates, but I ended up removing it because it complicated certain optimizations. I think the main issue was having to reason through weird edge cases such as constant folding of string addition when two unpaired surrogates are joined together. I think it's still possible to do this but I'm not sure how much of a win it is.
I assume by zero-copy you mean that identifiers in the AST are slices of the input file instead of copies? I was also careful to do this in both the Go and Rust versions. It's somewhat complicated because some JavaScript identifiers can technically have escape sequences (e.g. "\u0061bc" is the identifier "abc"), which require dynamic memory allocation anyway. See "allocatedNames" in the current parser for how this is handled.
Note that strings aren't slices of the input file because JavaScript strings are UTF-16, not UTF-8, and can have unpaired surrogates. So I represent string contents as arrays of 16-bit integers instead of 8-bit slices (in both Go and Rust).
In the past I tried using WTF-8 encoding (https://simonsapin.github.io/wtf-8/) for string contents, since that can both represent slices of the input file while also handling unpaired surrogates, but I ended up removing it because it complicated certain optimizations. I think the main issue was having to reason through weird edge cases such as constant folding of string addition when two unpaired surrogates are joined together. I think it's still possible to do this but I'm not sure how much of a win it is.
They are also very similar (basically the Go version was a direct port of the Rust version) so the performance should be very comparable.
Sure, but different approaches are going to be more optimal for different languages.
I assume by zero-copy you mean that identifiers in the AST are slices of the input file instead of copies?
Yes. From the README:
zero-copy: if a parser returns a subset of its input data, it will return a slice of that input, without copying
Geal also makes claims that nom is faster than hand-written C parsers.
It's somewhat complicated because some JavaScript identifiers can technically have escape sequences (e.g. "\u0061bc" is the identifier "abc"), which require dynamic memory allocation anyway.
Nom comes with 'escaped' and 'escaped_transform' combinators. In theory it should be possible, with relative ease, to return a slice if there are no escape characters and an allocated string if expansion is required. Presumably you'd have to use a Cow<str> though.
Note that strings aren't slices of the input file because JavaScript strings are UTF-16, not UTF-8, and can have unpaired surrogates. So I represent string contents as arrays of 16-bit integers instead of 8-bit slices (in both Go and Rust).
Of course it is. My opinion (which is worth what you've paid for it) is that I'd just go for UTF-8 support. I can't remember the last time I've seen UTF-16 in the wild (thankfully).
Performance-wise the other thing that I'd keep in mind with rust is that in debug mode string handling is painfully slow.
Edit: here's the URL for nom: https://github.com/Geal/nom
Sure, but different approaches are going to be more optimal for different languages.
I assume by zero-copy you mean that identifiers in the AST are slices of the input file instead of copies?
Yes. From the README:
zero-copy: if a parser returns a subset of its input data, it will return a slice of that input, without copying
Geal also makes claims that nom is faster than hand-written C parsers.
It's somewhat complicated because some JavaScript identifiers can technically have escape sequences (e.g. "\u0061bc" is the identifier "abc"), which require dynamic memory allocation anyway.
Nom comes with 'escaped' and 'escaped_transform' combinators. In theory it should be possible, with relative ease, to return a slice if there are no escape characters and an allocated string if expansion is required. Presumably you'd have to use a Cow<str> though.
Note that strings aren't slices of the input file because JavaScript strings are UTF-16, not UTF-8, and can have unpaired surrogates. So I represent string contents as arrays of 16-bit integers instead of 8-bit slices (in both Go and Rust).
Of course it is. My opinion (which is worth what you've paid for it) is that I'd just go for UTF-8 support. I can't remember the last time I've seen UTF-16 in the wild (thankfully).
Performance-wise the other thing that I'd keep in mind with rust is that in debug mode string handling is painfully slow.
Edit: here's the URL for nom: https://github.com/Geal/nom
Thanks for this really detailed and informed comparison - it's as interesting as your project itself.
Would you say go language is more fun to code in than rust ? How does it compare with nodejs and Typescript in terms of developer productivity ?
Not them but my personal opinion:
- Go is more fun if you are trying to be productive and push stuff out. The programming experience feels fluid and there’s not much agonizing over small details; the language is simple and you don’t need to think as much about things.
- Rust is more fun if you have a focus on perfection. It offers a lot of tools for abstraction and meta programming. These tools can be challenging at times. I do think even with NLL you will find yourself fighting the compiler, trying for example to resolve how you can avoid overlapping a borrow with a mutable borrow in some complicated bit of code, but you definitely get a lot of nice guarantees in exchange. I also do find it frustrating when something as simple as passing the result up can end up being really tricky.
- Go is more fun if you are trying to be productive and push stuff out. The programming experience feels fluid and there’s not much agonizing over small details; the language is simple and you don’t need to think as much about things.
- Rust is more fun if you have a focus on perfection. It offers a lot of tools for abstraction and meta programming. These tools can be challenging at times. I do think even with NLL you will find yourself fighting the compiler, trying for example to resolve how you can avoid overlapping a borrow with a mutable borrow in some complicated bit of code, but you definitely get a lot of nice guarantees in exchange. I also do find it frustrating when something as simple as passing the result up can end up being really tricky.
There's truth in what you say, but it only describes the learning phase of Rust. Rust is quite challenging to learn and you spend a pretty long time in the uncomfortable place you describe. But one day, you end up internalizing the borrow checker's rules and you just don't think about it anymore and don't have any productivity penalty at all.
I don’t know how long it takes to fully get past stumbling through borrow checking and learning the intricacies of Result but it’s long enough to be a detriment. Obviously learning curve on its own is a downside, but also this complexity does not disappear when you understand it. It’s similar to, but less severe than, C++, in this regard.
I also think it depends heavily on the type of program you are writing and how. I’ve certainly hit cases where I still don’t know the optimal way of structuring things. In other cases people have managed to help me figure out what I need to do.
Concurrent memory safety is a huge plus, without a doubt, though there are applications where its not enough and applications where its too much. I think that puts Rust in a spot where it has use cases where it is clearly the best option but many use cases where it is overkill. As an example, Go shines particularly well for servers thanks to Goroutines and the fact that many servers have a shared-nothing architecture these days.
I also think it depends heavily on the type of program you are writing and how. I’ve certainly hit cases where I still don’t know the optimal way of structuring things. In other cases people have managed to help me figure out what I need to do.
Concurrent memory safety is a huge plus, without a doubt, though there are applications where its not enough and applications where its too much. I think that puts Rust in a spot where it has use cases where it is clearly the best option but many use cases where it is overkill. As an example, Go shines particularly well for servers thanks to Goroutines and the fact that many servers have a shared-nothing architecture these days.
> I don’t know how long it takes to fully get past stumbling through borrow checking and learning the intricacies of Result but it’s long enough to be a detriment.
That's right, but no one expect to learn quantum physics in a few weeks either. Rust is indeed way longer to learn than Python, Javascript & others, but it's also much more powerful. And with the same level of power, both C and C++ are way harder to master than Rust (and arguably, nobody really master them in practice since even the most brilliant programmers shoot themselves in the foot from time to time. Yes, even DJB [1].
> Rust in a spot where it has use cases where it is clearly the best option but many use cases where it is overkill.
Indeed. It would make no sense to switch to Rust if Python is good enough for the task. I was just arguing that once you've learned Rust, you can do pretty much anything you want with it without friction, and I personally wouldn't bother writing anything in Python nowadays, because I can write Rust as fast and get the static typing guarantees and sane error handling that comes with it.
> Go shines particularly well for servers thanks to Goroutines and the fact that many servers have a shared-nothing architecture these days.
Having done quite a bit of Go, I don't agree with you. It's way too easy to accidentally share data between goroutines, and then cause data race or deadlocks . The day they introduced the race detector, we found 6 data races in our code (a few thousand loc) and a few others in our dependencies, and in the next year we found two not caught by the race detector (because it's a dynamic thing, it can't catch all races). More than generics (which are being introduced if they don't change their mind like they did for error handling) Go really need something akin to Send and Sync in Rust. M,.or maybe like Pony's capabilities system, but Go definitely needs improvement on that front. Multithreading is a hard thing, and Go makes it too easy to use, to people without the necessary experience (because Go is so easy it attracts a lot of junior or self-taught devs) without safety net and this generally doesn't end well.
[1]: https://www.cvedetails.com/product/16058/D.j.bernstein-Djbdn...
That's right, but no one expect to learn quantum physics in a few weeks either. Rust is indeed way longer to learn than Python, Javascript & others, but it's also much more powerful. And with the same level of power, both C and C++ are way harder to master than Rust (and arguably, nobody really master them in practice since even the most brilliant programmers shoot themselves in the foot from time to time. Yes, even DJB [1].
> Rust in a spot where it has use cases where it is clearly the best option but many use cases where it is overkill.
Indeed. It would make no sense to switch to Rust if Python is good enough for the task. I was just arguing that once you've learned Rust, you can do pretty much anything you want with it without friction, and I personally wouldn't bother writing anything in Python nowadays, because I can write Rust as fast and get the static typing guarantees and sane error handling that comes with it.
> Go shines particularly well for servers thanks to Goroutines and the fact that many servers have a shared-nothing architecture these days.
Having done quite a bit of Go, I don't agree with you. It's way too easy to accidentally share data between goroutines, and then cause data race or deadlocks . The day they introduced the race detector, we found 6 data races in our code (a few thousand loc) and a few others in our dependencies, and in the next year we found two not caught by the race detector (because it's a dynamic thing, it can't catch all races). More than generics (which are being introduced if they don't change their mind like they did for error handling) Go really need something akin to Send and Sync in Rust. M,.or maybe like Pony's capabilities system, but Go definitely needs improvement on that front. Multithreading is a hard thing, and Go makes it too easy to use, to people without the necessary experience (because Go is so easy it attracts a lot of junior or self-taught devs) without safety net and this generally doesn't end well.
[1]: https://www.cvedetails.com/product/16058/D.j.bernstein-Djbdn...
- A trade off is a trade off; no need to justify it. Increased cognitive load is a con.
- Sorry your Go experience was bad. I can only say my anecdotal experience was the opposite. Mostly for shared nothing architectures, but I also worked on an MQ-style software in Go and had a relatively good time. I think things that are well-suited to CSP concurrency fare pretty OK. Rust could’ve prevented things like accidental concurrent map accesses, but it still can’t guarantee you are implementing concurrency correctly on the application level (from perspective of say, coherency or atomicity.) So for many apps I’ve written, even somewhat complicated ones, I don’t feel like Rust would always be the best option. To me Rust makes most sense when you really can’t ever afford a memory error. Web browsers seem like an obvious winning case.
- Sorry your Go experience was bad. I can only say my anecdotal experience was the opposite. Mostly for shared nothing architectures, but I also worked on an MQ-style software in Go and had a relatively good time. I think things that are well-suited to CSP concurrency fare pretty OK. Rust could’ve prevented things like accidental concurrent map accesses, but it still can’t guarantee you are implementing concurrency correctly on the application level (from perspective of say, coherency or atomicity.) So for many apps I’ve written, even somewhat complicated ones, I don’t feel like Rust would always be the best option. To me Rust makes most sense when you really can’t ever afford a memory error. Web browsers seem like an obvious winning case.
While Rust can be technically superior to Go for a lot of use cases, Go has been most fun to program in for me for last 2-3 years. Coming from mostly Python/JS and having maintained systems in Java and Ruby as well, I still feel happiest when writing Go code. I don't know why but I think it's because I've never had deal with a system in Go where I had to peel layer after layer to find how something worked no matter who wrote it. All Go projects I've come across and contributed to have been extremely simple to read, understand and contribute to. _I think_ that is what makes me enjoy maintaining systems in Go so much.
Also not them, but I've worked quite a bit with both Golang and TypeScript.
I find TypeScript's interface flexibility to be pretty clutch when working with high-level code that deals with input that's.. Complex data structures. So I'm thinking configuration files, REST APIs, user/developer inputs. Having worked with many async paradigms, I also favor async/await for developing asynchronous business logic workflows. If you imagine your call graph for a certain workflow, and everything is written to use "async", you can imagine just drawing a circle around a portion of it and then easily "stamp out" more of those to occur in parallel. The way you can collect results and handle errors with the async/await paradigm is a bit nicer than working with channels and go-routines IMHO.
I like Golang for lower-level tooling and network services of course. It also has seamless support for parallel processing in process. The OPs project is something I would certainly look to Golang for.
Now on a tangent, C# has async/await and most of the speed but doesn't quite have the flexibility of TypeScript's interfaces and of course doesn't have the compile speed of Goglang. I would honestly use C#/F# a lot more or stuff if I thought it would fly in my work environment. Would love to work on some open source projects in C# to get it more exposure :)
I find TypeScript's interface flexibility to be pretty clutch when working with high-level code that deals with input that's.. Complex data structures. So I'm thinking configuration files, REST APIs, user/developer inputs. Having worked with many async paradigms, I also favor async/await for developing asynchronous business logic workflows. If you imagine your call graph for a certain workflow, and everything is written to use "async", you can imagine just drawing a circle around a portion of it and then easily "stamp out" more of those to occur in parallel. The way you can collect results and handle errors with the async/await paradigm is a bit nicer than working with channels and go-routines IMHO.
I like Golang for lower-level tooling and network services of course. It also has seamless support for parallel processing in process. The OPs project is something I would certainly look to Golang for.
Now on a tangent, C# has async/await and most of the speed but doesn't quite have the flexibility of TypeScript's interfaces and of course doesn't have the compile speed of Goglang. I would honestly use C#/F# a lot more or stuff if I thought it would fly in my work environment. Would love to work on some open source projects in C# to get it more exposure :)
After spending the last couple days in Rust and then diving back into Java for 30mins, I know what you mean by un-fun. Rust is such a PITA. I really want to like it, but I think I might end up liking C++ more, when Java doesn't cut it...
I think ESbuild is more comparable with Pax [1] (or Pax + SWC) in the alternatives written in Rust. ESbuild mostly focuses on bundling for now, and does little in terms of source code transformation compared to SWC.
[1] https://github.com/nathan/pax
[1] https://github.com/nathan/pax
Note that what Babel does and what Webpack does are orthogonal. They're frequently used together and the challenges of converting them from JS to native are similar, but the OP isn't really in "competition" with SWC.
Sure, well aware that Babel and Webpack are not the same thing.
SWC has a Webpack plugin, whereas ESBuild seems to aim to replace both Webpack and Babel. Correct me if I'm wrong.
This is a big reason why I think SWC is more likely to succeed, and why I say they're in "competition" since they overlap partially, even if it's not 1-1.
SWC has a Webpack plugin, whereas ESBuild seems to aim to replace both Webpack and Babel. Correct me if I'm wrong.
This is a big reason why I think SWC is more likely to succeed, and why I say they're in "competition" since they overlap partially, even if it's not 1-1.
Hmm, I got a little thrown off by the words "bundler and minifier", which are both Webpack tasks. Scrolling further down I do see "JSX transformation" thrown in, which is definitely normally a Babel thing.
If this project is trying to replace everything in a single pass then yeah, it's probably doomed to failure. Not only from the ambition of the task, but from the lack of the modular structure that allows community support to thrive: anybody can write a Webpack loader or a Babel plugin without getting their hands dirty in those codebases. That doesn't seem to be the case here.
If this project is trying to replace everything in a single pass then yeah, it's probably doomed to failure. Not only from the ambition of the task, but from the lack of the modular structure that allows community support to thrive: anybody can write a Webpack loader or a Babel plugin without getting their hands dirty in those codebases. That doesn't seem to be the case here.
I think the intention of the project is somewhat different to "classic" transformation tasks, in that it intends to cover the 80% workflow - take some js files, a few libraries, and make an efficient bundle from it.
That's what I need to do often and exactly what's annoyingly time intensive with the current JavaScript tooling.
That's what I need to do often and exactly what's annoyingly time intensive with the current JavaScript tooling.
Sir, it's a hobby project and can be appreciated for its efforts
SWC doesn't have a bundler.
[deleted]
Alas, there's such a massive ecosystem at this point around Webpack and Babel and family (extra features like tree-shaking, loaders support for different things, project configurations, etc.), that I think you'd be hard-pressed to make an outright replacement take off.
What you could do (and I'm surprised this hasn't been done yet) is lower some of the hot paths of Webpack itself into a lower-level language. Possibly even moreso than Webpack, Babel seems like a natural target since it has much simpler input/output.
Although one barrier I can see with that (unless it's C/C++) is requiring people to install a whole other build system just for it to be used in an npm install. C/C++ compilers generally come with the system, but Go/Rust ones do not.
Edit: further info https://stackoverflow.com/questions/20728255/could-one-write...
What you could do (and I'm surprised this hasn't been done yet) is lower some of the hot paths of Webpack itself into a lower-level language. Possibly even moreso than Webpack, Babel seems like a natural target since it has much simpler input/output.
Although one barrier I can see with that (unless it's C/C++) is requiring people to install a whole other build system just for it to be used in an npm install. C/C++ compilers generally come with the system, but Go/Rust ones do not.
Edit: further info https://stackoverflow.com/questions/20728255/could-one-write...
https://what-problem-does-it-solve.com/webpack/interoperabil... nailed head-on what I detest about Webpack and many Javascript build solutions for that matter: the burning desire to shove the kitchen sink of operations into a single tool with its own, incompatible-with-everyone-else, plugin framework and an ungodly configuration language. Where all I really wanted was a somewhat more ergonomic Make and a bunch of well-written, decoupled tools to use with it.
This is on my back burner of ideas to run with one day, but as a back-end engineer working in other langs, it hasn't quite bubbled up to the surface yet.
This is on my back burner of ideas to run with one day, but as a back-end engineer working in other langs, it hasn't quite bubbled up to the surface yet.
Agreed. A decent build system language and separate tools would be a lot easier to understand than one megatool that does everything in a magic way with plugins and a mega-config file that's impossible to understand.
The biggest bottleneck in the JS ecosystem is the minified, and that is already kind of decoupled from the other tools. Webpack plays with Terser pretty closely, but it's not particularly hard to do the minification as a separate step, or even write a Webpack plugin to replace the existing ones.
On large builds, minification is often as much as 80%+ of the build time, sometimes even if you include the unit tests! Most projects aren't particularly opinionated about their minification solution, either, so you can just swap it in. That makes it a very good place to start.
For development where minification isn't needed, Webpack is still slow, but it doesn't matter nearly as much. Even very large projects rebuild fairly quickly in dev server mode (assuming they're setup right), and if Webpack 5's persistent cache delivers, startup shouldn't be too bad either. It would still be great if the community made changes as you suggest to speed things up, but minification is really where efforts should start, IMO.
On large builds, minification is often as much as 80%+ of the build time, sometimes even if you include the unit tests! Most projects aren't particularly opinionated about their minification solution, either, so you can just swap it in. That makes it a very good place to start.
For development where minification isn't needed, Webpack is still slow, but it doesn't matter nearly as much. Even very large projects rebuild fairly quickly in dev server mode (assuming they're setup right), and if Webpack 5's persistent cache delivers, startup shouldn't be too bad either. It would still be great if the community made changes as you suggest to speed things up, but minification is really where efforts should start, IMO.
Check out this project that is an attempt to rewrite Babel in Rust. It includes a Webpack plugin and their benchmarks show a major speedup.
https://github.com/swc-project/swc
https://github.com/swc-project/swc
Very cool. Gotta say though, that logo is impossible to read...
I think that’s a good observation. Babel is probably a perfect place to start.
Two things: (1) In Go you can trivially distribute pre built binaries for <num arch> * <num os> targets. Just building with GOOS set to windows, darwin, and linux (with GOARCH set to amd64 and arm64) would cover 99% of all users. (2) Depending on where bottlenecks are, RPC/HTTP/WS can be just as effective as dynamic linking. So maybe even the need for a C/C++ toolchain could be avoided.
Two things: (1) In Go you can trivially distribute pre built binaries for <num arch> * <num os> targets. Just building with GOOS set to windows, darwin, and linux (with GOARCH set to amd64 and arm64) would cover 99% of all users. (2) Depending on where bottlenecks are, RPC/HTTP/WS can be just as effective as dynamic linking. So maybe even the need for a C/C++ toolchain could be avoided.
You're right about cross-compilation, but
> Depending on where bottlenecks are, RPC/HTTP/WS can be just as effective as dynamic linking.
...is just wrong. Even when performance doesn't matter at all (and it matters significantly: replacing even a low constant number dynamic library calls with optimized host-local RPCs using a fast transport can be enough to transition latency from "unnoticeable" to "a human can feel the lag"), the complexity undertaken when orchestrating any sort of RPC or webservice/webhook is substantial--absent LD_LIBRARY_PATH hacks and getting libraries installed, you don't have to think about any of that stuff when (dynamically or statically) linking.
> Depending on where bottlenecks are, RPC/HTTP/WS can be just as effective as dynamic linking.
...is just wrong. Even when performance doesn't matter at all (and it matters significantly: replacing even a low constant number dynamic library calls with optimized host-local RPCs using a fast transport can be enough to transition latency from "unnoticeable" to "a human can feel the lag"), the complexity undertaken when orchestrating any sort of RPC or webservice/webhook is substantial--absent LD_LIBRARY_PATH hacks and getting libraries installed, you don't have to think about any of that stuff when (dynamically or statically) linking.
I didn't know that about Go, that's pretty cool! I assumed it was like Rust, where the compiler/toolchain makes builds on arbitrary specific platforms really easy to do, but where you still have to actually do those very narrow builds.
I think you may have misunderstood: each “go build” incantation lets you produce a single OS/arch binary, not a “fat binary”. So you make a build per OS/arch pair and distribute it.
> I assumed it was like Rust, where the compiler/toolchain makes builds on arbitrary specific platforms really easy to do, but where you still have to actually do those very narrow builds.
What's the difference?
What's the difference?
Most teams I know using Webpack would love to spend the 1-2 "upgrade bullshit" days annually they already have to spend on Webpack porting to something else instead, if it actually had some longevity and advantage (and Parcel does not, if you already have a working Webpack config - still slow, still accretes config bullshit at the same rate, it just has a more mature 0 epoch).
It's not just the up-front config change, it's the constellation of loaders and such around it. I assume this new project doesn't have a TypeScript loader, or an Elm loader, or a Polymer-specific loader.
https://webpack.js.org/loaders/
https://webpack.js.org/loaders/
The problem with Webpack is that it is not a JavaScript bundler. It is a JavaScript bundler creation toolkit. That’s great if you want to write a bundler. But it’s a huge waste of time if you just want to bundle your damn JavaScript.
I suspect the supposed sluggishness attributed to Webpack actually comes from Babel instead because it's been fast ever since I removed Babel from my pipelines (because Typescript already covers what I was using Babel for).
Actually, webpack does a lot of additional stuff that might be needed for some complex configurations, so it can't be easily replaced with this tool. But also almost 75% of the time (in webpack) is spent inside the minimizing plugin, and that is very simple task. So it would be great to have just the minimize plugin based on this tool. Then maybe investigate what other performance critical parts of the webpack can be replaced without requiring to change the ecosystem.
Finally. The extreme slugishness of webpack&co has always bugged me any time I ventured into JS development, to the point of me getting immediately discouraged.
It’s a natural cycle though. New bundler created that is fast and simple to use -> New features are requested and added to the bundler which makes it more sluggish -> New bundler created that is fast and simple.
I wonder how this compares to ParcelJS.
I’m a big fan of Go but there are usually fringe benefits in having the bundler written in the language that it is bundling I.e. JavaScript.
I wonder how this compares to ParcelJS.
I’m a big fan of Go but there are usually fringe benefits in having the bundler written in the language that it is bundling I.e. JavaScript.
Benchmark against parcel is in the documentation. 200x faster.
That’s exceptional, if the benchmarks test identical workloads and features I would definitely consider switching to this.
FWIW it may sound like I’m hostile to this framework when I’m just trying to be critical in a constructive way. I use golang as my main driver at work and find most projects in Go to be better then its common counterparts, however I have been through so many JS bundlers that I scratch my head when anyone claims superior results on (Self-declared) hobby ones.
OP should know this seems really solid and like great work. I just don’t want junior devs running head first putting this in production.
FWIW it may sound like I’m hostile to this framework when I’m just trying to be critical in a constructive way. I use golang as my main driver at work and find most projects in Go to be better then its common counterparts, however I have been through so many JS bundlers that I scratch my head when anyone claims superior results on (Self-declared) hobby ones.
OP should know this seems really solid and like great work. I just don’t want junior devs running head first putting this in production.
If these numbers are true, it would be nice to have more documentation and tutorials. This would help the javascript community a lot.
I built a gaming computer to work with javascript codebases, true story.
I built a gaming computer to work with javascript codebases, true story.
I think if somebody replaces webpack with something that is 100x faster has 80% of webpacks most important features, many companys would pay real money for it.
[deleted]
I made my last webpage in 1999 using HTML tables and cut-up "design" jpgs so I am a bit out of date in this area.
I see all these preprocessors and crap for "web apps" and I wonder why nobody just writes HTML/CSS/Javascript instead of all of these other "languages" and abstractions or is the base web language now so bad that nobody touches it manually?
I see all these preprocessors and crap for "web apps" and I wonder why nobody just writes HTML/CSS/Javascript instead of all of these other "languages" and abstractions or is the base web language now so bad that nobody touches it manually?
Desktop-to-web deserter here, I’ll try to sum it all up.
- Browsers still have bad and ugly environments. You don’t have all features everywhere, even in recent versions. Things like “class Foo { static x = new this }”-level are missing, not some rare magic (like proper modules). Node.js is a different beast, despite a [vaguely stated] promise to be indistinguishable. Lots of env-neutral code written in node doesn’t work in a browser as is. Think msvc vs gcc/c99.
- CSS is just too low-level, people do like sass, less, etc very much. These have “includes”, functions, variables, flow control, non-css inheritance, bem-related isolation, common shorthands for a syntax. Modular apps cannot live without that.
- DOM is so flipping slow that it requires precise differential updates to it if you don’t want to charge your battery four times a day. It is also much less “cool” than desktop ui data-controls, and it requires a framework, which usually comes from node_modules for reasons above (prebundled versions exist for syntaxless fws). Also, too few people like to write a big single html (again, no “includes” etc) and bind it to the code directly, so there is vue, jsx, pug, etc. The more you abstract it away (dom, not html), the better it is for everyone.
In terms of programming, if you want an analogy, pure browser feels like a single qbasic file on steroids, while bundlers make it feel almost like a visual studio project (put here any decent ide if you don’t like vs).
- Browsers still have bad and ugly environments. You don’t have all features everywhere, even in recent versions. Things like “class Foo { static x = new this }”-level are missing, not some rare magic (like proper modules). Node.js is a different beast, despite a [vaguely stated] promise to be indistinguishable. Lots of env-neutral code written in node doesn’t work in a browser as is. Think msvc vs gcc/c99.
- CSS is just too low-level, people do like sass, less, etc very much. These have “includes”, functions, variables, flow control, non-css inheritance, bem-related isolation, common shorthands for a syntax. Modular apps cannot live without that.
- DOM is so flipping slow that it requires precise differential updates to it if you don’t want to charge your battery four times a day. It is also much less “cool” than desktop ui data-controls, and it requires a framework, which usually comes from node_modules for reasons above (prebundled versions exist for syntaxless fws). Also, too few people like to write a big single html (again, no “includes” etc) and bind it to the code directly, so there is vue, jsx, pug, etc. The more you abstract it away (dom, not html), the better it is for everyone.
In terms of programming, if you want an analogy, pure browser feels like a single qbasic file on steroids, while bundlers make it feel almost like a visual studio project (put here any decent ide if you don’t like vs).
Thanks for this it makes it a bit more clear, but I'm now never going to touch web again lol.
It sounds like hell to me.
However, what you describe above sounds like what you need to write a full blown application. Why does one need this to display simple information like a blog or news? Tasks that I used the HTML tables stuff for. Nowadays I can enter a personal blog with maybe 100 lines of actual text but it still pulls several megabytes of javascript.
I just can't wrap my head around why all this is necessary when all you want is to display information that could be equally served by a text or html file.
It sounds like hell to me.
However, what you describe above sounds like what you need to write a full blown application. Why does one need this to display simple information like a blog or news? Tasks that I used the HTML tables stuff for. Nowadays I can enter a personal blog with maybe 100 lines of actual text but it still pulls several megabytes of javascript.
I just can't wrap my head around why all this is necessary when all you want is to display information that could be equally served by a text or html file.
I feel your disdain! I'm an old-timer and remember complaining about using those new-fangled cascading style sheets instead of just putting everything in html tag attributes.
And don't get me started about having been forced to use div's for layouts instead of tables. In fact, I still don't understand why the html table construct couldn't have been evolved to accommodate a layout role.
And don't get me started about having been forced to use div's for layouts instead of tables. In fact, I still don't understand why the html table construct couldn't have been evolved to accommodate a layout role.
What is funny, both grid and flex still fail at some use cases. For one, grid is unaware of “media print” and cannot paginate itself (that was a big facepalm after an hour of preparing). It is also 10k rows/cols max by definition, 1k max by a common implementation. Can’t style gaps too. Otoh, flex is non-tabular, so you cannot simulate both with just one. I also met some trouble with inner flex scrolling (“overflow-xy” hierarchy forwarding issues), but these details I forget as fast as I meet them.
The entire css thing is overly situational. In a generalized model, any problem is solved by a finite set of building blocks. In css, any problem except few premodelled ones is a pile of hacks and worms that never get back into a can.
The entire css thing is overly situational. In a generalized model, any problem is solved by a finite set of building blocks. In css, any problem except few premodelled ones is a pile of hacks and worms that never get back into a can.
Four issues:
1) JavaScript has to be sent over the wire, so it should be minified.
2) Caching is easier if you hash your assets, ie put the MD5 into the URL so browsers know if they’ve downloaded it before or not.
3) Old browsers don’t support modern JS syntax but it can pretty much all be backported by eg turning () => {} into function() {} etc.
4) It’s convenient to use other people’s code by importing it from open source libraries with NPM.
Small issues individually but they add up to meaning any non-hobbyist project ends up needing a bundler.
1) JavaScript has to be sent over the wire, so it should be minified.
2) Caching is easier if you hash your assets, ie put the MD5 into the URL so browsers know if they’ve downloaded it before or not.
3) Old browsers don’t support modern JS syntax but it can pretty much all be backported by eg turning () => {} into function() {} etc.
4) It’s convenient to use other people’s code by importing it from open source libraries with NPM.
Small issues individually but they add up to meaning any non-hobbyist project ends up needing a bundler.
No, base HTML/CSS/JS are currently brilliant - however when you are building complex interfaces, web editors, React/Angular allow you to do it much easier
Yes so you have libraries to help.
But what I don't really understand is why do you have to have all the "build tools" that translates one language into another and a lot of all the other stuff. I have no idea what they really do, but it seems so strange that it would be a large process for a scripting language that runs directly in the browser.
Wouldn't be easier to specify the UI directly in javascript using this library like with C++/qt? C++/qt also has something similar to html for declaring UI/UI actions called QML but I never managed to see the improvement over just writing code.
But what I don't really understand is why do you have to have all the "build tools" that translates one language into another and a lot of all the other stuff. I have no idea what they really do, but it seems so strange that it would be a large process for a scripting language that runs directly in the browser.
Wouldn't be easier to specify the UI directly in javascript using this library like with C++/qt? C++/qt also has something similar to html for declaring UI/UI actions called QML but I never managed to see the improvement over just writing code.
Sure you can use plain JS, but then you don’t get async/await, destructuring, module system, type checking, JSX, and countless other modern affordances that make writing JS more palatable. If you can stomach OldJS, then yeah, none of this has much use for you.
Well, last time I checked, you could use a lot of these out of the box, as long as you didn't mind dropping support for old Internet Explorer versions?
A lot of these, but not all (JSX for example). And then you have to worry about support - caniuse is great help, but it gets very tiring.
But to me the main benefit is something else. React & co. let you define the state and then rendering based on that. It is incredible what this means for code maintainability... Try it, it really is worth the effort.
But to me the main benefit is something else. React & co. let you define the state and then rendering based on that. It is incredible what this means for code maintainability... Try it, it really is worth the effort.
I've since decided to boycott JavaScript as a developer, and to block it everywhere I could as a user. (And rendering should be decided by the user's browser.)
Web apps, basically. You couldn’t make Google Sheets that way, for example.
So it would be impossible to write google sheets in pure html/javascript (except for the server bits that is)?
The matter is that it wouldn't make sense to do it. It's certainly possible.
If you try building a large scale application (not merely a website or simple CRUD-style service), something as complex and demanding as a Google Sheets in pure html/javascript, you're going to reinvent a lot of wheels in the process and waste an enormous amount of time, for reasons (hopeful performance gains, and minimizing external requirements) that will not be strong enough to justify the time cost. More likely in the process of reinventing wheels, you'll lose a lot of your hoped for performance and simplification gains, as you create your own monsters and bloat.
If you try building a large scale application (not merely a website or simple CRUD-style service), something as complex and demanding as a Google Sheets in pure html/javascript, you're going to reinvent a lot of wheels in the process and waste an enormous amount of time, for reasons (hopeful performance gains, and minimizing external requirements) that will not be strong enough to justify the time cost. More likely in the process of reinventing wheels, you'll lose a lot of your hoped for performance and simplification gains, as you create your own monsters and bloat.
Well, the issue these days is that JavaScript is becoming a serious security problem. Hopefully more and more people will just start blocking JavaScript outright, and this will just have been a misguided attempt to try and turn the web browser into a full blown virtual machine...
It's not impossible. That's obvious since the frameworks ultimately compile down to plain HTML/CSS/JS. The consideration is around how workable the codebase becomes/is.
Seconded. I mean, computers made of rubber bands and tinker toys that play tic tac toe exist, but I wouldn’t make a chess playing computer that way.
After a few years of mostly doing React apps, I’m considering vanilla js for a new thing. It’ll be best for performance purposes, but man, I’m gonna miss components.
After a few years of mostly doing React apps, I’m considering vanilla js for a new thing. It’ll be best for performance purposes, but man, I’m gonna miss components.
I think it legacy browsers is to blame. You can certainly write web apps in plain HTML/CSS/JavaScript, but it will be limited. So, some transpiler are invented to solve this problem, and bundler is built to solve module problem. Of course, you don't have to use them if you don't care about old browsers, but many companies do care.
I was thinking that a parser and tokeniser distributed as wasm could increase the build speed while maintaining the js part for customisation. But this is a full fledged native bundler. Though less flexible it's incredibly fast!
This is really cool - our dashboard uses Vue for the frontend and a Go backend. Everytime I deploy, I think about how insanely slow the frontend 'compilation' + unit tests step is compared to testing and building the Go code. It's one of the things that makes Go a joy to code in for me.
On a similar note, we have some services written in Python and I was doing some research for some static type checking tools. At least in my testing, Pyright (written in Node) outperforms the pants out of similar tools written in Python. Using Go to bundle + minify Javascript sounds perfectly reasonable to me.
On a similar note, we have some services written in Python and I was doing some research for some static type checking tools. At least in my testing, Pyright (written in Node) outperforms the pants out of similar tools written in Python. Using Go to bundle + minify Javascript sounds perfectly reasonable to me.
reading the go code is a true pleasure and great resource to study the language further.
I've been using parcel because it's the only thing I know and I always thought it was slow but it claims to be fast and I'm new to js so I took it's word for it. According to this it's the slowest of the lot, though. Why does it claim to be "blazing fast"?
Yes this was a surprise for me too. The benchmark I used for esbuild deliberately tests full build time without a cache, since that is an important metric to me. I assume Parcel is talking about incremental build times when the cache has already been populated. In that case I'm sure Parcel is not as slow.
Depends on what you are measuring. If 95% of what you care for is development mode, then it may not even be faster. Individual files only have to be compiled once in watch mode. For some bundlers even between builds.
If you measure production builds, it's a different story. As every change has to most likely redo tree shaking and minifying over all the code. If you want source maps, it gets even worse.
If you measure production builds, it's a different story. As every change has to most likely redo tree shaking and minifying over all the code. If you want source maps, it gets even worse.
For my project with maybe 30 dependencies in package.json, the incremental build finishes within a second on watch mode when I edit a file, so it's quite fast in that case.
> Parsing, printing, and source map generation are all fully parallelized
I think this the major cause of speedup. Now that NodeJS supports WorkerThreads, it can be done in parallel. Along with this if right choice of data structures used there is no need to depend upon writing a build manager in another language instead of host language.
Also, as of today JS(V8) performance is getting closer to Go runtime.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
I think this the major cause of speedup. Now that NodeJS supports WorkerThreads, it can be done in parallel. Along with this if right choice of data structures used there is no need to depend upon writing a build manager in another language instead of host language.
Also, as of today JS(V8) performance is getting closer to Go runtime.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
> I think this the major cause of speedup.
This simply can't be true. On a 4 core laptop you might expect parallelism to provide up to 4x performance. But this has 75x / 200x performance, that must come partly from other reasons.
This simply can't be true. On a 4 core laptop you might expect parallelism to provide up to 4x performance. But this has 75x / 200x performance, that must come partly from other reasons.
First of all you didn't even care to read I also mentioned that choosing the right data structures also. ;)
You are deluded by the benchmark posted too. I terms of different build process steps that library isn't doing as much as other js build frameworks.
You are deluded by the benchmark posted too. I terms of different build process steps that library isn't doing as much as other js build frameworks.
I understood your phrase "the major cause" to mean majority reason, that couldn't be true since the remaining difference is greater. If you meant something different than that, then I misunderstood you. We both agree that the remainder of the performance (the bulk of it) must come from other reasons than parallelism.
As well as your suggestions (doing less work, choosing the right data structures) - I would also propose (A) no V8 JIT warmup time (B) no need for fstat on many node_modules files (C) fewer AST passes (D) possibility of value struct types => less boxing => lower GC pressure (E) less dynamism/no plugin architecture.
As well as your suggestions (doing less work, choosing the right data structures) - I would also propose (A) no V8 JIT warmup time (B) no need for fstat on many node_modules files (C) fewer AST passes (D) possibility of value struct types => less boxing => lower GC pressure (E) less dynamism/no plugin architecture.
Superb project. Did you try comparing against Packem. We leverage Rust as a backend for the bundling and resolving phases:
https://packem.github.io
https://packem.github.io
Nice work! Building from scratch without any caching is not the build time devs will be seeing 99% of the time, so it’s probably not the best benchmark.
How fast is an incremental rebuild with webpack though? That esbuild finishes from scratch in .5s (though probably with the code previously loaded from disk) means it’ll still likely be the fastest.
Posters up thread claim that minifying is the slowest part of webpack, so presumably that’s pretty consistently slow relative to input size.
Posters up thread claim that minifying is the slowest part of webpack, so presumably that’s pretty consistently slow relative to input size.
You don’t need to minify during development, so like building from scratch I’m not sure how much it matters for a practical benchmark.
I don’t at all doubt esbuild is faster than the others, just in practice I doubt it’s as much faster as the benchmark suggests.
I don’t at all doubt esbuild is faster than the others, just in practice I doubt it’s as much faster as the benchmark suggests.
Yass. This is a really well written go codebase. Go read it and see what life was like before the framework kids showed up.
Eh. I’ve been working in the industry since before the framework kids showed up. I’ve seen plenty of truly awful C++, C#, and COBOL codebases.
[deleted]
how does this compare with SWC made with rust?
Go is such a strange choice of language for this kind of thing compared to something in the ML family.
A bundler written in OCaml:
https://github.com/fastpack/fastpack/blob/master/README.md
Yes, think of all the great tools written in Haskell, like pandoc and uh… a second example someone will reply to me with that no one has ever fucking heard of.
Nah, the second one is also pretty popular.
https://www.shellcheck.net
https://www.shellcheck.net
Well, I can say I have heard of it. It still strikes me as a pretty much trivial tool compared to the bazillion things written in Rust and Go in the last five years, but it’s better than nothing.
Way to get overly salty.
ML is not just Haskell, there is OCaml and F# as other notable dialects. (and to a much more limited extent, Rust, which borrows a few features but is otherwise more traditionally C-like)
The language family lends itself very well for writing data transformation, especially tree processing (of which most parsers require), which is a very common kind of operation when writing minifiers and syntax transformers.
ML is not just Haskell, there is OCaml and F# as other notable dialects. (and to a much more limited extent, Rust, which borrows a few features but is otherwise more traditionally C-like)
The language family lends itself very well for writing data transformation, especially tree processing (of which most parsers require), which is a very common kind of operation when writing minifiers and syntax transformers.
I agree that syntax transformations are in the wheelhouse of ML-family languages, but this just makes the absence of any popular tools besides pandoc more striking.
Well, the Haxe compiler is written in OCaml!
But yes, I do agree that the ML family is pretty barren, at least in terms of publicly-available software.
It is, perhaps, the "functional programming" of functional programming! (as in, FP is already a couple orders of magnitude less popular than OOP/procedural, ML is definitely less popular than, say, Lisp)
But yes, I do agree that the ML family is pretty barren, at least in terms of publicly-available software.
It is, perhaps, the "functional programming" of functional programming! (as in, FP is already a couple orders of magnitude less popular than OOP/procedural, ML is definitely less popular than, say, Lisp)
Can we drop the "in Go" in links like these? I don't really see how that adds any value to the end user.
The thing that caught my attention about this post was specifically that it‘s a JavaScript bundler not written in JavaScript. The “in Go” in this case made the difference between my noticing and skipping a very interesting submission, so I think it’s relevant.
Almost all JavaScript tooling is itself written in JavaScript, so the fact that it's written in Go is actually notable to me.
That’s a no go for me.
X in Go seems to be the new Y in Rust.
I am curious to know what is the benefit of doing this in Go?
I thought that the reason why webpack is complex is because the whole JavaScript ecosystem is complex with respect to the ES20XX standards. I fail to understand how rewriting the logic in another language is going to reduce the complexity.
Speed is important, sure, but certainly not at the cost of correctness!
I am curious to know what is the benefit of doing this in Go?
I thought that the reason why webpack is complex is because the whole JavaScript ecosystem is complex with respect to the ES20XX standards. I fail to understand how rewriting the logic in another language is going to reduce the complexity.
Speed is important, sure, but certainly not at the cost of correctness!
I think X in Go preceded Y in Rust by several years.
In short Go is a simple language with minimal syntax and explicit design patterns that make it very easy to read and understand once you’ve learned the ropes. It’s a typed and compiled language with all the benefits that entails, with great concurrency support.
It’s standard library is fantastic, compile times are great (one of the original goals for Go was to reduce c++ compile times at Google), with out of the box support for networked/web applications on the standard library.
Finally the community for Go is very strong with things like the gophers slack channel which is insanely popular and friendly.
In short Go is a simple language with minimal syntax and explicit design patterns that make it very easy to read and understand once you’ve learned the ropes. It’s a typed and compiled language with all the benefits that entails, with great concurrency support.
It’s standard library is fantastic, compile times are great (one of the original goals for Go was to reduce c++ compile times at Google), with out of the box support for networked/web applications on the standard library.
Finally the community for Go is very strong with things like the gophers slack channel which is insanely popular and friendly.
Haha, yes.
Thanks, I’ll certainly check out the gophers community.
As the sibling points out babel/webpack are like compilers of JavaScript world so reducing compile times have a huge impact on developer experience.
I am just wondering where does the complexity of juggling various ES* standards go? Also, the webpack plugins and loads of config examples might require this project to support additional complexity. Will the compiled/typed nature of Golang make it difficult to correctly maintain parity with babel/webpack?
I am really looking forward to learning golang this year. I am almost done ~~suffering~~ learning JavaScript/React + CSS3/media-queries this winter.
Thanks, I’ll certainly check out the gophers community.
As the sibling points out babel/webpack are like compilers of JavaScript world so reducing compile times have a huge impact on developer experience.
I am just wondering where does the complexity of juggling various ES* standards go? Also, the webpack plugins and loads of config examples might require this project to support additional complexity. Will the compiled/typed nature of Golang make it difficult to correctly maintain parity with babel/webpack?
I am really looking forward to learning golang this year. I am almost done ~~suffering~~ learning JavaScript/React + CSS3/media-queries this winter.
Browsers lag so hard behind the ES standard that most developers are using shims provided within Babel or Typescript to use the features early. Normally you specify whatever ESNext features your using in your tsconfig, webpackconfig, babelrc, etc.
I'm not sure if the implementations are always formalized such as the shim you use is identical to whatever implementation that major browsers end up going with.
Ultimately you just end up bloating your bundle because you need added code in your shims for backwards compatibility of these advanced features.
I'm not sure if the implementations are always formalized such as the shim you use is identical to whatever implementation that major browsers end up going with.
Ultimately you just end up bloating your bundle because you need added code in your shims for backwards compatibility of these advanced features.
Not sure anyone said anything about reducing complexity. Webpack and Babel in tandem are pretty much "the compiler of modern JavaScript". They do a lot of crunch work and can take a long time to run. Makes perfect sense to me to try and do that in a faster language.
You could potentially not need node installed in order to do "modern" frontend development.
I'm trying to create a build tool that a) works well for a given sweet spot of use cases (bundling JavaScript, TypeScript, and maybe CSS) and b) resets the expectations of the community for what it means for a JavaScript build tool to be fast. Our current tools are way to slow in my opinion. Let's build faster tools!