10k Bounty for Rewriting Prettier in Rust(twitter.com)
twitter.com
10k Bounty for Rewriting Prettier in Rust
https://twitter.com/Vjeux/status/1722733472522142022
79 comments
"I care that people can have a faster version of prettier, not how it's implemented." & "... for any project written in Rust" are contradictory.
That was in the context of a question about whether external crates may be used. Rust is clearly implied.
Looks like the bounty is at 20k now with contribution from @rauchg https://twitter.com/rauchg/status/1723400569392656771
Prettier runs on a single thread only. Running it on each thread could make it significantly faster [0].
[0]: https://github.com/microsoft/parallel-prettier
[0]: https://github.com/microsoft/parallel-prettier
The performance gain is massive!
I don’t really get it. Prettier isn’t something that you need to run that often on the entire repo; you’re probably just running it on save in a handful of milliseconds or on a pre-commit on hopefully less than 10 files.
There are plenty other tools that often run on the entire repo, like tests, lint, build, type-checking, etc. focus on those. Bun showed that there’s a lot of space for improvement.
There are plenty other tools that often run on the entire repo, like tests, lint, build, type-checking, etc. focus on those. Bun showed that there’s a lot of space for improvement.
Adding on to the other comment about CI setups, when I benchmarked removing eslint-plugin-prettier from my work’s ESLint configuration we saw a 37.5% speedup. On our fairly slow CI agents that ended up being pretty significant (~34s per run).
I _wanted_ to drop the plugin and have Prettier automatically run on as a pre-commit hook (as you suggested) but in the end I lost :) We wound up keeping the plugin and eating the slower pipeline time.
Of course there’s more overhead than just Prettier in this case—the ESLint plugin itself will be contributing towards the added runtime!—but depending on your codebase and CI machines, Prettier can definitely slow things down quite a bit. CI taking over a minute or two is a big drag on developer productivity imo.
Ideal world is that everyone is on board with code formatting as a pre-commit task that only touches modified files so it doesn’t bloat CI unnecessarily, but it’s not always possible :(
I _wanted_ to drop the plugin and have Prettier automatically run on as a pre-commit hook (as you suggested) but in the end I lost :) We wound up keeping the plugin and eating the slower pipeline time.
Of course there’s more overhead than just Prettier in this case—the ESLint plugin itself will be contributing towards the added runtime!—but depending on your codebase and CI machines, Prettier can definitely slow things down quite a bit. CI taking over a minute or two is a big drag on developer productivity imo.
Ideal world is that everyone is on board with code formatting as a pre-commit task that only touches modified files so it doesn’t bloat CI unnecessarily, but it’s not always possible :(
I’ve done some research on this in the past. The reason that is so slow is because the eslint plugin runs prettier, does a diff between the result of prettier and the real file and then converts each diff entry into a eslint issues. Most of the pain comes from this remarshaling cost
I ran into the same issue and have been separating the prettier and eslint steps ever since: https://cpojer.net/posts/fastest-frontend-tooling-in-2022#es...
What we did to avoid that bottleneck was just turn off the prettier eslint rule in CI and just run the `prettier --check` command they recommend in their docs.
We also have it set to only run in changed files which helps a lot.
We generally don't link pre-commit hooks for standards though, hence the CI focus. Too easy to circumvent, and I'd rather pay for an external machine to check it when it matters (pre-merge by doing it on PR commits) than block my devs when it doesn't (every time they commit to a non-main branch).
We also have it set to only run in changed files which helps a lot.
We generally don't link pre-commit hooks for standards though, hence the CI focus. Too easy to circumvent, and I'd rather pay for an external machine to check it when it matters (pre-merge by doing it on PR commits) than block my devs when it doesn't (every time they commit to a non-main branch).
Many organisations have a CI step that runs a linter or formatter on all the code and fails if the tool makes changes.
The idea is to catch cases where contributors failed to run the tool on save.
On large code bases this can be time consuming.
The idea is to catch cases where contributors failed to run the tool on save.
On large code bases this can be time consuming.
The trick to make CI in large codebases fast, or how to make any system fast, is to only test what’s changed.
In CI you only need to lint the files that have changed, or run the tests that depend on code that has changed etc.
This way the time it takes to execute the tests scales with the amount of changes, and not with the total amount of code.
In CI you only need to lint the files that have changed, or run the tests that depend on code that has changed etc.
This way the time it takes to execute the tests scales with the amount of changes, and not with the total amount of code.
The difficulty with that is trust, if there’s any chance it will miss things by accident then I’d rather eat the slow runs.
Test runners that run new/changed tests first though are cool. Faster feedback on error but still comprehensive.
Test runners that run new/changed tests first though are cool. Faster feedback on error but still comprehensive.
I believe you don't see the big picture.
What you just described is running on thousands of machines several times a day.
What you just described is running on thousands of machines several times a day.
Or, better still, just remove prettier and free yourself to write more expressive code again.
it's very common to have a lint/format step that make sure all files are well-formatted in Pull request.
This is such a waste of electricity IMO. Just occasionally batch-fix all files for syntax. Semantic linting can run in CI, as it's about catching bugs. But formatting is for long-term consistency and doesn't need to be done on every commit.
I'm not a lint fascist, but enforcing them gave our team:
- less time reviewing: no more "you forgot to indent here"
- less git conflicts: because of differences in trailing comas, line breaks, ...
- less (to no more) code style debates: the linter is the benevolant dictator
- less time making our code clean, thinking about line breaks: just get it done, the linter and CI will handle formatting !
Occasionally batch-fixing syntax makes "git blame" far less useful.
Yup, but you need to reference the commit ID, meaning that you always need to create two commits, one for the reformatting, and a second one for updating the .git-blame-ignore-revs file. Additionally, you may not use git rebase, or any other workflow that modifies the history, or the references will be broken. Third, the git config needs to be set per-project (can't be set globally), so all developers need to manually adjust their config, for this feature to be useful.
If you value a clean commit history, then simply reformat as part of the code change. With autoformatting on save (e.g. through prettier), that's a complete non-issue anyways.
If you value a clean commit history, then simply reformat as part of the code change. With autoformatting on save (e.g. through prettier), that's a complete non-issue anyways.
Until you change the formatting rules and the whole codebase is in one diff. This seems to happen at least once a year from my observations.
only in node.js, it's a waste of electricity.
in other language, formatters are pretty damn fast. never heard anyone complain about gofmt or cargo fmt
in other language, formatters are pretty damn fast. never heard anyone complain about gofmt or cargo fmt
you may have to force approve edits by the formatter then, since they’re pretty annoying when included in changes for code review.
> The main issue is that none of them match the long tail of formatting logic of prettier. I'm putting up a $10k bounty for any project written in Rust that passes > 95% of the prettier JavaScript tests.
95 % is not the "long tail" in my opinion. Maybe 100 % is too hard, but 95 % is rather low.
95 % is not the "long tail" in my opinion. Maybe 100 % is too hard, but 95 % is rather low.
Why does everything need to be written in rust these days?
TBH this sounds like a great candidate for rewrite - likely millions of hours of runtime/hour. Prettier is written on a platform with a history of such rewrites getting 10x perf increases. The only alternative that comes to mind is Go but I'm not that offended by Rust choice - has pros and cons.
It's not just speed, but also correctness. A quick search in the Prettier repo shows plenty of issues for "undefined is not a function", "cannot read property X of undefined/null", and other such errors that are avoidable in a language with a good type system.
Makes me wonder if Prettier would be noticeably improved it it were migrated to TypeScript (first).
IIRC the maintainers are not currently open to migrating to TypeScript
It’s a great candidate for a rewrite buy why not Go, Zig, Ocaml, Haskell e. t. c.?
No matter what language they choose, someone will complain and ask "why X, why not Y or Z?"
Right which is part of why dictating the implementation language up front as part of the bounty is a mistake. They should have put the bounty on the speedup factor they're looking for and let it sort itself out.
My not-completely-naive guess is an expert optimizer can get the improvement they're looking for with only small parts or maybe even none of it rewritten in another language.
My not-completely-naive guess is an expert optimizer can get the improvement they're looking for with only small parts or maybe even none of it rewritten in another language.
familiarity. and rust seems to be winning the war for web development tooling, at the moment. zig (by way of bun) may make a good stand if bun were to produce such tools too, as standalone. it’s a high barrier if bun has to be adopted wholesale in order to have a pretty printer.
The swc project reimplemented some of the js build toolchain in rust which led to some great speed ups.
I’d imagine that this call for rewriting prettier is inspired by swc
Not sure if updating prettier will benefit as much as transpilation.
[0] https://swc.rs/docs/benchmarks
I’d imagine that this call for rewriting prettier is inspired by swc
Not sure if updating prettier will benefit as much as transpilation.
[0] https://swc.rs/docs/benchmarks
Any AOT compiled language will do, but only Rust will give extra hacker credo points.
As a rust dev I'm just as perplexed. I've seen preprocessors and stuff like tailwind being rewritten in Rust aswell. Why though (
As a (nominally) front-end developer the RIIR movement's energy is similar to how the front-end community approaches things.
Overall Rust and its community have many traits that make them very approachable to people not expected to know anything about systems programming, namely:
-Package manager with a familiar philosophy behind it (and not the sort of hell you have to deal with setting up a C++ development environment).
-Outputs WASM without much bureaucracy and advertises it.
-Friendly compiler messages. I don't recall seeing "perhaps" in a compiler error message before.
The people designing Rust put much effort into making the language as approachable as possible and that's the net effect.
That is not to say this is going to be necessarily successful, but so far the enthusiasm is there.
Overall Rust and its community have many traits that make them very approachable to people not expected to know anything about systems programming, namely:
-Package manager with a familiar philosophy behind it (and not the sort of hell you have to deal with setting up a C++ development environment).
-Outputs WASM without much bureaucracy and advertises it.
-Friendly compiler messages. I don't recall seeing "perhaps" in a compiler error message before.
The people designing Rust put much effort into making the language as approachable as possible and that's the net effect.
That is not to say this is going to be necessarily successful, but so far the enthusiasm is there.
Some dev workflow tools really benefit from being very fast. And parsing and ast building and the like are actually things that rust is quite good for.
That doesn’t mean it couldn’t be done in other languages, but it’s a weighing of trade-offs. And of course, groups of people, such as developers on socials, are quite sensitive to popularity.
That doesn’t mean it couldn’t be done in other languages, but it’s a weighing of trade-offs. And of course, groups of people, such as developers on socials, are quite sensitive to popularity.
> And parsing and ast building and the like are actually things that rust is quite good for.
Having built a compiler and typechecker in Rust, I don't know if I'd say that it was "quite good" for it. Going back, I'd still do it again, but it wasn't exactly a cakewalk. Rust's lifetime rules and visceral hatred of referential structs did not make my life easy at all. I definitely took the easy way out with a lot of string copies too.
Having built a compiler and typechecker in Rust, I don't know if I'd say that it was "quite good" for it. Going back, I'd still do it again, but it wasn't exactly a cakewalk. Rust's lifetime rules and visceral hatred of referential structs did not make my life easy at all. I definitely took the easy way out with a lot of string copies too.
Just as an example: Our eslint checks take 2 minutes to complete. Improving this would considerably improve developer productivity.
> I've seen preprocessors and stuff like tailwind being rewritten in Rust aswell. Why though
a) speed. On large codebases each preprocessor (and there may be many) can take a long time
b) Rust has several features which make it quite suitable for writing pasrers and suchlike
a) speed. On large codebases each preprocessor (and there may be many) can take a long time
b) Rust has several features which make it quite suitable for writing pasrers and suchlike
If it speeds up a build/CI pipeline it’s a worthwhile investment for a lot of folks
You can get a big speedbump with go without introducing the overhead of object lifecycle management.
Your sentence is confusing; a speedbump is designed to slow you down.
Haha, you're right. I meant speed bump. As a German I'm used to chaining words together.
[deleted]
It wasn't the chaining that was an issue: https://en.wikipedia.org/wiki/Speed_bump .
Perhaps "speed boost" would be a better alternative for your intended meaning.
Perhaps "speed boost" would be a better alternative for your intended meaning.
Interestingly "bump in speed" would be fine, but "speed bump", means the exact opposite.
It isn’t in this context. “got a speed bump” vs “hit a speed bump”
Then rewrite Prettier in Go and I'm sure people will use it :)
Not slightly interested. My point is: we have tools that fit the problem better instead of supporting the hype train of rust.
What do you suggest would be better? Command-line tools, parsing and parallelism are all things that Rust excels at.
Why on earth would you need object lifecycle mem management for these things? These are points where Go excels as well. Rust is meant as a systems language, I'd love to have it as a general purpose lang as well, but it's not imho.
So you think Rust is a bad choice because... the programmers will need to deal with the borrow checker? I think it's been demonstrated that programmers are fine with that.
Crab people [1]
[1] https://en.wikipedia.org/wiki/Crab_People
[1] https://en.wikipedia.org/wiki/Crab_People
It is a sad moment for me, because I have more difficulties understanding Rust than Haskell, and believe me, Haskell is very difficult already. I know C, Lua, OCaml, Go, and the like, but Rust? Not really. What I hate the most is reference implementations being in Rust instead of C. C is almost like a pseudo-code.
Because everything is written in nodejs.
Can you pls rewrite your comment in rust.
Is this going to be a race? Multiple people implementing it in parallel and one claims the prize?
It would be problem just as fast if rewritten in any compiled language. Maybe I should rewrite all the rust implementation in zig and see if it’s just as fast
I just imagined a transpiler from Rust to Zig.
This would probably be a good thing on net, but how would plugins for Rust work?
I use <https://github.com/NiklasPor/prettier-plugin-go-template> a bit and I'm not sure what one would have to do to make a plugin for a Rust binary.
I use <https://github.com/NiklasPor/prettier-plugin-go-template> a bit and I'm not sure what one would have to do to make a plugin for a Rust binary.
Well they could work any way the author wanted, but if it were me, I'd make the plugins WASM.
That's how my favorite code formatter (dprint) does it.
For that matter though, it's how I would implement just about any plugin, these days, except those that need to exchange large info rapidly (e.g. an editor plugin that needs to operate on every key press).
It's dead easy to implement on the app side, and plugin authors can use an increasing number of higher-level languages to build plugins.
That's how my favorite code formatter (dprint) does it.
For that matter though, it's how I would implement just about any plugin, these days, except those that need to exchange large info rapidly (e.g. an editor plugin that needs to operate on every key press).
It's dead easy to implement on the app side, and plugin authors can use an increasing number of higher-level languages to build plugins.
> except those that need to exchange large info rapidly
I've seen attempts at zero-copy memory sharing between JS and WASM, but I'm not sure if the same approach would allow different plugins to share memory and communicate via e.g. Cap'n Proto?
https://observablehq.com/@kylebarron/zero-copy-apache-arrow-...
I've seen attempts at zero-copy memory sharing between JS and WASM, but I'm not sure if the same approach would allow different plugins to share memory and communicate via e.g. Cap'n Proto?
https://observablehq.com/@kylebarron/zero-copy-apache-arrow-...
And on top of that it’s also easy to mostly guarantee the plug-in can only access the APIs you give it. Which is quite nice for third party code.
...which would have the advantage that plugin authors could also use other languages that compile to WASM.
Would love to learn more if you have any resources available?
Isn't a bounty like the worst way to compensate someone for a defined piece of work?
It got the word out so well that it made the front page of HN, so I’d say it wasn’t a complete failure.
The people from Algora have created a great homepage for the bounty: https://console.algora.io/challenges/prettier
Note: Guillermo Rauch and Wasmer also joined, adding $12,500 extra to the mix (total bounty is $22,500)
Note: Guillermo Rauch and Wasmer also joined, adding $12,500 extra to the mix (total bounty is $22,500)
[deleted]
i’ve argued for writing tools in technology fit for purpose. a lot of web development tools being written in javascript (and then typescript) was because web developers had to build their own tools, and typically these folks were extremely comfortable with ruby and/or javascript, so naturally they reached for them. some of these important tools can have massive speedup gains just by switching internal data structures and algorithms. in my opinion, every rewrite implicitly makes this switch but the importance of these decisions are subsumed by the detail of the implementation language.
prettier can be faster with a better algorithm. then maybe rust can help it take advantage of specific operating system improvements not naturally available to node. regardless, bravo and onwards with more rewrites!
prettier can be faster with a better algorithm. then maybe rust can help it take advantage of specific operating system improvements not naturally available to node. regardless, bravo and onwards with more rewrites!
> prettier can be faster with a better algorithm
Do you have any examples of this or is there anything in particular that Prettier uses a suboptimal algorithm for?
Do you have any examples of this or is there anything in particular that Prettier uses a suboptimal algorithm for?
Biome already gets pretty close.
It is now 22.5k if it can be compiled upto wasm.
Sick. Hope someone does it.
Next do rubocop
The dprint plugin wraps Prettier under the hood so compatibility is good. The perf wins come from formatting files in parallel and incrementally.
[1] https://david.deno.dev/posts/faster-prettier-with-dprint/