Turborepo 1.2: High-performance build system for monorepos(turborepo.org)
turborepo.org
Turborepo 1.2: High-performance build system for monorepos
https://turborepo.org/blog/turbo-1-2-0
54 comments
> However, I feel a bit sad at the state of general purpose build tools (Bazel/Pants/Buck, make, GitHub Actions, Maven, Nix) or cloud-based IDEs (Cloud9, Codespaces). These tools come off as too complex to operate and build on top of...
I definitely agree, although I've found Please (please.build) to potentially be a solution in this area. It is a lot simpler, smaller, and more modern than Buck and Bazel, but shares similar properties as to be familiar (I.e. the buildfile syntax). I think it is supposed to be easier to extend with other languages, but I haven't tries that myself.
I definitely agree, although I've found Please (please.build) to potentially be a solution in this area. It is a lot simpler, smaller, and more modern than Buck and Bazel, but shares similar properties as to be familiar (I.e. the buildfile syntax). I think it is supposed to be easier to extend with other languages, but I haven't tries that myself.
> There are a huge number of mostly non-JS-specific problems that monorepo tooling eventually needs to solve: distributed build artifact and test result caching, distributed action execution, sandboxing, resource management and queuing, observability, and integration with other CI tools to name a few.
Turborepo author/founder here....
I agree. I built Turborepo because existing tools weren’t meeting our needs.
To solve these problems and still be flexible, many existing build tools end up with lots of configuration bloat. We’re trying to avoid that. We want to reimagine the developer experience of monorepo tooling and make it accessible for everyone.
Turborepo author/founder here....
I agree. I built Turborepo because existing tools weren’t meeting our needs.
To solve these problems and still be flexible, many existing build tools end up with lots of configuration bloat. We’re trying to avoid that. We want to reimagine the developer experience of monorepo tooling and make it accessible for everyone.
Hey. Off topic but would you consider handing Formik and TSDX to a community member? Huge appreciation for turborepo and your work on these other libraries -- they have become key parts of the ecosystem and turborepo could follow.
However, this is all the more reason why it could be super helpful to address the governance issue on those projects. Thanks and sorry for disturbing!
However, this is all the more reason why it could be super helpful to address the governance issue on those projects. Thanks and sorry for disturbing!
Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular?
(Is Bazel designed in a way that make it impossible to do JS monorepos well?)
(Is Bazel designed in a way that make it impossible to do JS monorepos well?)
Yes, it's possible to use Bazel w/ JS tooling. At Uber we run a 1000+ package monorepo w/ Bazel and Yarn. Someone else mentioned rules_nodejs if you want to go with the popular option that is more or less in line with the Bazel philosophy[0]. We use a project called jazelle[1] that makes some different trade-offs that lets us do some interesting things like not busting the whole world's cache when lockfile changes, and source code generation (including automatic syncing of BUILD files to package.json)
[0] https://news.ycombinator.com/item?id=30959893
[1] https://github.com/uber-web/jazelle
[0] https://news.ycombinator.com/item?id=30959893
[1] https://github.com/uber-web/jazelle
> Is Bazel designed in a way that make it impossible to do JS monorepos well?
Not impossible, but you really need to go all in with it and follow its conventions and practices. See this for the main docs: https://github.com/bazelbuild/rules_nodejs
One thing in particular that doesn't work well in the bazel world is doing your own stuff outside its BUILD.bazel files. If you're used to just npm install and jam some code in your package.json scripts... that doesn't usually work in the bazel world. If you have a lot of logic or tools in your build you'll likely need to go all in and make bazel starlark rules or macros that recreate that logic. Nothing is impossible, but expect to spend time getting up to speed and getting things working the bazel way.
Not impossible, but you really need to go all in with it and follow its conventions and practices. See this for the main docs: https://github.com/bazelbuild/rules_nodejs
One thing in particular that doesn't work well in the bazel world is doing your own stuff outside its BUILD.bazel files. If you're used to just npm install and jam some code in your package.json scripts... that doesn't usually work in the bazel world. If you have a lot of logic or tools in your build you'll likely need to go all in and make bazel starlark rules or macros that recreate that logic. Nothing is impossible, but expect to spend time getting up to speed and getting things working the bazel way.
> Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular?
It's definitely possible, but I think the practical limitations would make it too complex to reason around and maintain. You'd end up creating two separate and overlapping systems to declare dependency graphs and input sources and manage caching and execution.
I haven't yet seen a case where the two are actually interleaved. Currently at Databricks, we use Bazel to provide the correctness guarantees and interop needed for CI, and we use JS-specific tooling (non-Bazel) locally to meet our performance needs, where the usage profile is different and where we're willing to make correctness tradeoffs.
> (Is Bazel designed in a way that make it impossible to do JS monorepos well?)
There are limitations in Bazel that don't play nicely with modern JS conventions. For example, Bazel's standard sandbox is based on symlink farms, and Node.js and the ecosystem by default follow symlinks[1] to their real locations, effectively breaking sandboxing. A FUSE or custom filesystem (Google's version of Bazel takes advantage of one[2]) would be better but is not as portable. As another example, Bazel's action cache tries to watch or otherwise verify the integrity of every input file to an action, and when node_modules is 100k+ files, this gets expensive and is prone to non-determinism. Bazel does this for correctness, which is noble but results in practical performance problems. You need to do extra work to "trick" Bazel into not reading these 100k+ files each time.
The problems feel solvable to me, but not easily without adding yet more configuration options to Bazel. The influx of new JS-specific tooling is a reset to this, building the minimum viable set of functionality that the JS ecosystem specifically needs, without the burdens of being a general purpose build system.
[1] https://nodejs.org/api/cli.html#--preserve-symlinks [2] https://dl.acm.org/doi/10.1145/2854146
It's definitely possible, but I think the practical limitations would make it too complex to reason around and maintain. You'd end up creating two separate and overlapping systems to declare dependency graphs and input sources and manage caching and execution.
I haven't yet seen a case where the two are actually interleaved. Currently at Databricks, we use Bazel to provide the correctness guarantees and interop needed for CI, and we use JS-specific tooling (non-Bazel) locally to meet our performance needs, where the usage profile is different and where we're willing to make correctness tradeoffs.
> (Is Bazel designed in a way that make it impossible to do JS monorepos well?)
There are limitations in Bazel that don't play nicely with modern JS conventions. For example, Bazel's standard sandbox is based on symlink farms, and Node.js and the ecosystem by default follow symlinks[1] to their real locations, effectively breaking sandboxing. A FUSE or custom filesystem (Google's version of Bazel takes advantage of one[2]) would be better but is not as portable. As another example, Bazel's action cache tries to watch or otherwise verify the integrity of every input file to an action, and when node_modules is 100k+ files, this gets expensive and is prone to non-determinism. Bazel does this for correctness, which is noble but results in practical performance problems. You need to do extra work to "trick" Bazel into not reading these 100k+ files each time.
The problems feel solvable to me, but not easily without adding yet more configuration options to Bazel. The influx of new JS-specific tooling is a reset to this, building the minimum viable set of functionality that the JS ecosystem specifically needs, without the burdens of being a general purpose build system.
[1] https://nodejs.org/api/cli.html#--preserve-symlinks [2] https://dl.acm.org/doi/10.1145/2854146
Turborepo author/founder here....
Yes, it's possible. Databricks currently uses both Turborepo and Bazel together in their monorepo[1]
[1] https://twitter.com/elado/status/1504216742393876483?s=20&t=...
Yes, it's possible. Databricks currently uses both Turborepo and Bazel together in their monorepo[1]
[1] https://twitter.com/elado/status/1504216742393876483?s=20&t=...
I'm sad JS needs a build step. The best build system is none at all IMHO. I'd love to see native support everywhere for typescript or other things we typically depend on a build for today.
>I'm sad JS needs a build step. The best build system is none at all IMHO.
Good news: JS doesn't need a build step. Modern webdev wants a build step mostly because it wants Javascript to feel more like a "serious" language. There are technical benefits to compiling to Javascript, most of which can be served by other means, but the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill.
>I'd love to see native support everywhere for typescript or other things we typically depend on a build for today.
Typescript is part of the problem. You can literally just accept Javascript for what it is - dynamically typed - and write it like any other scripting language.
Good news: JS doesn't need a build step. Modern webdev wants a build step mostly because it wants Javascript to feel more like a "serious" language. There are technical benefits to compiling to Javascript, most of which can be served by other means, but the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill.
>I'd love to see native support everywhere for typescript or other things we typically depend on a build for today.
Typescript is part of the problem. You can literally just accept Javascript for what it is - dynamically typed - and write it like any other scripting language.
> Typescript is part of the problem
Typescript is a solution to the jank-fucking-tastic type munging JS does (see: [1] About 1:20 in) and the problems that ensue.... if all you are doing is just making menus appear/disappear on click, by all means keep to JS.
The build steps/tooling are useful when you want to build actual applications rather than decorate a marketing page, and also when you need to support legacy browsers, being able to work with modern sensibilities and get code that'll work in IE11 is a blessing.
> the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill.
JS has much more of a "flavour of the week" problem than more mature ecosystems like PHP, I put that down to a relatively poor stdlib by comparison, rather than aesthetics or gatekeeping.
Typescript is a solution to the jank-fucking-tastic type munging JS does (see: [1] About 1:20 in) and the problems that ensue.... if all you are doing is just making menus appear/disappear on click, by all means keep to JS.
The build steps/tooling are useful when you want to build actual applications rather than decorate a marketing page, and also when you need to support legacy browsers, being able to work with modern sensibilities and get code that'll work in IE11 is a blessing.
> the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill.
JS has much more of a "flavour of the week" problem than more mature ecosystems like PHP, I put that down to a relatively poor stdlib by comparison, rather than aesthetics or gatekeeping.
1. https://www.destroyallsoftware.com/talks/watI'm with you in that a lot of TS use seems dogmatic or ritualistic today. I have a strong feeling in the near future we're going to see the bow string snap back and simple zero build, basic dynamic use of pure JS comes back in vogue.
TS is about readability and maintainability of code at scale. Having types helps immensely in understanding a new codebase and working in a large codebase with many other people.
The problem is Javascript has been entirely commoditized by enterprise, so solutions which should only be relevant to "code at scale" have become mandatory at any scale. You practically can't distribute a Javascript library without writing it in Typescript and submitting it to NPM, with the build step expected.
Yet writing even complex javascript without Typescript and having it work is still entirely possible, just as it is with other weakly typed languages. There should still be room for that, but the concept of simply writing javascript has become so alien it needs to be reintroduced as "vanilla javascript."
Yet writing even complex javascript without Typescript and having it work is still entirely possible, just as it is with other weakly typed languages. There should still be room for that, but the concept of simply writing javascript has become so alien it needs to be reintroduced as "vanilla javascript."
It seems to be coming back as 'vanilla JS' so maybe there's hope.
> I'd love to see native support everywhere for typescript
This (controversial) feature is currently at stage 1 in TC39.
This (controversial) feature is currently at stage 1 in TC39.
I like github actions better than learning a new monorepo framework
bare-metal pnpm (what Rush is built upon) has never let me down. Handles rather large monorepos without issue.
Hey! In case you missed it, Turborepo launched 1.0 a few months ago[1].
This 1.2 release is important, as it includes --filter[2] (highly requested). You're able to much more easily filter tasks, instead of the previous --scope, --include-dependencies, --since, and --no-deps flags.
Happy to answer any questions about Turborepo or monorepos in general (recently wrote an entire post about them[3]).
[1]: https://turborepo.org/blog/joining-vercel
[2]: https://turborepo.org/docs/features/filtering
[3]: https://vercel.com/blog/monorepos-are-changing-how-teams-bui...
This 1.2 release is important, as it includes --filter[2] (highly requested). You're able to much more easily filter tasks, instead of the previous --scope, --include-dependencies, --since, and --no-deps flags.
Happy to answer any questions about Turborepo or monorepos in general (recently wrote an entire post about them[3]).
[1]: https://turborepo.org/blog/joining-vercel
[2]: https://turborepo.org/docs/features/filtering
[3]: https://vercel.com/blog/monorepos-are-changing-how-teams-bui...
Hello! We are using pnpm (which as I understand it has some overlap with what turborepo can do when it comes to monorepo management).
Some teams are considering turborepo for the remote caching feature. I'm a security guy, and I'm a little worried that remote caching basically means that all dev are able to run arbitrary code on other devs computer (by poising the cache). Am I missing something? Or are there security workarounds? Would, for example, only allow master build from the CI to push to the remote cache and only read only access for the devs work and still provide enough benefits? (I expect most dev to branch from master, so I'd assume the cache would be good enough?)
Thanks and congrats on the release!
Some teams are considering turborepo for the remote caching feature. I'm a security guy, and I'm a little worried that remote caching basically means that all dev are able to run arbitrary code on other devs computer (by poising the cache). Am I missing something? Or are there security workarounds? Would, for example, only allow master build from the CI to push to the remote cache and only read only access for the devs work and still provide enough benefits? (I expect most dev to branch from master, so I'd assume the cache would be good enough?)
Thanks and congrats on the release!
Hey! Yeah, we recommend using pnpm. Excellent tool. You might be interested in this feature from 1.2:
"You can enable Turborepo to sign artifacts with a secret key before uploading them to the Remote Cache. Turborepo uses HMAC-SHA256 signatures on artifacts using a secret key you provide. Turborepo will verify the remote cache artifacts' integrity and authenticity when they're downloaded. Any artifacts that fail to verify will be ignored and treated as a cache miss by Turborepo"
https://turborepo.org/docs/features/remote-caching
"You can enable Turborepo to sign artifacts with a secret key before uploading them to the Remote Cache. Turborepo uses HMAC-SHA256 signatures on artifacts using a secret key you provide. Turborepo will verify the remote cache artifacts' integrity and authenticity when they're downloaded. Any artifacts that fail to verify will be ignored and treated as a cache miss by Turborepo"
https://turborepo.org/docs/features/remote-caching
HMAC are symmetric signatures, which means that anyone who can check the signature can also poison the cache (so pushing only from the CI are reading/checking on the dev machines would not be possible)
any way you're going to solve the linting problem? I had having to install eslint and produce a config on every little package in my repository.
Yes I can use a generator to handle this and other boilerplate, but why? Can't turbo just spawn eslint in sub processes to lint each package and stitch the input back together?
Yes I can use a generator to handle this and other boilerplate, but why? Can't turbo just spawn eslint in sub processes to lint each package and stitch the input back together?
If I'm hearing you correctly, your pain is having to configure ESLint manually, correct? Are you wishing for ESLint to be included in Turborepo natively? My opinion is that giving folks the option on their linter of choice is a good thing, even though ESLint is very popular.
No, I’m saying I’d like to install eslint at the root of the repo and turbo could just spawn a process to use the root install of eslint for each package in the monorepo using the root config. Could do the same with prettier etc. just use a sub process to spawn and control the directories it runs against.
I’d like to see this for other tools too. I know turbo knows what the root of my repository is, this would make tools and reusabilty more efficient. It’s be even better if I could pass arbitrary arguments to actions like that so commands know where they are operating
It gets tiresome having to copy configuration and what not. Just less files to manage and less places to have to update things overtime.
It’s almost like you need gulp tasks again or something like that and the whole idea is to get away from task runners right? Yet I still need to manage my per package configs and pipelines on top of turbo
Turbo still forces you to have to think about so many other parts of the pipeline. At least nx has plugins but it’s support outside of their happy path for customizations is horrid, but turbo just lacks any thought to this problem of re usability on a task basis.
I don’t want to litter the package.json for each packages with every dependency they need to operate actions be it formatting, minting, etc because that means I have to check them all for different versions, make sure new packages have the same layout etc. A task runner like turbo should be aware enough to know how to do segmented task running by spawning processes for each package without needing a copy of whatever tool installed for each package
I’d like to see this for other tools too. I know turbo knows what the root of my repository is, this would make tools and reusabilty more efficient. It’s be even better if I could pass arbitrary arguments to actions like that so commands know where they are operating
It gets tiresome having to copy configuration and what not. Just less files to manage and less places to have to update things overtime.
It’s almost like you need gulp tasks again or something like that and the whole idea is to get away from task runners right? Yet I still need to manage my per package configs and pipelines on top of turbo
Turbo still forces you to have to think about so many other parts of the pipeline. At least nx has plugins but it’s support outside of their happy path for customizations is horrid, but turbo just lacks any thought to this problem of re usability on a task basis.
I don’t want to litter the package.json for each packages with every dependency they need to operate actions be it formatting, minting, etc because that means I have to check them all for different versions, make sure new packages have the same layout etc. A task runner like turbo should be aware enough to know how to do segmented task running by spawning processes for each package without needing a copy of whatever tool installed for each package
> Turborepo is a high-performance build system for JavaScript and TypeScript codebases.
> Vercel, the well-funded front-end development platform from the team behind Next.js, today announced that it has acquired Turborepo, a high-performance build system for JavaScript and TypeScript monorepos, an increasingly popular way for organizing source code into a single repository that includes all of the necessary packages to build an application.
> Vercel, the well-funded front-end development platform from the team behind Next.js, today announced that it has acquired Turborepo, a high-performance build system for JavaScript and TypeScript monorepos, an increasingly popular way for organizing source code into a single repository that includes all of the necessary packages to build an application.
What does it mean that Vercel has "acquired" Turborepo?
Isn't Turborepo open source? How do you acquire open-source?
Isn't Turborepo open source? How do you acquire open-source?
>How do you acquire open-source?
You buy the company that develops it (or the main fork at least) and provides support for it.
You buy the company that develops it (or the main fork at least) and provides support for it.
Im pretty sure the acquisition was mentioned already a few months back
I'm working with some fairly large monorepos and compile times. To be completely honest, JS is the least of our concerns when it comes to build time, things are pretty snappy when compared to other code bases like C, and Java even with tools like Bazel.
While we're initially focused on JavaScript and TypeScript codebases, you can already use Turborepo with any language as long as you define tasks through `package.json` scripts and use npm/pnpm/yarn workspaces. Turborepo is written in Go and uses `turbo` to build `turbo` in its own monorepo[1].
We're discussing more native support for other languages. It would likely be a commitment to a small subset of popular languages (e.g. Rust, Go, C++, Python) while still maintaining our goal of nearly zero configuration.
[1]: https://github.com/vercel/turborepo/blob/main/turbo.json
We're discussing more native support for other languages. It would likely be a commitment to a small subset of popular languages (e.g. Rust, Go, C++, Python) while still maintaining our goal of nearly zero configuration.
[1]: https://github.com/vercel/turborepo/blob/main/turbo.json
This was my hunch, but it'd be awesome to make that a little more clear on the homepage! Especially if you can link to an FAQ with more detail on the tradeoffs.
When considering adopting a tool like this vs something more general-purpose like Bazel, I'd like to have a rough idea of how screwed I'll be when some non-JS/TS code gets added in the future.
When considering adopting a tool like this vs something more general-purpose like Bazel, I'd like to have a rough idea of how screwed I'll be when some non-JS/TS code gets added in the future.
Please consider .net / msbuild support too :)
Turborepo has been a welcome addition to the JavaScript tooling sphere.
Funnily enough, at nearly the same time as the initial version of Turbo became available, I had just about finished building my own monorepo tool for similar reasons (contemporary tools' inadequacies, etc). Haven't had a reason to switch personally, but I'm glad there are finally fast, small tools with good defaults (and minimal required configuration if any).
Funnily enough, at nearly the same time as the initial version of Turbo became available, I had just about finished building my own monorepo tool for similar reasons (contemporary tools' inadequacies, etc). Haven't had a reason to switch personally, but I'm glad there are finally fast, small tools with good defaults (and minimal required configuration if any).
Is there anything in the works to make this more feature fleshed like `nx` with plugins?
`nx` had alot of potential, but it falls really really short for non happy path projects when I tried integrating (this is as of 15 days ago) into a relatively simple monorepo. I really disagree with its plugin interface and the idea of separating your tests from your src folders (I don't want to litter a folder `e2e` every time I need to write some tests, just feels wrong. I also then have to publicly export certain things for testing if I'm doing it right)
In particular, I feel like using go templates to do something like `plop` does would be super nice.
`nx` had alot of potential, but it falls really really short for non happy path projects when I tried integrating (this is as of 15 days ago) into a relatively simple monorepo. I really disagree with its plugin interface and the idea of separating your tests from your src folders (I don't want to litter a folder `e2e` every time I need to write some tests, just feels wrong. I also then have to publicly export certain things for testing if I'm doing it right)
In particular, I feel like using go templates to do something like `plop` does would be super nice.
Turborepo is really good, but there are a few things I like to see implemented:
* tasks on the root package (e.g. tsc -b that typechecks all packages)
* treat tasks such as lint:eslint, lint:pretter as a single task lint (or maybe lint:*)
These are probably niche features though.
* tasks on the root package (e.g. tsc -b that typechecks all packages)
* treat tasks such as lint:eslint, lint:pretter as a single task lint (or maybe lint:*)
These are probably niche features though.
Turborepo author here...
> * tasks on the root package (e.g. tsc -b that typechecks all packages)
We are working on this as we speak! The first step is to add the ability to restrict hashing `inputs`[1] to the Turborepo `pipeline`. After that we are going to be adding root task running in the next minor release.
However, as your monorepo grows, you will likely want to move away from running tasks like tsc from the root and instead run them on a per-package basis. The reason is that tools like Bazel, Buck, Turborepo, etc. can become more incremental (and thus faster) as your dependency/task graph becomes more granular (as long as you maintain or reduce the average affected blast radius of a given change). The other argument against root tasks is that they break hermeticity and encapsulation of the package abstraction. That being said, root tasks are very useful for fast migration to Turborepo and also for smaller repos. Futhermore, we're happy to tradeoff academic purity for productivity with features like this.
> treat tasks such as lint:eslint, lint:pretter as a single task lint (or maybe `lint:*`)
You can run multiple tasks at the same time and Turborepo will efficiently schedule them at max concurrency.
turbo run eslint prettier --filter=@acme/...
However, it sounds like you like to see glob fan out of tasks. This is a really cool idea. I created a GitHub issue for it here [2] if you'd like to follow along.
[1]: https://github.com/vercel/turborepo/pull/951
[2]: https://github.com/vercel/turborepo/issues/1029.
> * tasks on the root package (e.g. tsc -b that typechecks all packages)
We are working on this as we speak! The first step is to add the ability to restrict hashing `inputs`[1] to the Turborepo `pipeline`. After that we are going to be adding root task running in the next minor release.
However, as your monorepo grows, you will likely want to move away from running tasks like tsc from the root and instead run them on a per-package basis. The reason is that tools like Bazel, Buck, Turborepo, etc. can become more incremental (and thus faster) as your dependency/task graph becomes more granular (as long as you maintain or reduce the average affected blast radius of a given change). The other argument against root tasks is that they break hermeticity and encapsulation of the package abstraction. That being said, root tasks are very useful for fast migration to Turborepo and also for smaller repos. Futhermore, we're happy to tradeoff academic purity for productivity with features like this.
> treat tasks such as lint:eslint, lint:pretter as a single task lint (or maybe `lint:*`)
You can run multiple tasks at the same time and Turborepo will efficiently schedule them at max concurrency.
turbo run eslint prettier --filter=@acme/...
However, it sounds like you like to see glob fan out of tasks. This is a really cool idea. I created a GitHub issue for it here [2] if you'd like to follow along.
[1]: https://github.com/vercel/turborepo/pull/951
[2]: https://github.com/vercel/turborepo/issues/1029.
> We are working on this as we speak!
Perfect!
> However, it sounds like you like to see glob fan out of tasks.
Yes, the idea here being that I don't want to list all similar tasks (such as linting) explicitly in the turborepo config. Teams should be free to add any additional lint task if they think it's useful for them (and possibly only for them). Similar to Maven(Java) where additional goals can be bound to the standard lifecycle phases.
Perfect!
> However, it sounds like you like to see glob fan out of tasks.
Yes, the idea here being that I don't want to list all similar tasks (such as linting) explicitly in the turborepo config. Teams should be free to add any additional lint task if they think it's useful for them (and possibly only for them). Similar to Maven(Java) where additional goals can be bound to the standard lifecycle phases.
I’ve been wanting something like this but there’s a few big things that stick out to me. Why is it configured via json instead of scripting? Declarative task/build tools are just not as nice as something where you can write code. Specifying a task input/output seems very limited and is where so much optimization comes from. The remote cache seems to only be available through some hosted service?
I use Gradle a lot (bazel a little) and would love a tool that’s got an api that’s easier to understand, but this seems like 100 steps backwards without a good path forward.
I use Gradle a lot (bazel a little) and would love a tool that’s got an api that’s easier to understand, but this seems like 100 steps backwards without a good path forward.
You can self-host your remote cache if you'd like. For example, with GitHub Actions. We're working on better docs for this.
https://turborepo.org/docs/features/remote-caching#custom-re...
https://turborepo.org/docs/features/remote-caching#custom-re...
The main page says it's for JavaScript and TypeScript codebases. Will it support other languages?
We're discussing more native support for other languages. It would likely be a commitment to a small subset of popular languages (e.g. Rust, Go, C++, Python) while still maintaining our goal of nearly zero configuration.
https://news.ycombinator.com/item?id=30959607
https://news.ycombinator.com/item?id=30959607
From what I've seen, the scope of this project is to compete directly against the likes of Rush and Lerna. I spoke to the creator last year and my impression was that general language support akin to Bazel's was not a primary/immediate goal.
Your best bet for monorepo building across a variety of languages is Nix. If money is no object you can also make do with Bazel.
Curious as to why "if money is no object" regarding Bazel? Where are the hidden costs? Appreciate it if you could elaborate
I haven't used it before but i'm considering Bazel, Please.Build or Nix for our monorepo.
I haven't used it before but i'm considering Bazel, Please.Build or Nix for our monorepo.
The main thing to be aware of with Bazel is its sandboxing philosophy. Instead of using the files in place, it basically creates a copy of the relevant file structure for any given build (i.e. minus the files that might exist on a given folder but be irrelevant for a build). Because of this, it generally requires strict hermeticity guarantees from everything you're doing, so many things have to be done in a "Bazel way" (i.e. there are specific ways in which you're supposed to download dependencies and there are restrictions in terms of file creation in e.g. tests that might deal w/ I/O)
The way that Bazel approaches flexibility is by being programmable (via a Python-like language called Starlark). There are open source rulesets for various languages that you can use to get off the ground faster, but they are usually fairly opinionated, so if you want to deviate from the happy path, you're going to hit face first into a fairly steep learning curve about depsets, graphs, file/symlink permissions and the like.
Another "weakness" of Bazel is that it's not aware of ultra granular file-to-file dependencies out of the box (e.g. how Golang imports work vs Java's), so either you have to manage dependency graphs manually or you invest in wrapper tooling like gazelle to automate some of the BUILD file generation for you.
The way that Bazel approaches flexibility is by being programmable (via a Python-like language called Starlark). There are open source rulesets for various languages that you can use to get off the ground faster, but they are usually fairly opinionated, so if you want to deviate from the happy path, you're going to hit face first into a fairly steep learning curve about depsets, graphs, file/symlink permissions and the like.
Another "weakness" of Bazel is that it's not aware of ultra granular file-to-file dependencies out of the box (e.g. how Golang imports work vs Java's), so either you have to manage dependency graphs manually or you invest in wrapper tooling like gazelle to automate some of the BUILD file generation for you.
If you're doing something new it's pretty straightforward to start with bazel and do everything from the start 'the bazel way'. If you're pulling in an existing codebase and a lot of dependencies along with it, you could be spending a lot of time to make it work 'the bazel way'. There's a very narrow path that bazel forces you to be on--this is what makes it work so well but make it difficult to stray off that path.
The primary difference is that in Nix it's fairly easy to wrap arbitrary build systems, so you can inject third-party dependencies into your codebase without vendoring and adapting their build system (plus, through nixpkgs[0], the majority of all relevant software has already been wrapped).
In Bazel you have to do a lot of that work yourself. Google has essentially ~unlimited manpower to do this for the things in their third_party tree, but for any other size of organisation this is not the case and people resort to all sorts of ugly hacks.
Depending on your needs these different types of projects can also coexist with each other in Nix. To use an example from my work, we have a Nix-native build system for Go[1] and code using this[2] co-exists with code that just uses the standard Go build system[3]. Both methods end up being addressable on the same abstraction level, meaning that they both turn into equivalent build targets (you can see both in a full build[4] of our repo).
And for what it's worth, some of the things Bazel gets extremely right (such as having a straightforward mapping from code location to build target, universal build command) are pretty easy to do in Nix (see readTree[5], magrathea[6]).
[0]: https://github.com/NixOS/nixpkgs
[1]: https://code.tvl.fyi/about/nix/buildGo
[2]: https://cs.tvl.fyi/depot/-/blob/ops/besadii/default.nix
[3]: https://cs.tvl.fyi/depot/-/blob/third_party/gerrit-queue/def...
[4]: https://buildkite.com/tvl/depot/builds/13067
[5]: https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md
[6]: https://cs.tvl.fyi/depot@f0e6d3498d6c0f905977ea9432c311b6808...
In Bazel you have to do a lot of that work yourself. Google has essentially ~unlimited manpower to do this for the things in their third_party tree, but for any other size of organisation this is not the case and people resort to all sorts of ugly hacks.
Depending on your needs these different types of projects can also coexist with each other in Nix. To use an example from my work, we have a Nix-native build system for Go[1] and code using this[2] co-exists with code that just uses the standard Go build system[3]. Both methods end up being addressable on the same abstraction level, meaning that they both turn into equivalent build targets (you can see both in a full build[4] of our repo).
And for what it's worth, some of the things Bazel gets extremely right (such as having a straightforward mapping from code location to build target, universal build command) are pretty easy to do in Nix (see readTree[5], magrathea[6]).
[0]: https://github.com/NixOS/nixpkgs
[1]: https://code.tvl.fyi/about/nix/buildGo
[2]: https://cs.tvl.fyi/depot/-/blob/ops/besadii/default.nix
[3]: https://cs.tvl.fyi/depot/-/blob/third_party/gerrit-queue/def...
[4]: https://buildkite.com/tvl/depot/builds/13067
[5]: https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md
[6]: https://cs.tvl.fyi/depot@f0e6d3498d6c0f905977ea9432c311b6808...
From what I've seen, Nix[1] is exciting and has some innovative ideas, some of which we're learning from with Turborepo. I believe Replit uses it[2].
[1]: https://nixos.org/
[2]: https://docs.replit.com/programming-ide/getting-started-nix
[1]: https://nixos.org/
[2]: https://docs.replit.com/programming-ide/getting-started-nix
I think you can benefit from using Nix and Bazel together.[1]
[1]: https://www.tweag.io/blog/2018-03-15-bazel-nix/
[1]: https://www.tweag.io/blog/2018-03-15-bazel-nix/
Personally I don't think this is a good idea. Yes, you can make it work if you for some reason have to use both tools. However, both Nix and Bazel are tools that expect to "control the entire world" and whichever way you decide to layer that onion will contain some amount of pain (not to mention that both are very complex tools and you're adding significant onboarding cost by using both).
Probably not, or it would become a Make competitor
Am I in the minority in thinking that we should stop trying to optimize for huge monorepos? The problems created by them seem far worse and more expensive than the problems they purportedly solve.
We now have a healthy competition between several JS domain-specific build tools bit.dev, Nx, Turborepo, and Rush. This is in addition to plugins to general purpose monorepo tooling like rules_nodejs (Bazel). I'm looking forward to the seeing the new ideas that come out from the community as this field matures.
However, I feel a bit sad at the state of general purpose build tools (Bazel/Pants/Buck, make, GitHub Actions, Maven, Nix) or cloud-based IDEs (Cloud9, Codespaces). These tools come off as too complex to operate and build on top of such that we, the JS community, seem to be choosing to build JS-specific tooling from scratch instead. There are a huge number of mostly non-JS-specific problems that monorepo tooling eventually needs to solve: distributed build artifact and test result caching, distributed action execution, sandboxing, resource management and queuing, observability, and integration with other CI tools to name a few. I wish somehow we could reorganize around a smaller set of primitives instead of what appears to be reinventing the wheel.
Regardless, I think all of this effort and attention has lent credence to the monorepo thesis, and I'm very excited to see what's next.