Monorepos: Please Don't (2019)(medium.com)
medium.com
Monorepos: Please Don't (2019)
https://medium.com/@mattklein123/monorepos-please-dont-e9a279be011b
43 comments
I've been saying "megarepo" because I tend to split things into a few large groups and then have dedicated repos for places it makes sense
I will start saying this, too. :)
Instead of e.g., "blah blah blah a monorepo — not necessarily a literal monorepo in terms of having only one repo with every single project in it, but a repo in which several related projects are colocated, and that uses some kind of monorepo tooling like Nx to avoid redundant work — blah blah blah".
Instead of e.g., "blah blah blah a monorepo — not necessarily a literal monorepo in terms of having only one repo with every single project in it, but a repo in which several related projects are colocated, and that uses some kind of monorepo tooling like Nx to avoid redundant work — blah blah blah".
I'd love to learn more about anyone using mono repos for small to medium projects.
This article is also 5 ish years old and I would hope some of the tooling for bigger orgs has started to make it downstream since then.
As parent comment mentioned, scaling welcomes teams to work on tooling.
A mix of Gitlab overall with some Gitolite to manage access to certain branches of the repo if needed seem to be one way to go.
This article is also 5 ish years old and I would hope some of the tooling for bigger orgs has started to make it downstream since then.
As parent comment mentioned, scaling welcomes teams to work on tooling.
A mix of Gitlab overall with some Gitolite to manage access to certain branches of the repo if needed seem to be one way to go.
You can npm install repositories, no need to merge frontend and backend just to share code. Break out what can be shared into utilities and you can share it in completely unrelated projects as well.
I get monorepos for SSR and the likes but why build and deploy an entire container to push a line of CSS?
I get monorepos for SSR and the likes but why build and deploy an entire container to push a line of CSS?
Polyrepo ends up with a mess of version management and flood of PRs to roll out minor changes
Imagine adding a field to a table that must be reflected in the front end and some other micro service consumers. Coordination can be painful.
Each system has its pros and cons, you're going to need tooling either way. I've moved to mono or mega repo as the default and break out smaller pieces into dedicated repo as needed
Imagine adding a field to a table that must be reflected in the front end and some other micro service consumers. Coordination can be painful.
Each system has its pros and cons, you're going to need tooling either way. I've moved to mono or mega repo as the default and break out smaller pieces into dedicated repo as needed
You're going to have to edit the code in 2 places either way. With a sane deployment strategy the backend would have the extra field, then you can deploy the frontend to render said change. Or sometimes the table is dynamic and doesn't even need changes on the frontend.
To me the ability to push changes independent of other pieces far outweighs the extra git commit and push. I've never had the need for any other tooling than a container to run in, git to manage code, and CI/CD for deploys. If there's a need for special tooling to manage the projects there's to much complexity for my taste and I remove the complexity instead.
To me the ability to push changes independent of other pieces far outweighs the extra git commit and push. I've never had the need for any other tooling than a container to run in, git to manage code, and CI/CD for deploys. If there's a need for special tooling to manage the projects there's to much complexity for my taste and I remove the complexity instead.
I get the feeling what you call a monorepo is microscopic compared to what the article is discussing.
Your monorepo sometimes approach is the polyrepo approach.
Your monorepo sometimes approach is the polyrepo approach.
At work, I pushed for a monorepo. We deploy everything as docker containers, both the backend, the frontend and the services. Each service imports and compiles what it needs, nothing more.
We could look to deploy CSS in a different way to optimize the use case for updating a single CSS line, and deoptimize everything else.
We could look to deploy CSS in a different way to optimize the use case for updating a single CSS line, and deoptimize everything else.
I don’t understand why deployment approach is related to mono vs multi repo?
Read the comment I'm replying to and you'll see the context, a single deploy for changes on backend and frontend is mentioned as a benefit. I'm suggesting deploying in order so a frontend doesn't expect a field that doesn't yet exist on the backend.
Nothing in any of these comments suggests you need to deploy in any particular order, or that you need to group any particular code together in order to deploy it.
Monorepos make sharing code easier. They don’t put any requirements on deployment. You can make good or bad decisions regarding deployment when using monorepos or multiple repos.
Monorepos make sharing code easier. They don’t put any requirements on deployment. You can make good or bad decisions regarding deployment when using monorepos or multiple repos.
How do you share config files, protobuf schema, etc. between frontend and backend if you have different repos for each?
I have only worked in large codebases with monorepo, so I'm curious.
I have only worked in large codebases with monorepo, so I'm curious.
You build a package and publish it to a private registry, so builds can reference specific versions of dependencies from other repos.
We use a jenkins job that watches the “source of truth” repo, which in our case is the backend.
Jenkins creates all the protobuf definitions and pushes them to their dependent repos.
A better way would be with a package repository for them, and having it versioned; and running multiple versions of the backend.
Jenkins creates all the protobuf definitions and pushes them to their dependent repos.
A better way would be with a package repository for them, and having it versioned; and running multiple versions of the backend.
I think of org-wide monorepos the same way I think of zealous microservice use, or microfrontends, or platform engineering, etc. It is a "shiny thing" that FAANGs use, which get a disproportionate amount of tech evangelist coverage, and then transition into becoming the perceived "best-practice" by bored devs (myself included) to then create some tech-debt work.
The reality is that these are somewhat niche usecases for very high scale orgs and/or very high-scale developer workforces which can afford to reinvent the wheel. The way the vast majority of companies should go about developing their systems is not new at all, its just to pragmatically apply tried and tested technologies, and transition to new shiny things where there is a demonstrable need.
Worth saying that I'm definitely a fan of team-scoped monorepos though. Being able to automate building and testing across all my apps, update dependencies across all apps, and deploy with a single merge/pull request is great.
The reality is that these are somewhat niche usecases for very high scale orgs and/or very high-scale developer workforces which can afford to reinvent the wheel. The way the vast majority of companies should go about developing their systems is not new at all, its just to pragmatically apply tried and tested technologies, and transition to new shiny things where there is a demonstrable need.
Worth saying that I'm definitely a fan of team-scoped monorepos though. Being able to automate building and testing across all my apps, update dependencies across all apps, and deploy with a single merge/pull request is great.
TFA assumes several things which may or may not hold for a particular company or team.
For example, the Linux kernel works with its tens of millions of lines just fine, even though ownership is heavily distributed among many subsystem maintainers. Checking out a branch or pulling in changes rarely takes more than a few seconds.
On the other hand, some organizations abuse their VCS, using it as a repository for large immutable files such as build artifacts, software packages or similar non-source archives. Orgs who let this happen are of course going to have a hard time. But that's hardly the monorepo's fault.
For example, the Linux kernel works with its tens of millions of lines just fine, even though ownership is heavily distributed among many subsystem maintainers. Checking out a branch or pulling in changes rarely takes more than a few seconds.
On the other hand, some organizations abuse their VCS, using it as a repository for large immutable files such as build artifacts, software packages or similar non-source archives. Orgs who let this happen are of course going to have a hard time. But that's hardly the monorepo's fault.
Having seen a large org attempt a poly -> mono repo conversion project (which took almost 2 years and failed and was in the process of being rolled back when I left) I agree with everything in this article
Once an org becomes large enough you need lots of custom tooling to make working across a huge codebase smoother, and that tooling is similar for mono and poly repos, but you need an additional metric ton of tooling to make monorepos work
And please pray you didn't decide to go with bazel for builds
My least favourite monorepo experience: I want to update my dependency for my tiny service for a security patch -> Oh, 400 other services depend on it and dozens of tests break when I try to update it -> this isn't worth my time I'll do something else instead
We were promised a world where every dependency was kept up to date by necessity but ended up in a world where all dependencies atrophied due to the increased difficulty to update them
Once an org becomes large enough you need lots of custom tooling to make working across a huge codebase smoother, and that tooling is similar for mono and poly repos, but you need an additional metric ton of tooling to make monorepos work
And please pray you didn't decide to go with bazel for builds
My least favourite monorepo experience: I want to update my dependency for my tiny service for a security patch -> Oh, 400 other services depend on it and dozens of tests break when I try to update it -> this isn't worth my time I'll do something else instead
We were promised a world where every dependency was kept up to date by necessity but ended up in a world where all dependencies atrophied due to the increased difficulty to update them
Having worked at a place with a monorepo it was one of the things I actually really liked about working there. It took a lot more tooling but I found it to be much better than a multi-repo would have been, even with a similar amount of tooling. I didn't have to be checking different projects out all the time and changes across projects weren't too bad (large scales ones still took a lot of managing but not as much as changes across a multi-repo).
I was always hoping that one of the big cloud providers would offer a monorepo and invest a lot in making tooling for it to actually be usable.
I was always hoping that one of the big cloud providers would offer a monorepo and invest a lot in making tooling for it to actually be usable.
> I want to update my dependency for my tiny service for a security patch
If the API changed it's not a security patch... Why would a change that only fixes a security bug cause tests to fail?
Sounds like this is more a matter of needing to cherry-pick temporarily and then actually pushing for the codebase to update, probably by the security team.
Sometimes you need to work with other people, that might necessitate doing "ugly" things to get the job done.
If the API changed it's not a security patch... Why would a change that only fixes a security bug cause tests to fail?
Sounds like this is more a matter of needing to cherry-pick temporarily and then actually pushing for the codebase to update, probably by the security team.
Sometimes you need to work with other people, that might necessitate doing "ugly" things to get the job done.
Yes, but very often you are trucking along just fine with some version of a dependency and then all of a sudden it gets a CVE and the fix has only been applied to the next major version and not backported because the version you are on is no longer supported. And now you are in dependency update cascading hell.
This actually explains why Google is so keen to retire products that don't necessarily cost much to run, but don't turn a profit yet.
> And please pray you didn't decide to go with bazel for builds
Why not? And what should one use instead?
Why not? And what should one use instead?
I get what the article is saying. But if `ls -d ../*/.git/` prints the names of all the company's repos, and the same happens in a few coworker's computers, I start to question things.
I worked at 4 other companies before Google, and Google's source code tooling was the best I'd ever seen and definitely something the rest of the world can work towards.
In particular, it fostered a culture where accepting code changes from outside teams was normal (though obviously not always uncontentious), and prototyping those changes was pretty easy.
I've been using pantsbuild.org since I left and I think it hits a good sweet spot between blaze, which has a lot of overhead, and the rest of the world.
In particular, it fostered a culture where accepting code changes from outside teams was normal (though obviously not always uncontentious), and prototyping those changes was pretty easy.
I've been using pantsbuild.org since I left and I think it hits a good sweet spot between blaze, which has a lot of overhead, and the rest of the world.
I’ve heard about how huge companies like google are using a giant monorepo, and I honestly don’t understand how that could work. But I guess it does…
It's sometimes important to remind ourselves that just because we can't see understanding in something doesn't mean there isn't understanding in it, and it might take effort on our part.
It's not a negative thing as much as an opportunity to grow if we ask ourselves to learn and see it as an opportunity.
It's not a negative thing as much as an opportunity to grow if we ask ourselves to learn and see it as an opportunity.
The company I work at is fairly large, with thousands of engineers, and we use monorepos. It works extremely well. What about them makes you think it can't work?
We use monorepos in my company too, but we use one for each product, and it works really well for that. But for the whole company to be in a single monorepo just seems like it would require a lot of effort to maintain.
We have several projects per monorepo, and they are massive, currently being merged into a single one. Anything specific about maintainability that scares you?
> ...we use monorepos.
MonorepoS, in the plural?
MonorepoS, in the plural?
Yes. We used to have dozens of repos, but started to merge them slowly. Nowadays we have 3 large ones, but the goal is 1.
with lots of controls and custom tooling
Facebook open sourced theirs
https://github.com/facebook/sapling
https://github.com/facebook/sapling
This looks interesting! I'll check it out and maybe it will help me wrap my head around how they handle the complexity.
If you aren't worried about partial checkouts & permissions
The biggest issue is deciding what to build across developers and their branches & products. You don't want to build the world on every commit, but you do want to ensure that you are building what the code would look like post-merge before the merge.
The biggest issue is deciding what to build across developers and their branches & products. You don't want to build the world on every commit, but you do want to ensure that you are building what the code would look like post-merge before the merge.
At a previous gig, they went down the monorepo rabbit hole. For all the reasons the author notes, in not terribly long, we were hearing about the monobuild project, and then the monodeploy project, which was strange, because I think the same people behind the monorepo project had been the ones who advocated for breaking up what had been a monolith service. Plus ça change, I suppose.
Monorepos are most painful when all the components are heavily coupled. Maybe a bunch of unrelated stuff in a single compilation unit (e.g. package) results in a lot of waste. And then people just end up not having clear interfaces and boundaries and ownership.
But I guess that's not a result of a monorepo inherently. It's just the result of poor engineering leadership.
But I guess that's not a result of a monorepo inherently. It's just the result of poor engineering leadership.
I'm in the opposite situation, poly-repo hell.
Our team produces 4 libraries. The code for them along with test apps is spread across 7 repos. So when I want to make a core change (like updating the version of Gradle we use) it results in 7 different PRs that have to go in at the same time, and that everyone has to fetch in sync. It's pretty rare that a change only requires a single PR because of this.
Releases are a mess. To do a release we tag the latest commit in each of the repos with the same release number and use Jenkins to pull that tag from each repo and put it all together. But that's just like a suggestion, and it's not uncommon for a release to be based off of master. Good luck doing the forensics to figure out the state of everything for a release.
The best part is that each repo needs to be in a specific folder on disk because some of them look for the others via hard-coded paths.
There is one repo that needs to be separate, because it's public. But I've been pushing hard to get most of the repos pushed together into a single one just to add some sanity to my PRs.
Our team produces 4 libraries. The code for them along with test apps is spread across 7 repos. So when I want to make a core change (like updating the version of Gradle we use) it results in 7 different PRs that have to go in at the same time, and that everyone has to fetch in sync. It's pretty rare that a change only requires a single PR because of this.
Releases are a mess. To do a release we tag the latest commit in each of the repos with the same release number and use Jenkins to pull that tag from each repo and put it all together. But that's just like a suggestion, and it's not uncommon for a release to be based off of master. Good luck doing the forensics to figure out the state of everything for a release.
The best part is that each repo needs to be in a specific folder on disk because some of them look for the others via hard-coded paths.
There is one repo that needs to be separate, because it's public. But I've been pushing hard to get most of the repos pushed together into a single one just to add some sanity to my PRs.
Keeping prod stable is more important than alpha testing every team's latest commit, but monorepos tend not to accommodate staying on known-good versions of your dependencies and opting into updates when ready.
Medium: Please Don’t (2024)
Moving to monorepo is often a win; also the name sucks, we have a a few "monorepos" at work, and I think it's the sweet spot. Rust and C firmware doesn't need to live with TypeScript frontend apps, but it's pretty wasteful to have apps of the same ecosystem unable to share any and all dependencies and utility code trivially.
(Where "trivially" means literally one line of code
and not one single step more.)