Improving large monorepo performance on GitHub(github.blog)
github.blog
Improving large monorepo performance on GitHub
https://github.blog/2021-03-16-improving-large-monorepo-performance-on-github/
132 comments
You can compute the checksums outside of the lock. You just need to compare them inside the lock.
The key thing here is that prior to the lock if data changes you recompute the checksums. As long as any change outside the lock triggers a recompute of the corresponding checksums and no changes can occur during the lock, there is no race condition.
I imagine that this may result in data getting de-synced/failing the checksum comparisons more often however it's still a net performance increase as long as the aggregate time spent re-syncing the data is less than the extra time spent waiting for checksums in the lock.
The key thing here is that prior to the lock if data changes you recompute the checksums. As long as any change outside the lock triggers a recompute of the corresponding checksums and no changes can occur during the lock, there is no race condition.
I imagine that this may result in data getting de-synced/failing the checksum comparisons more often however it's still a net performance increase as long as the aggregate time spent re-syncing the data is less than the extra time spent waiting for checksums in the lock.
I think the crucial part are the retries, which havent been mentioned.
In general, you cant assume your code will always read the new checksum before entering the critical section, unless you synchronize it. That is, when leaving the critical section you have to make sure that all threads waiting to enter it will recompute the hash.
But, from the entering thread's perspective, you never know when youll be forced to recompute. So you've got to wait on the lock. And recompute when you are woken up.
I'm not stating what they've written is wrong. Just for me it's a bit vague and looks like potential race condition.
In general, you cant assume your code will always read the new checksum before entering the critical section, unless you synchronize it. That is, when leaving the critical section you have to make sure that all threads waiting to enter it will recompute the hash.
But, from the entering thread's perspective, you never know when youll be forced to recompute. So you've got to wait on the lock. And recompute when you are woken up.
I'm not stating what they've written is wrong. Just for me it's a bit vague and looks like potential race condition.
I don't think that would enter a race condition.
1. All the zones start precomputing the checksums using a worker pool. Changes are fed into the pool with the change time stamped. The workers don't store the checksum if a newer checksum is already present.
2. The zones enter the lock/critical section. At this point new changes are no longer added to the work queue and must wait until the lock exits. The worker pool continues to process the queue.
3. The work queues are empty and the checksums are compared between zones. Those that match are "locked in" and those that don't are set aside for the next time the zones enter the lock/critical section.
4. The zones exit the lock, a timer starts, and step 1 starts again.
At no point here would there be a race condition. As long as non-matching changes are pruned and retried in the next cycle, progress will be made. In certain conditions the retries could degrade overall performance but consensus is eventually achieved and the system continues to make progress.
1. All the zones start precomputing the checksums using a worker pool. Changes are fed into the pool with the change time stamped. The workers don't store the checksum if a newer checksum is already present.
2. The zones enter the lock/critical section. At this point new changes are no longer added to the work queue and must wait until the lock exits. The worker pool continues to process the queue.
3. The work queues are empty and the checksums are compared between zones. Those that match are "locked in" and those that don't are set aside for the next time the zones enter the lock/critical section.
4. The zones exit the lock, a timer starts, and step 1 starts again.
At no point here would there be a race condition. As long as non-matching changes are pruned and retried in the next cycle, progress will be made. In certain conditions the retries could degrade overall performance but consensus is eventually achieved and the system continues to make progress.
No, it’s just switching from a pessimistic locking approach to an optimistic one: https://en.wikipedia.org/wiki/Optimistic_concurrency_control
I think its not entirely optimistic locking, because the condition is based on checksum value, which is expensive to compute.
I don't recall the article mentioning it, but if you don't compute the actual checksum under the lock then you either cache it (under the lock) and update only successful writes, or you risk a race condition between time of the check and action.
That is, optimistic locking assumes that if the condition is true I immediately get exclusive lock. But here, condition check is completely separate from the action. When the checksums are equal, but another thread manage to enter and leave the critical section before you, then that invalidates the condition. But you need to synchronously let other threads know they must recompute.
I don't recall the article mentioning it, but if you don't compute the actual checksum under the lock then you either cache it (under the lock) and update only successful writes, or you risk a race condition between time of the check and action.
That is, optimistic locking assumes that if the condition is true I immediately get exclusive lock. But here, condition check is completely separate from the action. When the checksums are equal, but another thread manage to enter and leave the critical section before you, then that invalidates the condition. But you need to synchronously let other threads know they must recompute.
I can bet whatever you want they did this improvement for microsoft/windows repo.
Microsoft/windows is hosted on Azure DevOps, and they have also blogged about what they've done to improve its performance!
Here's a recent post: https://devblogs.microsoft.com/devops/introducing-scalar/
Here's a recent post: https://devblogs.microsoft.com/devops/introducing-scalar/
Rumors are Azure DevOps and GitHub are converging "soon", and maybe "Project Cyclops" wasn't specifically to improve Microsoft/Windows repo performance, but it seems reasonable given the convergence rumors it could be a step in the direction of preparing for/migrating the repo to GitHub. Of course Microsoft doesn't want to panic Enterprise developers on Azure DevOps just yet so they are extremely quiet right now about any convergence efforts, so I take the rumors with a grain of salt. It is something that I wish Microsoft would properly announce sooner rather than later as it might provide momentum towards GitHub in capital-E Enterprise development world (even if will panic those that are still afraid of GitHub for whatever reasons).
Did they contribute the repack optimizations upstream?
Nice work, really interesting blog post!
On a sidenote, git itself can also get painfully slow with large monorepos. Hope GitHub can push some changes there as well.
I know FB moved off git to mercurial because of performance issues.
On a sidenote, git itself can also get painfully slow with large monorepos. Hope GitHub can push some changes there as well.
I know FB moved off git to mercurial because of performance issues.
My understanding is that neither Git nor Mercurial can do this well out of the box, and FB and Google both have their own extensions to Mercurial to make this possible (because even though Mercurial is often slower than Git, it’s extensible)
e.g. https://facebook.github.io/watchman/ - used as part of Facebook’s Mercurial solution, I think.
e.g. https://facebook.github.io/watchman/ - used as part of Facebook’s Mercurial solution, I think.
Git also has a file system monitor interface which can use Watchman. We (GitHub) are working on a native file system monitor implementation in addition - https://github.com/gitgitgadget/git/pull/900.
And then from mercurial extensions to our own server, mononoke, which apparently has been moved under the Eden umbrella: https://github.com/facebookexperimental/eden
I thought Google used some custom fork of perforce.
From what I understand, Piper is not a fork of Perforce, but instead a completely different system with the same interface. You know, built on top of a BigTable or Spanner cluster instead of whatever Perforce uses.
The Mercurial extensions are then an alternative client for Piper.
The Mercurial extensions are then an alternative client for Piper.
It's my understanding that it started as a fork of perforce, Google bought a source license for perforce. Whether you'd still call a fork of perforce anymore is like asking if a Computer is the same computer after you've replaced all the components over the years.
That was the case many years ago (extremely customized perforce server), but after that there was a full rewrite as a distributed system built on Bigtable etc. - https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... is an overview. It wasn't a ship-of-Theseus type situation where they incrementally replaced components.
You might be interested in scalar [1] developed by Microsoft for handling large repos.
[1]: https://github.com/microsoft/scalar
[1]: https://github.com/microsoft/scalar
It's also interesting to note how much of Microsoft's work for handling large repos in git has merged upstream directly into git itself.
One very interesting part of that is the effort that has gone into the git commit-graph: https://git-scm.com/docs/commit-graph.
It's part of what makes scalar interesting compared to some of the projects you hear mentioned used inside the FB and Google gates: not only is scalar itself open source, but a lot of what scalar does is tune configuration flags to turn on optional git features such as the commit-graph, sparse checkout "cones", etc that are all themselves directly supported by the git client. Even if you aren't at the scale where it makes sense to use all of the tools that scalar provides, you can get some interesting baby steps by following scalar's "advice" on git configuration.
One very interesting part of that is the effort that has gone into the git commit-graph: https://git-scm.com/docs/commit-graph.
It's part of what makes scalar interesting compared to some of the projects you hear mentioned used inside the FB and Google gates: not only is scalar itself open source, but a lot of what scalar does is tune configuration flags to turn on optional git features such as the commit-graph, sparse checkout "cones", etc that are all themselves directly supported by the git client. Even if you aren't at the scale where it makes sense to use all of the tools that scalar provides, you can get some interesting baby steps by following scalar's "advice" on git configuration.
If you are willing to adapt to a different structure and workflow, you can filter the scope of git down dramatically with sparse checkouts (as @WorldMaker also mentioned).
https://github.blog/2020-01-17-bring-your-monorepo-down-to-s...
https://github.blog/2020-01-17-bring-your-monorepo-down-to-s...
Sparse-checkouts are amazing. I wrote some small tools that use dependency information in Flutter packages to drive a sparse-checkout. We use it at $dayjob now.
Google's approach to that is to make submodules better, so you can have a "superproject" which has submodules of multiple repos that looks like a monorepo. If you look at https://android.googlesource.com/ all the projects named as "superproject" are for this purpose.
Or at least that was the plan at 2016 when I left Google.
Or at least that was the plan at 2016 when I left Google.
Offtopic, but I often wonder if there are people using `git worktree` to have several related code trees within the same repo.
Technically it works the mostly the same as multiple repos, but theoretically allows to have something like a bootstrap script with everything self contained in the same repos. Looks like an alternative tradeoff between a monorepos with shared history and multiple repositories.
Technically it works the mostly the same as multiple repos, but theoretically allows to have something like a bootstrap script with everything self contained in the same repos. Looks like an alternative tradeoff between a monorepos with shared history and multiple repositories.
I tried that, and discovered that it won't let you have two worktrees with the same branch checked out?
Offtopic (I think), but I just learned that "For instance, git worktree add -d <path> creates a new working tree with a detached HEAD at the same commit as the current branch." (from the manpage).
It's offtopic because the second worktree has a detached HEAD, so that doesn't help in the case you mention.
It's offtopic because the second worktree has a detached HEAD, so that doesn't help in the case you mention.
Which is very sane - what should git do if you modify the branch in one tree but not in the other? The least painful solution would require something like multiple-refs-per-remote-branch, which would be (to my understanding) a re-architecture.
Well, I wanted the same effect as two separate checkouts (ie entirely separate branch structures) but with a bit of the disk space shared between them, but that's not how it works.
You can use “git clone” for that; it will set up an alternates link in the local clone, so you only store the git object data in the origin but apply deltas to the clone.
Do you have an example of this setup?
As far as I know git worktree is just to have different branches of the same repo checked out in different locations. At least, that's the only way I use it (and it's great!). Are you suggesting to have different projects on different branches? So an empty "master", then "project1", "project2" etc. as branches?
As far as I know git worktree is just to have different branches of the same repo checked out in different locations. At least, that's the only way I use it (and it's great!). Are you suggesting to have different projects on different branches? So an empty "master", then "project1", "project2" etc. as branches?
I'm not sure worktree works exactly as you think.
I use worktree locally so that, for example, I can have my working copy that I am doing development in and then a separate working copy where I can do code review for someone else, without having to interrupt what I am doing in my own worktree.
My own experience is that if you are using branches with radically different content for different purposes in the same tree, it's going to end up a mess at some point. Worktrees, as far as I am aware, do not help with that in any special way.
I use worktree locally so that, for example, I can have my working copy that I am doing development in and then a separate working copy where I can do code review for someone else, without having to interrupt what I am doing in my own worktree.
My own experience is that if you are using branches with radically different content for different purposes in the same tree, it's going to end up a mess at some point. Worktrees, as far as I am aware, do not help with that in any special way.
Are you thinking of git subtree?
https://www.atlassian.com/git/tutorials/git-subtree
https://www.atlassian.com/git/tutorials/git-subtree
I'm slightly surprised that GitHub is still basically storing a git repo on a regular filesystem using the Git CLI. I would have expected that the repos were broken up into individual objects and stored in an object store. This should make pushes much faster as you have basically infinitely scalable writes. However it does make pulls more difficult. However computing packfiles could still be done (asynchronously) and with some attention to data-locality it should be possible.
This would be a huge rewrite of some internals but seems like it would be a lot easier to manage. It would also provide a some benefits as objects could be shared between repos (although some care would probably be necessary for hash collisions) and it would remove some of the oddness about forks (as IIUC they effectively share the repo with the "parent" repo).
I would love to know if something like this has been considered and why the decided against it.
This would be a huge rewrite of some internals but seems like it would be a lot easier to manage. It would also provide a some benefits as objects could be shared between repos (although some care would probably be necessary for hash collisions) and it would remove some of the oddness about forks (as IIUC they effectively share the repo with the "parent" repo).
I would love to know if something like this has been considered and why the decided against it.
Vincent Marti's legendary and funny "My Mom Told Me That git Doesn't Scale" presentation explains why GitHub chose filesystems instead of something fancier. https://www.youtube.com/watch?v=Ri8hSZNKzu4
Thanks, I hadn't seen that before. I really liked the line "People think we're a Ruby shop, but actually we're a UNIX shop. Everything else is an implementation detail".
> This should make pushes much faster as you have basically infinitely scalable writes. However it does make pulls more difficult.
I bet GitHub has much more read traffic than write traffic, so this trade-off does not make sense.
I bet GitHub has much more read traffic than write traffic, so this trade-off does not make sense.
Seriously, imagine the compute and requests costs to assemble a large git pull.
Sounds a lot like this here https://github.com/Homebrew/brew/pull/9383
I said "difficult" not expensive. Once you assembled the packfiles (much like they do today) it should be roughly the same cost.
Yes and I would also imagine that the trade-offs between writing a proprietary object storage and reusing the battle-tested object storage that everyone else uses would have been considered as well.
It seems like the sort of thing that would be an interesting open source research topic if you could build an object database for git that performs better than its packed in filesystem object store. But it's probably not something you want to do as a proprietary project with fewer eyeballs on its performance trade-offs and more engineering work every time git slightly changes its object storage behavior which would remain tuned for the filesystem object store because it was entirely unaware of your efforts.
It seems like the sort of thing that would be an interesting open source research topic if you could build an object database for git that performs better than its packed in filesystem object store. But it's probably not something you want to do as a proprietary project with fewer eyeballs on its performance trade-offs and more engineering work every time git slightly changes its object storage behavior which would remain tuned for the filesystem object store because it was entirely unaware of your efforts.
Have you ever tried it? It's not remotely performant and wouldn't make sense since GH is read heavy. Plus I'm sure they spend a lot of time thinking about this stuff, no?
If you want to get your feet wet, check out go-git[1]. It's a native golang implementation of git. They have a storage layer abstracted over a lean interface that you quickly create alternative drivers for in golang. You'll be effectively implementing poorly sharded file system on a database, then it becomes obvious why scaling the FS is just easier.
[1] https://github.com/go-git/go-git/tree/master/storage
If you want to get your feet wet, check out go-git[1]. It's a native golang implementation of git. They have a storage layer abstracted over a lean interface that you quickly create alternative drivers for in golang. You'll be effectively implementing poorly sharded file system on a database, then it becomes obvious why scaling the FS is just easier.
[1] https://github.com/go-git/go-git/tree/master/storage
> Plus I'm sure they spend a lot of time thinking about this stuff, no?
I think this is unfair - the author was not insinuating that the people who designed this system at Github are stupid in some way, but just asking if other architectures have been considered.
I think this is unfair - the author was not insinuating that the people who designed this system at Github are stupid in some way, but just asking if other architectures have been considered.
> ...Github are stupid in some way, but just asking if other architectures have been considered.
To me, asking an engineering org if they’ve considered alternative architectures for their main engineering problem is silly at best, overconfident at worst.
To me, asking an engineering org if they’ve considered alternative architectures for their main engineering problem is silly at best, overconfident at worst.
I think the main point of the comment was asking why they decided against it. At one point the wording mildly suggests the possibility that no one at github has thought about it:
> I would love to know if something like this has been considered and why the decided against it.
... but that still sounds more like a grammatical hedge than an actual suggestion that github didn't think it through.
imo it's fair to lay out why you're surprised about some decision in the hopes that someone will enlighten you, even if it can be tricky to phrase that without coming off like a "why didn't you just..." comment.
> I would love to know if something like this has been considered and why the decided against it.
... but that still sounds more like a grammatical hedge than an actual suggestion that github didn't think it through.
imo it's fair to lay out why you're surprised about some decision in the hopes that someone will enlighten you, even if it can be tricky to phrase that without coming off like a "why didn't you just..." comment.
Sure, but also sometimes the only way to learn is to ask.
> but just asking if other architectures have been considered.
That’s the polite way of calling them stupids ;)
That’s the polite way of calling them stupids ;)
The problem with recognizing that a lot of phrasings are just the polite way to call someone stupid/tell someone to fuck off/etc is that you start seeing assholes whenever someone is just trying to be polite. :(
They're just expressing curiosity. Jeeze.
He is literally actually insinuating the GitHub team have not architected things correctly.
He didn’t even open with a question, rather he opened with a statement. I don’t think there is even a question mark in the post as it stands right now.
I hate tiptoeing around to avoid offence, but it would have been better to frame the post as a question, it would likely avoided people reading negative intent behind the post.
He didn’t even open with a question, rather he opened with a statement. I don’t think there is even a question mark in the post as it stands right now.
I hate tiptoeing around to avoid offence, but it would have been better to frame the post as a question, it would likely avoided people reading negative intent behind the post.
I always had the impression GitHub was not preemptively investing in the fundamentals like that. So yeah, agree it's bummer but also not surprised.
And hey, at least that means a post GitHub FOSS world won't be leaving fundamental improvements behind!
And hey, at least that means a post GitHub FOSS world won't be leaving fundamental improvements behind!
I am not a github employee, but my 2 cents.
An object store lacks an index which your typical FS will provide with a relatively high degree of efficiency. FS's can be distributed to arbitrary write velocity given an appropriately distributed block storage solution ( which will provide the k/v API of an object store that you're looking for ). Distributed FS's are conveniently compatible with most POSIX operations rather than requiring bespoke integration. Most object stores are optimized for largish objects and lack the ability to condense records into an individual write (via the block API) or pre-emptively pre-fetch the most likely next set of requested blocks.
In the GitHub's case the choice of diverging from GitCLI/FS based storage APIs could lead to long term support issues and an implicit "github" flavor of git rather than improving the core git toolchain.
Object Stores are great, but if you need some form of index they get slow and painful really fast.
An object store lacks an index which your typical FS will provide with a relatively high degree of efficiency. FS's can be distributed to arbitrary write velocity given an appropriately distributed block storage solution ( which will provide the k/v API of an object store that you're looking for ). Distributed FS's are conveniently compatible with most POSIX operations rather than requiring bespoke integration. Most object stores are optimized for largish objects and lack the ability to condense records into an individual write (via the block API) or pre-emptively pre-fetch the most likely next set of requested blocks.
In the GitHub's case the choice of diverging from GitCLI/FS based storage APIs could lead to long term support issues and an implicit "github" flavor of git rather than improving the core git toolchain.
Object Stores are great, but if you need some form of index they get slow and painful really fast.
You should be able to split the object store into 2 systems: 1 metadata (think rdbms/nosql/etc) and a blob-data service, keeping large files, think 10KB+. Both systems should be able to be more efficient than the current method.
Example: you can add erasure coding to the blob-data service for better efficiency. You can add fancy indexing to your metadata store. etc etc.
But somebody has to create it, that's the issue.
Example: you can add erasure coding to the blob-data service for better efficiency. You can add fancy indexing to your metadata store. etc etc.
But somebody has to create it, that's the issue.
That's exactly how distributed filesystems are built.
Systems such as HDFS use the NameNode for this task, but depending on the exact characteristics of the fileSystem a multi-master setup is often used. I know of at least one NFS implementation which uses postgres as its metadata layer.
Systems such as HDFS use the NameNode for this task, but depending on the exact characteristics of the fileSystem a multi-master setup is often used. I know of at least one NFS implementation which uses postgres as its metadata layer.
https://github.com/juicedata/juicefs
has an index implementation and is backed up by object storage.
aye - as it says it provides an FS. The second diagram shows how they chop up files into blocks then store them in the backing object (k/v) storage.
A filesystem really is just an index over k/v storage (blocks). You can find similar diagrams for the implementation of EXT/XFS/GFS/HDFS etc. Like databases there are many tradeoffs and nuances in terms of how these concepts are implemented.
A filesystem really is just an index over k/v storage (blocks). You can find similar diagrams for the implementation of EXT/XFS/GFS/HDFS etc. Like databases there are many tradeoffs and nuances in terms of how these concepts are implemented.
> I'm slightly surprised that GitHub is still basically storing a git repo on a regular filesystem using the Git CLI.
Maybe I'm a bit dense, but how did you get that from the article? I'm fairly certain that in other pieces of writing they showed that they are using an object store, and I'm guessing that's what the "file servers" in the article are.
Maybe I'm a bit dense, but how did you get that from the article? I'm fairly certain that in other pieces of writing they showed that they are using an object store, and I'm guessing that's what the "file servers" in the article are.
`git repack` is an operation that is fairly specific to git's default file format: if they were storing objects in any (other) database, it is very unlikely that they would experience blocking repack operations, as that is an area where databases are highly optimized to execute incrementally.
I think the GitHub folks have written more than one article about this. I'm not sure I can find the one I'm thinking of, but here's another one: https://github.blog/2016-04-05-introducing-dgit/
> Perhaps it’s surprising that GitHub’s repository-storage tier, DGit, is built using the same technologies. Why not a SAN? A distributed file system? Some other magical cloud technology that abstracts away the problem of storing bits durably? The answer is simple: it’s fast and it’s robust.
> Perhaps it’s surprising that GitHub’s repository-storage tier, DGit, is built using the same technologies. Why not a SAN? A distributed file system? Some other magical cloud technology that abstracts away the problem of storing bits durably? The answer is simple: it’s fast and it’s robust.
This kind of comment is why every single project needs to be justified with "What problems are you solving?" and "What usecase are you supporting?". Because I could 150% imagine somebody getting excited about this and then:
1) Framing it as such with poor justification "a lot easier to manage"
2) "This would be a huge rewrite of some internals" Becoming a multi-year migration quagmire
3) The dawning realization that you have used a write-heavy architecture in a read-heavy system
1) Framing it as such with poor justification "a lot easier to manage"
2) "This would be a huge rewrite of some internals" Becoming a multi-year migration quagmire
3) The dawning realization that you have used a write-heavy architecture in a read-heavy system
I actually implemented this one day--storing all of the git objects into a PostgreSQL database, as a modification to the normal C program copy of git (not some go fork or "libgit2" or whatever people seem to expect you have to use)--and it was really easy: git already is designed to do its object access via a simple internal abstraction, so it was a trivial tap.
Why postgres? Wouldn't SQLite be better?
> infinitely scalable writes
Scalable receive, but having to send hundred of thousands of objects one-by-one is very inefficient. The fact that there are hundreds of thousands of servers to receive them does not help you.
(and same problem when you pull)
Scalable receive, but having to send hundred of thousands of objects one-by-one is very inefficient. The fact that there are hundreds of thousands of servers to receive them does not help you.
(and same problem when you pull)
Not to pile on (as the idea isn't necessarily, immediately crazy):
1) The filesystem is a surprisingly decent datastore,
and,
2) Unless your write-volume is unreasonably high for a GH repo, your main problem is making sure your locking (and unlocking) is rock-fucking-solid; Git has built-in locking, but I bet Github has either replaced it or has some serious state-machine stuff going on to manage it outside of Git itself,
also and,
3) Reads on files are ultra-cacheable, obviously, and reads on HTML pages are too, and serving a second-or-two-stale read isn't a problem ~100% of the time in GH's use case, so read volume isn't really a problem.
[EDIT] source: have designed and developed a git-backed, client-accessible product at a much smaller scale than GH, but enough to see where the problems & strengths could/would be, at greater scale.
[EDIT EDIT] oh and with a bunch of clients using a bunch of different repos that effectively never interact except through normal git-push/PR behavior, obviously horizontal scaling is a breeze and a half. All you need is a little routing to figure out which repo is where. Git itself, plus your locking solution (whatever that may be), give you the tools you need to migrate from one server to another if you need to move a repo, probably with an unnoticeable amount of downtime (=locked repo) if you're clever about it.
1) The filesystem is a surprisingly decent datastore,
and,
2) Unless your write-volume is unreasonably high for a GH repo, your main problem is making sure your locking (and unlocking) is rock-fucking-solid; Git has built-in locking, but I bet Github has either replaced it or has some serious state-machine stuff going on to manage it outside of Git itself,
also and,
3) Reads on files are ultra-cacheable, obviously, and reads on HTML pages are too, and serving a second-or-two-stale read isn't a problem ~100% of the time in GH's use case, so read volume isn't really a problem.
[EDIT] source: have designed and developed a git-backed, client-accessible product at a much smaller scale than GH, but enough to see where the problems & strengths could/would be, at greater scale.
[EDIT EDIT] oh and with a bunch of clients using a bunch of different repos that effectively never interact except through normal git-push/PR behavior, obviously horizontal scaling is a breeze and a half. All you need is a little routing to figure out which repo is where. Git itself, plus your locking solution (whatever that may be), give you the tools you need to migrate from one server to another if you need to move a repo, probably with an unnoticeable amount of downtime (=locked repo) if you're clever about it.
When is GitHub going to finally add support for Microsoft’s VFSforGit?
https://github.com/microsoft/VFSForGit
https://vfsforgit.org/
https://github.com/microsoft/VFSForGit
https://vfsforgit.org/
I'm not sure that will ever happen [1] as Microsoft itself is limiting active development of VFS for Git in favor of Scalar [2] by the same team, which aims to improve client-side big repo performance without having to use OS-level file system virtualization hooks.
I don't believe VFS for Git will ever be abandoned by Microsoft, but I'm doubtful it will ever get any more major improvements from them.
Scalar does use the VFS for Git client-server protocol, and both Scalar and VFS for Git rely on the same improvements to the git app itself, so I could imagine that GitHub would adopt the GVFS protocol and support Scalar without formally supporting GVFS itself.
[1] GitHub did announce future GVFS support in 2017 - https://venturebeat.com/2017/11/15/github-adopts-microsofts-... - but if anything came out of that I don't see it in GitHub help today.
[2] https://github.com/microsoft/scalar
I don't believe VFS for Git will ever be abandoned by Microsoft, but I'm doubtful it will ever get any more major improvements from them.
Scalar does use the VFS for Git client-server protocol, and both Scalar and VFS for Git rely on the same improvements to the git app itself, so I could imagine that GitHub would adopt the GVFS protocol and support Scalar without formally supporting GVFS itself.
[1] GitHub did announce future GVFS support in 2017 - https://venturebeat.com/2017/11/15/github-adopts-microsofts-... - but if anything came out of that I don't see it in GitHub help today.
[2] https://github.com/microsoft/scalar
You know Microsoft runs windows repo with VFS right?
Yes, I do know the Windows os git repo uses GVFS. In fact, I shared my personal experience with git in the os repo some time ago:
https://news.ycombinator.com/item?id=20748778
When I left Microsoft about half a year ago, GVFS and Scalar were both in heavy use there.
When I left Microsoft about half a year ago, GVFS and Scalar were both in heavy use there.
I should clarify that Scalar does not require a VFS for Git server to work correctly, even though it can get significant benefits if a VFS server is available. This means you can use Scalar today with GitHub, but not VFS.
Scalar also supports Windows and macOS, while VFS only supports Windows: https://github.com/microsoft/VFSForGit/blob/v1.0.21014.1/doc...
Scalar also supports Windows and macOS, while VFS only supports Windows: https://github.com/microsoft/VFSForGit/blob/v1.0.21014.1/doc...
Hey, I'm the product manager for Git Systems at GitHub. Can you share more about how you'd use VFS for Git / GVFS protocol if we had it on GitHub?
Right now we don't plan on supporting it; most of our work is focused on upstreamable changes and opinionated defaults. But that could change if we're missing some important use cases.
Feel free to email me - my HN alias @github.com - if you prefer to discuss privately.
Right now we don't plan on supporting it; most of our work is focused on upstreamable changes and opinionated defaults. But that could change if we're missing some important use cases.
Feel free to email me - my HN alias @github.com - if you prefer to discuss privately.
kind of an aside, but what's the best practice for pushing and building separate projects in a monorepo?
say you have a structure like: projectA projectB sharedUtils
Each time you push you might have a build for projectA and projectB but it builds both each time you push to master. Ideally you could use Git to see if anything in projectA or sharedUtils changed to trigger projectA's build and same for projectB, but I'm curious what others are doing.
say you have a structure like: projectA projectB sharedUtils
Each time you push you might have a build for projectA and projectB but it builds both each time you push to master. Ideally you could use Git to see if anything in projectA or sharedUtils changed to trigger projectA's build and same for projectB, but I'm curious what others are doing.
If separate projects have independent builds maybe a monorepo was not a great idea to begin with?
I have a big monorepo at work but whenever anything changes I want to rebuild everything to generate a new firmware image. I have ccache setup to speedup the process given that obviously only a tiny fraction of the code actually needs to be rebuilt.
It's a bit wasteful, sure, but if I were to optimize it I'd be worried about ending up with buggy, non-reproducible builds. Easier to just recompile everything every time and make sure everything still works the way you expect.
So basically my approach is KISS, even if it means longer build times.
I have a big monorepo at work but whenever anything changes I want to rebuild everything to generate a new firmware image. I have ccache setup to speedup the process given that obviously only a tiny fraction of the code actually needs to be rebuilt.
It's a bit wasteful, sure, but if I were to optimize it I'd be worried about ending up with buggy, non-reproducible builds. Easier to just recompile everything every time and make sure everything still works the way you expect.
So basically my approach is KISS, even if it means longer build times.
IIRC you can specify filters "paths" and "paths-ignore" when you define a github action that should only be triggered when a subdirectory changes.
See this documentation page: https://docs.github.com/en/actions/reference/workflow-syntax...
Their example is:
See this documentation page: https://docs.github.com/en/actions/reference/workflow-syntax...
Their example is:
on:
push:
paths:
- '\*.js'
Perhaps check out a tool like please[1]. There are other tools in this space, but that one has worked well for me without the complexity of some other, similar tools.
[1] https://please.build
[1] https://please.build
I can't speak for using it in a massive monorepo, but I started using https://please.build for some of my personal projects recently just as an alternative to the dominant Java build systems (Ant/Maven/Gradle). It's far more straightforward to use, and incremental builds actually work reliably.
That's what build systems aim to do, and there are many of them. In general, I've found all the tooling required around monorepos to be a job for a full-time team. Shortcuts (as suggested in other replies) or full builds on every commit tend to stop scaling relatively quickly. If you take shortcuts, you will find that it becomes "tribal knowledge" to do a full build every time you edit a single line of code, and people who were once making multiple changes a day start making one change a week, or they start committing code without ever having run it. (It happened to me on a 4 person team. We had so many things that needed to be running to test your app, that people just started committing and pushing to production without ever having run their changes locally! That is the kind of thing that happens if you stop caring about tooling, and it happens fast. I addressed it by taking a couple of days to start up the exact environment locally in a few seconds, without a docker build, and people started running their code again.) Be very, very careful.
If you do a full build on every commit, it gets slow much sooner than you'd expect, and people are going to do less work while they context switch to posting to HN while waiting for their 15 minute build for a 1 line code change.
I worked at Google and we had a monorepo, and there were hundreds if not thousands of engineers working on build speed and developer productivity, and it was still significantly slower to "bazel run my-go-binary" versus "go run cmd/my-go-binary". In many cases, it was worth it, but in very isolated applications, it was definitely not worth it. (And people did work around it, by just setting up Git somewhere and using Makefiles or whatever, and that ended up being even worse. But it gets worse incrementally over time, and you're kind of the frog getting boiled alive.)
Where I'm going with is to advise you to be very careful. The tools to support real productivity in a monorepo are expensive in terms of your org's time. If you can get by with a repo per app and a common modules repo, and just update the app to refer to a version of the modules repo as though it's some random open source project you depend on, you're going to get much farther with much less tooling work than you would with a monorepo. But, the modules repo is going to break apps without knowing, and that's going to be a pain. Monorepos do exist for a good reason.
(The other thing I like about monorepos is that you do less per-project setup work. Want to make some new app? You can just start writing it, and you get the build, deploy, framework, etc. for free. It can be very productive if you're finding yourself starting new projects regularly. In my spare time, I write a software, and I really regret splitting it up into multiple projects. But, it's kind of necessary for open source stuff -- people don't want to download ekglue if they want to just run jlog. So I split them, but it costs me my valuable free time to do something I've already done ;)
My TL;DR is that you will be tempted to take shortcuts and the shortcuts will suck. If your project has the resources to have someone set up Bazel, distribute the right version of Bazel and the JRE to developer workstations, setup CI that is aware of Bazel artifact caching, and SREs to be around 24/7 to support your now-custom build environment, you will have a good experience. Be aware that a monorepo is that level of investment.
Meanwhile, if you just have a frontend and a backend in the same repo, you can probably get away with a full build for every commit. And you don't need that shadow team of tooling engineers to make it work, you just need a docker build, and a script that runs "go test ./... && npm test" or whatever ;)
If you do a full build on every commit, it gets slow much sooner than you'd expect, and people are going to do less work while they context switch to posting to HN while waiting for their 15 minute build for a 1 line code change.
I worked at Google and we had a monorepo, and there were hundreds if not thousands of engineers working on build speed and developer productivity, and it was still significantly slower to "bazel run my-go-binary" versus "go run cmd/my-go-binary". In many cases, it was worth it, but in very isolated applications, it was definitely not worth it. (And people did work around it, by just setting up Git somewhere and using Makefiles or whatever, and that ended up being even worse. But it gets worse incrementally over time, and you're kind of the frog getting boiled alive.)
Where I'm going with is to advise you to be very careful. The tools to support real productivity in a monorepo are expensive in terms of your org's time. If you can get by with a repo per app and a common modules repo, and just update the app to refer to a version of the modules repo as though it's some random open source project you depend on, you're going to get much farther with much less tooling work than you would with a monorepo. But, the modules repo is going to break apps without knowing, and that's going to be a pain. Monorepos do exist for a good reason.
(The other thing I like about monorepos is that you do less per-project setup work. Want to make some new app? You can just start writing it, and you get the build, deploy, framework, etc. for free. It can be very productive if you're finding yourself starting new projects regularly. In my spare time, I write a software, and I really regret splitting it up into multiple projects. But, it's kind of necessary for open source stuff -- people don't want to download ekglue if they want to just run jlog. So I split them, but it costs me my valuable free time to do something I've already done ;)
My TL;DR is that you will be tempted to take shortcuts and the shortcuts will suck. If your project has the resources to have someone set up Bazel, distribute the right version of Bazel and the JRE to developer workstations, setup CI that is aware of Bazel artifact caching, and SREs to be around 24/7 to support your now-custom build environment, you will have a good experience. Be aware that a monorepo is that level of investment.
Meanwhile, if you just have a frontend and a backend in the same repo, you can probably get away with a full build for every commit. And you don't need that shadow team of tooling engineers to make it work, you just need a docker build, and a script that runs "go test ./... && npm test" or whatever ;)
One thing that works well in "monorepo" is that API specs are in the same repo for "queries" and "answers". In a multi-repo world, you have three distinct possibilities, the API spec goes in the "answers" repo (this is pretty natural, but requires a build system that triggers "queries" rebuilds/retests on changes), the API spec goes in a dedicated repo, or the API spec goes into one of the "queries" repos.
This ordering is probably in order from "most" to "least" obvious, and I don't have an answer for what the correct solution is. Most probably "separate repo", making the interface not privileged to either side of the "interface barrier".
FWIW, building at Google when at a 8h offset from Pacific Time worked fine, indicating that part of the problem is "does your repo/build solution scale to tens of thousands of simultaneous users".
This ordering is probably in order from "most" to "least" obvious, and I don't have an answer for what the correct solution is. Most probably "separate repo", making the interface not privileged to either side of the "interface barrier".
FWIW, building at Google when at a 8h offset from Pacific Time worked fine, indicating that part of the problem is "does your repo/build solution scale to tens of thousands of simultaneous users".
As a few others have mentioned this is something that build systems handle since they understand the dependency graph. For example, Bazel is often used to this end.
However... I would strongly advise not going for a monorepo. No, I don't mean something like tensorflow where you have a bunch of related tools and projects in a single repo. I mean one repo for the entire org where totally unrelated projects live.
Every company I've been at that used a monorepo found themselves struggling to make it work since you need a ton of full time engineers just to keep things working and scaling. Many of the problems that monorepos try to solve (simplifying dependency and version management) are traded for 10x as many problems and many of them are hard (incremental builds, dependency resolution).
Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted.
However... I would strongly advise not going for a monorepo. No, I don't mean something like tensorflow where you have a bunch of related tools and projects in a single repo. I mean one repo for the entire org where totally unrelated projects live.
Every company I've been at that used a monorepo found themselves struggling to make it work since you need a ton of full time engineers just to keep things working and scaling. Many of the problems that monorepos try to solve (simplifying dependency and version management) are traded for 10x as many problems and many of them are hard (incremental builds, dependency resolution).
Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted.
I'll try to tread at least a little lightly here because this topic does tend to be a bit flammable, but caveat emptor.
My contrasting anecdotal experience is that whether at BigCo or on a small team monorepo is almost always the right answer until your requirements get exotic enough that you're in special-case land anyways (like a separate repo for machine-initiated commits, or something that's security-sensitive enough to wall off some contributors).
Both `git` and `hg` scale easily to to really big projects if you're storing text in them (at FB our C++ code was in a `git` monorepo on stock software until like 2014 or something before it started bogging down, I'll gloss over the numbers but, big): the monorepo-scaling argument is brought out a lot but rarely quantified.
The multi-repo problem that gets you is dependency management, which in the general case requires a SAT-solver (https://research.swtch.com/version-sat), but of course you don't have a SAT-solver in your build script for your small-to-medium organization, so you get some half-assed thing like what `pip` and `npm` do.
Again purely anecdotal, but in my personal experience multi-repo too often gets pushed by folks who want to make their own rules for part of the codebase ("the braces go here"), push an agenda around unnecessary RPCs, or both. That's not true of all cases of course, but it's a common enough antipattern to be memorable.
My contrasting anecdotal experience is that whether at BigCo or on a small team monorepo is almost always the right answer until your requirements get exotic enough that you're in special-case land anyways (like a separate repo for machine-initiated commits, or something that's security-sensitive enough to wall off some contributors).
Both `git` and `hg` scale easily to to really big projects if you're storing text in them (at FB our C++ code was in a `git` monorepo on stock software until like 2014 or something before it started bogging down, I'll gloss over the numbers but, big): the monorepo-scaling argument is brought out a lot but rarely quantified.
The multi-repo problem that gets you is dependency management, which in the general case requires a SAT-solver (https://research.swtch.com/version-sat), but of course you don't have a SAT-solver in your build script for your small-to-medium organization, so you get some half-assed thing like what `pip` and `npm` do.
Again purely anecdotal, but in my personal experience multi-repo too often gets pushed by folks who want to make their own rules for part of the codebase ("the braces go here"), push an agenda around unnecessary RPCs, or both. That's not true of all cases of course, but it's a common enough antipattern to be memorable.
I personally have seen the opposite problem - the friction of making small changes to "utility" libraries becomes a huge pain point for developers when you have to make changes, test locally, push to package manager, update all consumers to use the new version... It's much easier, in my experience, to just consume a class that's already in the same project / repo.
I have also experienced this pain where a company I worked for went too hard on splitting every thing into separate repos, such that updating something deep in the dependency tree becomes very painful and involves a protracted "version bump dance" on dependent repos. There's no silver bullet here.
> Google has a huge team in charge of helping their monorepo scale and work efficiently. You are not google... don't be tempted.
It’s funny, I’ve heard this exact same argument for why you should not use micro services.
It’s funny, I’ve heard this exact same argument for why you should not use micro services.
"You are not google" is also an argument for why you don't have to worry about scaling a monorepo.
The other half of needing to use a build system that understands the dependency graph like Bazel is that Bazel _keeps state_, so that it knows which part of the graph does not need to be re-built when you push commit B because it was already built in commit A.
Well, sure. if you have a pile of totally unrelated things that never need to change in lock-step, then you don't need a "monorepo." But on the other hand if you're building an entire software system such as a collection of API services, a database schema, embedded device firmware, and a website, and all of these things are interdependent and incompatible across versions then please for the love of god use a monorepo.
At my job our cloud team uses multiple separate repositories which makes sense, but it also moves the burden of versioning to run-time. This is because they have to interface with multiple different versions of the device firmware. So they deploy different run-time versions of the APIs to support legacy and current production firmware versions. But our firmware repository is a monorepo in that the sources and build system builds the artifacts for multiple devices from the same source tree.
So it's not so cut and dried as "never use a monorepo" or "always use a monorepo." It involves engineering tradeoffs and decisions that are made in a context, and you can't extract your advice from the context in which it exists. What works for our cloud team would be a terrible mess on the embedded side simply because of how the software is deployed and managed.
At my job our cloud team uses multiple separate repositories which makes sense, but it also moves the burden of versioning to run-time. This is because they have to interface with multiple different versions of the device firmware. So they deploy different run-time versions of the APIs to support legacy and current production firmware versions. But our firmware repository is a monorepo in that the sources and build system builds the artifacts for multiple devices from the same source tree.
So it's not so cut and dried as "never use a monorepo" or "always use a monorepo." It involves engineering tradeoffs and decisions that are made in a context, and you can't extract your advice from the context in which it exists. What works for our cloud team would be a terrible mess on the embedded side simply because of how the software is deployed and managed.
There are definitely pros and cons to either approach. My personal preference/bias is a monorepo architecture, even for smaller teams and orgs.
Working effectively in a monorepo does require suitable tooling (full disclosure: I am a core contributor to Pants, a monorepo build system with, currently, a Python focus). But with that tooling in place a monorepo can make it a lot easier to collaborate, to manage changes and dependencies, and to avoid balkanization and fiefs. Granted there are no silver bullets, but we are working to make tooling good enough to avoid needing the large in-house support team you allude to.
Having a large number of small repos can work if they don't have a lot of interdependencies, and maybe this is what you mean by "totally unrelated projects". But if there are significant interdependencies (and repos usually end up this way, in my experience, which may or may not be representative) things become very brittle, and it's very hard to make changes and reason about how they affect downstream code, not to mention the challenge of version resolution, aka "dependency hell".
In the end, managing a large codebase, whether it follows a monorepo or multi-repo architecture, requires suitable tooling. On balance I think that monorepo is often a better choice, but YMMV for sure.
Working effectively in a monorepo does require suitable tooling (full disclosure: I am a core contributor to Pants, a monorepo build system with, currently, a Python focus). But with that tooling in place a monorepo can make it a lot easier to collaborate, to manage changes and dependencies, and to avoid balkanization and fiefs. Granted there are no silver bullets, but we are working to make tooling good enough to avoid needing the large in-house support team you allude to.
Having a large number of small repos can work if they don't have a lot of interdependencies, and maybe this is what you mean by "totally unrelated projects". But if there are significant interdependencies (and repos usually end up this way, in my experience, which may or may not be representative) things become very brittle, and it's very hard to make changes and reason about how they affect downstream code, not to mention the challenge of version resolution, aka "dependency hell".
In the end, managing a large codebase, whether it follows a monorepo or multi-repo architecture, requires suitable tooling. On balance I think that monorepo is often a better choice, but YMMV for sure.
Monorepos require much more care to be put into the integration/CI side of the process.
This is worth a read: https://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-y...
This is worth a read: https://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-y...
That argument is super, super weird. I agree that if you are a complete jackass with a demonstrated tendency to make poor choices, a monorepo may not work well for you. On the other hand, software development is probably not the right career for such a defective person, or the right line of business for such a defective organization. The anti-pattern described at the beginning of the article is why you should adopt trunk-based development, not a good argument for why you should avoid monorepos.
https://trunkbaseddevelopment.com/
https://trunkbaseddevelopment.com/
Trunk based development is definitely the ideal, and the mode of operation forced on users of Gerrit/Repo, which tries to implement much of Google's internal monorepo workflow with multiple external standalone repos.
The main point is the "better tooling than average" section near the end - you need tools that can understand relationships between different parts of the codebases, and test appropriately, which is where the integration process for monorepos starts to fall apart.
The main point is the "better tooling than average" section near the end - you need tools that can understand relationships between different parts of the codebases, and test appropriately, which is where the integration process for monorepos starts to fall apart.
If you’re in the Javascript/Typescript world Yarn 2 does a really good job with local packages, and letting you depend on them, and optionally allowing you to push for soundness in your repo. (vendoring your deps, tooling, and declaring all dependencies).
Building on that dependency graph and soundness in Yarn 2, I built yarn.build [0] which works in a similar way to all the monorepo build tools.
It builds your packages based on their dependency graph, skips whats already built, and has a bundle command for creating a zip ready for lambda/docker/etc
[0] https://yarn.build
Building on that dependency graph and soundness in Yarn 2, I built yarn.build [0] which works in a similar way to all the monorepo build tools.
It builds your packages based on their dependency graph, skips whats already built, and has a bundle command for creating a zip ready for lambda/docker/etc
[0] https://yarn.build
I’m curious what counts as a “large” monorepo?
This is a very subjective evaluation. You could look at # of files versioned, total bytes of the repository on disk, # of logical business apps contained within, total # of commits, etc.
For me, its any repository where I would think "damnit im going to have to do a fresh clone" if the situation comes up. There isnt a hard line in the sand, but there is certainly some abstract sensation of "largeness" around git repos when things start to slow down a bit.
For me, its any repository where I would think "damnit im going to have to do a fresh clone" if the situation comes up. There isnt a hard line in the sand, but there is certainly some abstract sensation of "largeness" around git repos when things start to slow down a bit.
> Improving repository maintenance
There's one thing I'd really like to see there: the ability to lock out the repository and perform a really aggressive repack. I'm talking `-AdF --window=500` or somesuch. On $dayjob's repository, the base checkout is several gigs. Aggressively repacking it reduces its size by 60%.
There's also a git-level thing which would greatly benefit large repositories: for packs to be easier to craft and more reliably kept, so it's easier to e.g. segregate assets into packs and not bother compressing that, or segregate l10n files separately from the code and run a more expensive compression scheme on that.
There's one thing I'd really like to see there: the ability to lock out the repository and perform a really aggressive repack. I'm talking `-AdF --window=500` or somesuch. On $dayjob's repository, the base checkout is several gigs. Aggressively repacking it reduces its size by 60%.
There's also a git-level thing which would greatly benefit large repositories: for packs to be easier to craft and more reliably kept, so it's easier to e.g. segregate assets into packs and not bother compressing that, or segregate l10n files separately from the code and run a more expensive compression scheme on that.
This might be something Github can do for you manually. I've requested similar things before through support email as one-off fixes, and they were able to accommodate.
It really looks like such a repack ought to be possible in a lockless incremental way. Simply take a snapshot of all the objects in the repo, repack them, and then take a new snapshot of the repo, repack the new objects, and repeat until there is nothing new.
Most repos could be done in one operation, and If bet that as long as repos stay under 10 commits per second this strategy will always terminate.
Most repos could be done in one operation, and If bet that as long as repos stay under 10 commits per second this strategy will always terminate.
Can you clone, do maintenance, push to new repo with `—-mirror‘, then swap repos to get that benefit?
You'd need to have some automated process cherry-picking changes across if you did it this way. To be honest, I think a flag day is the easiest way to handle this, but it does require tooling support to a) lock the repo out while it's going on, and b) prevent users from pushing back in all the removed content immediately afterward.
There’s no real removed content to an aggressive repack. The dangling stuff is GC’d but for the most part it’s just storing everything in packs wit more expensive options than the normal “online” maintenance: a very aggressive full repack can take an hour or two in a really big repo.
Ah cool, thanks for the clarification; I had not realised that repacking was distinct from rebasing/filtering.
> On $dayjob's repository, the base checkout is several gigs.
Why is it several gigs? Is that really necessary?
Why is it several gigs? Is that really necessary?
> Why is it several gigs?
A lot of code written by a lot of engineers over a lot of years.
I'm not sure what other answer you're expecting?
I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual.
> Is that really necessary?
What would you do? Delete history every year or so? I regularly annotate files and see useful history from ten years ago that I need to do my work.
A lot of code written by a lot of engineers over a lot of years.
I'm not sure what other answer you're expecting?
I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual.
> Is that really necessary?
What would you do? Delete history every year or so? I regularly annotate files and see useful history from ten years ago that I need to do my work.
> I'm not sure what other answer you're expecting?
I've seen some things. Binaries, large files, isos, generally just any large file with no lines that somebody just didn't know better and committed, changed a few bytes a few times, and generated huge deltas quickly.
I'm far more surprised you don't know what he was expecting than I am that he asked, considering how many times I've run into this in the past.
I've seen some things. Binaries, large files, isos, generally just any large file with no lines that somebody just didn't know better and committed, changed a few bytes a few times, and generated huge deltas quickly.
I'm far more surprised you don't know what he was expecting than I am that he asked, considering how many times I've run into this in the past.
Nope, the answer is exactly what’s above, this is a large and old project with a fair amount of churn. There are assets in there but they’re a small minority of the issue.
The biggest issue is actually the l10n stuff. It’s all text with lines but there is a lot of it and historically the exports were not really ordered so every major update to the translation files scrambled everything.
The biggest issue is actually the l10n stuff. It’s all text with lines but there is a lot of it and historically the exports were not really ordered so every major update to the translation files scrambled everything.
If people want a concrete example, here's a 6 GB repo that's 90% Java, 5% C, then some other languages.
https://github.com/oracle/graal
It's not even a mono-repo - this is just part of the project.
Maybe someone's got some tools that let them dig around in the history and find large things or explain why it's so large? I don't think they've been checking ISOs in.
https://github.com/oracle/graal
It's not even a mono-repo - this is just part of the project.
Maybe someone's got some tools that let them dig around in the history and find large things or explain why it's so large? I don't think they've been checking ISOs in.
git filter repo has tools for analyzing history
I'm totally not surprised that large repos exist, without having fallen prey to any antipatterns. Just saying that I know "have you committed any binaries, dependencies, generated code, etc" are the first questions I tend to ask when somebody tells me their repo is massive. In part selection bias, though, because usually the people working on super old projects know exactly why.
I was really taken aback by this given my previous experience with the Linux kernel git repo size. So, I just checked out Graal.
Receiving objects: 100% (981372/981372), 187.08 MiB | 6.39 MiB/s, done.
% du -sh graal 328M graal
% du -sh .git 221M .git
And that includes many binary files that were added and removed over time, but not filter-branched out of existence, including jars, pngs, pdfs :)
(https://stackoverflow.com/questions/10622179/how-to-find-ide...)
Receiving objects: 100% (981372/981372), 187.08 MiB | 6.39 MiB/s, done.
% du -sh graal 328M graal
% du -sh .git 221M .git
And that includes many binary files that were added and removed over time, but not filter-branched out of existence, including jars, pngs, pdfs :)
(https://stackoverflow.com/questions/10622179/how-to-find-ide...)
Huh... maybe I have a degenerate copy for having it so long and I should try running the compression commands myself.
See also “git prune” to remove references to branches and objects no longer on the remote server. Check the docs though I think in some cases I might delete local branches or objects so make a backup first.
That just seems an excessive amount of text! But the Linux kernel repo (.git + working copy) is over 7GB.
It turns out folks can write an ungodly amount of code.
It turns out folks can write an ungodly amount of code.
but you have to write ungodly amounts of code to get a repo that big, and it's not even that big
that or check-in binaries, logs, database backups or other things that don't belong in VC. TBH it's pretty much always this
that or check-in binaries, logs, database backups or other things that don't belong in VC. TBH it's pretty much always this
> I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual.
That's a lot of code! I think that is at least a little unusual. The monorepos I see are usually large because people gratuitously commit large binary files (and sometimes change them, generating huge changesets).
That's a lot of code! I think that is at least a little unusual. The monorepos I see are usually large because people gratuitously commit large binary files (and sometimes change them, generating huge changesets).
It is. Let's say we are talking 30 person-years, that is 60000 days. It means 100KB of text written per day and engineer.
Can't edit anymore, but ofc the math is off by factor 10: 'tens of engineers over more than a decade' is more in the realm of 300 person-years. Still 10KB per day and engineer, which is maybe 200-500 lines of code. Far above average still, but approaching something realistic. This is if we are talking 6GB with full history and without any compression.
Just clone any large open source software you know, most of them are several gigs.
I don't know if it's necessary, but that's just the way it is: large monorepos are everywhere.
Just depends on the size of the company and amount of developers. With monorepo approach, all of the projects are in one repo, all kids of technologies/languages, forks of various open source projects, sources of tools/utilities...
At one of my work places one of the monorepos was over 50GB... took an hour to clone and many minutes to pull/rebase if I was a few days behind...
I don’t like monorepo approach because of such monstrosity, but the upside is it’s easier to commit/sync changes that span multiple products/components, and easier to merge many commits a minute from thousands of engineers...
At one of my work places one of the monorepos was over 50GB... took an hour to clone and many minutes to pull/rebase if I was a few days behind...
I don’t like monorepo approach because of such monstrosity, but the upside is it’s easier to commit/sync changes that span multiple products/components, and easier to merge many commits a minute from thousands of engineers...
> Why is it several gigs? Is that really necessary?
Just wait until you have to store deep learning training data.
Just wait until you have to store deep learning training data.
Sounds like something to store in not-git.
Git has large file storage now: https://git-lfs.github.com/
That is not-git, storing files this way will not increase your repo size. Same with git-annex, same with commiting .txt files with a URL to a file on S3.
I'm using git-shell to serve my repo. Can those extensions run over git-shell? Or do I need to set up a new server?
They are separate. git-lfs needs a separate server, that will communicate with the Git client over HTTP (not SSH), and git-annex can use a variety of different backends for storing the data (which doesn't need to be related to the Git repository) such as S3, FTP, webdav, Google Drive, ... and even git-lfs.
GitLab and Gitea come with a git-lfs server, but I don't know if there is any standalone server that you could use with git-shell.
GitLab and Gitea come with a git-lfs server, but I don't know if there is any standalone server that you could use with git-shell.
Ok, thanks. This is why I hope large file support will be integrated in Git.
It's just too much of a hassle to figure out the security implications of yet another server, and administrating it, setting up the connection between Git and the plugin, etc.
It's just too much of a hassle to figure out the security implications of yet another server, and administrating it, setting up the connection between Git and the plugin, etc.
Additionally to everything else in this thread it'd be nice to see better support for monorepos in the github UI as well.
Something like the ability to have github.com/<org>/<repo>/<subproject>/issues be a shard of all the issues for a subproject.
You can do that with tagging, but that's a bit of a PITA because that's all fairly bad and unscalable of a UI.
Something like the ability to have github.com/<org>/<repo>/<subproject>/issues be a shard of all the issues for a subproject.
You can do that with tagging, but that's a bit of a PITA because that's all fairly bad and unscalable of a UI.
What geographic feature is pictured in the hero shot of this blogpost? At first I thought it was the Golden Throne in Capitol Reef but I now think it's something else. I'm 90% sure it's either in Capitol Reef or Grand Staircase.
https://www.flickr.com/photos/23155134@N06/7132776459
Forest Mountains of Zion National Park, Utah
Forest Mountains of Zion National Park, Utah
Close but so far! Thanks so much.
Just 10 years too late, I remember when Facebook switched to Mercurial because the Git community wouldn't care about big monorepos. Mercurial is great!
Mercurial didn't exactly "care" either, they were just more open to outside contributions along these lines
I absolutely love this.
Isnt this changeset introducing a race condition? One of the replicas' checksum could change between the checksum is computed and the lock is taken. Otherwise, there is no need for the lock at all.