This is a really core and famous problem in extremal combinatorics and a lot of people think about this problem and many related ones. I think the biggest impediment is that too many people were looking for the upper bound instead of looking for a lower bound. The construction here is still incredibly novel and requires sophisticated mathematics to prove it is correct.
But specifically, the ORT strategy can cache computed renames across multiple commits being rebased, so the performance benefits are even greater for "git rebase".
That history that looks tangled and awful would look a lot better if the commits were sorted by `--topo-order` instead of `--date-order`. That sort “groups” commits that are in a single line of history.
You're right that we used what we learned from Scalar's background maintenance an applied that to Git itself.
Putting background maintenance into Git was actually part of our effort to get Scalar on Linux.
You might be interested in our "Philosophy of Scalar" [1] document, which includes this paragraph:
> Scalar intends to do very little more than the standard Git client. We actively implement new features into Git instead of Scalar, then update Scalar only to configure those new settings. In particular, we are porting features like background maintenance to Git to make Scalar simpler and make Git more powerful.
The delta compression in Git is about storing the file contents of an object as a diff against another object. This changes the literal size on-disk, but it doesn't change the logical unit.
In fact, the delta chains used by Git for space compression have no direct relation to the object model DAG. From the perspective of a user using Git, these deltas are completely invisible.
Edit: perhaps to help this point... If Git stores an object using a delta, that doesn't change the object ID of that object compared to storing it uncompressed.
If I have a 100-line file and on 'main' it changes near the top, but in my 'topic' branch it changes near the bottom, then I can cherry-pick 'topic' onto 'main' and Git will resolve the diff correctly. The resulting diff or patch would only change in the line numbers for the context of the diff.
This is of course a very simple example. You might hit a conflict in your "git cherry-pick" command which gives you an opportunity to resolve the unexpected diff issue in an appropriate way, which ends up with a different diff than before.
If you don't disable fetch.writeCommitGraph and gc.writeCommitGraph there is a chance that those operations will overwrite your commit-graph and delete the changed-path filter data. (fetch.writeCommitGraph uses the --split option so it might not actually erase the data until you have accumulated enough "new" commits)
You don't need to rewrite the commit-graph file every time you want to run "git log". The "git log" command will parse the newer commits the old-fashioned way until you reach the commits encoded in the commit-graph file. If you do it once now, then you'll still be fast even if a few commits are added on top of your existing history.
If you update the commit-graph with that command once a week, then you'll stay fast even in a very large repository.
The data is stored in your `commit-graph` file by this command, but those other ways to update the `commit-graph` file don't pay attention to the fact that the data exists in order to persist it.
Thanks for the link to the documentation. That is updated with every major Git version, and can be used to track what features are present. Also, the release notes can be helpful.
In particular, the recent Git v2.27.0 release does include an implementation of Bloom filters with speedups for `git log` and `git blame`. You need to manually run the command to make it work. The version I prefer is this:
By writing these filters, you will speed up most `git log` and `git blame` calls. There is an improvement coming in the next version that includes speedups for `git log -L`.
Caveat: The biggest reason these improvements have not been widely advertised is that the user experience has not been completely smoothed out. In particular, you can only write the changed-path Bloom filters using the command(s) above. If a commit-graph is written during GC (due to `gc.writeCommitGraph` config setting) then the filters will disappear. Similar for `fetch.writeCommitGraph`. We plan to have these resolved in time for v2.28.0, along with more performance improvements.
(Full disclosure: I am a contributor to Git, Scalar, and VFS for Git, which are referenced by the article.)
LFS only downloads the files required by the checkout. From Git’s perspective, those files are very tiny, and only include the information required so LFS can download the files on-demand.
Git’s partial clone is a more natural way of achieving the same outcome.
The sparse-checkout patterns match at the file level, so you can always use that (without “cone mode”) if you want. It becomes difficult to match an exact file list as people add files to projects: you require every other user to update their patterns to match the newly-added file.