ByronBates

قبل 3 سنواتdiscuss

You could try the alternative version of the database-generator: https://github.com/jmforsythe/Git-Heat-Map/pull/6 - it shouldn't crash.

ByronBates

قبل 3 سنواتdiscuss

Computing diffs is what takes large amounts of time as the object database is used intensively along with limited efficiency of object caches.

I couldn't resist and threw `gitoxide` at it, and it turned out to be more than 2x as fast (even though it uses way more CPU to do that, there is definitely room for improvement).

The PR which adds the `db-gen` program: https://github.com/jmforsythe/Git-Heat-Map/pull/6

ByronBates

قبل 4 سنواتdiscuss

`gitoxide` can currently perform many of the tasks required, and closing the loop for a full clone with worktree is going to happen at the end of this year.

That said, here is how to receive a pack from the remote and resolve it: `mkdir out && gix -v no-repo pack receive https://github.com/Byron/gitoxide out`.

ByronBates

قبل 4 سنواتdiscuss

That number is already based on using `zlib-ng`. Sometimes I wonder if it's cheating since `git` might not actually use it. In any case, improving the efficiency of `zlib` along with the Rust integration for it has immediate impact on the git object database performance of `gitoxide`.

ByronBates

قبل 4 سنواتdiscuss

> This is fast enough for small repositories, but as the repository increases in size the overhead of parsing these plain-text files to get the graph relationships becomes too expensive. Even the fact that we need a binary search to locate the object within the packfile begins to add up.

From my experience and measurements as the author of `gitoxide` both the parsing of commits as well as the binary search to find the object are negligible costs here. The majority of the time is spent in `zlib` which has to inflate objects and deltas prior to use. This makes me wonder what `zstd` would do to these kinds of workloads.

In any case, the object database decoding performance is the bottleneck when traversing commits, and one can expect to get about 120000 commits per second on the linux kernel pack on a modern CPU core like M1 with `gitoxide`s implementation.

Besides that, I find the explanation of the commit-graph file interesting and how it relates to commit-graph queries. There is so much to learn, and so much still to implement :).

ByronBates

قبل 6 سنواتdiscuss

I frequently use `tig` to see the commit history, and pull out a single commit hash for use in `git checkout` or to reference it elsewhere. Probably it's just me not knowing all the `tig` hotkeys to make that easy, but it's my hope to have `gitoxide` as an obvious way to do things at least I need commonly.

ByronBates

قبل 6 سنواتdiscuss

It's a project goal to make the API so accessible and easy to use, that's it's straightforward to quickly throw together your very own custom commands that do exactly what you need. This should allow building these slot-in commands as the need arises, and without breaking into sweat.

ByronBates

قبل 6 سنواتdiscuss

Great point! Windows is already tested on CI, but I used the opportunity to make Windows support explicit in the project goals.

ByronBates

قبل 6 سنواتdiscuss

I am not interested in competing with libgit2, and have no plans in adding and maintaining C-bindings to the project. This doesn't mean, however, that other parties couldn't maintain them out-of repo at first, and once things stabilize, it just makes sense to make them part of an organization that maintains it along with the project. After all, C-bindings open it up to be used in interpreters, which might be especially interesting for users of `GitPython`. It has been around for a decade and is definitely in for an alternative (I am the author and maintainer of that one, too).

ByronBates

قبل 6 سنواتdiscuss

That sounds like such a project should definitely see improvements when using GitOxide. Even in projects of the size of the Linux Kernel checkouts can take time as those only reach about 3000 files per second. You will see only 70% of a single core being busy and a lot of kernel overhead. On my machine, using all the 4 cores, this number can be 7500 when checking out in parallel, something that `gitoxide` is definitely going to do from day one. After all, there is no value in just being as fast as git.

The same applies when checking for changes on disk - right now git does not do that in parallel. These are low hanging fruit that I plan to pick for my own sake.

There is another contributor who is very interested in increasing pack performance, which will directly impact fetch and pull speeds. Judging from my experience with packs thus far, I believe a lot of options are still to be tapped in that field as well. Rust will open it up for contributions and experiments to a greater audience, so I would hope that this will go way beyond of what I can do.

Even though this project is not very contributor friendly right now, I will be working on improving this so more people can join in (see https://github.com/Byron/gitoxide/issues/8).

ByronBates

قبل 6 سنواتdiscuss

Right now, the project is clearly missing contribution guidelines, but now that it's a bit more public these will be added soon, possibly along with some tickets that are ready for pickup. For now I am very focussed on implementation. That said, I have created a quick issue for you with a small tasks and high value: https://github.com/Byron/gitoxide/issues/7 Thanks a lot for your consideration <3

ByronBates

قبل 6 سنواتdiscuss

Let me reply with the most tedious parts, as I wouldn't want to make the impression of labelling git as 'bad' - I have been fascinated by it for more than 10 years now and it's time to scratch my itch for good.

Please note that everything I name can certainly be fixed by some tool that already exists, and that's great. It's just that I would like to have 'this one tool' that gets most things right and be happy with it alone.

That said, here is my list:

* adding individual files to the staging area/index

* looking at the commit history and copying individual commit hashes

* pulling while assuring no merge commit is created, or: simplified trunk-based development

* resumable clones

* seeing what's going on during all remote operations - here in China GitHub is incredibly slow most of the time and connections can be flaky

ByronBates

قبل 6 سنواتdiscuss

Absolutely! My plan is to proclaim a version 1.0 once the basic workflow of clone-commit-push can be performed, which should be enough for many to start using it. From that point on, the API should remain stable, following semantic versioning as usual.

It is my hope that the quality and stability will be high enough to convince people [in the Rust ecosystem] to move away from libgit2 and instead contribute to the maintenance and development of gitoxide, rewarding everyone with better performance and improved usability.

ByronBates

قبل 6 سنواتdiscuss

As this is part of the every-day workflow, I will think long and hard on how to make it as painless and user-friendly as possible. Having a more user-friendly experience on the command-line is one of the project goals, and it will also be me benefiting from it so the motivation will be high to achieve it. By then, that thinking process will hopefully be public enough to allow more people to chime in and get a better result that way. And yes, I think a lot is possible, and I can't wait to test some ideas of mine :)

ByronBates

قبل 6 سنواتdiscuss

Unfortunately I can't yet tell, as the corresponding code does not yet exist. Object and pack lookup is competitively fast so I would hope that translates well to everyday operations like that.

Actual numbers for what's there, pack access and traversal, can be found here: https://github.com/Byron/gitoxide/issues/1 and here: https://github.com/Byron/gitoxide/issues/5

ByronBates

قبل 6 سنواتdiscuss

I am the author, and will be here for a few hours in case there are any questions.

(proof: https://keybase.io/byronbates)

ByronBates

قبل 8 سنواتdiscuss

The same author also wrote hyperfine, a tool to compare performance of various program runs.

    hyperfine './target/release/hexyl ./target/release/hexyl' 'xxd ./target/release/hexyl' 'hexdump ./target/release/hexyl'
    Benchmark #1: ./target/release/hexyl ./target/release/hexyl
      Time (mean ± σ):      1.529 s ±  0.028 s    [User: 1.476 s, System: 0.050 s]
      Range (min … max):    1.491 s …  1.581 s    10 runs

    Benchmark #2: xxd ./target/release/hexyl
      Time (mean ± σ):      70.5 ms ±   0.5 ms    [User: 68.0 ms, System: 1.2 ms]
      Range (min … max):    69.5 ms …  72.3 ms    41 runs

    Benchmark #3: hexdump ./target/release/hexyl
      Time (mean ± σ):     262.4 ms ±   2.8 ms    [User: 260.1 ms, System: 1.5 ms]
      Range (min … max):   259.8 ms … 268.8 ms    11 runs

    Summary
      'xxd ./target/release/hexyl' ran
        3.72 ± 0.05 times faster than 'hexdump ./target/release/hexyl'
       21.70 ± 0.43 times faster than './target/release/hexyl ./target/release/hexyl'

Currently hexyl seems nearly 22x slower than xxd.

ByronBates

Submissions

Check out the Linux kernel in less than a second with gitoxide

comments