We've used this extensively at my work and it has been great. As someone else pointed out it's a bit tricky if the code isn't idempotent. We used this as part of a data migration project and for writing we ran the experiment 100% of the time and read back the result to check it was correct - a bit of a performance impact but nothing that was noticeable. One super useful extension we've made to the gem is to have it update Prometheus metrics so you can easily see how often your experiment is showing that the candidate is different from the control.
These might be the clunky way your talking about but you can have “private” git ignored either in ‘.git/info/exclude’ in the repo or in the ‘core.excludesfile’ set in your git config. The later is quite nice because you could actually version control that elsewhere if you version control your system config.
[^1]: https://github.com/mystor/git-revise