Cracking the Python Monorepo(gafni.dev)
gafni.dev
Cracking the Python Monorepo
https://gafni.dev/blog/cracking-the-python-monorepo/
6 comments
Most of the time, the CI resources in a python monorepo is not spent on packaging. It’s spent on running the tests.
I would love to read more about how the author is tackling the testing problem in their setup.
I would love to read more about how the author is tackling the testing problem in their setup.
Hey, I’m the author.
At the bare minimum tests of unchanged code and dependencies would be skipped (cached) as well in this setup.
More sophisticated rules would have to be set up by hand, but again it’s easy to do with Dagger (as you can expected any kind of logic there).
But the whole point of using Dagger in this setup is to get tests caching out of the box, and for that you need to assemble the container correctly (by only including relevant dependency files).
At the bare minimum tests of unchanged code and dependencies would be skipped (cached) as well in this setup.
More sophisticated rules would have to be set up by hand, but again it’s easy to do with Dagger (as you can expected any kind of logic there).
But the whole point of using Dagger in this setup is to get tests caching out of the box, and for that you need to assemble the container correctly (by only including relevant dependency files).
Reads more like an advertisement for dagger than anything else, unfortunately.
At work we have a pretty large project with many teams having moved to using nested git submodules for their stuff.
Whenever you checkout a commit you basically have do a `git submodule --init --recursive` and pray there's no random files left over because some submodule has moved and git-submodule thinks it's your job to clean up its mess. This becomes really annoying when you want to bisect something.
Surely there must be a saner way to deal with trees of git repos. I guess AOSP uses its own `repo` tool to do multirepo stuff which might be better. But honestly this _should_ be fixable in git-submodule itself if they just make it behave sanely.