i never found myself struggling with submodules, but at times i found myself just slightly annoyed (especially when having to remove/replace submodules), especially when they are used for simpler use cases.
i actually ended up creating https://carvel.dev/vendir/ for some of the overlapping use cases. aside from not being git specific (for source content or destination), its entirely transparent to consumers of the repo as they do not need to know how some subset of content is being managed. (i am of course a fan of committing vendored content into repos and ignore small price of increasing repo size).
> It would be nice if there was a separate state reconciliation system that one could adapt to use with Cue or Dhall or any other frontend
this exactly was thinking behind https://carvel.dev/kapp for Kubernetes (i'm one of the maintainers). it makes a point to not know how you decided to generate your Kubernetes config -- just takes it as input.
> In particular the ability to import other files as semantic hashes seems like a great feature.
it's an interesting feature but seems like it should be unnecessary given that config can be easily checked into git (your own and its dependencies).
i think kubernetes is not a great example in favor of more client state (like tf) since k8s has uniform resource structure (metadata.*) and first class labeling support. but as you point out kubectl doesnt use labels well (at least imho).
when building https://carvel.dev/kapp (which i think of as "optimized terraform" for k8s) the goal was absolutely to take advantage of those k8s features. we ended up providing two capabilities: direct label (more advanced) and "app name" (more user friendly). from impl standpoint, difference is how much state is maintained.
"kapp deploy -a label:x=y -f ..." allows user to specify label that is applied to all deployed resources and is also used for querying k8s to determine whats out there under given label. invocation is completely stateless since burden of keeping/providing state (in this case the label x=y) is shifted to the user. downside of course is that all apis within k8s need to be iterated over. (side note, fun features like "kapp delete -a label:!x" are free thanks to k8s querying).
"kapp deploy -a my-app -f ..." gives user ability to associate name with uniquely auto-generated label. this case is more stateful than previous but again only label needs to be saved (we use ConfigMap to store that label). if this state is lost, one has to only recover generated label.
imho k8s api structure enables focused tools like kapp to be much much simpler than more generic tool like terraform. as much as i'd like for terraform to keep less state, i totally appreciate its needs to support lowest common denominator feature set.
common discussion topics:
* whats the lowest common denominator for apis that need to be supported
* how much state to store client side vs server side (in the api itself e.g. tags or in "assistive service" e.g. s3 api)
* is it enough to just store resource identifiers vs whole resource content (e.g. can resource content be retrieved at a later point; if content is stored, is it sensitive)
* how easy is it to recover from complete state loss
i actually ended up creating https://carvel.dev/vendir/ for some of the overlapping use cases. aside from not being git specific (for source content or destination), its entirely transparent to consumers of the repo as they do not need to know how some subset of content is being managed. (i am of course a fan of committing vendored content into repos and ignore small price of increasing repo size).