The nature of the basic research beast. There are grad student written astrophysics/comp chemistry spaghetti codes that continue to get big funding for the sole reason (it feels like) that they scale huge and eat up DOE supercomputing time "look how fast (we burn money)". Maybe a hot take.
Intended as analogy - but it is essentially a description of the DMRG algorithm (quantum chem). Only pair-wise operators there but the theory approaches exact when there are enough terms in your tensor product (iterations ~ depth) and a large enough embedding dimension.
> There are no constraints on how the data transforms.
Except those implicit in your learned representation. And that representation could be the MB WF.
QM would tell us the order of your Hamiltonian (attention operator) doesn’t limit the complexity of the wave function (hidden state). It might be more efficient to explicitly correlate certain many-body interactions, but pair-wise interactions, depth and a basis (hidden state dimension) approaching completeness "are all you need”.