I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared.
I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.
People will run into subtle bugs and weird behavior when starting to compile things themselves and replacing libraries I think. Not worth the effort for me, fun as an exercise, but would only do it if it's really (really) needed for performance.
As far as I understand, you can't just use any gcc binary as it exists today. The program needs to be represented as a specific, mathematical expression that is suitable for zero-knowledge proofs.