Hey, thank you for your question. In fact it is not one CPU program driving thousands of GPUs but rather the opposite: independent processes, each driving one GPU, and all running the same program on different data. The host process (CPU) is just moving the data to the GPU and driving the network. The tensor (sub)tiles are allocated to individual nodes.
I also want to be clear and I hope I was upfront enough about it in the post: I haven't validated any of this at scale myself. Phobos is a learning project, and the numbers in the post are single-GPU. This is my understanding of how the big clusters work, not something I've run as I do not have access to a large cluster.
[ my public key: https://keybase.io/joa; my proof: https://keybase.io/joa/sigs/XIuyRObYDr46LCgYLIWJt5DKlYIBq3DYbjRIOqUzocE ]