You can already buy cheap but powerful old servers. But newer hardware tends to be more power efficient. So, depending on time horizon you consider, it might be cheaper to buy newer hardware.
Assuming that GPUs power efficiency will increase, the same will be true about them.