We're a distributed GPU cloud with 10k+ consumer GPUs on our network.
We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s.
The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.
This is what we've done at Salad (www.salad.com). SaladCloud (officially launching on July 11th) has 10k+ Nvidia GPUs at the lowest prices in the market. Not conducive for training but for serving inferences at massive scale.
We're a distributed GPU cloud with 10k+ consumer GPUs on our network. We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s.
The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.