Yes! P4 and the other small cards are fantastic. I've been more focused on developing a cooling system around the 2 slot sized cards so I don't have any of these lying around. Are you using P4 for anything outside of LLM work? I'm interested in seeing their image processing capabilities.
16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length?
Do you have a set of models you'd like me to look at?
Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.