Not arguing against 'great', but cost efficiency is questionable. for 10% you can get two used 3090. The good thing about LLMs is they are sequential and should be easily parallelized. Model can be split in several sub-models, by the number of GPUs. Then 2,3,4.. GPUs should improve performance proportionally on big batches, and make it possible to run bigger model on low end hardware.
it may be even cheaper to use big models through API. Claude, GPT, whatever. Rental is efficient only for big butches, while API is priced per call/size and is cheap for small models.
from https://www.asacomputers.com/nvidia-l40s-48gb-graphics-card....
nvidia l40s 48gb graphics card Our price: $7,569.10*
Not arguing against 'great', but cost efficiency is questionable. for 10% you can get two used 3090. The good thing about LLMs is they are sequential and should be easily parallelized. Model can be split in several sub-models, by the number of GPUs. Then 2,3,4.. GPUs should improve performance proportionally on big batches, and make it possible to run bigger model on low end hardware.