What is Azure thinking with this VM size NVadsA10 v5-serieslearn.microsoft.com1 points·by ssheng·2 yıl önce·1 comments
ssheng·2 yıl önce·discussTo get a full A10, one must also get 36 vCPU and 440 GB of memory. I must be missing something.
ssheng·2 yıl önce·discussQuality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
ssheng·2 yıl önce·discussCreative idea. Any data on how much it takes to load the LoRAs and how much latency it adds to the generation speed?
ssheng·2 yıl önce·discussHello! We're the authors of this blog post. Please let us know if there are other models and inference backends you'd like us to benchmark next.