ssheng

2 ปีที่แล้วdiscuss

To get a full A10, one must also get 36 vCPU and 440 GB of memory. I must be missing something.

2 ปีที่แล้วdiscuss

Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.

2 ปีที่แล้วdiscuss

How does Exllama rank among these? Heard good things about it.

2 ปีที่แล้วdiscuss

Creative idea. Any data on how much it takes to load the LoRAs and how much latency it adds to the generation speed?

2 ปีที่แล้วdiscuss

Hello! We're the authors of this blog post. Please let us know if there are other models and inference backends you'd like us to benchmark next.

Submissions