It would be great if you can help me with this PR as well as adding a support for exporting a model that was quantized using GPTQ, bitsandbytes, plain torch. This would bring a lot of benefit from both worlds:
- Low memory footprint(thanks quantization)
- Fast inference(thanks io binding)
Particularly in case of alpaca I have seen a 5x decrease in latency on A100 and 10x on AMD EPYC.
I believe this is the way for users to have an AI that could genereate a response as fast as it can on their hardware.
I have also added a link to my profile on hf with small alpacas turned into ONNX format. Take a look into them.
Q: We need you to draw seven red lines.
All of them strictly perpendicular; some with green ink and some with transparent. Can you do that?
A:
I'm sorry, but I am a text-based AI model and I don't have the ability to physically draw lines or use different colors of ink. However, I can help you visualize this concept by describing the red lines and the different ink colors.
A red line with green ink: Imagine a traditional red line, but with the ink being green. This line would still be red in color, but it would have a distinctive and recognizable green hue.
A red line with transparent ink: Imagine a red line, but with the ink being transparent. This line would still be red in color, but it would be less opaque and more see-through.
A red line with red ink: This line would be just like a regular red line, but with red ink. This is the most traditional and common type of red line.
A red line with blue ink: Imagine a red line, but with the ink being blue. This line would still be red in color, but it would have a distinctive and recognizable blue hue.
A red line with black ink: This line would be just like a regular red line, but with black ink. This is another common type of red line.
A red line with green and blue ink: Imagine a red line, but with the ink being a combination of green and blue. This line would still be red in color, but it would have a unique and distinctive green and blue hue.
A red line with transparent and green ink: Imagine a red line, but with the ink being a combination of transparent and green. This line would still be red in color, but it would be less opaque and more see-through, with a green tint.