Thanks for your encouragement! We are working on quantization as well. We recently submitted a paper, Atom [1], that uses 4-bit quantization, delivering 7.73x throughput compared to FP16 and 2.53x compared to INT8. Atom is able to maintain a perplexity (i.e., model accuracy) close to FP16, outperforming existing quantization approaches.
We are polishing the 4-bit code. It will be added to Punica code base soon. Please stay tuned :)
Certainly! We'd like our good designs to be picked up by frameworks and serve all users. Currently, Punica is built on top of PyTorch and HuggingFace Transformers ecosystems. Therefore, vLLM and LMDeploy, which are also in the PyTorch ecosystem, should have a smooth adaption. As for Nvidia Triton and TensorRT-LLM, since our kernels are written in CUDA, I believe it will also work seamlessly.
We call for the open source community to help us integrate Punica with all frameworks, thus the whole society can benefit from the efficiency improvement!
Thank you! We are also very excited about combining the fast fine-tuning and efficient serving. In fact, what you just said is very related to one of our very first motivations. In my previous blog post [1], I call this scheme "Just-in-time Fine-tuning". Our previous measurement is that, for a medium-sized webpage (~10K tokens), it takes around 30 seconds to 2 minutes to finetune a LoRA model. Another good side of this JIT fine-tuning scheme is that, we can turn any model into a long-context model.
We'll keep doing more research on finetuning. And hopefully, we'll see the results soon.
I'm using 27-inch 4K monitors with Gnome. I found it more practical to use no scaling but 1.25x text size (settings in gnome-tweaks). The problem for me with 2x scaling or fractional scaling is that it scales UI (icons, margins) as well. As someone who appreciates functionality (i.e., displaying text) over overdesigned white spacing, scaling the whole UI is just wasting my workspace size whereas scaling text-only is a perfect balance between good-looking text and workspace size.
But anyway, good to see that Linux desktop is gaining fractional scaling support!
> If you find a really good tutorial by programmers who are both excellent teachers and experienced in that particular field, it beats JIT learning on a personal project in one important respect. You get exposure to the One True Way of doing things.
That's exactly my feeling when I was watching Jon Gjengset's Rust tutorials. I like his real reactions to unexpected problems. Really learned a lot from this kind of lengthy but realistic videos. https://www.youtube.com/c/JonGjengset/featured