Talk on optimizing matrix multiplication with Triton kernels, focusing on low-bit processing and efficient quantization for high-performance AI models.
It addresses key challenges in multimodal AI development:
- Managing diverse inputs
- Scaling Generative AI apps
- Ensuring extensibility
Built on Ray for seamless scaling, Aana offers a unified framework for multiple data types, easy integration with popular ML frameworks, and a modular architecture.
We are releasing new 2-bit Mixtral models. These ones use a mixed HQQ 4-bit/2-bit configuration, resulting in a significantly improved model (ppl 4.69 vs. 5.90) with a negligible 0.20 GB VRAM increase.
No data calibration needed, extremely fast , works on both language and vision models!
* Why does it matter?
Quantization significantly reduces GPU memory requirements but degrades the quality of the models. Having faster and more accurate quantization methods is extremely valuable for the ML community.
* Approach:
Sparsity-based error formulation between the original weights and their dequantized version. We used a Half-Quadratic solver to derive a closed-form solution that is 100x faster than backprop via Pytorch's Autograd.
* Quantization speed:
~ 1 minute for Llama2-13B
~ 4 minutes for LLama2-70B (over 50x faster than GPTQ)
* Findings:
- Larger models quantized to 3/2-bit outperform smaller full-precision models with similar or lower memory requirements.
- Successful 2-bit quantization requires a lower group-size (e.g., 32 or 16) and compression of both the zero-point and the scaling factor for lower memory usage.
While we acknowledge our view might be slightly biased, we genuinely believe that our work will significantly benefit the open-source software (OSS) machine learning community. Code and model are in Apache permissive license.
It is not distilling the model, it is reducing the model weights on the fly and uses LoRA for training/fine-tuning. After the training phase, we explain how to merge the LoRA weights with the pruned weights to achieve faster inference speed
In a nutshell, we've managed to reduce the model's parameter count by up to 50%, double the training speed, and increase inference speed by 1.25 times.
For those interested in the technical details or looking to replicate our results, the code is openly available for community use and contributions
While agreeing to the general principle, incentive structures are wired quiet differently in academia vs end consumer oriented gig/service industries.
Publications ( number, when, where, citations) is the primary currency/value in which one is judged within the peers in academia, and reputation outside the immediate academic community has a much lower weight. Whereas for online market places solid revune is the first priority and then comes reputation ( which is a means for the higher reveune). In academia it is the reverse, with reputation ( in a small clique) being the primary motivator, and funding being the means to gather it.
First principles of doing a PhD and taking up an industrial jobs are quite different, which this article sidesteps. I am talking from the perspective of someone who did a PhD, postdoc and migrated to be a founder/CEO.
A PhD system trains you to think about unsolved problems in an given domain deeply with a larger time runway. The end goal is not a tangible product that reaches millions of people, but rather a set of ideas that can take a crack at the unsolved problems in your field in a novel way. A good work should inspire others in the field, and eventually a larger audience to pick them up and expand and build on top of it. To give a small example, a majority of the fundamentals of machine learning was charted out by many, many PhD works over the last 40 years. Implementing a linear classifier is 2 lines of code in 2018, but many Bothans died to bring us this information :-) .
The goals of industry are more immediate. Expect for a privileged few research labs in industry, your work is expected to be monetized, and rightly so. The goal is for you, if you run the business, else your management team to first figure out a problem of high relevance and monetary value. Build products/solutions for that problem, that can be used by someone who is less versed/ambivalent of your technical solutions. Efficacy of solving that particular problem often defines the merit of your contribution.
The fundamental of choosing the PhD or industry should be taking stock of what kind of contribution you want to make as an individual. If it is a few set of ideas to science, which on a later date might become something fundamental in our understanding of the world, then PhD is a good path. If it is a set of contributions towards a product/solution that eases the pain of many users then go into the industry first.
Author here. Just to clarify, motive of this work is to ease curation, with the massive amount of content being created; but by no means an attempt at creativity or originality.
The work is not at all contradictory to Adorno, especially in the sense that it is explicitly trying to as non-reductionist as possible, and assuming notion of aesthetics is a dynamic entity .
There is a finite pattern in the dataset; more interesting, it has its interesting share of subtleties ( for example, as opposed to a image classification problems), and the technological question is whether we can capture these.
But there is another interesting data question. For our work, we curated our training set with the help of expert curators. But the dataset itself is a metamorphising entity; i.e. it is subject to revision ( it is a continuous process for us at the moment), but more interestingly it is a chance for open debate between our curators. In some sense, technology allow to codify and challenge our notion of aesthetics ( especially with the evolution in our training sets) at a given point of time.
The author here. I used the term "understanding", not as in machines understanding the images, but more as scientific attempt in understanding aesthetics. ( <snippet from the text>"empowering me to develop systems for understanding images from a computational and scientific perspective"</snippet ends> ).