Talk on optimizing matrix multiplication with Triton kernels, focusing on low-bit processing and efficient quantization for high-performance AI models.
It addresses key challenges in multimodal AI development:
- Managing diverse inputs
- Scaling Generative AI apps
- Ensuring extensibility
Built on Ray for seamless scaling, Aana offers a unified framework for multiple data types, easy integration with popular ML frameworks, and a modular architecture.
We are releasing new 2-bit Mixtral models. These ones use a mixed HQQ 4-bit/2-bit configuration, resulting in a significantly improved model (ppl 4.69 vs. 5.90) with a negligible 0.20 GB VRAM increase.
No data calibration needed, extremely fast , works on both language and vision models!
* Why does it matter?
Quantization significantly reduces GPU memory requirements but degrades the quality of the models. Having faster and more accurate quantization methods is extremely valuable for the ML community.
* Approach:
Sparsity-based error formulation between the original weights and their dequantized version. We used a Half-Quadratic solver to derive a closed-form solution that is 100x faster than backprop via Pytorch's Autograd.
* Quantization speed:
~ 1 minute for Llama2-13B
~ 4 minutes for LLama2-70B (over 50x faster than GPTQ)
* Findings:
- Larger models quantized to 3/2-bit outperform smaller full-precision models with similar or lower memory requirements.
- Successful 2-bit quantization requires a lower group-size (e.g., 32 or 16) and compression of both the zero-point and the scaling factor for lower memory usage.
While we acknowledge our view might be slightly biased, we genuinely believe that our work will significantly benefit the open-source software (OSS) machine learning community. Code and model are in Apache permissive license.
It is not distilling the model, it is reducing the model weights on the fly and uses LoRA for training/fine-tuning. After the training phase, we explain how to merge the LoRA weights with the pruned weights to achieve faster inference speed
In a nutshell, we've managed to reduce the model's parameter count by up to 50%, double the training speed, and increase inference speed by 1.25 times.
For those interested in the technical details or looking to replicate our results, the code is openly available for community use and contributions
While agreeing to the general principle, incentive structures are wired quiet differently in academia vs end consumer oriented gig/service industries.
Publications ( number, when, where, citations) is the primary currency/value in which one is judged within the peers in academia, and reputation outside the immediate academic community has a much lower weight. Whereas for online market places solid revune is the first priority and then comes reputation ( which is a means for the higher reveune). In academia it is the reverse, with reputation ( in a small clique) being the primary motivator, and funding being the means to gather it.
First principles of doing a PhD and taking up an industrial jobs are quite different, which this article sidesteps. I am talking from the perspective of someone who did a PhD, postdoc and migrated to be a founder/CEO.
A PhD system trains you to think about unsolved problems in an given domain deeply with a larger time runway. The end goal is not a tangible product that reaches millions of people, but rather a set of ideas that can take a crack at the unsolved problems in your field in a novel way. A good work should inspire others in the field, and eventually a larger audience to pick them up and expand and build on top of it. To give a small example, a majority of the fundamentals of machine learning was charted out by many, many PhD works over the last 40 years. Implementing a linear classifier is 2 lines of code in 2018, but many Bothans died to bring us this information :-) .
The goals of industry are more immediate. Expect for a privileged few research labs in industry, your work is expected to be monetized, and rightly so. The goal is for you, if you run the business, else your management team to first figure out a problem of high relevance and monetary value. Build products/solutions for that problem, that can be used by someone who is less versed/ambivalent of your technical solutions. Efficacy of solving that particular problem often defines the merit of your contribution.
The fundamental of choosing the PhD or industry should be taking stock of what kind of contribution you want to make as an individual. If it is a few set of ideas to science, which on a later date might become something fundamental in our understanding of the world, then PhD is a good path. If it is a set of contributions towards a product/solution that eases the pain of many users then go into the industry first.
Author here. Just to clarify, motive of this work is to ease curation, with the massive amount of content being created; but by no means an attempt at creativity or originality.
The work is not at all contradictory to Adorno, especially in the sense that it is explicitly trying to as non-reductionist as possible, and assuming notion of aesthetics is a dynamic entity .
There is a finite pattern in the dataset; more interesting, it has its interesting share of subtleties ( for example, as opposed to a image classification problems), and the technological question is whether we can capture these.
But there is another interesting data question. For our work, we curated our training set with the help of expert curators. But the dataset itself is a metamorphising entity; i.e. it is subject to revision ( it is a continuous process for us at the moment), but more interestingly it is a chance for open debate between our curators. In some sense, technology allow to codify and challenge our notion of aesthetics ( especially with the evolution in our training sets) at a given point of time.
The author here. I used the term "understanding", not as in machines understanding the images, but more as scientific attempt in understanding aesthetics. ( <snippet from the text>"empowering me to develop systems for understanding images from a computational and scientific perspective"</snippet ends> ).
I agree with you completely about "bad industry-focused research". It serves no end. My question is that, is it just a reflection of mediocracy and gaming/dishonesty being everywhere, including academica ?
In this case, please trying to take shortcuts towards their goal of improving the amount of publications and grant approvals. There is reward in the system for this kind of behaviour. You being a reviewer/jury position unfortunately do not have the luxury of a filter.
Is there a way to early catch this , by looking at past trends ?
Slight detour is that this is one of my rationale for spending time reviewing papers for journals/conferences. In average, only 10 to 20% of papers I review really stands out or appeal to me, which is correlated to acceptance rate of a top journal/conference.
Pushing away from color lines is very easy. For negative values of lambda in Eq.1 of the paper, (i.e. reverse the cost for color lines ), the optimization tries to push the puzzle shapes away from the color lines. We had tried a few in this configuration, but personally my co-authors and I liked the puzzles generated by the scheme of adhering to the color lines.
Hardness/fun factor in puzzles is a matter of personal taste. Hence the ability to personalize is very interesting. Not everyone like to solve 10000+ piece puzzles , nor color line adherence, but people invest time and effort in solving them.
One of the authors here. Since it is a optimization, the difficulty can be controlled as a parameter ( the lambda parameter in Eq.1 in the paper ).
But you are right, some of the puzzles can be super-hard ( for example, the Seurat puzzle ) that we used to joke between ourself to name our paper "taking the fun out of puzzles".
Personally, what was fascinating for me is the shape of the puzzle curve it produced. Most of the common puzzles are grid based (i.e. four neighbours - up , down, left, down ). But in this scheme, there can be strange neighborhood pieces, with even stranger shapes.