Working on project-hami.io, but recently decided that all the new markdown based presentation tools have far too much HTML dependency, so I decided to build my own, with custom extensions and plugins. It's supposed to support some basic structures that allow easy creation of presentations, but the editable odp conversion is proving to be a nightmare. It should be possible, but I need help.
All sorts of optimizations. Of course vision is huge. Lots of production use in all sorts of manufacturing. Lam research had a few talks a semiconductor manufacturing optimization. There is also CUDA assisted RAN.
Maybe AI is a bit of a misnomer, since everything ML at some point just started getting called AI.
Gentle reminder that while most money is spent on LLM inference, the vast majority of useful AI use is in fact not LLMs. Also, more and more work is poured into making small models. One thing I like about the whole export controls saga is that people are finding creative ways to squeeze performance out of these devices as witnessed in this post. But, if you then look at solutions like vLLM, vLLM will just fill whatever VRAM is available, no matter the context size, or the model size. So then you have two things to worry about:
First, where do you know exactly what the optimal VRAM assignment per model, per context size is, which seems to be currently based purely on experience and second how do you make sure that only that amount is available to your infra/containers, which is being handled by DRA and stuff like https://project-hami.io
While only tangentially related to the blog post here. The title is picked in such a way that I couldn't help, but put the shameless plug here. When he wrote popping the bubble, I thought we're talking about devices and reducing NVIDIA dependency, but this seems very focused on Cuda.
Disclaimer: I work with Dynamia.ai, the founders of which created HAMi.
Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k.
Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.
Is it hyperbolic though? One of the best things about the compute and memory shortage is that people are going to insane lengths to optimize things to run on lower memory / lower compute devices. If we keep this up for a while and then ramp up memory and local compute production, that AI inflection point may actually come.
I'm extremely concerned about the state of Open Source. The gamification of the whole thing & devstats means that people that are good at gaming metrics are rising up the ranks and people that are genuine high quality contributors and pushed to the sidelines unless they have a very popular profile. Mass generated AI slop and AI content gives people massive devstats boosts.
Funny you should bring up Germany, since it's an a place where a monopoly class of Notary's gets to suck you dry every time you add a new investor to your business in a reading session where the only value he brings to the table is to read the whole thing for you.
It's interesting that decades into this we are still calling the massmurder and pillaging of dozens of countries, turning entire geographic regions into war ridden wastelands with slave markets, a "war on terror". Meanwhile the leaders of those so-called terror groups are praised and invited to ALL the western capitals.
Unfortunately Europeans are terrible customers for making money. They ask a lot of questions and they're very stingy with their wallets. Americans on the other hand ...
The whole internet AND local computing. This may be cool for people on SF salaries but for the vast majority of the world memory has become a luxury good
I'm not sure how much of it is just an unintentional side-effect of greed from promises of international capital based in NY, and Dubai, and how much was intentional malicious behavior to destroy home compute to force people to pay for openai subscriptions, but the role of a functioning government typically is to keep corporations from doing exactly this.
Regardless of which one it is, I absolutely despise the cartel that is running the US government right now, that created this situation for their crony big tech buddies.
It's very broken, and I'm not sure if it's possible to write everything original given that you're expected to repeat 2/3rds of past research to fill pages when you write your thesis. For a master thesis that was at least 100 pages. For a PhD nowadays each one of those is published as a book. At least it was like that in my engineering department.
And I feel like the reason why OpenAI was so aggressive with messing up the RAM market, was specifically to make it hard for us to run models on our own hardware.