When you watch on vhs or laserdisc the loss of resolution only bothers you til the movie sucks you in.
At that point it’s irreverent because your eyeballs are not watching a long sequence of pretty still pictures, but rather your brain is watching a story in a way similar to reading a good book.
It’s not a process monitor, really, but to me the AWS Lightsail monitor tab feels like this. The “sustainable” line hits me right in the OCD to keep me grinding on cpu usage of the workload to keep extra spend at zero.
I have this as well, but run a heavily locked down and isolated BIND server with NSD and Unbound for external authoritative and internal caching DNS respectively.
Its easy to feed an RBL to unbound to do pi-hole type work, I use pf to transparently redirect all external DNS requests to my local unbound server but I get the bind automation around things like DNSSEC, DHCP ddns and ACME cert renewals.
2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card.
I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.
Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server.
I had it running at 40t/s with zero effort and 50t/s with a brief tweaking.
Its just too bad that even the 120b isn't really worth running compared to the other models that are out there.
It really is amazing what ggerganov and the llama.cpp team have done to democratize LLMs for individuals that can't afford a massive GPU farm worth more than the average annual salary.
Thanks Daniel. I know you upload them, but I was hoping for some solid numbers on your dynamic q8 vs a naive quant. There doesn't seem to be anything on either of those links to show improvement at those quant levels.
My gut feeling is that there's not enough benefit to outweigh the risk of putting a middleman in the chain of custody from the original model to my nvme.
However, I can't know for sure without more testing than I have the time or inclination for, which is why I was hoping there had been some analysis you could point me to.
I generally download the safetensors and make my own GGUFs, usually at Q8_0.
Is there any measurable benefit to your dynamic quants at that quant level?
I looked at your dynamic quant 2.0 page, but all the charts and graphs appear to cut off at Q4.
Trees are pure carbon. I have heard a number of weak “yeah, but…” arguments that try to diminish the fact, but a central, common sense thesis remains.
If we are truly worried about climate change and are unable to curb our consumption, then we should plant as many trees as we can and aggressively shift as much of our long-lived infrastructure to using wood products as possible.
In my opinion GPT-SoVITS is the best if you can put in the effort. I'm still using v2 since the output is so good.
Its also the best multilingual one in my testing on Japanese inputs.
It's a bit harder when they've provided the safetensors in FP8 like for the DS3 series, but these smaller distilled models appear to be BF16, so the normal convert/quant pipeline should work fine.
At that point it’s irreverent because your eyeballs are not watching a long sequence of pretty still pictures, but rather your brain is watching a story in a way similar to reading a good book.