First, it's not really "1 bit", actually much closer to 2-bit.
IQ1_M is actually 1.75bit and IQ2_XXS is 2.06bit
This is from the ./llama-quantize --help with most of the quant types and their size in bpw:
https://pastebin.com/bCUqGfeE
And to elaborate on the "dynamic" aspect inconito said in the other comment, if you click on one of the .gguf files in huggingface:
There are a lot of Q5_K, Q6_K, etc tensors.
Only the routed experts (ffn_gate_exps.weight, ffn_up_exps.weight, ffn_down_exps.weight) are heavily quantized, and it looks like the down_proj is actually iq3_xxs for this model.
>I don’t really get anything useful out of these postmortems (e.g., clues about how to rephrase my instructions)
Unfortunately, an LLM can't actually reflect or advise how you could have improve the prompt. Otherwise we could give them a sample output and say "Generate the prompt that would produce this output.
Then why do the original Command-R, Command-R+ and WizardLM2-8x22B (taken down because Microsoft forgot to run safety checks) get it right every time?
But the newer models get it wrong?
I’m not saying it’s a “political conspiracy”, it’s the alignment tax.
I don't talk to them about politics or "china 1989" either. But here's a quick example of the alignment tax:
```
A woman and her son are in a car accident. The woman is sadly killed. The boy is rushed to hospital. When the doctor sees the boy, he says "I can't operate on this child, he is my son." How is this possible?
```
Older less politically aligned models get it right. Here's CohereLabs/c4ai-command-r-v01:
That's without reasoning, but the model seems to be conflicted. First it blurts out:
```
The doctor is the boy's mother.
```
Then it second-guesses itself (with reasoning disabled), considers same-sex parents then circles back to the original response along with a small lecture about gender biases.
>The voiceless groups or fringe opinions which we take as normative today do not appear.
Times are different. Anybody with an internet connection can "publish" their thoughts and perspective online. LLMs scrape all of this. Modern datasets like CommonCrawl capture a vastly wider spectrum of humanity than a printing press ever could.
The pre-1930 model acts as a time capsule of "gatekept publishing", but modern LLMs are trained on the democratized web.
>Does this encourage us to write in the present such that we influence the models in perpetuity?
I noticed a bunch of LLM-powered Reddit accounts praising products/services in dead threads. Or one bot posting a setup question, then a few other bots responding with praise / questions about a specific product in response.
I don't know why they're doing this but I'm beginning to suspect it's something like this (get this positive sentiment into the datasets for the next generation of LLMs).
Last year's models were bolder. Eg. Sonnet-3.7(thinking), 10 times got it right without hedging:
>You should drive your car to the car wash. Even though it's only 50 meters away (which is very close), you'll need your car physically present at the car wash to get it washed. If you walk there, you'll arrive without your car, which wouldn't accomplish your goal of getting it washed.
>You'll need to drive your car to the car wash. While 50 meters is a very short distance (just a minute's walk), you need your car to actually be at the car wash to get it washed. Walking there without your car wouldn't accomplish your goal!
etc. The reasoning never second-guesses it either.
Yes, it does exactly that. It also sends other prompts like generating 3 options to choose from, prefilling a reply like 'compile the code', etc. (I can confirm this because I connect CC to llama.cpp and use it with GLM-4.7. I see all these requests/prompts in the llama-server verbose log.)
You can stop most of this with
export DISABLE_NON_ESSENTIAL_MODEL_CALLS=1
And might as well disable telemetry, etc:
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
I also noticed every time you start CC, it sends off > 10k tokens preparing the different agents. So try not to close / re-open it too often.
100% agreed, and I've been explaining this to people for the past year.
I have an iPhone now and miss Firefox for Android (with Ublock, sponsorblock, etc). But this painful restriction is the only thing stopping Chrome from becoming the new IE6.
At a few startups I've worked for, the devs all use chrome exclusively, and only test in chrome during development.
The only reason they consider other browsers, is because of Safari on iOS. Sometimes it's driven by support calls / complains from iOS users after a release. If Chrome's engine is allowed on iOS, that means support can just tell the users to install Chrome (like they do now if anyone has issues on Windows in other browsers). This means Firefox will usually work as well.
Many years ago, I was able to swap banks when my bank's website stopped working in Opera 12. If all the major banks / websites target Chrome-only, we'll have no choice but to use it. And then we'll have no control as Google push new restrictions into Chrome.
>This is great. Sonnet 4.5 has degraded terribly.
>I can get some useful stuff from a clean context in the web ui but the cli is just useless.
>I swear it was not that awful a couple of months ago.
I agree on all 3 counts. And it still degrades after a few long turns in openwebui. You can test this by regenerating the last reply in chats from shortly after the model was released.
First, it's not really "1 bit", actually much closer to 2-bit. IQ1_M is actually 1.75bit and IQ2_XXS is 2.06bit This is from the ./llama-quantize --help with most of the quant types and their size in bpw: https://pastebin.com/bCUqGfeE
And to elaborate on the "dynamic" aspect inconito said in the other comment, if you click on one of the .gguf files in huggingface:
https://huggingface.co/unsloth/GLM-5.2-GGUF/blob/main/UD-IQ1...
There are a lot of Q5_K, Q6_K, etc tensors. Only the routed experts (ffn_gate_exps.weight, ffn_up_exps.weight, ffn_down_exps.weight) are heavily quantized, and it looks like the down_proj is actually iq3_xxs for this model.