It is believed dense models cram many features into shared weights, making circuits hard to interpret.
Sparsity reduces that pressure by giving features more isolated space, so individual neurons are more likely to represent a single, interpretable concept.
The OpenAI fine-tuning api is pretty good - you need to label an evaluation benchmark anyway to systematically iterate on prompts and context, and it’s often creates good results if you give it a 50-100 examples, either beating frontier models or allowing a far cheaper and faster model to catch up.
It requires no local gpus, just creating a json and posting to OpenAI
One interesting anecdote about this bill was that the European Commission allegedly funded digital advertisements promoting it, targeting specific political demographics, which is something that could possibly be prohibited by their own regulations.
HHI is a pretty interesting metric. It’s calculated by taking each firm’s market share, squaring it, and summing across all firms.
This gives the probability that two randomly chosen customers belong to the same firm.
In one micro models of oligopoly, Cournot competition, it lines up directly with the markup firms can sustain.
Outside of theory, it’s an intuitive way to average together the market power of all firms, with increases in market share for bigger players being weighted more heavily.
E.g. map <think> -> THINK <user> -> USER <tool> -> TOOL
If they learn something specific in the chat finetuning stage, this might show LLM its user input text not these tag references.