Distillation is a specific named technique from a pretty famous paper "Distilling the Knowledge in a Neural Network," by Hinton and others.
What makes it different from just training on any data is that you train the student model on "soft targets" that includes the full output distribution (logits) from the teacher model. Regular training uses one-hot targets and penalizes anything else; distillation will partly reward a student for getting in the distribution. This teaches the student to think like the teacher, not just imitate it.
Generating training data is not distillation, not in the technical sense, and I dislike Anthropic undermining the nomenclature to set a narrative.
Still debating whether these are actually information dense or not. Complexity
gives the appearance of density, but these sentences always come in whole pages at a time while often failing to deliver the needed information.
I wish people would stop using Anthropics incorrect use of the term distill. They don’t share logits so you can’t distill. You can generate training data, which doesn’t sound nearly so scary.
Way easier said than done, and hiding that behavior isn’t trivial, and huge waste of compute budget if it’s found and never used. Also not difficult to run in contained environments where it doesn’t have access to Internet to begin with.
Not impossible I agree, but seems like a really impractical way to ship a trojan while much weaker channels exist.
Yeah I recently downgraded my subscription. The paternalism is out of control. Secret weights, secret guardrails, secret stenography, secret dumbing-down of triggers, secret token allowances.
For anyone wondering what the actual purported security weaknesses are in this article (I used the slop machine to reduce the slop):
- Cloud backups — by default, backups to iCloud/Google Drive contain plaintext messages, and E2EE backup is opt-in. Even if you enable it, a weak password collapses the effective security, and any other person in the chat with an unencrypted backup exposes the conversation.
- Metadata — who you talk to, when, how often, IP, contact graph, etc. This is the "reading your life without reading your messages" argument, and it's the part that's genuinely well-established.
- Pen register / FBI — the claim that WhatsApp uniquely delivers near-real-time metadata (~every 15 min) to law enforcement.
- Group chat membership integrity — a server-level adversary can inject a member into a group; messages stay encrypted but get delivered to the injected party.
Endpoint compromise (Pegasus / CVE-2019-3568) — encryption is irrelevant if the device is owned.
- Closed source, Meta AI, business accounts — content can leave the E2EE envelope in those flows.
Nothing really new here, and as everyone else is pointing out Telegram might be worse.
I think it's important to multiply "likelihood of the human making a mistake" by "how many times the human must avoid a mistake". If your hobby presents 100 avoidable-but-life-threatening mistakes an hour, it's a dangerous hobby.
Absolutely. Imagine forking VSCode and then selling it for $60 BILLION in a few years. And right as you lose your competitive advantage to foundation providers. Amazing.
I mean there's also just sheer luck. Getting billions is probably more about luck than hard work OR corruption, even if those two things can increase your chances.
But luck is also not "earning" in most senses of the word.
I thought he might be onto something interesting when he began with the growth-rate example, since that's most-obviously tied to valuation. But he skipped over valuation entirely and decided to lecture a bunch of Oxford grads about middle-school math as if he were onto something profound.
I think the way network effects and valuation creates billionaires is a more earnest defense. But that would start to lean into sociology and luck more than the hard-work he must insist upon, so I guess we sweep that under the "exponential because hard work" rug too.
I think maybe a simpler explanation is that tech has been such a story of purportedly humble people becoming wildly successful. Classic rags to riches. Makes it easy to think of nerds as one of the common people, even the rich ones.