InkCanon

848 karmajoined vor 2 Jahren

comments

vor 4 Tagendiscuss

They phrase it in a confusing way, with a lot of fluff. But to my understanding what happens is 1. At some part in the transformer, calculate the Jacobian (matrix gradients for that forward pass). Do this for 1000 prompts and average them. 2. For another pass take an "activation vector" (the input vector x to that part). Apply the matrix in step 1 to this vector. 3. Unembed the output to get a token. Presumably to make this have meaning, they use the reading of past tokens after embedding, instead of a random vector in the middle of the transformer. The high level idea appears to be you're trying to find which token, if the value were changed slightly, creates the largest downstream change.

vor 12 Tagendiscuss

Completely stopping people from using open source models is impossible, but if it became a federal law there could be incredible pressure exerted on US companies. For example, OpenRouter might be compelled to not accept open source requests on threat of fines.

vor 12 Tagendiscuss

Paraphrasing him, he says that open source software which tool tens or hundreds of millions to build should be classified as a different category. And the implication being they should be banned. Why?

vor 15 Tagendiscuss

I feel obliged to point out the disingenuousness of what this post says and the post it quotes. The most egregious parts are payment fraud and reselling - these are speculated but not actually said to be known to be happening, which you have left out.

1. Claims of payment fraud. I actually clicked the BBC article linked about payments fraud it referred to. It was an article about a criminal syndicate stealing credit cards. It mentions buying cryptocurrencies, AI API purchases are not mentioned.

2. The claims that they are reselling the chats to AI labs. The post you cite is speculating it could be, but this is unverified.

The claims of reselling is also bizzare. Arbitrary user prompts are low quality data. If I were an AI lab, wouldn't I just pay for the API proxy and get targeted output for far less?

Also calling it bot accounts is a stretch. Bots mimick human input. These are proxy accounts.

vor 15 Tagendiscuss

The openness of AI is currently being held up only by Chinese companies (previously Meta, but they stopped). They're not saints, but there is not even a question in the open weight/HF community that the immense mass of Chinese talent, knowledge and resources are the only thing stopping a monopoly/duopoly from forming. In a very Cathedral vs Bazaar-esque way, China severely lacks compute, but are extremely ingenious in coming up with new optimizations, architectures, etc, which they all detail in their papers.

vor 15 Tagendiscuss

"You're trying to kidnap what I've rightfully stolen."

vor 15 Tagendiscuss

I'd hazard anthropic perceives their number 1 enemy as open weight models. If alternatives to their business exist (which is mainly coding tokens currently), they will get into a nasty fight and the nightmare of all tech companies - losing their monopoly. It threatens their ability to extract value, and could reduce their valuation to a tenth of what it was. They cannot make open weight models worse, so they're using lawfare to try and get them banned. And of course we previously know they attempted to block Chinese companies under the guise of national security by lobbying for restricting GPU sales.

vor 16 Tagendiscuss

Anthropic is master of Newspeak (see previously bugs -> vulnerabilities wrt Mythos). Distillation violates their terms of service, which is a civil offense, not a criminal one. It is not illicit, illegal nor breaks any laws.

vor 20 Tagendiscuss

Is that a Unitree?

vor 28 Tagendiscuss

I'm confused, is the pitch here to train (to the extent that using an LLM needs training) like Red Cross volunteers to use Claude?

vor 29 Tagendiscuss

The checking has nothing to do with AI, despite the (massively funded) marketing done to make you think so. It is based on formal methods/theorem provers.

vor 2 Monatendiscuss

Also, slightly stretching the definition of terms consecutively, so the multiplicative meaning is really far from the truth. For example, 271 vulnerabilities were really mostly bugs - generally incorrect states, but which almost never led to any exploit.

vor 2 Monatendiscuss

Wait till you hear what Rocq was originally called!

vor 3 Monatendiscuss

The Stinger is an anti air weapon, the Javelin is an anti tank weapon.

vor 3 Monatendiscuss

The field is massively hampered by the wishful mnemonics and anthropomorphization of LLMs. For example, even the hallucination idea arbitrarily assigns human semantics to LLM results. By the actual mathematical principles by which LLMs work, any hallucination is another output, with no clear definition between it and every other output.

vor 3 Monatendiscuss

The more accurate version is only Chinese companies (plus Facebook briefly) really open source their frontier models. The rest are non frontier. They are either older or specialized for something.

vor 3 Monatendiscuss

I do think there's value in science communication, but it does take an intelligent understanding of it on a case by case basis as to whether it's genuine or hype marketing.

Side note: talking to someone from such a "elite" university, I discovered many labs in these unis have standing orders by PIs to tweet their papers/preprints when published. Varies by field, in AI it is by far the most common.

vor 3 Monatendiscuss

I strongly disagree with the claim that it's a phenomenal paper on exploits, the exploits themselves are nowhere near significant in the cybersecurity research sense. It's saying that implementations of these benchmarks has exploits on the way they conduct their tests. It doesn't discover that current LLMs are doing it (they highlighted several other exploits in the past), they only say it's a possible way they could cheat. It's a bit like they've discovered how to hack your codeforces score.

What they claim as exploits is also deeply baffling. Like the one where they say if you exploit the system binaries to write a curl wrapper, you can download the answers. This is technically true, but it is an extremely trivial statement that if you have elevated system privileges, you can change the outputs of programs running on it.

I'm actually deeply confused about why this is a paper. This feels like it should be an issue on GitHub. If I were being blunt, I'd say they are trying really hard to make a grand claim about how benchmarks are bad, when all they've done is essentially discovered several misconfigured interfaces and website exploits.

vor 3 Monatendiscuss

Often at the start yes. So the students gets a bit of recognition, a bit of experience and a bit of knowledge.

vor 3 Monatendiscuss

Yes, and it's a very interesting use case for Wasm. Firefox has a sandbox called RLbox built on this, and has been published in a few papers.

Performance is one benefit, but the real killer feature is Wasm's guarantees are incredibly strong and formally proved. So by definition, you won't get out of bounds memory reads, memory corruption etc, assuming the implementation is correct. And because of the thorough specification, these kinds of exploits are far rarer in wasm runtimes.

https://hacks.mozilla.org/2020/02/securing-firefox-with-weba...