gpugreg · HackerTrans

Anthropic probably trained Mythos on their own code and found that it is too got at reproducing it.

Some suits with no understanding of how LLMs work are scared that the models might hack them, or believe that they'd have to send data to China because they do not know that open models can be run on your own infra.

gpugreg

先月議論

Not as far as I can tell. Are we seeing different things?

For deepseek-v4-pro:

- $0.350 in, $0.003000 cache, $0.80 out https://crof.ai/pricing

- $0.435 in, $0.003625 cache, $0.87 out https://api-docs.deepseek.com/quick_start/pricing

gpugreg

先月議論

Those were amazing times. You could vibe code an entire prototype in seconds (200 tps). With Qwen3.6-35B-A3B and MTP, you can program at that speed on a single GPU at home now, but Kimi K2 is of course much smarter at almost 30 times the size.

I'm also looking forward for the Cerebras Kimi K2.6 release, which should be even better at 1000 tps. It is hard to overstate how important speed is for programming. Instead of having to wait for a few minutes until a task is done, it is just done instantly, and you don't have to context switch from whatever else you were working on while waiting.

I hope they will make it available to regular customers.

gpugreg

先月議論

Groq stopped serving Kimi K2 (1T params) when they got aquihired by NVIDIA, so I guess NVIDIA took most of the hardware in addition to the employees. The largest model they serve now is the relatively minuscule gpt-oss-120b.

The community support forum is also getting retired and there haven't been any posts by support employees in forever anyway, so they are probably gone, too. Also, the number of issues have been piling up, suggesting that the developers are gone as well. https://community.groq.com/c/forum/4 (archive link for when it goes down https://web.archive.org/web/20260602064050/https://community...)

To me, it looks they are trying to raise 650M with a few remaining (ancient) LPUs and no employees.

gpugreg

先月議論

For Anthropic, 5 minute caching costs 1.25x base input price and 1 hour costs 2x base input price. https://platform.claude.com/docs/en/about-claude/pricing#pro...

For OpenAI, it seems like you can't prolong the caching duration for money. Duration is longer during off-peak hours for in-memory caching and up to 24 hours for extended prompt caching. https://developers.openai.com/api/docs/guides/prompt-caching

For DeepSeek, caching duration of at least 12 hours (and likely longer) have been observed. Cache writes are free. https://zhuanlan.zhihu.com/p/2035737726952194774

gpugreg

先月議論

> The demo shows how every case gets successfully decoded without any hangups or a crash.

I am always baffled by the audacity of those LLMs to suggest that anything else would even be acceptable.

gpugreg

先月議論

> There exist a large number of people who are absolutely convinced that LLM providers are all running inference at a loss in order to capture the market and will drive the prices up sky high as soon as everyone is hooked.

> I think this is often a mental excuse for continuing to avoid engaging with this tech, in the hope that it will all go away.

Thanks for that psychological explanation. I was wondering why people were simply ignoring the math that shows that inference at API pricing can be quite profitable, e.g. published here for DeepSeek V3/R1 with 545% profitability: https://github.com/deepseek-ai/open-infra-index/blob/main/20...

gpugreg

2 か月前議論

Same here. LLMs are great at spitting out well-known solutions to problems instead of the best one. The "long tail" of solutions is usually lost due to how tokens are sampled from the LLM's probability distribution.

What I found to help a lot is to ask for e.g. 10 different solutions to a problem and then choosing one of them. Sometimes, this even leads to borderline creative solutions if there aren't 10 different ones.

gpugreg

2 か月前議論

> What's your source for Opus being a 5T model?

Elon Musk tweeted that Grok is 0.5T or 1/10th the size of Opus. https://xcancel.com/elonmusk/status/2042123561666855235#m

While this source's reliability is certainly debatable, the size matches the results of this paper, in which researchers estimated the parameter count from model knowledge. https://01.me/research/ikp/

gpugreg

2 か月前議論

Another factor is that DeepSeek is not just doing inference, but also training models, so they can use underutilized compute nodes for training during off-peak hours, as described in their DeepSeek v3 article: https://github.com/deepseek-ai/open-infra-index/blob/main/20...

But I agree that the main driver is that they are really good at optimizing. They will have chosen their architecture in such a way that it will be as efficient as possible on their own infrastructure, so they have a massive head start. Inference framework developers still have to catch up.

gpugreg

2 か月前議論

We can at least put an upper limit on it. From https://www.anthropic.com/glasswing

    Claude Mythos Preview will be available to participants at $25/$125 per million input/output tokens
    ...
    Anthropic is committing up to $100M in usage credits for Mythos Preview

Although I'd expect reduced prices for cached tokens, which is not mentioned on their website at this point in time.

gpugreg

2 か月前議論

Maybe the human brain also does other things besides interpolation?

gpugreg

2 か月前議論

> Please go run some numbers.

- DeepSeek serves DeepSeek V4 Pro at 27 tps: https://openrouter.ai/deepseek/deepseek-v4-pro

- At 27 tps per user, a B300 GPUS will give you around 800 tokens per second (serving 30 users): https://developer-blogs.nvidia.com/wp-content/uploads/2026/0...

- That's 800 * 60 * 60 generated tokens per hour, at a cost of $0.87 per 1M tokens, or $2.50 per hour.

- For input and output tokens, the math is a bit more complicated because we have to make assumptions about their ratio. Using the published values from OpenCode, we get another $2.50 for cached tokens (which are almost free for DeepSeek) and another $3.40 for input tokens (which are a lot cheaper to compute than output tokens), which gives us a total of $8.50 per hour per B300 GPU.

- B300 GPUs can be rented for as low as $3.40 per hour, which is less than $8.50, so hosting DeepSeek V4 Pro is profitable.

You could also host it at fewer tps per user to raise the efficiency and therefore the profit even higher.

gpugreg

2 か月前議論

I think I was using GitHub Copilot when I made the experience that led me to this statement. I guess the experience of using LLMs can be quite different depending on model version and harness.

gpugreg

2 か月前議論

> Uncensoring a model also doesn't necessarily improve generic use cases.

While the following is not a generic use case, I have a funny anecdote about how censorship is holding back flagship models.

I was asking an uncensored version of Qwen3.6 how a CLI option of llama.cpp worked, and to my horror and amazement, it rudely went and decompiled the binary to figure it out. It felt like the computer-equivalent of asking a vet why my dog looks sick, who then proceeds to cut it open to check. Flagship models usually do not do that without some convincing, but it sure is effective.

We will need much better sandboxes when less restricted models become more common. I can already see them hammering out 0-days when they are prompted to do some task that usually requires root.

gpugreg

2 か月前議論

Do your 20 year old university essays really fulfill all those criteria at once?

gpugreg

2 か月前議論

Sorry to say, but it almost certainly is AI.

- 51 EM-dashes

- Section headings

- Excessive repetitions: "The [...] are real. The [...] are real. The [...] is real. All three things are true at once."

- Excessive use of "genuine", "genuinely", "honest", "real", "true"

- Excessive use of "gap": "near-term gap", "the Compute Gap", "the Narrative Gap", "critical gap"

- Corny and meaningless closing sentence: "Understanding both parts is the beginning of taking AI deployment decisions seriously."

gpugreg

2 か月前議論

> Open weights will remain open only if they’re significantly worse than the frontier weights.

This makes the assumption that you earn more money by selling access to the model than by releasing the weights. That might be true for a company, but a US adversary might profit more from tanking the US economy. NVIDIA's stock dropped by 17% in a single day after DeepSeek-R1 was released, and the share of tech companies in the S&P 500 has only risen since then.

gpugreg

2 か月前議論

> Should we interpret this to mean that in the new world Windows is more resistant to attacks than say Linux.

LLMs can read assembly better than most, so probably not. But reality has never stopped people from trying to obfuscate.

コメント