throwdbaaway

20 часов назадdiscuss

If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.

throwdbaaway

4 дня назадdiscuss

That's exactly what I said. They do care when FLOPs are involved. Restoring an old session with 900k tokens will require a lot of FLOPs to reprocess the 900k token.

Meanwhile, they don't really care if you use hundreds of millions of cached input tokens, which doesn't consume any FLOP.

throwdbaaway

4 дня назадdiscuss

Different sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4, 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take less than 9GB of VRAM for 1M context, based on the numbers shared in https://arxiv.org/html/2606.19348v1.

throwdbaaway

4 дня назадdiscuss

Well I wouldn't call it a low bar, since some of the edits were quite complex. And 1M context in less than 6GB of VRAM is truly impressive, but somehow this gets way less attention than the crappy turbo quant from Google.

throwdbaaway

5 дней назадdiscuss

While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089

> The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for users. In an extreme case, if you had 900k tokens in your context window, then idled for an hour, then sent a message, that would be >900k tokens written to cache all at once, which would eat up a significant % of your rate limits, especially for Pro users.

Using the current Opus pricing, that pre-lunch 900k tokens should roughly consist of:

720k input tokens = 0.72 x $5 = $3.6

180k output tokens = 0.18 x $25 = $4.5

900k 1h cached writes = 0.9 x $10 = $9

500M cached input tokens = 500 x $0.5 = $250

$267.1 in total, with 93.6% from cached input tokens. The portion that requires GPU compute is about 3% of the total.

Post-lunch, the 900k tokens should consist of:

900k input tokens = 0.9 x $5 = $4.5

900k 1h cached writes = 0.9 x $10 = $9

So Anthropic is fine with the $267.1 accumulated over 3~4 hours before lunch, but not fine with the $13.5 incurred immediately after lunch. Why?

The only plausible explanation is that the actual cost of caching is way less than the API pricing. If you use a coding plan, Anthropic doesn't really care about your cached input tokens usage. Indeed they want you to show your ccusage screenshots. On the other hand, if you pay by API tokens, the margin is huge for cached input tokens.

Only when you do something that requires a lot of FLOPs, e.g. the post-lunch 900k input tokens, the cost becomes real.

throwdbaaway

5 дней назадdiscuss

Indeed they are all lossy. Not sure how much they contribute to the quality loss in long context though. I got a 700k session with DSV4 Pro (official API), and the model was still coherent and didn't make any tool call error.

throwdbaaway

5 дней назадdiscuss

The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion:

> That is, for your $100/month fee, you get $3600 equivalent of API usage. This is presumably because Anthropic has figured out some clever things to do with model routing and input caching, and also can subsidize with investor money and take a hit on their operating margins.

My take: this is exactly what Anthropic wants everyone to think. In reality, 90% of that $3600 are for cached input tokens, that can be made to cost next to nothing, as shown by DeepSeek.

throwdbaaway

5 дней назадdiscuss

Seems like a pretty pointless post that still centers around output tokens.

In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require GPU compute, and DeepSeek has shown that it can be done 50~100x cheaper with MLA/CSA/HCA, and a whole bunch of disks. This should collapse the margin.

throwdbaaway

7 дней назадdiscuss

And somehow they claimed that it is "lossless".

throwdbaaway

18 дней назадdiscuss

On ZFS with zstd compression, I am getting 1.34x compressratio for the BF16 weights (across multiple models).

Here's the du output for GLM-5.2:

    $ du -s -BG /cube/models/zai-org/GLM-5.2/
    1099G   /cube/models/zai-org/GLM-5.2/

throwdbaaway

2 месяца назадdiscuss

And their disk-based caching is amazing. I got a long 700k context session spanning more than a week, with pauses in between that was longer than a day, and some rewinds mixed in as well.

Stats from pi:

↑400k ↓438k R432M 71.9%/1.0M

Half a billion tokens, $2.12

throwdbaaway

2 месяца назадdiscuss

Hah, that's because the prompt itself was only about 30 tokens. We need a much bigger prompt to properly test PP.

throwdbaaway

3 месяца назадdiscuss

Huh that's not what I gathered from the tweet at all. If I am going to write a five why's analysis, the immediate cause is the LLM wrongly decided to delete a volume, while the root cause is the bad design to co-locate staging and production data in the same volume. The writing was quite vague though, let's wait for a response from railway.

throwdbaaway

3 месяца назадdiscuss

If I understand correctly, both the staging database and the production database share the same volume. Thus, production data was gone as well after deleting the volume.

1st hint - the API call only contains one volume:

    curl -X POST https://backboard.railway.app/graphql/v2 \
      -H "Authorization: Bearer [token]" \
      -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}'

2nd hint - this gem from the tweet:

> No "this volume contains production data, are you sure?"

throwdbaaway

3 месяца назадdiscuss

Should be about 10~20 GiB per session. Save/restore is exactly what DeepSeek does using its 3FS distributed filesystem: https://github.com/deepseek-ai/3fs#3-kvcache

With this much cheaper setup backed by disks, they can offer much better caching experience:

> Cache construction takes seconds. Once the cache is no longer in use, it will be automatically cleared, usually within a few hours to a few days.

throwdbaaway

3 месяца назадdiscuss

Based on the release schedule of 3.5 previously, my optimistic take is that they distill the small models from the 397B, and it is much faster to distill a sparse A3B model. Hopefully the other variants will be released in the coming days.

throwdbaaway

3 месяца назадdiscuss

His Vibe Coding book is invaluable as a textbook example of slop.

throwdbaaway

3 месяца назадdiscuss

https://github.com/anthropics/claude-code/issues/46829#issue... - Have you checked with your colleague? (and his AI, of course)

throwdbaaway

3 месяца назадdiscuss

> EC2 instances on shared hardware showed up to 30% variance between runs due to noisy neighbors.

Based on this finding, I suppose the better way is to rely on local hardware whenever possible?

throwdbaaway

3 месяца назадdiscuss

Very nice TG improvement from Flash Attention KQ fusion. Is it something that was already done in ik_llama.cpp? If not, then it will be a welcomed addition for hybrid CPU/GPU inference.

comments