Yes, disappointing given that Chronicle actually does have legit expertise here. (Chronicle Map is not very well-known even in the Java space but it's by far the best larger-than-memory Map available.)
That's a reasonable option, just be aware that you get about 1/3 as much memory bandwidth with the M5 Pro, or 2/3 with the M5 Max [now you're at $4100 for the lowest-end]. So both your prefill (flops-bound, M5 has a lot less) and decode (bw-bound) will be slower.
1. Yes, "gain" is a vanity metric but it's harmless, nobody is being "fooled" here.
2. This could be a problem in principle, sure, but unless you're actually vetting bug reports you're just spreading FUD.
3. Again, do you have any reason to believe that the thousands of devs using rtk are silently tanking their performance without noticing? here's a thought: instead of reporting that SOMEONE SHOULD MEASURE THIS, you could, you know, measure it yourself.
4. Good lord, what is this doing in a purportedly technical article?
5. Yes, this is inherent in the problem domain, again, nobody is being "fooled".
Yes, I'm grumpy; reading this article was a waste of time.
Bias: had my first RTK pr accepted today, so I guess I probably know more about it than this guy who got offended by "gain" and spit out the first thoughts that came to mind.
it is hard to understand what the actually meaningful innovations are here / what TileRT is bringing to the table.
- dflash: new-ish but February is ancient by the standards of the pace of AI innovation lately, I guess applying it to a 1T model is new-ish in the sense that the dflash researchers don't have the hw budget to prove that out
- persistent engine kernel: this is like CUDA 101
- warp specialization: I think this just means "keep different gpu resources all busy w/ pipelining" which is CUDA 201, some of it is even baked into pytorch now
- MXFP4 QAT: not new
- TileRT: hard to tell what this actually does, there's a PyPi wheel with support for DS 3.2 and GLM 5 but binary only
Because you need kv proportional to context length during inference of a single token to avoid quadratic recomputation. So compressing the kv lets you handle longer contexts in the same amount of vram.
Generically, I would say, just start building it and ask your favorite coding agent for advice when you get stuck. This is the first technology that can teach you how to use it! (But do ask a model with a recent knowledge cutoff, i.e. not gemini.)
As someone who has been writing harnesses for a year: the people at opencode etc aren't stupid, when they decide to break the prefix cache [usually partially] it's always because they've tested it and it gives better results overall.
If you think that dsv4 behaves differently enough from the aggregate of other models, submit a PR with a patch to special case that to your harness of choice with evidence. Just blindly assuming "append only all the time because cache" is a waste of everyone's time.
How should I update my simplistic understanding that decode is bw-bound with these results that show the B70 decoding faster than a 4090 (about 50% more bw)?
Brokk keeps LLMs on-task in million-line codebases by adding compiler-grade understanding of your code's structure and semantics.
Previously: author of JVector, co-founder of DataStax, founding project chair of Apache Cassandra.
Twitter: http://twitter.com/spyced