Similarly curious about this. The intuition I extracted:
Let’s say we have 10 requests, where 9 of them take 1 second to complete but one that takes 100 seconds. The average time to complete a request is about 10 seconds, but if you experience the requests in series, at any given time you’re much more likely to sit and wait in one of those 100 second requests.
So if you imagine a long series of requests from this distribution and place yourself randomly in the series, the average time to completion is just a bit less than 50 seconds.
This is what is meant by t-weighted, that events with a large t take a larger place.
In this case it could be solved by not letting the LLM consume the transaction message. Effectively the same as preventing user supplied info going to the first argument of printf().
The transaction in question can remain opaque to the LLM and a %transaction.message% string is resolved in the layer between the LLM and the user.
This looks very interesting. Possible to get those rates without exotic hardware.
But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.
We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.
Feedback: I think it would be beneficial to clarify that there is a trial at the top of the page. Currently we have download and buy buttons next to each other, which I assumed meant that the software can be downloaded first, but will require a key you get from ”Buy”.
The images show Switch 1 controllers. Since the Switch 2 is already out, does this imply that the tracking doesn’t work on Switch 2 controllers?
I don’t like the 3 PC activation thing. It implies that the software needs to call home and will stop working if the company goes under. Or, at least, will not be possible to install anew.
Naively it seems odd that running multiple checks in parallel is faster than just running the autoregressive model multiple times in series. It’s the same amount of compute right?
But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. And so checking multiple tokens is cheap because we can batch and thus reuse the read weights for multiple tokens.
The verification step is similar to a prefill with a small batch size. The difference is what we do with the generated logits.
This is an amazing idea, and congrats on getting so far through it.
I personally would be wary about fire. Custom electronics without experience, and then putting (assuming here) high energy density batteries in a soft toy handled by little kids. Any accident will be absolutely disastrous. And when you scale up those very low probability failures are bound to happen.
How do you think about this, is this handled already?
This is very interesting and exciting, but IMHO the comparisons read as a bit disingenuous with the other models at 16 bit weights.
The 16 bit releases of the others models are not optimized for size, making it difficult to take the comparison seriously.
Would be interesting to see a comparison to quantized versions of the other models. If this model beats the others also in a fair comparison it gives more credibility to it.
Instead of anchoring the sun and thus noon at the top it would be interesting to have the sun move around the clock face as the year progresses. Noon then moves around as the year progresses.
”Up” could be said to point towards the center of the galaxy instead.
What they're saying is that the error for a vector increases with r, which is true.
Trivially, with r=0, the error is 0, regardless of how heavily the direction is quantized. Larger r means larger absolute error in the reconstructed vector.
This is cool. It makes storage of the KV cache much smaller, making it possible to keep more of it in fast memory.
Bandwidth-wise it is worse (more bytes accessed) to generate and do random recall on than the vanilla approach, and significantly worse than a quantized approach. That’s because the reference needs to be accessed.
I guess implied is that since the KV cache is smaller, the probability is higher that the parts it that are needed are in fast memory, and that bandwidth requirements of slow links is reduced, and performance goes up.
Would be interesting with a discussion about benefits/drawbacks of the approach. Ideally backed by data.
> Instead of expecting it to understand my requests, I almost always build tooling first to give us a shared language to discuss the project.
This is probably the key. I’ve found this to be true in general. Building simple tools that the model can use help frame the problem in a very useful way.
Tbh shrinking the image is probably the cheapest operation you can do that still lets every pixel influence the result. It’s just the average of all pixels, after suitable color conversion.
I think you’re implying that it would be useful to have the LLM predict the end of the speaker’s speech, and continue with its reply based on that.
If, when the speaker actually stops speaking, there is a match vs predicted, the response can be played without any latency.
Seems like an awesome approach! One could imagine doing this prediction for the K most likely threads simultaneously, subject by computer power available, and prune/branch as some threads become inaccurate.
Let’s say we have 10 requests, where 9 of them take 1 second to complete but one that takes 100 seconds. The average time to complete a request is about 10 seconds, but if you experience the requests in series, at any given time you’re much more likely to sit and wait in one of those 100 second requests.
So if you imagine a long series of requests from this distribution and place yourself randomly in the series, the average time to completion is just a bit less than 50 seconds.
This is what is meant by t-weighted, that events with a large t take a larger place.