Good question, I hope the answer doesn't disappoint you too much:
1. I made no benchmark comparisons to other existing projects
2. TRiP is CPU-only
3. the matmul kernel is not hand-optimized. I have some experience in this, and made several attempts, but could not achieve significant improvement against simple gcc13 -Ofast, so I decide to leave it readable, and just moved forward. The only optimization hint left is probably the directive to align allocs to the size of the cache line.
I considered adding flash attention, but CPU memory hierarchy does not benefit at the same level of GPUs.
I considered (shortly) using optimized libraries, but actually I got bad results, and still - that was not my main focus (learning the transformer architecture in the details).
This does not mean that TRiP is horribly slow! Keeping the kernel straight-forward, plus the alignment thing, should help the optimizer to make its fair use of unrolls, strides and vectorization.
If you have any suggestion to improve it (and for sure there's room for improvement), I'd be glad to get it, and if this does not complicate the things up to the point of messing up with the educational purpose, I could put it in!
Thank you for your interest!
It's not, except some utilities (json parser and picture handling). Please refer to the readme for full details. The main purpose was to understand the internals of transformer models by coding an engine from the ground up, so using AI to generate the machine learning related code would have made the whole project pointless. The most exciting moments: when I first ran successfully a decode; when I managed to fine tune a Gemma model by having it learn things about me; when Paligemma boxed correctly a bee in a picture I presented to it.
For sure Opus 4.7 is more chatty and talkative, I had to explicitly state a "be concise" preference in the settings. Is anyone experiencing some (very very rare) glitch in the output? Broken words, I mean. I'm using the WEB interface extensively, adaptive thinking ON, PRO plan.
don't miss the (formerly over-hyped) Google Research's Titans+Atlas thread, they are quite inspiring and informative on the subject; even if I don't think they released anything (open, at least) as a follow-up to their research.
https://arxiv.org/abs/2501.00663https://arxiv.org/abs/2505.23735
I remember a pic of a shirt, which a former colleague of mine was showing on his monitor ten years ago, and the office splitted half in recognizing the colours, can't remember if they were gold vs black; but we went crazy because nobody could believe that the other half was seeing a colour different from the one they were seeing!