Sorry, what? Passing a pointer is a matter of wrapping the value into the CPU register. OTOH passing an offset into a shared memory is a write to main memory so several magnitudes slower.
Obviously, transformers architecture is just one of the ingredients. Otherwise we wouldn't be seeing competing labs in this race. I also read all his interviews as a marketing material.
> Nah, coroutines/async/etc often lives in various threads (ie, the workers can schedule one of them on different threads during the lifetime and they live concurrently), so you still have all the issues of threading (+ new ones since things like thread-local variables aren't reliable if an async/coroutines moves threads between calls).
In cooperative scheduling how is it possible to have two coroutines running concurrently?
SIMD programmers have to be paid very well then in the China ... Jokes aside, some 2 or 3 years ago I thought that it is becoming inevitable for CPU designs to become an extended versions of their already quite capable vectorized execution engine units.
No, they don't force the model to use a json schema, they simply use the model to extract the data, and then they feed that OCR result into the pipeline further to evaluate the OCR results against the ground truth, and this is where JSON schema is used, and also another model (gpt4o).
Quite unlucky CPU to run the experiments. The article doesn't mention it but I hope that the measurement numbers OP got were extracted by re-running the experiment on same type of cores. Intel Core Ultra 7 255H is a mix of performance- (6x), efficient- (8x) and low-power (2x) cores.
I don't think you're wrong. Virtual functions is a two-pointer dereference operation (vptr, vtable[vptr]), and there we can have a d-cache miss but the main cost of using virtual functions is the increased likeliness of the i-cache miss. Cost of 30-60 cycles as per article assumes an icache-hit, and since virtual call is an indirect call (jump), it also heavily depends on the branch-target predictor and its buffer. I can easily imagine that iterating over a heterogeneous collection of objects would incur much larger cost than ~50 cycles/iteration. Branch target misprediction flushes the whole pipeline (15-20 cycles) and icache miss can easily end up being a fetch from main memory (200-300 cycles)
The article in general is interesting since it gives a rough idea of cost of operations relative one to each other but since CPUs are much more complex beasts it also gives us an incomplete picture, and if you're unaware of it the chance is that you will use it derive incomplete conclusions from it - understanding performance implications of a software running on an actual hardware is much more involved than what one article can fit.
You're missing the point that when you're building the product, and not using SBC just for fun, the image you are building, and yes even the userspace, is highly customized. Running a full-blown Linux distro is a non-requirement, and often incorrect.
I never tackled the reproducible builds topic but this is also something I had in mind while having a thought about it. Building object files for translation units, apart from macros introducing variance, should be reproducible no matter how high the number of compile jobs is running. So, isn't this largely a linking stage problem?
And what happens when E-core decoder encounters the AVX-512 instruction set? Does this mean that E-cores will have to implement some sort of (downsized) vectorized execution engine too? Otherwise, I don't see how this would work. Migrating on the HW core level would be too complicated.
And do you know about Linux? After cursory search I couldn't find anything as well. In this case it seems very expensive, if not impossible without the huge penalty, to implement such a workaround in the hardware.
Heterogeneous compute aware kernel scheduler seems like the most natural approach to me.
I don't think the open source is the differentiator here. I'm sure there's plenty of solid ideas. It would be interesting though if nlnet would open source the contributions they received in which case we would have a solid understanding how many of them have been AI or not