It's very unclear what's special in Rubin to be optimized for inference? I can see disaggregated bit (with having separate prefill and decoding nodes), but what else?
The core insight there is to separate value semantics (no identity) from reference (itself) semantics (nullability). While this particular change can bring very limited amount of improvements it’s still does some - probably smaller to no object header + more guaranteed optimizations for variables on stack.
It’s when they land next part (nullability) it will shine fully - particularly on the intersection of not null and value. Alternatively if they introduce tearable semantics it will also shine - it would be possible to still optimize array of value classes, even if they are nullable (for example by having correspondent nullability mask).
So they are taking right step in a right direction. They are just trying to land this incrementally.
DOJ puts an accusation with clarifying text in semi private document? They don’t do this, they do much worse things (and get, rightly, much worse response).
This document isn’t great, but comparing it to Trump administration actions isn’t great either. As well as focusing on it rather than on a substance of the article in question (which is, about Garry Tan accusations in a first place).
Thank you for your comment though it made me go back and reread the linked text more critically.
I somewhat agree that the whole document is sloppy, but I also think this conversation overemphasizes what is wrong with it. It is not as if the document simply accuses the journalist and leaves it there; it actually elaborates on what it means by “violated HIPAA” in that context (and please don’t make me start comparing it with current administration behavior).
There is also a broader question of how to properly handle what are, quite honestly, likely bad-faith actors — the journalist in this case. Should the office simply ignore the smearing campaign and the lies? Maybe. But this is already an issue even before accounting for the significant amount of money — pushed by a very small number of people — being spent against the attorney’s office.
This is what great reporting looks like: well-written, transparent, and rigorous. It’s sad to see how hatred toward progressives can distort people’s judgment.
It's unclear why the most probable next token given the context "please pick random number" won't be distributed uniformly across all the possible numbers (in the end it's totally possible for LLM return 10 logits of around same value for numbers 0..9 for example).
However billionaires don’t own tiny part of US wealth, more like 5%-10%. And top 1% (and grandparent was talking about rich people) own 1/3 of US wealth.
Total US wealth is ~170T so obviously it will be enough to cover federal and state government for a year (and more like 20 years).
Even considering obvious issue of wealth going down like crazy in such hypothetical scenario in its ends this would be enough. Because in the end it’s all part of same economy.
While I mostly agree with you, it worth noting modern llms are trained on 10-20-30T of tokens which is quite comparable to their size (especially given how compressible the data is)
It's not an issue of warmup time, it's an issue of jit compilation.
On my server (AMD EPYC 7252):
1) base time of the java program from the repo is 3.23s (which is ~2 worse than the one in linked page, so I assume my cpu is about 2 slower, and corresponding best c++ result will be ~450ms
2) if you count from inside of java program you get 3.17s (so about 60ms of overhead)
3) but if you run it 10 times (inside of same java program) you cut this time to 1570ms
It's still much slower than c++ version, but it's between rust and go. And this is not me optimizing something, it's only measuring things correctly.
update: running vector version of java code from same repo brings runtime to 392ms which is literally fastest out of all solutions including c++.
update2: ran c++ version on same hardware, it takes 400ms, so I would say it's fair to say c++ and vectorized java are on par (and given "allows vectorization" comment in cpp code I assume that's the best one can get out of it).
There might be more than one reason for an ongoing crisis, and different takes on who’s responsible. However Maduro is responsible of huge number of refugees fleeing Venezuela, and we (and some other countries around) have some obligations to help asylum seekers.