The catch is that it's not really doing anything. It "stores" 16GB of quants by... storing the base 2GB model and quantizing it on user's machine on demand.
The extra diff files are only there because the quantization is not fully reproducible*, and author's 3-line llama.cpp fix PR** supposedly fixes this. With it fixed, then in my understanding the author's tool is literally just a wrapper around llama-quantize.
* I also wonder if this even matters for quant quality. In fact, isn't it possible for contracted FMAs to be more numerically accurate (even if less correct from IEEE pov) than non-contracted math?
** I'd give the PR like 50% chances of not surviving the day, since AFAIK on llama.cpp, vibed PR descriptions usually end up rejected as a rule. Maybe the diff being super trivial could save it from this fate, dunno.
Also side observation, the last HN user to regularly mention b7r6 (with the strong implication that it's them) got banned here several months ago: https://news.ycombinator.com/item?id=47119245
The "dissertation" linked there (https://github.com/b7r6/cassandra-dissertation) is also incredibly interesting; looks like the HN user asked an LLM to prove/validate that they are "right" in their comments more often than not.
In general, these GH accounts and their repos/gists are kind of a rabbit hole.
> The GDPR is almost trivial to comply with if you’re not harvesting data willy-nilly.
I buy a VPS. I apt install nginx. Is it okay that by default, opening http://IP/index.html logs the IP address to /etc/log/nginx/access.log? Maybe yes, maybe no, maybe yes but I need a privacy policy (for an empty index.html). Maybe I need to ask a lawyer (who usually errs on side of caution) because people have been arguing about it for 10 years (and please don't answer here). And in the end, even if I didn't need to do anything, it sure is _some_ nonzero drain of my resources to have think about it at all (completely ignoring whether it's justified or not).
> Also, as you're using full double/f64-precision all the time, you're leaving a fair bit of performance on the table
There's another issue that popped up on my quick naive profiling run: std::shared_ptr<Material> in the HitRecord/HittableLightSample is assigned/copied and destroyed a lot, and somehow these refcount operations show up as half of all samples on my profile (presumably because even if there's no hit and the pointer stays nullptr, the smart pointer still must check if there's anything to deallocate).
Reading this leaves a weird taste in my mouth, since the author tends to regularly make nontrivial >1k LOC PRs (sometimes several per day) and merge them on the same day with no reviews at all. This is even ignoring the LLM aspect; I don't know what % of them are assisted, but even if it was 0%, this isn't the pace of development I'd be comfortable with.
I think this is 6.3.2.3.7 in C99 about casting between pointer types:
> If the resulting pointer is not correctly aligned for the
pointed-to type, the behavior is undefined.
However, unless I’m missing something, producing such a pointer from an integer is apparently not insta-UB? 6.3.2.2.5:
> An integer may be converted to any pointer type. Except as previously specified, the
result is implementation-defined, might not be correctly aligned, might not point to an
entity of the referenced type, and might be a trap representation
And later on 6.5.3.2.4:
> If an invalid value has been assigned to the pointer, the behavior of the unary * operator is undefined.
Which implies that the invalid pointer must have been obtained without being already undefined, right?
> First off, templates are the opposite of opaque due to the fundamental requirement that the implementation be visible to every translation unit using a template.
That's not strictly true, you can have an implementation hidden in a separate TU, as long as that TU instantiates the template for all template arguments that the users are going to use.
I wonder how often this happens in practice - by "this", I mean industry/LLM world not noticing* some research until a bigger player repeats it with louder PR.
At a glance, I see they do gather analytics about how much the app is used (model downloads, model invocations etc) without message content, pretty much just the model used.
It already has a fast path, from (I think) 3.11. If you run `object.x` repeatedly on the same type of object enough times, the interpreter will swap out the LOAD_ATTR opcode to `LOAD_ATTR_INSTANCE_VALUE` or `LOAD_ATTR_SLOT`, which only makes sure that the type is the same as before and loads the value from a specified offset, without doing a full lookup.
I also did some reading and experiments, so quickly talking about things I've found out re: refcount elimination:
Previously given an expression `c = a + b`, the compiler generated a sequence of two LOADs (that increment the inputs' refcounts), then BINARY_OP that adds the inputs and decrements the refcounts afterwards (possibly deallocating the inputs).
But if the optimizer can prove that the inputs definitely will have existing references after the addition finishes (like when `a` and `b` are local variables, or if they are immortals like `a+5`), then the entire incref/decref pair could be ignored. So in the new version, the DECREFs part of the BINARY_OP was split into separate uops, which are then possibly transformed into POP_TOP_NOP by the optimizer.
And I'm assuming that although normally splitting an op this much would usually cost some performance (as the compiler can't optimize them as well anymore), in this case it's usually worth it as the optimization almost always succeeds, and even if it doesn't, the uops are still generated in several variants for various TOS cache (which is basically registers) states so they still often codegen into just 1-2 opcodes on x86.
One thing I don't entirely understand, but that's super specific from my experiment, not sure if it's a bug or special case: I looked at tier2 traces for `for i in lst: (-i) + (-i)`, where `i` is an object of custom int-like class with overloaded methods (to control which optimizations happen). When its __neg__ returns a number, then I see a nice sequence of
_POP_TOP_INT_r32, _r21, _r10.
But when __neg__ returns a new instance of the int-like class, then it emits
_SPILL_OR_RELOAD_r31, _POP_TOP_r10, _SPILL_OR_RELOAD_r01, _POP_TOP_r10, etc.
Is there some specific reason why the "basic" pop is not specialized for TOS cache? Is it because it's the same opcode as in tier1, and it's just not worth it as it's optimized into specialized uops most of the time; or is it that it can't be optimized the same way because of the decref possibly calling user code?
That's not true. I mean: it's true that it has little to do with OOP, but most imperative languages (only exception I know is Rust) have the issue, it's not "Python specific". For example (https://godbolt.org/z/aobz9q7Y9):
struct S {
const int x;
int f() const;
};
int S::f() const {
int a = x;
printf("hello\n");
int b = x;
return a-b;
}
The compiler can't reuse 'x' unless it's able to prove that it definitely couldn't have changed during the `printf()` call - and it's unable to prove it. The member is loaded twice. C++ compilers can usually only prove it for trivial code with completely inlined functions that doesn't mutate any external state, or mutates in a definitely-not-aliasing way (strict aliasing). (and the `const` don't do any difference here at all)
In Python the difference is that it can basically never prove it at all.
I saw some eve-specific logic in Destiny repo, like warp enter condition and warp velocity math, or entity visibility between grids.
(Also, it’s full of std::(unordered_)map/set. Surprised they didn’t try squeeze some more perf there.)