The readme has strong LLM smells. Was the code written by an LLM as well?
What is your experience with cryptographic engineering, in particular avoiding common implementation pitfalls that bite first-time implementers of cryptographic primitives?
Are the primitives tested against Wycheproof vectors, and proofed against the common implementation mistakes they document?
This is probably just a difference in how your brain and the author's work. A variety of home organizers have told me that people mean different things when they say they want an organized space. Some people want everything precisely labeled and sorted into narrow categories, and hidden away in drawers or closets. Others want everything visible and coarser categories. Each system looks and feels very distressing to brains of the other types.
It's especially a problem for people with ADHD, because the "very sorted and hidden" mode of organizing is heavily socialized as the _only_ way to be organized, but it's also the exact opposite of how (some) ADHD brains want to operate. OTOH the very exposed and "emergent" organization that works for an ADHD brain probably is mild torture to an OCD brain :)
For myself, the sorting system in this post looks pretty ideal. All the stuff is right there where I can see it and scan for what I'm after, it explicitly allows for emergent organization where classification happens incrementally over time, and the dots thing has near zero activation energy but still gives me long-term information I can use. It's much better than an electronic or "clean" inventory system precisely because I'll never be able to consistently keep using those, whereas slapping a dot on a box, even on bad brain days I can manage that!
Strictly speaking: no. The "forward pass" terminology does not imply that there exists a "reverse pass" that does the same kind of computation. Rather, it's describing two different kinds of computation, and the direction they occur in.
The forward pass is propagating from inputs to outputs, computing the thing the model was trained for. The reverse/backwards pass is propagating from outputs back to inputs, but it's calculating the gradients of parameters for training (rougly: how much changing each parameter in isolation affects the output, and whether it makes the output closer to the desired training output). The result of the "reverse pass" isn't a set of inputs, but a set of annotations on the model's parameters that guide their adjustment.
The computations of the forward pass are not trivially reversible (e.g. they include additions, which destroys information about the operand values). As a sibling thread points out, you can still probabilistically explore what inputs _could_ produce a given output, and get some information back that way, but it's a lossy process.
And of course, you could train a "reverse" model, one that predicts the prefix of a sequence given a suffix (trivially: it's the same suffix prediction problem, but you train it on reversed sequences). But that would be a separate model trained from scratch on that task, and in that model the prefix prediction would be its forward pass.
Both, with caveats. The attention computation is fundamentally quadratic: for every token in the sequence, you're doing a computation that has to compute over every other token in the sequence. So it's O(N) per token, O(N^2) for the whole sequence.
The big mitigation for this is that in causal transformers (i.e. all the chatbot type applications, where each token is only allowed to see tokens before it), you're running inference repeatedly on the same prefix in order to grow it by one token at a time. So if you cache the computations for tokens 0..N-1, on each inference pass you only have to compute O(N) for the newly added token at the end of the sequence.
That's why caching (and caching charges) appear so prominently everywhere in the pricing of inference.
In practice, caching is most beneficial at inference time, because you typically have relatively long conversations that start with the same cacheable prefix (the system prompt). At training time the same optimization can apply, but you're typically not pushing the same prefixes through the model repeatedly so you end up paying the quadratic cost more often.
The quadratic cost of attention is the fundamental compute bottleneck for transformer architectures, which is why there's research like this trying to find shortcuts in computing attention, as well as research into completely new primitives to replace attention (e.g. SSM, which is O(N) on a cold cache and O(1) on a warm cache).
I don't know about all hyperscalers, but I have knowledge of one of them that has a large enough fleet of atomic frequency standards to warrant dedicated engineering. Several dozen frequency standards at least, possibly low hundreds. Definitely not one per machine, but also not just one per datacenter.
As you say, the goal is to keep the system clocks on the server fleet tightly aligned, to enable things like TrueTime. But also to have sufficient redundancy and long enough holdover in the absence of GNSS (usually due to hardware or firmware failure on the GNSS receivers) that the likelihood of violating the SLA on global time uncertainty is vanishingly small.
The "global" part is what pushes towards having higher end frequency standards, they want to be able to freewheel for O(days) while maintaining low global uncertainty. Drifting a little from external timescales in that scenario is fine, as long as all their machines drift together as an ensemble.
The deployment I know of was originally rubidium frequency standards disciplined by GNSS, but later that got upgraded to cesium standards to increase accuracy and holdover performance. Likely using an "industrial grade" cesium standard that's fairly readily available, very good but not in the same league as the stuff NIST operates.
I wrote MetalLB, a bare metal load-balancer for Kubernetes, because I needed one for myself. It gained some popularity because for a couple years, it was the only way to get working L4 LB outside of clouds. These days I believe a couple of the CNIs added support for external BGP peering and integration with k8s's LB machinery, but that came years later.
As a result, I became network troubleshooting tech support for a large chunk of people trying to run kubernetes on bare metal. If you've not looked at k8s's networking, debugging even your own cluster's networking is a nightmare, never mind debugging someone else's over slack, while (usually) simultaneously having to give them a crash course in intermediate/advanced networking stuff like asymmetric routing and tracing packets through netfilter so that you can then tell them that networks can't do the thing they wanted and no amount of new features I can add will change that.
Meanwhile companies selling bare metal k8s services started bundling MetalLB, but kept sending their customers to my bugtracker instead of taking some of the load themselves.
The experience burned me out severely. It's been several years and I still have a visceral negative reaction to the idea of open-sourcing code I wrote, and when I infrequently do they come with a fairly blunt "contributions no welcome" message and a disabled issue tracker. I handed over the keys to MetalLB a long while back now. I hope the new maintainers and the project are doing okay.
I'll mention a positive of that time as well, to balance it out: as an experiment I opened a pinned issue asking happy users to drop me a note (https://github.com/metallb/metallb/issues/5), and many did. It was nice occasionally getting a notification that wasn't a complaint or demand for support. At one point someone left me a note that it was being used to support research projects at NASA JPL and DARPA. That was pretty neat.
Our epic treatise on how NAT traversal works (in general, not specific to Tailscale) mentions this. IPv6 greatly reduces the amount of pain for p2p connections, but does not eliminate some of the fundamentals (stateful firewall traversal) if you want it to be zero-config: https://tailscale.com/blog/how-nat-traversal-works/
But until deployment hits 100%, and until ISPs start caring about IPv6 reliability the way they do about IPv4, "just use IPv6" can't be your answer. It's lovely when it works, but you need to do something other than "give up" when it doesn't. (also, as long as the internet is dual-stacked, doing IPv6 right also implies figuring out if NAT64 is in play, and wielding it correctly; so arguably IPv6 adds more complexity to the overall story, for now :) )
AUR package maintainer here (also tailscale employee). The poor experience was definitely on me. Relaynode's initial setup flow is a bit weird, and I didn't make the package explain anything. I think you were the first user of the Arch package other than me, so you got to experience the fun :)
Next release will have a better daemon, with a more typical setup flow. If you want to test drive it, `tailscale-unstable-bin` is the AUR package for it.
What is your experience with cryptographic engineering, in particular avoiding common implementation pitfalls that bite first-time implementers of cryptographic primitives?
Are the primitives tested against Wycheproof vectors, and proofed against the common implementation mistakes they document?