Ahh good to know about those -builder repos. It would be awesome to publish the compilers as container images, could also publish the squashfs as an artifact to a registry. And it shouldn't be too hard to back-publish the existing tarballs as containers without rebuilding. Might poke around a bit later
Thank you! I'm also excited at the possibilities and today you can share "long links" (with all the data embedded in the URL's fragment) but in the future short links with storage on the server or loading from a gist for example would be another cool feature. And having a variety of interesting examples ready to go like you mention would be nice too. There's almost too much there that I was worried I would never ship and would still be playing with examples. For fun, here is the most basic version of compiler explorer https://programexplorer.org/#s=eyJjbWQiOiJzaCAvcnVuL3BlL2luc...
You can run any program that is in the container you select. Right now it ignores the entrypoint from the container config (though you could manually copy that into the Cmd field). Shell is a convenient program to run though so you can pass it a script to then run multiple commands. But you can run `date` directly for instance: https://programexplorer.org/#s=eyJjbWQiOiJkYXRlIiwic3RkaW4iO...
Yeah names are hard and it is just a working name. I'm simultaneously using containers as the distribution and runtime mechanism but also wanted them to be an implementation detail and not the spotlight
To add on since this took me a while to understand: for a single token, self attention is permutation invariant because we take the qK (one query dot all the other keys) weighted sum of all the values; that sum is what gives the invariance because + is commutative. But for all the tokens, the mha output matrix will not be invariant, but rather equivariant, where you apply the same permutation to the output matrix as you did to the input tokens. What might be a more useful example is to take one position, like the last one, and compute its mha for every permutation of the previous tokens; those will/should all be the same.
Great read and visuals. I think they typo'd the pun on basically/basisally. It got me thinking about program synthesis in the following scheme: data is embedded as vectors and program operations are metric tensors (or maybe just fields in general?) which tell the data how to move. Then, if you have an input/output pair we seek some program to move the data from input to output along some low energy path. Model a whole program as a time varying (t 0-1) metric tensor (is that a thing?) and optimize to find such an object. Maybe you choose ahead of time the number of operations you're searching over and these are like spline basis points and then you lerp between the metric tensors of each op; or you do it continuously and then somehow recover the operations. Then you want to find one program which satisfies multiple input/output pairs, ie one time varying metric tensor (or generally field) such that if you integrate from the input points they all end up at (or close to, which makes me think that you want some learned metric tensor for closeness) the output points. Right now I'm only thinking of unary ops with no constants, maybe the constants could be appended to the input data symbolically and you also get to optimize that portion of the input vectors, with the constraint that it is a shared parameter across all inputs.
surprised they didn't go straight into cloud-hypervisor, though I haven't actually tested with gpu yet but it is on my todo list. OCI layers can use zstd compression. I wonder if they are defeating layer sharing by splitting in 500 mb chunks. Lambda splits your image into chunks and shares at the block layer (I believe even same chunk different (user's?) container on a single host). Esp for 15 GB images I'd think using lazy pulling with nydus/stargz or whatever would be beneficial. I'd like to test out snapshotting, though my testing already boots a guest and runs a container in ~170ms; and I'm not actually sure how you write the guest init to signal it is ready for snapshotting and then wait properly (maybe you just sleep 1000?) so it resumes from the snapshot in a good state. I know fly has written about their use of snapshotting but I don't think it went into that detail. Cool stuff overall though, not worrying about locations and the yucky networking to do so seems nice
Wow thanks for the info! This was a good overview of one system https://academic.oup.com/mt/article/25/4/14/6815410 - 5um slice thickness at 0.7um per pixel and ~1 TB/cm^3, max sample size 50x50x20mm. Humans are about 75k cm^3 so a 75 PB scan isn't impossible. Probably start with a coarser scan and work on the compression algorithms. "Just" need a (much) bigger scanner. And figure out if the knife edge is okay with bones!
Bit late and I posted essentially this same comment on their site in hopes they'll see it, but this article jumped out at me because a while ago I was thinking about what it would take to get a whole organism single cell atlas so that you could explore it on the computer. The only thing I can really think of is to take a CNC machine, put it in a big freezer, then scan one layer of the specimen with a microscope and whatever multispectral lighting/imaging you need, then mill away a layer and repeat. Pathology slides are prepared with stains to enhance the contrast of membranes and other features which is probably not an option in the frozen state, so that's a big hurdle, but maybe you can do it with some fancy multispectral imaging. And if you can't, maybe you can at least start with lower resolution structures like anatomy (though I think there's still contrast challenges for that). I know there are some single cell atlas efforts for parts of or whole brains for mice and maybe other organs, but like the author's dream idea of getting order thousands of specimens to study variation, it would be great dataset if you could capture everything in one go.
Very nice writeup and I appreciate the effort put into showing the process. I got nerd sniped yesterday playing around with how to find the isle_opt.rs filepath from the core file and didn't succeed but left some notes on scripting with lldb here https://gist.github.com/aconz2/aef366a7b198b8ac151df147fec32...
Nice article, really like the survey bit. I know titles are just one tiny aspect, but they do set the tone, and in this case framing as a negative ("sad state") might put people off and may undermine the goal of getting people to adopt property-based testing. Though perhaps specifically this article is about stateful property-based testing, so it kinda comes off as negative against the existing library authors. I assume your intent is pure so rephrasing as "property-testing could be so much better" might be more positive. I am likely on the far end of sensitive on this only because I've spent a decade stuck in the negative view ("sad state" "everything sucks" "why did they design it that way" etc) and I'm trying to work my way out of it now. Anyways, too long rambling and mostly not even directed at the author necessarily but comment readers more generally.
As for how to get more adoption of property-based testing and/or stateful and/or parallel property-based testing, is there any way to do this from outside the language level? So you'd write your tests to consume commands and produce results from stdio, then an external process can drive the testing process. Can you run these in a VM to get deterministic thread scheduling for any language? Fakes could be implemented in any language and use the same stdio mechanism. And then build up a public set of known-good fakes. Ideally the fakes could be composable but not sure that is easy. Of course for bespoke things you'll need a bespoke fake, but I am thinking of the general stuff like a dictionary, queue, etc. This is all sparked because I've been thinking of how to build a giant inter-language inter-library testing database, initially with unit tests but this made me think about how to incorporate property based tests.
Late to the party, but I think my summary is (L is context length, C is hidden dimension, H is headsize, C = H * nh):
3.1 Optimised attention: Instead of using a learned W_V to project from C to H, slice V into H sized vectors. (V is just the input tokens X). This is because the matrix multiply is to a lower dimension anyway, so why not just slice. Slicing is just reshaping (L, C) -> (L, nh, H)
3.2 Efficient attention: I think this opens with a typo, "In the last section, we discussed how and why we can remove W_O..." should be W_V not W_O I think. Anyways, same as above, just for the keys this time. Reshape K (which is just X) from (L, C) -> (L, nh, H)
3.3 Super attention: Introduce an (L, L) W_A (lower triangular for masked) that transforms V on the left (X again) from (L, C) -> (L, C) (whereas standard attention has W_V (C, C) that transforms (L, C) -> (L, C) from the right). And they share W_A between heads.More efficient when C > L, so for long context models, probably not more efficient.
I think the first two modifications are equivalent to just setting W_V and W_K to the constant identity matrices right? So that makes me think what would happen if you instead restrict W_V (and/or W_K, W_Q) to be block diagonal (though non square) such that each head has in effect an (H, H) matrix which transforms the slice of X it receives. This is different than standard attention right? Because there the W_V acts over the full C dimension. Almost surely someone has thought of this so I will try to find out
Nice, this is really helpful to compare with minigpt in pytorch. I think I like the dimensions in the variable names. Kinda miss seeing the modules used in a module being setup in __init__ as in pytorch, but I get its all cool to jit. I prefer einsum to transpose etc for multi head so at least you only have one line to stare at, but that could be done in torch too (sidenote, I kinda wish einsum let you have longer names so it could be something like `... Lq H Dh, ... Lk H Dh -> ... H Lq Lk`). Jax is more modular in libraries so then you gotta get up to speed on linen, optax, orbax ... but those are usual tradeoffs. Why are the commit messages removed?
very cool, love the website design and the blog posts had a lot of interesting bits, like the backplate removal jig. I was kinda surprised to see an fpga in there but makes sense. Could that design be repurposed as a general image sensor -> sd card module or is it pretty specific to the sensor?
Sounds cool. Does the DRAM exist both in the SPU (4 kelvin) area and 77 kelvin area? This paragraph is a bit confusing:
> We call it a superconductor processing unit (SPU), with embedded superconducting SRAM, DRAM memory stacks, and switches, all interconnected on silicon interposer ... Next, there is a glass bridge to a warmer area, a balmy 77 K that hosts the DRAM.
Having 20 exaflops in a 20x20x12 cm volume is cool, but aren't you going to need a memory bandwidth close (factor 10 - 1000 lets say depending on arithmetic intensity) to that to be useful? And total memory capacity as well. I feel like the bytes/second/area (bandwidth flux) would be the limiting factor to make use of that compute density
> Recent multimodal foundation models are very widely adopted but still model different modalities separately, often using modality specific encoders or decoder
Is this accurate? I thought for example gemini pro used image tokens and gpt4-o similar
> without the need for separate image/text encoders
but then they say they pre-trained two different tokenizers, so maybe they just mean that the tokens go into the same attention layer? But then I thought that is how all the multi-modal stuff was happening already?
Very interesting, do wish we could play with it since it is a show hn, maybe repost when it is open sourced/usable. Looking forward to reading about how it works. How does it know which holds are on? How does it know what the top is? Are those inputs or inferred or is it just end to end completing likely body positions one after the other?
I have been playing around in pytorch with an a770 16GB card and hit this error. The response seems to be https://github.com/intel/intel-extension-for-pytorch/issues/... that larger than 4gb allocations aren't supported even though the card is 16gb. I haven't seen a ton of stuff on intel arc for machine learning so wanted to share my experience