I am a huge fan of mm0, the thesis[0] is so brilliantly written, and MMC is such a great step towards the kind of verified computing I really want to be doing
Ahh I thought I was the only one! One line per sentence makes the diffs so much nicer too, maybe we need git hooks to reject multiple sentences per line?
I didn't find note-taking particularly useful until I started keeping everything in a single notebook with dated pages.
This worked a lot better than (for example) trying to organise notes by category - it's often easier to remember when you were working on something than how you categorised it, and once you know roughly when, you can find it by binary search
I saw a similar (I think!) paper "Grassmannian Optimization Drives Generalization in Overparameterized DNN" at OPT-ML at neurips last week[0]
This is a little outside my area, but I think the relevant part of that abstract is "Gradient-based optimization follows horizontal lifts across low-dimensional subspaces in the Grassmannian Gr(r, p), where r p is the rank of the Hessian at the optimum"
I think this question is super interesting though: why can massively overparametrised models can still generalise?
Which repos worked well? I've had the same experience as op- unhelpful diagrams and bad information hierarchy. But I'm curious to see examples of where it's produced good output!
Really nice! Had a quick read, here's my quick summary:
- Arrays are typed `S : D` with shape S and strides D
- Each of `S` and `D` is a nested tuple (instead of the flat tuples one typically sees in a tensor framework)
- Together `S` and `D` define the layout of a tensor
- Not every layout is "tractable", but the tractable ones form a nice category
A really good exposition, my only criticism is that it's quite front-heavy- it would be nice to see a detailed example like in 2.3.8 earlier in the document; there is a lot of detail presented early that doesn't seem necessary to understand the core ideas.
Last comment: I suspect there is a connection to strictification[0], would love to know more if the authors see this!
I only had a quick look, but it looks like they tweaked the state update so the model can be run with parallel scan instead of having to do it sequentially.
[0]: https://digama0.github.io/mm0/thesis.pdf