fmap · HackerTrans

vor 24 Tagendiscuss

It's in Anthropic's best interest to focus the conversation on "distillation".

Imo the more interesting thing to focus on is that there are now several more labs with the expertise and capabilities to train trillion parameter models. That's a serious technical accomplishment and the main reason why open models are catching up to Anthropic and OpenAI (and local models are typically distillations of much larger models).

Who cares that they got some small amount of training data out of Claude. The crux is that the big US labs are not special, they just have a first mover advantage that's slowly shrinking as incremental progress becomes harder.

fmap

vor 27 Tagendiscuss

> The actual opposing argument is that it's impossible to create a billion dollar enterprise without a group effort

The other thing that we're often ignoring is that it's impossible to create a billion dollar enterprise without luck. You have to be at the right place at the right time.

For the most part only capital gets to roll the dice, but even before that it's a sign of the times that we take it seriously at all when people talk about "earning" a billion dollars. We could all do with a bit more humility.

fmap

vor 4 Monatendiscuss

I agree with most of what the author is saying, but the slogan that "a sufficiently detailed spec is code" can be misunderstood as "a sufficiently detailed spec is a program". The statement is only true if you read "code" as "statement in a formal language". Here's a (sketch of a) specification for a compiler:

> For every specification satisfied by the input program, the output program satisfies the same specification.

This is not a program and it does not become a program once you fill in the holes. Making the statement precise clearly requires a formal language, but that language can work at a higher level of abstraction than a programming language. So yes, a specification can absolutely be simpler than a program that implements it.

fmap

vor 10 Monatendiscuss

Apologies if this came across the wrong way. I really do want to know what the novel contributions of the post are, because the author implies that something about what they're doing is solving previously open problems:

> I figured out how to solve manifold Muon in the square case late last year, but I was unable to solve the full rectangular case and thus posed the problem as an open problem on the Modula docs. Jianlin Su solved the problem this summer

It sounds like the generalisation of projected gradient decent to "Muon" is what they're focusing on, but the derivation is all about the retraction map on the Stiefel manifold? I think I'm missing some background here.

fmap

vor 10 Monatendiscuss

Isn't this an old idea? E.g., here's a textbook on optimization algorithms for matrix manifolds https://press.princeton.edu/absil and here's a library that implements this in python for the Stiefel manifold that's the subject of this blog post: https://pymanopt.org/docs/stable/manifolds.html#module-pyman...

What is novel about the approach in the blog post? Serious question, I really can't tell after reading the post.

fmap

vor 9 Jahrendiscuss

I don't know, but Russel O'Connor got his PhD in 2008, the same year that type classes were introduced in Coq.

There has been a lot of development on the Coq proof assistant since then. I don't imagine that working with Coq in 2008 was particularly pleasant. :)

fmap

vor 9 Jahrendiscuss

There is an easy design choice - which Haskell didn't take - which would allow us to have transparent Identity functors, associative Compose and many others: Add a conversion rule to your language and don't eagerly expand definitions at the type level.

For example, in Coq you would write

  Definition Identity a := a.
  Definition Compose F G a := F (G a).

  Instance: Functor Identity. Proof[...]

and so on. This works, since explicit conversion during type checking allows you to associate type classes to otherwise transparent definitions.

Of course, this is no silver bullet and complicates other things. Everything works out beautifully though, if you combine conversion with bidirectional type checking, at the expense of less powerful type inference.

comments