radarsat1

4,045 karmajoined 15 лет назад

Info: https://sinclairs.gitlab.io

Email: <myusername>@gmail.com

Submissions

My Calculator Is a Transformer

sinclairs.gitlab.io

4 pointsby radarsat12 месяца назад1 comments

comments

3 дня назадdiscuss

I can sympathize, the Spanish naming system makes things legitimately complicated sometimes. My wife has a Spanish style name Firstname(s) Fathername Mothername. But very very often she just goes by Firstname Fathername but it's not technically her legal name and is confusing in the non Spanish world because people assume otherwise that Mothername is her last name.

So for our son we decided to try to skip any confusion by doing what you are alluding to, and making her Fathername into his middle name, and giving him just my last name, in the English style.

And it worked, sort of, but then we discovered it was absolutely not legal to do that in my wife's country of origin. So kind of hilariously his registration in that country is: Firstname Mothername Fathername Mothername.

They forced us to repeat the middle name as part of his last name too. Just ridiculous. I thought at the least they would allow us to reverse them, but no.

5 дней назадdiscuss

You're of course right but I can't help but point out one thing..

> but c'mon: tomatoes originated in the Americas!

Yes. But they were brought to Europe like right after Columbus landed and have been part of Italian and European cooking for longer than the US has existed. So you only sort of have a point here. (Same for corn, eg. polenta )

I mean, tomatoes are definitely a big part of Italian cuisine, despite any differences that American Italians introduced to the menu.

https://en.wikipedia.org/wiki/Columbian_exchange

9 дней назадdiscuss

> The primary goal of this is support for old/obscure hardware with no LLVM/GCC support.

If you're going to go to all this effort for an old target though, wouldn't the effort be better spent on making it an LLVM target? Then you'd get Rust and a bunch of other languages for free.

But maybe there are required parts of the LLVM IR that make this undesirable for certain targets, maybe requiring specific hardware features, I don't know. I guess also WASM-as-IR is a possible way to go. (Is that a thing?)

Edit: sorry I see that this point was already raised in this thread by ivanjermakov. Ignore.

20 дней назадdiscuss

is it strictly necessary to use different models or can you get similar results by doing the same thing but just using eg Codex in different agents & persona? curious if you've compared this

25 дней назадdiscuss

if you don't like the article that's fine, but it gets really tiring reading this kind of side-tracked comment thread in like.. every post.

people use LLMs for writing. we know! get over it.. or don't... i don't really care.. but I'd rather read a discussion about the article contents and not the writing style.

this kind of comment is the new "discuss the font choice / background color / anything but what the article is actually saying."

2 месяца назадdiscuss

You can talk all you want about generalization and reasoning ability and AGI, but the fact is that it's also useful simply as a really user friendly database.

Even if it's only able to report facts from its dataset or perform simple synthesis of search results.

That it can actually reason to a certain extent is bonus points.

2 месяца назадdiscuss

Not everything is about making money.

Also I don't get the impression the idea is intended for "most musicians".

2 месяца назадdiscuss

> It's wildly better at tight and limited scope than large scale changes but even then I would rather code it myself.

One thing I would like to see is the use of LLMs for smarter semi-manual editing.

While programming I often need to make very similar changes in several places. If the instances are similar enough I can get away with recording a one-off keyboard macro to repeat, but if there are differences that are too difficult to handle this way I end up needing to do a lot of manual editing.

It would be nice to see LLMs tightly integrated into the editor so I can do a simple "place the cursor at things like this" based on an example or two. I'm sure more ideas for using LLMs more quickly perform semantic changes you intended are possible, instead of just prompting for a big diff. I feel there's a lot more innovation possible in this direction, where you're still "coding it yourself" but just faster.

2 месяца назадdiscuss

Recently I got interested in how to "compile" a program definition into the weights of a Transformer. I settled for distilling the MLPs individually, but the attention weights are fully "calculated".

The example program [1] generates a Transformer that executes an RPN expression, using "breadcrumb" tokens to track its progress. The output looks like:

  Prompt: 3 4 + EXEC
  Output: c2 c1 c0 7
  
  Prompt: 3 4 + 3 3 + * EXEC
  Output: c2 c1 c0 7 c5 c4 c3 6 c6 c5 c2 42
  
  Prompt: 10 2 3 * + 2 + EXEC
  Output: c3 c2 c1 6 c4 c3 c0 16 c6 c5 c4 18

I think there's still a lot that could be improved but I wanted to document what I have done so far. It turned out very interesting and made me think about transformers, attention and particularly the structure of the residual stream in a new way.

[1]: https://github.com/radarsat1/rpn_transformer/blob/main/src/p...

3 месяца назадdiscuss

you gave me the idea of using it to explore weird random scifi ideas, ended up spending way too much time clicking through details about the role of astrophage in the development of intelligence in deep sea life. Fun!

3 месяца назадdiscuss

Wow, love this idea.

3 месяца назадdiscuss

That was a fascinating read! Thanks for the link.

3 месяца назадdiscuss

This reminds me a lot of the tricks to turn BERT into a generative model. I guess the causal masking that keeps it to essentially be autoregressive is an important difference though. Kind of best of both worlds.

3 месяца назадdiscuss

Because the nature of transformers is that running a bunch of pregenerated tokens through them is a parallel operation, not autoregressive. That's how it works at training time, but speculative decoding uses it at inference time. So if you just want to check whether a set of known tokens is "likely" given the base model, you can run them all through and get probability distributions, no need to sample.

It's the same reason there's a difference in speed between "prompt processing" and "generation". The former is just taking the pre-generated prompt and building the KV cache, which is parallel, not autoregressive and therefore way faster.

3 месяца назадdiscuss

Consider also that even reattaching nerves that are supposed to be there is not exactly a walk in the park. Look into finger reattachment surgery and post operation care. Think pain, tingling, a year or more of physiotherapy.. and that's in the best case that it actually works and you don't end up with a "dead" finger. Now, imagine that for your whole body.

4 месяца назадdiscuss

> "Redefine" is a favorite word of AI. Honestly no need to read further.

You're not wrong, but it certainly is an annoying outcome of AI that we're not allowed to use.. words.. anymore.

4 месяца назадdiscuss

> Should you be counting on confusion of an underpowered text-merge to catch such problems?

This does not really follow from my statement.

I said that underpowered text merge should not silently accept such situations, not that it is the only way to catch them. It doesn't replace knowing something about what you are merging, but it is certainly a good hint that something may be wrong or unexpected.

> Post-merge syntax checks are better for that purpose.

Better, yes, but I was addressing semantic issues, not syntactical. I have seen syntactically valid merges result in semantic inconsistency, it does happen.

I do agree with your last statement.. unit & integration tests, agent checks or whathaveyou, these all contribute to semantic checking, which is a good thing.

Can they be relied on here? Maybe? I guess the jury is still out. My testing philosophy is "you can only test for what you think of testing". And tests and agent checks have a signal to noise ratio, and are only as useful as their SNR allows.

There is no guaranteed way to stop bugs from happening, if there were it likely would have been discovered by now. All we can do is take a layered approach to provide opportunities for them to get caught early. Removing one of those layers (merge conflicts) is not clearly a good thing, imho, but who knows.. if agent checks can replace it, then sure, I'm all for it.

4 месяца назадdiscuss

Is it a good thing to have merges that never fail? Often a merge failure indicates a semantic conflict, not just "two changes in the same place". You want to be aware of and forced to manually deal with such cases.

I assume the proposed system addresses it somehow but I don't see it in my quick read of this.

4 месяца назадdiscuss

Something I want to add to the discussion is that the only time I've encountered this was not with a specific company but with an "AI recruitment agency", which I'm seeing getting more and more popular.

And while I get the idea of an agency handling hiring, what bothered me is that the terms of the AI interview were that it was relatively standardized for a given role, and that they would record it and put it on file to show to other companies, with the selling point being: do well in one interview and we'll shop your profile around for you!

Which is.. great if you do well I guess, but.. really unsettling if you don't. I mean, there was zero information that you'd be able to do it over, no advance details of the format, no practice session. So if you fail, or stammer, or get surprised by some detail of some question.. what, you're just "on file" now, out of reach for their entire client portfolio?

At least if you're doing it one company at a time, you mess up, then ok you move on and try again somewhere else. But the idea of making some random mistake (which happens all the time!) just blacklists you for some unknown number of companies, forever..

No way, that's too high stakes. I noped out.

4 месяца назадdiscuss

> There are no details about training

my understanding was that they are not training at all, which would explain that. they are compiling an interpreter down to a VM that has the shape of a transformer.

ie they are calculating the transformer weights needed to execute the operations of the machine they are generating code for.