Something I’ve noticed in my own experimentations is that it’s really only limited by the length of its replies. It can happily generate perfect code but then stop in the middle of a function simply because the website only asked for 200 tokens or something. It’s really quite something to say that one of my gripes lies not in the model but in the user interface wrapping the model.
Exactly. As the OP of the blog, the amount of handholding I had to do for it to understand the syntax of an extremely tiny language was a lot. On the other hand, I’ve messed around with codex and other models before, and something about explaining in normal English, as though I was having a conversation rather than just listing some commands made it much easier. I’m excited not because of what exists right now, but because this shows so much promise even just 1 or 2 papers down the line :D
While what’s on the blog isn’t cherry picked, it often requires way more context than a human would to solve a problem. For instance I omitted the 100+ message back and forth where I explained the syntax of this extremely simple language.
Hi I wrote the op article. The previous prompts were me copy pasting verbatim 10 line chunks of the spec for my language: https://github.com/randomsoup/sack into the chat box, since the whole spec at once was longer than it would intake as an input.
Generally longer inputs can help reduce the amount of cherry picking you need. And of course there are many jailbreaks to get around no access to the internet. In this demo I actually didn’t use any! :>)