andyk

I hope the competition will inspire people to make breakthroughs in the open, so I won't take any rights to the IP, instead the winning solutions must use open source code and open weight models.

andyk

2 anni fadiscuss

yes the prize money is from me to the winners

andyk

2 anni fadiscuss

That has a double meaning - half tongue in cheek.

1) since we are creating a contamination-free version of SWE-bench (i.e. scraping a new test set after submissions are frozen) it is guaranteed that agents in this contest can't "cheat", i.e., models can't have trained on the benchmark / agents cant memorize answers.

2) as a general rule in life, don't cheat on things (not that there aren't exceptions)

andyk

2 anni fadiscuss

(reposting from locallama and lower down here) yep that's true.

one of my goals is to inspire and honor those that work on open source AI. Those people tend to be motivated by things like impact and the excitement of being part of something big. i know that's how i always feel when i'm around Berkeley and get to meet or work with OG BSD hackers or the people who helped invent core internet protocols.

those people are doing this kind of OSS work and sharing it with the world anyway, without any cash prize. i think of this as a sort of thank you gift for them. and also a way to maybe convince a few people to explore that path who might not have otherwise.

andyk

2 anni fadiscuss

andy here - happy to answer questions.

Also, I answered a bunch of questions yesterday on LocalLLaMA that people here might find interesting https://www.reddit.com/r/LocalLLaMA/comments/1hdfng5/ill_giv...

andyk

2 anni fadiscuss

yeah i agree. one of my goals is to inspire and honor those that work on open source AI.

people who work on open source tend to be motivated by things like impact and the excitement of being part of something bigger than themselves - at least that's how i always feel when i'm around Berkeley and get to meet or work with OG BSD hackers and people who helped invent core internet protocols or the guys who invented RISC or more recently RISC-V

those people are going to do this kind of OSS work and share it with the world anyway, without any cash prize. i think of this as a sort of thank you gift for them. and also a way to maybe convince a few people to explore that path who might not have otherwise.

andyk

2 anni fadiscuss

I tweeted this on stage at NeurIPS on weds: "I'll give $1M to the first open source AI that gets 90% on this sweet new contamination-free version of SWE-bench - http://kprize.ai" - K Prize is a new kaggle competition

andyk

2 anni fadiscuss

done! in the "Alternatives and related projects" section I just added to the ht readme -- https://github.com/andyk/ht/blob/main/README.md#alternatives...

andyk

2 anni fadiscuss

Sorry, my wording wasn't very clear. I wasn't trying to imply that ht is more geared towards scripting than `expect` (in fact I'd say `expect` is more scripting-oriented being an extension of a scripting language) but rather that ht is more geared towards scripting the terminal as a UI than `expect`.

Am I wrong about that? (I may very well be since I haven't used `expect` before)

andyk

2 anni fadiscuss

I tried to contrast to `expect` in a couple of my other responses, but yeah this is my sense too after looking briefly at `expect` - that ht always transparently sets up a terminal for you under the hood and you interact with that so you can always grab a screenshot of any terminal UI.

I don’t think `expect` is targeted at this use case (though I am only learning about `expect` right now so could be wrong)

andyk

2 anni fadiscuss

andyk here. it's clear our readme is lacking use cases! adding some now. When we introduced ht on twitter I gave a little more context -- https://x.com/andykonwinski/status/1796589953205584234 -- but that should have been in the project readme.

Also a few people comparing to `expect`. I haven't used `expect` before, but it looks very cool. Their docs/readme seem only slightly more fleshed out than ours :-D Looks like the main way to use expect is via:

  spawn ...
  expect ...
  send ...
  expect ...
  etc.

so, the expect syntax seems targeted more towards testing where you simultaneously get the output from the underlying binary and then check if it's what you expect (thus the name I guess). I can't see if there is a way to just get the current terminal "view" (aka text screenshot) via an expect command?

ht is more geared towards scripting (or otherwise programmatically accessing) the terminal as a UI (aka Terminal UI). So ht always runs a terminal for you and gives you access to the current terminal state. Need to try out expect myself, but from what I can tell, it doesn't seem to always transparently run a Terminal for you.

There might already be some other existing tool that overlaps with the ht functionality, but we couldn't find it when looked around a bunch before building ht.

andyk

2 anni fadiscuss

thanks for surfacing `expect` to our attention. I'll add a compare/contrast to the ht readme

andyk

2 anni fadiscuss

Interesting. When we decided to build ht we didn't compare it to expect (which I hadn't heard of or used) but I'm comparing the two now as they seem related.

How exactly did `expect` fall over?

From what I can tell, expect does not provide the functionality of a stateful terminal server/client under the hood for you so it isn't as easy to grab "text" screenshots of a Terminal User Interface, which is one of the main motivations behind ht (will update the readme to make this main use-case more clear)

andyk

2 anni fadiscuss

Include in your input json the ascii control character that the keyboard combo would generate (e.g., \x03 for ctrl-c).

To send control-c to the terminal, for example, you'd send the following JSON message to ht:

  { "type": "input", "payload": "\x03" }

andyk

2 anni fadiscuss

Yep this was my main reason for wanting it, though lots of other good ideas in these HN comments.

andyk

2 anni fadiscuss

I shared the motivating use case for why Marcin and I built this (LLM agents using terminals) in a diff comment but I’ll also expand the readme to give examples of use cases.

andyk

2 anni fadiscuss

Oh this is cool. I looked at using tmux before we built ht because I’ve used screen and tmux forever. I didn’t find libtmux though. Will def check it out.

andyk

2 anni fadiscuss

Hey, project lead here. I had a very specific use case in mind: I’m playing with using LLM agent frameworks for software engineering - like MemGPT, swe-agent, Langchain and my own hobby project called headlong (https://github.com/andyk/headlong). Headlong is focused on making it easy for a human to edit the thought history of an agent via a webapp. The longer term goal of headlong is collecting large-ish human curated datasets that intermix actions/observations/inner-thoughts and then use those data to fine-tune models to see if we can improve their reasoning.

While working on headlong I tried out and implemented a variety of ‘tools’ (i.e., functions) like editFile(), findFile(), sendText(), checkTime(), searchWeb(), etc., which the agents call using LLM function calling.

A bunch of these ended up being functions that interacted with an underlying terminal. This is similar to how swe-agent works actually.

But I figured instead of writing a bunch of functions that sit between the LLM and the terminal, maybe let the LLM use a terminal more like a human does, i.e., by “typing” input into it and looking at snapshots of the current state of it. Needed a way to get those stateful text snapshots though.

I first tried using tmux and also looked to see if any existing libs provide the same functionality. Couldn’t find anything so teamed up with Marcin to design and make ht.

playing with the agent using the terminal directly has evolved into a hypothesis that I’ve been exploring: the terminal may be the “one tool to rule them all” - i.e., if an agent learns to use a terminal well it can do most of what humans do on our computers. Or maybe terminal + browser are the “two tools to rule them all”?

Not sure how useful ht will be for other use cases, but maybe!

andyk

2 anni fadiscuss

I just read that article and enjoyed it. Thanks for sharing! I don’t think the author was arguing biological processes can’t be reverse engineered, but rather that the tools and approaches typically used by biology researchers may not be as effective as tools and approaches used by engineers.

Submissions

Pplx and Dbrx founder giving $1M to first OSS AI that gets 90% on SWE-bench

#1 agent on swe-bench wrote 7% of its own code

Headless Terminal – making terminals easy for LLMs to use

AI benchmarks should be like unit tests

OpenAI Dropped Work on New 'Arrakis' AI Model in Rare Setback

comments