Our main argument is that outputs will become increasingly indistinguishable, but the processes won't. E.g. in 5 years if you watch an AI book a flight it will do it in a very non-human way, even if it gets the same flight you yourself would book.
This is a fair criticism we should've addressed. There's actually a nice study on this: Vong et al. (https://www.science.org/doi/10.1126/science.adi1374) hooked up a camera to a baby's head so it would get all the input data a baby gets. A model trained on this data learned some things babies do (eg word-object mappings), but not everything. However, this model couldn't actively manipulate the world in the way that a baby does and I think this is a big reason why humans can learn so quickly and efficiently.
That said, LLMs are still trained on significantly more data pretty much no matter how you look at it. E.g. a blind child might hear 10-15 million words by age 6 vs. trillions for LLMs.
The cross-tile challenges were quite robust - every model struggled with them, and we tried with several iterations of the prompt. I'm sure you could improve with specialized systems, but the models out-of-the-box definitely struggle with segmentation
We have an example of a failed cross-tile result in the article - the models seem like they're much better at detecting whether something is in an image vs. identifying the boundaries of those items. This probably has to do with how they're trained - if you train on descriptions/image pairs, I'm not sure how well that does at learning boundaries.
Reload are challenging because of how the agent-action loop works. But the models were pretty good at identifying when a tile contained an item.
Same! As we talk about in the article, the failures were less from raw model intelligence/ability than from challenges with timing and dynamic interfaces
While running this I looked at hundreds and hundreds of captchas. And I still get rejected on like 20% of them when I do them. I truly don't understand their algorithm lol
Roundtable is a research and deployment company building the proof-of-human layer in digital identity. Roundtable seeks to research and build real-world Turing Tests, and to productize these developments in bot detection, fraud prevention, and continuous authentication.
We're hiring a Member of Technical Staff. This is not a typical “Founding Engineer” position. In particular, this role will involve more research than similar positions. You should be prepared to spend around half of your time on research and half on engineering.
The ideal candidate has experience in both. However, we don’t strictly require research experience as long as you have a strong quantitative background and an eagerness to learn. We do require proficiency in both web development (Javascript, Node.js) and Python.
If you're interested shoot an email to [email protected] with your resume and a short blurb on why you're interested (note: we are unable to offer visa sponsorship at this time).