I love this idea, but have a hypothesis that 90% of agents that people actually use today would fail this test inadvertently (false negative).
Industry best practice + standard implementation for most agents right now is to do web browsing / fetching via subagents. Their output is summarized using a cheaper model and then passed back to the parent. It's very unlikely that without preserving the actual content the subagents see that the `CANARY-` strings would be found in the output.
Any thoughts on how you'd change the test structure with this in mind?
Anyone have thoughts or insights about why Lego is ending the partnership with FIRST? Thirty years is a long enough track record that it doesn't seem like an overnight decision...
Major League Hacking (MLH) is rethinking technical education for the modern world. Now that every company is a technology company, we need faster and more effective ways to train the talent that everyone depends on. Our unique hands-on, community-driven approach has helped more than 500,000 technologists worldwide launch their careers and stay up-to-date with the latest trends. If you're passionate about technology, education, and the future of work – this this the place for you!
As our Head of Engineering you’ll be responsible for building out a team of developers who will create and maintain engineering products that help aspiring software engineers level up and launch their careers. The software that you and your team write will power the largest community of early career developers in the world and help redefine the future of work & technical education.
Stack: Ruby on Rails, Next.js, PostgreSQL, Redis, Heroku
Industry best practice + standard implementation for most agents right now is to do web browsing / fetching via subagents. Their output is summarized using a cheaper model and then passed back to the parent. It's very unlikely that without preserving the actual content the subagents see that the `CANARY-` strings would be found in the output.
Any thoughts on how you'd change the test structure with this in mind?