Confirmed that mini uses ~30x more tokens than base gpt-4o using same image/same prompt: { completionTokens: 46, promptTokens: 14207, totalTokens: 14253 } vs. { completionTokens: 82, promptTokens: 465, totalTokens: 547 }.
Page 7 of their technical report [0] has a better apples to apples comparison. Why they choose to show apples to oranges on their landing page is odd to me.
We've been using Braintrust for evals at Zapier and it's been really great -- pumped to try out this proxy (which should be able to replace some custom code we've written internally for the same purpose!).
This was a ton of fun to build! We'll also be releasing a NLA enabled version of our Chrome extension [0] within the next couple of days which will be similar (but way more convenient than) the demo on the landing page above.
We're super bullish on LLMs for pulling "no-code" forward, helping more knowledge workers build automations. Already, folks are using our OpenAI [1] + ChatGPT [2] integrations to build very cool automations with summaries, categorization, copy writing, and more. We think there is a ton more to do here.
If anyone is interested in this problem space, shoot me an email [email protected]!