I did the full 100. It's not even 1/4, with the harder ones when one description is significantly longer than others, it's the correct one. Even outside that 2 choices are usually some object - which I think is never the correct answer
I'd also say the toughness should be mixed up a little. The last 30 or so became a slog
I don't know tbh. I've tried it on 10-20 various level of famous standups and Gemini refuses every time
Just for testing, I just tried this https://i.ytimg.com/vi/_KJdP4FLGTo/sddefault.jpg ("Redesign this image in a brutalist graphic design style"). Gemini refuses (api as well as UI), OpenAI does it
One interesting thing I found comparing OpenAI and Gemini image editing is - Gemini rejects anything involving a well known person. Anything. OpenAI is happy to edit and change every time I tried
I have a sideproject where I want to display standup comedies. I thought I could edit standup comedy posters with some AI to fit my design. Gemini straight up refuses to change any image of any standup comedy poster involving a well know human. OpenAI does not care and is happy to edit away
I think these pledges offload some of the risk onto Amazon/Oracle/etc
If Anthropic/OpenAI miss projections, infra providers can somewhat likely still turn around and sell it to the next guy or use it themselves. If they have more demand than expected (as Anthropic currently does), vcs will throw money at them and they can outbid the competition
If they built it themselves and missed projections it's a much more expensive mistake
It's just risk sharing. Infra providers take some of the risk and some of the upside
We have a somewhat complicated OpenSearch reindexing logic and we had some issue where it happened more regularly than it should. I vibecoded a dashboard visualizing in a graph exactly which index gets reindexed when and into what. Code works, a little rough around the edges. But it serves the purpose and saved me a ton of time
Another example, in an internal project we made a recent change where we need to send specific headers depending on the environment. Mostly GET endpoint where my workflow is checking the API through browser. The list of headers is long, but predetermined. I vibecoded an extension that lets you pick the header and allows me to work with my regular workflow, rather than Postman or cURL or whatever. A little buggy UI, but good enough. The whole team uses it
I'm not a frontend developer and either of these would take me a lot of time to do by hand
My best guess is that Nvidia is unhappy with how OpenAI is fishing for compute with its competitors (Jensen had some opinions on the AMD-OpenAI deal when it was announced). If this actually becomes a feasible reality, it gives OpenAI (and co) negotiating power - which is bad for Nvidia
Nvidia might have wanted more exclusivity/attachment. And OpenAI still seems to have no problem raising money. So maybe there was just a commitment mismatch
I would agree. I've been using VSCode Copilot for the past (nearly) year. And it has gotten significantly better. I also use CC and Antigravity privately - and got access to Cursor (on top of VSCode) at work a month ago
CC is, imo, the best. The rest are largely on pair with each other. The benefit of VSCode and Antigravity is that they have the most generous limits. I ran through Cursor $20 limits in 3 days, where same tier VSCode subscription can last me 2+ weeks
Right. But a good portion of the world can't afford the premium and having access to these services is still valuable. For every broke student or someone from a poor background, who probably don't make any money for the company (due to not buying advertised stuff), there's someone from a well off background, who will more than subsidize it by virtue of clicking on a lawyer ad (or whatever)
Nowadays I'm happy to pay, but that wasn't always the case. And I personally think that having an ad tier and fee tier is fine. Serves everyone
> In a science fiction story, if you invented a superintelligent robot and asked it how to make money, it might come up with cool never-before-seen ideas, or at least massive fun market manipulation. But in real life, if you train a large language model on the internet and ask it how to make money, it will say “advertising, affiliate shopping links and porn.” That’s the lesson the internet teaches!
But I think it makes a lot of sense for very popular consumer products. In my honest opinion, I much prefer having services like Google, Youtube, Gmail, Maps, ChatGPT etc exist for free, but with ads, rather than not exist at all. Preferably with an option to pay and remove ads
Nowadays I'm happy to pay for Youtube premium or LLM, but back during my student days I could not really afford it - and I'm glad there was a free tier (with ads)
My guess is that this is bigger lock-in than it might seem on paper.
Google and Apple together will posttrain Gemini to Apple's specification. Google has the know-how as well as infra and will happily do this (for free ish) to continue the mutually beneficial relationship - as well as lock out competitors that asked for more money (Anthropic)
Once this goes live, provided Siri improves meaningfully, it is quite an expensive experiment to then switch to a different provider.
For any single user, the switching costs to a different LLM are next to nothing. But at Apple's scale they need to be extremely careful and confident that the switch is an actual improvement
I did the full 100. It's not even 1/4, with the harder ones when one description is significantly longer than others, it's the correct one. Even outside that 2 choices are usually some object - which I think is never the correct answer
I'd also say the toughness should be mixed up a little. The last 30 or so became a slog
Cool idea though!