How 'Open' Is OpenAI, Really?(dot.la)
dot.la
How 'Open' Is OpenAI, Really?
https://dot.la/openai-elon-musk-2659434979.html
77 コメント
One reason why Microsoft was able to execute so fast was that they had nothing to lose in doing so. The entirety of the Bing product and brand is mostly immaterial to the company's bottom line. What better way to try and revitalize it than by taking a huge leap with some hot new tech that is capturing the public imagination? If that leap is successful, you get to open up a huge new line of business and challenge Google. If not, well it just...goes back to being Bing.
For Google – whose entire business is tied up in that one search box – it is a very different equation. Any reduction of consumer confidence in their search engine is an existential threat to the company. Changes have to be rolled out slowly and with an immense amount of deliberation and testing.
Microsoft would be stuck in the same dilemma if, say, someone came up up with a futuristic-looking disruption in the spreadsheet space and Google Sheets was quick to adapt to it.
For Google – whose entire business is tied up in that one search box – it is a very different equation. Any reduction of consumer confidence in their search engine is an existential threat to the company. Changes have to be rolled out slowly and with an immense amount of deliberation and testing.
Microsoft would be stuck in the same dilemma if, say, someone came up up with a futuristic-looking disruption in the spreadsheet space and Google Sheets was quick to adapt to it.
How about… 3D spreadsheets, man. You finish tallying up all your credits and debits, and then wham! You’re sprung into another dimension. It’s Google Tensorsheets, man, Microsoft won’t even know what hit them.
This is what got me into programming way back when. I was a recruiter and doing things in spreadsheets, and I was like, I need a way to link data in a 3rd dimension, not just on the x and y axis. And then I was like actually if 3 dimensions why not 4 or more? So I asked on like, stackoverflow or something, do spreadsheets like that exist, and someone answered "you're basically describing a database with linking between tables," which led to me rabbitholing, learning more, getting into programming, and changing careers.
> Microsoft would be stuck in the same dilemma if, say, someone came up up with a futuristic-looking disruption in the spreadsheet space and Google Sheets was quick to adapt to it.
I feel like they basically are. It’s probably been close to a decade and Google’s online collaborative features in their office suite is far and above what Microsoft has managed to cobble together.
I feel like they basically are. It’s probably been close to a decade and Google’s online collaborative features in their office suite is far and above what Microsoft has managed to cobble together.
I don't know if I buy the whole Microsoft got lucky part. I can remember years ago seeing Azure ads pushing machine learning. Github Copilot is now almost 2 years old (since the first "tech preview" announcement). Copilot is built on OpenAI/GPT-3 just like ChatGPT... and then a few months after Copilot, they made it available on Azure's OpenAI service. This is all in 2021. So they've been working on this and collecting data for a while through Azure and Github.
Edit: Azure ML launched in 2015.
Edit: Azure ML launched in 2015.
They also did a course correction for "the internet" years ago...
I have to give them credit, but the other side of the equation is that though they react quickly, they tend to duct-tape feature and products together. I'm thinking of how office went online yet has always lacked cohesion. Or how teams competes with zoom/slack/etc, yet is frustrating to work with.
I have to give them credit, but the other side of the equation is that though they react quickly, they tend to duct-tape feature and products together. I'm thinking of how office went online yet has always lacked cohesion. Or how teams competes with zoom/slack/etc, yet is frustrating to work with.
Microsoft treats working software as secondary to marketing. They only build software to the point where it makes their marketing plausible to people who will never use the software and don't understand what it does, or what characteristics matter for it.
Non-profit to for-profit is one thing, but it shouldn't prevent them to actually open their technology. There are scores and scores of for-profit companies that do exactly that.
Continuing to call themselves "OpenAI" is disingenuous and frankly ridiculous at this point.
Continuing to call themselves "OpenAI" is disingenuous and frankly ridiculous at this point.
I hope they eventually give the word Open another meaning, for example, they can make their technology and method more open (source, maybe), and/or create a standardized platform to allow people to freely play with and extent their technology without the fear of big daddy restriction.
On the other hand, if they ended up chooses a different route, go patented everything and starts suing, then maybe it's better off for them to remove "Open" in their title just to make things more clear.
On the other hand, if they ended up chooses a different route, go patented everything and starts suing, then maybe it's better off for them to remove "Open" in their title just to make things more clear.
They take the mathematicians meaning. Open does not mean not closed.
I hope their api pricing goes down over time.
It depends on how many competitors will enter the market. At current stage, I would say the chance of them to increase the price is high. The current price is so cheap, it might as well work for a saturated market, which is not what the actual market is currently at.
I guess their current pricing is designed to attract new users to bring use cases/ideas to them, which would be a field where their the imagination is lacking.
I guess their current pricing is designed to attract new users to bring use cases/ideas to them, which would be a field where their the imagination is lacking.
[deleted]
It was very impressive for an org the size of Microsoft to pivot as quickly as they did.
I thought it was so cool how a non-profit made one of the most game changing tech products in years. Then I was incredibly bummed when I realized they weren't a non profit anymore.
They are still dedicated towards trying to advance humanity and I think that's more important than non-profit status. (For the record, I'm uncertain whether they are net positive or not given that I am skeptical of their approach to safety).
Im skeptical of their dedication towards trying to advance humanity versus more mundane objectives. It's a tale as old as time that noble initial goals are corrupted by power and greed when opportunity presents itself.
I think one should also acknowledge that OpenAI has been perhaps the leading provider of cutting edge AI to the public.
They let the public play against Dota Five, they provide their GPT playground and API, and they were the first to give something to the public with Dalle2 too. From the perspective of a developer this os mot very open, but from the perspective of the public it is quite open. But more "open access" or "open visibility" instead of "open source".
What they really don't want is that google or whatever develops an AGI secretly.
What they really don't want is that google or whatever develops an AGI secretly.
Question for someone who follows this space: are there any significant models/tech/etc that OpenAI has made open? Either before or after they changed business models. What they've kept private is well-known, but are there any checkboxes in the other column?
Whisper. Speech recognition and translation. Probably the best general-purpose speech recognition available right now. Certainly the best you can run on your own hardware. Open-sourced code and publicly free and available models. https://github.com/openai/whisper
Off the top of my head, CLIP the vision-language model, and Whisper, the audio to text model, have their trained weights and the architectures open. They did not open source the training dataset, although I fear it may be too much of an ask given all the possible legal issues.
GPT-2
Here is one cool (and recent) research project using GPT-2 model weights.
https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a...
Here is one cool (and recent) research project using GPT-2 model weights.
https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a...
TFA claims GPT-2 isn't open. Is it that the trained weights are open but the project that did the training isn't?
Realistically the most difficult part of training a LLM is curating a good data set. OpenAI never published their training data.
[deleted]
whisper
OpenAI are legends. They gave us fire. Until then we only heard about this tech from inside Google. It was always like "AI ethicists think this is too dangerous" and so we couldn't have it.
Each time, OpenAI made it happen. They were the driving force behind all of this stuff. Dall E changed the game entirely. ChatGPT changed the game again!
Lots of credit to Stability.AI - absolutely grand work by then. But OpenAI took what was once the tool of wizards in secret towers and gave it to all of us.
Then Stability and Meta gave us the means to build the tools. Who would have thought that Mark Zuckerberg would be crucial to the democratization of cutting edge tech. What a legend!
Each time, OpenAI made it happen. They were the driving force behind all of this stuff. Dall E changed the game entirely. ChatGPT changed the game again!
Lots of credit to Stability.AI - absolutely grand work by then. But OpenAI took what was once the tool of wizards in secret towers and gave it to all of us.
Then Stability and Meta gave us the means to build the tools. Who would have thought that Mark Zuckerberg would be crucial to the democratization of cutting edge tech. What a legend!
on the meaning of open, a tangential issue, let's have Sherman set the wayback machine to when the word "open" first started being used. "OK Mr Peabody"
open has a number of meanings that are useful, but have been overwhelmed by the concept of open source.
but open standards used to be the meaning. "ethernet is this, here's how you talk to it, but we're not going to write the code for you" (i'm not claiming that ethernet was originally an open standard, just using that as an example of what an open standard means)
and hardware manufacturers used to include blueprints/schematics (for repair, or for interconnection, for example) and commercial software vendors would show you the source code, but at the same time, you weren't being given intellectual property rights or licenses to anything, it was just part of what was required to encourage customers to stick with the product by making it easier to fit in where it was going.
And all of this was a really useful kind of openness. Just mentioning it because when people talk about how open OpenAI is, I'm never quite sure what they're expecting (I don't mind any expectation, not judging, I just don't know)
open has a number of meanings that are useful, but have been overwhelmed by the concept of open source.
but open standards used to be the meaning. "ethernet is this, here's how you talk to it, but we're not going to write the code for you" (i'm not claiming that ethernet was originally an open standard, just using that as an example of what an open standard means)
and hardware manufacturers used to include blueprints/schematics (for repair, or for interconnection, for example) and commercial software vendors would show you the source code, but at the same time, you weren't being given intellectual property rights or licenses to anything, it was just part of what was required to encourage customers to stick with the product by making it easier to fit in where it was going.
And all of this was a really useful kind of openness. Just mentioning it because when people talk about how open OpenAI is, I'm never quite sure what they're expecting (I don't mind any expectation, not judging, I just don't know)
> open has a number of meanings that are useful, but have been overwhelmed by the concept of open source.
How is Open Source not open? You have the source, and it is open (you can modify, hack, and distribute it). Hence, “Open Source”. What’s wrong with it?
> but open standards used to be the meaning. "ethernet is this, here's how you talk to it, but we're not going to write the code for you" (i'm not claiming that ethernet was originally an open standard, just using that as an example of what an open standard means)
Open standards are not better than Open Source, if you go that way. Most of them are not free, for a start.
> and hardware manufacturers used to include blueprints/schematics (for repair, or for interconnection, for example) and commercial software vendors would show you the source code, snip
That is a fantasised past. Some companies did; most companies did not. Just as nowadays.
How is Open Source not open? You have the source, and it is open (you can modify, hack, and distribute it). Hence, “Open Source”. What’s wrong with it?
> but open standards used to be the meaning. "ethernet is this, here's how you talk to it, but we're not going to write the code for you" (i'm not claiming that ethernet was originally an open standard, just using that as an example of what an open standard means)
Open standards are not better than Open Source, if you go that way. Most of them are not free, for a start.
> and hardware manufacturers used to include blueprints/schematics (for repair, or for interconnection, for example) and commercial software vendors would show you the source code, snip
That is a fantasised past. Some companies did; most companies did not. Just as nowadays.
i didn't say open source is not open, i said that other things that are not open-source can also be called open. OpenAI is not called OpensourceAI because that's not the extent of what they meant.
I didn't say open standards are better than open source, I said open standards are also open.
don't have time to read the rest of your comment, chatgpt's waiting for me on hold and getting impatient.
I didn't say open standards are better than open source, I said open standards are also open.
don't have time to read the rest of your comment, chatgpt's waiting for me on hold and getting impatient.
Neither their name nor history matter to me so, as long as their services are useful and pricing is reasonable, I will continue using OpenAI services.
That's a reasonable position to take and probably makes a lot of business sense.
Yet it's great to see that there are alternatives emerging as this forces OpenAI/Microsoft to innovate and hopefully limits their ability to hike up prices once they've captured the market in a year or two.
We don't really need another quasi uncontested tech monopoly
Yet it's great to see that there are alternatives emerging as this forces OpenAI/Microsoft to innovate and hopefully limits their ability to hike up prices once they've captured the market in a year or two.
We don't really need another quasi uncontested tech monopoly
Perhaps worth noting that Anthropic was founded by former OpenAI employees after the funding from the Microsoft: https://en.wikipedia.org/wiki/Anthropic
Openai changed it course.
Time for rebranding!
ClosedAI? MicrosoftAI? What do you think it should be?
Time for rebranding!
ClosedAI? MicrosoftAI? What do you think it should be?
I actually think that open source AI will be a terrible idea for sufficiently advance AI for the same reasons that open source nukes or bioweapons would be a mistake. So I'd be very happy to see them change their name to ClosedAI.
That may well be but a single for profit company like Microsoft will be a terrible guardian of that.
Future OpenAI usage limitations:
- anything illegal in the US and your country
- anything MS deems immoral and ethically problematic
- and anything that competes with Microsoft.
Future OpenAI usage limitations:
- anything illegal in the US and your country
- anything MS deems immoral and ethically problematic
- and anything that competes with Microsoft.
Why is that?
command.com
It may not be 'Open' in sense of open-source or free, but it is open that I can access it and use it in my projects. Before this companies like Google/Facebook had such models but closed in sense that we can get some papers and blog posts but can't rely on them to build our startup.
Because of financial and people resources required, availability of data etc. rest always stayed. Now I can use ChatGPT, Gpt-3/4, Whisper etc. at a reasonable cost and it is forcing rest of the industry to also "Open" up.
Because of financial and people resources required, availability of data etc. rest always stayed. Now I can use ChatGPT, Gpt-3/4, Whisper etc. at a reasonable cost and it is forcing rest of the industry to also "Open" up.
That, like, an incredibly low bar as to what's considered "open".
(For all we know ChatGPT might be just an outsourced Indian call center under the hood. We don't actually know how it works.)
(For all we know ChatGPT might be just an outsourced Indian call center under the hood. We don't actually know how it works.)
OpenishAI
The Indian was too expensive, it is a Kenyan [1].
[1] "OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic" https://time.com/6247678/openai-chatgpt-kenya-workers
The Indian was too expensive, it is a Kenyan [1].
[1] "OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic" https://time.com/6247678/openai-chatgpt-kenya-workers
I wish we could interview those scientists to hear what they think of this AGI goal.
I'm not convinced this will work.
I'm not convinced this will work.
When I was a teenager, there was a local chain of record stores over here named "free record store"
But guess what, you couldn't just walk in, grab some CDs and walk out without paying.
But guess what, you couldn't just walk in, grab some CDs and walk out without paying.
I was walking around downtown Toronto and saw some shop with large lettings on the outside saying "Pop Up Shop" advertising wild sales, etc. Turns out, the company is actually called Pop Up Shop. Very clever
It's a bad name there, too, but physical products are expected to have a cost; software frequently is FOSS and gratis, so there's more room for a reasonable person to think that maybe they actually are what they claim to be.
AAAS (AI as a Service)
You heard it here first.
You heard it here first.
Now that it runs on our laptops a lot of the initial shock, mystique, and novelty has already worn off, just like with DALL-E before it.
What the leaks show is that the community can run circles around OpenAI and the others when it comes to optimizing these things and taking them in interesting directions on a shoe string budget.
For example of what one sesh of tinkering can produce:
https://github.com/geohot/tinygrad/blob/master/examples/llam...
https://www.youtube.com/watch?v=nctqc8FBJ2U&t=4954s
The cat is out of the bag, you don't need a beowulf cluster of A100's and unlimited Azure credits. You do need a budget to train that is out of reach for hobbyists but the moat is not insurmountable and I don't think VC's will be scared off anymore as a lot of startup ideas are viable and a lot of outsiders can demonstrably make a real go of it.
Time for BasedAI. Game on.
What the leaks show is that the community can run circles around OpenAI and the others when it comes to optimizing these things and taking them in interesting directions on a shoe string budget.
For example of what one sesh of tinkering can produce:
https://github.com/geohot/tinygrad/blob/master/examples/llam...
https://www.youtube.com/watch?v=nctqc8FBJ2U&t=4954s
The cat is out of the bag, you don't need a beowulf cluster of A100's and unlimited Azure credits. You do need a budget to train that is out of reach for hobbyists but the moat is not insurmountable and I don't think VC's will be scared off anymore as a lot of startup ideas are viable and a lot of outsiders can demonstrably make a real go of it.
Time for BasedAI. Game on.
Yeah, the community always seems to figure out how to do things more effective.
My girlfriend asked me if I could transcribe some audio files for her with my "programming stuff". I immediately thought of Whisper from OpenAI.
I first used the official CLI tool. With the largest model it took long 8 hours to transcribe a 30 min long file. I noticed it was running on the CPU - tried switching it to use the GPU instead with no luck. Running it on WSL was probably not helping.
Then I found this gem: https://github.com/Const-me/Whisper A C++ Windows implementation of Whisper. I opened the program, fed it with the largest model and the file. The transcript was done in 4 minutes, instead of 8 hours... Downside? The program has a GUI, lol.
Of course, I could probably get the CLI tool to run on the GPU with some tinkering and installing some Nvidia packages for Whisper to use. But frankly, I have so little experience with that kind of stuff, that installing the Windows implementation was a much easier choice.
My girlfriend asked me if I could transcribe some audio files for her with my "programming stuff". I immediately thought of Whisper from OpenAI.
I first used the official CLI tool. With the largest model it took long 8 hours to transcribe a 30 min long file. I noticed it was running on the CPU - tried switching it to use the GPU instead with no luck. Running it on WSL was probably not helping.
Then I found this gem: https://github.com/Const-me/Whisper A C++ Windows implementation of Whisper. I opened the program, fed it with the largest model and the file. The transcript was done in 4 minutes, instead of 8 hours... Downside? The program has a GUI, lol.
Of course, I could probably get the CLI tool to run on the GPU with some tinkering and installing some Nvidia packages for Whisper to use. But frankly, I have so little experience with that kind of stuff, that installing the Windows implementation was a much easier choice.
[deleted]
People have decent oss solutions for running inference. There are currently no high quality oss code for training and the best checkpoints are still unavailable (gpt3.5+4/palm/claude)…
There are only a few hundred folks who have actual experience training at this scale.
llama is both not oss and not as good as da-vinci-003/etc. Prob best actual oss model is like flan-t5-xxl.
There are only a few hundred folks who have actual experience training at this scale.
llama is both not oss and not as good as da-vinci-003/etc. Prob best actual oss model is like flan-t5-xxl.
So we can't lay eggs yet, at least we got one. Who knows what might hatch.
I guess I just felt an objection about “[oss] running circles around OpenAI.” The reality is that the core technology is still very unapproachable (making these models). OSS solved like 1% of the total problem/difficulty.
My biggest hope is that meta/google/anthropic actually decide to release training code + data + checkpoint. Meta is most likely to do it but still unlikely
Without that it would take eutherai/carperai to really exceed expectations for OSS to be competitive.
My biggest hope is that meta/google/anthropic actually decide to release training code + data + checkpoint. Meta is most likely to do it but still unlikely
Without that it would take eutherai/carperai to really exceed expectations for OSS to be competitive.
ChatGPT can run on our laptops? How? The only thing I could find was that it can't because it requires 128 GB VRAM.
https://github.com/ggerganov/llama.cpp/issues/39
40GB for the big llama.
40GB for the big llama.
Even biggest Llama is like 1/3 size of ChatGPT
Might as well compare it to AlpacaGPT, they are trained different. Carefully prompted the output shows flashes of excellence and really whips ass and it is still early days.
We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.
We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.
But that's not ChatGPT?
Not it isn't but what the OP is saying that with the llama models (and others) being runable on consumer hardware it's only a matter of time until we get something that's "good enough" for many use cases.
Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.
Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.
No one said ChatGPT. It was a vague reference to the recent LLaMA developments from the community.
[deleted]
LLaMa [1] works on Apple Mx processors due to unified memory. The 7B model works on a 8GB Apple MacBook Pro M1, ~40 seconds/token.
[1] https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac
[1] https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac
I get 220ms/token on an M1 8GB Macbook Air, so ChatGPT speeds. on a 32GB M1 Max it babbles faster than my ex-gf.
Weird, I ran it alongside quite a few other processes, didn't expect to even be able to run and didn't bother to optimize it in any meaningful way.
Not ChatGPT, but LLaMa. Not as good as ChatGPT, but seeing the pace of progress when performance has been demonstrated, we'll have ChatGPT running on our laptops by summer. End of the year at the latest.
Does it matter? Openai finally built something resembling ai. Big tech wasn’t innovating, I personally don’t care if employees at openai capture a ton of the upside
> Does it matter?
Yes.
> Openai finally built something resembling ai. Big tech wasn’t innovating, I personally don’t care if employees at openai capture a ton of the upside
And if they called themselves ClosedAI I don't think people would mind so much; it's the dishonesty that's grating.
Yes.
> Openai finally built something resembling ai. Big tech wasn’t innovating, I personally don’t care if employees at openai capture a ton of the upside
And if they called themselves ClosedAI I don't think people would mind so much; it's the dishonesty that's grating.
> And if they called themselves ClosedAI I don't think people would mind so much; it's the dishonesty that's grating.
Isn’t this part for the course for an Elon project? Jokes aside (in this case I actually agree with him that openness is critical if the society starts really relying on AI), there would still be a need for a free model. There is too much at stake to rely on a single black box in that area.
Isn’t this part for the course for an Elon project? Jokes aside (in this case I actually agree with him that openness is critical if the society starts really relying on AI), there would still be a need for a free model. There is too much at stake to rely on a single black box in that area.
At least they publish their papers!
> Does it matter?
It does. Very much. We do not want to have to rely on government intervention and unforced errors to avoid an AI wasteland like we used to have with operating systems and web browsers. Particularly as these things are trained using our collective “wisdom” (well, production anyway) only to be sold back to us.
It does. Very much. We do not want to have to rely on government intervention and unforced errors to avoid an AI wasteland like we used to have with operating systems and web browsers. Particularly as these things are trained using our collective “wisdom” (well, production anyway) only to be sold back to us.
[deleted]
[deleted]
OpenAI started out with the intention of being a non-profit, but realized over time that it made it difficult to compete in terms of salary and hardware, so pivoted to a 'capped' for-profit.
As for Microsoft, my guess is their investment was just another investment in a hot AI 'startup'. It was sheer luck, and being in the right place at the right time, that they hit a gold mine with that investment as ChatGPT took off, but otherwise this would have been just one of their many investments that disappeared into the ether.
The most surprising thing to me is how hard and fast Microsoft pivoted once they realized they had a golden goose on their hands, with the Bing / Edge and other integrations they are doing. Never though I would see them move that fast, kudos.