OpenAI API(beta.openai.com)
beta.openai.com
OpenAI API
https://beta.openai.com/
164 comments
Agreed on the democratization front!
We saw this OpenAI demo: https://player.vimeo.com/video/427943452
and were just blown away. Very cool!!
I guess a spreadsheet is never too old [1] to learn new tricks :)
[1] Founder of https://mintdata.com here, so a bit biased (& opinionated about) spreadsheets, take the above with a pound or 10 of salt.
[2] I've sent them this example how we'd invoke their APIs, hopefully they'll let us into the beta, fingers crossed :) https://mintdata.com/docs/learn/core-mechanics/work-with-dat...
We saw this OpenAI demo: https://player.vimeo.com/video/427943452
and were just blown away. Very cool!!
I guess a spreadsheet is never too old [1] to learn new tricks :)
[1] Founder of https://mintdata.com here, so a bit biased (& opinionated about) spreadsheets, take the above with a pound or 10 of salt.
[2] I've sent them this example how we'd invoke their APIs, hopefully they'll let us into the beta, fingers crossed :) https://mintdata.com/docs/learn/core-mechanics/work-with-dat...
How is this "democratization"? OpenAI trains a model, then they make it available through an API. You have no say in what that model is trained on or how (other than to say whether they can use your data- but not how), neither can you modify the model according to your needs. And of course, with no ability to modify the product you're buying you have no opportunity to innovate. You can wrap it up in a different kind of application, sure, but the nature and number of applications that it can be wrapped up in is restricted by the abilities of the model and therefore is entirely dependent on the choices made by OpenAI.
Imagine MS saying they "democratised" operating systems because, hey, you can buy their binaries, so everyone can use their operating system. Compare that kind of "democratisation" with open source oSs.
No, the truth is that as more and more resources are necessary to wring the last few drops of performance out of the current generation of deep neural net models it is only large, well-funded companies that have the resources to innovate - and everyone else is forced to follow in their wake. Any expectations that progress would lead to "democratisation" of deep neural networks research has gone out the window.
Imagine MS saying they "democratised" operating systems because, hey, you can buy their binaries, so everyone can use their operating system. Compare that kind of "democratisation" with open source oSs.
No, the truth is that as more and more resources are necessary to wring the last few drops of performance out of the current generation of deep neural net models it is only large, well-funded companies that have the resources to innovate - and everyone else is forced to follow in their wake. Any expectations that progress would lead to "democratisation" of deep neural networks research has gone out the window.
Odd analogy to use with Microsoft Windows, since GPT-3’s source is available, along with a series of papers that enables anyone with the money and knowledge to implement it themselves.
The reasons why MS windows and GPT-3 cannot easily be modified by anyone are different, but the result is the same: you're stuck with what you're sold.
To clarify: MS windows is closed source, but you can't very well train a large GPT model unless you're someone with the resources of OpenAI. So you're stuck with whatever they choose to train and make available to you.
To clarify: MS windows is closed source, but you can't very well train a large GPT model unless you're someone with the resources of OpenAI. So you're stuck with whatever they choose to train and make available to you.
The API allows you to fine-tune existing models on your own dataset [1]
[1] Cf second paragraph of https://openai.com/blog/openai-api/
[1] Cf second paragraph of https://openai.com/blog/openai-api/
"fine tuning", i.e. transfer learning is still limited by the training of the original model.
For instance, if the original model is trained on English text exclusively and you want to fine-tune it on Greek text you are S.O.L.
For instance, if the original model is trained on English text exclusively and you want to fine-tune it on Greek text you are S.O.L.
It's only Open™️ if I can run the API on my own machines.
Indeed. I really don't understand how proprietary SaaS is "Open". It's just as locked down as IBM Watson and even moreso than Google's WaveNet-aaS.
If big LM's are the future then even if you had the model you couldn't run it on your own machines without having a DGX or two laying around.
Some of us do. And we can’t run OpenAI’s model, so it’s not open.
The essence of open source is that the resources are made available to you (without warranty). That isn’t the case here.
The essence of open source is that the resources are made available to you (without warranty). That isn’t the case here.
[deleted]
Run the model yourself and you can.
Here's the Bloomberg story https://www.bloomberg.com/news/articles/2020-06-11/trillions...
Concrete numbers from the various pullouts:
> They saw ratings hover around 60% with their original, in-house tech — this improved by 7-8% with GPT-2 — and is now in the 80-90% range with the API.
> The F1 score of its crisis classifier went up from .76 to .86, and the accuracy went up to 96%.
> With OpenAI, Algolia was able to answer complex natural language questions accurately 4x as often as it was using BERT.
I think the most informative are the first two, but the most _important_ is the final comparison with BERT (a Google model). I am, uh, a little worried about how fast things will progress if language models go from a fun lil research problem to a killer app for your cloud platform. $10m per training run isn't much in the face of a $100bn gigatech R&D budget.
> They saw ratings hover around 60% with their original, in-house tech — this improved by 7-8% with GPT-2 — and is now in the 80-90% range with the API.
> The F1 score of its crisis classifier went up from .76 to .86, and the accuracy went up to 96%.
> With OpenAI, Algolia was able to answer complex natural language questions accurately 4x as often as it was using BERT.
I think the most informative are the first two, but the most _important_ is the final comparison with BERT (a Google model). I am, uh, a little worried about how fast things will progress if language models go from a fun lil research problem to a killer app for your cloud platform. $10m per training run isn't much in the face of a $100bn gigatech R&D budget.
$10m per training run gets me a lot of engineering time to build our own version of this system and lease it to other customers. Just skip one training run and I've got a pretty good team.
Putting aside the question of whether it would ever be a choice between spending $10M on a training run and hiring a team for $10M, GPT transformers were the end result of decades of language research and innovations. You’re making it sound as though you can build the next iteration past GPT-3 for $10M, which I don’t think is the case.
no but poach the right resource and it puts you quickly on par with the competitors. What's that poaching cost?
Since the demos on this page use zero-shot learning and the used model has a 2020-05-03 timestamp, that implies this API is using some form of GPT-3: https://news.ycombinator.com/item?id=23345379 (EDIT: the accompanying blog post confirms that: https://openai.com/blog/openai-api/ )
Recently, OpenAI set the GPT-3 GitHub repo to read-only: https://github.com/openai/gpt-3
Taken together, this seems to imply that GPT-3 was more intended for a SaaS such as this, and it's less likely that it will be open-sourced like GPT-2 was.
Recently, OpenAI set the GPT-3 GitHub repo to read-only: https://github.com/openai/gpt-3
Taken together, this seems to imply that GPT-3 was more intended for a SaaS such as this, and it's less likely that it will be open-sourced like GPT-2 was.
But since the resources required for training such a model are only available to well-funded entities, it seems like offering the model as an API while releasing the original source-code is the best practical method of getting the model into the hands of people who would otherwise not have access?
That depends on which GPT-3 model they're using, and from both the API and the blog page, it's unclear.
Easy access to the 175B model would indeed be valuable, but it's entirely possible they're using a smaller variant for this API.
Easy access to the 175B model would indeed be valuable, but it's entirely possible they're using a smaller variant for this API.
It's worth noting that at least one of the demos is not few-shot: the code completion one notes it was trained on Github.
Looks like OpenAI is going head to head with huggingface.
This makes a lot of sense and it seems they are telegraphing to monetize what they have been doing. It also seems like this is why they don't release their models in a timely manner.
This makes a lot of sense and it seems they are telegraphing to monetize what they have been doing. It also seems like this is why they don't release their models in a timely manner.
The notable difference is that the base Huggingface library is open source, so you could in theory build something similar or more custom to the OpenAI API internally (which then falls into the typical cost/benefit analysis of doing so).
Wow I actually used your code to do experiments to synthesize tweets. I didn't realize you responded to my comment!
So its like Github vs Gitlab which makes more sense. I can see huggingface have a hosted version because now you can share your models on their platform.
Whoa -- Speech to bash commands? That's a pretty novel idea to me with my limited awareness of NLP. I could see this same idea in a lot of technical applications -- Provisioning cloud infrastructure, creating a database query.. Very cool!
Cool indeed! While language-to-code (where code is a regular, general-purpose language) has only recently started to be workable, text-to-SQL has been a long running application/research area for semantic parsing.
Some interesting papers and datasets:
NL2Bash: https://arxiv.org/abs/1802.08979
Spider: https://yale-lily.github.io/spider
Some interesting papers and datasets:
NL2Bash: https://arxiv.org/abs/1802.08979
Spider: https://yale-lily.github.io/spider
Can they train it to write regex commands too? That would be useful
It is not a novel idea and I don't think it is practical. If the natural language was practical for bash we would already have already "list directory" instead of "ls" and so on. "ls" is just 3 keystrokes while the natural language option is 15, 5 times more.
I was imagining more of a "list of files that contain word "hello" in them at least 5 times". Would be useful to easily write longer and pipe-chained commands, especially for people that don't use bash-like scripting on a daily basis.
But the character length would matter less when you can move to the speech domain.
ls is 2 syllables list dir is also 2 syllables with more meaning.
Ultimately, with natural language, the effectiveness seems to be when it is coupled with speech-to-text
ls is 2 syllables list dir is also 2 syllables with more meaning.
Ultimately, with natural language, the effectiveness seems to be when it is coupled with speech-to-text
It could be good for programmers who can't type and have to use voice. I couldn't use my hands much for a couple months, and the state of the art for voice access.. leaves something to be desired. If I lose function for longer in the future I'd need something like that.
It could be useful for learning tho (but at that point it could also become a crutch).
This is incredible. I can't tell how much this is cherry-picked examples vs. revolutionary new tech.
Yeah I can't tell exactly which ones but I really feel like some of the OpenAI demos of products could be potentially huge if fleshed out.
Sign up for the beta if you'd like to be the one to flesh them out :)!
I definitely did immediately after seeing this. Being neither an academic nor representing a recognizable name brand company, I don’t know if I should have my hopes up too high for getting access soon, but I certainly hope so. I’d love to play around with this and push its limits for some creative hackathon-style side projects!
Just wanted to add: It’s amazing all the negativity in this discussion. Whatever happened to the creative tech community who loves to push boundaries? Isn’t that still part of the hacker ethos, isn’t this still hacker news? Just because a tool has the potential to be used for bad doesn’t mean we shouldn’t be excited to find new ways to use it for good.
Just wanted to add: It’s amazing all the negativity in this discussion. Whatever happened to the creative tech community who loves to push boundaries? Isn’t that still part of the hacker ethos, isn’t this still hacker news? Just because a tool has the potential to be used for bad doesn’t mean we shouldn’t be excited to find new ways to use it for good.
OpenAI started off wide-eyed and idealistic but it made the mistake of taking on investors for a non-profit mission. A non-profit requires sponsors, not investors. Investors have a fiduciary responsibility to maximize profits, not achieve social missions of open AI for all.
OpenAI LP, our "capped-profit" entity which has taken investment, has a fiduciary duty to the OpenAI Charter: https://openai.com/blog/openai-lp/
Mind changed. Keep leading the way!
I guess Sama plans on manufacturing growth metrics by forcing YC companies to pretend that they're using this.
Generic machine learning APIs are a shitty business to get into unless you plan on hiring a huge sales team and selling to dinosaurs or doing a ton of custom consulting work, which doesn't scale the way VCs like it to. Anybody who will have enough know how to use their API properly can jus grab an open source model and tune it on their own data.
If they plan on commercializing things they should focus on building real products.
Generic machine learning APIs are a shitty business to get into unless you plan on hiring a huge sales team and selling to dinosaurs or doing a ton of custom consulting work, which doesn't scale the way VCs like it to. Anybody who will have enough know how to use their API properly can jus grab an open source model and tune it on their own data.
If they plan on commercializing things they should focus on building real products.
Not everyone wants to be an admin to their infrastructure. Real existing services like Heroku and Squarespace exist as useful services because even though you might know how to design and build a website from scratch, sometimes you just need something done quickly without too much worrying about details of the system that do not matter for your project at this point. I really don't see how this wouldn't apply to AI projects as well.
I could make a much better site coding my own website from scratch and setting up servers myself, but for some projects I wouldn't even think about it that way, because using Heroku or Squarespace I can save a LOT of time and get the results I need much quicker.
I could make a much better site coding my own website from scratch and setting up servers myself, but for some projects I wouldn't even think about it that way, because using Heroku or Squarespace I can save a LOT of time and get the results I need much quicker.
That's true, but machine learning models are not twilio or sendgrid, you have to tune them for your use case, monitor their performance and handle the uncertainty of their outputs. Doing that well requires a data scientist and if you have one they will be much more productive iterating on their own models instead of depending on a 3rd party black box.
Not a data scientist myself, but plenty of data scientists in a consultancy company that I used to work in said that they have to implement variants of a limited set of models over and over again, because they couldn't reuse code and infrastructure. The project contracts demanded that all IP created by the consultant is the property of the client. This even caused some of the data scientists to lose motivation, because the job wasn't challenging to them intellectually as it involved setting up the same stuff again and again. Very rarely would their actual expertise be needed in the job.
I am not sure if this particular service solves the problem for them in any way, but to my ear it sounds like there is a need for code and infrastructure reuse in the data scientists domain that is ripe for innovation.
I am not sure if this particular service solves the problem for them in any way, but to my ear it sounds like there is a need for code and infrastructure reuse in the data scientists domain that is ripe for innovation.
I'm pretty sure people said the exact same thing about Algolia when it was getting started (you have to tune search for your use case! How could you possibly use a search provider?!?)
Truth about the situation: - Transformers generalize well and don't need much fine tuning - OpenAI can probably fine tune for your use case better than you can - Getting new models into production takes 6 months to a year at companies of this size, if you did have Data Scientists in house, it might just be better to go with a solution like this for velocity - Not every company has the talent to make an in house ML program successful.
Truth about the situation: - Transformers generalize well and don't need much fine tuning - OpenAI can probably fine tune for your use case better than you can - Getting new models into production takes 6 months to a year at companies of this size, if you did have Data Scientists in house, it might just be better to go with a solution like this for velocity - Not every company has the talent to make an in house ML program successful.
Except the point of these larger transformer models is they generalize well over a wide range of domains or only require a small amount of transfer learning for really specific domains.
I'd say they're perfect candidates for the API as a service model.
I'd say they're perfect candidates for the API as a service model.
> I guess Sama plans on manufacturing growth metrics by forcing YC companies to pretend that they're using this.
That's wrong in almost too many ways to list. Sam left YC over a year ago, nor would he do such a thing. Nor does YC have that kind of power over companies, nor would it use it that way if it did. That would be wrong and also dumb.
That's wrong in almost too many ways to list. Sam left YC over a year ago, nor would he do such a thing. Nor does YC have that kind of power over companies, nor would it use it that way if it did. That would be wrong and also dumb.
Sorry, that was supposed to be sarcastic. What I meant to say is that Sam has a huge network and is a phone call away from pitching any CEO in the valley. One of the biggest benefits of YC these days is the huge network of companies in your portfolio, which makes getting intros and pilots a lot easier, leading to "traction" and more VC dollars.
[deleted]
I imagine they’re considering offering GPT-3, which would be cost prohibitive to fine-tune for most people. I also I heard inference was too slow to be practical. Perhaps they have some FPGA magic up their Microsoft sleeves.
Nobody is putting these huge models in production, even the smaller transformer models are still too expensive to run for most use cases.
With the way the field is moving, GPT-3 will be old news in a month, when more advances are made and open sourced.
With the way the field is moving, GPT-3 will be old news in a month, when more advances are made and open sourced.
i don't understand. if they run it for you and you apply transfer learning and fine tuning on your specific use case that would reduce drastically the costs hence why their offer make sense
[deleted]
Precisely my point. If they could put a model as large as GPT-3 into production (at a reasonable price to the consumer), wouldn’t that be a 10x improvement?
GPT-3 isn't a 10X improvement. (At least from everything we know so far.)
If the OP is right that nobody is putting the largest models into production (which I think is in inaccurate statement), then GPT-3 in production would be a 10x (ok, 5x?) improvement over the small GPT-2s and BERTS in production? So 10x in practice, if the hypothesis is correct? Which like I said, I don’t believe to be the case.
OpenAI started as a non-profit, went for-profit. Still owned by the big players.... Something isn't right.
Is OpenAI just a submarine so the tech giants can do unethical research without taking blame??? Its textbook misdirection, nonprofit and "Open" in the name, hero-esque mission statement. How do you make the mental leap from "we're non-profit and we won't release things too dangerous" to "JK we're for-profit and now that GPT is good enough to use its for sale!!". You don't. This was the plan the whole time.
GPT and facial recognition used for shady shit? Blame OpenAI. Not the consortium of tech giants that directly own it. It may just be a conspiracy theory but something smells very rotten to me. Like OpenAI is a simple front so big names can dodge culpability for their research.
Is OpenAI just a submarine so the tech giants can do unethical research without taking blame??? Its textbook misdirection, nonprofit and "Open" in the name, hero-esque mission statement. How do you make the mental leap from "we're non-profit and we won't release things too dangerous" to "JK we're for-profit and now that GPT is good enough to use its for sale!!". You don't. This was the plan the whole time.
GPT and facial recognition used for shady shit? Blame OpenAI. Not the consortium of tech giants that directly own it. It may just be a conspiracy theory but something smells very rotten to me. Like OpenAI is a simple front so big names can dodge culpability for their research.
> so the tech giants can do unethical research
I know it's trendy (and partly justified) to look down on OpenAI, but can you actually give any basis for this claim?
What kind of research is OpenAI doing that all the other big AI players (Google/DeepMind, FB, Microsoft) aren't also invested in? And even if others are doing the same, what part of OpenAI's research do you consider unethical?
> It may just be a conspiracy theory
Yea, it very much looks like that to be honest.
I know it's trendy (and partly justified) to look down on OpenAI, but can you actually give any basis for this claim?
What kind of research is OpenAI doing that all the other big AI players (Google/DeepMind, FB, Microsoft) aren't also invested in? And even if others are doing the same, what part of OpenAI's research do you consider unethical?
> It may just be a conspiracy theory
Yea, it very much looks like that to be honest.
> What kind of research is OpenAI doing that all the other big AI players (Google/DeepMind, FB, Microsoft) aren't also invested in? And even if others are doing the same, what part of OpenAI's research do you consider unethical?
I believe all of them are doing unethical research, especially facial recognition. Notice the public backpedaling this week from all the big tech companies on this too. By directing their cash through OpenAI they can avoid whatever fallout comes from unleashing things like GPT3 on the world.
The most straightforward use case for GPT3 is generating fake but believable text. AKA spam. That's what it was designed to do. If you think fake news is a problem now, wait till someone is generating a dozen fake but believable news articles per minute by seeding GPT3 with a few words and hitting a button.
Its a conspiracy theory with some circumstantial evidence. We will probably never know either way, because who would admit to it if it was true.
I believe all of them are doing unethical research, especially facial recognition. Notice the public backpedaling this week from all the big tech companies on this too. By directing their cash through OpenAI they can avoid whatever fallout comes from unleashing things like GPT3 on the world.
The most straightforward use case for GPT3 is generating fake but believable text. AKA spam. That's what it was designed to do. If you think fake news is a problem now, wait till someone is generating a dozen fake but believable news articles per minute by seeding GPT3 with a few words and hitting a button.
Its a conspiracy theory with some circumstantial evidence. We will probably never know either way, because who would admit to it if it was true.
> I believe all of them are doing unethical research, especially facial recognition.
Yes all of them are doing facial recognition research, except... OpenAI, so how exactly is OpenAI used as a scapegoat to be able to do that kind of research without public backlash?
> By directing their cash through OpenAI they can avoid whatever fallout comes from unleashing things like GPT3 on the world.
GPT-3 si not unethical research. It is what you decide to with it and how you decide to release it that can potientially be unethical.
Also, OpenAI is just ahead of other labs because they have an insane compute budget and really talented people, but if you have been following a little bit the NLP news, you will see that your theory of OpenAI being a front for unethical research just makes no sense. OpenAI release GPT-2, 1.5B billions parameters, then NVIDIA realeased Megatron, 8B parameters, Google released T5 at 11B and recently Microsoft did turing-nlg at 17B. So they are clearly working on this in their own names and very much publicizing their work.
Yes all of them are doing facial recognition research, except... OpenAI, so how exactly is OpenAI used as a scapegoat to be able to do that kind of research without public backlash?
> By directing their cash through OpenAI they can avoid whatever fallout comes from unleashing things like GPT3 on the world.
GPT-3 si not unethical research. It is what you decide to with it and how you decide to release it that can potientially be unethical.
Also, OpenAI is just ahead of other labs because they have an insane compute budget and really talented people, but if you have been following a little bit the NLP news, you will see that your theory of OpenAI being a front for unethical research just makes no sense. OpenAI release GPT-2, 1.5B billions parameters, then NVIDIA realeased Megatron, 8B parameters, Google released T5 at 11B and recently Microsoft did turing-nlg at 17B. So they are clearly working on this in their own names and very much publicizing their work.
Interestingly, by serving gpt3 as an API like this, they can actually monitor to see if companies are using it to generate spam
conspiracy hypothesis
It looks like a similar organizational structure as Mozilla Foundation + Mozilla Corporation.
They redefined the org from non-profit to "capped profit", whatever that means.
They're directly selling GPT 3 even though they originally said they wouldn't release it because of potential bad uses.
They paid MS a ton of money for hardware and got a huge equity investment from them.
And lets be honest here, the easiest and most straight-forward use of GPT3 is generating spam and low quality clickbait. Its the only use case that requires zero effort. The whole thing is built to generate fake but believable text. Its DeepFakes for text.
I'm not saying the whole thing is nefarious and evil, just suggesting that OpenAI may not be what it seems. There's a lot of odd things going on with it. They should have done what universities do, spin off the technology into a different for-profit company and sell it. Instead of redefining their entire org structure to make money.
They're directly selling GPT 3 even though they originally said they wouldn't release it because of potential bad uses.
They paid MS a ton of money for hardware and got a huge equity investment from them.
And lets be honest here, the easiest and most straight-forward use of GPT3 is generating spam and low quality clickbait. Its the only use case that requires zero effort. The whole thing is built to generate fake but believable text. Its DeepFakes for text.
I'm not saying the whole thing is nefarious and evil, just suggesting that OpenAI may not be what it seems. There's a lot of odd things going on with it. They should have done what universities do, spin off the technology into a different for-profit company and sell it. Instead of redefining their entire org structure to make money.
wow you just made the connection for me. GPT2 was too dangerous to release, and now GPT3 is so much better - is there no point at which things become too dangerous anymore? what was the conclusion on that one?
The blog post directly addresses this question: https://openai.com/blog/openai-api/
> What specifically will OpenAI do about misuse of the API, given what you’ve previously said about GPT-2?
> We will terminate API access for use-cases that cause physical or mental harm to people, including but not limited to harassment, intentional deception, radicalization, astroturfing, or spam; as we gain more experience operating the API in practice we expect to expand and refine these categories.
> What specifically will OpenAI do about misuse of the API, given what you’ve previously said about GPT-2?
> We will terminate API access for use-cases that cause physical or mental harm to people, including but not limited to harassment, intentional deception, radicalization, astroturfing, or spam; as we gain more experience operating the API in practice we expect to expand and refine these categories.
With Amazon having a moratorium of their rekognition API, I wonder if a Cambridge Analytica type event could happen to OpenAI where someone abuses and escapes the terms of service.
ah i've been caught not reading the linked post
hmm i dont love this. either OpenAI has implicitly promised to monitor all its users, or has adopted a "report TOS violations to us when they happen and we will judge" stance. neither are great roads to go down.
hmm i dont love this. either OpenAI has implicitly promised to monitor all its users, or has adopted a "report TOS violations to us when they happen and we will judge" stance. neither are great roads to go down.
GPT2 being "too dangerous to release" was a marketing stunt from the very beginning.
Who are you quoting here?
Maybe this article: https://blog.floydhub.com/gpt2/
More fake news and generated AI content there is more people would stop trusting social media. It will saturate to that tipping point that humanity will need to find more genuine ways to communicate. So I say bring it on.
Yeah bring the noise floor up so high that signal is lost.
Burn all social media to the ground, I say.
Burn all social media to the ground, I say.
You replied within an hour and now Im seeing it in 20 hours.
We exchange short texts, we both dont have a context, we dont know what each of us feels at that moment and we have absolutely no feedback about each other's mental states.
THIS isnt working! The conversational part of the social media is out of sync with reality and timespan. Evolution never had to optimize for this b/c there was never a need for it.
We struggle to understand, we just throw words at each other in passing and fill in the dots in our own minds - which is terrible b/c those dots are too far apart.
Signal is a needle in a haystack. Its not worth trying to keep fixing and reshaping the haystack so we dont keep loosing the needle. Lets just admit this tool isnt working and move on to better alternatives.
Edit: Clarifications (Need for post-editing also supports my point btw. )
Signal is a needle in a haystack. Its not worth trying to keep fixing and reshaping the haystack so we dont keep loosing the needle. Lets just admit this tool isnt working and move on to better alternatives.
Edit: Clarifications (Need for post-editing also supports my point btw. )
Based on my experience with non-profits, they are just like regular corps except they don't pay taxes, and they're always attached to a for-profit interest. The real community organizations don't tend to incorporate, as then you have to hire people to manage the corp or do it yourself.
This OpenAI work is almost certainly a way for these bigger corps to collude. Proving that would be impossible, though.
This OpenAI work is almost certainly a way for these bigger corps to collude. Proving that would be impossible, though.
GPT-2 is hard to do "shady" things right now
(speaking from experience)[1] but maybe GPT-3 might do better?
I could get poems to generate well. Tweets were a bit harder but I don't think we are at the point where you could just use a generative model to fool people that would be cheaper than actually hiring someone to write fake news. (Also shameless plug below)
[1] 1400 - TALK.8 - "A way to make fake tweets using GPT2" - Joshua Jay Herman https://thotcon.org/schedule.html
I could get poems to generate well. Tweets were a bit harder but I don't think we are at the point where you could just use a generative model to fool people that would be cheaper than actually hiring someone to write fake news. (Also shameless plug below)
[1] 1400 - TALK.8 - "A way to make fake tweets using GPT2" - Joshua Jay Herman https://thotcon.org/schedule.html
When I learned that Sam Altman (sorry Sam) was involved, I understood the direction, you mentioned.
And yes, there is often no need to call something open explicitly, if it really is. Is into OpenOS, or just Linux?
And yes, there is often no need to call something open explicitly, if it really is. Is into OpenOS, or just Linux?
Well said. I think they need to change the name. It is misleading on many levels.
A reshuffling might give: NopeAI or PeonAI
If that's the case, they need to change they're name.
I think it's simply because OpenAI is fundamentally created and controlled by venture capitalists, and the tech they created turned out to be just too juicy an opportunity to not monetize.
I can’t say I blame them, when they realize they are sitting on the technological equivalent of a mountain of gold. What would you do?
I can’t say I blame them, when they realize they are sitting on the technological equivalent of a mountain of gold. What would you do?
[deleted]
> sitting on the technological equivalent of a mountain of gold. What would you do?
Greed is not justified. I get that people are weak, selfish, they can't stop themselves. Some feel sympathy because they've been weak too. "Maybe it's justified," they like to think. "Everybody lies." But seriously, those who care so much about money and power they can't do things in a civilized respectable way: they are not yet an adult and must be hard barred from the upper tiers of capitalism until they learn that life does not revolve around them.
I blame them for being shitty, and blame everyone around them for letting it happen.
Greed is not justified. I get that people are weak, selfish, they can't stop themselves. Some feel sympathy because they've been weak too. "Maybe it's justified," they like to think. "Everybody lies." But seriously, those who care so much about money and power they can't do things in a civilized respectable way: they are not yet an adult and must be hard barred from the upper tiers of capitalism until they learn that life does not revolve around them.
I blame them for being shitty, and blame everyone around them for letting it happen.
Why is it not justified, beyond a moral belief? Seems rational that people would strive to make as much money as possible.
Rational is building community. Rational is searching for inner peace. Rational is being the best you can, and helping others.
Greed is not rational. It's just rationalized.
Greed is not rational. It's just rationalized.
I don't really understand how it's not rational. What's irrational about trying to get as many resources as you can?
The literal millennia of humans who've achieved that then got to the end of their life just to look back and say "I wish I'd focused on family and friends more."
Basically everyone on their death bed says focus on the experience, not the material. And everyone who does it agrees.
Basically everyone on their death bed says focus on the experience, not the material. And everyone who does it agrees.
What is wrong with selling technology?!
I miss the opposite: the old openAI gym and other testbeds. I still don’t know why they shut those down.
What alternatives do people like?
What alternatives do people like?
Why are there no live examples on the page. All I see is video presentations and some cached API response.
Is it a confidence problem? Are the OpenAI folks not confident on a single use case? Or did I miss the live demo somewhere?
Is it a confidence problem? Are the OpenAI folks not confident on a single use case? Or did I miss the live demo somewhere?
You can use the API live in multiple products, such as AI Dungeon (https://play.aidungeon.io/)!
Thanks Greg. Will check it out. Would love to see AI move from the "it's fun" zone to "make some money" zone soon though. We are all invested in the success of AI :)
Natural language search is approximately $100B business. This might be first AI application that changes the search landscape from 1990s and finally puts an end to the question “where is money in AI?”.
I wonder if there are any legal complications in the transition from a non-profit to a regular company (especially from a tax perspective)
In NLP there is a very clear and powerful new paradigm: train a HUGE language model using vast amounts of raw text. Then to solve the problem of interest, either fine-tune the model by training on your specific dataset (usually quite small), or 0/1-shot the learning somehow.
The crucial question is : is this paradigm viable for OTHER types of data?
My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
I'm especially optimistic that this paradigm will work for image streams from autonomous vehicles. Classic supervised learning has proved to be difficult if not impossible to get to work for AV vision, so the new paradigm could be a game-changer.
The crucial question is : is this paradigm viable for OTHER types of data?
My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
I'm especially optimistic that this paradigm will work for image streams from autonomous vehicles. Classic supervised learning has proved to be difficult if not impossible to get to work for AV vision, so the new paradigm could be a game-changer.
> My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
This has been demonstrated for many years, it's not news. Many of the SOTAs like BiT require pretraining on JFT-300M, or Instagram, or what have you.
This has been demonstrated for many years, it's not news. Many of the SOTAs like BiT require pretraining on JFT-300M, or Instagram, or what have you.
An API that will try to answer any natural language question is a mind blowing idea. This is a universal thinking interface more than an application programming one.
I just sent in a request to join the waiting list, for the company I work at, Kognity. The potential for this in the EdTech field is mindblowingly amazing!
There are a few good examples of educational help on the list but it's really only scratching the surface.
I'm really excited and hope Kognity and EdTech in general can use this for even more value-full (both for students and teachers) tasks soon.
There are a few good examples of educational help on the list but it's really only scratching the surface.
I'm really excited and hope Kognity and EdTech in general can use this for even more value-full (both for students and teachers) tasks soon.
Seems potentially more simple to get up and running then the Azure and Google Cloud alternatives which seemed involved when I last tried them.
OpenAI seems like a completely disingenuous organization. They have some of the best talent in Machine Learning, but the leadership seems completely clueless.
1) (on cluelessness) If Sama/GDB were as smart as they claim to be, would they not have realized it is impossible to run a non profit research lab which is effectively trying "to compete" with DeepMind.
2) (on disingenuity) The original openAI charter made OpenAI an organization that was trying to save the world from nefarious actors and uses of AI. Who were such users? To me it seemed like, entities with vastly superior compute resources who were using the latest AI technologies for presumably profit oriented goals. There are few organizations in the world like that, namely FAANG, and their international counterparts. Originally OpenAI sounded incredibly appealing to me, and a lot of us here. But if their leadership had more forethought, they would perhaps not have made this promise. But given the press, and the money they accrued, it has now become impossible to go back on this charter. So the only way to get themselves out of the whole they dug into was by making it into a for profit research lab. And by commercializing perhaps a more superior version of the tools Microsoft, Google and the other large AI organizations are commercializing, is OpenAI any different from them?
How do we know OpenAI will not be the bad actor that is going to abuse AI given their self interest?
All we have is their charter to go by. But given how they are constantly "re-inventing" their organizational structure, what grounds do we have to trust them?
Do we perhaps need a new Open OpenAI? One that we can actually trust? One that is actually transparent with their research process? One that actually releases their code, and papers and has no interest in commercializing that? Oh, that's right, we already have that -- research labs at AI focused schools like MIT, Stanford, BAIR and CMU.
I am quite wary of this organization, and I would encourage other HN readers to think more careful about what they are doing here.
1) (on cluelessness) If Sama/GDB were as smart as they claim to be, would they not have realized it is impossible to run a non profit research lab which is effectively trying "to compete" with DeepMind.
2) (on disingenuity) The original openAI charter made OpenAI an organization that was trying to save the world from nefarious actors and uses of AI. Who were such users? To me it seemed like, entities with vastly superior compute resources who were using the latest AI technologies for presumably profit oriented goals. There are few organizations in the world like that, namely FAANG, and their international counterparts. Originally OpenAI sounded incredibly appealing to me, and a lot of us here. But if their leadership had more forethought, they would perhaps not have made this promise. But given the press, and the money they accrued, it has now become impossible to go back on this charter. So the only way to get themselves out of the whole they dug into was by making it into a for profit research lab. And by commercializing perhaps a more superior version of the tools Microsoft, Google and the other large AI organizations are commercializing, is OpenAI any different from them?
How do we know OpenAI will not be the bad actor that is going to abuse AI given their self interest?
All we have is their charter to go by. But given how they are constantly "re-inventing" their organizational structure, what grounds do we have to trust them?
Do we perhaps need a new Open OpenAI? One that we can actually trust? One that is actually transparent with their research process? One that actually releases their code, and papers and has no interest in commercializing that? Oh, that's right, we already have that -- research labs at AI focused schools like MIT, Stanford, BAIR and CMU.
I am quite wary of this organization, and I would encourage other HN readers to think more careful about what they are doing here.
Why is it "impossible"? Academic labs are non-profit, and they are also effectively trying "to compete" with DeepMind.
Have a look at this discussion and the article from earlier today [0]. Of course, a singular lab could compete with something DeepMind does, but not without massive amounts of money in their pockets. The state of the art has become pretty expensive, really fast.
[0]: https://news.ycombinator.com/item?id=23486163
[0]: https://news.ycombinator.com/item?id=23486163
State of the art can be (and usually is) born in academic labs.
I don't think this is true. The ResNet was born at Microsoft, DQN was born at Deepmind, the Transformer was born at Google, and GPT2 was born at OpenAI.
I'm obviously biased since I work at an industry AI lab, but we both have important roles to play.
I'm obviously biased since I work at an industry AI lab, but we both have important roles to play.
ResNet lead authors were from UC San Diego and the Transformer was a collaboration with U Toronto. There absolutely is innovation coming from industrial labs, but industrial ties to academia run deep -- especially at Stanford-born Google and affiliated organizations.
It'd be great if OpenAI also introduced CAPTCHA. I'd be much more willing and understanding to resolve those than anything Google makes.
It’s been a long time coming, but I am curious to see how OpenAI’s research output is directed and impacted by market forces.
It seems like a step towards OpenAI becoming something like a utility provider for AI capabilities
Natural Language Shell seems fun
Interesting to see this. Is this similar to Google and Azure's ML apis?
From AGI to money machine...
What happened to working on AI for the good of humanity, including AGI, and making sure it didn’t fall into the hands of bad actors? Wasn’t that the original aspiration? Now this reads like next generation Intercom/Olark tools.
It seems it has translation. How does it compare to Google Translate?
AGI in text is < 3yrs away.
there's zero understanding in any of this. This is still just superficial text parsing essentially. Show me progress on Winograd schema and I'd be impressed. It hasn't got anything to do with AGI, this is application of ML to very traditional NLP problems.
> Show me progress on Winograd schema and I'd be impressed.
The paper evaluated Winograds: https://arxiv.org/pdf/2005.14165.pdf#page=16
The paper evaluated Winograds: https://arxiv.org/pdf/2005.14165.pdf#page=16
i think you are assuming that what is happening under the hood is that a human-inputted sentence is being parsed into a grammar. it is not.
I know that it isn't. That's part of the problem. There is no attempt to generate some sort of structure that can be interpreted semantically and reasoned about by the model. The model just operates on the input superficially and statistically. That's why there has been virtually no progress on trivial tasks such as answering:
"I took the water bottle out of the backpack so that it would be [lighter/handy]"
What is lighter and what is handy? No amount of stochastic language manipulation gets you the answer, you need to understand some rudimentary physics to answer the question, and as a precondition, you need a grammar or ontology.
"I took the water bottle out of the backpack so that it would be [lighter/handy]"
What is lighter and what is handy? No amount of stochastic language manipulation gets you the answer, you need to understand some rudimentary physics to answer the question, and as a precondition, you need a grammar or ontology.
Have you tried feeding this to GPT and seeing if it continues it in a way that reveals understanding?
It sounds like you're saying "It doesn't work because it can't work", but you haven't actually shown that it doesn't work.
It sounds like you're saying "It doesn't work because it can't work", but you haven't actually shown that it doesn't work.
yes, I have. You can paste these into the website of the Allen Institute for AI, yourself here. (https://demo.allennlp.org/reading-comprehension/MjE1MzE1Mg==)
In the example above it guesses wrongly, but again this is not surprising because it can't possibly get the right answer (other than by chance). The solution here cannot be found by correlating syntax, you can only answer the question if you understand the meaning of the sentence. That's what these schemas are constructed for.
In the example above it guesses wrongly, but again this is not surprising because it can't possibly get the right answer (other than by chance). The solution here cannot be found by correlating syntax, you can only answer the question if you understand the meaning of the sentence. That's what these schemas are constructed for.
The problem for me was how to formulate the sentence in a way so that the natural next word would reveal the thing the network had modelled.
edit: Retracted a test where it seemed to know which to select, because further tries revealed it was random.
edit: I did some more tries, and it does seem to be somewhat random, but the way it continues the sentence does seem to indicate that it has some form of operational model. It's just hard to prompt it in a way that it is "forced" to reveal which of the two it's talking about. Also, it seems to me its coherence range is too short in GPT-2. I would love to try this with GPT-3.
edit: Retracted a test where it seemed to know which to select, because further tries revealed it was random.
edit: I did some more tries, and it does seem to be somewhat random, but the way it continues the sentence does seem to indicate that it has some form of operational model. It's just hard to prompt it in a way that it is "forced" to reveal which of the two it's talking about. Also, it seems to me its coherence range is too short in GPT-2. I would love to try this with GPT-3.
FWIW, I fed this into AIDungeon (running on OpenAI) and got this back: “The bottle is definitely lighter than the pack because you can throw it much further away than you can carry the pack.
You continue on into the night and come to an intersection.”
I'm skeptical. Amazing progress has been made in the last 5-10 years but it still feels like we need more paradigm shifting in the ML/AI field. It feels like we're approaching the upper limits of what stuffing mountains of data into model can do.
But with the speed of the field, maybe we can figure it out in three years. It just seems like we're still missing some key components. Primarily, reasoning and learning causality.
But with the speed of the field, maybe we can figure it out in three years. It just seems like we're still missing some key components. Primarily, reasoning and learning causality.
What breakthrough occurred?
Zero shot and few-shot learning in GPT-3 and lack of significant diminishing returns in scaling text models. Zero-shot learning is equivalent to saying "i'm just going to ask the model something that it was not trained to do"
And how do we get from zero shot to AGI? You're making gigantic leaps here.
For those who are wondering about reasoning behind this being the path to full AGI I recommend this Gwern post that goes into detail: https://www.gwern.net/newsletter/2020/05
From what I understand, its not just that the GPT-3 has impressive performance but more what is signifies and that is the fact that massive scaling didn't produce diminishing return, and if this pattern persists, it can get them to the finish line.
From what I understand, its not just that the GPT-3 has impressive performance but more what is signifies and that is the fact that massive scaling didn't produce diminishing return, and if this pattern persists, it can get them to the finish line.
what is the difference between zero-shot learning in text and AGI? not saying there isn't one, but can you state what it is?you can express any intent in text (unlike other media). to solve zero-shot in text is equivalent to the model responding to all intents.
many people have different definitions for AGI though. for me it clicked when i realized that text has this universality property of capturing any intent.
many people have different definitions for AGI though. for me it clicked when i realized that text has this universality property of capturing any intent.
Zero-shot learning is a way of essentially building classifiers. There's no reasoning, there's no planning, there's no commonsense knowledge (not in a comprehensive, deep way that we would look for it call it that), and there's no integration of these skills to solve common goals. You can't take GPT and say ok turn that into a robot that can clean my house, take care of my kids, cook dinner, and then be a great dinner guest companion.
If you really probe at GPT, you'll see anything that goes beyond an initial sentence or two really starts to show how it's purely superficial in terms of understanding & intelligence; it's basically a really amazing version of Searle's Chinese room argument.
If you really probe at GPT, you'll see anything that goes beyond an initial sentence or two really starts to show how it's purely superficial in terms of understanding & intelligence; it's basically a really amazing version of Searle's Chinese room argument.
I think this is generally a good answer, but keep in mind I said AGI "in text". My forecasting is that within 3 years you will be able to give arbitrary text commands and get the textual output of the equivalents of "clean my house, take care of my kids, ..." like problems.
I also would contend that there is reasoning happening and that zero-shot demonstrates this. Specifically, reasoning about the intent of the prompt. The fact that you get this simply by building a general-purpose text model is a surprise to me.
Something I haven't seen yet is a model simulate the mind of the questioner, the way humans do, over time (minutes, days, years).
In 3 years, I'll ping you :) Already made a calendar reminder
I also would contend that there is reasoning happening and that zero-shot demonstrates this. Specifically, reasoning about the intent of the prompt. The fact that you get this simply by building a general-purpose text model is a surprise to me.
Something I haven't seen yet is a model simulate the mind of the questioner, the way humans do, over time (minutes, days, years).
In 3 years, I'll ping you :) Already made a calendar reminder
Pattern recognition and matching isn’t the same thing as reasoning. Zero shot demonstrates reasoning as much as solving the quadratic equation for a new set of variables does; it’s simply the ability to create new decision boundaries leveraging the same set of classifying power and methodology. True agi isn’t bound to a medium — no one would say Helen Keller wasn’t intelligent for example.
I look forward to this ping :)
I look forward to this ping :)
What exactly is the difference between pattern matching and reasoning?
I think pattern matching can be interpreted as a form of reasoning. But it is distinct from logical reasoning. Where you draw implications from assumptions. GPT seems really bad at this kind of thing. It often outputs texts with inconsistencies. And in the GPT-3 paper it performed poorly on tasks like Recognizing Textual Entailment which mainly involves this kind of reasoning.
Exciting!
"Powered by Azure" -- Elon clearly distrust Amazon.
Elon is no longer part of OpenAI. Microsoft invested $1b.
https://en.wikipedia.org/wiki/OpenAI
https://en.wikipedia.org/wiki/OpenAI
Does this mean Microsoft isn't going to sue their customers for patent infringement?
Microsoft's stake in OpenAI doesn't seem to be publicly known.
> Exactly what terms Microsoft and OpenAI have agreed on with this $1 billion investment isn’t clear.
https://www.theverge.com/2019/7/22/20703578/microsoft-openai...
> Exactly what terms Microsoft and OpenAI have agreed on with this $1 billion investment isn’t clear.
https://www.theverge.com/2019/7/22/20703578/microsoft-openai...
Why would you say this?
I presume it's reference to OpenAI's patent pledge:
> Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world.
I'm not sure if it's ever been publicly elaborated on.
https://openai.com/blog/introducing-openai/
> Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world.
I'm not sure if it's ever been publicly elaborated on.
https://openai.com/blog/introducing-openai/
Still seems like a low effort, bad hot-take.
It's not just OpenAI, most of his ventures uses Azure.
Awesome! Just signed onto the wait list.
"OpenAI technology, just an HTTPS call away"
'an' is only mean to proceed a vowel. Should say
"OpenAI technology, just a HTTPS call away"
'an' is only mean to proceed a vowel. Should say
"OpenAI technology, just a HTTPS call away"
This depends how you pronounce 'h'. If you pronounce it "aitch", then "an HTTPS" is correct. If you pronounce it "haitch", then "a HTTPS" is correct. There's no universal pronunciation, and therefore no universally-right answer.
It depends how you pronounce "H". If you pronounce it aitch instead of haitch then using "an" in this context is totally correct. https://blog.apastyle.org/apastyle/2012/04/using-a-or-an-wit....
You're right! Wow, I never realised rules get changed by pronunciation before. It's not vowels at all but whatever sounds like one in the writers mind.
On a side note, has anyone noticed a lack of diversity on the group photo on their careers page: https://openai.com/content/images/2020/04/openai-offsite-jul...
I remember coming across it not too long ago and felt unwelcomed/disappointed.
I remember coming across it not too long ago and felt unwelcomed/disappointed.
I dunno - looks fairly representative of the Silicon Valley population demographic. Maybe not so much an issue with this company?
I notice (as usual) under-representation of women. But that's endemic to the industry.
To an extent I agree that most startups are made up of Caucasians/Chinese males, I think one of the reasons I left tech professionally is because I didn't feel comfortable/enjoy the lack of diversity.
Has anyone noticed a substantial rise in people noticing skin colour ?
Define diversity?
fwiw my best friends are SE Asian (not Chinese), white, black, and Mexican. I didn't feel this photo is as equally representative of that diversity. I'm not making a case it has any obligation to do so, just noting that it impacted my decision to not apply here a while ago.
If it had people of different color of skin but all men, would that bother you (no women)?
Or if it had different color of skin but all of them were Christian? ( no other religions)
Or different color of skin but all Canadians would that bother you? (No other nationalities)
As a European I am much more used to diversity meaning national or religious diversity. Still getting used to this notion of it mainly been used in the context of skin color in US.
As a European I am much more used to diversity meaning national or religious diversity. Still getting used to this notion of it mainly been used in the context of skin color in US.
Its just code for too many whites.
1. Just apply and you will find out if you are welcome or not. Do not assume.
2. Asians are not diverse enough? There are plenty on the photo.
This is what I submitted for beta list:
I want to create a software that can generate new code given business case hints, by studying existing open source code and their documentation.
I know this is vague, but sounds like what we eventually want for ourselves right?
I want to create a software that can generate new code given business case hints, by studying existing open source code and their documentation.
I know this is vague, but sounds like what we eventually want for ourselves right?
Remember how Microsoft trained their bot from reddit comments and it went anti human? Well I guess I have to start dropping hints for the skynet in all my repos.
Bloomberg reports the API is based on GPT-3 and “other language models”.
If that’s true, this is a big deal, and it epitomizes OpenAI’s namesake. The largest NLP models require vast corporate resources to train, let alone put into production. Offering the largest model ever trained (with near-Turing results for some tasks) is a democratization of technology that would otherwise have been restricted to well-funded organizations.
Although the devil will be in the details of pricing and performance, this is a step worthy of respect. And it bodes well for the future.