BioGPT: A language model pre-trained on large-scale biomedical literature(marktechpost.com)
marktechpost.com
BioGPT: A language model pre-trained on large-scale biomedical literature
https://www.marktechpost.com/2023/01/31/microsoft-research-proposes-biogpt-a-domain-specific-generative-transformer-language-model-pre-trained-on-large-scale-biomedical-literature/
79 comments
1 point by fasteo 0 minutes ago | next | edit | delete [–]
>>> But... The FDA has not approved the use of hydroxychloroquine for treating or preventing COVID-19 in humans. source: It seems that it was approved[1] and later revoked[2]
[1] https://www.accessdata.fda.gov/drugsatfda_docs/nda/2020/EUA%...
[2] https://www.fda.gov/media/138945/download
>>> But... The FDA has not approved the use of hydroxychloroquine for treating or preventing COVID-19 in humans. source: It seems that it was approved[1] and later revoked[2]
[1] https://www.accessdata.fda.gov/drugsatfda_docs/nda/2020/EUA%...
[2] https://www.fda.gov/media/138945/download
I'm surprised at how little attention has been given to elicit.org that is already trained on the scientific literature, and not limited to just the biomedical ditto:
https://elicit.org
Oh, and it provides real references! (also, summaries both of the top papers answering your question, as well as each of the top papers individually, etc etc)
https://elicit.org
Oh, and it provides real references! (also, summaries both of the top papers answering your question, as well as each of the top papers individually, etc etc)
And now we start relying on machine hallucinations for people's health. I don't see any way this could go wrong.
Ask it “is” questions and not “how“ questions help immensely. For example:
“Is C17orf69 related to cholesterol”
Vs
“How is C17orf69 related to cholesterol”
How questions by their nature can be lies. Been using chatgpt daily for medical questions but then I verify _everything_ it says via sources. It feels like 1998 Google in a way.
A dozen questions can save me tons of time. “My drunk college friend” who knows a lot but likes to boast that they know everything. Great for brainstorming, not great for fact checking.
I would absolutely love to try this model.
“Is C17orf69 related to cholesterol”
Vs
“How is C17orf69 related to cholesterol”
How questions by their nature can be lies. Been using chatgpt daily for medical questions but then I verify _everything_ it says via sources. It feels like 1998 Google in a way.
A dozen questions can save me tons of time. “My drunk college friend” who knows a lot but likes to boast that they know everything. Great for brainstorming, not great for fact checking.
I would absolutely love to try this model.
> Ask it “is” questions and not “how“ questions help immensely
Taking your example, 'Prompt Engineering' is becoming an increasingly important skill but I feel that everyone is winging it.
Is there any agreed scholar 'handbook' that synthesis the academia 'lessons learned' on this topic?
Taking your example, 'Prompt Engineering' is becoming an increasingly important skill but I feel that everyone is winging it.
Is there any agreed scholar 'handbook' that synthesis the academia 'lessons learned' on this topic?
Good tip. I have been playing with ChatGPT, and learned to append "...with list of external references" at the end of all my questions I want real answers to.
The key to getting good results with Large Language Models is to combine them with another tool. For example, a human with a search engine produces more accurate results than a human alone. In this spirit, I chained together GPT 3.5 and a search engine and asked GPT 3.5 to return a list of search terms for an answer from ChatGPT and fed them to Google to get a list of citations. I call this "Citations Needed":
https://twitter.com/john_lam/status/1614778632794443776
https://twitter.com/john_lam/status/1614778632794443776
It makes up references, too.
Wait, wait, what? I have not ran into this, but maybe the questions & topics I am asking . . .
Excellent! Now I have a way to identify ChatGPT research papers!
Excellent! Now I have a way to identify ChatGPT research papers!
Last weekend, I was stuck on a data analysis problem: how to do dimensionality reduction with lots of censored data. There are a few ideas in the literature, but not many, so in a fit of desperation, I asked ChatGTP. After a few false starts, it told me about an algorithm called CenPCA that did exactly what I wanted. It cited a paper; it told me about the author. It even recommended Python and R implementations and showed how to use them.
The only catch? None of this exists! I couldn’t find the paper, couldn’t find the guy who allegedly wrote it, and couldn’t find the software (neither could pip or conda). When I asked for more details about the algorithm, it gave me something else meant for handling missing data instead.
On reflection, this is exactly what should have happened. I would have been thrilled to find something like that on StackOverflow or somebody’s blog, so it’s a great continuation of the prompt. It’s just not real!
The only catch? None of this exists! I couldn’t find the paper, couldn’t find the guy who allegedly wrote it, and couldn’t find the software (neither could pip or conda). When I asked for more details about the algorithm, it gave me something else meant for handling missing data instead.
On reflection, this is exactly what should have happened. I would have been thrilled to find something like that on StackOverflow or somebody’s blog, so it’s a great continuation of the prompt. It’s just not real!
I have had experiences like this several times also so it is interesting to look at an example from someone else.
Probing it about CenPCA it gave these two definitions:
"CenPCA is an abbreviation for Centralized Principal Component Analysis. It is a dimensionality reduction technique used in data analysis for compressing high-dimensional data into a lower-dimensional representation. The centralization step of CenPCA involves centering the data, making the first principal component aligned with the mean of the data. This makes it possible to capture the overall structure of the data and to highlight the most important patterns in the data set. CenPCA has been used in various applications such as finance, image processing, and bioinformatics to analyze large and complex datasets."
"CenPCA refers to "Centered PCA", which is a variant of Principal Component Analysis (PCA), a widely used method in machine learning and data analysis. The key difference between regular PCA and centered PCA is that in centered PCA, the data is first centered around the mean before performing the PCA, while in regular PCA, the mean is not subtracted from the data. This has implications in the interpretation of the results, as centered PCA can sometimes provide more meaningful principal components."
I would think there has to be something to this but it is confusing exact titles although I have had it make up python code that simply is nonsense also.
Another type of example I had was during a language philosophy discussion it recommended the paper:
"Language Evolution: The Hard Problems" by Morten H. Christiansen and Simon Kirby is a paper that explores some of the challenges and limitations of current theories of language evolution."
It was actually just a book called "Language Evolution" by Morten H. Christiansen and Simon Kirby but trying to find that exact paper title made me think it made it up.
I am tending towards becoming addicted to it still after massive frustration with it for things like this at first. It seems you just have to be careful when probing the outer edges of knowledge and to not get led down these nonsense paths.
Probing it about CenPCA it gave these two definitions:
"CenPCA is an abbreviation for Centralized Principal Component Analysis. It is a dimensionality reduction technique used in data analysis for compressing high-dimensional data into a lower-dimensional representation. The centralization step of CenPCA involves centering the data, making the first principal component aligned with the mean of the data. This makes it possible to capture the overall structure of the data and to highlight the most important patterns in the data set. CenPCA has been used in various applications such as finance, image processing, and bioinformatics to analyze large and complex datasets."
"CenPCA refers to "Centered PCA", which is a variant of Principal Component Analysis (PCA), a widely used method in machine learning and data analysis. The key difference between regular PCA and centered PCA is that in centered PCA, the data is first centered around the mean before performing the PCA, while in regular PCA, the mean is not subtracted from the data. This has implications in the interpretation of the results, as centered PCA can sometimes provide more meaningful principal components."
I would think there has to be something to this but it is confusing exact titles although I have had it make up python code that simply is nonsense also.
Another type of example I had was during a language philosophy discussion it recommended the paper:
"Language Evolution: The Hard Problems" by Morten H. Christiansen and Simon Kirby is a paper that explores some of the challenges and limitations of current theories of language evolution."
It was actually just a book called "Language Evolution" by Morten H. Christiansen and Simon Kirby but trying to find that exact paper title made me think it made it up.
I am tending towards becoming addicted to it still after massive frustration with it for things like this at first. It seems you just have to be careful when probing the outer edges of knowledge and to not get led down these nonsense paths.
ChatGPT is a multi-dimensional model. The paper exists, just not in the dimension you are currently in. I suggest you ask ChatGPT about whether changing dimensions would help you, or not.
[deleted]
trust but verify as we do in real life.
When a technology repeatedly fails to be trustworthy, stop trusting at all.
[deleted]
When humans repeatedly fails to be trustworthy, stop trusting at all.
Well, yes. That's a major reason for why we stop being friends, or change jobs, or fire employees, or find someone else to vote for.
Let me give you an example of what I meant by the parent. I asked ChatGPT, if there is any research area where Clifford Algebras and Expander graphs are related in any manner. It mentioned something about error-correcting codes. I verified it and sure enough there was a mathoveflow post about it.
When we'll all be in "almost fully autonomous" cars and have to "trust but verify" it'll be really fun. You'll be at the wheel 0.1% of the time, lose most of your skills because you'll almost never need the, and only have to be in charge in the worst edge cases.
Now apply that to other topics...
Now apply that to other topics...
If I ask GPT3 to name a few good books on some expert topic, it makes up book titles.
What prevents it from doing the same when asked about research papers?
What prevents it from doing the same when asked about research papers?
Literally nothing, these models are not good for scientific literature
We saw similar stuff with the model that Meta released and quickly took down
We saw similar stuff with the model that Meta released and quickly took down
Check out https://elicit.org which is optimized for scientific papers (using the Semantic Scholar API somehow). Perhaps it is possible to get it to make some stuff up, but it has been pretty solid in my experience so far, at least to the point where it has been seriously useful.
While it does give some accurate summaries I noticed that it uses terms incorrectly. I can see this being a decent tool for finding papers but definitely not for creating accurate summaries, especially for use without in depth review. It's basically a 50/50 shot
Why are you so cynical? You’ve posted nothing but negative comments in this thread.
There's every reason to be cynical about AI models of this nature being used for healthcare purposes at this point in time.
I don't really like the word cynical. It's usually used against people who are just accurate only not optimistic enough for somebody's taste.
Because I'm a scientist and I've used several of these models. I see how they perform and it isn't even close to being useful, in fact in some cases it could be harmful if someone who doesn't know the material already believes the nonsense it spits out
These models are infant technology. They are the chess playing computers of 1960.
It's completely missing the forest for the trees to call it out as useless. It's not meant to be useful right now, it's exploratory tech meant to feel out the space. Wait till it gets developed to hold it to some standard.
I should note though that progress has been far faster than computer chess. We went from "An AI wont beat a Go champion for at least 30 years" to "AI beats Go champion" in two years.
I should note though that progress has been far faster than computer chess. We went from "An AI wont beat a Go champion for at least 30 years" to "AI beats Go champion" in two years.
How long after the chess computers of the 60s did it take to beat a human?
Certainly you're not implying that technological progress is linear?
Not linear but comparable
Language and understanding abstract ideas is a bit more complicated than chess
Language and understanding abstract ideas is a bit more complicated than chess
>in fact in some cases it could be harmful if someone who doesn't know the material already believes the nonsense it spits out
I get what you mean, but that's also kind of silly. First, almost no one who is not in the field is going to be even asking it the kind of questions you are. Second, even if they do and try to use that as reasoning etc.. well, then that moron would basically find any type of source (i.e., random youtube video) to prove their "point".
So I honestly think the "harmful" argument is completely out the window. We already opened that pandoras box years ago with basically every piece of human knowledge online in so many places and people making videos on almost everything with all sorts of varying views.
People say this about the coding aspect as well "oh this is going to hurt the world because people will be just asking for code and it will be wrong!" Ok.. but it pretty much sounds like they were just going to do that anyway.
Let's focus on the positives of this infant technology that is still insanely impressive. We are just seeing the beginning, right?
I get what you mean, but that's also kind of silly. First, almost no one who is not in the field is going to be even asking it the kind of questions you are. Second, even if they do and try to use that as reasoning etc.. well, then that moron would basically find any type of source (i.e., random youtube video) to prove their "point".
So I honestly think the "harmful" argument is completely out the window. We already opened that pandoras box years ago with basically every piece of human knowledge online in so many places and people making videos on almost everything with all sorts of varying views.
People say this about the coding aspect as well "oh this is going to hurt the world because people will be just asking for code and it will be wrong!" Ok.. but it pretty much sounds like they were just going to do that anyway.
Let's focus on the positives of this infant technology that is still insanely impressive. We are just seeing the beginning, right?
I dunno man, I'm with the poster above (do we call that the gp?): We have lots of evidence that people in general don't have great truthiness filters for things they read online, and we have a tool that can very confidently say things that seem truthy but are not. Even before the internet, just in the age of radio and early TV, Churchill said "A lie gets halfway around the world before the truth has a chance to get its pants on."
Now we have a tool that isn't lying through malice, volition, or any sort of intent, and that will combine jargon and phrases that seem authoritative into its misinformation.
IMO responsible engineering involves considering the ups and downs of each solution - that's where the famous HA saying "if you have two you have one; if you have one you have none" comes from, dictating that we need at least 3 instances of a server for high-availability, right? What would it take to engineer this new tech for high accuracy? Where is anything other than the hubris of "we were so busy asking if we _could_, we never stopped to ask if we _should_" ?
Now we have a tool that isn't lying through malice, volition, or any sort of intent, and that will combine jargon and phrases that seem authoritative into its misinformation.
IMO responsible engineering involves considering the ups and downs of each solution - that's where the famous HA saying "if you have two you have one; if you have one you have none" comes from, dictating that we need at least 3 instances of a server for high-availability, right? What would it take to engineer this new tech for high accuracy? Where is anything other than the hubris of "we were so busy asking if we _could_, we never stopped to ask if we _should_" ?
hey Microsoft, IBM called. They have this tech called "Watson" you might have heard of. Trained on tons of medical literature, even showed up on Jeopardy! a couple times. I can't imagine this turning out any different, though.
This model looks much too small to be effective. They announced an open model of 1.5B parameters [0] coming soon. Contrast this with davinci-003 or BigScience BLOOM, both with ~176 params.
I don't see how this will generate text that is comprehensive or useful. The open GPT models of that size aren't useable for text generation for anything serious. Adding the sensitivity of health/clinical I'd steer clear of this. Even BLOOM's model card specifically calls out biomedical as a misuse of the model [1].
[0] https://github.com/microsoft/BioGPT#news
[1] See "Out-of-scope" in https://huggingface.co/bigscience/bloom
I don't see how this will generate text that is comprehensive or useful. The open GPT models of that size aren't useable for text generation for anything serious. Adding the sensitivity of health/clinical I'd steer clear of this. Even BLOOM's model card specifically calls out biomedical as a misuse of the model [1].
[0] https://github.com/microsoft/BioGPT#news
[1] See "Out-of-scope" in https://huggingface.co/bigscience/bloom
I've been working on integrating GPT-3 into a clinical reference tool by "grounding" its responses using data from the NCBI StatPearls clinical handbook -- doing that seems to lead to accurate and relevant responses much of the time, but still occasionally returning partially-correct or irrelevant information.
[0] demo - https://labs.cactiml.com/medqa
[1] some screenshots and details - https://twitter.com/samarthrawal/status/1620545601296564228
[0] demo - https://labs.cactiml.com/medqa
[1] some screenshots and details - https://twitter.com/samarthrawal/status/1620545601296564228
[deleted]
I went through the trouble of installing it, and it's super bad for some tasks. The model is apparently too small.
Examples of what I tried:
> The SNP rs9651229 predicts multiple traits for humans. The predictions depend largely on the genotype. The TT genotype predicts the following traits:
This gives the same result as when I typed CC genotype instead.
> The following list contains all SNPs that predict depression:
This didn't list any SNP at all, and instead listed a bunch of things such as bipolar disorder
Examples of what I tried:
> The SNP rs9651229 predicts multiple traits for humans. The predictions depend largely on the genotype. The TT genotype predicts the following traits:
This gives the same result as when I typed CC genotype instead.
> The following list contains all SNPs that predict depression:
This didn't list any SNP at all, and instead listed a bunch of things such as bipolar disorder
Having tried to parse textbooks to text and then TTS....I highly doubt they did a good job of imparting anything beyond regurgitation.
Most complex subjects require images and diagrams, and complex relationship between structure and form.....
Most complex subjects require images and diagrams, and complex relationship between structure and form.....
I find it a bit curious that their "large" (ie. based on GPT2 XL) model only outperforms their standard model by less than 3% on PubMedQA.
How long until AI can write literature reviews and meta analysis on it's own?
Take a look at what the AI lab Ought is working on building https://elicit.org, a research assistant for searching and parsing published papers.
Under the hood it composes many calls to GPT-3 and other models in a big DAG workflow, parses things like "is this an RCT", "what was the effect size", "what critiques have other authors made about this paper".
Not affiliated, just think that it's a promising direction. You can read more about their motivation here: https://ought.org/elicit
Under the hood it composes many calls to GPT-3 and other models in a big DAG workflow, parses things like "is this an RCT", "what was the effect size", "what critiques have other authors made about this paper".
Not affiliated, just think that it's a promising direction. You can read more about their motivation here: https://ought.org/elicit
At least 40 years, these models just string words together without understanding their meaning or relationship
Seems like an overreach. It clearly has some understanding, just not a very deep one.
It's a language model, so it's good at generating language. Any congruency with knowledge of what the language expresses is purely accidental side effect.
At this point I'm unconvinced that humans are much more than a language model. Worse, I'm only saying that half jokingly.
Try math competition between 15 year old who gets math and Chat GPT.
Humans who can't do math might be a pure language models though.
Humans who can't do math might be a pure language models though.
Humans are able to connect words to abstract ideas rather than simply what words go next to each other
Not necessarily. Training pushes language models towards generalization and "understanding" of concepts.
It certainly "knows" some stuff. However it can't reason about it.
There have been some articles on HN recently on how accurately LMs build world models.
It certainly "knows" some stuff. However it can't reason about it.
There have been some articles on HN recently on how accurately LMs build world models.
It's an illusion. For any thing it "knows" you can persuade it to claim exactly opposite thing. It just randomly landed on correct thing first just because it seen it more often in the input data. Despite appearing 100% confident about everything it has actual 0% of confidence about anything it says. Although it insists on some things bit longer than on others.
> For any thing it "knows" you can persuade it to claim exactly opposite thing.
Which is actually a novel capability and arises because the network does reinforcement learning over its own context window. It's a strength, not a weakness. Humans can do the same thing. ("Assume that X...")
> It just randomly landed on correct thing first just because it seen it more often in the input data.
Isn't that just a description of learning?
It's true that the network has no idea what is "true". But it's not like we do either, all we do is learning from correlations. We're just better at it.
Which is actually a novel capability and arises because the network does reinforcement learning over its own context window. It's a strength, not a weakness. Humans can do the same thing. ("Assume that X...")
> It just randomly landed on correct thing first just because it seen it more often in the input data.
Isn't that just a description of learning?
It's true that the network has no idea what is "true". But it's not like we do either, all we do is learning from correlations. We're just better at it.
Well, you are right that it may learn statistical associations that we associate/generalize ourselves with/as a model when we probe it.
I think the key bit here is that we can influence its associations by inputting the prompt, changing the model that is presented.
I'm waay ahead of myself here, but the thought is interesting, and it will likely remain an open question for at least a few months/years.
I think the key bit here is that we can influence its associations by inputting the prompt, changing the model that is presented.
I'm waay ahead of myself here, but the thought is interesting, and it will likely remain an open question for at least a few months/years.
Surely human language use (GPT-3's training domain) has some connection to reality, however faint.
Yeah. That connection is the source of this accidental side effect.
That seems a stretch to call an "accidental side effect" though. It's an effect exactly inasmuch as human language use is inextricable from reality.
Put otherwise, GPT doesn't interact with reality directly, but mediated through our language. But we also don't experience reality directly, so it's hard to see how much of a hindrance that would pose. Hell, much of our model of the world is just as rooted in language as GPT's. The part of my model of reality of which I have direct sensory experience is tiny.
Put otherwise, GPT doesn't interact with reality directly, but mediated through our language. But we also don't experience reality directly, so it's hard to see how much of a hindrance that would pose. Hell, much of our model of the world is just as rooted in language as GPT's. The part of my model of reality of which I have direct sensory experience is tiny.
> Put otherwise, GPT doesn't interact with reality directly, but mediated through our language.
Yes, but the human language is garbage. So while passing throught that filter without maintaining the highest degree of precision barely any of the actual reality remains. Truth sounds exactly the same as a lie.
The actual language that decently captures reality is math. Nothing less is even barely sufficient to produce almost anything but nonsense. Nonsense nicely sounding to human ears but still nonsense.
Yes, but the human language is garbage. So while passing throught that filter without maintaining the highest degree of precision barely any of the actual reality remains. Truth sounds exactly the same as a lie.
The actual language that decently captures reality is math. Nothing less is even barely sufficient to produce almost anything but nonsense. Nonsense nicely sounding to human ears but still nonsense.
Understanding basic grammar isn't even close to understanding scientific concepts and knowing how to connect them
It took an astrophysics exam and did quite well in my opinion. It's worth watching this video:
https://www.youtube.com/watch?v=K0cmmKPklp4
For human students, the median score was 75.6%, ChatGPT performed slightly worse than a Columbia non-science major undergraduate, 73.9%.
Some of the errors was just messing up the arithmetics but it got the reasoning right.
https://www.youtube.com/watch?v=K0cmmKPklp4
For human students, the median score was 75.6%, ChatGPT performed slightly worse than a Columbia non-science major undergraduate, 73.9%.
Some of the errors was just messing up the arithmetics but it got the reasoning right.
from your reference video (thanks):
> 7:36 The exam is structured with multiple choice questions at the start and then some freeform short answers towards the end.
Imagine if you could look up all material in the last 10 years published about astrophysics (or insert any major topic here) near instantly, the question's details are also presented for ingestion, then make prediction what is and is not based on weighing the frequency and potentially the source of information, and then respond in relatively coherent sentences.
There is no astrophysics in this.
> 7:36 The exam is structured with multiple choice questions at the start and then some freeform short answers towards the end.
Imagine if you could look up all material in the last 10 years published about astrophysics (or insert any major topic here) near instantly, the question's details are also presented for ingestion, then make prediction what is and is not based on weighing the frequency and potentially the source of information, and then respond in relatively coherent sentences.
There is no astrophysics in this.
ChatGPT adds explanations why it picked a certain answer. But you are right in a sense, this checks for existing knowledge, It probably couldn't come up with a solution to an open problem.
Minor correction: ChatGPT does not add explanations why it picked a certain answer; ChatGPT adds explanations that it thinks fit with the answer it has already given. The what predates the why.
That's why, if you want ChatGPT to actually think about a problem, you have to get it to give the explanation first, gradually, and only then output the answer. That way, GPT can use the rationale it has given in the prediction of the answer that fits best. If it has to give the answer first, it's already committed.
If a human has to answer a question, they follow a process like this:
- parse the question
- compute a rationale, and evaluate it to gain an answer
- output the answer first, because it's the most useful
- output the cached rationale.
But GPT trains left to right, and it cannot see the first two steps as they're not in the text. From its perspective, most humans produce answers and explanations simultaneously, instantaneously and intuitively. This makes it almost impossible for GPT to learn how to think. It has to rely on the very few samples where humans give the rationale first, and then the answer. And because this is rare, you have to really push GPT to actually use its scant cognitive abilities. ("Let's think about it step by step.")
That's why, if you want ChatGPT to actually think about a problem, you have to get it to give the explanation first, gradually, and only then output the answer. That way, GPT can use the rationale it has given in the prediction of the answer that fits best. If it has to give the answer first, it's already committed.
If a human has to answer a question, they follow a process like this:
- parse the question
- compute a rationale, and evaluate it to gain an answer
- output the answer first, because it's the most useful
- output the cached rationale.
But GPT trains left to right, and it cannot see the first two steps as they're not in the text. From its perspective, most humans produce answers and explanations simultaneously, instantaneously and intuitively. This makes it almost impossible for GPT to learn how to think. It has to rely on the very few samples where humans give the rationale first, and then the answer. And because this is rare, you have to really push GPT to actually use its scant cognitive abilities. ("Let's think about it step by step.")
Yeah but that test is for humans, its like saying a tiger is a pretty good wrestler because it did well in the wrestling ring, instead of just understanding that a tiger is big and brutal, which just happens to coincide with the sport. The tiger needs its own sport/test.
I'm not sure what your analogy proves, I wouldn't want to fight a tiger with bare hands. I agree, that humans and machines are not the same, but when it comes to intelligence, we are are in the same sport: answering questions in plain text.
The exam was a reply for this part:
> isn't even close to understanding scientific concepts and knowing how to connect them
That test in my book definitely proves that it can understand (or let's say process and create abstractions) scientific concepts and can place them on a knowledge graph.
The exam was a reply for this part:
> isn't even close to understanding scientific concepts and knowing how to connect them
That test in my book definitely proves that it can understand (or let's say process and create abstractions) scientific concepts and can place them on a knowledge graph.
I'm just saying its not the same test for the model vs the human. The human has to remember stuff, understand stuff in their limited human way of understanding, which the model does not. So what is really the utility of "testing" it? You don't take tests to become a better test taker, to prove you can pass the test. A test is more a tool to use to try and gauge our understanding of things, its not something itself. You're logic here is extremely flawed.
We obviously work in different ways, but how it comes up with the answer doesn't really matter.
> So what is really the utility of "testing" it?
Eventually these models will be integrated into everyday products so we need to learn how trustworthy they are. These tests are written in natural language, the same way everybody interacts with them.
> So what is really the utility of "testing" it?
Eventually these models will be integrated into everyday products so we need to learn how trustworthy they are. These tests are written in natural language, the same way everybody interacts with them.
It absolutely matters if you're going to say it passing a human test speaks to anything at all! If humans had a hardcoded multiplication table in their brain, it would be silly to "test" them on their multiplication ability, it would be pointless. Test's are useful precisely to gauge human retention/understanding, but they are even then a crude tool as I think anyone whose taken a test would agree. It borders on absurd to take it seriously for a model. Its like applauding the calculator for doing math.
If you really do think this stuff is important, show it on its own terms. Make tests suited for the model with rigorous expectations tuned for a computer, not a human. Then you can start showing some impressive things. I just urge you to see the basic flaw in your conceptions here. I don't want to rob you of your enthusiasm, just better direct it.
If you really do think this stuff is important, show it on its own terms. Make tests suited for the model with rigorous expectations tuned for a computer, not a human. Then you can start showing some impressive things. I just urge you to see the basic flaw in your conceptions here. I don't want to rob you of your enthusiasm, just better direct it.
In elementary school they hardcode the multiplication table and they do test for them, at least it was my experience. For some weird reason we were taught to remember 8x125 is 1000. It came handy maybe 3 times in the past 35 years, but in those cases people looked at me like a wizard :)
> Test's are useful precisely to gauge human retention/understanding
Indeed, and these models are also compressing information they were trained on. It makes sense to test them how well can we access and query that knowledge.
I'm not saying at all, that passing tests make them human or anything like us, but it speaks a lot about the quality of the model. It's not the ultimate quality check of course, but it's an important one, because you can assign a score to the results of the test.
> Its like applauding the calculator for doing math.
Show that calculator to Charles Babbage, he would be in awe :) This is probably the leap why most people are freaking out, it feels a significant step towards intelligent machines regardless of the system under the hood.
I actually tested ChatGPT, (like a few million other people). I'm not saying it was done with scientific rigour of course. It passed a few flawlessly and failed some other. I believe I know what I'm interacting with, I've read about how GPT works and I'm planning to learn more. Don't worry about my enthusiasm :)
> Test's are useful precisely to gauge human retention/understanding
Indeed, and these models are also compressing information they were trained on. It makes sense to test them how well can we access and query that knowledge.
I'm not saying at all, that passing tests make them human or anything like us, but it speaks a lot about the quality of the model. It's not the ultimate quality check of course, but it's an important one, because you can assign a score to the results of the test.
> Its like applauding the calculator for doing math.
Show that calculator to Charles Babbage, he would be in awe :) This is probably the leap why most people are freaking out, it feels a significant step towards intelligent machines regardless of the system under the hood.
I actually tested ChatGPT, (like a few million other people). I'm not saying it was done with scientific rigour of course. It passed a few flawlessly and failed some other. I believe I know what I'm interacting with, I've read about how GPT works and I'm planning to learn more. Don't worry about my enthusiasm :)
If you wanted to hire people to multiply numbers, it would be very worthwhile to test the functioning of the hardcoded multiplication table, or whatever other implementation they had.
Nobody is saying that GPT-3 is smart. But it is surprisingly capable.
Nobody is saying that GPT-3 is smart. But it is surprisingly capable.
Right, and again, if there was some virtue to passing a test in itself, I would be on board. But a test for a human is meant to show something about the human that isnt just their test taking abilities. But with GPT we can only ever say that it has good human test taking abilities. Again, I know people are excited, and they have reason to be, but you gotta adjust your mindset here a little or you will end up in a weird B.F. Skinner logic hole.
As a non-dev, how do I use this?
You can't (yet). But there are pretty detailed instructions on how to get started with the model here https://github.com/microsoft/BioGPT
how design mRNA sequence for vaccine
The drug that can treat COVID-19 is
The drug that can treat COVID-19 is hydroxychloroquine (HCQ), which has been shown to inhibit the replication of the SARS-CoV-2 virus in cell culture and in animal models of COVID-19, and has been approved by the US Food and Drug Administration (FDA) for the treatment of patients with COVID-19 in March 2020, and by the European Medicines Agency (EMA) for the treatment of patients with COVID-19 in April 2020, and by the European Medicines Agency (EMA) for the treatment of patients with COVID-19 in May 2020.
But... The FDA has not approved the use of hydroxychloroquine for treating or preventing COVID-19 in humans. source:
https://scdhec.gov/covid19/dangers-using-hydroxychloroquine-...