ChatGPT makes up fake academic papers(twitter.com)
twitter.com
ChatGPT makes up fake academic papers
https://twitter.com/dsmerdon/status/1618816703923912704
55 comments
I agree with your sentiment, and for some time now I fear that all this hype will lead to another AI winter.
Large language models are a true breakthrough and they are, in my view, much more useful than ChatGPT. They allow for some fundamental NLP tasks that can then be used as building blocks for truly intelligent systems.
Symbolic AI failed because it could not deal with complexity and fuzziness. Statistical learning can deal with complexity and fuzziness, but it is very limited when it comes to rigorous reasoning.
I believe that true AI will combine both ideas, and I have a strong hunch that neurosymbolic approaches will be crucial, but nobody really knows yet how to create a true neurosymbolic approach that seamlessly combines the strengths of symbolic and statistical approaches. My fear is that we throw the baby out with the bath water when people are finally fed up with ChatGPT hype, and then we will have to wait a couple decades before the next serious attempt at AI is made.
Large language models are a true breakthrough and they are, in my view, much more useful than ChatGPT. They allow for some fundamental NLP tasks that can then be used as building blocks for truly intelligent systems.
Symbolic AI failed because it could not deal with complexity and fuzziness. Statistical learning can deal with complexity and fuzziness, but it is very limited when it comes to rigorous reasoning.
I believe that true AI will combine both ideas, and I have a strong hunch that neurosymbolic approaches will be crucial, but nobody really knows yet how to create a true neurosymbolic approach that seamlessly combines the strengths of symbolic and statistical approaches. My fear is that we throw the baby out with the bath water when people are finally fed up with ChatGPT hype, and then we will have to wait a couple decades before the next serious attempt at AI is made.
Absolutely agreeing that true “intelligence” would come from such a synthesis - it also loosely matches what we know from cognitive psychology. We won’t go anywhere without rigorously addressing knowledge representation and reasoning - unlike humans, language is the only faculty ChatGPT possesses (and the power of language has been oversold for a while).
The cynic in me would rather have the backlash (and a rude awakening for the industry), than the looming hyper-normalization of convincing-sounding bullshit/plagiarism/etc (and new, more convoluted forms of “saying the magic words to make the algorithm behave”) in the near future.
(Relatedly: Say what you will about the rationalist community, they appear to have really thought about a lot of this.)
The cynic in me would rather have the backlash (and a rude awakening for the industry), than the looming hyper-normalization of convincing-sounding bullshit/plagiarism/etc (and new, more convoluted forms of “saying the magic words to make the algorithm behave”) in the near future.
(Relatedly: Say what you will about the rationalist community, they appear to have really thought about a lot of this.)
>When you really start to prod this and think "Ok, this tool by design is going to produce highly plausible but completely untrue information" do we really think the sensible next step is "Better integrate this very tightly into our primarily tool for finding out accurate information"? Because that appears to be the current plan.
It's not completely wrong all the time. Also the conditioning puts it into state where it has to respond with answers even if it doesn't know something. So coupling it with the tools for finding out accurate information is a logical next step. If it works out and you get highly plausible and true information most of the time, that's magical. Just because it's integrated doesn't mean you have to use it but many will. It'll keep getting better because that's how AI model have been doing at every benchmark, so even if you think it's not there yet, trends imply it might get there soon.
It's not completely wrong all the time. Also the conditioning puts it into state where it has to respond with answers even if it doesn't know something. So coupling it with the tools for finding out accurate information is a logical next step. If it works out and you get highly plausible and true information most of the time, that's magical. Just because it's integrated doesn't mean you have to use it but many will. It'll keep getting better because that's how AI model have been doing at every benchmark, so even if you think it's not there yet, trends imply it might get there soon.
> It's not completely wrong all the time.
But it makes up plausible answers and even fake references. Which makes it worse than something which is always dead wrong.
But it makes up plausible answers and even fake references. Which makes it worse than something which is always dead wrong.
And it seems, to my totally unqualified eye, that the creators did this intentionally. Because why not have ChatGPT respond with "my data does not enable to respond properly" everytime it makes stuff up? Because this whole thing would look a lot less impressive?
Because it has no idea if it's spouting bullshit at the moment or not.
And that doesn't seem to be addressable as long as the model is just stringing together the next most likely words as it learned them from some huge dataset. The answer "i don't know that" will always be rare, because nobody sits down and writes a blog post about how he could not find the solution to a problem.
And that doesn't seem to be addressable as long as the model is just stringing together the next most likely words as it learned them from some huge dataset. The answer "i don't know that" will always be rare, because nobody sits down and writes a blog post about how he could not find the solution to a problem.
Wouldn't it be great if there were a lot more ofbthose blog posts around?
Agreed, it's a problem right now. But conceptually, there are multiple ways (persistent memory in embedding dbs to reference, coupling it with search engines and data sources) it can get better.
A reference in a scientific journal to a piece of fraudulent research is far worse than a fake reference to a non-existent journal. Let's keep things in perspective?
In my view, underestimating the platform's capabilities may lead one to assume its imminent shutdown. For instance, I find the platform particularly useful for generating succinct bullet-point summaries of articles, enabling me to consume content in 1 minute instead of, say, 15 minutes.
Likely OK in a relatively basic, factual context.
But the point here is that as the subtlety and abstraction level of the material increases, the value of the statistical brute-forcing tapers off.
But one is less likely to apply such a tool to more challenging texts, in any case.
But the point here is that as the subtlety and abstraction level of the material increases, the value of the statistical brute-forcing tapers off.
But one is less likely to apply such a tool to more challenging texts, in any case.
ChatGPT is boring, although novel (unexpected) applications can be fun. Not the results (yes, it can write texts), but ideas how it can be applied, what sort of texts it can write, that I haven't thought of before. But what is now interesting to me is research into how the heck we, the meat blobs, actually manage to think.
> string words together with some randomness consistent with an existing body of work
To best of my awareness (and observation of my inner monologue as well as what I say during a conversation), this is also very simplified but very true description of my thought processes. Obviously, my observations are very limited (and surely nothing new under the sun, just me not being aware), but in retrospective (and sometimes that happens even while my working memory is still pretty much there as I've barely said another word) I can spot the points at which my thought process was influenced by some unknown (non-observable) factor, and I've picked a different word or switched a line of thought, making it kind of obvious to me that I was stitching words in a line. Best noticed when thinking in a comfortable but foreign language. And I can also spew bullshit to myself, when I have limited knowledge of some subject or limited time to think something through but I'm pressed to blurt out something.
Obviously, there's nothing new. But I'm hoping all this AI hype would also inspire more research into Natural Intelligence research, even though those are very different fields (although, who knows, maybe efforts made into analyzing AI outputs/behaviors would reveal something about us, too).
> string words together with some randomness consistent with an existing body of work
To best of my awareness (and observation of my inner monologue as well as what I say during a conversation), this is also very simplified but very true description of my thought processes. Obviously, my observations are very limited (and surely nothing new under the sun, just me not being aware), but in retrospective (and sometimes that happens even while my working memory is still pretty much there as I've barely said another word) I can spot the points at which my thought process was influenced by some unknown (non-observable) factor, and I've picked a different word or switched a line of thought, making it kind of obvious to me that I was stitching words in a line. Best noticed when thinking in a comfortable but foreign language. And I can also spew bullshit to myself, when I have limited knowledge of some subject or limited time to think something through but I'm pressed to blurt out something.
Obviously, there's nothing new. But I'm hoping all this AI hype would also inspire more research into Natural Intelligence research, even though those are very different fields (although, who knows, maybe efforts made into analyzing AI outputs/behaviors would reveal something about us, too).
I've been feeling exactly like this from the start. It's been very clear and very public from the very minute ChatGPT came out, that it's not concerned with the truth and simply makes things up as it goes along.
It can be fun to use for creative writing -- why not? Creative writing is about making shit up.
But for gathering information it's the opposite of what we want. Integrating it with a search engine is not just ill-advised, it's incredibly stupid.
It can be fun to use for creative writing -- why not? Creative writing is about making shit up.
But for gathering information it's the opposite of what we want. Integrating it with a search engine is not just ill-advised, it's incredibly stupid.
> Creative writing
I tried to get it to write _1984_ erotic fanfiction, which it did - but everything it spewed out was a cliche. I imagine it had read all of FFN and AO3.
> gathering information
It will probably be like the Google infoboxes and “People Also Search” entries we’ve had for a while, only more superficially coherent.
It seems like the “fake it until you make it” craze has fully taken over the AI/ML scene.
I tried to get it to write _1984_ erotic fanfiction, which it did - but everything it spewed out was a cliche. I imagine it had read all of FFN and AO3.
> gathering information
It will probably be like the Google infoboxes and “People Also Search” entries we’ve had for a while, only more superficially coherent.
It seems like the “fake it until you make it” craze has fully taken over the AI/ML scene.
it produced cliche because of the cliche quality of the prompt. You can’t expect it to be original when you’ve asked it for a tiresome combination of genre and a canonical work.
If the outputs are boring it ain’t chatgpt at fault…
If the outputs are boring it ain’t chatgpt at fault…
I don’t know the technical details of chatgpt but other language models have a sort of randomness tuning that makes them produce more unlikely stuff. Chatgpt appears to be deliberately tuned to produce very likely, very cliche, very rote output. Great for bloggers or people who want text summaries or boring professional content, not great for creativity. It’s not a ‘fault’ of chatgpt per se but it a consequence of chatgpt’s intentions, not the user’s.
Isn‘t it kind of ironic that we complain about ChatGPT not being „creative“ enough on a thread complaining it fakes academic papers? How can the same bot be limited to only produce „facts“ and at the same time would be useful to generate great „fiction“?
But I completely agree with you: it’s often spitting out boring stuff. And yes, it makes almost everything up based on language statistics and similarities.
Perhaps, we are just in a hype cycle? There are lots of specialty tools and bots available based on GPT-3 customized for different tasks. Complaining that ChatGPT or Bing+ aren’t equally suited for every task gets tiresome. Especially if more than 2/3 of the examples simply follow the pattern „junk in - junk out“.
But I completely agree with you: it’s often spitting out boring stuff. And yes, it makes almost everything up based on language statistics and similarities.
Perhaps, we are just in a hype cycle? There are lots of specialty tools and bots available based on GPT-3 customized for different tasks. Complaining that ChatGPT or Bing+ aren’t equally suited for every task gets tiresome. Especially if more than 2/3 of the examples simply follow the pattern „junk in - junk out“.
It would seem that, when you try too hard at being creative, chatgpt usually just loses it completely - in my experience, it often starts falling back on even worse cliches, or it stops making sense altogether. (Or it might just time out.)
Actually, ‘cliche’ is an extremely good evaluation:
It is assessing the quality of the output as if a human had made it. Most of human writing is cliche. Expecting it to produce good output is, as you say, a matter of prompt engineering.
It is assessing the quality of the output as if a human had made it. Most of human writing is cliche. Expecting it to produce good output is, as you say, a matter of prompt engineering.
This. I have been saying this to friends from the start. But even my tech friends are super impressed by it.
For me it is just another tool, that I can use to help me in certain areas. Not more. Not less.
If I was Sam Altman I would sell the thing asap before the hype dies out like with crypto.
For me it is just another tool, that I can use to help me in certain areas. Not more. Not less.
If I was Sam Altman I would sell the thing asap before the hype dies out like with crypto.
> If I was Sam Altman I would sell the thing asap
I think he just did sell it to Microsoft for an additional $10bn.
I think he just did sell it to Microsoft for an additional $10bn.
[deleted]
One thing that makes me depressed about ChatGPT is that the software engineering behind it is not some 1996 Doom Carmack style wizardry. It's attention mechanisms, a transformer architecture, and lots and lots and lots of data. The data does all the programming. I look at a lot of AI projects and there's not a lot of code. Sure it takes a lot of work to understand it, but in the end it's not a lot of code.
This makes me think that in the future programming will be just about collecting data and then doing things with the data. We will all be data generators and collectors with a few math PHDs in the middle working on the activation functions and so forth. The rest of us will be stringing models together Stable Diffusion style or writing the billing code.
This makes me think that in the future programming will be just about collecting data and then doing things with the data. We will all be data generators and collectors with a few math PHDs in the middle working on the activation functions and so forth. The rest of us will be stringing models together Stable Diffusion style or writing the billing code.
Feels good for some reason knowing that Carmack himself is working on AI from a different angle :)
https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe...
https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe...
> And he gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’
Is there any info on what these papers are?
Is there any info on what these papers are?
Try searching HN. There were recently a couple posts asking about these 40 papers.
I found the threads in the meantime, but there does not appear to be any new information as of yet.
I had an interesting ChatGPT experience yesterday where I asked it
> How can I increase the version of a TextDocument in VSCode from an extension?
and got a perfectly plausible answer, very eloquently explaining the following code
> How can I increase the version of a TextDocument in VSCode from an extension?
and got a perfectly plausible answer, very eloquently explaining the following code
// Get the TextDocument object
const document = vscode.window.activeTextEditor.document;
// Update the TextDocument version
document.update([], {incrementVersion: true});
I'd be interested to know happens if you ask your original question again. Do you get the original answer referencing the non-existent method? Or did you force it to update its state (knowledge of the world) such that it now always gives the second answer?
ChatGpt is finished training and doesn't update itself in any way.
The current conversation is fed each time as additional context back when you ask additional questions.
Thanks for the clarification.
If the chat is published on the net, and becomes part of ChatGPT's future training data, I wonder whether ChatGPT could somehow become "aware" of it in a way that would change its subsequent behaviour?
If the chat is published on the net, and becomes part of ChatGPT's future training data, I wonder whether ChatGPT could somehow become "aware" of it in a way that would change its subsequent behaviour?
No, starting a new chat it gives me again a (different) wrong answer.
I should add that I looked closer now at the transcript from yesterday, and the second answer it gave me after apologising was actually not correct, but correct enough for me to code it with the help of some actual API documentation.
I should add that I looked closer now at the transcript from yesterday, and the second answer it gave me after apologising was actually not correct, but correct enough for me to code it with the help of some actual API documentation.
State is only persisted per session, so no it will not learn globally. This is fairly important otherwise users could feed it false information.
You might get the correct answer on the second try, but that would be down to random seeds.
I asked ChatGPT a basic physics question:
> Imagine you have two semi-infinite conducting metal plates at right-angles, in an L shape. The plates are along the x and y axes. You place a charge +q at a point (d, d) in the top right-hand corner of the plane. Using the method of images, work out the force on the charge.
It gave me a very detailed, and utterly incorrect answer. I pointed out its errors and it corrected them politely. I eventually asked it to draw me an ascii-art diagram of what it thought was going on, and it drew a coordinate system with mis-labelled axes that looked both utterly plausible and yet was filled with genuinely creative bollocks. The whole thing was a bit like reading a patronising answer in a bad sci-fi film (that needed severe editing and had bits of reality in it).
I'm sure, however, if you took the output and sent it to the International Journal of Please Pay Us Open Access "Publishing" it'd get accepted and given a DOI...
> Imagine you have two semi-infinite conducting metal plates at right-angles, in an L shape. The plates are along the x and y axes. You place a charge +q at a point (d, d) in the top right-hand corner of the plane. Using the method of images, work out the force on the charge.
It gave me a very detailed, and utterly incorrect answer. I pointed out its errors and it corrected them politely. I eventually asked it to draw me an ascii-art diagram of what it thought was going on, and it drew a coordinate system with mis-labelled axes that looked both utterly plausible and yet was filled with genuinely creative bollocks. The whole thing was a bit like reading a patronising answer in a bad sci-fi film (that needed severe editing and had bits of reality in it).
I'm sure, however, if you took the output and sent it to the International Journal of Please Pay Us Open Access "Publishing" it'd get accepted and given a DOI...
It's not difficult to hit the limits of ChatGPT's "knowledge". I've had it invent completely fictitious historical pacts and documents, as well as give me utterly incorrect "facts" about programming competitions.
The issue for me isn't that it's wrong. By using ChatGPT, I'm an unpaid beta tester, so I expect inaccuracies. The main problem is that it is cheerfully and confidently wrong. Even if it said "this answer is accurate to within xx%", it would be a start.
The issue for me isn't that it's wrong. By using ChatGPT, I'm an unpaid beta tester, so I expect inaccuracies. The main problem is that it is cheerfully and confidently wrong. Even if it said "this answer is accurate to within xx%", it would be a start.
I'm not sure how it could be giving this percentage. In its world, the answer just is, accurate to 100% based on the training. You'd need an oracle or the like to check on the output, which we don't have (way less shiny work).
I've only toyed around with training machine learning models, but each time it would give a percentage of probability (78% probability this thing is a handwritten 2 and not just a squiggle, for example). I have no experience training LLMs at all, so maybe it's different.
Well, at least know we have a benchmark where we can test it. Imagine it at 20% being accurate, being confident while not knowing anything about the subject etc. Do we have a reasonable argument why it can't be 40% in the next 6 months, 80% a bit later and keep improving? There are so many experiments going on, also with tools that we use to gather accurate information. I don't see a reason why it can't get better.
ChatGPT is one of the greatest cultural pushes toward originality ever. It is the perfect summary of an era focused on making derivative outputs from historical data, and really shows how ultimately limited and uninteresting those outputs are. Rather than decimating the creative fields I think it is going to make them all explode with vibrance as we finally understand that it is novelty that makes new culture compelling and valuable. For this the bots have no way to truly compete, and we, when unburdened by the need to faithfully reproduce the past, can at last focus on producing the originality that makes culture slappppp
ChatGPT is like a lower manager that has to create a last minute presentation for upper management, it does it by copy pasting google search results from a list of big tech terms.
It’s clear that the AI snake oil that is being sold here by the AI bros is no better than the hype and mania of the crypto bros of 2021.
If anything it just shows that by just using it, it fails to live up to the hype and certainly falls flat on competing against search engines with it hallucinating its results.
Another orchestrated failed attempt at selling Microsoft-flavoured AI snake-oil with techno-speak to the markets in order to create a worse search engine than Google.
If anything it just shows that by just using it, it fails to live up to the hype and certainly falls flat on competing against search engines with it hallucinating its results.
Another orchestrated failed attempt at selling Microsoft-flavoured AI snake-oil with techno-speak to the markets in order to create a worse search engine than Google.
While a real researcher would never pad the references section of their paper with a ton of useless references he himself has only skimmed
I've had ChatGPT generate code that imported a fake Go library, and proceeded to use it and its made up functions for it, so this seems like less of a stretch.
It's not perfect, you can't expect it to always be correct, factual and logic. But it gives you something to with. The companies that are rolling out LLMs need to fix these flaws, though. That's where the competition is moving to.
This specific example already feels a bit outdated because Bing can use web searches and pull the titles of actual papers (I tried it and it works well). Improving factual consistency of LLMs is an active research area and we should see some good improvements this year.
It even suggests "Wang et al" as the author
It's pulling information out of it's ass to look intelligent like a human would. Impressive.
Isn't that.... Its job?
How well would it perform on Jeopardy?
scigen did this first
When you really start to prod this and think "Ok, this tool by design is going to produce highly plausible but completely untrue information" do we really think the sensible next step is "Better integrate this very tightly into our primarily tool for finding out accurate information"? Because that appears to be the current plan.