On GPT-3.5 and now on GPT-4, I told it a line I could remember from a sonnet, and asked it to give me which sonnet it came from. It failed, and fabricated a sonnet that was a mashup of other sonnets.
It seems like maybe GPT-4 is not good at knowing when it does not know something?
Is this a common issue with LLMs? Also surprising (to me), it seems to give a slightly different wrong answer each time I restart the chat.
Here is the transcript in case you're curious:
```
Me: hello
GPT: Hello! How can I assist you today?
Me: I can remember a line I love from a sonnet, but can't remember which one.
GPT: Sure, I can help you identify the sonnet based on the line you remember. What is the line that you can recall?
Me: "Like a jewel hung in ghastly night"
GPT: The line you remember is from Sonnet 66 by William Shakespeare. The full line is: "And art made tongue-tied by authority, And folly (doctor-like) controlling skill, And simple truth miscalled simplicity, And captive good attending captain ill: Tired with all these, from these would I be gone, Save that, to die, I leave my love alone. But, like a jewel hung in ghastly night, Makes black night beauteous and her old face new."
```
Incorrect, the response is a mashup of at least 2 different sonnets. Still I think it's really impressive how it understood what to do with "I can remember a line I love from a sonnet, but can't remember which one." That's not a well phrased question, or even a question at all, but still it knew what I was getting at.
How soon until the 1/25-scale cray C90 gets as many MIPS as the original? Seems like the one he built is within shouting distance.