> To this end, we analyze 101 subjects recorded with functional Magnetic Resonance Imaging while listening to 70 min of short stories. We then fit a linear mapping model to predict brain activity from GPT-2’s activations. Finally, we show that this mapping reliably correlates (R=0.50,p<10−15) with subjects’ comprehension scores as assessed for each story.
Note that this is exactly the wrong way to form and attempt to refute a scientific hypothesis. The authors don't start with some new observations that require explanation, they start with a hypothesis already fully-formed ("...these models encode information that relates to human comprehension..."), and then go out and collect observations to confirm this hypothesis.
I'm sure that if asked, the authors would say that they are simply trying to answer a scientific question, but it's obvious that they already have the answer they want and they're just trying to find data to support it. The problem of course is that if one is already convinced of the answer, one can always find evidence to "prove" it. It's a kind of confirmation bias.
Note that this is exactly the wrong way to form and attempt to refute a scientific hypothesis. The authors don't start with some new observations that require explanation, they start with a hypothesis already fully-formed ("...these models encode information that relates to human comprehension..."), and then go out and collect observations to confirm this hypothesis.
I'm sure that if asked, the authors would say that they are simply trying to answer a scientific question, but it's obvious that they already have the answer they want and they're just trying to find data to support it. The problem of course is that if one is already convinced of the answer, one can always find evidence to "prove" it. It's a kind of confirmation bias.