LLMs with a verification layer work great (code with tests)
I know my field quite well and I can one-shot many useful things. I can't trust any of it but I can trust tests and verification tools.
When you set up your Claude Science instance you can see that they're connecting to crossref, semantic scholar, pubmed, ArXiv, FDA. They instruct the LLM to validate citations.
My testing with this technique indicates that method they seem to be using (rag with an instruction to check sources) will reduce the confabulation rate for citations from the base rate 50-60% for regular models (e.g. regular Claude) to 5-15% (depending on how they implemented it).
On the one hand this is way better. On the other hand it's just good enough that your spot check will look good and your work will still contain hallucinations (which is probably worse than obviously bad).
Getting to zero confabulation would require a different process. (stand-alone validation engine running in parallel in real-time which is hard but not impossible.)
Quite beneficial to keep fake science out of the scholarly record. The good news is that bad papers will show multiple signals of misconduct.
This paper was not retracted for citation fraud, but a citation check gives it a zero score (desk reject)
"Overall Score
0 / 100
High Risk
Feedback Summary
The citation analysis identified significant issues verifying the existence of cited references and references with potentially weak topical alignment.
Document-Level Findings 3
Source location and matching issues
We were able to verify 117 of 144 references against sources. 1 not found, 26 potential mismatches.
One or more references exhibit patterns associated with higher citation risk, which can arise from citation errors or automated reference generation."