Do you have ideas for what would make a better experiment? The methodology for a literature search comparison, while simple, is the best I could come up with. We developed ~250 multiple choice questions which require a deep dive into a paper to answer, ideally with very convincing distractor answers. Then we gave 9 evaluators (post-docs and grad students in biology) a week to answer 40 questions each, without any limitations on their search. The evaluators were incentivized by providing a base pay per question completed, with a 50-100% bonus if they got enough questions correct.
Under those circumstances, the evaluators had an answer precision of 73.8%, and the AI system (PaperQA2) was 85.2%. Both the evaluators and PaperQA2 could choose not to answer on a particular question. If you look at accuracy, which takes into account not answering a question, evaluators were 67.7% and PaperQA2 was 66%. So in terms of overall accuracy -- humans still did a touch better. But when actually answering, the AI was more precise.
In terms of literature synthesis comparison, I think the methodology was pretty solid too, but would love more feedback. We had PaperQA2 write cited articles for ~19k human genes, of which there are (non-stub) Wikipedia articles for ~3.9k. It's worth noting that this is a particularly technical subset of Wikipedia articles. We sampled 300 articles that were in both sources, then extracted 500 statements from each (basically a paragraph block). These statements could be compound, or even multi-sentence statements. These statements were shuffled and obfuscated such that the origin could not be determined from the statement alone.
The statements were given to a team of 4 evaluators, who were each asked to evaluate if the information was correct as cited, i.e. did the source actually support the statement. So they had to access (if they could) and actually read all the sources. After we got the evaluator gradings back, we could compile and map each statement back to its origin for comparison. Under these circumstance, the PaperQA2 written articles were 83% cited and supported, while the Wikipedia articles were 61.5% cited and supported. Wikipedia had comparatively more uncited claims, so if we eliminate those and only focus on the cited claims themselves, then PaperQA2 had 86.1% of claims that were supported by the source and Wikipedia had 71.2%. We did an analysis of every single un-supported claim, and on Wikipedia, claims are often attributed to arbitrary or really broad sources, like a landing page to a database.
We measured PaperQA2 (https://github.com/Future-House/paper-qa) against the science portion of the RAG-Arena benchmark (https://arxiv.org/abs/2407.13998), it's the first time we've compared PaperQA2 against other systems based on Cohere or Contextual.ai. PaperQA2 achieves a 12.4% higher score than Contextual.ai on the same dataset (1,404 questions and 1.7M documents).
Great article, I’ve had similar findings! LLM based “document-chunk” ranking is a core feature of PaperQA2 (https://github.com/Future-House/paper-qa) and part of why it works so well for scientific Q&A compared to traditional embedding-ranking based RAG systems.
This is awesome! If you’re interested, you could add a search tool client for your backend in paper-qa (https://github.com/Future-House/paper-qa). Then paper-qa users would be able to use your semantic search as part of its workflow.
We used an open-source AI RAG library, PaperQA2 (https://github.com/Future-House/paper-qa), to generate well cited articles for every gene in the human genome, ~15k of which had no existing prior articles. In terms of factuality, we tested our generated claims against the same gene's human written Wikipedia article in a blinded study evaluated by PhD biologists. Our system's articles were more precise on average than cited claims from existing articles. (https://paper.wikicrow.ai)
The system is scalable in that we can comfortably generate all 19.2k gene articles once per week, building a repository of cited articles that automatically syncs with all published literature.
We're sharing some experiments in designing RAG systems via the open source PaperQA2 system (https://github.com/Future-House/paper-qa). PaperQA2's design is interesting because it isn't concerned with cost, so it uses expensive operations like agentic tool calling and LLM based re-ranking and contextual summarization for each query.
Even though the costs are higher, we see that the RAG accuracy gains (in question-answering tasks) are worth it. Including LLM chunk re-ranking and contextual summaries in your RAG flow also makes the system robust to changes in chunk sizes, parsing oddities and embedding model shortcomings. It's one of the largest drivers of performance we could find.
The understanding of amyloid-beta's role in Alzheimer's disease has evolved significantly over time:
Initially, amyloid plaques were considered the primary cause of Alzheimer's, leading to the amyloid cascade hypothesis (Fantini2020Progress pages 4-7).
Research shifted focus from plaques to small oligomeric structures of amyloid-beta as the neurotoxic culprits (Fantini2020Progress pages 4-7).
It was discovered that physiological (low) levels of amyloid-beta enhance memory, while only pathological (high) levels are harmful (Morley2014The pages 1-6).
The view changed from Aβ peptides being purely harmful to recognizing their important physiological functions in the brain (Fantini2020Progress pages 1-4).
Clinical trials targeting amyloid have had mixed results, leading to questioning of the amyloid hypothesis (Hardy2009The pages 2-3).
Some researchers now propose rejecting the amyloid cascade hypothesis as insufficient to explain Alzheimer's complex biology (Herrup2015The pages 1-1).
There's growing recognition that tau pathology may correlate better with cognitive impairment than amyloid accumulation (Mullane2020Alzheimer's pages 16-16).
Current approaches are shifting towards considering amyloid-beta as one factor among many in the aging process, rather than the central cause of Alzheimer's (Josepha2001Copernicus pages 1-2).
References:
Fantini2020Progress: Jacques Fantini, Henri Chahinian, and Nouara Yahi. Progress toward alzheimer’s disease treatment: leveraging the achilles’ heel of aβ oligomers? Protein Science, 29(8):1748–1759, July 2020. URL: http://dx.doi.org/10.1002/pro.3906, doi:10.1002/pro.3906. This article has 48 citations and is from a peer-reviewed journal.
Morley2014The: John E. Morley and Susan A. Farr. The role of amyloid-beta in the regulation of memory. Biochemical Pharmacology, 88(4):479–485, April 2014. URL: http://dx.doi.org/10.1016/j.bcp.2013.12.018, doi:10.1016/j.bcp.2013.12.018. This article has 96 citations and is from a domain leading peer-reviewed journal.
Hardy2009The: John Hardy. The amyloid hypothesis for alzheimer’s disease: a critical reappraisal. Journal of Neurochemistry, 110(4):1129–1134, July 2009. URL: http://dx.doi.org/10.1111/j.1471-4159.2009.06181.x, doi:10.1111/j.1471-4159.2009.06181.x. This article has 615 citations and is from a domain leading peer-reviewed journal.
Josepha2001Copernicus: J Josepha. Copernicus revisited: amyloid beta in alzheimer’s disease. Neurobiology of Aging, 22(1):131–146, January 2001. URL: http://dx.doi.org/10.1016/s0197-4580(00)00211-6, doi:10.1016/s0197-4580(00)00211-6. This article has 146 citations and is from a domain leading peer-reviewed journal.
Hamley2012The: I. W. Hamley. The amyloid beta peptide: a chemist’s perspective. role in alzheimer’s and fibrillization. Chemical Reviews, 112(10):5147–5192, July 2012. URL: http://dx.doi.org/10.1021/cr3000994, doi:10.1021/cr3000994. This article has 775 citations and is from a highest quality peer-reviewed journal.
Herrup2015The: Karl Herrup. The case for rejecting the amyloid cascade hypothesis. Nature Neuroscience, 18(6):794–799, May 2015. URL: http://dx.doi.org/10.1038/nn.4017, doi:10.1038/nn.4017. This article has 593 citations and is from a highest quality peer-reviewed journal.
Jacobs2022It’s: Noortje Jacobs and Bert Theunissen. It’s groundhog day! what can the history of science say about the crisis in alzheimer’s disease research? Journal of Alzheimer’s Disease, 90(4):1401–1415, December 2022. URL: http://dx.doi.org/10.3233/jad-220569, doi:10.3233/jad-220569. This article has 4 citations.
Mullane2020Alzheimer’s: Kevin Mullane and Michael Williams. Alzheimer’s disease beyond amyloid: can the repetitive failures of amyloid-targeted therapeutics inform future approaches to dementia drug discovery? Biochemical Pharmacology, 177:113945, July 2020. URL: http://dx.doi.org/10.1016/j.bcp.2020.113945, doi:10.1016/j.bcp.2020.113945. This article has 68 citations and is from a domain leading peer-reviewed journal.
We are announcing PaperQA2 (https://github.com/Future-House/paper-qa), the first AI agent to achieve superhuman performance on a variety of different scientific literature search tasks. PaperQA2 is an agent optimized for retrieving and summarizing information over the scientific literature. PaperQA2 has access to a variety of tools that allow it to find papers, extract useful information from those papers, explore the citation graph, and formulate answers. PaperQA2 achieves higher accuracy than PhD and postdoc-level biology researchers at retrieving information from the scientific literature, as measured using LitQA2, a piece of the LAB-Bench evals set that we released earlier this summer. In addition, when applied to produce wikipedia-style summaries of scientific information, WikiCrow, an agent built on top of PaperQA2, produces summaries that are more accurate on average than actual articles on Wikipedia that have been written and curated by humans, as judged by blinded PhD and postdoc-level biology researchers.
PaperQA2 allows us to perform analyses over the literature at a scale that are currently unavailable to scientists. At FutureHouse, we previously showed that we could use an older version (PaperQA) to generate a Wikipedia article for all 20,000 genes in the human genome, by combining information from 1 million distinct scientific papers. However, those articles were less accurate on average than existing articles on Wikipedia. Now that the articles we can generate are significantly more accurate than Wikipedia articles, one can imagine generating Wikipedia-style summaries on demand, or even regenerating Wikipeda from scratch with more comprehensive and recent information. In the coming weeks, we will use WikiCrow to generate Wikipedia articles for all 20,000 genes in the human genome, and will release them at wikicrow.ai. In the meantime, wikicrow.ai contains a preview of 240 articles used in the paper.
In addition, we are very interested in how PaperQA2 could allow us to generate new hypotheses. One approach to that problem is to identify contradictions between published scientific papers, which can point the way to new discoveries. In our paper, we describe how ContraCrow, an agent built on top of PaperQA2, can evaluate every claim in a scientific paper to identify any other papers in the literature that disagree with it. We can grade these contradictions on a Likert scale to remove trivial contradictions. We find 2.34 statements per paper on average in a random subset of biology papers that are contradicted by other papers from anywhere else in the literature. Exploring these contradictions in detail may allow agents like PaperQA2 and ContraCrow to generate new hypotheses and propose new pivotal experiments.
Under those circumstances, the evaluators had an answer precision of 73.8%, and the AI system (PaperQA2) was 85.2%. Both the evaluators and PaperQA2 could choose not to answer on a particular question. If you look at accuracy, which takes into account not answering a question, evaluators were 67.7% and PaperQA2 was 66%. So in terms of overall accuracy -- humans still did a touch better. But when actually answering, the AI was more precise.
In terms of literature synthesis comparison, I think the methodology was pretty solid too, but would love more feedback. We had PaperQA2 write cited articles for ~19k human genes, of which there are (non-stub) Wikipedia articles for ~3.9k. It's worth noting that this is a particularly technical subset of Wikipedia articles. We sampled 300 articles that were in both sources, then extracted 500 statements from each (basically a paragraph block). These statements could be compound, or even multi-sentence statements. These statements were shuffled and obfuscated such that the origin could not be determined from the statement alone.
The statements were given to a team of 4 evaluators, who were each asked to evaluate if the information was correct as cited, i.e. did the source actually support the statement. So they had to access (if they could) and actually read all the sources. After we got the evaluator gradings back, we could compile and map each statement back to its origin for comparison. Under these circumstance, the PaperQA2 written articles were 83% cited and supported, while the Wikipedia articles were 61.5% cited and supported. Wikipedia had comparatively more uncited claims, so if we eliminate those and only focus on the cited claims themselves, then PaperQA2 had 86.1% of claims that were supported by the source and Wikipedia had 71.2%. We did an analysis of every single un-supported claim, and on Wikipedia, claims are often attributed to arbitrary or really broad sources, like a landing page to a database.
(here's the paper fwiw: https://arxiv.org/abs/2409.13740)