Long-form factuality in large language models(arxiv.org)
arxiv.org
Long-form factuality in large language models
https://arxiv.org/abs/2403.18802
4 comments
Hmmm... checking against external sources is an interesting idea -- but using Google as a source of ground truth is a little bit tricky, given how often these days Google itself is spitting up confabulated AI-generated crud (or other low-quality stuff).
Use books and papers from the Library of Genesis - that gives you good context, even while the search engines collapse
End Google
Long live The Library!
End Google
Long live The Library!
For those interested in using search-augmented "reasoning", I implemented something similar in Emerging Trajectories[1], an open source package that forecasts geopolitical and economic events. We extract facts[2] from various websites (Google searches, news articles, RSS feeds) and have the LLM generate a hypothesis on a metric.
We're tracking the info forecasts to see how well this does for future events. For example, we're pitting the LLMs against each other to predict March 2024 CPI[3].
[1] https://emergingtrajectories.com/
[2] Sample code: https://github.com/wgryc/emerging-trajectories/blob/main/eme...
[3] https://emergingtrajectories.com/a/statement/28
We're tracking the info forecasts to see how well this does for future events. For example, we're pitting the LLMs against each other to predict March 2024 CPI[3].
[1] https://emergingtrajectories.com/
[2] Sample code: https://github.com/wgryc/emerging-trajectories/blob/main/eme...
[3] https://emergingtrajectories.com/a/statement/28