Fact-checking is really hard. How do you know what sources are reliable? How to balance or compare contradict evidences? We don’t have a clear answer yet. Therefore, instead of making the decision simply based on AI, we assist users with annotated evidences.
This past week has been transformative since Loki went open source. We're thrilled by the progress and eager to share our development plan. Thanks to your input, we have enhanced the tool's flexibility and optimized its installation.
I will take it as a compliment, lol. But I do hope ChatGPT or some agents could help me with this. Btw, our recent study on machine-generated text detection might be interesting to you.
Hi siffland! Thank you for your feedback. We understand your concern about the potential confusion given the popularity of Grafana Loki in the logging space. When naming our project, we sought a name that encapsulates our goal of combating misinformation. We chose Loki, inspired by the Norse god often associated with stories and trickery, to symbolize our commitment to unveiling the truth hidden within nonfactual information.
When we named our project, we were unaware of the overlap with Grafana Loki. We appreciate you bringing this to our attention! I will discuss this issue with my team in the next meeting, and figure out if there is a better way of solving this. If you have any suggestions or thoughts on how we can better differentiate our project, we would love to hear them.
Thank you for your suggestions, axegon!!! We will definitely consider them and add the features in a future version shortly.
Regarding the first version, we are currently working on enabling customized evidence retrieval, including local files. Our plan is to integrate existing tools like LlamaIndex. Any suggestion is greatly appreciated!
Regarding the second point, we have found OpenAI's JSON mode to be greatly helpful, and have optimized our prompts to fully utilize these advances. However, we agree that it would be beneficial to enable the use of other models. As promised, we will add this feature soon.
Lastly, we appreciate your suggestion and will work on improving the installation process for the next version.
Hello vinni2, thank you for mentioning the paper. However, I noticed that it hasn't gone through peer review yet. Also, the paper suggests that fine-tuning may work better than in-context learning, but that's not a problem. You can fine-tune any LLMs like GPT-3.5 for this purpose and use them with this framework. Once you have fine-tuned GPT, for example, with specific data, you'll only need to modify the model name (https://github.com/Libr-AI/OpenFactVerification/blob/8fd1da9...). I believe this approach can lead to better results than what the paper suggests.
I apologize for any confusion caused earlier. The core components have been defined separately (https://github.com/Libr-AI/OpenFactVerification/tree/main/fa...) to make customization easier. We understand that switching between different LLMs isn't particularly easy in the current version. However, we will be adding these features in future versions. You are most welcome to collaborate with us and contribute to this project!
I wholeheartedly agree on the necessity of linking fact-checking tools to credible sources. Currently, our team's expertise lies primarily in AI, and we find ourselves at a disadvantage when it comes to pinpointing authoritative sources. Acknowledging the challenges posed by the rapid spread of misinformation, as highlighted by recent studies, we developed this prototype to assist in information verification. We recognize the value of collaboration in enhancing our tool's effectiveness and invite those experienced in evaluating sources to join our effort. If our project interests you and you're willing to contribute, please don't hesitate to reach out. We're eager to collaborate and make a positive impact together.
Hi there, I agree that fact-checking is not something that current generative AI models can directly solve. Therefore, we decompose this complex into five simpler steps, which current techniques can better solve. Please refer to https://github.com/Libr-AI/OpenFactVerification?tab=readme-o... for more details.
However, errors can always occur. We try to help users in an interpretable and transparent way by showing all retrieved evidence and the rationale behind each assessment. We hope this could at least help people when dealing with such problems.
Hi Der_Einzige, thanks for pointing out these two great datasets! We are currently working on including customized evidence sources internally and will definitely consider these two datasets in the future version of this open-source project.
Thanks for your response. When discussing fact-checking capabilities, the key question is always: Can we guarantee that it will always offer the correct justification? While it's unfortunate, errors can occur. Nonetheless, we prioritize making the checking process both interpretable and transparent, allowing users to understand and trust the rationale behind each assessment.
