Towards accurate and efficient document analytics with large language models(arxiv.org)
arxiv.org
Towards accurate and efficient document analytics with large language models
https://arxiv.org/abs/2405.04674
7 comments
RAGFlow (github.com/infiniflow/ragflow) use OCR/layout recognition/TSR(table structure recognition) to understand the document structure and context. Is there any difference between RAGFlow and ZenDB?
I read the paper and there are some similarities between ZenDB and RAGFlow, but also many differences.
The goal of RAGFlow is to use computer vision models to recognize the structure of a document, including diagrams and tables, and then to slice these structures into appropriate formats, such as table information combined with table definitions into text, which is then sent to the RAG system to be used for retrieval and answering questions.
ZenDB also makes use of computer vision models to understand documents, but it is mainly used to understand the semantic structure of documents, such as headings, phrases, etc., which also involves semantic-based text clustering. ZenDB also defines a query language specifically for querying these semantics. ZenDB is pretty useful to query and summarize long text.
I think some combination of RAGFlow and ZenDB for processing unstructured document data could be interesting to work on.
The goal of RAGFlow is to use computer vision models to recognize the structure of a document, including diagrams and tables, and then to slice these structures into appropriate formats, such as table information combined with table definitions into text, which is then sent to the RAG system to be used for retrieval and answering questions.
ZenDB also makes use of computer vision models to understand documents, but it is mainly used to understand the semantic structure of documents, such as headings, phrases, etc., which also involves semantic-based text clustering. ZenDB also defines a query language specifically for querying these semantics. ZenDB is pretty useful to query and summarize long text.
I think some combination of RAGFlow and ZenDB for processing unstructured document data could be interesting to work on.
This is an ad for ZenDB.
EDIT: Having reread the paper in full I don't hold this view anymore. Leaving up for reply posterity.
EDIT: Having reread the paper in full I don't hold this view anymore. Leaving up for reply posterity.
What? This is a paper describing a system/technique that they happen to call ZenDB.
Apologies; having read the full paper you are correct.
This paper is describing a technique to query Semantic Hierarchical Trees (SHTs) constructed from documents. I would say that the data itself *is* structured, it just exists in an unstructured medium, but now I'm just arguing...semantics.
That being said I suspect they didn't think ShtDB would catch on very well, and so went with ZenDB. Shame really.
This paper is describing a technique to query Semantic Hierarchical Trees (SHTs) constructed from documents. I would say that the data itself *is* structured, it just exists in an unstructured medium, but now I'm just arguing...semantics.
That being said I suspect they didn't think ShtDB would catch on very well, and so went with ZenDB. Shame really.
The authors are all academics…
[deleted]