Yes, it evaluate using frontier model for parsing from all 3 major provider (google, anthropic and openai). It is also easy to extend to evaluaye new model (code/dataset is available)
Build a benchmark to evaluate how good document parser work on a dataset of 2000 PDFs manually annotated, trying to evaluate accross multiple dimensions: charts, tables, text styling, text correctness, and attribution.
The benchmark evaluate performance on full page (not selected part of the pages), and evaluaye different OSS / crobtier model / commercial approach.
For transparency it is available as a HF leaderbaord.
Do you have more technical details on your process? Did you publish anything? I try to dig more on your website but the content is rather thin (I learn more about what you are doing by reading your post here).
What are the by product? You mention cement and drinkable water, but there should be other byproduct to handle such as brime?