vikp · HackerTrans

It's an interesting illustration of the state of the AI market that immediately after arguing that AI cannot do anything complex...we have an ad arguing that AI can actually do those things. Even the people telling you that AI is a hype cycle buy into the hype cycle.

First, we have this section:

But the noise continues on volume 11/10. So this is my desperate plea... Can we PUHLEASE stop this AI Confidence Theater, people? It’s doing more harm than good.

Then immediately after:

This post is sponsored by Firecrawl. Firecrawl is the web data API to search, scrape, and interact with the web at scale. Turn the live web into clean, structured data your agents can actually use.

(P.S. I personally use Firecrawl in my Lovable apps all.the.time - most recently to scrape this very blog so my AI double could use my latest posts as context.)

vikp

7 miesięcy temudiscuss

Hey, I'm the founder of Datalab (we released Chandra OCR). I see someone requested it below - happy to help you all get setup. I'm [email protected]

vikp

7 miesięcy temudiscuss

Yes, we can sign a BAA!

vikp

7 miesięcy temudiscuss

Hi, I'm a founder of Datalab. I'm not trying to take away from the launch (congrats), just wanted to respond to the specific feedback.

I'm glad you found a solution that worked for you, but this is pretty surprising to hear - our new model, chandra, saturates handwriting-heavy benchmarks like this one - https://www.datalab.to/blog/saturating-the-olmocr-benchmark ,and our production models are more performant than OSS.

Did you test some time ago? We've made a bunch of updates in the last couple of months. Happy to issue some credits if you ever want to try again - [email protected].

vikp

w zeszłym rokudiscuss

Thanks for sharing! I'm training some models now that will hopefully improve this and more :)

vikp

w zeszłym rokudiscuss

Benchmarking is hard for markdown because of the slight formatting variations between different providers. With HTML, you can use something like TEDS (although there are issues with this, too), but with markdown, you don't have a great notion of structure, so you're left with edit distance.

I think blockwise edit distance is better than full page (find the ground truth blocks, then infer each block separately and compare), but many providers only do well on full pages, which doesn't make it fair.

There are a few different benchmark types in the marker repo:

  - Heuristic (edit distance by block with an ordering score)
  - LLM judging against a rubric
  - LLM win rate (compare two samples from different providers)

None of these are perfect, but LLM against a rubric has matched visual inspection the best so far.

I'll continue to iterate on the benchmarks. It may be possible to do a TEDS-like metric for markdown. Training a model on the output and then benchmarking could also be interesting, but it gets away from measuring pure extraction quality (the model benchmarking better is only somewhat correlated with better parse quality). I haven't seen any great benchmarking of markdown quality, even at research labs - it's an open problem.

vikp

w zeszłym rokudiscuss

I ran a partial benchmark against marker - https://github.com/VikParuchuri/marker .

Across 375 samples with LLM as a judge, mistral scores 4.32, and marker 4.41 . Marker can inference between 20 and 120 pages per second on an H100.

You can see the samples here - https://huggingface.co/datasets/datalab-to/marker_comparison... .

The code for the benchmark is here - https://github.com/VikParuchuri/marker/tree/master/benchmark... . Will run a full benchmark soon.

Mistral OCR is an impressive model, but OCR is a hard problem, and there is a significant risk of hallucinations/missing text with LLMs.

comments