Yes this is generally the fallback approach if finding the objects via the index (xref) fails. It is slightly slower but it's a one time cost, though I imagine it was a lot slower back when PDFs were first used on the machines of the time.
If you don't have a known set of PDF producers this is really the only way to safely consume PDF content. Type 3 fonts alone make pulling text content out unreliable or impossible, before even getting to PDFs containing images of scans.
I expect the current LLMs significantly improve upon the previous ways of doing this, e.g. Tesseract, when given an image input? Is there any test you're aware of for model capabilities when it comes to ingesting PDFs?
You're right, this was a fairly common failure state seen in the sample set. The previous reference or one in the reference chain would point to offset of 0 or outside the bounds of the file, or just be plain wrong.
What prompted this post was trying to rewrite the initial parse logic for my project PdfPig[0]. I had originally ported the Java PDFBox code but felt like it should be 'simple' to rewrite more performantly. The new logic falls back to a brute-force scan of the entire file if a single xref table or stream is missed and just relies on those offsets in the recovery path.
However it is considerably slower than the code before it and it's hard to have confidence in the changes. I'm currently running through a 10,000 file test-set trying to identify edge-cases.
Yes, you're right there are Linearized PDFs which are organized to enable parsing and display of the first page(s) without having to download the full file. I skipped those from the summary for now because they have a whole chunk of an appendix to themselves.
The point, which seems to be routinely massively downvoted on here, is that both things can be true at once:
- these drugs are good and a paradigm shift in the treatment of obesity (and have other benefits)
- we must not lose sight of the need to address a thoroughly sick food industry that necessitate so many people needing to use these. Junk food advertising, lack of subsidies for fresh vegetables, HFCS, food deserts, etc.
Chile is experimenting with banning junk food ads to children and is seeing some early behaviour changes.
The point which people seem to be wilfully missing is that we can have both these drugs and advocate for cracking down on a food system that deliberately poisons everyone in society. Having everyone be on this drug because we shrug and say "free market innit" while big corps continue to feed us crap is not a solution, obviously.