Context size limits are usually the reason. Most websites I want to scrape end up being over 200K tokens. Tokenization for HTML isn't optimal because symbols like '<', '>', '/', etc. end up being separate tokens, whereas whole words can be one token if we're talking about plain text.
Possible approaches include transforming the text to MD or minimizing the HTML (e.g., removing script tags, comments, etc.).
TypeSpec is great, but if you're working with Rust and you're about to write a new project that will require an OpenApi spec sooner or later, I'd like to recommend a web framework that has spec generation baked in:
All you need to do is derive a trait on your response structs and in return you get an almost perfectly generated spec. Unions, objects, enums are first class citizens.
Also, if you're from coming from PHP, the controllers feel very much like symfony controllers.
P.s. Please do recommend an ORM that would feel closer to doctrine. I miss doctrine.
Every single time I see these scraping discussions I get the same thoughts:
Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app?
At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.
Is there really no way we can make it technologically impossible for them to exfiltrate user data?