Context size limits are usually the reason. Most websites I want to scrape end up being over 200K tokens. Tokenization for HTML isn't optimal because symbols like '<', '>', '/', etc. end up being separate tokens, whereas whole words can be one token if we're talking about plain text.
Possible approaches include transforming the text to MD or minimizing the HTML (e.g., removing script tags, comments, etc.).
TypeSpec is great, but if you're working with Rust and you're about to write a new project that will require an OpenApi spec sooner or later, I'd like to recommend a web framework that has spec generation baked in:
All you need to do is derive a trait on your response structs and in return you get an almost perfectly generated spec. Unions, objects, enums are first class citizens.
Also, if you're from coming from PHP, the controllers feel very much like symfony controllers.
P.s. Please do recommend an ORM that would feel closer to doctrine. I miss doctrine.
Is there really no way we can make it technologically impossible for them to exfiltrate user data?