Using NLP on raw HTML from scrapers More efficient than training them directly?
4 comments
It’s worth trying.
I personally don’t have the tools to do this and I don’t have enough of a budget to do this as an experiment. Where would you go to find the right people to evaluate this concept? I’ve looked around stack overflow, Reddit, etc but can’t seem to find anyone talking about this.
I found these papers:
https://arxiv.org/abs/2210.03945
https://arxiv.org/abs/2202.00217
https://arxiv.org/abs/2201.10608
My take is that fine-tuning a language model to parse HTML shouldn't be terribly difficult but you probably do need a computer with a good GPU. The one problem that current LLMs have is that they all have a limited attention window. BERT has a 512 token window and ChatGPT has a 4096 token window, where typically a token is less than a word. There are models with much longer windows (reformer) but those don't work as well as the state-of-the-art models, at least not yet.
Practically that means you can't feed a huge HTML document into the model without splitting it up first, if you do split it up you're going to lose the ability of the model to see the document as a whole (for instance match up the <div> and </div> tags)
https://arxiv.org/abs/2210.03945
https://arxiv.org/abs/2202.00217
https://arxiv.org/abs/2201.10608
My take is that fine-tuning a language model to parse HTML shouldn't be terribly difficult but you probably do need a computer with a good GPU. The one problem that current LLMs have is that they all have a limited attention window. BERT has a 512 token window and ChatGPT has a 4096 token window, where typically a token is less than a word. There are models with much longer windows (reformer) but those don't work as well as the state-of-the-art models, at least not yet.
Practically that means you can't feed a huge HTML document into the model without splitting it up first, if you do split it up you're going to lose the ability of the model to see the document as a whole (for instance match up the <div> and </div> tags)
Thank you! This is really helpful. As someone who knows a bit about NLP but isn't from that world, I don't know if I'd have found these papers or known where to look. I appreciate the help.
It seems to me that finding these items in an HTML doc would be a trivial task for NLP and this would allow us to forgo scraper maintenance in the event that websites change. Assuming that I already have hundreds of thousands or millions of records in structured data that could be used for training, would modern tools make this a more efficient way of ingesting data?