Ask HN: How to approach writing a web crawler that can handle JavaScript?
3 comments
I would probably use node-horseman [1], but then what do I know?
[1] https://github.com/johntitus/node-horseman
[1] https://github.com/johntitus/node-horseman
I'll check it out. Thank you!
You want something like phantom.js
http://phantomjs.org/
There are other ones that do well also, but phantom will likely do anything you need to do.
http://phantomjs.org/
There are other ones that do well also, but phantom will likely do anything you need to do.
I am interested in writing a web crawler that can handle JavaScript, i.e. that can access the DOM after any JavaScript has run.
I recognize that this could get arbitrarily complicated; however, I wanted to know whether anyone had any obvious pointers. There do seem to be some nice Java--my preferred language--crawlers out there, i.e. https://github.com/yasserg/crawler4j. However, they of course do not handle JS.
Is the standard approach to handling JS to use something like Selenium? i.e. load the page in a browser and then pull the DOM into the crawler for processing?
Thanks.