Typescript adds static type definitions to javascript. Microsoft's own VSCode is written in typescript. Being able to gradually move from one to another as you augment your existing code with type definitions makes it easier to integrate typescript in existing projects.
Not if you're using probabilistic data structures like Count-Min[0]. You don't care about the exact count. You just want an estimate with a certain probability. Such structures only support addition, not removal.
I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content.
https://github.com/get-set-fetch/scraper - I've been working (intermittently :) ) on a nodejs or browser extension scraper for the last 3 years, see the other projects under the get-set-fetch umbrella. Putting a lot more effort lately as I really want to do those Alexa top 1 million analysis like top js libraries, certificate authorities and so on. A few weeks back I've posted on Show:HN as you can do basic/intermediate? scraping with it.
Not capable of handling 1 mil+ pages as it still limited to puppeteer or playwright. Working on adding cheerio/jsdom support right now.
I've started this library as a means to isolate my own scraping logic from all the boilerplate a scraping project requires.
- Start with a working set of built-in plugins capable of identifying, scraping (based on CSS selectors) and storing binary or html web resources.
- Insert your own plugins containing your own custom logic.
- Query the scraped data directly from one of the supported databases or export it as csv or zip.
The initial github commit from two months ago is misleading, if you take the look at the first project (now obsolete) under
the github get-set-fetch organization it all started a few years back. If I don't run out of steam, I'll port the features from the extension next (yes, there's also a browser extension :)), mainly the ability to click and scrape dynamic javascript pages.
What functionality would you like to see in such a project? More built-in plugins covering a wider range of scraping scenarios? More storage options? Additional browser support?