I'm happy to answer any questions about how the project works. Feedback is very much appreciated.
The big challenge this project has faced from the beginning is how to make the matching algorithm as accurate as possible and at the same time fast enough to scan an entire website worth of javascript within a couple of seconds. As someone with zero prior experience with search/information retrieval this was a hard task.
Accuracy-wise it's not quite there yet. In my benchmark, around 5% of identified libraries are false positives and something like 15% of bundled libraries are missed. The false positives mostly stem from cases where two libraries have almost identical content, or cases where one library has bundled a dependency into its own code.
Performance has gotten quite good. In a benchmark of popular websites, it can scan through ~1.4 websites / second, or ~3MBs of minified javascript / second (running on a 65€/month VPS).
Show HN: Bundle Scanner – Identify which NPM libraries are used on any website · HackerTrans
I'm happy to answer any questions about how the project works. Feedback is very much appreciated.
The big challenge this project has faced from the beginning is how to make the matching algorithm as accurate as possible and at the same time fast enough to scan an entire website worth of javascript within a couple of seconds. As someone with zero prior experience with search/information retrieval this was a hard task.
Accuracy-wise it's not quite there yet. In my benchmark, around 5% of identified libraries are false positives and something like 15% of bundled libraries are missed. The false positives mostly stem from cases where two libraries have almost identical content, or cases where one library has bundled a dependency into its own code.
Performance has gotten quite good. In a benchmark of popular websites, it can scan through ~1.4 websites / second, or ~3MBs of minified javascript / second (running on a 65€/month VPS).