In general when I start a search, the patterns are somewhat pathological. For example when learning about a new codebase I might start with 10-100 regex and 100+ keywords. With each iteration the complexity is reduced until I find the most relevant parts of the codebase.
I know rg performs significantly better than grep out of the box.
I think grep by default is compiled without optimizations and does not use concurrency.
I would be interested in comparing the performance characteristics between the tools in more detail.
I am a big fan of ripgrep. I think the tool is great for quick simple searches.
On larger data sets with sufficiently complicated regex patterns the performance gap is small. I have also encountered situations using rg on med-large datasets where the files may contain Unicode, regular expressions, random garbage... and rg failed. I don't recall a situation like this where the error handling in grep didn't recover and move on to the next file.
Here is an example showing the performance gap to be small using (enwiki-20200401-pages-articles-multistream.xml 72G) split into 24 files on a ramdisk.
I ran this 50 times and the results are similar.
du -sch .
72G .
72G total
time ls|xargs -i -P24 rg -c "<.?>" {}
real 0m6.841s
user 2m8.766s
sys 0m5.637s
time ls|xargs -i -P24 grep -Ec "<.?>" {}
real 0m6.168s
user 1m35.058s
sys 0m24.177s
I think tools that enable formal reasoning about code are incredibly valuable.
Just going through the process the author was able to find and fix three bugs. Could you imagine a pipeline where people submitted code that had already been proven to be correct?
I currently use http://surfraw.alioth.debian.org/ along with w3m and it works great but does not appear to be currently maintained. Would you have any interest in adding similar search options to your application? Query Wikipedia, HN or Stack Overflow.... directly?
I apologize for mentioning this without submitting bug reports. I will do so in the near future.
The dataset can be found here. https://dumps.wikimedia.org/enwiki/20200420/enwiki-20200420-...
In general when I start a search, the patterns are somewhat pathological. For example when learning about a new codebase I might start with 10-100 regex and 100+ keywords. With each iteration the complexity is reduced until I find the most relevant parts of the codebase.
I know rg performs significantly better than grep out of the box. I think grep by default is compiled without optimizations and does not use concurrency. I would be interested in comparing the performance characteristics between the tools in more detail.