wolfgarbe · HackerTrans

Peter Norvig shows that an edit distance = 2 will cover 98.9% spelling errors. https://impythonist.wordpress.com/2014/03/18/peter-norvigs-2...

That's the reason why the default maximum edit distance of SymSpell is 2.

Now, all your 6 out of 6 examples are chosen from that 1.1% margin that is not covered by edit distance 2, presenting a rather unlikely high amount of errors within a single word.

The third-party SymSpell port from Justin Willaby, which you were using for benchmarking, clearly states that you need to set both maxEditDistance and dictionaryEditDistance to a higher number if you want to correct higher edit distances. That you neither used nor mentioned. This has nothing to do with accuracy; it is a choice regarding a performance vs. maximum edit distance tradeoff one can make according to the use case at hand.

https://github.com/justinwilaby/spellchecker-wasm?tab=readme...

pronnouncaition IS within edit distance 3, according to the Damerau-Levenshtein edit distance used by SymSpell. The reason is that adjacent transpositions are counted as a single dit. https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_di...

wolfgarbe

5 ay öncediscuss

Author of SymSpell here. Congrats on the launch of Lexiathan.

Unfortunately, the comparison of Lexiathan vs. Symspell on your website regarding accuracy is misleading.

1. SymSpell has two parameters to control the maximum edit distance. Once you set both to 3, then also terms with an edit distance of 3 are accurately corrected:

  pronnouncaition -> pronunciation

  inndappendent -> independent

  unegspeccted -> unexpected

  soggtwaee       -> software

2. SymSpell comes with dictionaries in several sizes. Once you load the 500_000 terms dictionary, then also the two remaining terms will be corrected:

  maggnificntally -> magnificently

  annnesteasialgist -> anesthesiologist

https://github.com/wolfgarbe/SymSpell/blob/master/SymSpell.B...

SymSpell accurately corrects all of your examples if used properly with the correct parameters and dictionary.

Apart from that, your methodology of comparing correction accuracy by cherry-picking specific terms without statistical significance, where your product seemingly performs better, is questionable.

One would use large public corpora to measure the percentage of accurately corrected terms as well as the percentage of false positives.

Because SymSpell is Open-Source, everyone can integrate it into their applications for free, modify the code, use different dictionaries in various languages, or add terms to existing ones.

https://github.com/wolfgarbe/SymSpell

https://github.com/wolfgarbe/symspell_rs

wolfgarbe

8 ay öncediscuss

The stopword list in SeekStorm is purely optional, per default it is empty.

The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete results and perform quite well in the benchmark: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#be...

Both Lucene and Elastic still offer stopword filters: https://lucene.apache.org/core/10_3_2/analysis/common/org/ap... https://www.elastic.co/docs/reference/text-analysis/analysis...

wolfgarbe

9 ay öncediscuss

Can the index size exceed the RAM size (e.g., via memory mapping), or are index size and document number limited by RAM size? It would be good to mention those limitations in the README.

wolfgarbe

9 ay öncediscuss

Sure, but it says "High-performance" Full Text Search Engine. Shouldn't that claim be backed up by numbers, comparing it to the state of the art?

wolfgarbe

9 ay öncediscuss

Great work! Would be interesting to see how it compares to Lucene performance-wise, e.g. with a benchmark like https://github.com/quickwit-oss/search-benchmark-game

comments