Show HN: Vocab extractor for language learners using Stanza and frequency ranks(huggingface.co)

6 pointsby crivlaldovor 4 Monaten0 comments

Show HN: Vocab extractor for language learners using Stanza and frequency ranks

https://huggingface.co/spaces/vladvlasov256/vocab-nlp

I'm building a Telegram bot to practice Dutch. GPT-4o-mini kept picking vocabulary words I already knew, so I built a classical NLP pipeline to do it instead.

It takes a short text + learner level (A0–B1) and returns the best words to study, using Stanza for parsing and corpus frequency ranks (SUBTLEX-NL, srLex, SUBTLEX-US) for scoring. Wins at A1/A2, loses at A0 where the LLM picks more obvious words.

I also tried adding multi-word phrases (ADJ+NOUN, VERB+NOUN, phrasal verbs) backed by NPMI-scored collocation whitelists. Couldn't beat GPT there because it just "knows" which phrases matter.

For the phrase work I had to extract collocations from 100M+ OpenSubtitles lines. Published them as a free dataset: https://huggingface.co/datasets/vladvlasov256/opensubs-collo... There are 43K bigrams across English, Dutch, and Serbian.

Source https://github.com/vladvlasov256/vocab-nlp

0 comments

—