An alternative algorithm which would probably converge faster than 100 questions would be something like Elo or Glicko 2.
A word's "difficulty" would be some function of how rare it is. Once you have a reasonable estimate of the user's "skill" you can infer that a user won't know more difficult words. The benefit of this is you're not spending time asking the user about words they probably know.
Of course it's possible at an individual level, difficulty does not monotonically increase as a function of how rare the word is. A person might be very familiar with a domain-specific subset of English. But the "stratified sampling" approach will also have this problem.
There is a similar problem in chess, where players have ratings which really only change on one dimension. So there can theoretically be a mismatch when puzzles are also scored on a single axis, since a "harder" puzzle that contains a motif a player is familiar with will actually be easier for the player.
I’m working on an iOS app, One Million Checkmates [1]. It scratches an itch I had of chess puzzles for a long plane ride. This app has a functionally unlimited number of puzzles, all offline.
There was a decent amount of work involved in getting the download size reasonable since we need to store all valid moves in a position. There are puzzles with over 40 million valid move sequences, so I had to aggressively prune and compress the move trees.
I'll second this. Much better to set up a second machine you can ssh/tailscale into. If a training run takes down your training machine, you don't want it to also take down your home entertainment server.
You'll also have some fun pinning down the difference between an "inaccuracy", a "mistake", and a "blunder". These are meaningful delineations for humans but not for a chess algorithm. Objectively, any amount of centipawn loss either changes the best possible outcome for the player or it does not.
So in practice, a drop in win probability greater than 14% is considered a blunder on Lichess.
From an ML perspective, this is basically logistic regression with a single feature. However, once we leave the realm of theoretical centipawn value and begin to optimize predictive power, we could imagine adding in other things like the players' ELOs or time remaining per player, etc.
I think there are some interesting theoretical differences between predicted win probability derived from Stockfish CP and actual outcomes. As in, you could even imagine predicting positions where certain players struggle and steering them towards those positions. [0]
A word's "difficulty" would be some function of how rare it is. Once you have a reasonable estimate of the user's "skill" you can infer that a user won't know more difficult words. The benefit of this is you're not spending time asking the user about words they probably know.
Of course it's possible at an individual level, difficulty does not monotonically increase as a function of how rare the word is. A person might be very familiar with a domain-specific subset of English. But the "stratified sampling" approach will also have this problem.
There is a similar problem in chess, where players have ratings which really only change on one dimension. So there can theoretically be a mismatch when puzzles are also scored on a single axis, since a "harder" puzzle that contains a motif a player is familiar with will actually be easier for the player.