If you think that a woman has to meet a modesty standard that you decided on in order to participate in "nerd stuff", then you are the one with the problem.
Finite fields have multiplicative inverse only if the base is a prime, and that's what makes the rest of your usual algebra work (or not).
For example, for N = 5, 0 * 2 = 2, 1 * 2 = 2, 2 * 2 = 4, 3 * 2 = 1, 4 * 2 = 3, so the inverse of "* 2" is uniquely defined. On the other hand, for N = 4, 0 * 2 = 0, 1 * 2 = 2, 2 * 2 = 4, 3 * 2 = 2, so the inverse of "* 2" is not uniquely defined.