I'm concerned too, and awareness of this externality is what stays my hand, even if I set aside other concerns with LLMs. Mere efficiency, efficiency that we were fine without, and one we haven't built out infrastructure around it yet, doesn't seem worth the negative impact.
> people will always end up being frightened of "the difference"
I've also come to accept this about myself, but I had to stumble through a dark tunnel of feeling inadequate and feeling like an inhuman monster.
The typical list of traits that should not be used as a basis for discrimination is on a spectrum of how instinctual or fear-based it is, which I don't think have seen mentioned in training materials on unconscious bias.