It is a very real threat. At the worst, we see cases like this, where female prisoners are raped and sexually assaulted by males who were transferred to women's prisons:
To give another example, bathroom facilities for disabled people would be practically non-existent if left up to the owners of the space to provide. It's only through legislation that this has been mandated, to the benefit of those with disabilities.
Similarly with female-only spaces, legislation that protects these, for the benefit of women and girls, is becoming increasingly necessary in today's society.
If a male has personal beliefs that lead to him imposing himself upon female-only spaces, should that be allowed just because those are his personal beliefs?
Sometimes it's better to prevent people from acting on their personal beliefs, in particular where it adversely affects others.