That's really the crux of the problem, isn't it? If the organizer of a conference decides not to give a platform to neofascists who want to end democracy, they MUST also bar women who are being harassed by *chan trolls.
There's simply no way that a conference organizer could possibly handle those situations differently without being a hypocrite.