Oh My Zsh adds bloat but sets sensible default settings compared to vanilla zsh.
I tried using zsh without OMZ and I realized how many things are in OMZ which I took for granted, especially keybindings.
~150ms with minimal setup is good enough for me, although I would prefer <50ms.