I don't understand why anybody without access to serious inside information is currently betting on prediction markets. It's clear that insiders are absolutely going to eat your lunch. So who are the suckers who are losing all this money?
I guess this isn't true for all things you might bet on, but it seems to be true for a lot of them.
On the flip side, if you say "don't do xyz", this is probably because the LLM was already likely to do xyz (otherwise why say it?). So perhaps what you're observing is just its default behavior rather than "don't do xyz" actually increasing its likelihood to do xyz?
Anecdotally, when I say "don't do xyz" to Gemini (the LLM I've recently been using the most), it tends not to do xyz. I tend not to use massive context windows, though, which is where I'm guessing things get screwy.
I guess this isn't true for all things you might bet on, but it seems to be true for a lot of them.