Will their users appreciate that they disregard the intent of the authors of what they index?
I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.
Yes, sorry, it was a rhetorical question in response to previous.
Taking either step you suggest (along with robots.txt or eqiv.), it would seem fair to expect that Brave, Bing, whomever, would not feel it their neutral/natural domain to include in a public index.
What if I want what I publish to be known only by word of mouth?
What if I consider (some or any of) my ideas to be un-indexable, not directly suitable to representation in any hierarchy other than those I may set them in?
I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.