It's probably mostly due to legal liability. Real or perceived. It's gonna be risky to convince a jury you did your fiduciary duty to either consumers or stockholders when opposing lawyer is saying "so you subjected my client's data to you didn't even write? Code anyone one on internet can change at anytime, etc. etc.
legal is not about what is true or right or fair or probably it is about risk reduction/mitigation. A 20% chance to lose court case is too much. Or even chance of bad PR is something to be avoided.
You put a resource on an open network and don't use any of the standard, recognized methods to indicate don't index, don't share, (nor lock it away with auth).
It's like if you put a sculpture in front yard and get upset when someone points it out in their neighborhood tour company, even worse cause yard ornaments don't have standard accepted methods of saying "don't use".
1st amendment issue. As in persons, including those that own companies have the right to associate or not associate(1) with whomever they choose. As upheld in USA with https://en.wikipedia.org/wiki/NAACP_v._Alabama
1) The "not associate" is not clearly as protected and is frequently abridged to protect "protected classes" eg housing, employment, ADA laws. I doubt any US court will uphold "bigot" as a protected class.