What Google Won't Let MS Click(google.com)

7 pointsby byrneseyeviewil y a 19 ans5 comments

http://www.google.com/robots.txt

5 comments

gojomoil y a 19 ans

Misleading headline: the disallows affect all crawlers equally.

Missing context: what is someone, expert at reading robots.txt or not, expected to find interesting in the target link?

Bad submission.

Goladusil y a 19 ans

robots.txt is not a security feature. It's a courtesy that saves resources and makes life a lot easier for everyone who is willing to play by the rules.

rainsillil y a 19 ans

This is a bad submission. False headline. How do you vote an article down?

rmsil y a 19 ans

Making a comment saying it is a bad submission is great, just please don't say "This article should be on reddit!!!!!!!!"

chengmiil y a 19 ans

Does this even matter? How many bots actually respect robots.txt?