I am no longer with this company, but it was a small web dev company that had a problem with insecure legacy code on just about all 200 of its clients.
I wrote a software firewall to prevent the exploitation of these insecure websites as well as a monitoring system to identify any websites that got compromised in other ways.
It moved us from spending 80% of our time recovering endlessly defaced websites to 5%, pushed us on to newer and bigger projects, and made several hackers and at least one pen tester actively curse the systems in their exploit code.
This is true, but only technically. Google won't actively scrape anything disallowed in robots.txt, but those resources can still be indexed if found in the many other ways Google aggregates data, all of which is automated.
Robots.txt isn't something that bars access to information. It's just a notice that the administrator does not want large amounts of queries against certain resources.
I wrote a software firewall to prevent the exploitation of these insecure websites as well as a monitoring system to identify any websites that got compromised in other ways.
It moved us from spending 80% of our time recovering endlessly defaced websites to 5%, pushed us on to newer and bigger projects, and made several hackers and at least one pen tester actively curse the systems in their exploit code.