The Robot Exclusion Standard is an example of voluntary cooperation on the Internet, something we could definitely use a lot more of. At the same time, this standard does of course not solve every possible problem there may be with robots.
For one thing, using the robots.txt file to discourage
robots that are known to violate the Robot Exclusion Standard anyway is
rather pointless, and may even be counter-productive (since the violation is
thereby both recognized and tolerated by the site subjected to it).
In order to deal with robots such as the
Whois Source SurveyBot (which openly
refuses to discriminate between sites served from the same IP address,
effectively requiring hundreds of webmasters to agree on a single policy
with respect to SurveyBot), other measures may be necessary.