crawling security
"Other security measures the attackers circumvented included “Sentinel”, which limits the number of requests permitted from a single IP address, and the restrictions on crawling that are implemented in its robots.txt file."
I thought that robots.txt was a polite request; hardly a security measure, especially against people who have a tendency to break your rules.