Re: oxymoronic
Fending off spiders is a little more difficult than that. Once again, the Internet was built for a club of gentlemen, not a jungle of thugs. If you have a website, give a look to what crawls it.
There's a chance they used apparently innocuous-looking sites, where people knowing what to look for could use to exchange messages - hence the need of using "advanced" searches to found out the relevant data.
A hard remainder about how some corporations became nosy to sustain their business...