Reply to post:

besides, even if the robots.txt was a magic solution, it's got no legal standing. Copyright is granted/assumed by default. You have to opt to give it away, not opt to keep it (as the magic robots.txt would imply)