It has never failed to amaze me that most people think it's perfectly acceptable for Google to copy anything it wants, in its entirety. Whether it's web sites, newspaper articles, or entire books, people think it's OK simply because "It's Google" and "Well you should have used the robots.txt file". Yet if another organization tried the same thing, these people would be up-in-arms. I'm not going to start a war here, nor will I offer an opinion on whether or not I think Google has the right to do what it does. My point is simply that brand-recognition ("It's OK because it's Google") should not be a factor in whether or not you support the organization's actions.
My secondary point is that copyright protection is automatic by law. It is not opt-in or opt-out. I cannot go to the library and use a photocopier (or scanner) to copy/scan all of their books, only stopping when an author or publisher issues a request for me to not copy/scan an individual book. I don't have that right. And neither does Google or anyone else. Unless a work is in the public domain, or the author has explicitly given permission for such copying, it is illegal, no matter who does it (except under certain circumstances). Similarly, it should not be up to website designers to use the robots.txt file. I know it's there for a purpose, but there are a lot of people out there (many of them with websites) that don't know the specifics of a robots.txt file or how a search engine indexes sites. The lack of any person or company to use a technical means to stop a search engine (or anyone else) from copying their content does not remove the protections granted to them under the copyright act(s).