Re: Not only that
Actually, that is a bit different. Without even going into htaccess properties, the robots.txt file allows you to tell Google which pages of your site it is allowed to crawl. But the problem of web sites is another one.
Currently, either you allow Google to crawl a piece of information in your site, or you don't. And if you do allow Google to crawl it, Google can analyse it and display it to the user in all kinds of ways.
So you are in a Catch-22 situation: You might have super nice information on your web site, but if you don't allow Google to crawl it, then Google will not show your web site in results, because it does not know you have the information. And if you allow Google to crawl it, Google will show your web site in results, but it might also display the same information in a big splashy frame on the right side, and then users will not click on the link to your web site, because they have your information already. They don't need you anymore.
The way I understand it, under the new propositions, web sites will be able to allow Google to crawl their web sites, but stop it from showing the information in the big splashy frame, so that users will click on the link to your web site. It is slightly more work for the users, but your contribution is better rewarded.