Reply to post:

Robots.txt tells hackers the places you don't want them to look

Indolent Wretch

>> I have, several, and can you imagine how many pages I've had to prevent from being indexed by Google?

Luckily for you there's a solution to this, just organize the site so that the pages you don't want indexed have one or a few common roots and then there's a special file you can put in your root.

Can't remember what it's called though. Damn!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon