Publishers punt new web crawler blocking standards

Back to the article

Anonymous Coward

Or... #

Posted Thursday 29th November 2007 12:54 GMT

Just make a really horrible puke coloured website so nobody will bother reading the conent - like this: http://www.the-acap.org/

Mark

Regarding Copiepresse #

Posted Thursday 29th November 2007 13:38 GMT

They also sued Google when they stopped indexing their site.

Missed that bit out. Maybe Orlowski has to agree your copy before it can be printed...

Chris Williams

Re: Regarding Copiepresse #

Posted Thursday 29th November 2007 14:40 GMT

Erm, it's not in the copy because it's not relevant.

I mention the Copiepresse-Google News case as one of a few examples of publishers worried about how their content is used, which is why the APAC project was started - relevant.

The same publisher sues because the dominant search engine doesn't index them in tit for tat action - interesting, but irrelevant to this story.

See you at the tinfoil hat shop, anyway.

Mark

tinfoil not wanted #

Posted Thursday 29th November 2007 15:22 GMT

I *like* to see how people are trying to control me!

Gareth

Solution looking for a problem? #

Posted Thursday 29th November 2007 15:48 GMT

Doesn't HTTP already support an "expires" header? Can't a page be unpublished from a site or moved into an authenticated area of the site when it should no longer be visible to search engines?

robots.txt is a small, elegant, simple solution for keeping crawlers away from non-content areas which still need to be publically accessible (ie. Javascript files, site templates, etc).

This seems like rather a lot of committee-designed overly complex, redundant cruft which is forcing the hand of the "old media" way of doing things.

Mark

Re: Solution looking for a problem #

Posted Thursday 29th November 2007 17:33 GMT

The problem was that the content producers didn't want to go to the effort of understanding and obeying the current standards for the internet (such as, if your webserver is asked for a page, your webserver handing it over means you're allowing the recipient to copy it).

They want all the web indexers to change THEIR stuff to make their life easier.

DV Henkel-Wallace