Post: Not completely surprising
Not completely surprising →
Posted Wednesday 31st December 2008 01:32 GMT
In BusinessWeek hails quantum porn engine
I blocked Cuil's "Twiceler" spider with htaccess after it repeatedly ignored my site's robots.txt (which only allows Googlebot) and spidered everything it could to the tune of 1GB of data... I've wondered how many other site owners have had to do the same.
Top stories
Popular Whitepapers
- Robin Johnson: Questions and Answers
Dell CIO Robin Johnson discusses the process of transforming its own IT infrastructure - A Cure for Server Sprawl
Dell Global Infrastucture Consulting helps DeKalb Medical - Virtualization with the Intel Xeon Processor 5500 Series
A proof of concept - Data Center Savings
Realizing Data Center Savings with an Accelerated Server Refresh Strategy - Business-Critical Applications
The Benefits of Intel Xeon Processors and Windows Server 2008 R2 for Business-Critical Apps - Total cost of ownership of Dell, HP and IBM blade solutions
A comparison conducted by Principled Technologies
