Misfiring Microsoft search bots managed to render a site used by Perl Testers almost unusable last week. A post on the CPAN Testers' blog reports that its servers were being scanned by "20-30 bots every few seconds", resulting in what developers likened to a denial of service attack. The IP addresses of the bots - which failed …
Computer mis-use act
Launching a DOS attack, can't these people be banned from the Internet?
Errm, slow down a bit..
I'm no fan of MS, but I have come across a few sensible people who work there (not at the top, no). I find it encouraging that someone at MS did bother to acknowledge and try to fix the issue (no idea how Google would deal with it either).
Secondly, assume ignorance before malice. All you need is a bug or a config mistake, so I'm not yet convinced this was an intentional DoS. Let he who is without fault throw the first MFM or RLL encoded hard disk..
All I can say is..
"Bing goes your network"
Was his comment
Sorry - we are waiting for service pack 1 :D
Failing to obey robots.txt
That's just unprofessional, or have MSFT introduced a their own proprietary bot directive technology that we are unaware of?
Re : Failing to obey robots.txt
It's THEIR internet so they'll do what they want
Oh no it's not- is it?
MSN has been doing this sort of thing on and off for ages. Several years ago I had to deny 220.127.116.11/16 on one of our company websites to deal with an identical DOS attack. (Hmm, come to look at it, I never actually removed that configuration...)
They forgot to include that line:
That's what it must be like inside MS development offices. The company has a long track record of disorganization with no clear lines of authority, plus it has such a huge staff that (as it turns out, but not unexpectedly) a fair number of them aren't fully competent. They end up with some schmoe way down deep in the bowels of the outfit writing this bot without proper specifications, without proper testing, perhaps even unknown to his/her superiors.
Or to put it slightly differently, no one's in charge. Ballmer runs around frothing at the mouth, but that's not "being in charge".
With a corporate culture like that, with billions of lines of code that they have admitted they don't completely understand, with crap development software that does nothing to auto-block certain fundamental programming errors like array overflows, you can't help but wonder whether MS is really a bloated corpse of a company, gradually swelling in the hot sun from internal decomposition — it just hasn't exploded yet.
Why is there no icon for "dead animal bloating in the sun"?
NOT attack bots ?
F*ck me, I've been unable to access a Myspace account since November last year. Maybe it isn't the work of hackers after all, who have replaced the authentic log in panel with a false data stealing one. It is only Microsoft search robots looking for some decent free music, which are running amok and causing my DOS with the error message "Too many failed log in attempts using this eMail address".
Please someone bellow in the ear of the sys admins at MySpace, saying, "it's OK you can go back to sleep !" This DOS trick may have been in effect since 1996, according to Netcraft.
"their own proprietary bot directive technology"
All your stuff are belong to us.
Who cares if a Perl site goes down?
Well, it's a Perl-related site, so who honestly cares?
Die Perl, Die!
screams the asp.net
programmer to his three imaginary friends
MS attacks free programming language site ...
... accident? Incompetence? Opening of hostilities because FUD doesn't work any more?
As MS distribute Perl in SFU, I'd go with accident.
Microsoft? Good software engineering concepts ignored. Duh!
Microsoft routinely ignores every basic concept of software engineering. I have written spiders, and reading the robots.txt and paying attention to the meta tags is trivial.
Not just me then?
I experienced similar behavior of the bots on my server.
While the googlebots are almost unnoticable, the msn bots visit each page on the site like 10 times in a row, from different addresses sometimes, which makes me think that Microsoft failed at parallelizing the thingies; more bots visit the time at once, and the same page.
I was once considering banning those retards in .htaccess, but after a few hours it stopped, so I still allow them... but not for long, Microsoft, if it starts again.
This is very bad strategy in the long term, by the way, if they think that visiting a lot of times, and very often will help their search results or something. More and more people will get annoyed, and block the bots out of their sites, which will result in a massive fail of Bing (oh wait, that already happened, let's say very massive then).
Let out early for Good Behaviour
"The behaviour of the bonkers Bing bots contrasted with the good manners of other search engine agents from the likes of Google"
The rantings of somebody who has never seen Yahoo's monster in the wild.
Plus a good server admin knows how to use IPTables (or better yet a hardware firewall) in anger - but then I guess these guys are Perl people so calling them good server admins may be a little premature...
"aimed at web developers"?!
What is the "aimed at web developers" about? Are they confusing Perl with the single-trick PHP?
John Leyden obviously
hasn't found the Universe::Galaxy module, though I'm not sure it's the real deal - the Author's name is wrong, and the new method is not documented.
As much as I hate Microsoft and hate google slightly less; a gang of 70 simultaneous googlebots did manage to bring my car owners club phpBB forum to it's knees for a few days until we used a .robots to prevent it going down below the index pages.
Not just MS ignoring robots.txt
I happened to notice in our logs yesterday that search engine bots (inc yahoo & google) access our site very frequently but our robots.txt very rarely 50+:1 ish.
"...bonkers Bing bots ..."