Feeds

back to article Reply-all email lightning storm STRIKES TWICE at Cisco

Cisco is tackling another email storm today, just weeks after its servers were deluged with millions of "reply-all" messages being sent by the networking giant's staff - with many requesting they be removed from the list. The Register has heard from Cisco workers who are currently in the, er, eye of the storm, which is …

COMMENTS

This topic is closed for new posts.

Page:

Silver badge

Ahh this is always funny when that happens.

I remember it happening at my firm (about 15000 staff) loads of people going "Please stop!!!" and not getting the fact that they were not helping.

Then the wags started going "I'm off out for coffee anyone want anything?"

That was a fun morning.

8
0
Silver badge

Even better when it includes a large attachment and has been sent to the mailing list by somebody who isn't even at the company because a friend told them it got technical support (and some twunt hadn't locked the list to accept posts from internal addresses only).

1
0
Silver badge

Even better when it includes a large attachment

I once had the good fortune to work for a large operation that connected many smaller satellite operations to its e-mail directory, and it turned out that there were suddenly two of me. The real me in NYC, and someone with the same name in Upper Bumfuck, NY.

I ended up having to write a robot to deal with the floods of misdirected time sensitive and legally sensitive mail people would send me that was intended for the other one. I tried engaging the other guy to get the word out, but he remained willfully thick and insulting, until I changed the robot to stop forwarding the mail to him and simply bin it with a notification to the sender that they had the wrong guy. Things reached a head when I got notification that my brand new computer was scheduled for replacement. I managed to sort that out with his manager and I think I got his replacement machine delayed a month too. Hope so.

One particular manager had a group who always felt the need to say something when she mailed out stuff, even if it was just "Me Too" (Reply to all of course). One Monday I arrived to find my mailbox had buckled under multiple copies of huge PDFs and Word docs as everyone replied to all about each others replies about how wonderful it was. I also had sixty four notifications from the admins that my e-mail was turned off because of misuse.

So I wrote a special robot just for that manager. It recognized every member of the claque and forwarded all mail sent to me by one of them to her (which meant she got two copies for each reply), along with her original (making three copies) and an explanatory cover note ("Wrong Stevie again, Ms Manager"). Three misdirected mailings and she finally got the message (I calculate she got it a minimum of 192 times in fact).

Matters weren't helped by shirt-thick mail admins who would not grant update permissions to people so they could fill out their address book entries in a more helpful way than the minimalist default of name, phone number and SMTP mail. One of the admins offered to change MY email address when approached to help solve the issue. When I pointed out it wouldn't solve the problem but would break the address books of everyone I had a legitimate business relationship with outside our mail directory, they grudgingly agreed to add "NYC" to my address card for me.

10
0
Silver badge

Re: large attachment

Reminds me of the time I was in Tokyo during the late 90's with dial-up email and some moron at US HQ who I had never heard of sent out an email to everyone in the entire company with some nice pictures of some other person who I had never heard of who had apparently just had a birthday party in an office I had never been to.

Outlook at the time could not be told to not download that email so it sat their trying to download megabytes on a 28K dial up link.

Yeah, thanks for that.

0
0
Silver badge

Re: large attachment

Even with broadband that still happens. My roomie works in an office with a very limited mail space. They are expected to monitor mail, archive/download and delete for the server. Periodically secretaries for high level mucky-mucks send out "All Hands" type messages with html wallpapers, flashing GIFs, or overly large PDF fliers for upcoming celebrations. In fact just the other day her "funny for the day" comment when she got home was one such message had taken more than a fifth of her mail space that morning. Mind you, there are cases in which she is legitimately expecting largish images to be coming in so she can look at photos of stuff to determine failure modes of equipment.

0
0
Anonymous Coward

"I can't find my friend Spartacus in the phone book"

7
0
Trollface

Troll nostalgia

Reminds me of something classic I did at school:

In outlook web access three friends and I set up an e-mail rule each,

If a received e-mail had "SPAM" in the subject then forward it to the other three and then delete it

one person then sends one e-mail and sits back. The admin realised after a few 10's of minutes why the servers were bogged down and tried to *e-mail* us to ask us to stop. Naturally given the rules we didn't even get that e-mail for 5 hours.

Not sure how I didn't receive any punishment. Nor when we used a similar technique to bombard a different friend with 15,000 e-mails in 10 minutes

5
1
Bronze badge
Mushroom

Re: Troll nostalgia

A troll on alt.www.webmaster tried the email-bomb tactic on me once. He worked for an ISP and left a trail. What he didn't know was, though I'd moved away, I knew his boss from when we were younger. Last laugh = Best laugh.

2
0

Re: Troll nostalgia

If Outlook existed when you were at school then you are not permitted to use the word nostalgia you young whipper-snapper. :p

6
0

$600,000?!

Anyone care to enlighten me as to how a flood of emails add up to that much damage? I mean, that's some fairly hefty pocket change for what, a few minutes pressing delete one morning? A mild nuisance?

Or is this one of those silly-bugger accountant sums from never-never land where you say 35,000 employees (citation needed) each had to spend (citation needed) an 30 minutes (citation needed), at an average cost to the company of $35/hour (citation needed)...

5
0

Re: $600,000?!

If you count the lost productivity from the entire workforce being without email for at least half a day, you'll probably get somewhere around that figure. Then if you add any on-call workers and their expenses claims that will add a bit more. It's not so much that everyone had to delete a thousand emails, but that the servers had to cope with 34 million emails within a few minutes and promptly crashed in flames.

0
0

Re: $600,000?!

Service providers always refer to lost productivity as an "inconvenience".

Surely this number comes from something else...

1
1
Silver badge

Re: $600,000?!

Reminds me of oil companies and their claims they have lost several million in production for being down for a few hours. That oil is still in the ground, its not been lost. You just have a small team sitting about idle for a few hours.

0
1

Re: $600,000?!

Particularly since they'd probably point to lost production as a reason to push the price up a fraction for the next week, generating 10 times that in extra profits.

Cynical? Me?

0
0
Facepalm

I used to work at a large international firm...

One of the guys accidentally sent out a message to all 50k+ employees and even worse had read receipts turned on (he was finance and they always have them on by default)....So within seconds he realised what he had done so he tried to do a recall.

Then followed about 2 days of him sat there unable to do any work as every couple of seconds up would pop a notification of his email being read (followed by a "fail to recall").

OK so I could have killed the response messages and turned off his notifications etc. but no 'cos he wouldn't learn anything that way... :)

22
0
Silver badge

Certification Programme, anyone?

Obviously someone should set-up an enormously expensive Certified Emailing Professional scheme, requiring regular renewal, and get everyone at Cisco to enrol in it.

It's the only solution, Shirley?

19
0
Silver badge

Re: Certification Programme, anyone?

...with Bronze, Silver and Platinum level certification, required annual training hours, 3-day refresher courses in exotic locales like Kalamazoo, Michigan, proctored testing...

5
1
Anonymous Coward

Re: Certification Programme, anyone?

Don't knock Kalamazoo, some of the finest small Jazz on the planet and really good coffee

1
0
Bronze badge

Re: Certification Programme, anyone?

Yeah, and Kalamazoo is on I-94, just a couple hours from Chicago1 and a direct shot from Toronto, via Port Huron. And it's not far from the junction with I-69.

If you want out of the way, try holding the exam in Marquette.

1Or a couple of days, depending on traffic, since this is Chicago we're talking about.

1
0
Anonymous Coward

"Just as night follows day, some of those staffers committed the mortal sin of hitting reply all to the reply-all message requesting to be removed from the list."

LOL The office trolls, they just cant resist an opportunity like that.

"Other staffers than piled in by berating the "idiots" who failed to realise they were creating yet another email storm."

LOL^2 A combination of The Gullable people that dont know about the office trolls yet, and the office trolls who (only barely) managed to resist the first round opportunity temptation of "please remove me from this mailing list, me too, and me" in wait of the inevitable 2nd round of "dont reply all!".

If you're lucky, it all happened in another timezone while you were asleep, or drinking by yourself at home infront of the tv after dinner, and you get to review the whole shitstorm over your morning coffee to pick whether you want to be part of LOL or LOL^2, with the innocent defence of "oh sorry i didnt see those other e-mails yet, my INBOX was full of ...".

But seriously, think of all the innocent photons and electrons needlessly excited across ocean floors and continents to deliver all this rubbish. No one ever thinks of the poor photons and electrons....

4
0
Anonymous Coward

Blame the originator

Whoever sent out the original email is at fault. When you send an email to a large list you should use the Bcc: field for the list name and not the To, From or Cc fields. That way a "reply all" will never include the list.

4
4
Anonymous Coward

Re: Blame the originator

Thank god for Captain Obvious :^\

15
0
Anonymous Coward

Re: Blame the originator

Obviously it wasn't obvious enough

10
0
Anonymous Coward

Re: Blame the originator

"Whoever sent out the original email is at fault. When you send an email to a large list you should use the Bcc: field for the list name"

That would depend on what you're trying to achieve. If your intent is to broadcast information then you're right. If, however, your intent is to ask something from a large group of people, and the answers may prompt others to answer with more detail then it's entirely appropriate to not only use the To field but also to use the reply all function. Sometimes this is the only way to get useful and complete info, and at work I often resend a mail reply back out to the group to get further feedback because people did a direct reply to me.

Anonymous because this post is going to be deeply unpopular - it is at work generally because it's nerds I send them to who don't like to communicate.

1
4
Silver badge

Re: Blame the originator

For God's sake man, if you are going to write something like this at least take time to proof read!

You used "you're" correctly and had no spelling errors at all as far as I can see.

Intolerable laxity for someone aspiring to Anonymous Coward credentials.

6
0
Bronze badge

Re: Blame the originator

"That would depend on what you're trying to achieve": NO.

If you want to share info with a lot of people and get their input: REPLY, with the list as said in BCC.

If you want the above, but share the input with all the others: GET A WIKI/FACEBOOK/FORUM/whatever. You can even stick in a poll. It's like the future, you know.

2
0
Silver badge

Re: Blame the originator @ "what you are trying to acheive" AC

As pointed out, you are wrong. You are one of the arseholes that cause these problems - there are much more effective tools designed for multiple inputs that have been around for a long time now.

(The trouble is, you modus operandi is strangely familiar - do you work for a British university?)

3
0

Re: Blame the originator

Folks, "Blame the originator" is absolutely correct... this was like leaving a series of landmines for others to step in.

0
0
Bronze badge

Need some free publicity?

Use the Cisco Reply All Procedure.

or CRAP for short.

14
0

Re: Need some free publicity?

You can only use CRAP after you have received Special High Intensity Training of course

4
0
Silver badge
Thumb Up

Re: Need some free publicity?

Classic.

0
0
Anonymous Coward

Re: Need some free publicity?

"You can only use CRAP after you have received Special High Intensity Training of course"

Don't you mean the Special High Intensity Training Simulation Tracking Override Remedial Mailers?

1
0

Re: Need some free publicity?

Once you SHIT and CRAP, don't forget to engage in a Waste Immobilisation Process Experiment

1
0
Anonymous Coward

Re: Need some free publicity?

"You can only use CRAP after you have received Special High Intensity Training of course"

or unless you've gone through the course to setup an autoresponder, the (S)He's Out of Office Training thing.

0
0
Anonymous Coward

@AC " Blame the originator"

You can bet they'll never make that mistake again. Ever.

Or the sender was an innocent victim, having stepped away from their desk to fill their waterbottle for their drink-4L-of-water-a-day-and watch-the-pounds-dissappear-from-your-gut diet , forgetting to lock the screen in their enthusiasm for the aqua aquisition, and an opportunistic office troll took advantage of their oversight and sent the email while they were away...

(You can bet they'll never make that mistake again. Ever.).

1
0
Silver badge

I once had email accounts with different university departments. I decided to save hassle by forwarding emails from account A to account B. And also forwarding from account B to account A.

I recall I got to about 2 or 3 thousand emails before everything fell over.

8
1

@AC 101

Upvoted because I like silly people.

8
0
Bronze badge

Legend

There's an old story of the 'email laser' said to have occured at IBM, back in the early days of email.

Employee A went on holiday, setting an email rule to forward all email to his co-worker, Employee B.

Employee B, however, was also away - and set an email rule to reply to all incoming email with an out-of-office message.

As the rule was enforced by the email server, there was no network delay - the rate of back-and-forthing was limited only by the email server's own processor speed. The problem was noticed when it ran out of disk space.

4
0
Happy

Handling that volume of email must be why they came out with UCS! gr

1
0
Anonymous Coward

Back in school some chums figured out you could send modal pop up messages to every computer on campus. Turns out this included both student and staff computers, so after some particularly raunchy messages the admins locked that down.

0
0
Bronze badge

yeah...

'net send' lasted about a fortnight of my first year at Uni. At least we had the consideration to target them :P

0
0

My favorite is when you get someone in IS do a reply all to the original replay all explaining why you never do a reply all....

1
0
Anonymous Coward

Bloody annoying

I'm a Cisco employee and I could have used my phone as a vibrator for all of the action it got today.

There were two mail storms - the first was about some mundane technical question emailed to 5 people as well as (for some reason) a mailing list which is for everyone who works in engineering under one of the VPs - 99% of it was the usual crap like "how am I on the list' and "please unsubscribe me" as apparently no one in Cisco has the mental capacity to look at the list of recipients and see that one of them has "mailer list" next to it.

The second one was started when someone emailed a blank email, subject "unsubscribe" to the same list, followed by hundreds of other people doing it thinking it'd amount to anything. You can't unsubscribe from this list, it is automatically generated and permanent for as long as you work under this VP, and even if you could this is not how you do it.

I haven't got an email for a while so I guess someone in San Jose has got to work and finally put an access control list on the damn thing. Not sure why the average employee is theoretically able to email about half of the company.

Anon just bcuz.

7
0
Bronze badge

Re: Bloody annoying

" You can't unsubscribe from this list, it is automatically generated and permanent for as long as you work under this VP"

Not using modern Outlook and Exchange then? I think it was 2010 they put the "ignore" function in to block whole threads in your inbox.

0
1
Bronze badge

Re: Bloody annoying

In fact, come to think of it, for several years Outlook has also warned people that they are about to mail a lot of people if they use a big group. Presumably because Microsoft found it cheaper to change the software than watch this kind of nonsense. Cisco ought to upgrade :)

2
0
Anonymous Coward

Re: Bloody annoying

This multi-billion dollar giant only finally upgraded to the latest Exchange as of last year. Before then it was on good old 2003.

The ignore button was mentioned by someone but it was drowned by the sea of idiots (and by people changing the subject which would probably defeat the filter anyway). I counted probably close to 500 emails before they stopped flowing.

There are rumors that the original email that started it all didn't have the mailing list CC'ed when the sender sent it, and that Outlook or Exchange mangled it to include it. I can believe that, the original email was certainly not worthy of being sent to a VP's mailing list.

0
0
Anonymous Coward

Re: Bloody annoying

I'm also a cisco employee and can confirm the ordeal lasted from UK morning time till about 4pm. I see it starting with a very innocent email on a technical discussion. As for the ignore button in outlook. A lot of staff have their emails forwarded to work phones. I don't have a blackberry myself but I have the feeling that it doesn't have the 'ignore' button. One email to the list from a US staff complained about being woken up at 3am by a non stop vibrating phone. Also many of the replies changed the subject like to 'PLEASE STOP THE SPAMMING' or 'I DONT WANT TO BE ON THIS LIST'. I ended up creating a rule in outlook to ignore everything sent to that VP's mailing list.

0
0

Re: Bloody annoying

"One email to the list from a US staff complained about being woken up at 3am by a non stop vibrating phone"

Which illustrates how totally screwed up modern communications technology is: forwarding your email to a phone, then leaving it on all night. How messed up is that?

4
0
Anonymous Coward

A final tweak if its your last day there anyway

Change the subject line

Add confirm read receipt

Add a forwarding rule to your home email.

The read confirmations were showing up FOUR MONTHS LATER

More fun than putting two telephones together on an all staff conference call to make it nice and echoey

1
0
Anonymous Coward

Got a good one years ago, saying someone had left their car lights on.

From Tokyo.

4
0

Page:

This topic is closed for new posts.