Reply-all email lightning storm STRIKES TWICE at Cisco Cisco is tackling another email storm today, just weeks after its servers were deluged with millions of "reply-all" messages being sent by the networking giant's staff - with many requesting they be removed from the list. The Register has heard from Cisco workers who are currently in the, er, eye of the storm, which is …
Don't feed the alligators in the pond
What an unhelpful message. Where am I supposed to feed them?
it habituates them to human contact
In much the same way that McDonalds has habituated me to cheeseburger "contact", I suppose.
Our facility had a pond as well... but we were in Minnesota. Any gator that could live in our pond would have been truly terrifying!
Tonight on SyFy: SnowGators!
Stuff like this happens all the time in large companies, e.g. Microsoft. Yes, it bogs things down. No, it's not a cataclysm.
To start a storm: send an email out to a distribution list.
Someone replies, "I don't want this email. Please remove me."
Someone else replies, "Me, too!"
Someone else replies, "Me, three!"
And then really stupid comments follow.
Solution: fire people who can't rationally deal with email. Also, fire the people who have flagged themselves as idiots.
Personally, I'm always amazed that the mail servers aren't configured to reject messages with large attachments. I have a number of ... helpful ... coworkers who insist on sending emails with multi-megabyte attached files that are already available on internal web servers. So they've gone through the trouble of downloading the file and attaching it to their email, rather than just, y'know, putting the URL in the message. (Or letting me find it myself, which would take me approximately two seconds.)
If I were BOFH around here, the MTAs would be locked down tight against this sort of thing. And messages to lists with attachments would be filtered, too. And messages to lists with empty bodies, or with bodies that just say something like "unsubscribe". (Of course, if I were in charge of corporate IT, there would be many changes indeed. Oh, the wailing and gnashing of teeth that would ensue. But then what's the point of tyranny if you don't enjoy it?)
People won't get in the habit of using appropriate tools until you provide the incentives, and that includes making it more difficult to use the wrong tools.
"MAIL THE AUTHOR
Sorry, we experienced an internal error trying to handle your request."
I noticed in today's email storm that the "Subject: Unsubscribe" chain later turned into "Subject: UN-UNSUBSCRIBE". Peeking back in the message bodies showed why. Someone dropped this gem:
---
Subject: Re: Unsubscribe
Ah, but you can't unsubscribe like that.
You need to use:
UN-UNSUBSCRIBE
In the subject line. Capitalization is important.
For the especially bright people it might be worth digging out a recipe of casserole
---
The previous storm included some casserole recipes from snarky recipients making fun of the clueless.
Productivity killer? Yeah, because I keep laughing at the stupidity.
30 comments in and no one has asked why Exchange and/or Outlook (it will be them presumably, right?) don't have a capability to have a rule that says "if recipient_count > (say) 50 then (make extra checks before continuing)"
That used to be possible back in the 1990s, albeit probably via local customisation, but presumably Outlook and Exchange have improved since then and IT departments have got worse since then.
My "world class" employers Communications Department love to send out "All Employees" messages, typically relating yet another reorganisation of "senior leadership" which is few tens of KB of PDF. And a few hundred KB of distribution list, which would be trivial to "reply all" to. Maybe IT should have a word with them. Maybe IT don't know how to do it properly?
"That used to be possible back in the 1990s, albeit probably via local customisation, but presumably Outlook and Exchange have improved since then and IT departments have got worse since then."
Actually, the IT department has been downsized (what a polite way for management to say that they sacked all of the the smart wonks so they could have bigger bonuses come the holidays) since it's so simple now for people to configure their own mail programs (on the computers they had to bring with them).
There is an email training course at Lynda.com. The instructor covers the scenario that lead to Cisco's mess and many of the other mistakes that MBAs make when trying to use anything more complicated than a biro.
That's a bit too limited. What you actually need is a way to specify the people who can send the list, not that they be administrators. Assuming of course when you say "administrators" you mean mail admins instead of secretaries. Oddly enough my work place has some ancient Unix system for the LDAP service which provides exactly that functionality. And I mean ancient as in the current real* mail admins are getting tired of trying to keep it running.
*I'm shown in the system as a mail admin, but I consider myself a fake mail admin. I create user accounts and mailing lists. I don't know the first thing about standing up an actual mail server.
>. What you actually need is a way to specify the people who can send the list,
What I posted above is the setting for the DEFAULT value. If you want to specify a different value for a group, you have define and select the group. It's no more difficult, but it would be kind of boring what with explaning the names and groups. I didn't think anyone here would want to read it.
What surprises me is that a modern e-mail system would still buckle from this kind of thing. I mean, with postfix, one portion expands the "all@cisco.com" or whatever into 36,000 addresses, a chunk at a time, into the send queue. This chunk size is lower than the send queue size, so normal e-mail still gets into the queue and gets processed. This makes e-mails to some ill-advisedly large list like "all@cisco.com" (or whatever) slow but ordinary e-mail is not too impacted. Of course you still have the excessive number of messages to deal with then.
Last place where i worked anywhere near the email stuff, the mail servers were well resourced & tuned for high volumes and peak loads, unfortunately the same attention wasnt given to the virus/spam scanners that sat infront of them. The virus/spam filters were pedantic, buggy as hell and borked under load. And when they borked, the mail stopped flowing...
In $former_job in commercial web hosting, customers had the ability to create simple mailing lists. One of them ran a mailing list for a large, disparate charity organisation - and also ran several of said organisation's branch mail systems.
For some unknown (still) reason, they used Microsoft's SBS product with the dumbest POP3 receiver code I ever saw. The following happened:
1. Mail gets sent to list
2. Mail gets expanded to recipients
3. Mail goes to one of the aforementioned SBS POP3 connector widget, which looks at the incoming message and thinks 'this isn't for one of my addresses, it's been sent to some-list@some-domain...'
4. SBS POP3 widget sends message to mailing list
5. GOTO 1.
The list had thousands of subscribers. I lost count how many of these SBS machines were involved - at least 4 - but the resulting storm saw our customer attempt to claim £250k in damages from us for lost business! The last I knew about it, they'd backed down and as far as I know it never got to court.
Hilarious at the time, but the aftermath was bloody irritating.
No, I'd put this one on failure to have a restricted send ability on the mail group.
For the BCC you are depending on humans to be infallible as well as having unlimited willpower. Restricting who can send to the mail group/list is a technical solution that IT can implement. Others have noted other mitigation strategies IT can implement to put breakers in the system, but for a cold hard stop, it has to be a restricted mail group. Restricted send should be automatic on ANY large mail group (I'd say 50+ recipients but YMMV) and should be a consideration for any mail group. Yes it is more of an annoyance to the mail admin to maintain (but that can be delegated from real mail admins to fake ones like me), but not nearly the annoyance of dealing with a mail storm.
There is the ability to lock down the mailing lists quite well so that only owners or approved people can send email to the list, including with escrow where messages have to be approved by someone before they are fired off. I suspect that this, and the actual mailing list infrastructure runs on some shonky old Sun boat anchor as there's still a surprising amount of it at Cisco.
I have just got three "unsubscribe" emails from people who are emailing the *same* list so clearly someone still hasn't used any of the tools that are at their disposal to lock it down. I wonder if that person saw this article and wanted to try to go for three in a week.
The list is basically used by the VP / high up execs to send their usual feel good crap to the slaves in the engine room, I can't see any reason at all why someone like me should have the ability to send mail to it.
Is there any mail server that does clever de-duplication stuff that'd stop this being such a problem?
As in, an individual email and content is only stored once, and then the notifications and read monitoring is managed in the background. Shirley the big email providers would like something like that.
"an individual email and content is only stored once, and then the notifications and read monitoring is managed in the background"
DEC's ALL-IN-1 (tm) office automation and email system did that, if I remember rightly.
On VMS, on VAX
In the 1980s.
When 450MB was a big disk (RA81).
There's been lots of progress since then. Disks are bigger and cheaper and faster. Software is bigger and cheaper. IT departments are bigger.
>As in, an individual email and content is only stored once, and then the notifications and
Yes, that is what mail stores do. But when you respond to a mail message, you create a new and different mail message with new content. If you don't, then wtf are you doing pressing the send button? And your email client is modifying the message anyway -- adding that "in reply" or "forwarded by" text to the top of the message.
I mean, say I emailed n people internally. As I understand it, Exchange (or similar) will put one copy of that email into each of the n mailboxes. Each of those emails is identical, so the server is using n -1 times more disk space than it needs to.
The AC above points out that I'm not having an original thought as it was done years ago, but I'd have thought that at least the giant-scale mail providers like Google and Microsoft would have thought about something similar because at their scale the level of duplication must be huge.
At some point some prankster replied with the following:
------------
Ah, but you can't unsubscribe like that.
You need to use:
UN-UNSUBSCRIBE
In the subject line. Capitalization is important.
For the especially bright people it might be worth digging out a recipe of casserole
------
Immediately a ton of e-mails started arriving with "UN-UNSUBSCRIBE" subject line!
I think the stop to the whole madness was put by a guy in Israel who wrote the following:
-----------
Maybe this link will help unsubscribe: http://bit.ly/16t2VYr
-----------
The link contained (and I believe still contains) an HTML form that in Q and A format convinces people that the way to stop the storm is just to stop replying to the thread.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
