back to article NHS IT bod sends test email to 850k users – and then responses are sent 'reply all'

A test email sent by accident to 850,000 NHS workers has caused utter chaos after being sent from an apparently incorrectly configured* email distribution list. The sender, whom The Register will identify only as R, sent the blank message with a subject line that simply read "test" to a distribution list called …

Page:

  1. Your alien overlord - fear me

    Well, at least it was done by an IT profesional and not a dumb user !!!!

    1. Anonymous Coward
      Anonymous Coward

      I'm seeing more and more posts like this on el reg that take the side of the user.

      The thing is, nobody but you has even made the IT dept vs User comparison. It doesn't make sense here?

      I've made a separate post, but I actually have IT experience in the NHS...

      All dist lists were managed by a manager or clinical lead. The actual IT function was merely dictated to by staff

      I felt lonely then pushing back against 'testing' a dist list, and I doubt the culture has changed.

      Everyone, be it a user or especially IT staff, could do with understanding the logic behind email.

      I would like to know if the person implicated was following orders....

      NHS culture suggests they won't be going anywhere though.

      1. Anonymous Coward
        Anonymous Coward

        Restrictions?

        Where I used to work they imposed restrictions on the use of the "all" mailing list after an employee used it to express an ethical conflict with a bonus scam scheme. Naïve or not, it was relatively well formulated, but the next day that email had magically vanished from every mailbox in the company as if it never existed (except from my machine because I dumped it in print :) ). I heard later that he was marched off the premises but I wouldn't feel sorry for him - I'll get to the why later.

        From that moment on, the "all" mailbox could only be used by a restricted set of people. Not that it mattered much, to entertain myself on the last day I was working there I dumped the list of all the department mailing lists and did one leaving email which copied in all those lists, in effect bypassing the restrictions on "all" just to show how pointless it was. Thanks to the "read" flag I attached I could see that I'd managed to get my email read by over 85% of the company, and that included 3 separate hits by the CEO :).

        Oh, that employee? He went on to become a multi-millionaire. Well worth the exit IMHO :)

        1. Version 1.0 Silver badge

          Re: Restrictions?

          > Thanks to the "read" flag I attached I could see that I'd managed to get my email read by over 85% of the company, and that included 3 separate hits by the CEO

          Which is why I strip all receipt request headers at the mail server.

      2. Anonymous Coward
        Anonymous Coward

        Can any one of you thumbs down people explain WHY you voted me down?

        If not I'm taking my moral victory :)

        1. Brewster's Angle Grinder Silver badge
          Headmaster

          "Can any one of you thumbs down people explain WHY you voted me down?"

          The question mark at the end of the second paragraph; El Reg grammar pedantry is legendary.

          Honestly, you didn't get the joke and provided an incoherent response in a whiny tone. I didn't down vote you but you deserved them all. Be informative, be funny, or advance the conversation.

          1. Anonymous Coward
            Anonymous Coward

            Did you miss the rest of my post where I gave insight into the NHS and how to look at this properly?

            Sod you all, I'm right, you're wrong. :)

          2. Anonymous Coward
            Anonymous Coward

            @ Brewster

            I'm afraid I can't take your (smaller text) criticism at all seriously when after the first, throwaway line, which is valid if admittedly not "with" the joke, I go through my NHS experience.

            Which, oddly, got upvotes elsewhere in this very thread..

            I strongly suspect bandwagon jumping and not reading the full post. Your loss, not mine. It was informative and advanced the conversation. Sorry it wasn't a lame slice of sarcasm.

            Goodbye, anyway. I have better things to do than be downvoted for not laughing at crap jokes :)

            1. Jamie Jones Silver badge

              Re: @ Brewster

              Um, I didn't downvote you, but you just realise that he more you question it, the more you'll get!

              As for your message, it was written very confusingly, but what I could get out of it was that you felt it necessary to pompously "show off" to a bunch of techie readers, on a techie site, that when you worked for the NHS, it was the unit managers who actually had a say in what they used their computers for.

              The bit about them not taking IT advice is valid, but you just "authoritatively" told us something that 90% of us have experienced, and the other 10% could have guessed anyway.

              Oh yeah, it bugs me too when people put question-marks on sentences that aren't questions? :-)

              Anyway, no offense intended, (really) - I'm sure you're a nice guy, but you did keep asking for a response!

        2. Anonymous Coward
          Anonymous Coward

          Can any one of you thumbs down people explain WHY you voted me down?

          Yes, of course I can. Why?

          1. Anonymous Coward
            Anonymous Coward

            DrLifecandy: Can any one of you thumbs down people explain WHY you voted me down?

            If you have to ask this question, then that explains the down votes.

            Alternatively, of the (current) 111 comments, 14 (yes FOURTEEN) are from you. That might be called "putting yourself out"...

        3. Anonymous Coward
          Anonymous Coward

          Baaahh

          They won't let me down vote you twice ?!? Spoil sports

        4. Anonymous Coward
          Anonymous Coward

          I upvoted you to provide some much needed balance.

          * An up vote does not imply agreement in all cases

    2. herman

      MS Exchange?

      I'm amazed that MS Exchange can handle that many users and mails. At least, since it uses a database back-end, it didn't replicate the messages 80 * 1.2 million times.

      1. katrinab Silver badge

        Re: MS Exchange?

        I wonder if that is the biggest email deployment in the world that isn't an ISP or webmail provider?

        In the list of world's largest employers:

        US Department of Defense[sic] That one could be larger, but they are spread over loads of .mil and .gov domains, so maybe each one has a separate email service? Also, I don't know if all the regular squaddies have email addresses.

        People's Liberation Army Again, the regular squaddies probably won't have email addresses.

        Walmart The likes of asda.com (their UK operation) will presumably be on a separate email service, and checkout operators, shelf stackers, warehouse operatives, cleaners and so on probably won't have email addresses.

        McDonalds Only the largest if you include the employess who work for franchises, and the burger flippers probably won't have email addresses.

        NHS is 5th on the list. Just over half the employees are admin staff, and they will all have email addresses. The clinical staff and most maintenance staff will also need to use email. That leaves the likes of the cleaners, hospital porters and catering staff who maybe don't need email.

    3. asdf

      made me laugh

      I got a chuckle out of the BBC article showing a tweet of someone congratulating someone else for telling the original poster about the email going to everyone but doing so also with a reply all and a read receipt request as well. Sure that person really thought they were helping matters.

    4. Warm Braw

      at least it was done by an IT profesional

      Well, the article refers to ICT staff. In my experience, they're a very different kettle of fish...

      1. John Brown (no body) Silver badge

        "Well, the article refers to ICT staff. In my experience, they're a very different kettle of fish..."

        Yup, it means they work in Government or education.

  2. Whitter
    Boffin

    Not a bad test

    Not a bad test, the actual test being "Have we implemented sensible limits on the number of recipients in to/cc/bcc fields?"

    Sadly the result was "Fail".

    1. serendipity

      Re: Not a bad test

      Oops!

      I suspect alcohol intake of said "senior associate ICT delivery facilitator" is going to exceed NHS guidelines this evening;

      Good day at the office darling? Give me that damn bottle now!!!

      1. Peter2 Silver badge

        Re: Not a bad test

        From the article it seems that the problem is not that they had 1.2 million users in the CC box, but that somebody created a distribution group which contained other distribution groups, which probably contained other distribution groups etc.

        The perils of making changes to things when asleep on Monday mornings or Friday afternoons...

        1. VeNT

          Re: Not a bad test

          This is why the distribution lists are supposed to be vetted

          1. Anonymous Coward
            Anonymous Coward

            Re: Not a bad test

            Yes, I read this an immediately recalled nestled dist lists in my own NHS time.

            It screams out 'lack of administration'.

            I mean Admin, too. Just having a service desk who maintain stuff for secretaries... No thank you

          2. Anonymous Coward
            Anonymous Coward

            Re: Not a bad test

            Not just vetted, but have appropriate permissions applied. e.g. "All Staff" messages can only be sent by members of the executive committee or their P.A.s.

            1. Terry 6 Silver badge

              Re: Not a bad test

              A good point. Especially for the NHS and like organisations. Any risk of sending messages to every one, without really serious reasons, is an an unimaginable waste of time, and a serious risk that confidential information will get sent out to everyone accidentally.

        2. Anonymous Coward
          Happy

          Re: Not a bad test

          We had this very same cock-up happen a long time ago at a company I worked at: someone in HR (of course) sent an email to every email address in the corporate directory including all groups. Our email platform at the time didn't filter for duplicate recipients, so how many mail groups you were a member of determined how many copies of the pointless HR verbiage you got.

          We did an informal correlation of number of duplicates received with position in corporate food chain. Perhaps not surprisingly, we found high correlation up to middle management; the correlation then rapidly reduced as seniority increased. Our theory, born out by experience, was that our very senior managers had little or no interest in the day-to-day happenings of the company they were running, but instead had their rose-tinted view supplied to them by more junior drones.

        3. Anonymous Coward
          Anonymous Coward

          Re: Not a bad test

          The perils of making changes to things when asleep on Monday mornings or Friday afternoons...

          Yup. Classic PBC (posting before coffee :) ).

  3. Aqua Marina

    Just keep adding people!

    Reply to All was the bane of my life at one of the places I worked a few years ago. It got so bad I was asked if we could remove it from up high because what started off as an email conversation between 2 people, ended up going back and forth with more people added, to the point that towards the end, 30 or mo people were now in the To: / CC: field.

    There was no automated way at the time to remove or even displace it automatically without visiting each user, and making all the relevant clicks in their profile.

    1. herman

      Re: Just keep adding people!

      That is why every MS IT droid needs a Gamer Keyboard with macro keys.

  4. m0rt

    Someone is bound to get a promotion soon ... :)

  5. Doctor_Wibble
    Facepalm

    Who needs Kournikova when you have users?

    As per subject*, this proves people-power can be just as effective as automation!

    Or we add an 'are you sure?' to the reply-all function so we can pretend something was done about it when we know full well that most users will just click 'yeah whatever' like with every other prompt**.

    * complete with only-for-old-people reference

    ** did this once, had a lucky escape, but the memory of those moments of shtshtshtfufufu-phew won't ever go away

  6. David_H

    Very slow at the best of times

    As we have limited mobile signal at work, my wife, who works at an isolated doctor's site, used to occasionally send me an email to get milk on the way home, or more importantly pick up a child.

    The mails from the NHS are always delayed by an hour or two, but the worst we have seen is 9 days - 7 within the NHS servers and then 2 when my companies system threw a wobbly (presumably as the email was so old). For important and time critical communication they have to revert to printing the email and faxing an image!

    1. This post has been deleted by its author

  7. wolfetone Silver badge

    In a previous life I had to inform volunteers who used the IT systems of the charity I worked for that if they had to send an email to over 5 people (which is very common within this charity) they shouldn't put all the recipients emails in to the To: field or the CC: field, and if they could they should put them in the Bcc: field instead.

    I had this one thick (as in deliberately obtuse) person flat out say she wouldn't do what I told her to do. In front of 20 other people. I asked her why, she said she didn't want to. I ask her why again (two can be obtuse as well dickhead) and she repeated her answer. I ask her 2 more times and she fires up "Oh my God I told you already why do you keep asking the same question!?". I answered "Imagine now that was an email you sent to 200 people, all of their emails in the CC field, and every one of those recipients hit Reply All asking you why you didn't use the Bcc field and why everyone now has their email address, and why can they see everyone's reply".

    She looked fairly stupid, especially with the 20 other people sat around looking either a bit awkward at her dumbness or smiling that she had hung herself with her own dumbness.

    1. Anonymous Coward
      Anonymous Coward

      I wonder what the logic behind your two down votes is?

      1. Anonymous Coward
        Anonymous Coward

        I wonder what the logic behind your two down votes is?

        That same user and all of their friends?

      2. This post has been deleted by its author

      3. fruitoftheloon
        Happy

        @DrLifeCandy

        ...flawed...

      4. Gareth 7

        probably because the NHS.net distrubution list hid the recipients just like a BCC

        So this wasn't at all like the problem reported.

        The problem in this case was that there was an NHS.net distribution list with 1.2 million email addresses in it in the first place that anyone in the NHS could send an email to.

    2. Anonymous Coward
      Anonymous Coward

      Or how about next time instead of being an unprofessional dick, you just gently let her know that distributing others' personal details without their permission is a data protection act breach, not to mention being terribly impolite.

      1. Anonymous Coward
        Anonymous Coward

        How about you post without AC? You've been haunting here a lot just tossing out anti - IT sentiments.

        1. Ken Hagan Gold badge

          OK, I'll bite ...

          What is an anti-IT sentiment? Is "IT" now some kind of minority group that needs protection from persecution by anti-IT bigots (who probably read dreadful newspapers and vote for even worse)? And is there a way of distinguishing one Anonymous Coward from another that allows "IT" people to know when it is the same anti-IT bigot repeatedly posting the Wrong Kind of sentiment?.

  8. Anonymous Coward
    Anonymous Coward

    This was definitely caused by a very poorly configured dynamic distribution list, being that that function is now no longer available....

    DDL's were only made available 6 weeks ago or so, I'm amazed it's taken this long for the 'reply to allpocalypse' to happen.

    I'm told the offending list has been deleted, but *a lot* of people replied to all to it before it was removed - the system is choking on the backlog

    1. Anonymous Coward
      Anonymous Coward

      Watch someone forget about this and set up a different dist list doing the same thing.

      The only solution is knowledge.

      1. Anonymous Coward
        Anonymous Coward

        The ability to create a new Dynamic Distribution List is currently disabled.

        That's not to say an existing DDL couldn't be modified to create another NHSmail 'dirty bomb', but I'm not stupid enough to try it.

      2. Aladdin Sane
        Trollface

        The only solution is knowledge

        A better solution involves a cattle prod, a roll of carpet, a shovel and some lye.

        Beer is also a solution.

        1. PNGuinn
          Headmaster

          Re: The only solution is knowledge

          That's a helluva lot of work just to make some soap. Did you mean lime?

          1. Aladdin Sane

            Re: The only solution is knowledge

            Pretty sure lye does the job, forgot the large volume of polythene as well.

  9. adam payne

    I'm thinking that the Senior Associate ICT Delivery Facilitator will be having a few meetings with higher ups to explain how you can mess up the setting up /editing of a simple email distribution group.

    Well either that or everybody takes the piss out of the person for the next couple of months and then someone brings it back up every now and again just so that everybody doesn't forget.

    1. Anonymous Coward
      Anonymous Coward

      The word Facilitator is almost certainly a giveaway that this is a low position.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like