back to article Email! HUH! Yeah. What is it good for? Absolutely nothing...

Bid farewell to the weekend and a cheery hello to the week with another seepage from The Register's confessional in the form of our Who, Me? column. Eagled-eyed readers may remember Jon from Friday's data centre disaster. While he might have been forgiven for failing to sniff the burning of UPS circuitry then, today's tale is …

  1. Oliver Mayes

    I've got no sympathy for companies who don't pay attention to this kind of thing. People make mistakes, it's happened to everyone. The company should be taking steps to make sure things like this are caught and fixed. This mailing list was so important to them that no-one at all ever checked how many people had subscribed to it?

    1. Anonymous Coward
      Anonymous Coward

      Someone made a mistake in not catching it

  2. Pascal Monett Silver badge

    What a string of cockups

    Okay, a lot of things happened to muddy the waters, but I retain one thing from all this : nobody did an end-to-end test and validated the results.

    Was it really necessary to have an email generated for a separate database ? When it was found that that confused the system and ended in spam, I would have tried to find another way of doing things.

    Also, no monitoring of the applications' activity took place. The system went live and everyone was certain that it was working. Nobody made any sort of checks until months later, when the results were below expectations. Jon is not the only person responsible for losing two and a half million subscriptions - every manager implicated in the application was guilty of just assuming it worked without checking everything thoroughly for its first month of production.

    If someone had checked the mail status regularly, they would have noticed the amount of incoming mail from server logs and the amount of actual subscriptions, and that would have revealed the issue well before losing even a hundred thousand subscriptions.

    1. simonlb Silver badge

      Re: What a string of cockups

      This is why you have automated scripts that run on a daily basis to get this data and email it to you or your team every morning which you then review as part of your daily checks. And if the software you have doesn't/can't do it, make something that can do it for you even if its a batch file that uses curl with some Perl chucked in to generate a text file saved to a network location you can look in if it can't be emailed out. Setting something up then not even checking it is working on even a daily basis is like writing your own P45.

      1. brotherelf

        Re: What a string of cockups

        "which you then ignore as part of your daily checks"

        FTFY. Because yup, I get that kind of mail, and 80% of the entries are stuck on "error" and I'm in no position to fix them and even when I tell the person responsible to fix it, I don't know how long it will take until it actually is fixed. Red tape at its flypapery best.

        1. Anonymous Coward
          Anonymous Coward

          Re: What a string of cockups

          "Red tape at its flypapery best."

          That's not red tape, its just a broken system.

          Anything that "tests" something and passes the result to a team that is unable to address the issue directly or raise a ticket to have it addressed is almost impossible to differentiate from a system that is untested.

          Red tape would imply that internal procedures were preventing this system working or working quickly when the reason appears to be "no one gives a shit"

          Solution? Raise a ticket for each failed test and wait for either the test to be stopped or the problem to be fixed depending on the capabilities of those involved.

          1. Phil O'Sophical Silver badge
            Unhappy

            Re: What a string of cockups

            Raise a ticket for each failed test

            And watch each ticket being put into a "no resources to address this at the moment" state.

            1. lglethal Silver badge
              Go

              Re: What a string of cockups

              maybe, but then it's not your a$$ on the line when management descends months down the track for the system not working and you ignoring the reported failures. If you raised tickets, you can successfully point to the fact that you've raised tickets continuously and that the people whose job it was to fix it, were the ones not doing their job...

              1. A.P. Veening Silver badge

                Re: What a string of cockups

                Even better, they in their turn can successfully point to manglement for failing to provide the necessary resources, preferably one level above the level responsible and coming down on them.

              2. veti Silver badge

                Re: What a string of cockups

                What, you knew all those tickets were just being ignored for how long and didn't escalate it to management?

                Of course, if you had escalated it, all you'd see is a bunch of tickets being closed (very likely without any updates being made) and nobody would tell you what had happened, but you'd see the issue crop up again within a week. Then raise a new ticket and start over.

                Escalate to management too frequently, and you can be faulted for not going through proper channels and "using the system". Not frequently enough, and you're "just letting it fester". Now, the trick is: guess what is the correct frequency at which to pester management about the same issue?

                1. UKHobo

                  Re: What a string of cockups

                  hello mystery colleague!

            2. Trollslayer Silver badge

              Re: What a string of cockups

              Which goes into their metrics - assuming management care.

            3. Anonymous Coward
              Anonymous Coward

              Re: What a string of cockups

              EXACTLY! Where I work, we got a new computer system which is a pig's breakfast. It wasn't anywhere near ready for prime time, but our PHBs wanted to look good to their PHBs, so "GO LIVE!," they said. That didn't work out too well for anyone. They look like the idiots they are, which is good, but we have to suffer with the fallout of trying to use broken software and get our jobs done, which is not so good.

              It's been most of a year, and the system still has tons of serious bugs. The programmers have so many bug reports they probably don't know where to start. They fix a few bugs now and then, but there are so MANY it will take years to get this working right. I figure they'll have it all fixed up in about 15 years or so, by which time manglement will decide it's time for new software.

              Just another government IT project doing what government IT projects are famous for.

              Anon for obvious reasons.

              1. DanceMan

                Re: another government IT project

                Sounds like Phoenix.

      2. John 110

        Re: What a string of cockups

        "....This is why you have automated scripts that run on a daily basis...."

        That's what you would do now, back then when everything was run out of our garage, we fudged it together and hoped for the best, like "Jon" did...

    2. olderbutnowiser

      Re: What a string of cockups

      So who runs a mailing list and doesn't add themselves (and their private hotmail address - this was the 90s) to it, so that they can see exactly what their customer sees? Did it arrive? It sounds like there's more guilty parties (including every single person who ever posted a marketing message) here than just the IT dept.

  3. Anonymous Coward
    Anonymous Coward

    Black-holed marketing email! A heartfelt thanks to you, sir!

    And that testing? Almost worthy of a Windows10 developer

    1. Anonymous Coward
      Anonymous Coward

      Or..

      Truly worthy of a MacOS developer?

      1. TeeCee Gold badge
        Meh

        Re: Or..

        Assume It Just Works.

  4. Spanners Silver badge
    Pint

    20 Years Ago

    I was busily making sure that the Y2K bug did not affect my employer.

    It was successful. If the work had not been done they, like many others, would have faced a mess worthy of these pages.

    As it was successful, we get comments in the press describing it as a load of nonsense because little happened. Little happened because people were fixing where possible and replacing where it wasn't.

    1. Venerable and Fragrant Wind of Change

      Re: 20 Years Ago

      I was in Italy, where noone could be arsed with the Y2K fuss. Nothing happened there either.

      1. Dan 55 Silver badge

        Re: 20 Years Ago

        So which is the worst drain on productivity - fixing it before it happens or sorting out the problems created afterwards by hand then hurriedly fixing it anyway?

        1. Doctor Syntax Silver badge

          Re: 20 Years Ago

          I can answer that one: a combination of both.

          We had the new Y2K system set up to cut over for Jan 1. It was a new pair of boxes because the hot standby for the old main box couldn't run the Y2K version of the database.

          Because the year end accounts still had to be finalised the client's accountants insisted the old system had to be kept going for a couple of weeks whilst they finalised did that. So for a couple of weeks the non-Y2K system was crapping all over new orders and the vendors dialling in on a daily basis to fix the problems.

          I don't know why they didn't let us move operations over to the new box and let the accountants play with the old one. If they'd given me some notice that they wanted to do that I could have arranged to bring over data on work completed in January, if that's what was needed and set the clock back to 31 Dec every day or some similar fudge.

      2. A.P. Veening Silver badge

        Re: 20 Years Ago

        I was in Italy, where noone could be arsed with the Y2K fuss. Nothing happened there either.

        Of course nothing happened, nothing ever happens in Italy so there was no difference whatsoever.

    2. Anonymous Coward
      Anonymous Coward

      Re: 20 Years Ago

      @ Spammers

      What a load of bollocks.

      Though after looking at your handle I congratulate you. You got me to reply. I usually do my best to ignore spam. I must try harder…

      Ishy

      1. Alister Silver badge

        Re: 20 Years Ago

        @Ishy

        What the fuck is your problem? I suppose you believe that nobody needed to do anything to fix Y2K?

        1. Doctor Syntax Silver badge

          Re: 20 Years Ago

          Reading obviously. Spammers and spanners are two different things.

      2. mr-slappy

        Re: 20 Years Ago

        It's 'Spanners', you spanner

      3. Dan 55 Silver badge
        Facepalm

        Re: 20 Years Ago

        If I knew then while slaving away on Y2K changes for programs* written in a variety of languages* that I was a figment of your imagination, I wouldn't have tried so hard.

        * Some of the programs and languages I hadn't even touched before because in the PHB's opinion it was all Y2K.

    3. Terry 6 Silver badge

      Re: 20 Years Ago

      Yes and no. Mostly yes.

      But then there were the snake oil merchants that took the piss.

      Obvious example, non-networked, ageing, stand alone laptops, with no complex software knocking around the office. that we were told had to be Y2K compliant. Actually, no. They were used for routine tasks, like typing stuff up, lesson planning or preparing materials. A back-up to floppy of any work in progress might have been a good idea. But that's all. And in fact they were so old that we hid some of them away from the Y2K team to save a few quid. A few years later I needed to use one ( network was being fully used and needed to type something up quickly). I remembered that we'd left them in a cupboard and got one out . It was fine. If it hadn't been, no loss, though I'm buggered if I knew what could have gone wrong.

  5. Venerable and Fragrant Wind of Change

    Timeline

    Marketing list, spam ... that means no earlier than late '90s. By which time, surely Windows had long-since beaten NotWindows into an old relic?

    Around the turn of the century, I had an office, with a NotWindows man just downstairs (let's call him Fred, for that was not his name). He serviced a set of regular clients, many of whom were just beginning to discover email. And they wanted email paper trails (sometimes literally on paper) that NotWindows couldn't provide.

    So for several of Fred's clients, I ended up setting up a Linux mailserver with a set of custom scripts to meet their needs.

    This was of course the era when M$ was absolutely evil in a big way. I suspect, but don't know for certain, that Windows would've been no better than NotWindows for Fred's Clients' needs.

    1. Anonymous Coward
      Anonymous Coward

      Re: Timeline

      Desqview

      Which, for getting the job done, beat todays shovelware, sorry I mean't , "hugely bloated buggy as fuck rubbish operating systems".

      Ishy

    2. EVP

      Re: Timeline

      “ This was of course the era when M$ was absolutely evil in a big way. I suspect, but don't ”

      When did that era end, again?

      1. Venerable and Fragrant Wind of Change

        Re: Timeline

        I think my first sight of them taking steps towards something non-evil after that period was some folks I encountered at ApacheCon in Dublin, 2006. Since then they've taken more steps.

  6. chivo243 Silver badge
    Windows

    First rule of support

    Always Follow UP!!! Many times if necessary!

    1. Anonymous Coward
      Anonymous Coward

      El Reg, I call you out on running this fiction

      there are dense ignorant lazy salespeople, and then there are salespeople who aren't getting commissions or bonuses. I call bunk on this because if someone's paycheck was directly tied to the results of this marketing program and nothing was happening, the pointy haired manglements in sales (who usually have the most bonus to lose), would have been charging in from throwing dollars at the pole

      and knocking down the doors within days, not weeks or months. You don't expend the kind of resources they did and then just sit back for months wondering why nothing was happening.

  7. Dan 55 Silver badge
    Unhappy

    RIP BeOS

    Because it was BeOS, wasn't it?

    1. GrumpenKraut Silver badge

      Re: RIP BeOS

      NotWindows = NetWare ?

      1. AIBailey Silver badge

        Re: RIP BeOS

        I thought NetWare too.

        The "competitor to Windows" thing confused me at first as I was thinking of desktop clients (which would probably have been either Mac or OS/2), but once you think server-side (and then take the obvious clue from the capitalised NW), it's clear who it was.

        1. Dan 55 Silver badge
          Thumb Up

          Re: RIP BeOS

          Makes sense, I was thinking of desktops too. I'm a little less cut up over NetWare.

      2. simonlb Silver badge

        Re: RIP BeOS

        I thought that was a Novell interpretation!

      3. Anonymous Coward
        Anonymous Coward

        Re: RIP BeOS

        There was that theory that NotWindows was pretending to be a MS competitor. They would buy out other companies, such as office suite software for example and then break them.

        What's that expression? Something like - don't attribute to malware what can be down to incompetence?

  8. Boris the Cockroach Silver badge
    Windows

    20 years ago..

    Looking forward to life on min wage/the dole after the company I attended went TITSUP* (nothing to do with me, the owner was a crook and did a runner with the money after ripping off the customers, the staff, the suppliers , the inland revenue, customs and excise, his own brother )

    MMMM making a £5 tescos value pizza last all week... bleergh

    Tramp .. because thats nearly what I became

    *feel free to invent your own phrase here

    1. The First Dave Silver badge

      Re: 20 years ago..

      £5 for a 'Value' Pizza ???

      1. Giles C

        Re: 20 years ago..

        It is £1.50 for a value pizza, the premium ones are a fiver. Or if you contract for the factory that makes them a lot lot less....

        1. Boris the Cockroach Silver badge

          Re: 20 years ago..

          Glad to see other people here have been down the 'almost too skint to eat' road.

          1. Long John Brass Silver badge
            Windows

            Re: 20 years ago..

            Survived for almost a month on the tea rooms bikkies. Thursday was always a treat... Choccy Bikkies!

            Couldn't look a biscuit in the face for years after that.

            1. Kiwi Silver badge
              Pint

              Re: 20 years ago..

              Couldn't look a biscuit in the face for years after that.

              KFC for me. Was so poor I had KFC for dinner just about every night... [1]

              Cheap noodles as well. Lived on them for way too long. And cheap chips (what you lot call "crisps").

              Damn I miss those days.. NOT! I'm not rich now, but my fridge is mostly full, my cupboards have at least a week's worth of food, my spice-rack is near full (need to replenish some stocks). Fruit and veg bins are full, freezer has plenty including meat. And not a pack of chips or biscuits in site, I eat way too good for junk like that these days! (just ignore what looks like chocolate wrappers in the rubbish bag...)

              [1]It helped that I had a good friend who worked there, and she dropped off some of their unsolds on the way home each night.

            2. Sapient Fridge
              Unhappy

              Re: 20 years ago..

              When I was in my late teens I had a mate who ended up homeless so crashed on my couch for a while. I was on the dole so got money fortnightly, and had only just enough money to survive. My mate was paid weekly but didn't have any money when he arrived so we agreed I'd pay the first week's food for both of us then he'd pay for the second week.

              Great plan, but the stupid <redacted> blew his wages gambling and we ended up living on sweet and sour spaghetti for a week as all I had left was a big jar of sauce and a load of pasta! Bad week.

              1. Kiwi Silver badge
                Pint

                Re: 20 years ago..

                Bad week.

                Yet those weeks can also teach you a lot - about how others deal with things, about what you can get by on and so on.

                You've learned valuable things that many others won't ever get the chance to experience! (least I hope you learned :) )

  9. SVV Silver badge

    seperate databses

    A short and innocuous sounding phrase, that sounds temptingly nice because what goes on in one cannot possibly "interfere" with what goes on in another.

    One which will lead you to discover lots of hellish work creating verbs such as reconcile, replicate, deduplicate, reformat, and if you're lucky the nicest phrase of all : "distributed transactions".

  10. Not Enough Coffee
    FAIL

    So no one in the company subscribed to their own marketing email as an easy way to check the basic process was working?

    1. Anonymous Coward
      Anonymous Coward

      So no one in the company subscribed to their own marketing email as an easy way to check the basic process was working?

      That seems like the best possible solution, combined with a calendar reminder to verify receipt of the marketing email.

      1. A.P. Veening Silver badge

        Won't work, way too much common sense.

      2. Anonymous Coward
        Anonymous Coward

        If I read it correctly, the server would allow the first 50 emails through. And of course your test subscribers would be added first, wouldn't they?

        1. MacroRodent Silver badge

          speculating

          That is what I also thought, but since the problem was the subscriptions all came from the same address from the point of view of the list server, the tester would not have received any marketing mails (all going to the database servers address), and would have noticed there is some problem.

  11. 1752
    Happy

    Version 1.1

    I know what was need, Support Pack 1 for the mailing list!

    1. bpfh Silver badge
      Pint

      Re: Version 1.1

      Ah, NT 4 Service Pack 6. Lets just lock all PC's out of TCP/IP until you either give them admin permissions and/or you wait 2 weeks for 6.0a. Those were the days, when you could could just flip the licensing to per seat and get unlimited client access to the servers, the BSA was the bogeyman, Winamp could fit on a floppy and really whip the llama's ass, your new modem may have had a clipper chip and my employer's email system was brought down by a chain mail by "bill gates" promising a free trip to disney in Orlando to anyone who forwarded that email. How the "tracking" would have worked over the internal firewalled network, who knows, how the rest 1990's IT press were crying about the end of IPV4 and that we were all but out of available addresses and that the end of the internet was nigh while we laughed and deployed our internal park of workstations and servers using our own public Class A ip addresses, just because we had 16 million IP's available and we could... I can almost remember the IP of our ad-hoc PC converted to web server that someone loaded smut on and gave it an FQDN too just for gits and shiggles, 9.212.something .... Ahh those were the days!

      1. sbt Silver badge
        Devil

        9.212.something

        So, an IBM employee, then. Hello.

        1. bpfh Silver badge

          Re: 9.212.something

          Yep, used to be, the site is abandoned now, according to the last photos on google maps. Fun times though :)

  12. Brian Miller

    Me too, me three

    Oh, this reminds me of an incident when I was a contractor at a not-a-monopoly company in Redmond, WA.

    During the development cycle of well-known and poorly designed email server software, one of the testers was testing mail lists. The tester had created many huge mailing lists just to create them. Now, you can look at your account, and see what mail lists you're on, and see who originated the mailing list. But some bright spark decided to mail the list at large, "Who owns this list, and why am I on it?" Someone else replied, "Me, too." Then someone else also replied, "Me, too." Then an even brighter spark replied, "Me, three."

    And the messages being bandied about went downhill from there.

    The design of the software caused replication of messages to all accounts, across all connected mail servers. The mail system was brought to a complete halt for three days while everything was set right, by the developers going directly into the databases and deleting everything associated with the list emails.

    Later on, "Me, too" t-shirts were printed.

    This was not the last time that stupid bright capable people brought the system to its knees with the help of mailing lists.

    1. Anonymous Coward
      Anonymous Coward

      Re: Me too, me three

      And that reminds me of a similar story with the NHS national email system that nobody actually wants.

      At the time, there was over 3/4 of a million users - 850k according to some.

      A manager decided to send a test email to a group of users but managed somehow to send it to all users. This would have just been funny and source material on the Register in a few years.

      The problem was that, amongst the 850 thousand, there was an unhelpful number of managers who were so brain dead that they "replied to all" asking to be taken off the mailing list. Then there were other oxygen thieves who complained about being asked to do that so they hit Reply-To-All and it spun up. The system ground to a halt, Microsoft had a psychological boost for Office 365 sales, uninformed comments from everything from the Daily Mail to Have I Got News for You, some politicians sure that we would have done better private and others sure that this was a result of creeping privitisation.

      Was there an enquiry afterwards? How come the rest of us never heard the result?

      What preventative actions were taken? How about...

      Stopping anyone being able to send a message to all users.

      Making the Reply To All option less obvious - perhaps a couple of confirmations.

      Education for the managers who caused it by emailing their uninformed comments to 850 thousand users.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019