Email! HUH! Yeah. What is it good for? Absolutely nothing... Bid farewell to the weekend and a cheery hello to the week with another seepage from The Register's confessional in the form of our Who, Me? column. Eagled-eyed readers may remember Jon from Friday's data centre disaster. While he might have been forgiven for failing to sniff the burning of UPS circuitry then, today's tale is …
I've got no sympathy for companies who don't pay attention to this kind of thing. People make mistakes, it's happened to everyone. The company should be taking steps to make sure things like this are caught and fixed. This mailing list was so important to them that no-one at all ever checked how many people had subscribed to it?
Okay, a lot of things happened to muddy the waters, but I retain one thing from all this : nobody did an end-to-end test and validated the results.
Was it really necessary to have an email generated for a separate database ? When it was found that that confused the system and ended in spam, I would have tried to find another way of doing things.
Also, no monitoring of the applications' activity took place. The system went live and everyone was certain that it was working. Nobody made any sort of checks until months later, when the results were below expectations. Jon is not the only person responsible for losing two and a half million subscriptions - every manager implicated in the application was guilty of just assuming it worked without checking everything thoroughly for its first month of production.
If someone had checked the mail status regularly, they would have noticed the amount of incoming mail from server logs and the amount of actual subscriptions, and that would have revealed the issue well before losing even a hundred thousand subscriptions.
This is why you have automated scripts that run on a daily basis to get this data and email it to you or your team every morning which you then review as part of your daily checks. And if the software you have doesn't/can't do it, make something that can do it for you even if its a batch file that uses curl with some Perl chucked in to generate a text file saved to a network location you can look in if it can't be emailed out. Setting something up then not even checking it is working on even a daily basis is like writing your own P45.
"which you then ignore as part of your daily checks"
FTFY. Because yup, I get that kind of mail, and 80% of the entries are stuck on "error" and I'm in no position to fix them and even when I tell the person responsible to fix it, I don't know how long it will take until it actually is fixed. Red tape at its flypapery best.
"Red tape at its flypapery best."
That's not red tape, its just a broken system.
Anything that "tests" something and passes the result to a team that is unable to address the issue directly or raise a ticket to have it addressed is almost impossible to differentiate from a system that is untested.
Red tape would imply that internal procedures were preventing this system working or working quickly when the reason appears to be "no one gives a shit"
Solution? Raise a ticket for each failed test and wait for either the test to be stopped or the problem to be fixed depending on the capabilities of those involved.
maybe, but then it's not your a$$ on the line when management descends months down the track for the system not working and you ignoring the reported failures. If you raised tickets, you can successfully point to the fact that you've raised tickets continuously and that the people whose job it was to fix it, were the ones not doing their job...
What, you knew all those tickets were just being ignored for how long and didn't escalate it to management?
Of course, if you had escalated it, all you'd see is a bunch of tickets being closed (very likely without any updates being made) and nobody would tell you what had happened, but you'd see the issue crop up again within a week. Then raise a new ticket and start over.
Escalate to management too frequently, and you can be faulted for not going through proper channels and "using the system". Not frequently enough, and you're "just letting it fester". Now, the trick is: guess what is the correct frequency at which to pester management about the same issue?
EXACTLY! Where I work, we got a new computer system which is a pig's breakfast. It wasn't anywhere near ready for prime time, but our PHBs wanted to look good to their PHBs, so "GO LIVE!," they said. That didn't work out too well for anyone. They look like the idiots they are, which is good, but we have to suffer with the fallout of trying to use broken software and get our jobs done, which is not so good.
It's been most of a year, and the system still has tons of serious bugs. The programmers have so many bug reports they probably don't know where to start. They fix a few bugs now and then, but there are so MANY it will take years to get this working right. I figure they'll have it all fixed up in about 15 years or so, by which time manglement will decide it's time for new software.
Just another government IT project doing what government IT projects are famous for.
Anon for obvious reasons.
So who runs a mailing list and doesn't add themselves (and their private hotmail address - this was the 90s) to it, so that they can see exactly what their customer sees? Did it arrive? It sounds like there's more guilty parties (including every single person who ever posted a marketing message) here than just the IT dept.
I was busily making sure that the Y2K bug did not affect my employer.
It was successful. If the work had not been done they, like many others, would have faced a mess worthy of these pages.
As it was successful, we get comments in the press describing it as a load of nonsense because little happened. Little happened because people were fixing where possible and replacing where it wasn't.
I can answer that one: a combination of both.
We had the new Y2K system set up to cut over for Jan 1. It was a new pair of boxes because the hot standby for the old main box couldn't run the Y2K version of the database.
Because the year end accounts still had to be finalised the client's accountants insisted the old system had to be kept going for a couple of weeks whilst they finalised did that. So for a couple of weeks the non-Y2K system was crapping all over new orders and the vendors dialling in on a daily basis to fix the problems.
I don't know why they didn't let us move operations over to the new box and let the accountants play with the old one. If they'd given me some notice that they wanted to do that I could have arranged to bring over data on work completed in January, if that's what was needed and set the clock back to 31 Dec every day or some similar fudge.
If I knew then while slaving away on Y2K changes for programs* written in a variety of languages* that I was a figment of your imagination, I wouldn't have tried so hard.
* Some of the programs and languages I hadn't even touched before because in the PHB's opinion it was all Y2K.
Yes and no. Mostly yes.
But then there were the snake oil merchants that took the piss.
Obvious example, non-networked, ageing, stand alone laptops, with no complex software knocking around the office. that we were told had to be Y2K compliant. Actually, no. They were used for routine tasks, like typing stuff up, lesson planning or preparing materials. A back-up to floppy of any work in progress might have been a good idea. But that's all. And in fact they were so old that we hid some of them away from the Y2K team to save a few quid. A few years later I needed to use one ( network was being fully used and needed to type something up quickly). I remembered that we'd left them in a cupboard and got one out . It was fine. If it hadn't been, no loss, though I'm buggered if I knew what could have gone wrong.
Marketing list, spam ... that means no earlier than late '90s. By which time, surely Windows had long-since beaten NotWindows into an old relic?
Around the turn of the century, I had an office, with a NotWindows man just downstairs (let's call him Fred, for that was not his name). He serviced a set of regular clients, many of whom were just beginning to discover email. And they wanted email paper trails (sometimes literally on paper) that NotWindows couldn't provide.
So for several of Fred's clients, I ended up setting up a Linux mailserver with a set of custom scripts to meet their needs.
This was of course the era when M$ was absolutely evil in a big way. I suspect, but don't know for certain, that Windows would've been no better than NotWindows for Fred's Clients' needs.
there are dense ignorant lazy salespeople, and then there are salespeople who aren't getting commissions or bonuses. I call bunk on this because if someone's paycheck was directly tied to the results of this marketing program and nothing was happening, the pointy haired manglements in sales (who usually have the most bonus to lose), would have been charging in from throwing dollars at the pole
and knocking down the doors within days, not weeks or months. You don't expend the kind of resources they did and then just sit back for months wondering why nothing was happening.
I thought NetWare too.
The "competitor to Windows" thing confused me at first as I was thinking of desktop clients (which would probably have been either Mac or OS/2), but once you think server-side (and then take the obvious clue from the capitalised NW), it's clear who it was.
There was that theory that NotWindows was pretending to be a MS competitor. They would buy out other companies, such as office suite software for example and then break them.
What's that expression? Something like - don't attribute to malware what can be down to incompetence?
Looking forward to life on min wage/the dole after the company I attended went TITSUP* (nothing to do with me, the owner was a crook and did a runner with the money after ripping off the customers, the staff, the suppliers , the inland revenue, customs and excise, his own brother )
MMMM making a £5 tescos value pizza last all week... bleergh
Tramp .. because thats nearly what I became
*feel free to invent your own phrase here
Couldn't look a biscuit in the face for years after that.
KFC for me. Was so poor I had KFC for dinner just about every night... [1]
Cheap noodles as well. Lived on them for way too long. And cheap chips (what you lot call "crisps").
Damn I miss those days.. NOT! I'm not rich now, but my fridge is mostly full, my cupboards have at least a week's worth of food, my spice-rack is near full (need to replenish some stocks). Fruit and veg bins are full, freezer has plenty including meat. And not a pack of chips or biscuits in site, I eat way too good for junk like that these days! (just ignore what looks like chocolate wrappers in the rubbish bag...)
[1]It helped that I had a good friend who worked there, and she dropped off some of their unsolds on the way home each night.
When I was in my late teens I had a mate who ended up homeless so crashed on my couch for a while. I was on the dole so got money fortnightly, and had only just enough money to survive. My mate was paid weekly but didn't have any money when he arrived so we agreed I'd pay the first week's food for both of us then he'd pay for the second week.
Great plan, but the stupid <redacted> blew his wages gambling and we ended up living on sweet and sour spaghetti for a week as all I had left was a big jar of sauce and a load of pasta! Bad week.
A short and innocuous sounding phrase, that sounds temptingly nice because what goes on in one cannot possibly "interfere" with what goes on in another.
One which will lead you to discover lots of hellish work creating verbs such as reconcile, replicate, deduplicate, reformat, and if you're lucky the nicest phrase of all : "distributed transactions".
That is what I also thought, but since the problem was the subscriptions all came from the same address from the point of view of the list server, the tester would not have received any marketing mails (all going to the database servers address), and would have noticed there is some problem.
Ah, NT 4 Service Pack 6. Lets just lock all PC's out of TCP/IP until you either give them admin permissions and/or you wait 2 weeks for 6.0a. Those were the days, when you could could just flip the licensing to per seat and get unlimited client access to the servers, the BSA was the bogeyman, Winamp could fit on a floppy and really whip the llama's ass, your new modem may have had a clipper chip and my employer's email system was brought down by a chain mail by "bill gates" promising a free trip to disney in Orlando to anyone who forwarded that email. How the "tracking" would have worked over the internal firewalled network, who knows, how the rest 1990's IT press were crying about the end of IPV4 and that we were all but out of available addresses and that the end of the internet was nigh while we laughed and deployed our internal park of workstations and servers using our own public Class A ip addresses, just because we had 16 million IP's available and we could... I can almost remember the IP of our ad-hoc PC converted to web server that someone loaded smut on and gave it an FQDN too just for gits and shiggles, 9.212.something .... Ahh those were the days!
Oh, this reminds me of an incident when I was a contractor at a not-a-monopoly company in Redmond, WA.
During the development cycle of well-known and poorly designed email server software, one of the testers was testing mail lists. The tester had created many huge mailing lists just to create them. Now, you can look at your account, and see what mail lists you're on, and see who originated the mailing list. But some bright spark decided to mail the list at large, "Who owns this list, and why am I on it?" Someone else replied, "Me, too." Then someone else also replied, "Me, too." Then an even brighter spark replied, "Me, three."
And the messages being bandied about went downhill from there.
The design of the software caused replication of messages to all accounts, across all connected mail servers. The mail system was brought to a complete halt for three days while everything was set right, by the developers going directly into the databases and deleting everything associated with the list emails.
Later on, "Me, too" t-shirts were printed.
This was not the last time that stupid bright capable people brought the system to its knees with the help of mailing lists.
And that reminds me of a similar story with the NHS national email system that nobody actually wants.
At the time, there was over 3/4 of a million users - 850k according to some.
A manager decided to send a test email to a group of users but managed somehow to send it to all users. This would have just been funny and source material on the Register in a few years.
The problem was that, amongst the 850 thousand, there was an unhelpful number of managers who were so brain dead that they "replied to all" asking to be taken off the mailing list. Then there were other oxygen thieves who complained about being asked to do that so they hit Reply-To-All and it spun up. The system ground to a halt, Microsoft had a psychological boost for Office 365 sales, uninformed comments from everything from the Daily Mail to Have I Got News for You, some politicians sure that we would have done better private and others sure that this was a result of creeping privitisation.
Was there an enquiry afterwards? How come the rest of us never heard the result?
What preventative actions were taken? How about...
Stopping anyone being able to send a message to all users.
Making the Reply To All option less obvious - perhaps a couple of confirmations.
Education for the managers who caused it by emailing their uninformed comments to 850 thousand users.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
