back to article Canadian ISP Telus launches novel solution to deal with excess email: Crash your servers and wipe it all

Dealing with email is possibly the most tedious daily exercise that the modern digital world has forced on us. But 13 million customers of Canadian ISP Telus have discovered that not having that problem is more of a burden. Starting Thursday morning, the connection biz's entire email system went down, with not a single email …

  1. Will Godfrey Silver badge
    Facepalm

    Vultures are so cruel

    {snigger}

  2. billdehaan
    Facepalm

    This has all happened before, this will all happen again

    Back in the late 1990s, I had an account with Bell Canada. I was one of their first ADSL customers, which caused numerous problems. I was assigned a five digit account ID, but then Bell realized that in a country of 20M+ people, more than 99,999 people might want high speed internet, so they changed their system to be six digit IDs, which my account couldn't be used with. And no, it wasn't a matter of adding a leading or trailing zero.

    But that's another story.

    In April of 1999, the DSL system went down. For four days. Bell's response was that people should use dialup during the outage (which they would charge you extra for, of course). Those that did, and activated their dial up email address, discovered that in so doing, it replaced their DSL email address of the same name. In other words, DSL customer joe.smith@sympatico.ca may have had 1800 emails on the server, but if he logged in as a dialup user, and created a new user account also called joe.smith@sympatico.ca, it deleted the previous account, and all history.

    Then, two months later, Bell's mail servers all went down anyway. When they were rebooted, they were restored from a week old backup (the most recent they had), so everything sent in the interim was lost.

    Bell's response? "It is recommended that if you were anticipating any emails to be sent to your account after the date of the restored backup, you contact the sender and ask them to resend the message to you". This was, of course, then followed by the Bell billing department refusing to resend lost bills, and actually told customers on the phone that they should have a reliable email provider.

    Good times, good times.

    As for Dell hosting, I dealt with them in 2001. I had a customer who was hosted on them, and one day, Dell discovered that he hadn't paid his most recent three bills. In fact, about 800 customers had not paid their most recent three bills. This was due to Dell not having sent the most recent three bills, but that was beside the point.

    So, Dell then sent out email notices telling the customers that they were 90 days overdue, and if they didn't pay their amount owing, their accounts would be suspended at 5pm Friday. Of course, it took a while to send out 800 emails, especially since they only started sending them out at about, oh, 3pm on Friday.

    As customers found their sites going offline, and called the 1-800 number to address the issue, they discovered two things. First, paying the account was easy enough to do with a Visa card (my customer's account was something like $350 a month; a $1,000 bill was not hard to pay). But secondly, that the tech department had gone home for the weekend already, and that the sites could not be restored until at least 8am next Monday.

    But wait, customers said, we're paying for 7/24 support. Yes, Dell replied, in a message for the ages, but customers needed to realize that "24 hour support only applies to the telephone support, not networking support".

    That's right. If you paid for hosting with 7/24 support, that meant that if you went offline at 5:01pm, you could call them, and they'd take a message to tell the tech team two days later. Even though your site was offline for 64 hours or so, it would be looked at first thing. Support!

    It's good to see we've made so much progress in the past 20 years...

    1. Any other name

      Re: This has all happened before, this will all happen again

      Speaking about Bell Canada and e-mail: Didn't they also undertake to build the unified, government-wide e-mail system in, oh, around 2013? [yes, they did: https://o.canada.com/business/bell-canada-gets-400-million-contract-for-federal-government-email-system]. The rumour at the time was that they've never buit a business e-mail system of that size (or even 1/10-th of the required size - we are talking about 300,000 direct employees, and many more actual e-mail accounts) before; apparently their plan was to run the entire thing on a single Exchange cluster. Five years later, the system was still not ready [https://globalnews.ca/news/3709736/canadian-government-email-project-stalled/]. I sort of lost the interest in following the story since then ...

      If there is one thing all Canadians unite in, it's the mild dislike of Bell, Telus, and Shaw ;o)

      1. Barry Rueger

        Re: This has all happened before, this will all happen again

        If there is one thing all Canadians unite in, it's the mild dislike of Bell, Telus, and Shaw ;o)

        Ahem. From my experience, Rogers is still the worst of the bunch.

      2. billdehaan

        Re: This has all happened before, this will all happen again

        If there is one thing all Canadians unite in, it's the mild dislike of Bell, Telus, and Shaw

        Shaw and Cocego are actually held in less contempt than the big three. Rogers, Bell, and Telus are collectively referred to as "Robelus", or Big Telephone. They are interchangeable, essentially. Despite denying they are a cartel, it's pretty much a given that when one of the three announces a new service, price hike, or (rarely) price drop, that the other two will announce the same within 30 days.

        Fortunately, there are alternatives. And despite being horrible consumer services, all three provide very solid back end networking, so third party MNVOs and ISPs are both plentiful and competitive. Unfortunately, the average user either doesn't know about them, or is not confident with having a no-name provider for something as important as their phone service and internet connection.

        1. Anonymous Coward
          Anonymous Coward

          Re: This has all happened before, this will all happen again

          I have a cell phone with a VNO running on the Rogers network, an ISP running over a dry loop from Bell, and an email provider / ISP who is neither of the above. Someday I will likely get a second ISP running over whatever they call the cable equivalent of a dry loop.

          It works much better than dealing directly with the three evil telecom empires.

      3. keith_w

        Re: This has all happened before, this will all happen again

        If there is one thing all Canadians unite in, it's the mild dislike of Bell, Telus, and Shaw ;o)

        Mild? That is good old fashioned Canadian understatement! Personally, I abhor all of our providers.

    2. Anonymous Coward
      Anonymous Coward

      Re: This has all happened before, this will all happen again

      Did they not have a new advertising campaign at the time (and new logo) after laying off thousands of employees: Welcome to the New Bell?

  3. JLV Silver badge
    Trollface

    Sad looking cheetah?

    Their corporate branding is all about various cute animals.

    Right now, though extinct, a befuddled-looking dodo chick seems a more appropriate mascot.

    1. Anonymous Coward
      Anonymous Coward

      Re: Sad looking cheetah?

      ...or perhaps a Lemming?

    2. Nolveys Silver badge

      Re: Sad looking cheetah?

      Their corporate branding is all about various cute animals.

      That's one of the things they got out of Cadvision when they consumed it about 15 years ago.

      Cadvision was an amazing ISP that served the Calgary area. Fantastic service, prices and speed. Also, cute animal advertising.

      Then Telus did what Telus does best, it ate Cadvision and turned it into feces.

      1. Huw Leonard

        Re: Sad looking cheetah?

        Pretty sure the cute animals marketing was originally Clearnet, which Telus bought in 2000. Cadvision's marketing was the stylized men and women (often sitting behind a computer) with the blank glasses that looked vaguely 50's-ish.

  4. AdamWill

    "rich history"

    "As a technology company with a rich history in providing excellent customer service"

    Hold on, hold on, you must be talking about someone else, because that's definitely not the Telus we know and 'love'...

    I'll stick with Shaw. About as mediocre as most telcos in a lot of ways, but at least their business support lines still employ some people who live in the same timezone and know what the hell they're talking about. And leave you on hold for less than two hours.

    1. keithzg

      Re: "rich history"

      Maybe they just meant they have a history in claiming to provide excellent customer service, and that that claim is pretty rich ;)

      In my experience Teksavvy is even better than Shaw for customer support, albeit hamstrung by being a reseller, so sometimes there's nothing they can do either but wait for Shaw or Telus or whatnot to actually fix a physical problem.

      Have had mostly good experiences with Shaw's business support, save for a particularly galling instance where they replaced a cable modem with a modem+router combo box after having claimed they wouldn't, and would not swap it back for a purely modem box because they no longer offered such. Luckily their putting that box into passthrough mode has mostly stayed set, only very occasionally knocking my entire office at work off of the internet . . .

      1. AdamWill

        Re: "rich history"

        Yeah, I've heard Teksavvy are good too but I need a business plan (as I run servers and need a non-blacklisted static IP) and last I checked they didn't really handle that very easily. They now seem to have a better business site, but Shaw's doing fine so I'll stick with them till they screw something up...

    2. CrazyOldCatMan Silver badge

      Re: "rich history"

      I'll stick with Shaw. About as mediocre as most telcos in a lot of ways

      It's like the reasons for using BT: "they are not as bad as some of the others".

      Of course, in the technical paradise that is the UK we have lots of ISPs - some of whom have A Clue(TM) and also understand customer service..

      (Zen - I'm looking at you)

      1. CountCadaver

        Re: "rich history"

        Happy Zen customer here, albeit took issue that their site doesn't let you opt out of cookies yet as "we're working on that"

        Apart from that £29.99 for no data limit 40mbps FTTC with line rental included with a "promise" of no price increases as long as I keep the service, happy enough, though new folk are now paying £31.99, better and cheaper than Sky who implemented transparent proxying and then claimed it was a faulty update, when in fact its their new "Sky Buddy" so you can turn off the net and foisted on every customer whether they have kids or not.

    3. herman Silver badge

      Re: "rich history"

      Hrrmfff... The only way to get tech support with Telus is to phone them and hold for the next 45 minutes until some poor sod answers - immediately swear heavily at this bloke/blokess - which will then get you bumped up to a supervisor, who is usually quite knowledgeable.

  5. AJames

    It's particularly sad and inadvertently funny if you live here in Telus territory and you get to watch the story on the evening news ("Angry customers demanding to know when Telus will fix the problem") immediately followed by a Telus internet ad with cute animals bumbling around while they tell you what great service they offer. Was the placement accidental, or was someone at the news station getting in a dig at Telus? :)

    1. billdehaan

      Was the placement accidental

      I'm sure it was accidental. Adverts are paid for and planned long in advance. And the news guys don't know what commercials are playing in what order, so the can't really time the stories to match the ads.

      I worked for a newspaper for a bit, and while that editorial/advertising firewall was very real and very necessary, it also produced a number of embarrassments that I can remember. The most vivid was a Gazprom ad in a German magazine (possibly Der Spiegel). On the left, a full page advertisement for a Russian gas company, proudly talking about the rich shared history of German and Russian in using gas together. On the right, page one of a historical retrospective of Auschwitz, and the Molotov Ribbentop pact. Awkward...

  6. I'm Dugly

    Telus is a corporate cheapskate

    They underpay contractors by 20 - 30%. If you pay peanuts, you get monkeys.

    1. Anonymous South African Coward Silver badge

      Re: Telus is a corporate cheapskate

      They underpay contractors by 20 - 30%. If you pay peanuts, you get monkeys.

      And this is what beancounters does not understand... and they never will.

      1. Spanners Silver badge
        Flame

        Re: Telus is a corporate cheapskate

        They underpay contractors by 20 - 30%. If you pay peanuts, you get monkeys.

        And this is what beancounters does not understand... and they never will.

        And this is despite beancounters always wanting the best hardware or nice #shiny iThing for themselves because "you get what you pay for".

  7. Anonymous Coward
    Anonymous Coward

    Been there, done that.

    Gentleman's agreement between two parties saw me hosting an email server for a client, and not getting paid for doing so.

    Result - I did not deliver a professional service, only a best-effort service, and when they wanted some old emails, guess what... whoops, buggrit, no archives.

    They moved over to another provider, but they had to pay for the email hosting. Good riddance.

    Now it's somebody else's headache and PITA.

  8. Alister Silver badge

    Our entire leadership team is committed to learning from this experience to augment the strength of our globally-leading networks

    and not be such penny-pinching bastards...

  9. Doctor Syntax Silver badge

    It's somebody else's computer

    If you want to be sure of access to your own email save it locally and do whatever you feel necessary to back it up. Otherwise you're reliant on what somebody else thinks is adequate.

    1. billdehaan

      Re: It's somebody else's computer

      I have always pulled my email down by POP3 and saved it in a local archive. Prior to Mozilla releasing Thunderbird, I used a couple of other options, but since then, it's been Thunderbird all the way.

      I pay a third party (mailbox.org) 1 Euro a month for a secure email, rather than use the free Hotmail/Gmail/Outlook, because (a) I don't trust Big Email, and (b) being a paying customer, I have a right to complain. You're either the customer or the product, and I'd rather be a customer.

      But even then, Mailbox is just a server with a POP3 and web interface. Although they offer to archive the mail, I don't rely on them to; I do that myself.

      Unfortunately, that's not for everyone. Too many mundane/nontechnical people just assume email storage is some magical thing that's handled by "the internet". I've had clients with defective hard drives say it was fine to wipe the hard drive and re-image it, confirm that five separate times, say that there's nothing of value and they can redownload anything the need, and then totally freak out when you tell them that wiping their hard drive means losing all the emails they downloaded.

    2. Anonymous Coward
      Anonymous Coward

      Re: It's somebody else's computer

      If you want to be sure of access to your own email save it locally and do whatever you feel necessary to back it up

      ---------------------------------------------------------------------------------------------

      I run Thunderbird on all my computers.

      With IMAP, Thunderbird automatically updates the server copies of Drafts and Sent directories, and downloads all the incoming mail. As a result, every computer contains a backup of my mail, as of the last time it was on the Internet.

      Nothing gets deleted unless I delete it, either manually or via a rule... certain types of sort of legitimate advertising, for example, is deleted after 30 days.

      And if there is a problem with my computers or internet, there is also a copy of everything not deleted on the ISP server, accessible via web browser... which gets replicated again whenever I configure a new computer.

      The only thing I have not done is set up the web browsers to handle encrypted messages... I trust Thunderbird a lot more for that.

  10. CrustyPlugweed

    If only userfriendly still existed (or rather, were written/updated)....

  11. JimC

    "novel" solution??

    Seems to me its been done many times before.

    And of course changing email platform is a well established means for cutting down storage, since if the users have to migrate their archives manually most of them won't bother...

  12. Nolveys Silver badge
    Windows

    It's crap, don't use it.

    Telus's email hosting is an ancient pile of garbage and should not be used by anyone for any reason.

    But beyond that, ISPs should be used for internet access and nothing else, especially if the additional service is based on a domain owned by the ISP. The reason being that hosting with an ISP makes it difficult to dump them if need be.

    1. JLV Silver badge

      Re: It's crap, don't use it.

      Most definitely this. +1.

      Do you want to move somewhere and have to notify _everyone_ you know you can’t be reached at jsmith@telus.net anymore? Never use ISP-bound email.

  13. CrazyOldCatMan Silver badge

    Backups, Backup, Backups..

    Or did they get forgotten about in the rush to reduce costs?

    (My email spools get backed up to my Backblaze buckets - I can (theoretically) restore to any of the 4-per-day points that backups happen. And, since I'm using qmail, the restore won't nuke anything that currently exists unless the filenames clash (unlikely). You may end up with deleted emails reappearing but that's easily solved by using the delete key.

  14. Disgruntled Telus User

    "... starting immediately, customers still experiencing issues can access their inbox by visiting webmail.telus.net from an Internet browser. "

    This statement is not true. Try out the URL - this is what you'll get: "Internal Server Error / The server has encountered an internal error. / Please try again later. / ERROR: 500".

    I phoned up Telus yesterday and let them know that they were lying on their site where they state that all customers can access webmail. The customer service rep asked me try out different browsers, as if that mattered.

    So, this is really nothing more than another marketing ploy to save their reputation.

    1. Robert Moore
      Pint

      So, this is really nothing more than another marketing ploy to save their reputation.

      Telus have no reputation to save. I have never regretted the day I switched away from them. I still smile when I remember the day when they called and tried to get me to come back. I laughed and laughed.

  15. Bitsminer
    Angel

    "ensure our customers never find themselves in this situation again"

    That's easy, isn't it?

    Don't be a Telus customer!

  16. StargateSg7 Bronze badge

    Looks like Telus is mostly back up again!

    Since I'm in the Vancouver area myself, I use Telus on and off for various personal stuff but I always laugh when people say the ENTIRE Telus email system went down and needs to be restored! In reality it's ONLY about 1500 Terabytes (1.5 Petabytes) of Total Email Data for all of the as-of-August-2019 around 14 million Telus internet subscribers (about 100 megabytes of email per customer). That is NOT a lot of data at about 188 of the Eight Terabyte Western Digital Enterprise class drives or around $120,000 US at bulk prices. That is NOT a very large amount of money for a company the size of Telus which is currently at the 33rd position on the world's largest telecom providers at $9.7 Billion US of revenue versus British Telecom (BT) at $27.3 Billion US. Of course AT&T ($163.8 Billion) and Verizon ($131 Billion US are the world's largest telecom operators which are gargantuan-sized when compared to the the tiny pip-squeaks of BT and Telus!

    I store and use that amount of data in about every three hours which technically means i'm bigger than ALL of Telus just in my tiny office and the living room-sized storage centre on the other side of my office! When you get into the ZETTABYTES PER DAY the parent company stores and uses on the Monster-Machine supercomputer THEN call us to cry on our shoulders about your email restoration difficulties!

    In terms of actual bytes of data stored and transferred per month, I would say our parent company is about #4 on the list after AT&T, Verizon and NTT so basically were are one of the 1000 Pound Gorillas of Data Storage and Transfer! The parent company actually had to buy an entire Memory Chip and Disk storage systems maker and take it's intellectual property and manufacturing expertise fully in-house to keep up with internal demand!

    .

    So when Telus cries Holy Mercy, I say it's just crocodile tears when they could have EASILY chartered a private jet to Singapore to buy and bring back 300 to 500 Eight Terabyte enterprise class Western Digital disk drives from their manufacturing facilities! And Micron down south in Idaho could have supplied 1000 4-terabyte SSD's in about 48 hours for a measly $750,000 US.

    .

    Bah! Humbug Telus !!! Quit yer yappin' and go buy some drives! Cry Me a River!

    .

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019