back to article Scanning an Exchange server for a virus that spreads via email? What could go wrong?

Just like clockwork, another weekend is over and Monday is here again. To lighten the load, El Reg is offering you the latest instalment of Who, Me?, our weekly sysadmin confessional column. This time we meet "Romeo", who was working at a large music company in London at the time in question. It was his first job for a big …

Page:

  1. Giovani Tapini

    There is a little bit of me

    That thinks the actions could be close to doing the right thing?

    Although a company culturally can create a great dependence on email as a business tool for variously archiving, signoff approvals, support messages, and other unwise use-cases.

    At least the server was clean, and probably caught other things too.

    My worry would be how the messages got to this point without being cleaned already...

    1. LDS Silver badge

      Re: There is a little bit of me

      No, the server was not clean. It didn't clean the mail database, it just deleted it, so a previous copy had to be restored - it still contained all the bad emails the previous one contained but the last day.

      And why a company should not rely on email? Should they install pneumatic mail, and have people with carts going around desks bring paper documents to read and sign? Companies always relied on internal document shifting...

      1. Giovani Tapini

        Re: There is a little bit of me

        @LDS Fair call, but the article does not go into much detail on what was done with the restore. I assume some tool more suitable for cleaning mailboxes was deployed. Server was therefore clean albeit at the expense of deleting everything. The process did have an excess of collateral damage, but did remove the infection.

        Now about those carts, they used to be filled with snacks and coffee...

        1. LDS Silver badge

          Re: There is a little bit of me

          For what I read, it had big troubles restoring the mail database because in Exchange just replacing an older file is not enough - as database systems are usually picky when data files, logs and other things don't match. So, sure, he deleted the ILOVEYOU mails of that day - but whatever else was lurking there from previous days was still there.

          Cleaning such stuff exactly needs tools which are able to read the database correctly and clean infected messages one by one - but you usually need to have the mail database open and accessible to run them, because accessing the on-disk structure of such files - often undocumented - it's a very risky task.

          While in certain circumstance you may not have other options that wiping everything, running an AV against database files is usually a very bad idea - especially if the default actions is "delete".

          1. heyrick Silver badge

            Re: There is a little bit of me

            especially if the default actions is "delete"

            He must have been really new to virus scanning. People with experience know that there are such things as false positives which are more often than not important documents and critical Windows DLLs... as such the only sensible action is to quarantine suspect files to ensure you aren't about to nuke something important, and if it needs to go (of it's a real virus and not a "this code looks odd" heuristic), you know what to replace from the installation media before the machine gets itself into a bluescreen-at-boot state.

            Never ever give an antivirus program the ability to automatically delete stuff...

      2. phuzz Silver badge
        Thumb Up

        Re: There is a little bit of me

        "didn't clean the mail database, it just deleted it, so a previous copy had to be restored"

        That is cleaning it. The same way I clean my car, by sandblasting all the paint off, and then re-painting ;)

        1. Waseem Alkurdi

          Re: There is a little bit of me

          @phuzz

          That is cleaning it. The same way I clean my car, by sandblasting all the paint off, and then re-painting ;)

          In this case, the backup contained the virus.

          This is like re-painting with the same old paint melted into a liquid form or something.

        2. LDS Silver badge

          "by sandblasting all the paint off, and then re-painting ;)"

          But if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk...

          1. amanfromMars 1 Silver badge

            Re: "by sandblasting all the paint off, and then re-painting ;)"

            But if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk... ..... LDS

            The I Love You Virus a Corpse in a Trunk whenever Phantom Bodied? I Don't Think So, Mes Amis/Mon Brave.

            Are you Bitten and Smitten and Rooting for More LOVE Bugs to Display the Bounty of ITs Wares with Immaculate Temptations to Sate and Supply ie Fully Realise in COSMIC Great Order for SMARTR TerraPhorming Nations?

            Try Contemplation of IT as Advanced IntelAIgent Driver/Virtual Mentor and Practical Monitor.

            One of those Funky Clunky NEUKlearer HyperRadioProACTive IT AIdPrograms Perfectly Suited for Princes and Princesses with Visions in Peril? ...... Saudi Vision 2030

            And there be Kings and Queens, Princes and Princesses, Nymphs and Satyrs Everywhere. And that Convenience makes More LOVE Bugs AIdPrograms Astronomically Wealthy and Certainly Worthy.

            1. Anonymous Coward
              Anonymous Coward

              Re: "by sandblasting all the paint off, and then re-painting ;)"

              I have no idea what is going on anymore ...

          2. Adam 1 Silver badge

            Re: "by sandblasting all the paint off, and then re-painting ;)"

            > ... if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk...

            Asking for a friend?

      3. chivo243 Silver badge
        Pint

        Re: There is a little bit of me

        Buttle or Tuttle? Tubes going everywhichway! Love it!

    2. David Knapman

      Re: There is a little bit of me

      It's not like this one organization would even have been in the minority here, ILOVEYOU was a major wake up call for many orgs to put more work into their email scanning.

      Many orgs, if they had incoming scanners at all were just using signature based checks so of no use against a rapidly spreading worm based on social engineering.

      1. JimboSmith Silver badge

        Re: There is a little bit of me

        At the time of that particular virus spreading I was working for a firm doing some tech related planning. We were using outlook and hit quite hard when one just person was sent it. Whilst IT were cleaning everything the rest of the place descended on the local public houses. We had an afternoon spent doing not much except trying different beers etc. Then returned to the office to collect our things at 5:30 whereupon some people went back to the pub. IT support told us that they had cleaned all our mailboxes/computers and beefed up the mail filter.

        Everything was fine after that except a few months later when we all started to get these emails again. IT were a bit annoyed that it had made it through the mail filter The culprit was the Intern mailbox which was only used when we had an intern. There wasn't an intern at the time so the mailbox never got cleaned. One of the staff left immediately for the pub when the first email appeared. He was called back before he could order anything to drink.

  2. Evil Auditor Silver badge

    Sweet memories...

    I remember well when ILOVEYOU broke out on our university campus. It was that morning when I found an e-mail in my inbox with this love confession from a girl in administration. As sweet as she was, I had some mixed feelings about this - can't remember her name since we cruelly only called her The Nose. For obvious reason. Needless to say, our love relationship wasn't meant to last. Not longer than a few seconds anyway.

    I was safe with my Linux environment. And the next thing was to start up the Windows sandbox - no virtual machine at that time - and investigate what this charming virus actually did.

    1. sandman

      Re: Sweet memories...

      Yep, that's rekindled a few grey cells. We thought we were on top of it, repeated warnings sent out, 24pt, bold, underlined, bright red, dire threats and all. Then, one person in PR decided to completely ignore all that... When asked, they said, "Oh, I never read messages from IT, you're always just sending out warnings."

      1. tiggity Silver badge

        Re: Sweet memories...

        Exactly what you (sadly) expect from some users

      2. Nick Kew Silver badge

        Re: Sweet memories...

        "Oh, I never read messages from IT, you're always just sending out warnings."

        The boy who cried Wolf springs to mind.

        Can't comment on your individual situation, but warnings are more effective if you pick your cases with some care to avoid overloading users with esoterica that'll only baffle them.

        1. Aladdin Sane Silver badge

          Re: Sweet memories...

          This would be just post Y2K, which people put a lot of effort into fixing only for their efforts to be dismissed as scare-mongering.

          1. katrinab Silver badge

            Re: Sweet memories...

            And while there was stuff that needed to be fixed, a lot of it was scaremongering, especially related to embeded systems. For example, the idea that a washing machine might, on 01/01/00, think it was 1900, and, that as it hadn't been invented yet, it ought to shake itself to bits and then spontaneously combust.

            1. Adam 1 Silver badge

              Re: Sweet memories...

              > For example, the idea that a washing machine might, on 01/01/00, think it was 1900, and, that as it hadn't been invented yet, it ought to shake itself to bits and then spontaneously combust.

              ... And I would have gotten away with it if not for you pesky kids.

              1. John Brown (no body) Silver badge

                Re: Sweet memories...

                "... And I would have gotten away with it if not for you pesky kids."

                Should've bought a Scooby! :-)

          2. steviebuk Silver badge

            Re: Sweet memories...

            That's what I always point out to people. I wasn't involved in any fixes, I'd just finished college. But the amount of people that say "That Y2K thing was bollocks wasn't it. Nothing happened". Especially so called comedians. Yeah nothing happened because people worked fucking hard to fix the issues before they could happen. Tits.

            1. Andrew Moore Silver badge

              Re: Sweet memories...

              I had a journalist approach me in 1998 looking for scary Y2K story- I told her that nothing was going to happen because we'd been on top of it for a number of years now and everything should be in place by then. Needless to say, she completely ignored me and found a nutjob that gave the doomsday scenario that she was looking for.

              1. John Brown (no body) Silver badge

                Re: Sweet memories...

                "Needless to say, she completely ignored me and found a nutjob that gave the doomsday scenario that she was looking for."

                It would have been funny if all those media types touting the scare stories had their own orgs IT fall over, but sadly their own IT people were also on top of things. Shame the journos didn't interview their own IT bods and got the real story.

            2. Mike 16 Silver badge

              Re: Sweet memories...

              @steviebuck

              While I agree that a lot of conscientious people worked a lot of hours in the run-up to Y2K, IIRC a patch for Windows believing 2000 would be a leap year came out in something like November 1999. This despite earlier complaints from fin-tech people that computations of future value or the like were odd. The thing is,, sometimes you don't just need to know what day today is, but what day 60 or 180 days from now will be.

              1. katrinab Silver badge

                Re: Sweet memories...

                2000 was a leap year. 1900 and 2100 are not.

                Most computers at the time assumed any year divisible by 4 is a leap year, when in fact years divisible by 100, but not divisible by 400 are not leap years.

                Excel still thinks 29th February 1900 is a valid date, and most other spreadshhets copy this bug for compatibility.

          3. Anonymous Coward
            Anonymous Coward

            Re: Sweet memories...

            Y2K was scare mongering.

            Had to be to get all the CEO's and bean counters on board to allow all the time and money to be spent to check all the software for problems. In the end the CEO's and bean counters all knew they would get blamed by the stock holders, customers and little kids on the street if their company had a Y2K problem.

            1. Unicornpiss Silver badge
              Meh

              Y2K was scare mongering..

              I remember people freaking out because they thought their cars wouldn't start on 1/1/00. As though cars (at least of that era) cared what date it was, with the exception of somehow knowing when they're 1 week out of warranty..

              1. katrinab Silver badge
                Trollface

                Re: Y2K was scare mongering..

                I suppose the software that says, "I'm one week out of warranty, I'm going to shake myself to bits and spontaneously combust" might not work, and it might think it has to keep going for another 100+ years.

      3. Mark 85 Silver badge

        Re: Sweet memories...

        Then, one person in PR decided to completely ignore all that... When asked, they said, "Oh, I never read messages from IT, you're always just sending out warnings."

        Those types should be strung up by the front door as example and with an email or group meeting explanation. The second one who does it should be drawn and quartered.

  3. Nick Kew Silver badge

    Still baffled

    ... at how noone sued MS for damages at the time.

    The means by which this email evaded detection in a simple and sensible email scanner was MS's deliberate breaking of MIME standards dating back to 1992. And the RFC even contains an informational section under the heading of security implications explaining exactly why what MS subsequently did would leave their users wide open to attack.

    1. heyrick Silver badge

      Re: Still baffled

      "what MS subsequently did would leave their users wide open to attack"

      You mean like how the created user profile for users on a home installation of XP had admin rights by default, and how the restricted user profile was so restricted it was near useless for many (you couldn't even change the time FFS). There was, I believe, a tool to tweak what rights users had, in the enterprise version...

      It was pretty much a wide open door back then, just marginally less open than Win32 machines.

    2. jake Silver badge

      Re: Still baffled

      "at how noone sued MS for damages at the time."

      Read the fine print. MS' code isn't even guaranteed to work as advertised when used as intended. It's use at your own risk, at least according to MS's own EULA. You HAVE read the EULA, and fully understand it, right? And your corporate lawyers have vetted it as OK for use by your business, right?

      1. heyrick Silver badge

        Re: Still baffled

        Read the licence blurb?

        Eight years ago a company demonstrated how many people bother reading all that rubbish: https://www.geek.com/games/gamestation-eula-collects-7500-souls-from-unsuspecting-customers-1194091/

  4. adam payne Silver badge

    Scanning an Exchange server for a virus that spreads via email? What could go wrong?

    It deletes the EDB file and then lots of people shout at you.

    I would say how did the emails get that far anyway but this was back in 2000 anyway so I suppose that answers that.

    I've never deleted an EDB file but I have had an old boss add an extra drive to the Exchange server and then proceed to import the RAID config. Oops.

    1. jake Silver badge

      "I would say how did the emails get that far anyway but this was back in 2000 anyway so I suppose that answers that."

      By 2000, real MTAs had been dropping malware long before it got anywhere near userspace for over a decade. Milters (introduced in 2000) and the like made it easier to admin. Toys like Exchange were never really considered an option by professionals.

  5. Anonymous Coward
    Anonymous Coward

    I deleted an EDB file once by accident - and no backups then.

    Oops.

    Learnt a hard lesson then, never, ever again. It was not a fun experience, and one I don't wish on any Exchange admin.

    Anon because.

    Since that incident I've started to use ntbackup to back up the mail store - and funnily enough, that issue was never repeated.

  6. Anonymous South African Coward Silver badge

    Remote Desktop in the ILOVEYOU virus era?

    1. GlenP Silver badge

      PC Anywhere and the like have been around since the mid 1980s. I was certainly using it (via modems) in around 1988.

    2. Waseem Alkurdi

      The first version of RDP shipped with Windows NT Terminal Server 4.0.

      1. Freddellmeister

        Well there was NCD wincenter and a few other options before RDP..

        1. jake Silver badge

          Wincenter was OK, at least for the thin client set. Surprisingly, DESQview/X was a rather good option for remote GUI support of Windows boxen using *nix as the admin box. Spendy, though.

    3. Evil Auditor Silver badge

      @ASAC

      Indeed. My earliest recollection of a remote desktop solution dates back to the mid 90s. And it wasn't exactly avant-garde back then either. It might have been with NT 4.0.

    4. DJV Silver badge

      Yes

      Very common! A short while after the ILOVEYOU outbreak I was working for a "famous insurance company" that was (mainly) based in and named after the city I lived in (Norwich). We used remote access from our base in the city centre to the servers in the "lights out" data centre server farms four miles away out on the outskirts. It was fantastic!

      Well...

      ...only for a weird value of "fantastic" that meant that...

      ...the lights in the server farm were never actualy out because remote access was bloody slow and, for "security", sessions were set to always time out after 15 minutes which, due to the slowness of the network in general and remote access in particular, meant that you barely got more than 20 mouse clicks and 10 fields filled in remotely before the whole thing shut down on you (if it had managed to stay up for the full 15 minutes, itself a rare feat). So, it was often quicker to catch the regular company-provided shuttle bus to the data centre and go and access the server non-remotely (hence the lights never being out as the data centre was full of pissed off people all doing the same non-remote thing).

    5. jake Silver badge

      telnet was standardized in 1973. NCSA's version ran on DOS in 1986. Wall Data's Rumba was in wide use on corporate Windows desktops by 1990.

      Maybe not RemoteDesktop[tm], but certainly remote desktop capability.

  7. defiler Silver badge

    Restoring EDBs...

    One of our clients accidentally started a restore in Exchange. I think it was a block-level restore of the database rather than of a mailbox or folder - it was a while ago and I (luckily) wasn't there. When she realised her mistake she pulled the power on the email server...

    My colleague had to regedit the hell out of it to force the database out of restore mode, and then restore a complete copy of the database from before the errant command. I don't think that database was quite right ever again.

    Still, after I'd left that job, my ex-line-manager managed to torpedo the server nicely in a different way, but that's a story for another Monday...

    1. Tom 7 Silver badge

      Re: Restoring EDBs...

      We had trouble with Exchange Server 4.5. I remember running some DB repair program that took 4 or 5 hours to scan the DB half a dozen times to get it into a state that Exchange Server was happy to load.

      I was pissing about with some VB for some web app and discovered VB would allow you to dump a whole exchange DB and read everyone's emails and I think I could have used that to rebuild a corrupted DB better than the MS repair program but never have the courage to try it in action. Some nice reading while repairing the DB mind.

  8. Anonymous Coward
    Anonymous Coward

    +1 for nom de guerre "Romeo"

    (n/t)

  9. Anonymous Coward
    Anonymous Coward

    Thing of the past, thank god!

    It's Stories like this that make me relieved to get out of the email game having just shifted 10,000 users to O365 in the last couple of months....

    Having lived through a corrupt EDB file recovery on our Exchange environment last year, i know how terrifying email issues can be. No matter how often you spout the line "Its a communication tool, not a file server" at people it never stops them from retaining everything ever sent since God was a boy.....

    Also, micromanaging mailbox sizes on-prem is becoming harder and harder, users just do not get why you need to limit their mail to 2Gb and exceptions pop up all over the place (largest user mailbox we had was 80Gb ffs...)

    It doesn't help these days that File Shares and collab SharePoint seems to be a dirty word at C-Level, where they just want to send 100Mb PowerPoint Decks and Business plans to each other...and who are IT to tell them how to do their job.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019