back to article Scottish brewery recovers from ransomware attack

Staff at Arran Brewery were locked out of its computer systems this week following a ransomware attack. The attack against the Isle of Arran-based Scottish beer maker appears to have been a targeted strike. Prior to the infection, adverts for an already filled finance post at the brewery were placed on recruitment sites …

  1. GnuTzu Bronze badge
    Joke

    Pay Us in Beer

    Somebody had to say it.

    1. gotes
      Pint

      Re: Pay Us in Beer

      No beer icon? Flabbergasted.

    2. EddieD

      Re: Pay Us in Beer

      I'd go for that - Arran Beer is really rather tasty.

  2. TRT Silver badge

    Barry Shteiman...

    say what? Before my dyslexia kicks in...

  3. Dwarf Silver badge

    Thanks for not paying the scammers.. The scheme only works if they profit from it.

    1. Spazturtle Silver badge

      Paying the ransom should be considered funding organised crime and people who pay should be prosecuted.

      1. Anonymous Coward
        Anonymous Coward

        >Paying the ransom should be considered funding organised crime and people who pay should be prosecuted.

        How very black and white Daily Mail of you, when it's your loved ones arriving a bit a time by post you may think differently.

        1. Spazturtle Silver badge

          Ransoms only work because people pay them, if people stopped paying them then people would stop taking things ransom. Also we were talking about ransomware, not people kidnapping other.

          In regards to kidnapping, it is already illegal to pay a ransom under anti-terrorism laws.

          https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/540539/CTS_Bill_-_Factsheet_9_-_Kidnap_and_Ransom.pdf

        2. DavCrav Silver badge

          "How very black and white Daily Mail of you, when it's your loved ones arriving a bit a time by post you may think differently."

          Sorry, this is a ransomware attack. You mean your database is your loved one, and they send it back row by row?

          1. David 132 Silver badge
            Happy

            Sorry, this is a ransomware attack. You mean your database is your loved one, and they send it back row by row?

            No, read what he said... “a bit at a time”.

            So one day you get a 1 in the post, then a 0, then maybe another 0... maybe they’ll take a byte out of the victim, or if the extortionists are really cruel they’ll communicate by the time-honored method of cutting out words.

  4. Anonymous Coward
    Anonymous Coward

    Customer caught

    I had a customer caught like this a couple of years ago.

    Removed the hard drive, re-imaged the machine with a new drive, the user was back up and running in 2 hours.

    Luckily it just encrypted the boot partition and rebooted, it didn't try and do the file shares. An encrypted share would have been a bit more work, but a maximum of 3 hours work lost.

    On the other hand, I know one company, where the directors ran their own NAS, because they didn't trust ops and ops weren't allowed to back up the NAS either, that was the job of the directors... Only they never got around to it, because, you know, RAID... They got hit by ransomware and, while the director concerned noticed it quite quickly, they still lost a couple of years worth of financial information and had to cough up.

    1. Paul Crawford Silver badge

      Re: Customer caught

      RAID != Backup

      But a NAS that supports automated daily snapshots would have had a sporting chance of recovery with but a day's lost data (e.g. the feature on FreeNAS that comes free with ZFS' inherent copy-on-write operation).

      1. Spazturtle Silver badge

        Re: Customer caught

        "But a NAS that supports automated daily snapshots would have had a sporting chance of recovery "

        An automated backup likely means the backup drive is attached to the system, in which case it could also be encrypted by the ransomware. I remember a guy telling me about how the company he used to work for used USB hard drives for backups and they were only connected to the servers during the backup, one day in the middle of a backup lighting struck the building and killed not only the servers but also the USB hard drives.

        If a backup drive is connected to the system then it no longer counts as a backup for the duration of the time it is connected, this is why you need to rotate backup drives so that you always have an isolated backup.

        1. Brian Miller

          Re: Customer caught

          Every time I read about, "we lot everything due to ransom-ware," I think about all of the easy, good practices that have been developed over the years. And I think about how they are not followed, because it requires even a minimum of effort.

          Good backups means that the recovery process takes place in maybe three to four hours. Bad backups means that data spanning years is lost. Suck it up, and put good practices in place!

          1. Mark 85 Silver badge

            Re: Customer caught

            Suck it up, and put good practices in place!

            "But, but, but that costs money!!! And reduces our profit and bonuses.... " quoth the board.

        2. Jamie Jones Silver badge
          Devil

          Re: Customer caught

          An automated backup likely means the backup drive is attached to the system, in which case it could also be encrypted by the ransomware.

          With FreeNAS and zfs snapshots? Not a chance!

      2. vtcodger Silver badge

        Re: Customer caught

        Also, an offsite backup wouldn't be a bad idea. Buildings do burn down or washed away. Offsite backup is likely to be a PITA to do, so it may only get done weekly or monthly. But losing only a few weeks worth of data will probably look like a blessing when confronted with the loss of all the configuration information and data that the company owned other than what can be recovered from surviving scraps of paper and a random selection of files and eMails from personal machines.

        1. Paul Crawford Silver badge

          Re: offsite backup

          There are many ways to destroy data integrity, not just the obvious ransom-ware or HDD failure, but also examples of electrical surge, fire, flood or some oik nicking the thing.

          Having an off-site copy is a VERY GOOD IDEA and if you want to DIY then you could sync two NAS locally, move one off-site and then have an rsync job (ideally taking a copy of the most recent snapshot so it is all consistent in time).

          Of course you also need to check it is working, not just initially but also months down the line, and to try your recovery process as well. You REALLY don't want to find out its not quite right after a major event!

          1. vtcodger Silver badge

            Re: offsite backup

            All really good advice. But be aware that, as with many other things, what's simple in concept may not be so simple in practice. For example, it'd take a good part of a week to stash a copy of my PC hard drive to the "cloud" over my suburban US DSL line. And I don't have any video data. Folks (including businesses) in neighboring towns have even slower connections BTW. Moreover, tying up the home data pipe with a massive, days long, upload is likely to annoy the other folks that reside here. Therefore my home system offsite backup is on a usb stick in the spare tire well of my car. THAT only takes about five hours to build

            One very likely wants to encrypt offsite data. Easy enough, if one does something like tar-compress-encrypt on high level directories. Why tar? Because I really don't want to deal with data recovery from a file system with tens of thousands of files with obsfucated names. Rsync isn't going to work very well. Solvable? Yes, I think. I haven't actually tried to integrate rsync into the workflow. Easy? Not so much I'm pretty sure.

        2. Anonymous Coward
          Anonymous Coward

          Re: Customer caught

          I am sure my manglement would say "It's not an issue in the cloud..."

      3. katrinab Silver badge

        Re: Customer caught

        Volume Shadow Copies in Windows also works at least some of the time.

        Obviously you should do backups as well, but the VSC or ZFS snapshot might be more recent than last night's backup.

        1. Danny 14 Silver badge

          Re: Customer caught

          another call for VMs. the host is isolated from the normal network and has an isolated backup. Then when the guests are infected you first power them off forcefully. recover from backup. carry of as normal.

          if the guests can get to the hosts then you are doing something wrong. the whole point of having a management network is to keep your infrastructure away from production network.

          1. Danny 14 Silver badge

            Re: Customer caught

            oh and for backups get a pair of cheap synology. they will even rsync to each other. keep then in totally different buildings (different power etc) and vlan'd apart from the production network (if you cant dedicate cabling). for less than 1500 you can have terabytes of independent backups that live in different buildings.

            hyperv server is free. it can even cluster and can use server storage nowadays - no need for san. a pair of r410 can be bought (refurbished ) for under 3k with enough ram and storage to have a 2 node clustered VM platform for your linux servers. no licensing needed.

            shame that you need datacenter for stretch cluster as stretch cluster works brilliantly for multisite cluster resiliency (thats what we do)

            1. big_D Silver badge

              Re: Customer caught

              Rsync replication won't help. It will sync the encrypted files and delete the originals!

              We had a CEO who decided mirrored, redundant servers were the solution, hot stand-by, so no need for backups. Until he managed to corrupt the database and realised the hot stand-by also now had the corrupted database as well.

              Offsite, offline backup on a separate medium is the only real answer. And offsite != cloud backup. Tapes on another site in a fire safe or in a bank valut, for example, are proper offsite backups. Cloud backup, unless you use glazier style is online and disk based, which breaks 2 two rule in 3-2-1 backup (no file exists, until there are at least 3 copies, on 2 different medium and 1 is offsite). Although a backup on an HDD array half way around the world is probably OK these days.

              I use replication to an external drive, replication to a NAS and cloud backup at home. At work we have Veeam onto hot near-line storage and onto external media, into the fire safe.

              1. donk1

                Re: Customer caught

                Exactly, you need and off-site AIRGAPPED backup....tapes anyone?

                These days people seem to want everything online and lost interest in AIRGAPPING!

                Also Dave's rule 1 : Test your restores not your backups!

  5. Chewi
    Pint

    Well worth a visit

    I'm very surprised to see the Arran Brewery featured here as I've visited the place and it's tiny! A reminder that these bastards can take down businesses large and small. Anyhow, it's well worth a visit, the island is beautiful and the beer is excellent. I recommend Red Squirrel.

    1. defiler Silver badge

      Re: Well worth a visit

      Not for me. When I went to visit we all paid, got inside and the place was shut down. They'd gone out of business and the new owners weren't up and running yet. I'm guessing that's around 10 years ago. They showed us some nice bottles as we came out in the style of "look at what you could have won"...

      Their beer is very pleasant, and it is indeed a lovely island, but the brewery tour was certainly disappointing that day.

  6. Mystic Megabyte Silver badge

    Webmail at UK2.net

    I've recently been getting some phishing "invoices" with attached "Office365 docs". Strangely when I tried to forward them to NFIB Phishing I got an error message, "This contains malware" or something similar. Why is it not scanned on delivery but is scanned on sending?

    1. Danny 14 Silver badge

      Re: Webmail at UK2.net

      we run pfsense and snort (paid version). i get an alert when some idiot opens the .doc/.xls/.pdf that try to phone home to compromised servers, sometimes the AV gets them too. luckily we havent been hit with anything snort hasnt known about.

      still, at least i can educate people when the alert goes off. education is better than AV or IPS reliance.

    2. Spazturtle Silver badge

      Re: Webmail at UK2.net

      IDS is run on the local side normally, you can run it on the WAN side if you want but then you will be notified about things that your firewall is blocking anyway.

  7. Michael H.F. Wilkinson Silver badge
    Pint

    Good to hear they are up and running again

    especially as it is beer o'clock here!

  8. fidodogbreath Silver badge

    the decision meant accepting the loss of three months worth of sales data from one infected server

    "Recovers?" Sounds more "gives up and moves on." But good on them for not paying the scammers.

    They can probably recreate most of the sales data from credit card transaction logs and their customers' records, anyway.

    1. Sgt_Oddball Silver badge

      If only....

      There was some other method of storing data. Some kind of write once read many format in an easily read container. Though I will admit matters of scale can be an issue, and the file system has to be regularly updated. Data dispolsal can get interesting too (I personally burn mine).

      It's times like this that as much as I like moving forward with the march of progress, I wonder if we lose too much in the process (no one's ever managed to encrypt hard copy once it's been created. The only other thing I can think of is if the language dies but that's slightly different and can't be held to ransom over that one)

  9. ThePieMan

    Backup? How long?

    There’s no excuse these days to lose more than a few hours of data if it’s important to your business.

    Fire the boss I say!

  10. Phil Endecott Silver badge

    “Barry Shteiman ... said ... If the downtime caused by data being unavailable, or by the backup restoration process, is more expensive than paying the ransom, then organisations should pay.”

    Mr Shteiman is overlooking the wider effect on society of paying.

    Perhaps, if he really believes in this selfish “only our bottom line matters” attitude, we need to tell him that we’ll boycot businesses that pay ransoms.

    Does anyone have a list of businesses that have admitted to paying ransoms?

    1. JimC Silver badge

      this selfish “only our bottom line matters”

      "And that is called paying the Dane-geld;

      But we've proved it again and again,

      That if once you have paid him the Dane-geld

      You never get rid of the Dane.

  11. Twanky

    'organisations should pay'?

    If I were running a ransomware operation I'd want to take a look at whatever was so important that a victim agreed to pay for it. If it's that important to the victim then it's probably worth a higher ransom.

    Why would you trust the bad guys to provide the decryption key once you've paid? They're bad guys.

    As for 'losing 3 months worth of sales data'... it's a valuable lesson which the brewery could probably have learned at a lower cost.

    I very much approve of small breweries in general though - I'll make a point of sampling their product soon.

    1. Adrian 4 Silver badge

      Re: 'organisations should pay'?

      "Why would you trust the bad guys to provide the decryption key once you've paid? They're bad guys."

      Because they want repeat business. If they have a reputation for failing to decrypt when paid, why would anyone pay ?

  12. Anonymous Coward
    Anonymous Coward

    Been hit a couple of times by ransomware, (Trend AV on client PCs not being as useful as you’d expect, and users not being as vigilant as... ah what am I saying...)

    Data held on NetApps was restored via Previous Versions or using ndmpcopy from the netapp console to restore direct from .snapshot folder once the clients were taken offline.

    Relatively painless.

  13. Henry Minute

    Freedom

    They may take our sales data but they'll neverrr take our beerrr

  14. Mike 137

    Not an IT problem

    Despite the general trend of comments here emphasising the technical aspects, this is not really an IT problem at all.

    One has to ask why unsolicited emails responding to a non-existent recruitment drive got opened at all, let alone their attachments.

  15. Jakester

    Don't have server write to backup shares, have backup device pull data..

    The approach I use for backups is to have a backup system (Linux running 'backintime') pull data from the servers. The backup systems do not have shares on them, they only attach to shares on the server. I have nightly and hourly backups - some kept locally - and at least 2 backup media kept off-site. Standard portable hard drives (currently using 2 TB) keep nightly backups for 3 weeks, then a couple months of weekly, and finally monthly backups kept. Periodically phasing in a new drive allows years of monthly data readily available. Using this approach, the chance of any backup getting infected is reduced by orders of magnitude.

  16. Giovani Tapini

    disagree with the economics statement

    Value of data/recovery over extorted amount is not the entire equation.

    There is also a judgement call relating to the likelihood of getting your data back at all even if you do pay. This is certainly not guaranteed at all even if you do pay.

  17. Toltec

    BOFH darkside solution

    "While many security experts warn about paying ransoms or entering into negotiations, the answer in reality comes down to simple economics."

    What if the ransom is less than it would cost to track them down and remove bits from them until you had the encryption key?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019