back to article KCL staff offered emotional support, clergy chat to help get over data loss

Staff at King's College London are now being offered counseling and prayers to help them get over the data loss suffered during October's catastrophic IT failure. A month ago, departments across the university suffered “irretrievable data loss” when a 3PAR's one-fault-tolerant RAID Array kicked the bucket. Almost every system …

  1. Mutton Jeff

    Pray? Or prey

    nothing fails like prayer, as they say

    1. Ol' Grumpy
      Coat

      Re: Pray? Or prey

      "nothing fails like prayer"

      .... except KCL's backups by all accounts! :)

  2. Anonymous Coward
    Anonymous Coward

    I wonder if the Chaplain says:

    "Do not worry my son - it is undoubtedly all part of God's plan"

    - or "That is God's punishment for your sins".

    1. Anonymous Coward
      Anonymous Coward

      Misread it - read it as Chaplain version 1.1

      1. TeeCee Gold badge
        Meh

        Shame really. If they had Chaplain 2.0, they could be part of the Internet of Chaplains and air their problems via Social Chaplaincy.

    2. Anonymous Coward
      Trollface

      I wonder if the Chaplain says: "Do not worry my son - it is undoubtedly all part of God's plan" - or "That is God's punishment for your sins".

      The former for social sciences data, the latter for real science data. Sadly the arts faculties are unlikely to have data to lose.

      Now, ignoring the mechanics of the IT department's ineptitude, people will be searching for a reason, and I think the Chaplaincy WILL be able to offer a reason: God is angry with KCL.

    3. TRT Silver badge

      God's plan...

      offered at Bronze, Silver or Gold levels, giving 10Tb, 100Tb and Unlimited storage respectively in His cloud facility?

  3. GeezaGaz

    counselling for data loss????

    you couldn't make this shit up...

    ..or maybe you did?

    1. Anonymous Coward
      Anonymous Coward

      Re: counselling for data loss????

      Many of the world's politicians are once again touting their belief in $deity$ as justifying their violations of human rights.

      1. Anonymous Coward
        Anonymous Coward

        Re: once again touting their belief in $deity$ as justifying their violations of human rights.???

        Because there's no way they'd find another excuse if that one stopped working.

        Religion isn't to blame for people being dicks, they manage that all by themselves.

        1. Teiwaz Silver badge

          Re: once again touting their belief in $deity$ as justifying their violations of human rights.???

          Religion isn't to blame for people being dicks, they manage that all by themselves.

          - It does seem to be a primary source of permission for being dicks, just ahead of football, politics and sometimes race/community*.

          * I grew up in Northern Ireland, were all of the above were often in alignment.

    2. Paul Crawford Silver badge

      Re: counselling for data loss????

      Well, lets look at the hard facts...

      OK, praying is no less efficacious then KCL's previous data protection plans.

    3. allthecoolshortnamesweretaken

      Re: counselling for data loss????

      Isn't there an app for this?

    4. Captain DaFt

      Re: counselling for data loss????

      Would make for a surreal scene in a monster movie:

      Angry crowd with torches and pitch forks storm Frankenstein's castle because his monster has wreaked havoc.

      Baron Frankenstein then appears on the parapet and announces:

      "Good News! I've arranged anger management counseling classes for all!"

  4. Destroy All Monsters Silver badge

    Jesus saves!

    Unfortunately he hides the tapes.

    1. allthecoolshortnamesweretaken

      Re: Jesus saves!

      Since 20 years or so I have stickers on my car and on my motorbike saying "Jesus not only safes - he also makes frequent backups".

      1. Korev Silver badge
  5. David Tallboys

    KCL has quite a large and respected War Studies department. I guess the importance of cyber warfare has just been demonstrated by this KCL own goal.

  6. Pyrofer

    Jesus Saves!

    But GOD does the backups.

    1. pakman
      Angel

      Re: Jesus Saves!

      Maybe the chaplain will be handing these out: http://shipoffools.com/gadgets/personal/176_1.html

      Oh wait, that would be against KCL's rules about making personal backups, wouldn't it.....?

      1. Robert Carnegie Silver badge

        Re: Jesus Saves!

        I wondered if I was going to see an update to USB of Angus Deayton's (was it he?) crucifix necklace when playing an Anglican priest on the "terrible TV station" show "KYTV" quite a few years ago. The long leg of the crucifix unclipped - it was a handy pen.

    2. Anonymous Coward
      Anonymous Coward

      Re: Jesus Saves!

      God may do the backups, but when you try to restore them, they may turn out to be the dead part of Schrödinger's cat.

  7. Phil O'Sophical Silver badge
    Coat

    Yesterday

    I can't claim any credit for this, I saved it from Usenet a long time ago:

    Yesterday,

    All those backups seemed a waste of pay.

    Now my database has gone away.

    Oh I believe in yesterday.

    Suddenly,

    There's not half the files there used to be,

    And there's a milestone hanging over me

    The system crashed so suddenly.

    I pushed something wrong

    What it was I could not say.

    Now all my data's gone

    and I long for yesterday-ay-ay-ay.

    Yesterday,

    The need for back-ups seemed so far away.

    I knew my data was all here to stay

    Now I believe in yesterday.

    1. this

      Re: Yesterday

      er, a milestone hanging over me?

  8. Anonymous Coward
    Anonymous Coward

    Hummm...

    How does this Staffers were informed: “It is now safe to save new work on all drives.” work?

    Does it include individuals using their own USB connected hard disks? If why not, because that should give individuals the security of their data they need.

    As an aside, did the head of IT get his P45 for the monumental cockup?

    1. monty75

      Re: Hummm...

      More likely he'll get a promotion for his sterling work in managing the crisis.

  9. Dan 55 Silver badge
    Devil

    Get the IT head's head on a stick.

    That would offer more 'closure', I think the word is.

    1. Just An Engineer

      Re: Get the IT head's head on a stick.

      "It's too bad they got rid of all the old punishments", otherwise impalement might be an option for complete incompetence.

  10. Chris King Silver badge

    "The Chaplaincy has arranged two special drop-in sessions where people can either speak to a Chaplain 1:1, or share experiences with other people affected".

    I read that as "Chaplain 1.1" and wondered if "Chaplain 2.0" was a bot on social networking ! Or even...

    "Computer, activate Emergency Theolgical Hologram !"

    "Please state the nature of the Ecumenical emergency, my son"

    (I think we need an "I need to get out more often" icon for such misunderstandings...)

  11. FuzzyWuzzys
    Facepalm

    FFS!

    Get yer backup tapes out and fecking restore the data. If you ain't got any backup tapes, a) you're a fecking eejit and 2) get a job in a different industry 'cos you shouldn't be working in ours!!!

    1. Anonymous Coward
      Anonymous Coward

      Re: FFS!

      my partner works for a company that also lost a load of their own data - mainly all the customers account details and bills etc. The funny bit being - they are an I.T company selling , amongst other things, backup services !

  12. Mark 85 Silver badge

    IT Services has pressed ahead with the new back-up system which was going to replace the old one — and which in the actual act of replacing it, managed to destroy the backups — using “next generation technology”.

    So the new tech wiped out the data from the old tech???? Or is it the new tech can't read the old tech's data? I hope there's a serious flogging in store for someone then for not testing compatibility.

  13. Anonymous Coward
    Anonymous Coward

    Always back up your data!

    RAID is not backup - it's redundancy against some hardware component failure

    Disk/Volume Mirroring is not backup - it's resiliency against loss of source disk, but will still happily replicate corruptions and deletions of data

    The only way to do this is make a regular, complete copy of your data sets, logically separate to the source data, preferably in an offsite location, that cannot be overwritten (unless it's expired and replaced by other copies elsewhere by design/policy).

    Backup products are with big tape silos and disk silos and de-duplication tech are not cheap, and not always 100% reliable, but they are very necessary insurance policy over disk failure, data corruption and user error.

    1. Prst. V.Jeltz Silver badge

      Re: Always back up your data!

      Can we take this opportunity to discuss our own domestic data backup arrangements and weigh them up? I'll start:

      Main resource is a dlink NAS with 2x 2Tb drives in RAID , once weekly a vbscript/robocopy copies this data alternately to

      1) Another dlink NAS with a single 2tb drive in

      2) A 2tb drive stuck in the back of one of my computers,

      So i have 4 x 2tb drives with the same data on . Am i safe? ...must move that 2nd NAS to the garage really for off site storage

      1. The Onymous Coward

        Re: Always back up your data!

        Seems like you're covered for everything except a house fire, burglary, electrical surge and virus that attacks D-Link NAS boxes.

        Unless your data is almost worthless, which doesn't sound like the case here, it's worth paying for an offsite backup service. The one I use costs £4 a month and definitely works, having had my laptop's SSD fail shortly after my NAS was misplaced in a house move. I don't think Mrs Coward would ever have forgiven me if I'd lost 400GB of photos.

      2. Ogi

        Re: Always back up your data!

        > Can we take this opportunity to discuss our own domestic data backup arrangements and weigh them up?

        Sure thing.

        Main resource is a 4U rackmount server, running freeBSD and ZFS (24TB RAM, 6 core AMD), and 4 * 3TB raidz2 for main storage, 3x 1TB raid0 for scratch, with a UPS. Main file server, and runs quite a few VMs and background processing.

        Backup is:

        - daily snapshot && backup to a 6TB external drive

        - monthly full zfs backup to another 6TB external drive

        - weekly rsync's of core data to my friends server in France (and in return he syncs his backups to my machine, bit of a virtual "disk swap" backup on each others boxes going on).

        So far have suffered lightning strikes, floods, two massive array failures and an over active squirrel without losing data. ZFS in particular is amazing, as it can restrict failures to block level, so even when array was trashed, 70% of the data was still accessible because different parts of different disks had failed. So far have had to restore from external disk twice.

        It seems like a very resilient setup, and everything is scripted so I have nothing to do (except put the external backup drive in the slot). The two external drives are key for quick restores, the offsite backup is just to fill in anything that may be missing locally after the restore.

        Likewise when my friends server bit the bullet, he was able to get his core data back from my machine, so it really helps to have an off site backup. This setup is a like-for-like swap, and it implies a certain level of trust (otherwise you can just encrypt everything before you send it across).

      3. Vince

        Always back up your data!... more than once.

        Sure, we can do the "my backup is better than yours..."

        So let's start... at home..

        Servers are Replicated to Secondary Servers offsite - with hourly snapshots - handy for some basic recovery stuff.

        Servers are backed up locally to Hard Discs (in an array or 3, every 15 minutes)

        All data (documents, music files, photos, that kind of stuff) backed up to 2 different off-site services (one I control, one I don't directly control) - real time as it happens

        Daily Image sync/updates and backups to another offsite location.

        And yes that's at home. Because I'm paranoid. So multiple places, types and shapes of backup imaging and recovery options.

        I don't want to be THAT guy at work who preaches about backups to everyone else - staff, customers and so on, and then be the one who has to admit they lost stuff. It also serves as a handy way to check how good or bad this stuff is in reality.

      4. Pen-y-gors Silver badge

        Re: Always back up your data!

        I have one major work laptop to backup.

        I have a 2x1TB NAS in separate building. Nightly differential backups of main data directories for 30 days. Weekly differential backups of entire hard drives

        Approx monthly, complete backup to external hard drive that is then unplugged and put in a safe drawer.

        Also a lot of 'live' code is available on production servers as well (which are also backed up to AWS by me, as I don't entirely trust the hosting providers backups)

        Main backups largely cover accidental deletion or corruption (can go back to yesterday/last week), also total disk failure. Also loss or theft of laptop. Having NAS in separate building provides safety from fire and a thorough burglar. If one of our occasional overflying RAF jets ploughs into the office and destroys everything I'll probably be beyonf caring.

        The external hard drive backups give an extra level of protection against nasty ransomware that encrypts the hard drive.

        I'm sure I've missed something...

        1. This post has been deleted by its author

      5. TheVogon Silver badge

        Re: Always back up your data!

        "Am i safe?"

        Not if you need data from a version before your last 2 backups...

    2. Mage Silver badge

      Re: Always back up your data!

      I AM my own IT manager and I don't trust my backups. I make additional personal ones and tell users to additionally back up everything as well as having it on their laptop AND the server.

      You can't be too careful. Fire, Flood, plague, locusts, rats, mice, rooks etc. I also backup "off site".

      1. Just Enough

        Re: Always back up your data!

        I've spent my entire career in IT doing my own backups. Reasons are fourfold;

        1/ I don't trust an organisation-wide backup to restore something in both good order and good time.

        2/ When I mess up, I can fix it myself without having to go bothering others.

        3/ Having to redo a week's worth of odd-jobs, if you can remember them all, is the stuff of nightmares.

        4/ If you want a job done right, do it yourself.

        1. TRT Silver badge

          Re: Always back up your data!

          I think we just saw death of the first born.

        2. Anonymous IV

          Re: Always back up your data!

          I always thought the saying was,

          "If it [data] moves, back it up."

          And back it up even if it doesn't move...

  14. Anonymous Coward
    Anonymous Coward

    This article should be accompanied by a pic of someone doing a Gendo Pose.

    Terminal Dogma: Being forbidden to rely on one's own backups.

  15. TRT Silver badge

    I'm sort of concerned...

    that if that's the level of pastoral care being offered... what the fuck kind of data have been lost?! I mean, are we talking about somebody's life work, worth millions in research grants?

    1. Anonymous Coward
      Anonymous Coward

      Re: I'm sort of concerned...

      Probably a Don Brown novel McGuffin.

    2. Anonymous Coward
      Anonymous Coward

      Re: I'm sort of concerned...

      Yes, and yes.They have lost research data, admin data, student data. They did not have proper tape backups, they were relying on 1st and 2nd level backups stored on the same 3par as the live data. Unadulterated management incompetence, these idiots should not be allowed to flip burgers.

      1. TRT Silver badge

        Re: I'm sort of concerned...

        I believe the branch of McDonalds hosted in the corner of one of the Student Union buildings has now moved away.

  16. Anonymous Coward
    Anonymous Coward

    Outsourced

    I hope none of KCL's IT is outsourced. That would explain how the system was bungled...

  17. TheWeenie

    Backups for data in theological organisations strike me as about as logical as lightning conductors on church steeples. Surely if you place your belief in a God, then under what circumstances would your deity of choice decide to smite the building in which you worship, or destroy your data-set?

    Unless, of course...there's...

    1. TRT Silver badge

      Unless, of course...there's...

      a law requiring public buildings over a certain height to incorporate certain features?

      1. Destroy All Monsters Silver badge

        Re: Unless, of course...there's...

        Like suicide interdiction devices?

        1. TRT Silver badge

          Re: Unless, of course...there's...

          Have you heard about the suicidal pizza? Tried to top itself.

  18. Androgynous Cow Herd

    Faith based computing

    Is never a good idea.

    1. Anonymous Coward
      Anonymous Coward

      Re: Faith based computing

      Backup to the Cloud - it's closer to God, and just as reliable.

  19. Antron Argaiv Silver badge
    FAIL

    No personal backups permitted?

    Watch me...

    Now, if it's just "business information", I'm happy to abide by the rules. But, if it's data on which the grant of my degree depends, I, not some IT director, will decide how to keep that information safe and available. Because, if you want a job done right...and it's your ass in the sling if they screw up (and they will, sooner or later).

    Data security issues? Pull the other one. The risk of my losing the backup drive on public transit pales in comparison to the damage that could be done should some IT numpty manage to destroy the data.

    1. Pascal Monett Silver badge

      Re: No personal backups permitted?

      Effin A.

      Nobody is going to tell me I can't back up my own data, and nobody is going to keep me from doing so. Anybody with a brain at KCL should have a stack of optical discs at the ready, because completely losing data when you have an IT department with dedicated staff and adequate funding is just proof of total incompetence from the top down.

  20. John Styles

    Your data's going to be living on a (server?) farm upstate where it can run and play

  21. Ironclad

    Prayers and Poetry

    Reminded me of these, a little old but worth revisiting:

    https://www.gnu.org/fun/jokes/error-haiku.html

  22. Scroticus Canis
    Devil

    Our Tron who art in the data stream, forgive us our disc fails as we forgive those who failed us ...

    Nope... data didn't magically reappear. Data sins are not forgiven - just hidden if you are lucky.

  23. The Vociferous Time Waster

    Backups

    I sync to three different competing cloud providers from my laptop and my wife's laptop. If we lost both laptops it would be fairly trivial to recover.

    1. Anonymous Coward
      Anonymous Coward

      Re: Backups

      > If we lost both laptops it would be fairly trivial to recover.

      Would be? Or know it to be? Have you tested?

  24. Starting

    Let the uni do your backups

    Fortunately my uni only backs up the standard Windows system, which only poor students and other losers use. I wouldn't let them anywhere near the data on my data. So I take the stuff home each evening. It's against the data protection laws? Catch me! Prosecute me! Lol!

  25. G0HJQ

    Have they contacted GCHQ? Surely they have a copy of it all.

  26. Allan George Dyer Silver badge
    Joke

    A fine tradition...

    Nice to see KCL getting back to the vision of their founders, but perhaps a little more emphasis on the other half of their motto is required: Sancte et Sapienter "With Holiness and Wisdom". The backups are certainly holey, but where's the wisdom?

    Disclosure: I'm a UCL Grad.

  27. Anonymous Coward
    Anonymous Coward

    Padre

    My porn collection! :'( :'( :'(

  28. Anonymous Coward
    Anonymous Coward

    They still haven not learnt

    A new job advert from King's IT:

    https://www.hirewire.co.uk/HE/1061247/MS_JobDetails.aspx?JobID=74209

    "The post holder will be responsible act [sic] as a bridge between stakeholders in the organisation and will perform a range of activities which help ensure the operational effectiveness and excellence of the organisation through identifying, defining and validating internal and external changes."

    Poor English aside, King's IT are still doubling-down on appointing people to these kinds of pseudo-ITIL non-jobs and outsourcing to Cornwall (!) while the in-house IT rots. They've learnt nothing. Disgraceful.

    1. David Neil

      Re: They still haven not learnt

      Or maybe they decided that putting some sort of controls in place might help highlight these sorts of risks before they find out they've shit the bed?

    2. Anonymous Coward
      Anonymous Coward

      Re: They still haven not learnt

      £48K??? Is that all a Senior Business Analyst makes in London? Is that full-time? Or are they low-balling it?

  29. Fruit and Nutcase Silver badge
    FAIL

    Masters/PhD/MBA etc

    KCL should capitalise on the mass of material there for cast studies, theses etc.for students of data loss prevention - at least, how not to do it

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019