Pro tip: Servers belong in dry server rooms, not wet cloakrooms Welcome again to On-Call, our regular look at the things your fellow readers find when they're asked to go out and fix things. This week we hear about the time that “Stuart” was handed an unusual assignment: his boss asked him to visit his employer's solicitor. “They felt they had been ripped off by their IT contractor,” …
OK, how about a very large well known UK organisation.
Server room with full heavy 19 inch rack, the floor it was standing on was chipboard.
The air conditioning baffle was above the rack, it got a blockage and dripped water down the rack, which found its way to the floor and turned the chipboard into Weetabix.
The rack went through the disintegrating floor and was thus inclined like the tower of Pisa.
With no downtime possible the solution was to tie the rack to a roof girder with a rope, so eventually the rack was hanging in mid air. Still live.
I hope Stuart charged the solicitor at the rate that the solicitor would have charged him for doing something completely elementary. Having recently done probate and conveyancing twice each, it was clear to me that they are money for old rope. However, there is some niggling detail at a level comparable to the questions about computers that are answered for free on numerous websites.
It isn't 300GBP per hour. It's normally time-sliced to be 30GBP per six minutes (or part thereof).
Thus do not be surprised when the lawyer (lawyeress in my case) spends 30 seconds of chitchat time to find some common interest (eg a hobby) that turns into an extra 10 minute (60GBP) of added value (to her, not me).
Generally speaking a solicitor stays put and you go and see them. The only costs are their time and their seat value (which includes equipment, furniture, their share of the building operating costs etc).
The IT consultancy firm typically visits you and therefore shells out for employee expenses. If you're located anywhere expensive (anywhere remotely close to London) those expenses can easily run to £150+ a day just for accomodation and food, plus any travel expenses (mileage/parking/trains). All that goes on top of their time and seat value. Covering said expenses is why you'll struggle to get IT consultancy for cheap, unless it's someone located locally who can 'pop in'.
totally agree I did my parents probate myself. Admittedly it was fairly straight forward no trusts, offshore accounts, loads of property, etc The estate was over £550k but it was very easy to do, download forms and documentation, read documentations fill in forms, send off to HMRC, piece of pi$$ Total cost feck all. As you say money for old rope like a lot of things solicitors do.
In your case it may well have been a piece of the proverbial, but it is different when things look that way but there's an adder under the grass - you won't catch that but a solicitor will, perhaps even saving you half or more of an entire estate! You won't know that until you're well in it... then you might think a quick telephone call to the solicitor will clear up your question, and be surprised to receive a significant invoice for thier fees - again, they'll have given you information over the telephone with minimal information about the whole case.
NO - I am not a solicitor nor is anyone else in my family a solicitor, however I have received their services and seen some of the work that went into it and so appreciate what I paid for.
How is it possible to not know that there is only one backup tape, and not be aware of what that means ?
Not knowing IT is not an excuse here, you don't need to be a mechanic to know that your car needs refueling. People know tapes, we've had VCR technology for more than a decade (okay, that was a decade ago now, but still). So people should know that a backup is done on a tape, then you put in another tape. It's not rocket science.
This is a clear case of "It's all magic and I don't understand anything anyways, so nothing can go wrong". Well that only works as long as you're lucky, and these idiots were extremely lucky.
"How is it possible to not know that there is only one backup tape, and not be aware of what that means ?"
The story is a bit vague but as a new backup tape was bought every month it was used to replace the previous tape in the drive. The only way to know that there hadn't actually been a backup would be to check and it seems that until "Stuart" came along there'd been nobody near the machine with the knowledge to do that.
As to location of kit when the lab I worked in first acquired a server back in the early '80s (anybody remember the Onyx?) it was put in a cubbyhole next to the lift. Transients from the lift mechanism meant it reset at frequent intervals until we persuaded TPTB that this wasn't a good arrangement it got it moved.
"Transients from the lift mechanism"
Reminds me of very long (200+ m) cable runs of RS422 Multidrop running Wyse terminals and a bunch of printers to a SCO 3 server (386, 4Mb RAM) in a superyacht shipyard running frikkin big welders. It worked most of the time, only requiring terminal resets a few times each day.
Also, a similar system in an aluminium refinery. Of course, it wasn't the transients that were impressive there, it was shovels standing up by themselves in the eddy fields. I can't remember what voltage they ran the refinery vats at, but it consumed hundreds of kiloamps.
A very long time ago I worked for a company that, in addition to flogging bespoke business apps, built the multi-user machines that they ran on. Everything there was done on the cheap, and the base configuration had just a 5 1/4" floppy for backup. One day a service engineer was called out to a site that had suffered a head crash. He replaced the disk and demanded the backup media. And was presented with a single floppy disk. Total data loss.
But the worst of it is that every night a secretary had sat there watching the backup program and pressing the Enter key when it periodically asked for the next diskette...
-A.
I now have a job for a school as my predecessor not only did not backup, but when the first RAID disk in the only RAID set on the only server (don't go there!) fell over, he just left it. A second disk eventually failed and took all their data with it.
They had to send all the array off to a specialist data recovery (proper, expensive clean-room recovery that cost more than all the server hardware put together, not to mention RAID reconstruction after that), then recover the user's files from cached local profiles (I kid you not.... they basically sent all of the machines on-site to the IT department one-by-one for user folder recovery), then stitch it all back together by hand until they had enough to be getting on with.
Ironically, to this day, I still get emails from backup software suppliers addressed to that guy - everything from cheap freeware junk up to Backup Exec. And the drives? I took over shortly after they'd got back up and running, found the original drives in a box, still with the data recovery re-seals on them, and had them framed. They hang proudly on the IT Office wall with the pig-latin of "Cogito Ergo Facsimile" (I think, therefore I make copies!) as a warning to future IT guys there. And for at least the first year I had a very nice budget, especially where redundancy, resiliency, backups, disaster prevention etc. were concerned.
The school quite like me now. They accidentally overwrote the main salary file last week. I had something like fourteen copies of various ages. I recovered the first (and most recent) within ten seconds of getting back to my office. They are quite happy to have that peace of mind, now!
We had a customer, they had a backup running every day, but nobody knew Linux, so nobody checked the backups ran, they just assumed the 7 year old tape in the 7 year old drive was backing up properly.
When the 7 year old server died, they had total data loss. Luckily we still had some of the configuration information for their ERP software, but after 3 weeks, we were still fine tuning their set-up.
Er.... Never heard of the scumbags who setup a company create havoc and then liquidate it before the shite hits the fans and then they setup another company (often with an almost identical name) and ... wrinse/repeat all the way to the bank.
Now try taking legal action against an entity that no longer exists. What answer do you think that your £300/hour is going to get?
"If you're going to rip anyone off, someone who's versed in the minutiae of the law and understands exactly how courts work, would NOT be my first choice"
Stuart's story doesn't cover all aspects of the case.
Was the original contractor overseeing the server regularly or did they install the kit 3 years ago and were never contacted again to check it? Some companies just skimp on everything IT related.
Backups require - at minimum - monitoring the backup logs, and if there was no such thing then the original contractor blew it. That the system wasn't automatically overwriting the tape also tells that the system was only tested with empty tapes, and the fault obviously is with the original contractor. If the contractor was administering the server until Stuart came in then they obviously were just a bunch of cowboys.
The original IT contractor may haven chosen the cloakroom since the law firm may not have had any sort of spare room for the kit. Unless soggy clothes and umbrellas were directly dripping on the equipment or they were standing on water puddles I don't see any problem here. Servers are usually spec'd to operate in high humidity (at least 80%) and bunch of wet clothes shouldn't turn the environment into a jungle (or the whole building could get moldy).
Where did Stuart move the server and was there a good reason why it wasn't moved there three years ago? In the end, the customer has the final say for placing equipment and the three legged contraption was likely provided by the customer too.
Where better to put the coats ? We sometimes hang our coats in the hot side of the server room - it's running external air cooling so the humidity goes goes straight out the outlet, but the warm blast from the back fo the servers does wonders for drying time.
Worse still are those who have their own 'IT' who happens to be the managing partner's son.
Who may have been dicking around with group policy.
And managed to lock out everyone. Including the administrator groups..
Who had no backups worth talking about.
Or those who demand a cast iron guarantee that their seven year old hardware won't fail again. Ever.
I could go on. There's plenty of stupid out there.
Or those who demand a cast iron guarantee that their seven year old hardware won't fail again. Ever.
Yes, I've had that conversation with numerous idiots customers who insist that the cheapy laptop they bought in 2002 must still be fine, because "it's worked fine up until now".
Yes I say, "but the drive's failed"
"Really" they reply, "Well, what did you do to it"?
I have a client who's "server room" is a closet on the edge of the building. It is uninsulated and the ceiling is the bare metal of the roof. Every spring the snow on the roof melts and it rains in the "server room". When I brought this to the attention of The Deciders they solved the problem by placing the servers on blocks of wood so they wouldn't be sitting directly in pooling water.
There was a time when this would have bothered me.
A friend of mine had a similar situation, a council's server room was located underground directly under some public toilets. Every time they overflowed or there was a bit of rain, it rained in the server room and it filled up with several feet of water.
Their solution: Raise the servers up AND build a corrugated plastic roof over the servers with guttering to move the rain away. Wellies were provided at the door with explicit instructions to not ask what was in the water!
we had plans drawn up for a new data centre in our lab. Two possible locations, one next to our existing server room, so with all services within easy reach, risers, power, cooling ducts, lift, etc. Or another location in an extension of the building with no easy route for power, very poor access, etc, but the killer was its right next door i.e. other side of an internal wall, to a VERY large salt water lab, which has and does flood! and of course its a very humid and salty atmosphere there as well. Now guess where of senior management wanted to put the DC! as they wanted to use where we wanted to go for meeting rooms! Lucky for us of the 6 DC designers only 2 offered plans for the unsuitable site and one of those said I REALLY wouldn't put it there. But STILL senior management wanted to put it there! In the end a planning issue (residential area over looks the lab and we probably wouldn't have got planning permission to put the AC units where they needed to go without loads of hassle doing environment noise level tests, etc) put pay to it, thank feck! Potential total disaster area adverted!
Like the time I setup a split-mirror, remount and backup process on an HR system. Then 6 months later, tried to restore only to find I was splitting one disk set but mounting another with an image from 6 months ago. Oops!
From that point on, there was a flag file set with the epoch time on a reserved directory, then the post-split-and-mount script would check how long ago that value was compared to now. Anything more than 900 seconds, alert to operator.
Mid-80s, I was working for a company that built gear to dynamically allocate bandwidth between voice and data.
Incredibly Big Monster of a company (not HAL!) started getting weird bit error rates on their global T1 (E1, T3 etc ... ) network. I was assigned to track down the problem after lower level techs couldn't figure it out.
Going thru' the data, I discovered that once the problem started occurring at any one site, it gradually became worse ... It was never bad enough to actually take down a connection, but network errors ramped up over time.
Further review showed that the same team of installers had installed the gear at the sites with the problem.
I flew out to Boca and discovered that they had installed punch-down blocks in a janitors closet ... directly over a mop bucket full of ammonia water. Seems it was the only wall space that was unused almost universally in such spaces.
Corroded wire replaced and blocks relocated, no more bit-errors ...
I was called as the supervising manager by an out of hours engineer who told me it was raining in a server room ..... I met him and another engineer 'who wanted a look' and lifted the ceiling tiles in the server room and found a network of metal troughs zigzagging around the ceiling.
Someone had put a washing machine directly above the server room and not checked their joints were tight ..... dribbling, not dripping, had swamped the floor above, raised the water level to where the radiator pipes in the wall came through the turned up lino - i.e. the water was 2 inches deep and now running down the back of the lino, dripped through the floor void and landed in a trough above the ceiling. The troughs were so haphazardly bodged in that the joints leaked and water was dripping on the Siemens ISDX phone switch (a 5 cabinet 'large' if your interested). Eventually someone from the estates department turned up turned off the water, drained the troughs and sealed the floor upstairs. I shut down the ISDX and we booked a call with the isdx engineers to check out whether it was seriously damaged. We were in the throes of migrating to voip so the isdx only needed to hold out for a couple of weeks. We deployed 20 voip phones as an interim measure, circulated the details and went home.
Two weeks later I had a missed call on my phone from the boss at about 2am, I called back but got voicemail so I left a message and went back to bed. I wasn't on call so when I got to work I asked if anything notable had happened last night .... everyone was a little surprised, but I asked around some more and another manager mentioned that he thought the ISDX had caught fire and was wondering if that's what the missed call was about. I eventually got hold of the boss and he confirmed that yes, the isdx had caught fire, but he hadn't wanted to turn it off without knowing where the call traffic would go if it was shut down ..... my reply? Its on fire, turn the fucker off and fast - what kind of a moron doesn't think that killing the power to a piece of equipment that's on fire is a good idea.
It turns out that an on call electrician had been summoned to shut it down but had not realised that the front of this thing was a door, behind which was an isolater so after he'd isolated the mains at the upstream distribution panel he had levered his way in deep enough through the side casing to get a spanner on the hundreds of 1 volt lead acid battery terminals and had disconnected everything he could reach, manually, in the dark with his bare arm up a machine full of little buckets of acid the like of which he'd never seen before.
it took 2 weeks to get the funding approved to replace the network switches with poe models so we could finish getting the other 400 voip handsets deployed.
AC because that was 4 years ago and I'm still desperately looking for a way out of this crap, currently considering being a 3 star burger flipper if prospects don't improve :-)
Approximately 20 years ago I was in a situation when I had to take over the running of some systems. When I asked for the backup tapes I was handed a box which at 1st glance looked good, however at further investigation all the tapes were labelled XXXX 1 of 2, there were no XXXX 2 of 2. When I asked why I was told that 90% of the data fitted on a single tape and the operators couldn't be bothered to hand around to change the tape.. Anyway it was OK because "most of the data was backed up"..
I explained this was not OK, and that processes needed to change PDQ..
All the tales of leaky server rooms reminds me of Colossus, the Computer Wot Shortened the War, which was installed in a hut that flooded. Much more fun standing in water working on a machine with a couple of hundred volts everywhere.
Obviously not much has changed since then.
I worked in a very small publishing company where, on day one of my employment, I set up a back-up process, as i nearly had heart failure when I found that they had never, never, never backed-up. I did a once-a-day and a once-a-week, then a once-a-month, which I took home with me, as we'd had several small break-ins and things had got smashed.
As my fellow commentards have anticipated: yes, we was robbed once again and the safe with the back-ups was broken open and everything not taken was wrecked. Luckily, it was the second day of the month, so I nipped home and got the tape from the previous month and we were out only two days.
And then got bollocked for taking office equipment out of a 'secure area' and off the premises.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
