Crashed and alone in a remote location: When paid help is no help

Wilds of Yorkshire

"The Third World" sketch in the Python film "Meaning of Life" springs to mind here for some reason...

The Server CPU Swap game

There we were working away in Central Asia when one of the Two servers died. We found that it was a CPU board failure. The nearest one was in Moscow (3 time zones away). After several long phone calls we dispatched one of the client team to the Airport. He caught a flight to Moscow where he was met by the Field Service Manager. A CPU Board swap took place in Sheremetevo Airport and the return flight was duly caught. The Local went because he didn't need a visa to enter Russia. Us westerners woul dhave needed one. The airfare at the time for locals was 1/4 of that of us rich westerners.

A little under 10 hours after the crash the System was up and running again.

The IT director took us out to dinner for fixing the system in the way we did.

This was in the Mid 1990's. Those were the days.

Quality service from DEC

An employer in the 1980s had scheduled an upgrade to a VAX. The engineer was supposed to be on site for a few hours during which the system would be unavailable -- timed to cause minimum impact to customers in Europe and Japan.

Everything was going well until the engineer stood on the last board to be fitted. Ouch. DEC found a replacement in Manchester and it arrived three hours after the accident. That's a pretty good time for handling a distress call after normal working hours, finding the board in a warehouse and driving it to the Midlands. The VAX was back in service later than scheduled but no customer complained.

It was an early learning experience:

* Competent plans go wrong in unexpected ways;

* Great suppliers are the ones who do a good job when things go wrong, not organisations that (unrealistically) never make mistakes;

* Set honest expectations -- with staff and customers -- and be frank about errors and problems.

Well, it does reiterate what a redundant system is:

One box here, the other box over there. Hopefully on a different network + power supply.

In another building would be good.

Its not a good idea to have a 'redundant' server that can wiped out by a single cup of coffee.

First off, it's always almost faster to listen to your user base and their ability to detect service failures than to rely on autonomic monitoring systems. Even remote telemetry solutions like Sun/Oracle, etc have are generally slower than the user picking up the phone to cut a ticket. Still helps with failures that don't cause an outage that you might not notice until you scrubbed a log, but, when the server completely craps out you will likely know before your monitoring tools do.

Remote locations often consider an onsite parts agreement so that critical components are in the DC already for the engineer or the customer themselves to use to restore service without waiting for delivery of parts that could be delayed due to weather, traffic, or because the one part you need is out of stock at your local stocking location.

Most of the time its probably cheaper (long term TCO) to have N+1 redundancy than to rely solely on Premium support SLA to keep you in business. Depending on the costs of an outage you might be able to get by with business hours support on gear that you can afford to lose availabilty on for a few hours. Clustering, load-balancing, now "serverless" application designs or VM/container mobility strategies can buy you time to diagnose and restore individual nodes without having to make the panic call to the vendor at 0-dark-thirty. Of course back in the day of this story there were less options on that front and the redundant gear tended to be a little pricey to be left idle.

Cool Story Bro

Tisleton Scars - not very remote

I could be in the Hill Inn in twenty minutes from there.

Hmmm

Then one evening at about 6:45 I was having dinner

I do think sysops should really get into the habit of saying things like "18:45 ZULU". It lightens the load on ticket wrestlers and possibly lawyers...

Interesting from the point of view of support

Always found support in the midlands to be 3-4 hours for one vendor, fascinating to hear that Yorkshire has much better response (but I guess they were paying for it).

SSD wasn't even heard of back then

Well, NAND flash SSD, maybe.

Basically, core memory is SSD too. And in the 1990's several manufacturers had a couple of solid state drives in their program DEC had one, physically the size of a HSC50 (can't recall the model number; ESE50?) which was essentially a backplane filled with 150MB worth of DRAM boards and a SDI interface, plus a MVAX board with an RD54 hooked up and an UPS. If the power went out, the UPS was to keep the lot running while the memory contents were transferred to disk. Later they had a drive with a 3.5" form factor, SCSI interface, static RAM and a rechargeable battery. Couple hundred MB, IIRC. No idea of the list price of either, but definitely well over that of their size in spinning rust.

Service? What is this - service?

I have only worked for one business where the owner was willing to pay for a service contract. For the rest, we made up a song, "The Electron' Swap" to cover how "service" was done:

"I entered the office late one night,

The hardware systems were a ghastly sight,

Our two 'hardware specialists' had their screwdrivers out,

there were pieces of gear all strewn about.

They did the swap, the electron' swap..."

The nearest Frys was over an hour away.