Lateral thought saves sizzling server

That's nothing

I had a 'faulty' ceiling mounted projector in a Primary school classroom. Often it would run for the whole morning, then switch itself off at lunch. Sometimes it would switch itself off after a couple of hours. Sometimes (as far as I knew) it would work fine all day. It always started up immediately after switching itself off and had no unusual settings. I tried changing the bulb, but that didn't help.

Unfortunately I didn't have a spare at the time, but the holidays were coming up. The following week, I swapped the projector over with another of the same model. The problem stayed with the room, and not the projector. So, it must be the mains - but everything else in the room was fine, and not cutting out. The ceiling socket looked fine, and Estates new of no fault with the electrics in the building.

OK, so more investigation. I connected an anglepoise lamp to the projector and left the room for half an hour. When I came back I peered in through the classroom window, and the light had gone out. I walked back into the room, and, as if by magic, the lamp was back on! Then a moment of inspiration. In the corner of the room was what looked very much like some kind of motion sensor. I left the room for half an hour, waited for the light to go out, then walked to the doorway and stuck my arm in. The light came on.

After Estates took a look, it turned out that the electricians who had put the extra ceiling socket in for the projector had tapped it off a mains power cable running along the top of the room. This cable was from an old lighting circuit connected to a motion sensor that everyone had forgotten about. The teacher of the classroom had her pupils working so quietly that sometimes the motion sensor thought the room was empty and so switched off the projector.

Phone problem

When I was a student, the hose phone started behaving oddly. This was about six months or so, after changing from BT to NTL (against my wishes!) sometimes the phone would chirrup and there would be someone at the other end, sometimes people would complain that they had been ringing us all night and no answer, other times we would pick up the phone and someone would be there, but it wasn't ringing. All this, while the phone was, occasionally, appearing to work normally. NTL visited about three or four times over a three month period until I got fed up and demanded that BT be put back in.

The BT guy came round and asked where we wanted their box, I told him and he asked if we minded if he used the old NTL wire across the house, which we would no longer be needing, I said that was fine. About twenty minutes later he said that something was wrong with the wire and he needed to go out to the van to get some test kit. Upon connecting the tester he pinpointed the problem being behind a radiator. The NTL guys had put a metal staple through the cable.

The phone was fitted in the summer, when there was no need to use the radiator, when we turned the heating on it caused expansion which made a short, when the radiator was off, everything was working fine.

Ah, sounds familiar...

I had a machine that suffered from sunburn too..

My favourite weird one was the machine that would never crash when left on it's own.

I could access it remotely, and run it's little cpu 100% 24/7, no problem at all.

But if I sat down at the desk with it, within an hour it would crash.

Until the one day it didn't... The one day I used it for hours whilst sitting at the desk, and it didn't put a foot wrong. The one day when my mobile phone was off being repaired!

Since then I've never owned a PC with a plastic case.

Dataplex

Many years ago, I gleaned from a far-away operator that the kit only went wrong when the sun came out and she had the covers off.

The kit was controlled by photo-sensors....

WOW

It's amazing how simple problems seem difficult if you don't think.

A new lease of life for an aging processor

I've had similar experiences with a desktop machine which I converted to a web and file server for development purposes at home.

This particular machine (an AMD XP 3000+) was forever failing on me. First it told me that the hard drives had gone. Not having the cash to buy new drives, I was in a bit of a panic about this until for some unknown reason I decided to try replacing the IDE cable instead. Funnily enough, this solved the fact the drives had seemingly packed up. Never did work out why I decided to try changing the IDE cable but I'm glad I did. A cable is a damn sight cheaper than 2 new hard drives and the frustration of restoring 200GB of data from backup

Around the same time as this happened the system started to shut itself down, freeze, reboot and do all kinds of peculiar things. At first it would only do these things when it was hot. I was forever cleaning it out from dust and grime (I was living in a pretty grotty hole when this started). Some times it would run for weeks or months on end without incident and others it would fail 8 or 9 times a day.

My initial thoughts were that it was likely to be an overheating problem. Most of the time it played up was during hot or humid weather although the fans never seemed to be working unduely hard. With this in mind I began to suspect that perhaps the temperature sensor on the MB had packed up or was in the process of failing.

This problem has been going on for about 3 years (yes, I'm still that skint I cant afford a new system yet) although since march it had been getting much much worst to the point I actually couldn't boot the system for more than an hour at a time or until I tried to open any applications.

About 2 months ago I decided to look more closely at the matter.

After careful thought I concluded that the system only crashed on me during hot weather or when I was placing undue load on the system. This narrowed it down to one of two things. Either memory or the CPU. Seeing as the memory had been upgraded, I wondered about the CPU.

Rather than replace the CPU itself (again through lack of money) I thought to try throttling it back. The clock base defaults at 166MHz. I brought this down to 100MHz and it was stable over the weekend. Wanting to find out how much the system could handle, I took it up to 150MHz and the system was stable overnight but died in the morning with a bios error relating to clock frequency. I've now dropped it down to 140MHz and its been stable for just under 2 months, and has taken everything I've thrown at it so far including indexing over 110GB of audio tracks, a feat that has not been completed in a single session in nigh on 3 years.

So now its time for a new processor although this one is currently performing quite nicely even though it is on its last legs.

I probably could have solved this a long time ago but to be honest, until March I wasn't all that bothered. I rarely use it as a desktop itself, preferring to do most of my work from my laptop and just use it as a server, and whilst it was frustrating at times, I could live with the occasional 5 minutes downtime whilst it reset and cooled enough to boot up again.

Nightmare on Elm St.

Many years ago when I was "last level" support for a hardware supplier, we got a call from a disti who had installed a network for a high-profile client. It suffered from occasional, but disastrous networking problems. Back then, networking was a black art (remember those old, thick, yellow networking cables?). After piling in their own people, analysers, reflectometers, tracing software and everything else they could throw at it, they finally called us in.I sat in a small cubby room for days (on charge, natch') squeezed in with seized/evidence equipment - yes it was No. 10 Elm St. and all the disti's kit, and nothing happened. After a while, the account manager thought the problem must've somehow fixed itself and was considering declaring the problem "solved". You guessed it - massive packet loss, collisions, machines crashing. The TDR showed up which cabling segment was at fault and off we went to find out what was going on.

It turned out that the cable was running under a staircase - with one rocky step, just by a window. When a courier had made a delivery, or a pickup, he would call in to get the next job. It turned out that mobile reception in the building was lousy and the only place his phone would work was near that particular window. There he'd stand for a couple of minutes, squashing the 10 base-2 cable and causing networking ructions. While he was not the only person to use that staircase, no-one else loitered on that particular step for any length of time, so the problems from people passing by were too small to notice.

Heard a better one

Can't give any specifics, but where I used to work there was a story going around about a network connection failing every time the toilet was flushed.

Dodgy wiring to the water pump.

Solving another intermittent problem

When I was at Uni studying Engineering some years ago, we had a Professor who did a bit of external consultancy. He would occasionally tell us about some of the jobs, and this one was my favourite.

He was called to a paper mill to try and find out why just every now and then the paper was coming out with uneven thickness and, at considerable cost, had to be thrown away.

He looked around at the operation and asked them a few questions. He then said that he could fix it and named (by his own admission !) a hefty price. They were so keen to get it resolved they went for it.

He said "See that window up there - put some blinds on it". He'd worked out that when the sun was in a certain position it was falling on just part of the mill rollers and they were expanding with the heat !

I do hope there are still guys like him teaching ....

never trust control panels

One place I worked, room full of Vax's (ooo 18 years ago). One day im round the back of the big one, feels a bit warm. i tell the sysadmin, he wanders over to the air con control panel, no error lights, everythings ok. "yes xxxx tends to get a bit warm, its a big machine"

first really hot day of summer, mid morning i cant log into any of the vaxes. helpdesk reports come in saying the same. i go over to the other building to see whats going on, only to find the loading doors wide open and the sysadmin powering down the (19"/340mb) hard drives.

the 5 machines log printers were going mental as being in a cluster everything got reported across the whole suite. one goes "I've lost connection to A!" everything else going "B has lost connection to A!" as well as their own problems.

what had happened is 2 out of the 3 aircon units had failed but due to a fault the fail lights weren't working on the panel. 1st hot day, the last working one gives up the ghost, machines overhead and its goodbye productivity for 1000 people.

@David Perry

I can see why pruning trees so they don't collide with double decker buses (for example) is the council's job, but I can hardly see how it is the council's job to ensure that line of sight RF links don't get obstructed by trees. The council isn't the stupid entity in your story, rather it's the university. That is, if your story isn't apocryphal...

A Bit Harsh

Come on, guys, we all have to learn some of them painfully. Netware servers are old enough that it was probably many years ago when overheating wasn't so much of a problem, especially in Scotland. I've had occasion to impress the management a few times when a problem appeared during a customer demo (actually a spurious signal on a bit of radar test kit) which I solved by turning off the nearby monitor. All of a sudden the 18kHz spurious disappeared. What they didn't know is that a few months before that, I'd spent most of a day trying to find out why one of my test circuits, that was working happily the day before, seemed to be oscillating at about 21kHz. Then near the end of the day someone who'd been using the department computer (in the days when EGA was the bees knees and an IBM AT cost seven grand) shut it down, the monitor went off and my problem disappeared.

Some problems are only obvious now because of hard, painful experience or, if you're lucky, a good tale of woe from a colleague down the pub who had the experience.

re: eh

"So the server was randomly crashing and it took you how long to consider it might just be overheating?"

It was in Scotland! Trust me overheating is not the first thing you would think of!

@Martin Proffitt

>I was forever cleaning it out from dust and grime (I was living in a pretty grotty hole when this started).

And

>After careful thought I concluded that the system only crashed on me during hot weather or when I was placing undue load on the system. This narrowed it down to one of two things. Either memory or the CPU. Seeing as the memory had been upgraded, I wondered about the CPU.

PSU, either dodgy or needs cleaning, in a dusty environment probably the latter.

Cash only please

<no title>

A stern talking to a council actually had an effect besides being passed to someone else time and again? Wonders will never cease. Must have finally spoken directly to the council's tree pruning chap/chapess.

Networking

I remember a network hub that was run by our engineers (before standardisation and telling them It kit was our role only) that would shut down at "random".

We eventually found it connected up to a power switch that they had taken off the same line as the nearest hand dryer.

When the senior engineers used that one rather than the usual toilet ones it would spike the power and kill the hub.

Three biggest causes of failing systems

Dodgy wiring.

Faulty Power Supply.

Heat.

Check those first and 9/10 you'll fix the problem.

I recall....

..doing an installation in a quarry of an IBM cluster controller with a few screens 'n printers linked to a S/36 up the way over a BT leased line with BT sync modems.

Couldn't get the bloody thing to work at all. To add insult to injury, every time I reported the line faulty to BT, they insisted that they'd tested it and it was OK. The only evidence of something being up was a set of cable clips running up both sides of a dividing wall and ending at a hole that didn't go through the (rather thick) wall.

Eventually, I managed to get BT to get their engineer at the Exchange to call me directly while I was on-site. He was seriously pissed off 'cos he'd tested the same line repeatedly for the thick end of a month now and he normally only worked nights (big clue here). "It's up and fine, I can see carrier from both ends", he says. "So it is" say I. A couple of moments later, "It's bloody down now", I say. "F*** me! So it is." he exclaims. This up 'n down process continues for a while and he agrees to send an engineer on site.

The clue was the cable clips. That was where the original installing engineer had found that his drill wouldn't reach through the wall (not even half-way through). He'd then noticed a phone in the back office where the modem was required and "borrowed" a couple of free pairs off the existing cable. One of said pairs was a tad dodgy and every time a truck went over the weighbridge outside, the resulting vibrations caused an intermittant short and the line would drop.

It sounds like an urban legend...

...but it really happened.

Way back, the company I worked for had an NCR mini-tower server in the main office of a local theme park. We'd get regular weekend call-outs from staff based in other offices because the database had crashed. The server was fine, sitting there waiting for logins but the database process had terminated. Easy enough to restart the database but it was annoying as the software (Progress) was usually pretty stable.

Didn't take long to check the server logs and find that it usually went down on the Friday evening so I asked one of guys in the main office to work late on Friday and see if anything happened.

Six o'clock and the cleaning lady came in and asked if he minded her doing the office while he was there. "No", he replied, "go ahead". So first thing she does is unplug the server and plug her hoover into the socket.

Problem solved for the cost of a sticker with "DO NOT UNPLUG" written on it.

I am shure I typed

www.theregister.co.uk

not

www.anoldfokeshome.org

must be wed morning

@ Pooper Scooper

ditto.