Those among you who are prone to vocal outbursts of rage at cantankerous kit should be aware that hardware has feelings too, and directing your ire at disk drives can provoke "unusually high disk I/O latency". That's according to Sun's Fishworks engineer Brendan Gregg, who explains just how to upset your drive on his blog and …
Don't get mad ....
... get a hammer.
That is all.
Mine's the one with the 7lb lump hammer marked 'Fine Adjuster' in the pocket
Quite right too...
...shouting and verbal abuse is reserved for users, percussive maintenance is for hardware.
I always find 14 lb ones, available from most DIY stores, are more effective (and satisfying)...
Just don't forget to buy a pair of safety goggles while you're at it...
Alternatively, "earth" your equipment to the building's lightning conductor and wait for a storm, cut a hole in the roof before ordering a skylight, etc....
Mine's the one with a copy of that entertaining read "50 ways to shut down or crash a Netware Server" in the pocket...
Talking of which...
Some of the suggestions in November's "Back the F:\ Up" thread would also be satisfying (e.g. initiating a thermit reaction on the equipment - hence the icon).
And if your server uses RAID, you can hot swap a healthy disk in to solve the immediate problem, then have fun "fixing" the dodgy disk...
At last, some common sense engineering experimentation for 2009 instead of some whacky partial financial modelling from those crazy maths guys...
I find a bit of percusive engineering works wonders
Nothing teaches new hardware to work properly than sitting the unit it replaced beside it, after you've stuffed it with oily rags and burnt the misbehaving unit out as an example to all.
It's pickin' up bad vibrations
Is anybody concerned that this guy works every day in an environment this noisy without any form of hearing protection. He could probably lower the latency of the drives still further IF THERE WASN'T SO MUCH BLOODY NOISE IN THE LAB IN THE FIRST PLACE.
When a machine really deserves it, I personally like to switch the power supply (on back) to the wrong voltage.
With the right PSU your box will make a screaming sound and begin to spew white smoke from the back. Now that's satisfying.
@get a hammer
that used to be (semi-)official support advice from Sun engineers ... the original Sun4 SparcStation had hard-drives that fairly regularly would get stuck in the "head park" position if they were powered down. The advice to fix this was to give the unit a sharp hit on the side to dislodge the heads!
Of course, while the effect of shouting at your discs and causing vibration is quite diverting, the point of this video is to demonstrate the detailed instrumentation available in these new 7x10 arrays.
The UI not only shows the momentary increase in latency, it identifies which spindles this is occuring for. When you remember how inexpensive these arrays are, this is an extraordinary level of detail. Oh, and this is available out of the box, it's not an optional extra.
Re:Quite right too...
"...shouting and verbal abuse is reserved for users, percussive maintenance is for hardware."
If only it could be the other way around. If only ...
I wonder what the device is above the array in question ?
Re Device above
It's the head unit for the 7410. It's actually a 2U AMD based server.
To Simon and AC
"It's pickin' up bad vibrations"
- Come now Simon... Who works all day in the lab? That's what remote access is for. Of
course it's possible that this guy does, but I doubt it. Your other comment on reducing the
level of noise in a lab... Have you ever been in a "real" data center? They are loud! The air
conditioning alone is loud, but then when you add a ton of spinning disks and server fans to
the mix it's incredibly loud. The lab shown in this video is not any louder than the average
DC (try putting a mic in your data center and see how loud it sounds). Ear protection is a good
idea though. I know many old mainframe guys that are now deaf because of data center
noise. I often spend countless hours in data centers without hearing protection and I doubt
that I am alone in that regard.
"I wonder what the device is above the array in question?"
- That's the actual 7440 NAS Head. Then above that is another JBOD hooked up to another
7440 NAS Head. You can check them out on Sun's website.
I for one am very impressed with the level of detail shown by the analytics of this box. Very impressive! EMC, HDS, and NetAPP don't even provide this level of detail in such an easy format for fee.
Re: Device above
It's not the head unit for the the 7410.
The 2RU unit for the 7410 is a x4440. The device in the picture is 1RU, and has 4 non drive modules
in the front.
@To Simon and AC
"Have you ever been in a "real" data center? They are loud!"
Not as loud as the silence when the power and the UPS fails. That is deafening.
Paris, 'cos she likes a good back up.
It's a fundamental design problem...
The basic mathematics of binary computing is the cause of it all.
Subtraction is done by two's-complement addition (it's faster).
But two's complementation requires a bit flip to be propogated through an entire byte value. This often results in a carry at the top end, which either goes to the low order if that's a zero, or has to be dropped.
These spare bits left over simply crash to the floor, which is why there is so much noise in server rooms, and the poor hard drives can only take so much vibration.
Consequently, they are hyper sensitive to any additional vibration induced by our angry vibes.
Yes, mine's the one with the bunch of April dates hanging out of the pocket.
That approach might have fixed one of my drives which had stuck heads, if I had known the problem before opening it. The difference being that the heads were stuck to the platters. I don't think the motor had enough torque to free them – it took quite a bit of force by hand. Nothing particularly important on that drive, but I make better backups now.
re: @To Simon and AC
"not as loud as the silence when the power and the UPS fails"
You've never heard a UPS alarm in close proximity then? You're right, you won't hear much after that. Except a high pitch whistling noise
That might explain it.
A couple of years ago, contractors were installing attachments for a false ceiling on the floor below mine, using hammer drills. I could feel the vibration through my feet, my chair and my desk.
I got the distinct feeling my PC was running slowly, but thought at the time that it was just the racket making me impatient and affecting my thinking. It was driving me up the wall.
My system box was on the (vibrating) floor, so this research could provide the real answer, and I was not going crazy.
Phew! Thanks for that.
To paraphrase Beldar Conehead
Sure, and let me know when elvis gets here ...
Close, but wrong. The problem does lie with the binary nature of computing, though.
The issue isn't so much with the zeros as it is with the ones. Zeros, being round and smooth, flow along just fine. The ones, however, knock about, tumble, and get jammed together, resulting in noisy data flow (just like water in a pipe). Today's faster computers with their smaller conductors in all the ICs compound the problem.
The old curly keyboard wires were really notorious for getting the ones stuck in them, which is why you tend to see the non-curly wires on modern equipment.
Mine's the one with the BOFH excuse of the day calendar in the pocket.
This probably also means "don't put speakers next to your PC" and "Don't turn the volume up too high"?
If it doesn't work...
...hit it with a bigger hammer. If it breaks, it needed fixing anyway.
Motto of engineers everywhere.
the magic smoke escape while you had your back turned while going for the hammer
stick that in your drive bay and format it!
re: two's-complement addition
Is two's-complement addition really faster, or is does it just require fewer transistors (reuse those for addition when doing subtraction)? At one time minimizing transistors was very important, but today the question is usually how can you best use the available ones.
Is it time to reevaluate the choice between one's complement and two's complement arithmetic? A one's complementation (nice verb) is a simple XOR operation and does not require a bit flip to be propogated through an entire byte value. This would mean no spare bits left over to crash to the floor. Perhaps the noise associated with these bits was not as significant years ago when the processors were several orders of magnitude slower.
foot work ?
I find kicking the case often makes the clicking noises go away, usually a lose cable catching on a fan, soon goes away, the last drives that failed got some large nails pushed thru with a sledgehammer as not want personal data to re-appear :)
If only I could forcechoke some of the users or do a "Scanners" number on them, I wouldn't need percussive maintenance. No physical evidence either...
Paris, because forcechoke wouldn't work on her, her throat muscles are too well disciplined
Am I the only one who remembers having to gently warm IBM drive spindles with a lighter to get them to spin up after a long shutdown?
Re: Hard drive
Probably, everyone else died when they got doused by an overzealous fire control system.
re: two's compliment
I had to laugh when I read the posts on two's compliment ooperations as both of the previous posters ignored the endedness problem as though it didn't even exist! In my day we had to balance the big endian rigs with the little endian ones so's they complimented each other. We were young in those days and no problem was insurmountable, when I see the kids these days practically counting on their fingers I despair.