I'm willing to bet my 1996 bonus that Alan did not get "a job in devops" back then.
Good Monday morning, Reg readers, and welcome once more to Who, Me? – our regular trip down memory lane for those with something to get off their chest. This week, we meet “Alan” who once took out an entire trading floor at an arm of a US bank. Back in 1996, Alan had just graduated from a software engineering degree and had …
While it might be the buzzword of the CIO, CTO, et al these days, the concept has been around a long time, long before 1996.
No such thing as a new thing. I remember when people were getting all excited about the "new" thin client model a few years ago...."oh, you mean like the mini/mainframe I was using at university a couple of decades back?"
I think we called that a Client/Server Model of Computing like BACK IN THE DAY of 1980's era IBM AS-400 or even the 1960's/1970's era VAX-780 and IBM 360 mainframe. DevOps is basically Top-Down development and VERY BASIC 1950's era Systems Analysis.
a) Define the business problem you're trying to solve.
b) Define the FUNCTIONALITY of current or new components needed to solve a part of the overall business problem. (i.e. Do we need hardcopy printing, email, calendar functions, streaming videophones, SMS/instant messaging, calendar input, etc)
c) Define the TYPES of processing modules NEEDED to solve each require functionality as defined above. (i.e. print PDF reports, Enter data via touch screen or keyboard, send email to clients on POP server, use RTSP for video, etc)
d) Define the functionality of sub-components needed within each large-scale component
(buy RTSP DLL from Datastead for video streaming, use Thunderbird components for email,
code custom data entry screen in-house, use PDF generator component in compiler, etc)
e) Code, Link and Combine components into test runtime folder
f) test on SEPARATE MACHINE with SEPARATE TEST DATA ONLY!!!!!!!
DO NOT USE LIVE DATABASE !!!
g) test 1000 times with local admin department heads and team of each component and data entry and processing module over 3 months.
h) Do bug report log and send back for recode and retesting of 'E" and "F" and "G" over another 60 days.
i) Get Go-ahead for One-Branch-Office-at-a-Time deployment with initial two weeks allocated for in-house training and another two weeks of real-time use per office for real-world feedback and fixes.
If no major issues, then go ahead and use same two-week training and two week real world test schedule for next three months at another three branch offices.
j) If no major problems at the four branch offices, then roll out to ABOTHER four branch offices at the same time again with two week training and two weeks in-office testing until all offices update to new version of application. Update ONLY four branch offices at a time and train/test for 30 days!
h) Shut down last database from old system ONLY AFTER ALL OFFICES are converted but still keep on online version of the old software on ONE machine in every branch office connected to a separate head office server for at least one year as backups just in case something BAD happens to new system!
i) do a 3 month, 6-month and 1 year review of each office interviewing certain users personally on fixes or updates required for next app update version. Send responses back to coders and management for review and add-in and THEN DO ANOTHER office-by-office rollout but with an accelerated one week of in-house training and two weeks of testing for NEW bugs.
THAT is how you do a PROPER Roll-out of a new app using Top-Down Modeling and Deployment!
Amen, brother. After 12 years of dealing with that same scenario on the same application, for each update, I can attest to the absolute desire to chuck a bunch of flaming bags of dog/cat/horse/human/elephant/cow poo at the doors of anybody of that vendor that has a 3-letter-acronym in their title. Why haven't I moved on? The pay is fantastic, I determine my own (over 40 hours) workweek, and I know enough about the software that I commonly correct their developers on their mistakes. Sometimes, you find a tech job that actually is worth the frustration, irritation, and insanity that comes with it. I got lucky on that.
"It may not have been called "DevOps", but we most certainly used the same concepts and methods in the mid 90s."
One of the joys of being a grey-beard is that you can watch all the young folk [re]discovering so much stuff. All we need is a buzzy name and we can get waterfall development back in fashion.
All we need is a buzzy name and we can get waterfall development back in fashion.
Total Quality Artificial Intelligence Administered Stateful Software Management.
I can feel the dollars rolling in already. I'll write it once I've been funded enough to retire to a distant tropical island (volcano lair optional).
I'm willing to bet that "screaming traders assaulted the frame room baying for my blood” didn't happen either. From the traders I met when I worked in the city most of them couldn't tell you where their phone was plugged in, never mind where the server room was and even if they did find it , the door would have been auto locking even back then so unless 1 of them had the combination or he jammed the door open its total BS.
I remember someone unplugging the keyboard on a Sun box to tidy up the cabling.
The reason I remember this is that my boss at the time - a normally placid man - started screaming obscenities at the offender, as he happened to be in the machine room working on another system.
I thought I heard the words "force-fed", "floor tiles" and "without salt" somewhere in that rant.
Icon, but don't get another one - just plug the old one back in and step well away, there's a good chap. Or as the boss-man put it "GET OUT AND LEAVE MY KIT THE FUCK ALONE !"
David, this was a non-delightful feature of SPARC hardware for a long time. Probably still has it. Bitten by same "feature" two decades ago. Which reminds me, must dump the Solaris 8 pizza boxes lurking in shed so I have more room for something useful.
At least AIX, HPUX and PCs hardware was not so scream inducing. I loved the old PARA-RISC HP hardware. One could do a full backup without starting the OS using embeded firmware. Great for major OS upgrades and raw disk databases.
"A long, long time since I used IPX or Sparc but I can't remember that being a feature."
They don't panic, but they DO stop - at least IPX and SLCs did.
They also had this wonderful feature of stopping the entire machine to scroll one line of text on the screen, so the first thing you did after installing them was to install and start X
My recollection is that messing about with the console like this tended to cause the firmware to suspend the OS and throw you back out to the OpenPROM prompt.
Thing is, this isn't a kernel panic. If you kept calm, it was possibly to simply 'resume' the OS and life would return to normal. Only damage done would be that anyone's X11 session would have frozen for the duration, and mysteriously come back to life a short time later.
I remember these days well. We used to have stacks of Sparcstation20 and Ultra60 workstations with 10-bay SCSI disk enclosures chained off them running under desks in the dev area. Flipping noisy, but kept the office comfortably warm in winter.
No, I remember well that you could take down certain SPARCstations by unplugging the keyboard. Weird, but true. I was working for a trading firm where the Traders were in fact quite smart, and did have access to the glassed in room with the raised floors, so nothing about this story sounds at all improbable to me (except the duly noted anachronism of the buzzword 'DevOps'). About that same time, working late I accidentally did an rm -rf ./src in the wrong place, deleting all the source code staged for our current build. As I frantically called my boss (for visibility), I remembered that our NAS took frequent snapshots, and was able to restore the company's tens of millions of lines of proprietary source code, and kicked off a validating build that came up clean. This was way before "git" so source control systems were centralized, and our developers were not overly dedicated to committing code frequently, so this no doubt saved my job, if not the company. Ah, the good old days...
Almost as good as the infamous IBM PC boot error:
"Keyboard not found. Press F1 to continue"
I don't get why people think that is so stupid or funny.
Plug in a keyboard and press F1 to continue booting... What else were they supposed to do? Shut the system back down immediately? It's a clear, concise error message with an easy resolution.
"Plug in a keyboard and press F1 to continue booting" would therefore have been a better and unambiguous message.....
I think the reason that message was "funny" is precisely that it generally happened when a perfectly good keyboard was indeed correctly plugged in all the time. The problem lay elsewhere.
I was using mostly sparc workstations around the time of the story. But I don't recollect ever yanking a keyboard out, so I can't say one way or t'other whether anything bad happens. I suspect it depends on what is listening to the keyboard, and how it reacts on losing it, hence some seeing huge overreaction while jake saw no problem.
My one and only experience with SPARC workstations was when visiting a VLSI lab and the tech there was proudly showing off the multi-user, remote desktop capabilities. I noticed the mouse buttons didn't work and he calmly pointed out that numlock was on. I calmly pointed out that overpriced crap is still crap. The look of hurt on his face still haunts me and I have since learnt to temper my honest opinions when not on home turf.
Anonymous because the hard working tech guy could very well be reading this and still holding a grudge, or wielding a type4 keyboard.
I don't get why people think that is so stupid or funny.
Because when confronted with a message on a screen, people's understanding becomes astonishingly literal. The stupidity really isn't as one sided as it seems at all.
(I wonder how many have headed for the exits when seeing the lpd "printer on fire" message)
Because when confronted with a message on a screen, people's understanding becomes astonishingly literal.
Like people who phone the helpdesk saying that they can't find the "Any" key.
"I am told lots of Russian keyboards have "ANY" (in Latin characters) handwritten on the space-bar (Russian has no definite article)"
If translated there are several ways of doing it which would not involve a possible confusion (you might write "press any out of keys", just as you would write "take any out of books".
If untranslated the problem is more with word order. In Russian if the noun comes first it tends to be treated as definite, whereas if it comes last it's indefinite (Stakan na stole, the glass is on the table, versus Na stole stakan, a glass is on the table.) So yes, "Press any key" could be read as "Press the 'any' key" but it's a little bit of a stretch.
So my guess is that Russian IT people have simply got used to the abysmal standard of error messages and may simply be taking precautions. Computing is Anglocentric enough without factoring in poor syntax as well.
This is pre-dates USB by a long time. Hotplug hardware was somewhere between rare and fictional so plugging the keyboard back in could break the keyboard, motherboard or both. As the hardware did not support hotplug there was no reason to have the software handle it. If the hardware survived then the firmware in the keyboard controller could hang so the machine would need a power cycle anyway.
Now imagine you have got your server almost ready to install on site. You remove the keyboard and video card because no-one will be typing on that machine, switch on and the damn thing beeps at you. You read the manual for the motherboard and discover the beep code means the BIOS has not detected the video card and refuses to boot. You put the video card back in and try again: the machine hangs. Attempt three with a monitor shows the infamous "Keyboard not detected. Press F1 to continue". After summoning a horde of demons to hunt down and sandpaper the programmer responsible you dust off a coffee stained keyboard and rip the keycaps off it so the machine will boot (and the customer will not press the wrong button).
Years later, Microsoft came up with: "Mouse not detected. Click here to change."
Had to set up some display screens, so used some old PCs that were laying about. Didn't want anyone messing about with them, so disconnected the keyboard. And get the error message come up.
My solution was to remove one of the back plates., take the keyboard apart and remove the PCB. Wrap it up in tape and feed it through the gap and fasten it inside the PC. The cable would still be connected to the PS2 socket so it would detect a keyboard and boot up. Added advantage is that it saves space!
IBM PCs and their "Keyboard not found - press F1 to continue" and its very irritating cousin "Mouse not found, click 'OK' to continue" are funny on the way that clonking your funnybone or touching an HT lead when messing around under your car bonnet with the engine running, is funny. As in, you have to laugh because screaming obscenities loudly is not considered acceptable behaviour in the modern work environment.
As the OP has obviously never experienced either of those errors, we should take pity on him or her and explain that you could swap the "broken" keyboard or mouse to another PC and all would be fine on that one, and the keyboard or mouse that had been working without problem 'over there' would also give the same error when attached to the offending machine.
As a Conehead in a previous life, I found percussive maintenance often resolved the issue...
"Plug in a keyboard and press F1 to continue booting"
I think it more likely to have been the result of a specific error message and the automatic concatenation of a standard phrase to any boot-time non-fatal error message.
The Intel board on my Mythtv box does something similar. It has a setting in the BIOS for running keyboard-less but on boot still reports that there's no keyboard and that the error is "logged" (where? - no don't tell me I'm not really interested) but carries on booting which seems to be the only effect of the BIOS setting.
"I am also fairly sure if I did grow a beard it would be of all colours"
My beard used to look like that. Now the red and brown hairs have faded to grey, as have some of the black ones, the blond ones are hard to tell if they are now gray or not. No grey anywhere else though.
@onefang, lucky you. Up top I've got salt and pepper going on, down under though.... slap a red hat on it and it'd look like an ugly santa impersonation. Only reason I can think of is how when some people get a sudden shock their hair turns white. Truth be told over the years it has seen things, things that would terrify H.R.Geiger so perhaps I shouldn't really be surprised.
I don't feel like grey beard yet..
That's OK. We welcome diversity here. Beards of all colours and even non-beards welcome.
I agree with Korev: there are times when a greybeard icon would be useful here. And of course it would be open to honourary greybeards as well as us literals.
Having replaced terminals on all kinds of running Sun gear before, during, and after that time frame, I'm fairly certain that unplugging the keyboard wouldn't cause a kernel panic. And having just pulled the keyboard from a running Sun 4/50 down in the machineroom/mausoleum/morgue/museum, I can fairly confidently state that that still doesn't cause a kernel panic.
Just a few days into my first proper job the unix sysadmin wanted to show off the insides of the shiny new Ultrix mainframe to me.
Unfortunately, removing the backplate caused an instant loss of power. Oops.
I learned a lot of lessons that week...
Unfortunately, removing the backplate caused an instant loss of power. Oops.
I learned a lot of lessons that week...
Lessons like "power interlock switch on the cabinet"? Very valuable lesson. Also slamming one's fist in frustration on the cabinet would sometimes trip those switches.
And having just pulled the keyboard from a running Sun 4/50 down in the machineroom/mausoleum/morgue/museum, I can fairly confidently state that that still doesn't cause a kernel panic.
Surely it'd depend on the OS, though?
Memory and Wikipedia both claim the 4c architecture ran SunOS (BSD-based) and Solaris 2 et seq. (SysV plus chunks savagely ripped from BSD). Or you could run stock BSD on it. And apparently there are Linux distributions, though I doubt a bank would have been using that in '96.
And then there's the question of OS version. And could the 4c's firmware be updated? (I haven't had one of my own since, oh, 2002, maybe. So I've forgotten many of the details. I did like those pizzaboxes as personal workstations, though, back in the day.)
Who remembers the early days of the 80286, which had the A20 gate in the keyboard
You left out an important word here: "controller". The original IBM PC/AT used the keyboard *controller* (not the actual keyboard) to manage the gate that suppressed access to what was later called the "High Memory Area".
The reason was that the 80286 had what amounted to a bug in its implementation of "real address mode", where the CPU itself did not suppress the carry out of A19 in the addition that calculated the physical address from the segment:offset virtual address. (It made the emulation of an 8086/8 faulty, in that FFFF:0010 was not the same address as 0000:0000.)
Actually I'm pretty sure the keyboard controller was used to get the 286 out of protected mode by giving it a reset. There was no instruction capable of exiting protected mode on the 286 as Intel had assumed that once you went in, you would never leave.
There was then a bit of a dodge in the 386 that meant you could set up extended segment selectors in protected mode which were not cleared on exiting back to real mode, this gave you access to 4GB of memory from real mode. It was a bug, but people started to use it and then Intel guaranteed to keep it there for all future versions of the x86 architecture. So yes, along with Gate A20, I assume it is still there in current CPUs and chipsets!
If it was a classic cherry keyboard then they probably didn't want the keyboard nicked :) and thought "Lets make the system go down if the keyboard is removed. That will then make everyone notice and we won't have a stolen keyboard on our hands".
I'll get my coat.
It wasn't a kernel panic, but the Sun machines had a tough time differentiating between Stop-A and the keyboard being unplugged. Stop-A was a means to break out into the rommon to debug the kernel, and was reasonably difficult to preform, and was introduced into an era when machines were built to be serviced by kernel systems programmers to find kernel bugs. Then continued on long past the day when this was useful.
It was just unfortunate that the Stop-A procedure was confused a bunch by the keyboard being unplugged too.
Some SE courses wouldn't have intersecting circles, believe me.
I worked at a place where students on a so-called "Informatics" degree could avoid programming altogether right up until their final year, and even then they only did Visual Basic.
I confiscated their old Linux box because a departing lecturer thought he could trust the students with root access - as you can guess, it wasn't long before the Conch was smashed and Piggy was killed.
Yes... @Chris King is right... I know this is true because a class colleague of mine ended up at a fancy uni studying for a B.A. Informatics degree, whereas I was at the equivalent of a polytechnic and got a diploma for the exact same thing. Where they studied on all the theory of everything that is business administration, we got dumped in the deep end with our first COBOL classes within two weeks of arriving there and not even knowing where to switch the workstation (an ancient, even for that time, Olivetti one) on.
To add more fun, some machines had the old 5.25" 360K drives, whereas others had the more modern 5.25" 1.2MB ones, and... even more fun, the third-year lab had... *GASP* 386es with both 5.25" *AND* 3.5" 1.44MB drives! That lab was always busy... with DOOM players.
It was bizarre to have my class colleague begging me to teach her COBOL at the end of her second year, where I was already having loads of fun with things like ADABAS/NATURAL, Pascal and C (COBOL? Who dat?). Her course was 4 times the price of mine, yet here I was teaching *her* stuff that her lecturers didn't bother to teach them!
Yeah. Now she's a house wife (nowt wrong with that, but a bit of a waste of 3 years at uni). I'm messing about in science, aviation and IT. Funny how that works.
YEAH POLY! POLY FOREVER. :-D
P.S. And yeah, I am of the vintage where dodgy hardware things are all too familiar... and this story is not unplausible at all.
Seems very unlikely that unplugging the keyboard would cause a panic, and even plugging it in again the panic does not just "go away". It's possible that it caused a hang, in a similar way to the STOP key on a Sun workstation IIRC. And plugging it in again, might well un-stop the workstation.
Mine just logs an error and continues on, happily toggling transistors without a care in the world. As you would expect from a network aware multi-user system that supports remote logins. In fact, many (most?) pizza boxen ran headless as a matter of course, as did a lot of the lunchboxen (such as the IPX) but it was routine to attach a console to them occasionally. Or to use a KVM switch to move between them.
I once had a music CD with the Sony copy protection. It said "will not play on PC or Mac". Just out of curiosity, I popped it into our departmental Sun Server. = instant kernel panic. I quickly took it out again and was hard at work fixing the machine when the management arrived to find out what happened.
I once had a music CD with the Sony copy protection. It said "will not play on PC or Mac". Just out of curiosity, I popped it into our departmental Sun Server. = instant kernel panic.
An excusable event, assuming that the Sun Server didn't carry a sticker saying "will not play with a Sony CD"
Many many moons ago, somewhere around 1995 or 1996, I accidentally (really) tried booting an already-installed Slackware 3.1 (I did say it was many moons ago) Linux(1) with an audio CD in the CD-ROM drive.
It was a proper Red-book audio CD since none of the later "innovations" about CDs were widespread at the time I bought it (which was, in turn, some time before 1995 - I bought my first CD player in 1987, ffs), and the 1.2.5 or maybe 1.2.13 kernel didn't like what the CD-ROM drive told it during the pre-init hardware detection phase, and promptly panicked.
I took the CD out and rebooted and all was well. Needless to say, I never again left a CD in the drive at boot time.
(1) A development machine for $JOB. Running Linux. In 1995. fvwm, tcsh, and a Pentium with the FDIV bug.
At my first IT job as the operator trainee tending a Honeywell mainframe in early 1980's, we were told to be quick about adding paper to the console printer (which collected status messages, rather like syslog or journald), as the system would supposedly crash if the printer was offline for more than a minute or so. Luckily I never found out if this is true.
'twas true for some kit, back in the day. When the printer ran out of paper, the computer buffered the output until more paper was loaded. If the buffer filled, you crashed. Try to remember, though, this was back in the day when a meg of core memory was considered extremely extravagant ... and I don't think that Honeywell had that problem, although I vaguely remember some people thinking it was a normal problem with computers in general.
Back in the 1980s I worked for a large Government department in Gloucestershire. The local tech college (not Gloucester) ran a course on "C" programming so our boss signed us up for it. The "notes" looked as though they'd been written in the pub the night before and the machines were on a flaky network where you had to quit the editor in the "proper" way otherwise the whole network crashed. We reported this to our boss who managed to get the course fees returned to his acount.
This one has to be anon.
About twenty-five years ago, I was working for a large bank on their market data systems. The vast majority of the Reuters market data was page based, each page being 17 rows x 64 columns of data. However Reuters had just released a new set of page data (in beta) which was 25x80.
I typed in the new page name (I think it was BP.L or something like that) and lo and behold, up came a new page, 25x80, with lots of useful data. I showed this to a friend of mine (really, it was a friend of mine), who said "I've got a mate at XXX bank which runs YYY system (different to ours) - I'll tell him about it"
So he rang him up, and this is what I heard.
"Have you seen these new large pages from Reuters?"
"Well, try typing in BP.L"
"Ah. Well, I'd better let you go then."
He put down with the phone, turned to me with a grin and said "We've just brought down the whole trading floor at XXX bank."
Yes, sometimes your machine won't work if the system teletype (it was the 70's) is on the fritz. Luckly, we transported the inner workings of the ASR35 (EBCDIC) off to get fixed. On the way back from doing the transport, I thought a bit, and hooked up an idle computer (HP 2114) to take the place. A little software later (translation from EBCDIC to ASCII) and it worked. Good thing as well. The repair was to take a day, and ended up taking three as I remember it. When returned, it snapped back in, and all was well.
Of course things like this are hard to forget. I needed to a more urgent repair a year or so later, and took out the old hardware and slapped it together again. We had it that way for about 5 days as I recall. Somehow those "temporary" lashups have a life of their own.
"Luckly, we transported the inner workings of the ASR35 ..."
Just looked that up - made me feel v sentimental. At school we had occasional access to a PDP12 (I think) with mark-sense card reader and one of those Teletype terminals with tape punch. Regret not keeping a few of those cards and a bit of paper tape - those were the days when you could hold bits and bytes in your hand and look at them.
"Just looked that up - made me feel v sentimental."
Remember seeing KSR33 in the "really old computers" section of a technology timeline at the Bradford Media Museum and similarly feeling very nostalgic ... my first computing was done on one of those connected to a Data General Nova at school (had similar waves of nostalgia at the Computer History Museum on a visit to Mountain View where they had a Nova ... took great temptation not to pretend to load the bootstrap on the front panel switches!). My abiding memory of teletypes is that, unlike virtually every other keyboard, if you hit it in anger you came off worse!
Luckly, we transported the inner workings of the ASR35 (EBCDIC) off to get fixed.
I worked on Teletypes when in uni. It's extremely unlikely that the ASR-35 was EBCDIC.
The amount of mechanical redesign to make that happen would be prohibitive. I'm not saying it's impossible, but if true, it would require an impressive amount of hardware bodging.
Now, I *might* believe that, squirreled away somewhere in the voluminous base of the '35, there was an electronic ASCII-EBCDIC converter board...
"Now, I *might* believe that, squirreled away somewhere in the voluminous base of the '35, there was an electronic ASCII-EBCDIC converter board..."
Somewhere in a junkbox at one of my parents places there's a bit of perfboard with an ASCII<->BAUDOT converter on it. The (entirely mechanical) Creed Model 7A teletype it drove is long-gone.
perfboard with an ASCII<->BAUDOT converter
Built one of those myself, and it would copy the shortwave Baudot RTTY transmissions on an ASCII 33 machine. I had loads of fun with it, still have it...somewhere in the attic.
The UARTs today don't support 5-level, so you'll need to bit-bang the Baudot to drive whatever.
I have a TT-253/UG "typing reperforator" which could stand to see some data...must get to work on that.
I recall a tale told to me by someone who allegedly worked in a trading bank in the UK many years ago. Apparently one trader was so incensed by him taking more than 2 seconds to diagnose what was wrong with his PC (something along the lines of cables kicked out by user) that the user actually started kicking him and screaming abuse. His response was "in that case fix it yourself", followed by "I earn more than you, I'll get your boss to fire you" and the bosses reply of "you can fix it yourself, you don't earn more than him."
Not sure how true the tale is but every job I've seen for the financial sector has stipulated experience within the financial sector or the job is specifically for a graduate with no real life job experience. Something to do with not knowing that you're supposed to be treated with respect by other employees apparently...
The Sun IPX was never meant to be a server, but a tiny workstation. Typical configuration was something like 40MHz CPU, and 16MB or 32MB of RAM. It came in a tiny "lunchbox" case. (about 1 foot by 1 foot by 8 inches tall).
I doubt a PC era hardware was more powerful, but almost certainly, SunOS was 1000 times more stable and capable then anything running on a PC at the time.
"The Sun IPX was never meant to be a server, but a tiny workstation. "
Perhaps, but it packed 16MIPS and 16MB RAM when PCs were lucky to push 1MIPS and have 2MB.
It took several years for PeeCee technology to outclass them and those PCs needed substantially more memory and CPU MHz to do so, which was surprising given that SPARCs are RISC and Intel are CISC, so it's supposed to be the other way around.
My first one cost $2500 (secondhand) in 1993, and another $1000 for the staggeringly large 1GB drive to go with it. It was the core server of my network for about 6 years (and that keyboard thing was annoying)
I used to support a lot of video editing and effects systems running on various Silicon Graphics workstations. The Tezro was particularly good in that it would display a GFX error on the front panel LCD if the mouse was unplugged. What it was trying to tell you was that it couldn't move a pointer in X11 without a mouse, but it halted the entire system and made it look like an expensive hardware fault.
It was 1987 and although the start-up I was working for had just flamed out, the one client we had committed to deploying our system was building their new trading floor and were not about to fill it with Quotron terminals. Instead, we deployed a fleet of Macs, both color and black&white, both ethernet and localtalk, to handle the traffic. I did the transport layer, both server-side and client-side. Riding up on the train one day, inspiration hit me and I came up with a semi-reliable (reliable enough, as measurements proved) UDP broadcast protocol that solved our "TCP can't possibly handle hundreds of machines" - at that point, with the Mac II being the most powerful machine we could buy. And yes, it handled Black Monday later that year with aplomb.
should be taken out and shot. Lesson #1: All cords to hinged or sliding equipment need to be secured. Most rack mount keyboard/display units have a hinged or flexible arm just for this.
I did work for an electric utility where rack mounted relays had to be slid out for calibration. Before s substation had been commissioned, the relay techs found that the leads to the racks had been made too short by the electricians. They proceeded to take side cutters and lop off every cable to every racked relay. Then called the electricians to come back and try it again. Correctly this time.
> Most rack mount keyboard/display units have a hinged or flexible arm just for this.
It did take manufacturers several decades to get to this point, and neither low-end kit, nor reassuringly expensive boxes were necessarily quick to join this trend. Not necessarily feasible to have everything secured too, when a rack's keyboard would be moved between servers at the top/middle//bottom of the cab. Especially in places which didn't have the budget for expensive serial console servers (only for a moderately expensive proprietary keyboard).
While I have never seen a kernel panic from a yanked keyboard cable I wouldn't write it off as BS. Pulling the plug cleanly would have caused a stop-a I think (in-head memory is faulting when I try and access long-term storage for useless facts) but if the plug deformed and the pins separated from he motherboard connector in haphazard order, who knows?
I once worked at a place that issued their staff those neato IBM Thinkpads with the butterfly keyboard - the one that sprang out all steampunk-like and assembled itself into a full-sized keyboard when you opened the lid of the decidedly tiny computer.
The way IBM managed the miniaturization trick was to move all the peripherals normally fitted into the case (floppy, CD etc) out into external units that connected via a parallel or serial port as appropriate.
Problem was, the back of the lappy had a proprietary port the width of the laptop designed to dock into a "workstation" frame, and to break out the pins of this enormous harmonica-like port you needed to snap in a special accessory that hooked over one side of the laptop and snapped onto a spring-loaded pawl on the other. Still with me?
A colleague was madly typing away one Wednesday afternoon when the plastic pawl broke and the port adaptor thingy ejected. But it did so in an arc, starting on the left and not completely disengaging from the right side when the laptop reacted.
The bizarre unplanned unplug resulted in the lappy hardware sending all sorts of interrupts to Windows 95 in rapid succession, which decided that the laptop obviously needed to load a new profile as it was clearly undocking. From something. It didn't have a profile to load, but didn't let that get in the way of an attempt to do so.
It got about halfway through this before the laptop hung (or was turned off in a panic; I was suspicious but couldn't prove anything).
There were lots of company-revenue-generating ideas on that machine's hard-drive, but because of the new-fangled RIDs 'n' SIDs 'n' Bellz 'n' whistlez all now were unavailable without recourse to advanced low-level digital spanners and hammers.
They were still recovering the machine when I left work on Friday afternoon.
Many years ago, back when I worked for a telco, I was given the job of replacing a noisy fan tray in the processor module of a NEAX61E-VS (very small) digital exchange, the purpose of which was to provide the 0800 service for a smallish country.
Unlike its big brother, the NEAX 61E, the VS version had both of the hot-standby processors in a single module, rather than two separate modules.
What wasn't made very clear in the procedural documentation was that the 50v power feed to processor modules was via relays which were held operated by a signal from the module's fan tray - obviously to shut the processor down should the fans ever fail. As it turned out, this was a fine concept for the 61E, but not so clever for the VS version.
So, having undone the retaining screws I carefully removed the fan tray, only to be greeted by a distinct lack of blinky lights on the processor consoles and the cards which filled the processor module.
By the time I worked through the convoluted boot process (by modern standards) and then loaded the core software followed by the latest data backup (all from tape cartridge) people had most certainly become aware of my activities.
No point in posting as an AC for this, it was all documented and acknowledged at the time.
"By the time I worked through the convoluted boot process (by modern standards) and then loaded the core software followed by the latest data backup (all from tape cartridge) people had most certainly become aware of my activities."
Not as aware as they became when someone futzed up PMR switch.
That reminds me of the time in the mid 90's that i went to clean the old vax/vms consoles keyboard.
In came the the boss and everyone into the computer room, to find me, cleaning cloth in hand, proud of the shiny keyboard which had been a little crusty before.
Apparently, the pause/break key really did what it said.
But you just had to press resume and the system started where it left off..... phew(ish)
I remember those days where there desktop machines were really just mainframes / mini-computers that were trimmed down with the standard TTY and printer were replaced with a video controller and a keyboard. They'd go all weird weird when one of those wasn't working because the system never expected those things to not be available (what with them supposed to be soldered on...).
I also remember some of the models that tried to be smart and if a keyboard and/or monitor was missing at boot, the system assumed you are wanting to use Serial 0 as the console. So if you accidentally knocked out the keyboard at boot, the system would work just fine (OS would boot, daemons would start and begin doing their thing, etc), but nothing would be displayed on the console, nor would keyboard input do anything (Since the keyboard input is now routed to TTY1, but TTY0 is attached to the kernel).
A lot of it was just really teething issues and programmers needing to unlearn a bunch of assumptions from before the beginning of the transition from computers having their own rooms to them being out in the office.
I worked for a large investment bank in the late 2000's with multiple trading desks and about 2000 employees, plus two huge server farms spread across two datacentres for DR purposes. The whole trading infrastructure was an absolute rat's nest of duplicated systems and people's vanity projects which all had to be kept in sync with a sprawl of horrible middleware and half-arsed trade feeds, all built completely differently and inconsistently. One day one of the tech team managed to issue a remote shutdown with some kind of crazy wildcard that rebooted every Windows desktop and server across the whole organisation. The time it takes the machines to come back up is trivial, but in a normal (for example post patching) environment the machines would normally all be brought up in tiers in a particular order - domain controllers first, then things like email servers, then servers housing SQL Server databases, then servers housing application engines, then finally things like desktops. This was important as a lot of the services were set to auto-start and they'd get confused if things they depended on weren't already up. Once all these machines had been brought up the application teams would start bringing up remaining services, synchronising FIX sequence numbers for trade feeds, resolving issues with things that are partially processed or transactions rolled back as part of the shut down etc etc. So resumption of trading varied desk by desk depending on their systems, but in many cases they weren't able to trade for several hours and ironing out every single issue and making sure all trade and pricing data across all systems was correct and synched up took days. On the desk I worked for, what was the impact of not being able to trade? Well - after hours of the traders screaming at us that they had positions to close out, markets had moved unexpectedly and the P+L on the positions they'd been unable to close out was actually substantially up. That one IT guy literally made millions for the bank - he still works there now in fact.
Biting the hand that feeds IT © 1998–2019