A boss pinching pennies may have cost his firm many, many pounds Welcome once more to Who, Me?, where El Reg readers share their IT catastrophes. And it doesn't get much more catastrophic than this week's story from "Marty". At the time of the incident, Marty was working in the trenches at a financial institution. "When I was first employed, the rack-mount servers for our division were …
I think it was Special Reserve that I bought the Star Trek: The Next Generation - A Final Unity game from and then entered (and won) the competition to win the entirety of TNG on VHS!
They changed the rules slightly after that as I had put as many postcards in an envelope as would fit within the first class weight limit of the time to maximise my chances of winning :-)
I used one of those along with 20 other engineers and things could get slow - a 3 minute jobbie would take half an hour or more. Until the day I discovered that a program I had would crash and leave me in whatever superuser debugging mode and then I could lift the priority of my batch job to 2 below max (any more and the OS hung) and my job would be done in 3 minutes and the system managers never found out why everything else ground to a halt.
I never did it on bigger jobs because they meant trips to the really good library we had there to further my knowledge of obscure computing ephemera.
>> Slow reports and breakfast breaks
Ah, the collision between technology and internal politics and procedures. Too often manglement ignore these things. You can almost be sure that a new accounting/auditing system will attract the attention of whoever is filching supplies or cooking the books. Some would say it's worth faking such a system in order to flush out anybody who is on the fiddle. Just look for whoever is sneaking in a sledge hammer....
And the rest.
Back in my first job (late 1970s) was working in a time-sharing bureau (sort of cloud on a single machine) which also had an application development department, and had a colleague developing an early client/server booking application for a holiday camp company. Grand programmer, but not in favour of writing code in separate compilable modules - umpteen thousand lines in a single COBOL program generally took the best part of overnight to compile! Clearly, he's not happy and I guess neither were management or the client. As in those days it was quite normal for the source code of the system programs (including the COBOL compiler) to be distributed, I spent some time profiling how the compiler was working and concluded that the bulk of the time was being spent sorting and re-sorting (virtually every time a new overlay of the compiler was brought into memory) the symbol table for the being compiled code.
Looking at the coding of the sort routine, I saw that it was the most basic sort possible (Order n^2 or worse) - so looked out my recently discarded University textbooks and re-coded it using a much more efficient algorithm in what I thought was really neat machine code exploiting the machine architecture in some "interesting" ways. Compile time for the application dropped from several hours to circa 20 minutes. Still not brilliant (breaking up into modules was really required) but at least several compile runs became possible each work session, rather than the one or two previously.
I did submit the revised compiler code to the manufacturer (Digital Equipment) as a "bug"(sorry, Software Performance Report) but I was never sure if it was incorporated into the production compiler as I never ran across such daft sizes of application design in the remaining ten or so years I continued working with DECsystem-10/20s.
10 minutes? Pah! I raise you Data General Model 30 compiling Fortran for up to four hours. Trying to look busy during that time was challenging. Let's just say that the devil finds work for idle hands!
So, exactly how much pr0n could you view on a Model 30 ? Or did it take that long to generate one ascii image?
25 years ago. Slackware 1.0. I decided to see if recompiling the kernel for my hardware was really all that worthwhile. Keep in mind that I was evaluating Linux (and Slack)[0] on a spare 386SX16. 8megs of RAM and a 16meg swap file should be plenty, right? 27 hours later ... a reboot into the new kernel, run a couple speed test scripts and ::drumroll:: I managed to speed up the system about 3.5% :-)
Was several years before I bothered recompiling the kernel at home again.
[0] Main (Intel) machine was a 486DX2 with a bank-breaking 16megs, dual booting BSD and Coherent. Slack soon joined the pair, and rapidly became my OS of choice. Still is.
"So, exactly how much pr0n could you view on a Model 30 ? Or did it take that long to generate one ascii image?"
Ha! This was in the days of the stash of a certain type of magazine in a colleague's desk, some of which featured a young lady who used to be a secretary in our department.
10 minutes! That's a blink of an eye!
I've worked on projects where it's taken a whole half hour to build and deploy a project to a dev environment. But it was ok! My tech savvy line manager told me to make as few mistakes as possible to reduce debugging time.
Worked at a company on a *huge* C++ app where one of my good mates was working on the 'main module' for the system. Over the years, the 'main module' turned into a dumping ground for everything which didn't have an obvious home (or if the devs were too lazy to figure it out). Inspite of having the fastest hardware money could buy, linking the main module took over 20 *minutes*! My mate could only try about two dozen changes a day. He did a *lot* of web surfing.
Actually I'm typing this at work, on a PC sporting a sticker proudly proclaiming its "Pentium inside".... and 4Gb RAM and a 32 bit version of Windows.... complex tasks (like highlighting a cell in Excel or switching between Outlook and Notepad++) make the PC thing twice or thrice before moving.... ahh, these are the good days!
Our company has a policy of renewing laptops every 3-5 years and installing Office of them whether you want it or not. Great, you'd think.
Now try asking for developer tools. You must ask your current project manager, who of course sees no reason to do this. Why should his budget pay for something that you'll still be using after you've left this project and are assigned to another?
Static code analyser? IDE? You're lucky if the VM you've been assigned to do build work on a) is not shared between 100 other VMs on the same server and b) has the full set of standard UNIX commands.
And you have to use C V Fucking S for the code repository. I don't know whether to be thankful it's not SCCS or cry.
In case anyone's confused I'm not waxing lyrical about 20 years ago. I'm talking here, now, in 2018.
If it helps, I'm a Programme Manager (Project Manager squared) and I'd fund that.
The amount of deliveries that have fallen on their arse because of this kind of thinking.... Perhaps that's how I got demoted and now actually have to deal with the PHBs as the Developers' Champion?
I always try and make sure my devs have the appropriate tools for the project and will go to bat if need be.Sometimes even the PM is at the mercy of the PHB's I took over a project many years ago the week TOAD became a commercial product. Just my luck that it needed to be licensed separately for the farcical price of £1500 per copy. My poor devs were given brand new top of the range machines but were forced to use the Oracle development set and vt100 terminal emulators.
I did have a huge bust up with a dev team manager once when I was a tech support manager. He got hold of an evaluation copy of the COBOL compiler for my IBM VM mainframe (*1 off offer 30 day license) because he appointed a contractor who only wrote COBOL (in 1990 when were were a SQL developer shop). He had the contractor in for 4 weeks developed the code and de-installed the compiler. Then it failed User Acceptance Testing. Unfortunately for him I wasn't willing to pick up the £30,000 PA cost of the compiler licence for one module in one app and one of the other devs had to reverse engineer it in Oracle.
And you have to use C V Fucking S for the code repository. I don't know whether to be thankful it's not SCCS or cry.
Fnarrrh - Try using Rational Clearcase with Rational Team Concert. Before diving into the void, one sniffed a generous lump of wasabi in the morning and the rest of the day would be somewhat better than that.
10 minutes? Oh, right, others have long-since pooh-poohed that.
Anyone else remember[1] the era when to build gcc, you would bootstrap a skeleton gcc using whatever native or other cc you could find, then use that gcc in a second pass to build the real thing?
My first encounter with gcc, I had to run the first pass with Sun's bundled cc. Then go back over reams of error messages and retry. Iterate quite a few times before I have a working install. And each build wasn't ten minutes, it was an overnight job.
About 30 years ago - when I had "mathematician" in my job title - I played my part in shortening some long jobs, by coming up with better algorithms. A mathematical model to calculate and plot coverage maps, down from over 4 hours to under a minute. Bootstrapping a tracking device (predecessor to satnav), down from 45 minutes minimum to about 90 seconds average to acquire a fix from cold.
[1] Silly question in this forum: of course half of you remember that!
Taking another detour from waiting for compiling...
I left a modern laptop running over night in the office yesterday, coz -
It hadn't finished installing the June Windows 10 update yet, coz -
No one had left it turned on for long enough since June, coz -
By my estimate, it was gonna take 15 hours to complete the update, having already downloaded it when I checked it yesterday morning. Maybe longer, I'll check it on Thursday.
"Anyone else remember[1] the era when to build gcc, you would bootstrap a skeleton gcc using whatever native or other cc you could find, then use that gcc in a second pass to build the real thing?"
Remember? I'll be doing that later today. An embedded system I'm responsible for uses Aboriginal Linux, where as an "air-lock" step, you first build gcc to build a host gcc (moving from "whatever compiler you happen to have laying around" to "we know what this compiler is, we just built it"), that is then used to build a target gcc, that is used to build the rest of the OS in a VM (in my case, a 486 VM). Linux From Scratch does something similar, you first build a gcc that is used to build a target gcc, that is then used to build the rest of Linux from Scratch in a chroot. Apparently Linux From Scratch was the inspiration for that process in Aboriginal Linux.
There was the time our company upgraded our NetWare 2.01 (running on a '286) all the way up to NetWare 2.15. That seemingly small jump (I don't think the 3.x series was out yet) turned a decently-performing network into a SLUG. A slug on a sub-zero January morning.
I managed to memorize the key sequences needed to log-into my system in the morning (depending on whether I had to run an EDI pull at the time). I would boot my machine (which loaded it's basic boot from floppy, no HDDs here) and type in as much of my morning login procedures into the keyboard buffer as possible, then go get my morning coffee. Then go back and pick up at whatever point it had ended up at. Entering customer orders was horrible, you could watch the order entry screen redraw line by line. Fortunately the boss accepted the point that the server HAD to be upgraded (a shiny-new '386, woo-hoo!).
Of course, running on ArcNet, there was only so fast the system was ever going to be (ethernet still being wildly expensive at the time), fortunately RealWorld accounting was text-only at the time.
This sounds terribly familiar. I worked for a life insurance company that used a similar setup. The computer room had a bunch of '386s for production, but developers got the old '286s. The network was ArcNet and there was a special network concentrator on the back wall of the computer room.
Their client server system was a bunch-o-Novell servers running TriMark software, aka Magic PC database with the HDD-less clients connected to it.
In 2010 I was working for a certain large software company in a suburb of Seattle, and my manager didn't care that the PC the company had provided to me didn't work.
"That's the machine we bought for you, that's the one you have to use."
So after months of suffering random hardware failures (losing several disks and having to reinstall my development environment many times) and making no progress with IT or my manager, I finally just went out and spent my own money on a very nice developer class machine. (I won't get into the shitty software designs the manager forced me to adopt -- I'd been programming computers since before this guy had been *born*, and he was an idiot).
I moved on to another, better project and much better managers. Took the PC home, since it was mine.
Sweet, sweet revenge a couple years later, when I got a call from Google asking about that manager. He'd been interviewing and had used me as a reference. That was a fun ten minutes.
I'm ASTONISHED at your story because (a) they let an outside piece of hardware onto the network and (b... and something I have experience in) once you bring your own hardware it becomes property of the company. My new pc-from-home became the bosses' new PC. Fortunately that company is in my past.
a.) Sometimes it allows you to be more productive. Good developers know more than System Administrators, that is why they are paid more. It is not like the silly guy from sales bringing his malware-infested, 5 years out of date Windows PC without AV, full of pr0n ....
b.) Since you fell for this, I understand why you are mumbling about a.)
Do you happen to administer a Windows network ?
10 minutes? Pah! I just wrote a compiler which takes *17* minutes to compile a one-line 'Hello, World!' program, and I have the video to prove it:
https://www.youtube.com/watch?v=1wLATW7sVXs
Admittedly, it is running on a BBC Micro. (See http://cowlark.com/cowgol/ for the main project page.)
That is nothing. Back in the 90s a few weeks after I took over as IT manager where none of the PCs had been changed for years,
One manager came in and demanded a new PC for the new starter he had hired and hadn't advised IT. He demanded a PC NOW. I said I haven't got anything suitable.
Look he said you have some in a pile over there. I explained the pile was the ones I had replaced as the slowest in the building then raided for memory to get the other ones to the giddy heights of 1 MB (I think it was a 286 or early 386sx when were putting out Pentium IIs. I said that if they wanted the staff member to resign immediately then sure I could build one of those. Personally I would recommend we order a new one. Don't be difficult I was told.
2 days later I had installed windows & office from the network (I had discovered network install meant I didn't need to swap 40 floppies so I loved it).
The machine was pronounced "ready", the manager watched it boot windows for 10 minutes complaining all the way. Twenty minutes later I had a signed purchase order.
sometimes you just have to let them figure it out for themselves.
... who had a boss who refused to pay for training. And then said boss yelled at the staff who couldn't support the thing the training was for. I've seen it zillions of times. You'd think that somewhere in management school they'd point out that that trick never works.
Ah, well. I get more loot picking up the pieces, I guess. But what a waste.
This round's on me.
Experienced that one. Some appalling "Enterprise Service Bus" thing that comes with zero documentation - the vendor makes a stack of money from training courses. You don't even really program the thing, it's some process diagram driven thing with cryptic icons representing processes that you pass around typeless collections of key value pairs to.
Reminds me of what happened here a couple of years ago.
I'm a 20-year veteran, so have been there and done that on most things. But we have a new tool-type which is different to our existing run of the mill stuff (I work for a semiconductor manufacturing tool vendor), and I was asked to support it. Also as background I'm a certified trainer on the older tool types.
So get trotted of around the globe for a week's training on aforesaid tool. All very nice and jolly, except I got back home to an email proudly congratulating me on now being a certified trainer for that new tool type too.
Yup, after a grand total of a week's hand-on with the new tool, I was expected to (and indeed actually had to) train both colleagues and customers on them. Shall we say the first couple of courses were "interesting", but at least they sharpened up my skills at winging it and educated guesswork...
Ah yes, experienced this recently.
It was brought up in a regional meeting that I wasn't HP Gen-9 Server certified.
When asked why, I told them that it had been decided (higher up) that it could not be justified to take me out of the field for 5 days of courses considering how few servers I repair since I am primarily a printer specialist.
Doesn't stop them sending me out on warranty repair calls for Gen-9 servers and then complaining that HP won't reimburse the labour...
In a school where I worked I found a locked room and got one of the caretakers to open it up and saw a dusty room filled old PCs that looked like they have never been used. Turned out very soon after the school got a suite of brand new computers in a properly wired network, some kid stuck a knife into the cabling conduit. They survived this unscathed but the school didn't have the money to get the contractors back in to do the repairs so the room was locked up (for context I dealt with some contractors (probably not the same ones) and it would be between £50-£250 for a new plug outside the spec they were working from and that was only because they were already onsite and had the stuff sitting in their van). Am guessing some were scavenged for individual classrooms but what a waste.
Worked for a POS (Point Of Sale, but the other meaning was also appropriate) software developer back in 2015. The owner had only installed two phones in the whole building that we worked in - one on his desk and the other for the first line support team to share. Sales and anyone at a customer site were expected to use their personal mobile phones.
The development machines were a bunch of clapped out machines cobbled together from off the shelf parts. The worst thing was that going on site meant lugging a battered tower case, LCD screen, keyboard and mouse along. Great impression to give the customers - although they were pretty clueless or else they wouldn't have bought the crap POS system in the first place.
The owner was also a control freak and would only pay things by cheques that only he was authorised to sign. That included our pay. On one occasion he went to sail his yacht around the Caribbean leaving us unpaid for several weeks after the date we were supposed to be.
All that and the regular bawling outs that the boss gave people were enough to convince me to change job.
Not actually IT related, but in a previous career (before I saw sense, packed myself off to Uni and changed to being a technician, I was doing general admin work for a local hospital (now demolished). The staff and visitor canteen needed a refit, as it had not been refitted since the early 70s (this was the early 90s), so the hospital got a company in to do it. They produced a lovely plan, showing the canteen (which had been the hospital chapel in a previous era) with beautifully positioned concealed lights which really did an excellent job of highlighting the extremely intricate detailing on the original ceiling that the fitters in the 70s had just slapped a false ceiling on. It did look absolutely stunning. The trouble is, the manager did not want to pay the thousands the company wanted to add a proper glass entrance hall, so asked them to remove it, then got a local fitter in to install a home conservatory. As a result, the door (because the home conservatory wasn't designed to stand up to hundreds of people walking though it) was out of action for repairs more than it is worked.
Also, in my current job, we had a room fitted out as a small studio. We spent tens of thousands of pounds on proper, good quality studio lighting, then my boss had to cut the cost of the project. So, he left the lights, and asked the installers to remove the computerised control system we'd asked for. The only control we had over those lights was the on/off switch on the wall, and whatever controls the lights offered on the control panel on the back of the light (assuming there was one). He also asked them to remove the control booth that was supposed to be at the back of the room, and most of the wiring. So, any users had to borrow equipment from us, or provide their own, and the only thing they had to record on on the room was a PC or Mac we provided..
I'm afraid that this sort of thing is not uncommon in the wonderful world of academia. Grants often include the purchase of big-ticket equipment, services or whatever, but the smaller, routine items come from a general overhead account or even aren't budgeted at all. Training is the obvious one. I was fortunate and worked for an organization that recognized the necessity of training, but you could find that you'd got the money to buy a vastly costly piece of equipment and then struggle to get it installed! We often found ourselves moving heavy, awkward equipment using members of our (highly skilled) team. Fortunately, H&S abolished that activity, as it was recognized that using untrained and ill-equipped personnel to move heavy equipment wasn't the safest of things!
I had the opposite experience. Working for a charity, a nice company (OK, it was Autodesk and I bet you haven't heard them described as "nice" before) gifted us a copy of Autocad (last DOS version). They also threw in three days training for two of us. However... the charity wouldn't buy a machine to run it on. Cue sudden panic when the head of Autodesk UK decided to visit to see how their gift was being used. Which is how I ended up with a 386 and an A2 pen-plotter almost overnight. Of course, in the intervening 6 months my memory of the training was a little shaky. As a side issue, rendering and plotting whole very large buildings took a looooong time on the equipment mentioned - 24 hours wasn't unheard of.
"I had the opposite experience. Working for a charity,"
Same here, I do volunteer work for a charity that looks after seniors, basically as the onsite IT guy, helping seniors with their technology. They survive on donations and grants. The charity has existed for a very long time (1948 if I recall correctly). The top executive positions are elected positions, and they have a high turnover of volunteers. When I started early last year, I was given a small office that had a variety of computer hardware, most of it supplied through grants, some of it purchased, some of it I have no idea where it came from. None of the computers had been updated for years, since that was the last time they had an IT volunteer. The paid for IT support company only works on the office computer systems, not the donated freebies used for training. There is stuff in there that every one forgot about.
Often I'm asked to find low cost solutions to their IT problems, coz they just don't have the money for more expensive solutions. So far I have managed to solve all but one of their problems by either re-purposing old equipment they know about that was being unused, or finding stuff that had been hiding in a cupboard for years, but solved their problem. The one exception was their need for a Chromecast, to hook up to the projector, to demonstrate Android stuff to a bunch of people, some of which have bad eyesight and can't see the details on a small phone screen. I had initially been using my own, but eventually had them purchase one. I suspect they had one before, or was borrowing one before, as some of their office computers had the software for it. I just couldn't find the old one anywhere.
Maybe the opposite problem for me, the boss HAD paid for triple PSU redundancy on a legacy comms switch many years ago which supported a Critical National Infrastructure service. Power loading meant that the two shelf rack of cards could run on just one PSU so having three should have meant near 100% uptime.
Enter stage left engineer (err, me) who when moving some cables (the old heavy shielded RS-232 type) let one fall from height miraculously tripping all three power switches which were located in a row at the bottom of the rack. It took 10-15 mins to get the rack back online but for most customers the legacy serial connections were spread across two racks which even I didn't manage to break.
I 'fessed up to the mistake and thankfully management supported me, albeit change control processes were tightened.
I had a similar problem with an HP blade chassis. It had four redundant PSUs, and given the load, should have been able to run on any two of them. I was moving some power cables around to make our rack a little bit neater, and after double checking that all of the other PSU's were online, I pulled the power out of one of them.
The entire blade chassis died straight away, with me left standing there wondering WTF?
After some investigation we discovered that one of the PSU's in the machine was faulty, and couldn't actually sustain any load. Most of the time this wasn't an issue, because the other PSUs took all of the load, but me pulling the power to a good PSU, put load on the bad one which immediately died, taking everything else with it.
Fortunately at that job spare cash wasn't so hard to find, so we bought the full complement of six (I think) PSUs for that chassis, just so it couldn't happen again.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
