We at Freeform Dynamics have been running surveys and interactive workshops with The Register’s readership since 2006 on a wide variety of topics. Looking back, we see the same themes coming up time and time again. One of these is the perpetual problem of IT staff feeling they are not able to respond quickly enough to change …
Some interesting views on what can go wrong here: http://itshambles.wordpress.com/2011/10/27/some-history-half-a-century-of-failure/ - admittedly related to software projects but still relevant.
A very funny blog (not my own), check it out.
Poor change testing
Going in to a change with no certainty (or even a reasonable prospect) that it will work. Whether that's because of lack of time to work out how to do the change and then carry it out, live - or a lack of preparation due to test/mirror systems that don't actually reflect the production environment - changes being assigned to staff who have no real clue what they're doing (but it was their turn) - or even just systems that are intrinsically untestable and have to be changed "hot". One place I was involved with made over 12,000 production changes a year. Of those 60% were to undo earlier changes, correct improperly applied changes or to add diagnostics so the support teams could work out what was going on - oh yes: then another change to fix the problem and another change after that to take the diagnostics off again.
The change manager!
Re: Don't forget
Change the management who uses CHANGE MANAGEMENT.
Reading over the history and the stupid outflow of data to people in depts who do not need to get wohless email on this crap.
The old stuff is based on DOS and converted to HTML.
It is old and converted to keep it running like putting a V6 in a FORD Model T.
How ever They keep this beast rolling is beyond me. But the sad truth is anything modern in competition agaist this software tool would cost an extra 5 million and take an extra 3 years to build then once done an extra 10 million needed to support the backend.
Sending everyone to ITIL training then not implementing anything from the framework.
My personal favorite: Complicated online Change Managment forms, where the important data (like server names and what exactly is changing) is all in a single unsearchable freeform text box.
Yeah! Cause ITIL always works??
Hey I'm ITIL qualified, it has added extra grease to my sloppy shoulders but other than that it's just more management cr*p.
Too much demarcation
Frequently the thing is designed then the security bods are told about it with the expectation that they will accept it or can just bolt security on afterwards.
It never works, and is much more expensive than getting security involved early on in the deisgn stage. "Sorry chaps, you can't do it that way" is a lot cheaper than "Sorry chaps, you can't do that."
I got one for automation
I worked once in a company that over the course of a year automated everything as much as possible, the drive to automate came from the IT department as they were overworked.
Most of what they did proved to be a success.
That gave management even more reasons to off source jobs, essentially they opened their eyes to efficiency and not requiring too many people knowing much.
A year after that I joined the company, by that time the IT dept had been decimated from 25 to 8 people.
The remaining people (including the IT manager) refused to ever mention to management anything that we did with regards to any automated way of doing anything.
I automated many database procedures/cleaning/exports/merges, and no one outside the IT dept knew anything about it. I was told recently that this is still the case. From the point of view of management the databases maintenance was something you had to invest weeks working on, had to be done manually and required a month of planning.
From that moment on I have been wondering how many people does similar things.
Re: I got one for automation
easy answer - that would be every system, network or DB administrator.
Re: I got one for automation
So they fired 25 highly efficient people since they had no use for them anymore? Brilliant !
Re: I got one for automation
And the sad fact is that the people who are made to leave are often those that understood the automation, so as soon as something changes, the automation breaks and nobody knows how to fix it, so it becomes a manual process again.
About 7 years ago, I was part of a project automating the build of servers (IBM Power 5 servers running AIX) in a server farm. Could deploy an OS image on a virtualised machine with all patches, management and security software (and some frequently used application as well if required) installed and registered in about 40 minutes from bare metal to hand it over to the application installation team. Did all the work from base packages, no Golden Image in sight. Brilliant (and also stunned IBM when they came to see what we were doing!)
Came back to the company a year and a bit later, to find that the people running the process were all low skill process monkeys who had reverted to manual processes when new machine types came along, and they did not know how to tweak the process (even though it was fully documented!).
Broke my heart!
Re: I got one for automation
I wonder what happened when everyone who knew how it worked was made redundant 4 years ago?
There is a wider issue here in that in today's environment users have very high expectations in terms of what can be delivered, how fast and for what cost. Take an old, but simple example: you want an email address, you go to hotmail.com, type in a couple of details and press submit. Job done - and it was entirely free.
Now go to your 1990's IT department and say, "right then chaps, I'd like you to get an email account for everyone". Cue wringing of hands, pursing of lips and warnings of months of work and hundreds of thousand pounds of cost.
This is becoming a much more wide-spread problem as "cloud" services move from being just email to a wide variety of enterprise apps. The business can still visit a website and click submit to get access to storage, CRM, video editing, etc etc. And they expect their internal IT department to be as responsive.
There are two principle issues here:
1) The self-service cloud solutions don't have to worry much about a lot of the issues that internal IT departments worry about: integration and security being two of the main ones.
2) IT departments generally don't (yet) have the ability to respond quickly and cheaply to new business needs.
Issue (1) is about understanding when these things matter and when they don't. Then leveraging the opportunities to use rapidly provisioned on-demand services appropriately. Issue (2) is about improving the underlying systems that provide business services and moving towards a Private, Enterprise cloud solution.
I don't really agree therefore with the thrust of the article. It isn't about process or communication for most IT departments anymore - they generally have this in fairly good shape. The issue is that business expectations are outstripping the ability of IT departments to deliver. Tooling would certainly help, but the underlying infrastructure architecture is the key issue - and this is something that typically doesn't change overnight.
Will the internal IT department ever be able to catch up, or will we need to become enablers to help the business use cloud services appropriately?
Re: It isn't about process or communication for most IT departments anymore
Then count yourself lucky to have been working in excellent jobs for the last decade. Many of the issues I see in my current job and saw in my last result from precisely this issue.
And that's before we get to the still legitimate issues you raise.
Your email example is actually an excellent one. Where I am we just moved to using GMail Cloud services for Mail, Calendaring, and all the rest of the kitchen sinks and utensils they throw into the mix. On one level, the change makes sense. They'd been planing a *nix to Exchange migration for a couple of years before somebody noticed they forgot to include sufficient storage for records retention requirements, and the backups to support it and there was no way in hell they could get the increased budget to support it. But the extensive sets of mail lists based on LDAP queries won't migrate to the new system (don't know why, I'm just a help desk monkey). And we're seeing issues with people unable to authenticate to their email accounts for 24 to 48 hours, then they magically work again. Not a large number of people, just a small percentage, say about 0.5%. Of course in an organization our size that means 1300 people without mail for at least one day last week.
Good take, just multiply by 100
Everything pointed out in the main article is true, especially in large organizations (200+ IT staff and 10 times that in users). In fact the bigger the IT org gets, the more applicable Moore's law applies to things like change (and configuration) management. It's not the number of physical or virtual devices, servers, etc. alone, it's the processes used to manage them, and more importantly the quality of communications between the "managers" (or should I say, "individual contributors").
Not that anyone who counts is listening...
we definately have two of those here
Too Much Process and Poor Communication are what cripple our efforts here.
Especially the Too Much Procees one, the Blame Game is perhaps the single largest activity of any of the management team.
Forgetting who the customer is
Yes, IT departments that think users are there only to justify IT's existence.
IT serves users not the other way around.
RE: Forgetting who the customer is
Exactly - the "Disjoints within the business" section paints the picture of IT striving to cope with unrealistic demands; in my experience the reality is usually of IT departments empire building and holding their users hostage rather than giving business users the tools they need.
Re: RE: Forgetting who the customer is
IT = problem to be worked around.
Not read or failed to understand the impact
"Sometimes business stakeholders and IT don’t talk to each other enough, and sometimes they talk but fail to communicate.
Take the simple example of a request made for something that is technically impossible. IT proposes an alternative way of dealing with the requirement that can be delivered quickly and economically, albeit with some caveats.
The proposal is approved by busy stakeholders who either don’t look at the detail or fail to understand its significance. When the change is delivered, the response is: “That’s not what we asked for.”"
That point just needs to reitterated! Yes we serve customers, but the process is a *two way* street and that point is forgotten or not understood!
Re: delivery of business tools
At my last job the customers were always complaining about the delivery of business tools. Problem was, they never wanted to PAY for them* and the first budget cut was always IT.
*In one instance someone got a special deal on an application software upgrade but failed to read the server requirements. We didn't have any servers on which it could be installed, and getting their required pretty much upgrading all the server software because that particular mix of MS Server software didn't play well together. So yeah, we got about 4 of the 6 article items in one shot.
I've never known an IT department fail because of too *little* process...
...quite the opposite - it's the only way to get anything done.
Let the government handle it. That seems the most sure-fire way of messing things up.
Overheard in the Lift one day.
"Why do we employ IT they don't bring in profits when they just cost money?"
This is sadly the way business is, unless it's a problem and you can fix it they don't want to know.
From IT Crowd:
Roy: "Define hit it off?"
Moss: "Did you continue the conversation after you fixed her problem?"
Roy: "No, and she rested a cup on my back!"
...are generally the root cause.
Too much AND too little process
Both at the same time, if you can believe it: Requests get multiple approvals and reconfirmations, time is accounted for meticulously, and things wind up misallocated and miscommunicated anyway because none of the process really addresses anything about doing the actual work in the correct manner.
Re: Too much AND too little process
Same here. It's probably more common than you'd think.
Leaving the IT group out completely
I'm in the middle of this situation right now. A complete redesign of the corporations website with new whistles and bells, plus the merging of all the business groups into one entity. Yet the marketing department holds seminars and demos of the new site to staff and the executive group without including any IT staff. So any feedback, complaints, problems, questions are now filtered through the non-tech savvy Marketing group. So now I have to translate change requests that start out, "The thingy on the page isn't looking right." Just Effing wonderful....
A solution to some of this might be.....
.... Project Managers.
IT People [techies] are good at being techies.
I work for a large telecomms provider as a PM and work with 'techies' every single day. Everything that we deliver is exactly what the customer wanted.
That is because, part of my job, is to find out what the customer wants, not how they want it done. Let the clever techies deliver the 'how' once the customer says the 'what'.
As a PM, you make sure that the requirements are fully understood, the solution is clearly defined, and, where the one is different from the other, the customer decides that they either need to increase the budget [the usual reason why there is a gap] or they accept that it is 80% of what they thought they wanted, and not 100%.
Re: A solution to some of this might be.....
Oh, that *might* be a solution to some of this, if you get a project manager who actually knows what the job is for and how to do it. And then again, maybe your company gives a couple random people the title "Project Manager" and they have no idea how to do the job. I'm... familiar with the second outcome.
Re: A solution to some of this might be Pt2
<RANT> sack instantly anyone who uses the acronym ITIL without spitting. Nothing wrong with the library itself, but it attracts the thickest, cowardly, incompetant, technically and human hostile clods. Why I don't know, but I suspect the Peter Principal concept of Advanced Confabulation is the root cause. Author is correct about silo processing of changes where techies are in remote groups with no-one keeping a complete high level picture of the IT part of the organisation. Oh CIO, wherefore art thou CIO ?
Add in multivendor support and the ITIL fiends have a field day, lots of back covering and no-one gives a stuff about the poor customer or equally fuming techs, but the process paperwork provides hours of ego stroking oportunity to lambast IT for not delivering as the change teams argue. </RANT>
A decent PM can assist mightily in reducing this. However these people are rare. The good ones are usually too well employed to be available to an organisation needing them. Still for the PHBs and CEOs who want to enjoy failures, cost over-runs and user hatred, outsource to a multinational that uses ITIL. You too can then enjoy the same quality IT as governments.
All of them
I've worked in places where you had all of those problems at the same time. Including too little process and too much process.
Re: All of them
don't you love explaining detailed technical stuff to clerks whose eyes glaze at the word volume group or zone, but want lots of step by step instructions? The love of power is the root of all IT evil.
Paris because it is about the same level of intellect in change control all too often. Where is she anyway ?
"....the handling of change requests can become either a lottery or a question of who shouts the loudes."
Usually the highest ranking staff member who shouts the loudest.
"The proposal is approved by busy stakeholders who either don’t look at the detail or fail to understand its significance. When the change is delivered, the response is: “That’s not what we asked for.”"
Bang on the head! In over 25 years in the industry, I have only worked for one organisation who were able to effectively manage change requests to the satisfaction of both parties.
"Every request, regardless of its magnitude and urgency, is forced through an intensive process of form filling followed by review and approval by every man and his dog."
Back protectors need to be "managed out" of the organisation! Never happens because it's not what you know it's who.
"Poor automation and the time spent keeping systems in sync, often manually, lead to unnecessarily high costs, slow response times and increased risk."
The usual, poor systems/procedures in place. Following "industry standards is not always the best path to follow.
I think nearly every reader can relate to at least one item raised.
It's always office politics, those who look the part but don't learn from failures (back covering).
When someone namedrops someone higher up and says "He has said this needs to be pushed through, can you just f***ing Do It!".
JFDI - it ALWAYS goes wrong...
Change management by Committee.
"After consultation with all the stakeholders we want the moon, on a stick, with marshmallow and chocolate sauce"
"Oh, and you've got a £250 budget, and it's due Tuesday Morning (on a Friday)".
Yeah thanks for that.
Re: Change management by Committee.
>> "Oh, and you've got a £250 budget, and it's due Tuesday Morning (on a Friday)"
You lucky b***ard - you get budget AND time !
Electronic Data Systems, the king of excess paperwork.
Make the issue visible,...
we had a "exception" sheet on the wall, one of those annual planners splitting months into different rows, and a different coloured "star" for each section (DBAs, Network, Developers, etc) for each time that they asked us to do something with less than the required notice or analysis (aka the JFDI board).
One day the director walked past and said "my, we are doing well, with all those stars". When we explained what it was, our boss was ordered to tell us to take it down later that day (We stuck it back up in the swipe-carded workshop). And all of a sudden, the numbers of exceptions started to reduce. Real change can be done, but it sometimes takes the "rubbing the puppies nose in its own mess" approach to get the point across to management.
A lot of people will read the article without reading the title and implement this as a check list then find a label - usually best practice . . .
A previous employer of mine starting getting a real hard-on for ITIL. It's done nothing for performance or quality. The process you pick does not matter if you have clowns running the show.
One missed off the list is having a "one size fits all" change management process. You can't really treat a comparatively minor technical change e.g. upgrading a print server in the same way as a major business process change. This is the most common situation where you end up with both too heavyweight and too light a process.
Not being able to get through to the requestors what the impact of a change will be is another big problem. I've found telling the marketing bod that the "little" change he's after will actually cost £200k and put the project back about 6 months (can't remember the exact figures, but it was that ballpark) suddenly means the "critical~" change isn't all that important any more!
Does it have a purpose, apart from to make money for companies selling ITIL training? It's just a meaningless bureaucracy to tick boxes for management to be hoodwinked into believing they know what's going on. ITIL 2012 will be 'launched' soon which will be more opportunity to sell training to companies who don't know their arses from their elbows. It's a con - and an expensive one at that.
Eliminating the CAB
In line with too little process but my current employer is considering eliminating the weekly CAB meeting because none of the managers can ever agree on anything and it turns into one loud shouting match. So rather than maintain a clear, department-wide direction we'll just eliminate any communication benefits from a robust change process. Brilliant!
Change requests or change control (AKA management)?
As I see it, the article is about Change Requests from the business (that spawn mini-projects) and have very little to do with Change Control (Change Management), i.e. putting a change into production in a controlled/managed fashion (often driven by the IT department).
Any process that confuses the two is likely to be...confused. And therefore useless. Having to justify the business benefits of applying security patches and beg every "stakeholder" + his dog to let you do the work, on top of the inflated test & implementation planning, drags out the day-to-day work so IT staff don't have time to handle actual Change Requests.
But back to the article - I can see how Change Requests would suffer from the problems listed - IF there's no program/project manager to link the ends & resources.
My most gobsmacking project experience? As the recently-joined Unix guy on a project with a clueless PM:
- Got the boxes installed & cabled up. Installed the OS. Installed monitoring & scheduling tools. Proudly announced in the project meeting that I'd done my job and the other teams could start theirs.
- Monitoring team wanted ME to raise a Change Request for them to start monitoring the boxes.
- Schedulers wanted ME to raise a Change Request and write detailed specs on what jobs there were - hmm, surely their task?
And so on, for every department - even if they'd been attending the meetings for weeks, they hadn't started their planning.
Apparently the Unix guys "always did that stuff". I told the PM it wasn't my job to write other departments documentation, but I'd help him compile a list of tasks for the project and the lead times so we could create a... "project plan", I guess you'd call it. He and the other PMs actually appreciated that help!
My boss backed me up and wanted me to help him drive the model throughout the IT function. But I got bored with trying to change the world and moved on to a company where tech teams cooperated and took responsibility for their own areas.
@tfewster: Given the capitalized IF, I'd have to say
you've been a lucky bastard too.
Many organizations can't even afford a PHB for a PM let alone one who knows how to do his job.
- Mounties always get their man: Heartbleed 'hacker', 19, CUFFED
- Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
- Samsung Galaxy S5 fingerprint scanner hacked in just 4 DAYS
- Feast your PUNY eyes on highest resolution phone display EVER
- AMD demos 'Berlin' Opteron, world's first heterogeneous system architecture server chip