Complexity killed the IT quality of service ...

Budgets and metrics

The problem is often in IT management. Most existing helpdesks have their workload and performance measured through a simple issue-tracking application. Team performance is evaluated based on how many calls per hour they can resolve. In that environment, there is no incentive for helpdesk operatives to undertake preventative maintenance - in fact the incentive is to wait until things blow up, then keep themselves busy for hours resolving the crisis.

It's even worse for outsourced services. Your boss's boss has signed a contract with another company's boss, to provide a hosted (cloud / outsourced / managed / insert buzzword) service. When something goes wrong, you (the mere peon) call the remote helpdesk (also mere peons) and complain that it's "a bit slow". Their response is typically "oh it's just like that", or "it's always slow on Tuesdays"; when in truth they've massively underprovisioned their system. However you have zero leverage over them and even your boss doesn't take much interest. Even if he did, you're locked into a contract for years and all your data is on their system Result: lost productivity, wasted time and money.

Not that expensive or difficult anymore

I have seen many extremely complex and costly Service Assurance or wider OSS implementations as they call it in the Telco industry... sometimes totally failing the main "Assurance" point.

We have since built through experience an approach and ready to roll-out methodology and toolset (open source based.. but really integrated slashing the TCO many times) to finally get to a point where user-level and platform monitoring can anticipate those dreaded user calls (the calls not the users of course). If there's a call, as said above, it is past the point of annoyance and it is affecting the business already... it means it is way too late and burning. For web applications at least, and even legacy ones, there is no excuse not to have at least something to learn from incidents and know exactly where to look over time.

If you put a bit of pragmatic IT Service Management methodology on top, IT engineers working life finally have a chance to stop scratching their heads figuring out when those seconds of response time started to become an issue.

Rube Goldberg Was A Piker

The fundamental problem is simple: most implementations are stiched-together monsters made up of parts that should have been left at rest; and mainly not intended for their current purpose.

There is no getting around that; and many "solutions" (hung on like a bag on a kludge) only make things worse. I have seen multiple instances of "monitoring" schemes that turned out to be the performance issues they meant to prevent. Some of them even create fake transactions that then produce errors in processing; spitting messages that then need to be ignored. Of course; once you start crying wolf, no one notices when the pack really is at the lambs.

(Sometimes, programs abused by these "tools" will be hard-coded to ignore messages for certain user IDs.The IDs are then issued to real users a couple of months or years down the road. That's real fun to figure out.)

So long as we continue to build serious systems out of Tinker Toys (like IP and Web Servers) we will continue to create shambles that need constant patching and propping up; and sometime the structures will just implode or burst into flame.

all too true

These types of events happen more often than not, this is all user mentality and in many is the perception that they don't want to bother the IT people for a self righting problem. Sadly those types of problems are mearly temmors towards something bigger and its when it goes bang or breaks is when It hear about things. Not all It kit is created equal, big tin IBM mainframe type enviroments are less likely to experience such issues becasue the monitoring is more hands on and the internal hardware monitoring is levels beyond desktop PC's. Indeed rack servers still catching up on many levels in the x86 market. Not long ago I noticed a server that had this burnt solder smell to it, ans suspected something was wrong, yet all management/server diagnostics/logs showed nothing, few days later it died. In that situation the server was working fine, the logs etc showed nothing but at some level it was starting to show signs of going wrong but nothing that could be attributable to justify action at that stage beyond being mindful and keeping an eye out. Now had it had flames comming out of it or smoke then nomatter what diags/logs say and despite it running fine it would be pulled, simple as. but having a smell that I'll admit not all could smell, though another was able to descern as well *yay non-smokers*, was no justification for direct action. And this is the crux in many cases; whilst the signs are there, are they documentable enough to justify action. In many case's and the one you outline, user rings up - system going slow, then after logging lots of details, oh it seems to be fine now, case closed. That's it in a nutshell on near on everything, if its not an issues its closed off. Once closed its just a statistic and nothing of use beyond it was running slowly at this date/time, that could lead somebody to check what changed on the server back then and delay the fact the network card was failing due to a heat issue in placement that only became apparent due to extra load from the yearly accounting runs, small details like that get swallowed up from you looking at it from the IT perspective and sidetracked by some date in the past. Thats what happens when issues are closed off quickly as the problem fixed itself. It's also the case of how do you map business knowledge like the increased periodic workloads into mangamnet software that monitors hardware today. Its's hardware/software aware not business aware. The fact that the accounts department has increased for a 2 week period from 10 people to 15 people is something most servers are never told funnily enough. Sure logins added, but that tells you nothing about workloads.

Though it gets down to the real issue of most people being prevented from being proactive and those that do end up hitting a wall of others who don't see an issue now, when it happens then we can look at it. Companies as they work at the worker level end up preventing there staff from proactivly addressing problems and it gets down to the fact that they dont have a charge code for a issue that don't exist yet. As for Quality Assurance; It's such an open aspect of IT that boils down to the rebranding testing and calling that QA and thats it sadly. As for the user they just want something that works, and as far as there concerned call it and do what you like as long as it works. How do you firstly define what QA is needed for a user who outlines there requirements like that and then does the user need to outline there requirements beyond that! This then gets down to we tested it you approved/tested it and a sign off is agreed, beyond that its bugs which we know will happen as nothing is perfect. But do your accounts department budget and account for mistakes and if so if software costs 1m to build, would you seriously see them in reality budget 3m for bugs/oss issues. probably not, but what is the reality in relationship to a hardware/software projects costs to deploy and what it costs down the line overall once apart fo the business. Realy gets down to listerning to staff and paying them to speak there minds saftly without a p45 for going and speaking the truth sometimes.

Simpler solution.

Instead of designing from the client down, design from the backend up. That way the only software any specific desktop machine runs is the software required for that specific desktop.

Me, I use BSD on the servers & Slackware as a base on the desktops. Try it, you might like it.

Or, you can continue struggling with a commercial based top-down approach ... I mean, seriously, who in the fuck REALLY thinks that basing a corporate network around an OS that most users don't even touch 0.1% of the lines of code of the software installed on their desktops is a good idea? HOW many gigs for a minimal system these days? I can do the SAME EXACT office work with DOS 3.3, Wordstar & Lotus![1]

I wonder how fast the US and State budgets would be balanced if Redmond & Cupertino were banned from all state & federal systems ... Yeah, there is a learning curve, but you only have to learn it once.

[1] Don't give me "but what about Power Point?" crap ... I fire Power Point aficionados on sight ... and have done similar since the days of Ashton Tate, and I'm not talking dBase ;-)