Your pointy-haired boss 'bought a cloud' with his credit card. Now what? Shadow IT strikes fear into the hearts of many businesses. Unfortunately, most businesses fear shadow IT for all the wrong reasons. It is easy to have a discussion about Shadow IT with different areas of the business by talking about risks that affect them directly. Legal can be made afraid by bringing forth the bogeymen of …
In my workplace, everyone had a clause added to their contract that if they move customer data off official IT systems, they take full criminal and civil liability for protecting that data. There is even a provision where the company can sue said employee if their negligence causes the company to be sued. We bought some very high-priced lawyers that specialize in InfoSec (their most junior lawyer was top 2% of their class at Columbia Law).
These contracts have even held up in Federal Court after a a high-level exec hosted some data on his own systems and the systems got hacked. The company settled out of court with the resultant class-action by affected customers, after which the company sued the guy and the settlement amount out of him plus court and lawyer fees. The guy was senior enough that he had tens of millions in stock, all of which was sold to pay back the company, as was his fancy house, his car, and the yacht. Not that he needed any of those things as he was thrown in prison by the FTC / SEC for gross mishandling of PCI and SOx data.
No one in the company has so much plugged external storage into their machine since the lawsuit, let alone copied anything to them (we implemented some pretty heavy-duty device management software since then).
Strange thing is, DevOps is always the way me and my teams rolled. Not to go all virtuous and all, just that there wasn't any dividing line for the sys-admin side on through all the other facets of IT including development of new capabilities. Different time.
Unfortunately that means that you fail at even the bare basics of info sec. There are times when DevOps make sense, but if your company is large enough to employ more than one or two "IT guy(s)" that is not the case. As a consequence you consistently fail to deliver on even the most basic level of infosec, simply by not separating development from operations.
Quite often, the PHB has been trying to accomplish something for a decade or more. Often this is something that should be easy. Their IT department have been blocking them for one reason or another (usually a lack of time brought on by doing the rest of IT badly), and now finally someone has given them a way to bypass the awkward negative people in IT and just get the job done. One swipe of the credit card later, and they have achieved everything they always wanted, and didn't need IT to help them. The reality is that most cloud computing is better protected from a security and regulatory standpoint than almost any internal IT I've seen over the years, so it's not the end of the world. The main issue here is that IT are being cut out of the loop because they have been unable to achieve the promises our industry has been making.
I'm sure I'll get down voted for this post, even though deep down you all probably know the above is true from the perspective of management.
I'm certainly not going to downvote you because you're at least half right. It's a thing that happens. Sometimes tho, it's not that IT is doing things badly, it's that they're not successfully explaining why they won't do things at all.
Sometimes the reason this happens is that whatever the PHB is trying to get done is stupid and dangerous, IT have said "Not in a million years and here's precisely why", the PHB hasn't understood the "here's precisely why" bit, and then spends ages trying to circumvent the process instead of rethinking the original plan.
Our industry has a bit of a history of promising a pet bear. Sure, it sounds cool, and everyone is going to be impressed if you have one, but sooner or later it's going to bite your face off. Most PHB's don't have the technical understanding to realize that, and most IT staff aren't good at explaining complex problems to people who don't have a technical background.
The reason Dunhumby are a thing is because Tesco IT said it'd take 2 years and £10m.
Dunhumby knocked up a working prototype in a couple of months for less than £100k and shock-horror, bosses jumped all over it.
Anon, because my employer has also pilfered a handful of contracts from companies where internal IT said it'd cost far more than it needed to and fucked the client department around so much that they "lost" what should have been an internal inter-departmental contract and instead got outsourced to us instead of a department just shuffling some wooden dollars into IT's coffers.
Same employer had a habit of going shadow at previous employers - namely when he was in Web Dev and IT insisted that code for the website be delivered to them to upload to the webservers (including the Dev servers FFS) instead of allowing the web team to actually have access to their own dev servers for their own development work. The mind boggles. That place also ended up with his boss buying some external hosting as "office supplies" on the department card, because IT were so resistant to anyone else being allowed to touch anything computer-related, even on the occasions it was actually necessary for their work.
The role of IT is to provide a safe and secure environment for the business to operate in. If the answer to a query about a business requirement is "no", it is generally the job of IT to present an alternative solution - just saying no is interfering with a business process (unless the question is a genuinely stupid/illegal one, in which case a sensible conversation needs to be had to show the asker the error of their ways). I love the idea above of Strategic IT providing stable management, with tactical IT providing the slightly more "edgy" response to immediate business requirements which a stodgy, slow-moving centralised group simply couldn't shift to.
it's going to bite your face off.
We're faced with a multitude of PHB's that don't seem to understand something as plain as this. Or maybe they stop listening when we say "No" because they think someone is challenging their authority. Either way it's frustrating to get asked the same question year after year and have to say, "Yes, I could do that, but we'd all get fired and sent to prison after I did, so, no, I'd rather not."
"Most PHB's don't have the technical understanding to realize that, and most IT staff aren't good at explaining complex problems to people who don't have a technical background."
No, it's more that PHB's aren't willing to listen. All they care about is, "We need X, Y, and Z--of which at least one is a Unicorn--done, yesterday--and yes, he DOES mean yesterday." The instant you say "here's why" your speech is auto-DEtranslated into something like Xhosa, meaning they never hear or understand the why of it, and it's like that everywhere so jumping ship may just mean jumping into a worse situation.
But there might be a reason for IT dragging their heels over certain things. Will a PHB actually listen to their IT staff? [EDIT - Ha! Beaten to it.]
Like the request I just had in to make the staff list on the website paginate by first letter of the surname. (1) What's the point? There's a search function anyway. (2) There's only a couple of hundred names and the webpages paginate in a rather clever way that utilises the height of the browser window to minimise scrolling and the calculations behind that, if the pages are to display where they break rather than just show a number, would have to calculate everything for every page and iterate the arrays four times over needlessly. So if it happens to break between James and Jones do you want A-Ja Jo-Z or A-J K-Z? In which case it shows the first n letters of the surname until there's a difference (and let's hope it doesn't break between A Smith and B Smith).
"Oh, but the such and such institute do that". "Yes, but I know the sysadmin at the such and such institute and they manually edit their staff pages every time someone joins or leaves rather than just lift if off the staff database, and they employ someone pretty much just to do this job. Thus their IT/web-maintenance staff is 12 people and they outsource on top of that for a department only twice the size of ours; you only employ just 1 person – me."
The problem is that having to quietly explain issues like this to PHBs over and over and over again gets bloody annoying in time; PHBs are like toddlers and have short memories. Eventually "No, and here's why" turns into "No, now bugger off and stop bothering the grown-ups!"
My current place of employment tends to suffer quite badly from this, together with management having "good ideas" which would make Baldrick blush, such are their knuckle-dragging flaws. Tricks such as re-developing some storage space to make a huge open-plan office, despite the long-held loathing of all staff anywhere for such space plans.
" (1) What's the point? There's a search function anyway. (2) There's only a couple of hundred names and the webpages paginate in a rather clever way that utilises the height of the browser window to minimise scrolling and the calculations behind that, if the pages are to display where they break rather than just show a number, would have to calculate everything for every page and iterate the arrays four times over needlessly. So if it happens to break between James and Jones do you want A-Ja Jo-Z or A-J K-Z? In which case it shows the first n letters of the surname until there's a difference (and let's hope it doesn't break between A Smith and B Smith)."
Or just do a sorted query on the driver database and use that to drive the website so the data's sorted already?
There's the problem the PHB thinks is the problem, and then there's the real problem.
"you only employ just 1 person – me."
You might want to check your ego a bit there. If you've built a system where it is difficult enough to implement any kind of dumbass sorting that a PHB is *going* to ask you for that you have to complain about it on the internet, you're doing it wrong.
@TRT I hate to say it but what you describe there is what's driving this. If your job is to code the website and they ask you to code it a certain way then your opinion is not important - the business wants it that way and they actually don't have to explain their reasoning to you. Yes, I agree that your way probably is better. Yes, it makes more sense, saves some CPU cycles etc. but is it what the business wants? NO.
IT suffers from a lot of people who feel a driving urge to do things correctly. We suffer massively when some arty or managey type decides to do things differently because we can't accept their requirement - it's not correct therefore I can't implement something inferior. Sometimes the solution is JFDI - save the argument for the big stuff. In your case, pagination won't cause any actual issue so you may as well comply before you are worked around. When PHB asks you to retain customer data indefinitely, breaking the DPA that's a business issue and you would be right to argue.
Amen to the O.P. Working somewhere right now where there are genuine business needs that IT refuse to address. Really unnecessary and whimsical things such as: ensuring all trades are accurately captured in the central trade capture system rather than having supplementary deal information help in countless downstream systems, validating deal capture to ensure contradictory information isn't being entered, ensuring there is a suitable mechanism to interrogate this central data store without needing to resort to direct database and table access thus hard-wiring systems together. At every stage the answer has been no so now the business attitude is "fuck you we'll just get it done ourselves". We already have more highly skilled developers with far deeper domain knowledge. Not to mention that for several things we have requested they have tried pushing us towards "the cloud" despite our protests that systems are far better kept in house especially as we're creating and processing up to 100GB of data per day.
Centralised IT will always be behind the curve which is why you need centralised strategic IT and business-aware tactical IT to help prototype systems and functionality without creating a record macro mess.
"Centralised IT will always be behind the curve which is why you need centralised strategic IT and business-aware tactical IT to help prototype systems and functionality without creating a record macro mess."
Ooh! I like that phrase. It encapsulates in business-bingo exactly what the problem is. Though nowhere does it say "Shadow IT" (whatever that is, it would help if there was a definition in the article) is the answer. Perhaps a better understanding by management would help.
We face similar problems where the central directorate impose change managers brought in from other organisations onto central IT who then declare that "IT provision and administration has grown organically over the years without central coordination and an overarching vision resulting in a proliferation of incompatible systems", like it's a bad thing. The "organic" response to a business need usually results in something tailored and efficient, a tactical response. What we don't need is for this to be ignored and e.g. vLANs ripped out and fully randomised DHCP imposed resulting in expensive network license servers going rogue, walled-garden data capture rigs appearing on the open intranet, VOIP systems falling over, IP-based semi-authentication systems failing.
"Working somewhere right now where there are genuine business needs that IT refuse to address. "
In our case, the IT department was asked eighteen months ago (not an exaggeration, if anything it's an underestimate) if it's possible for them to provide data connectivity for a particular group of mobile devices to the corporate network whenever they are in range of corporate wifi. As we are still waiting for an answer, we're probably going to have to move the "Band Aid" 4G solution into some form of production use. Even if they'd given us a "No", we could have formed a Plan B - but with no answer at all, we don't have authorisation to look into other options as it is assumed central IT will do it until they say they can't - but they have explicitly said they can't answer question yet, and when asked for wifi coverages maps we can't have them as they are "for IT use only".
I find it hard to criticise the existence of Shadow IT in a place where that kind of thing is allowed to happen, and when "end user" departments in the business are hiring their own IT people (because we're being told "this is not an exception, this is the way Central IT works"). Especially as I'm one of the hires ;)
That doesn't mean I support PHBs going off and doing their own thing without some professional input, as that way leads to chaos - but the "official" way of doing things is probably slightly less fun than running a marathon through molasses?.
If your start point is "current IT systems are failing to meet requirements" then yes, obviously IT will be part of the problem. Often it's more a manglement issue, with the symptoms visible in IT, but you may find similar bollocks exists in other areas of the business.
Since you're at the stage where you're solving your work issues yourself, and have your own dev team, I'm a little at a loss why this is an issue. You can do your job currently (presumably making trades) and you already have the tools for it. You'd like nicer tools, that do some of the dull but vital parts of your job, and you've employed people to build them for you. But the issue is that IT weren't prepared to build them for you?
My guess would be (assuming decision makers know the issues) is that there is a suitably slow and comprehensive update planned to solve all these issues, but telling the troops about this is a Bad Idea, and since no-ones bothered to get input from them, it's also going to have a bunch of problems, Which is why it's delayed, and other options to address it are given a "no".
My advice would be:
- document all the change requests, detailed plans and suggestions, along with examples of currently produced solutions.
- show the business case for doing things your way (follow the money etc)
- get some feedback, especially from the hostile groups. That's when you'll (hopefully) discover the real reasons why you've been getting denied
If you've really got things the way you say, then take complete ownership of the systems from IT. Including all support, running costs, and risk coverage.
Just a general comment on traders (which I presume you are). Since they are time and results focused, traders often overlook (or deliberately avoid) anything that can slow them down or stop a trade. This almost always ends up with them getting very close to the line of legality or other complete failure risk. It's also why traders usually hate Risk and Compliance, since all we (appear) to do is shit on perfectly good deals, since no trader believes* they are making bad deals. Having traders who can get around certain checks and balances has led to a number of high profile, and many low profile bankruptcys of firms that should have been rock solid.
I'd presume you where one of the good 'uns, that you're not trying anything dodgy, but it's very hard (from the IT/Risk management perspective) to prevent "tactical" IT solutions from circumventing the strategic ones.
I wouldn't downvote you tho. Even the basic details you've given indicate that a cloud based solution would either be so massivly specialised it wouldn't really count as cloud (maybe hybrid cloud), or someone is telling a pile of porkies to get what they want. Well, more porkies than usual
* or they believe they can pull themselves out of the hole before anyone notices
"....We already have more highly skilled...." Ah, the sweet smell of opportunity! Sorry, but as a contractor that is exactly what I'm looking for in a target company - resource-hiding (if you have higher skilled staff why aren't you sharing them?), knowledge hoarding, lack of tactical and strategic communication, and a breakdown in trust between the business and the IT Ops and Dev departments. And the good news (for me) is employees like you will make it even easier for me to sidle in and sell you something you probably didn't need or could have done better yourself, if you'd only had a better CIO.
All the big consultancy companies, they look out for things like different arms of the business having their own business analysts and/or project teams (guaranteed it will be because they are not sharing and working together) - it screams shadow IT and opportunities!
@Matt Bryant: "....We already have more highly skilled...." Ah, the sweet smell of opportunity! Sorry, but as a contractor that is exactly what I'm looking for in a target company
Sorry dude, buy I am a contractor and have been for the last 20 years. I have the domain knowledge and the skills which is why we now bypass the internal IT team that has zero, and I do mean zero, desire to learn the business they support. We don't share the higher skilled staff because IT are control freaks. As soon as you lend them a resource they'll totally hamstring what you can and can't do, tie them up in bullshit and nothing will get done. It's why all their good people left and we spend IT budget on what we want. Competition, it's a good thing isn't it?
"....bypass the internal IT team....." I'm sure you think you do, indeed you may actually have all the relevant skills, and be employing them in line with your company security policies, but probably not. I remember going to a big corporate in London (a highstreet name) to discuss a centralised and virtualised Windows farm (the Big Thing before "cloud" became the Big Thing). Their CIO assured me they knew everything going on in their network. I bet him a hundred quid he didn't. A quick port scan showed up over 200 unauthorised and insecure MS SQL server instances, set up without the knowledge of his IT team, many with such bumbling flaws as the admin password set to "password", and many holding customer data that was covered by the corporate's data security, retention and privacy policies. Even worse, the numpties that had set up their own database servers had tied them to their own web servers, again with awful security. One web server was also set up as a BitTorrent system! Needless to say, not only did I get my hundred quid but we got the contract and several employees were shown the door.
More often as not, in my experience, shadow IT is a massive opportunity for consultancies to come in and scare management with security and compliance blather. If, as a contractor being paid to look, I come in and find it then it will make you look very untrustworthy (or worse) to management that are often terrified of being sued or fired over privacy blunders, and myself as both more trustworthy and skilled, regardless of how good a job you think you did. If my company aims to supplant you as the trusted technical advisor in the account (which we will do if you are perceived to be the barrier to making business in the account) then don't make it easy for us by putting a shadow IT rope around your neck.
And no matter how cool you think your boss is, when push comes to shove he will probably not put his hand up and stop you getting fired, he will probably have already covered his backside. I would advise that, if you get asked to create anything "off the books", then make sure you keep a record of all the emails (print them out, do not rely on having access to the email server if you are being fired for a security issue!), and make sure you are witnessed asking "Does this comply with our security policies?"
Just too add my tuppenyworth, if you are in a position of responsibility and some internal department is blocking you from an action, there is probably a damn good reason for it. Be it IT, Legal or HR, there is usually a very sensible reason why you *can't* have it.
But those reasons *will* be ignored if the PHB wants their pet project. I've experienced more than one company wrecked by such activities, where the IT requests where denied, then denied with full explanations, then denied with full explanations from Legal as to why an expensive internal solution was essential versus cloud storage. Then it somehow managed to make it to the board, who at least knew a critical risk when they saw it.
PHB still went ahead with a "pilot", which turned out to not only be slower than our internal kit, but also managed to leak trade secrets via AWS. Lots of log ins from Russian and Chinese IPs...
At least that one has a happy ending. Some of those trade secrets where classified as national security. PHB talked his way out of getting fired, only to end up doing 4 years for espionage.
> if you are in a position of responsibility and some internal department is blocking you from an action, there is probably a damn good reason for it.
Which may be a business reason, or it may be the macdonaldisation of IT with skills dumbed down and a large amount of management & coordination of disparate groups eating up the budget, instead of more expensive techies.
Do your PoC in the cloud, but then do a proper analysis of what's required based on the results of that PoC, don't run up cloud PROD at the same time.
Just too add my tuppenyworth, if you are in a position of responsibility and some internal department is blocking you from an action, there is probably a damn good reason for it. Be it IT, Legal or HR, there is usually a very sensible reason why you *can't* have it.
No, actually there isn't. Sometimes in life you just happen to come up against obstructionist pricks in a cosseted position of power and they say no because: 1) they likely don't understand and don't want to look stupid and reveal the tenuousness of their tenure, 2) love the feeling of power they have. Believe me when I say I have experienced this nonsensical bullshit first hand. Some people are just tossers. You might want to believe that everything is done for a valid reason but I'm afraid your outlook is far too clean-room for the real world.
Of course, a lot of people now slapping themselves on the back for being elite system administrators were precisely the PHBs complained of here: they were able to buy a couple of Suns or, more recently, a little x86 server off a local budget and therefore declare UDI from the enterprise mainframe herders.
I was once privileged to be part of a DBA conference call with my clueless colleagues in a more-rural-than-my-site head office who were outraged that their upperest mostest boss had brought in outsiders to do a logical database design. "We've been trained to do that! Why bring in that shower?" they cried in unison.
"I'll hazard a guess" I said, rising to the challenge. "I'll bet it's becuase we have the reputation that by the time we've figured out how to address a user requirement it doesn't matter any more".
I was made to sit in the uncooperative corner. But it was worth it. I was championing Relational Database Technology (all caps because it was A Cause) in an effort to defeat that very perception in our user base, but had recently been told by my head of department that for the next project he wasn't at all interested in discussing speed of delivery, and that relational databases were a ( and I quote) a passing fad.
You can't argue with facts like that.
[Edit: Pretty much says what others have been saying]
"The reality is that most cloud computing is better protected from a security and regulatory standpoint than almost any internal IT I've seen over the years, so it's not the end of the world."
Well external IT suppliers need to be regulated, whereas your own IT department answers only to the needs of your business.
Adding two more external providers to deal with (ISP and Cloud) is not going to make life easier. Oh sure! In the outset, when the provider is hungry, they will bend over backwards for you. What if they can't manage rapid growth and you want/need to leave their cloud? How difficult will it be to do so? Sure! The contract states you "can leave" but that doesn't mean they have to help you. Perhaps you're own IT department can do the work, if short term thinking hasn't eviscerated it. Who has the leverage?
A cloud, used to get a business or new IT application up and running is a good idea; but it should be done with an eye to the horizon. A business needs the ability to take control and sail the vessel with their own crew.
"Well external IT suppliers need to be regulated, whereas your own IT department answers only to the needs of your business."
Isn't it more the case that external providers have a contract, and you get everything in that contract and nothing else, whereas internal IT has to respond to any and all requests at all levels of approval and sanity. Plus if IT says "x is possible" they'll be held to it, whereas if external group says "x is possible, it'll cost y"
For a comparable case, if I'm consulting for a company, and some PHB wants me to do a task he should really give to his own minions, as long as I get paid (and it's legal) I'll do it. Data entry at $200 per hour? Sure thing. Fix your shitty formatting? Why sure, just sign off here. Sure, there's a bit of a stink when they realise their management by dumping shit elsewhere doesn't work so well when they have to pay for it, but the PHB has to own up to it at some point.
If I'm working as an employee, I'd tell them to fark off and have their staff do it, as there is not a simple way to "back charge" the PHB. And I'll get in shit for taking on tasks that should be someone else's problem. So then rather than the "fly tipping" PHB being the problem, you are for not saying "no".
So a lot can depend on the decision the IT department is making is going to set a precedent, or is a one off. Or is the start in a long series of one offs...
Their IT department have been blocking them for one reason or another (usually a lack of time brought on by doing the rest of IT badly), and now finally someone has given them a way to bypass the awkward negative people in IT and just get the job done.
I know, but it IS a real problem. All these managers going off and buying IBM PC microcomputers for their employees on their own budget. Don't they realize that these toys will never be able to accomplish anything like what the mainframe does? Computer time is expensive, and has to be allocated fairly according to the company's goals.
"Quite often, the PHB has been trying to accomplish something for a decade or more."
And failing to explain exactly what they want, to furnish the same attempt at explanation more than once or to answer questions as to the little details they omitted. Not that any of these things will stop them trying to do something themselves nor from expecting someone else to sort it out a few months down the line.
The popularity of the PC is because it was used to bypass having to get it done on the Mainframe. The department could write their own program rather than get the IT department to do it. IT managed to claw that back.
I agree that people doing their own IT can cause problems. I provide email services and build servers that customers can keep their email on yet still they open a GMail account.
Well, there's a couple of reasons on that though.
As others about have (rather exhaustively) pointed out, IT usually says no because what they're being asked to do is fundamentally insecure or impossible. Moving literally all your workloads to the cloud if you make 4K movies is a recipe for disaster which is insanely obvious to IT staff but sounds hugely attractive to management ("hey, we can get them to work from home! No-one will ever have an excuse to miss a deadline again!"). Storing all your data on cheapinsecurecloud.com might save some money too, but will result in your IT staff who actually understood basic infosec training screaming in horror. Having to explain this sort of thing to supposedly intelligent people gets tiresome after the 15 millionth time.
But also, the big delay factor might just be down to overwork and lack of training in your IT department. If you have 15 IT staff and haven't bothered to train them in 10 years then yeah, they probably don't know how to cut down their own workload through modern, rapid automation systems that are built in to most modern kit. If you have 5 IT staff trying to do all the project and 3rd-line support work for 80,000 end users, then yeah, they probably have a 2 year+ waiting list on change requests. These are not IT staff's fault. These are management failures. Blaming IT staff for management failures caused by other management failures is hardly fair, especially given that most of the engineers at the coalface are pulling 12 hour days just trying to keep up with an ever-expanding workload and not getting the training they need to stay on top of the rapidly shifting techs they work with day-to-day.
Basically, if your IT department hasn't done anything about that project you've wanted to roll out for 8 years, then the answer might be that you need more IT staff (not IT managers, IT STAFF), not that you need to go find some unvetted service you don't understand to upload all your sensitive data to.
@naselus most of your assumptions about cloud are incorrect. Infosec on cloud is every bit as mature as on premise, and 4k video is fine in the cloud if you understand the full lifecycle of the data and work with the cloud rather than against. It's not always the solution, but you seem to assume its always not the solution, and to paraphrase your post "Having to explain this sort of thing to supposedly intelligent people gets tiresome after the 15 millionth time."
Funny how no-one here has mentioned the biggest problem with PHBs that I have found, is that THEY say NO, for years and years and years.
"We would like to upgrade the email system, we would like to upgrade the desktop, we would like to put in instant messaging, we would like to automate our patching and monitoring, we would like to develop a better reporting system for end users."
And every goddamned time I've been involved in such an initiative, it's been knocked back - sometimes for half a decade - by PHBs who don't want to spend money. Or who are to chickensh*t to try something new.
And then some vendor comes along with a cloud offering, which actually ends up costing MORE in the medium-long term, and suddenly the credit card comes out. And WE get the blame for management inertia and not being "agile" enough.
how can we get to the boardroom table (or cocktail cabinet or golf club) to pre-empt shadow IT, by campaigning to broaden the services IT proper offers to cover the need that spawned this issue in the first place?
The first problem with fighting battles on who provides IT Services is knowing there is actually a battle going on.
(me, I tend to provide hourly reports when things fail to my manager and his manager, even if it's 3am. That tends to get the root cause analysis focused on making sure it does not happen again.)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
