Dear sysadmin: This is how you stay relevant Who are the sysadmins of tomorrow? Are they today's sysadmins, evolved? Or are they something new – a different breed of administrator that looks at the world differently, lacking the biases of those who built their careers hugging servers? Anything that touches even tangentially on hot-button fear topics like job security is …
This smacks of someone who doesn't actually have a clue what being a sysadmin is like but thinks they do.
Plenty of the helpdeskers, desktop technicians, and other vaguely technical staff where I've worked think they could do a better job than me or my team.
Unfortunately those same people are also the ones who think it's our fault when a product doesn't work as advertised, or some undocumented bug prevents us from using it the way they want, or don't understand when we won't do something because of the range of impact or potential risks involved in something they insist we should do.
Not coincidentally these are often recent graduates who feel they have vast technical knowledge from their wealth of education and they fully understand how things should work, not realising that due to lack of experience they have no idea how things actually work.
@AC
It is certainly the case that in some companies you get the wrong end of every stick. People of the more self-righteous type* will tell you that it's your fault if the business doesn't listen to you. And some times those people are right. Other times, it's just a bad culture or overly head-strong string pullers and you're powerless.
And, again, in some of those instances, you could possibly improve the situation with enough effort but it's not always worth it; sometimes it's just better for your sanity to find another role.
Without knowing your situation I can't really make too informed a comment but I would suggest that the best way is to put forward a well-reasoned proposal. If you perceive a problem with the current system(s) and have concrete ways to address those problems then make your recommendations. But be aware that there may be angles to this you haven't considered.
Again, I don't know your situation but are these problems you see really a problem for the business itself? (Rather than the IT team.) Will fixing it cost more than it does to manage? Are there business processes or legal/tender requirements that dictate certain configurations as necessary and/or others as forbidden?
As a sysadmin of some years experience, I can sympathise with dealing with disconnect caused by a 'problem' that I am technically able to fix but am prevented from doing so due to reasons. Some times I agree with those reasons and sometimes I don't. Either way, when this happens, I ask those who have made the decisions exactly what they would like me to tell the person I am speaking with.
And yeah, sometimes you just have to take the knocks and press on.
* - I say MORE self-righteous because deep down (and sometimes not so deep,) we're all a little self-righteous.
" IT operations staff may be deeply wedded to risk aversion, change avoidance and a perceived stability, but the businesses they supposedly serve aren't."
That's what the business say they want - agile, flexible, cheaper - right until the point service all of this DevOps/agile/on-the-fly/cheaper stuff causes service to go to shit, and then they scream they need stable, reliable, quality service - i.e. all these things Ops told them they needed to maintain happy customers and the revenue stream that this provides..
Sysadmin or developer, the advice is the same. Talk to people. Sysadmin to developer & vice versa but also talk to users, the people actually sat in front of screens, not just their managers. Get to know how the business uses IT in reality and get to know each others' concerns. That way you might be able to anticipate what the business needs, not just react to it.
Sysadmin or developer, the advice is the same. Amen, although I'd say it of everyone who wants to be successful (CIO on down in our fields). Several times a week I'd go walk-about and hang with the supervisors and especially the line animals to see how they were doing and if there are any pain points and possible requirements (wants/needs) adjustments. It had an additional benefit. Whenever the business had an new or additional requirement (want/need), I could almost always address the possible resources we'd need right off the bat simply due to knowing the people involved (especially personalities), budget, and what we can use that's on-hand. It's something you have to work at, especially if you are as socially inept as I am.
"The truth is that most organisations today are IT organisations"
I've go to disagree with that. Most organisations might have a reliance on IT by that's a far cry from being an IT organisation. Most are retail, banking, logistics etc organisations that need IT to function but they see it as a necessary evil. It isn't their core business which is why it never gets the investment or focus it needs.
Until it goes TITSUP and suddenly management discovers they are an IT business because with out IT there is no business.
PS if you don't think your business is an IT business ask yourself this question.
How long can your business last without
a) marketing
b) HR
c) management
d) facilities
e) R&D
f) IT
"The truth is that most organisations today are IT organisations"
I think the point of the argument for the above quote is that most businesses may not be selling IT for its own sake but they are selling a service that would not be able to function without IT, effectively selling an IT service to facilitate the delivery* of their end product. None of the businesses you mention "retail, banking, logistics" could function without IT, just look what happens when a banks IT goes tits up.
* By delivery I mean the wider business sense of providing a product or service, unless you are a logistics company then your deliverable is a delivery of course!! And just try to manage that without IT!!
"just look what happens when a banks IT goes tits up."
That's a poor example, banks have lots of downtime. Just look at the story about ING which was posted on the register just a few days ago. Result is that people won't be able to access their account anymore for a few hours. It may be an issue for a few people that need to get something done quick, but normally it's only a small nuisance. And card payments are automatically approved up to a certain amount if the bank servers are down, so you can still pay in the shops.
I'd personally rather have my bank down for 24 hours than my internet connection.
"The truth is that most organisations today are IT organisations"
I'll argue with the "most" there.
For many organisations, IT is simply a tool, much like a PABX switchboard or telex used to be.
In other words, their use of IT does not differentiate them from their competitors in any significant way.
"In other words, their use of IT does not differentiate them from their competitors in any significant way."
Leaving aside the obvious differentiation between the mom and pop store with an ebay shop and the one without there are other things that can make an important difference like whether they have voicemail or whether they use caller ID to call back a customer whose call they missed because they were "on the other line" or whether they implement a telephony system that allows queuing and if they do, whether they choose to tell you where you are in the queue or not. Even the way they deal with the humble telephone can affect and effect repeat business.
My local butcher got a card payment terminal recently - that differentiates them against all the local butchers who don't take card payments.
Just because they only have to worry about cooling dead pigs instead of servers, it doesn't mean they're immune to the march of the machines.
"Most are retail, banking, logistics etc "
That may be their primary business, but IT innovators in any field can maximise returns on how they deliver their product and even open new revenue streams, new business models. Organisations that stick to the old way of viewing IT will be out-paced by the innovators and will perish.
In fact, change "innovators in any field can" to "innovators in EVERY field WILL"!
It's like discussing the difference between a hand and an arm. The hand will have you believe it is a completely separate entity to the arm but in truth, the point where the hand ends and the arm begins is literally only skin deep - wrap a hand around your forearm near your elbow and wiggle the fingers of the other hand if you have any doubts.
The hand is just a specialised part of the arm. The hand would have you believe otherwise but wouldn't last long if it ceased to be part of the arm.
It's all very well saying sysadmins must be this and that, the truth is most are at the mercy of IT Directors or whoever else and don't necessarily make decisions based on business needs.
Lost count of the number of times I've heard variations of we need to move to the cloud from the higher ups, followed by OK, what do you want to migrate and what business requirements will be fulfilled by moving.
Usually there isn't a (coherent) answer, as the higher ups don't want to say we read it online and it sounded good.
<snip>
"Usually there isn't a (coherent) answer, as the higher ups don't want to say we read it online and it sounded good."
Preempt the buggers. Dump a load of encrypted data on the cloud, keep it up to date and carry on as normal. When they ask about the cloud enthuse about it , tell them that you have it and carry on as normal ignoring it.
Worked for me and the CEO was 100% behind me. I said to him that the cloud was like lending me his wife, letting me take her home and do what I wanted to her and giving him access when I felt like it. I said there is a risk that you might never see her again for various reasons but you have to accept that as part of the deal.
He thought for a few days and did not let me have either his wife or the Cloud as a solution. He almost let me have the wife for a 3 month test run but did not fancy the repercussions if I found her incompatible.
Say no to the cloud.
My cloud analogy was:
"It's like getting all your companies documents/bank accounts/secrets, putting them into a big box, running up to a stranger in the street and asking him/her to look after them (with the added bonus that you'll pay them loads of money to do it).
But your "Bosses Wife" version made me cry with laughter, so I might have to change!
"Dump a load of encrypted data on the cloud, keep it up to date and carry on as normal. When they ask about the cloud enthuse about it , tell them that you have it and carry on as normal ignoring it."
We just call our on-premise "private cloud" a "cloud". If they don't care were it's located and who is running it, it might as well be on-site. Running products with names like owncloud also helps. And since we're running VDI everything is accessible from everywhere and we have a better uptime than the public cloud. Costs are lower than the public cloud and we respond immediately when something is down. OPEX/CAPEX could be solved with lease should there be a discussion in the future.
"Usually there isn't a (coherent) answer, as the higher ups don't want to say we read it online and it sounded good."
Moreover, their "cloud research" didn't involve costs. When the financial implications start to become apparent, I find the Boss's enthusiasm [about the cloud] tends to wane - especially if you are an EU-based company with GDPR looming on the horizon, when all IT costs (especially cloud costs) will start to rise.
...then there's sysadmin burnout due to high stress levels as management does not see IT as core to their business, and the sysadmin must work round the obstacles presented by management as well as do his/her daily duties without letting core IT go down ~
~ because if it goes TITSUP, the whole company sits with no work.
Obstacles = no budget for IT, new servers, new kit - everything is purchased on an "oh buggrit, nobby nobbs broke it sarge" basis.
http://www.theglobeandmail.com/news/politics/cost-to-fix-payroll-system-could-hit-50-million-ottawa-reveals/article31755962/
" For instance, the government will pay to ensure that system maintenance takes place during evenings, overnights and weekends to improve capacity during the day" Gosh, was there someone actually signing contracts which allowed disruptive system maintenance to occur during the day?
"A long-standing trope in IT is that there is a "best way" to do it. One magical approach to infrastructure design, one software stack, one security checklist that defines the "industry standard" and determines how things "should" be done."
Anyone who thinks that is a flat out idiot and has no business touching anything that has buttons on it.
There are certain best practise principles you can apply to a given situation, some regarding the situation, some regarding the specific software you're applying, but there is *never* "one true way". There never has been, there never will be. Computers et al, are an engineering problem. That means looking at the different variables, find the best compromise between cost, effort, maintenance, tools, etc, that is appropriate for your situation.
Unfortunately, I've worked with multiple technical people who learned how to do a particular set of things one way and after that, it was set in stone. This person was a literal IT graybeard, and he refused to implement ssh instead of telnet until, oh, 2008 or so (my first ssh deployment was in 2000, and even then it was past time). You've got people who learned to set all network ports to 100/Full instead of auto because of that one problem that Cisco had, and they kept doing it into the gigabit era. The nature of learning is that it's an iterative process, but some people never revisit their "knowledge" to see if it still holds true or if what they learned is the best way to accomplish a task.
Trevor's point is ultimately about being flexible, and if there's a lesson that the commentards of El Reg can teach us all, it's that many IT practitioners (in whatever sub-discipline you care to name) believe that their way is the right way and to hell with any evidence to the contrary. All you have to do is look at the eternal, pointless pissing contests over operating systems, programming languages, methodologies, or what have you. It all basically comes down to dick-waving and people defending themselves against the merest possibility that someone else may have an actual, valid point.
It's not just IT that suffers from this - any complex function that a corporation carries out becomes the focus of managers and micro-managing policy makers thinking that there must be a simple solution/universal template for said function. And if you ever make the mistake of trying to explain just some of the degree of complexity they say things like "Just give me the idiots guide to this" - the correct response in this situation is "If you're an idiot you won't understand this"
Can put the need of others above their own. In a way I've always considered my sysadmin job to be a bit ironic, because in a way you're always busy by making yourself obsolete. To a certain extend.
Think about the things you do in day to day operations. A good practice is to have those documented so that others can take over your tasks if the need arises. Yet this also means that theoretically they could kick you out from doing that task in its entirety and have someone cheaper simply follow the procedures which you laid out.
Even so, this has always been my philosophy on sysadmin work: you're basically doing things which make yourself obsolete. Because you always strive to make things better, but making things better also often means less interaction and more automation (not always!).
Of course there is much more to this. But that's also where the danger lies: the moment when the shit hits the fan, then we're getting where things count. Now your expertise and knowledge will be put to the test, and that is something which your "procedure following John Doe" won't be able to cope with. But that part is dangerous, especially when the upper brass starts to assess risk calculations. How often could this happen? How much costs would it take for them to get "ad-hoc" help? Maybe outsourcing so you only pay when you need a sysadmin's help?
For the record: I'm also one of those old guys who despises the fact that in many companies the helpdesk (sorry: service desk) is where they park sysadmins, claiming that the job is actually the same. It is not, and any managers who claim otherwise are in my humble opinion stupid idiots. As a sysadmin I do not mind doing service desk help, absolutely not. But don't think that I can actually work on server maintenance or other specific tasks while there's a risk that I'll be interrupted at any given time. Concentration is key with some things, and that's an issue many "highly trained managers" apparently never heard of, or can't even remotely imagine.
Just my 2 cents here.
Excellent points. One other thing on the "job security" front...I have run into so many people in my career (I'm old too!) that have defended their lack of documentation as job security. That's not where it comes from. Like you mentioned, real job security comes from the fact that things run smoothly under your watch, stuff you design doesn't blow up randomly, etc.
In the end, there's very little job security anymore. Once an MBA-wielding middle manager or management consultant comes to your CIO with a spreadsheet showing you as expensive and offshoring as cheap, the decision is as good as made. It sounds defeatist, but it's true for most cases where the CIO has no clue what you do. Your job security comes from a good reputation and the ability to jump ship when this happens and land somewhere else, possibly doing something totally different. Don't think that not documenting something will save you -- I've seen offshore guys come in after people have been fired for not cooperating...they'll just systematically pull everything apart as their onboarding effort and finish that documentation!
The good thing (for now) is that companies are constantly waffling back and forth between offshoring/outsourcing and in-house IT, and not all of them are on the same cycle. Usually the MBA guy comes in, gets the CIO to sign an outsourcing agreement, things go nuts after a while, CIO gets fired, new CIO comes in and in-houses everything -- and the cycle repeats!!
"Your job security comes from a good reputation and the ability to jump ship when this happens and land somewhere else, possibly doing something totally different."
If you've got the good reputation go freelance. That way you work for a company which is totally focussed on your career.
A lot of people I've talked to are absolutely unconvinced that cloud computing, microservices, containers and all that fun stuff are going to catch on. In some industries they may be right -- there are tons of good reasons to keep local control over your data -- but the truth is that it doesn't matter. The cloud has definitely been sold to most CIOs out there. Microsoft was very smart to get Microsoft shops into it by shifting them to Office 365 with Azure AD as a baby step. The pay as you go, fire the IT guys, OpEx cost model arguments are just too loud for most organizations to hear availability, data integrity, etc. arguments over.
There are a bunch of forces at work that are causing shifts away from the traditional admin roles and responsibilities:
- Software defined everything is reducing the need for specialists in a particular vendor's proprietary hardware ecosystem
- Virtualization (as a first step) and cloud VMs (as a logical conclusion) are also hurting people whose focus is proprietary hardware -- I can't remember the last time I directly interacted with some of the VMWare hosts we have at the place I work now.
- DevOps is actually starting to become a thing. It was total startup hipster stuff a few years ago, but developers of real world applications are really starting to adopt this.
- The lines between software and hardware, developers and sysadmins are getting blurred. All good sysadmins should know at least scripting and "glue coding" but increasingly it's possible for developers to bypass the admins and deploy their own stuff.
I guess my advice for the future is to stay flexible and try to be as much of a generalist as you can. People who know one hardware stack or OS are about as valuable these days as coders who know a single JavaScript framework and can't do anything outside of it.
increasingly it's possible for developers to bypass the admins and deploy their own stuff.
Yep and the story goes something like ....
Dev: Hey Mr Sysadmin, the new-foo.bar isn't working all our clients are complaining
Sys: New-foo.bar? I thought we were running bar.baz for our clients
Dev: Yeah we did a dev-ops implementation & migration last night and now it doesn't work anymore
Sys: it's 8am ... When did you test it?
Dev: Last night, it worked fine from my desktop
Sys: Just cut back to the old system
Dev: We can't we migrated all the data in the DB the schema's aren't compatible
Sys: Ok ... where is all this running and where is the documentation
Dev: In the cloud????
Sys: Fuck
the end.
This sounds like the people who say "You never get fired for specifying IBM" (old version), or "Windows and an Intel PC" (new version).
The whole idea of a vendor is to get you locked into their own ecosystem. It doesn't matter how it is done (from the vendor or you), but it is being done. I once worked for a boss that wanted everything "DEC", even if there were better solution out there. This was back in the mid 80's where there were vendord EVERYWHERE.
The theme now is to get you hooked (addicted) to a particular vendors product (hopefully for the vendor one that provides continuing income), and lock you in. With every vendor's "cloud" a little bit different, this is quite easy, as there is NO incentive for a vendor to be compatible with any other (see lock-in, above).
As for needing "the cloud" the biggest thing to do is ask "why?" and "what will it do for me that isn't already being done?". Doing "the cloud" for its own sake, is like building palaces in North Korea. They look nice, but they don't get used, and sit empty. Hopefully that doesn't happen.
Was in a meeting recently in which one of the CI vendors was waxing poetic on the value of not having to do the rack/stack/release notes/bug reports/patches dance and providing more value to the business as a result of your Sys Admin time being freed up. One of the silo chaps (compute, I think) chimed up and said "but I like reading release notes and patch reports, it's relaxing.." There you have it - the Luddites and those who can and will re-invent themselves into something of more value the business. Which camp are you in?
The Sysadmin Directives
1. Make sure nothing under your control ever quite works: see "The Microsoft Laws of Directed Chaos".
2. Continue your air of aloof arrogance that, born from your inability to make it as a developer, you've already cultivated to perfection.
3. Obstruct, obfuscate, deliver late, patch irregularly, update inconsistently, opine constantly, argue incessantly.
4. Manage upwards, block downwards.
5. Knowing the words is far more important than knowing the technology. Facts near your brain should be treated like strong magnets near the tape backups.
6. Just say no. All the time, as long so doing as it doesn't clash with directive 4.
7. When asked about directive 7, just give your infuriating sysadmin smile (practice this with other sysadmins). Secrecy is a sword best wielded widely and often.
8. Wear your phone in a belt holster, your work pass around your neck, and your gut proudly, so other sysadmins can check you visually, give you the respect you so obviously deserve, and stay off your patch.
9. Always apply the directives. Learn them well and destroy this list.
Most of us have actually moved on from 1994, actually, and we even no longer wear belt holsters and pagers.
Yes, there are some old farts who fit your lovely stereotype of wanting to block progress of any kind, but the majority of us - by far - do not.
I was a sysadmin for 15 years, I've been a cloud architect for 2 years now, no way will I ever go back. Way more money and way more interesting.
- The leading public clouds are more secure than your Data Centre
- You can save money by migrating to cloud, if you do it right, knowing how to do it right is worh $$$
- The real advantage of cloud is agility, not just cost
- Cloud adoption takes skill and re-architecting, if you just migrate your on-prem servers you're doing it wrong
How long can your org avoid public cloud? In most cases not very long. Sysadmin skills are absolutely relevant but so are cloud skills, Syssadmin is a great place to start from. There are heaps of job opportunities for cloud skilled folk, best career move I ever made. But, if you're a Sysadmin and you want to bury your head in the sand and hope it goes away (it won't) while your peers update their skills for public cloud your job prospects will suffer.
Your choice...
'Cloud architect'!
Oh my word. That made me laugh. Do you tell your friends that? I'm imagining you standing outside in dense fog, waving your hands and impotently raging because you can't get it to go in the shapes you want it to.
Honestly, what on earth is a 'Cloud Architect' and how is that different from 'installing applications written by other people on machines owned by another set of people'.
Anyway, I'm off to the kitchen to be a 'Coffee architect'. I just can't stop smiling about this - you've really made my day!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
