back to article 'Amazon has destroyed the unicorn factory' ... How clouds are making sysadmins extinct

The rise of the cloud is wiping out the next generation of valuable sysadmins as startups never learn about how to manage data-center gear properly, a Pasadena tech biz boss has said. The problem, according to Steve Curry – the president of managed OpenStack provider Metacloud – is that modern upstarts are going directly to …

COMMENTS

This topic is closed for new posts.
  1. Nate Amsden

    as one of those unicorns

    As you call them..

    It is pretty depressing to see how much companies have wasted on cloud services, at any scale. It's shocking to see so many clueless morons who think it's normal to be spending a half a million a month or more on cloud services. When it's quite common to have ROI of well under 12 months (some times 6 months or less) to build stuff out yourself. Of course you may have to pay people more to attract the talent. But hey if paying someone(s) more means your able to save an extra few million a year it's probably worth it (biggest "doh" face I can make).

    I remember one cloud discussion where the installation fee alone was 4x my own costs for building things outright(with tier 1 equipment and 4 hour on site support).

    But word is getting around slowly but surely what a scam most cloud services are(have yet to see one that isn't myself) from either a cost, features, or availability standpoint(or all of them combined). I talk to more and more folks who want out, or have moved out, but those don't make the news.

    I heard of one rarity of a startup in Seattle who was spending 25% of their REVENUE on cloud have since moved out(or are in the process of moving out I forget -- with a 6 month ROI going to of all things high end Cisco UCS gear not exactly bottom barrel), and the rarity bit is the CEO is apparently overly happy to talk with anyone about how terrible their experience with Amazon was. Usually companies just keep quiet and try to forget about it or something.

    Fortunately I have not had to bang my head against incompetent management over cloud in a couple years or so(having spent my own time managing crap in Amazon for about two and a half years - by far the worst experience in my professional career - and the management at Amazon cloud whom I met with personally are equally frustrating to deal with - one of them tried to get me fired once for a blog post), and not anticipating having to do that again anytime soon.

    I sort of wish I could do more, I mean I see so much fail going on but I just can't come close to even starting to address all of it. I wish I could. I wish I could help more companies from making stupid f#@$@ mistakes by going to providers like amazon.

    this image has been making it's way around recently amongst folks I know and it makes me feel good to see it.

    http://thenubbyadmin.com/wp-content/uploads/2014/01/SAY_CLOUD_AGAIN.png

    1. Nate Amsden

      Re: as one of those unicorns

      A bit more to my post, there is also a massive sink of fail when it comes to many orgs and their own hosted stuff too, so obviously running your own isn't always the best thing unless you have some good folks running it. It's almost equally astonishing to me the leverage some IT executives have at massive over spending on solutions for problems that are just obscenely over spec'd because they don't know any better. But somehow are able to get their boards of directors etc to approve their massive budgets..

      I work at smaller companies so don't see that angle of things very often at least first hand anyway. Someone I used to work with just joined up with a much bigger company and they have quite a bit of gear and somehow they have measured that their utilization of their own equipment is roughly 7% (and they have done quite a bit of virtualization apparently). At the same time they seem to be constantly "out of capacity". just poorly operated....sad to see.......situations like that repeated again and again.

      --

      The caveat to my posts I suppose are for my experience, all of the companies I have worked for over the past 14 years have been for companies that wrote their own software. In cases like that it just makes sense(pretty much always) to run your own infrastructure. Though it is still cost effective to do co-location at even moderate scale (I think even folks like Twitter use colocation extensively they are a big tenant in the QTS facility I am in). Building data centers from the ground up is really for those at truly massive scale. When I say data center I'm talking minimum a megawatt or two of power, not talking server closets or small server rooms.

      1. dan1980

        Re: as one of those unicorns

        I don't think 'the cloud' - as a rule - is a scam.

        It is a quick-start, hands-off option. Kind of like renting space in a serviced office to get going quickly and avoid all the hassle of looking for an office and getting it fitted-out and cabled-up, engaging phone companies, ISPs and insurance agents.

        1. This post has been deleted by its author

      2. I Am Spartacus
        Mushroom

        Re: as one of those unicorns

        Ahh the sweet sound of someone who feels their jobs are under threat.

        Nate, it's time to wake up to the hard cold truth of reality: unless you work for Google, Amazon, Facebook or another purely high tech company then the companies major reason for being in business is to do something, or make something, or trade something. It is not to run a server room of equipment run by a bunch of over paid technoweenies. In many cases, a vast majority of cases, outsourcing the actual computers to other people is a very sound business decision.

        Yes, it may cost several 10's of thousands of pounds per month, and yes YOU may be more effective and efficient. But YOU are one in a million. There are a lot of very average, of actually very poor, sysadmins out there. There are lot of added costs to running a server room: the maintenance, the upgrades, the HVAC, and the sheer space it takes up. Do I want a server room or do I want the space back for 20 extra traders? Hmmm. Tough choice, let me think on that one.

        Cloud services, or even better SaaS are the way that smart business are going. IT is now a commodity. Got the wrong cloud service? Not up to what you expected? Then churn. We all do this with our mobile phones. Now businesses will do it with their IT infrastructure.

        My company trades coffee, We want to trade coffee. We do it very well. So, now, we trade coffee and we are not a computer company that also has a coffee trading arm. IT staff costs are way down and profit margins are up.

        What's not to like?

        1. boltar Silver badge

          Re: as one of those unicorns

          >My company trades coffee, We want to trade coffee. We do it very well. So, now, we trade coffee >and we are not a computer company that also has a coffee trading arm. IT staff costs are way >down and profit margins are up.

          >

          >What's not to like?

          Oh I dunno, when the cloud supplier goes bust or has a major failure or the link to your provider goes down and you suddenly find you can't trade coffee anymore and all your customers go elsewhere for the day. Or week. Or permanently. All for the sake of saving a few pennies on some servers and a medium sized room. Still, if there are more mugs like you out there I'll start buying shares in cloud providers since if there's one born every day I don't see why I shouldn't get some of their money.

          1. Charles Manning

            Re: as one of those unicorns

            > Oh I dunno, when the cloud supplier goes bust or has a major failure

            So having a BOFH in the house prevents failures does it?

            A company down the road from me runs their own servers. A couple of years back the servers failed. They then found their back ups were not working properly. They lost a whole lot of source code + important business docs.

            Sure, Google or other clouds can go offline, but most of them do a better job of doing backups and provide more 9s than you can do yourself.

            > saving a few pennies

            Joking right? Look at your pay packet + server costs + leccy bill + cost of renting the space. That likely adds up to more than "a few pennies" per employee.

            For larger companies, it might make sense to run your own but for for smaller companies (most companies are small), clouding is (a) way cheaper and (b) way more reliable than they could do themselves.

            1. boltar Silver badge

              Re: as one of those unicorns

              >So having a BOFH in the house prevents failures does it?

              If they know what they're doing - yes.

              >A company down the road from me runs their own servers. A couple of years back the servers failed.

              ALL the servers failed at the same time? BS.

              >They then found their back ups were not working properly. They lost a whole lot of source code +

              >important business docs.

              If they're too incompetant to use RAID and do multilpe back ups to tape then is unlikely they'd be able to use a cloud service properly. But I'll tell you whats far more likely for a small company especially one not in a large city , and thats the network going down. Good luck using a cloud service then.

              >Joking right? Look at your pay packet + server costs + leccy bill + cost of renting the space. That >ikely adds up to more than "a few pennies" per employee.

              No I'm not joking. Cheap multiply redundant servers are commodity items and in a lot of cases cost little more than a decent desktop. If you have a company of 100 people with 100 desktops, buying 10 servers is chicken feed. And those servers can easily sit in a rack in the corner minding their own business. Once you're made the hardware investment you've got the cost of the staff but that'll be less than a decent cloud contract would cost over time.

        2. John Sanders

          Re: as one of those unicorns

          Dear coffee trader,

          You just wait until the current price-throat cutting competition dries up, and you see your fees sky-rocket.

          Then you will try to go back to a less cloudy model, but you will find out that you can not hire a sysadmin, because you can not find one.

      3. Charles Manning

        Differnt organisations need different solutions

        I can fully understand the BOFHs having a negative knee-jerk reaction to clouds. Just like electric street lighting put the gas lamp lighters out of business, this will impact the BOFH market. It won't eliminate the BOFHs though.

        For a small company (say up to 20 people) cloud is great. If you're spending $20 per person per month on clouding then that's $400 per month. That's not going to get you much BOFHery or hardware. Clouds are going to be cheaper and will do the job better.

        One small consulting company I deal with has around 15 employees. They're mostly programmers and could do all the BOFHery themselves. But they've pushed pretty much everything into the cloud.

        What they found was that the BOFHery (fixing crap, updates, backups etc) was eating 2-3 man hours per week. That was 2-3 less hours they could bill out or around $250 per week/$1k per month, not even considering the cost of hardware, storing backups, tripping over boxes etc. They replaced most of it with $200 per month of cloud, keeping just some low-maintenance stuff that is harder to push into the cloud (license servers etc).

    2. Anonymous Coward
      Anonymous Coward

      Re: as one of those unicorns

      It might not make sense for a server room or a couple racks to do it in house. However, if you don't do it there, how will you cut your teeth? I think this is what this article brings out, and it's very true.

      I talk to Java programmers from time to time and it always amazes me. It's obvious they didn't have to use a piece of software that ran on a Java VM before they learned the language. Otherwise they would vomit just knowing they were creating yet another abomination.

      Apply this to any technology including my butt. People who have not used services that are in my butt, will never have that vomit feeling when programming services to be used in my butt. Anyway, this whole thing about my butt is really out of hand. The breaking news should be: "World realises my butt is not the best place to put your marbles."

    3. sapidmolecules

      Re: as one of those unicorns (No you are not...)

      You admit yourself you work for small companies.

      Here the issue is to find skills on massively scalable systems which, by itself. is outdated on the medium term because more and more applications will rely on loosely coupled resources in the future. DCs will still host 1000s of machines but they might need less orchestration because it will be built in software

    4. Fazal Majid

      Re: as one of those unicorns

      We ran on AWS for about a year, with terrible performance and availability, and switched as soon as we could hire a top-flight sysadmin. The tipping point for where it makes more sense to build than rent is about $20K/month in cloud fees.

    5. dz-015

      Re: as one of those unicorns

      "It is pretty depressing to see how much companies have wasted on cloud services, at any scale" etc.

      Just because you've come across retarded companies making the wrong decisions and doing things stupidly, doesn't mean that running your infrastructure in the cloud is automatically wrong. It makes a lot of sense in a lot of situations for a lot of companies, especially smaller ones. When doing AWS deployments I've never experienced crazy costs like the ones you're talking about, and running in the cloud gives companies a lot more freedom and flexibility than they would have with physical hardware, minus all the hardware management hassle.

      To be a professional sysadmin you need to drop your prejudices and choose the right tool for the job. Sometimes AWS or similar is the right tool, and sometimes it isn't.

    6. (AMPC) Anonymous and mostly paranoid coward

      Re: as one of those unicorns

      Re: as one of those unicorns

      Yup,

      IMHO, there are two or maybe three ways to correctly use cloud services:

      1) as a palliative, quick fix for certain start-up data processing problems (not having enough to build and run your own data center is an obvious one). Of course, if your business goes viral, AWS and its brethren quickly become the more costly option, so remember to stay in touch with your disgruntled, unemployed but talented techie friends and associates.

      2) A properly analyzed, properly specced solution to existing data processing problems (like not having enough resources to manage demand spikes, resiliency needs or other scale-out, scale-up scenarios).

      3) A solution for rationally compressing and optimizing traditional data centers, after doing all the homework and analysis.

      All these scenarios presume that the management and IT bods (if any are involved) can do the math that will honestly and accurately assess, measure and monitor real work-flow capex and opex requirements.

      By extension, that also presumes their ability to distinguish between snake oil and offerings which can provide measurable ROI. Use of a cloud service, provider or technology does not automatically eliminate the need for sysadmins, support or other technical people in the enterprise. But these roles will certainly evolve, some may disappear, some new ones may be created. In the old days, it was called skilling up. Nowadays, pink slips seem to be the preferred solution and are the uglier side of technological disruption.

      But those who believe cloud services can cure any disease and solve any problem deserve what they get. There are many vultures waiting in the clouds ready to pick apart their plump, juicy carcasses.

      Again and again, the technology is never the real issue, it is the PEOPLE who buy, deploy and make the wrong technological decisions that cause the heartbreak.

      Cloud technologies are just tools in a box, not the Second Coming. Realizing this is the first step towards more success stories.

    7. ecofeco Silver badge

      Re: as one of those unicorns

      Thank you. I've been saying this for years.

      The cloud is not your friend.

      Have an upvote.

    8. Darkflame007

      Re: as one of those unicorns

      While cloud does has it place for some services, example if you are using an app, having your own infrastructure is a lot less cost. I estimated about 4-6 times less than running in the cloud. This is comparing a co-located rack in a Tier 4 data centre with a SQL cluster, 3 Hyper V host servers, iSCSI SAN all in a redundant setup with Cisco switches & firewalls etc.

      Unless you just buy an App you log into be very careful with putting your infrastructure in the cloud.

  2. SVV Silver badge

    I'm glad people believe sysadmin skills are becoming extinct

    As my own sysadmin skills acquired in big hosting environments with complex infrastructures will probably become a "COBOL skills circa 1999" commodity in terms of contracting rates when the deficiencies of the cloud-will-solve-everything hype become manifest and I can expect to be treated as a rare genius for my ability to write shell scripts........ well, maybe in my dreams.........

    1. Christian Berger Silver badge

      Re: I'm glad people believe sysadmin skills are becoming extinct

      Seriously, compared to what most people have as sysadmins, you _are_ a genius.

      Most sysadmins out there have never heard of rsync. They have never used a package manager or scripted something via ssh. Many probably wouldn't even find their way out of vim.

      Those are people who live in the Microsoft bubble where people believe they don't need to be able to program, and that somehow it is normal that e-mail is something complex.

      1. Trevor_Pott Gold badge

        Most sysadmins out there have never heard of rsync

        *puts fingers in ears*

        LALALALALALALALALALALALALALALALALALA

        I refuse to believe this is true.

        As for the "Microsoft bubble", well, there's robocopy, richcopy, DFSR (just off the top of my head) for file-level replication and Starwind for block-level replication (well there's lots more than Starwind, but Starwind's the only one I trust.)

        But not heard of rsync? I...just...wha?

        Beer. I need beer. I am going to go format floppy disks and drink beer while I dial up to one of the last remaining BBSes with my 9600 baud modem. Because I still can. (But for how long?)

        1. Rufus McDufus

          Re: Most sysadmins out there have never heard of rsync

          There's always cygwin, with rsync loaded. :)

          1. e^iπ+1=0

            Re: Most sysadmins out there have never heard of rsync

            Must say that if I have to use a Windows machine for mote than a very short while I install cygwin. Most of the time this works fine for me.

      2. Anonymous Coward
        FAIL

        Re: I'm glad people believe sysadmin skills are becoming extinct

        "Those are people who live in the Microsoft bubble where people believe they don't need to be able to program, and that somehow it is normal that e-mail is something complex."

        Really? Good news! I can stop using Powershell or Perl when I need to carry out bulk data transformation and complex automated processes and everything will still be possib... oh. wait.

        The primary difference between the scripting experience on Windows and the Linux World is that Linux had the better tools for years and years and years, not that it somehow didn't have to be done. If you're running SBS for 5-10 users, sure, you can live without it, otherwise Powershell is the order of the day and it's now under the bonnet of pretty much everything, everywhere in MS land. Try doing the Windows Sysadmin role in even a medium size environment without scripting, and please let me know how you get on.

        1. NogginTheNog
          Pint

          PowerShell?

          PowerShell?? Some of us have been scripting complex system-admin tasks using VBS, Perl, and even good old 'DOS batch scripts, long before this funny PS thing came along my friend...

          1. Anonymous Coward
            Anonymous Coward

            Re: PowerShell?

            "PowerShell?? Some of us have been scripting complex system-admin tasks using VBS, Perl, and even good old 'DOS batch scripts, long before this funny PS thing came along my friend..."

            I've been one of those some of us for 20 years. The point I'm making is that previously you had to use a mishmash of whatever the heck worked to get things done, VBScript, Perl, Batch and in my case the UnxUtils Win32 ports to give me stuff like sed, awk, grep, wget, etc when Perl wasn't allowable or practical.

            Powershell has massively reduced the patchwork tendency, not least because of the ability to use .NET system methods. You can go from one liners to stuff that resembles C# more than Powershell - I have to resort to Heath Robinson-esque workarounds much less these days.

            1. Keith Langmead

              Re: PowerShell?

              "I've been one of those some of us for 20 years. The point I'm making is that previously you had to use a mishmash of whatever the heck worked to get things done, VBScript, Perl, Batch and in my case the UnxUtils Win32 ports to give me stuff like sed, awk, grep, wget, etc when Perl wasn't allowable or practical."

              Oh yes, been there, done that. Batch scripts basically acting as a wrapper to call vbs scripts and Win32 ports of Unix commands. Those Win32 port files are quite possibly the most used utils I've ever found. It's so much nicer now finally having a uniform language that can do all that (I know you probably could in VBScript, but I never really got my head into it), without having to worry whether I've already copied those scripts to a server. Being able to include much better help and error trapping is a benefit as well of course, using something designed for SysAdmins rather than programmers.

            2. Paul 129
              Happy

              Re: PowerShell?

              From getting rsync working with dos PROPERLY ( ie link it in to volume shadow copy,so you can backup registry etc, vshadow.exe ) Ugggghhh what fun! And then a bit more of a code nightmare getting it to network properly. FUN FUN FUN.

              Each new version of windows had a new way of doing things, and significant changes to the capabilities, so when powershell came out yeah it'll be about as useful as vbscript, capable but you still have to dig out the old tools, and forgotten when the NEW thing comes along. So with my meddling in powershell recently, I have actually been plesantly surprised by it...

              Mind you, these days for scripting I'm finding myself going for python more and more. Just simply wrap any command line and the ease of getting stuff done well

              http://xkcd.com/353/

      3. Keith Langmead

        Re: I'm glad people believe sysadmin skills are becoming extinct

        "Those are people who live in the Microsoft bubble where people believe they don't need to be able to program, and that somehow it is normal that e-mail is something complex."

        To be fair, I imagine there are people like that on both sides of the OS divide, with people considering themselves "Linux admins" because they admin a cPanel install on a Linux box, yet have no idea how to do anything at a bash prompt.

    2. rnturn

      Re: I'm glad people believe sysadmin skills are becoming extinct

      Being asked by recruiters looking for sysadmins whether I know shell scripting reminds me of the days when recruiters looking for VMS system managers would ask me if I knew DCL. (I think my response back then was that I wouldn't let anyone that couldn't write DCL scripts even know about the SYSTEM account let alone give them the password to it.) It boggles my mind that sysadmins would /not/ know how to write shell scripts even though I've met some who could barely write one and yet had been given the responsibility of managing critical systems at former employers.

  3. This post has been deleted by a moderator

  4. Blarkon

    Sysadmins - the new buggy whip manufacturers

    That analogy isn't so much fun when we are the buggy whip manufacturers is it?

    1. P. Lee Silver badge

      Re: Sysadmins - the new buggy whip manufacturers

      I'm not sure the analogy holds. Clouds still use buggy-whips, its just someone-else wielding them.

      The point of the article was that the skills required to build the infrastructure are being sucked up by big tech and no-one outside that area is bothering to learn.

      It's kind of like introducing Windows into the DC. Sure, it might be a good point-solution, but people will end up assuming email requires some very complicated and expensive software with relational databases and proprietary protocols.

      1. Anonymous Coward
        Anonymous Coward

        Re: Sysadmins - the new buggy whip manufacturers

        Email doesn't require complicated and expensive software, sure.

        Sadly, "running Exchange" does, and when the overboss's edict comes down that he *will* be using Outlook, and it *will* be "working properly" (their term for having everyone using activesync on their iWossname and having group calendars and room booking) your options begin to get fewer, and it ends up being less trouble just to run Exchange after all.

        Then you look at Office365, and realise that can be (mostly) someone else's problem, and do you want cloud? Because that's how you get cloud.

        Please, opensourceland, make a mail server that the boss can use with Outlook and not tell the difference, that I can get running without 10 years of linux and development experience, and I will happily put the time in to learn how to drive it.

        1. launcap Silver badge
          Mushroom

          Re: Sysadmins - the new buggy whip manufacturers

          "Please, opensourceland, make a mail server that the boss can use with Outlook and not tell the difference, that I can get running without 10 years of linux and development experience, and I will happily put the time in to learn how to drive it."

          They do - it's call Zimbra. OK - so not open-source any more but it doesn't involve the use of the Windows Spawn of Satan..

      2. Anonymous Coward
        Anonymous Coward

        Re: Sysadmins - the new buggy whip manufacturers

        "The point of the article was that the skills required to build the infrastructure are being sucked up by big tech and no-one outside that area is bothering to learn."

        But why should they? I'd accept that there's many situations, even the majority, where private infrastructure is cheaper and more controlled than cloud. But I can save money and have more control by doing lots of jobs that (in both business and personal contexts) I choose to outsource. Sometimes the quality of the outsourced work is poorer than a DIY approach would offer, so I tolerate higher cost, lower control, and worse quality, because the outsourced activity is something I simply don't want to do myself. At a personal level I choose to pay somebody else to maintain my car, my lawn, and do my ironing. In a business context we outsource IT infrastructure and the all desktop activity, we outsource catering, premises security, even entire facilities operations at some sites, and so forth, again because the costs and downsides of outsourcing are (if well done) less than the benefits of having somebody else do it for us.

        This "get somebody else to do it" approach doesn't and never will fit all companies. But the situation hitherto was that you generally needed to build and operate your own infrastructure because there weren't adequate alternatives. There is now an adequate alternative, and there's no going back. As soon as the server virtualisation genie was let out of the bottle it opened the way for businesses like AWS to offer commodity grade solutions to businesses who don't want their fingers dirty.

        1. Tom 13

          @ Ledswinger

          I think the problem is that many of us have come to believe in all or nothing solutions. It's either all insourced or all outsourced and there's never any balance.

          On one level I get managers wanting to outsource the risk of data loss to someone else and blaming them when something goes wrong. For me I'd always want to be able to put my hands on the server, the disk drive, and the tape backups.

  5. Anonymous Coward
    Anonymous Coward

    Driven by bad accounting

    Servers and sysadmins might be way cheaper than cloud services, but they are counted under "fix cost", as opposed to cloud services, that are "variable cost".

    In the company I work for, "fix cut" is the newest management fad. All managers have fix cost targets in their target setting (which directly affects their bonus).

    Most management fads blow over, sooner or later, so there is still hope.

    1. Tom 13

      Re: Most management fads blow over

      Yes, but there also seems to be a never ending stream of new ones, and they seem to follow Sturgeon's law religiously.

  6. Christian Berger Silver badge

    How I learned the company I'm at was using Amazon services

    The company I currently work at has an out-sourced wiki and issue tracker. Recently we have found out that it sends it's e-mail via an Amazon e-mail service.

    Amazon advertises that service by claiming that those mails will not land in spam filters... which is a problem for people using servers in the Amazon IP-range as it's crowded with spammers.

    Well so far so bad, the funny thing is how we found out about that. E-Mail from that issue tracker... which uses Amazon to not be considered as spam, landed in the spam folder. :)

  7. Michael Habel Silver badge

    But has this to do with the Failbox360, and it RRoD exactly?

  8. R69

    It isnt just sysadmins

    ...its infrastructure architects, project managers and other roles involved in the IT project lfiecycle as well.

    Im all for using the right tools for the job - and cloud has its place - however its not the answer to every CTOs problems - and its those idiots that are turning cloud into the new outsourcing - i.e clouds the answer, whats the question, whereas it used to be (and still is) outsourcing.

    Interesting to hear that as startups reach critical mass, they are looking for autonomy over their key IP - which does reinforce my earlier point - everything has its place

  9. Unicornpiss Silver badge

    As I like to point out to people...

    If you had email on AOL in 1996, you were using "the cloud". What's next, "XXtreme Cloud"? I do not understand the fascination for dumping responsibility for data and services with another partner instead of doing them in house if you have the capacity. Is it easier? Somewhat. Is it more reliable and secure? Not likely.

    Our company inked a deal in blood with Microsoft for mail and hosting services some time ago. I have to say that my job has never been less threatened---since MS took it over, we've had more problems, slowdowns, and outages than the last 5 years put together.

    1. Tom 13

      Re: As I like to point out to people...

      The only thing I ever used AOL disks for was drink coasters.

  10. Anonymous Coward
    Anonymous Coward

    But....

    Unicorns I heart them......

  11. Caoilte

    AWS is cheaper than outsourcing your data centres to Accenture...

  12. dan1980

    What _is_ a 'sysadmin' anyway?

    I think my problem with this article is not the sentiment so much as the term used: "sysadmin".

    What does that even mean in the context of "large-scale operations at a global scale", where you have entire teams devoted to just storage, others to databases, specific applications teams (with their own internal sub-specialist), network admins, dedicated backup techs, and so on?

    Systems Administrator' is necessarily ill-defined until you specify what systems are being administered.

  13. JLH

    I don't agree

    As a sysadmin who looks after large scale HPC systems, I do not see cloud as a threat to my job.

    Whether or not the kti is actually on my premises doesn't threaten my job.

    Companies will STILL need people with a clue about networking - ie how in the heck does this magic 'data' get to and from the 'servers' - ie people who understand and can troubleshoot layer 2 and layer 3 problems.

    Still need people who have a clue abotu exactly how processes are run on a system, and about performance measurement and right sizing of systems.

    As a for instance, I just gained a 15% speedup on one of your systems just by having a chat with someone about how he was running things, and made a simple suggestion (turning off hyperthreading actually). My experience showed through there.

    1. Robert Grant

      Re: I don't agree

      Software defined networking may reduce networking jobs, for example. I'm not saying it will, just that every time a company is created to solve problems such as "how do we turn networking into a platform" then it's a threat to those who tweak it by hand.

    2. Charles Manning

      Don't be too smug

      "I do not see cloud as a threat to my job." I would.

      As the cloud displaces other sys admins, they end up in the market circling for other sys admin jobs, including YOURS.

      At the least, other sys admins will start accepting lower wages and your employer will wonder why they're paying their sysadmin 50-100% more than the median sysadmin rate. That will put pressure on your future wage increases.

      Markets are highly interconnected. When buses came along, people walked less, so shoes lasted longer, so cobblers had less work and many were driven out of business.

    3. ecofeco Silver badge

      Re: I don't agree

      Companies will STILL need people with a clue about networking - ie how in the heck does this magic 'data' get to and from the 'servers' - ie people who understand and can troubleshoot layer 2 and layer 3 problems.

      You know that we know that, but often management does not. ESPECIALLY at the executive level.

  14. JLH

    Rates of pay

    Finding people with skills in "large-scale operations at a global scale" is "really rare," he told The Reg in San Francisco on Thursday, and worried it may become almost impossible to recruit such techies.

    Well if that is the case, why aren't hosting companies shovelling money at sysadmins?

    I wish they were...

    1. ecofeco Silver badge

      Re: Rates of pay

      Because it's all bullocks, innit?

      What they mean is "cheap sys admins."

  15. hoola Bronze badge

    It is where it is sold...

    The really big issue with colud services is that they are sold to management teams by woolly suits who turn up with the trendy PowerPoint presentations.

    It all looks cool and the business appear not to care about any of the implications. It all centres round revenue rather than capital costs.

    Revenue is great, it can be fixed and they know year on year what it will be (at the outset). The costs then go up but it just gets absorbed.

    The technical teams are simply ignored as whining jobsworths. It then all comes home to roost, but the techies have all gone.

  16. squigbobble

    Every time I hear employers whining that they can't find people with the skillz...

    ...I think the same thing- "Spend some effort on training and development, you fucking lazy bastard."

    Reminds me of a local TV news piece from around the time I left school (late 90s) with some PHB of an insignificant local company moaning that the school leavers he'd employed had no idea how to use a photocopier. I think I would've punched the guy on camera if I'd been there. If you're running a haulage company you don't go whinging that kids these days aren't coming out of school with an HGV licence. Just shows that that people with a misplaced sense of entitlement that the whole world should be set up in their favour at no cost to them have existed far longer than the Millenials-bashing craze.

    1. Tom 13

      Re: Every time I hear employers whining that they can't find people with the skillz...

      I was thinking "the pay for it you cheap bastage" actually. But your point has validity too. Because after you've paid to train them, you will have to pay them or they will leave.

    2. ecofeco Silver badge

      Re: Every time I hear employers whining that they can't find people with the skillz...

      Training? That's just crazy commie talk!

  17. IT Hack

    Work for Amazon?

    Think I'd rather rip my gnads off.

    1. Anonymous Coward
      Anonymous Coward

      Amazon GENERATING sysadmins?

      My understanding (and it could be wrong) is that green people go into AWS, work very very hard, for not much money, burn out and leave taking what they learned with them.

  18. James 36

    da plan

    so I see my retirement in 20 years being contracting as infra arch /sysadmin at exorbitant rates to fund my dreams of classic sportscar racing,

    awesome

    though on a more sensible note I have been experiencing more customers just thinking that infrastructure is easy because that complexity is hidden to an extent by cloud. They fail to grasp that setting up an ipad is different to building systems across geographically diverse data centers servicing 10s or 100s of thousands of users will multiple conflicting requirements. This lack of understanding may lead to a future full of opportunity or it may not.

  19. Michael H.F. Wilkinson Silver badge
    Joke

    Beware of sysadmins bearing cattleprods

    "So you think you can get rid of me, and replace all our kit in the cloud, do you?

    Enjoy that elevator ride, mr. CFO"

    1. Paul Crawford Silver badge

      Re: Beware of sysadmins bearing cattleprods

      More likely the BOFH will arrange for the CFO to get a visitation of ghosts of other companies who bet their corporate crown jewels on SLAs that turned out to be not worth the paper they were written on. Not even soft and absorbent.

      1. Anonymous Coward
        Anonymous Coward

        Re: Beware of sysadmins bearing cattleprods

        No. The BOFH will become a cloud services provider, rent the companies datacenter with an ironclad contract requiring his company hosts it and charge more.

        With the BOFH it's always interesting to see how there is a balance between enjoying using the cattleprod-of-education vs the more-money-for-less-work.

        Classic episodes feature both I think.

  20. Rabs80

    Use the best system for the job

    I'm OS agnostic - always have been. Used to work with Andrew Fernie (post above) and he speaks sense. You use whatever system is best for the job. In my mind there are a lot of blokes out there who have cut their teeth in large organisations, used a raft of different systems but because of the size of the organisation weren't allowed to do any REAL sys admin stuff. I'm taking all menu driven sys admin.

    I cut my teeth getting down and dirty with DOS, Win3.x, Win9x, NT4, Solaris, Centos, AiX, OS/400, MacOS, etc. I grew up in the days before GUI and have always been comfortable with the command-line. However, use whatever you have at your disposal. If there is a pretty GUI tool which will do the job as well the command-line, why not use it?

    As far as the scripting thing goes... If you can automate it by whatever means then do it. I've done some really elegant stuff that I am still proud of to this day as well as some really horrible kludges where one script generated a flat file, another script read the flat file, processed it a bit and spat out another flat file. Hey, if it works with the tools that you have at hand then do it.

    My only request is that you document it with comments and user guides!

  21. dz-015

    "suffer no fools gladly – much like our own Trevor Pott"

    Sure, if by "fool" you mean "person who doesn't agree 100 percent with Potty".

    1. Trevor_Pott Gold badge

      Oh, I like people who don't agree with me. If any two men agree about everything then one of them is redundant. I don't, however, suffer fools.

      Guess which you are?

      1. dz-015

        Well done for so effectively demonstrating my original point in such a hilariously self-deluded way.

        1. Trevor_Pott Gold badge
          Pint

          The fact that I believe you, personally, are an imbecile doesn't mean I don't appreciate people who disagree with me. It proves nothing except that I believe you're an imbecile.

          If you had twelve functional brain cells to rub together you'd realize that I truly adore people who disagree with me. I love arguing. I love learning. I absorb knowledge on every topic I can for no reason other than pure enjoyment. When I'm wrong, I get a thrill out of having it proven to me, because it means I now know something I didn't before. That's happy fun times for me.

          There exist, however, people who believe wholeheartedly they are right, that I am wrong, but who cannot prove this to even the remotest degree. You are one such. Having a great big ego and demanding that I believe something different doesn't make you right. When what you assert goes against every bit of evidence I have then I am going to rapidly come to the conclusion that you're a tool, thank you, and extract my enjoyment from trolling you.

          So I return to the statement at the top of this comment: the fact that I believe you, personally, are an imbecile doesn't mean I don't appreciate people who disagree with me. It proves nothing except that I believe you're an imbecile.

          Deal with it, or don't; that's your bellyache. I have no need for your validation, though you are clearly upset that you lack mine. Cheers.

  22. dz-015

    "If all companies use the cloud, then the pool of available talent to create new infrastructure-specialist companies will shrink – and that strikes us as a bad thing."

    As far as the industry is concerned, that may well be true. As far as I personally am concerned, I'm a seasoned sysadmin currently getting plenty of business from AWS deployments and associated "devops" stuff because I stay up to date and ensure that my skill set remains dynamic. If there comes a point where the fashion changes back to physical hardware, I've a whole bunch of skills I can dust off and utilise in that area too. So whilst I feel a degree of concern about this particular issue, there are definitely pros as well as cons for those willing to keep up with the times and continually keep themselves educated and relevant.

  23. Phil W

    Hybrid

    Hybrid services are the future. Cloud services make sense for certain things, for instance the storage of non-sensitive data. Storage can be expensive, especially when you factor in replacement of failed disks or support contracts to cover them.

    There are certain things where Cloud services don't work though, in direct contrast to the above sensitive data. Anything you have to keep control of for confidentiality, data protection or other legal reasons should never be stored in a Cloud service because you can't guarantee the data won't be seen by employees of the Cloud service provider and more importantly you can't guarantee the data will remain within its country or origin as is probably required by law.

    Some of it also comes down to the attitude of your company with regards to accountability. Are the bosses/staff/users/customers happy to accept finger pointing at a Cloud provider when something is broken with a particular service. Because in some cases that may be all you can do when there's a problem.

    Where the answer is yes, then it may be viable to host those services with a Cloud provider.

    Where the answer is no, and they expect explanations of the problem and/or timescales to fix, then an in house solution is likely the way to go.

    The future skillwise I think, is knowing where the appropriate boundaries for in house and Cloud services are in relation to your businesses functions, and the ability to provision hybridisation and as seemless a user experience as possible of those in house and Cloud services.

  24. Permidion

    Sysadmin are finally free

    to work on interesting stuff instead of "losing" their time on boring repetitive task.

    with the help of some server management system, like chef or puppet, it has become a piece of cake to build new servers in the blink of an eye, which free more time to work on more interesting aspect, like architecture, explore new tools and finally be able to improve legacy stuff thats always put in last priority otherwise.

    imo sysadmin are less unicorn than wild mustang now, which is a good thing

    1. Tom 7 Silver badge

      Re: Sysadmin are finally free

      To learn a little coding to get the computer to do the 'boring repetitive task' that computers really excel at?

      Or you could if you didn't use a damned GUI for every last action.

  25. David Fetrow

    Clouds and Unicorns, parallels in recent history

    Yeah, we obviously haven't worked out exactly where clouds make sense and they don't. It's all still rapidly evolving...but I am old enough to have been through this a few times. Here are some previous experiences:

    The VAX will never replace the tasks done by PDP-10's and, even if it can, it won't do them well enough.

    A database on Unix? Ha! It doesn't have the special file formats like VMS.

    Email server on Windows? Insane!

    Move the mainframe tasks to a server farm? Ha!

    We don't need http, we have ftp already.

    Virtualization, what a scam.

    In each of the above cases, people weren't entirely wrong given the assumptions of the day but those assumptions change over time and the classic example is mistaking "less efficient" or even "not as good" or for "not what happens".

    Also, please note: in each example above there are still cases where it's true for each and every one of them.

    Clouds are doing some things well I really didn't expect to see. I have no doubt they'll be oversold, that's the way of waves in IT. We find a better hammer and we use it pound nails AND turn a few screws. Some companies get burned, there is a minor backlash, the wave becomes the new normal. Repeat.

    As for hybrid clouds: I am so convinced that's going to be the "right thing" I predict it doesn't catch on. I hope I'm wrong about that.

    1. Phil W

      Re: Clouds and Unicorns, parallels in recent history

      "As for hybrid clouds: I am so convinced that's going to be the "right thing" I predict it doesn't catch on. I hope I'm wrong about that."

      Alas you're probably right about that, despite my earlier comment. At 28 my cynicism doesn't kick in as soon as it should.

      1. Destroy All Monsters Silver badge

        Re: Clouds and Unicorns, parallels in recent history

        > At 28 my cynicism doesn't kick in as soon as it should.

        Kids these days.

  26. Javapapa

    Even the NSA hates sysadmins

    http://www.theregister.co.uk/2013/08/09/snowden_nsa_to_sack_90_per_cent_sysadmins_keith_alexander/

  27. William Boyle

    Misleading

    For small deployments, the AWS tools may be satisfactory for some applications. On the other hand, for large deployments, there are still issues that require sysadmin, NetOps, and other competencies to deal with that are NOT easily available from Amazon, et al. We have deployed about 2500 servers in the AWS cloud, and trust me when I say that sysadmins are just as needed now as ever. I should know, since I am the one that administers just a small bit of that for our performance engineering group. We have maybe 100 servers, but they still are standard (if cloud-based) Linux servers. Who is going to set up the cron jobs, remote access, load balancers, gateways, DNS servers, etc? NOT Amazon...

    1. (AMPC) Anonymous and mostly paranoid coward
      Gimp

      Re: Misleading

      Et voila.....

    2. Trixr Bronze badge

      Re: Misleading

      You get an upvote for "administering" your systems, not "administrating" them.

  28. Phil Dalbeck

    Glad to hear I'm actually getting more employable!

  29. StuartC

    I think this article is pretty much spot on. I've interviewed a lot of people over the years (technically, not as management), and it's pretty hard to find someone who can act as an escalation point for multiple dissimilar operating systems, multiple hypervisors, program managers for month-long projects involving hundreds to thousands of OSI's, and oh have you ever worked in datacenters supporting more than 10000 physical servers? We can find people all day long who supported 100 servers, or managed 20 virtualization hosts. Finding people who understand the challenges that occur when you go from 500 servers to 10000+ is much more rare. Finding someone who understands those challenges from the host through the hypervisor, storage, and network layers in my (admittedly limited) experience is nearly impossible.

    On the one hand I should be glad that people don't have to go through the struggles I've had to overcome in 15+ years of enterprise level experience. On the other hand, it can be frustrating when you're trying to explain to other 'senior' technical people why they can't trunk several thousand VLANs to every port on a L2 only switch. If you haven't saturated the CAM table as part of your growing pains, you either haven't grown enough or grown fast enough.

    AWS, Google, MS, and a few other players who can afford to customize physical servers as well as write their own operating code can run servers cheaper than other people. In order to pay for the excess operating expenses of their human capital, they do have to make a significant profit however, which is one of the reasons why some workloads end up being more expensive to run in those environments. For low intensity, small workloads however it's hard to compete in price. A good sysadmin who understands the financial side of doing business would also acknowledge that using 'cloud' providers in some cases is the best business decision. Cloud providers are a tool, just like the multiple hypervisors and hardware platforms all of you people saying the cloud is bad are already running in multiple geographical locations right?

    1. Mike Pellatt

      I'd have thought that the reason there are so few people on the job market who've gone through the pain of going from 500 servers to 10000+ is:

      a) There aren't that many organisations who've done it.

      b) For those who have done it, they recognise that keeping the skilled people who did it (and gained said skills through doing it) is crucial to their ongoing business success. So everything is done to keep them sweet.

      Until, of course, the organisation changes ownership, at which point all bets are off.

    2. Getriebe

      @StuartC

      A few days late to the party, me - but gotta say - bang on. Almost 6,000 physical spread across 3 centres and all time zones; I understand your scope.

      And the comment "A good sysadmin who understands the financial side of doing business" stands alone.

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019