back to article You, yes YOU: DevOps' people problem

You’ve no doubt heard of DevOps. This is the process of getting developers and sysadmins working together closely on the same team to support a company’s custom-written software. I know, I know, Dear Reader: you’ve been doing this ever since operating that AS/400; no one really needs weekly releases; and, of course, the …

  1. NinjasFTW

    I wish I could believe in the promise of Dev Ops.

    The problem is that the developers think that Dev Ops simply means that they are now the Operations/engineers/architects.

    Governance and strategy simply goes out the window in the rush to get the latest widget into prod. They don't care that they are building on extremely shoddy foundations and that while it works now, once you start extending the rest of the platform onto it, it's either going to come crashing down or require huge amounts of infrastructure resource to keep it limping along.

    I'm sure that at its core that's not how its meant to work but that what all the developers I work with think and management goes along with it because of the promise of rapid delivery.

    I've checked the deployment logs before and found instances of 22 deployments in a single day to a microservice! WTF!?!?

    I would be interested to hear the experiences of others (Non developers) on their experiences on how they work in Dev Ops environments.

    1. Anonymous Coward
      Anonymous Coward

      Re: I wish I could believe in the promise of Dev Ops.

      Experience so far is for most companies it's a pipe dream as you need total control over all of the development for all of your core IT applications. I mean DevOps seems more of an App by App thing. But yeah we apparently do "devops" to do one of our multiple web UIs, it's fine... Everything else... old fashioned hard work.

      Now test driven infrastructure is probably far more interesting for those of us in the real world. Who have off the shelf software, 3rd party developed bespoke software, internal developed software, software that was written by some guy some time in a shed in Korea because it was cheapest, etc. All while juggling a mix of managed service hardware, private cloud kit, public cloud kit, saas, colo and kit across global headquarters.

    2. meetra

      Re: I wish I could believe in the promise of Dev Ops.

      So, I'm going to write a little about my experience as a non-developer. I've worked in the DevOps business in 2010 building Ant scripts and using Hudson to CI and deploy applications.

      I usually separate DevOps in two things: Developer side and Infrastructure side.

      --- On the developer side ---

      This type of DevOps thing is perfect because there's a real lack on knowledge of infrastructure at the core of the typical software projects. Teams need both Application Architects and Infrastructure Architects in ALL software projects from beginning to the end.

      Examples:

      - Deployed new code on a 3-tier application:

      -- Developer: If it works on my notebook and doesn't work in production, that's not my problem;

      -- Me: Checking the logs and there was a web service declaration in web.config with URLs 127.0.0.1 or localhost (btw, it happened like 3 times a week and stopped when the App Architect told them if they do that mistake again, they had to buy a cake the next week);

      - Receiving files via web services and sometimes the file didn't show up in the App (3-tier application with 2 servers per tier with LB):

      -- Developer: I receive a file and I write to disk, if it doesn't work in production, that's not my problem;

      -- Me: Checking the AppServers, the file was in AppServer01 and not on AppServer02. When user logs and goes to AppServer02, the file doesn't exist there. Solution, another server exporting NFS to both Servers (not perfect but worked);

      Developers really don't care about infrastructure. They really don't know it, they are afraid of it, they just take it for granted and "that's not their problem". I can understand that because they have to work 12 hours/day translating business to code and the PMs whips them if they get behind schedule. That's why Docker/Vagrant/CI/CD hype is huge for them. They just type one command and done! The reality is that it doesn't solve the problems in software development.

      --- On the infrastructure side ---

      People, please! Real SysAdmins use scripting to facilitate repetitive work and Configuration Management software always existed in enterprise environments! Ever heard about PXE, TFTP, kickstart or running remote commands with SSH on multiple servers? The philosophy of automating tasks was already in the infrastructure side! Now there's puppet, docker and others. OK, cool! Changed the tools, maintained the philosophy! Just use your head and help the developers!

      Tips for infrastructure guys when dealing the developers:

      - Charge the developers infrastructure usage;

      - If the production environment has firewalls, load balancers and multiple servers per tier, just implement the same structure in DEV/QA environments (helps spot configuration errors);

      - Try to allocate minimal resources as possible (CPU/MEM/DISK) for DEV/QA environments (forces them to optimize code);

      1. Vic

        Re: I wish I could believe in the promise of Dev Ops.

        Developers really don't care about infrastructure.

        IME, many developers just don't get multi-user environments.

        So they produce code that works on their local workstation. But when it comes time to deploy to the production server, it suddenly needs to be run with sudoer privilege, because it starts a dedicated server process. On the same port as the OS-supplied server that is being used by others...

        I don't have a decent solution, but I do start with refusing to accept such things. It might be a bit more work to get the proper daemon to do the job, but that's what needs to be done. Building your own, semi-functional replacement is simply a maintenance nightmare.

        Vic.

  2. yoganmahew

    It's not me, it's you?

    The resistant to change line is trotted out time and again. Time and again it is wrong. DevOps, like every other top-down theoretical imposition is 9 parts BS, 3 parts extra paperwork and, yes, it adds up to 120% Marvellous.

    'Written for Apps' is exactly why DevOps is a crock. Many of us have a more serious view of out responsibilities, where fixes are loaded as required, not on a monthly release schedule and where one size does not fit all.

    When DevOps can present a method of managing CD on an integrated application, a method that is better than the current home-grown practices (grown based on the application, app-centric if you like), then I'll cease resistance. Until then, it's just another lowest common denominator theory - written from the simplest implementation and expected to work on the most complex.

  3. jake Silver badge

    DevOps is yet another ...

    ... excuse for otherwise useless middle management.

    TGIF

  4. dogged

    So you're a marketer.

    Okay.

    1. Anonymous Coward
      Anonymous Coward

      > So you're a marketer.

      Ever since starting my career as a programmer, and through being an industry analyst, strategist, and, now, marketer

      Yep, that's where I stopped reading. Well actually, my BS early warning system keyed on "analyst" first. A cursory skim through the article confirms it was not a false alarm.

      This article says absolutely nothing of substance regarding Devops.

  5. Brewster's Angle Grinder Silver badge

    "to be continued."

    Is about the best thing in this article. Because a sentence like:

    "A fair number of people think they’re doing continuous delivery, but when compared to the textbook definition, they’re more like dabblers, picking and choosing practices that are easiest and leaving out the rest."

    needs expanding. What's textbook CD? How are those of us dabblers who've been delivering updates since the Cretaceous getting it wrong? Why should I spend my time looking into DevOps?

    1. Pascal Monett Silver badge

      Re: Why should I spend my time looking into DevOps?

      Ideally you shouldn't. Ideally, companies should be monitoring their requirements to ensure that they are being met and that future requirements are being planned for. Whatever is this week's flavor of name for that doesn't really matter.

    2. Steve Button

      What's textbook CD?

      Well, I'd guess it's "all the things" in the book Continuous Delivery by Jez Humble. It's a little bit old now, but still has a hell of a lot of lessons that most companies could learn from. This book is pretty much the DevOps bible.

      Automated Testing. Blue/Green deployments. Canary releases.

      Can you deploy your software by running a single command (tell it which version and which environment)? Will that command also fire off a bunch of tests to tell you it does what it's supposed to? (unit tests AND full acceptance tests). After that you're good to let testers (real people) do some exploratory testing.

      Everything is in source control, right? Including your description of the infrastructure? Including the network?

      There's a few more bits, such as linting to enforce style guidlines, autoscaling, putting code into components. Security would be nice. And compliance. Oh, and backups. We want backups, right?

      Oh, and Docker. Don't forget Docker, as you've got to have that too. I've never worked out why, but you've gotta have it. ;-)

      Nothing says you HAVE to deployments every two weeks. Just you don't have to shit your pants every time you deploy!

      That's about it really.

      1. Brewster's Angle Grinder Silver badge

        Re: What's textbook CD?

        Yeah, I pretty much do all of that or it's on the roadmap. Except Docker.

        As I said, the good people do this stuff intuitively. And it is obvious that the probability of an error is O(εN) where N is the number of times a human being has to intervene. But even if that wasn't the case, I'm a dev: I transform boring tasks into programming projects, even if I'm only going to do the task once and programming it takes 100 hours more than the 10 minutes it would take to just do it.

    3. sed gawk Bronze badge

      I'm doing it "wrong" and I don't care.

      We pxe boot a host, which pulls down images over tftp.

      The host builds qemu images using debootstrap and our own boostrap (configs and repo information).

      On first boot the guest installs a meta-package with the code needed for doing it's job example corp-httpd.

      When we push updates, we rebuild the guest images.

      C/I is dumb as dirt, each job is the same, set a couple of ENV vars echoed into a text file, checkout the script repo and the "code" repo.

      Run the build script, jenkins takes care of pushing it to the next job, and eventually a packaged deb ends up in our repo.

      Is this C/D ? who cares, our devs can't deploy broken code to production, any issue is fixed with roll forward. We have real ops people, who get ops, and dev's safely in their sandpit.

  6. gumbril

    To paraphrase Donald Rumsfeld - you can have the right people, and no process and it can work, you have the right process and the wrong people, it won't. Waterfall? Agile? DevOps? You start out with highly motivated teams, working together, passionate, and guess what - good things happen. Marketing gimboids and booksellers get on board, and suddenly it's a pancea, but if you've got shit people, shit will happen. And it's not about culture, it's about competence. Management, Customer and Engineering all need it to manage entropy/debt. Christ - just get a good team, and let them choose the framework for delivery, and whatever they choose will include CI

    1. Brewster's Angle Grinder Silver badge

      Cargo Cult processes

      These fads do seem to be about making bad people behave like the good ones in the hope that it will somehow make them good.

    2. Destroy All Monsters Silver badge

      > and whatever they choose will include C

      Out. Now!

  7. Anonymous Coward
    Anonymous Coward

    Torture

    We're getting it rammed down out necks at the moment.

    It's torture. Just let me stare at a screen all day, I don't want to talk these other people...

    1. Destroy All Monsters Silver badge

      Re: Torture

      I hear Xanax helps with social anxiety disorders.

  8. Lysenko

    You’ve no doubt heard of DevOps.

    HAHAHAHAHA !!

    You jest of course. The only way someone wouldn't have heard of DevOps around here would be if it were temporarily obscured by a landslide of storage articles.

    How about a more useful analysis than this formulaic rehash. Examples:

    OCD is equally as common as ADHD. Some people want/need an app to behave *exactly* the same today as it did six months ago. Just faster. How does CD help in such cases?

    Many industries have statutory or regulatory requirements. Where does CD fit in when bugs mean deaths and gaol sentences? Closely related: how does CD fit in when user viewpoints are secondary to what HMRC (IRS) and the CAA (FAA) think?

    "Continuous Delivery" is a concept very familiar to sewage engineers. Discuss relevant parallels.

    1. Destroy All Monsters Silver badge
      Holmes

      Re: You’ve no doubt heard of DevOps.

      Where does CD fit in when bugs mean deaths and gaol sentences?

      Nothing changes. If anything, you should have HIGHER assurance that your latest bowel movement meant to control information processing actually does what it says on the tin and that all requirements are being met.

      1. Lysenko

        Re: You’ve no doubt heard of DevOps.

        So, you write your code, run your TDD suite on it, notice the time (release day!), upload the code ... to the plane ... then you get in the pilot seat (DevOps remember) and fly the thing. Good luck with that.

        You'll avoid prosecution by the CAA only because you're likely dead because you assumed a suite of highly specific test cases is a substitute for proper field testing.

        Hyperbole aside, how does the DevOps/Agile mantra (scarily close to: "Release early, release often") cope with mission critical systems? TDD is not a panacea. Neither is rewriting everything in Haskell and hand waving away the fact that a physical world full of sensors, actuators and wetware has side effects that push your TDD cases into a geometrically expanding number of combinations.

        How do you "DevOp" a process that is already test driven and still finds most bugs via three month plus field beta tests? Something like the "fast lane" of the Win10 update system? Or just our old friend: "Beta test in production"?

  9. jMcPhee

    At our outfit, IT's been on the path to serious suckage for about 20 years.

    - The sysadmin and devloper work has been farmed out to the extent that they don't know what the operations unit really needs. (Natural outcome of outsourcing and putting IT in a separate organization which joins at the CEO. It wasn't always this way)

    - The operations unit doesn't know IT. Consequently, they don't know what to ask for.

    So, most of our projects are crap. They deliver the 'scope deliverables' but don't really work that well.

  10. Bbbbit
    Thumb Down

    Is it just me?

    Or does that chap have a really pushy, invasion-of-personal space, bully tone of voice? What a rude man. "Rub some DevOps on it".

    I have some advice for him: What he needs to do is not "rub" but "shove" his DevOps and I shall let the rest of you guess where.

    This alpha-clown is as bad as that chap who was carping on about "T-shaped developers" a few weeks back.

    Stop serving me this corporate crap, El Reg; I will not eat it.

    1. Disko

      Re: Is it just me?

      nope.

      YOU! WRITER! You've no doubt heard of readers, and read some of those pesky little tidbits that are commonly called "Comments", that seem to want to drag you out of your comfy cubicle kicking and screaming, and pour gasoline over you to see if you go WOOSH and how fast you can run.

      Let me stoop down to a slightly more condescending tone, as I tell you what you already know: words are made of letters, and require proper coordination to form useful sentences. Now that your brain has exploded in a rage of annoyance, nothing. It was just a trick to get your attention. I will not really tell you anything new, meaningful, or even funny.

      I know, I know, Dear Typist: you’ve been doing this ever since writing about that AS/400; no one really needs weekly storage articles; and, of course, the favorite: “this is just the current way for lazy writers to make money.”

      Now if you let me continue I will lock the door of the conference room and allow you to listen to some Metallica covers, as played by the Westboro Short Bus Marching Band and recorded over a Japanese telephone line while I skip out and breathe in some fresh winter air to clear this smell of cheap cologne and carpet glue.

  11. kmac499

    Regular releases

    What other industry would say that every set time period we will release a new version just because our process says we should. Even Formula 1 teams don't always hit the next race deadline for new tweaks; and they are about as agile as you can get.

    I genuinely don't understand. Is all the work from request through spec, code and test, however integrated it all is, to be completed in a week? or is it just writing the code in a week? or sensibly it's a pipeline of planned changes one arriving every week?

    How would you feel if the software on you're radiotherapy machine or your holiday jet was subject to this process.

    The biggest money saving technique in IT is knowing when to pull the plug.

    1. Destroy All Monsters Silver badge
      Holmes

      Re: Regular releases

      1) That's not how it works

      2) There is confusion between agile delivery of the scrum sort and "dev ops"

      3) This is not how agile is supposed to be done; it'a about having feeback on a pipeline of short actions (as opposed to a long stovepipe of large actions); it is not about ramming half-processed stuff into operations at fixed intervals. Plus building the artifacts that keep the mutating product in the "requirements funnel".

  12. LDS Silver badge

    You will find true resistance...

    .... as soon as you show management the costs of implementing devops - hardware, software, the number of people required and their skills - including what they expect to be paid for those skills.

    How many truly afford the cost of full test rigs capable to simulate really the production system load? How many will pay the cost for new software, and the needed training to exploit it adequately, and to orchestrate everything? How many will be ok with the many hours you'll charge for writing *tests*?

    Recently a well known consulting company was requested to analyze our division, to improve business... well, it advised to incentivate PMs with bonuses if products are delivered earlier than estimated - guess what it will mean? More/better hardware, software, tools, training? More skilled people? Believe me, it will end up in taking shortcuts: "do you really need that much to test it? Can't we do it in less time and spending less? [so I can reap my bonus, eh eh!])" - or - "We'll outsource this and that to company X that will do it in less time and a cheaper price!"

    1. Brewster's Angle Grinder Silver badge

      Re: You will find true resistance...

      "We'll outsource this and that to company X that will do it in less time and a cheaper price!"

      Well, at least you'll be able to savour the schadenfreude of managers failing to receive their bonuses.

  13. cschneid

    Creating good software is more like inventing Lego all over again, each time.

    Not if you want it to work reliably.

    [...] metaphoric “innovation factories.”

    That doesn't even make sense. Visions of management shouting, "Have an idea!"

    1. Anonymous Coward
      Anonymous Coward

      @cschneid

      > Creating good software is more like inventing Lego all over again, each time.

      On the other hand reinventing the wheel is what we do best in software engineering...

  14. G Watty What?

    How many more of these articles must I read before I learn my lesson.

    Don't get me wrong, Agile, DevOps, CD, CI, automate this, measure that. I get it. Useful stuff, it has a place, great stuff.

    BUT, if I have to wade through one more article on The Register that is just some polished marketing techno-crapola that then ends in a shameless plug then I am going to need to find a new news provider.

    Less of this sort of thing!

    Not one to bring a problem without a solution. Get Dabbsy to write more, he wouldn't put up with this turd. In fact get him to write about DevOps, then we can all laugh together.

  15. Stretch

    My company adopted this

    Productivity has dropped through the floor. Releases now take months instead of minutes. Basically some muppets heard some buzzwords written by muppets similar to author of this article, ruined everything and proclaimed it better.

  16. Anonymous Coward
    Anonymous Coward

    Well, yes it's a people problem

    To quote from the article:

    "Creating good software is more like inventing Lego all over again, each time. Fostering that kind of continuous learning requires putting the process in place"

    And the problem is that there is a limited pool of people who really can do "continuous learning". The best of them stay in academia. If they get into industry, the change-lovers are forever wanting to "improve things" which can make life a misery for those who don't have the same "continuous learning" ability. And yet it is the plodders who get things _finished_ & will happily maintain systems that to a bleeding edger are just asking to be rewritten.

    As a bleeding edger myself, continuous tinkering (oops, I mean devops) is my natural mode of working but most people I've worked with over the years are _not_ wired that way. And forcing them into that mould will cause more chaos than a traditional "chuck it over the wall" IT structure.

    1. Brewster's Angle Grinder Silver badge

      So it's this scene from Jabberwocky?

      (It's the moment where Michael Palin encounters a pair of blacksmiths nailing rivets into a piece armour. Palin peers at them and says, "Excuse me, I couldn't help noticing you'd improve your efficiency if you moved your box to here." Palin then moves the box of rivets, the blacksmith putting in the rivets get his hand hit and his reaction leads to the whole tent collapsing.)

    2. Vic

      Re: Well, yes it's a people problem

      As a bleeding edger myself, continuous tinkering (oops, I mean devops) is my natural mode of working but most people I've worked with over the years are _not_ wired that way

      IME, most people are actually quite amenable to change. What they want is to be consulted about it first; the coal-face crew are the ones with knowledge about how their life works, and the last thing they want is for some seagull to dump change upon them. But a quick chat can mean you produce something that fits your idea of improvement as well as theirs...

      Vic.

      1. Anonymous Coward
        Anonymous Coward

        Re: Well, yes it's a people problem

        Vic, I absolutely agree with what you say about unwanted change from above vs. working with users/customers to find out what will work for them. "chuck it over the wall" is never right when it comes to the relationship with users!

        What I meant was that most ops people I've known could not have been developers, and a lot of the developers would not understand the world of ops. So, while it's a good thing to bridge that divide it's not everyone who can do it.

        "original AC"

        (don't get me started about managers, the classic was a newly hired manager in a different team who drifted into geek central to ask the question "so what is the difference between a flat file and a database?").

        1. Disko

          Re: Well, yes it's a

          PHB problem

          FTFY

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019