You, yes YOU: DevOps' people problem

Re: I wish I could believe in the promise of Dev Ops.

Experience so far is for most companies it's a pipe dream as you need total control over all of the development for all of your core IT applications. I mean DevOps seems more of an App by App thing. But yeah we apparently do "devops" to do one of our multiple web UIs, it's fine... Everything else... old fashioned hard work.

Now test driven infrastructure is probably far more interesting for those of us in the real world. Who have off the shelf software, 3rd party developed bespoke software, internal developed software, software that was written by some guy some time in a shed in Korea because it was cheapest, etc. All while juggling a mix of managed service hardware, private cloud kit, public cloud kit, saas, colo and kit across global headquarters.

Re: I wish I could believe in the promise of Dev Ops.

So, I'm going to write a little about my experience as a non-developer. I've worked in the DevOps business in 2010 building Ant scripts and using Hudson to CI and deploy applications.

I usually separate DevOps in two things: Developer side and Infrastructure side.

--- On the developer side ---

This type of DevOps thing is perfect because there's a real lack on knowledge of infrastructure at the core of the typical software projects. Teams need both Application Architects and Infrastructure Architects in ALL software projects from beginning to the end.

Examples:

- Deployed new code on a 3-tier application:

-- Developer: If it works on my notebook and doesn't work in production, that's not my problem;

-- Me: Checking the logs and there was a web service declaration in web.config with URLs 127.0.0.1 or localhost (btw, it happened like 3 times a week and stopped when the App Architect told them if they do that mistake again, they had to buy a cake the next week);

- Receiving files via web services and sometimes the file didn't show up in the App (3-tier application with 2 servers per tier with LB):

-- Developer: I receive a file and I write to disk, if it doesn't work in production, that's not my problem;

-- Me: Checking the AppServers, the file was in AppServer01 and not on AppServer02. When user logs and goes to AppServer02, the file doesn't exist there. Solution, another server exporting NFS to both Servers (not perfect but worked);

Developers really don't care about infrastructure. They really don't know it, they are afraid of it, they just take it for granted and "that's not their problem". I can understand that because they have to work 12 hours/day translating business to code and the PMs whips them if they get behind schedule. That's why Docker/Vagrant/CI/CD hype is huge for them. They just type one command and done! The reality is that it doesn't solve the problems in software development.

--- On the infrastructure side ---

People, please! Real SysAdmins use scripting to facilitate repetitive work and Configuration Management software always existed in enterprise environments! Ever heard about PXE, TFTP, kickstart or running remote commands with SSH on multiple servers? The philosophy of automating tasks was already in the infrastructure side! Now there's puppet, docker and others. OK, cool! Changed the tools, maintained the philosophy! Just use your head and help the developers!

Tips for infrastructure guys when dealing the developers:

- Charge the developers infrastructure usage;

- If the production environment has firewalls, load balancers and multiple servers per tier, just implement the same structure in DEV/QA environments (helps spot configuration errors);

- Try to allocate minimal resources as possible (CPU/MEM/DISK) for DEV/QA environments (forces them to optimize code);

> So you're a marketer.

Ever since starting my career as a programmer, and through being an industry analyst, strategist, and, now, marketer

Yep, that's where I stopped reading. Well actually, my BS early warning system keyed on "analyst" first. A cursory skim through the article confirms it was not a false alarm.

This article says absolutely nothing of substance regarding Devops.

Re: Why should I spend my time looking into DevOps?

Ideally you shouldn't. Ideally, companies should be monitoring their requirements to ensure that they are being met and that future requirements are being planned for. Whatever is this week's flavor of name for that doesn't really matter.

What's textbook CD?

Well, I'd guess it's "all the things" in the book Continuous Delivery by Jez Humble. It's a little bit old now, but still has a hell of a lot of lessons that most companies could learn from. This book is pretty much the DevOps bible.

Automated Testing. Blue/Green deployments. Canary releases.

Can you deploy your software by running a single command (tell it which version and which environment)? Will that command also fire off a bunch of tests to tell you it does what it's supposed to? (unit tests AND full acceptance tests). After that you're good to let testers (real people) do some exploratory testing.

Everything is in source control, right? Including your description of the infrastructure? Including the network?

There's a few more bits, such as linting to enforce style guidlines, autoscaling, putting code into components. Security would be nice. And compliance. Oh, and backups. We want backups, right?

Oh, and Docker. Don't forget Docker, as you've got to have that too. I've never worked out why, but you've gotta have it. ;-)

Nothing says you HAVE to deployments every two weeks. Just you don't have to shit your pants every time you deploy!

That's about it really.

I'm doing it "wrong" and I don't care.

We pxe boot a host, which pulls down images over tftp.

The host builds qemu images using debootstrap and our own boostrap (configs and repo information).

On first boot the guest installs a meta-package with the code needed for doing it's job example corp-httpd.

When we push updates, we rebuild the guest images.

C/I is dumb as dirt, each job is the same, set a couple of ENV vars echoed into a text file, checkout the script repo and the "code" repo.

Run the build script, jenkins takes care of pushing it to the next job, and eventually a packaged deb ends up in our repo.

Is this C/D ? who cares, our devs can't deploy broken code to production, any issue is fixed with roll forward. We have real ops people, who get ops, and dev's safely in their sandpit.

Cargo Cult processes

These fads do seem to be about making bad people behave like the good ones in the hope that it will somehow make them good.

> and whatever they choose will include C

Out. Now!

Re: Torture

I hear Xanax helps with social anxiety disorders.

Re: Is it just me?

nope.

YOU! WRITER! You've no doubt heard of readers, and read some of those pesky little tidbits that are commonly called "Comments", that seem to want to drag you out of your comfy cubicle kicking and screaming, and pour gasoline over you to see if you go WOOSH and how fast you can run.

Let me stoop down to a slightly more condescending tone, as I tell you what you already know: words are made of letters, and require proper coordination to form useful sentences. Now that your brain has exploded in a rage of annoyance, nothing. It was just a trick to get your attention. I will not really tell you anything new, meaningful, or even funny.

I know, I know, Dear Typist: you’ve been doing this ever since writing about that AS/400; no one really needs weekly storage articles; and, of course, the favorite: “this is just the current way for lazy writers to make money.”

Now if you let me continue I will lock the door of the conference room and allow you to listen to some Metallica covers, as played by the Westboro Short Bus Marching Band and recorded over a Japanese telephone line while I skip out and breathe in some fresh winter air to clear this smell of cheap cologne and carpet glue.

Re: You will find true resistance...

"We'll outsource this and that to company X that will do it in less time and a cheaper price!"

Well, at least you'll be able to savour the schadenfreude of managers failing to receive their bonuses.

@cschneid

> Creating good software is more like inventing Lego all over again, each time.

On the other hand reinventing the wheel is what we do best in software engineering...

So it's this scene from Jabberwocky?

(It's the moment where Michael Palin encounters a pair of blacksmiths nailing rivets into a piece armour. Palin peers at them and says, "Excuse me, I couldn't help noticing you'd improve your efficiency if you moved your box to here." Palin then moves the box of rivets, the blacksmith putting in the rivets get his hand hit and his reaction leads to the whole tent collapsing.)

Re: Well, yes it's a people problem

As a bleeding edger myself, continuous tinkering (oops, I mean devops) is my natural mode of working but most people I've worked with over the years are _not_ wired that way

IME, most people are actually quite amenable to change. What they want is to be consulted about it first; the coal-face crew are the ones with knowledge about how their life works, and the last thing they want is for some seagull to dump change upon them. But a quick chat can mean you produce something that fits your idea of improvement as well as theirs...

Vic.