A series of disorderly events

@AC

Old printers were giving up the ghost. (Some were pushing 10+ years.) We needed a new set of higher volume printers. Had the things arrived on time, the old printers would have gone away with the old network, and the new printers would have started working on Monday morning with the new network. Alas, this was not to be as the – get this – paper trays where backordered.

Yeah. Paper trays.

*sob*

Contractor. HAH.

Full time regular staff. Salaried, and I don't get overtime. There wasn’t enough gear to run both networks in parallel. We had to pull of the network change without new equipment; almost everything is virtualised, so the “new network” was largely a set of new virtual machines. Given that, I don’t actually see how you can move from one forest to another, migrating all services “one at a time.”

I would love to say I am totally in charge of making all such decisions, but unfortunately I do have to work with what I’m given. I do largely get to buy what want, but I have to justify it all; part of that justification is that all equipment be utilised 80%+ for it’s lifespan. Buying equipment just to handle the changeover then letting it sit idle would /not/ have gone over well. Most especially since we would have required somewhere around 30% of our yearly budget’s worth of gear to do it.

Sometimes, you just gotta do what you gotta do…

@The original steve

Actually, I *AM* the one who said it had to be done in a weekend. Here's the scoop:

We absolutely /had/ to have the updates done by September 1st. (This per CTO requiring updates and the fact that Sept – Dec is silly season around here.) As per above, I could see no way except to do it “all in one go.” I was not present through July, as I was in two of the other locations installing Wyse clients, recabling and prepping the locations local hardware for the changeover.

It took us the first two weeks of August to get the Domain Controllers, E-mail server, and BES/OCS/WSUS/Teamviewer manager server (yes those 4 share a VM) installed and for the new network. (I was going to have THOSE at least installed, if not configured by Doomsday.) In addition, we roughed out a template Windows 7 VM and a few template Windows XP VMs for render boxes.

We didn’t have to go ENTIRELY from scratch, but I promise you that even to get what we had prepped in advance we were scraping the bottom of the excess Virtual Server capacity barrel.

I managed to create all the user accounts prior to hitting the wall, but that’s about it. Exchange wasn’t configured, OCS was not only not properly configured…it had to be uninstalled THRICE and reinstalled specifically to get the blessed thing to cooperate. As to the rest…well…there’s more articles on that.

Suffice it to say that the actual call “hey guys, we need to do this all in one go over a weekend” was mine…however there were ZERO other choices that would have had us meet the deadlines imposed on us…

An additional comment...

I should also point out that when server upgrade time rolls around, there won’t be any side-by-side coexistence of servers. Go read everything in the article that we support, and then understand that I have to shove all of that, 4x 100Mbit fiber connections and the salaries of the two sysadmins and the bench tech into less than a quarter-mil a year. When the virtual servers in use are done with being servers, they will be removed from their chassis, given a hearty dose of TLC and placed into desktop chassis.

They will then serve an additional three years as “medium-demand physical workstations” for our Photoshop geeks. The next server replacement cycle doesn’t occur until 2012. We are using “tick-tock.” This year was desktops. Two years from now is Servers. Two years after that will be desktops again. The sole exceptions are the Photoshop geeks who get trickle-down servers as part of the server refresh cycle.

When you see me talking in my articles fairly constantly about the need to do things cheaply, (or constantly trying to find the most cost-efficient method of doing something,) this is why. It should also then carry some weight when I recommend spending money on something. The cost of a Windows Server Enterprise license can to me mean 4 years of service life from a virtual server capable of running 30 personal virtual machines.

I constantly work on the razor’s edge of what is actually possible with the hardware and software I can get my hands on. I can’t even big myself up and say that this is because I am somehow superhuman or a great sysadmin. I am just willing to work long hours to get things done. I will tell you now, honest and true, that without my partner-in-crime fellow sysadmin (and one of my very best friends) this wouldn’t be possible.

I specialize in the impossible; making a system do what it was never designed to. Pushing the limits and doing the research. I am an IT MacGuyver, but as such I am only one part of the equation. My buddy is the polar opposite. He is the living embodiment of “by the book.” He keeps me in check, goes over all the nightmare hacks and kludges I have created to put out fires and get things works. He takes my quick fix or cobbled together solution and produces reams of documentation, tests it in alternate conditions and works out something that is reproducible and far more production ready.

Together, I honestly think we make a great team for an SME environment. We do what I am constantly told by me peers simply isn’t possible. I have an enormous amount of pride in that, but I do have to say the stress gets to you. It isn’t the stress of the long hours…but the thanklessness of the job. Heroics aren’t rewarded in IT. Nobody cares that you are pulling all nighters or that you are doing the impossible with no budget. What they care about…the ONLY thing they care about…is what doesn’t work, or isn’t set up the way they want it.

IT Operations isn’t a field where people pat you on the back and say “attaboy.” It’s a field where you can damn near kill yourself for seven years and then get reamed out because you collapsed in exhaustion before you remembered to tell someone some minor detail about something. It is a field where hard work and technical achievement pale in importance to ego stroking and pandering to the whims of users and managers.

Operations guys like me are viewed as little more than digital plumbers. When the cutbacks come at the large corporations, Operations are the first to feel it. Those who remain are told to do more with less. In the SME space, we are constantly up against the wall on budget, manpower and time. Through it all there is always the threat of having your job outsourced to a consultancy or another country.

So why did I put in 82 hours straight? Because the network needed to be ready for September 1st and I could see no other way. One day, my editor might even be able to teach me enough that I might be able to make writing into a career that keeps food on the table. That would be a great day. Until then, I do what I must because there is no other choice.

I am not afraid of hard work. I am afraid of letting people down; especially if the people in question employ me. If keeping my job requires stupid hours, then that’s what I’ll do. There aren’t a lot of IT good jobs for IT Operations folks in Alberta. There are however plenty of significantly worse ones than the gig I’ve got….

Hey SuperM

If you are looking for case study material you might find the reasoning behind the migration interesting.

The biggest reason for the move was the damage to the schema. In truth, some of it was caused by inexperience several years ago. Installing one product or uninstalling another caused all sorts of crap to accumulate in the scheme that I couldn’t clean out by hand no matter how hard I tried. (I simply didn’t know exactly where all the bits hung out, and Google wasn’t helping me much.)

Similarly, before we moved to virtual machines, it wasn’t uncommon for domain controllers to just up and die. As small as the company is (read the article again to get an exact count of how many systems we have) seven years ago we had one domain controller that was also a file server, firewall, FTP server and everything else all in one. When I started with the company that domain controller was being run off of a Pentium 4 desktop board with a single desktop-class hard drive.

As you can imagine it was a few years before we started to get DCs that didn’t experience random and sudden hardware failures; so there were remnants of these old DCs, despite my best efforts to nuke all references to them out of the AD. (The “Proper procedures” don’t get them all. Especially if your previously an heroed DC was a certificate authority, etc.) Not to mention that going from OCS 2003 -> 2005 -> 2005 R2-> 2007 R2 creates a whole bunch of zomfgwtf hanging about in the AD. Similarly Exchange 2000 -> 2003 -> 2007 -> 2010

In theory if I had thrown enough time at the old AD I might have been able to clean it. There comes a point however where you have to look at the whole mess and say “I only have 75 users. Let’s just restart from scratch; it’s significantly less effort.” I always have this sneaking suspicion that if I was a true active directory expert, I might have been able to avoid all of this. I’m not though. I’m not an “expert” at any one field of IT. I haven’t spend my career specialising. As an SME sysadmin I have to maintain all of those systems you read about (and more that didn’t make it into the article.)

There simply isn’t any possible way for one human being to develop true expertise in all of those various systems. (There is no such thing as a modern-day polymath.) The best I can reasonably hope for is to understand the fundamentals and as many of the commands/quirks/specialised ballyhoo of every application, operating system, hypervisor, file system, database, piece of hardware, networking, crypto etc as I can fit into my brain.

Where that falls apart is the truly in depth knowledge of things like Active Directory. I know more than your average bear; but by the same token I only really have to deal with it once every two or three years. It might have some real arguments for/against IT generalists like me. I believe I am a rare breed in IT; most folks seem to have specialised in some particular part of it (databases, AD, LAMP, whathaveyou) by this point in their career.

I am capable of dealing with a wider variety of systems than your average specialised IT body…but in order to have that capability I have to sacrifice a great deal of the super-specialised knowledge that comes from spending over a decade dedicated to a single type of product. At first glance, someone like myself might seem ideal for an SME. It certainly gives me a breadth of knowledge and experience that allows me to write about various topics here on El Reg.

What I begin to wonder however is if the truly efficient way to deal with SME IT administration isn’t to have SMEs handled by largish consultancies. A large consultancy can afford to have one (or more than one) of each relevant kind of specialist. When they run up against a challenge like I described above they don’t have to restart the whole AD from scratch. They summon their in-house AD super-specialist and he deals with it.

Where’s the line between the utility of an IT generalist and the advantages of a small cluster of IT specialists? I’ll be honest when I say that I don’t know. I am however exceptionally curious about the answer…

@Loogie

A large part of what we needed to do was shed the "cruft" a decade of disparate naming schemes, hirings, firings, e-mail address changes, remaining of users etc had caused to the AD.

You had USER_D who had been created as USER_A originally. The name was changed in order to ensure that USER_D could access USER_A's e-mail for business continuity purposes, but many of the various hooks for that user in the AD still reference USER_A. Similarly, many of the users were simply Firstname instead of Firstname.Lastinitial (which we started to make all new users several years ago.) There are other examples, but you get the idea. For this reason ADMT was pointless. We didn’t want to migrate the users. We wanted entirely new users with clean information that would follow the naming convention from the CEO to the digital janitors.

Recreating 75 users was the absolute least of our worries. That was about an hour’s work. Create the users, assign them a SIP address, make them an e-mail address. Link the home folders. Set the dial-in permissions on the small handful who needed them. Everything else was handled through GPO.