@Devon_Custard

The applications in question don't support said automation. It is not an issue of not being willing to put the time in to figure out how to automate them. The exception here is Firefox, to which I throw a “mea culpa”. I made an assumption (which made an ass out of u and me) that add-ons were global. I was caught entirely by surprise when I logged on as the user to load up the PST and discovered that the Firefox add-ons installed under the administrative context weren’t there.

I am 100% certain it is possible to automate the Firefox portion of this rollout. In fact, it’s on my list of new GPOs to create in the next month. That said, since we had to go into the profiles to manually configure applications for which there simply is no possible way to automate a rollout anyways, we did the Firefox as part of that.

When I say that there is no possible way to automate the rollout of these applications, I mean it. I have spent four years of off-and-on research trying.

@Colin Millar

I visited two out of three remote sites ahead of the change. I talked about the change with folks on the third. I made the changes known in meetings as far back as January, though the exact details of course were far from concrete at that point. While I certainly didn’t have “details” on what was going to go down (that is what the e-mails were for,) the general notion that “something big involving computers” was going to occur in mid August has been known by all parties for the entirety of the year.

I believe the term “network overhaul” was used more than once. I am absolutely certain I mentioned new virtual machines with Windows 7, Office 2010 and a new version of Communicator. The printers bit crept up at the 11th hour. (Old printers died.) Overall, if there was one major issue that could have been thrown down as my failure, it was that I didn’t have this all planned six months early.

My only excuse for that is lack of manpower and budget. Until the 11th hour we weren’t 100% sure what hardware resources of software licenses we would have available to us. We had a general idea, but no plan survives contact with the enemy. In addition to planning this rollout this year, I had to keep the entire network running (putting out fires) do research for the rollout itself, do testing/prototyping/etc. and we rollout out a complete desktop replacement of Wyse thin clients in June/July as well.

Somewhere in there we also completely redid the website and tacked on an e-store. I had to redo the spam server. Again. (http://www.trevorpott.com/?p=275) Which is more of a project (research wise) than I like to admit to. There was a major upgrade of the industry-specific software package (that hasn’t been fully completed yet, actually…) and a review/overhaul of the phone system. Oh, and we replaced all our Treo 700Ws with blackberries and switched providers.

ALL of that had to be planned, sourced, tested and implemented before September. It’s also not abnormal. This is what it’s like every year here. I will not spend the next month and a half cleaning up any handing threads from those projects. Towards the middle/end of October, I do a network audit. What needs done in 2011? Then I start putting together proposals. January brings budget meetings and we do it all again.

Did I fail miserably in making the users fully aware of how tough this overhaul was going to be? Yes. I deserve a good smack on the wrist and a “do better next time” for that. I can’t say I didn’t try; I simply wasn’t experienced enough to succeed.

@Bullseyed.

I watch water polo. The folks who play it get paid what I consider a pittance for their efforts. Other than that, I do watch Hockey like a good Canadian, but not the NHL. In fact, I'd like to see the players in the league I watch get paid more. I'd be willing to pay more for my season's tickets if they would.

Furthermore, CEOs in IT get paid excessively compares to their peers in most other industries. There are a few industries that are as bad, but overall IT is among the worst of the lot.

I also don’t see a “jealous rage” at all. I said I think he is grossly overpaid. I believe that the CEO of a multi-billion-dollar company deserves to be paid in the quarter-million range regular salary. That would cover the stress and the long hours. It would put them on par with other folk who work for them putting those stupid hours in.

The company should indemnify them against lawsuits incurred on the job.

They should also get performance bonuses TIED TO PERFORMANCE OF THE COMPANY that is where they make their millions from. My beef is that the base salary is disproportionate to the effort. My beef is NOT with the man making millions.

Also: there are plenty of folk that Hurd employs with far greater education than hi that I assure you he doesn’t pay nearly that well. You can tell me I am not qualified to comment on this all you want, but I am merely comparing IT to the scale of available other industries and looking at the scale of BASE SALARIES and how they are calculated.

So I maintain my statement: effort isn’t rewarded. Luck and making the correct decision at the correct time is. Bully for Hurd managing to cheat the system, but it doesn’t make the system any more just.

@Bluegreen

The local experts thing was tried. See above post by me. Getting "buy-in" from the users is impossible...they see any change as a hostile act and the changes were dictated from above my head as necessary. We didn't have the funds to run both systems in parallel.

And no, you DON'T, EVER do it how we did it. It was a mess. Please, anyone reading these articles and these comments....learn from that.

As much as I would love to blame my managers and say they have explaining to do…they sort of didn’t have much choice either. Well…one of them did. The one pushing the new software as requisite. The rest…if you take the new software requirement as a given, then no…there was no other choice. The funds simply weren’t available to it properly. It’s not a matter of being cheap; it is a matter of IT requirements outstripping capacity. The company I work for is in a odd place: we have the IT requirements of a company three times our size. Periodically, we are all of us forced to bite the bullet and do what we don’t want to do.

I am sure I shoulder some of the blame. I explain this through my articles as I talk about the things I should have done but didn’t. I could have done some things to make this go smoother, but didn’t think of them in time. Still, the combination of what I should have and could have done combined with what I should have and couldn’t have done made for a weekend in which pretty much everything that could go wrong…did.

I don’t think it’s a neat little bow where all the blame can be borne by one person. I deserve to take some crap for it. The managers do as well. The users were warned about all of this before hand (months beforehand) and honestly could have read their damned emails too. I think there’s plenty of blame to go around, but unfortunately the issues involved are so complex you can’t lay them all on one person’s head.

It was a learning experience for all involved. One I hope my readers learn from as well.

@Alex Rose

A question instead of a snipe! Hurray! Okay, here’s the skinny: We use a Point Of Sales (POS) application that is absolutely vital, and forced us into the VDI route. In Addition, we have a piece of industry specific software that isn’t compatible with multiple sessions running on a single machine.

The POS software is only barely multi-session aware. It will work if it is installed on a terminal server, with several people running several instances of it. However the application over a userbase of about 50 individuals does tend to crash on someone’s machine at least once a day. When that thing crashes in a terminal services environment for one user, it crashes for them ALL. For the same reason, application virtualisation is out; App-V doesn’t fully containerize the application. It just presents a TS session consisting of one application. Tank one copy of the app in an App-V scenario, you still tank every other copy running on the Windows Desktop Services server.

Since it’s a POS application, when it crashes it leaves a nightmare for the bean counters to clean up, let alone an inconvenience for the users. Because of weirdness in the design of the POS application, it absolutely *must* be run from the same location. We can’t deploy it centrally and have folks on remote sites run it locally. They have to be using VMs at a central location to use it.

Even if we could iron that out, we still wouldn’t be able to run the industry-specific piece of software that the other 25 users require in a multi-session environment.

Oh, and both those apps as well as Firefox require user-specific configurations, so we’d still have to have logged on as each user to do that, as well as port their PST files from the old network.

And that sir is how we ended up with VDI.

@Pandy06269

We are running almost fully VDI. We did use templates. Office doesn't like bing cloned. Firefox has user-specific configurations and add-ons must be installed /per user/. We have industry-specific applications that absolutely require user specific configuration. PST files had to be loaded from the old forest's exchange server into the new forest's exchange server manually. The change from XP to 7 also brought with it the change in profile types. We changed from roaming profiles to folder redirection and did so across forests. That meant manually importing files from the old network and placing them into the appropriate places on the new network.

There were other things, but no. It wasn't something that could be automated. I promise you we automated everything we could, and made extensive use of template VMs. Sadly, the ability to filly configure applications which require user-specific setup depends on support from the application vendor. Two of the applications we use simply don’t support it, and I have zero say in the fact that we use them.

@Kempsy

Ubitmenu. <3.

Training?

As in where you spend money to get users trained? Not in the seven years I've worked there. Myself and the other sysadmin have gotten MCP exams paid for. The other sysadmin and the CFO periodically are flown out to learn some things about how the point of sales software works. One or two of our production folks attends a conference every year about how a piece of industry-specific software works.

As to users...no sir. It is generally considered our job as IT Operations to train them. The production fellow has to train users in the use of that application. That's as far as it goes. As to chaninging my job...why do you think I took up writing? It will be a slow process, but maybe in ten years...

An additional comment...

I should also point out that when server upgrade time rolls around, there won’t be any side-by-side coexistence of servers. Go read everything in the article that we support, and then understand that I have to shove all of that, 4x 100Mbit fiber connections and the salaries of the two sysadmins and the bench tech into less than a quarter-mil a year. When the virtual servers in use are done with being servers, they will be removed from their chassis, given a hearty dose of TLC and placed into desktop chassis.

They will then serve an additional three years as “medium-demand physical workstations” for our Photoshop geeks. The next server replacement cycle doesn’t occur until 2012. We are using “tick-tock.” This year was desktops. Two years from now is Servers. Two years after that will be desktops again. The sole exceptions are the Photoshop geeks who get trickle-down servers as part of the server refresh cycle.

When you see me talking in my articles fairly constantly about the need to do things cheaply, (or constantly trying to find the most cost-efficient method of doing something,) this is why. It should also then carry some weight when I recommend spending money on something. The cost of a Windows Server Enterprise license can to me mean 4 years of service life from a virtual server capable of running 30 personal virtual machines.

I constantly work on the razor’s edge of what is actually possible with the hardware and software I can get my hands on. I can’t even big myself up and say that this is because I am somehow superhuman or a great sysadmin. I am just willing to work long hours to get things done. I will tell you now, honest and true, that without my partner-in-crime fellow sysadmin (and one of my very best friends) this wouldn’t be possible.

I specialize in the impossible; making a system do what it was never designed to. Pushing the limits and doing the research. I am an IT MacGuyver, but as such I am only one part of the equation. My buddy is the polar opposite. He is the living embodiment of “by the book.” He keeps me in check, goes over all the nightmare hacks and kludges I have created to put out fires and get things works. He takes my quick fix or cobbled together solution and produces reams of documentation, tests it in alternate conditions and works out something that is reproducible and far more production ready.

Together, I honestly think we make a great team for an SME environment. We do what I am constantly told by me peers simply isn’t possible. I have an enormous amount of pride in that, but I do have to say the stress gets to you. It isn’t the stress of the long hours…but the thanklessness of the job. Heroics aren’t rewarded in IT. Nobody cares that you are pulling all nighters or that you are doing the impossible with no budget. What they care about…the ONLY thing they care about…is what doesn’t work, or isn’t set up the way they want it.

IT Operations isn’t a field where people pat you on the back and say “attaboy.” It’s a field where you can damn near kill yourself for seven years and then get reamed out because you collapsed in exhaustion before you remembered to tell someone some minor detail about something. It is a field where hard work and technical achievement pale in importance to ego stroking and pandering to the whims of users and managers.

Operations guys like me are viewed as little more than digital plumbers. When the cutbacks come at the large corporations, Operations are the first to feel it. Those who remain are told to do more with less. In the SME space, we are constantly up against the wall on budget, manpower and time. Through it all there is always the threat of having your job outsourced to a consultancy or another country.

So why did I put in 82 hours straight? Because the network needed to be ready for September 1st and I could see no other way. One day, my editor might even be able to teach me enough that I might be able to make writing into a career that keeps food on the table. That would be a great day. Until then, I do what I must because there is no other choice.

I am not afraid of hard work. I am afraid of letting people down; especially if the people in question employ me. If keeping my job requires stupid hours, then that’s what I’ll do. There aren’t a lot of IT good jobs for IT Operations folks in Alberta. There are however plenty of significantly worse ones than the gig I’ve got….

Backgrounds?

Nein! We use RDP to access virtual machines. You vill use solid colours und you vill like ut!

Dude...

So many applications and settings for which that simply isn't possible. It sucks. Greatly.

Reality Journalism

I think I need time to digest that. I abhor reality TV with a passion I can't properly describe. Am I truly participating in reality journalism? Have I committed some form of unpardonable sin?

"Most read author?" I honestly have no idea. I don't get any stats, so I fly blind except for the scathing (though occasionally nice) comments from El Reg's famously cynical commenttards.

I can’t see it though. That could my bias as a dedicated reader of El Reg for over a decade, but I really don’t think I’m quite as good as the regulars around here. I would point you in the general direction of Lewis’s DARPA battleboffinry descriptions as a fantastic example. In time though, I hope to learn enough to be able to write at that level.

As for my favourite author, actually, it’s Tad Williams.

@jackifshadows

I did it because it had to be done. I doubt I'll eve be trying something like it again. We didn't have the money for IBM Globalservices. We did have the money for a sysadmin who doesn't get paid overtime. What I did was make a decision that was good for the company, but bad for me personally.

If I have a flaw as a sysadmin it is honestly that I work too hard. When I put my “company hat” on, I push out my own needs and focus entirely on what is best for the company. The problem with that is that I then rapidly burn out, which doesn’t do the company any good and sure as heck doesn’t do me any good.

So the thing is to know when to spend the money and when to bur myself out. Sometimes the money isn’t there and so the call is taken out of my hands. Sometimes I make a bad call, sometimes I make a good one. I’m still learning where the balance lies.

I agree with your assessment though. The IT generalist is becoming an individual who manages a series of contractors and outsourcers. It’s something that saddens me, because it means the end of my career in IT. I don’t have a degree, nor any management credentials beyond running an SME IT department for seven years. I might be able to get a PMP designation or somesuch, but then I am fighting eleventeen squillion unemployed IT Operations guys with PMP designations for the small handful of jobs left on this continent.

Once the SME administration job market dries up, I honestly don’t have a clue how to make the jump to being an admin in a larger enterprise, nor do I think that I have the resume to go up against the many other vastly more experienced contenders for the “outsource manager” positions.

This is why I’ve taken up writing. I think that ten years hence there will be more of a career in writing than in SME IT administration. Have to change with the times. If I can stick with IT writing, then all those years of experience as an IT generalist won’t have gone to waste.

@Jane Fae

In all honesty, I prefer a middle ground. I believe that a legitimate new organisation should have to register with a central office, but that it not be a form of "accreditation." It should simply be a registry with a fee attached. The fee should be high enough to discourage Joe and Jane random blogger who just wants a press pass to wave about for ego purposes, but low enough that it is not unachievable for dedicated bloggers or not-for-profit new organisations.

An organisation like The Register which may not produce a dead-tree edition is still nonetheless a legitimate news entity. It is the entity itself that I believe is important, not the individual reporter or blogger. The reputation of that entity should be the backing behind validity of a press pass. If Joesblog.com registers as a news organisation, it should be able to issue a press pass to one of it’s bloggers. Individual events should have the right not to recognise joesblog.com if they choose, but police should not be given that choice.

An event (such as Macworld for example) may choose to be selective about which press organisations it reveals information to. There are reams of business reasons why an event or company may choose to do this. A government or police organisation on the other hand should be allowed no such right of discrimination. Any news outlet which has gone to the trouble of registering it’s existence should have to be recognised by the powers that be as a news organisation capable of fielding a journalist on scene.

I think that’s a reasonable compromise between enabling citizen journalism and preserving the integrity of the press pass concept.

If The Register wanted to give me a press pass then, nothing could prevent it from doing so. I may not have a degree in journalism, but it may make sense for me to have one: I could be attending a conference, or simply stumble into a relevant situation in which I could gather information as a journalist/blogger/whatever-the-hell-I-am that would benefit El Reg. By the same token, if they felt I was a loose cannon nutjob, they should rightly withhold a press pass on the basis that my running around bearing a slip of paper with their name on it could degrade their reputation with companies/event organisers/etc.

To summarise my beliefs:

In order to be allowed to issue a press pass you should have to register centrally in the country in question as a news agency. This discourages every blogger in the universe issuing press passes to themselves.

Event organisers/corporate PR departments should be allowed to discriminate against organisations based on internal criteria regardless of registered status. This encourages news organisations to be judicious to whom they hand out press passes.

Governments/Police Forces/Militaries etc. should not be allowed to discriminate against organisations based on any criteria except registered status.

I am very interested to hear why people agree/disagree with the views I have expressed above.

@AC

Bringing in other help. THAT is a whole other story. The interesting part is that I did try to get a buddy of mine who runs a computer consultancy to lend a hand (for appropriate remuneration) during the op. He bailed at the eleventh hour adding another layer of fun and happiness to this entire exercise. Additionally, another individual I was hoping to be able to talk into assisting got distracted by Starcraft 2. (He was at the time considering pursing IT as a career only to change his mind and head towards electrician in the last couple of weeks.)

So what I was seriously hoping was going to be a five man operation became a three man operation. Then Xerox got tripped up on the printer delivery due to backordered paper trays (sonofa…) OCS blew up. The Spam server ate half a night because I typoed the domain name. Office 2010 had some ridiculous upgrade guarantee thing that made getting the 2010 key from our 2007 installs a screaming nightmare and took five times as long as we figured it would. We discovered too late that Firefox add-ons are USER SPECIFIC and didn’t have time to figure out how to push those through a GPO, so ended up installing them manually.

To top it off, we had two different industry-specific applications that absolutely required user-by-user configurations and took about an hour each. There is no facility whatsoever for centralised deployment and configuration of those applications. Throw into the mix that each user is a “special case” with their own unique set of software requirements and you can’t just roll out one base image and be done with it.

It’s so very easy for people looking at something like this in hindsight, or reading about only part of the reasons and events that occurred to spew vitriol and negativity about the whole thing. Actually working in an SME environment like this is a completely different story…something I hope that my blog articles can help convey.

@Ammaross Danan

Amen brother. You have hit upon the purpose behind the entire exercise. There were some things we simply couldn't accomplish with the changeover except by logging in as those users and setting their profiles up for them. What folks with the anger-making don’t see to get is that had everything gone to plan, we would indeed have reset the users’ passwords and forced them to change upon first login that Monday.

Things went all pear-shaped when folk started showing up on the Monday…and oh, damn…we weren’t done customising their VMs. By the time we got done with the customisation, we were deep into Tuesday. This means the users had logged in for a couple of days with the “new” passwords they were given.

Going around and forcing a password reset on the Wed morning, after Monday and Tuesday had been filled with printer issues, incompletely-customised VMs and changes to new and bewildering versions of software like Office 2010 would have been heap bad juju for IT.

The solution in the end was to phone the users one at a time and walk them through the password changes whilst taking the time to hear out any complaints they had, teach them any bits the needed to know about the new software and help them customise Windows 7 to behave less new and scary.

The sheer APATHY towards the “changing passwords” part of that phone call shocked me however. Users really, really don’t give a damn about security. Enough that I am starting to come round to the opinion that they honestly do need a periodic kick in the ass about it.

Unlike Unix, Windows administrators don’t have the opportunity to completely abstract the security away from users, nor do they have the control that a Unix admin would have to customise profiles and the like from afar. This means some level of user cooperation is required in the Windows world. When you run up against the sheer user apathy we did…that’s an eye opener worthy of an article.

@Keith 21

Well, first off...we don't have compliance issues to worry about. Were i operating in an environment where that was a concern, then things certainly would play out differently. We did have a contingency plan if things went wrong: step through issues with the users one at a time. There are only 75 users.

Phoning them up one at a time to deal with their passwords, listen to their complaints and solve any problems they have one at a time really didn’t take that long. Less than a working week and we had dealt with each and every user in the company. A contingency plan doesn’t have to be a set of automated tools or a whitepaper. Sometimes it can be putting one’s nose to the grindstone or adding a human touch.

There are “proper” ways to do things. I agree with you wholeheartedly that if you have the time, the resources and the manpower then how things went down during doomsday weekend would look frighteningly inefficient. What my Doomsday Weekend articles are about is not how things work in a corporation of 300,000 users with a crack team of administrators and management that understands why IT Operations need time to execute.

What my Doomsday Weekend articles are about is what life is like at the coalface of an undermanned, underfunded SME IT department with nearly zero resources and a management staff that doesn’t remotely understand why IT Operations need time to execute. My Blogs aren’t a statement of how I have accomplished perfection and found the one true path. They are a demonstration of the mistakes I have made, the neat products I have encountered and the insights I have gained from it all.

I operate in a far less than perfect environment. Contrary to your stated belief, I am not lazy. I am willing to put the time and effort in whenever and wherever I can. If you read the article previous to this one, you will see the network I have to run, with only two sysadmins and a bench tech to hold it up. I promise you that I know the “by the book” ways to run a network. I also promise you that we don’t have the resources or the time to manage by whitepaper, as easy as that would be.

As to “lashing out,” I don’t believe I did any such thing. If anything, I think you will find I took a good poke at both users and systems administrators in my article. What I do have to question though is why you feel the need to be so negative towards me? What did I do to offend you so?

@James Pond

Biometrics and Wyse Thin Clients. *Wince.* I am not sure I want to start mixing magics like that.

@ITSMeagain

I'm Canadian. I can't speak for the rest of my countrymen, but when it comes to this all pervasive spying, be it from Google or our own governments, I will fight to my dying breath. You will pry my personally hosted servers from my dead, rotting hands.

Correction:

This should have read:

My statement was /merely/ that it was a common approach, but in all truth not one I support in any way.

My experiment of using my HTC Desire as the only source of comment input for my own articles over the past month is truly proving both how bad I am at spelling/grammar without squiggly little lines under everything and how far these devices really do have yet to go as regards text input.

More fodder for the smartphone-as-a-work-tool article I suppose…

@AC

Got a link for that job in Kandahar? Sounds like less stress than my regular. Better pay too.

@Timo

The pre-edited version of the article had this line: "I should have printed off a copy on each of the remote sites’ printers such that the information they required was on hand when they walked in."

Very small difference to the published article, but it essentially mirrors your advice. I have a few users who would agree with you that it was indeed the correct approach to have taken.

@AC

Overall this is excellent advice. The dynamics of this appraoch in a smaller company become far more complicated...but the advice is no less true.

@nichomach

Honestly, I don't believe in this approach. My statement was mere that it was a common approach, but in all truth not one I support in any way.

@AnachismEvolved

Would have been in compliance if we had managed to reset all the passwords before the bell. Sadly...

Hey SuperM

If you are looking for case study material you might find the reasoning behind the migration interesting.

The biggest reason for the move was the damage to the schema. In truth, some of it was caused by inexperience several years ago. Installing one product or uninstalling another caused all sorts of crap to accumulate in the scheme that I couldn’t clean out by hand no matter how hard I tried. (I simply didn’t know exactly where all the bits hung out, and Google wasn’t helping me much.)

Similarly, before we moved to virtual machines, it wasn’t uncommon for domain controllers to just up and die. As small as the company is (read the article again to get an exact count of how many systems we have) seven years ago we had one domain controller that was also a file server, firewall, FTP server and everything else all in one. When I started with the company that domain controller was being run off of a Pentium 4 desktop board with a single desktop-class hard drive.

As you can imagine it was a few years before we started to get DCs that didn’t experience random and sudden hardware failures; so there were remnants of these old DCs, despite my best efforts to nuke all references to them out of the AD. (The “Proper procedures” don’t get them all. Especially if your previously an heroed DC was a certificate authority, etc.) Not to mention that going from OCS 2003 -> 2005 -> 2005 R2-> 2007 R2 creates a whole bunch of zomfgwtf hanging about in the AD. Similarly Exchange 2000 -> 2003 -> 2007 -> 2010

In theory if I had thrown enough time at the old AD I might have been able to clean it. There comes a point however where you have to look at the whole mess and say “I only have 75 users. Let’s just restart from scratch; it’s significantly less effort.” I always have this sneaking suspicion that if I was a true active directory expert, I might have been able to avoid all of this. I’m not though. I’m not an “expert” at any one field of IT. I haven’t spend my career specialising. As an SME sysadmin I have to maintain all of those systems you read about (and more that didn’t make it into the article.)

There simply isn’t any possible way for one human being to develop true expertise in all of those various systems. (There is no such thing as a modern-day polymath.) The best I can reasonably hope for is to understand the fundamentals and as many of the commands/quirks/specialised ballyhoo of every application, operating system, hypervisor, file system, database, piece of hardware, networking, crypto etc as I can fit into my brain.

Where that falls apart is the truly in depth knowledge of things like Active Directory. I know more than your average bear; but by the same token I only really have to deal with it once every two or three years. It might have some real arguments for/against IT generalists like me. I believe I am a rare breed in IT; most folks seem to have specialised in some particular part of it (databases, AD, LAMP, whathaveyou) by this point in their career.

I am capable of dealing with a wider variety of systems than your average specialised IT body…but in order to have that capability I have to sacrifice a great deal of the super-specialised knowledge that comes from spending over a decade dedicated to a single type of product. At first glance, someone like myself might seem ideal for an SME. It certainly gives me a breadth of knowledge and experience that allows me to write about various topics here on El Reg.

What I begin to wonder however is if the truly efficient way to deal with SME IT administration isn’t to have SMEs handled by largish consultancies. A large consultancy can afford to have one (or more than one) of each relevant kind of specialist. When they run up against a challenge like I described above they don’t have to restart the whole AD from scratch. They summon their in-house AD super-specialist and he deals with it.

Where’s the line between the utility of an IT generalist and the advantages of a small cluster of IT specialists? I’ll be honest when I say that I don’t know. I am however exceptionally curious about the answer…

@Loogie

A large part of what we needed to do was shed the "cruft" a decade of disparate naming schemes, hirings, firings, e-mail address changes, remaining of users etc had caused to the AD.

You had USER_D who had been created as USER_A originally. The name was changed in order to ensure that USER_D could access USER_A's e-mail for business continuity purposes, but many of the various hooks for that user in the AD still reference USER_A. Similarly, many of the users were simply Firstname instead of Firstname.Lastinitial (which we started to make all new users several years ago.) There are other examples, but you get the idea. For this reason ADMT was pointless. We didn’t want to migrate the users. We wanted entirely new users with clean information that would follow the naming convention from the CEO to the digital janitors.

Recreating 75 users was the absolute least of our worries. That was about an hour’s work. Create the users, assign them a SIP address, make them an e-mail address. Link the home folders. Set the dial-in permissions on the small handful who needed them. Everything else was handled through GPO.

@AC

Two problems with that idea.

1) I actually have no idea where to rent server equipment from. Honestly not a clue where to even begin.

2) I didn't think of it until we were about halfway through the weekend. Hindsight and a great deal of “d’oh” suggests you might well be on the proper track, however…

@AC

The printers were largely upgrades to an extant lease of Xerox printers. Some were replacements for really, REALLY old workcenters (Think N32s, N40s, etc.) No futzing with forms was necessary, and I did spec delivery for the middle of the month. (Doomsday Weekend took place on the 20th of August.) Negotiations for the whole thing started at the beginning of August.

The nice lady seemed to be trying to do everything she could to get the widgets on time, but sometimes expletive happens. Sadly, it tends to happen at the worst possible time…

@alphawave

I use no alias. I make mistakes, same as any other human being. I will fess up in public in the hopes that others can learn from my mistakes. The fact that I have made these mistakes is called “experience.” You screw up in some way once and then tend not to do so again. For the record though: I don’t happen to work in one of those lovely environments with unlimited budgets, massive amounts of free time and adequate manpower for every task. Quite the opposite: I am regularly tasked with doing the impossible on a shoestring budget. Some times I make it…many times I don’t.

It certainly doesn’t make for bragging rights. I don’t get to stand up in front of all the commenters on El Reg and proclaim “here’s how I pulled off the perfect everything with no mistakes whatsoever.” It does is keep me humble. With luck, it will help a junior sysadmin or two avoid the same errors that I have made. Additionally it gives some of the commenters here on El Reg a reason to feel superior: many of you have avoided the mistakes I have made. Some have avoided these issues due to superior foresight, some due to superior experience and many due to superior availability of resources.

Whatever the case…I don’t try to hide my faults, or my mistakes. I would rather honestly make a mistake (and own up to the consequences) than bury the truth and be hired/respected/whatever on false pretences.

My blogs are my experiences in IT. The good, the bad, the hideously ugly. If it means that an IT recruiter looks at my articles and believes that I am completely incompetent then so be it. That is a consequence of my choice to try to pursue writing. I have often been told “write what you know.” Much of my life has been devoted to IT. Combine that with a personal philosophy of never sugar coating anything and you get El Reg’s Sysadmin blog.

Warts and all.

@The original steve

Actually, I *AM* the one who said it had to be done in a weekend. Here's the scoop:

We absolutely /had/ to have the updates done by September 1st. (This per CTO requiring updates and the fact that Sept – Dec is silly season around here.) As per above, I could see no way except to do it “all in one go.” I was not present through July, as I was in two of the other locations installing Wyse clients, recabling and prepping the locations local hardware for the changeover.

It took us the first two weeks of August to get the Domain Controllers, E-mail server, and BES/OCS/WSUS/Teamviewer manager server (yes those 4 share a VM) installed and for the new network. (I was going to have THOSE at least installed, if not configured by Doomsday.) In addition, we roughed out a template Windows 7 VM and a few template Windows XP VMs for render boxes.

We didn’t have to go ENTIRELY from scratch, but I promise you that even to get what we had prepped in advance we were scraping the bottom of the excess Virtual Server capacity barrel.

I managed to create all the user accounts prior to hitting the wall, but that’s about it. Exchange wasn’t configured, OCS was not only not properly configured…it had to be uninstalled THRICE and reinstalled specifically to get the blessed thing to cooperate. As to the rest…well…there’s more articles on that.

Suffice it to say that the actual call “hey guys, we need to do this all in one go over a weekend” was mine…however there were ZERO other choices that would have had us meet the deadlines imposed on us…

@AC

Sounds to me like you have a bad case of "Schrödinger’s rack." Tough break; I hear the cure is quite expensive…

@AC

Old printers were giving up the ghost. (Some were pushing 10+ years.) We needed a new set of higher volume printers. Had the things arrived on time, the old printers would have gone away with the old network, and the new printers would have started working on Monday morning with the new network. Alas, this was not to be as the – get this – paper trays where backordered.

Yeah. Paper trays.

*sob*

Contractor. HAH.

Full time regular staff. Salaried, and I don't get overtime. There wasn’t enough gear to run both networks in parallel. We had to pull of the network change without new equipment; almost everything is virtualised, so the “new network” was largely a set of new virtual machines. Given that, I don’t actually see how you can move from one forest to another, migrating all services “one at a time.”

I would love to say I am totally in charge of making all such decisions, but unfortunately I do have to work with what I’m given. I do largely get to buy what want, but I have to justify it all; part of that justification is that all equipment be utilised 80%+ for it’s lifespan. Buying equipment just to handle the changeover then letting it sit idle would /not/ have gone over well. Most especially since we would have required somewhere around 30% of our yearly budget’s worth of gear to do it.

Sometimes, you just gotta do what you gotta do…

Perchance?

I still use it. In regular conversation. I'm not even British; I'm Canadian. So, dude...wtf? Might as well lay into someone for using "whilst" or "thrice." Internet word snobs...*sigh*...

"This is loads more fun."

Um...

Is it useless though?

I thought the purpose of such entities was to make the hoi polloi believe something was being done about whatever subject was at hand while in truth protecting the quangos and individuals the folks in power liked. You make it seem like you believe that there exist government agencies (ourside of health care/social services and firefighting) that actually help people.

Madness.