Save us from our users Many sysadmins among us certainly have a god complex. The truth is, no matter how well-prepared for a large project we try to be, we can't control our users. I took great pride before Doomsday Weekend in typing detailed emails about the changes that users should expect to see. The documents were necessary. They helped users …
"Next time I won't care about bad PR for IT. I’ll just tick the “reset password" box and be done with it"
Speaking as an IT person, I woudl sincerely hope that "next time" you will not be lazy and will instead spend the extra effort to provide each user with a unique secret password - problem completely solved.
Oh, sure, you will have to do a little more work, but that's what we in IT do - we do a little extra work to ensure the safety of our customers' data and to make their lives easier.
you want an easy life? Get out of IT.
Sorry, but I have zero sympathy for yuou on the password issue, that is purely down to laziness on your part (and I speak from experience here).
Dear Keith 21,
When you have over 1000 employees which don't read emails from IT, exactly how do you provide each user with a unique secret password?
When you are dealing with a handful of users in a single site you can consider things like this. Otherwise you have to make compromises just to keep an organisation operational. This is the nature of the article and an unpleasant fact of large scale IT.
...that you provided them with the same password as all the others.
Doh!
"When you are dealing with a handful of users in a single site you can consider things like this"
I'm sure you can.
Doesn't mean you can't take that TINY little bit more effort when, like me, you are supporting 25,000 users at 9 globally-distributed sites.
Unless, of cours,e you are just lazy. Like the author.
You're going to generate dozens/hundreds/thousands of secure passwords, go through adding them to all the user accounts, then find a secure way of informing all the users about all their new passwords? That's incredibly time-consuming and utterly absurd, not least because 98 percent of the users will forget the password you've given them within 24 hours and you'll then have to let them set a password of their own choosing anyway.
Not sure what 'experience' you speak of here - administering the family PC in your house, maybe?
There are plenty of programs to do this, give the user a big list to choose from via a webpage, create lots of passwords on the page and then let them choose, its even easier if the passwords are able to be spoken phonetically.
If you want to distribute passwords, send the user a text, then they can keep the password at hand but out of sight, there are plenty of sms gateways out there. Then for the few people that do not have mobiles you can manage by hand.
simples!
My employer has a policy of forcing a password change every 28 days, and no re-use of a password within 12 months. From talking to many of my co-workers it seems there are two systems in widespread use across the company. One is the user's christian name followed by the number of the month, the other is PASSWORDxx where xx is again the month number. I figure I could probably get into at least 50% of the computers on site using one of these combinations so where is the security in forcing people to change passwords too frequently?
I suppose you could argue that it's the people that are wrong and not the system, but human nature says that when faced with password overload we tend to opt for the simple.
"...or they will punish them with even more exacting security measures." is made of wrong; I appreciate the frustration, but if the sanction applied for disregarding security measures that are considered overly onerous is more onerous security measures that will also be disregarded, what have you accomplished other than getting yourself more bad press?
This should have read:
My statement was /merely/ that it was a common approach, but in all truth not one I support in any way.
My experiment of using my HTC Desire as the only source of comment input for my own articles over the past month is truly proving both how bad I am at spelling/grammar without squiggly little lines under everything and how far these devices really do have yet to go as regards text input.
More fodder for the smartphone-as-a-work-tool article I suppose…
"I’ll just tick the “reset password" box and be done with it".
That check-box is there for a reason.
And force them to change at least 10 different passwords (from last one), otherwise they will change it, and immediately change it back. I do it myself wherever I can.
Plus, nobody reads IT mail because you are used to sending "we'll be experiencing intermittent outages between 2 am and 3 am this Sunday" mails, that had proved most helpful to 9-5 drone workers.
Oh yes, they feel the urge to access their outlooks at wee hours in the Sunday morning.
Not even if it is Monday morning in Japan, they won't.
"... detailing exactly how users could change passwords, why doing so was a good idea..."
"See, we set all of your passwords to the same thing, so you need to change yours for security reasons..."
At which point the office scallywags went and logged in as everyone else and flooded your mail system with;
"From: Dan
To: Mike
Subject: I've been meaning to tell you...
...I want you to call me Susan and be my cuddle-bunny."
(Just kidding. I'm sure you didn't really do that).
I learnt a long time ago that users never bother to read those emails you've spent ages lovingly crafting to give them a full and helpful understanding of what's going on. Nowadays I just keep it down to a couple of lines or bullet points with the vital info and nothing more. If they want to know more they can ask.
Everything you will try to do to help users will fail. A large percentage of them (not all of them) simply seem to like to make you (the sysadmin) miserable. The only user-approved change is no change at all. So, go on and make your life easier while ignoring their selfish requests when you (need to) roll out changes.
Follow the way of the BOFH.
Setting every users password to the same would fail compliance on at least half a dozen different standards and opens the floodgates for masquerading users. We went through a similar project and set every users passwords to a random password generated and got them to change at first logon, simple and effective.
But I work as an IT head and my support team continuously gets moaned at for having such complicated password requirements with requiring users to change their passwords every 60 days following US DOD standards. But ultimately we didnt get a opinion in the discussion, we have to adhere to security standards for compliance and those restrictions are stated in black and white, we just have to live with them, keep them enforced and take the brunt of the complaints.
So what did your Contingency Plan say for handling that case and still maintaining compliance?
You did do contingency planning, right? After all, that is a pretty essential part of the process, plaqnning for how to handle the situations where things do not go according to plan.
Every time we do any work, we always have a full contingency plan to account for how we will handle any failures, from minor things through to everything going tits-up. This we share with the management teams and, for big projects, our end-users as well, to give them the confidence that we've got the situation under control and that even if chaos reigns throughout, we'll have them working safely and securely in some defined manner.
Things go wrong in even the most meticulously prepared project. That's just a fact of life.
The true test of a properly planned and managed project is how you cope with those unforeseen things, how you ensure your customers are not screwed.
Sure, I get that users don't read their emails and miss important info, I really do get that (and with that aspect, I fully sympathise and empathise with you).
But I do not for one second think you have any justification to complain about them with regard to the passwords, given that was entirely a failure of your own making - lashing out and blaming them and threatening them with dire consequences next time because YOU failed is not an attractive trait.
Failures happen.
We can either learn from them, or do what you are doing and lash out at others.
Well, first off...we don't have compliance issues to worry about. Were i operating in an environment where that was a concern, then things certainly would play out differently. We did have a contingency plan if things went wrong: step through issues with the users one at a time. There are only 75 users.
Phoning them up one at a time to deal with their passwords, listen to their complaints and solve any problems they have one at a time really didn’t take that long. Less than a working week and we had dealt with each and every user in the company. A contingency plan doesn’t have to be a set of automated tools or a whitepaper. Sometimes it can be putting one’s nose to the grindstone or adding a human touch.
There are “proper” ways to do things. I agree with you wholeheartedly that if you have the time, the resources and the manpower then how things went down during doomsday weekend would look frighteningly inefficient. What my Doomsday Weekend articles are about is not how things work in a corporation of 300,000 users with a crack team of administrators and management that understands why IT Operations need time to execute.
What my Doomsday Weekend articles are about is what life is like at the coalface of an undermanned, underfunded SME IT department with nearly zero resources and a management staff that doesn’t remotely understand why IT Operations need time to execute. My Blogs aren’t a statement of how I have accomplished perfection and found the one true path. They are a demonstration of the mistakes I have made, the neat products I have encountered and the insights I have gained from it all.
I operate in a far less than perfect environment. Contrary to your stated belief, I am not lazy. I am willing to put the time and effort in whenever and wherever I can. If you read the article previous to this one, you will see the network I have to run, with only two sysadmins and a bench tech to hold it up. I promise you that I know the “by the book” ways to run a network. I also promise you that we don’t have the resources or the time to manage by whitepaper, as easy as that would be.
As to “lashing out,” I don’t believe I did any such thing. If anything, I think you will find I took a good poke at both users and systems administrators in my article. What I do have to question though is why you feel the need to be so negative towards me? What did I do to offend you so?
There are two ways to approach passwords. Assume the users password will be compromised every x number of days and force them to change it. Which usually results in the simpl pattern passwords. The other is to force the user to choose a complex password that they will not be forced to change, and works for multiple applications. This results in a password that the user will not need a note to remember and that is harder to break, but if it is broken the breach is much worse.
Hi Trevor
I'd like to thank you for having the balls to put your own experiences in the open like this. Not a lot of people would, especially in a forum such as this where every issue you come accross is poured over by commentards who don't realise that if they actually did projects like this then "un-forseen circumstances" do arise and cause problems. Also, hindsight is a wonderful thing especially when talking about someone else's project.
Secondly, users eh? Can't live with em...... can't kill them all with a hammer either.
Most don't see IT security or getting things up and running as anything to do with them what-so-ever and given half a change will gladly not co-operate with any instruction given over email. I agree with that stance to a large degree, they shouldn't be expected to sort this stuff out. As far as they're concerned, that's what the IT dept is paid for.
Anyway it's all good stuff!
Hey. Thanks for the nice post. In a sea of negativity, it is a refreshing change. Comments such as this that demonstrate an understanding of the issues and frustrations I face are the reason I enjoy writing. It’s so easy for people to be negative when they have the advantage of hindsight, or external objectivity. Being in the thick of it and against a deadline always leaves less time for consideration than reading about it on a blog.
Writing for El Reg has really changed my attitude towards being a commenter here; I can’t bring myself to take the piss out of the authors nearly as often as I used to. (Other commenters…that is another story.) I have been informed by various commenters that I simply need to develop a thinker skin about such things. They are probably right…but it doesn’t make a comment like yours any less nice to read.
Made my day.
Is to make sure the managers are on side first.
Then any problems with the users not following instructions from IT, become a local management issue.
The downside is instead of a mob of complaining users that you can largely ignore, you can end up with a more forceful group of complaining managers. But at least its a smaller group to manage.
Not sure why you posted AC, as it's an excellent piece of advice and I like to know who I'm upvoting. :-)
Existing management comms channels are ideal for making sure that the message gets put across thoroughly and effectively. Users will read and acknowledge mail from their line manager; they are less likely to read it from Joe.Bloggs@ITSupport.
There are endless studies (that I can't be bothered googling for you) that show regular forced password changes are stupid. You are actually forcing your users to come up with a predictable scheme or write it down. Let them choose a strong password with mixed case, punctuation and numbers that they can remember without writing it down.
So if the users won't read your email, maybe you could have posted signs in conspicuous places around the office (by the elevator/stairs, printers, water fountains, bathrooms, etc) that would have given them a hint that the network was completely redone over the weekend. That would be one more source of a clue besides having the info printed out.
Or even easier - send 25 copies to each printer everywhere in the company. Sure you kill trees but you'd cover a lot of ground that way.
The pre-edited version of the article had this line: "I should have printed off a copy on each of the remote sites’ printers such that the information they required was on hand when they walked in."
Very small difference to the published article, but it essentially mirrors your advice. I have a few users who would agree with you that it was indeed the correct approach to have taken.
...and if I have to change my password every 60 days, that allows somebody to steal it and use it for up to 59 days. And the breach is never detected.
So how much of a benefit is that, really? How much damage can someone do in 61 days on our network with my user credentials that they can't do in 60?
Okay, I'm a fan of handing out physical keys or tokens that get you onto the system - could be expensive, though.
How about this: you change user's password to a long random string, and issue them with a card carrying a barcode of that string - which you can print easily. At the workstation, they present that card to the scanner, and they're in.
Maybe make it double-sided - user name and password.
But, but, they can and must change their password to the one you gave them PRECEDED by a 4 digit PIN that THEY choose.
So, the user now types in their PIN and THEN flashes the password at the card reader.
...because a lot of barcode scanners key in the code that they read like keyboard input, and then press Enter.
I love this!
No cohersion, you are free my fellow end-user, IT is your friend, do as you want.
Same effect as checking the change-required box, but on a volutary basis.
Is Joseph Goebbels your favourite reading by any chance?
Keep them coming Trevor, you must be the most read author by far. Reality journalism is here.
I think I need time to digest that. I abhor reality TV with a passion I can't properly describe. Am I truly participating in reality journalism? Have I committed some form of unpardonable sin?
"Most read author?" I honestly have no idea. I don't get any stats, so I fly blind except for the scathing (though occasionally nice) comments from El Reg's famously cynical commenttards.
I can’t see it though. That could my bias as a dedicated reader of El Reg for over a decade, but I really don’t think I’m quite as good as the regulars around here. I would point you in the general direction of Lewis’s DARPA battleboffinry descriptions as a fantastic example. In time though, I hope to learn enough to be able to write at that level.
As for my favourite author, actually, it’s Tad Williams.
My usual method is when moving between boxes to copy the existing passwords and then expiring them immediately. That way the user knows their existing password and has to set a new one of their choice.
If users are complaining that they have to change passwords then there is a user expectation issue which needs managing. I have to admit though its easier in the finance industry. You just make sure that this is tagged as a compliancy issue and that the FSA will frown down on anyone not obeying and most just do it.
"all non-administrative users were set to the same default password"
Wouldn't the obvious Windows enhancement be a "login as user:" option once you have validated yourself as "Administrator"? 'sudo' comes to mind in the *nix world. The root of the problem is each user has personal preference/config data that, AFAIK, CAN NOT BE CHANGED except by logging in as said user and ticking the boxes for IE9, removing/adding toolbars on the start-bar (who's idea was throwing the language bar on the taskbar by default when installing Office07?), etc, etc, etc.
Fortunately, some relief for this has come to us through Group Policy Preferences, but it's still not enough.
Amen brother. You have hit upon the purpose behind the entire exercise. There were some things we simply couldn't accomplish with the changeover except by logging in as those users and setting their profiles up for them. What folks with the anger-making don’t see to get is that had everything gone to plan, we would indeed have reset the users’ passwords and forced them to change upon first login that Monday.
Things went all pear-shaped when folk started showing up on the Monday…and oh, damn…we weren’t done customising their VMs. By the time we got done with the customisation, we were deep into Tuesday. This means the users had logged in for a couple of days with the “new” passwords they were given.
Going around and forcing a password reset on the Wed morning, after Monday and Tuesday had been filled with printer issues, incompletely-customised VMs and changes to new and bewildering versions of software like Office 2010 would have been heap bad juju for IT.
The solution in the end was to phone the users one at a time and walk them through the password changes whilst taking the time to hear out any complaints they had, teach them any bits the needed to know about the new software and help them customise Windows 7 to behave less new and scary.
The sheer APATHY towards the “changing passwords” part of that phone call shocked me however. Users really, really don’t give a damn about security. Enough that I am starting to come round to the opinion that they honestly do need a periodic kick in the ass about it.
Unlike Unix, Windows administrators don’t have the opportunity to completely abstract the security away from users, nor do they have the control that a Unix admin would have to customise profiles and the like from afar. This means some level of user cooperation is required in the Windows world. When you run up against the sheer user apathy we did…that’s an eye opener worthy of an article.
Trevor, your experience brings back painful memories of a few complete rebuilds I've gone through, some scheduled, some not.
The inability to test everything in advance is probably the top factor causing a high risk of things not going well. The other is lack of support from management--probably the biggest problem we faced. We received little more than abuse from senior management (quote: "You don't need any help; you don't do anything anyway." From a Board of Directors member.) and instructions to accomodate whatever the users wanted. This was in a company of less than 1000 employees, where people had no reservations about calling one of the co-owners and complaining.
We tried the same password for all users, and not forcing changes; I have to agree with those who say this is a bad idea. It didn't work for us, either. Prepare a script in advance to use something like their employee number as part of a password, so they're all unique, but you don't have to actually send passwords to them. Something like "ABCIncnnnnnCBA".
We concluded that sending email to managers and tagging them to respond when the message was read, then calling the ones who _didn't_ read it, was about the only way to be effective about getting the word out. "If your employees don't read this, they won't be able to work Monday." tends to get the managers attention. Having a notification sent out a few days in advance by the CEO or President of the company also helps--if they'll send it.
I also don't think you had enough help. Bringing in several more people who could handle tasks like rejoining a system to a domain and configuring printers would have made life much easier for you. Again, this assumes management will allow/pay for the extra people. (No, ours didn't.)
Congratulations on surviving; these are rarely smooth and never fun, particularly when you can't get a budget for the tools that would simplify it.
netgeek
Bringing in other help. THAT is a whole other story. The interesting part is that I did try to get a buddy of mine who runs a computer consultancy to lend a hand (for appropriate remuneration) during the op. He bailed at the eleventh hour adding another layer of fun and happiness to this entire exercise. Additionally, another individual I was hoping to be able to talk into assisting got distracted by Starcraft 2. (He was at the time considering pursing IT as a career only to change his mind and head towards electrician in the last couple of weeks.)
So what I was seriously hoping was going to be a five man operation became a three man operation. Then Xerox got tripped up on the printer delivery due to backordered paper trays (sonofa…) OCS blew up. The Spam server ate half a night because I typoed the domain name. Office 2010 had some ridiculous upgrade guarantee thing that made getting the 2010 key from our 2007 installs a screaming nightmare and took five times as long as we figured it would. We discovered too late that Firefox add-ons are USER SPECIFIC and didn’t have time to figure out how to push those through a GPO, so ended up installing them manually.
To top it off, we had two different industry-specific applications that absolutely required user-by-user configurations and took about an hour each. There is no facility whatsoever for centralised deployment and configuration of those applications. Throw into the mix that each user is a “special case” with their own unique set of software requirements and you can’t just roll out one base image and be done with it.
It’s so very easy for people looking at something like this in hindsight, or reading about only part of the reasons and events that occurred to spew vitriol and negativity about the whole thing. Actually working in an SME environment like this is a completely different story…something I hope that my blog articles can help convey.
Depending on how email is used in your environment standard globals may never get read, long explanations, although they indicate a willingness on your part to engage fully, are probably not as good as very short and to the point.
Try using non-it communication methods. If you have a limited number of sites try big banners with a "countdown to "Doomsday Monday"" (although probably not with Doom in the title). If people have regular team meetings invite yourself along and do a very short briefing. If you have the budget send them all a jokey postcard, if you don't have the budget for that send them a jokey bit of A5.
In environments where users are becoming IT (and particularly e-mail) jaded some old-fashioned methods might just be different enough to raise a bit more awareness.
I visited two out of three remote sites ahead of the change. I talked about the change with folks on the third. I made the changes known in meetings as far back as January, though the exact details of course were far from concrete at that point. While I certainly didn’t have “details” on what was going to go down (that is what the e-mails were for,) the general notion that “something big involving computers” was going to occur in mid August has been known by all parties for the entirety of the year.
I believe the term “network overhaul” was used more than once. I am absolutely certain I mentioned new virtual machines with Windows 7, Office 2010 and a new version of Communicator. The printers bit crept up at the 11th hour. (Old printers died.) Overall, if there was one major issue that could have been thrown down as my failure, it was that I didn’t have this all planned six months early.
My only excuse for that is lack of manpower and budget. Until the 11th hour we weren’t 100% sure what hardware resources of software licenses we would have available to us. We had a general idea, but no plan survives contact with the enemy. In addition to planning this rollout this year, I had to keep the entire network running (putting out fires) do research for the rollout itself, do testing/prototyping/etc. and we rollout out a complete desktop replacement of Wyse thin clients in June/July as well.
Somewhere in there we also completely redid the website and tacked on an e-store. I had to redo the spam server. Again. (http://www.trevorpott.com/?p=275) Which is more of a project (research wise) than I like to admit to. There was a major upgrade of the industry-specific software package (that hasn’t been fully completed yet, actually…) and a review/overhaul of the phone system. Oh, and we replaced all our Treo 700Ws with blackberries and switched providers.
ALL of that had to be planned, sourced, tested and implemented before September. It’s also not abnormal. This is what it’s like every year here. I will not spend the next month and a half cleaning up any handing threads from those projects. Towards the middle/end of October, I do a network audit. What needs done in 2011? Then I start putting together proposals. January brings budget meetings and we do it all again.
Did I fail miserably in making the users fully aware of how tough this overhaul was going to be? Yes. I deserve a good smack on the wrist and a “do better next time” for that. I can’t say I didn’t try; I simply wasn’t experienced enough to succeed.
So good, I have registered on the register for the first time in 12 years of reading to contribute!
Management buy in is the most important thing in these situations. I especially like the idea of e-mailing managers with read receipts switched on. That way you can track the inevitable majority who don't bother reading your e-mail and who need a more direct interaction.
But in terms of getting users to submit to password security policies, that is actually very easy. You simply flip the script. Rather than making it seem like an edict from those "geeky, jumped up freaks in IT", you sell it. Remember secure passwords are for THEIR protection. After all a compromised password can really put an end user in the merde. Passwords are not only used for securing resources but as part of audit accountability. If Freddy Feasy has been writing his password down on a bit of paper, sticking it to his monitor and using it for the last 3 years, when the inevitable data breach happens, the sacrificial lamb tossed to the prosecutors or discipline board will be pretty obvious. This is particularly effective with those users who INSIST on sharing their passwords with significant others.
But if the carrot is your preferred educational tool (and you have the time and patience) I would recommend demonstrative Password Security training. The reasons for this are simple. To most end users, passwords are a barrier between them and their computer. It's only after they see "the most difficult password they can come up with" cracked in 0.2 seconds, that it suddenly all makes sense. Then all those special characters; numbers; upper-case and lower-case letters; maximum password durations and preventions of reused passwords make a little more sense.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
