back to article Rogue SF sysadmin may cost city over $1m

The disgruntled sysadmin accused of locking San Francisco out of its IT network may cost the city more than $1m in upgrades, consultants and repairs to undo the damage, according to the City's Department of Technology. Terry Childs, a 43-year-old from the Bay Area city of Pittsburg, is accused of creating a super password for …

COMMENTS

This topic is closed for new posts.
Joke

Why am I reminded...

...of the Despair, Inc. demovitational: "If you're not part of the solution, there's always good money in prolonging the problem."

0
0
Anonymous Coward

Upgrades

Hmmm, not sure those can really be included in the figure of 'damages'.

Oi, you ran into my Fiesta, you will now buy me a Porsche.

0
0
Coat

Actually...

Consider that one of the jobs of a sysadm is to keep all data safe from misuse by the unauthorised and the incompetant, and that disclosure of a major password is actually a security breach, a felony in its own right as some might have it.

Sounds like this "rogue" was actually doing his job to me! Too many bosses try to treat their IT staff like shit...

Now the shit is on the other boot!

0
0

But locking out idiots is a great policy!

You'd have to be incompetent to NOT lock out idiots from your system. That is what passwords are for! It's the pointy-haired bosses who should be in the clink, not the admin.

0
0
Silver badge

One Million Dollars

Just goes to show that so much involved in IT is a scam. They've only spent ~$190k to resolve the issue - but now they want another $800k to spend on "upgrades" and "consultant" (often cousin Tommy), and flat panel monitors for the mid-managers. This is horseshit! There's no need for a password fix to take months to fix once you've got the password. The Admin should be publicly flogged then everyone gets on with their business.

0
0
Flame

The Admin is in the right

First, if you fire me, I no longer am obliged to tell you anything, you should have thought to ask before you decided to fire me.

Second, If I am still working for you, it is my responsibility to keep others from doing something stupid, therefore I cannot give out the password, lest you do something stupid, and I get fired for letting you have the password.

Finally, in situations like this you should always have two admins, in case something "bad" happens to one of them.

0
0
Paris Hilton

No crime has been committed. I See a Big Settlement Coming

So far the majority of offenses that San Francisco DTIS, city officials, the police, and the District Attorney have alleged against Terry Childs has been exaggerated, standard network engineering practice, or just plain wrong. The remainder is grounds for counseling, reprimand, or at the most discharge. Prison time? Please.

For those who are interested I highly recommend infoworld.com's Paul Venezia and his blog series on this event. I think there is a 90% chance that Terry Childs will win a big settlement from this case.

San Francisco officials at all levels are so far demonstrating worse judgement than even UK local councils, couldn't resist!

Paris, because she is WAY smarter than San Francisco politicians and DTIS managers.

0
0
Silver badge
Happy

Overtime pay!

Yeah, higher-ups figure there's a problem with the network. Go fetch a couple of pizzas, we will finally be able to play Defcon tonight. And bring me one of those forms for the overtime tallying, thanks.

0
0

The admin was wrong

K folks, I'm an IT admin, so I understand the impulse to circle the wagons, so to speak. But this guy is clearly in the wrong ( as well as his managers ). When you are working on a critical system, you document the heck out of it, including passwords. You make sure your boss knows where this documentation is too. It's part of being a "Professional".

I'm not clear on the details of this guy's termination, but I could very easily see him getting prison time. Not for not turning over the passwords, but rather blackmail.

Oh, and Mike? While the admin does have the obligation to protect the network, he does not have the right to refuse a request from his boss. If he were so worried about his job, all he needed to do was get the request in writing, along with his objections, and then fulfill the request. This assuming that he was doing his job wrong to begin with and NOT documenting passwords.

0
0
Flame

Re: The admin was wrong

Documenting passwords? Great, let's right them all down on post-its for everyone to see! Any IT manager that approves a system to rely on standalone authentication should be shot. Haven't these people heard of LDAP, RADIUS, etc?

Sadly, the real loser here is the locals in SF supporting this idiotic gov't with their taxes, because they'll have to fit the bill in all cases, regardless of who wins. And ultimately, no one will end up being responsible for the whole situation, even if the admin takes the blame legally. There should be some policitians in SF that bear some of this blame too.

1
0
GF

Two Wrongs Do Not Make It Right

Just because management is perceived to be screwballs, the IT admin should not act the same. Terry Childs may truly have major work performance problems and he may personally be a real as**oe with everyone he works with, and if so, his behavior is definitely inexcusible.

I had worked in government positions for some time, and there are employees who think they are indestructable and treat everyone like crap, while management act like sheep because their hands are tied due to employment rules and laws. These employees do not do anything illegal, so they could not get fired, but they treat every contractor like crap, launch "gernades" in almost every meeting to disrupt teamwork, and generally interfere with any manager's ability to manage. But yet they can still keep their job because there are government employees.

0
0
Boffin

over $1m ?

I was wondering how it was going to cost so much until I saw:

'plans to set aside $1m to pay for consultants and upgrades to the network.'

So in other words that would be $999,997:50 for the consultants and $2:50 for a new Patch cable....

I think the picture of the faceless, mindless CON-sultant fits best here

0
0
Anonymous Coward

On being a pro

being a professional means you get paid, that is all.

It doesn't mean you get to apply your made up ethics to everyone, it is just a matter of getting payment for something.

Oh and of course you have the right to refuse a request, were are not in Nazi Germany quite yet :)

They have to pay you up to the moment of your dismissal, they owe you not the other away around. If he wishes to refuse the information then that is his right, they pay up to the moment he leaves or they dismiss him.

It is no more complicated than that. What is going on here is an abuse of the legal system, he has not done anything morally wrong that anyone can see at the moment.

If the job required him to document passwords, which would have to have been laid down in a contract, then who is to say they were not documented, and has anyone seen the contract stating that?

I thnk you are just making things up, in business you would be a loose cannon, quite dangerous to work with, but I think in your mind you think your approach is correct - interesting.

0
0
Silver badge
Joke

Amerikan vays ov confincing zem do dalk...

"He was eventually convinced to cough up the correct code, when San Francisco Mayor Gavin Newsom visited his jail cell."

'Cough up'... a nice way of putting it. I assume a little spot of waterboarding didn't go astray during the Mayor's little chat with this fellow...

0
0

Your Nuts

This guy is in NO way criminally liable. Have you done ANY reasearch into this case. The network admin tried THREE TIMES to have HIS superiors come up with a disaster recovery plan. He even wrote one. He WANTED to document everything. Management shot it down because they knew their asses would be in a sling if they had a plan on paper and something went wrong from something they ordered the admin to do.

You do realize they promoted a help desk type person to Security Manager. The first thing this manager said is give me the keys to the kingdom. This Admin was responsible for the Police, Fire, EMS and government networks. How responsible do you think it would be to hand over the keys to such a raw cadet? What do you think this manager would do with the passwords?

When the mayor came and spoke with him, the passwords were freely given.

This is a clear cut case of a good IT worker, probably a little eccentric, who was severly overworked. When the new "Security Manager" showed up and could not get the keys to the kingdom she trumped up a poor performance report.

Anyone who does not see this does NOT work as Systems or Network Administrator. We are treated like the power company. We are only called when something is not working. We are never called and thanked for keeping the systems working trouble free.

0
0
Stop

Re: The admin was (supposedly) wrong

"he does not have the right to refuse a request from his boss"

Bollocks! His obligation is to the quality of service provided to the customer, not to the twat in a suit ranking above him! Once you get above the basic "please perform this narrow ranging task" mode of work, and into the "please provide infrastructure solutions" mode, you get into politics, where spin and blame-sharing are rife and the customers interpretation depends on who tells him what and how they are pandered to, NOT necessarily the actual facts.

The admin here is in the difficult position of knowing what is wrong, but not being in the "circle" of those who distrbute information about what was wrong (and who can therefore influence the facts by ommission or outright falsehood). In this case, he has chosen to escalate the issue to the point and place (court of law) where he can reasonably be expected to put forward his side of the story without "taint" from the PHB. A risky proposition, if he is wrong or presents his arguments badly, he can get sent down, but he also has the chance to have his decisions vindicated.

True Story: I was against a junior member of staff implementing a large print solution but was ordered to allow him/her/it to do it anyway, upon veiled threat of my position as Team Leader. It failed, caused outage and I spent ages fixing it.

About 2 months later, an off-hand comment from a user in a meeting about me "not dropping the ball like the last time we changed the print system" lead to me finding out that the boss told the users "I organised the implementation of that change", neglecting to mention he over-rode my professional and technical opinion.

If I ever meet that fat b'stard in a dark alley,....

0
0

Keeping stuff safe...

How can he be keeping stuff safe if he was introducing new risks to the system. That password should have been in a safe somewhere that someone else had access to in case the sysadmin got hit by a bus.

0
0
Coat

I think some admins may have an inflated opinion of themselves.

You guys acting like he was the champion of the system.

Making out he's the hero protecting the computers from the Idiots.

He's an Admin! He greases the wheels and makes sure the system keeps working. Just like the cleaners, and the janitors (except the plumbing is a bit more complicated). These are the systems that enable the real workers to actually do the work.

I'm a developer so I understand the need to overinflate our own importance but in an environment such as the case above the networks are not the reason for the organisations existance. Just a utility.

0
0
Joke

Why Managers should not get superuser access

"....is accused of creating a super password"

and to hide it he no doubt chose the username "administrator"

They'd be better off spending that cash on some education courses for the current crowd of idiots.

0
0
Anonymous Coward

@Sean Kennedy

"While the admin does have the obligation to protect the network, he does not have the right to refuse a request from his boss"

Um, would this be a request from a boss after he was fired? If so, then he's no longer got a boss to refuse requests from.

If you are going to fire someone you get all the information you need to know to continue performing that person's job *before* you sack them, that's rule of letting staff go number 1. Not complain that the worker didn't document things for his replacement after the event. The workers boss is the person responsible for the lack of documentation for not making sure it got done, not the fired worker.

0
0
Dead Vulture

Only on the Reg...

...will you find people defending this prick. If there's a clearer case of a petty little gobshite abusing his power... well, I'll let you fine people add the punchline to that one.

0
1
Paris Hilton

Agreed - The admin was wrong

Far too many posts in this comments section are heavily biased towards IT admins who believe they hold their companies togeather... clever you!

IT Admins are payed to keep the network secure and working for the best interests of their employer. Witholding passwords to a system because you have been fired is the IT equivelant of throwing all your toys out of the pram.

Seriously, what happened to intelligent deduction of facts round here!

0
0
Ash

@Caffeine Junkie

Good analogy with the Site Staff.

Would you let your PHB fix a leaking gas pipe? Service the heating system? Would you even trust him with your (rather expensive) Mikita 18v drill?

There is a good reason these people tell you to Arkell Vs. Pressdram; You don't know what you're doing, and you could very easily break something vital.

This guy did everything right except not hand over the password to the Police. In the UK, that's an instant 2 years prison. I'm not sure how it works over there. That's the crux of the "blackmail" case.

0
0
Alert

Locking out idiots!!?

WTF are you thinking!?

It's the IDIOTS that cause all the 'problems' that provide all the WORK, Duh!

No Idiot users = no work = no jobs for sysadmin!

Can't believe they've only spent $15k (£8k) in O.T. on this. How many people are/were working on it, two, three?

0
0
Pirate

@Caffeine Junkie - Developer

And I have lost count of the number of times a twerp Developer has come up with a messiah complex asking for complete control / casting vote on all infrastructure changes, rather than expend a little effort to change their program. SAP Basis admins are especiallty good at this. (and yes, I cut code for 8 years before I moved to being a sysadmin).

The battle for best fit of an IT solution goes ever on.

0
0
Flame

Yes, documenting passwords!

Doesn't mean writing on the back of your notebook or on a post-it under the fecking keyboard, you gonks! You can really tell the cowboys around here, can't you?!

Document all your root and admin passwords, seal them in envelopes, give them to the company security officer and then they are stored in an offsite firesafe. At my site you need two upper managers and security officers approval for a root password, then sign the password approval register which the CTO and his PA hold, it is countersigned by one other authorised person, not the security officer. All the admins have their own admin accounts and they are all audited, the audit records stored in a software vaults which only a handful of people have access to.

That's paranoia at work, but at least I know that I cannot be blamed when some numb-nuts decides to go the IT equivalent of "postal" and hold an entire infrastructure to ransom!

0
0
Boffin

@ Agreed - The admin was wrong

Jon, I wouldn't hand out passwords outside of a contract, that were not immediately related to my current job (match "!=*" if unemployed). I would interpret that as a breach of my non-disclosure agreement, and refuse to give the information after my contract has been terminated. The wording of many contracts actually supports this interpretation quite well. Besides, it wouldn't hurt my security clearance to be tight-lipped when in doubt.

Yes, it would be pedantic - but anyone stupid enough to fire me BEFORE finding out the passwords I hold deserves nothing less. I'm quite happy to hand over all passwords (together with - not separately from, my responsibilities) - and to be escorted from the premises, in one atomic process. But, ask me for company information after I've left the building, and my standard response will be "I will not discuss that information outside a contract of employment with you."

By the way, many companies DO use a safe to keep passwords in, and they DO document company-critical passwords. The key is usually held by the company's Security Officer(s), not the administrator. Passwords should be documented (and kept in a secure place) - and their use constantly audited and controlled by Security staff (this is required for SOX compliance anyway, these days). I would even venture to say that any company that does not control its passwords does not deserve to remain in business.

0
0

Wow

I admit, I wasn't expecting such a response. But let's take it point by point, shall we?

1) He should have kept information from his boss

a) No, he shouldn't. He works for the organization, and the organization put someone else in charge. Them are the breaks. I will grant you that his bosses are almost as liable for not making him document things ahead of time. Further, if it really did shake out the way it's claimed ( he was fired and then asked for the passwords ), then there's nothing criminal about that.

2) You shouldn't document passwords

a) Maybe in a small mom'n'pop shop. In the real world, real admins aim to protect the network. That includes from you dropping dead, or more likely, moving on to greener pastures and forgetting to tell someone something before you go.

3) Professionals are anyone who gets paid to do a job

a) A professional is anyone who takes pride in their work and does the job right. I don't grant that title to a vast majority of IT workers out there; they are unreliable ( the afore mentioned ones who don't document passwords ) and untrained. Essentially, kids playing with their toys.

As an IT professional, I know the network doesn't belong to me ( as child's apparently didn't ). I know my purpose there is to increase productivity through the use of IT services and equipment. To that end, I do everything in my power to safeguard that primary purpose.

I do not throw a fit like a child because someone I didn't like got promoted above me. Hell, I don't even care if the only use they have for a keyboard is to drool on it. If they ask me for a piece of information that I failed to document ( which is my mistakes anyway ), then I give it to them. To protect myself, I will get everything in writing, so when the idiot deletes everything on the SAN I have documentation, but that's it.

But let me reiterate; I agree with one premise. If he was let go before the requests were made, it's not criminal for him not to give it up. As an employer, I wouldn't touch him, both because he is being stubborn and because he didn't document, but it's not criminal.

Were I the City of San Fran, I'd fire his handlers too for being so incompetent as to let someone like Childs bring the city to it's knees.

0
0
Dead Vulture

Re: Re: The admin was (supposedly) wrong (AC)

"True Story: I was against a junior member of staff implementing a large print solution but was ordered to allow him/her/it to do it anyway, upon veiled threat of my position as Team Leader. It failed, caused outage and I spent ages fixing it."

Your biggest mistake was not attaching ownership of this idea, clearly and firmly, to the junior member of staff - and never letting anyone forget (even for a moment) that it was his baby, until he botched it. By attaching ownership, you look like a team player by awarding credit and responsibility to a junior member of staff, in a world where so many bosses try to swipe the credit; by criticizing him, you made yourself look weak by not appearing to be a team player.

Bad move.

If you think something will fail - and you know you don't have the required political support from managament, sometimes the best course of action is to keep the information to yourself and set the guy up to take a fall. Yes, it's evil, Machiavellian and positively devious. It's called Office Politics 101 - and trust me, no amount of technical skill exempts you from having to play this game - as you just found out.

Some companies I've worked in demanded an intricate political bent for 80% of the workload - just 20% was related to satisfying customer requirements. I'm not kidding! Such companies also tend to be real snake pits, as everyone has their own agenda (and most people would like to see their friends doing your job, rather than you). Sad but true... it's a dog-eat-dog world we live in!

0
0

Idiot

A few things to say.

I have gone in and sorted out networks on contract before where the previous admin had "thrown a wobbly". Not to this scale, but with similar stunning moves on the part of the admin.

No way should he have locked out the network and made himself the single point of failure. That smacks of ego and not admin or engineer thinking. For years we have had it drilled into our head about redundancy and load balancing being good and then he goes right off and throws his toys out of the pram and makes himself the weakest link in the whole network (but with the most leverage). Earlier posters had it right, either have a backup admin or a key of some sort stored on a flash drive in a safe off-site. Even better, have both.

0
0
Pirate

Re: Oliver jones - response from AC

Yep, totally agree, get it on record about whose decision it was and who overrode who, by mail, meetings minutes or group discussion.

Except when the slimy sod in charge knows that and is always "taking it offline" (No witnesses) "too busy right now to reply to your email" (Avoidance of evidence of confirmation), follows up later with "I understand you are being inflexible in situations that requires dynamic intervention" (distraction from root cause). Followed even later by "I would hate issues like this to become your legacy here" (Veiled threat to give a bad reference after I left).

Office politics 102 - never announce ahead of time your intention to leave

So, no record on email, no record on paper, and if this is the way they treat the team leader, there is no way my staff can expect fair treatment if they side with me on this issue. And I never knew the Users knew the details until well afterwards, at which point any response I make sounds defensive or self-serving.

Office Politics 103 - get your retaliation in first, either by innuendo before the event (I wouldn't be surprised if....) or afterwards (pity about X doing....)

BTW, the junior involved socialised frequently with the Director of the Company and was hired at his behest.

Office Politics 104 - never go up against the "annointed one" of management, you will always lose.

Will write a whitepaper about all this one day.

0
0
Dead Vulture

Easy fix...

Enter the password into the court documents -- Public info. Every joe, tom, dick and sally would have it. You wanted the password, hell, yell it out during a hearing.

Run, run like the wind!!! Change all the passwords on all of the equipment!!!!

Bringing the courts into the mess to get a password was S T U P I D. I guess the need the 'consultants' to hook up a blue Cisco terminal cable and do the old password reset feature. It also sounds like Cisco or whom ever sold them 'upgrades' so 'this didn't happen again.' Time will tell. SF may be converting to Dark Fiber.

0
0

AC and password resets

Cisco has a feature where you can not do password resets. It's often not used because of how painful a lost password can be, but it is there. And it's bad mojo if you do lose a password with this feature turned on.

0
0
This topic is closed for new posts.

Forums