back to article Rogue SF sysadmin coughs up passwords

San Francisco City Council regained access to its own computer network today after Mayor Gavin Newsom convinced network administrator Terry Childs to give them the passwords. Childs is in jail until he can raise $5m in bail. He is accused of blocking all access to the city's network and routers by resetting passwords. He …

COMMENTS

This topic is closed for new posts.
  1. Mark Lockwood

    Passwords

    For "After initial confusion" read "After they turned off the Caps Lock"

  2. Andraž Levstik

    "become a bit maniacal"

    I guess he got the BOFH upgrade recently :)

  3. lansalot
    Joke

    you see...

    This just wouldn't have happened if he had written the passwords for everything on a post-it note and stuck it above his desk - like everyone else does....

  4. Anonymous Coward
    Anonymous Coward

    I wonder what the password was ?

    Give me a F&*(ing payrise you bastards ?

  5. Anonymous Coward
    Anonymous Coward

    Wrong guy is in jail

    The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed.

  6. TeeCee Gold badge
    Joke

    Caps lock?

    Nah! That wouldn't be "confusion" that would be a "hardware error". I reckon it's that they had trouble interpreting the Mayor's handwritten version of "Th3M4y0R1S4f5ckw1T".

  7. I. Aproveofitspendingonspecificprojects
    Unhappy

    After they turned off the Caps Lock

    How do you do that then?

  8. Anonymous Coward
    Anonymous Coward

    I don't know what kind of computer system they use...

    ...but surely they had at least two options:

    1. (Not recommended, but workable) Get some people off the net who are penetration testers to hack back into it.

    2. Call me naive, but I'm sure that most OS's have a kind of recovery mode where if you have physical access to them, you can boot them up manually and log in and override them. (E.g. if on a Linux machine you accidentally forget the root password, it is possible to force a certain kind of boot that you can log in and reset the password). Not necessarily so workable for the routers perhaps but still definitely possible.

    The only other question this begs is whether it will now give the next generation of terrorists a new idea on bringing down the establishment, whichever establishment this is.

  9. Philip Teale
    Happy

    Re:Passwords

    Mark: Best comment ever on El Reg!

  10. Echowitch

    Hmm

    "Childs is accused also of installing hardware on the network to enable remote access."

    Could this possibly be so as to remotely access the network and fix problems from home out of hours, rather than have to get up, get dressed and travel X amount of time to come in to the office to do something that could potentially take 5 minutes to fix ???

    Sounds like he's a bit of a belligerent BOFH who doesn't like the bosses interfering in how he runs "his" network. And in this case they've totally over-reacted !!

  11. Lars Silver badge
    Happy

    funny?

    I cannot se how having access to the harware loosing passwords could be such a big problem.

    I once hade to take back a Unix machine from a customer who had stopped paying for the machine.

    Asking the boss for the root password he smiled and said "sorry I just forgot it".

    I could have left it at that but I had to boot the machine from a floppy, mound the HDD and erase the root password.

    The boss did not smile anymore.

    There must be ways to deal with Windows too.

  12. Anonymous Coward
    Heart

    Obviously not Windoze then

    As any sensible Desktop Support Operator knows, all you need to do is talk nicely to your nearest (insert flavor of Unix here)-using geek and (s)he will be able to furnish you with a password hacking tool... sorry, emergency boot disk.

    Anon as I'm at work and the Big Bosses would go uber-ballistic if they realised just how fekkin stupid we really think they are.

  13. Slimey
    Paris Hilton

    You seriously telling me...

    That they couldn't find a hacker in the Bay area, if not California that could crack the passwords? Instead they go pleading to the culprit?

    Clear case of incompetent bureaucracy.

    SF is a BIG city so their budget must be large enough to suggest he had a team rather than be working alone - what were they doing while he was setting all this up?

    I was torn between the S&C (a hacker could have sorted them out) and Paris. Paris got it in the end (oooerr) to represent the administration...

  14. alistair millington Silver badge
    Thumb Up

    @Mark

    Or they realised the I was a 1...

  15. Anonymous Coward
    Anonymous Coward

    Remote Access - Huh?

    Sorry, but don't most Sysadmin's have remote access to the stuff they manage so they can get to it out of hours.

    There's a whole load of questions here, not least around the city's governance procedures, if they have them.

  16. Neil

    "Convinced"

    In this hand, I have a brick. In my other hand, I have another brick. See these two meatballs? Now, passwords please...

  17. Steve
    Happy

    Resetting passwords

    "He is accused of blocking all access to the city's network and routers by resetting passwords."

    I think we all know what this means - the passwords were all "admin" or "password" and no-one in SF thought to try them.

  18. Anonymous Coward
    Anonymous Coward

    @Passwords

    Couldn't read their own writing...

    "Is it an 'o' or a '0'?"

  19. Destroy All Monsters Silver badge
    Coat

    He's NOT good at it.

    Otherwise those passwords would have been easily available at a safe nearby.

    I know what the back of people who are "good at it" looks like...

  20. Al
    Black Helicopters

    They couldn't have recruited a hacker because....

    ... aren't they all 'terrorists' now? It's probably a lot easier all round for the city authorities to lock up one bloke until he tells them the password, rather than prove that an outside hacker could get through their security.

    Pretending that access to the system is impossible without the correct password gives the impression the system is, if nothing else, impregnable to unauthorised users. Getting someone else to hack in and set it right would have the US press howling in full-on 'Chicken Licken' mode that any 'terrorist' could have done the same - cue the banning of 'War Games' and every IT professional going on a 'no fly' list.

    My money's on the mayor telling our man that they'd already got in, but the trial would go a lot easier if the fiction was maintained.

  21. Paul Rogers
    Linux

    @Mark

    Or were using a Mac (fanboys or technopleges) to log in and the password had a # in it.....

  22. Dr. Mouse Silver badge

    "become a bit maniacal"

    What, like a politician by any chance?

    And I agree, the initial confusion was probably misspelling, leaving the caps lock on, or general stupidity. And as for remote access, I also agree that it was probably for remote admin so he could do his job better. I have left back-doors open into systems when I have been admining for just this purpose.

    Of course, I am an ethical man and have always closed them up when I left the job ;)

    God save us all from eejits, erm, I mean users.

  23. Anonymous Coward
    Anonymous Coward

    Cisco?

    Didn't the original story say this was Cisco kit? As long as you have physical access to the kit you can recover the password on most Cisco kit.

  24. Anonymous Coward
    Joke

    After initial confusion

    that'd be the 1 in c1sco then?

  25. Anonymous Coward
    Happy

    routers only, not any servers

    He was in charge of WAN routers, all Cisco gear, and the passwords were all for those routers, there were no servers nor any desktops involved.

    Apparently, the Ciscos were configured such that password recovery was turned off, or something like that. This was all in an online article a few days ago where another IT guy working there gave some further details.

  26. Anonymous Coward
    Anonymous Coward

    He's not a BOFH

    As usual, the media got this wrong, he's not a sysadmin, he's a network administator.

  27. Jon Green
    Paris Hilton

    "Many have questioned why Childs' bail is so high"

    Well, duh!

    "Give us the passwords, and we can talk about cutting the bail to something sensible. That is, if you want to have a last little bit of freedom before all this becomes your second home. You do, don't you? Or have you come to enjoy Big Bubba's night-night 'cuddles'?"

    Paris could have worked that one out for herself.

  28. dodge
    Dead Vulture

    The inside skinny

    I can't vouch for veracity of this, but here is apparently the inside story...

    (from infoworld, linked by geekpress.com)

    http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/07/18/30FE-sf-network-lockout_1.html

  29. Mark
    Thumb Up

    Airline trip

    I'm surprised Homeland Security didn't fly him off somewhere and have the sh*t beaten out of him.

  30. frymaster

    re: Wrong guy is in jail

    "The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed."

    Agreed, because one person's incompetancy excuses another person's willful damage.

    ...oh wait, it doesn't

    Not disputing that in the aftermath of this, the IT manager should be investigated and at least reprimanded if not sacked or sued, but I don't see why that means the other guy gets to go free

  31. ben

    Get a life

    Hasn't this guy got anything better to do, if he doesn't like the job, leave, forget about it and get on with stuff. He must have had a massive complex about this position in the company and needed to feel powerful. That's what being a network administrator does to you... No life and his only friend the computer, looser.

  32. Echowitch
    Paris Hilton

    I may be rusty but.....

    I used to be responsible for Cisco password security at a rather large multi-national many years ago and we had numerous cases of Network Engineers setting up routers and forgetting to update the password file. (Wonderful flat text file available to some 500+ users who could easily copy it to floppy......I know as my Manager and I did once. Left the building, went to lunch, and no one knew. Informed the 3rd line manager and he just grunted at us.)

    As routers with lost passwords were at customer sites we had one of two options to recover them.

    1. Use the Cisco Configuration Tool for dragging back the config, editing it, and then uploading it to the router again. (Cisco wouldn't allow us to have it, but we had the IBM versions which worked great.)

    2. Send an engineer to site at a cost of £100 per router and get them to manually locally download the config to their laptop, reset the passwords, and upload the new config.

    Surely they could have done the above ???

    Even Paris could have done better.

  33. Anonymous Coward
    Happy

    Re: routers only, not any servers

    Yep you've hit the nail on the head - the guy disabled the password recovery mechanism which locks out access to ROMMON which would be the only way of traditionally recovering the hardware (the config is destroyed regardless). Basically this guy had the keys to the kingdom.

    Whilst it is obviously crazy that all of this was entrusted to one guy (what if he died unexpectedly?) based on my experience of configuring Cisco equipment for corporates I would say it wasn't that unexpected for one guy (or girl) to end up with absolute control over the network. Suits seem to generally only care about the network staying up, not the particulars of how it is administered, until - of course - the s**t hits the fan.

  34. Stephen Usher
    Alert

    Paranoia....

    The problem was that the sysadmin was paranoid.. to the point where he wouldn't even write the router configuration to the router's flash memory. (Yes, if the power failed the router would lose its configuration unrecoverably. Maybe it was safe from hackers but it wasn't safe for hardware failure.. stupid sysadmin!)

    Apparently he didn't give anyone the password or write it down because he didn't trust them.

  35. David Cornes
    Paris Hilton

    Case?

    After all this, I'm confused as to why he's still pleading NOT guilty...?

    Paris, 'cos she wasn't guilty either, just a little confused.

  36. Sceptical Bastard

    Bail?

    Give the bloke a medal for pwning SF thereby showing up what was obviously negligent network administration and management.

    I wonder if Mayor Boris's outfit uses Cisco kit ...?

  37. Anonymous Coward
    Joke

    re: get a life - ben

    he probably used to post on TheRegister style comments pages as well

  38. Egons Proton Pack
    Paris Hilton

    If the film Hackers has taught me anything...

    Its that all admin passwords are either GOD or SEX....

    I wonder if they tried those?

    Paris cause i reckon all her password are related to sex...

  39. Anonymous Coward
    Joke

    How they found the passwords,...

    the Abu-Graib way.

    http://www.catb.org/~esr/jargon/html/R/rubber-hose-cryptanalysis.html

  40. Danny
    Linux

    @Mark Lockwood

    >For "After initial confusion" read "After they turned off the Caps Lock"

    ROFL!

    Real keyboards don't have Caps Lock...

    http://www.pfusystems.com/hhkeyboard/hhkeyboard.html

  41. Danny Silver badge

    (different Danny)

    I got a SysAdmin job once where the previous guy had been fired. After a week of getting to grips with the kit I still hadn't found any root passwords for the comm's equipment - and there was a lot of unexplained traffic. So I had to open up the boxes, remove the batteries. Now the previous guy had been quite a bit more techie than me, and had not only kept full access to the system, he'd rewritten the drivers for some of the kit. So I had to download new drivers offsite and repeat the process. All of which took downtime that I was blamed for - after all, the last guy never had these problems! I got so much grief from users and management I regretted not just leaving the guy full access and keeping my mouth shut.

  42. Mark

    Re: re: Wrong guy is in jail

    But I thought managers were paid more because they were in the "risky" positions. Ones requiring the RESPONSIBILITY of the actions of their subordinates.

    Or is that a load of pigshit?

    Yeah. The latter.

  43. Anonymous Coward
    Thumb Up

    A quick lesson,...

    Why he is there now,..

    Middle Manager: The network is unmaintainable while only you hold the passwords and configs. Please arrange to document these in a suitable manner for other staff.

    Senior Engineer: No, I do not believe you or any of the other staff have the necessary skills to maintain this network.

    [Lots of back and forth]

    Middle Manager: Last chance, documentation or suspension.

    Senior Engineer: Suspension.

    [More waiting]

    Middle Manager: Passwords and config please?

    Senior Engineer: No

    Middle Manager: Last chance, documentation or incarceration.

    Senior Engineer: Incarceration.

    [More waiting]

    Middle Manager: Passwords and config please?

    Senior Engineer: No

    Middle Manager: Last chance, documentation or prosecution?

    Senior Engineer: Documentation

    [Try passwords]

    Middle Manager: Proper passwords and config please?

    Senior Engineer: No

    Middle Manager: Last chance, proper passwords or prosecution?

    Senior Engineer: Proper passwords

    LESSON: All Senior Engineers are still only cogs in a larger machine.

    Why he did it,…

    Middle Manager: Please provide passwords to Junior to allow him to make changes.

    Senior Engineer: Those changes are outside his ability to perform, and are an unacceptable risk.

    Middle Manager: I don’t think your job is as complex as you make it out to be. Passwords please.

    [Receive passwords]

    Middle Manager: Junior, please make this network change with the passwords I have provided.

    [Network crash – 36 hours for Senior Engineer to recover]

    Director: What the heck happened last week?

    Middle Manager: Senior Engineer made a mistake, despite being told it was not a sound change to make.

    LESSON: All Middle Managers are cnuts.

  44. Anonymous Coward
    Anonymous Coward

    Initial Confusion..

    The proper BOFH response..

    "OK, OK.. I'll tell you - the password is the Mayor's wife's first name and the surname of his favourite hooker."

  45. Edward
    Thumb Down

    Password Recovery.

    Assuming the Hard Disks aren't encrypted, with physical access to the machines you can:

    Windows:

    Reset the Local Machine and Active Directory passwords by modifiying SAM

    Extract hashes from SAM and crack the passes using Rainbow Tables.

    *nix:

    Reset the passes by modifying /etc/shadow.

    Crack /etc/shadow to get plain-text passwords.

    I'd put money on the HDs not being encrypted, its a drawn out, expensive process with very little actual ROI.

    Who wants to bet this chap is one of, if not the only person managing the system. He probably set it up as well. This is a storm in a teacup, exacerbated by the City's unwillingness to properly staff their infrastructure.

  46. Mitch Russell

    Odds are that the password was one of these:

    password

    cisco

    ******

    foobar

    iknowitandyoudont

  47. Anonymous Coward
    Stop

    Network DOWN!

    I make the following prediction:

    Now the dullards in SF have the passwords the fibrewan network will work no more.

    Up until Childs handed over the passwords the network was working great, you just could not make any alterations to it. Now the city has the passwords some PFY will be given the job of making an apparently minor change that will result in partial or total breakdown.

    Mark My Words, your Doomed SF!

  48. Anonymous Coward
    Anonymous Coward

    @ Edward

    You failed to read all the information. The passwords withheld were for Cisco WAN routers (neither Windows nor *nix) which had been configured with password recovery disabled. If they had performed a hard reset on those routers, then they would have wiped the configuration, their WAN would have stopped working. And the only person who had the knowledge to configure that gear is the guy who is in jail. Catch 22.

  49. Ian Michael Gumby Silver badge

    BOFH in training?

    One has to ask what was he thinking?

    Of course had he read the entire saga of BOFH, he wouldn't have made the mistakes that he did.

  50. I. Aproveofitspendingonspecificprojects
    Coat

    SF Cisco

    What does SF stand for? SanAndreas' Fault? Send fail? Systems failure? Sentry fled? Soft Fu....errr ....geddit.

  51. Gav
    Boffin

    Partial Info

    This report shows only half the story. This Admin didn't steal or re-set any passwords. He had possession of the passwords from the very start and he, and he alone, knew what they were. His bosses demanded them off him, he decided they weren't to be trusted with messing with his network and said no. They fire him. He still says no. Bosses call police.

    Sounds like an over possessive admin, but maybe he had his reasons. Either way, this is a whole different story from the way its been presented. He wasn't a rogue admin, he was the sole admin, which also makes his bosses liable in a number of ways.

  52. Marc

    So the password is 1, 2, 3, 4, 5?

    That's the stupidest password I've ever heard in my life! The kind of thing an idiot would have on his luggage!

  53. pctechxp

    Childs and the PFY

    What'll we do tomorrow, Terry?

    The same thing we do everyday, PFY.

    Plot to take over the world!

  54. I. Aproveofitspendingonspecificprojects
    Happy

    SFSico II

    From a link posted about it earlier:

    Sole administrator

    A key point made in the e-mail is that Childs' managers and coworkers all knew that he was the only person with administrative access to the network. In fact, it was apparently known and accepted in many levels of the San Francisco IT department. Again, quoting from the e-mail:

    “This is where it gets tricky for the prosecution, IMO, because the localized authentication, with Terry as sole administrator, has been in place for months, if not years. His coworkers knew it (my coworkers and I were told many times by Terry's coworkers, 'If your request has anything to do with the FiberWAN, it'll have to wait for Terry. He's the only one with access to those routers'). His managers knew it.

    "Other network engineers for the other departments of the City knew it. And everyone more or less accepted it.

    "No one wanted the thing to come crashing down because some other network admin put a static route in there and caused a black hole; on the other hand, some of us did ask ourselves, 'What if Terry gets hit by a truck?' If a configuration is known and accepted, is that 'tampering'?”

    My source appears to believe that Childs' motivation was the antithesis of tampering, and that Childs did everything possible to maintain the integrity of the network, perhaps to a fault:

    “He's very controlling of his networks -- especially the FiberWAN. In an MPLS setup, you have 'provider edge' (PE) routers and 'customer edge' (CE) routers. He controlled both PE and CE, even though our department was the customer; we were only allowed to connect our routers to his CE routers, so we had to extend our routing tables into his equipment and vice versa, rather than tunneling our routing through the MPLS system.”

    http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/07/18/30FE-sf-n

    He shouldn't be in gaol at all!

    He should be free to hold the secrets he was entrusted with -until payment was made for the knowledge he was allowed to leave his employment with.

    You don't sack a man in that position until you have made him release his secrets. Once you stop paying him you can't expect any different.

  55. Doug Glass
    Flame

    Must be Gomer Pyle's Dumber Brother

    Surprise, Surprise, Surprise

    "Many have questioned why Childs' bail is so high and how he apparently so easily gained complete control of the city's computer system."

    The very fact they this sort of activity still brings surprise should tell us something about those who confess to not understand how easily it was done.

    Idiots who are surprised at this must also be surprised that babies puke and pee.

  56. Robert Moore

    @Marc

    That would be the combination to the air shield.

  57. James O'Brien
    Paris Hilton

    I know what the password was

    ÅÎÎΜ@ñÀg£ŗšǻřΈΤǒЅЅєЯs

    Seriously though I can see why he did what he did (known many retarded idiots who shouldnt be trusted with a spoon let alone network access) but at the same time once he left he should have turned it over right away if only to laugh when the new people made a royal mess of everything seeing as he was no longer responsible. Then again he would problem have still been arrested cause the new people would have denied they messed it up and blamed it on some sort of logic bomb that was hidden.

    /Paris because, well shes been as Fu#!ed as this guy would have been one way or the other.

  58. James Pickett

    Bargaining

    I'm surprised he didn't use his knowledge to gain a bit more leverage, such as a written undertaking not to pursue the matter. Or forget them, even...

  59. PunkTiger
    Alert

    I'd walk a mile for a calomel.

    I can only imagine how part of the conversation went...

    "What's the password?...And if you say swordfish, I'LL LOSE IT!!"

  60. Matthew
    Thumb Up

    real life BOFH story

    nice one mate.

  61. Anonymous Coward
    Anonymous Coward

    Is mayor a code word for sodium penthanol

    Normally it is a word with the Abbott ;), perhaps they don't have one in San Fran.

    So this story looks like this:

    Man with passwords gets fired, and they expect him to remember those passwords after the trauma of being fired, and hand them over?

    He has not done anything wrong, and his is on 5,000,000 dollars bail!

    It beggars belief that San Fookedcisco (as it is now known), did not have contingency plans in place. And one has to wonder if the person this guy reported to was the mayor, therefore only the mayor should have been the recipient.

    There potentially is one hell of an interesting counter claim case here. I hope he has played it by the letter, and I have a suspicion he has, who knows what their security docs say, perhaps even something like: 'you may not divulge any IT passwords of the system, except directly to the person you report to.' Those security docs, that 'people' always recommend you do first - may just have shot themselves in the foot with this one.

  62. tony trolle

    The mayor

    The only man with a higher rank that did not fire him.

    ALL started with an audit............

  63. 2Fast

    well....

    When Childs get out of prison id like to buy him a beer :)

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019