back to article Rogue SF sysadmin coughs up passwords

San Francisco City Council regained access to its own computer network today after Mayor Gavin Newsom convinced network administrator Terry Childs to give them the passwords. Childs is in jail until he can raise $5m in bail. He is accused of blocking all access to the city's network and routers by resetting passwords. He …

COMMENTS

This topic is closed for new posts.

Page:

  1. Mark Lockwood

    Passwords

    For "After initial confusion" read "After they turned off the Caps Lock"

  2. Andraž Levstik

    "become a bit maniacal"

    I guess he got the BOFH upgrade recently :)

  3. lansalot
    Joke

    you see...

    This just wouldn't have happened if he had written the passwords for everything on a post-it note and stuck it above his desk - like everyone else does....

  4. Anonymous Coward
    Anonymous Coward

    I wonder what the password was ?

    Give me a F&*(ing payrise you bastards ?

  5. Anonymous Coward
    Anonymous Coward

    Wrong guy is in jail

    The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed.

  6. TeeCee Gold badge
    Joke

    Caps lock?

    Nah! That wouldn't be "confusion" that would be a "hardware error". I reckon it's that they had trouble interpreting the Mayor's handwritten version of "Th3M4y0R1S4f5ckw1T".

  7. I. Aproveofitspendingonspecificprojects
    Unhappy

    After they turned off the Caps Lock

    How do you do that then?

  8. Anonymous Coward
    Anonymous Coward

    I don't know what kind of computer system they use...

    ...but surely they had at least two options:

    1. (Not recommended, but workable) Get some people off the net who are penetration testers to hack back into it.

    2. Call me naive, but I'm sure that most OS's have a kind of recovery mode where if you have physical access to them, you can boot them up manually and log in and override them. (E.g. if on a Linux machine you accidentally forget the root password, it is possible to force a certain kind of boot that you can log in and reset the password). Not necessarily so workable for the routers perhaps but still definitely possible.

    The only other question this begs is whether it will now give the next generation of terrorists a new idea on bringing down the establishment, whichever establishment this is.

  9. Philip Teale
    Happy

    Re:Passwords

    Mark: Best comment ever on El Reg!

  10. Echowitch

    Hmm

    "Childs is accused also of installing hardware on the network to enable remote access."

    Could this possibly be so as to remotely access the network and fix problems from home out of hours, rather than have to get up, get dressed and travel X amount of time to come in to the office to do something that could potentially take 5 minutes to fix ???

    Sounds like he's a bit of a belligerent BOFH who doesn't like the bosses interfering in how he runs "his" network. And in this case they've totally over-reacted !!

  11. Lars Silver badge
    Happy

    funny?

    I cannot se how having access to the harware loosing passwords could be such a big problem.

    I once hade to take back a Unix machine from a customer who had stopped paying for the machine.

    Asking the boss for the root password he smiled and said "sorry I just forgot it".

    I could have left it at that but I had to boot the machine from a floppy, mound the HDD and erase the root password.

    The boss did not smile anymore.

    There must be ways to deal with Windows too.

  12. Anonymous Coward
    Heart

    Obviously not Windoze then

    As any sensible Desktop Support Operator knows, all you need to do is talk nicely to your nearest (insert flavor of Unix here)-using geek and (s)he will be able to furnish you with a password hacking tool... sorry, emergency boot disk.

    Anon as I'm at work and the Big Bosses would go uber-ballistic if they realised just how fekkin stupid we really think they are.

  13. Slimey
    Paris Hilton

    You seriously telling me...

    That they couldn't find a hacker in the Bay area, if not California that could crack the passwords? Instead they go pleading to the culprit?

    Clear case of incompetent bureaucracy.

    SF is a BIG city so their budget must be large enough to suggest he had a team rather than be working alone - what were they doing while he was setting all this up?

    I was torn between the S&C (a hacker could have sorted them out) and Paris. Paris got it in the end (oooerr) to represent the administration...

  14. alistair millington
    Thumb Up

    @Mark

    Or they realised the I was a 1...

  15. Anonymous Coward
    Anonymous Coward

    Remote Access - Huh?

    Sorry, but don't most Sysadmin's have remote access to the stuff they manage so they can get to it out of hours.

    There's a whole load of questions here, not least around the city's governance procedures, if they have them.

  16. Neil

    "Convinced"

    In this hand, I have a brick. In my other hand, I have another brick. See these two meatballs? Now, passwords please...

  17. Steve
    Happy

    Resetting passwords

    "He is accused of blocking all access to the city's network and routers by resetting passwords."

    I think we all know what this means - the passwords were all "admin" or "password" and no-one in SF thought to try them.

  18. Anonymous Coward
    Anonymous Coward

    @Passwords

    Couldn't read their own writing...

    "Is it an 'o' or a '0'?"

  19. Destroy All Monsters Silver badge
    Coat

    He's NOT good at it.

    Otherwise those passwords would have been easily available at a safe nearby.

    I know what the back of people who are "good at it" looks like...

  20. Al
    Black Helicopters

    They couldn't have recruited a hacker because....

    ... aren't they all 'terrorists' now? It's probably a lot easier all round for the city authorities to lock up one bloke until he tells them the password, rather than prove that an outside hacker could get through their security.

    Pretending that access to the system is impossible without the correct password gives the impression the system is, if nothing else, impregnable to unauthorised users. Getting someone else to hack in and set it right would have the US press howling in full-on 'Chicken Licken' mode that any 'terrorist' could have done the same - cue the banning of 'War Games' and every IT professional going on a 'no fly' list.

    My money's on the mayor telling our man that they'd already got in, but the trial would go a lot easier if the fiction was maintained.

  21. Paul Rogers
    Linux

    @Mark

    Or were using a Mac (fanboys or technopleges) to log in and the password had a # in it.....

  22. Dr. Mouse

    "become a bit maniacal"

    What, like a politician by any chance?

    And I agree, the initial confusion was probably misspelling, leaving the caps lock on, or general stupidity. And as for remote access, I also agree that it was probably for remote admin so he could do his job better. I have left back-doors open into systems when I have been admining for just this purpose.

    Of course, I am an ethical man and have always closed them up when I left the job ;)

    God save us all from eejits, erm, I mean users.

  23. Anonymous Coward
    Anonymous Coward

    Cisco?

    Didn't the original story say this was Cisco kit? As long as you have physical access to the kit you can recover the password on most Cisco kit.

  24. Anonymous Coward
    Joke

    After initial confusion

    that'd be the 1 in c1sco then?

  25. Anonymous Coward
    Happy

    routers only, not any servers

    He was in charge of WAN routers, all Cisco gear, and the passwords were all for those routers, there were no servers nor any desktops involved.

    Apparently, the Ciscos were configured such that password recovery was turned off, or something like that. This was all in an online article a few days ago where another IT guy working there gave some further details.

  26. Anonymous Coward
    Anonymous Coward

    He's not a BOFH

    As usual, the media got this wrong, he's not a sysadmin, he's a network administator.

  27. Jon Green
    Paris Hilton

    "Many have questioned why Childs' bail is so high"

    Well, duh!

    "Give us the passwords, and we can talk about cutting the bail to something sensible. That is, if you want to have a last little bit of freedom before all this becomes your second home. You do, don't you? Or have you come to enjoy Big Bubba's night-night 'cuddles'?"

    Paris could have worked that one out for herself.

  28. dodge
    Dead Vulture

    The inside skinny

    I can't vouch for veracity of this, but here is apparently the inside story...

    (from infoworld, linked by geekpress.com)

    http://www.infoworld.com/archives/emailPrint.jsp?R=printThis&A=/article/08/07/18/30FE-sf-network-lockout_1.html

  29. Mark
    Thumb Up

    Airline trip

    I'm surprised Homeland Security didn't fly him off somewhere and have the sh*t beaten out of him.

  30. frymaster

    re: Wrong guy is in jail

    "The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed."

    Agreed, because one person's incompetancy excuses another person's willful damage.

    ...oh wait, it doesn't

    Not disputing that in the aftermath of this, the IT manager should be investigated and at least reprimanded if not sacked or sued, but I don't see why that means the other guy gets to go free

  31. ben

    Get a life

    Hasn't this guy got anything better to do, if he doesn't like the job, leave, forget about it and get on with stuff. He must have had a massive complex about this position in the company and needed to feel powerful. That's what being a network administrator does to you... No life and his only friend the computer, looser.

  32. Echowitch
    Paris Hilton

    I may be rusty but.....

    I used to be responsible for Cisco password security at a rather large multi-national many years ago and we had numerous cases of Network Engineers setting up routers and forgetting to update the password file. (Wonderful flat text file available to some 500+ users who could easily copy it to floppy......I know as my Manager and I did once. Left the building, went to lunch, and no one knew. Informed the 3rd line manager and he just grunted at us.)

    As routers with lost passwords were at customer sites we had one of two options to recover them.

    1. Use the Cisco Configuration Tool for dragging back the config, editing it, and then uploading it to the router again. (Cisco wouldn't allow us to have it, but we had the IBM versions which worked great.)

    2. Send an engineer to site at a cost of £100 per router and get them to manually locally download the config to their laptop, reset the passwords, and upload the new config.

    Surely they could have done the above ???

    Even Paris could have done better.

  33. Anonymous Coward
    Happy

    Re: routers only, not any servers

    Yep you've hit the nail on the head - the guy disabled the password recovery mechanism which locks out access to ROMMON which would be the only way of traditionally recovering the hardware (the config is destroyed regardless). Basically this guy had the keys to the kingdom.

    Whilst it is obviously crazy that all of this was entrusted to one guy (what if he died unexpectedly?) based on my experience of configuring Cisco equipment for corporates I would say it wasn't that unexpected for one guy (or girl) to end up with absolute control over the network. Suits seem to generally only care about the network staying up, not the particulars of how it is administered, until - of course - the s**t hits the fan.

  34. Stephen Usher
    Alert

    Paranoia....

    The problem was that the sysadmin was paranoid.. to the point where he wouldn't even write the router configuration to the router's flash memory. (Yes, if the power failed the router would lose its configuration unrecoverably. Maybe it was safe from hackers but it wasn't safe for hardware failure.. stupid sysadmin!)

    Apparently he didn't give anyone the password or write it down because he didn't trust them.

  35. David Cornes
    Paris Hilton

    Case?

    After all this, I'm confused as to why he's still pleading NOT guilty...?

    Paris, 'cos she wasn't guilty either, just a little confused.

  36. Sceptical Bastard

    Bail?

    Give the bloke a medal for pwning SF thereby showing up what was obviously negligent network administration and management.

    I wonder if Mayor Boris's outfit uses Cisco kit ...?

  37. Anonymous Coward
    Joke

    re: get a life - ben

    he probably used to post on TheRegister style comments pages as well

  38. Egons Proton Pack
    Paris Hilton

    If the film Hackers has taught me anything...

    Its that all admin passwords are either GOD or SEX....

    I wonder if they tried those?

    Paris cause i reckon all her password are related to sex...

  39. Anonymous Coward
    Joke

    How they found the passwords,...

    the Abu-Graib way.

    http://www.catb.org/~esr/jargon/html/R/rubber-hose-cryptanalysis.html

  40. Danny
    Linux

    @Mark Lockwood

    >For "After initial confusion" read "After they turned off the Caps Lock"

    ROFL!

    Real keyboards don't have Caps Lock...

    http://www.pfusystems.com/hhkeyboard/hhkeyboard.html

  41. Danny

    (different Danny)

    I got a SysAdmin job once where the previous guy had been fired. After a week of getting to grips with the kit I still hadn't found any root passwords for the comm's equipment - and there was a lot of unexplained traffic. So I had to open up the boxes, remove the batteries. Now the previous guy had been quite a bit more techie than me, and had not only kept full access to the system, he'd rewritten the drivers for some of the kit. So I had to download new drivers offsite and repeat the process. All of which took downtime that I was blamed for - after all, the last guy never had these problems! I got so much grief from users and management I regretted not just leaving the guy full access and keeping my mouth shut.

  42. Mark

    Re: re: Wrong guy is in jail

    But I thought managers were paid more because they were in the "risky" positions. Ones requiring the RESPONSIBILITY of the actions of their subordinates.

    Or is that a load of pigshit?

    Yeah. The latter.

  43. Anonymous Coward
    Thumb Up

    A quick lesson,...

    Why he is there now,..

    Middle Manager: The network is unmaintainable while only you hold the passwords and configs. Please arrange to document these in a suitable manner for other staff.

    Senior Engineer: No, I do not believe you or any of the other staff have the necessary skills to maintain this network.

    [Lots of back and forth]

    Middle Manager: Last chance, documentation or suspension.

    Senior Engineer: Suspension.

    [More waiting]

    Middle Manager: Passwords and config please?

    Senior Engineer: No

    Middle Manager: Last chance, documentation or incarceration.

    Senior Engineer: Incarceration.

    [More waiting]

    Middle Manager: Passwords and config please?

    Senior Engineer: No

    Middle Manager: Last chance, documentation or prosecution?

    Senior Engineer: Documentation

    [Try passwords]

    Middle Manager: Proper passwords and config please?

    Senior Engineer: No

    Middle Manager: Last chance, proper passwords or prosecution?

    Senior Engineer: Proper passwords

    LESSON: All Senior Engineers are still only cogs in a larger machine.

    Why he did it,…

    Middle Manager: Please provide passwords to Junior to allow him to make changes.

    Senior Engineer: Those changes are outside his ability to perform, and are an unacceptable risk.

    Middle Manager: I don’t think your job is as complex as you make it out to be. Passwords please.

    [Receive passwords]

    Middle Manager: Junior, please make this network change with the passwords I have provided.

    [Network crash – 36 hours for Senior Engineer to recover]

    Director: What the heck happened last week?

    Middle Manager: Senior Engineer made a mistake, despite being told it was not a sound change to make.

    LESSON: All Middle Managers are cnuts.

  44. Anonymous Coward
    Anonymous Coward

    Initial Confusion..

    The proper BOFH response..

    "OK, OK.. I'll tell you - the password is the Mayor's wife's first name and the surname of his favourite hooker."

  45. Edward
    Thumb Down

    Password Recovery.

    Assuming the Hard Disks aren't encrypted, with physical access to the machines you can:

    Windows:

    Reset the Local Machine and Active Directory passwords by modifiying SAM

    Extract hashes from SAM and crack the passes using Rainbow Tables.

    *nix:

    Reset the passes by modifying /etc/shadow.

    Crack /etc/shadow to get plain-text passwords.

    I'd put money on the HDs not being encrypted, its a drawn out, expensive process with very little actual ROI.

    Who wants to bet this chap is one of, if not the only person managing the system. He probably set it up as well. This is a storm in a teacup, exacerbated by the City's unwillingness to properly staff their infrastructure.

  46. Mitch Russell

    Odds are that the password was one of these:

    password

    cisco

    ******

    foobar

    iknowitandyoudont

  47. Anonymous Coward
    Stop

    Network DOWN!

    I make the following prediction:

    Now the dullards in SF have the passwords the fibrewan network will work no more.

    Up until Childs handed over the passwords the network was working great, you just could not make any alterations to it. Now the city has the passwords some PFY will be given the job of making an apparently minor change that will result in partial or total breakdown.

    Mark My Words, your Doomed SF!

  48. Anonymous Coward
    Anonymous Coward

    @ Edward

    You failed to read all the information. The passwords withheld were for Cisco WAN routers (neither Windows nor *nix) which had been configured with password recovery disabled. If they had performed a hard reset on those routers, then they would have wiped the configuration, their WAN would have stopped working. And the only person who had the knowledge to configure that gear is the guy who is in jail. Catch 22.

  49. Ian Michael Gumby

    BOFH in training?

    One has to ask what was he thinking?

    Of course had he read the entire saga of BOFH, he wouldn't have made the mistakes that he did.

  50. I. Aproveofitspendingonspecificprojects
    Coat

    SF Cisco

    What does SF stand for? SanAndreas' Fault? Send fail? Systems failure? Sentry fled? Soft Fu....errr ....geddit.

Page:

This topic is closed for new posts.

Other stories you might like