San Francisco City Council regained access to its own computer network today after Mayor Gavin Newsom convinced network administrator Terry Childs to give them the passwords. Childs is in jail until he can raise $5m in bail. He is accused of blocking all access to the city's network and routers by resetting passwords. He …
For "After initial confusion" read "After they turned off the Caps Lock"
"become a bit maniacal"
I guess he got the BOFH upgrade recently :)
This just wouldn't have happened if he had written the passwords for everything on a post-it note and stuck it above his desk - like everyone else does....
I wonder what the password was ?
Give me a F&*(ing payrise you bastards ?
Wrong guy is in jail
The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed.
Nah! That wouldn't be "confusion" that would be a "hardware error". I reckon it's that they had trouble interpreting the Mayor's handwritten version of "Th3M4y0R1S4f5ckw1T".
After they turned off the Caps Lock
How do you do that then?
I don't know what kind of computer system they use...
...but surely they had at least two options:
1. (Not recommended, but workable) Get some people off the net who are penetration testers to hack back into it.
2. Call me naive, but I'm sure that most OS's have a kind of recovery mode where if you have physical access to them, you can boot them up manually and log in and override them. (E.g. if on a Linux machine you accidentally forget the root password, it is possible to force a certain kind of boot that you can log in and reset the password). Not necessarily so workable for the routers perhaps but still definitely possible.
The only other question this begs is whether it will now give the next generation of terrorists a new idea on bringing down the establishment, whichever establishment this is.
Mark: Best comment ever on El Reg!
"Childs is accused also of installing hardware on the network to enable remote access."
Could this possibly be so as to remotely access the network and fix problems from home out of hours, rather than have to get up, get dressed and travel X amount of time to come in to the office to do something that could potentially take 5 minutes to fix ???
Sounds like he's a bit of a belligerent BOFH who doesn't like the bosses interfering in how he runs "his" network. And in this case they've totally over-reacted !!
I cannot se how having access to the harware loosing passwords could be such a big problem.
I once hade to take back a Unix machine from a customer who had stopped paying for the machine.
Asking the boss for the root password he smiled and said "sorry I just forgot it".
I could have left it at that but I had to boot the machine from a floppy, mound the HDD and erase the root password.
The boss did not smile anymore.
There must be ways to deal with Windows too.
Obviously not Windoze then
As any sensible Desktop Support Operator knows, all you need to do is talk nicely to your nearest (insert flavor of Unix here)-using geek and (s)he will be able to furnish you with a password hacking tool... sorry, emergency boot disk.
Anon as I'm at work and the Big Bosses would go uber-ballistic if they realised just how fekkin stupid we really think they are.
You seriously telling me...
That they couldn't find a hacker in the Bay area, if not California that could crack the passwords? Instead they go pleading to the culprit?
Clear case of incompetent bureaucracy.
SF is a BIG city so their budget must be large enough to suggest he had a team rather than be working alone - what were they doing while he was setting all this up?
I was torn between the S&C (a hacker could have sorted them out) and Paris. Paris got it in the end (oooerr) to represent the administration...
Or they realised the I was a 1...
Remote Access - Huh?
Sorry, but don't most Sysadmin's have remote access to the stuff they manage so they can get to it out of hours.
There's a whole load of questions here, not least around the city's governance procedures, if they have them.
In this hand, I have a brick. In my other hand, I have another brick. See these two meatballs? Now, passwords please...
"He is accused of blocking all access to the city's network and routers by resetting passwords."
I think we all know what this means - the passwords were all "admin" or "password" and no-one in SF thought to try them.
Couldn't read their own writing...
"Is it an 'o' or a '0'?"
He's NOT good at it.
Otherwise those passwords would have been easily available at a safe nearby.
I know what the back of people who are "good at it" looks like...
They couldn't have recruited a hacker because....
... aren't they all 'terrorists' now? It's probably a lot easier all round for the city authorities to lock up one bloke until he tells them the password, rather than prove that an outside hacker could get through their security.
Pretending that access to the system is impossible without the correct password gives the impression the system is, if nothing else, impregnable to unauthorised users. Getting someone else to hack in and set it right would have the US press howling in full-on 'Chicken Licken' mode that any 'terrorist' could have done the same - cue the banning of 'War Games' and every IT professional going on a 'no fly' list.
My money's on the mayor telling our man that they'd already got in, but the trial would go a lot easier if the fiction was maintained.
Or were using a Mac (fanboys or technopleges) to log in and the password had a # in it.....
"become a bit maniacal"
What, like a politician by any chance?
And I agree, the initial confusion was probably misspelling, leaving the caps lock on, or general stupidity. And as for remote access, I also agree that it was probably for remote admin so he could do his job better. I have left back-doors open into systems when I have been admining for just this purpose.
Of course, I am an ethical man and have always closed them up when I left the job ;)
God save us all from eejits, erm, I mean users.
Didn't the original story say this was Cisco kit? As long as you have physical access to the kit you can recover the password on most Cisco kit.
After initial confusion
that'd be the 1 in c1sco then?
routers only, not any servers
He was in charge of WAN routers, all Cisco gear, and the passwords were all for those routers, there were no servers nor any desktops involved.
Apparently, the Ciscos were configured such that password recovery was turned off, or something like that. This was all in an online article a few days ago where another IT guy working there gave some further details.
He's not a BOFH
As usual, the media got this wrong, he's not a sysadmin, he's a network administator.
"Many have questioned why Childs' bail is so high"
"Give us the passwords, and we can talk about cutting the bail to something sensible. That is, if you want to have a last little bit of freedom before all this becomes your second home. You do, don't you? Or have you come to enjoy Big Bubba's night-night 'cuddles'?"
Paris could have worked that one out for herself.
The inside skinny
I can't vouch for veracity of this, but here is apparently the inside story...
(from infoworld, linked by geekpress.com)
I'm surprised Homeland Security didn't fly him off somewhere and have the sh*t beaten out of him.
re: Wrong guy is in jail
"The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed."
Agreed, because one person's incompetancy excuses another person's willful damage.
...oh wait, it doesn't
Not disputing that in the aftermath of this, the IT manager should be investigated and at least reprimanded if not sacked or sued, but I don't see why that means the other guy gets to go free
Get a life
Hasn't this guy got anything better to do, if he doesn't like the job, leave, forget about it and get on with stuff. He must have had a massive complex about this position in the company and needed to feel powerful. That's what being a network administrator does to you... No life and his only friend the computer, looser.
I may be rusty but.....
I used to be responsible for Cisco password security at a rather large multi-national many years ago and we had numerous cases of Network Engineers setting up routers and forgetting to update the password file. (Wonderful flat text file available to some 500+ users who could easily copy it to floppy......I know as my Manager and I did once. Left the building, went to lunch, and no one knew. Informed the 3rd line manager and he just grunted at us.)
As routers with lost passwords were at customer sites we had one of two options to recover them.
1. Use the Cisco Configuration Tool for dragging back the config, editing it, and then uploading it to the router again. (Cisco wouldn't allow us to have it, but we had the IBM versions which worked great.)
2. Send an engineer to site at a cost of £100 per router and get them to manually locally download the config to their laptop, reset the passwords, and upload the new config.
Surely they could have done the above ???
Even Paris could have done better.
Re: routers only, not any servers
Yep you've hit the nail on the head - the guy disabled the password recovery mechanism which locks out access to ROMMON which would be the only way of traditionally recovering the hardware (the config is destroyed regardless). Basically this guy had the keys to the kingdom.
Whilst it is obviously crazy that all of this was entrusted to one guy (what if he died unexpectedly?) based on my experience of configuring Cisco equipment for corporates I would say it wasn't that unexpected for one guy (or girl) to end up with absolute control over the network. Suits seem to generally only care about the network staying up, not the particulars of how it is administered, until - of course - the s**t hits the fan.
The problem was that the sysadmin was paranoid.. to the point where he wouldn't even write the router configuration to the router's flash memory. (Yes, if the power failed the router would lose its configuration unrecoverably. Maybe it was safe from hackers but it wasn't safe for hardware failure.. stupid sysadmin!)
Apparently he didn't give anyone the password or write it down because he didn't trust them.
After all this, I'm confused as to why he's still pleading NOT guilty...?
Paris, 'cos she wasn't guilty either, just a little confused.
Give the bloke a medal for pwning SF thereby showing up what was obviously negligent network administration and management.
I wonder if Mayor Boris's outfit uses Cisco kit ...?
re: get a life - ben
he probably used to post on TheRegister style comments pages as well
If the film Hackers has taught me anything...
Its that all admin passwords are either GOD or SEX....
I wonder if they tried those?
Paris cause i reckon all her password are related to sex...
How they found the passwords,...
the Abu-Graib way.
>For "After initial confusion" read "After they turned off the Caps Lock"
Real keyboards don't have Caps Lock...
I got a SysAdmin job once where the previous guy had been fired. After a week of getting to grips with the kit I still hadn't found any root passwords for the comm's equipment - and there was a lot of unexplained traffic. So I had to open up the boxes, remove the batteries. Now the previous guy had been quite a bit more techie than me, and had not only kept full access to the system, he'd rewritten the drivers for some of the kit. So I had to download new drivers offsite and repeat the process. All of which took downtime that I was blamed for - after all, the last guy never had these problems! I got so much grief from users and management I regretted not just leaving the guy full access and keeping my mouth shut.
Re: re: Wrong guy is in jail
But I thought managers were paid more because they were in the "risky" positions. Ones requiring the RESPONSIBILITY of the actions of their subordinates.
Or is that a load of pigshit?
Yeah. The latter.
A quick lesson,...
Why he is there now,..
Middle Manager: The network is unmaintainable while only you hold the passwords and configs. Please arrange to document these in a suitable manner for other staff.
Senior Engineer: No, I do not believe you or any of the other staff have the necessary skills to maintain this network.
[Lots of back and forth]
Middle Manager: Last chance, documentation or suspension.
Senior Engineer: Suspension.
Middle Manager: Passwords and config please?
Senior Engineer: No
Middle Manager: Last chance, documentation or incarceration.
Senior Engineer: Incarceration.
Middle Manager: Passwords and config please?
Senior Engineer: No
Middle Manager: Last chance, documentation or prosecution?
Senior Engineer: Documentation
Middle Manager: Proper passwords and config please?
Senior Engineer: No
Middle Manager: Last chance, proper passwords or prosecution?
Senior Engineer: Proper passwords
LESSON: All Senior Engineers are still only cogs in a larger machine.
Why he did it,…
Middle Manager: Please provide passwords to Junior to allow him to make changes.
Senior Engineer: Those changes are outside his ability to perform, and are an unacceptable risk.
Middle Manager: I don’t think your job is as complex as you make it out to be. Passwords please.
Middle Manager: Junior, please make this network change with the passwords I have provided.
[Network crash – 36 hours for Senior Engineer to recover]
Director: What the heck happened last week?
Middle Manager: Senior Engineer made a mistake, despite being told it was not a sound change to make.
LESSON: All Middle Managers are cnuts.
The proper BOFH response..
"OK, OK.. I'll tell you - the password is the Mayor's wife's first name and the surname of his favourite hooker."
Assuming the Hard Disks aren't encrypted, with physical access to the machines you can:
Reset the Local Machine and Active Directory passwords by modifiying SAM
Extract hashes from SAM and crack the passes using Rainbow Tables.
Reset the passes by modifying /etc/shadow.
Crack /etc/shadow to get plain-text passwords.
I'd put money on the HDs not being encrypted, its a drawn out, expensive process with very little actual ROI.
Who wants to bet this chap is one of, if not the only person managing the system. He probably set it up as well. This is a storm in a teacup, exacerbated by the City's unwillingness to properly staff their infrastructure.
Odds are that the password was one of these:
I make the following prediction:
Now the dullards in SF have the passwords the fibrewan network will work no more.
Up until Childs handed over the passwords the network was working great, you just could not make any alterations to it. Now the city has the passwords some PFY will be given the job of making an apparently minor change that will result in partial or total breakdown.
Mark My Words, your Doomed SF!
You failed to read all the information. The passwords withheld were for Cisco WAN routers (neither Windows nor *nix) which had been configured with password recovery disabled. If they had performed a hard reset on those routers, then they would have wiped the configuration, their WAN would have stopped working. And the only person who had the knowledge to configure that gear is the guy who is in jail. Catch 22.
BOFH in training?
One has to ask what was he thinking?
Of course had he read the entire saga of BOFH, he wouldn't have made the mistakes that he did.
What does SF stand for? SanAndreas' Fault? Send fail? Systems failure? Sentry fled? Soft Fu....errr ....geddit.
- Updated Zucker punched: Google gobbles Facebook-wooed Titan Aerospace
- Elon Musk's LEAKY THRUSTER gas stalls Space Station supply run
- Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
- Mounties always get their man: Heartbleed 'hacker', 19, CUFFED
- Android engineer: We DIDN'T copy Apple OR follow Samsung's orders