For "After initial confusion" read "After they turned off the Caps Lock"
San Francisco City Council regained access to its own computer network today after Mayor Gavin Newsom convinced network administrator Terry Childs to give them the passwords. Childs is in jail until he can raise $5m in bail. He is accused of blocking all access to the city's network and routers by resetting passwords. He …
...but surely they had at least two options:
1. (Not recommended, but workable) Get some people off the net who are penetration testers to hack back into it.
2. Call me naive, but I'm sure that most OS's have a kind of recovery mode where if you have physical access to them, you can boot them up manually and log in and override them. (E.g. if on a Linux machine you accidentally forget the root password, it is possible to force a certain kind of boot that you can log in and reset the password). Not necessarily so workable for the routers perhaps but still definitely possible.
The only other question this begs is whether it will now give the next generation of terrorists a new idea on bringing down the establishment, whichever establishment this is.
"Childs is accused also of installing hardware on the network to enable remote access."
Could this possibly be so as to remotely access the network and fix problems from home out of hours, rather than have to get up, get dressed and travel X amount of time to come in to the office to do something that could potentially take 5 minutes to fix ???
Sounds like he's a bit of a belligerent BOFH who doesn't like the bosses interfering in how he runs "his" network. And in this case they've totally over-reacted !!
I cannot se how having access to the harware loosing passwords could be such a big problem.
I once hade to take back a Unix machine from a customer who had stopped paying for the machine.
Asking the boss for the root password he smiled and said "sorry I just forgot it".
I could have left it at that but I had to boot the machine from a floppy, mound the HDD and erase the root password.
The boss did not smile anymore.
There must be ways to deal with Windows too.
As any sensible Desktop Support Operator knows, all you need to do is talk nicely to your nearest (insert flavor of Unix here)-using geek and (s)he will be able to furnish you with a password hacking tool... sorry, emergency boot disk.
Anon as I'm at work and the Big Bosses would go uber-ballistic if they realised just how fekkin stupid we really think they are.
That they couldn't find a hacker in the Bay area, if not California that could crack the passwords? Instead they go pleading to the culprit?
Clear case of incompetent bureaucracy.
SF is a BIG city so their budget must be large enough to suggest he had a team rather than be working alone - what were they doing while he was setting all this up?
I was torn between the S&C (a hacker could have sorted them out) and Paris. Paris got it in the end (oooerr) to represent the administration...
... aren't they all 'terrorists' now? It's probably a lot easier all round for the city authorities to lock up one bloke until he tells them the password, rather than prove that an outside hacker could get through their security.
Pretending that access to the system is impossible without the correct password gives the impression the system is, if nothing else, impregnable to unauthorised users. Getting someone else to hack in and set it right would have the US press howling in full-on 'Chicken Licken' mode that any 'terrorist' could have done the same - cue the banning of 'War Games' and every IT professional going on a 'no fly' list.
My money's on the mayor telling our man that they'd already got in, but the trial would go a lot easier if the fiction was maintained.
What, like a politician by any chance?
And I agree, the initial confusion was probably misspelling, leaving the caps lock on, or general stupidity. And as for remote access, I also agree that it was probably for remote admin so he could do his job better. I have left back-doors open into systems when I have been admining for just this purpose.
Of course, I am an ethical man and have always closed them up when I left the job ;)
God save us all from eejits, erm, I mean users.
He was in charge of WAN routers, all Cisco gear, and the passwords were all for those routers, there were no servers nor any desktops involved.
Apparently, the Ciscos were configured such that password recovery was turned off, or something like that. This was all in an online article a few days ago where another IT guy working there gave some further details.
"Give us the passwords, and we can talk about cutting the bail to something sensible. That is, if you want to have a last little bit of freedom before all this becomes your second home. You do, don't you? Or have you come to enjoy Big Bubba's night-night 'cuddles'?"
Paris could have worked that one out for herself.
"The city manager and head of IT should be in jail, not this guy. They are responsible for the lack of security and procedures which allowed a single BOFH to change admin passwords without being noticed."
Agreed, because one person's incompetancy excuses another person's willful damage.
...oh wait, it doesn't
Not disputing that in the aftermath of this, the IT manager should be investigated and at least reprimanded if not sacked or sued, but I don't see why that means the other guy gets to go free
Hasn't this guy got anything better to do, if he doesn't like the job, leave, forget about it and get on with stuff. He must have had a massive complex about this position in the company and needed to feel powerful. That's what being a network administrator does to you... No life and his only friend the computer, looser.
I used to be responsible for Cisco password security at a rather large multi-national many years ago and we had numerous cases of Network Engineers setting up routers and forgetting to update the password file. (Wonderful flat text file available to some 500+ users who could easily copy it to floppy......I know as my Manager and I did once. Left the building, went to lunch, and no one knew. Informed the 3rd line manager and he just grunted at us.)
As routers with lost passwords were at customer sites we had one of two options to recover them.
1. Use the Cisco Configuration Tool for dragging back the config, editing it, and then uploading it to the router again. (Cisco wouldn't allow us to have it, but we had the IBM versions which worked great.)
2. Send an engineer to site at a cost of £100 per router and get them to manually locally download the config to their laptop, reset the passwords, and upload the new config.
Surely they could have done the above ???
Even Paris could have done better.
Yep you've hit the nail on the head - the guy disabled the password recovery mechanism which locks out access to ROMMON which would be the only way of traditionally recovering the hardware (the config is destroyed regardless). Basically this guy had the keys to the kingdom.
Whilst it is obviously crazy that all of this was entrusted to one guy (what if he died unexpectedly?) based on my experience of configuring Cisco equipment for corporates I would say it wasn't that unexpected for one guy (or girl) to end up with absolute control over the network. Suits seem to generally only care about the network staying up, not the particulars of how it is administered, until - of course - the s**t hits the fan.
The problem was that the sysadmin was paranoid.. to the point where he wouldn't even write the router configuration to the router's flash memory. (Yes, if the power failed the router would lose its configuration unrecoverably. Maybe it was safe from hackers but it wasn't safe for hardware failure.. stupid sysadmin!)
Apparently he didn't give anyone the password or write it down because he didn't trust them.
I got a SysAdmin job once where the previous guy had been fired. After a week of getting to grips with the kit I still hadn't found any root passwords for the comm's equipment - and there was a lot of unexplained traffic. So I had to open up the boxes, remove the batteries. Now the previous guy had been quite a bit more techie than me, and had not only kept full access to the system, he'd rewritten the drivers for some of the kit. So I had to download new drivers offsite and repeat the process. All of which took downtime that I was blamed for - after all, the last guy never had these problems! I got so much grief from users and management I regretted not just leaving the guy full access and keeping my mouth shut.
Why he is there now,..
Middle Manager: The network is unmaintainable while only you hold the passwords and configs. Please arrange to document these in a suitable manner for other staff.
Senior Engineer: No, I do not believe you or any of the other staff have the necessary skills to maintain this network.
[Lots of back and forth]
Middle Manager: Last chance, documentation or suspension.
Senior Engineer: Suspension.
Middle Manager: Passwords and config please?
Senior Engineer: No
Middle Manager: Last chance, documentation or incarceration.
Senior Engineer: Incarceration.
Middle Manager: Passwords and config please?
Senior Engineer: No
Middle Manager: Last chance, documentation or prosecution?
Senior Engineer: Documentation
Middle Manager: Proper passwords and config please?
Senior Engineer: No
Middle Manager: Last chance, proper passwords or prosecution?
Senior Engineer: Proper passwords
LESSON: All Senior Engineers are still only cogs in a larger machine.
Why he did it,…
Middle Manager: Please provide passwords to Junior to allow him to make changes.
Senior Engineer: Those changes are outside his ability to perform, and are an unacceptable risk.
Middle Manager: I don’t think your job is as complex as you make it out to be. Passwords please.
Middle Manager: Junior, please make this network change with the passwords I have provided.
[Network crash – 36 hours for Senior Engineer to recover]
Director: What the heck happened last week?
Middle Manager: Senior Engineer made a mistake, despite being told it was not a sound change to make.
LESSON: All Middle Managers are cnuts.
Assuming the Hard Disks aren't encrypted, with physical access to the machines you can:
Reset the Local Machine and Active Directory passwords by modifiying SAM
Extract hashes from SAM and crack the passes using Rainbow Tables.
Reset the passes by modifying /etc/shadow.
Crack /etc/shadow to get plain-text passwords.
I'd put money on the HDs not being encrypted, its a drawn out, expensive process with very little actual ROI.
Who wants to bet this chap is one of, if not the only person managing the system. He probably set it up as well. This is a storm in a teacup, exacerbated by the City's unwillingness to properly staff their infrastructure.
I make the following prediction:
Now the dullards in SF have the passwords the fibrewan network will work no more.
Up until Childs handed over the passwords the network was working great, you just could not make any alterations to it. Now the city has the passwords some PFY will be given the job of making an apparently minor change that will result in partial or total breakdown.
Mark My Words, your Doomed SF!
You failed to read all the information. The passwords withheld were for Cisco WAN routers (neither Windows nor *nix) which had been configured with password recovery disabled. If they had performed a hard reset on those routers, then they would have wiped the configuration, their WAN would have stopped working. And the only person who had the knowledge to configure that gear is the guy who is in jail. Catch 22.
This report shows only half the story. This Admin didn't steal or re-set any passwords. He had possession of the passwords from the very start and he, and he alone, knew what they were. His bosses demanded them off him, he decided they weren't to be trusted with messing with his network and said no. They fire him. He still says no. Bosses call police.
Sounds like an over possessive admin, but maybe he had his reasons. Either way, this is a whole different story from the way its been presented. He wasn't a rogue admin, he was the sole admin, which also makes his bosses liable in a number of ways.
From a link posted about it earlier:
A key point made in the e-mail is that Childs' managers and coworkers all knew that he was the only person with administrative access to the network. In fact, it was apparently known and accepted in many levels of the San Francisco IT department. Again, quoting from the e-mail:
“This is where it gets tricky for the prosecution, IMO, because the localized authentication, with Terry as sole administrator, has been in place for months, if not years. His coworkers knew it (my coworkers and I were told many times by Terry's coworkers, 'If your request has anything to do with the FiberWAN, it'll have to wait for Terry. He's the only one with access to those routers'). His managers knew it.
"Other network engineers for the other departments of the City knew it. And everyone more or less accepted it.
"No one wanted the thing to come crashing down because some other network admin put a static route in there and caused a black hole; on the other hand, some of us did ask ourselves, 'What if Terry gets hit by a truck?' If a configuration is known and accepted, is that 'tampering'?”
My source appears to believe that Childs' motivation was the antithesis of tampering, and that Childs did everything possible to maintain the integrity of the network, perhaps to a fault:
“He's very controlling of his networks -- especially the FiberWAN. In an MPLS setup, you have 'provider edge' (PE) routers and 'customer edge' (CE) routers. He controlled both PE and CE, even though our department was the customer; we were only allowed to connect our routers to his CE routers, so we had to extend our routing tables into his equipment and vice versa, rather than tunneling our routing through the MPLS system.”
He shouldn't be in gaol at all!
He should be free to hold the secrets he was entrusted with -until payment was made for the knowledge he was allowed to leave his employment with.
You don't sack a man in that position until you have made him release his secrets. Once you stop paying him you can't expect any different.
Surprise, Surprise, Surprise
"Many have questioned why Childs' bail is so high and how he apparently so easily gained complete control of the city's computer system."
The very fact they this sort of activity still brings surprise should tell us something about those who confess to not understand how easily it was done.
Idiots who are surprised at this must also be surprised that babies puke and pee.
Seriously though I can see why he did what he did (known many retarded idiots who shouldnt be trusted with a spoon let alone network access) but at the same time once he left he should have turned it over right away if only to laugh when the new people made a royal mess of everything seeing as he was no longer responsible. Then again he would problem have still been arrested cause the new people would have denied they messed it up and blamed it on some sort of logic bomb that was hidden.
/Paris because, well shes been as Fu#!ed as this guy would have been one way or the other.
Normally it is a word with the Abbott ;), perhaps they don't have one in San Fran.
So this story looks like this:
Man with passwords gets fired, and they expect him to remember those passwords after the trauma of being fired, and hand them over?
He has not done anything wrong, and his is on 5,000,000 dollars bail!
It beggars belief that San Fookedcisco (as it is now known), did not have contingency plans in place. And one has to wonder if the person this guy reported to was the mayor, therefore only the mayor should have been the recipient.
There potentially is one hell of an interesting counter claim case here. I hope he has played it by the letter, and I have a suspicion he has, who knows what their security docs say, perhaps even something like: 'you may not divulge any IT passwords of the system, except directly to the person you report to.' Those security docs, that 'people' always recommend you do first - may just have shot themselves in the foot with this one.
Biting the hand that feeds IT © 1998–2019