All right, what's all this, then?!
So they needed a network audit before firing him. And they really needed an audit after he 'left'. And now they need a miracle? Oh, and an audit.
An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …
What icons? A Planchette has none. Original had a hole for a pencil for 'automatic writing', later version was a pointer for the Ouija board, which might only have decorations, no icons, as it's Yes, No and alphabet. Released originally as a game and a game company still has the name as a trademark.
So what are these Oracle icons?
I don't think an audit will solve the problem... It might have prevented it but it's too late now. Maybe the answers are on the mysterious #4 server as I've known at least one admin who kept a server "hidden" for emergencies. Turned out to have config info, some nasty info on manglement, and a pile of server and network scripts.
I'm the AC who mentioned Radius/TACACS farther down in the comments as being helpful. Here's a link with brief explanations of AAA, RADIUS and TACACS+. http://www.pearsonitcertification.com/articles/article.aspx?p=2449614
I have no ties to Pearson and am not able to speak to the efficacy their certification course ware.
That is true, but it really depends how large the place was. For example, I am mostly a developer, but I volunteer some system administration for a local charity that I appreciate. They used to have an administrator, but they left and they don't have that many systems. When I arrived to look over the systems and start my work, I found the following:
1. A server that contains a domain controller and shared network folders.
2. A UPS for aforementioned server. Not plugged in to the wall socket or, thankfully, the server.
3. A backup system that seemed to be set up properly. It used removable disks that were swapped out every week, when there was an administrator. Since that admin left, they had one disk inside the system that contained the most recent backup and two disks that contained backups from eight months previous.
4. A firewall that nobody had the access codes to. Nobody knew what this firewall was or wasn't doing and I just wanted to get rid of it once I felt confident to rebuild the network.
This is what happens when there is only one person working on the system and the company lacks the ability to manage that person. The charity is small, the director is nontechnical, and the system was consequently chaotic. There wasn't a clear person at fault, but we could all agree that there was a problem.
I volunteered to take care of the network in my condo because the existing situation was a mess.
Well, not completely a mess, at the origins, it had been set-up properly, 2 WiFi per floor, one in each wing of the building, a WiFi bridge over to the other building, using Linksys and Ubiquity (I mean, not the cheapest possible hardware), under DD-WRT... So a good built. But obviously, the company managing the system was not the one that once installed it, so when the network hanged, the only solution was a power-off to reboot the system (and because there are 15 WiFi, it is faster to turn off the central power of the building! I joke you not. Just hope nobody is in the lift during these 5 seconds!).
So, that was the situation when I moved in 2 years ago, so I volunteered to look after the system, I changed the main router/authentication platform and now I am left with a bunch of WiFi under DD-WRT without knowing the password.
If the few hacks I found online are not working, I will be left with the solution of a reset-reconfigure, but I am not really looking forward to that.
I would suggest, if possible, that you get them to buy new hardware and set up a replacement in parallel. Otherwise, I hope you are good at network administration. I have tried long enough to get a multiple-AP network going with multiple openwrt devices, and I found it to be a terribly long and painful process involving far too much fiddling with DHCP. I'll be the first to say that my network admin experience is suboptimal, but there is still a lot of complexity and ways for that to completely fail.
Having had to adopt a Ubiquiti wifi network left in a similar state, I'd say you're not in too bad a place. Not sure what you're using as a controller (an old laptop running debian will do it fine) but, as long as you use the same SSID and passwords, you can wipe and reset the whole system pretty easily - start with one AP (reset, then adopt) and then roll it out from there when you're happy with the config. The Ubiquit forums are your friend in this situation.
Happens in every IT department I've worked in. There are always staff members who see knowledge as power and always managers who allow it to persist.
I have audited my own current IT department (I work in infosec) and have pointed this issue out repeatedly over the past couple of years, it's in almost every report I write but still isn't addressed. They state that since they have sharepoint up and running this isn't a problem, but as always there's bugger all on there from the staff members in question. It's all full of content from the team-minded people, the ones who play well with others etc.
Thing is one staff member in particular has been suspended several times in the past few years for not following change management processes and causing the network to drop, screwing DNS, breaking the web filtering etc. He's one of those who is a full believer in "knowledge is power" and has an inept manager who won't challenge him. I am desperate for that staffer to end up at a disciplinary so I can grill him but until then my hands are tied.
It's bloody frustrating, not just for me but all the IT staff who have to put up with this crap.
There are always staff members who see knowledge as power
Ah, I think that's where I've gone wrong then. I've always sought to be professional and leave documentation so that someone coming along after me has something to work from. Of course, that makes me dispensable ...
Also, I happen to know that at my last job my carefully engineered and fully documented network was ripped out by a "I don't understand it so I'm ripping it out" imbecile who only thinks he has a clue about networking (barely understands the basics of IPv4 addressing). Amusingly, I'd had no outages on the DNS we hosted for something like 600 customer domains for several years thanks to proper engineering with suitable redundancy - and he killed about 100 that were left on our servers when he killed the master and didn't know he needed to promote a slave before the zones timed out ! Prior to that we'd had a policy in the technical section of doing detailed network diagrams when doing a customer install - IP addresses, WiFi details, all the useful stuff that an engineer could do with to look after the network. Same person demanded all that useful stuff be removed because it didn't make for a "pretty drawing" to hand over to the customer (we kept our hand drawn originals !)
At my previous job to that, I had a database of what was patched to where, a database of all IP allocations, detailed network diagram, where all the data sockets were located, the patch leads were colour coded by function, etc. Before long all the patch leads were replaced with black (and didn't use the management bars well), the database was tossed, etc, etc ...
It's "slightly disheartening" to know that you've left a situation where any half-capable person can come in, look at the documentation, and take over without too much hassle - only to find they've replaced you with a monkey (in at least one case, that's being unkind to primates !) who's set out to wreck things.
There is a picture in reddit https://i.redd.it/ejlc2pmrd9n21.jpg where the joke is that the angry girl is going to shoot the author's chromebook and the script has him say "don't do that that is the only copy of my book". Most commenters note that the author is probaby writing in G Suite and only needs to get a new chromebook if she shoots. :-)
Maybe time to start clouudifying configs. :-)
If it wasn't for the fact that this switch was so integral to their network, a simple solution might be to expose the thing to the internet and offer a bounty to the first enterprising hacker who could get in, reset the login credentials and then report in with the solution.
The password is BOB. Unless he was dyslexic, in which case it will be BOB.
Seriously, to blame a lame engineer for this stuff is ridiculous. The guys boss should be on the chopping block for allowing his network to be run that way. Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process.
"Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process."
Well, yes agreed, but also, say a manager asks sysadmin for full audit of network including providing backup configs and passwords, the manager is still dependent on the sysadmin's honesty and competence, either of which could be lacking. How would the manager even notice if anything related to this particular switch was missing, given that it was literally months before anyone even noticed the switch was there?
Not to mention the process usually goes
"Bob, I need a full network inventory...oh wait, the sales department are complaining that their VPN isn't working, can you fix that first..."
Aaaaaaaaaaaaaaaaand the audit (and any other documentation) never gets done because it's always less important than keeping everything running.
We have a core switch that only one person can log into, not only because they are the only person with a password, but also because it's broken and will only respond to packets from certain MAC addresses.
Manager: So we need to document this flow. And we to fix these things ASAP.
Me: You can have one or the other.
Manager: Why? Why can't we have both?
Me: Because you insisted on a three hour meeting, face to face in a conference room to go over this. Now I have time to do one or the other.
Manager: Hmmmmm... Ok, just fix the things. Then we'll have to have another meeting to go over the plan to document everything.
Me: *sigh* Sure thing, boss. Just put the meeting notice on my calendar. (Screaming inwardly to myself, "And you've still learned NOTHING!")
Just a lightly edited extract of a real situation. Much of the problem is that for managers their work product is meetings, so they think things are getting done if you have a meeting about a subject rather than actually working on the problem.
This touches on something that's been really pissing me off lately.
I consider myself a fairly competent IT person. I've worked as a systems admin, databases admin, software developer, and a network engineer for two ISP's.
Lately, I've been applying for a bunch of IT manager positions. I'm not having much luck because I'm told I need x years of management experience. They don't seem to give a shit about my 26 years of technical experience!! No, I could have a shit technical background, but if I had 5 years of experience as some PHB, I would be in.
I (being a little pissed about getting turned down without even an interview) even mentioned to a couple of companies that maybe if they hired managers with real technical skills, it may make things run better at their company?
What RFC822 said.
Free advice for new sysadmins: Take as many business related courses as you can stomach. Haul your ass to your nearest post-secondary school that offers night courses and talk to a career counselor. Tell 'em that you are a techie, but are interested in management. You want to take courses that can be applied to a future MBA (should you want to go that route later).
If you already hold a four year degree, and you can code fluently in one or more upper level languages, chances are you can snooze through an MBA in two years (or less, if the classes line up right). Lest you think getting an MBA is difficult, think about all the feckless idiots you know who hold one ;-)
I realize that not all of us are cut out for management ... the objective isn't necessarily to become a manager, but rather to learn their lingo. It's amazing how fast long-closed doors open once you learn to talk to Moneybags in his/her own language. On top of that, an MBA will better prepare you for when the time comes to strike out on your own and become a consultant.
In a prior life, I was tasked with the boots-on-ground closet to closet network hardware audit of an office complex.
Serial numbers, model numbers, port count, etc etc. One of the duties was pulling configs from everything that could give them up.
Was it password protected without the password being known by anyone? Hold this button on the panel, connect via terminal emulator over the serial port, enter this obscure command, and then set this password after hoovering up the config.
Mostly CISCO gear, but I'd assume with hardware access anything can be changed.
Dell might just be evil though?
That is something you can do on Dell networking equipment. The problem is that they can't afford to take it down for the few minutes it would take to do that and why they are screwed until April when they have a scheduled maintenance period.
From the sounds of it, this data-center is mission-critical and even a minute of downtime would be quite costly, especially since nothing is broken.
How do you know nothing is broken if you can't log into it? Just because it is routing traffic properly? How do you know that on top of it's normal duties, it isn't logging all LAN traffic and shipping it off to your competitor? Or that it hasn't otherwise been rooted, given the lack of clues possessed by the former admin?
To me, the box is a corporate hazard and needs to be airgapped, preferably last December!
No problemo! I would put all this in writing to the management and wait patiently until it becomes really broken. Sometimes in the future they will have to approve the maintenance and it might be at an even worse moment than it is right now.
but Management sure as hell isn't going to agree
"Err boss - we appear to have had a sudden power outage in part of the data centre and the switch went down for a minute or so.."
"On the good side, I now have a full config of that switch and can log into it"
"No - the two are not connected - why do you ask?"
It is always "mission critical" right until it isn't - When the neglected plant blows up, falls apart, seizes up, burns up, then the "business decision" is to wait weeks and weeks by holding daily crisis meetings while the replacements trundles trough procurement and everything goes to hell.
Since the beancounters cloaked themselves with power, it is always more important to "control costs" than it is to run the business.
Went to add a new Server to a Dell UPS which had already had the software installed and configured by the previous tech but he hadn't documented the Admin password to the software. Called Dell nonHelpdesk to see how I could reset the password to default but no way could it be done. Was told that even uninstalling the software may leave the changed password in the Server.
Not happy Jan!
... if you were to report the make (Dell, apparently), model and serial number (last 4 digits XXXXXed out) here, one of ElReg's esteemed readers would have an avenue of approach to get in. Might involve a soldering iron, but I've rarely been stymied when the chips are down and I have access to the hardware.
How is that only one person would have the password? All the configs should be backed up to a Rancid server and all the network engineers should have Radius/TACACS log in credentials. Several senior engineers or managers should be able to grant/restrict access to devices as required. The devices themselves should be restricted to having only a senior engineer or manager change critical passwords.
"How is that only one person would have the password?"
The real world isn't always as sensible a place as it should be. In an ideal world, you would use RADIUS/TACACS to manage the device and have a local username/password safely stored away somewhere in case this failed. Or you have a single username/password in a restricted password safe.... Or default credentials that no one ever bothers changing... Or username/password just one person knows... All options...
For recovery, it depends on the model as Dell has mixed and matched vendors across their switch range. With Cisco and some other vendors, you maybe able to use SNMP RW details to backup the config to a TFTP server. Or there may be a web GUI/API that gives you RO access to at least document the config. Even SNMP RO can provide a good chunk of information if you are trying to replicate a config, assuming you use SNMP management tools in your environment.
Otherwise you are stuck with figuring out how it works. It may not be easier than you expect, especially if there's some basic documentation to go along with it or you have access to connected switches to get shared information off.
In a former life as an IT manager with a large company, I was ensured key passwords were stored off-site in a secure fire safe. I made it a matter of policy that "anyone can get run over by a bus" so procedures had to be in place to handle such an event. Unfortunately I was subsequently hit with a real bus which landed me in hospital briefly and wrote off my motorbike. The irony wasn't lost on me.
Several years on, I was working as a contractor / software developer for my own business. I gave all relevant documents to a bookkeeper / accountant who did all the necessaries for submissions to Companies House, HMRC etc leaving me to get on with developing software. I didn't have much of a clue about book keeping and accounting. Then one day I got a letter out of the blue from a debt collection agency. I'd been fined for non-submission of annual accounts to Companies House. This was the first I'd heard, so I got straight onto the phone to my accountant... or rather spoke to his widow. He'd died unexpectedly a few months earlier from cancer. He was only in his thirties with a couple of young kids. His widow helped the best she could and between us we managed to piece together bits of information from random files, documents and spreadsheets on his computer so I could reconstruct my accounting position and make a late submission to companies house. It was a complete crash course in accounting for me.
It is too easy to have too many eggs in one basket. Anyone can die unexpectedly and potentially leave you in deep doo-doo without adequate procedures in place. It was somewhat ironic that in my former employment I put procedures in place to handle dead-employee scenarios, but when I ran my own business I fell foul to a lack of such procedures. Sigh.
"The irony wasn't lost on me."
I feel your pain! For years and years I used to tell the guys in the workshop to make sure that rope/cables didn't present a trip hazard. You can guess who then tripped over a bit of string they'd thought they'd tied up neatly, but hadn't. Cue several v expensive visits to the dentist :(
As a (very) long time network engineer, I can't accept this at face value. If the only switch they were locked out of was the core switch, then the bulk of the config could be extracted from all of the neighbors. Once you have the information from the neighbors, then some network events debugging enabled on the neighbors would enable the bulk of the remaining missing information to be derived.
Yeah... and we have only the new guy’s word that he was incompetent... if there’s one universal rule in tech it’s a new engineer slagging off his predecessor to management. You will never see lawyers, accountants etc do this. They understand that solidarity to the profession trumps any short term advantage they may gain from doing so.
"You will never see lawyers, accountants etc do this."
I assume that Patisserie Valerie's new accountants are not singing the praises of the previous lot.
"They understand that solidarity to the profession trumps any short term advantage they may gain from doing so."
'Solidarity to the profession' sounds quite a lot like 'close ranks and disavow any fault' to me.
'I assume that Patisserie Valerie's new accountants are not singing the praises of the previous lot.'
I don't see why not, ok, maybe not in public, but at least in private, as, after all, they've got a nicely paying gig out of their predecessors' sterling work, and as a bonus they've no doubt also got lots of doubleplusuncheap forensic work on the go as well just to keep the old expenses meter spinning that wee bit faster..
Lawyers, as officers fo the court, are required to report incompetent, fraudulent, crooked of dodgy fellow lawyers. They might have pride intheir profession, but I have known them to blow the whistle, somethimes descreetly, but never-the-less. OTOH I have seen IT colleagures closing ranks around a useless and dangerously-inept IT mate because simply went all dog-pack when one of them was threatened.
the ones integrated in the M100 chassis were easy to get into, pull the out, set/unset a jumper and plug them back in. The Juniper switches they resold can be broken from the serial console.
Also, they most certainly can determine exactly what this switch/router is doing by reverse engineering. You just have to map it all out based on where the connection go, and how those devices/ports are configured.
Been there, done that, still have the t-shirt...
Indeed, buy a new switch and rack mount it, have a good guess (and by guess I mean work out as best you can), what the problematic switch has to be doing, configure the new switch accordingly, swop all the connections over. Then it's a case of seeing who screams about loss of connectivity and adjust config accordingly.
Management will absolutely hate this plan, just calmly point out that it's either this approach (where there is at least the possibility of backing out), or wait until the old switch catastrophically fails, at which point people start losing their jobs, possibly starting with the management!
And afterwards you'll have a nice new (well actually old) switch to be factory reset and used for something else.
... wait until the old switch catastrophically fails, at which point people start losing their jobs, possibly starting with the management!.
Your argumentation, while factual, is not aligned with the usual incentives structure: Getting sacked only pops the Golden Parachutes for Management earlier than the planned and people losing their jobs will always boost stock prices, which will boost Management stock options even more, so it is a good thing.
So? Worry about what you control: Write an ass-covering report/action-plan, present it at the management meeting, it will be rejected, and then leave it at that. Nobody cares. If you care, consider a different career.
Dell has nothing to do here. If I remember correctly there was a sysadmin in a Californian city who accepted to go to jail instead of providing access to Cisco switches* he was managing.
* see "service no password-recovery" command on Cisco switches
This is why one should Radius/TACACS for authentication. You can provide various levels of control to your engineers. There are a variety of commands that engineers doing their daily work in a network do not need to have access and all of which can be controlled, at least using decent equipment.
Even if you use a password manager, you still have to be in control of at least one of the three parts, and this depends how you configure access. You either need to be an administrator of the password manager, administrator of the directory service, or administrator of your 2fa system.
If you manage the password manager, you can reassign resources to other users.
If you manage the directory service you can modify group membership to grant access to other users.
If you manage the 2fa system, you can reset/re-issue 2fa for accounts.
These are questions for your password manager vendor.
The things I recommend for companies that use password managers are, ensure that all interaction gets logged, and never allow a single individual to responsible for anything. This means having a second individual with admin permissions, even if they would only be capable of doing anything while on the phone with tech support.
I use Password Manager Pro from ManageEngine.com. It runs on Windows or Linux and you can install a copy that is the full enterprise version for a trial. I have used the free version and the Enterprise version, and while the Enterprise version has some very nice features, my company's needs are such that the free version works just fine.
I have a friend who's company uses LastPass, and they're happy with it.
That being said, I think you should try several password managers and determine which works best for your situation.
We've been going through this sort of thing with our succession planning for our parish council (2.5 employees!). The God passwords are written down and stored in a safe, the key to which are held (locked up) by the two Responsible Officers and the (external) IT Auditor. If all three manage to die at the same time - especially something that takes the IT Auditor as well - it will probably be due to an event causing more concerns than parish council continuity. ;)
The external IT Auditor is an external organisation. They would only ever approach our offices if both Clerk and Deputy Clerk fell under the same bus at the same time. And even if all the parish councillors went under the bus as well resulting in no executive authority, things would devolve up to the district council. If the district council was wiped off the earth, there's be other more important things to worry about.
I had configs and layout documentation stored on a hidden server on the network. I backed everything up to it. Then I archived it and encrypted it so it could be stored in plain site on several servers, on tape backup, and store the tape in a bank vault. The network didn't change much, so this was a viable option. But when the network went down, the other admins had no clue how stuff was configured because they weren't doing the backups like they were supposed to. Some managers were let go. I figured out what the problem was and reloaded the router with the config off my hidden server. When I left, I still don't think they found the server.
The backup server is a Raspberry Pi v1.0. It's in a small, non-descript case with a label that has the IP address and a message that reads "Critical network monitoring equipment. Do not disconnect."
It sounds like he was a bad admin. But he may have seen the writing on the wall about getting fired and purposely felt like making things as difficult as possible for his successor, not knowing that it would be his successor in a literal sense. People seem to be speaking ill enough of the dead in the article that it didn't sound like he was very cooperative.
And I can't deny that in a crap job situation many of us would not have outright sabotaged anything, but would have felt this way, right or wrong. In the end, it was management's fault for letting things get this way, and possibly for both either maintaining an incompetent employee or so neglecting a once-useful employee that he ended up indifferent and apathetic.
The deceased either knew the writing was on the wall so came up with a scenario that meant if he got fired he'd have some satisfaction from the shit-storm he'd leave behind or knew someone didn't like him and was trying to make it impossible for them to get rid of him.
I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.
I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.,
I had fun with that once or twice. The boss had been badmouthing me to suppliers, customers and contractors.
One day there was a fault with the internet (telco end, not ours) and he came in while I was talking to one of said customers and mentioned it. I just nonchalantly said "Well, you know much more about networking than I do, I'll go off clock and watch in awe as you fix it".
Both the customer and I found it quite a joy to be sharing a moment together watching him get more and more flustered as he proclaimed his knowldge, went in with a 'fix', only to have that fail so go with another thing. He'd heard me talk of a bad LAN card flooding the network with "martian packets" (an early outage before I got some resiliency built in) so decided one of the machines was causing that and unplugged them one by one, claimed bad cabling so opened up packets of patch cables (all the shop ones were made by me, the ones he opened were on the shelf IE saleable stock), and various other things..
What made it more enjoyable for me is it was a "brief" (as in 3-4 hours) but major outage. The customer worked a couple of blocks away and had everything with the one telco so had lost phones and internet, and had come to me in person to see if I was aware what was up. As our net and phone line was also out, I'd already phoned the telco to make sure they were aware of a fault when the boss came on the scene. Given we could log into the router and it had a "disconnected" status icon, I was fairly certain the internal LAN was fine and the other side was dead, especially as someone a couple of blocks away had the same problem :)
Love being able to show that kind of manager up when the time comes.
for a small/medium size organization to protect against this kind of attack. You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen. Instead, the organization can outsource it completely and it's no longer a problem.
Instead, the organization can outsource it completely and it's no longer a problem.
Of course then you have an entirely new set of problems starting with GDPR compliance, data leakage, cloud providers deciding you're last in line for a fix, connection troubles, being held for ransom with rate hikes or mandated software "upgrades", deciding your app isn't worth supporting any longer, etc.
When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.
But don't let that all get in the way of your cloudy sales pitch!
When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.
Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.
As opposed of a CEO who is more likely to think it is their company and they can and know how to play with the password.
Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.
So you've seen this too. It's an interesting phenomenon, some form of "absolute power corrupts" I gather.
One organization I worked with was effectively run by a very competent secretary (that was also a force to be reckoned with if you did wrong). Her "boss" was a miserable twat....
They run the place.
Free hint to all consultants: ALWAYS ask the secretary about the Boss's computer knowledge. You can save a lot of time and trouble for a lot of people over the long haul.
I know of several CEO-types of Fortune-500s who make a big show of "checking the computer", even though their network cable was "accidentally" never installed.
I can't count the number of times I've swapped the Boss's top of the line CPU, gathering dust and spiderwebs under his credenza/return, artfully changing screensavers every couple minutes, for his secretary's underpowered kit ... without the Boss noticing.
My late wife was (officially) the Financial Controller of a small craft brewery, but as the two Partners were complete imbeciles, who squandered their father's legacy, she actually ran the company. Then she was struck down by an incurable disease, and after she died, the company folded.
You don't deserve the up votes in the same way I don't deserve the down votes.
Are you trying to tell us the CEO should login three times a day just to make sure those passwords have not been changed ? It happened to me when organizations asked me to help and provided me with two or three written down passwords that were no longer valid. By the way I'm not a cloud stuff salesman, I'm currently configuring telecom equipment but I've learned there is more to security than TACACS/RADIUS and a password kept in an envelope in CEO's office. Clue for you, do a Google search for privilege separation and privileged access management and start from here.
As someone mentioned in a post below, you're just changing the nature of the problem but your rogue admin scenario is gone. Yes, the outsourcer might have a rogue employee too but your company can hope to be compensated financially for the down time. As for compliance you'll have to do it anyway. Outsourcing or not, you're the owner of the data so act on securing it wherever it is stored. And it is better and cheaper to focus on securing data instead of caring for every switch and wiring closet door in addition to data. My career too is threatened but there's nothing I can do, developers have stolen the show. I would really like to adapt to the new reality but time is a serious constraint.
No, that just means that you've traded one problem (rogue employee) for another problem (rogue employee of outsourcer) but I agree with your statement "You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen.", but you left out the most important part: good people. If you don't have good people, you can buy tech, etc, to no avail.
>for a small/medium size organization to protect against this kind of attack. ... Instead, the organization can outsource it completely and it's no longer a problem.
It is still a problem, I've come across many small/meduim businesses that prove the rule that businesses like to do business with similar sized organisations. So the small business has outsourced their IT to a 1~2 man operation (1 tech who actually knows the passwords and where they are kept & 1 sales/consultant who instructs the tech but doesn't actually know how to access the tech's folders).
This is exactly why companies need oversight, you need to badger colleagues to document and record, get organised people. I hate this idea that just because you're asked to document your work and record things, that you're getting the boot next week. No one is trying to steal your job! We need the knowledge so that we can keep you and your job. You vanish on holiday for 2 weeks and no one knows what you do you put the whole company a risk and everyone's job.
Get your head out of your arse and get stuff written down, and sensitive stuff stored securely, even if it's just a print out given to the IT manager or even get Finance to put it in their safe. On the other hand the more you hoard knowledge, the more complaints HR may receive about you and then you're considered a risk for not complying with regs, this means they may find some way to demote or remove you for not complying.
You may not be going full on "DevOps" but the TPJ has a lot of sensible ideas about ensuring work is documented, critical people who cause bottlenecks are utilised correctly and knowledge is recorded so others can step in when needed to keep companies in crisis moving along.
And do you really want Imminent Failure to reveal Special Forces AIMissions for Universal Resolution with AISolutions for Systems in a Failure withTotal Collapse of Non-Cooperative Operating Systems ......with Exclusive Elite Executive Officer Suites in Stasis?
Yes would be the Correct Answer there whenever ready for Everything Quite Wonderfully So Easily Different.
And here be but one Blank Canvas upon which such Futures are Wrought and Writ with Almighty 0Day Provider Protection Testing Vulnerabilities to Distractions and/or Destruction with Enigmatic Ethereal Exploitation of Earthly Resources with Depleted Intelligence Sources.
What do you Think? AI Fact and/or Pump Fiction? :-)>
And just whenever Theresa was praying for it not to get any worse ........ An AI LifeRaft just Simply Appears out of Nowhere.
That Project is Magic. Pure and Simple.
That may be true, but companies will, at times, ask for documentation under the theory that I'll write down everything that a replacement working with a lot less knowledge and for a smaller bill can simply pick up. That isn't a reason to refuse to document, and I have never done that and would not suggest that anyone else do so. Still, some people don't understand that the hundreds of pages of documentation and procedures, while as organized and clear as I can make them, are long and require thorough reading to understand. I've been praised frequently on the quality and quantity of my documentation, but it has not prevented others from contacting me after I've left to ask questions that were answered in my documentation with more information and clarity.
If the admin struggled with the basics of FTP and ESX, then there is a good chance that they struggled with networks too, so I’d expect it to be a factory config plus the minimum required.
As there is only one admin fo the whole environment then the environment can’t be that large either, so reverse engineering all of it should be fairly trivial. Sounds like a nice short term contract for someone in the local area.
Fully agree that an audit and some guidance on how to do things properly is in order.
> so I’d expect it to be a factory config plus the minimum required.
I would hope that one of the "about 15 different" ways to get in, was to try the default password...
One client I work with, I ensure they maintain all records they have of the various passwords used by their ex-IT guy and his email account. The chap left 4 years ago and even now they encounter situations where this list of passwords has proved helpful, in part because people do stuff and rather than go through the account/contact change process, simply leave everything as they found it. Unfortunately, they also tend not to communicate which set of credentials actually got them into a particular account/system etc.
When I was writing the Dartmouth Time Sharing System I realized that I was the only one who knew how to bring the system up. So one Saturday morning during experimental time sharing (when the sysprogs could hack away, cause crashes etc. - not a critical time for anyone) I made myself unavailable for bringing up the system. It took them 2 hours to bring up the system - there were really sharp people there - and then I had their attention so they listened when I explained the details of bringing up and running the system.
I maintain a website for the local square dancing federation. I made it quite clear that there should be people backups for me. The master password for the web site is known to several officers of the federation and to some others who are actually skilled to maintain the website. There is complete documentation for procedures for maintaining the website stored on the website which anyone is free to download (without the master password).
This same federation lost the entire subscriber database when the person managing it was secretive and stored everything on her personal computer. She died in a car accident and nobody could get into her password-protected computer.
I have seen in several companies, the "wilfull employee" and how much they sabotage attempts to discipline them. And how bad management are at dealing with them. The various futile attempts to enforce documentation and redundancy rules, the meetings to convince them to change their ways and the powerlessness of the hierarchy to enforce the rules.
I had one subordinate who refused to show me anything she was doing, and would lock her terminal every time I approached. If I asked her to do something she didn't want to do, she would claim she was working on something far more important and then refuse to elaborate (what she was doing was stuffing around with linux settings she was unqualified and unauthorised to change, resulting in several outages). But as a team leader, I had no authority to discipline her and our manager's nickname was Homer (fat, bald and really "yellow"), so no resolution there. When I would email her with work (or put it on her desk), she would simply ignore it or place it back on mine (one charming instance, I gave her a print out of a Priority 1 issue to resolve, went to the computer room for an hour and came back to find the printout back on my desk and the issue still outstanding)
I would love to hear anyone who has actually gotten a rogue employee either fired or buttoned down.
I ran across that four times in the 9-5 portion of my career. I handled it by carefully documenting everything, and once I had my ducks in a row, I went over "Homer's" head. I got fired for this effrontery once (turned out that Homer was the owner's nephew). The other three times, I was given Homer's job. Two of those three times it would have been easier getting fired. Choose your battles wisely.
We had a particularly inept senior developer who had done the rounds of the development teams. I ran the tech support team and was told in a management meeting that it was my turn to have him. I responded that's fine you'll have my resignation letter on your desk by the end of the day.
We had a restricted access office with access to hardware consoles for mainframes, the x.25 network management station etc, we were just introducing UNIX and had root / sysadmin access to every server in the organisation.
I had spent 3 years turning around a failing team and was delivering to all SLAs with a 5 9's availability record and was not prepared to inflict this asshat on a team who had worked so hard to turn things around.
My intransigence ended up starting a wave of revolt amongst the other team managers and no technical manager was willing to have him back due to the damage he did to service delivery and team morale. This lead (eventually) to a disciplinary process and the exit of the guy from the organisation voluntarily. He beet being fired by about 2 hours as he resigned just before his disciplinary hearing.
This tactic worked because I actually was willing to resign over the principle.
The system is still working. Therefore, the sadly departed admin likely wasn't as bad as he is made out to be.
Lots of system problems are due to 'finger trouble'. Not having the password effectively prevents that, with the result that the system is still working, long after the admin moved on to the great cloud network in the sky.
So, why exactly are they annoyed with him?
On crucial core network kit (at small firms), I often print the password and attach it to the device (along with IP address, SSID, etc). Normally the back/underside. Anyone with enough physical access to see it could do a factory reset (or just unplug it) anyway.
I know that I won't be around forever, and I also know that there's unlikely to be a formal, structured, handover when that time comes. I want to make it as easy as possible for my successor because when the shoe has been on the other foot I have appreciated such acts too.
There was a joke about a prison van colliding with a concrete truck and ended with the police looking for a gang of hardened criminal. Took my fancy so I used concrete truck instead of bus after that.
I was looking after a small school with about 300 devices and any time I changed a password, I email it to the principal and the IT Co-ordinator (aka ITC) and to my account which was accessible to the agent I contracted for/to and also update the electronic documentation. The first 3 or 4 times I would mention that I had changed the password and emailed the new password to specific staff just in case I got hit by a concrete truck on the way home. The prick principal laid a complaint against me because I was going round making suicidal comments. Just couldn't win at that site.
Are all well and good, so long as the person tasked with keeping them knows if they're accurate or up to date.
I've taken over IT networks, been handed a "handover package" which comprised of nothing more than "we have switches, servers and computers." or being shockingly out of date.
In the face of an 'inaccessible switch, no known config, mission critical' situation am I the only one to think of the bleedin' obvious? As in: check the configs of all the *other* devices plugged into that switch and thus just reverse-engineer what it was doing? Switches don't work in isolation. It's not that hard. Been there, done that. Doing it again at the moment, as there's one on a network I've just 'inherited' :-(
And this is the reason why there used to be a backdoor by using the direct serial connection.
I the "trunk" line feeding the switch is on a single VLAN and not showing up in a traceroute, it can be thought to be a dumb switch. Most likely it has a 802.1q trunk line and may require additional network tracing. This tracing may include visiting all devices connecting to the switch and seeing the settings there.
Ah, the good old days of tracing a ethernet cable through the network floor......
Ok couple of things, first one; almost all the managed switches I came accross (used in businesses) has a local console port and anyone with the right tools can reset a lost password without loosing any configuration information. I guess people are getting too comfortable with click here and click there user interface. Start typing. Second, the guys at the company saying for a coworker "not a good engineer" yet "STRUGGLING TO REPLACE AN EXPIRED ESXI CERT" at the time. Well I guess the whole IT department needs to go considering what happened and what is happening.
"No clue about VLANs, no clue about if it has STP, or trunking, or anything."
Yes you do - if you have access to the attached switches/devices to the core then you have the other half of the config. Totally agree with the previous comments on the mgmt side should ensure this situation doesn't happen, and backups (all things fail human and silicon) but as a network guy with 20yrs +, STP is a mutual thing as should be the vlan and trunking (packet sniffer will show up most of this). You will need down time for the replacement but hunt for the clues first on the wire and attached devices first if you really are situation. You should not get in to this situation and is bad working practise from all parties, it is a fire fight but not all is lost (if you do not know what your core switch is doing for your critical traffic with out seeing the config please find the door....oh you cant do that with out being provided a map I see your problem!)
>You will need down time for the replacement but hunt for the clues first on the wire and attached devices first if you really are situation.
This is what Dylan should be doing now, aided by Dell support!
Such preparation means that the most can be obtained from the scheduled downtime in April.
Perhaps Dylan has received guarantees from Dell that with the network down they will be able to gain access to the switch without losing the configuration. But even then with senior management visibility and attention, Dylan would be well advised to do the leg work and document the As-Is network configuration; in part to provide basic information necessary to start configuring a replacement device - something I would be doing in any case - in part because it is highly likely the existing network gear is running old software..
Not quite the same, but I got a query this morning from a guy asking if I happened to have a copy of an old Cisco IOS because a customer had an ancient switch that held a vital part of their network together that had gone down
(as it happened we did have something suitable on an equally ancient XP PC with 20Gb drive running as a TFTP server... plenty of space left on the drive, so we never bothered deleting old files, and no big deal if it went BANG! tomorrow)
Almost ALL Enterprise grade manageable switches and routers have a password recovery method upon booting the device and having physical access. Dell is no exception.
How they can not recover this device is beyond me. This place sounds sketchy with untrained technicians running everything.
How they can not recover this device is beyond me. This place sounds sketchy with untrained technicians running everything.
Might be something hidden in the bits about "critical infrastructure" and requiring scheduled downtime a month or two in advance?
Sure, it would be nice if every manager and the like knew exactly how every bit of network kit was run, and they knew how to build to best practices etc (including not relying a single piece of kit that cannot be shut down for a few minutes without a quick way to bring things back up - they do have a 4 hour replacement contract after all).
But then if managers could do our jobs, we would not have jobs now would we? If the boss could build the network, why hire someone else? And if you hire someone who passed enough of your checks to be allowed to build your network, there's a fair bet he could at least fake it well enough to appear that he could be trusted to do his job.
Sure, it's a simple matter of rebooting the switch. But sometimes, that is not so simple. I used to run a couple of seperate lots of web/email server (with mirroring) in different locations, and if I had to reboot either routers or servers I could change the DNS, wait a bit (set to 15min TTL IIRC), then reboot knowing full well that if anything failed to come up no one but me would know. Not every one has this set up though.
Until you get one that isn't. I've worked for myself most of of the time but there have been some employee jobs mixed in. It was years and years until I finally got a manager that knew his job and did it well. They're a pretty rare breed and this situation is due to a poor manager.
Systems change all of the time. When I worked in aerospace, stuff changed at least once a week. This forced me to come up with a way of documenting the hardware I was working on in a way that was easy to update. I also would document from 3 different approaches that each worked the best depending on what made the most sense when troubleshooting. That spilled over to how I managed the software on my computer. Since I was the lead avionics person, being the only avionics person, all of the e-CAD was on my computer along with licenses for Solidworks and other other SimWare. On my desk was a folder that contained all of the serial numbers, logons and passwords so if I were to get hit by a bus somebody could pull up whatever they needed that wasn't checked in at the time as a .pdf or other portable format. The folder also contained a whole bunch of other inane company procedure crap so it wasn't obvious that there was a page of credentials in there. Security wasn't a huge issue since there were only about 9 of us in the design office. When we had a proper engineering manager, he got a copy of my notes to keep and made everybody do the same thing. With just 9 people, nearly all of us were a one person departments. A big part of my job became documentation management since we started doing some government contract work and The Man is all about paper. Yes, it took away time from my working on electronics, but it also helped the company win a million dollar aerospace prize after we had a fire and had other people come in to help recover in one long night. A whole system was rewired with a single error while I slept.
I still got grief for spending time documenting stuff and trying to weed out the endless useless files in SVN that were automatically checked in since nobody ever received any training on which bits go in and which get tossed out.
If there is something in a company that is massively critical, it can't bottleneck through one person. Even worse if the person is a complete tosser. There has to be somebody in management that skips the 3 martini lunches and thinks about "What happens if this breaks?" What happens if the power goes out? What happens if this person leaves with no notice/dies/gets sick/defects to a competitor? If it's a janitor, easy, hire a new janitor. If it's an EVP, what procedure needs to be in place to do a secure exit process? If it's the guy in IT that has all the passwords, hire in a third party analyst to figure out what passwords that person better hand over to keep their job and good reference.
Biting the hand that feeds IT © 1998–2019