back to article Dead LAN's hand: IT staff 'locked out' of data center's core switch after the only bloke who could log into it dies

An IT department is pulling its hair out this month after realizing a coworker who died last year was the only person who could log into a crucial network switch. This is according to Dylan, a sysadmin at a small US healthcare company, who today told El Reg a story of how he and his colleagues ended up locked out of the …

  1. Notas Badoff

    All right, what's all this, then?!

    So they needed a network audit before firing him. And they really needed an audit after he 'left'. And now they need a miracle? Oh, and an audit.

    1. Yet Another Anonymous coward Silver badge

      Re: All right, what's all this, then?!

      "talk -ouija" isn't an option ?

      1. Unicornpiss Silver badge

        Re: All right, what's all this, then?!

        Perhaps they should consult with Oracle. After all, Oracle's some of Oracle's old icons were a ripoff from Ouija's pointer or "planchette" as I think it's called.

        1. Mage Silver badge
          Coat

          Re: Oracle's old icons

          What icons? A Planchette has none. Original had a hole for a pencil for 'automatic writing', later version was a pointer for the Ouija board, which might only have decorations, no icons, as it's Yes, No and alphabet. Released originally as a game and a game company still has the name as a trademark.

          So what are these Oracle icons?

    2. Mark 85 Silver badge

      Re: All right, what's all this, then?!

      I don't think an audit will solve the problem... It might have prevented it but it's too late now. Maybe the answers are on the mysterious #4 server as I've known at least one admin who kept a server "hidden" for emergencies. Turned out to have config info, some nasty info on manglement, and a pile of server and network scripts.

    3. Anonymous Coward
      Anonymous Coward

      Re: Speed impacted my children's education

      Selling audits is so much fun.

    4. Anonymous Coward
      Anonymous Coward

      Re: All right, what's all this, then?!

      I'm the AC who mentioned Radius/TACACS farther down in the comments as being helpful. Here's a link with brief explanations of AAA, RADIUS and TACACS+. http://www.pearsonitcertification.com/articles/article.aspx?p=2449614

      I have no ties to Pearson and am not able to speak to the efficacy their certification course ware.

      also see:

      http://wiki.freeradius.org/guide/Getting%20Started

      http://networkradius.com/doc/FreeRADIUS%20Technical%20Guide.pdf

  2. Anonymous Coward
    Anonymous Coward

    Sounds like poor management allowed this to happen.

    1. jake Silver badge

      ITYM

      "It was clearly bad management which allowed this to happen."

    2. smudge Silver badge

      Indeed. "Over-reliance on key personnel" is considered in even the most basic of risk assessments.

      1. doublelayer Silver badge

        That is true, but it really depends how large the place was. For example, I am mostly a developer, but I volunteer some system administration for a local charity that I appreciate. They used to have an administrator, but they left and they don't have that many systems. When I arrived to look over the systems and start my work, I found the following:

        1. A server that contains a domain controller and shared network folders.

        2. A UPS for aforementioned server. Not plugged in to the wall socket or, thankfully, the server.

        3. A backup system that seemed to be set up properly. It used removable disks that were swapped out every week, when there was an administrator. Since that admin left, they had one disk inside the system that contained the most recent backup and two disks that contained backups from eight months previous.

        4. A firewall that nobody had the access codes to. Nobody knew what this firewall was or wasn't doing and I just wanted to get rid of it once I felt confident to rebuild the network.

        This is what happens when there is only one person working on the system and the company lacks the ability to manage that person. The charity is small, the director is nontechnical, and the system was consequently chaotic. There wasn't a clear person at fault, but we could all agree that there was a problem.

        1. Olivier2553 Silver badge

          Similar situation.

          I volunteered to take care of the network in my condo because the existing situation was a mess.

          Well, not completely a mess, at the origins, it had been set-up properly, 2 WiFi per floor, one in each wing of the building, a WiFi bridge over to the other building, using Linksys and Ubiquity (I mean, not the cheapest possible hardware), under DD-WRT... So a good built. But obviously, the company managing the system was not the one that once installed it, so when the network hanged, the only solution was a power-off to reboot the system (and because there are 15 WiFi, it is faster to turn off the central power of the building! I joke you not. Just hope nobody is in the lift during these 5 seconds!).

          So, that was the situation when I moved in 2 years ago, so I volunteered to look after the system, I changed the main router/authentication platform and now I am left with a bunch of WiFi under DD-WRT without knowing the password.

          If the few hacks I found online are not working, I will be left with the solution of a reset-reconfigure, but I am not really looking forward to that.

          1. doublelayer Silver badge

            I would suggest, if possible, that you get them to buy new hardware and set up a replacement in parallel. Otherwise, I hope you are good at network administration. I have tried long enough to get a multiple-AP network going with multiple openwrt devices, and I found it to be a terribly long and painful process involving far too much fiddling with DHCP. I'll be the first to say that my network admin experience is suboptimal, but there is still a lot of complexity and ways for that to completely fail.

          2. Anonymous Coward
            Anonymous Coward

            Ubiquiti

            Having had to adopt a Ubiquiti wifi network left in a similar state, I'd say you're not in too bad a place. Not sure what you're using as a controller (an old laptop running debian will do it fine) but, as long as you use the same SSID and passwords, you can wipe and reset the whole system pretty easily - start with one AP (reset, then adopt) and then roll it out from there when you're happy with the config. The Ubiquit forums are your friend in this situation.

          3. M.V. Lipvig
            Trollface

            "I volunteered to take care of the network in my condo "

            Found the problem.

    3. Anonymous Coward
      Anonymous Coward

      Happens in every IT department I've worked in. There are always staff members who see knowledge as power and always managers who allow it to persist.

      I have audited my own current IT department (I work in infosec) and have pointed this issue out repeatedly over the past couple of years, it's in almost every report I write but still isn't addressed. They state that since they have sharepoint up and running this isn't a problem, but as always there's bugger all on there from the staff members in question. It's all full of content from the team-minded people, the ones who play well with others etc.

      Thing is one staff member in particular has been suspended several times in the past few years for not following change management processes and causing the network to drop, screwing DNS, breaking the web filtering etc. He's one of those who is a full believer in "knowledge is power" and has an inept manager who won't challenge him. I am desperate for that staffer to end up at a disciplinary so I can grill him but until then my hands are tied.

      It's bloody frustrating, not just for me but all the IT staff who have to put up with this crap.

      1. Anonymous Coward
        Anonymous Coward

        There are always staff members who see knowledge as power

        Ah, I think that's where I've gone wrong then. I've always sought to be professional and leave documentation so that someone coming along after me has something to work from. Of course, that makes me dispensable ...

        Also, I happen to know that at my last job my carefully engineered and fully documented network was ripped out by a "I don't understand it so I'm ripping it out" imbecile who only thinks he has a clue about networking (barely understands the basics of IPv4 addressing). Amusingly, I'd had no outages on the DNS we hosted for something like 600 customer domains for several years thanks to proper engineering with suitable redundancy - and he killed about 100 that were left on our servers when he killed the master and didn't know he needed to promote a slave before the zones timed out ! Prior to that we'd had a policy in the technical section of doing detailed network diagrams when doing a customer install - IP addresses, WiFi details, all the useful stuff that an engineer could do with to look after the network. Same person demanded all that useful stuff be removed because it didn't make for a "pretty drawing" to hand over to the customer (we kept our hand drawn originals !)

        At my previous job to that, I had a database of what was patched to where, a database of all IP allocations, detailed network diagram, where all the data sockets were located, the patch leads were colour coded by function, etc. Before long all the patch leads were replaced with black (and didn't use the management bars well), the database was tossed, etc, etc ...

        It's "slightly disheartening" to know that you've left a situation where any half-capable person can come in, look at the documentation, and take over without too much hassle - only to find they've replaced you with a monkey (in at least one case, that's being unkind to primates !) who's set out to wreck things.

      2. Kiwi Silver badge
        Devil

        I am desperate for that staffer to end up at a disciplinary so I can grill him but until then my hands are tied.

        Er, you don't read much of the BOFH do you?

        1. jake Silver badge

          "you don't read much of the BOFH do you?"

          No, I don't. It was tired & derivative of itself before it left Usenet.

          But thanks for asking.

  3. stu_san
    Joke

    Old cartoon

    I recall an old cartoon (from Datamation? no idea) of a mother and children who are mourning next to a grave in the rain. A portly gentleman is leaning close and saying, "I know this is a bad time, but do you remember him saying anything about source code?"

    1. jake Silver badge

      Re: Old cartoon

      I think I saw that one in Infoworld. Probably alongside one of Metcalfe's columns poo-pooing Linux & FOSS back in the mid-late '90s.

    2. Anonymous Coward
      Anonymous Coward

      Re: Old cartoon

      There is a picture in reddit https://i.redd.it/ejlc2pmrd9n21.jpg where the joke is that the angry girl is going to shoot the author's chromebook and the script has him say "don't do that that is the only copy of my book". Most commenters note that the author is probaby writing in G Suite and only needs to get a new chromebook if she shoots. :-)

      Maybe time to start clouudifying configs. :-)

  4. revenant Silver badge

    There's help out there ...

    If it wasn't for the fact that this switch was so integral to their network, a simple solution might be to expose the thing to the internet and offer a bounty to the first enterprising hacker who could get in, reset the login credentials and then report in with the solution.

    1. jake Silver badge

      Re: There's help out there ...

      Reading between the lines, chances are good that this has already occurred ... Well, the "cracker got in" bit, anyway.

    2. Terry Barnes

      Re: There's help out there ...

      I don’t know, I think you’d be inviting someone malicious to hold the config to ransom.

      1. jmch Silver badge

        Re: There's help out there ...

        "I don’t know, I think you’d be inviting someone malicious to hold the config to ransom."

        How is that a bad thing? They don't have the config anyway, I'm pretty sure they would be willing to pay a reasonable amount to get it back.

        1. ChrisBedford

          Re: There's help out there ...

          Yeah, but given the predominant mentality out there, the chances are good that someone even more malicious gets in first and without backing it up screws the entire config so nothing works.

        2. Terry Barnes

          Re: There's help out there ...

          Malicious as in - pay us or we delete it and hard reset the switch back to factory defaults. No IT problem is solved by inviting criminals to solve it for you.

  5. Kicker of Metaphorical Cats

    BOB

    The password is BOB. Unless he was dyslexic, in which case it will be BOB.

    Seriously, to blame a lame engineer for this stuff is ridiculous. The guys boss should be on the chopping block for allowing his network to be run that way. Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process.

    1. LDS Silver badge
      Coat

      Re: BOB

      The only bad admin that changes the default password form root/calvin?

      1. stiine Silver badge

        Re: BOB

        And I change them both, so neither root nor calvin are valid.

      2. gerdesj Silver badge
        Windows

        Re: BOB

        That's iDRACs you insensitive clod.

      3. bpfh Silver badge

        Re: BOB

        Which just reminded me of a few years ago where scott/tiger got me into more databases than it should, but I blame that on the consulting firms that had excellent sales in managing to sell Big Red To small businesses where Access would have done...

    2. jmch Silver badge

      Re: BOB

      "Managers get a pass when someone goes rogue, but not when they ignore an ongoing problem with a critical process."

      Well, yes agreed, but also, say a manager asks sysadmin for full audit of network including providing backup configs and passwords, the manager is still dependent on the sysadmin's honesty and competence, either of which could be lacking. How would the manager even notice if anything related to this particular switch was missing, given that it was literally months before anyone even noticed the switch was there?

      1. phuzz Silver badge

        Re: BOB

        Not to mention the process usually goes

        "Bob, I need a full network inventory...oh wait, the sales department are complaining that their VPN isn't working, can you fix that first..."

        Aaaaaaaaaaaaaaaaand the audit (and any other documentation) never gets done because it's always less important than keeping everything running.

        We have a core switch that only one person can log into, not only because they are the only person with a password, but also because it's broken and will only respond to packets from certain MAC addresses.

        1. nerdbert
          Pint

          Re: BOB

          Exactly this.

          Manager: So we need to document this flow. And we to fix these things ASAP.

          Me: You can have one or the other.

          Manager: Why? Why can't we have both?

          Me: Because you insisted on a three hour meeting, face to face in a conference room to go over this. Now I have time to do one or the other.

          Manager: Hmmmmm... Ok, just fix the things. Then we'll have to have another meeting to go over the plan to document everything.

          Me: *sigh* Sure thing, boss. Just put the meeting notice on my calendar. (Screaming inwardly to myself, "And you've still learned NOTHING!")

          Just a lightly edited extract of a real situation. Much of the problem is that for managers their work product is meetings, so they think things are getting done if you have a meeting about a subject rather than actually working on the problem.

          1. swm Silver badge

            Re: BOB

            I once had a boss who wanted a task done in three months. He then asked for another task taking two days. I said, "OK, but the three month task is now three months and two days."

            He had an interesting look on his face.

      2. Anonymous Coward
        Anonymous Coward

        Re: BOB

        This touches on something that's been really pissing me off lately.

        I consider myself a fairly competent IT person. I've worked as a systems admin, databases admin, software developer, and a network engineer for two ISP's.

        Lately, I've been applying for a bunch of IT manager positions. I'm not having much luck because I'm told I need x years of management experience. They don't seem to give a shit about my 26 years of technical experience!! No, I could have a shit technical background, but if I had 5 years of experience as some PHB, I would be in.

        I (being a little pissed about getting turned down without even an interview) even mentioned to a couple of companies that maybe if they hired managers with real technical skills, it may make things run better at their company?

        1. RFC822

          Re: BOB

          Why do you think that 26 years technical experience qualifies you to be a manager? Do you think that 26 years management experience would qualify somebody for a technical position?

          1. jake Silver badge

            Re: BOB

            What RFC822 said.

            Free advice for new sysadmins: Take as many business related courses as you can stomach. Haul your ass to your nearest post-secondary school that offers night courses and talk to a career counselor. Tell 'em that you are a techie, but are interested in management. You want to take courses that can be applied to a future MBA (should you want to go that route later).

            If you already hold a four year degree, and you can code fluently in one or more upper level languages, chances are you can snooze through an MBA in two years (or less, if the classes line up right). Lest you think getting an MBA is difficult, think about all the feckless idiots you know who hold one ;-)

            I realize that not all of us are cut out for management ... the objective isn't necessarily to become a manager, but rather to learn their lingo. It's amazing how fast long-closed doors open once you learn to talk to Moneybags in his/her own language. On top of that, an MBA will better prepare you for when the time comes to strike out on your own and become a consultant.

    3. Tom 7 Silver badge

      Re: BOB

      That would involve Manglement having a clue! Have you ever tried to get them to understand what their own job is let alone yours!

      1. Anonymous Coward
        Anonymous Coward

        Re: BOB

        I would suggest the password is just the letter a

    4. Anonymous Coward
      Anonymous Coward

      Re: BOB

      If Nick is as bad as they say, no one tried "password"?

  6. Zarno

    In a prior life, I was tasked with the boots-on-ground closet to closet network hardware audit of an office complex.

    Serial numbers, model numbers, port count, etc etc. One of the duties was pulling configs from everything that could give them up.

    Was it password protected without the password being known by anyone? Hold this button on the panel, connect via terminal emulator over the serial port, enter this obscure command, and then set this password after hoovering up the config.

    Mostly CISCO gear, but I'd assume with hardware access anything can be changed.

    Dell might just be evil though?

    1. Crazy Operations Guy Silver badge

      That is something you can do on Dell networking equipment. The problem is that they can't afford to take it down for the few minutes it would take to do that and why they are screwed until April when they have a scheduled maintenance period.

      From the sounds of it, this data-center is mission-critical and even a minute of downtime would be quite costly, especially since nothing is broken.

      1. Zarno

        I remember not having to do a reboot or introduce downtime, but IIRC it did flag physical access in the logs.

      2. Yet Another Anonymous coward Silver badge

        You are doubly screwed if for whatever reason it doesn't come back up - and a replacement is 4-6 weeks delivery

        1. pavel.petrman Bronze badge

          The article says they have a 4 hour replacement on it from Dell. This to me says something about it being a critical piece of network equipment, because why else would a company with _this_ level of management incompetence buy a 4 hour replacement.

          1. jmch Silver badge
            Happy

            "why else would a company with _this_ level of management incompetence buy a 4 hour replacement?"

            Erm... you're kind of answering your own question there :)

      3. jake Silver badge

        "especially since nothing is broken."

        How do you know nothing is broken if you can't log into it? Just because it is routing traffic properly? How do you know that on top of it's normal duties, it isn't logging all LAN traffic and shipping it off to your competitor? Or that it hasn't otherwise been rooted, given the lack of clues possessed by the former admin?

        To me, the box is a corporate hazard and needs to be airgapped, preferably last December!

        1. Crazy Operations Guy Silver badge

          Re: "especially since nothing is broken."

          Sure, you and I would call it broken, but Management sure as hell isn't going to agree. If money is coming into the company at or above, the normal rate means everything works.

          1. Anonymous Coward
            Anonymous Coward

            @Crazy Operations Guy - Re: "especially since nothing is broken."

            No problemo! I would put all this in writing to the management and wait patiently until it becomes really broken. Sometimes in the future they will have to approve the maintenance and it might be at an even worse moment than it is right now.

          2. CrazyOldCatMan Silver badge

            Re: "especially since nothing is broken."

            but Management sure as hell isn't going to agree

            "Err boss - we appear to have had a sudden power outage in part of the data centre and the switch went down for a minute or so.."

            "On the good side, I now have a full config of that switch and can log into it"

            "No - the two are not connected - why do you ask?"

      4. fajensen Silver badge
        Pint

        Pah. Rubbish.

        It is always "mission critical" right until it isn't - When the neglected plant blows up, falls apart, seizes up, burns up, then the "business decision" is to wait weeks and weeks by holding daily crisis meetings while the replacements trundles trough procurement and everything goes to hell.

        Since the beancounters cloaked themselves with power, it is always more important to "control costs" than it is to run the business.

    2. JJKing Silver badge
      Unhappy

      Bloody Dell.

      Went to add a new Server to a Dell UPS which had already had the software installed and configured by the previous tech but he hadn't documented the Admin password to the software. Called Dell nonHelpdesk to see how I could reset the password to default but no way could it be done. Was told that even uninstalling the software may leave the changed password in the Server.

      Not happy Jan!

  7. jake Silver badge

    I'll bet a dollar ...

    ... if you were to report the make (Dell, apparently), model and serial number (last 4 digits XXXXXed out) here, one of ElReg's esteemed readers would have an avenue of approach to get in. Might involve a soldering iron, but I've rarely been stymied when the chips are down and I have access to the hardware.

  8. Anonymous Coward
    Anonymous Coward

    UK Retailer who is no longer trading

    I was tasked to replace a failing tape drive on one of their stores servers, so I rang their Helpdesk to ask for the admin password so it could be shutdown gracefully, turns out they didn't use a password and you just typed in admin and click OK

  9. Anonymous Coward
    Anonymous Coward

    How is that only one person would have the password? All the configs should be backed up to a Rancid server and all the network engineers should have Radius/TACACS log in credentials. Several senior engineers or managers should be able to grant/restrict access to devices as required. The devices themselves should be restricted to having only a senior engineer or manager change critical passwords.

    1. Korev Silver badge
      Thumb Up

      I guess rancid is a typo... I think I'll use it from now on as it sounds much better than radius :)

      1. chris 143

        https://en.m.wikipedia.org/wiki/RANCID_(software)

        1. LDS Silver badge
          Facepalm

          "RANCID uses Expect to connect to the routers, send some commands and put the results in files"

          Very useful description... !

        2. Korev Silver badge
          Pint

          I've learned something. Thanks

      2. gerdesj Silver badge
        Gimp

        Rancid is a configuration management system for network gear. RADIUS .... isn't

      3. hmv Bronze badge

        You'll have a hard job backing up switch configs to a RADIUS server (rancid is an entirely different beast).

    2. Excused Boots

      And meanwhile back in the real world.........

    3. Anonymous Coward
      Anonymous Coward

      "How is that only one person would have the password?"

      The real world isn't always as sensible a place as it should be. In an ideal world, you would use RADIUS/TACACS to manage the device and have a local username/password safely stored away somewhere in case this failed. Or you have a single username/password in a restricted password safe.... Or default credentials that no one ever bothers changing... Or username/password just one person knows... All options...

      For recovery, it depends on the model as Dell has mixed and matched vendors across their switch range. With Cisco and some other vendors, you maybe able to use SNMP RW details to backup the config to a TFTP server. Or there may be a web GUI/API that gives you RO access to at least document the config. Even SNMP RO can provide a good chunk of information if you are trying to replicate a config, assuming you use SNMP management tools in your environment.

      Otherwise you are stuck with figuring out how it works. It may not be easier than you expect, especially if there's some basic documentation to go along with it or you have access to connected switches to get shared information off.

      1. Anonymous Coward
        Anonymous Coward

        @AC - Unfortunately

        SNMP RW and web GUI are among the first things I remove from configuration even though it has happened twice in my long career when these features were useful.

    4. Criggie
      Thumb Up

      Oxidized is the new RANCID (sorry bout the spelling)

      https://github.com/ytti/oxidized

      1. jake Silver badge

        "sorry bout the spelling"

        My Big Dic[0] prefers that spelling, damn your ise! What are you, French?

        [0] OED, second edition.

  10. lglethal Silver badge
    Joke

    Have they tried switching it on and off again.... Oh...Right.... Umm i better pass you over to level 2 support... *click*

    1. Korev Silver badge
      Joke

      Isn't this a level 8 issue? Although "Nick" may have now made it a level 0...

      1. chivo243 Silver badge
        Coat

        I think we've reached DEFCON levels now... I would think Nick is at level -6 at this time. Unless he was cremated.

    2. Tom 7 Silver badge

      RE switching it on and off again

      That is a good way of finding out if its actually doing anything useful. Often the only way!

  11. Anonymous Coward
    Anonymous Coward

    Been there...

    In a former life as an IT manager with a large company, I was ensured key passwords were stored off-site in a secure fire safe. I made it a matter of policy that "anyone can get run over by a bus" so procedures had to be in place to handle such an event. Unfortunately I was subsequently hit with a real bus which landed me in hospital briefly and wrote off my motorbike. The irony wasn't lost on me.

    Several years on, I was working as a contractor / software developer for my own business. I gave all relevant documents to a bookkeeper / accountant who did all the necessaries for submissions to Companies House, HMRC etc leaving me to get on with developing software. I didn't have much of a clue about book keeping and accounting. Then one day I got a letter out of the blue from a debt collection agency. I'd been fined for non-submission of annual accounts to Companies House. This was the first I'd heard, so I got straight onto the phone to my accountant... or rather spoke to his widow. He'd died unexpectedly a few months earlier from cancer. He was only in his thirties with a couple of young kids. His widow helped the best she could and between us we managed to piece together bits of information from random files, documents and spreadsheets on his computer so I could reconstruct my accounting position and make a late submission to companies house. It was a complete crash course in accounting for me.

    It is too easy to have too many eggs in one basket. Anyone can die unexpectedly and potentially leave you in deep doo-doo without adequate procedures in place. It was somewhat ironic that in my former employment I put procedures in place to handle dead-employee scenarios, but when I ran my own business I fell foul to a lack of such procedures. Sigh.

    1. H in The Hague Silver badge

      Re: Been there...

      "The irony wasn't lost on me."

      I feel your pain! For years and years I used to tell the guys in the workshop to make sure that rope/cables didn't present a trip hazard. You can guess who then tripped over a bit of string they'd thought they'd tied up neatly, but hadn't. Cue several v expensive visits to the dentist :(

    2. tim 13

      Re: Been there...

      Our secure offsite fireproof safe is at a bus station....

  12. Sandgroper

    As a (very) long time network engineer, I can't accept this at face value. If the only switch they were locked out of was the core switch, then the bulk of the config could be extracted from all of the neighbors. Once you have the information from the neighbors, then some network events debugging enabled on the neighbors would enable the bulk of the remaining missing information to be derived.

    1. Anonymous Coward
      Anonymous Coward

      ^ this.

      I'm befuddled how they couldn't know if it's running STP. Perhaps check its neighboring devices for evidence of BPDUs and TCNs? I suspect "Nick" isn't the only "not good" engineer at this idiot factory...

    2. Anonymous Coward
      Anonymous Coward

      Yeah... and we have only the new guy’s word that he was incompetent... if there’s one universal rule in tech it’s a new engineer slagging off his predecessor to management. You will never see lawyers, accountants etc do this. They understand that solidarity to the profession trumps any short term advantage they may gain from doing so.

      1. DavCrav Silver badge

        "You will never see lawyers, accountants etc do this."

        I assume that Patisserie Valerie's new accountants are not singing the praises of the previous lot.

        "They understand that solidarity to the profession trumps any short term advantage they may gain from doing so."

        'Solidarity to the profession' sounds quite a lot like 'close ranks and disavow any fault' to me.

        1. Anonymous Coward
          Anonymous Coward

          'I assume that Patisserie Valerie's new accountants are not singing the praises of the previous lot.'

          I don't see why not, ok, maybe not in public, but at least in private, as, after all, they've got a nicely paying gig out of their predecessors' sterling work, and as a bonus they've no doubt also got lots of doubleplusuncheap forensic work on the go as well just to keep the old expenses meter spinning that wee bit faster..

        2. Anonymous Coward
          Anonymous Coward

          It’s more about not being a backstabbing shitweasel, and knowing that your successor is likely to stitch you up too unless we all stop doing it.

          All we have in this story is that someone who evidently doesn’t know what he’s doing slinging mud at someone who can no longer defend himself

      2. Hollerithevo Silver badge

        Actually

        Lawyers, as officers fo the court, are required to report incompetent, fraudulent, crooked of dodgy fellow lawyers. They might have pride intheir profession, but I have known them to blow the whistle, somethimes descreetly, but never-the-less. OTOH I have seen IT colleagures closing ranks around a useless and dangerously-inept IT mate because simply went all dog-pack when one of them was threatened.

  13. stiine Silver badge
    Happy

    re: dell switches

    the ones integrated in the M100 chassis were easy to get into, pull the out, set/unset a jumper and plug them back in. The Juniper switches they resold can be broken from the serial console.

    Also, they most certainly can determine exactly what this switch/router is doing by reverse engineering. You just have to map it all out based on where the connection go, and how those devices/ports are configured.

    Been there, done that, still have the t-shirt...

    1. Excused Boots

      Re: re: dell switches

      Indeed, buy a new switch and rack mount it, have a good guess (and by guess I mean work out as best you can), what the problematic switch has to be doing, configure the new switch accordingly, swop all the connections over. Then it's a case of seeing who screams about loss of connectivity and adjust config accordingly.

      Management will absolutely hate this plan, just calmly point out that it's either this approach (where there is at least the possibility of backing out), or wait until the old switch catastrophically fails, at which point people start losing their jobs, possibly starting with the management!

      And afterwards you'll have a nice new (well actually old) switch to be factory reset and used for something else.

      1. Anonymous Coward
        Anonymous Coward

        Re: re: dell switches

        ... wait until the old switch catastrophically fails, at which point people start losing their jobs, possibly starting with the management!.

        Your argumentation, while factual, is not aligned with the usual incentives structure: Getting sacked only pops the Golden Parachutes for Management earlier than the planned and people losing their jobs will always boost stock prices, which will boost Management stock options even more, so it is a good thing.

        So? Worry about what you control: Write an ass-covering report/action-plan, present it at the management meeting, it will be rejected, and then leave it at that. Nobody cares. If you care, consider a different career.

  14. The Vociferous Time Waster

    Further advice...

    Don’t use dell network kit anywhere important.

    1. jake Silver badge
      Pint

      Re: Further advice...

      Well, yeah. But people do, and then we have to deal with it.

      Blargh. Pints all 'round.

    2. Anonymous Coward
      Anonymous Coward

      @The Vociferous Time Waster - Re: Further advice...

      Dell has nothing to do here. If I remember correctly there was a sysadmin in a Californian city who accepted to go to jail instead of providing access to Cisco switches* he was managing.

      * see "service no password-recovery" command on Cisco switches

      1. Anonymous Coward
        Anonymous Coward

        Re: @The Vociferous Time Waster - Further advice...

        This is why one should Radius/TACACS for authentication. You can provide various levels of control to your engineers. There are a variety of commands that engineers doing their daily work in a network do not need to have access and all of which can be controlled, at least using decent equipment.

  15. deevee

    Just contact the NSA or the FBI, they have built in backdoors to access everything. Lucky it isn't a Chinese/Huawei switch, otherwise it would be totally unhackable!

    1. Mark 85 Silver badge
      Devil

      Nah... if it's one of those just contact the nearest Chinese embassy. I'm sure they'd be glad to help out.

      1. Anonymous Coward
        Anonymous Coward

        Not outside of China and not for non-Chinese gov ;)

  16. Doctor Syntax Silver badge

    buy download a password manager

    1. Anonymous Coward
      Anonymous Coward

      And a time machine...

    2. Anonymous Coward
      Anonymous Coward

      > download a password manager

      Can anyone recommend a password manager (paid or not) that is multi-user and allows RBAC to certain passwords (to match the access allowed to the kit itself) but also allows "3 of 5" type overrides for when people leave, die etc?

      1. stiine Silver badge

        Even if you use a password manager, you still have to be in control of at least one of the three parts, and this depends how you configure access. You either need to be an administrator of the password manager, administrator of the directory service, or administrator of your 2fa system.

        If you manage the password manager, you can reassign resources to other users.

        If you manage the directory service you can modify group membership to grant access to other users.

        If you manage the 2fa system, you can reset/re-issue 2fa for accounts.

        These are questions for your password manager vendor.

        The things I recommend for companies that use password managers are, ensure that all interaction gets logged, and never allow a single individual to responsible for anything. This means having a second individual with admin permissions, even if they would only be capable of doing anything while on the phone with tech support.

        I use Password Manager Pro from ManageEngine.com. It runs on Windows or Linux and you can install a copy that is the full enterprise version for a trial. I have used the free version and the Enterprise version, and while the Enterprise version has some very nice features, my company's needs are such that the free version works just fine.

        I have a friend who's company uses LastPass, and they're happy with it.

        That being said, I think you should try several password managers and determine which works best for your situation.

  17. J.G.Harston Silver badge
    Mushroom

    We've been going through this sort of thing with our succession planning for our parish council (2.5 employees!). The God passwords are written down and stored in a safe, the key to which are held (locked up) by the two Responsible Officers and the (external) IT Auditor. If all three manage to die at the same time - especially something that takes the IT Auditor as well - it will probably be due to an event causing more concerns than parish council continuity. ;)

    1. Tom 7 Silver badge

      Do you have rules to prevent all 3 keyholders walking on the same side of the road on the way to the pub?

      1. j.bourne

        Or travelling together on the same bus/train/plane or in the same car etc...

        1. hplasm Silver badge
          Happy

          Or travelling together on the same GoodieCycle...

        2. J.G.Harston Silver badge

          The external IT Auditor is an external organisation. They would only ever approach our offices if both Clerk and Deputy Clerk fell under the same bus at the same time. And even if all the parish councillors went under the bus as well resulting in no executive authority, things would devolve up to the district council. If the district council was wiped off the earth, there's be other more important things to worry about.

  18. Maelstorm

    I had configs and layout documentation stored on a hidden server on the network. I backed everything up to it. Then I archived it and encrypted it so it could be stored in plain site on several servers, on tape backup, and store the tape in a bank vault. The network didn't change much, so this was a viable option. But when the network went down, the other admins had no clue how stuff was configured because they weren't doing the backups like they were supposed to. Some managers were let go. I figured out what the problem was and reloaded the router with the config off my hidden server. When I left, I still don't think they found the server.

    The backup server is a Raspberry Pi v1.0. It's in a small, non-descript case with a label that has the IP address and a message that reads "Critical network monitoring equipment. Do not disconnect."

    1. tellytart

      Hmm, wonder if I worked the same place as you! I wondered what a RPi was doing in one of the racks on the lower ground floor apps room with a label strikingly similar to yours!

    2. Anonymous Coward
      Anonymous Coward

      Congratulations, you are part of the problem.

      Running an organisation's IT/network is a -team- sport.

      1. jake Silver badge

        Horse crap.

        IT isn't any kind of sport. It's a job.

        Oh, wait ... you're one of those new-agers who expects a trophy for participating, and do-overs when you make a mistake, right?

        1. Kiwi Silver badge
          Pint

          Re: Horse crap.

          and do-overs when you make a mistake

          Isn't that the purpose of backups? :)

          1. jake Silver badge

            Re: Horse crap.

            No. Backups aren't a do-over. Backups are a roll-back mechanism.

  19. Unicornpiss Silver badge
    Meh

    Maybe he wanted it that way?

    It sounds like he was a bad admin. But he may have seen the writing on the wall about getting fired and purposely felt like making things as difficult as possible for his successor, not knowing that it would be his successor in a literal sense. People seem to be speaking ill enough of the dead in the article that it didn't sound like he was very cooperative.

    And I can't deny that in a crap job situation many of us would not have outright sabotaged anything, but would have felt this way, right or wrong. In the end, it was management's fault for letting things get this way, and possibly for both either maintaining an incompetent employee or so neglecting a once-useful employee that he ended up indifferent and apathetic.

  20. NateGee

    I smell a rat

    The deceased either knew the writing was on the wall so came up with a scenario that meant if he got fired he'd have some satisfaction from the shit-storm he'd leave behind or knew someone didn't like him and was trying to make it impossible for them to get rid of him.

    I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.

    1. Kiwi Silver badge
      Pint

      Re: I smell a rat

      I've worked with people in the past who were actually very good at their jobs but were constantly being bad mouthed by management. This was often when a PHB thought they knew everything (often loudly) and the colleagues were constantly being forced to prove them otherwise to the PHB's boss.,

      I had fun with that once or twice. The boss had been badmouthing me to suppliers, customers and contractors.

      One day there was a fault with the internet (telco end, not ours) and he came in while I was talking to one of said customers and mentioned it. I just nonchalantly said "Well, you know much more about networking than I do, I'll go off clock and watch in awe as you fix it".

      Both the customer and I found it quite a joy to be sharing a moment together watching him get more and more flustered as he proclaimed his knowldge, went in with a 'fix', only to have that fail so go with another thing. He'd heard me talk of a bad LAN card flooding the network with "martian packets" (an early outage before I got some resiliency built in) so decided one of the machines was causing that and unplugged them one by one, claimed bad cabling so opened up packets of patch cables (all the shop ones were made by me, the ones he opened were on the shelf IE saleable stock), and various other things..

      What made it more enjoyable for me is it was a "brief" (as in 3-4 hours) but major outage. The customer worked a couple of blocks away and had everything with the one telco so had lost phones and internet, and had come to me in person to see if I was aware what was up. As our net and phone line was also out, I'd already phoned the telco to make sure they were aware of a fault when the boss came on the scene. Given we could log into the router and it had a "disconnected" status icon, I was fairly certain the internal LAN was fine and the other side was dead, especially as someone a couple of blocks away had the same problem :)

      Love being able to show that kind of manager up when the time comes.

  21. Anonymous Coward
    Anonymous Coward

    It can become cost prohibitive

    for a small/medium size organization to protect against this kind of attack. You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen. Instead, the organization can outsource it completely and it's no longer a problem.

    1. whitepines Silver badge
      Alert

      Re: It can become cost prohibitive

      Instead, the organization can outsource it completely and it's no longer a problem.

      Of course then you have an entirely new set of problems starting with GDPR compliance, data leakage, cloud providers deciding you're last in line for a fix, connection troubles, being held for ransom with rate hikes or mandated software "upgrades", deciding your app isn't worth supporting any longer, etc.

      When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.

      But don't let that all get in the way of your cloudy sales pitch!

      1. Olivier2553 Silver badge

        Re: It can become cost prohibitive

        When just making sure the CEO had a copy of all passwords available would have fixed this at no extra charge.

        Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.

        As opposed of a CEO who is more likely to think it is their company and they can and know how to play with the password.

        1. whitepines Silver badge

          Re: It can become cost prohibitive

          Or better, the CEO's secretary, as that is a person that can usually be entrusted, but would never think of abusing the power that has been let with them.

          So you've seen this too. It's an interesting phenomenon, some form of "absolute power corrupts" I gather.

          One organization I worked with was effectively run by a very competent secretary (that was also a force to be reckoned with if you did wrong). Her "boss" was a miserable twat....

          1. jake Silver badge

            A good secretary is like an NCO in the army of your choice (was: Re: It can become cost prohibitive)

            They run the place.

            Free hint to all consultants: ALWAYS ask the secretary about the Boss's computer knowledge. You can save a lot of time and trouble for a lot of people over the long haul.

            I know of several CEO-types of Fortune-500s who make a big show of "checking the computer", even though their network cable was "accidentally" never installed.

            I can't count the number of times I've swapped the Boss's top of the line CPU, gathering dust and spiderwebs under his credenza/return, artfully changing screensavers every couple minutes, for his secretary's underpowered kit ... without the Boss noticing.

            1. Anonymous Coward
              Anonymous Coward

              Re: A good secretary is like an NCO in the army of your choice (was: It can become cost prohibitive)

              My late wife was (officially) the Financial Controller of a small craft brewery, but as the two Partners were complete imbeciles, who squandered their father's legacy, she actually ran the company. Then she was struck down by an incurable disease, and after she died, the company folded.

          2. This post has been deleted by its author

        2. Is It Me Bronze badge

          Re: It can become cost prohibitive

          When I was sole IT guy for a company I gave the HR guy a sealed envelope with the relevant passwords in and instructions to only pass it on if I was dead/injured etc.

      2. Anonymous Coward
        Anonymous Coward

        @whitepines - Re: It can become cost prohibitive

        You don't deserve the up votes in the same way I don't deserve the down votes.

        Are you trying to tell us the CEO should login three times a day just to make sure those passwords have not been changed ? It happened to me when organizations asked me to help and provided me with two or three written down passwords that were no longer valid. By the way I'm not a cloud stuff salesman, I'm currently configuring telecom equipment but I've learned there is more to security than TACACS/RADIUS and a password kept in an envelope in CEO's office. Clue for you, do a Google search for privilege separation and privileged access management and start from here.

        As someone mentioned in a post below, you're just changing the nature of the problem but your rogue admin scenario is gone. Yes, the outsourcer might have a rogue employee too but your company can hope to be compensated financially for the down time. As for compliance you'll have to do it anyway. Outsourcing or not, you're the owner of the data so act on securing it wherever it is stored. And it is better and cheaper to focus on securing data instead of caring for every switch and wiring closet door in addition to data. My career too is threatened but there's nothing I can do, developers have stolen the show. I would really like to adapt to the new reality but time is a serious constraint.

    2. stiine Silver badge

      Re: It can become cost prohibitive

      No, that just means that you've traded one problem (rogue employee) for another problem (rogue employee of outsourcer) but I agree with your statement "You need a complex mixture of technology, policies, procedures and auditing to make sure this doesn't happen.", but you left out the most important part: good people. If you don't have good people, you can buy tech, etc, to no avail.

      1. Anonymous Coward
        Anonymous Coward

        @stiine - Re: It can become cost prohibitive

        Just assume you'll never have the good people you'd like and use this assumption as a foundation for building the security in IT systems. Be trustful but put safeguards in place.

    3. Roland6 Silver badge

      Re: It can become cost prohibitive

      >for a small/medium size organization to protect against this kind of attack. ... Instead, the organization can outsource it completely and it's no longer a problem.

      It is still a problem, I've come across many small/meduim businesses that prove the rule that businesses like to do business with similar sized organisations. So the small business has outsourced their IT to a 1~2 man operation (1 tech who actually knows the passwords and where they are kept & 1 sales/consultant who instructs the tech but doesn't actually know how to access the tech's folders).

  22. FuzzyWuzzys
    Facepalm

    Read The Phoenix Project!

    This is exactly why companies need oversight, you need to badger colleagues to document and record, get organised people. I hate this idea that just because you're asked to document your work and record things, that you're getting the boot next week. No one is trying to steal your job! We need the knowledge so that we can keep you and your job. You vanish on holiday for 2 weeks and no one knows what you do you put the whole company a risk and everyone's job.

    Get your head out of your arse and get stuff written down, and sensitive stuff stored securely, even if it's just a print out given to the IT manager or even get Finance to put it in their safe. On the other hand the more you hoard knowledge, the more complaints HR may receive about you and then you're considered a risk for not complying with regs, this means they may find some way to demote or remove you for not complying.

    You may not be going full on "DevOps" but the TPJ has a lot of sensible ideas about ensuring work is documented, critical people who cause bottlenecks are utilised correctly and knowledge is recorded so others can step in when needed to keep companies in crisis moving along.

    1. amanfromMars 1 Silver badge

      Re: Read The Phoenix Project!

      And do you really want Imminent Failure to reveal Special Forces AIMissions for Universal Resolution with AISolutions for Systems in a Failure withTotal Collapse of Non-Cooperative Operating Systems ......with Exclusive Elite Executive Officer Suites in Stasis?

      Yes would be the Correct Answer there whenever ready for Everything Quite Wonderfully So Easily Different.

      And here be but one Blank Canvas upon which such Futures are Wrought and Writ with Almighty 0Day Provider Protection Testing Vulnerabilities to Distractions and/or Destruction with Enigmatic Ethereal Exploitation of Earthly Resources with Depleted Intelligence Sources.

      What do you Think? AI Fact and/or Pump Fiction? :-)>

      And just whenever Theresa was praying for it not to get any worse ........ An AI LifeRaft just Simply Appears out of Nowhere.

      That Project is Magic. Pure and Simple.

    2. doublelayer Silver badge

      Re: Read The Phoenix Project!

      That may be true, but companies will, at times, ask for documentation under the theory that I'll write down everything that a replacement working with a lot less knowledge and for a smaller bill can simply pick up. That isn't a reason to refuse to document, and I have never done that and would not suggest that anyone else do so. Still, some people don't understand that the hundreds of pages of documentation and procedures, while as organized and clear as I can make them, are long and require thorough reading to understand. I've been praised frequently on the quality and quantity of my documentation, but it has not prevented others from contacting me after I've left to ask questions that were answered in my documentation with more information and clarity.

  23. chivo243 Silver badge

    Where are they?

    Documentation, drawings, config backups, DR plans? I would be ripping heads off and shitting down necks of Nick's bosses right about now...

    There must be some documentation on Nick's workstation, something surely?

  24. StuntMisanthrope Bronze badge

    Rest in profile.

    Be excellent and careful. #tombstone

  25. Dwarf Silver badge

    If the admin struggled with the basics of FTP and ESX, then there is a good chance that they struggled with networks too, so I’d expect it to be a factory config plus the minimum required.

    As there is only one admin fo the whole environment then the environment can’t be that large either, so reverse engineering all of it should be fairly trivial. Sounds like a nice short term contract for someone in the local area.

    Fully agree that an audit and some guidance on how to do things properly is in order.

    1. Roland6 Silver badge

      > so I’d expect it to be a factory config plus the minimum required.

      I would hope that one of the "about 15 different" ways to get in, was to try the default password...

      One client I work with, I ensure they maintain all records they have of the various passwords used by their ex-IT guy and his email account. The chap left 4 years ago and even now they encounter situations where this list of passwords has proved helpful, in part because people do stuff and rather than go through the account/contact change process, simply leave everything as they found it. Unfortunately, they also tend not to communicate which set of credentials actually got them into a particular account/system etc.

      1. swm Silver badge

        When I was writing the Dartmouth Time Sharing System I realized that I was the only one who knew how to bring the system up. So one Saturday morning during experimental time sharing (when the sysprogs could hack away, cause crashes etc. - not a critical time for anyone) I made myself unavailable for bringing up the system. It took them 2 hours to bring up the system - there were really sharp people there - and then I had their attention so they listened when I explained the details of bringing up and running the system.

        I maintain a website for the local square dancing federation. I made it quite clear that there should be people backups for me. The master password for the web site is known to several officers of the federation and to some others who are actually skilled to maintain the website. There is complete documentation for procedures for maintaining the website stored on the website which anyone is free to download (without the master password).

        This same federation lost the entire subscriber database when the person managing it was secretive and stored everything on her personal computer. She died in a car accident and nobody could get into her password-protected computer.

        1. Roland6 Silver badge

          >when the person managing it was secretive and stored everything on her personal computer. She died in a car accident and nobody could get into her password-protected computer.

          Must have had full disk encryption enabled, so filesystem couldn't be read by another OS.

  26. OzBob

    Was his name Terry Childs Jr?

    I have seen in several companies, the "wilfull employee" and how much they sabotage attempts to discipline them. And how bad management are at dealing with them. The various futile attempts to enforce documentation and redundancy rules, the meetings to convince them to change their ways and the powerlessness of the hierarchy to enforce the rules.

    I had one subordinate who refused to show me anything she was doing, and would lock her terminal every time I approached. If I asked her to do something she didn't want to do, she would claim she was working on something far more important and then refuse to elaborate (what she was doing was stuffing around with linux settings she was unqualified and unauthorised to change, resulting in several outages). But as a team leader, I had no authority to discipline her and our manager's nickname was Homer (fat, bald and really "yellow"), so no resolution there. When I would email her with work (or put it on her desk), she would simply ignore it or place it back on mine (one charming instance, I gave her a print out of a Priority 1 issue to resolve, went to the computer room for an hour and came back to find the printout back on my desk and the issue still outstanding)

    I would love to hear anyone who has actually gotten a rogue employee either fired or buttoned down.

    1. jake Silver badge

      Re: Was his name Terry Childs Jr?

      I ran across that four times in the 9-5 portion of my career. I handled it by carefully documenting everything, and once I had my ducks in a row, I went over "Homer's" head. I got fired for this effrontery once (turned out that Homer was the owner's nephew). The other three times, I was given Homer's job. Two of those three times it would have been easier getting fired. Choose your battles wisely.

    2. Anonymous Coward
      Anonymous Coward

      Re: Was his name Terry Childs Jr?

      We had a particularly inept senior developer who had done the rounds of the development teams. I ran the tech support team and was told in a management meeting that it was my turn to have him. I responded that's fine you'll have my resignation letter on your desk by the end of the day.

      We had a restricted access office with access to hardware consoles for mainframes, the x.25 network management station etc, we were just introducing UNIX and had root / sysadmin access to every server in the organisation.

      I had spent 3 years turning around a failing team and was delivering to all SLAs with a 5 9's availability record and was not prepared to inflict this asshat on a team who had worked so hard to turn things around.

      My intransigence ended up starting a wave of revolt amongst the other team managers and no technical manager was willing to have him back due to the damage he did to service delivery and team morale. This lead (eventually) to a disciplinary process and the exit of the guy from the organisation voluntarily. He beet being fired by about 2 hours as he resigned just before his disciplinary hearing.

      This tactic worked because I actually was willing to resign over the principle.

  27. herman Silver badge

    The system is still working...

    The system is still working. Therefore, the sadly departed admin likely wasn't as bad as he is made out to be.

    Lots of system problems are due to 'finger trouble'. Not having the password effectively prevents that, with the result that the system is still working, long after the admin moved on to the great cloud network in the sky.

    So, why exactly are they annoyed with him?

    1. Martin Summers Silver badge

      Re: The system is still working...

      Read the article mate, also don't work in IT please. Thanks.

    2. hmv Bronze badge

      Re: The system is still working...

      Thanks a lot.

      My desk now has a new dent in it.

  28. Anonymous Coward Silver badge
    Alien

    Security

    On crucial core network kit (at small firms), I often print the password and attach it to the device (along with IP address, SSID, etc). Normally the back/underside. Anyone with enough physical access to see it could do a factory reset (or just unplug it) anyway.

    I know that I won't be around forever, and I also know that there's unlikely to be a formal, structured, handover when that time comes. I want to make it as easy as possible for my successor because when the shoe has been on the other foot I have appreciated such acts too.

  29. TrumpSlurp the Troll Silver badge
    Facepalm

    Not fired just died

    For all those commentards who have said that he was fired, the article says they were thinking of firing him when he died unexpectedly.

  30. JJKing Silver badge
    Facepalm

    Some people; no humour.

    There was a joke about a prison van colliding with a concrete truck and ended with the police looking for a gang of hardened criminal. Took my fancy so I used concrete truck instead of bus after that.

    I was looking after a small school with about 300 devices and any time I changed a password, I email it to the principal and the IT Co-ordinator (aka ITC) and to my account which was accessible to the agent I contracted for/to and also update the electronic documentation. The first 3 or 4 times I would mention that I had changed the password and emailed the new password to specific staff just in case I got hit by a concrete truck on the way home. The prick principal laid a complaint against me because I was going round making suicidal comments. Just couldn't win at that site.

  31. user0

    Was it running RIP or OSPF? Must have been RIP

    I'll get my coat.

  32. Sequin

    Was he also the only person to have the password to the laptop holding all of the bitcoin?

  33. andy gibson

    Handover documents

    Are all well and good, so long as the person tasked with keeping them knows if they're accurate or up to date.

    I've taken over IT networks, been handed a "handover package" which comprised of nothing more than "we have switches, servers and computers." or being shockingly out of date.

  34. Anonymous Coward
    Anonymous Coward

    Maybe check the configs on the kit plugged into it?

    In the face of an 'inaccessible switch, no known config, mission critical' situation am I the only one to think of the bleedin' obvious? As in: check the configs of all the *other* devices plugged into that switch and thus just reverse-engineer what it was doing? Switches don't work in isolation. It's not that hard. Been there, done that. Doing it again at the moment, as there's one on a network I've just 'inherited' :-(

  35. gnarlymarley Bronze badge

    serial connection hackable

    And this is the reason why there used to be a backdoor by using the direct serial connection.

    I the "trunk" line feeding the switch is on a single VLAN and not showing up in a traceroute, it can be thought to be a dumb switch. Most likely it has a 802.1q trunk line and may require additional network tracing. This tracing may include visiting all devices connecting to the switch and seeing the settings there.

    Ah, the good old days of tracing a ethernet cable through the network floor......

  36. Decox

    Easy fix

    They do know that you can just boot into recovery and just pull the running config.txt over to a machine and take out the encrypted login yeah? I'm a Cisco certified network admin and do work on government contracts, not that hard to do guys.

    1. Anonymous Coward Silver badge
      Facepalm

      Re: Easy fix

      How do you boot into recovery without interrupting service?

      1. dnicholas Bronze badge

        Re: Easy fix

        At night. Order pizza and tell everyone to **** off.

        That's how I do it

  37. OneITGuy

    Ok couple of things, first one; almost all the managed switches I came accross (used in businesses) has a local console port and anyone with the right tools can reset a lost password without loosing any configuration information. I guess people are getting too comfortable with click here and click there user interface. Start typing. Second, the guys at the company saying for a coworker "not a good engineer" yet "STRUGGLING TO REPLACE AN EXPIRED ESXI CERT" at the time. Well I guess the whole IT department needs to go considering what happened and what is happening.

  38. Kijoma

    They could of course attack other monitoring routers set as bridges to log what vlan's and traffic eminate from this things ports and build up a picture of it's config ? Use the Mikrotik ones as they are cheap enough unless you have a cisco fetish.

  39. W60
    Facepalm

    rubbish statement - takes two to tango

    "No clue about VLANs, no clue about if it has STP, or trunking, or anything."

    Yes you do - if you have access to the attached switches/devices to the core then you have the other half of the config. Totally agree with the previous comments on the mgmt side should ensure this situation doesn't happen, and backups (all things fail human and silicon) but as a network guy with 20yrs +, STP is a mutual thing as should be the vlan and trunking (packet sniffer will show up most of this). You will need down time for the replacement but hunt for the clues first on the wire and attached devices first if you really are situation. You should not get in to this situation and is bad working practise from all parties, it is a fire fight but not all is lost (if you do not know what your core switch is doing for your critical traffic with out seeing the config please find the door....oh you cant do that with out being provided a map I see your problem!)

    1. Roland6 Silver badge

      Re: rubbish statement - takes two to tango

      >You will need down time for the replacement but hunt for the clues first on the wire and attached devices first if you really are situation.

      This is what Dylan should be doing now, aided by Dell support!

      Such preparation means that the most can be obtained from the scheduled downtime in April.

      Perhaps Dylan has received guarantees from Dell that with the network down they will be able to gain access to the switch without losing the configuration. But even then with senior management visibility and attention, Dylan would be well advised to do the leg work and document the As-Is network configuration; in part to provide basic information necessary to start configuring a replacement device - something I would be doing in any case - in part because it is highly likely the existing network gear is running old software..

  40. mildy bemused

    No Postit on the poor chap's monitor?

  41. Christian Berger Silver badge

    Uhm... wait for the downtime...

    ... shut it down, pull the CF-card and read the configuration if it's a larger device.

    Alternatively boot into "factory config", log in with the default credentials and read the starup config.

  42. Anonymous Coward
    Anonymous Coward

    Not quite the same, but I got a query this morning from a guy asking if I happened to have a copy of an old Cisco IOS because a customer had an ancient switch that held a vital part of their network together that had gone down

    (as it happened we did have something suitable on an equally ancient XP PC with 20Gb drive running as a TFTP server... plenty of space left on the drive, so we never bothered deleting old files, and no big deal if it went BANG! tomorrow)

  43. darksurf

    Surely this is a joke

    Almost ALL Enterprise grade manageable switches and routers have a password recovery method upon booting the device and having physical access. Dell is no exception.

    https://www.dell.com/support/article/us/en/19/how12045/how-to-recover-from-forgotten-password-on-dell-networking-n-series-switch?lang=en

    How they can not recover this device is beyond me. This place sounds sketchy with untrained technicians running everything.

    1. Kiwi Silver badge

      Re: Surely this is a joke

      How they can not recover this device is beyond me. This place sounds sketchy with untrained technicians running everything.

      Might be something hidden in the bits about "critical infrastructure" and requiring scheduled downtime a month or two in advance?

      Sure, it would be nice if every manager and the like knew exactly how every bit of network kit was run, and they knew how to build to best practices etc (including not relying a single piece of kit that cannot be shut down for a few minutes without a quick way to bring things back up - they do have a 4 hour replacement contract after all).

      But then if managers could do our jobs, we would not have jobs now would we? If the boss could build the network, why hire someone else? And if you hire someone who passed enough of your checks to be allowed to build your network, there's a fair bet he could at least fake it well enough to appear that he could be trusted to do his job.

      Sure, it's a simple matter of rebooting the switch. But sometimes, that is not so simple. I used to run a couple of seperate lots of web/email server (with mirroring) in different locations, and if I had to reboot either routers or servers I could change the DNS, wait a bit (set to 15min TTL IIRC), then reboot knowing full well that if anything failed to come up no one but me would know. Not every one has this set up though.

  44. dnicholas Bronze badge

    Well at least...

    ... he changed the default username and password.

    That's several steps above some of the messes I've had to sort out.

  45. MachDiamond Silver badge

    Managers are shite

    Until you get one that isn't. I've worked for myself most of of the time but there have been some employee jobs mixed in. It was years and years until I finally got a manager that knew his job and did it well. They're a pretty rare breed and this situation is due to a poor manager.

    Systems change all of the time. When I worked in aerospace, stuff changed at least once a week. This forced me to come up with a way of documenting the hardware I was working on in a way that was easy to update. I also would document from 3 different approaches that each worked the best depending on what made the most sense when troubleshooting. That spilled over to how I managed the software on my computer. Since I was the lead avionics person, being the only avionics person, all of the e-CAD was on my computer along with licenses for Solidworks and other other SimWare. On my desk was a folder that contained all of the serial numbers, logons and passwords so if I were to get hit by a bus somebody could pull up whatever they needed that wasn't checked in at the time as a .pdf or other portable format. The folder also contained a whole bunch of other inane company procedure crap so it wasn't obvious that there was a page of credentials in there. Security wasn't a huge issue since there were only about 9 of us in the design office. When we had a proper engineering manager, he got a copy of my notes to keep and made everybody do the same thing. With just 9 people, nearly all of us were a one person departments. A big part of my job became documentation management since we started doing some government contract work and The Man is all about paper. Yes, it took away time from my working on electronics, but it also helped the company win a million dollar aerospace prize after we had a fire and had other people come in to help recover in one long night. A whole system was rewired with a single error while I slept.

    I still got grief for spending time documenting stuff and trying to weed out the endless useless files in SVN that were automatically checked in since nobody ever received any training on which bits go in and which get tossed out.

    If there is something in a company that is massively critical, it can't bottleneck through one person. Even worse if the person is a complete tosser. There has to be somebody in management that skips the 3 martini lunches and thinks about "What happens if this breaks?" What happens if the power goes out? What happens if this person leaves with no notice/dies/gets sick/defects to a competitor? If it's a janitor, easy, hire a new janitor. If it's an EVP, what procedure needs to be in place to do a secure exit process? If it's the guy in IT that has all the passwords, hire in a third party analyst to figure out what passwords that person better hand over to keep their job and good reference.

  46. dnicholas Bronze badge

    https://youtu.be/UduILWi2p6s

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019