Sysadmin's three-line 'annoyance-buster' busts painstakingly crafted, crucial policy Monday, bloody Monday. But fear not – Who, Me? has a suitably stressful story to remind you things can always be worse. This time El Reg's weekly column of tech catastrophes comes from "Todd" who worked as an operations sysadmin for a medium-sized regional ISP. His work was consistently plagued by an irritating alert that one …
The engine that is so overengineered that even proper introductions start with "a security context is made up of these three things, of which we will never use the second and third". (The improper introduction is of course "setenforce permissive".)
I've never ever seen a tool that was so firmly "unless you are big enough to have two people doing these things full-time, don't bother".
"It (SELinux) was originally engineered by the American government"
You mean the NSA...so they can watch everyone's mobile phones... Andoid even lists it... Apple doesn't because they don't want you to know the truth...
I think that covers all of the SELinux mobile phone conspiracy.
People who will believe that SELinux is an NSA conspiracy are going to believe that no matter what I say, no matter what you say, no matter what anyone says, so addressing that part of it seemed to be pointless.
The conspiracy theories will need to come up with a new conspiracy about iOS. Maybe they can find a link between Carnegie Mellon and the NSA and decide that secret NSA code in Mach is how they have p0wned iOS...
This post has been deleted by its author
Depends on how prone to configuration errors you want to be. The error-free way is to load the default first always, then check for a system specific configuration and load that over the top, then check for a user specific configuration and load that.
That way, anything that wasn't specifically modified by the system or user configurations will use the default and the user only has to configure those things he cares about.
Alternatively, you load the user configuration if it exists, the system configuration is the user configuration doesn't exist and a system one does and the default only if neither of the previous exists.
It's probably going to read /etc/myprog/config first, then ~/.config/myprog/config , and ignore the living daylights out of /usr/share/myprog/default/config .
The canonical order is supposed to be to read configuration from /etc/ first, and then from somewhere under the owner's /home/ folder; that way, users' own options override system-wide ones. (But there are always a few exceptions .....) /usr/share/ is not for loading configuration files from; any "configuration file" you find under there should be just an example.
But if in doubt, read the Source Code.
The real mistake was not to the system to make sure that an existing security policy does not have the same name. It does open the question however - were the security policies actually documented?
Saying that this is the sort of mistake that all IT admins make sooner or later; once the dust has settled, the scream silenced (possibly using the sort of techniques taught at the BOFH school of IT admin) and the system been restored to what it should have been, you tend to be a whole lot more cautious/paranoid when pushing out any subsequent changes.
"It does open the question however - were the security policies actually documented?"
Of course they were documented. See that dusty cabinet over there with loads of carefully printed documentation that is almost untouched aside from the thick layer of dust? It's in there....probably
You don't even have to introduce a negative. Make the documentation very verbose, include something in it approximately 68% of the way through, and watch people lose their mind when they try to find it. Otherwise known as the irritating API reference technique, where you know there should be a function that does what you want, and it's in this category, but this category includes thirty APIs, each of which implement 60 functions. And the search box only searches API names, but not functions. They're watching me to see how long I search, aren't they?
From Foobar to FUBAR?
Best I managed was adding a security policy to a remote server.. Forgetting to add the office's IP address to the whitelist. Server suddenly stopped talking to our office Cue just about every department in the office howling whilst I made an embarrassing call to the hosts to talk them through what they needed to change to get us back onto the server.
(other customers could see it just fine but because of remote access ports attempts triggering firewall blocks, we got the ban hammer straight away)
Don't worry AC, we've all been there. Firewall rules on colo servers, VPN to customer sites, etc etc.
Sometimes you can cobble together something to get you back in, sometimes you have to phone up and eat humble pie. Sometimes you can just put it off until you're next on-site and fix it discretely.
We screwed up once when converting a telco into an ISP. We had some real good people to do this, but one in particular was a tad too impressed with himself, so the Gods of Technology decided to take him down a peg.
He was busy configuring the big bad box that to route it all, and all of a sudden he came and quietly asked if I still had dialup access to my old account - he put the filters the wrong way round and locked himself out. That was the moment he finally fitted into the team - he learned he was just as fallible as everyone else.
There are two things you do during your work that create experience: screwing up and digging yourself out of that hole (with or without help), and teaching others so you get people asking things you never thought about. You don't become really good at something until you have done both IMHO.
To quote Ramirez from Thief:
Boys -
While I appreciate your enthusiasm, I must point
out that breaking legs is, really, rather inefficient,
if we wish our clients to be able to pursue their
rather active lives in order to immediately repay
their debts. I encourage your initiative in this
area - perhaps you might take a lesson from the
gardener and his hedge clippers, who trims off one
branch while leaving the rest to grow... and most
of our clients do start with fingers to spare.
There seems to be some confusion.
I work in IT and try and keep my assistance or Mr Darwin to that which is strictly necessary to keep important systems running. And avoid unnecessary use of my time. Oh and occasionally when I'm bored on a Friday afternoon...
I'd never dream of expecting users to pay to evolve...
Our division had been spun off and therefore had a new name. A new longer name. Much longer than three initials.... So I found where that setting was in the configuration and I changed it from the old 3 letter name to the new 18 character name. Soon thereafter, calls started coming in from the shop that they couldn't log into the product tracking system (which is as you woulld imagine, rather important). I went downstairs to the closest one (there were a lot) and logged into my account from their terminal and it worked just fine (including displaying the long company name), but when they logged into their production account, it would disconnect them. I quickly came up with a workaround that would get them connected, but required them to log in twice...they didn't much like that, but it did get them tracking product again. Then I opened a case with support. What it came down to was the 18 character name I had inserted was 10 characters longer than the field into which it was written....which, when downloaded to the FEPs, caused part of the transaction processing code to be overwrritten, and crash on login. After shortening it to 6 characters, the transaction processing logins started working again.
The moral of the story is if there's unclear or no documentation, don't make any changes to a production system until you ask what happens if I change ______, especially if its never been changed before...
"The moral of the story is if there's unclear or no documentation, don't make any changes to a production system until you ask what happens if I change ______, especially if its never been changed before..."
Er, no. The moral of this story is that user input should *ALWAYS*, no exceptions, be checked, starting with how long it is.
"Er, no. The moral of this story is that user input should *ALWAYS*, no exceptions, be checked, starting with how long it is."
Wrong, you should start with checking whether it is there at all, length is secondary to that (unless you equate it not being there with a zero length, which I don't, zero length can be acceptable).
a) Testing.
b) Change Management.
c) Proper naming conventions (which should include the date and/or author). Name a policy with 20190204 on the end when you make it today and it's impossible to get confused with one people wrote years ago called the same but with THAT date. Plus, you instantly know how old that policy is, i.e. how long it's been useless and/or working, and can modify your interference behaviour accordingly.
It reads like a catalogue of errors from the start.
Only lucky that the "overwriting" that was being prevented originally didn't overwrite something much more critical when deployed to all the relevant servers and leave you with, say, a blank DNS database for example.
No, don't put dates in names. That just makes the names harder to understand. If the files are called "system_restart" and "system_test_dr_capability", they are less likely to be run by accident than if they're called "20170204_ada_lovelace_at_company_dot_com_fixes20160419_system_test_dr_capability".
The latter approach not only makes long names that are hard to remember, but it can also result in multiple versions of the file that may or may not do the same thing.
Put instructions at the top of any file that can handle them, and of course test before you push things into production. Also, if you can, don't use a system that cheerfully replaces one config with another config without asking; if they had gotten a single confirmation box or terminal warning*, this would have been detected before it caused a problem.
*As it turns out, neither mv nor cp complains about copying a file over another one. I thought they did. Time to become more nervous.
*As it turns out, neither mv nor cp complains about copying a file over another one. I thought they did. Time to become more nervous.
Yes they do. Both of them. Have you tried with -i ?
Disclaimer: Presence and function of -i option may depend on your implementation.
Perhaps I should clarify my statement. Neither mv nor cp complain about copying a file over another one *by default*, which is how everyone runs them. Since these tools don't do a lot of, to me, more obvious things without my having to tell them with switches, I would think that -i should be on by default. So much did I think this that I assumed that it was.
me:
The SQL queries that produce KPI reports for the CEO, CFO and other top management use the content of the kpiName column to find data. This means we cannot change names of these KPI's. If there is a typo or we wish to rename them for a more accurate description to those entering data we can't. Perhaps a kpiID column with a lookup table for the kpiName would be the way to go. Then we can change kpi names at any time. A new column called kpiID is all that is needed. I will change the Insert query to send the kpiID as well as the origninal kpiName.
Business Analyst and dbAdmin:
That make sense good idea
A day later
IT Manager:
WTF is going on? why do these reports not work?
CEO is going to tear me a new one.
I write the code that that presents a web interface for users to enter KPI's
It is obviously my fault.
Me:
I will find out why, will update you soon
I look at the column names. They have been changed... kpiName to kpiID and a new empty kpiName column added.
An email to the Analyst requesting they revert the changes to column names and bit of SQL jiggery pokery later and it all works again
me:
It is sorted now, some column names were changed.
IT manager:
Thanks for the quick fix, make sure is doesn't happen again.
I didn't have access rights to alter the table but I kept my mouth shut.
IT manager still thinks it my fault. Perhaps it was, I over estimated the ability of the analyst. Unfortunately I had no rights to change the queries generating the reports.
Perhaps I should have suggested that the queries that produced the reports needed updating too.
Was fskn obvious to me. Not so to the Analyst.
adnim,
Your problem was the old familar one ........ You thought something was obvious so did not mention it.
Never make assumptions about other peoples knowledge or understanding.
Document *everything* that relates to your problem/process .... even the blindingly obvious.
Better to 'teach your grandmother to suck eggs' than realise 'Oops !!!' something was missed because you thought someone else was doing it !!! :)
Yes AC, I thought is was obvious because it fskn is!
I am self taught, I have no qualifications except a C&G in 'C' applications programming from the early 90's. I am not even sure what I am doing most of the time. And when my code works I am sometimes surprised.
I expect someone with a university degree who is a database admin to have a clue, and teaching them to 'suck eggs' so to speak seems patronising if not insulting.
You are right though... I will just add "sorry if this is obvious but make sure that ...." to any future request for change. :-)
That's just a degree of not just incompetence but brain-beggaring stupidity.
Process:
1. run a single line of DDL
2. do not populate the new columns -- fail
3. do not resynch the downstream code -- fail
I mean, to not even populate the Names...
.
I'd have sacked the guy on the spot. How the HELL did he blag his way into the job in the first place??
"Your problem was the old familar one ........ You thought something was obvious so did not mention it."
Tee hee. I've got one of those.
I once worked for a small bleeding edge firm with a contract to a large cruise line to write a ticketing system. It was not going well. I had just been hired and was rushed to the angry customer with a bunch of their younger, wiser programmer analysts to see what could be done to make them happy.
I was an Old Mainframe Guy. Customer rep was an Old Mainframe Guy. We locked eyes.
"Forgive me if I'm covering old ground, but I'm new here and need to fully understand the problem we are trying to solve" I said in my best humble voice. There were rolled eyes from the younger members of my own team but no reaction from the customer rep.
"Could you tell me, does the SS Saucy Sal always have the same number of staterooms?" I asked. My team groaned and rolled their eyes some more. The customer rep's mouth twitched and, after a suitable pause, he said "No".
You could have heard a pin drop. We were still eye-locked.
"I see. Could you explain how and under what conditions you change the number of staterooms on the SS Saucy Sal? Again I'm sorry if this has already been covered." The customer rep proceeded in a level tone to explain how and why this transformation would come to pass. We were still eye-locked (though blinking was allowed) but I could detect a palpable thaw in the customer rep's attitude re: contracted software "specialists" from Somewhere Not Here.
My own mob were trying not to do the Bonehead Gape Face.
"That's very interesting. Thank you. Now, when you say you ticket up until the last minute, could you tell me exactly what you mean and how you go about delivering that service?" A very revelatory explanation involving servers loaded on beer trolleys, long extension cords and fistfuls of cash and paperwork was delivered, demolishing the model my own team had laboriously constructed from assumptions and guess.
By the end of it the customer rep was smiling and we were on a much better footing with him and about 3/4 of the staff. (I had to impress someone else in their DBA department early one morning by demonstrating mad skillz on the fly before the other 1/4 came grudgingly around).
It took a couple more of these before my own team forgave me for being new and old at the same time, and mired in the mainframe world instead of flying high on a balloon made of PCs, Visual Basic and the Light of Jesus in Their Eyes.
It CAN mean the system that connects "stuff" to the coax cable that goes to all the houses. Maybe one for a small town long ago. Once it might have been merely TV receivers with aerials and C-band Dish+LNB+analogue FM receivers and then loads of modulators and amps.
Now it might be a fibre fed CMTS in a street cabinet for Internet with Digital Video, Switched video, Some analogue video and even some Band II VHF radio. Unlike the Cat3 telephone pairs, the coax can have nearly 1000MHz bandwidth rather than less than 30MHz DSL (designed for 3.5KHz!) and run at higher QAM. Distance can even be a few kilometres with no penalty. Twisted pair phone line speed / bandwidth falls of a cliff beyond 200m. VDLS2 maybe only 12Mbps at 1km.
You could make an undocumented change in Prod which kills people:
https://www.brisbanetimes.com.au/world/north-america/behind-the-lion-air-crash-20190204-p50vhf.html
Just this evening I was reconfiguring a switch port and knocked everybody offline for about 15 minutes when I made a small mistake changing vlans. Fortunately not many people affected. The biggest nuisance was finding a way to connect to the switch to fix the mistake, given the network was broken!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
