back to article First-day-on-the-job dev: I accidentally nuked production database, was instantly fired

"How screwed am I?" a new starter asked Reddit after claiming they'd been marched out of their job by their employer's CTO after "destroying" the production DB – and had been told "legal" would soon get stuck in. The luckless junior software developer told the computer science career questions forum: I was basically given a …

Gold badge
Unhappy

" Step away from IT for a minute - Induction 101 "

Dumbest f**king orientation I ever had was along the lines of (on day one)

"Ask me anything"

At this point I know s**t about s**t.

How would I know what I need to know since I have been told nothing?

4
0
Silver badge

Re: " Step away from IT for a minute - Induction 101 "

"Where do you see yourself in five years?"

2
0

The database must have been written in Basic using file records. Or 100 times worse yet, dataflex.

0
0

Holy crap...

The yutz that wrote the documentation should be the one escorted out the door. Geez... You should _never_ put production passwords in new hire documentation....

3
0

The problem is staring you in the face

El Reg posted a bullet list form to say who was responsible.

Clicking on one entry on the form cleared the previous entry.

THAT IS THE PROBLEM

Think about it.

2
0

Re: The problem is staring you in the face

>El Reg posted a bullet list form to say who was responsible.

>Clicking on one entry on the form cleared the previous entry.

>HAT IS THE PROBLEM

>Think about it.

Long story short, it should have been a multiple choice poll... ;-)

3
0
FAIL

This is 300% on the CTO!

1. The documentation for the new dev was FUBAR. That's on the CTO, he has to sign off on that.

2. The new dev should not have had access to the production database at this point to begin with. Again, that's falls within the CTO's responsibility

3. Not having working backups means that they do not have a valid DR procedure in place. And that is absolutely on the CTO, he has to sign off on that as well...

6
0
Facepalm

Documentation issue.

Definitely a docs issue, clearly written by a really 'clever' person who knew what they meant and assumed everybody else reading (using) the doc was as clever as them.

Docs like this need to be properly idiot proof, you need to ensure that whoever is using them cannot get anything wrong.

Oh and working, verified backups is a good thing too!

1
0
Gold badge
Unhappy

"Definitely a docs issue, clearly written by a really 'clever' person "

I can think of at least 3 cases where I've been dropped in the s**t by some "genius."

One case was down to a MENSA member, so he really was a genius. That I could fix quite easily, although that will fail in about 60 years. When I did it I assumed the code would be junked long before it was an issue. Now I'm not so sure.

The others were completely out of my hands to fix. Why would you not use a case statement for multi way decisions based on a state field?

0
0
Silver badge

Re: Documentation issue.

by a really 'clever' person who knew what they meant and assumed everybody else reading (using) the doc was as clever as them.

Hope that was sarcasm. But from a communication POV the first rule is that you can't assume the reader knows what is in your head. If it isn't in the explanation you haven't explained it. To some extent this may go with the territory of IT people. The tendency, expressed in El Reg a few years back, to people with Aspergers type thinking to go into IT and be really good at it, probably means that some quite important people have a very poor understanding what is in other people's heads.

2
0

Re: Documentation issue.

Well to be fair, most of the time the documentation people don't know what they are talking about.

They rely on "SME" or "subject-matter-expert".

That's probably the guy who should be fired. Or one of them...

Or the guy who never proofread the docs, or the guy who never asked them to be corrected/updated, etc.

1
0
Anonymous Coward

Re: "Definitely a docs issue, clearly written by a really 'clever' person "

"One case was down to a MENSA member, so he really was a genius. "

Not necessarily.

Mensa is supposed to be open to the top 2% of the population by IQ, which is only about an IQ of 130, depending on the test (two standard deviations above the mean, near enough).

IIRC, 'genius' starts somewhere around 145 to 160 (about 1 standard deviation above the Mensa lower limit), depending on who is doing the definition... one source quoted 140, and another stated that 125 was 'necessary but not sufficient'. I expect that last can be discounted as an idiosyncratic outlier, using a rather different conception of the term.

0
0

This post has been deleted by its author

Gold badge
Unhappy

"Not necessarily."

I stand corrected.

So just three ordinary f**kwits then.

0
0

Gordon Ramsay

I just binge watched a whole lot of Hotel Hell, Kitchen Nightmares (UK and USA)

Seems to me the whole IT industry needs something like it and the muppets in the mentioned IT firm are cooking up some tarts the wrong way.

2
0

The purpose of making backups is not to hide them away in a cabinet, but to have those be the ones users are making changes to. Then at the end of each day the modifications are rolled up into the real database.

No one uses the main database, live.

The one responsible for a bulletproof system is the CTO.

Not doing this right is his fault.

That is why he is paid the most.

2
1
Anonymous Coward

@kirk_augustin@yahoo.com

> The purpose of making backups is not to hide them away in a cabinet, but to have those be the ones users are making changes to. Then at the end of each day the modifications are rolled up into the real database.

All you have done is redefined "backup" to "real database" and "real database" to backup.

The live production database is, by definition the one that's being used live and in production. The backup is by definition the one you update every night.

2
0

It's entirely possible that the database concerned was one of countless that are left installed "open to the skies" as discovered in the MongoDB trouble in January. Database systems often tend to be unsecured by default, on installation, and if no-one gets around to adding it, that's what's going to happen. Presumably that introductory document dates back to when there were only five people working there in the same room?

2
0
Anonymous Coward

Deleted a manager, first action.

I was given admin access to corporate Navision for local user administration, I wanted to create a new user with the same permissions and roles as an existing production manager.

In the version of Navision that was in use at the time, the roles looked like a spreadsheet so I selected them Ctrl-C, then opening the roles in the newly created user, Ctrl-V, nothing obvious happened except the roles closed and the orginal user was deleted, I had to ask corporate to recover him from the backup and I was not asked to administer Navision any more.

Was actauly quite pleased as if that sort of cock up is possible, and not controlled in the software only luck will save the user from another one.

Just because software is f^cking expensive doesn't make it well designed.

4
0
Silver badge

Why was he even allowed access to the production server on his first day without supervision?

2
0
Anonymous Coward

To be fair, it depends on who you are.

At one job, I did a days long upgrade to the core application for the company starting on my fourth day, on an OS I had not previously worked on (though probably my 9th OS, so that was less traumatic than it could have been)... paying close attention to the manufacturer docs was critical.

1
0

back in the day when Novell was used i was on work experience and given the role to look after the network, while logged in as the admin account i was cleaning up user accounts and thought it would be a good idea to delete the admin account and create a new one, for some reason back then it allowed me to delete the master admin account i was logged in under and from that point had no privileges to create user accounts, resulting in the network no longer having an admin account to maintain the network. I wasn't popular from a while.

1
0

If it was bindary it was always good to have a copy of burglar.nlm.

0
0
JLV
Silver badge
Facepalm

dev or admin?

If admin, then yes, still the CTOs fault for the docs and the lack of working backups. Most of you seem to be answering from the viewpoint that he was an admin, just questioning the context and procedures, but assuming he would have business on that db later on.

If dev... WTF is a dev supposed to be doing with live root passwords to the prod db? Most places don't, and shouldn't, give even system specialists devs (like me) access to the live db, figuring that any interaction should take place through admin-run scripts and/or the product SMEs. I find that sometimes that's a bit overkill and it's nice to have access to support troubleshooting. But I specifically ask for read-only access - no way to screw things up (well, except for mis-written 5M row Cartesian Joins). And none of this justifies root.

That's a system management fail, task segregation fail, people fail and a security fail by writing down those credentials in a disseminated document, all in 1.

3
0
Silver badge

Oninit -iy on a live Informix database also.does fun stuff*

*try it at your own risk.

0
0

This is soooo stupid... Firing this junior won't solve anything. He'd never do it again and this was clearly due to the stupidity of the people of this company. Not protecting prod data, not having a solid backup in place, requiring prod access to create a dev/test database. All of this is so amateuristic. I can hardly feel sorry for the company. It's their own stupid fault, if...

If it's true. Maybe this guy screwed up another way and just made it all up to save his reputation.

1
0

To Be Fair: at least in Canada, you can fire anybody you hire within their first 3 months with no notice or explanation and no legal problem. Similarly, the new hire has the right to quit on the spot, within the same time period, without giving any notice or reason, either.

(Because giving notice goes Both Ways).

Firing the ignorant clumsy new guy who wrecks the whole operation seems wrong, but is it?

No, it is not.

0
0

Yup.

I was once fired from a programming job after three days for using a subroutine call. No one had told me that local practices banned them because of the overhead.

1
0
Anonymous Coward

@Grunchy

> Firing the ignorant clumsy new guy who wrecks the whole operation seems wrong, but is it?

No, it is not.

You seem to be confused between 'wrong' and 'illegal'.

This was clearly wrong.

1
0
Flame

Caught Out At Least Once By Dodgy Doc's

Installing & configuring new servers in branches of a Canadian bank.

They came pre-configured up to a point & then it was just a case of following the documentation until one crucial point where the required configuration details have just been input by yourself........

...... the document tails to a point where the bottom half of the page is blank, the next is also blank, as is the first half of the next one..........

Which finally puts up a image of a Stop sign & on the next full page says "Do not press enter at this point, if you have you will have to reimage the server from scratch using your provided reimage media"

This is not what you want to realise you have done at 12.30am.

3
0
Bronze badge

"canned" seems a bit excessive

but if someone expected their "new" pro to be able to figure out what's actually deployed instead of following a script verbatim, then they company didn't get the caliber of employee they expected. Not gonna weigh in on realistic or unrealistic expectations in this regard.

But legally actionable? I don't see any what that's gonna fly. Especially if there IS a documented script and those values were followed.

Unless you got the guy on Social Media bragging about how he's gonna "destroy" this company because he's got some agenda, then talking about "Legal" is someone who's clueless and realizes HE's got some responsibility too.

Restore from backup and move on. No backup? no policies for such? Mr CTO calling for "Legal" is doing so because HE screwed the pooch.

2
1

I Love You

This reminds me of the Unix version of the "I Love You" virus:

----------------------------------------------------------------

If you receive this email, delete a bunch of GIFs, MP3s, and binaries from

your home directory, then send a copy of this email to everyone you know.

----------------------------------------------------------------

So who to fire?

1. the creator of the virus

2. the supervisor of the creator of the virus

Proceed with legal action against the pair.

What else?

3. put CTO on administrative leave (if he wasn't already caught by #1 or #2)

4. investigate CTO

5. audit all documentation, beginning with all new employee orientation

6. instigate a new orientation regime where the new employee is assigned a mentor

7. apologize to the poor victim of this prank whose career is now upended, perhaps offering them a try at the CTO position

This is obvious stuff. If any business wanted to do it any other way, I wouldn't want to work there. So:

8. Turn down the CTO position at such a lax place of work. They probably have passwords on stickies, if they have them at all.

2
1

Everybody should be fired

The new guy, because he's so ignorant/dumb he doesn't know what he's doing (for the job he's been hired for).

The CTO for same reason.

The docs team, same reason.

The rest of the company? Yes, they are indeed all fired.

Trump field day!!

1
1
Anonymous Coward

a) The developer should not have had access to production at all. Separation of Duties. Fundamental tenant of security. Enforced by network and host separation.

b) WTF was confidential material, DBA account details, doing in the developers document in the first place?

c) No backups. W.T.F.

The CTO should have been marched out of the door.

The fired developer can undoubtedly claim unfair dismissal.

0
0

I have trouble believing this really happened....then again people on this planet really are that stupid! Presuming it is true certainly not the new guy's fault....surely, I would have though, he's got legal claim for unfair dismissal.....

Even if he did it deliberately and maliciously....it shouldn't have even been possible as that production DB should have been password protected, at the very least.

0
0
Happy

Now on Slashdot

https://developers.slashdot.org/story/17/06/10/0450234/developer-accidentally-deletes-production-database-on-their-first-day-on-the-job

with reference to thereg poll results

0
0
Anonymous Coward

As the US has "fire at will" laws, who gives a shit. The CTO can do what they want. Don't like your accent. Fired. Secretly a racist and you smell of curry. Fired.

0
0

Should be more surprising than it is.

This person did nothing wrong at all. The story should be unbelievable, but unfortunately this kind of halfarsedness is not surprising. Halfarsedness is rampant in the industry. An investigation into security, configuration, documentation, and other things would undoubtably reveal plenty of people and processes at this company that are to blame, and no shortage of people who have privately or loudly complained about it and expected something like this to happen eventually. This person is a victim of the company's collective stupidity, and the CTO and the company deserve no sympathy. Anyone there who tried to highlight such problems, and this poor person do deserve sympathy.

0
0
Anonymous Coward

Principle of Least Privilege and good RBAC

CTO 100% at fault, documentation just gave more fuel for the fire.

None of this is the new guys fault at all.

They had given him high level access as a junior??

Principle of least privilege within a good role based access control (RBAC) model could of helped them. However their backups should of been tested...

0
0
Mushroom

Documentation and Backup fail - buck stops with the CTO

But its easier and cheaper to assigned blame rather than find and fix the cause after the event.

0
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018