back to article Techie was bigged up by boss… only to cause mass Microsoft Exchange outage

Welcome once more to Who, Me?, our weekly column in which readers confess to their worst IT cock-ups. This time, we meet “Freddie”, who must have a good supply of luck on his side, as he managed to not only get away with his error, but win praise for fixing it. At the time, Freddie was a busy man. He worked for a managed …

Silver badge

we’re on the hunt for spooky tales for a Halloween special of On Call

To be titled 'Ghost in the Machine'?

46
0
Anonymous Coward

Or "Ghost in the Shell Server."

35
1

It'll be a crossover with the movies which starred Matt Damon as an undercover spy: "Ghost in the Bourne Shell". The sequel will be "Ghost in the Bourne Again Shell".

55
0
Silver badge
Coat

JSON Bourne.

{

"firstName": "David",

"lastName": "Webb",

"isAlive": true,

"age": 48,

"address": {

"streetAddress": "415 East 71st Street",

"city": "Nixa",

"state": "Missouri ",

"postalCode": "65714"

},

"Preferred shell": [

{

"type": "linux",

"name": "Bourne"

},

{

"type": "when out and about",

"name": "bash"

}

],

"children": "2",

"spouse": "Dao"

}

42
0

Re: JSON Bourne.

Great, now the damn assassins are going to be after me again, thanks for that mate!

26
0
Silver badge
Pint

Re: JSON Bourne.

Well done, Symon, well done. E-beer for you.

8
0
Silver badge
Pint

we’re on the hunt for spooky tales for a Halloween special of On Call

To be titled 'Ghost in the Machine'?

And here I was thinking they were looking for more horror stories about Windows 10. I'll go to the naughty corner after I offer this virtual beer.

7
1
Anonymous Coward

Re: JSON Bourne.

Your cover is safe - in Arkansas they call each other "Bubba", not mate, so either you are an import (not easy to do in this day and age), or you are living in a different country, possibly in the southern hemisphere .

2
0
Silver badge

Which is why you don't work direct at the server but over RDP unless something really critical has happened, because then "shutdown" is something you can only really do with a command-line (or if some installed program offers to do it for you, but then you test that on a non-server first, and generate yourself a mental install script, right?)

1
28
Anonymous Coward

Ehm, no. you can do a shutdown from RDP too - even if it will warn you - if you have the permissions.

Anyway, there was a time when MS server applications had great remote administration applications, which let you working from your workstation without any need to access the server directly, or via RDP or whatever.

Then came the Linux crowd where there were no such applications, and everything must be done directly connected to a server via SSH to manipulate some text files (while maybe wiping your machine with rm -rf *, or shutting it down if you happen to work in the wrong terminal).

Unluckily MS saw how cheaper it was (and easier to offshore), compared to developing full remote administration applications, and copied it with PowerShell. Now, even management applications are wrappers over PowerShell scripts, so everything is far slower, bloated, and error handling a joke.

26
26

Re: while maybe wiping your machine with rm -rf *

Well, who wants remote clicky-mousy for server administration instead of a text file that can be copied, edited, copied back just in case, etc. might actually be "wiping your machine with rm -rf *".

From the point-of-view of the universe, he or she might also do less harm in a Windows environment.

12
0
K
Silver badge
Trollface

RE: Then came the Linux crowd where there were no such applications

CLI works for linux because every application is decoupled... With Windows its a peace of dogs meat, as all they've done is take the same monolithic products and stripped away the GUI - For analogies,

Linux is like a sleek F1 car, each part is self-contained and held together with screws. It can be hard to get the components working together at first, but they're easily swappable.

Windows is like a big American RV, humongous and put together with rivets... it works and is very comfortable, but don't expect to swap anything without rebuilding the whole thing.

Troll, cause its Monday morning and I'm stuck on a train going out of my mind --->

44
15
Anonymous Coward

Re: RE: Then came the Linux crowd where there were no such applications

"Linux is like a sleek F1 car, each part is self-contained and held together with screws. "

Nowadays, that's unfortunately less true than ever. To illustrate this, I suggest you try the Gentoo installation process. When you start with a stage 3 tarball and use OpenRC rather than systemd, the number of dependencies required to get a *lightweight* functioning desktop with a suite of useful applications which use your hardware properly is scary.

12
4

Re: RE: gentoo

what are you on about?

xorg-server mate slim firefox thunderbird conky dconf-editor libreoffice galculator corefonts dejavu roboto vlc audacity spotify ghex gimp conky

there. you have a gentoo desktop.

9
1
Silver badge

Re: RE: Then came the Linux crowd where there were no such applications

@K

You do know that modern PowerShell is actually available and actually worlks... and that headless MS servers do indeed exist...

5
0
Silver badge

Re: RE: Then came the Linux crowd where there were no such applications

"Linux is like a sleek F1 car"

You have to replace half the components after every few hours of use, and just chuck the old ones away?

A rally car might have been a better choice (they're designed to have components swapped in and out), but even then it's still not a good analogy.

17
5
Linux

Re: RE: Then came the Linux crowd where there were no such applications

"...try the Gentoo installation process. ... the number of dependencies required to get a *lightweight* functioning desktop with a suite of useful applications which use your hardware properly is scary."

At least Portage deals with all those dependencies for you automagically, even if you do need to type a few package.use incantations to keep it all clean & tidy and free of unnecessary cruft and avoid filling the OCD-pleasing 4GB partition. (I'm afraid I had to cave in and resort to allocating 8GB on my desktop systems some years ago.)

Getting a fully-tricked-out Windows system requires downloading Mammon knows what from Baal knows where (they'll give you hints if you bribe them with shares in your soul), then trying to install it all in the right order without accidentally letting any of those packages install the bundled spyware, take over a bunch of file associations that you would rather stayed with some other package or replace your preferred media player with SuperVideoMediaDoober Plus Max (Cloud Edition). Then you find that one of the things you need only works properly if you install .Net Framework 1.1 _after_ you've installed .Net 4 but before you install .Net 2, unless you install .Net 3.5 first, in which case you have to install the Visual C++ 10,11,12,13,14,15,16,17 and 18 runtimes in an order determined by consulting a medium, after slaughtering a goat at New Moon.

(Oh wait, I'm getting confused. The bit with the goat is from the NetBSD install procedure.)

38
1
TRT
Silver badge

Re: "... headless MS servers do indeed exist..."

ARRRGGGHHHHH! Are the halloween stories starting already?

Or there's the JK Rowling one, "Nearly headless 'nix server".

31
0
Mushroom

Simply seeing shutdown on the same menu as log off sends chills down my spine every time I see it.

Sure I can understand how it makes perfect sense for a desktop system, but for a production server where I don't have access to restart the box. Not good.

28
0

Re: RE: Then came the Linux crowd where there were no such applications

Just dreamin':

[Quote]

Linux is like a sleek F1 car, each part is self-contained and held together with screws. It can be hard to get the components working together at first, but they're easily swappable.

[/Quote]

Then there was systemd: https://youtu.be/hYOWIdPHXts

17
0
SVV
Silver badge

Re: "... headless MS servers do indeed exist..."

Headless MS servers are rather like headless chickens. They can run for a little while, but soon conk out.

They can also both be described as "dead fowl".

5
6
TRT
Silver badge

Re: "... headless MS servers do indeed exist..."

I suppose they might be a turkey solution.

6
0
Anonymous Coward

shutdown/reboot

Agreed. Where's the Windows version of Molly-guard?

6
0
Windows

Re: RE: Then came the Linux crowd where there were no such applications

@khaptain

You do know that modern PowerShell is actually available and actually worlks...

Hahahahahaha

and that headless MS servers do indeed exist...

No dooont... stooooppp... my sides are hurting...

11
6
Bronze badge

especially when your in the office in slough, and the server is in telecity, and you dont have inteligent hands.....

2
0
Anonymous Coward

..."where I don't have access to restart the box"

That is quite rare nowadays though. Usually they'll be VMs which you just start up through the relevant console or for hardware boxes/vm hosts then you normally have a management interface that works with the box switched off (but obviously still powered). Failing that the intelligent power supply can be cycled and as long as you have it on power on on power restore that should work. I've never let a truly remote box get zombied by an errant shutdown.

2
1

Re: RE: Then came the Linux crowd where there were no such applications

*No dooont... stooooppp... my sides are hurting...*

Psst! Don't tell anyone, but there's a Linux port of PowerShell! There's even a Debian repository:

https://docs.microsoft.com/en-us/powershell/scripting/setup/installing-powershell-core-on-linux?view=powershell-6

1
1

Re: RE: Then came the Linux crowd where there were no such applications

A sleek F1 car... where everyone brings one piece of the car along when they come to the racetrack. They all go out to the pit and put the car together piece by piece, arguing non-stop about what kind of car they are supposed to be building. Violent fights erupt between KDE / Gnome and VI / emacs, Linux/Debian user groups ... In the end the bodywork is not installed because nobody can agree on the color scheme to be used. There's not steering wheel ,shifter or pedals. Instead there's a panels with some cryptically labelled buttons. And a MAN page explaining in which order to press the buttons to steer the car, accelerate and brake...

The car takes off without paint or bodywork but is declared a huge success because.. well look at the 'open'-ness of it... you can see every part ... and it's very lightweight too... so it's fast. But it still is only half baked. If it breaks the people who built it simply tell you : we gave you the list of parts (the sourcecode) you go fix it. We've moved on to the next car. If you dare raise the issues of missing bodywork and paint : prepare to be scoffed at. As for the steering steering wheel and pedals : this is not a kids bike. Only little children need steering wheels and pedals.

coat..

21
6
Bronze badge
Pint

Re: RE: gentoo

>> "Linux is like a sleek F1 car, each part is self-contained and held together with screws. "

>> Nowadays, that's unfortunately less true than ever. To illustrate this, I suggest you try the Gentoo installation process. When you start with a stage 3 tarball and use OpenRC rather than systemd, the number of dependencies required to get a *lightweight* functioning desktop with a suite of useful applications which use your hardware properly is scary.

> what are you on about?

> xorg-server mate slim firefox thunderbird conky dconf-editor libreoffice galculator corefonts dejavu roboto vlc audacity spotify ghex gimp conky

> there. you have a gentoo desktop.

Um. You put conky in twice. I know it's good, but it is not that good. Good effort, though, so please accept the virtual pint --->

And for most normals, that would be part of a truly scary command line, which would be accompanied by pity when you proudly point out you composed it from memory.

4
0
Anonymous Coward

Re: RE: Then came the Linux crowd where there were no such applications

Linux is like a sleek F1 car, each part is self-contained and held together with screws. It can be hard to get the components working together at first, but they're easily swappable.

Ha Ha Ha Ha Ha Ha

Linux is like a series 2 land-rover defender with bits and bobs screwed on from Later models, discoveries and Range rovers.

Of course in reality some bits aren't screwed on and need welding or you may have to drill some holes in the chassis rails or even remove the engine.

Some bits wont work together at all and may cause your defender to stop in the middle of a stream.

3
1

Linux is like a sleek F1 car ... Windows is like a big American RV

Good analogy! Especially considering the fact that F1 cars usually are designed with less than ~100 hours of life span.

0
2

shutdown on the same menu as log off

Did you know that you're supposed to click on your avatar (instead of the power icon) on the start menu when you're trying to log off or lock the screen?

0
3
Silver badge
Gimp

Re: RE: Then came the Linux crowd where there were no such applications

"Getting a fully-tricked-out Windows system requires downloading Mammon knows what from Baal knows where"

Ninite. One installer, no spyware, done.

To stick with the point about upgrading a CLI based OS to a full GUI, in Windows you just do: install-windowsfeature server-gui-mgmt-infra,server-gui-shell.

1
0

"“I had just finished with five minutes to spare, but instead of hitting 'sign-out', I accidentally hit 'shut-down' on the primary server,” Freddie said."

Placing the logout button next to the button for "shutdown" is one of the worst pieces of design in the entirety of Windows, and that's saying something. Freddie must be one of thousands of people who have made and continue to make this mistake.

With Windows 10, the buttons have been parted (but only by half an inch), and are no longer in the same submenu. And they are better marked, but it is still one of the most obtuse and needless mistakes ever to come out of Microsoft. Putting a massive risk at customers' feet for no reason.

7
0
Anonymous Coward

Been there got the t-shirt

When using robocopy to move an empty (as in no users) EDB file to a new drive, make sure you specify the Database Name in the script.

We were running out of space so added some more storage, then moved some of the databases to that new drive.

We had 9 days worth of space left so it was a bit of a rush job once the storage was ready.

Anyway, without the variable set, I robocopied the half empty drive over the live one...

Script cancelled but damage done.

Thankfully Exchange sorted it all out, failing over to another copy with about 10-15 mins downtime. It took it's time to think about it, but it sorted itself out! Phew.

You learn by your mistakes!!!

17
0

Re: Been there got the t-shirt

When you are lucky you learn by others' mistakes!

4
0

Re: Been there got the t-shirt

Our previous contractor was asked to leave his employment as a result of an errant Robocopy statement.

He left the MIR switch in his script when copying over some errant files that hadn't copied due to permission issues, 2 days after the file server had been replaced. Result? We lost 2 days data. His biggest mistake though was doing all this before checking to make sure we could actually access the backups of the new file server. Small mercy being only 2 days data lost, and it wasn't the database server.

0
0
Windows

Note to Microsoft

the default on an option on server OS should be logoff not shutdown.

36
0
Silver badge

Re: Note to Microsoft

At least on later versions of Windows Server (from 2008 onwards I believe), you're required to provide a reason for the shutdown/restart that goes into the event log. This is usually about the same time that you realise you hit the wrong option when trying to select "Log off", and does at least give you chance to cancel the impending doom you've initiated.

29
0

Re: Note to Microsoft

I think that feature was there in Server 2003 as well.

And Windows desktop had an "are you sure?" screen.

5
0
Silver badge

Re: Note to Microsoft

Having heard all of the horror stories of people shutting down servers since the 2k days I have always put in a GPO on my sites to remove the "shut down" and "restart" options from the start menu of my servers. (plus other housekeeping bits like logging off people who leave their sessions connected after a decent period.)

This forces shutting machines down to be done with the "shutdown" command (start->run "shutdown /i") which forces very deliberate action be taken in shutting machines down.

19
0
Silver badge

Re: Note to Microsoft

"the default on an option on server OS should be logoff not shutdown."

I think it is these days

Also you get that "Let me record reason for shutdown" box - thats saved me a couple times. or is that only on reboot?

2
0
Bronze badge

Re: Note to Microsoft

All well and good until it restarts at 0930 because 'updates' Server 2012. I hadn't logged on locally to that server for several weeks and the updates should restarts should have happend outside the backup window / between 7-8 am. Thanks MS.

1
0

Re: Note to Microsoft

Real computers boot at power up and are shut down when they outlived their usefulness and decomissioned. They don't have reboot options and don't need them. Ask anyone running a Cray...

7
0
Silver badge
Facepalm

Re: Note to Microsoft

@Peter2

"I have always put in a GPO on my sites to remove the "shut down" and "restart" options from the start menu of my servers"

I tried that with one particular customer. They were in the habit of rebooting the server whenever there was any problem, so I thought that removing the menu items would stop them. Instead, they resorted to yanking the power cable every time.

7
0
Mk4

Is there anybody out there?

I did the same thing, accidentally shutting down the file and print server for the London HQ of AT&T (this was a very long time ago, so I feel safe enough naming the company) at 14:30ish on a Thursday afternoon. I was not sure I had really hit "shutdown" until I saw all the lights go off on the server. Starting up was a slow process - the disk arrays spun up their drives sequentially and the whole thing was offline for 5-10 minutes.

We had all the big wigs in the building, the hundreds of staff etc. but there was not a single call logged. SMB mounts on clients would only have noticed if they tried to r/w while the server was down and obviously also no-one tried to print anything (or just tried it again 5 minutes later and it worked).

Not a truly impressive story I will admit, but in those minutes before I could check that everything was back online I was convinced I was going to get the sack. :-)

45
0
K
Silver badge

Reminds me of something I heard a long time ago...

Take 10 mins to fix a business crippling outage, then get shouted at for a) causing it and b) taking so long to fix it... But, take 5 hours to fix it, then you're a rockstar, praised for being a saviour!

31
0
Silver badge

Re: Reminds me of something I heard a long time ago...

If it took 5 hours it must have been a difficult job with lots of hard work involved, obviously, er ..... as I always told management.

26
0
Silver badge

Re: Reminds me of something I heard a long time ago...

Never finish anything too quickly- it gives management unrealistic expectations...

2
0
TRT
Silver badge

Re: Reminds me of something I heard a long time ago...

To be honest, Captain, I think you gave me too much time...

So how long will it really take, Laddie? Eh? You told him exactly how much time it would take to fix? How are you going to keep your reputation as a miracle worker doing that?

Montgomery Scott.

3
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018