back to article Techie's test lab lands him in hot water with top tech news site

Monday morning arrives once more for those of you holding the fort while colleagues are on holiday.So why not enjoy this extra special instalment of Who, Me?, El Reg’s weekly confessional column. Because this week, Vulture Towers plays a leading role in the tale that our brave reader "Victor" recounts. "Perhaps 13 years ago, …

Silver badge

I wonder if Victor will own up in the comments?

34
0
Silver badge
Alien

And leave Victor to the err Vultures?

21
0

...

You mean owning up on the front page is not good enough?

11
0
Anonymous Coward

I wonder if Victor will own up in the comments?

Or will it like the end of that movie Sparticus? Or like "Life of Brian"?

5
0

I would like to know what happend subsequently. What's your vector victor?

41
0
Mushroom

"They bought their tickets. They knew what they were getting into. I say, let 'em crash!"

37
0
Joke

What's your vector victor?

Is that... is that a Despicable Me reference?

Where's the minion icon when you need it?

1
22
Silver badge

@Zippy

I'm guessing it's a reference to the film Airplane.

47
0
Silver badge

Re: @Zippy

Roger, Roger...

34
0
Silver badge

Silly! Minions don't speak... English.

...I'm not going to count the word "Banana", used in the Minion epic adventure, "Banana".

7
0
Silver badge

Re: @Zippy - enlightenment ahead of it's time

Rodger Roger

10
0
Silver badge

Re: What's your vector victor?

You need clearance Clarence.

28
0
Coat

Banana isn't English

It's Spanish ....

according to my Spanish word of the day app.

1
0
Silver badge

Re: @Zippy

Okay for clearance, Clarence...

2
0
Silver badge

"Surely you don't mean that!"

"I do, and don't call me Shirley"

24
0

Re: What's your vector victor?

You have clearance, over

It’s “Clarence Oveur”.

6
0

Re: @Zippy

I'm guessing it's a reference to the film Airplane.

Oh, of course it is. Obvious lack of coffee error there!

1
0

@Robert Carnegie

Silly! Minions don't speak... English.

But Vector does. And he was the villain in the first movie (he stole the shrink ray.) And also, his real name was Victor.

I didn't spot it was an Airplane! reference (although I should have), probably because of coffee underload.

1
0
Silver badge

So The Reg didn't have a redundant backup elsewhere? Just two servers in the same place?

One day you really need to write an article about the tech behind the site, because sometimes it just sounds quite worrying.

It's like having a bunch of sports journalists who comment on every match but have never had any involvement with the sports themselves.

22
2
Silver badge

Re: commenting

It's like having a bunch of sports journalists who comment on every match but have never had any involvement with the sports themselves.

Isn't that sort of mandatory for most [insert sport of choice] journo's?

26
1

Geographic redundancy, thirteen-fourteen years ago? I'm vaguely surprised they had servers in an actual colo at that point, and not sitting underneath the junior journalist's desk.

81
1

Re: ...top tech news site

I can't help remembering the classical "And that is a baaad miss" in David Mitchell's voice.

16
0
Silver badge

You mean in 2005?

Er... yes...?

"Geographic redundancy" = "buy one each from two different companies, rather than two from the same".

I'm not talking hot-swap, fast-failover, high-availability, but I'm also talking "website doesn't fall over just because someone pressed a switch".

Go hit Google for "site:theregister.co.uk" and change the date range from to 2005-2006. They were just as big, just as sarky about downtime, doing articles about grid-computing, etc. etc.

We're talking the .xxx TLD era, not the dark ages.

15
3

That works until you find that the two companies you brought from are both buying out the same colo space.... (i.e. the cheapest one)

26
0
Silver badge

re sports journalists

IMO, that is no worse, probably better, than say the BBC's practice of "employing" a multitude, let's call it a magnitude, of ex-pro sports people who endlessly babble on about mundane matters or the blatantly obvious, or their relentless agenda. At exhorbitant rates too.

The less sport they show, the more they chat about it.

15
3
Anonymous Coward

So The Reg didn't have a redundant backup elsewhere? Just two servers in the same place?

Even El Reg started up at some point, and the Net wasn't such a resource intensive place it is now (IMHO mainly because of the large volume of sh*ts trying to abuse your resources for something else).

2
3

Nah.

Back then it was not as trivial as it is today - it was somewhat expensive to do properly with some pricey F5 GTM devices.

Two well setup servers behind a load balancer would give you 99.99% without breaking a sweat so it's understandable why they only had two servers back then.

At this particular hosting company we had staff operating 24x7 who would run over with a crash cart the second anything played up and we had a room full of spares.

So really unless you are doing something requiring a lot of reliability, why bother with geo-redundancy, it's more ballache than it's worth in many cases.

26
0
FIA

Re: Nah.

So really unless you are doing something requiring a lot of reliability, why bother with geo-redundancy, it's more ballache than it's worth in many cases.

This. Sometimes being 'good enough' is really 'good enough'

Also, best practices have to be married with some level of competence too. I remember a previous company I worked at had their primary B2B site running on a solaris box in the server room with an apache front end hosted by a 3rd party. No real redundancy (other than the text box that could be repurposed if needed) however it pretty much just 'worked' for most of the time as the servers were setup correctly and well maintained. (Sure, we were at risk of serious outage if a fire or flood occurred, but still).

Years passed and we got acquired by another company, they came and gave us the fancy presentation on their shiny new data center. The site was moved to this wondrous place, hosted across 6 servers with load balancing and all the 'best practice' boxes well and truly ticked.

Unfortunately it ticked boxes, but perhaps wasn't set up quite right, that site had more downtime in the next 3 months than it did over the previous 5 years. (Mainly due to randomly crashing servers, sticky sessions and load balancers configured to blithely ignore if the servers were actually up and responding or not...) Whilst the site was now more 'disaster proof' our customers were much less happy.

Sometimes working is good enough.

(Ironically, we did have full multi site geographically separate DR for our core systems, it's just this new fangled web thing had appeared in the meantime...)

7
0

Has elReg bought a 3rd server yet?

29
0
Silver badge
Coat

Has el Reg bought a 3rd server yet?

Nevemind that, has they upgraded the hardware, or are they still running on the original servers?

Pocketses got lots of nifty new nanoservers.

13
0
Silver badge
Joke

Has elReg bought a 3rd server yet?

No, but they've upgraded from the original 640K RAM.

23
0

Upgraded the servers

Of course they've upgraded!

Don't you see the shiny new boxen?

Of course the internals are the same, the new internals having been pilferred by the BOFH for home use...

7
1
Silver badge

Top Boss

[The boss] said, ‘I’m not worried that you broke something, that's expected, the most important thing is how you deal with it’,

And that is how a boss should react. Shouting at people for making mistakes doesn't mean fewer mistakes are made. It just means more are covered up.

Admit the mistake and learn from it. I know it sounds like management bull****, but mistakes are fantastic learning experiences. Just don't learn by mistakes too often...

88
0

Re: Top Boss

This. Punishing mistakes doesn't avoid mistakes in the future, it makes people better at covering them up.

34
2
Silver badge

Re: Top Boss

I would never shout at my staff for bringing down a client's server, failing to spot a bad code merge request or similar.I just make them buy me a beer for every failed code review that gets notified to me by SonarQube

Hic

18
0
Anonymous Coward

Re: Top Boss

And that is how a boss should react. Shouting at people for making mistakes doesn't mean fewer mistakes are made. It just means more are covered up.

Not just bosses. One of the reasons I've always had the keys to the server room & the root password is that the admin not only has reasonable trust that I won't screw up, he knows that if I do I will call him, own up, and help to fix whatever I did.

21
0
Gold badge

Re: Top Boss

I heard an interview with an old actor who did some TV in the early 60s. Back when all TV plays were broadcast live, as recording was fearfully expensive.

So the director is doing his final pep talk. Firslty the helpful, "Remember if you fluff you lines just keep going, there's no time to stop." Then the less helpful, "and anyone who dries up will never work for me again!"

Because I'm sure that helped with nerves... What an arse.

19
0
Silver badge
Facepalm

@I ain't Spartacus Re: Top Boss

On a similar vain, I heard the head of a sports body said to an athlete as the athlete went out for their Olympic qualifying trails: "The future funding of our sport depends on you qualifying for the Olympics."

Yep, the athlete crashed & burned within seconds.

A textbook example of how not to give a motivational speech.

7
0
Silver badge

We don't kill our own...

...we just point and laugh.

5
0

To quote Shaggy

It wasn't me!

10
0
Silver badge

The timing is perfect

Literally just had to bring up a server that was refusing to come back up.

Long story short, new servers being built. To facilitate the new servers we power down a QNAP NAS temporarily, plug it back into the correct UPS and power it back up. At this point I'll explain that the job of doing this is actually that of the IT contractor who has come in to build the server.

So the QNAP is powered down, plugged back into the correct UPS and all is happy. Until about 10 minutes later when the file server starts becoming non-responsive. Decision made to reboot the file server, but it won't boot back up (it's a VM). Turns out it's complaining about a missing virtual drive.

We disable the virtual drive, boot up the server and check to see which drive appears to have failed. At which point I notice that the drive that's actually stored on the QNAP is the missing one. And the IT contractor suddenly remembers he forgot to turn it back on. QNAP turned back on, missing drive added back to the VM and all back up and running.

In case anyone thinks it looks like things are now back to normal, we then find out that the new server can't be built because a vendor who's name rhymes with Hell has installed the wrong HD controller card and the nice shiny hard drives can't be configured in RAID. So project delayed by another couple of days.

14
0
Anonymous Coward

Yes, I *know*, shut up anyway!

Fortunately, Victor learned his lesson, so El Reg's servers will never again face the risk of being inadvertently shut down at an inopportune mo

Broadcast message from victor@highlyimportantserver.theregister.co.uk

(/dev/pts/0) at 16:50 ...

The system is going down for halt NOW!

13
0
Silver badge

Re: Yes, I *know*, shut up anyway!

"Fortunately, Victor learned his lesson"

Only if he installed Mollyguard

5
0
Joke

So... were there any other sites hosted in the same server?

The title mentions a "top tech news site" that went down, but the article text only mentions El Reg, so I think something is missing.

21
0
Silver badge

I wonder

if the downtime was any longer that el-reg would have set the BOFH on 'victor'

Lift shaft accident or fell out of the window...... the world must know

4
0
Silver badge

Re: I wonder

Well, he lived to tell the tale..... this time. If he had to drive down the road a bit to get his coffee, we have had a different ending involving lifts, windows.

4
0

I once included theregister.co.uk in one of our proxy blacklists and we also took it down for about 15 minutes. Once we realised what we did we removed it and it came back up, phew!

I hope no one else noticed.

3
0

Shirley you can't be serious?

1
0
swm

And then there is Major Major Major.

1
0

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018