back to article You shrunk the database into a .gz and the app won't work? Sigh

Welcome yet again to On-Call, our unimaginatively-named regular recap of readers' recollections of jobs gone wrong. You all liked a double-barrelled story so much last week we've decided to do it again, starting with “Gary” wrote to tell us of his time working for Burroughs and the job he was sent to do for a bank with a …

Silver badge

Re: When life imitates art...

That's assuming that you could have found out which port it was plugged into, which would require a network engineer who both knew the password for the switch, and was willing to actually log in and tell you the information, and if they were that helpful, they probably wouldn't have ended up as a network engineer.

(I'm joking network ops people. Mostly.)

26
0
pig

Re: When life imitates art...

"and if they were that helpful, they probably wouldn't have ended up as a network engineer"

LMAO. Funny and true.

12
0
Silver badge

Re: When life imitates art...

You could have told them which switch port - if all switches are intelligent. There are plenty of installations where only core switches are intelligent, or none of them are.

8
0
Facepalm

Re: When life imitates art...

Ah, server location... One previous place had a server which no-one quite knew where it was as there were a few comms cupboards in the place. After setting up something to ping a message to /dev/console didn't attract any attention (screen was probably in standby, or just no-one was in the room), I figured out how to use the speakers (Sun Ultra-1) to make a noise. Not just any noise, you understand, somehow I thought it would be a good idea to nab the bit from Holy Grail - "help,help, I'm being repressed". Cranked the volume up to full and set a cron job to cat the .au file to /dev/audio every 30 minutes.

Of course, through a comms room door, the only bit passers by could hear was "help, help". Which led to security being called to find who was trapped & where. Of course, yelling "hello, hello?" at the area to identify the "trapped person" didn't elicit a response. Until the next cron run and they checked the console and tagged me. Of course, the security guard being someone I knew, he didn't call me direct, he called our mutual friend who also worked to let him know about it which gave him no end of amusement :)

So, slight embarrassment all round, but we found the server. Which was very useful a couple of weeks later when some builders needed to get a cable out of the way and just snipped through the cat 5...

15
0
Silver badge

Re: When life imitates art...

"You could have told them which switch port - if all switches are intelligent".

Unless of course you were on a site full of 3Com switches and half your management modules were permanently in transit to/from 3Com.

7
0

Re: When life imitates art...

That's almost like the story of the Netware server whose location was lost, yet it was still working. It was eventually found behind some drywall.

14
1

Re: When life imitates art...

This is where servers with speakers and CD/DVD-drives are useful... "We know it is in the server room somewhere and we have remote access it it - eject the drive and listen/watch": I've had to do this more than once....

13
0

Re: When life imitates art...

Well, our network people are absolutely brilliant! (are you listening, Denise)

5
0
Silver badge
Thumb Up

Re: When life imitates art...

...3Com switches and half your management modules were permanently in transit to/from 3Com.

Those wouldn't have been 3CEME modules, would they? They used integrated RTC/SRAM/Battery modules from Dallas that had solder joint issues...after a while, the SMT solder joints on the SRAM inside the modules would open up. I was at 3Com and did a redesign to eliminate them.

Then they canceled the whole Eclipse product line.

4
0
Anonymous Coward

Re: When life imitates art...

"You Could have told them which switch and port to go follow the cable from."

Did that, it wasn't much help as the switches were in a room 20 metres away and the cabling for it (5 long Cat5 leads ziptied together) went into the space above the ceiling tiles ;)

I have no idea why, given that there were closer switches in the room in question. I can only assume there weren't any ports available when the server was originally stood up and someone bodged it.

3
0
Silver badge

Re: When life imitates art...

Had to do that today. KVM switch to a stack of half a dozen PCs. One was booked for replacement. Ejecting the DVD drive was easier than painfully tracking all the cables.

1
0
Anonymous Coward

Re: When life imitates art...

That story about the Netware server was a VAX when I last read it.

1
1

Re: When life imitates art...

Don't count on the switches to be intelligent. Ping (continuously) the lost host, and start unplugging cables. If ping continues to work, plug the cable back in. If it doesn't, then the lost host is somewhere at the end of that wire.

3
1

Re: When life imitates art...

http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/

1
0
Silver badge

Re: When life imitates art...

"Those wouldn't have been 3CEME modules, would they?"

These switches were bog-standard SuperStack series, nothing to do with the Eclipse series but the management modules on those things dived more often than continental footballers.

Management was happy with the free "lifetime warranty", even though I was frothing at the mouth about six-week turnaround time and not being allowed to carry spares.

1
0

Re: When life imitates art...

but, but, but, it's NEVER the network that's at fault!!!!

at least according to the network engineers I regularly have "energetic" discussions with.....................

2
0
Anonymous Coward

Re: When life imitates art...

Ping (continuously) the lost host, and start unplugging cables.

Not a recommended approach for any switch that could even potentially be carrying important traffic.

Cue: "Our production Oracle database just failed over, the standby then failed, and the original isn't failing back properly. Were you doing something in the server room?"

2
0
Silver badge

Re: When life imitates art...

Your software can't recover from a few seconds of disconnection ?

Time to change vendors.

0
0
RW
Boffin

Re: When life imitates art...

"some builders needed to get a cable out of the way and just snipped through the cat 5..."

Builders (usually "contractors" in Canada and the US) do that kind of thing. The Atlantic City Music Hall contained a huge pipe organ with pneumatic action. Eventually it fell into disuse, but remained intact....until contractors working on renovations simply cut through the bundles of tubes that connected keyboard with pipes.

[I'm paraphrasing this from memory and hope I got it right.]

0
0
Silver badge

Re: When life imitates art...

"wasn't much help as the switches were in a room 20 metres away and the cabling for it (5 long Cat5 leads ziptied together) went into the space above the ceiling tiles"

That's where one of these (https://www.youtube.com/watch?v=PKgzPYNu6-g) comes in handy.

There are a bunch of tracers on the market with greater and lessor ranges. If truly lost, invest in a Cat and Genny. These are somewhat bulky in a server room but if you know how to use them properly you can tell where a cable is in the wall/floor/ceiling/ground and _how far_ buried it is (it's nice to know "6 inches vs "10 feet")

0
0
Silver badge

Re: When life imitates art...

"Ejecting the DVD drive was easier than painfully tracking all the cables."

Labelling the fecking things avoids the issue in the first place - using labels that won't fall off.

Brother TZe-S641 label cassettes for the equipment (sticks well) and TZe-FX641 for cables (other colour combos are available) are worth buying and can be obtained for far less than the £25 apiece that Brother flog 'em for.

If you have "problems" with people removing labels, then "security" labels will dissuade that practice (TZ-SE4)

Dymo, Brady and others all sell similar labelling systems. Whatever you have make sure you have a good stock of tapes and a labeller in every area (they're not particularly expensive compared with having to trace cables) or people won't do it.

My estimate is that it costs about 8p/label.

0
0

Point of order - you mean the database's native compression, right? That's not the same as encryption!

8
0
Silver badge
Facepalm

In another life

I worked for a restaurant that had a computerized ordering system. When I started, I was told that the system needed to be backed up every night. When I was shown the procedure, I asked which floppy disk to use next I was told it didn't matter just grab one out of the drawer which had about 25 unlabeled floppies!

17
0

"Shrank"

see above.

3
1
FIA
Bronze badge

Gary mis-heard the address and instead went to the bank's branch in the town of Otley, about 15 kilometres away and a half-hour's drive when the traffic is kind.

I'm from around here and I can promise you no-one travels anywhere using one of those fancy modern fangled kilometre things. (Especially now we've left Europe), infact I'm not even sure most folk know what one is.

We travel to our underground places of gainful excavation in Gods own miles.

And nowt else.

37
3

We used to dream of travelling underground....

20
0
Silver badge

"our underground places of gainful excavation"

I didn't think there were any in Otley. Maybe you're thinking of Ossett.

1
0
Anonymous Coward

"We used to dream of travelling underground...."

night Night Tube ....?

1
0
Silver badge

I grew up in Downingtown, Pa. Several Burroughs plants there and nearby. My father worked there, as did my best friend's father, the fathers of everybody in the high school computer club... except Alan, whose brother did.

Ah, those were the days. Dad's in a nursing home these days, my best friend is the spitting image of his father back then, and I've pretty much lost track of the rest. Me? I develop firmware for a flight simulator company.

17
1

@jeffrey

I see you got a downvote for no apparent reason - unless it was your best friend who didn't need to be reminded that his youth is now far behind him...

Anyway, have a balancing upvote

8
0
Anonymous Coward

Takes me back to when I was in the Army and had a very nice deployment to the Sun Shine state in the US of A. Having just done by sysadmin course on the kit that I was to maintain I was surprised to hear they didn't do backups because 'they took to long'. That changed..

There was also a security account that all emails were copied to so they could be (manually) checked for illegal words. There was a weekly job to check this which hadn't been done for a long time.. It took the whole night shift to delete them. I've no idea how many there were as the total never got calculated completely.

4
0

I started at a new company a while ago and did all the usual checks and measures to see how the land lies.

Backups was first. Every day we received a "Backup Complete" email, great! Backups are being done....until you look at the amount of data backed up. Approx 1GB of data was being backed up, successfully, to the single tape backup. Alarm bells started ringing, especially as the mailstore itself totals 90GB. There's no compression algo in the world that is that efficient right?!

After further investigation, it transpired that we were backing up a single database only. The database was essentially an index of what was supposed to be backed up, but not the actual files themselves. Yosemite Server Backup had decided that, despite not backing up well over 400GB of data, the fact it had backed up 1GB of an index was successful enough.....

Sorted now though, with a successful backup AND restore tests!

13
0

@rjstua - There are 2 types of companies in the world.

Those who have yet loose data through failed backup routines, and who don't do test restores...

14
0
Silver badge
Joke

"Yosemite Server Backup had decided that..."

What? People still use Tapeware?

0
0
Anonymous Coward

MOJ

Why am I not surprised that MOJ and the company building one of their systems thinks its OK to use production data in an offsite development environment :-(

10
0
Silver badge

Re: MOJ

If you can't trust a jiffy bag of CDs in the post to securely transfer confidential data - what can you trust?

11
0
Silver badge

Re: MOJ

To paraphrase the old joke:

"Real government departments don't do backups, they just leave the CD of data on the train and let someone else back it up to Wikileaks"

13
0
Anonymous Coward

Re: MOJ

It's not ideal, but then where do you get fake data while taking full advantage of the ability of human stupidity to confound any imaginable system?

3
0
Silver badge
Facepalm

Re: MOJ

Why am I not surprised that MOJ and the company building one of their systems thinks its OK to use production data in an offsite development environment :-(

How often have great systems turned out to be a total clusterscrew because real world data turned out to contain lots of stuff that no one ever thought to test for? eg people from Scunthorpe and Arsenal not being able to sign in to certain websites/services because of "bad word" filters....

I wrote a little util some years back that did some work manipulating files from certain other programs. A user had a problem with it though, it created a temp file that grew in size until it had consumed his HDD space (pre GB days) then crashed. Thankfully it was a DOS system so easy to find and delete the file. What had happened was where my text editor transformed tabs into spaces, his transformed spaces into tabs, and I hadn't thought to handle any tab characters in the program - it didn't occur to me and I didn't have any data files that contained tabs. I couldn't re-produce the error untill he sent me some live data and then it took only a few minutes to figure out what was wrong. Icon for how I felt when I figured it out.

(I do hope like hell that the MOJ made absolutely certain that the data was secure.. Ah screw it, what could go wrong? He looks legit....)

2
0
Anonymous Coward

Half backups all the way

Arrived as an external contractor sap basis at a small SMB running Sap on NT and Oracle. DB had grown over the years and they had added discs. Wanted to check backup before performing a complex Intervention. Turned out they just backup the files online (no budget for the proper do online backup tool), and had not included the new disk in the file backup. So not valid backup ever.

Decided to not escalate but ask the sys admin to ensure we have a full, offline backup and would come back the next day.

Next day my boss gets a call from an angry SMB owner that I did something due to which the backup takes a lot more time, doesn't fit on the tape anymore and he would refuse for the second day of consulting fees and we were responsible for the downtime.

Next time I will just throw the local sys admin under the buss.

Worked out well in the end, but I always struggle to snitch on incompetent local sys admins, knowing some don't get the required training. It's not really their fault. I wish they would do the same, unlike this one, who blamed me

14
0
Anonymous Coward

Re: Half backups all the way

After thirty plus years I'm not sure which stories to share - decided on this one.

The internationally known financial services client in the US (one with a large hand in the recent 2008 unpleasantries) had called us in to look at their 12 year's out of support records management system.

I designed a migration and replacement plan and as part of that we were discussing backup. Outsourced guy at the table says "we have that covered, just a simple shell script that runs at midnight with a copy command to an external disk". On a running, production SQL Server database with 10+GB of data. Yep, just a folder copy should be fine.

14
0
Silver badge

Compression and encryption

Err..

Yes. They compressed the live production database. With something other than the database's native encryption too.

What has encryption got to do with the compression of the file - they are very different things.

Also, compressing something then trying to send it to tape is unlikely to result in a reduction of tape use, since all drives support on the fly compression and attempting to re-compress compressed data just makes it larger again.

If you really want to encrypt and compress, then compress first, then encrypt. The randomisation of encryption should (if the algorithm is any good) result in virtually uncompressable data.

6
0
Silver badge

Re: Compression and encryption

I'm guessing that stopping the application also meant stopping the Oracle instance, because a running instance isn't going to let you zip up its data files.

6
0
Anonymous Coward

Useless backups

I remember a new fairly large contract, where during due diligence, we were proudly shown a huge portacabin on site that was full of rusty steel cupboards packed full with various types of tape media. Not one of them was in the plastic box that protected from dirt and dust - since that took up space.

There was no environmental control of the portacabin and none of the tapes had any labels or barcodes giving any indication of what system they were from, what date they were created, what software was used or anything that might help in working out what treasures they may have contained.

We also learned that only a small volume of the tapes were active and the admins picked the tape they thought was next and used that, which might have been OK if there was some form of reliable rotation other than "pick one".

Obviously, no test restores had ever been done as everyone was too busy.

It was an interesting discussion point in the resulting report about the lack of any usable backups and one of the first tasks was to sort out that mess.

Anon for obvious reasons !

8
0
Silver badge

Re: Useless backups

On a similar note, but a smaller scale, I once got sent out to a local branch office of a major UK insurance company to have a look at a failed backup tape drive. Got there, had a quick look at the logs, went to the manager and told him the tape had expired. ie it was a dated tape which the backup software would refuse to use after said expiration date. "Can I have a new one please?" says I. "A new one?" says he, "what new one, we only have the one head office sent us when the it was installed.

A quick call to their HQ IT dept, and the guy says to me "ah good, did you find the fault? We've had 4 other offices call in with the same fault (voice in background "we just got another 5 called in"). So I tell him the reason for the backup failure and to likely expect calls from *every* local office if all the tapes came from the same batch, and the lack of any tape rotation (hard to do with just one!) and lack of spares. Cue a minutes silence. I suspect it was the time to pick up his jaw after it dropped rather than respect for the dead tape. Eventually he says they'll get on that right away and send a dozen tapes to every office and a new procedure description to do proper tape rotation. I also pointed out that the backup software config had an option to email the status report but that was currently blank. He sounded rather sheepish now he was over the shock but did explain he'd only been there a month and was still trying to get up to speed on the mess left by a previous sysadmin.

In all that chatting back and forth, I never did ask why no one had simply remoted in and looked at the logs themselves in the first place instead of just assuming a hardware fault. Maybe he didn;t have passwords? That'd not surprise me in the least.

5
0
Silver badge

Re: Useless backups

"rusty steel cupboards"

Yup. BTDT. The amazing thing is how frequently IT management refuse to spend money on doing backups right until you run a few quiet queries further up the food chain about what the cost of extended downtime of critical services would be (the usual answer is "It would break the company")

0
0

A lesson some other Oracle DBAs have probably learned over the years. Do not give your redo logs a .log suffix - A sysadmin WILL think they can be archived/deleted

7
0
Anonymous Coward

Disk failure

Started a new job and created new team. IT supplied drive space and and we dutifully wrote source code and filled it.

Two years later IT came to me and explained there had been a drive failure and they discovered that they'd not added my department's drive space to the backup schedule.

Being head of department I'd had my own backup schedule with offsite backups. Lost nothing. Had most of the QA department's data too.

3
0

Something similar....

I had a call that was very similar in nature.

Working one afternoon, I had a call from a customer in the US complaining that their document management system was down. Further investigation showed that the database was failing to start (which was a little odd). So, after 15 minutes I finally got to the bottom of the whole sorry state of affairs.....

The original guy looking after the system had left, leaving a very inexperienced person with the root password to the UNIX system, and very basic notes that said...

If the disk space fills up, delete any large files you find.

Guess what happened to the database files.

Oh, and the last back they took was over a week before.

5
0

Oh, and the time.....

....where I went to restore a system from tape backups and found the tapes all useless.

Worked at a company where backups were done it a bit of a strange way. I had to write a header file, followed by all the operational data, then the DBAs would back up the database (using code that they wrote and I had to call), and I would write a trailer file (so we could tell that we had a complete backup for that system).

That's all fine - until one of the DBAs changed the tape device from no-rewind to a rewind without telling anyone. I backed up my stuff, he backed up his (rewinding the tape afterwards), and I wrote the trailer file....

3
0

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018