back to article Bombastic boss gave insane instructions to sensible sysadmin, with client on speakerphone

Hey, hey, it's Friday! Which means weekend frolics aren't far away once you get through this edition of On-Call, The Register's weekly reader-contributed tales of workplace woe. This week, meet “Craig,” who shared a story of working for a small IT services company that hired a new “team leader”. Craig used italics because …

Silver badge

So this law firm, with business-critical emails, were ONLY storing them in an outlook PST on a local machine?

God, they deserve everything they get, not a recovery.

I mean, if they had Exchange - LITERALLY no excuse. Just delete the mail account on the client and re-load and it'd pull back down from the database.

But LOCAL Outlook folders? Obviously with "delete from server" turned on rather than just syncing to a huge IMAP folder held in the cloud somewhere? And then stored on only ONE client, but they just-so-happen to expect the tech to back everything up for them?

Law-firms have serious and quite clear obligations on the retention of emails. These guys failed from the start, even if the tech-guys managed to restore something.

32
57
Bronze badge

Your high horse is getting tired...

Back in the day we had a 4GB limit on the Exchange mailstore. Not per mailbox, but for the whole database. Sure, at the time that was a lot, but when you have 100 staff in a solicitor's office, it's amazing how quickly it goes.

"Well you can pay for Enterprise, or we can extract the data to PST, but please be careful with them" becomes "Well you can pay... eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee... or we can extract... eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee..." which is quickly answered by "the second one".

Also, bear in mind that around these parts, partner solicitors don't draw a (meaningful) salary, but take a split of the profit at the end of the year. So when you ask them to pay for something, that's taking money out of their pocket.

This story is agonisingly familiar - I have been the miracle-worker for this kind of crap more times than I really want to consider.

115
1
Anonymous Coward

Re: Your high horse is getting tired...

We had so many problems due to that 4Gb limit when I worked in a local authority. Our policy was very clear that e-mail wasn't a file storage tool and they should be keeping important stuff on their network drives so it could be backed up. The PSTs couldn't be as our infrastructure wouldn't support dragging such massive files back and forward on our piddly little broadband connections between sites even if we'd wanted to.

The "fix" wasn't to train and educate users in why we got them to do it (basically to avoid the problems in this article) but instead to simply create an additional PST when the old one filled up but keep both attached so users could double dip to their hearts content.

Needless to say the first time a director lost his PST due to a HDD failure we were forced to pay over 2 grand for an attempt to recover the files then suddenly this policy became rather important.

.. for about two weeks before everyone just continued as before.

49
0
Anonymous Coward

Re: Your high horse is getting tired...

4Gb... you where lucky. I used to work on a system using a FoxPro database (complete with 1Gb limit) for its accounting package that also hooked into the website to show customer accounts. When it hit that limit... everything stopped until we could free up some space and get it limping along again.

Oh and the only tool it would talk to was MS access...*shudders*

17
0
Bronze badge

When it hit that limit... everything stopped until we could free up some space

I know that feeling, too. But with Exchange it was a case of "everything stopped and God couldn't make it work again".

There may have been ways to coax it back to life, but I didn't take the chance. When Exchange 2003 SP2 came out, I'd always set the limit to 72GB in the registry, keeping those 3GB in hand for the inevitable emergency when it would hit the stops.

Telling the boss "all of the company email will stop in 4 months if nobody clears out" had the desired effect (unusually). That company was still using Exchange 2003 in 2012, with the same 72GB mailstore limit...

16
0
Silver badge

Re: When it hit that limit... everything stopped until we could free up some space

Really, people?

I'm sure historically such things were problems. But the cheapest of RAID storage in an Exchange server (let's not get into VMs, SANs, etc. okay) will give you so much space your user's email really shouldn't be an issue.

If they're using it for file storage, set a limit on the EMAIL SIZE not the mailbox size.

But solicitors et al do have very specific retention requirements. Expecting a local PST to do anything is failing to abide by them before you start (e.g. the guy could delete them and claim he never got them). If nothing else, if they get hit with a order to produce, your IT guys are going to be held accountable for not being able to provide it. The fact that the machine went titsup six months ago and took everything with it is not acceptable to a court, or client.

Even a SOHO Exchange or IMAP setup should be retaining those emails beyond what the user can generate in a year, and then you can backup to long-term storage and clear out the mailbox if necessary. TBH, who do you think will win in court - your solicitor's suing you for failing to provide adequate retention despite a service contract, or you trying to argue that they never instructed you to?

0
30
Silver badge
FAIL

Re: When it hit that limit... everything stopped until we could free up some space

You have no clue what you are talking about. The 4 GB was hard coded into Exchange. You can put the data store on a 1 Exabyte drive and make not wit one of a change that limit. Things are marginally better these days. I, for one, refuse to touch it beyond finding out what might be bogging up Exchange. I'll leave it up to the even more insane people than I to deal with it.

12
0
Silver badge

Re: Your high horse is getting tired...

When it hit that limit... everything stopped until we could free up some space and get it limping along again.

Sound better than dbm on 32-bit Solaris just wrapped around, once file size hit 2GB, and began overwriting itself.

6
0
Silver badge

Re: When it hit that limit... everything stopped until we could free up some space

"The 4 GB was hard coded into Exchange."

I'm wondering whether that number shared some common ancestry at some point with the file size limit of 4GB of FAT32 file systems...

5
0
WTF?

Re: When it hit that limit... everything stopped until we could free up some space

I'm utterly baffled about these limits. My Oracle database hit 1TB - back in 2001 on Windows NT 4, and with no sign of getting anywhere a limit. Doesn't Outlook use SQL server as it's backend? I'm pretty sure that even back then, SQL server could go over 2TB.

And in 2012, we still had that Oracle database running on Windows 2003, then over 200TB and growing by at least 2TB a month.

0
0
Silver badge

I really do not understand why you have suffered such a high level of down-voting. This is exactly how I would have expected any company to operate - centralised IMAP storage with workstations simply caching a month or so's emails on their local HD.

0
0
Silver badge
Facepalm

rather than just syncing to a huge IMAP folder held in the cloud somewhere?

Er, you do realise this is a LEGAL firm being talked about here - with significant expectations of privacy and so on. Backing up stuff "on the cloud"?

The cloud.,, Where many companies disappear with little or no notice, or where the feebs etc can come in and seize servers etc.. Confidentiality, only the lawyers emails (and other data) are maybe housed on a data store that miraculously gets seized by the feebs investigating totally random and honestly-just-coincidence-we-had-no-idea-it-was-the-same-place child porn allegations from "an anonymous tipoff".

Any law firm that uses anything other than some form of private cloud (at worst) should not have any customers, you cannot trust them with your data.

And you complain about them failing "from the start".....

4
0
Bronze badge

Down votes for attitude?

The article is really about managing incompetent bosses who are unaware of their own ignorance or lack any belief in their team but the discussion immediately went to explaining what the client was doing wrong without the full details of the client's situation.

Looking at any entrenched IT system within an established organisation, in my experience, always throws up a handful of WTF items. No need to immediately assume that the business in question is being run incompetently. My working assumption when finding something looks unbelievably stupid for no good reason is that I don't yet know the full details.

14
0

Re: "I'm utterly baffled about these limits."

The 4GB limit (and similar limits from that era) is not usually due to concerns about physical storage capacity (primarily).

On a 32 bit architecture, the highest value you can store as a native integer is 4GB, assuming you don't store negative values. Allowing higher numbers involves either using more bits to represent them or switching to non-integers. Both of these would be less memory efficient and slower (and less accurate in the case of floating point values).

The practical outcome is that if the highest number your code stores is 4GB then that becomes the effective maximum file size - as you could never store the location of any data past that point in a file.

These days, systems are either 64 bit or are performant enough to take the hit of not using native 32 bit integers, so the "addressable locations" in the memory store rise and files can be larger.

2
0

Re: When it hit that limit... everything stopped until we could free up some space

iirc (been so long I may be wrong) but it was totality of the edb that mattered not the available drive space.

0
0
Anonymous Coward

Re: When it hit that limit... everything stopped until we could free up some space

That would have been horrible because typical disk sizes then were 10s GB and commercial disks were smaller and more expensive (In 2000 laptops shipped with 4GB drives - I had a 26GB drive in my desktop - 16GB drives were still standard in servers so 60 drives for 1TB) .

Disks were not hot-swap and RAID mirrors still broke in horrible ways. The only way I'd seen this done was with large storage arrays and vendor specific raw AIO drivers (typically the disks are not handled by the OS but by the application)

1
0

Re: When it hit that limit... everything stopped until we could free up some space

>I'm utterly baffled about these limits. My Oracle database hit 1TB - back in 2001 on Windows NT 4, and

>with no sign of getting anywhere a limit.

How did you get 1TB of drive space for a single database back in 2001?

On Windows NT 4 of all OS, which had a bug that prevented you from creating any partition size above 136GB?

>Doesn't Outlook use SQL server as it's backend? I'm pretty sure >that even back then, SQL server could go over 2TB.

Well, no. Neither Outlook (the email client) nor Exchange (the email server) are using SQL server. Both use their own, internal database engine(s)...

>And in 2012, we still had that Oracle database running on Windows 2003, then over 200TB and growing

>by at least 2TB a month.

Well, Windows Server 2003 had a maximum partition size of 256TB...

2
1

Re: When it hit that limit... everything stopped until we could free up some space

> How did you get 1TB of drive space for a single database back in 2001?

Two ways:

a. don't use windows NT (2 - 4 Gigabytes was the address space limit for 32 bit addresses and in 2001 there were systems that used 64 bit addresses)

b. on MS Windows construct the database from more than one database file spread across several Windows partitions

Since you asked.

1
0
Silver badge
Headmaster

Can I get that in writing?

I've been in the position of "pulling the trigger" and I also protested, and said with clarity that I wouldn't be responsible for the outcome. Needless to say, I spent weeks cleaning up the mess, but someone above me had their ass in the sling for quite some time.

51
0
Anonymous Coward

Re: Can I get that in writing?

I was asked to do something I was uncomfortable with on a database. I said I wasn't happy doing this and was told to get on with it. I said can I have that instruction in writing please. The person who had told me to do these changes then got cold feet and said "You're not filling me with confidence". I nearly said "Likewise".

50
0
Silver badge
Coat

Staff?

he'd previously fixed their jammed printers, added new users to the company domain and lots of other mundane staff.

So he was a recruiter too? And if he was surely he wants to bring in better than "mundane" people ;)

28
4
Anonymous Coward

Re: Staff?

No, I was a techie, employed by a different company to provide on site tech support.

Nothing to do with hiring.

9
4

Re: Staff?

Back to sub-editing school for me, then :-(

7
0

"Mundane staff"

I've worked with those. Heck, I've been one of those.

41
0
Silver badge
Headmaster

Re: "Mundane staff"

Haven't we all (to both statements)...?

It was one of the most valuable lessons I learned in post-graduate academia - always know who actually moves things along and gets things done, as opposed to who appears to be in charge but just orders them to be done. And to get on their good side and in their good books....

Forget being nice to professors and directors and suchlike, if you were on friendly terms with the secretaries, storesmen (or probably storespeople these days, although back then they were all white-coated males) and technicians you could get anything done and circumvent all sorts of delays and bureaucracy.

And it was amazing how quickly and easily things got signed off by the senior staff when you went via their secretaries, whom you took care to always be nice, friendly and helpful to of course.

58
0

Re: "Mundane staff"

Secretaries & Security - I always tried to stay on their side when I was posted to customer sites, and while it didn't pay off too often, the investment didn't hurt.

One of the secretaries was English working in the US, I just need to pack a packet of Maltesers every trip to keep her sweet. That customer was tricky & she was a senior person's PA, so THAT was worth the grid

32
0
Silver badge
Windows

The worst I've had is a team leader who one week ordered me to spend more time looking after a critical system as the weekend and the next week ordered me to not record doing it (and therefore not get paid for it).

29
0
Silver badge

I had a similar situation- I told him to fuck off.

34
0

So he knew how to fix it anyway?

'Had a third party PST repair tool that worked'

So why didn't he just run that and get out of there in the first place? I guess he was trying to drop the new team leading in the shit but if I was the Lawyer firm I'd be asking why 'Diannes' time was so utterly wasted by the clown on site playing stupid games when he knew how to resolve the situation in the first place.

68
12
Bronze badge

Re: So he knew how to fix it anyway?

My only guess on this front is that as a "third party tool" it was an "off the internet" application and not one that had been vetted by the company - so he would have needed to confirm use of it to CYA.

If not, I'm just as puzzled as you.

50
0
Silver badge

Re: So he knew how to fix it anyway?

He phoned his office to consult with a colleague and instead got stuck dealing with his boss.

RTFA.

55
12
Silver badge

Re: So he knew how to fix it anyway?

Are you suggesting there was some sexual tension between 'Dianne' and 'Craig'? There's currently on the TV a toothpaste ad where an attractive, 21 ish year old woman, who's free spirited enough to make stupid faces at a child playing with a glove puppet on a train, gets shunted up against a pretty boy with a quiff. They then exchange flirty, faux-embarrased looks, and the woman does the classic 'lady di' pretending to be shy and avoiding eye contact. Genius Ad Execs, trying to persuade overweight women from Rochdale that their life is crap, and that they too can have exciting encounters on a train if they use that particular brand of toothpaste. Needless to say, this ad is annoying the shit out of me.

Anyway, that's 'Craig' and 'Dianne', when looking at a paper jam.

24
0

Re: So he knew how to fix it anyway?

Sounds to me like Craig is a bit of a dick who cared more about scoring points off the people back at base (and the new boss he didn't like in particular) than saving the client's emails. Dianne should not have hailed him as a hero.

14
59
Silver badge

Re: So he knew how to fix it anyway?

Surely we've all met someone who thinks he's an expert because he can turn a computer on?

I meet these people constantly at work and they tend to attract too much attention in the first place.

13
0
Silver badge

I don't get it.

His boss was an arse, yes.

Can registry changes make any difference to the loss of data in a PST file?

No. No it cannot. so other than the TL being a tosser, this just doesn't stack up. The data was either present before the registry change, or it was not. The registry changes had no effect on this at all.

Then we move on to this this tool he's familiar with, but totally didn't think to use until phoning base and asking for advice. He then remembered and "saved the day". What exactly did he try other than recreating the mail profile and (presumably) running the baked in scan/fix PST jobby in office?

He ran this on a backup he'd taken, so basically the PST file as it was when he arrived (which suggests the data was not lost at this stage) just a corrupt PST, that';s probably fucking huge.

He was given poor advice, by a dickhead, but the dickhead didn't cause or make the issue worse.

I also suspect he walked off site leaving the same person with the same mega-gigabyte PST file that will corrupt again in short order.

16
7
Silver badge

Re: So he knew how to fix it anyway?

"if I was the Lawyer firm I'd be asking why 'Diannes' time was so utterly wasted by the clown on site playing stupid games when he knew how to resolve the situation in the first place."

For one thing I'd guess Dianne was Craig's lawyer appointed minder. Craig would be handling client confidential information. In those circumstances the lawyer would expect to have someone keep and eye on things and, if he'd any sense, Craig would also want want that too.

40
0

Re: I don't get it.

"Can registry changes make any difference to the loss of data in a PST file?"

Hmm. Mostly I suspect that the team leader believed that messing about in the registry performed some chicken bone ritual. It had worked for him before so try it again... There are so many implementations of Outlook and so many app versions that there might be a bug which prevents correct reading of a PST file.

If running an enterprise Outlook/Exchange environment, wiping out HKCU settings and anything in %AppData% for Outlook would be relatively harmless. The user would lose a few preference settings but the mailbox would be auto configured to work again.

Wiping out HKCU etc in a non-enterprise environment would probably screw up Outlook and require tech intervention.

Wiping out HKLM entries for Outlook is bad advice. You cannot be sure that Outlook will apply sensible defaults when it finds them missing.

14
0
Anonymous Coward

Re: I don't get it.

Its nothing personal, but you weren't there.

You have no possible idea of what happened.

You can speculate all day.

I walked off site knowing that the email issue was resolved, my reputation remained in good standing and that when the inevitable complaint(s) came in I was able to fight my corner with evidence and a witness.

Say what you like about the circumstances but facts are facts.

42
7

This post has been deleted by its author

Anonymous Coward

Re: So he knew how to fix it anyway?

Running cracked software on site was a no-no.

The inbuilt tool to repair them was failing to do so.

I wanted to speak to the senior tech to bat ideas back and forth.

I had limited options and ONLY when I had exhausted the "legal" ones, did I employ "illegal" ones.

Sometime, you have to take the bull by the horns and do what's needed to get the fucking job done.

Again, its easy to analyse what I did and find fault. I was there, you weren't.

60
3
Anonymous Coward

Re: So he knew how to fix it anyway?

I managed both. Thanks.

And I assure you, for saving his emails, no matter how trivially easy to a techie, will ALWAYS get you in the good books.

9
0

Re: So he knew how to fix it anyway?

To quote Aladdin Sane,

"He phoned his office to consult with a colleague and instead got stuck dealing with his boss.

RTFA."

14
2
Anonymous Coward

Re: So he knew how to fix it anyway?

To quote Aladdin Sane,

"He phoned his office to consult with a colleague and instead got stuck dealing with his boss.

RTFA."

6
2
Anonymous Coward

Re: So he knew how to fix it anyway?

Furthermore, you must also take into account, I only have .a certain amount of time to do the job, I only had the minimum of tools to do the job, there is only so much you can do on site within allotted timeframe before you have to wave a white flag and RTB with it.

There were so many other daft things I had tried, deleting temp files (even hidden ones), clearing cookies, resetting IE, short of reinstalling windows I did *everything* I could. But the FIRST thing I had done was backed up that .pst file. As I have said, I tread cautiously.

I've worked with IT since the mid 70's and although I'm far from a sys-admin (not my particular field) I am a bloody home-pc technician and I've used Windows more or less exclusively since 3.1 (I had Amiga whilst others struggled with DOS) so I know how to fix most errors and then, reinstalling the entire OS then updating, putting all the custom software back on etc was a bloody pain. At this point, I introduced them to Norton ghost!

It's difficult to cram a couple or 3 hours work into a short, succinct story.

So forgive me if I didn't do things your way. I did it my way! ;-)

36
1
WTF?

Re: So he knew how to fix it anyway?

Me too me too. I actually thought it might have been made in some less sophisticated country (Romania, USA etc.) because this stuff was passe in the Noughties.

Anyway they deserve each other.

2
0
Anonymous Coward

Re: I don't get it.

You have no possible idea of what happened.

Wasn't that the point of the action of telling the story - to let us know what happened??

3
6
Silver badge

@Lewis

"So why didn't he just run that and get out of there in the first place?"

For the record: I totally agree with you.

However... Hindsight is a wonderful thing, isn't it? That's the part we both ignored at first. Heat of the moment, you don't always think straight and perfect. Especially not with that kind of manager looming over your shoulder.

6
0

This post has been deleted by its author

John Cougar Mellencamp Song? (wasRe: So he knew how to fix it anyway?)

Maybe Craig and Dianne were fantasising about the John Cougar Mellencamp song Jack and Diane?

Yes sad but it's 1am in Sydney!

1
0
Silver badge

"Here's a little ditty about speakerphones that On-Call finds amusing"

Error loading Youtube: Video could not be played...

It's mildly amusing...

6
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2017