* Posts by jwkinneyjr

1 publicly visible post • joined 16 Jun 2011

A sysadmin's top ten tales of woe

jwkinneyjr

Staff Efficiency

Many years ago in a universe far far away, I was the IT managing partner in a small architectural firm. We used a Wang minicomputer (remember those?) for bookkeeping duties that was generally managed and maintained by an outside service.

One day the service technician walked into my office with a face the color of fresh clean photocopier paper. He explained that he had made an error in upgrading some routine and the Wang was not working. He mounted the supposedly current backup (the techs always called ahead and asked for the staff to do a fresh backup before they came over to make changes) however the newest backup was showing 8 week old data. These backups were done on 8 inch floppy disks that were verified at the end of each set. We had 4 disk sets in rotation, all properly numbered, verified and dated. Sure enough, all the current sets including the one dated earlier in the day had 8-week old data.

So I asked the bookkeeper to show us how she had done the backup. Instead of going to the regular backup screen, she pressed a combination of keys which flashed a warning signal, "For computer technician use ONLY. Bookkeeping personnel DO NOT USE." Near the center of the screen was an option titled "Add updates to current backup." The bookkeeper said, "I've been using this update for the past couple of months. It's a lot faster than the regular one. I don't know why they don't want us to use it."

The technician turned even whiter, checked the backups again and sure enough. The backup record showed that it had been run daily for the past two months, dutifully noting that there had been no system updates during that time. The technician finally asked the bookkeeper to do paper printouts of all the reports on the system, which the bookkeeper grudgingly agreed to do. Later that day, the technician walked out with two boxes of printout and the hard drive from the system. We hired some help and switched to manual bookkeeping (payroll that day as I recall).

A week later the service boss called and said they were unable to reconstruct the data from the hard drive in a meaningful way. He offered to loan us an identical computer so we could reconstruct the system by entering the data by hand from the paper reports and proposed that we share the cost. I finally convinced my senior partners that this was a fair arrangement. The service also modified the Wang dos so that updating the system would never overwrite valid data. We were back in operation in about 3 weeks.

We eventually outgrew the Wang and switched to a miniPDP, but that's another near-tragedy for another time.

John