back to article 'I crashed AOL for 19 hours and messed up global email for a week'

Welcome again to Who, me? In case you've missed previous editions of the column, it's a confessional in which readers share their stories of having broken stuff. Badly. This week, a fellow reader named "Bert" asked you to "Cast your mind back to the mid-90s when America Online (AOL) was the biggest online service and the …

I wonder if he is the same guy

I remember one guy explained something about AOL mail inner workings on Slashdot, a usual "citation needed" nerd asked "How would you know?", he replied "Because I coded it."

64
0

http://edition.cnn.com/TECH/9608/09/aol.fallout/index.html

" The Internet is developing into a mainstay, and cyber junkies are finding there are plenty of other on-line services out there, many with on-line fees significantly less expensive than AOL. "

Those were the days

31
0
Holmes

This brings back memories. I remember the early days using bbcnc, cnn and cnet, although this was at university. Icon for reminiscing.

6
0
Anonymous Coward

One of my network colleagues came into the office one morning to announce that the internet had finally reached a tipping point of public awareness in the UK.

He had just followed a van down the A1M motorway - which had a company URL painted on the back door. Probably late 1990s.

47
0
Silver badge
Pirate

In the late 90s there were a couple of cars in Truro, Cornwall which had email addresses printed in a very large font on. I guess they must have thought they were the coolest guys on the planet :)

As Penzance isn't far away -->

33
0

Probably fully formatted on the vehicle as:

http://www.company_name.co.uk/index.html

I had a friend who operated a mobile disco service, and considered himself to be quite “with it” on the internet. He wasn’t bad, as he was perfectly capable of setting up a hosted domain and coding a rudimentary HTML page complete with every flashing and marquee effect the W3C had made available. An early adopter of internet advertising, he had posted his URL on his brand new vehicle INSTEAD OF his phone number.

His business pretty much evaporated overnight. For two reasons. One of which was that it was 1997 and no-one that seemed to require his services owned a computer, let alone a modem. Secondly, the URL on his van had omitted all the non alphabet formatting... it said along the lines of httpwwwmobilediscocouk.

42
0
Silver badge

I was in USA in 95 , and noticed all the movie posters had URLs on them

5
0
Silver badge
Facepalm

"INSTEAD OF his phone number."

just wow

31
0
Anonymous Coward

When I was at university in Oxford in the early 80s there was a company selling audio equipement who used their phone number as their business name with regular ads in the student what's on listings with the tag line "you know our name, you know our number" ... which some of us referred to as "you don't know our number, you'll never find it!"

10
0
Silver badge
Trollface

In the late 90s there were a couple of cars in Truro, Cornwall which had email addresses printed in a very large font on. I guess they must have thought they were the coolest guys on the planet

FTFY.

28
1
Silver badge

'In the late 90s there were a couple of cars in Truro, Cornwall'

Typical northern metropolitan elite with their private transport.

24
0
Silver badge

"INSTEAD OF his phone number."

Do people get phone numbers from random vans in the street anyway?

I would normally look it up, back in the day in a phone book, these days online of course.

Vans don't come into it.

8
1
Silver badge
Devil

The guys with round wheels were unbearably smug too....

the work of the devil them -->

5
0
Silver badge

Re: "INSTEAD OF his phone number."

From random vans? No idea, never tried. From a passing cab? Hells yeah!

4
0
Silver badge

Re: "INSTEAD OF his phone number."

One of my license plate frames had my Berkeley bang-path address across the bottom in about 1980. Still does, in fact. The top line reads "Go Bears!". I got all kinds of shit about it when I got to Stanford ;-)

5
3
Anonymous Coward

Re: "INSTEAD OF his phone number."

One of my license plate frames had my Berkeley bang-path address across the bottom in about 1980

"bang-path" - we don't wish to know about your extra-curricular activities sir.

4
0
Coat

But

Truro, Cornwall is in the South West (They still can't do a cream tea correctly though).

2
0
Silver badge

Re: But

It may be in the south west of Great Britain, but it's still north of The Lizard.

And nothing tops clotted cream, you heathen.

3
1
Silver badge

Re: But

"nothing tops clotted cream, you clot."

FTFY

4
3
Silver badge

Re: But

'Truro, Cornwall is in the South West'

If you come from the right side of the 50th parallel Helston is dangerously northern.

0
0

Re: "INSTEAD OF his phone number."

Actually, this morning I took a photo of the local sandwich shop delivery van in Tesco's car park as it had the number on the side and we might need a buffet next week!

1
0
Silver badge
Go

With hindsight

Out of curiosity, would anybody be able to explain how you could fix/prevent this if the only load balancers available were wholly insufficient for the task?

20/20 hindsight is allowed but you only have the existing resources of 1995/6

35
0
Silver badge

Re: With hindsight

Yes.

6
2
TRT
Silver badge

Re: With hindsight

A the true face of capitalism. Theory would suggest that as there is a gap/need in the market for some meatier load-balancer, someone will invest the capital to exploit that gap. Reality suggests that it's easier to find a cheap kludge than capital.

15
3
Silver badge

Re: With hindsight

did the story actually tell us how the outage was resolved?

13
0
TRT
Silver badge

Re: With hindsight

I was wondering what triggered the cascade failure in the first place.

18
0

Re: With hindsight

A better solution would have been exim, seeing as this resource exhaustion tale has all the hallmarks of a sendmail shop. In the 1990's a friend of mine was running a forum mailing list for a very well known computer software company and was suffering the same sort of problem caused by sendmail's 1 msg then 1 delivery -> 1 process to deliver it "paradigm". A bit of a problem on a forum with 5000 people on it. He changed to exim and the problem went away.

10
0
Silver badge

Re: With hindsight

Commercial load balancers were a bit shit but you could do something with a fast PC with a handful of network cards installed. They had FDDI on the outside and even then you could install versions of Linux that could then fan out that FDDI with a host of machines with 10Mbit ethernet on the inside, either simple round robin or something slightly more intelligent.

7
0
Silver badge

Re: With hindsight

All they had to do to figure out load balancing was ask one of the major Usenet outfits. Wouldn't have changed the main problem, though ... AOL's mail system was a home-grown clusterfuck that had Internet email grafted onto it as an afterthought. The guy that ran it (not this "Bert" character, but rather someone I'll call "L") told me on more than one occasion that he wished he could rip it all out and start over with something sane like sendmail (!!), but the PTB wouldn't let him.

qmail and postfix had nothing to do with AOL's issues. qmail was written to address security issues that weren't seen as a problem when sendmail was written; postfix was written to be an easier alternative to sendmail. qmail's bones were laid down in late 1995, before the AOL meltdown. postfix was an IBM-Watson reearch project about a year later, and I have it on good authority that IBM didn't give a rat's ass about anything AOL was doing.

3
2
Silver badge

Re: With hindsight

exim wasn't really ready for prime time until around 1998, some would say 2000.

1
2
Silver badge

Re: With hindsight

qmail was written to address security issues that weren't seen as a problem when sendmail was written

Indeed. My home[1] use of sendmail was *extremely* short (as in a matter of days) I then switched over to this new, secure and lightweight qmail (and later added on ezmlm rather than listserv for mail list handling)

From the article - remember this 1996 when servers were feeble – the server would probably reboot

I've *never* had one of my linux boxes reboor itself because of swap exhaustion. Probably because I use the above-mentioned qmail (and postfix on FreeBSD boxes) rather than sendmail. Both of which handle queuing a hell of a lot better than sendmail does (not difficult!)

[1] I foolishly later took a job herding Sun boxen - all of which used sendmail. So I got to experience the joys later. Including on making it play nicely with Exchange 5.5 - which advertised that it supported ESMTP. At which point the Solaris sendmail tried doing batch SMTP delivery - which Exchange couldn't handle and so silently discarded the emails. Fixed by telling sendmail to ignore ESMTP announcments for the internal set of IP addresses that the various Exchange boxes used. Those were the good old days of packet-switched frame relay networks..

1
0
Silver badge

Re: With hindsight

I had a thought about this ...

They could have fiddled with the DNS to get a poor mans load balancer. Set the mx to (say) a.domain.tld with (say) a ttl of 3 hours. After (say) half an hour, change the mx to b.domain.tld, also with a tld of 3 hours. After another half hour, change to c.domain.tld. And so on. You could script the DNS updates to automate it.

Then each resolver would cache just one of a.domain.tld, b.domain.tld, etc and so (using the numbers originally given would try and contact only one of 5 different MXs. Different resolvers would cache different records depending on the timing of when they last fetched the records. That was definitely doable back then.

If they had geographically distributed servers then they could also have done some conditional DNS stuff to present different MXs to different area - can be done with BIND using views, but I don't know whether that feature was available then.

0
0

Re: With hindsight

Correct - Exim was "experimental" for a fair few years. Postfix came out in 1998, but I didn't use it till v2.2.

smtpd_timeout was a lovely thing, not to mention all the smtp client timeouts going the other way (no waiting forever tying up a process waiting for a receiving MTA to respond).

0
0

1995/96, using Claris Emailer on my Quadra 700, so exciting.......sigh....

2
0
Silver badge

'95/'96 we both had JANET email access at work and we had a rented property with a single phone point just inside the door so our Centris 650 was offline. It had to wait for the move up here to Dundee end of '98 and into our own house with a Telewest cable account and a separate phone line to get it online.

Eudora was our mail app of choice, at work for me as well as at home. Having two email addresses, one at home and one professionally seemed luxurious and somewhat decadent.

16
0
Thumb Up

Sendmail hacking to the rescue

I was working as a data network administrator at an Australian university when this happened. Luckily I had written local hacks into sendmail to do a form of exponential back off when emails were unduly delayed. At the time we hosted a number of listervs and other various sundries, as well as 25,000 users doing their normal thing.

So when the outage hit our mail queue grew to tens of thousands of emails, a fair number of which were to the mailing lists (hi Pavement fans mailing list) enquiring if anyone else hand noticed the outage and asking others to reach out to users who weren't answering; my response, "Are you helping? Good, well stop." Our poor Sparcserver20 reached a load average of 128 but it stayed up, one of only a handful of Aussie university mail servers that didn't bounce at least once during the outage. I know some other university mail admins null routed the email to AOL via DNS lame delegation hacks. All our email to AOL eventually delivered about 48-72 hours later.

Did I mention I hand wrote our sendmail.cf file?

40
0
Silver badge

Re: Sendmail hacking to the rescue

I vaguely remember it now, because of the effect on ListServe. I had a UK JANET account at work back then and other than the effects on email and listsserve we had ringside seats. That big fat, optical JANET pipe was as serious luxury. I could send multi megabite email attachments (science data) and apart from having to confirm that I wanted to do that it would go. Had to be to another JANET account though.

7
0
TRT
Silver badge

Re: Sendmail hacking to the rescue

Ah... I can remember the days when science data used to fit on a SSSD Floppy... and 10MB seemed to be an infinite limit for a mail attachment.

10
0
Anonymous Coward

Re: Did I mention I hand wrote our sendmail.cf file?

I once had a general sysadmin-type inteview where I was asked if I'd ever hand-editted a sendmail.cf. I think it was a trap, but couldn't be sure. However, I had done it, just once - I'd added a line (or commented one out, I forget exactly) to block the open relay that was there in the default, so I gave that reply.

Mind you, I didn't get the job. Maybe they needed gung-ho and/or expert sendmail hackers? Or was me even thinking about going near a sendmail.cf file with a text editor too crazy for words? I suppose I will never know.

14
0
Anonymous Coward

Re: Sendmail hacking to the rescue

"10MB seemed to be an infinite limit for a mail attachment"

When I was rputting together my first PC, Tiny Computers sold you the box but you had to add a HDD and I remember thinking as I bought a 40MB HDD that despite buying the smallest HDD available I was gettting something so vastly huge (twice the size of the disk on my work PC) that I was never likely to fill!

(Though going back another 5-10 years I can remember when adding a 4kB RAM *card* - that's really 4096 bytes - to the 6800 processor at school was a huge deal as it meant we could play StarTrek!)

13
0
Anonymous Coward

Re: Sendmail hacking to the rescue

"Did I mention I hand wrote our sendmail.cf file?"

You couldn't figure out how to work m4 macros either?

13
0
Silver badge

Re: Sendmail hacking to the rescue

"I bought a 40MB HDD [...] I was never likely to fill"

Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic if said PC was meant for anything beyond Haiku storage.

0
9
Anonymous Coward

Re: Sendmail hacking to the rescue

"Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic if said PC was meant for anything beyond Haiku storage."

Actually, small (physically - 5.25 inch) floppies used to hold 160 kb, compared to the 2nd generation full sized (8 inch) floppies which held a massive 250 kb.

I eventually provided office applications (word processing, spreadsheet, database) for an entire office (one floor of a 20 story office tower) running on a computer with a 5MB hard drive, enabling us to ditch the stand alone word processing machines.

0
0
Childcatcher

Re: Sendmail hacking to the rescue

Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic

You had 1.44M floppies? You kids didn't know when you had it good. We had 180K floppies, and were grateful for it, didn't have to hump cartons of punchcatds around anymore.

4
0
Silver badge

Re: Sendmail hacking to the rescue

1.44 megs? Luxury ...

Seriously, my first 8" floppys were 256K ... My first 5¼s were 160K. The system ran off one of those disks. In those days, the thought of a 40meg HDD at home was purely in the realm of fantasy. (By way of reference, in mid-1980 an 18 meg NorthStar drive cost in the neighborhood of $4,200 ... in 1980 dollars. About a year and a half later, Apple debuted a 5 meg drive for $3500 ... People lusted after these GIGANTIC storage devices.)

1
2
Silver badge

Re: Sendmail hacking to the rescue

Oh boo-hoo. And punched cards held mere bytes. Yeah, the 8" floppies on our CP/M Z80 boxes held stupid little, but it's irrelevant - by the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, nobody used anything other than 1.44 floppies; compared to which 40 megs were a luxury, but a vewwy-vewwy modest luxury indeed. Nobody I knew walked around with less than a full box of floppies by then, and when your existing data instantly takes up over a third of your allegedly humongolicious new HDD, starting to longingly ogle one at least four times as big before you even installed this one is what you do, not expecting it to never fill up.

0
9
Silver badge

Re: Sendmail hacking to the rescue

I do beg to differ: our first pc, an IBM .... 80something-or-other had a harddisk. I think something like 20MB? And two 5.25" floppy drives.

My first PC with the non-floppy 3.5" floppies had a 250MB hdd. That was... in the early-to-mid 90s

0
0
Silver badge

Re: Sendmail hacking to the rescue

By the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, the IBM XT came with a 10meg HDD and a 5¼ 360K floppy. 1.44s were a number of years in the future. Nobody I knew walked around with a full box of floppies, unless they had just purchased them.

3
2
Bronze badge

Re: Did I mention I hand wrote our sendmail.cf file?

dnl

I remember when Allman & co decided that sendmail.cf was too hard and what would really make it easier was getting everyone to write it...in m4. Cos yeah, we all dream in preprocessor languages.

dnl

0
0
Silver badge

Re: Sendmail hacking to the rescue

Did I mention I hand wrote our sendmail.cf file?

I take it that now far enough in the past for you to have recovered from the sanity loss?

(I've written a few in my time. Mostly by copying ones from other people and amending them..)

0
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums

Biting the hand that feeds IT © 1998–2018