I wonder if he is the same guy
I remember one guy explained something about AOL mail inner workings on Slashdot, a usual "citation needed" nerd asked "How would you know?", he replied "Because I coded it."
Welcome again to Who, me? In case you've missed previous editions of the column, it's a confessional in which readers share their stories of having broken stuff. Badly. This week, a fellow reader named "Bert" asked you to "Cast your mind back to the mid-90s when America Online (AOL) was the biggest online service and the …
Probably fully formatted on the vehicle as:
I had a friend who operated a mobile disco service, and considered himself to be quite “with it” on the internet. He wasn’t bad, as he was perfectly capable of setting up a hosted domain and coding a rudimentary HTML page complete with every flashing and marquee effect the W3C had made available. An early adopter of internet advertising, he had posted his URL on his brand new vehicle INSTEAD OF his phone number.
His business pretty much evaporated overnight. For two reasons. One of which was that it was 1997 and no-one that seemed to require his services owned a computer, let alone a modem. Secondly, the URL on his van had omitted all the non alphabet formatting... it said along the lines of httpwwwmobilediscocouk.
When I was at university in Oxford in the early 80s there was a company selling audio equipement who used their phone number as their business name with regular ads in the student what's on listings with the tag line "you know our name, you know our number" ... which some of us referred to as "you don't know our number, you'll never find it!"
A better solution would have been exim, seeing as this resource exhaustion tale has all the hallmarks of a sendmail shop. In the 1990's a friend of mine was running a forum mailing list for a very well known computer software company and was suffering the same sort of problem caused by sendmail's 1 msg then 1 delivery -> 1 process to deliver it "paradigm". A bit of a problem on a forum with 5000 people on it. He changed to exim and the problem went away.
Commercial load balancers were a bit shit but you could do something with a fast PC with a handful of network cards installed. They had FDDI on the outside and even then you could install versions of Linux that could then fan out that FDDI with a host of machines with 10Mbit ethernet on the inside, either simple round robin or something slightly more intelligent.
All they had to do to figure out load balancing was ask one of the major Usenet outfits. Wouldn't have changed the main problem, though ... AOL's mail system was a home-grown clusterfuck that had Internet email grafted onto it as an afterthought. The guy that ran it (not this "Bert" character, but rather someone I'll call "L") told me on more than one occasion that he wished he could rip it all out and start over with something sane like sendmail (!!), but the PTB wouldn't let him.
qmail and postfix had nothing to do with AOL's issues. qmail was written to address security issues that weren't seen as a problem when sendmail was written; postfix was written to be an easier alternative to sendmail. qmail's bones were laid down in late 1995, before the AOL meltdown. postfix was an IBM-Watson reearch project about a year later, and I have it on good authority that IBM didn't give a rat's ass about anything AOL was doing.
qmail was written to address security issues that weren't seen as a problem when sendmail was written
Indeed. My home use of sendmail was *extremely* short (as in a matter of days) I then switched over to this new, secure and lightweight qmail (and later added on ezmlm rather than listserv for mail list handling)
From the article - remember this 1996 when servers were feeble – the server would probably reboot
I've *never* had one of my linux boxes reboor itself because of swap exhaustion. Probably because I use the above-mentioned qmail (and postfix on FreeBSD boxes) rather than sendmail. Both of which handle queuing a hell of a lot better than sendmail does (not difficult!)
 I foolishly later took a job herding Sun boxen - all of which used sendmail. So I got to experience the joys later. Including on making it play nicely with Exchange 5.5 - which advertised that it supported ESMTP. At which point the Solaris sendmail tried doing batch SMTP delivery - which Exchange couldn't handle and so silently discarded the emails. Fixed by telling sendmail to ignore ESMTP announcments for the internal set of IP addresses that the various Exchange boxes used. Those were the good old days of packet-switched frame relay networks..
I had a thought about this ...
They could have fiddled with the DNS to get a poor mans load balancer. Set the mx to (say) a.domain.tld with (say) a ttl of 3 hours. After (say) half an hour, change the mx to b.domain.tld, also with a tld of 3 hours. After another half hour, change to c.domain.tld. And so on. You could script the DNS updates to automate it.
Then each resolver would cache just one of a.domain.tld, b.domain.tld, etc and so (using the numbers originally given would try and contact only one of 5 different MXs. Different resolvers would cache different records depending on the timing of when they last fetched the records. That was definitely doable back then.
If they had geographically distributed servers then they could also have done some conditional DNS stuff to present different MXs to different area - can be done with BIND using views, but I don't know whether that feature was available then.
Correct - Exim was "experimental" for a fair few years. Postfix came out in 1998, but I didn't use it till v2.2.
smtpd_timeout was a lovely thing, not to mention all the smtp client timeouts going the other way (no waiting forever tying up a process waiting for a receiving MTA to respond).
'95/'96 we both had JANET email access at work and we had a rented property with a single phone point just inside the door so our Centris 650 was offline. It had to wait for the move up here to Dundee end of '98 and into our own house with a Telewest cable account and a separate phone line to get it online.
Eudora was our mail app of choice, at work for me as well as at home. Having two email addresses, one at home and one professionally seemed luxurious and somewhat decadent.
I was working as a data network administrator at an Australian university when this happened. Luckily I had written local hacks into sendmail to do a form of exponential back off when emails were unduly delayed. At the time we hosted a number of listervs and other various sundries, as well as 25,000 users doing their normal thing.
So when the outage hit our mail queue grew to tens of thousands of emails, a fair number of which were to the mailing lists (hi Pavement fans mailing list) enquiring if anyone else hand noticed the outage and asking others to reach out to users who weren't answering; my response, "Are you helping? Good, well stop." Our poor Sparcserver20 reached a load average of 128 but it stayed up, one of only a handful of Aussie university mail servers that didn't bounce at least once during the outage. I know some other university mail admins null routed the email to AOL via DNS lame delegation hacks. All our email to AOL eventually delivered about 48-72 hours later.
Did I mention I hand wrote our sendmail.cf file?
I vaguely remember it now, because of the effect on ListServe. I had a UK JANET account at work back then and other than the effects on email and listsserve we had ringside seats. That big fat, optical JANET pipe was as serious luxury. I could send multi megabite email attachments (science data) and apart from having to confirm that I wanted to do that it would go. Had to be to another JANET account though.
I once had a general sysadmin-type inteview where I was asked if I'd ever hand-editted a sendmail.cf. I think it was a trap, but couldn't be sure. However, I had done it, just once - I'd added a line (or commented one out, I forget exactly) to block the open relay that was there in the default, so I gave that reply.
Mind you, I didn't get the job. Maybe they needed gung-ho and/or expert sendmail hackers? Or was me even thinking about going near a sendmail.cf file with a text editor too crazy for words? I suppose I will never know.
"10MB seemed to be an infinite limit for a mail attachment"
When I was rputting together my first PC, Tiny Computers sold you the box but you had to add a HDD and I remember thinking as I bought a 40MB HDD that despite buying the smallest HDD available I was gettting something so vastly huge (twice the size of the disk on my work PC) that I was never likely to fill!
(Though going back another 5-10 years I can remember when adding a 4kB RAM *card* - that's really 4096 bytes - to the 6800 processor at school was a huge deal as it meant we could play StarTrek!)
"I bought a 40MB HDD [...] I was never likely to fill"
Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic if said PC was meant for anything beyond Haiku storage.
"Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic if said PC was meant for anything beyond Haiku storage."
Actually, small (physically - 5.25 inch) floppies used to hold 160 kb, compared to the 2nd generation full sized (8 inch) floppies which held a massive 250 kb.
I eventually provided office applications (word processing, spreadsheet, database) for an entire office (one floor of a 20 story office tower) running on a computer with a 5MB hard drive, enabling us to ditch the stand alone word processing machines.
Considering floppy disks used to carry 1.44 megs meaning 40 megs held less than thirty disks, and even back then a single floppy held basically not a whole lot of anything, that sounds a tad bit optimistic
You had 1.44M floppies? You kids didn't know when you had it good. We had 180K floppies, and were grateful for it, didn't have to hump cartons of punchcatds around anymore.
1.44 megs? Luxury ...
Seriously, my first 8" floppys were 256K ... My first 5¼s were 160K. The system ran off one of those disks. In those days, the thought of a 40meg HDD at home was purely in the realm of fantasy. (By way of reference, in mid-1980 an 18 meg NorthStar drive cost in the neighborhood of $4,200 ... in 1980 dollars. About a year and a half later, Apple debuted a 5 meg drive for $3500 ... People lusted after these GIGANTIC storage devices.)
Oh boo-hoo. And punched cards held mere bytes. Yeah, the 8" floppies on our CP/M Z80 boxes held stupid little, but it's irrelevant - by the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, nobody used anything other than 1.44 floppies; compared to which 40 megs were a luxury, but a vewwy-vewwy modest luxury indeed. Nobody I knew walked around with less than a full box of floppies by then, and when your existing data instantly takes up over a third of your allegedly humongolicious new HDD, starting to longingly ogle one at least four times as big before you even installed this one is what you do, not expecting it to never fill up.
By the time buying a HDD for a home computer was a thing that even non-NASA personnel could reasonably do, the IBM XT came with a 10meg HDD and a 5¼ 360K floppy. 1.44s were a number of years in the future. Nobody I knew walked around with a full box of floppies, unless they had just purchased them.
Biting the hand that feeds IT © 1998–2018