This post is a translation of original.
Date: Sunday, November 24, 2002 9:03:02 PM
From: Trey Harris firstname.lastname@example.org
Subject: The 800 Km Email Case
Here's a problem that * sounded * impossible…
I almost regret posting the story to a wide audience because it's a great story to tell by drinking on a
The story is slightly altered to protect the
guilty, elude about irrelevant and boring details, and generally make the
whole more fun thing.
I was working on campus campus email a few years ago when I got a call from the president of the statistics department.
"We are having trouble sending emails outside the department."
"What's the matter?" I asked.
"We cannot send emails over 800 km," explained the president.
I choked on my coffee. "As well?"
“We can't send emails more than 800 kilometers from here,” he repeated. "A
little more, actually. I'd say about 830 km. But no further. ”
“Hmmm… email doesn't work like that, usually,” I said, trying
keep the panic out of my voice. No panic when talking to a
department chair, even from a relatively small department like statistics.
“What makes you think you can't send emails over 800 km?” I asked.
"It's not what I think," replied the president angrily. “See, when we noticed this happening a few days ago…”
"Did you wait a few days?" I interrupted, a slight tremor in my voice. “And you can't email this all the time? “
“We were able to send email. Just no more than - “
“- 800 km, yes,” I finished for him, “I get it. But why don't you
Did you call earlier? “
“Well, we didn't collect enough data to make sure what was going on so far. “
Right. This is the president of statistic.
"Anyway, I asked one of the geo-statisticians to take care of it -"
“- Yes, and she made a map showing the radius within which we can
email, just over 800 km. There are a number of
destinations within that radius that we cannot reach, or reach
sporadically, but we can never send emails beyond that radius. “
"I see," I said, and put my head in my hands. "When it started? A few days ago, you said, but have you made any changes to your systems recently? ”
“Well, the consultant came in and updated our server and restarted it.
But I called him and he said he didn't touch the email system. “
"Okay, let me have a look, and I'll call you back," I said, barely
believing I was leashing. It was not April Fools' Day. I tried to remember if I had done any pranks on anyone and that would be some kind of revenge.
I logged into your department's server and sent you some test emails. This server in the North Carolina Search Triangle, and a test email to my own address, was delivered smoothly. Same as an envoy to Richmond, Atlanta, and Washington. Another for Princeton (640 Km) worked.
But then I tried to send an email to Memphis (965 Km). Failed.
Boston failed. Detroit failed. I grabbed my address book and started trying to narrow the radius. New York (675 Km) worked, but Providence (933 Km) failed.
I was beginning to wonder if I had lost my sanity. I tried emailing a friend who lived in North Carolina but whose server was in Seattle.
Fortunately, it failed. If the problem had to do with the location of the
Human recipient and not your email server, I think I would have burst into tears.
Having established this - unbelievably - the reported problem was true, and replicable, I took a look at the file. sendmail.cf. It looked reasonably normal. In fact, it looked familiar.
I compared it to the sendmail.cf in my computer. It had not been
changed - was the sendmail.cf that I had written. And I was pretty sure I didn't enable the “FAIL_EMAIL_APÓS_800KM” option. Not knowing what to do, I connected via telnet to the SMTP port.
The server responded happily with a SunOS sendmail banner.
Wait… a SunOS sendmail banner? At the time, Sun was still
sending Sendmail 5 with your operating system, although Sendmail 8 was already quite mature. Being a good system administrator, I had standardized everything for Sendmail 8. And also being a good system administrator, I had written a
sendmail.cf well documented and with variables, available in Sendmail 8 instead of the cryptic settings and codes that were used in Sendmail 5.
The pieces fell into place at once, and I again choked on the dregs of my now cold coffee. When the consultant “upgraded the server,” he apparently had updated the SunOS version, and in doing so
outdated Sendmail 8 to 5. The update did not change the sendmail.cf, although now it was the wrong version.
It turns out that Sendmail 5 - at least the version Sun sent,
that had some adjustments - was compatible with the sendmail.cf from Sendmail 8, as most rules remained unchanged.
But the new rules were considered junk and ignored.
And the sendmail executable had no compiled defaults for most of these rules, so when it didn't find any proper settings in the file sendmail.cf, they have been set to zero.
One of the settings set to zero was the timeout to connect to the remote SMTP server. Testing a few things, I realized that on this server, with its typical load, a zero timeout would abort
the request in just over three milliseconds.
An odd feature of our campus network at the time was that it was 100% with switches. An outbound packet would not be delayed by a router until it reached POP and reached a router on the other side. So the delay to connect to a remote server on a nearby network would be largely governed by the speed of light to the destination rather than router delays.
Feeling a little dizzy, I typed in the terminal:
$ units 1311 units, 63 prefixes
You have: 3 millilightseconds
You want: miles * 558.84719 / 0.0017893979
“800 Km, or a little more.”
Trey Harris, 2002