Email is crazy
68 points by FlyingSnake
68 points by FlyingSnake
The envelope is like an address written on the physical letter, whereas the header is like the sender’s signature on the letter. SMTP doesn’t check if they match.
There are a few reasons for this that deserve a little more unpacking:
Traditionally, message submission was done using a command such as sendmail -t which takes a message and creates an envelope from it. As part of that process the BCC: header’s contents are added to the envelope recipients and the BCC: header is deleted. When email moved from timesharing to client-server and SMTP started being used for message submission, the responsibility for creating the envelope moved to the MUA. (It wasn’t until several years later that client-server message submission and server-server message transfer were clarified as distinct protocols, so it was always either too early or too late to add a sendmail -t feature to SMTP.)
The other thing that happens during message submission is the authenticated sender’s address is put in the Sender: header, if it is different to the From: header. That allows you to make an author/sender distinction (useful for secretaries or when multiple staff are handling a role address) or even list multiple authors in the From: header. (I tried sending a multiple-author message over 20 years ago: almost everything assumes a single author so it did not work very well. I assume multi-author messages are even more broken these days.)
When a message is delivered to a mailing list, traditionally the From: header is left alone (so that recipients can reply to the author as usual) but the sender is replaced with the mailing list’s bounce handler. This allows the mailing list software to deal with list members whose addresses are broken, and people who send messages to the list (and who cannot fix it) are not bothered by the problems. Nowadays it’s more common for mailing lists to replace the From: header as well, because that works better with DKIM and DMARC.
Great insight. Thank you for sharing it. Email is a deep rabbit hole, many parts of it are truly arcane for modern programmers. Stuff like mutt, POP3, UUCP, X.400 etc seem to be lost in time.
When I started writing, I ended up with a 5 part blog series and still I hadn't touched many areas. I decided to cut the scope and keep it light enough that a casual technical reader can understand it. Email is indeed very fascinating.
Stuff like mutt, POP3, UUCP, X.400 etc seem to be lost in time.
I just want to say that I still use mutt to check my email. So I don't think it is lost to time.
I do use popular email services such as GMail, Proton, etc. but I have also been running exim4 on Debian as my MTA for a long time now. It writes all incoming email to a mailbox file at /var/mail/. Every few days, I run mutt to read my email.
The setup has DKIM, SPF, DMARC, etc. configured as well. It is unfortunate how much arcane knowledge it takes to run an MTA properly these days. I sometimes wish it were as straightforward as running an HTTP server, but it is what it is.
X.400 was wild. The stance was "tcp/ip and the internet are this interesting academic experiment, but lets do it 'properly' in a way which suits US and EU national telcos".
Hence you get X.25, OSI 7-layer stack, application protocols (P1+P7) etc, plus a bunch of structural built-in assumptions. e.g. addressing was (almost) stricly hierarchical: country, ADMD, PRMD, organisation, etc. country and admd are mandatory, there is no way to have an extra-national addressing entity, or one which was not hierachically "part" of an ADMD (intended to be one of the national telcos). One minor 'hack' of this format was the use of 'single space' and '0' for ADMD to represent "this PRMD is connected to all ADMDs in that country and this PRMD is connected to no ADMDs, non-delivery the message if you don't have a direct route".
There is some good stuff on this here: https://www.alvestrand.no/x400/debate/addressing.html
X.400 is a classic case of design by committee. The reality always trumps over such attempts. This recent article goes the other way to praise X.400, but they're panned by HN commentators.
https://buttondown.com/blog/x400-vs-smtp-email https://news.ycombinator.com/item?id=47873323
classic case of design by committee. The reality always trumps over such attempts.
I disagree, I think there is a significant confirmation bias here where people only think of things as "design by committee" when they do fail. Of course X.400 failed spectacularly but there are a bunch of cases where design by committee was a great success.
For example, Unicode was designed by a committee (well, industry consortium, but same deal) through quite a drawn-out process. It has now taken the world, everything uses Unicode. It is also a counterexample to xkcd #927: Standards, as it shows that you actually can in some cases replace a bunch of competing standards with a single one that covers everything. You just have to do a good job at it. It's hard, but fatalism certainly doesn't help.
Which part of unicode? Not UTF-8, at least.
The character set itself. UTF-8 was indeed invented by a small bunch of clever hackers, but is much younger than Unicode.
edit: And I want to add, both Unicode and UTF-8 are great examples for when their respective design process works well. Unicode is a big, complex and deeply human (and even political) thing. It is trying to capture all signs used by humans for writing. It is at its core a "make everyone happy, or at least not too angry" kind of design problem. This is what committees can accomplish.
UTF-8 is "just" an encoding. It is a small, very elegant technical artifact. It's been designed once and set in stone. The only mildly political aspect it contains is the fact that, unlike UTF-16 et al., it assigns a privileged position to ASCII. This is something that I think is best designed by a small group of very clever people.
It's not fatalism to call out "design by committee" approach, if we can learn from the mistakes. We've seen failures like CORBA, WAP etc as a result of it. Even Fred Brooks calls them out in the mythical man month.
This kind of attitude is pervasive in engineering. Similar baked-in assumptions were present in XHTML proposals, for example. And it didn't work either, being replaced by the "whatever works" of modern HTML.
mutt is one of the best tools available for handling gigantic mail volumes. The basic workflow is adapted from nmh:
Construct a filter to select the messages of interest; inspect for mistakes; correct if needed. Use the filter to tag all those messages; apply an action to the tagged messages.
mutt can handle million-plus message Maildirs on the same filesystem or multi-hundred-thousand message folders over IMAP with reasonable speed. (Why do those exist? Either users or automated processes. Users can be educated and given other tools, automated processes can be fixed or filtered out -- but mutt is the best way to do the initial cleanup.)
As an aside, I think the best part of email is that it's decentralized and federated. You just know that if it were invented today all the big-name providers (Gmail, Outlook, etc) would be total islands.
all the big-name providers (Gmail, Outlook, etc) would be total islands.
"Oh sorry, you need to download the app and send me Signal Mail™️"
The problem is, email is being constantly re-invented by the BigNames .. pretty much every social network in existence is an attempt to, eventually, replicate the features already available in email.
If only folks were taught to use email properly, we wouldn't need any of these BigNames. It is precisely due to the deleterious nature of technology that these things happen .. and that is driven at the personal level more so than by BigName. Its the user who decides email is too hard, lets just use SocialMediaTool instead, because "its sexier..."
If only folks were taught to use email properly
I think this is both a losing fight and a misuse of resources. Why keep spending effort on teaching people a user-unfriendly technology over and over again (reminder that new people will keep getting born) instead of investing into something that actually works? And even if you do everything "the right way" today, email just doesn't offer an acceptable level of security. Really the only thing email has left going for it today is federation, and even that is no longer such a unique selling point. We are long overdue for a replacement.
Its not so black and white.
Email is here to stay and people should learn to use it properly - to write effectively, to quote each other properly, to attach files and use it well. I don't agree that EMail is 'broken' - I think peoples expectations that they don't have to learn to do things according to proper standards is how we get continuously broken standards.
But this is a battle long fought in the tech world. Do we do whats best for the user, or whats best for the organization that owns the code? This is a never-ending dialectic quagmire.
I've seen kids learn to use email properly and grow up to be very effective communicators as adults. I've also seen kids who learned the 'easy to use and fun stuff' who can't communicate worth a damn. The problem is not the technology - the problem is the ethics.
I think peoples expectations that they don't have to learn to do things according to proper standards is how we get continuously broken standards.
There's a saying that I keep being reminded of that goes "You have a set of levers to pull. Human nature is not one of them". Any solution to a society-level problem that goes "everybody just has to X" is doomed to fail. Now you might argue that morally, everyone should, but how does that help?
I wouldn't discount that computer technology used to be something that was explicitly, formally taught in an institutional setting, right alongside reading and writing and every other formal subject. Students went to the computer lab and spent time "learning to computer" just as they learned to read and write and do arithmetic. Education might not be human nature, but we nevertheless really do expect everyone to participate, and the vast majority do. Not all collective action problems are completely intractable :D
CS is still taught formally. And there has always been a large group of self-taught or informally taught programmers...
You think email is crazy? Wait until you see the World Wide Web, it's absolutely bonkers!
SMTP based email didn't do too bad for an old workhorse that had to keep up with the times. As a thought experiment imagine writing in the same style about modern web technologies.
I think the underlying protocols are not even close to the mess mail is. HTTP has evolved, but it's versioned and QUIC is a proper replacement. No duct taping like mail. HTML and Javascript are technically languages and not protocols, so you can't really compare them. Keeping them backwards compatible kinda makes sense.
But the utter chaos developers built ontop of it, truly funny to imagine the books it could fill!
I think the underlying protocols are not even close to the mess mail is.
I dunno, there’s a lot of brokenness in HTTP:
In my experience SMTP is much more solid at a basic syntactic level. The worst aspect is that the binary MIME standards were never widely implemented so everything has to be funnelled through printable ASCII.
QUIC is in no way a replacement for HTTP. It's an implementation of tcp on top of UDP and is oblivious to the content transmitted.
Opportunistic TLS is pretty crazy because most mail servers accept any certificate when connecting. Invalid, expired, and self-signed don't matter. MTA-STS and DANE, as the article mentions, could fix this up but adoption remains low. I'm betting on MTA-STS.
I wrote about that absurdity a while back. https://alexsci.com/blog/is-email-confidential-in-transit-yet/
Standard practice for transactional email does not require authenticated encryption, so stuff like password reset emails are vulnerable to pretty basic MITM attacks. I'd love to see a transactional email provider add MTA-STS support.
That’s a good article, thanks for writing it.
Part of the problem is that when SMTP STARTTLS was specified it didn’t clearly state how to validate a certificate nor what to do when validation failed. In particular it doesn’t say what domain name should appear in the cert, and there are at least three possibilities:
As a result mail servers began using STARTTLS without validation and were widely deployed with certificates that could not be validated. It has been very difficult to recover from that mistake.
To be fair, there were a few pre-existing difficulties that contributed to the problem.
Mail servers had a long history of hosting multiple mail domains, and long histories tend to involve messy setups. If strict validation had been a thing from the start, a postmaster who was deploying TLS would first have to go through a cleanup campaign to ensure the DNS records for all their domains are set up correctly. Otherwise enabling TLS would have broken mail delivery.
There are a couple of things that prevent STARTTLS from working with certificates that validate mail domains. Firstly, messages can contain recipients at multiple domains, so the certificate would need to validate all of them, which is problematic for mail servers with large numbers of domains. Secondly, the mail server doesn’t find out the recipient domain until after TLS is established, and there was no TLS SNI back then.
I hoped that DANE would fix the validation gap between mail domains and MTA host names but it has sadly been unsuccessful.
To be fair that's too much knowledge not true for most client to server connections and email due to its nature is the place where you should very much use E2E encryption. Sadly we live in a mess of tools on this front.
Sure, client-to-server SMTP handles encryption differently. It's a lot easier for end users to enable authenticated encryption for the one (or small number of) servers they use for email. I'd love widespread E2E too, but until a viable option appears I keep pushing for better server-to-server validation.
Most people think email is real-time, but it is not. It is a queuing system with retry logic. If the destination server is down, the message stays on the queue and is retried with exponential backoff at 5 minutes, 30 minutes, 1 hour and so on up to 4-5 days.
Wow! I did not know this. I recently screwed up the MX records on a domain that I was transferring to a new hosting service, and missed a few emails in the process. I was worried that these emails were just gone forever, but I was surprised to see the emails I missed (and my test emails) slowly trickle their way into my inbox over the course of several hours, once I fixed the MX records. Now I know why that happened!
Oh yes, this is why I laugh a bit at people saying "omg self hosting is insane how do you do 99.9999% uptime?!" cuz like email was designed when systems has 0.0001% uptime; they'd dial each other periodically to forward messages and retry later when they can't. So you'll be quite fine just letting it go down sometimes, it'll catch up eventually.
Actually, when I ran a backup email server, I was more likely to lose messages because I'd check the backup so infrequently I would forget all about it! So better to just have the one and let the system do its thing.
And spammers will definitely hit the back up MX in the hopes that it's less picky and more spam will get through. I run my own email server, and in twenty years, I haven't bothered with setting up a secondary MX server, and it's never been an issue.
I've had to learn all this stuff getting my own domain to reliably deliver and receive email. I cut my teeth in the early 90s forging emails by typing SMTP commands in telnet. What we have now is nearly incomprehensible.
The key unlock for me was to stop using Google services to send or receive email. They really don't do some of the modern encryption stuff right, particularly Google Workspace, and I was losing email because Gmail was bouncing Google's own forwards as spam. I'm now using Cloudflare to deliver email (to Gmail) and SMTP2Go to send email (from Gmail) and it works pretty well.
Yes, Cloudflare Email forwarding is pretty good. Their new Email sending is also quite easy, but it's still in beta.
do you know if they have an SMTP relay I can point Gmail at to send mail? I couldnt' find one, their sending product seems very oriented to being used in integrations for software.
No. Cloudflare doesn’t have an SMTP relay (like SMTP2GO). The Email Sending is a paid product and that’s tightly integrated in the Cloudflare Dev environment.
At work our phish report tool comes in the form of a button added into our mail client UI and when clicked it forwards the suspected phish to a reporting address with a [phish alert] prefix added to the subject.
One time Google started rejecting this phish report because it looked too much like a phishing email.
Email is bad, and continues to be bad, because the SMTP people want it to be that way. I don't say this idly.
I was involved in the preliminary anti-spam investigations in IETF ASRG working group the early 2000's. Sure, there were clearly anonymous spammers in the group trying to keep spam going. But it also became clear that the SMTP people didn't want it fixed, either.
If you proposed a solution which would "fix" the spam problem, they would mock you. They had a checklist for "anti-spam kooks", and had no shame about pointing out how your solution was stupid, naive, etc.
On the other hand, if you had a suggestion which would narrow down the scope of the spam problem (i.e. stop one particular behavior), they would mock you. They didn't have a checklist here, but they were very happy to point out how your solution was stupid, because it only solved part of the spam problem, and not all of it.
People who pointed out the contradiction in the two views were either mocked, or banned.
SMTP implementations could make simple changes to improve the ability to distinguish spam from non-spam. So far as I last checked, most (or all) of them don't do that.
It should be possible to design a series of small changes to SMTP which would address the vast bulk of the spam problem. When I pointed this out in ASRG, the SMTP people claimed that spam was a hugely difficult problem, and responded as above.
Except that my background is nuclear physics. In the late 1990s, CERN was getting something like 4 terabits a second of data. Then after a few years of running, would publish a paper with 10 data points. The methods which let them do this were simply a carefully designed series of discriminators (which was the technical word for the equipment). That equipment "sliced and diced" the data in various different ways.
I explained the above points in ASRG, and they responded predictably.
Many of the problems with email are due to the underlying design. Sure, there are reasons for it. Sure, it was designed a long time ago when we didn't have as much experience with designing protocols. So SMTP violates a large number of modern protocol design principles.
You (and billions of people like you) get spam for one reason, and one reason only: The STMP people have refused for 25+ years to fix the problems.
I'd genuinely love to hear your thoughts on spam prevention. While modern mail spam filters seem to be OK, I am seeing a similar level of spam from SMS and social media.
It's been a long time, and I haven't looked much into any spam issues around SMS or social media.
For SMTP, there are some basic design decisions which make it easier to send spam:
For SMTP implementations (MTAs and viewers)
SPAM filters don't seem to do basic statistics. i.e. if 99% of my traffic comes from the same people or lists I always talk to, why doesn't that get a different set of filters than the 1% of traffic from the unknown Internet swamp? Why not split traffic based on ongoing conversations, history, etc.
There is some tracking of reputations, but from when I last looked into it, it wasn't enough.
I could go on, but that's the high level summary.