Why is it so expensive to send an email?
55 points by wolfadex
55 points by wolfadex
Title should be "Why is it so expensive to send a lot of emails without getting them marked as spam".
Next up in the series: "Why does a single table row cost $15 per year?", a riveting deep dive into DNS.
The main reason that domain names cost anything is because cheap and free DNS providers always get spammed to hell with junk and abuse, then they don’t have enough money to tackle the ensuing problems. Pricing is not a complete solution to spam, but it can cut out the worst of it. (OK, the other reason is ICANN’s feudal tax, which is why the clearest examples of disastrous spam have been ccTLDs that don’t have to pay ICANN; happily they learned from their mistakes and cleaned up their acts.) (OK OK, and the trademark lawyers must be appeased.)
I pay $4.20/month for the VM that hosts my outgoing mail server. It's used by me and a few other people for a handful of domains. I haven't had any problems reported with mail delivery for ages, but I've also been using the same domain for around 20 years and that domain has built reputation. Being an outgoing mail server uses less than 1% of the power of that VM, so having an IP address that's able to send a bunch of email is fine. My provider retains a fairly good reputation for the IP block because their policy is that outbound port 25 is blocked by default, you have to request that it is unblocked, and they'll block it again if you are reported for sending spam.
I suspect that people like SendGrid exist because of the problem that I recall hearing from someone who ran anti-spam things for Yahoo! Mail back in the day: a lot of users find it easier to hit 'spam' than 'unsubscribe'. If someone actually did sign up for a mailing list but forgot or doesn't want it anymore, they'll often just mark the messages as spam and not bother to unsubscribe. In a big mail provider, spam filtering is shared to a reasonable degree between users, so if 20% of people who ever subscribed to your mailing start marking it as spam then you'll start to find that your mails are marked as spam for everyone.
I am a bit skeptical about companies like SendGrid for this reason: they seem to exist to do reputation laundering. If you send your emails with them, they'll send them from a set of IPs that sends so many emails that it's hard for big mail providers to block them and so that their spam:ham ratio is maybe not too high.
I haven't had any problems reported with mail delivery for ages, but I've also been using the same domain for around 20 years and that domain has built reputation.
There are two kinds of people who self-host email. People who say the above, and people who no longer self-host email because their emails don't arrive. Which group you belong to is half skill (getting DMARC, DKIM, SPF and email headers right, trawling through all the blacklists to get yourself unblacklisted) and half luck. Tomorrow, Google might change their spam filter and your emails now go straight to junk. You may never realize that your answer to the life changing job offer never arrived and you missed the window of opportunity.
Anyway, the first group of self-hosters telling the second group "it works for me" does not help. It's not that easy. The risk of emails not arriving is significant and troubleshooting such issues is a tremendous time expenditure. I personally gave up, it's just not worth it.
Jokes on you all: even the people paying for hosting don’t have reliable delivery. Half my emails from iCloud don’t reach Gmail.
Yup. I've seen GMail classify their own mail servers as spam sources.
Mail delivery is hard & outsourcing your mail hosting to one of the big mail hosters does not guarantee anything like reliable delivery.
My wife got a genuine “your storage is full” message from Google put in Spam, which is ludicrous. On the bright side, if they’re not even whitelisting their own stuff, maybe that gives others a better chance.
Certainly looking at her Spam folder I’m not impressed with their quality. Various at-least-semi-legitimate (mostly marketing to an existing customer) banking stuff went in there too (she has certainly never marked any such things as spam).
That's a different question than the article is addressing, though. (Unless I've misunderstood you!) It sounds like you're running mail service for a small group of people you know. Where you and those people are sending "handwritten" emails to people that the senders (mostly) know personally.
TFA is talking about transactional emails from their website that gets real traffic from the general public.
One of those things carries more deliverability challenges than the other.
a lot of users find it easier to hit 'spam' than 'unsubscribe'.
To be fair, I'm unable to unsubscribe from three advertising lists from very real sellers (two also with physical shops) and of course, I have been subscribed to these lists without my consent.
I've also unsubscribed from some other lists but even over several years, not many more than three. This means that I'm unable to unsubscribe from a very large part of the lists I'm on and from the vast majority of marketing emails I get.
In the old days, if someone sent spam, we'd write their network operator, and they'd get booted or otherwise punished. Once corporations and profit became involved, that stopped.
Like everything else in life, if we want something to stay nice, we need to punish those who want to make a profit from shitting on it.
It upsets me how much network-related stuff is treated like this. Focusing first on domestic cases (because international ones do complicate things more): spam phone calls, spam email, botnetting—all commonly treated as quite impossible to fix, when the solution is truly obvious and reliable, it just takes a directive to care and quite a bit of effort up-front to fix up decades of neglect, followed by gentler maintenance.
For all of its issues, i think the flexibility and now longevity of email is still quite impressive! Though, at the same time, I'll be the first person to agree that I wish we could create a new email system from scratch - of course, with properly applied learnings from all the decades that have been learned from conventional email.
Spam email is not something that can be fixed by only technical means.
Most of the problem arises from the economic structure of the protocol, in the broad sense of security economics: i.e. who is participating in the protocol and what their relationships are, in terms of contracts, payments, or the lack thereof. There’s also a macroeconomic version of Conway’s law (which I think is due to David Clark) which observes that an open protocol defines an interface between businesses that operate that protocol as a service, so when you are designing a protocol you need to keep that in mind because it determines who ultimately bears the costs.
The difficulty with spam is due to some of the basic attributes of email:
It’s common for techies to want to fix spam by improving sender authentication (that’s the whole point of SPF and DKIM) but it’s still easy for spammers to mint new identities, so spam continues.
The upshot is that anti-spam is mostly about reputation management. Receivers keep track of sender reputations and use that as a major signal for classifying spam; and senders police their customers to avoid damaging their reputation. Senders for this purpose can be identified by IP address as well as domain name or email address. That basic structure has been the same since Paul Vixie introduced the RBL, years before SPF and DKIM came along.
Much of the cost of sending email is to pay for that policing effort. Some of the cost is also to discourage the bottom-feeding spammers who rely on nearly-free sending to cope with a very low response rate. But there’s a somewhat delicate balance on setting the cost of email: it isn’t possible to make it too expensive for spammers but still affordable for legitimate senders. (Observe the ratio of paper junk mail to legit post that comes through your letter box.)
I lightly suspect that with the advent of AI bots and scrapers, we've finally brought this dynamic to HTTPS too.
I don't think spam is the only issue...but as it relates to spam: yep, i couldn't have said it better myself! :-)
I think basically all of the technical deficiencies of email can in principle be fixed by evolving the current system. I’ve ranted about this here before but the main reason for problems like shitty underspecified html is that the large email providers aren’t working with each other to improve things. (I suppose there are some things that have changed, such as DMARC and transport security, but they don’t have much positive effect on the way users perceive email.) And without a commitment to migrate the majority of email users to something better, all you have is yet another niche chat system, of which there have already been dozens.
Which is why it’s important to consider protocol economics when designing a replacement for email: there must be positive incentives in favour of adoption as well as negative incentives against abuse.
I just want to say thank you for your email comments, especially the linked rant. I share many of the same thoughts but am not as grounded in the history of it all. Your words help flesh things out.
And it's also nice hearing people beat the drum of the centrality of economics problems in protocols. Also the drum that email-qua-suite-of-protocols is Mostly Fine™ (I put DNS in a similar camp).
Something I am curious about is if this is avoidable in any federated/decentralized social media?
I guess I am not being spammed on Activitypub, but is that because of the protocol or because it is just not as popular?
I guess I am not being spammed on Activitypub, but is that because of the protocol or because it is just not as popular?
It’s a combination of a small community and the fact that it isn’t funded by advertisers. The people running instances have a fairly strong motivation to keep the network clean. Roughly what email was like before 1995ish. This can change for the worse if it’s easy enough to fake up a new server instance that the benefit of spamming outweighs the cost of playing defect, and if legit servers don’t have easy ways to vet the reputation of new instances.
I suspect it's popularity. I didn't get any spam on Signal until it got close to 100M users. Now I do and they spend a lot of time trying to block it.
Telephones, Email, Signal, and ActivityPub have the same root problem: they conflate identifiers with capabilities. If you know someone's identity (email address, ActivityPub address, Signal username or phone number) then you can connect to them. There is no way to give someone a capability that allows them to authorise a single sender to contact you but not a second one, for example. That alone would address a huge amount of spam (irrespective of underlying protocol) because I would be able to give a company a capability that let them add a connection to me, but didn't allow anyone else to use the capability a second time. If their contacts are breached, today they get a list of identifiers and these work as capabilities that allow their botnet to send spam. If they got single-use capabilities then the breach would leak keys that allow DKIM-signed email associated with a specific sender to send spam, and this would not be useful for spamming the customers.
I agree. There are some things to emphasize though.
Spam to a large degree comes from "reputable" big providers like Gmail, Hotmail, etc. They are legit email accounts used for spamming. A lot come from slightly smaller institutions, like marketing email services (Mailchimp, etc.), but the time when spam emails were silly scripts with no properly set up email server behind them is largely over.
Even with central authorities, costs involved, etc. people haven't "solved" spam. No matter if you use WhatsApp, Discord, Telegram, Signal, SMS or anything, you will still end up getting spam.
The lack of authentication (largely) isn't the issue anymore.
And as a side note.. this is also why the whole IP block or IP address as well as domain names aren't a good thing to rely on and mostly seems to be used by ancient setups.
Another problem in my personal opinion is that spam and legit newsletter/marketing emails have become way too similar. Things like embedding images fetched from a server to "detect open rates" and faking URLs (displaying one, but actually going somewhere else) are hacks first used by spammers and phishers but now are used by each and every service. I think spam filtering on a large scale would be way easier.
I also don't think that LLMs are the big solution here. AI as in ANNs and other statistical methods has (somewhat successfully) used for decades already. If anything there will be kind of an arms race between filtering and spam passing through, because they are both extremely suited to training.
I think if that happens then this is a resource race and then it would make sense to just use Hashcash from 1997 at a large scale.
Spam to a large degree comes from "reputable" big providers like Gmail, Hotmail, etc. They are legit email accounts used for spamming. A lot come from slightly smaller institutions, like marketing email services (Mailchimp, etc.), but the time when spam emails were silly scripts with no properly set up email server behind them is largely over.
The majority of the ones I get come from compromised hosts, usually some site running WordPress with a vulnerable plugin, since PHP has a convenient mail() function. After that there are a lot of smaller free providers. Large providers are pretty sparse, since they fight automated registrations pretty hard. However, pretty much all of them have a Reply-To pointing to a gmail.
I just looked over my spambox, and shoutouts to whoever has compromised crefito2.gov.br.
I tend to agree, too. It seems a no-brainer in hindsight to be able to do things like "pre-allow a given party to send me e-mail" (perhaps via revokable capability URLs)?
I would love a good way to revoke a sender's ability to send me email. There's a looooot of companies in the world where "unsubscribe" means "unsubscribe for the legally required 1 year or whatever, and then start spamming you again."
Give each company a different email address and revoke when you no longer want their emails. Of course, this is easier if 1) you have your own domain, and 2) you run your own email server, or use an email service that allows unlimited email addresses per domain.
I was thinking along these lines: instead of typing your e-mail address, your browser fills the field with a capability URL. This points indirectly to your inbox - kind of like mail forwarder services do today. Now that I think of it, it seems like it's achievable with no new protocols (with something like Firefox Relay), but service providers still treat temp-mail like a definite scam signal instead of a legitimate use case for normal people. Had this been the default since the early days, we might be able to enjoy revokable addresses, thanks to them being single-use only.
For these instances can't you just filter by sender domain? Probably would be nice to have a good UI in email clients.
I'm signing up for a new service on-line, and it wants to send me an email. How do I pre-authorize it? How do I even know what address the email is coming from (a complaint I have now)? Why get the web involved at all? (I know, I know, it's the "All Singing, All Dancing Protocol For Everything From Now Until The End Of Time", much like FTP was on the very early Arpanet).
Or, I'm reading a blog post from Alice, and I want to send her an email about it. How can she "pre-allow" my email? She doesn't even know I want to send her one.
I don't use it (because DHH) but this is close to the idea of hey.com's email service. Literally nothing comes to your inbox unless you've allowed it. It means you end up with two inboxes -- in a way -- one to check in case there's something interesting in there, and the one where you know the emails that you want are in it. (Similar to GMail's priority inbox, but with less promise of magic).
I think this is a good model, perhaps even a better default model.
If I think of a real inbox -- I go and fetch the physical post, 80% is mass-mailed and goes straight into recycling and only letters from suppliers and government tend to end up in my actual in-tray. The idea that the postman could put stuff in your inbox is nuts!
So...a spam folder+graylisting approach but with more steps? A different UX too, of course. Which is important.
with properly applied learnings from all the decades that have been learned from conventional email.
What are those learnings? I'd say the most important weakness of email is the spam problem; authentication is "solved" with TLS (non-TLS mailservers notwithstanding) and I personally don't consider encryption a very important core feature. So, spam: how to "fix" that without making email closed?
Learnings would include lots of stuff that apply elsewhere in tech, and tend to be something along the lines of trusting too much of other systems and devices back in the day, and yet nowadays knowing why we should not trust any other system or device on a network (either local or otherwise), and build up from there, etc. Now, as to your point about not considering encryption a very important core feature, i disagree...but that does not make you nor I correct or incorrect; merely that we differ in desired features...which i actually wish was easier to decide to turn on or off by a user. Because i know that we do indeed have encryption via email, but its implementation is not shall we say easy, etc.
It is not just „old and goofy“. There is some purpose and thought behind it.
The conversational character of the SMTP protocol makes perfect sense. You will not send e-mail data if the recipient mailbox does not exist, or if the attachment is larger than given server accepts. Why send several megabytes and then realize that it is over the limit? Why send message in an unsupported encoding? Semantics of a single request-response does not fit everywhere. (or you can see SMTP as a sequence of several requests and responses coupled together in a session).
And those „randomly“ disappearing e-mails that can not even be found in the spam folder? When I used to be a postmaster, it was a matter of honor to deliver every message and not miss anything. The suspect spam was just flagged and it was up to the user configurable filters to put it in the appropriate folder. We were essentially peer to peer and each of us was interested in making sure it worked reliably. Today, it is different. There are several global corporations and they would prefer it if everyone else would just disappear. These corporations can not simply nuke smaller providers or suddenly stop accepting their messages, but can make them look unreliable in the eyes of users.
From a technical perspective, [SMTP submission] is essentially the same as sending an HTTPS request.
It is not, and treating it this way will get your emails dropped. Email is fundamentally a store-and-forward system, and the protocol requires that you be able to do exponential backoff for delivery. That requires a persistent queue, for which there is no built-in analogue in HTTP, and it also makes the operation fundamentally asynchronous. HTTP, by contrast, is full of synchronous use-cases; most endpoints in most RESTish APIs are synchronous.
At minimum, this will get your emails dropped if the destination server goes down e.g. for maintenance or an outage and relies on senders doing standards-compliant retries. But some providers will always drop your mail as spam, because of greylisting.
Because it was specifically designed that way as an anti-spam measure.
Can anyone really be so ignorant to not know this and yet purportedly be smart enough to need to investigate it?!