Good intentions, unexpected results: Mailing lists and DMARC

If there’s anything that software people know, it’s that changing one part of a program can result in a change in a seemingly unrelated part of the program.  That’s why automated testing is so powerful; it can show you when you have made a mistake that you not only didn’t intend, but that you didn’t expect.

If unexpected results can happen in a system that you control and supposedly understand, it’s not hard to imagine what happens when the results of your changes involve many pieces of software other than yours, running on computers other than yours, being used by customers who aren’t yours.

This would appear to be the situation with one of the latest anti-spam and security features for e-mail, known as DMARC.

I’m not intimately familiar with this standard, but I’ve seen other standards relating to e-mail in the past to know that anything having to do with e-mail will be frustrating for some of the people involved.  E-mail is in use by so many people, on so many computers, and by so many different programs, that you can’t possibly make changes without someone getting upset.  Nevertheless, the DMARC implementation and rollout by a number of large e-mail providers over the last few weeks has been causing trouble.

Let me explain: DMARC promises, to some degree, to reduce the amount of spam that we get by verifying that the sender’s e-mail address (in the “From” field) matches the server from which the e-mail was sent.  So if you get e-mail from me, with a “From” address of “reuven@lerner.co.il”, DMARC will verify that the e-mail was really sent from the lerner.co.il server.  To anyone who has received spam, or fake messages, or illegal “phishing” messages, this sounds like a great thing: No longer will you get messages from your friend with a hotmail.com address, asking for money now that they’re stranded in London.  It really, admirably aims to reduce the number of such messages.

How? Very simply, by checking that the “From” address in the message matches the server from which the message was sent.  If your DMARC-compliant server receives e-mail from “reuven@lerner.co.il”, but the server was some anonymous IP address in Mongolia, your server will refuse to receive the e-mail message.

So far, so good.  But of course, for every rule, there are exceptions.  Consider, for example, e-mail lists: When someone posts to a list, the “From” address is preserved, so that the message appears to be coming from the sender.  But in fact, the message isn’t coming from the sender.  Rather, it’s coming from the e-mail program running on a server.

For example, if I (reuven@lerner.co.il) send e-mail to a mailing list (list@example.com), the e-mail will really be coming from the example.com server.  But it’ll have a “From” address of reuven@lerner.co.il.  So now, if a receiver is using DMARC, they’ll see the discrepancy, and refuse to receive the e-mail message.

If lerner.co.il is using DMARC in the strictest way possible, then reuven@lerner.co.il sending to list@example.com will have especially unpleasant consequences: lerner.co.il will refuse to receive its own subscriber’s message to the list, because DMARC will show it to be a fake.  These refusals will count as a “bounce” on the mailing list, meaning a message that failed to get to the recipient’s inbox.  Enough such bounces, and everyone at lerner.co.il will be unsubscribed.

Yes, this means that if your e-mail provider uses DMARC, and if you subscribe to an e-mail list, then posting to such a list may result (eventually) in every other user of your provider being unsubscribed from the list!

I’ve witnessed this myself over the last few weeks, as members of a large e-mail list I maintain for residents of my city have slowly but surely been unsubscribed.  Simply put, any time that a Hotmail, Yahoo, or AOL users posts to the list for Modi’in residents, all of these companies (and perhaps more) refuse the message.  This refusal increases the number of bounces attributed to the users, and eventually results in mass auto-subscriptions.

As if that weren’t bad enough (and yes, it’s pretty bad), people who have been passively reading (i.e., not participating) in the e-mail list for years are now getting cryptic messages from the list-management software, saying that they have been unsubscribed because of excessive bounces.  Most people have no idea what this means, which in turn leads to the list managers (such as me) having to explain intricate e-mail policy issues.

There are some solutions to this problem, of course.  But they’re all bad, so far as I can tell, and came without any serious warning or notification.  And when it comes to e-mail, you really don’t want to start rejecting message en masse without warning.  The potential solutions are:

1. Subscribers can receive the digest mode of the list, which is always “From” an address on the server.  If you get the digest, this problem won’t happen to you.  If you are a mailing-list subscriber, rather than a list administrator, this is really the only recourse that you have.
2. The list managers can change the list such that instead of each message being “From” the individual, it’ll come from the list’s address.  I know that there are some people who say that this is the right behavior for e-mail lists, but I have long subscribed (so to speak) to the school of thought that you don’t want to change the “From” address.  (For more on this subject, you can read “reply-to considered harmful” and its associated messages.)
3. Supposedly, Mailman (the list-management software that I use) now has some support for DMARC that might solve the problem.  But the more I learn about DMARC, the less I’m convinced that Mailman can do anything.

And by the way, it’s not just little guys like me who are suffering.  The IETF, which writes the standards that make the Internet work, recently discovered that their e-mail lists are failing, too.

E-mail lists are incredibly useful tools, used by many millions (and perhaps billions) of people around the world.  You really don’t want to mess with how they work unless there’s a very good reason to do so.  Yes, spam and fraud are big problems, and I welcome the chance to change them.  

But really, would it have been so hard to contact all of the list-management software makers (how many can there be?) and work out some sort of deal?  Or at least get the message out to those of us running lists that this is going to happen?  I have personally spent many hours now researching this problem, and trying to find a solution for my list subscribers, with little or no success.

This all brings me back to my original point: The intentions here were good, and DMARC sounds like a good idea overall.  But it is affecting, in a very negative way, a very large number of people who are now suddenly, and to their surprise, cut off from their friends, colleagues, workplaces, and organizations.  The fact that AOL and other e-mail providers are saying, “Well, you’ll just need to reconfigure your list software,” without considering whether we want to do this, or whether e-mail lists really need to change after more than two decades (!) of working in a certain way, is rather surprising to me.  I’m not sure if there’s any way back, but I certainly hope that this is the last time such a drastic, negative solution is foisted on the public in this way.