Maximizing Spam Filter Effectiveness

As we discussed earlier this year, spam filtering is a topic we think a lot about and a service we are always trying to improve. Meanwhile, the spammers are trying to figure out how to circumvent anti-spam measures, and so the spam 'arms race' is continuously evolving.

We recently ran some numbers to evaluate how we're doing with regards to keeping spam out of customer mailboxes. Using a dataset of around 650,000 recent email deliveries on our shared system, we compiled statistics on the effectiveness of our filter.  Our conclusion in a nutshell: the majority of our customers could easily reduce the amount of spam they receive by 50% or more with just a few clicks.

I'll explain. First though, a bit about how spam filtering works at Modwest.

How Spam Filtering Works

In addition to a number of 'front line' defenses, we use SpamAssassin,the world's #1 open source spam filtering system. The way it works is to compare each incoming message to hundreds of rules, assigning "points" for any matched rules to compute a total score. You can think about that score as a confidence level that a message is unwanted spam.

Some (but not all) the rules concern the actual text of the message. For example, a message that includes the all-caps phrase "FREE SHIPPING" might receive one point, but if it also contains "HERBAL PHARMACEUTICALS" and "NO PRESCRIPTION REQUIRED!!!", it could accumulate several more points for a higher total spam confidence score.  In OnSite, the "Normal" level of filtering corresponds with a score of 8 or more, "High" with 5 or more, and "Very High" with 2 or more. Really egregious and awful spam can score 15+ points.

You can see the spam-confidence score each message receives by checking the "X-Spam-Status" line in the full headers of any message in your Modwest-hosted mailbox. (The Mnenhy add-on to Thunderbird makes this even easier.)

Our philosophy is that you should have as much control over your mailbox as possible, and so each incoming message's spam score is compared to the mailbox owner's preferences and handled accordingly.  If you haven't set any preferences, then the server will tag messages based on the "Normal" level of filtering but deliver everything to your Inbox even if it was detected as spam.

Adoption Rate

Of course the spam filter is really only effective if mailbox owners choose to set it up according to their preferences. And here's where we see an opportunity (hence this blog post).

Of the thousands of mailboxes hosted by Modwest, only about 25% of them have spam-handling preferences set up. That means for the other 75%, every piece of incoming junk mail is delivered directly to the Inbox.

Effectiveness

The good news is that we're detecting a tremendous quantity of spam.  In our 650,000 message dataset, about half were detected as spam, but remember that at least three-quarters of our customer mailboxes only have the default filtering level of 8, which is relatively tolerant. The following chart (click to enlarge) illustrates spam scores in the data we analyzed:

With around half of all messages scoring 8 points or higher, and only a quarter of our customers actively filtering spam on the server, that represents a huge opportunity to cut down on spam deliveries.

Another interesting fact: 40% of messages earned a whopping 12 or more points. At that level of confidence, there's very little chance a message is legit (unless you really are in the "herbal viagra" business I suppose). With this in mind, a few of us in the office have been experimenting with multiple levels of spam preferences.

For example, if a message gets 12+ points, it's surely safe to delete it automatically, sight unseen. But something in the 2-5 range warrants a second look, since I sure don't want to miss any important business email. I'm using just such a system and it's working great for me. If it continues to work well, we'll consider building this functionality into OnSite. 

How you can reduce the amount of spam you receive:

If you're hosting a mailbox on the Modwest shared system, you need to set up the spam filter according to your preferences. Using the spam protection tool in OnSite, make sure you've turned the filter on, as well as chosen a Spam Handling option (subject-tag, filter into a folder, or delete). Monitor the results carefully, and lower the threshold as low as you're comfortable. You can add important senders to your 'Always Allow' list, one per line, to ensure that your business contacts and family members are exempt from the spam filter, too. This and other tips are available in the FAQ.

We'll continue to monitor the effectiveness of our spam filtering services in order to provide the best possible experience to our customers. We'd love to hear your feedback on the topic.

-JM

Comments

[Daniel Weltch]

I don't like ads HERBAL PHARMACEUTICALS" and "NO PRESCRIPTION REQUIRED!!!"
But everyday received mass mails

[partnership for prescription assistance]

hey,
well written.
i hate when my mail box is full of spam it gets on my every nerve when 70% of emails i get are spam i think im gonna try the mnehy add on =\

thanks.
cheers.
chrissy.