Modwest Blog

Escalating War on Spam

Herbal Vi@gra, no-prescription pills, bodily-enhancement miracles, mortgage scams, cable descramblers, non-existent lonely women fishing for a date with absolutely anyone, fake bank login reminders, designer watches and handbags and various junk you don't want. Sound familiar?

These past few weeks, we've noted a dramatic increase in spam delivered to customer and staff mailboxes on our shared hosting system. Our Support Team has heard from many customers recently who reasonably assumed our spam filtering technology was simply turned off. It was a problem, and it got noticed.

So, we began an incremental process of increasing the sensitivity of some of the tests which help determine whether an individual email is spam or not. Each test assigns points, and the total points represents a confidence level that the message is indeed spam. For more on how this works, see this post.

It's important to make changes gradually, because dramatic changes to the scoring system can result in "false positives", or legitimate email that gets flagged as spam, which we try to avoid for obvious reasons.

To measure the effects of our changes, we worked this month to track statistics on the amount of mail being handled, the percentage of it which was flagged as spam, and the level of "spamminess" of messages delivered. The overall dataset we examined was over four million deliveries from July 2009.

Spam Surge

The first thing we've learned is that while the amount of low-scoring spam certainly increased, it wasn't just a failure of the filtering software. There was a generalized surge in overall messages received -- probably due to the avalanche of new spam! Here's a visualization of this phenomenon that Erik made for us today:

There's one bar per day in July, and taller bars represent a higher volume of incoming mail. The darker inner bar represents the proportion of overall mail that day which was flagged as spam. July 31st is shorter because we pulled the stats about halfway through the day, but you can see that there was as much spam detected in the first half of the day than all mail combined from some days earlier in the month.

You can also see that the overall volume of mail increased dramatically, and the amount flagged as spam has increased near the end of the month.

How Spammy?

We wanted to not only know that more mail was being flagged as spam, we also wanted to know just how "spammy" it was. Here are two graphs that present that data.

First, a daily graph of average spam scores. For reference, a score of 8 or above is almost certainly spam.

The humps earlier in the month are weekends -- days on which there's a lot less legitimate email, so spam constitutes a higher ratio of deliveries.

Next, here's a representation of the distribution of spam scores at several points in the month. The first pie chart represents the 12th through 17th of July. The second represents the 26th through 31st of July, and the final pie chart is just for July 31st. We made our most recent filtering improvements on July 30th.

You can see that a few weeks ago, spam scoring 8 points or higher constituted about one third of all messages. Today, that proportion is roughly two thirds. That means that a lot of previously low-scoring spam is now being scored higher.

How You Can Help

Spam apparently isn't going away; in fact, it is increasing. We believe that mailbox owners should have as much control as feasible over deliveries to their mailbox, and that's why we built the OnSite Spam Protection tool.

In our previous review of spam stats, we came to the realization that a lot of our customers have the default "Moderate" setting for spam filtering enabled. Turning that up to "High" or even "Very High" is a safe bet for most people. This and other strategies are discussed in our spam FAQ.

We think the changes we made this month have improved the spam situation quite a bit. Do you agree? Please, let us know here or here.

-JM

First Fruits of the Modwest Community

A couple weeks ago we launched the Modwest Feedback Community.  Already, we have heard important ideas form customers and implemented changes in response!

Here's a few examples:

• radiohead correctly pointed out that while Subversion (svn) is available in all Modwest shared hosting accounts, we hadn't done a very good job of letting people know. We quickly made a few changes so that it shows up on the big features page as well as the main webhosting page under Site Management. Thanks radiohead!
• In a comment thread, squiggle advised that our System Status page should continue displaying maintenance and issue alerts for more than just a few days, to aid our clients in debugging any issues they may be tackling. We took care of that a few days later, and now the last 30 days of notices display. 
• In an OnSite feature request topic, bryanb suggested we provide a more direct way for site owners to get into webmail. Turns out that already existed but we never told anyone! There are two options:

http://webmail.modwest.com
https://beta-webmail.modwest.com

As an example of what's being discussed right now, here's a recently popular topic (click it for details!):

So, we'd say our "kick off" was a great success. If you haven't visited recently, check it out, and participate for a chance to win real prizes like this one:

-JM

Email Meltdown: Lessons Learned

One of our guiding principles is to continuously improve our systems by never getting bitten twice by the same problem. When disaster strikes, we review what happened and see what changes we can make to prevent similar headaches in the future. We don't always find a fixable “tragic flaw,” but the review  is still valuable.

Modwest hosts tens of thousands of mailboxes on our shared hosting mail cluster. We do this by spreading mail services out over numerous load-balanced servers. Some are devoted to processing messages on arrival--ascertaining whether the destination address is valid and then scanning the messages for viruses and spam content. Others are devoted to storage of the messages themselves. The first group (the “front end”) is like a strainer which does its best to ensure only legitimate messages get through to the second group, the “back end.”

Due to a series of headache-inducing events, it was the group of back-end storage servers that we had trouble with earlier this month. For the gory details, read on. Or skip to the lessons learned.

It all started with a report from one customer – they had a subfolder which was mysteriously inaccessible via webmail.  After some initial debugging on the appropriate back-end storage server, we concluded that the mailbox was corrupt.

Each mailbox can be thought of like a book and its index. Each message is a page in the book, and is referenced in the index. Every entry in the index has a corresponding page. However, if the index and the pages don't agree for some reason, a system administrator needs to “reconstruct” the mailbox to eliminate this corruption. This is very infrequent, but happens from time to time.

This reconstruction process usually takes a few seconds up to a few minutes for mailboxes that contain lots of messages and folders. On this mailbox, the process seemed to be still running after close to 24 hours. Actually, it was stalled, unable to begin work on the problematic folder.

Some additional research indicated the problem was not with the mailbox, but the underlying filesystem – the software glue between the mail server software and the physical hard drives. The only fix for that problem is an hours-long offline repair process. So, we began preparations for a period of downtime.

The next day, our patient took a turn for the worse. Right in the middle of the business day, the server's CPU utilization skyrocketed and performance plummeted. We did everything we could to mitigate but finally decided that performance was so bad, with webmail taking minutes to load every page, that we weren't really providing a “service” anyway, so we better go ahead and begin the offline filesystem repairs immediately. It took nine hours to complete.

And it didn't help.

So our next tactic was to move mailboxes off the problematic server as quickly as possible. Luckily, we had a brand new machine racked up and ready for exactly this purpose, and after some last-minute software upgrades and configuration changes, it was ready to help. We moved some 4,000 mailboxes late that night.

And that helped a lot! What a relief, for everyone.

Except that the upgrades required to integrate the new server left our oldest mailbox storage server out in the cold. Its older software was unable to communicate properly with all the new stuff, and so IMAP mailboxes, including webmail, were unavailable. Thousands of mailbox owners were denied access to their mailboxes unless they added some special settings or used an alternate webmail address we set up.

Our first plan was to move all the mailboxes on the older server to the brand new one. In theory, this could be done transparently and without interruption. But after exhaustive research on the plan, we concluded it just wouldn't work. An upgrade with data-in-place was the only way forward.

Meanwhile, since we're preparing for an office move (more on this in a future post), we are in the process of organizing and relocating various equipment. Late one night, we moved a backup server without changing its power settings. The resulting voltage mismatch caused a power fluctuation which resulted in several servers rebooting – including the brand new mail storage server. So, the thousands of mailboxes we'd just moved off the problem server were unavailable again while we ran the required filesystem repairs. This coincidence was just plain bad luck.

Near the end of last week, we scheduled and completed the final upgrade required to get the older storage server up to speed with the rest. The mail system has been functioning fine every since.

 

Lessons learned:

On the server-side, we haven't yet found any “silver bullets” to truly prevent this from happening again. There are no good ways for us to provide meaningful additional redundancy which would somehow host mailboxes in more than one place at once, and we still don't know exactly what caused the initial problem.

What we can do is communicate better about issues like this.

• When we're having server problems, our status page needs to be front and center. Our support team received hundreds of calls during the various problems, and many people simply didn't know the status page existed. We're working on ways to remedy that.

• The status page needs to more clearly explain the scope of the issues and their resolution. Despite thousands of mailboxes being non-functional for hours, the page claimed that POP3 service was “100% OK” throughout.  This is because the statistics are automatically generated based on a simple check of initial email connectivity – not whether any mailboxes are actually functioning. So, we need to explain this better, and/or implement more comprehensive checks.

As much as I'd like to, we can't prevent servers from breaking. But when problems do occur, we can and will do a better job of communicating what happened, who's affected, and what we're doing about it. And as always, we're happy to hear your feedback.

-JM

Sorry about the email

Update Aug 13th, 8:30PM: We've got the server back up and running and it's working hard to catch up with the backlog of messages. Check the status page for updates.

Current status:

A significant number of our customers (and all Modwest staff) with email boxes on the Modwest shared system have been without access to their mailboxes since about 11AM this morning. That's about when server alarms went off and we concluded it was necessary take one of our mailbox storage servers offline for diagnostics.

At the moment, we're running a filesystem repair utility which is examining every message and every mailbox to ensure data integrity; this process needs to complete before we can bring the server back online. Unfortunately it's impossible to ascertain how long this is going to take, but we'll keep the Modwest Status page updated as we know more.

As an educated guess, a repair process like this could take another 12 hours, though there are some indications it'll complete sooner. When it does, pending mail will be delivered. There's no indication that anything will be lost.

Some History:

Earlier this week we were alerted to a customer mailbox which had one folder which was inaccessible. Our standard mailbox diagnostic utilities couldn't even complete their analysis, which led us to conclude there was a filesystem problem.

The "filesystem" is the low level software that connects mailboxes to the hard drives in the server. The only way to fix a filesystem problem like this is to take the server offline and run repair utilities, and so we'd planned to conduct this maintenance this coming weekend. Then the alarms went off this morning, and so we had to perform the surgery right away.

The future:

As an immediate right now workaround, it's possible to create a new mailbox in OnSite and direct any new (but not pending) mail there. This has a downside: your mail will be split between your old mailbox (on the storage server undergoing maintenance) and the new mailbox (on an unaffected storage server). Because of that hassle and complexity (and the fact that we intend to have this issue resolved later tonight), we don't recommend this solution, but present it as option for those who want to do it. The only trick is that you need to log in as your administrative user and use the Add Mailbox page, not the Mailboxes & Addresses link.

We have spoken to some customers today who say they intend to move their mail elsewhere, which we can understand under the circumstances. Going with a specialist email provider might meet your needs better. These are not recommendations, but here are some options we've heard of:

• http://www.google.com/a/help/intl/en/admins/editions.html
• http://www.postini.com/
• http://www.messagelabs.com
• http://www.mailtrust.com/
• http://www.mxlogic.com
• http://www.gmsi1.com/index.html

In the meantime, we're working to restore service to the affected mailboxes as soon as possible. For additional updates, please check the Modwest Status page.

Sorry about the trouble.

-JM

Email Delays Resolved

Mail on the Modwest shared hosting system is flowing again now. Nothing was lost, but throughout the day yesterday (Friday), incoming messages could have been delayed several hours, webmail didn't work for most customers, and about half the mailboxes we host were totally unavailable for several hours last night.  A blow-by-blow is available on our offsite status page.

At the high-water mark for yesterday's problems, close to 200,000 messages were hung up in the queue, awaiting delivery to customer mailboxes. That included messages coming in to our support team, so it wasn't until late last night that a bazillion support request emails came in. Needless to say, we're a little behind.

The problem, as best as we could tell, was some sort of deadlock issue with our Cyrus IMAP software. The resulting behavior was all the 'stuff the message in a mailbox' processes believed they needed to wait for access to do so. All of them. I can sort of imagine them all politely saying "no please, I insist, you first" to each other, with no deliveries happening at all.

Anyway, the situation is resolved now; not necessarily in a permanent way, but everything is working at the moment and we're monitoring servers closely. Sorry about the trouble.

-JM

End of Forwarding to Comcast

Spam, spam, spam. The never-ending battle continues.

Some may remember about how we stopped allowing email forwarding to AOL last year. Spam that originates elsewhere, passes through Modwest, arrives at AOL, and is then reported to AOL by the recipient as being spam generates a complaint against Modwest as the source of the spam -- even though we were merely a conduit, forwarding mail as per our customer's preferences.

We now know that the same situation affects email auto-forwarded to Comcast, which has a similar policy. In the past month, we've asked Comcast to please resume accepting mail from Modwest customers at least a dozen times. They're usually responsive, but it's becoming an almost daily occurrence. At the moment I write this, nearly 1,000 messages are stuck here at Modwest because Comcast refuses to accept them. We hope to push them through to their final destinations at Comcast by the end of the day, but we need Comcast's cooperation to make that happen.

Therefore, because Comcast's anti-spam policies are regularly causing significant email delays (hours or days) for any Modwest customer who attempts to communicate with any Comcast customer, we cannot allow automatic forwarding to Comcast any longer. We'll soon be getting in touch with customers who rely on Comcast forwarding about their options.

(Incidentally, Yahoo is also blocking most Modwest customer mail at the moment, and we're working on clearing things up with them. UPDATE Feb 24: Yahoo is still blocking us, and not responding to requests for information and resolution.

Update Feb 27: Yahoo has not been terribly communicative but has let the vast majority of deferred messages through, finally.)

We always try to avoid removing features that customers have been using, and we may rescind this policy if other options become available, but we'll be preventing auto-forwarding to Comcast addresses starting next week.

-JM

Email changes: Attachment Size, Spam Filtering

This past week we made a couple improvements on our email systems that we think will be be better for everyone.

First, we changed the maximum email attachment size to twenty megabytes. This is more in line with other large email providers and will help some of our graphic designer and photographer customers send and receive the large files associated with their trade. Remote recipients might not be able to receive 20MB attachments sent through our system (since their email provider might have a different policy), but incoming attachments should come through just fine as long as they can make it through our dangerous content filters.

Second, all newly created mailboxes will have their spam filter turned on, and anything detected as spam will be placed in the Spam/ subfolder of the mailbox. Note that messages in the Spam/ and Trash/ folders are auto-deleted after a period of time.

We're committed to offering maximum control over your incoming messages and had to think hard about enabling the filter by default. But two things gave us confidence: a relatively commonly expressed concern from new customers that since transferring to Modwest, they seem to receive more spam, and the (directly related) discovery that the majority of our customers had not chosen to enable spam filtering at all. So this change should help with that phenomenon. Existing mailboxes are not affected by this change.

Questions? Comment below, or contact our support team.

-JM

Maximizing Spam Filter Effectiveness

As we discussed earlier this year, spam filtering is a topic we think a lot about and a service we are always trying to improve. Meanwhile, the spammers are trying to figure out how to circumvent anti-spam measures, and so the spam 'arms race' is continuously evolving.

We recently ran some numbers to evaluate how we're doing with regards to keeping spam out of customer mailboxes. Using a dataset of around 650,000 recent email deliveries on our shared system, we compiled statistics on the effectiveness of our filter.  Our conclusion in a nutshell: the majority of our customers could easily reduce the amount of spam they receive by 50% or more with just a few clicks.

I'll explain. First though, a bit about how spam filtering works at Modwest.

How Spam Filtering Works

In addition to a number of 'front line' defenses, we use SpamAssassin,the world's #1 open source spam filtering system. The way it works is to compare each incoming message to hundreds of rules, assigning "points" for any matched rules to compute a total score. You can think about that score as a confidence level that a message is unwanted spam.

Some (but not all) the rules concern the actual text of the message. For example, a message that includes the all-caps phrase "FREE SHIPPING" might receive one point, but if it also contains "HERBAL PHARMACEUTICALS" and "NO PRESCRIPTION REQUIRED!!!", it could accumulate several more points for a higher total spam confidence score.  In OnSite, the "Normal" level of filtering corresponds with a score of 8 or more, "High" with 5 or more, and "Very High" with 2 or more. Really egregious and awful spam can score 15+ points.

You can see the spam-confidence score each message receives by checking the "X-Spam-Status" line in the full headers of any message in your Modwest-hosted mailbox. (The Mnenhy add-on to Thunderbird makes this even easier.)

Our philosophy is that you should have as much control over your mailbox as possible, and so each incoming message's spam score is compared to the mailbox owner's preferences and handled accordingly.  If you haven't set any preferences, then the server will tag messages based on the "Normal" level of filtering but deliver everything to your Inbox even if it was detected as spam.

Adoption Rate

Of course the spam filter is really only effective if mailbox owners choose to set it up according to their preferences. And here's where we see an opportunity (hence this blog post).

Of the thousands of mailboxes hosted by Modwest, only about 25% of them have spam-handling preferences set up. That means for the other 75%, every piece of incoming junk mail is delivered directly to the Inbox.

Effectiveness

The good news is that we're detecting a tremendous quantity of spam.  In our 650,000 message dataset, about half were detected as spam, but remember that at least three-quarters of our customer mailboxes only have the default filtering level of 8, which is relatively tolerant. The following chart (click to enlarge) illustrates spam scores in the data we analyzed:

With around half of all messages scoring 8 points or higher, and only a quarter of our customers actively filtering spam on the server, that represents a huge opportunity to cut down on spam deliveries.

Another interesting fact: 40% of messages earned a whopping 12 or more points. At that level of confidence, there's very little chance a message is legit (unless you really are in the "herbal viagra" business I suppose). With this in mind, a few of us in the office have been experimenting with multiple levels of spam preferences.

For example, if a message gets 12+ points, it's surely safe to delete it automatically, sight unseen. But something in the 2-5 range warrants a second look, since I sure don't want to miss any important business email. I'm using just such a system and it's working great for me. If it continues to work well, we'll consider building this functionality into OnSite. 

How you can reduce the amount of spam you receive:

If you're hosting a mailbox on the Modwest shared system, you need to set up the spam filter according to your preferences. Using the spam protection tool in OnSite, make sure you've turned the filter on, as well as chosen a Spam Handling option (subject-tag, filter into a folder, or delete). Monitor the results carefully, and lower the threshold as low as you're comfortable. You can add important senders to your 'Always Allow' list, one per line, to ensure that your business contacts and family members are exempt from the spam filter, too. This and other tips are available in the FAQ.

We'll continue to monitor the effectiveness of our spam filtering services in order to provide the best possible experience to our customers. We'd love to hear your feedback on the topic.

-JM

End of auto-forwarding to AOL

We monitor just about everything at Modwest, and one of the types of alerts we occasionally receive is that one or more of the servers in our mail cluster has a backlog of messages awaiting delivery to remote destinations. That happened this week, and upon investigation we discovered just the latest occurrence of a phenomenon with which we're well acquainted by now: due to spam complaints, AOL was blocking all email from all Modwest customers hosted on our shared system.

How could this happen? Surely Modwest has strong anti-spam policies! (Yes, we do.)

Well, there is a small loophole: We allow our customers to set up mail forwarding, which means you can have mail addressed to yourdomain.com (hosted by Modwest) automatically forwarded to another email address elsewhere (such as AOL).  Sometimes, messages that get forwarded to a customer's AOL account are then reported via the "This is Spam" button at AOL; this results in Modwest being identified as the source of that message. Enough complaints, and AOL blocks Modwest altogether.

Therefore, beginning July 27th, 2007, we are going to reconfigure all forwarding rules destined for AOL addresses to deposit received messages in the main account mailbox. Further, we will be disallowing all new forwarding rules with an AOL address as the destination. We'll notify Modwest customers who currently have AOL forwarding rules soon.

We hate to make any changes that restrict features, but due to AOL's policies, continuing to allow these types of forwarding rules can adversely affect every customer of ours.  If you have any questions, don't hesitate to ask.

-JM

GoDaddy? Helo? Ehlo?

Earlier this week we learned that GoDaddy was blocking legitimate mail from Modwest customers to GoDaddy customers. A quick call to their HQ revealed that they were blocking one of our mail servers because of a "bogus helo" problem, which, in their words, generally indicated the presence of a spam or virus outbreak. Once we eliminated this possibility and ensured our strict spam policies were being followed, I asked Mike, our senior sysadmin and Operations Manager, for his analysis of the GoDaddy's policy. Here's what he had to say:

Why is the policy of checking SMTP/ESMTP HELO/EHLO bad?  RFC2821 (which supersedes RFC821 as the SMTP/ESMTP standard) Section 4.1.1.1 describes the SMTP HELO and ESMTP EHLO Command, response, and requirements.  An EHLO or HELO is a MUST for SMTP clients.  RFC2821 Sec. 4.1.1.1 states exactly this "The argument field contains the fully-qualified domain name of the SMTP client if one is available."  There are many cases where one is not available.  Or where an accurate one is not available.

Now, strictly speaking, if one can't determine a valid domain you're supposed to use what RFC2821 calls an "address literal".  This amounts to your IPv4 or IPv6 address enclosed in [].  All of this is problematic.  In the case of using a domain name, what is the "correct" domain name?  Well RFC2821 makes that very clear.  Section 3.6 addresses what a domain name is in this case.  Specifically it states:

   * The domain name given in the EHLO command MUST BE either a primary host name (a domain name that resolves to an A RR) or, if the host has no name, an address literal as described in section 4.1.1.1.
   * The reserved mailbox name "postmaster" may be used in a RCPT command without domain qualification (see section 4.1.1.3) and MUST be accepted if so used.

So the second bit isn't applicable in this case.  We're talking about a domain, not an address, but the second first part is.

In our case we have a large mail cluster whose primary host name is mail.modwest.com, which has an associated A record.  The RFC does NOT make any requirement as to that A record matching the IPv4 or IPv6 address we're connecting to the remote host from.  But when presented with this as a HELO/EHLO domain, GoDaddy refuses the message with a hard failure.  So our case of using the HELO/EHLO is clearly supported by the RFC.  GoDaddy refusing to accept the mail at that point, claiming a violation of HELO/EHLO being refused is unsupported by the governing standard in the case that our server says "EHLO mail.modwest.com".

Now, it is possible that GoDaddy is rejecting mail because of suspected virus load coming from a particular machine, but again, it should not be stating the problem to be an invalid HELO/EHLO if this is the case.  If you can't accurately identify to the sender the reason why you're rejecting a message, at the time you reject it, you should not reject it.

So why might the domain name not be available for HELO/EHLO?  It may be a case of DHCP environments, where you don't know what your rDNS should be. It may be the case of a large cluster of machines that appear as the same machine.  It may be the case of a single machine with multiple interfaces. (Only the kernel initially knows the interface and therefore IP that was used for an outbound connection, unless you explicitly force them to come from a particular IP, but that can have other nasty interactions with stateful firewalls and reverse path filters. )

So there you have it. GoDaddy, we'd love to work together with you to reduce spam and malware. But can you let our customers' legitimate mail through to your customers now? We're pretty sure both sides would appreciate it.

UPDATE 4/20: GoDaddy says they've unblocked the mail server in question, and it seems to be the case.

-JM