Modwest Blog

Escalating War on Spam

Herbal Vi@gra, no-prescription pills, bodily-enhancement miracles, mortgage scams, cable descramblers, non-existent lonely women fishing for a date with absolutely anyone, fake bank login reminders, designer watches and handbags and various junk you don't want. Sound familiar?

These past few weeks, we've noted a dramatic increase in spam delivered to customer and staff mailboxes on our shared hosting system. Our Support Team has heard from many customers recently who reasonably assumed our spam filtering technology was simply turned off. It was a problem, and it got noticed.

So, we began an incremental process of increasing the sensitivity of some of the tests which help determine whether an individual email is spam or not. Each test assigns points, and the total points represents a confidence level that the message is indeed spam. For more on how this works, see this post.

It's important to make changes gradually, because dramatic changes to the scoring system can result in "false positives", or legitimate email that gets flagged as spam, which we try to avoid for obvious reasons.

To measure the effects of our changes, we worked this month to track statistics on the amount of mail being handled, the percentage of it which was flagged as spam, and the level of "spamminess" of messages delivered. The overall dataset we examined was over four million deliveries from July 2009.

Spam Surge

The first thing we've learned is that while the amount of low-scoring spam certainly increased, it wasn't just a failure of the filtering software. There was a generalized surge in overall messages received -- probably due to the avalanche of new spam! Here's a visualization of this phenomenon that Erik made for us today:

There's one bar per day in July, and taller bars represent a higher volume of incoming mail. The darker inner bar represents the proportion of overall mail that day which was flagged as spam. July 31st is shorter because we pulled the stats about halfway through the day, but you can see that there was as much spam detected in the first half of the day than all mail combined from some days earlier in the month.

You can also see that the overall volume of mail increased dramatically, and the amount flagged as spam has increased near the end of the month.

How Spammy?

We wanted to not only know that more mail was being flagged as spam, we also wanted to know just how "spammy" it was. Here are two graphs that present that data.

First, a daily graph of average spam scores. For reference, a score of 8 or above is almost certainly spam.

The humps earlier in the month are weekends -- days on which there's a lot less legitimate email, so spam constitutes a higher ratio of deliveries.

Next, here's a representation of the distribution of spam scores at several points in the month. The first pie chart represents the 12th through 17th of July. The second represents the 26th through 31st of July, and the final pie chart is just for July 31st. We made our most recent filtering improvements on July 30th.

You can see that a few weeks ago, spam scoring 8 points or higher constituted about one third of all messages. Today, that proportion is roughly two thirds. That means that a lot of previously low-scoring spam is now being scored higher.

How You Can Help

Spam apparently isn't going away; in fact, it is increasing. We believe that mailbox owners should have as much control as feasible over deliveries to their mailbox, and that's why we built the OnSite Spam Protection tool.

In our previous review of spam stats, we came to the realization that a lot of our customers have the default "Moderate" setting for spam filtering enabled. Turning that up to "High" or even "Very High" is a safe bet for most people. This and other strategies are discussed in our spam FAQ.

We think the changes we made this month have improved the spam situation quite a bit. Do you agree? Please, let us know here or here.

-JM

A Day in the Modwest Life: Tom

"A Day in the Life" is our most requested blog topic currently, so here's another installment. Featured today is Systems Engineer Tom.

Here's the play-by-play:

Uptime: 16:33, Calories burned: ~4500, Calories consumed: Not enough, Emergencies: 0,  Commuting close calls: 3

6:47 - Alarm, in form of local child-of-the-80s radio station, goes off.  I press snooze 1 time, giving four more minutes of sleep.  The song playing was the single from U2's Pop album, Discothèque. I'm tired because my wife, N, and I stayed up too late last night watching 30 Rock episodes on Netflix. 

6:52 - Feed cat who purposely knocked food off platform.  Yes, master... 

6:53 - Move clothes from washer to dryer -- all of my running clothes in there, sitting overnight.  They are 12% musty, so I'm using a new dryer sheet to compensate.  Being cheap, I tend to use fabric softener sheets 3 or 4 go-arounds.  Not this time. 

6:54 - Check on sick kitten that we are fostering from the Humane Society -- she's cloistered in bathroom, with no explosive bodily expulsions today.  Yes! 

7:07 - Slicking in edgy hair fixative product, realizing hair has become too long overnight to be effectively chunky punk. 

7:11 - Go out and feed chickens.  The sun is up and they know I'm coming out at about this time, so they stack themselves next to the door and topple out when I open it.

7:14 - Attempt to wake up N.  No dice. 

7:16 - Get breakfast cereal.  I eat breakfast every morning before work, so I can be focused.  That's not true; it's like a fad diet, where the only allowed food is cereal.  My teeth will probably pay the price one day, but the double-pour-and-eat-action sure is convenient.  The stay-fresh container, or as Mitch Hedberg said, the fresher, is down to the dregs.  So, my meal is all bran, no raisin.  It turned into brown Jell-O when exposed to milk.  Sproing. 

7:17 - I sleep extremely heavily, so I now check phone for alerts I slept through.  If I do this before getting ready, it sucks me in and the whole day is thrown off.  There were only two load beeps for one of our VPS servers -- not uncommon when dozens of containers are performing maintenance at the same time.

7:22 - Attempt to wake up N.  Another swing and a miss. 

7:23 - Feed fish.

7:23 - Check if new MySQL backup solution worked last night, catch up on email. 

7:30 - Stall a bit because I don't want to be early for my run with Marianne, co-worker and 50% of Team Modwest Half-Marathon. 

7:35 - Actually wake up N!  Feed the dog. 

7:38 - Borrow N's Body Bug.  I've been meaning to see what the device thinks of my metabolism and the run today seemed like it would make me look bad-ass.  Unverifiable numbers! 

7:45 - Start biking downtown to meet Marianne for run.  My commutes are always by bike.  I've logged over 6000 miles while working at Modwest the past 6 years, according to the Way to Go Club.  My commuter for the whole time has been a 2003 Diamondback Topanga Comp.

7:50 - Avoid crazy drivers;  they're apparently confused about the road construction and are occupied telling their phones all about it. 

7:55 - Stow bike in bike locker leased by Modwest and run to meeting place.  Good on them for fostering alternative transportation by providing secure storage. 

8:05 - Shouldn't have dawdled, arrived late.  Starting run. 

9:30 - Completed run, walking it off.  Here's the route. I am good going up hills, Marianne's good at downhill.  Descents shred my knees.  This is my last longish run before the Missoula Half-Marathon. 

9:40 - Marianne bought me a Powerade Ion-4 Advanced Electrolyte System (Sour Melon).  While being unnaturally radiant green, it was not sour.

10:12 - Answer Ops chat question about automount maps.  We're using autofs in a clever and troublesome way so we can have multiple environments in parallel. 

10:15 - Install TV antenna Janiel picked up for the office -- we've had a TV in the rumpus room since we moved in nine months ago, and it doesn't get any channels.  Blasphemy!  Turns out we only can easily get PBS through all the concrete of the Wilma. 

10:45 - Back at desk for "queue cleanup day", the day the Ops team does digital janitorial work and slays a few odd jobs -- But, checking support first. 

10:55 - Repaired John's broken drop database attempt; he was testing some recent problems people had on db0. 

11:00 - Got some music going -- Muse, then Infected Mushroom. 

11:45 - Cleanup queue mostly handled.  Moving on to little queue inversion projects.

11:52 - Stomach growling, Powerade not cutting it.  Cheese'n'crackers time.  Sharp. 

12:15 - Cage flag created for system tickets.  The Ops team has two ticket queues, one for time-sensitive requests, and another for long-term projects.  These frequently need kung-fu in the data center.  This flag ought to alleviate the question, "Do you guys need anything in the cage?" 

12:30 - Working on wrangling our log/temp/etc file management utilities so they're doing their job with our storage changes.  Even though disk space is cheap, I don't believe in being sloppy, so I've made this my pet project for the day. 

1:13 - Music ran out.  Midnight Juggernauts and Kasabian.

2:30 - Finished adjusting and documenting mw_cleanusertmp for now, ran it and freed 20GB+ of space. 

3:00 - Finished tuning and documenting mw-compressoldlogs for now, ran it and freed almost 30GB of space.

3:15 - The day-in-the-life epic saga typing continues, in terrible form. 

3:20 - Back on log rotation.  If only we could just use logrotate. 

4:00 - Got error_log truncation working on a new fileserver.  Freed nearly 20GB of space 

4:21 - Found out N and I might not get to go fishing at nearby Georgetown Lake tomorrow because the ER where she works apparently thinks paying loads of overtime is a good idea. (We actually did get to go, and caught over ten Rainbow Trout between us.  It was all catch-and-release.) 

4:25 - Packing up to go home and check on le chat.  (N working a 10 hour shift, so kitty may be stressed without companionship.) 

5:00 - Kitten checked, fed, box cleaned.  Catching up on the day's email.  I try to ignore email for large portions of the day...for efficiency! 

5:15 - Making miva_shuffle script for support team.  This script does a song and dance to remedy some occasional file locking problems we encounter with Miva. 

6:00 - Jonathan helped me navigate our arcane postgresql backups for our beta webmail system.  Gah!  It was missing a directory, that is all. 

6:15 - Done with miva_shuffle.

6:22 - Wrapping up work, giving animals evening meals.  Headed back downtown to meet N for Downtown Tonight. 

6:50 - Talked to Marianne about POSIX file permissions -- a customer struggling with getting two users RW access on the same files.  Modwest HQ is right outside Caras Park, the hub of downtown summer events.  Lucky for me Modwest cares about my results, not my schedule, so it was easy to get some support work done as I came through to the park. 

7:00 - Filled John's Chair at the Downtown Tonight Modwest booth and talked with Heather.  He was evaluating the brews, she was certain she'd forgotten everything computers while on vacation.  Full-Grown Men was the band for the evening. 

7:10 - N's here! We're eating at the Thai Spicy booth, Me - Green Curry Chicken, N - Cashew Nut Curry Chicken 

7:23 - Ilsa's Lemonade for drinks. 

7:45 - Broccoli calzone, I can't remember the real name, for a chaser; I'm still hungry... 

8:00 - Bike loaded in car, going to see N's dad, self-deprecating fly-fisher extraordinaire.  (http://en.wiktionary.org/wiki/extraordinaire -- "Unlike most English adjectives, extraordinaire follows its subject."  I never noticed that was particularly unique.  Man, this writeup is informative AND entertaining.) 

8:30 - Got a fly fishing knot lesson -- the knot we learned is supposedly not in any of the books out there, it seems actually a passcode that grumbly guys with waders and wild hair use to identify the non-posers. 

9:00 - Three episodes of 30 Rock while doing house chores. 

11:20 - Bed.

Got other ideas for the Modwest Blog? Let us know in the Modwest Community.

-JM

Alternatives to FTP

We're not quite ready to say "R.I.P. FTP", but it's true that FTP has an inherent security flaw: your username and password are passed in clear text from your computer to the server. This means that  malicious software on your computer could trivially capture your FTP credentials the next time you update your site.

We were alerted recently when a customer reported that some files in their website had been modified to include a strange looking IFRAME tag, which apparently loaded malicious code from a server in Russia every time a visitor arrived at the page. This prompted Google to temporarily list the customer's site as "Dangerous" in search results.

A quick check of the logs indicated the customer's account had been logged into via FTP from many IP addresses around the world, some of which uploaded the various malicious changes to the site. Our customer was able to securely upload a clean backup, and changed of course their password immediately.

We're not sure how the bad guys got ahold of this customer's FTP credentials, but if this was a case of spyware on the customer's PC capturing his cleartext username and password during a site update, then using an alternative to FTP could have helped avoid this painful event.

Two of the simplest alternatives are SFTP and SCP. Both protocols encrypt the connection from beginning to end, so cleartext "sniffers" can't read the traffic. SFTP support is often built in to FTP clients such as FileZilla, Cyberduck, and the built-in client of Dreamweaver, but may not be enabled by default. Check the help files for your favorite file upload program to learn how to enable SFTP.

WinSCP provides SCP and SFTP support, and a familiar Windows Explorer style interface, too.

There are no important differences between SFTP and SCP (both encrypt authentication and file transfers), so use whichever one your program supports. If your file upload program only supports old-fashioned FTP, it is time to upgrade.

A Day in the Modwest Life: Jonathan

"A Day in the Life" is our most requested blog topic currently, so here's a first installment. Featured today is one of our Systems Administrators, Jonathan.

Here's the play-by-play:

00:00
Finally finished the migration of a Plesk-based domain for a Managed Server customer that has had me stumped for 2 days.  The Plesk Migration Manager is picky and opaque in many ways.  It faulted on one large domain (11GB) out of many, without any real indication of the problem.  Since this domain was migrating from a Plesk 8.2 instance to a Plesk 9.2 instance, the alternative migration strategies were somewhat convoluted.

00:30
Finished documentation of the migration adventure of that Plesk-based domain.
Checked the Modwest Support queue (we use Request Tracker for our ticketing system) for any late-night urgent requests before turning in for the night.

04:30
Z, the 2.5-week old newborn, is fussy and needs a diaper change.

05:30
Z is upset.  After a quick verification of diaper situation and other potential causes of this state, took her downstairs and we napped on the couch to give my wife C a much needed break.

07:30
Z and I are awake again.  Took her back upstairs for a diaper change and a reunion with C.

08:30
Working with one of our Customer Support Specialists (L1) on a shared SSL certificate issue for one of our shared hosting environment customers.

09:00
Checking hardware upgrade options for one of our Managed Server customers.  He wants more disk space and there is the option of adding more drives to his server, not just replacing the existing drives.  Adding drives is a lot less work on our part and gives the customer even more disk space, while utilizing all of the drive bays in his server.

09:20
Researching filesystem options for our Cyrus mail servers in our shared hosting environment.  ZFS is an option, but we'd only really want to do that with Solaris at this point and we've had recent bad experiences with our Solaris implementation.  So, really, that's not an option right now.  That leaves the two main contenders as ext3 and reiserfs.  Finding dialog and debate on the merits of each type going back years.  There's a lot of old information out there -- beware.
Our new Systems Administrator, who has expertise in storage systems, is working on some great new projects for our storage environment.  Interesting stuff in the works.

10:10
Helped L1 look at an unusual account information discrepancy for one of our customers, which turned into a handful of customers.  All with similar account IDs.  Seems to be LDAP-related, so maybe a typo in a script recently.  It doesn't seem to be affecting performance for these customers, but shows some odd information internally when we view their account/billing information.

10:15
Packing up to head to the office.  Departure delayed while watching over Z while C grabs a shower.  Mentally thanking Modwest (again) for allowing me flexibility in my work schedule.

11:15
Arrived at the Modwest offices.  It's about a 10 minute walk from home to the office; along the bike path, across the Clark Fork River via the Orange Street Bridge, and along the bike path again to the Modwest Offices in the historic Wilma Building. 

Along the way, thought about scheduling hardware and software maintenance later this month for one of our VPS servers.
Ah!  Forgot that today is monthly clean-up day.  It's the day that our Operations Team puts other projects aside to clean up the things that have been fluttering around begging for attention.  I'm focusing on some Managed Server clean-up tasks -- preparing for decommissioning of some hardware that former customers are no longer using.

13:00
Digging out of monthly task clean-up.
Need food.
Also need to stop by a specific local restaurant/bar to ask about catering next month.

13:15
Generated list of Managed Server hardware that needs to be decommissioned over in our data center a few blocks away.
Spent a good while looking for traces of disappearing email deliveries for a Managed Server customer.
Corresponded with L1 regarding a database restore for another Managed Server customer.
Still need food.

14:30
Of course, I needed to talk to the appropriate catering person who is only available in the morning.  Will try again on Friday or Monday.
Stopped by the data center and synchronized the data from a to-be-decomissioned Managed Server machine to an external drive.  We usually keep that data around for a month after a Managed Server customer leaves Modwest, because it is not infrequent that they come back to us a week or two later because they forgot something important.

15:00
Figured out the best options for the Managed Server disk upgrade that I was researching earlier in the day.  Sent a note to the customer detailing the different options and my recommendation.  This particular customer likes details.

15:30
Started decommissioning resources, IPs in particular, of a former Managed Server customer. 

16:00
Familiarized myself with older Postfix/Mailscanner implementation on an older Red Hat Managed Server.  Ended up throwing a brick at the problem and configured Mailscanner to not scan any mail from the domain of our customer's client.  Not an ideal solution, but since it was quite difficult to get detailed information from the remote client on the types and content of the emails that he was sending, we decided to just let it all through.

16:30
Working on a database restoration for a Managed Server customer.  Made a note to self to complete our internal documentation on how to correspond with customers with regards to restore requests, specifically with Managed Server customers.  Some customers want their live data to be just like it was a day or two ago.  Others want the recovered data placed in a specific location that they can view.  Either way, it saves time for the Operations Team if we know what the customer wants when the task to restore gets passed along to us.

17:15
Big Boss says that I have too much on my plate and to delegate a specific Wusage migration task to him.  Yay!

17:30
Still working on that database restore for the Managed Server customer.
- While reading more about Cyrus email server filesystem recommendations
- While discussing, with another Systems Administrator, upstream bandwidth providers (who we shall not call out by name)
- While checking on available cabin lodging at Glacier next weekend
- While... uh oh.  We blew out the keg in the Modwest Kegerator
Ought to head home soon -- where soon was 2 hours ago.

18:00
Finished the database restore for that Managed Server customer.
Helped our Systems Engineer track down a problem with PostgreSQL backups for an internal VPS container.

18:07
Headed out the door for home.

18:30
Got home to another Z-related hospital bill.  Sigh.  So,...
Took a short nap with Z on the couch.

19:30
Prepared for short grocery store trip to the Good Food Store and debating whether I should take Z solo or not.  Seems a bit risky having a 10 minute separation between a possibly frantic hungry baby and her food source.

21:00
Back from the store, which wasn't such a short trip.  We all ended up going, which was probably not the most efficient use of our time, but that's okay.

21:30
Yay!  Heard from a friend on Facebook that I haven't heard from since 1997.  I was wary of my late-night FB signup 6 months ago, but I admit it has been fun to catch up with 'certain' old friends.  She's actually doing what she thought she'd end up doing -- scientific illustrations.
Making dinner and listening to KCRW streamed from Santa Monica College in Santa Monica, CA.  KCRW is streaming at our house most of the day.

23:45
Sent out party announcement to Modwest staff.  C and I eloped last summer (a week before our 16th anniversary) and are planning a party for friends and family in August.
Changed Z's diaper.
Went to bed.

So as requested, there's 24 hours of diaper-changing and systems administration at Modwest. We have two more volunteers for this blog project (so far), so stay tuned and we'll post those soon.

Got other ideas for the Modwest Blog? Let us know in the Modwest Community.

-JM