Modwest Blog

2009 Thanksgiving Food Drive

Thanksgiving is next Thursday the 26th, and the Modwest Team will be taking the day off to spend with families and friends.

Last year, we ran a week-long food drive, and collected over 100 lbs of food for the Missoula Food Bank, an organization that distributes 250 tons of food every year to local families in need. We're going to try and beat that record this year.

If you would like to help out, please drop off your non-perishable food items at our office at 110 Caras Drive in Missoula; that's alongside Caras Park in the Wilma building downtown. We'd like to receive all donations before 5PM on the Tuesday the 24th so we can deliver them on Wednesday morning.

Here's an interactive map if you need help finding us.

View Larger Map


-JM

Migrate to a Faster Database Server in Minutes

Earlier this year, we deployed another MySQL database server (db2.modwest.com) for our shared hosting customers. It's a beast of a machine, and still has lots of horsepower to spare. In fact, compared to db.modwest.com, it's roughly 4x more powerful and currently doing a lot less work.

In addition to having lots of power to spare, db2 is also running MySQL 5, rather than MySQL 4.1. Version 4.1 will be "end of life" at the end of this year, meaning that MySQL will no longer provide bug or security fixes for it. MySQL 5 has some cool new features, as well, and so we encourage everyone with databases hosted on db.modwest.com to switch them over.

Most modern applications (such as Joomla, Wordpress, or Drupal) should be perfectly happy with MySQL 5; though if you have a custom application, you should probably ask your webmaster to read through MySQL's guide to upgrading from 4.1 to 5.

To make one step in the migration process easier, we've recently whipped up a web tool to copy a database from one server to another. It's here:

•  http://dbcopy.modwest.com

We hope you'll enjoy the increased performance of db2, and the increased features and security of MySQL 5, and let us know of any trouble along the way.

A Hint of Things to Come

Over the years we've been asked from time to time why we don't offer long-term discounts like other hosts.  The answer is we do, but it's not automated and not publicly advertised.

Recently, feedback in the Community (like this and this) has brought the topic back to top-of-mind, so for the past few months we've been working in earnest towards a robust automated prepayment system that offers lots of flexibility for customers and rewards you with discounts for prepaying for longer terms.

We've settled on some generalities, and now we're asking you for your feedback. Which prepayment option is most appealing to you?




-JM

September Downtime Explained

Last month, we had a couple glitches with the storage system of our shared hosting environment. A number of you asked what we were doing to prevent it from happening again, so here's an overview of what happened and what's next:

Our shared hosting system was built with reliability in mind, with many redundant elements. It's a load-balanced cluster of webservers, so that if any one webserver goes belly-up, we keep running without any interruption. The filesystem itself runs on dozens of hard drives, so losing a hard drive causes no significant headaches either.

There are parts of the shared system that aren't redundant in the way we'd like: most of all, the file storage servers. If a file server crashes, web pages and graphics cannot load. That's what happened in September.

When a server starts randomly crashing, despite no changes being made, it can mean it's suffering from a hardware problem that is difficult to diagnose. So, our first step will be to move everything to a different server as soon as possible. We've already deployed a new file server, and this initial migration will be done later this month.

But the real problem is our reliance on each file server, so we've been building a truly redundant storage solution since summer. In fact, a number of volunteers have had their websites hosted on this new system for a while already. We intend to migrate all customer sites to the new system by year's end.

Technical info: The new storage system (using OCFS, AOE, and DRBD) involves multiple storage servers, each of which stores all the data.That means if one has a serious hardware failure, then the other one can take over.

If you're interested in being a beta tester, knowing that that involves occasional interruptions while we test and reconfigure and reboot the new system, just let us know.

Minor OnSite Upgrade: Printable Invoices

We've just pushed live a small update to Modwest OnSite, our hosting control panel. This update includes two features requested in the Modwest Community, specifically: "What features do you want added to OnSite?"

• Printable Invoices: Previously, OnSite presented an aggregated view of invoices and payments, in the manner of bank statements which describe deposits and withdrawals. Many people found this confusing and requested a simpler, separated view of payments and invoices. The invoices aren't available in PDF format for now, but they are printable as standalone invoices.
• Blog Post Syndication: Recent blog posts, including this one, now appear under "News & Info" in the right-hand column.

Thanks to those of you who have participated in the community and made these suggestions. It really helps.

If you notice any issues, please use the 'Report bug" link at the bottom of every page within OnSite. And keep the suggestions coming!

-JM

A Day in the Modwest Life: Janiel

We had numerous requests earlier this summer for additions to our "Day in the Modwest Life" series, so here's one more. This a day in mid-July lived by Janiel, Modwest's Business Manager.

7:00 AM Visiting father in-law up and moving around so I’m up too. Make coffee & morning chit chat.

7:20AM Get in shower – water’s not very hot, hmm...

7:45 AM Go downstairs to check on water heater. Pilot light is out. Not good. Relight pilot and go up stairs for more chit chat – mother in-law is now up too. Wake up husband so he can chit chat and I can get ready for work.

8:20 AM Off to work. It’s about a 15 mile commute and traffic is light. The drive on Hwy 93 is lovely so when traffic isn’t a bear, and I enjoy the ride.

8:50 AM Park my car in the lot off Orange St. & Front and walk the river path to the office. It’s raining so the smell is absolutely wonderful. Very fresh & green, can smell the river, flowers, trees & grass. No better place to live!

8:56 AM Greet Marianne, and settle in to answer billing voice messages & emails. Emails & calls are pretty light considering I took a 3 day weekend.

9:32 AM On my way to get cup of coffee, I remember to remind a new employee that I need their health insurance enrollment forms.

10:00 AM We are doing a series of in-house workshops & training. Last week I along with Customer Care Specialists & Web Developer were assigned homework (due tomorrow) on PHP 101. Now’s a good time to start.

11:15 AM John lets us know he wants anti-spyware software installed on all Windows machines.

12:02 PM I install Spybot and check for any unwanted programs/infections on my machine. As usual Vista doesn’t play nice & makes the new software jump through extra hoops to do its thing. Ugh. Scan requires closing browser and I need multiple browsers open to help customers with support issues.

12:40 PM Business as usual, M & H back on phones after their Spybot work. Now we’re all hungry.

12:45 PM My kitchen cupboard is looking pretty bare – gotta go shopping this week and replenish my stock. Have little chicken salad & peaches. Fix big glass of water; have to remember to drink lots of water today.

1:35 PM Scans all done, machine is clean. Answer a few new emails & then back to my PHP lesson.

2:50 PM Stuck on a particular PHP variable. Searching internet to find out what the error I’m seeing is all about.

3:05 PM Missing an ending “;” in my code – doh – all good now. It’s not very pretty though and I’m wasting too much time trying to make it look nicer.

3:10 PM Still on the same variable. Can’t make it do what I want. Confer with M and she doesn’t know how either. Put this variable on hold and move on to the next section. Will come back to this later.

3:30 PM Mail’s here. Sort through mail and begin posting payments to accounts. Receive large UPS shipment – new VPS hardware.

3:45 PM Received a payment for a customer that I can’t find. Search under everything I can think of (no domain listed, or invoice number, or email address, or phone number, nothing comes up when I search by address either.) About to send check back, when I find them by cross referencing the city & amount. Turns out the street name is the same on the account & check but not the street number.

4:30 PM Mail is done, answer a new email and back to php lesson. Almost completed my lesson.

5:00 PM Completed all of the lesson except that one pesky variable. Tinker with it some more. Could not figure out how to make it do what I wanted. Just in case you are interested I wanted to use variables to calculate my age – 17 years, by using $dob, $actualage & $fantasyage. Cheat and use variable $fantasyage = 29.

Send my assignment to teacher (Tom).

5:24PM I’m done. Hope I drank enough water. Fill my water bottle with electrolyte supplement and head to car.

5:40PM Arrive at yoga. First class I’ve been able to attend in almost 2 weeks, because of entertaining visiting in-laws. In-laws are off to dinner with friends so here I am.

5:55 PM Enter studio to preheat. Hotter then Hades. Notice they got new humidifiers.

7:30 PM Exit studio. DID NOT DRINK ENOUGH WATER! Sit with fellow yogis and eat my popcicle. Drink 16 oz vitamin water.

7:45 PM Back to my car and the drive home. Drink remainder of my 16 oz water bottle.

8:15 PM Greet husband & in-laws, off to shower – water not very hot, hmm...

8:30 PM Check water heater – pilot light out again. Looks like water heater is on the fritz. Let husband know. He re-lights.

8:34 PM Mother in-law brought home chicken Caesar salad for me (lovely woman). Eat on the sofa while I visit.

10:00 PM In-laws are heading home tomorrow (going back to Arizona via the west coast.) We discuss what time and agree they will not leave before 7 AM. Father in-law tends to rise early (4 AM) so this was an excellent compromise. Say good night as they head off to sleep.

10:15 PM Join husband to finish watching ‘Tomb Raider’ and relax, just the two of us. First time I’ve gotten to sit in my favorite chair in almost 2 weeks. Feels good.

11:00 PM Off to bed. – Wonder if I will have hot water tomorrow morning?

Afterword: Janiel's husband identified the problem with the water heater and replaced the broken part the next day. All is well.

Got other ideas for the Modwest Blog?  Let us know in the Modwest Community.

-JM

Let There Be Wifi

Our office is located right on Caras Park, arguably the focal point of the Missoula community. There are community events there most days in the spring, summer, and fall, ranging from Out to Lunch and Downtown Tonight to Hempfest, Kidsfest, and Brewfest.

Since we're so close and sitting on a boatload of bandwidth, we thought it'd be good to share with the community, and so this summer, we deployed outdoor wireless devices which give pretty decent coverage throughout the park. Set your laptop or handheld to connect to the "Modwest_Public" network, and you should be surfing in no time.

Enjoy, Missoula!

Photo by Dan Leithauser for Missoula Hempfest.

Escalating War on Spam

Herbal Vi@gra, no-prescription pills, bodily-enhancement miracles, mortgage scams, cable descramblers, non-existent lonely women fishing for a date with absolutely anyone, fake bank login reminders, designer watches and handbags and various junk you don't want. Sound familiar?

These past few weeks, we've noted a dramatic increase in spam delivered to customer and staff mailboxes on our shared hosting system. Our Support Team has heard from many customers recently who reasonably assumed our spam filtering technology was simply turned off. It was a problem, and it got noticed.

So, we began an incremental process of increasing the sensitivity of some of the tests which help determine whether an individual email is spam or not. Each test assigns points, and the total points represents a confidence level that the message is indeed spam. For more on how this works, see this post.

It's important to make changes gradually, because dramatic changes to the scoring system can result in "false positives", or legitimate email that gets flagged as spam, which we try to avoid for obvious reasons.

To measure the effects of our changes, we worked this month to track statistics on the amount of mail being handled, the percentage of it which was flagged as spam, and the level of "spamminess" of messages delivered. The overall dataset we examined was over four million deliveries from July 2009.

Spam Surge

The first thing we've learned is that while the amount of low-scoring spam certainly increased, it wasn't just a failure of the filtering software. There was a generalized surge in overall messages received -- probably due to the avalanche of new spam! Here's a visualization of this phenomenon that Erik made for us today:

There's one bar per day in July, and taller bars represent a higher volume of incoming mail. The darker inner bar represents the proportion of overall mail that day which was flagged as spam. July 31st is shorter because we pulled the stats about halfway through the day, but you can see that there was as much spam detected in the first half of the day than all mail combined from some days earlier in the month.

You can also see that the overall volume of mail increased dramatically, and the amount flagged as spam has increased near the end of the month.

How Spammy?

We wanted to not only know that more mail was being flagged as spam, we also wanted to know just how "spammy" it was. Here are two graphs that present that data.

First, a daily graph of average spam scores. For reference, a score of 8 or above is almost certainly spam.

The humps earlier in the month are weekends -- days on which there's a lot less legitimate email, so spam constitutes a higher ratio of deliveries.

Next, here's a representation of the distribution of spam scores at several points in the month. The first pie chart represents the 12th through 17th of July. The second represents the 26th through 31st of July, and the final pie chart is just for July 31st. We made our most recent filtering improvements on July 30th.

You can see that a few weeks ago, spam scoring 8 points or higher constituted about one third of all messages. Today, that proportion is roughly two thirds. That means that a lot of previously low-scoring spam is now being scored higher.

How You Can Help

Spam apparently isn't going away; in fact, it is increasing. We believe that mailbox owners should have as much control as feasible over deliveries to their mailbox, and that's why we built the OnSite Spam Protection tool.

In our previous review of spam stats, we came to the realization that a lot of our customers have the default "Moderate" setting for spam filtering enabled. Turning that up to "High" or even "Very High" is a safe bet for most people. This and other strategies are discussed in our spam FAQ.

We think the changes we made this month have improved the spam situation quite a bit. Do you agree? Please, let us know here or here.

-JM

A Day in the Modwest Life: Tom

"A Day in the Life" is our most requested blog topic currently, so here's another installment. Featured today is Systems Engineer Tom.

Here's the play-by-play:

Uptime: 16:33, Calories burned: ~4500, Calories consumed: Not enough, Emergencies: 0,  Commuting close calls: 3

6:47 - Alarm, in form of local child-of-the-80s radio station, goes off.  I press snooze 1 time, giving four more minutes of sleep.  The song playing was the single from U2's Pop album, Discothèque. I'm tired because my wife, N, and I stayed up too late last night watching 30 Rock episodes on Netflix. 

6:52 - Feed cat who purposely knocked food off platform.  Yes, master... 

6:53 - Move clothes from washer to dryer -- all of my running clothes in there, sitting overnight.  They are 12% musty, so I'm using a new dryer sheet to compensate.  Being cheap, I tend to use fabric softener sheets 3 or 4 go-arounds.  Not this time. 

6:54 - Check on sick kitten that we are fostering from the Humane Society -- she's cloistered in bathroom, with no explosive bodily expulsions today.  Yes! 

7:07 - Slicking in edgy hair fixative product, realizing hair has become too long overnight to be effectively chunky punk. 

7:11 - Go out and feed chickens.  The sun is up and they know I'm coming out at about this time, so they stack themselves next to the door and topple out when I open it.

7:14 - Attempt to wake up N.  No dice. 

7:16 - Get breakfast cereal.  I eat breakfast every morning before work, so I can be focused.  That's not true; it's like a fad diet, where the only allowed food is cereal.  My teeth will probably pay the price one day, but the double-pour-and-eat-action sure is convenient.  The stay-fresh container, or as Mitch Hedberg said, the fresher, is down to the dregs.  So, my meal is all bran, no raisin.  It turned into brown Jell-O when exposed to milk.  Sproing. 

7:17 - I sleep extremely heavily, so I now check phone for alerts I slept through.  If I do this before getting ready, it sucks me in and the whole day is thrown off.  There were only two load beeps for one of our VPS servers -- not uncommon when dozens of containers are performing maintenance at the same time.

7:22 - Attempt to wake up N.  Another swing and a miss. 

7:23 - Feed fish.

7:23 - Check if new MySQL backup solution worked last night, catch up on email. 

7:30 - Stall a bit because I don't want to be early for my run with Marianne, co-worker and 50% of Team Modwest Half-Marathon. 

7:35 - Actually wake up N!  Feed the dog. 

7:38 - Borrow N's Body Bug.  I've been meaning to see what the device thinks of my metabolism and the run today seemed like it would make me look bad-ass.  Unverifiable numbers! 

7:45 - Start biking downtown to meet Marianne for run.  My commutes are always by bike.  I've logged over 6000 miles while working at Modwest the past 6 years, according to the Way to Go Club.  My commuter for the whole time has been a 2003 Diamondback Topanga Comp.

7:50 - Avoid crazy drivers;  they're apparently confused about the road construction and are occupied telling their phones all about it. 

7:55 - Stow bike in bike locker leased by Modwest and run to meeting place.  Good on them for fostering alternative transportation by providing secure storage. 

8:05 - Shouldn't have dawdled, arrived late.  Starting run. 

9:30 - Completed run, walking it off.  Here's the route. I am good going up hills, Marianne's good at downhill.  Descents shred my knees.  This is my last longish run before the Missoula Half-Marathon. 

9:40 - Marianne bought me a Powerade Ion-4 Advanced Electrolyte System (Sour Melon).  While being unnaturally radiant green, it was not sour.

10:12 - Answer Ops chat question about automount maps.  We're using autofs in a clever and troublesome way so we can have multiple environments in parallel. 

10:15 - Install TV antenna Janiel picked up for the office -- we've had a TV in the rumpus room since we moved in nine months ago, and it doesn't get any channels.  Blasphemy!  Turns out we only can easily get PBS through all the concrete of the Wilma. 

10:45 - Back at desk for "queue cleanup day", the day the Ops team does digital janitorial work and slays a few odd jobs -- But, checking support first. 

10:55 - Repaired John's broken drop database attempt; he was testing some recent problems people had on db0. 

11:00 - Got some music going -- Muse, then Infected Mushroom. 

11:45 - Cleanup queue mostly handled.  Moving on to little queue inversion projects.

11:52 - Stomach growling, Powerade not cutting it.  Cheese'n'crackers time.  Sharp. 

12:15 - Cage flag created for system tickets.  The Ops team has two ticket queues, one for time-sensitive requests, and another for long-term projects.  These frequently need kung-fu in the data center.  This flag ought to alleviate the question, "Do you guys need anything in the cage?" 

12:30 - Working on wrangling our log/temp/etc file management utilities so they're doing their job with our storage changes.  Even though disk space is cheap, I don't believe in being sloppy, so I've made this my pet project for the day. 

1:13 - Music ran out.  Midnight Juggernauts and Kasabian.

2:30 - Finished adjusting and documenting mw_cleanusertmp for now, ran it and freed 20GB+ of space. 

3:00 - Finished tuning and documenting mw-compressoldlogs for now, ran it and freed almost 30GB of space.

3:15 - The day-in-the-life epic saga typing continues, in terrible form. 

3:20 - Back on log rotation.  If only we could just use logrotate. 

4:00 - Got error_log truncation working on a new fileserver.  Freed nearly 20GB of space 

4:21 - Found out N and I might not get to go fishing at nearby Georgetown Lake tomorrow because the ER where she works apparently thinks paying loads of overtime is a good idea. (We actually did get to go, and caught over ten Rainbow Trout between us.  It was all catch-and-release.) 

4:25 - Packing up to go home and check on le chat.  (N working a 10 hour shift, so kitty may be stressed without companionship.) 

5:00 - Kitten checked, fed, box cleaned.  Catching up on the day's email.  I try to ignore email for large portions of the day...for efficiency! 

5:15 - Making miva_shuffle script for support team.  This script does a song and dance to remedy some occasional file locking problems we encounter with Miva. 

6:00 - Jonathan helped me navigate our arcane postgresql backups for our beta webmail system.  Gah!  It was missing a directory, that is all. 

6:15 - Done with miva_shuffle.

6:22 - Wrapping up work, giving animals evening meals.  Headed back downtown to meet N for Downtown Tonight. 

6:50 - Talked to Marianne about POSIX file permissions -- a customer struggling with getting two users RW access on the same files.  Modwest HQ is right outside Caras Park, the hub of downtown summer events.  Lucky for me Modwest cares about my results, not my schedule, so it was easy to get some support work done as I came through to the park. 

7:00 - Filled John's Chair at the Downtown Tonight Modwest booth and talked with Heather.  He was evaluating the brews, she was certain she'd forgotten everything computers while on vacation.  Full-Grown Men was the band for the evening. 

7:10 - N's here! We're eating at the Thai Spicy booth, Me - Green Curry Chicken, N - Cashew Nut Curry Chicken 

7:23 - Ilsa's Lemonade for drinks. 

7:45 - Broccoli calzone, I can't remember the real name, for a chaser; I'm still hungry... 

8:00 - Bike loaded in car, going to see N's dad, self-deprecating fly-fisher extraordinaire.  (http://en.wiktionary.org/wiki/extraordinaire -- "Unlike most English adjectives, extraordinaire follows its subject."  I never noticed that was particularly unique.  Man, this writeup is informative AND entertaining.) 

8:30 - Got a fly fishing knot lesson -- the knot we learned is supposedly not in any of the books out there, it seems actually a passcode that grumbly guys with waders and wild hair use to identify the non-posers. 

9:00 - Three episodes of 30 Rock while doing house chores. 

11:20 - Bed.

Got other ideas for the Modwest Blog? Let us know in the Modwest Community.

-JM

Alternatives to FTP

We're not quite ready to say "R.I.P. FTP", but it's true that FTP has an inherent security flaw: your username and password are passed in clear text from your computer to the server. This means that  malicious software on your computer could trivially capture your FTP credentials the next time you update your site.

We were alerted recently when a customer reported that some files in their website had been modified to include a strange looking IFRAME tag, which apparently loaded malicious code from a server in Russia every time a visitor arrived at the page. This prompted Google to temporarily list the customer's site as "Dangerous" in search results.

A quick check of the logs indicated the customer's account had been logged into via FTP from many IP addresses around the world, some of which uploaded the various malicious changes to the site. Our customer was able to securely upload a clean backup, and changed of course their password immediately.

We're not sure how the bad guys got ahold of this customer's FTP credentials, but if this was a case of spyware on the customer's PC capturing his cleartext username and password during a site update, then using an alternative to FTP could have helped avoid this painful event.

Two of the simplest alternatives are SFTP and SCP. Both protocols encrypt the connection from beginning to end, so cleartext "sniffers" can't read the traffic. SFTP support is often built in to FTP clients such as FileZilla, Cyberduck, and the built-in client of Dreamweaver, but may not be enabled by default. Check the help files for your favorite file upload program to learn how to enable SFTP.

WinSCP provides SCP and SFTP support, and a familiar Windows Explorer style interface, too.

There are no important differences between SFTP and SCP (both encrypt authentication and file transfers), so use whichever one your program supports. If your file upload program only supports old-fashioned FTP, it is time to upgrade.