Modwest Blog

Five Improvements for the New Year

We've been working for months on a number of improvements to our web hosting services, and we've just unveiled several new features and improvements. Check it out:

1. Auto-installed applications: We've started with Wordpress as requested, which may now be added easily in OnSite under the "Other Tools" menu.
2. Prepayment discount: As requested in the Modwest Community, we now offer substantial prepayment discounts on our new web hosting plan.
3. Affiliate program: For a limited time, we've increased our referral credit to $50. There's no better time to tell your friends & colleagues about Modwest!
4. Community upgrade: You'll now notice that we provide Google Images options while you're adding new topics and answers in the Modwest Community.
5. AWStats: now available as an alternative website traffic reporting program for just $1/month. Add it under 'Request Add-ons' in Onsite.

We hope you enjoy these upgrades, and as always, we're eager to hear your feedback in the Modwest Community. Here's to starting off 2010 right!

-JM

Minor OnSite Upgrade: Printable Invoices

We've just pushed live a small update to Modwest OnSite, our hosting control panel. This update includes two features requested in the Modwest Community, specifically: "What features do you want added to OnSite?"

• Printable Invoices: Previously, OnSite presented an aggregated view of invoices and payments, in the manner of bank statements which describe deposits and withdrawals. Many people found this confusing and requested a simpler, separated view of payments and invoices. The invoices aren't available in PDF format for now, but they are printable as standalone invoices.
• Blog Post Syndication: Recent blog posts, including this one, now appear under "News & Info" in the right-hand column.

Thanks to those of you who have participated in the community and made these suggestions. It really helps.

If you notice any issues, please use the 'Report bug" link at the bottom of every page within OnSite. And keep the suggestions coming!

-JM

Storage Server Issue: What Happened

Earlier this week we experienced a storage server problem on the Modwest shared hosting system. I just wanted to give you the low-down on everything that happened and how we responded.

Late Monday night, the server had warned us about a possible hard drive issue; it contains a whopping 24 drives, so one drive having a problem is no big deal. So, the next day, we prepared to replace the drive, which wasn't marked as "bad", just "unavailable" for some reason.

To make a long story short, we hit a glitch related to the confluence of Sun's Solaris operating system and LSI RAID hardware. The server crashed, and wouldn't restart. And, the 500GB of customer data on it was inaccessible.

Because something similar happened last year too, we immediately began preparing an alternate server to take this one's place. The new one runs good ol' Debian Linux.

But we needed access to the current website data. We opened a ticket with Sun commercial support and they couldn't figure it out beyond "hardware problem", but agreed that reinstalling the operating system might work. Some 12 hours later, after numerous attempts, our server team succeeded in accessing the customer data and began transferring it off the problem server onto the server we had standing by and ready to take its place.

It takes a long time to copy millions of files totaling nearly 500GB of data between servers, even on our fast internal network, and so then a waiting game began.

We know our customers' sites are important to them, and we are analyzing all aspects of the entire event so we can improve our operation and prevent a similar recurrence in the future.

During the outage, our awesome Support Team had the chance to speak with countless customers about the issue. We did our best to provide timely, accurate information about the work in progress, and kept the public Status Page up to date frequently. Did we do ok?

The Ops Team put in a tremendous effort over two days to get things up and running again as safely and quickly as possible. If you have a congratulatory haiku for the guys, please post it here: https://feedback.modwest.com/topic/103/Soothing_haikus_for_Modwest_engineers

We've recently hired a storage specialist who is now part of the team responsible for monitoring, maintaining, and improving our entire infrastructure. I'm much more confident now that the sorts of problems we endured with you this week will be less likely in the future. Thanks for your patience and understanding, and feel free to send us an soothing haiku!

-JM

First Fruits of the Modwest Community

A couple weeks ago we launched the Modwest Feedback Community.  Already, we have heard important ideas form customers and implemented changes in response!

Here's a few examples:

• radiohead correctly pointed out that while Subversion (svn) is available in all Modwest shared hosting accounts, we hadn't done a very good job of letting people know. We quickly made a few changes so that it shows up on the big features page as well as the main webhosting page under Site Management. Thanks radiohead!
• In a comment thread, squiggle advised that our System Status page should continue displaying maintenance and issue alerts for more than just a few days, to aid our clients in debugging any issues they may be tackling. We took care of that a few days later, and now the last 30 days of notices display. 
• In an OnSite feature request topic, bryanb suggested we provide a more direct way for site owners to get into webmail. Turns out that already existed but we never told anyone! There are two options:

http://webmail.modwest.com
https://beta-webmail.modwest.com

As an example of what's being discussed right now, here's a recently popular topic (click it for details!):

So, we'd say our "kick off" was a great success. If you haven't visited recently, check it out, and participate for a chance to win real prizes like this one:

-JM

Sorry about the email

Update Aug 13th, 8:30PM: We've got the server back up and running and it's working hard to catch up with the backlog of messages. Check the status page for updates.

Current status:

A significant number of our customers (and all Modwest staff) with email boxes on the Modwest shared system have been without access to their mailboxes since about 11AM this morning. That's about when server alarms went off and we concluded it was necessary take one of our mailbox storage servers offline for diagnostics.

At the moment, we're running a filesystem repair utility which is examining every message and every mailbox to ensure data integrity; this process needs to complete before we can bring the server back online. Unfortunately it's impossible to ascertain how long this is going to take, but we'll keep the Modwest Status page updated as we know more.

As an educated guess, a repair process like this could take another 12 hours, though there are some indications it'll complete sooner. When it does, pending mail will be delivered. There's no indication that anything will be lost.

Some History:

Earlier this week we were alerted to a customer mailbox which had one folder which was inaccessible. Our standard mailbox diagnostic utilities couldn't even complete their analysis, which led us to conclude there was a filesystem problem.

The "filesystem" is the low level software that connects mailboxes to the hard drives in the server. The only way to fix a filesystem problem like this is to take the server offline and run repair utilities, and so we'd planned to conduct this maintenance this coming weekend. Then the alarms went off this morning, and so we had to perform the surgery right away.

The future:

As an immediate right now workaround, it's possible to create a new mailbox in OnSite and direct any new (but not pending) mail there. This has a downside: your mail will be split between your old mailbox (on the storage server undergoing maintenance) and the new mailbox (on an unaffected storage server). Because of that hassle and complexity (and the fact that we intend to have this issue resolved later tonight), we don't recommend this solution, but present it as option for those who want to do it. The only trick is that you need to log in as your administrative user and use the Add Mailbox page, not the Mailboxes & Addresses link.

We have spoken to some customers today who say they intend to move their mail elsewhere, which we can understand under the circumstances. Going with a specialist email provider might meet your needs better. These are not recommendations, but here are some options we've heard of:

• http://www.google.com/a/help/intl/en/admins/editions.html
• http://www.postini.com/
• http://www.messagelabs.com
• http://www.mailtrust.com/
• http://www.mxlogic.com
• http://www.gmsi1.com/index.html

In the meantime, we're working to restore service to the affected mailboxes as soon as possible. For additional updates, please check the Modwest Status page.

Sorry about the trouble.

-JM

Email Delays Resolved

Mail on the Modwest shared hosting system is flowing again now. Nothing was lost, but throughout the day yesterday (Friday), incoming messages could have been delayed several hours, webmail didn't work for most customers, and about half the mailboxes we host were totally unavailable for several hours last night.  A blow-by-blow is available on our offsite status page.

At the high-water mark for yesterday's problems, close to 200,000 messages were hung up in the queue, awaiting delivery to customer mailboxes. That included messages coming in to our support team, so it wasn't until late last night that a bazillion support request emails came in. Needless to say, we're a little behind.

The problem, as best as we could tell, was some sort of deadlock issue with our Cyrus IMAP software. The resulting behavior was all the 'stuff the message in a mailbox' processes believed they needed to wait for access to do so. All of them. I can sort of imagine them all politely saying "no please, I insist, you first" to each other, with no deliveries happening at all.

Anyway, the situation is resolved now; not necessarily in a permanent way, but everything is working at the moment and we're monitoring servers closely. Sorry about the trouble.

-JM

E-Mail Delays

We are currently working on an issue that has caused email delivery delays today. For updates, See the Modwest System Monitor at http://status.modwest.com/.

A Holiday Message (and Gift!) from Modwest

Whichever holiday you celebrate, the team at Modwest would like to wish you a happy one! We look forward to serving you in the new year, and as a gesture of our appreciation, have decided to double our disk and bandwidth limits on our shared hosting Value plans and above. The new configurations will be applied immediately and automatically.

Also, some details about our phone support availability over the holidays:

• We'll be closed on December 25th and January 1st
• From December 24th to December 31st, phone support will be available M-F from 9-5PM (Mountain Time)

Naturally, operations staff will be tending to the servers as needed, and email support will be available throughout.

-JM

Howdy! And Network maintenance tonight

An introduction first - I'm John Masterson, a Modwest co-founder and general overseer of Support and Operations around here.

As recently announced in OnSite and our Status Page, we'll be taking our network offline briefly this evening for a router upgrade. This affects "everything": shared accounts, managed servers, and our new VPS accounts. Maintenance begins at 11PM Eastern Time and will last from 15-30 minutes.

This is a vendor-recommended upgrade from Juniper and offers numerous fixes and features, including "80,000 microsecond packet buffers", "ultimate-hop popping", and other network sorcery that ensures continued secure, reliable communication between your visitors' browsers, and our servers.