Sorry about the email

Update Aug 13th, 8:30PM: We've got the server back up and running and it's working hard to catch up with the backlog of messages. Check the status page for updates.

Current status:

A significant number of our customers (and all Modwest staff) with email boxes on the Modwest shared system have been without access to their mailboxes since about 11AM this morning. That's about when server alarms went off and we concluded it was necessary take one of our mailbox storage servers offline for diagnostics.

At the moment, we're running a filesystem repair utility which is examining every message and every mailbox to ensure data integrity; this process needs to complete before we can bring the server back online. Unfortunately it's impossible to ascertain how long this is going to take, but we'll keep the Modwest Status page updated as we know more.

As an educated guess, a repair process like this could take another 12 hours, though there are some indications it'll complete sooner. When it does, pending mail will be delivered. There's no indication that anything will be lost.

Some History:

Earlier this week we were alerted to a customer mailbox which had one folder which was inaccessible. Our standard mailbox diagnostic utilities couldn't even complete their analysis, which led us to conclude there was a filesystem problem.

The "filesystem" is the low level software that connects mailboxes to the hard drives in the server. The only way to fix a filesystem problem like this is to take the server offline and run repair utilities, and so we'd planned to conduct this maintenance this coming weekend. Then the alarms went off this morning, and so we had to perform the surgery right away.

The future:

As an immediate right now workaround, it's possible to create a new mailbox in OnSite and direct any new (but not pending) mail there. This has a downside: your mail will be split between your old mailbox (on the storage server undergoing maintenance) and the new mailbox (on an unaffected storage server). Because of that hassle and complexity (and the fact that we intend to have this issue resolved later tonight), we don't recommend this solution, but present it as option for those who want to do it. The only trick is that you need to log in as your administrative user and use the Add Mailbox page, not the Mailboxes & Addresses link.

We have spoken to some customers today who say they intend to move their mail elsewhere, which we can understand under the circumstances. Going with a specialist email provider might meet your needs better. These are not recommendations, but here are some options we've heard of:

• http://www.google.com/a/help/intl/en/admins/editions.html
• http://www.postini.com/
• http://www.messagelabs.com
• http://www.mailtrust.com/
• http://www.mxlogic.com
• http://www.gmsi1.com/index.html

In the meantime, we're working to restore service to the affected mailboxes as soon as possible. For additional updates, please check the Modwest Status page.

Sorry about the trouble.

-JM

Comments

[mark]

Can you tell us if any mail was lost during the downtime or would mail set to our modwest email addresses during this downtime be delivered once the service is fully restored?

Thanks.

[John]

A side effect of email being developed decades ago in an era of unreliable networks is that it's very good at dealing with temporary outages. We have no indication that anything was lost -- messages arriving during the outage were actually accepted by the Modwest system and delivered to the destination mailboxes once they became available again.

[mark]

Thank you for the update.

[Doug]

We're not moving. It would be a challenge to find a company as dependable, responsive and honest as yours. We appreciate your approach when problems arise.

Get some rest!