Email Delays Resolved

Mail on the Modwest shared hosting system is flowing again now. Nothing was lost, but throughout the day yesterday (Friday), incoming messages could have been delayed several hours, webmail didn't work for most customers, and about half the mailboxes we host were totally unavailable for several hours last night.  A blow-by-blow is available on our offsite status page.

At the high-water mark for yesterday's problems, close to 200,000 messages were hung up in the queue, awaiting delivery to customer mailboxes. That included messages coming in to our support team, so it wasn't until late last night that a bazillion support request emails came in. Needless to say, we're a little behind.

The problem, as best as we could tell, was some sort of deadlock issue with our Cyrus IMAP software. The resulting behavior was all the 'stuff the message in a mailbox' processes believed they needed to wait for access to do so. All of them. I can sort of imagine them all politely saying "no please, I insist, you first" to each other, with no deliveries happening at all.

Anyway, the situation is resolved now; not necessarily in a permanent way, but everything is working at the moment and we're monitoring servers closely. Sorry about the trouble.

-JM