MIVA Misadventures at Modwest

Modwest has offered MIVA as our e-commerce solution for several years. Despite some administrative complexity (both on the server and client side), we've found it to be the most feature-complete shopping cart application on the market. Recently, though, it's presented some new challenges, some of which have affected a minority of our customers' hosted stores.

Over the last week or two, with about 15% of our hosted stores, we've seen intermittent lock-ups that cause the store to become completely inaccessible. Since there is no good way for us to know immediately upon a store malfunction, sometimes the store owner notices first and notifies us, and sometimes our server administration team notices first. After a great deal of research, we've determined a workaround which frees the store up and allows it to continue functioning, at least temporarily. What we haven't found yet is a way to prevent it from happening in the first place.

While we don't anticipate it'll permanently fix the problem,  we will be upgrading MIVA Empresa on Sunday, just in case it helps (as recently announced in OnSite).

Because our confidence in our ability to reliably provide MIVA software has dipped, at least temporarily, we're suspending installation of new stores until we're confident the situation is resolved appropriately.

Technical analysis of the problem:

Modwest runs a load-balanced cluster of web servers which all attach to a central SAN-attached file server using NFS. There's a concept of 'advisory locking' of files over NFS, which means a file which is locked should probably not be accessed/overwritten by other process, but this is not enforced in the protocol. MIVA, unfortunately, seems to ignore this nuance, and assumes that file locking over NFS is authoritative. MIVA does quite a bit of reading and writing to database files during normal operation, and (as we've now learned) can get itself into a situation in which it believes a file is locked, and waits around for the lock to be freed... forever. This results in a store that never loads for visitors.

Our workaround for digging MIVA stores out of this state isn't pretty, and there's some chance it could cause file corruption and/or data loss, but it does seem to get a store back up and running. Basically we make a copy of /mivadata, remove the original, and then move the copy back into place. i.e.,

# cp -a mivadata mivadata.new && mv mivadata mivadata.old && mv mivadata.new mivadata && rm -rf mivadata.old

It's our hope that MIVA can fix this issue, but we're also working on a Linux kernel module which, if we can do it, might help avoid this situation in the future. In the meantime though, we won't be installing new MIVA stores. We'll post a followup as soon as we have more information.

-JM