Modwest Blog

Migrate to a Faster Database Server in Minutes

Earlier this year, we deployed another MySQL database server (db2.modwest.com) for our shared hosting customers. It's a beast of a machine, and still has lots of horsepower to spare. In fact, compared to db.modwest.com, it's roughly 4x more powerful and currently doing a lot less work.

In addition to having lots of power to spare, db2 is also running MySQL 5, rather than MySQL 4.1. Version 4.1 will be "end of life" at the end of this year, meaning that MySQL will no longer provide bug or security fixes for it. MySQL 5 has some cool new features, as well, and so we encourage everyone with databases hosted on db.modwest.com to switch them over.

Most modern applications (such as Joomla, Wordpress, or Drupal) should be perfectly happy with MySQL 5; though if you have a custom application, you should probably ask your webmaster to read through MySQL's guide to upgrading from 4.1 to 5.

To make one step in the migration process easier, we've recently whipped up a web tool to copy a database from one server to another. It's here:

•  http://dbcopy.modwest.com

We hope you'll enjoy the increased performance of db2, and the increased features and security of MySQL 5, and let us know of any trouble along the way.

September Downtime Explained

Last month, we had a couple glitches with the storage system of our shared hosting environment. A number of you asked what we were doing to prevent it from happening again, so here's an overview of what happened and what's next:

Our shared hosting system was built with reliability in mind, with many redundant elements. It's a load-balanced cluster of webservers, so that if any one webserver goes belly-up, we keep running without any interruption. The filesystem itself runs on dozens of hard drives, so losing a hard drive causes no significant headaches either.

There are parts of the shared system that aren't redundant in the way we'd like: most of all, the file storage servers. If a file server crashes, web pages and graphics cannot load. That's what happened in September.

When a server starts randomly crashing, despite no changes being made, it can mean it's suffering from a hardware problem that is difficult to diagnose. So, our first step will be to move everything to a different server as soon as possible. We've already deployed a new file server, and this initial migration will be done later this month.

But the real problem is our reliance on each file server, so we've been building a truly redundant storage solution since summer. In fact, a number of volunteers have had their websites hosted on this new system for a while already. We intend to migrate all customer sites to the new system by year's end.

Technical info: The new storage system (using OCFS, AOE, and DRBD) involves multiple storage servers, each of which stores all the data.That means if one has a serious hardware failure, then the other one can take over.

If you're interested in being a beta tester, knowing that that involves occasional interruptions while we test and reconfigure and reboot the new system, just let us know.

Minor OnSite Upgrade: Printable Invoices

We've just pushed live a small update to Modwest OnSite, our hosting control panel. This update includes two features requested in the Modwest Community, specifically: "What features do you want added to OnSite?"

• Printable Invoices: Previously, OnSite presented an aggregated view of invoices and payments, in the manner of bank statements which describe deposits and withdrawals. Many people found this confusing and requested a simpler, separated view of payments and invoices. The invoices aren't available in PDF format for now, but they are printable as standalone invoices.
• Blog Post Syndication: Recent blog posts, including this one, now appear under "News & Info" in the right-hand column.

Thanks to those of you who have participated in the community and made these suggestions. It really helps.

If you notice any issues, please use the 'Report bug" link at the bottom of every page within OnSite. And keep the suggestions coming!

-JM

New Database Server Deployed

We've just deployed another MySQL database server for customers on the shared system. It goes by the name of "db2.modwest.com", and it's running MySQL 5. It's a powerful, brand new server and we encourage anyone using the older database servers to switch.

Adding a new database on the new server is easy. Just log into OnSite and click Add or Drop Database. Right now, the migration process is manual, but we'll be glad to assist upon request. One caveat: if you're running an older PHP application and currently using "db.modwest.com" (which runs MySQL 4.1), double check that your application is compatibile with MySQL 5, because there are some differences, such as new reserved words.

Enjoy!

Storage Server Issue: What Happened

Earlier this week we experienced a storage server problem on the Modwest shared hosting system. I just wanted to give you the low-down on everything that happened and how we responded.

Late Monday night, the server had warned us about a possible hard drive issue; it contains a whopping 24 drives, so one drive having a problem is no big deal. So, the next day, we prepared to replace the drive, which wasn't marked as "bad", just "unavailable" for some reason.

To make a long story short, we hit a glitch related to the confluence of Sun's Solaris operating system and LSI RAID hardware. The server crashed, and wouldn't restart. And, the 500GB of customer data on it was inaccessible.

Because something similar happened last year too, we immediately began preparing an alternate server to take this one's place. The new one runs good ol' Debian Linux.

But we needed access to the current website data. We opened a ticket with Sun commercial support and they couldn't figure it out beyond "hardware problem", but agreed that reinstalling the operating system might work. Some 12 hours later, after numerous attempts, our server team succeeded in accessing the customer data and began transferring it off the problem server onto the server we had standing by and ready to take its place.

It takes a long time to copy millions of files totaling nearly 500GB of data between servers, even on our fast internal network, and so then a waiting game began.

We know our customers' sites are important to them, and we are analyzing all aspects of the entire event so we can improve our operation and prevent a similar recurrence in the future.

During the outage, our awesome Support Team had the chance to speak with countless customers about the issue. We did our best to provide timely, accurate information about the work in progress, and kept the public Status Page up to date frequently. Did we do ok?

The Ops Team put in a tremendous effort over two days to get things up and running again as safely and quickly as possible. If you have a congratulatory haiku for the guys, please post it here: https://feedback.modwest.com/topic/103/Soothing_haikus_for_Modwest_engineers

We've recently hired a storage specialist who is now part of the team responsible for monitoring, maintaining, and improving our entire infrastructure. I'm much more confident now that the sorts of problems we endured with you this week will be less likely in the future. Thanks for your patience and understanding, and feel free to send us an soothing haiku!

-JM

PHP upgraded... and everything else

We have known for quite a while that we needed to upgrade PHP. In fact, it was a top requested answer to the question "What else can Modwest do to help our clients succeed?". We're listening! The answer is is now marked as "Done"!  =)

We gave a brief introduction to the new environment for shared hosting accounts last month, and thanks to great feedback from some intrepid customer beta testers, we have now made the new environment the official environment for all new shared hosting accounts at Modwest.

The cool thing is that anyone who signed up before May 2009 can be switched over to the new setup easily.  Yesterday, we added a new FAQ for people ready to switch:

 Modwest FAQ: What do I need to know about the new environment?

So, if you're an existing customer and trying to install the latest version of Wordpress, Joomla, or Interspire products, you've got a modern version of PHP to work with.

Enjoy!

-JM

Modwest Mages versus Lazy Linking Sorcery

This post from Modwest Mage (aka Systems Engineer) Tom:

Once upon a time, we wrote a yarn of PHP Versions and I/O woe. Mired in a battle of attrition, we enlisted a tough, gritty, mysterious stranger to bolster the ranks of our demoralized filesystem army. However, even a legendary hero could not rescue us from inevitable performance peril, and our wizards have been furtively developing new majicks to provide salvation for us and our citizens. As one of these wizards, I have my own chronicle to relate, although it lacks the glory of front-line battle.

My most recent task was to freshen our shared system webservers with the newest release of their operating system. By doing so, I would be opening the portal that would, in the near future, allow our webserver software to communicate with our new filesystem architecture. Unexpected foes riddled the veins of the project, and the task became a trial that spanned many days. The challenges tore at my sanity; my therapy is to share the hurdles with you.

The most daunting problem was rooted in Linux's "Lazy Dynamic Linking." While I am not sure I even completely understand -- it is magic, after all -- I will do my best to paraphrase the situation and solution.

First, for perspective, let me describe how our webservers handle CGI scripts. A request for a CGI triggers our modified sbox. Sbox chroots the script inside the shared system environment, and then runs the script as the owner of the script, not as the webserver user. We call this process "jailing".

My upgrade plan included no changes to jailing. The only thing that was to change was the underlying operating system. So, after rebuilding a webserver from scratch, I was surprised to see this error when attempting to run any CGI script:


/usr/lib/apache/suexec: relocation error: /lib/libnss_compat.so.2: symbol _nss_files_parse_grent, version GLIBC_2.0 not defined in file libc.so.6 with link time reference.

While the error was clue enough to hint at C, I am not much of a C programmer. I was beginning to get a bad feeling about this. Like any good apprentice, hoping it was common lore, I asked the arcane compendium seer for knowledge. Alas, information was sparse, and relevant tomes did not exist. I was on my own.

After days entombed in my computing cave, I had gathered shockingly little helpful data in isolating the problem.

But then, a flicker of hope sparkled from a configuration in nscd. With passwd caching off, CGI scripts began to run just fine, without explanation!

However, as any Linux hedge wizard knows, nscd yields massive performance gains on name lookups, and webservers like ours do these lookups constantly. Without solving this problem, the webservers would be too lethargic to work as a foundation for our project. Growing weak from monitor radiation and frustration, the situation looked grim to me.

I was certain I had failed king and country. Desperate, we summoned a magician from another fiefdom. One afternoon later, filled with gdb, strace, and ltrace, his expertise finally revealed the true nature of our error:

warning: .dynamic section for "/lib/libnss_compat.so.2" is not at the expected address Error while mapping shared library sections: /lib/i686/mmx/libnsl.so.1: No such file or directory. Error while reading shared library symbols: /lib/i686/mmx/libnsl.so.1: No such file or directory. Error while reading shared library symbols: /lib/i686/mmx/libnsl.so.1: No such file or directory.

By using a clever technique of adding a sleep statement in the C code to provide time to bind to the process with the debugging programs, he finally received the output above and instantly knew the problem.

Lazy Linking normally maps library functions in the Procedure Linkage Table, but doesn't actually load the functions. The functions only load when called. This is fine for normal environments, but for this instance, ld.so was trying to load the functions while in the jail, but the mapping had been done outside of the jail. In other words, after deciding that we would use the hardware libraries, we then later tried to use the jail libraries instead. Or, as the computer would say, 'dynamic section for "/lib/libnss_compat.so.2" is not at the expected address.'

The key to modifying this behavior lies in a simple environment variable.

Running with "LD_BIND_NOW = 1" solved all of the library problems I had seen. The variable tells ld.so to map and load the functions at runtime. We lose a little bit of efficiency, because the linker may be loading functions the program doesn't actually use, but it guarantees we use the libraries we think we are using.

Another ritual complete, but does it mean we're any closer to smashing the stalemate?

Websites on our shared hosting cluster aren't affected by this change, aside from any miscast spells on my part. The change does mean that we can, within a few weeks, deploy our new Apache stack that talks to the new filesystem. Assuming that deployment goes well, the beta testing that we've hinted about will start to become a reality a few weeks later.

Soon after, I hope we may close the chapter on this epic battle and enter an age of renaissance.

-TRC

Email Meltdown: Lessons Learned

One of our guiding principles is to continuously improve our systems by never getting bitten twice by the same problem. When disaster strikes, we review what happened and see what changes we can make to prevent similar headaches in the future. We don't always find a fixable “tragic flaw,” but the review  is still valuable.

Modwest hosts tens of thousands of mailboxes on our shared hosting mail cluster. We do this by spreading mail services out over numerous load-balanced servers. Some are devoted to processing messages on arrival--ascertaining whether the destination address is valid and then scanning the messages for viruses and spam content. Others are devoted to storage of the messages themselves. The first group (the “front end”) is like a strainer which does its best to ensure only legitimate messages get through to the second group, the “back end.”

Due to a series of headache-inducing events, it was the group of back-end storage servers that we had trouble with earlier this month. For the gory details, read on. Or skip to the lessons learned.

It all started with a report from one customer – they had a subfolder which was mysteriously inaccessible via webmail.  After some initial debugging on the appropriate back-end storage server, we concluded that the mailbox was corrupt.

Each mailbox can be thought of like a book and its index. Each message is a page in the book, and is referenced in the index. Every entry in the index has a corresponding page. However, if the index and the pages don't agree for some reason, a system administrator needs to “reconstruct” the mailbox to eliminate this corruption. This is very infrequent, but happens from time to time.

This reconstruction process usually takes a few seconds up to a few minutes for mailboxes that contain lots of messages and folders. On this mailbox, the process seemed to be still running after close to 24 hours. Actually, it was stalled, unable to begin work on the problematic folder.

Some additional research indicated the problem was not with the mailbox, but the underlying filesystem – the software glue between the mail server software and the physical hard drives. The only fix for that problem is an hours-long offline repair process. So, we began preparations for a period of downtime.

The next day, our patient took a turn for the worse. Right in the middle of the business day, the server's CPU utilization skyrocketed and performance plummeted. We did everything we could to mitigate but finally decided that performance was so bad, with webmail taking minutes to load every page, that we weren't really providing a “service” anyway, so we better go ahead and begin the offline filesystem repairs immediately. It took nine hours to complete.

And it didn't help.

So our next tactic was to move mailboxes off the problematic server as quickly as possible. Luckily, we had a brand new machine racked up and ready for exactly this purpose, and after some last-minute software upgrades and configuration changes, it was ready to help. We moved some 4,000 mailboxes late that night.

And that helped a lot! What a relief, for everyone.

Except that the upgrades required to integrate the new server left our oldest mailbox storage server out in the cold. Its older software was unable to communicate properly with all the new stuff, and so IMAP mailboxes, including webmail, were unavailable. Thousands of mailbox owners were denied access to their mailboxes unless they added some special settings or used an alternate webmail address we set up.

Our first plan was to move all the mailboxes on the older server to the brand new one. In theory, this could be done transparently and without interruption. But after exhaustive research on the plan, we concluded it just wouldn't work. An upgrade with data-in-place was the only way forward.

Meanwhile, since we're preparing for an office move (more on this in a future post), we are in the process of organizing and relocating various equipment. Late one night, we moved a backup server without changing its power settings. The resulting voltage mismatch caused a power fluctuation which resulted in several servers rebooting – including the brand new mail storage server. So, the thousands of mailboxes we'd just moved off the problem server were unavailable again while we ran the required filesystem repairs. This coincidence was just plain bad luck.

Near the end of last week, we scheduled and completed the final upgrade required to get the older storage server up to speed with the rest. The mail system has been functioning fine every since.

 

Lessons learned:

On the server-side, we haven't yet found any “silver bullets” to truly prevent this from happening again. There are no good ways for us to provide meaningful additional redundancy which would somehow host mailboxes in more than one place at once, and we still don't know exactly what caused the initial problem.

What we can do is communicate better about issues like this.

• When we're having server problems, our status page needs to be front and center. Our support team received hundreds of calls during the various problems, and many people simply didn't know the status page existed. We're working on ways to remedy that.

• The status page needs to more clearly explain the scope of the issues and their resolution. Despite thousands of mailboxes being non-functional for hours, the page claimed that POP3 service was “100% OK” throughout.  This is because the statistics are automatically generated based on a simple check of initial email connectivity – not whether any mailboxes are actually functioning. So, we need to explain this better, and/or implement more comprehensive checks.

As much as I'd like to, we can't prevent servers from breaking. But when problems do occur, we can and will do a better job of communicating what happened, who's affected, and what we're doing about it. And as always, we're happy to hear your feedback.

-JM

Sorry about the email

Update Aug 13th, 8:30PM: We've got the server back up and running and it's working hard to catch up with the backlog of messages. Check the status page for updates.

Current status:

A significant number of our customers (and all Modwest staff) with email boxes on the Modwest shared system have been without access to their mailboxes since about 11AM this morning. That's about when server alarms went off and we concluded it was necessary take one of our mailbox storage servers offline for diagnostics.

At the moment, we're running a filesystem repair utility which is examining every message and every mailbox to ensure data integrity; this process needs to complete before we can bring the server back online. Unfortunately it's impossible to ascertain how long this is going to take, but we'll keep the Modwest Status page updated as we know more.

As an educated guess, a repair process like this could take another 12 hours, though there are some indications it'll complete sooner. When it does, pending mail will be delivered. There's no indication that anything will be lost.

Some History:

Earlier this week we were alerted to a customer mailbox which had one folder which was inaccessible. Our standard mailbox diagnostic utilities couldn't even complete their analysis, which led us to conclude there was a filesystem problem.

The "filesystem" is the low level software that connects mailboxes to the hard drives in the server. The only way to fix a filesystem problem like this is to take the server offline and run repair utilities, and so we'd planned to conduct this maintenance this coming weekend. Then the alarms went off this morning, and so we had to perform the surgery right away.

The future:

As an immediate right now workaround, it's possible to create a new mailbox in OnSite and direct any new (but not pending) mail there. This has a downside: your mail will be split between your old mailbox (on the storage server undergoing maintenance) and the new mailbox (on an unaffected storage server). Because of that hassle and complexity (and the fact that we intend to have this issue resolved later tonight), we don't recommend this solution, but present it as option for those who want to do it. The only trick is that you need to log in as your administrative user and use the Add Mailbox page, not the Mailboxes & Addresses link.

We have spoken to some customers today who say they intend to move their mail elsewhere, which we can understand under the circumstances. Going with a specialist email provider might meet your needs better. These are not recommendations, but here are some options we've heard of:

• http://www.google.com/a/help/intl/en/admins/editions.html
• http://www.postini.com/
• http://www.messagelabs.com
• http://www.mailtrust.com/
• http://www.mxlogic.com
• http://www.gmsi1.com/index.html

In the meantime, we're working to restore service to the affected mailboxes as soon as possible. For additional updates, please check the Modwest Status page.

Sorry about the trouble.

-JM

Saturday Night Maintenance

No, it's not a new John Travolta movie.

We're performing hardware maintenance on our shared hosting system on Saturday night, July 26th, 2008, starting at about 10PM Mountain Time. This will result in up to five hours of hosted websites being unavailable. Email (POP3/IMAP/SMTP) will work, but webmail won't.

We haven't had to schedule a five hour maintenance window in a few years, so we'll also be sending out email to affected customers and their technical contacts.

While our Modwest System Status page gives a brief overview of why this maintenance is necessary, I thought I'd take a moment here to provide some gory technical details.

As previously mentioned, we're in a transition phase, in which we are gradually replacing the storage hardware and software of our shared hosting system. This can mostly be done transparently,  but we're unavoidably relying on vendors' hardware and software working properly.

And there's the rub. We need to replace an Areca RAID controller that isn't behaving, and that requires an operating system reinstall (since the new controller is from a different vendor, LSI); hence, a five hour maintenance window.

I asked our Operations Manager to explain the technicalities of the problem this maintenance window addresses, and here's how he described it:

While it's not clear exactly who-is-causing what, what is clear is the Areca driver tries to de-reference a NULL pointer, and this is either because the adapter screws up, or the driver screws up somewhere.  The result is a Solaris kernel fault, pointing at the arcmsr driver, and apparently an adapter lockup.  It's not 100% clear what causes this condition.  It could be the driver not handling some buffer appropriately, it could be the card sending an error that the driver doesn't handle.  It's pretty likely though the issue is completely inside of the arcmsr driver and Areca hardware.  One thing we did discover that means we HAVE to replace the Areca hardware is that in JBOD mode (which is how we use it, since we're using Solaris' ZFS superset of RAID functionality), any disk failure seizes the whole card up until the failure clears, or maybe until some apparently long timer clears.  SATA and SAS have ethernet-like link failure detection.  You know within milliseconds when the cable is pulled. The Arecas in JBOD mode seem unable to handle hot-swap of any type, or even failures of any type.  When we tried to get Areca to address it, all we received was vague "you must have a failing drive" answers, which for a RAID card is a bad answer.  Even in JBOD mode the controller should signal/propagate an error.  Solaris would handle this condition.

Then there's the boot selection.  All logical drives appear in the boot selection.  Either the list fills up or the Areca's only show drives on the first controller.  That's a problem if you want to be able to boot an alternate drive on a second controller.

Catch all that?

The point is that we absolutely need our core storage technologies to be 100% rock solid. Now that we've detected this problem, we must address it ASAP, and that's what this Saturday Night Maintenance is all about.

-JM