Modwest Blog

Network Wrangling at Modwest: How One Hacked Site Can Affect Thousands of Customers

Tweet

by John Masterson

In late November, Modwest staff and a handful of customers noticed that we dropped off the internet for a few minutes. After a significant investigation, we struck out in identifying the source, and so we moved on to other concerns.

Then it happened again. And again.

Third time's a charm, and so we began working on improving our ability to monitor our network traffic in fine detail, and narrow down the range of possibilities as to the cause of the disruptions. Was it a failing switch or router? Some sort of misconfiguration? An attack? It was unclear, but we launched a thorough investigation.

The issue returned intermittently. A week would pass, and then we'd have four 5-minute events in a day. And for a time, while we had our suspicions (which eventually turned out to be correct), the specific source eluded us.

But we kept on the investigation.  We determined that something was creating a massive burst of internal network traffic, for a few minutes at a time. So much traffic that one of our core switches would stop responding briefly, causing a flurry of "host is down" alerts, and causing some customer websites and email to stop responding.

Our process involved upgrading and improving the configuration of our NMIS implementation to be sure we were monitoring virtually everything that can be monitored and graphed. We also improved our switches' "storm control" settings, a feature which can temporarily shut down affected network segments in the event of a sudden burst of traffic beyond a normal profile.  And we analyzed packet captures for source, destination, rate, and protocol. 

Once we'd narrowed the issue down to a group of Modwest Grid webservers, we tried to catch it in the act (using tools called tcpdump and wireshark, if you're curious). After a few days, we succeeded. And the new monitoring we'd installed on the webservers created logs of a gigantic burst of outbound traffic associated with one particular customer.

The culprit? The customer's outdated WordPress installation had been previously exploited by a remote hacker to install an attack program, which was being periodically accessed and used to launch a massive UDP packet flood at various external hosts.

We deleted the program, informed the customer, and locked down a few other parts of our network to detect and quaratine similar events in the future.

We try to turn every disruptive event into an opportunity to improve operations, and this case was no exception.  It highlighted the importance of having deep knowledge of your network setup, monitoring, reporting, and graphing -- which we have now accomplished. It also serves as a reminder of the potential impact of ignoring or overlooking those pesky security updates in common web applications like WordPress, Joomla, and Drupal.

At Modwest, we have one day a month we call "Update Day". It's the day on which we go through our list of servers and switches, and ensure everything has the latest patches for security, performance, and functionality.  Consider (or talk to your webmaster about) doing the same -- it makes your upgrades easier and helps keep your site secure.

There will always be bugs, exploits, and attackers. This particular issue is resolved, and we're much better equipped to deal with the next one. Onwards!

timthumb.php vulnerability

As Modwest hosting supports WordPress installs, we wanted to make you aware of a timthumb.php vulnerability that affects both themes and plug-ins which utilize the timthumb.php script. The vulnerability lies in a coding error that allows would-be attackers to backdoor malicious .php scripts while masquerading them as benign thumbnails.

According to articles I read, the original blog about this issue was posted on August 3, 2011 at ++Zen Blog. Here's an up-to-date list from Sucuri Research Blog (August 17, 2011) of plug-ins and themes being scanned by attackers.

If you utilize these products, mimize the risk of being exploited by acting promptly.

To POP or IMAP, that is the question

Posted by: John Masterson, Modwest co-founder

Modwest customers can choose to access their hosted email via either POP or IMAP access. How does a mailbox owner choose?

We've helped customers with this question many times over the years, and here are some selected highlights:

In a nutshell, IMAP frees me from having to wonder "where" my emails are, and whether I'm working with "copies" or not.

Think of IMAP as a way to work directly with the server itself, as opposed to "copies" of your messages. With a typical setup, any actions you take on your emails (deleting, for example) happen on the server and your local machine simultaneously. When you delete an email with IMAP, it is usually moved into your server-side trash folder. When you empty that folder (or individual mails there) they are deleted from the server permanently.

Unless you move messages to your local computer, you can only view your IMAP messages while connected to the internet.

And regarding POP:

The advantages:

• POP is somewhat simpler to manage. If you see a message, it is on your computer.
• You can view all of your messages even if you are not connected to the internet.
• More software supports the POP protocol.

The disadvantages:

• If you download a message it is only available on that computer.
• You cannot access your account if someone else is accessing it at the same time.

POP is often best if you are only going to view the messages from a single computer and your Internet connection is not always available.

I access my mail from four different computers (home, work, laptop, smartphone), and occasionally via webmail. In order to have a consistent perspective on my messages, I use IMAP exclusively.

No matter which method you choose, it's a good idea to figure out what you're doing with backups:

• How do I back up my email?
• What does Modwest back up?

 

A Look Back On The Year That Was

Posted by: John Masterson, Modwest co-founder

Another year has passed us by, and the time has come to reflect on the year that was.

First, a quick recap of new features and key events from 2010:

• In January, we celebrated our 10th birthday!
• We introduced our new "use what you need" Yep web hosting plan.
• We rolled out a new application autoinstall feature in OnSite, startiing with WordPress, Joomla, and Drupal.
• For thorny technical issues, we can now connect to your computer (with your permission) to assist via CoPilot.
• Due to popular demand, you may now contact us via live chat.
• We improved our spam filtering controls.
  
• Because we do web hosting differently, we renamed our system The Modwest Grid.
  
• VPS customers, we doubled your memory.
  
• We developed a coupon system that can give new customers discounts on signup. Keep your eye on Twitter and elsewhere for deals.
  

There were a number of valuable discussions in the Modwest Community in 2010, and the top five community topics were:

• What features do you want added to OnSite?
• What programs do you use to build your website?
• What applications should be available to auto-install on your site?
• What Modwest-branded goods or general giveaways would make good prizes for community contests?
• What browser do you use most often?

Finally, here's a few more stats that somehow illustrate what happened at modwest in 2010:

• number of miles that staff use alternative transporation to get to work: 5,476
• approximate # of web hits received on the grid: 4,700,000,000
• approximate # of spam messages identified: 130,000,000
• number of trouble tickets resolved: 9,255
• number of chat sessions handled since November: 189
• lbs of food donated to food bank: 90
• website uptime: 99.9%
• mail uptime: 100%
• database uptime: 99.99%
• number of pints poured from the kitchen kegerator: unknown
• approximate number of inches of snow that landed on our patio across from Caras Park: 26
• number of customer service phone calls answered = 4321
• number of customers that migrated to our newer hosting environment = 213
• number of people to join Modwest Community = 384

 

We hope your numbers were as good as ours. Happy New Year!

 

Blog Owners: WordPress Critical Security Update

Whether you installed WordPress yourself, or used our handy-dandy WordPress autoinstaller, you'll want to log in to your blog's admin page to upgrade. On the Modwest Grid, it's a one-click upgrade.

As they're saying over on the WordPress Development Blog, this upgrade is:

"...a very important update to apply to your sites as soon as possible because it fixes a core security bug in our HTML sanitation library, called KSES. I would rate this release as “critical.”

I realize an update during the holidays is no fun, but this one is worth putting down the eggnog for. In the spirit of the holidays, consider helping your friends as well."

 

Safe blogging,

-The Modwest Team

Don't Blacklist Me!

Posted by: John Masterson, Modwest Co-founder

Have you ever sent an email to a friend or colleague, only to find out the message was never received because your provider's mail server was on a "blacklist"? Or, perhaps you received an error message just a few minutes after sending, which cryptically explained that messages from your IP address are not being accepted?

As a defense against unsolicited email (spam) and viruses, most mail server administrators implement some sort of filtering that reject or delay mail from computers that send out lots of junk. These lists of junk-spewing computers are often referred to as "blacklists". There are a lot of reasons that servers can end up on blacklists, like when professional spammers set up servers to send billions of messages from countries with little or no law enforcement resources to deal with internet abuse. Nobody wants the email coming from those servers.

Unfortunately, there are also situations where innocent senders are erroneously labeled as spammers because of one "bad apple".

One way in which Modwest mail servers have found their way onto blacklists from time to time is due to a customer's website being exploited to send spam. This can happen when spammers discover a vulnerability in a popular web application, like WordPress, OSComerce, Joomla, or ZenCart, which allows them to send their junk mail right from the customer's website.

How You Can Help

Exploited sites are a problem for all web hosts and their customers. You can help ensure your site stays safe by promptly applying security updates from your web application vendor (e.g., WordPress, Joomla, Drupal, ZenCart, X-Cart, phpBB3, etc).

Until recently, that scenario would result in the junk being relayed through our central mail system, mail.modwest.com. If enough junk got out before we detected and stopped it, sometimes big email providers like Yahoo, AOL, Gmail and Hotmail would temporarily stop accepting mail from Modwest.

While we could generally convince them to remove the block within a day or two, it was a headache for everyone -- especially the innocent bystanders who just wanted to send mail to their friends and colleagues.

So, this week we're beginning a gradual change that separates mail generated by websites from everything else. Mail generated by websites includes forum reply notifications, ecommerce receipts, and so forth -- as well as junk mail generated by exploited sites.

The benefit of this change is an exploited site will now be much less likely to cause email delay or rejection problems for everyone else. And, you don't need to do anything different with your website or email.

A second, more esoteric change we made this week was in how our domain aliases are handled. A quirk of our system was that some remote mail servers would consider bounce-back messages from invalid addresses at domain aliases to be "backscatter", which they considered evidence of spam. We've solved that problem too.

Finally, we've expanded some of our front-line filtering to reject mail from a few persistent sources of spam.

Combined, these changes should reduce the spam you receive, and help prevent you from being falsely labeled a spammer. Of course, if you are a spammer, you'll be hearing from us. :)

New Affiliate Banners

It'd been a while, so today we launched some new affiliate banners for you to post on your site.

When someone clicks on one of the banners, you can get credit for free hosting. In fact, we have numerous customers who enjoy free hosting because of their stream of referrals. Thanks!

To check out the new banners and get the code to put on your site, just enter your Modwest login name or domain here.

 

 

Improved Spam Filtering Controls

If you are like most people, you spend a few minutes almost every day tediously plucking spam messages from your inbox, and occasionally, a real message from the spam folder.

Now there's a new option for our hosting customers to help keep spam out of the inbox: automatic spam removal. By using this option, the server will automatically remove the most blatant spam so you don't have to.

If you are already using other spam filtering preferences, adding the automatic removal option will allow you to more safely increase the sensitivity of the filter to file "maybes" into a folder for review. Plus, you won't have a zillion spam messages to sift through while searching for legitimate messages.

This has made a huge difference for many of us here in the office. For example, using the new controls, I feel confident setting my baseline filtering level to "possibly spam (2 or more points)", filing detected spam into my Spam/ folder, and then setting the auto-delete threshold to  "usually spam (8 or more points)". I can periodically check my Spam folder for legitimate messages, and add those senders to my 'Always Allow' list.

To get started, just log into OnSite, go to Spam Settings, and choose an option from the 'Delete Messages' drop-down box.

We've always felt strongly that our clients deserve as much control over their inbound mail as possible, and we've documented our filtering procedures and empowered our customer service team to track down the whereabouts of every message we handle. We hope this new feature adds to that control and flexibility, and is useful to everyone who wishes to use it.

Please let us know what you think, and of any questions.

Modwest technical support: Your liaison with the web hosting world

It is a fact that a large portion of our customers don't know what service Modwest provides them.  On the face of it, this may seem strange--could many other businesses claim this of their customer base?  And, if you're like many people I talk to, you, too, might be wondering what exactly Modwest does for you.

Quickly, let me just say: Modwest is your web host.  But, if you've ever wondered what a web host is or never heard the term, trust me, you're in very good company, company that includes my family, friends, and pretty much everyone I meet.  In fact, on the advice of a co-worker, I've developed the habit of responding to the question, "What do you do for work?" with the half-lie, "I work for an internet company."  The term "internet company" includes lots of jobs I don't do, of course, but at least the term doesn't puzzle the person I'm talking to, make him squinch up his face, or pause awkwardly and say, "Oh..."

It doesn't bother me that people are generally unfamiliar with my line of work, it's just the case that the role a web host fills is not widely understood (yet). Nor has the term "web host" permeated the non-technical sector (or what most people call "the world") with much force.  One reason might be that it's difficult to come up with a non-technical analogy for what web hosting is, but here are two, anyway. And, to make matters worse, most of the terms one would use to describe the services of a web host are less familiar than "web hosting" itself, such as IMAP, POP, SMTP, FTP, domain, DNS, server, and SSL certificate.  None of these are terms you hear casually tossed out in everyday conversation.

Yet, everyone who has a website--including those who have passions, hobbies, and businesses outside the realm of technology--needs a web host, so inevitably their world intersects with the web hosting one.  Many people have told me this intersection can feel more like a collision; Becoming the manager of a website when you have little technical knowledge of how to do so can be painful, confusing, and outright maddening. 

As an example, a dentist may learn that "if she wants to use Google Apps, she has to update her MX record"; a clean energy consultant may be told that he needs to "point his domain name at Modwest" or "use an FTP client to upload his files"; a bamboo fly rod enthusiast may be instructed by her web designer to "upgrade her website to the latest version of PHP" or even worse, "enable PHP's curl extension", to which they will all likely respond:  YOU. HAVE. GOT. TO. BE. KIDDING. ME.

So, what's the solution? 

Modwest's solution is our technical support team.  As you may know, Modwest is especially proud of its "insanely good technical support" (as one customer put it), which comprises groups of specialists who aim to solve our customers' email or website related woes quickly and expertly.  But this standard description of our support team fails to convey one key point: We are easy-to-talk-to and more than happy to play the role of your technical translator or of a liaison, between you and all the techno-jargon of the web hosting world.

Every web host has their strengths, but web hosting and demystifying web hosting is ours.  Please don't ask me about the technicalities of dentistry, clean energy, or bamboo fly rods unless you want to see my best blank stare.  However, I, and the rest of our support team, can definitely explain and help you complete such foreign tasks as importing a MYSQL dumpfile into your database or creating a Mod_Rewrite rule in a .htaccess file.  It's not as scary as it sounds, call me and I'll explain why.

Got a question about web hosting?  Ask: geekchick@modwest.com

New Applications Ready for auto-install at Modwest

As requested in the Modwest Community, we've recently added two more applications for you to auto-install in your accounts: Drupal and Joomla

Both applications are content management systems (CMS) that can stand alone as an entire website, and free templates to adjust the look and feel of the site exist for each.

Hope you enjoy them, and please keep the suggestions coming.