Modwest Blog

Network Wrangling at Modwest: How One Hacked Site Can Affect Thousands of Customers

Tweet

by John Masterson

In late November, Modwest staff and a handful of customers noticed that we dropped off the internet for a few minutes. After a significant investigation, we struck out in identifying the source, and so we moved on to other concerns.

Then it happened again. And again.

Third time's a charm, and so we began working on improving our ability to monitor our network traffic in fine detail, and narrow down the range of possibilities as to the cause of the disruptions. Was it a failing switch or router? Some sort of misconfiguration? An attack? It was unclear, but we launched a thorough investigation.

The issue returned intermittently. A week would pass, and then we'd have four 5-minute events in a day. And for a time, while we had our suspicions (which eventually turned out to be correct), the specific source eluded us.

But we kept on the investigation.  We determined that something was creating a massive burst of internal network traffic, for a few minutes at a time. So much traffic that one of our core switches would stop responding briefly, causing a flurry of "host is down" alerts, and causing some customer websites and email to stop responding.

Our process involved upgrading and improving the configuration of our NMIS implementation to be sure we were monitoring virtually everything that can be monitored and graphed. We also improved our switches' "storm control" settings, a feature which can temporarily shut down affected network segments in the event of a sudden burst of traffic beyond a normal profile.  And we analyzed packet captures for source, destination, rate, and protocol. 

Once we'd narrowed the issue down to a group of Modwest Grid webservers, we tried to catch it in the act (using tools called tcpdump and wireshark, if you're curious). After a few days, we succeeded. And the new monitoring we'd installed on the webservers created logs of a gigantic burst of outbound traffic associated with one particular customer.

The culprit? The customer's outdated WordPress installation had been previously exploited by a remote hacker to install an attack program, which was being periodically accessed and used to launch a massive UDP packet flood at various external hosts.

We deleted the program, informed the customer, and locked down a few other parts of our network to detect and quaratine similar events in the future.

We try to turn every disruptive event into an opportunity to improve operations, and this case was no exception.  It highlighted the importance of having deep knowledge of your network setup, monitoring, reporting, and graphing -- which we have now accomplished. It also serves as a reminder of the potential impact of ignoring or overlooking those pesky security updates in common web applications like WordPress, Joomla, and Drupal.

At Modwest, we have one day a month we call "Update Day". It's the day on which we go through our list of servers and switches, and ensure everything has the latest patches for security, performance, and functionality.  Consider (or talk to your webmaster about) doing the same -- it makes your upgrades easier and helps keep your site secure.

There will always be bugs, exploits, and attackers. This particular issue is resolved, and we're much better equipped to deal with the next one. Onwards!

Stop SOPA: How could SOPA affect web hosting?

Tweet

by Jennifer Stephens

Today is Blackout SOPA day and we're joining in by blacking out the Modwest logo on Facebook and Twitter.

If you haven't heard of SOPA, here's a summary by Wired: Why We've Censored Wired.com.

As the article states, "Under the current wording of the measures, the Attorney General would have the power to order ISPs to block access to foreign-based sites suspected of trafficking in pirated and counterfeit goods; order search engines to delist the sites from their indexes; ban advertising on suspected sites; and block payment services from processing transactions for accused sites. If the same standards were applied to U.S.-based sites, Wikipedia, Tumblr, WordPress, Blogger, Google and Wired could all find themselves blocked."

Read quotes from the bill in Mashable's op-ed piece, "Why SOPA is Dangerous."

If you want to get involved, here are a number of helpful links:

Blackout your own account at Facebook, Twitter, or Google+.

Blackout your website.

Sign the petition.

Join the growing list of participating companies.

Contact Congress.

Follow @BlackoutSOPA and use hashtag #BlackoutSOPA.

I like Google's simple statement posted at the top of its petition page: Let's "End Piracy, Not Liberty."

UPDATE: PIPA and SOPA Co-Sponsors Abandon Bill

Jennifer Stephens, MBA is the Customer Engagement Manager at Modwest, a Missoula native and Gonzaga University J-school grad who loves a good latte, music, and social media. She can be contacted at jstephens@modwest.com.

Customer Profile: Ednor Therriault aka Bob Wire

Tweet

This month Modwest highlights customer Ednor Therriault. Therriault has been a Modwest customer since November 2004. We're crazy about Therriault's work and happy to showcase him along with his alter ego, Bob Wire. Therriault hosts several websites with Modwest including Bob Wire Music, his blog Bob Wire Has a Point (which also appears on Make it Missoula), and the site for his new Christmas album Off White Christmas.

by Jennifer Stephens

"Let's have a credit card Christmas" wails Bob Wire in the background as I listen to the new album, Off White Christmas, written and performed by Bob Wire and Chip Whitson. "We can start over clean, I'll file Chapter 13 next Valentine's Day." Electric guitar, drums, and a riff reminiscent of "Jingle Bells" kick off the next song, "Santa's Getting Bigger," which entertains while commenting on the state of American health. Chip Whitson's harmony rounds out the country-rock sound, maybe best described as an auditory version of the Griswold's "Christmas Vacation." I can just see Cousin Eddie now ...

Last Thursday, I interviewed Ednor Therriault (pronounced "terry-o"), aka Bob Wire, to discuss his new album as well as his long history with Modwest web hosting. While listening to Off White Christmas, part of me finds it hard to believe that Therriault's stage presence, Bob Wire, is housed in the same body as the creative introverted family-man. Therriault's description of Bob Wire as his "larger than life" id is apt. "Bob Wire is an educator," Therriault explains, "who says what everyone's thinking but is afraid to say." Therriault is friendly, gregarious and eager to share his story about how he found his way to Missoula, why he's still here, and how Modwest helps him to promote his multiple business faces.

A Marine Corps pilot's son, Therriault moved all over during his childhood but had Montana connections which eventually drew him to the state. While attending the Seattle Art Institute in the mid-80s, Therriault connected with fellow non-traditional student Tim Ryan, from Montana, due to their common age (25-year-olds in a sea of 18-year-olds) and Montana connection. Through Ryan, Therriault cultivated many new Montana friendships. By the early 1990s, he says "Seattle had lost its small town feel due to an influx of gangs, drugs, and guns at school" and by then Therriault had more friends in Missoula than in Seattle. "Assured that [he] would eventually outgrow Missoula," Therriault moved to Missoula in 1993 which Therriault describes as "putting on an old worn leather jacket that perfectly fit." Within two months of moving to Missoula, Therriault met his wife-to-be while playing in his new band Betty for Sheriff. 

In 1995, Bob Wire and the Fencemenders was formed. Since he was the front man, Therriault was often referred to as "Bob" and the name stuck. Since then, Therriault's whole family has taken on "Wire" names: Barb Wire (Bob's wife), Rusty Wire (Bob's son), and Speaker Wire (Bob's daughter). My favorite is the grandmother: Byrdona Wire. Therriault's wife insists, however, that Bob's portrayal of Barb is not who she is. 

Besides his musician gig, Therriault is an author, graphic designer, and web designer. He is currently working on a novel after publishing his first book, Montana Curiosities, last year in paperback and Kindle editions.

Therriault's most recently hosted website with Modwest, offwhitechristmas.com, promotes Therriault's CD of rock 'n roll Christmas music. An online presence is important to Therriault in order to market his work worldwide. He notes that it's especially important for his CD, "which is full of original music that's an antidote to all the tired old Christmas songs that get trotted out every year."

We wrap up this customer profile with snippets in Therriault's own words regarding why he hosts his websites with Modwest.

Why did you initially host with Modwest?

I like to employ local businesses, but this decision was more pragmatic. With Modwest being right downtown and not tucked away in some cinder block building out by the airport or in some generic office park in Billings, I can walk right into the office and talk to the IT boys, the support techs, hell, chat with the owner about home brewing, if he's around. Accessibility is important to me. That, and the occasional free t-shirt.

Why do you continue to host with Modwest?

They know me. They know my local footprint. They come to my shows, they've bought my book. They have a good sense of the things I'm trying to achieve with my work, and they are unfailingly gracious and helpful whenever I get myself in too deep with website design and management. Which is often.

Describe one specific event that typifies your relationship with Modwest.

This fall, at the recommendation of my good friend at Modwest, Jonathan, I decided to have all my websites consolidated under one hosted account, ednor.com. I would be able to do this using domain aliases and redirects, and save money on paying for all the separate hosted accounts. While I use the computer and internet extensively for my various writing, recording and design projects, I've always been willfully ignorant when it comes to designing and deploying websites. Still, I knew what I was proposing was going to be a complex mofo.

I took two classes this fall at the Lifelong Learning Center in Wordpress and SEO strategy. Now I have just enough knowledge to be dangerous. Like most artists, I can be temperamental and moody. But through all the frustration of launching two new websites and revamping and resituating two others, the guys behind the stick at Modwest customer support went way above and beyond the call of duty every time I emailed/chatted/phoned tearfully for help. Aaron, Jeremiah and Walt basically functioned as a tag-team with my account. They were all intimately familiar with my sites and what I was trying to achieve, and went to great lengths to solve my problems and keep me from blowing my stack. It didn't matter if it was a misplaced database or a forgotten password, they were always eager to provide help and guidance, always with some well-placed, reassuring remarks. Jonathan, John, Tom, everyone I know there is whip smart but also very tuned in to the human aspect of what can be a very nonhuman industry.

A few weeks ago, at the end of his meticulously-crafted, lengthy and illuminating email describing the various behavioral ticks (and solutions to those ticks) of my Off White Christmas site, Aaron said, "I can honestly say that for once I'm looking forward to hearing some Christmas music."

Try getting that from HostGator.

---

Special Note: If you live in Missoula, Montana, you can catch Bob Wire live tonight at The Crystal. He'll be performing a set at the Holiday Variety Show. Billed as "Local Live Family Friendly Fun," the show will feature sets from David Semmens, Eden Atwood, and Bob Wire. Tickets are still being sold at The Bridge through 6 p.m. tonight. Tickets are $12 for adults, $10 for students/seniors/military, and children under 4 are free. The ninety minute show starts at 7:30 p.m. with doors opening at 7:00 p.m.

Jennifer Stephens, MBA is the Customer Engagement Manager at Modwest, a Missoula native and Gonzaga University J-school grad who loves a good latte, music, and social media. She can be contacted at jstephens@modwest.com.

November food drive: Can you help?

Tweet

by Jennifer Stephens

Modwest is holding its fourth annual food drive through Wednesday, November 23, 2011 in honor of the U.S. Thanksgiving holiday on November 24, 2011. 

We would like to invite our customers to join with us during our last week of this event to help us beat our 2009 top donation year of 138 lbs. (62.6 kg) of food donated to the Missoula Food Bank.

If you are close enough to Modwest headquarters, we invite you to drop off your food donations at our office located at 110 Caras Drive, Missoula, Montana 59802 facing the Caras Park Pavilion. Donations can be received between 9 a.m. and 5 p.m. Tuesday, November 15 through Tuesday, November 22 and on Wednesday, November 23 up until 10 a.m.

If you are unable to drop off a donation at Modwest and would really like to participate, we encourage you to hold your own food drive wherever you are. Let us know how much food you were able to donate to your local food bank and, after the holiday, we will let everyone know how much food was donated by Modwest and its customers. Across our nation and the world, there are record numbers of families who need that little extra help. This is a great opportunity to give back to your community, especially if unemployment is high.

Food donation guidelines from the Missoula Food Bank are listed below. For guidelines in your area, contact your local food bank or health department.

Suggested foods:

• Canned meats (i.e., tuna, chicken, salmon)
• Canned and boxed meals (i.e., soup, chili, stew, macaroni and cheese)
• Canned or dried beans and peas (i.e., black, pinto, lentils)
• Pasta, rice, cereal
• Canned fruits

Foods that cannot be accepted:

• Homemade items
• Noncommercial canned or packaged items
• Alcoholic beverages or drink mixes
• Open or used items
• Rusty or unlabeled cans

Thank you in advance for participating in the Modwest annual food drive! We look forward to hearing from you!

Jennifer Stephens, MBA is the Customer Engagement Manager at Modwest, a Missoula native and Gonzaga University J-school grad who loves a good latte, music, and social media. She can be contacted at jstephens@modwest.com.

Use WordPress? Protect your site from hackers: upgrade timthumb.php.

Tweet

by Jennifer Stephens

In mid-August, Modwest published a blog post regarding the timthumb.php vulnerability and how to protect yourself from having this file exploited.

Modwest is still feeling the effects of customers who have not yet upgraded this file to a safe version. Please take a few moments to see if you might be at risk and read about how to protect yourself if you are.

If you use WordPress for your website or blog, you should take the time to protect your site from timthumb.php vulnerabilty which leaves your site open to being hacked.

What can I do about it?

Use the free WordPress Timthumb Vulnerability Scanner, available as a plugin, to make sure that your WordPress site and/or blog are secure. This tool is designed to be easy to use by those without deep technical expertise; don't hesitate to give it a try.

Installation options for this plugin include:

1. Upload the timthumb-vulnerability-scanner to the /wp-content/plugins/ directory (alternatively, you could use the backend WordPress plugin installer, or install directly from the repository);
2. Activate the plugin through the 'Plugins' menu in WordPress; and
3. Visit the "Timthumb Scanner" page under the "Tools" menu.

What's in it for you?

As the WordPress plugin site states, "[The timthumb.php vulnerabilty is] the perfect combination of not so easy to fix for the technically disinclined, and easy to find and exploit for the malicious - resulting in a disastrous number of compromised sites."

When Modwest identifies a customer account compromise, the site must be disabled until the customer can resolve the problem. This protects all our customers from the potential impacts of compromise.

Jennifer Stephens, MBA is the Customer Engagement Manager at Modwest, a Missoula native and Gonzaga University J-school grad who loves a good latte, music, and social media. She can be contacted at jstephens@modwest.com.

Modwest features Jeremiah Goldson at First Friday

by Jennifer Stephens

Jeremiah Goldson is not unlike his comic book heros: quiet and efficient, he keeps his head down and gets his work done when he's in the office. Outside of the office, though, he reveals his alter-ego: tech support by day, Goldson is an artist and graphic novel illustrator by night.

Modwest proudly presents Goldson's work to the public for the first time at its offices during the upcoming First Friday on October 7, 2011.

But who is Jeremiah Goldson?

Name: Jeremiah "rez" Goldson
Age: 34  
Hometowns: Columbia, SC and Seattle, WA
Favorite beer: All of it/the one in front of me.
Favorite music at the moment: Underworld, Burial, Ben Frost, GvsB

Drawing inspiration from science fiction as a child (Blade Runner, Star Wars, Trek), Goldson naturally fell "in love at first sight" with comic books around 12 or 13 consuming works by Jim Lee, Miller, Silvestri, and McFarlane in the early 1990s and later on Hitch, Bachalo, and Frank Quitely. Manga and anime works by Otomo, Shirow, Mamoru Ishii, and Production I.G. naturally fed Goldson as well.

From this foundation, Goldson now works to remove remnants in his art of the work that once inspired him in order to reveal his own style. Current work finds its genesis outside the comic book realm from such areas as fashion, photography, and literature. Goldson describes the style he's developed as "intentionally unpleasant" and categorizes it in the scifi, fantasy, and cyberpunk genres.

Although he pursued formal art training for a while at the University of Montana's art department, Goldson primarily is self-taught and terminated formal training because "[he] already knew what he wanted to do."

Goldson's choice of media is not fancy: black and white illustration done in graphite and pen and ink with a few charcoal pieces were created on standard 8 1/2" x 11" copy paper until he upgraded to an artist sketch book

This image of Goldson's, titled Spinning in Daffoldils, draws from music of the same title by the band Them Crooked Vultures. Goldson explains that he draws much inspiration from music: "anything that makes me feel something real."

Besides stand-alone pieces, Goldson is the illustrator for a new work written by Roel Torres, Rogue Agent Zed, which is slated to be completed by the end of 2011. Goldson describes Zed as being about "a junkie British task force agent; a dystopian future; government corruption. The script reminds me of the X-Files, written by a mid-80's William Gibson, commissioned by Marvel Comics."

Previous projects by Goldson include an older cyberpunk webcomic, Monica Furious, and Goldson confesses that he "sometimes still gets emails" about the story.

Preview Goldson's work online at his website, blog, or on Flicker.

Setting the tone for Goldson's images, Avatar will spin electro, live. You can get a preview by listening to Avatar's radio show The Robot Maintenance Shop.

---

Jennifer Stephens is Modwest's Customer Engagement Manager, Missoula native, J-school grad, and MBA holder who loves a good latte, music, and social media. She can be contacted at jstephens@modwest.com.

Grid Maintenance Scheduled: Monday, September 19, 2011; 9:30 p.m. - 10:00 p.m. MDT

Technical Info:

Scheduled Grid maintenance will occur on Monday, September 19, 2011 from approximately 9:30 p.m. - 10:00 p.m. MDT. Due to two separate storage fail-overs during the maintenance window, some sites may hang for up to five minutes. The stalling will affect web traffic and FTP/SSH access.

Downtime Caused?

Up to two intervals of up to five minutes of hang time may occur during the half hour maintenance window from 9:30 p.m. to 10:00 p.m. MDT on Monday, September 19, 2011.

Questions of comments? Contact Modwest Support

Visit the Modwest Status Page for up-to-the-minute system status and system maintenance details.

Thanks!

Modwest Team

 

Grid Maintenance Scheduled: Wednesday, September 14, 2011; 9:30 p.m. - 10:00 p.m. MDT

Technical Info:

Scheduled Grid maintenance will occur on Wednesday, September 14, 2011 from approximately 9:30 p.m. - 10:00 p.m. MDT. Due to failing a storage server over, some accounts (web/FTP/ssh) will hang for up to five minutes.

Downtime caused? 

Up to five minutes of hang time during the half hour maintenance window from 9:30 p.m. to 10:00 p.m. MDT on Wednesday, September 14, 2011.

Questions of comments? Contact Modwest Support

Visit the Modwest Status Page for up-to-the-minute system status and system maintenance details.

Thanks!

Modwest Team

Missoula Hempfest and Sunday Streets Missoula

What are you doing this weekend?

As the air starts to cool a bit more at night and fall is imminent, Missoulians are stretching out summer for as long as they can.

A proud community supporter, Modwest sponsors both Missoula Hempfest and Sunday Streets Missoula which are both this weekend! Missoula Hempfest will be held on Saturday, September 10, 2011 in Caras Park and Sunday Streets Missoula is on Sunday, September 11, 2011 along Higgins Avenue.

If you're looking for something to do, please stop by and support these local events!

~Jenn

Customer Engagement Manager

Service Disruption Yesterday

Greetings, Modwesterners! Yesterday we had some intermittent service disruptions, so I wanted to give you the short and long version of what happened.

Thursday, as a result of some planned upgrades, our storage system began to show some faults, causing a portion of sites to become unavailable for short periods throughout the day. As you might expect, this service disruption was not part of our plan.  I usually try to explain exactly how much downtime is involved, but in this case, it varies depending on where your site lives on our storage system. At most, some sites may have experienced an hour of downtime, while other sites would have been fine for all but a few minutes.

We've kept our status page updated, but I want to mention that the downtime shown for HTTP (web) service is exaggerated. Modwest sites were not down for over 5% of the day (yesterday or today), as the numbers might lead you to believe.  The uptime calculations are complex and sometimes overly sensitive; Unfortunately for us, that hurts our perceived 6 month average, which had been well over 99.9% until recently.

Want to know more? Here's the technical scoop:

Over the last few weeks, our Modwest Grid has undergone some growing pains, largely due to the popularity of our Yep! hosting plans. We have known that additional storage servers in our cluster were needed in order to account for our continual data growth. However, the data growth outpaced our planned project and unfortunately, that has led to some service interruptions.

Our Storage Cluster expansion project is relatively simple -- we're adding additional storage servers to the cluster, thereby balancing the thousands of customers' accounts across additional pieces of hardware. In this particular case, our bottleneck is drive contention -- not enough spinning disks for the thousands and thousands of requests that our Grid Web Cluster receives. As the Storage Cluster, or server members, slow down, the entire Modwest Grid Web Cluster can be affected.

What are we doing about that? Well, in addition to fine-tuning active processes on the existing storage servers in order to prioritize web traffic, we just completed the hardware and software deployment of a new storage server this afternoon (Thursday, September 8, 2011). We'll be moving some busy accounts over to the newly deployed storage server this evening, which will reduce the disk contention on the other, over-taxed storage servers. This is the first part of the deployment plan. Once it's complete, we are planning to add yet another storage server to the Storage Cluster in order to
provide even more resiliency to our Grid system.

To learn more about how the Modwest Grid works, you can check out our blog post from last year that describes the architecture: http://blog.modwest.com/2010/08/introducing-the-modwest-grid.html

For those of you who experienced downtime yesterday, I sincerely apologize, and I hope you felt informed by our updates and responses by our outstanding support team. Please let me know if you have any feedback.

- Steven, Cofounder