Tuesday, June 3, 2008

Houston, We Have a Fire

The explosion and fire in Planet.com’s Houston, Texas, data center facility over this past weekend served as a scorching reminder of the importance of a strong backup plan.
Houston Fire Department officials said that the Planet.com Internet Services data center, where it creates, hosts and maintains Web sites for its clients, was rocked by an explosion in a network gear room. Apparently, no servers or networking equipment were damaged, and no one was hurt, but power was cut to the facility, affecting about 9,000 servers. The blast was strong enough to push three walls of the facility out of place. The incident was attributed to an electrical problem with a transformer. With an estimated 7,500 of its clients hoping the issue comes to a quick resolution, Planet.com is putting its recovery plan into action. Some servers will be relying on generator power for a week until normal utility connections are restored, according to Douglas Erwin, Planet.com’s CEO.
One of the interesting things that Planet.com employees are doing is providing updates on the data center’s progress through an online forum. This is seen as an important part of the disaster recovery plan. The latest updates say that the company is doing a rack-by-rack check for any servers that require technical support. Erwin said that 6,000 of the 9,000 servers have been restored, and the next step is to rebuild the electrical room, which will take place in the next week or so.
This fire certainly serves as a reminder that a good solid backup plan is critically important. Of course it is impossible to prepare for everything, but certain steps can be taken to assuage problems. Planet.com, for instance, added a backup server in March with continuous data protection. Turns out it wasn’t such a bad idea.

-- Jeff Feinman

No comments: