This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.I am sorry that this accident has occurred and I apologize for the impact.
-Kevin Hazard
NCS vends games and toys to customers around the world.
Saturday, May 31, 2008
NCSXSHOP down at the moment due to...
Dear Valued Customers,
In the interest of keeping you updated, tonight at 5:45 p.m. here in Houston, a transformer in our H1 data in Houston caught fire, which required us to take down all generators as instructed by the fire department. All servers in the H1 data center are down, as is our ServerCommand customer portal, which are working to move to another data center. None of our other data centers are affected.Our management team and facilities staff are on site evaluating the situation. In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.We are posting messages in our forum, in Orbit and on our customer support phone system.We appreciate your patience. We are working around the clock to resolve these critical issues.This link will give you updates as well:
http://forums.theplanet.com/index.php?showtopic=90185
In the interest of keeping you updated, tonight at 5:45 p.m. here in Houston, a transformer in our H1 data in Houston caught fire, which required us to take down all generators as instructed by the fire department. All servers in the H1 data center are down, as is our ServerCommand customer portal, which are working to move to another data center. None of our other data centers are affected.Our management team and facilities staff are on site evaluating the situation. In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.We are posting messages in our forum, in Orbit and on our customer support phone system.We appreciate your patience. We are working around the clock to resolve these critical issues.This link will give you updates as well:
http://forums.theplanet.com/index.php?showtopic=90185
Subscribe to:
Posts (Atom)