The second backup generator was installed at the data center yesterday evening and power was returned to the servers. Our site at www.ncsxshop.com is currently running and has been up since 5:38PM EST yesterday without interruption. There are still email issues however.
We've been with Rackshack and then EV1 Servers and now ThePlanet since 2002 when the "new" shop was built and released into the wild. It's been a great ride for all those years but this recent problem and a short server outage on www.ncsx.com from last Friday has got us thinking about selling the company to a larger operation. We don't have the onsite expertise to provision quickly against this type of critical issue when it happens and maybe it's time to relinquish the reins a bit and begin the search for a partner.
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.I am sorry that this accident has occurred and I apologize for the impact.
-Kevin Hazard
The second backup generator was installed at the data center yesterday evening and power was returned to the servers. Our site at www.ncsxshop.com is currently running and has been up since 5:38PM EST yesterday without interruption. There are still email issues however.
