ThePlanet Outage Redux

ThePlanet.com is experiencing issues again and www.ncsxshop.com is offline for the time being. Our backup shop at www.shopncsx.com is functioning properly. The latest updates from http://service-update.theplanet.com/ are:

June 3 – 7:30am CDT
As Doug mentioned in his audio message last night, "The explosion and electrical fire damaged, beyond repair, the electrical gear where the utility service enters the building as well as the transfer switch and main distribution panel that feeds the first floor of the data center."Because the transfer switch and distribution panel were damaged beyond repair, we are running H1 Phase I from a temporary generator, while Phase II is being powered by our permanent generator. We tested the temporary generator extensively prior to bringing it into service, and we did not find any indication of the faulty breaker.Our facilities group is working with the generator contractors to repair the faulty breaker as soon as possible.

June 3 – 6:38am CDT
Around 2:20 AM CDT, the backup generator being used to power H1 Phase experienced an electrical issue resulting in service loss for Phase I; Phase II remains unaffected at this time. Our data center operations and facilities teams immediately began investigating the cause of the failure to restore power to the Computer Room Air Conditioner (CRAC) units and Power Distribution Units (PDUs) for Phase I.The staff successfully tested the 2 megawatt generator without load, so they began powering up the CRAC units and PDUs to restore service to Phase I.

While working through this power restoration, the generator's breakers were tripped by their internal electronics. The generator is rated to handle more than the load required to power the phase, and the generator itself is fully functional, but the breaker system must be replaced to guarantee stable power distribution.

We have attempted to locate a replacement generator and are evaluating the time necessary to repair the breakers on the current generator so we can restore power as quickly as possible. We do not have an ETA for power restoration, but we will be updating you hourly with our current status or sooner, as developments warrant.

June 3 – 4:41am CDT
CRAC units are back online. The facilities and data center operations teams are verifying the stability of the generator, and they will restore power to the PDUs as quickly as possible.

June 3 – 3:33am CDT
The Customer Access Routers in H1, Phase 1 have been affected by the generator issues. While customer servers in racks may be powered on, they will not be accessible until the access routers have power restored.

June 3 – 2:54am CDT
The backup generator issue is affecting around 1/2 of phase 1. From most recent reports, servers have power, but until the issue is resolved, they may not be accessible.

June 3 – 2:25am CDT
Due to an issue with one of our backup generators, we've noted inconsistent power distribution to our CRACs (air conditioning units) and PDUs. Because these key components are fundamental to server racks, customers may note some downtime currently.We have our data center operations and facilities teams checking the generators, CRACs, PDUs and racks to restore connectivity. UPDATE: This issue appears to only affect phase 1.