USA - Omaha 1 Datacenter Incident Report for April 15, 2013

Start time: April 15, 2013, ~ 5:17 PM CST (GMT-6:00)

Conclusion: April 15, 2013, ~ 6:07 PM CST (GMT-6:00)

Duration: ~ 50 minutes

Type: Regional, affecting multiple datacenters throughout the Midwest USA

This information has been provided by the regional service provider serving the affected datacenters and includes additional actions to be taken by PowerDNN.

Multiple regional datacenters throughout the Midwest US were temporarily unavailable, either partially or completely, to each other and to the internet.  This included one of the PowerDNN datacenters in Nebraska.  Internal servers and other hardware were not affected and neither data nor security were ever at risk.

A regional Core switch servicing the Midwest United States experienced a hardware failure.  Due to the nature of the failure (concurrent supervisor card failure) the redundant hardware member did not immediately fail-over as designed.

The failed hardware was physically removed from the system and the redundant Core switch was rebooted, immediately restoring all connectivity.  New hardware has already been installed to return the Core switches to full redundancy (N+1).  No further service interruption is anticipated related to this maintenance.

The service providers are working with CISCO TAC to further diagnose the causes of both the Core switch hardware failure and the apparent faulty supervisor card to prevent any recurrence.

This appears to be a second occurrence of substantially similar issues impacting PowerDNN's Omaha 1 datacenter due to external sevice providers.  PowerDNN has determined that additional preventive steps are warranted.

To that end, PowerDNN has contacted fiber bandwidth providers (distinct from our datacenter providers) for bids to run separate connections into the PowerDNN network for both traffic load balancing and fail-over purposes.  This will ensure that our external connections at the Omaha 1 datacenter become redundant in vendor as well as in hardware and provider.  This will be a substantial investment, but we believe that it is important to ensure that we continue to provide the highest levels of service.

Even though this incident existed outside of the PowerDNN network, we understand the difficulties this presented for you and your customers.  We appreciate your continued support.

Jeff Hardy

Communications