USA Datacenter Incident Report - February 22, 2013

Start time: February 22, 2013, 8:50 AM CST (GMT-6:00)

Conclusion: February 22, 2013, 10:05 AM CST (GMT-6:00)

Duration: ~75 minutes

Type: Regional, affecting multiple datacenters throughout the Midwest USA

This information has been provided by the regional service provider serving the affected datacenters.

Impact
Multiple regional datacenters throughout the Midwest US were temporarily unavailable, either partially or completely, to each other and to the internet.  This included one of the PowerDNN datacenters in Nebraska.  Internal servers and other hardware were not affected and neither data nor security were ever at risk.

Root Cause
A regional Core switch servicing the Midwest United States experienced a hardware failure.  Due to the nature of the failure (concurrent supervisor card failure) the redundant hardware member did not immediately failover as designed.

Resolution
The failed hardware was physically removed from the system and the redundant Core switch was rebooted, immediately restoring all connectivity.  Spare hardware will be installed to return the Core switches to full redundancy (N+1) tomorrow, Saturday, February 23, between 3:00 and 4:00 AM CST (GMT-6:00) to complete.  No service interruption is anticipated related to this maintenance.

Preventative Action
The service providers are working with CISCO TAC to further diagnose the causes of both the Core switch hardware failure and the apparent faulty supervisor card to prevent any recurrence.

Even though this incident existed outside of the PowerDNN network, we understand the difficulties this presented for you and your customers.  We appreciate your continued support.

The PowerDNN Team