USA Datacenter Incident Report
Start time: February 22, 2013, 8:50 AM CST (GMT-6:00)
Conclusion: February 22, 2013, 10:05 AM CST (GMT-6:00)
Duration: ~75 minutes
Type: Regional, affecting multiple datacenters throughout
the Midwest USA
This information has been provided by the regional service
provider serving the affected datacenters.
Impact
Multiple regional datacenters throughout the Midwest US were
temporarily unavailable, either partially or completely, to each other and to
the internet. This included one of the
PowerDNN datacenters in Nebraska.
Internal servers and other hardware were not affected and neither data
nor security were ever at risk.
Root Cause
A regional Core switch servicing the Midwest United States
experienced a hardware failure. Due to
the nature of the failure (concurrent supervisor card failure) the redundant
hardware member did not immediately failover as designed.
Resolution
The failed hardware was physically removed from the system
and the redundant Core switch was rebooted, immediately restoring all
connectivity. Spare hardware will be
installed to return the Core switches to full redundancy (N+1) tomorrow,
Saturday, February 23, between 3:00 and 4:00 AM CST (GMT-6:00) to complete. No service interruption is anticipated
related to this maintenance.
Preventative Action
The service providers are working with CISCO TAC to further
diagnose the causes of both the Core switch hardware failure and the apparent
faulty supervisor card to prevent any recurrence.
Even though this incident existed outside of the PowerDNN
network, we understand the difficulties this presented for you and your
customers. We appreciate your continued
support.
The PowerDNN Team
Published: February 22, 2013 at 5:22 PM