RCA - Networking - Las Vegas
Summary of impact: Between 00:26:24 PST and 00:43:52 PST on 01/17/2019, some tenants may have experienced a disruption in private circuit connectivity to/from the Las Vegas data center.
Root Cause: True North engineers determined that a network switch experienced a firmware panic and rebooted.
Mitigation: True North engineers were alerted to the switch issue and performed an investigation with the hardware vendor. The switch rebooted cleanly and in a stable state.
Next Steps: We sincerely apologize for the impact to the affected customers. We are continuously taking steps to improve the True North Cloud platform and our processes to help ensure such incidents do not occur int he future. In this case, this includes (but is not limited to):
Updating the impacted switch firmware. Updating the matching secondary switch firmware. The firmware updates will take place within the next 2 weeks to allow for additional planning, testing and validation of the planned firmware code.
Additional communication for the planned maintenance will follow.
Provide feedback: Please help us improve the True North customer communications experience by taking our survey. https://bit.ly/2RXJUKg