Point to Point Connectivity
Incident Report for True North Cloud
Postmortem

RCA - Networking - Las Vegas

Summary of impact: Between 00:26:24 PST and 00:43:52 PST on 01/17/2019, some tenants may have experienced a disruption in private circuit connectivity to/from the Las Vegas data center.

Root Cause: True North engineers determined that a network switch experienced a firmware panic and rebooted.

Mitigation: True North engineers were alerted to the switch issue and performed an investigation with the hardware vendor. The switch rebooted cleanly and in a stable state.

Next Steps: We sincerely apologize for the impact to the affected customers. We are continuously taking steps to improve the True North Cloud platform and our processes to help ensure such incidents do not occur int he future. In this case, this includes (but is not limited to):

Updating the impacted switch firmware. Updating the matching secondary switch firmware. The firmware updates will take place within the next 2 weeks to allow for additional planning, testing and validation of the planned firmware code.

Additional communication for the planned maintenance will follow.

Provide feedback: Please help us improve the True North customer communications experience by taking our survey. https://bit.ly/2RXJUKg

Posted Jan 18, 2019 - 07:51 PST

Resolved
True North is moving this incident into a resolved state pending further permanent remediation efforts outlined in the provided RCA.
Posted Jan 18, 2019 - 07:34 PST
Monitoring
All point to point/private circuit connectivity has been restored. True North Cloud Services is placing this into "Monitoring Status". More information pertaining to this incident will follow.
Posted Jan 17, 2019 - 01:02 PST
Investigating
True North Cloud Services is aware of and investing an issue that may be impacting point to point connectivity into our Las Vegas facility.
Posted Jan 17, 2019 - 00:31 PST
This incident affected: True North Cloud Las Vegas Overall Availability (Las Vegas Private Circuit Connectivity).