STL - POD02 Downtime Incident.

Affected Services:
Customers hosted in location St Louis, limited to compute POD02.

Date and Time Issue Start:
27/March/2024 03:19 UTC

Date and Time Issue End:
27/March/2024 09:40 UTC

Root Cause:
High Availability (HA) network did not engage as designed

Contributing Factors:
Management Interface accessibility proved to take longer than setup was designed to facilitate

Future Corrective Actions

• Deploy a new secondary routing device with enhanced failover capabilities
• Implement proactive testing of HA devices to validate performance under real-world conditions
• Expand monitoring coverage to detect and mitigate potential failures earlier
• Optimize network design to improve management access during unexpected events

Management Notes:
We sincerely apologize for the disruption caused by this outage. We understand the impact downtime has on our customers, and while such incidents are rare we take them very seriously. Our team is committed to continuously improving our infrastructure to prevent similar occurrences in the future. The corrective actions outlined above will enhance our networks resilience and ensure a more stable and reliable service.