All repair work has been completed on the issues affecting AMS1.
Our network vendor had speculated this was a specific version bug on the firmware the AMS1 core routers were running. This bug theory could not be positively confirmed due to actions taken to quickly restore AMS1 connectivity.
After completing the full upgrade, a traffic diversion was manually triggered that verified the "bad path into AMS1" that had originally caused the US->AMS1 outage to now be working fine and passing traffic as originally expected.
Teraswitch has completed all testing and is considering AMS1 and EU network operations as normalized.
Posted Dec 13, 2024 - 05:10 UTC
Update
On the recommendation of our network vendor, Teraswitch will perform an emergency software upgrade on the AMS1 core network.
Teraswitch expects to start and complete these upgrades within the hour.
During this time, customers should expect to see internet traffic reroutes and possible moments of elevated latency.
Posted Dec 12, 2024 - 18:25 UTC
Monitoring
The fault has been identified as an issue with backbone traffic passing over the Amitie trans-Atlantic fiber cable. Only Teraswitch's AMS1 data center was unable to pass traffic over this backbone link. Traffic had shifted from the normal path onto Amitie during vendor maintenance and subsequent loss of the primary path. This caused a disruption of AMS1 communicating with US-based or US-routed internet traffic.
At this time, Teraswitch is still working with vendors to understand the specific issue with our network equipment, the Amitie cable, and AMS1.
AMS1 is operating in a mode that it will not use the Teraswitch backbone to eliminate the risk of any more traffic impacts.
Posted Dec 11, 2024 - 14:49 UTC
Update
We are continuing to investigate this issue. AMS1 sustained a brief outage around 22:05 UTC, but has since recovered and is stable at this time. We will provide updates as more information becomes available.
Posted Dec 10, 2024 - 23:31 UTC
Investigating
Teraswitch is investigating Internet connectivity issues at AMS1. We will provide an update shortly as more information becomes available.
Posted Dec 10, 2024 - 22:10 UTC
This incident affected: AMS1 - Amsterdam, Netherlands.