Connectivity Outage

In response to the outage experienced on the 29th April 2021 in which some customers were impacted, it has been identified that a configuration change inadvertently caused a network loop preventing some network traffic from being resolved correctly.

The configuration changes were reverted after being identified as the cause which saw services mostly restored to operation state.

After the network had stabilised, reports from some SD-WAN customers alerted to network performance degradation to a small number of SD-WAN platform aggregation nodes. This caused slow speed and packet loss to some SD-WAN customers. The root cause was identified and SD-WAN traffic effected was re-routed resolving the packet loss and performance issues.

Event Timeline:

14:45 - Configuration change made created a network loop for some TechPath connectivity services.

15:05 - Network loop removed, stabilising switching traffic for TechPath connectivity services.

16:30 - Slow network performance and packet loss on some SD-WAN services investigated.

17:30 - Issues with SD-WAN packet loss identified as SD-WAN nodes running in Active-Active mode creating packet loss and slow performance for some SD-WAN services.

18:00 - Secondary SD-WAN nodes disabled to remediate packet loss and slow performance.

18:30 - Secondary SD-WAN nodes enabled and monitored to confirm running in failover mode.

The issue has been resolved and has been stable for over 30 minutes.

A final update will be provided tomorrow on the resolution.

TechPath is currently experiencing an issue with multiple services including Internet, Cloud PBX and SD-Wan. We are investigating the issue and will update as soon as possible.