At 7:10 AM PST April 27 we detected that inbound calling to DID’s on the Thinktel, Distributel, and Navigata networks was unavailable. We reached out to the carriers and received sporadic information about a major national outage. Our network operations team quickly tested and confirmed that traffic to these networks was failing and we re-routed outbound calling accordingly, but unfortunately inbound numbers can only reside with one carrier and any number residing with these carriers was affected. At this time our testing confirmed that this was a Telco only issue and that IP trunks to both their Edmonton gateways and Toronto gateways were still responding to traffic.
At 8:01 AM PST our network operations team that monitors our carrier gateways observed both the Edmonton and Toronto IP gateways for these networks enter a hard down state and did not come back up until 8:57 AM PST and 9:14 AM PST respectively.
At 8:59 AM PST our team observed that the Safecall* failover system was now operational and that some customers were receiving calls. At 9:10 AM PST all DID’s were reported to be tentatively operational pending testing. At 9:20 our team confirmed operational status for all affected DIDs.
We are currently pressing our vendors for a full incident report and will pass along updates as we get them. We will also be reviewing any possible mitigation strategies internally.
*Safecall is a system whereby if a carrier loses IP connectivity an affected DID will be redirected to a DID on another carrier automatically through the traditional telco network. Unfortunately, in this case there was a total failure on both the telecom side and the IP side resulting in a loss of service.
We sincerely apologize for any inconvenience this may have caused and please know that we are working hard to mitigate any future issues.