Singtel refutes reports that its system upgrade caused Optus outage


Kwarkot/Getty Images

Singtel has refuted reports that a system upgrade carried out on its backbone network caused the massive outage suffered by its Australian subsidiary Optus. 

The “routine software upgrade” on a router was carried out at Singtel Internet Exchange (STiX), which is one of Optus’ international networks that facilitates peering connectivity with global internet networks. 

Also: The top cloud storage services you can buy

This upgrade was planned and prior notice was sent out to affected customers, including Optus and other telecom companies, Singtel said in a statement Thursday. During the upgrade, data traffic was directed to other points of presence on the STiX network and routed back to customers’ networks. 

The software upgrade was completed within 20 minutes and customers’ routers that were connected to it, including Optus’, were up and running, Singtel said. 

“We are aware Optus experienced a network outage after the upgrade, when a significant increase in addresses being propagated through their network triggered preset failsafes. However, the upgrade was not the root cause,” said the Singapore telco, of which Optus is a wholly-owned subsidiary. 

The November 8 service disruption left many without access to telephone and broadband services, affecting more than 10 million Optus customers, including 400,000 businesses. The hours-long outage also brought down payment systems, grounded train services in the state of Victoria, and left emergency call services inaccessible. Services resumed progressively more than six hours after the outage occurred and were fully restored 12 hours later. 

Also: What on earth happened to Cloudflare last week?

The STiX software upgrade was rolled out at 1 am local time, or 4 am in Sydney, on November 8. 

In an FAQ explaining the outage, Optus said its network had “received changes to routing information from an international peering network following a software upgrade” at 4:05 am on November 8. 

“These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers. This resulted in those routers disconnecting from the Optus IP Core network to protect themselves,” the FAQ said. 

Restoring the affected systems involved significant effort from its team, some of whom had to be sent across a number of sites to physically reconnect or reboot…