Optus outage caused by ‘software upgrade’

Optus has revealed a routine software change to its network sparked the outage that wreaked havoc for 10.2 million individuals and 400,000 businesses. 

The Australian telecommunications company revealed on 13 November that "changes to routing information” following the upgrade were at the root of the nationwide blackout. 

"These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers," Optus said in a statement. "This resulted in those routers disconnecting from the Optus IP Core network to protect themselves." 

According to Optus, the restoration required a large-scale effort of the team and "in some cases required Optus to reconnect or reboot routers physically" which necessitated dispatching people across a number of sites in Australia. "This is why restoration was progressive over the afternoon," the telco added. 

Optus' network first went down at around 4am 8 November, with early suggestions that the issue was caused by a "border gateway protocol", which is used by internet companies to route traffic. 

Within hours of the outage, which caused train services to go down in Melbourne and scores of businesses to lose EFTPOS access, Optus said it was investigating a network fault that is impacting Optus mobile and fixed customers and apologised for any "inconvenience caused". 

In its latest statement, Optus added: "Given the widespread impact of the outage, our investigations into the issue took longer than we would have liked as we examined several different paths to restoration.  

"The restoration of the network was at all times our priority and we subsequently established the cause working together with our partners. We have made changes to the network to address this issue so that it cannot occur again." 

Given last year's high-profile cyber attack, the telco added: "There is no indication at this stage that the network outage is related to a cyberattack or that customer data has been compromised. To ensure the safety of our customers against scams and fraud, we will not be sending communications about this outage with links." 

Optus has since offered mobile SIM and mobile data plans of at least 200GB; fixed users Internet Turbocharge for $0 and business customers are advised to contact their account manager or Optus Business Centre. 

Following the outage, Optus was criticised for the seemingly slow communications response by CEO Kelly Bayer Rosmarin, with Communications Minister Michelle Rowland stepping in to handle the initial press conference. 

Speaking to the ABC later on 8 November, Rosmarin apologised but clarified that further details would not be available until Optus had conducted a thorough investigation. 

“We had messages out early, letting our customers know of the outage," she told the ABC. "We gave more than seven media updates during the day, I went on a number of radio stations. 

“Unfortunately, when you have an outage like this, the only message customers want to hear is that we’ve restored the network, and so we gave information and updates as soon as they were available."