Australia's internet hit hard by massive Malaysian route leak

A misconfiguration at Telekom Malaysia late last week caused large-scale routing issues which cascaded through providers in Oceania and elsewhere in the world, resulting in customers experiencing severe network problems.

The issue started at at 6:43pm AEST last Friday when Telekom Malaysia made changes to its routers using the border gateway protocol (BGP), which is used by providers and networks to decide how data traffic flows via prefixes inserted into the internet routing table.

The system of sending traffic route announcements over BGP between providers is largely trust-based, and has experienced several mishaps as well as attacks in the past.

Monitoring site BGPmon said Telekom Malaysia mistakenly announced a large number of around 179,000 prefixes for traffic routes via its network to backbone provider Level 3, which wrongly accepted them. The global internet routing table currently has around 534,000 entries.

Level 3 then forwarded the routing prefixes to its peer networks as well as customers - meaning traffic from Level 3 and its associated networks was sent via Telekom Malaysia, which signalled it could accept the data and transfer it to other providers.

Telekom Malaysia's network buckled under the huge amounts of traffic sent to it from Level 3 and other providers, resulting in severe performance degradation for customers, BGPmon noted.

Graph from BGPmon showing a large increase in BGP route updates on Friday.

The "prefix hijacking" lasted for two hours and caused problems for users worldwide, but was most keenly felt in Australia and New Zealand where customers experienced site and service timeouts and connection failures, as well as packet loss and greatly increased round-trip times.

Telekom Malaysia took the blame for the large route leak and apologised for the error.

"We would like to clarify that during a network reconfiguration exercise, we had had unintentionally updated traffic routing information which caused congestion and packet loss to our international connectivity," the Malaysian incumbent said in a statement.

Level 3 also acknowledged the problem, but did not detail why it wrongly accepted the routing prefixes from Telekom Malaysia initially.