An independent investigation into the Telstra outage that occurred in July has determined that the incident stemmed from the replacement of a faulty power supply, which was mistakenly reset to an incorrect date.
The investigation revealed that alarms intended to alert staff about errors were not monitored after business hours, and there was an insufficient number of knowledgeable personnel available during that time.
Warnings Ignored
Despite prior warnings regarding vulnerabilities in its systems, Telstra did not adequately prioritize addressing these issues, which eventually led to a widespread outage affecting approximately 45 percent of calls and data sessions on its mobile network. This incident resulted in hundreds of failed connections to Triple Zero (000), the emergency service number in Australia.
The independent audit, conducted by Technology Audit Partners (TAP), highlighted that a lack of oversight and accountability for critical network components played a significant role in the outage. Notably, two key engineers were on mandatory leave when the incident occurred.
The outage began at 2:50 AM, nearly an hour earlier than Telstra had previously reported, following the replacement of a faulty power supply that caused a GPS card date to revert to 2006. This error cascaded through the mobile network.
The report indicated that the alarms that are typically monitored during business hours were overseen by a limited number of staff, resulting in significant delays—several hours—before the root cause of the outage was identified.
“The problems arose from several factors, including unclear responsibility for Network Time Protocol (NTP) management, lack of visibility regarding the planned replacement of the Melbourne chassis that night, and insufficient understanding of the alarms that signaled early network issues,” the report stated.
Furthermore, it was noted that Telstra did not consider its timing system a high-priority function within the network.
Telstra’s CEO, Vicki Brady, acknowledged the report’s findings and confirmed that the company is actively implementing its recommendations. “The causes we identified align with what we communicated at the time. This incident was not due to infrastructure failure but rather due to an undocumented design change made in October and the absence of a software update on a GPS card,” she explained during a media briefing.
Brady admitted that the company failed to treat its timing system as a critical capability that required high-level prioritization, which she recognized as a significant oversight. “We should have given this the attention it deserved,” she stated.
In response to the findings, Telstra has already begun reallocating resources to enhance the team’s capabilities.
The report follows earlier revelations by the ABC that warned Telstra had received alerts from federal government agencies and academic institutions regarding its vulnerability to such errors well in advance of the outage.
In 2024, the Cyber and Infrastructure Security Centre issued a public warning about Australia’s reliance on positioning, navigation, and timing delivery for critical infrastructure. An earlier alert in October highlighted that vital communication systems depend on timekeeping infrastructure, including satellite systems, and emphasized the legal obligations for providers to have backup systems in place.
During the same month, Telstra was addressing a timing issue in which one of its clocks in Melbourne frequently lost connectivity with another clock in Sydney, triggering NTP alarms. The report noted that a “Network at Risk” ticket should have been created to investigate the persistent issues affecting the Sydney source, but this did not occur. Personnel involved at the time failed to recognize the significance of the Melbourne NTP at stratum 5, resulting in no problem ticket being generated for further examination.
Reflecting on the situation, Brady acknowledged that there could have been greater inquiry into the circumstances surrounding the incident, particularly regarding the undocumented design change made in October of the previous year. “That is a lesson we must learn from,” she said.
Additional Preventable Failures
The TAP report also revealed that eight more Triple Zero calls were unable to connect during the July outage, in addition to the 604 calls previously reported to a Senate inquiry. Of these additional calls, five were classified as misdials or did not require assistance, one succeeded in connecting through an alternate network, one caller managed to connect after placing a second call, and one caller could not be identified despite police follow-up.
Brady confirmed that welfare checks have since been conducted for the eight calls, emphasizing that there were no adverse outcomes reported. “Recognizing the critical nature of Triple Zero, we prioritize rapid response to any issues that arise to reinforce the resilience of our processes,” she stated.
Communications Minister Anika Wells expressed her expectation for Telstra to promptly act on the report’s findings and recommendations. “This outage should not have occurred and resulted in significant disruption nationwide, affecting individuals, businesses, and transportation. The report highlights concerning preventable failures,” she remarked. “Australians rightfully expect to be able to reach Triple Zero when necessary.”
