Telstra managers realised there was trouble when internet searches for ‘mobile outage’ spiked at 4.30am

One of the worst network failures in Australian history was not detected because alarms started going off — they were only monitored during business hours — or because engineers realised a crucial component had reset the date to 2006.

No. Some of the earliest warning signs came from the internet.

Telstra managers began to realise there might be a problem when they saw customers who still had online access searching for “Telstra outage” or similar words at 4.30am.

Beyond the specific technical failures that July 8 morning, the life-threatening network collapse was due to poor culture, according to a picture presented by an independent report published by Telstra on Wednesday.

Staff at the former government-owned telecommunications monopoly lacked the curiosity needed to investigate problems, the report said, two engineers involved were on a mandatory rest break and an absolutely crucial component for running a modern telecommunications network wasn’t treated as a high priority.

“The outage should not have happened,” chief executive Vicki Brady said Wednesday morning. “We let our customers down and we take full accountability for what occurred.”

No one died, even though access to triple zero went down. The person who might have been most inconvenienced was Ms Brady, who suffered a $607,000 reduction in pay even though it was technicians far beneath her who stuffed up.

Work on the chassis

The outage was a serious moment for the company and the country, because it demonstrated the vulnerability of complex modern technology essential for life as we know it.

The report, by a US consulting firm, Technology Audit partners, is filled with telecommunications jargon that makes it difficult for outsiders to understand.

But it provides a detailed explanation of how routine maintenance disrupted a mobile network used by 19 million people for four days.

The problem began a few hours after midnight. The broken or defective power supply to a metal box, known as a chassis, at a Telstra facility in Melbourne was replaced and turned on.

Within the box was a GPS device that told the mobile network the time and date. Being turned on and off reset the device, which was part of a system known as network time protocol, or NTP. NTP tracks time precisely to allow messages travelling at the speed of light to arrive in the correct order.

Even a fraction of a second out can knock the network out of sync. In this case, because the GPS device was missing a software update (which the manufacturer had alerted customers was necessary), the reset didn’t shift the time one second.

It fell out 20 years.

Back to James Blunt

The mobile network should have had other sources of the time. It should not have been tricked into believing it was 2006.

In 2020 Telstra engineers replaced the chassis in Melbourne and redesigned how the NTP worked. Because of a problem with the new box, a second device used to track the time was removed from the system.

A GPS device, which gets the time from satellites, was installed as an emergency backup. It had to be manually turned on.

To protect the system if any device failed, the engineers made another pivotal change.

Instead of assigning a single source of truth of the time, the NTP devices were reprogrammed to look around the network for any time source that appeared accurate.

Last October, when one of the Melbourne clocks began malfunctioning, the engineers did not raise an alarm to have the problem fixed. They came up with a simpler solution: turn on the GPS.

Nine months later, at 2.50am on a Tuesday, that newly reset GPS device became the network’s primary source for the time and date. Without another clock to tell it George W. Bush was not the US President and no one was listening to James Blunt’s You’re Beautiful anymore, the central computer system became confused and overloaded.

People could not make or receive calls, including to triple zero, which Telstra is legally required to provide. They could not send or receive text messages or access the internet.

In other words, Telstra mobiles went dark — a catastrophic failure of an important national service.

Ramifications

Telstra, one of Australia’s biggest companies, has been dealing with the fallout since.

Some 30,000 customers have received almost $1 million compensation. The NTP system has been recognised internally as essential to the network and assigned more staff. Consultants have been hired to help.

The Australian Communications and Media Authority is conducting an investigation and may levy a fine.

Telstra executives have lost $2 million in pay. Ms Brady, an accountant and one of Australia’s top business leaders, has been humbled.

Communications Minister Anika Wells, who has become used to criticism of her performance, took advantage of the opportunity on Wednesday to chastise others. “What this report has unearthed is concerning and clearly shows there have been preventable failures,” she said.

One of the most intriguing parts of the story is what the report refers to as the “two primary NTP engineers” who replaced the power supply to the box were doing when alarms started going off around 4.20am.

They were, according to the company, inside the building on a break required under a workplace agreement with their union.

“Our team members completed their assigned work and began a rest break, in line with our enterprise agreement and our health and safety commitments to our people,” Telstra told The Nightly.

“This is standard practice for those working night hours on scheduled work.

“As the outage became clearer in the morning, they were called back in to support our response. This is a recognised exception to our standard rest break arrangements. Our processes are designed to look after our people, while keeping the flexibility to call on them when they’re needed, which is what happened here.”

No reasonable person would deny a worker on the night shift a tea break.

But if you want to refer to yourself as an engineer, a professional role, then maybe you should stick around to ensure your work isn’t going to embarrass yourself, your employer or your industry.