Counting the cost of a Telstra outage

Early last week the Warrnambool telephone exchange in South Western Victoria was gutted by fire. What followed should worry managers, entrepreneurs and ministers across the country as the consequences of the fire exposed the fragility of communications networks and the importance of business continuity plans to deal with unexpected problems.

The Warrnambool telephone exchange is the hub for much of Telstra's network in South Western Victoria and the fire knocked out 60,000 landlines, 15,000 internet customers and over 80 mobile phone towers. Across the region banks, shops and even the train service came to a halt.

With EFTPOS and ATM services disabled, businesses and households had to scramble to find cash or revert to paper based processing.

Jodi Heath, Regional Executive for the National Australia Bank's Geelong and South West Victorian region, described how her customers and staff at the ten affected branches in region dealt with the outage.

“We made sure we didn't run out of cash and kept the branch open until 6pm at Warrnambool so customers could have money over the weekend.” Heath told Technology Spectator

“I have to say customers have been really good about it, we've only had one or two people upset at the delays and problems.”

One of the problems for merchants was the switch back to processing credit card transactions with paper dockets, which required one of NAB's managers to do a late afternoon mercy dash from Melbourne with a carload of supplies.

Driving to Warrnambool, that manager would have passed many businesses heading the other way to buy Optus SIM cards and modems from stores in Geelong and Ballarat.

The Optus mobile network was largely being unaffected with only some 2G mobile and landline services impaired. The Warrnambool Standard reported the Ballarat Optus store was rushed by affected residents who made a five hour round trip to get hold of pre-paid SIM cards and mobile data modems to keep their business running.

Will the NBN be any better?

Some commentators have used the outage to call for the National Broadband Network (NBN) rollout to be accelerated given that South Western Victoria isn't scheduled to see the new system for at least three years.

The idea that the NBN would be immune to this sort of failures is unfortunately flawed as the high speed network has its own vulnerable points subject to similar risks as the old switched telephone network.

An NBN Co spokesperson told us that “if the actual electronics within the network at a Fibre Access Node (FAN) or at an Aggregation Node (AN) which contains the PoI were damaged, we would need to repair or replace this equipment.”

“We have put in place extensive practices and processes to handle a network outage of this type, and to recover services as quickly as possible. As you know these are very rare events, but contingencies are planned,” the spokesperson said.

The NBN Co's view is reflected by others in the industry, one former Telstra employee in Warrnambool said, “this is a one in a hundred year event and it's impressive how business continuity plans have stood up to this.”

Action stations   

For Telstra, their business continuity plans were the first to kick in as their alarm systems reported trouble at the exchange.

Bill Mundy, Telstra's area general manager for South West Victoria told Technology Spectator “we knew there was a big problem before the fire brigade arrived and we had equipment ready to go.”

Telstra also called in services from across the country, including the wonderfully named MEOWs – Mobile Exchanges On Wheels – to provide temporary services in Warrnambool and Hamilton.

“It really has been a ‘Team Telstra' effort to get things working as various arms of the organisation have had to come together to fix the problems,” Mundy said.

Five days after the fire the efforts of Telstra's staff have been bearing fruit with 80 per cent of landline services restored and priority premises like nursing homes and schools reconnected but there is no news on when ADSL services in the Warrnambool district will be back in action.

A good case for robust continuity

That many businesses like Telstra, NAB and dozens of smaller enterprises were able to continue trading reminds us of the importance of having contingency plans for the unexpected.

Despite the outage being a technology problem, the fallout should serve as a lesson for businesses. Simply assuming services like telephones and the web will be available in a crisis might render a business continuity plan useless when it's needed.

Indeed the reports that V-Line had to suspend passenger services into Warrnambool last Thursday because “train drivers had no alternative way to communicate with the control room” should raise the eyebrows of government, passengers and safety regulators.

Repercussions and compensation

For Telstra there are some serious questions about the lack of redundancy in their network and NBN Co and the Federal government will undoubtedly be looking at the design of the NBN, given the disruption a single point of failure caused.

In terms of making good on the losses from the outage, there is some discussion of Telstra making payments to affected customers. However, as any business that's been victim of an IT service outage knows, compensation from Service Level Agreements (SLAs) rarely comes close to covering the costs of disrupted operations. 

While people will be arguing about compensation over the Warrnambool outage for months, wiser folk will be pondering on how we can reduce the single points of failure in our telecommunications networks and how we can harden our business processes against the unexpected.

The widespread outage across Western Victoria is a stark reminder that technology can – and does – fall over. And our businesses, homes and communities have to be prepared for when they do.