Preventing the mayhem of a major Telstra outage

Imagine not being able to get money from an ATM, pay for fuel or make food purchases using EFTPOS? How is your boss going to call you into work or tell you to stay home? Should you take your children to school?

How are you going to call for an ambulance if an accident occurs?

How is your business going to function as you cannot use EFTPOS or call suppliers, and how will you contact your employees?

A fire broke out at 4.35am on November 22, 2012 in the vicinity of the maintenance control room in Telstra’s Warrnambool exchange.

What followed can only be described as a catastrophic telecommunications outage and for the people of South West Victoria mayhem ensued for several weeks.

RMIT University has carried out a consumer and social impact analysis supported by a grant from the Australian Communications Consumer Action Network. The research report highlights the devastating impact that the Warrnambool exchange fire had on the local community and the need for an increased focus on the telecommunications reliability and resilience.

The Warrnambool exchange fire has provided strong evidence that Australian society is reliant on telecommunications and for this reason telecommunications should now be considered an essential service. The research identified the need for telecommunications survival planning and telecommunication survival guides were prepared for government, business and families.

Telecommunication outages occur with surprising regularity, though the number of people affected by the South West Victoria telecommunications outage was larger than normal, as was the duration.

The fire and aftermath

By the time the fire was put out, over 60 per cent of the exchange building was affected by the fire, causing terminal damage to key telecommunications equipment and systems.

The fire caused a telecommunications outage that lasted for about 20 days, affecting 100,000 people in a region covering approximately 67,340 square kilometres.

Telstra reported that there were 135 exchange services, 85 schools, 20 hospitals, 27 police stations, 92 fire stations and 14 SES services affected by the outage. Water, power, gas and rail control, and signalling systems that utilised telephone connections went offline until repairs were carried out.

The financial cost to the region was estimated by the State Government to be at least $950,000 a day, however the true cost may never be known.

Telstra and the Department of Communications

In the months following the telecommunications outage Telstra and the Department of Communications issued reports into the Warrnambool exchange fire.

The Telstra investigation, led by David Piltz, highlighted in a final report released last year on March 23 the need for Telstra to take a range of steps nationally to improve fire awareness among employees and contractors, preparedness for loss of vital infrastructure and systems and to improve capability to react in the event of a major telecommunication outage.

The Telstra report included findings that should be of concern not only to Telstra but also to the nation, as Telstra is responsible for a large number of exchanges and other buildings that can be identified as single points of failure in the Australian telecommunications network.

Yes, even the National Broadband Network is being built with key systems located in Telstra’s single point of failure buildings, many of which are old and subject to flooding (Wollongong) and fire (Warrnambool). What this means is the NBN is not as resilient as it should be.

The Department of Communications' “inquiry to learn lessons from the Warrnambool exchange fire” report included a number of suggested actions, such as the need for regular updates while Telstra carried out the recommendations in the Telstra report.

To coincide with the RMIT report, Telstra has released an update saying “of the 22 recommendations made, 11 have been completed and implemented or are no longer required. The remainder are currently in progress with varying timeframes for their completion.”

The study

The RMIT University research team led by Dr Kaye Scholfield, located at Hamilton in Victoria, conducted three surveys, interviews and discussion groups over a nine-month period to ascertain the impacts of the telecommunication outage from a range of perspectives including government, business and individual.

Anecdotal stories around the telecommunications outage continue to be told. Scholfield recalls being told that “in small towns in the affected region publicans came into their own. Pubs are the local meeting place and publicans provide information and help where they can. One of the publicans lent money to people, but some of the locals were too embarrassed to ask for a loan to pay for groceries. Pubs, which often struggle for custom, missed out on Christmas bookings vital for their survival.”

The information gathered included quantitative and qualitative data that was analysed using sampling, keyword and phrase analysis techniques. The survey respondents and participants in either individual or group discussions were distributed throughout the affected region. Survey respondents were predominately from the 41-60 age group (50 per cent) and two thirds were female.

The extent of the telecommunications outage is illustrated by survey response statistics of the broad impact:

• 94.6 per cent of respondents suffered some inconvenience related to a business transaction
• 69.8 per cent were unable, or found it difficult, to make vital purchases such as food or fuel
• 86.5 per cent were unable to communicate with friends or family
• 24 per cent were unable to conduct business and had to shut down (for some period of time)

Over 60 per cent felt that the outage had a negative impact on community life.

Study recommendations

The study recommendations are:

1. Telstra and the Department of Communications implement the recommendations and suggested actions identified in their respective reports and publish the outcomes including the regular updates (Suggested action 1) being provided by Telstra to the Department of Communications.

2. The Department of Communications in conjunction with Telstra and NBN Co carry out a study to improve resilience of telecommunications infrastructure, including the increased use of mobile facilities that can be used to restore services quickly.

3. Incorporate the lessons learnt from the Warrnambool exchange fire into federal, state and local government emergency management and disaster planning processes and publications.

4. Raise awareness of the need for individuals, businesses, community organisations and local government to complete continuity planning that includes dealing with telecommunications outages.

5. Expand the ABC Emergency Service to include warnings and emergency notifications covering telecommunications outages affecting more than 100 people (or communities of less than this number) for one or more weeks. The ABC Emergency service be promoted more frequently on ABC radio, television and online.

The future

Understanding the far-reaching social impacts of a telecommunications outage must be central to impact assessment and to planning for risk mitigation in a society with an ever-growing dependence on telecommunications infrastructure.

Arguably, the social impact of a technology breakdown of the magnitude of the Warrnambool exchange outage is unlikely ever to be thoroughly quantified. What is clearly demonstrated in the data collected during and after the outage is the depth and breadth of the social impact. People were affected in all aspects of their lives.

Distinct, though interconnected, important themes emerging from the data included the impact on work, money, health and safety, communicating, and community life. Important data was also collected on the positive effects of the outage. Information gathered regarding the usage of alternate media and information sources have immediate relevance for planning at individual, family, community, business and government policy levels.

Analyses of the far-reaching social impacts of the outage have relevance to preparedness and risk mitigation for future telecommunications losses -- incidents that will grow in impact as reliance on telecommunications infrastructure becomes ever more integral to individual, business and community life. The ramifications of the outage may continue for a long time yet.

There is a need for the Department of Communications and Telstra to provide further information about the ongoing work to restore the Warrnambool exchange, the audits, reviews, policy and process modifications identified by Telstra and the suggested actions made by the Department of Communications.

While the immediate effects of the Warrnambool exchange fire have now receded, the lessons learnt should not be lost. There remains an unknown level of risk for other Telstra exchanges, particularly major urban facilities (including NBN Co’s facilities).

Mark Gregory was a chief investigator in the RMIT research team.