Ten years after epic blackout, US power grid smarter

It was just another hot August afternoon for New York electricity grid chief Rick Gonzales, as he sat in the control room helping to make sure the state's eight million homes had enough power to keep the lights on and the air conditioning running.

While the temperature pushed toward 90 degrees Fahrenheit, the grid serving New York City and the rest of the state of 20 million people wasn't under any strain, Gonzales recalled, as the clock ticked past 4pm on the East Coast.

But 500 miles away, a few benign tree branches were about to render Gonzales' territory – and that of 30 million unsuspecting others – powerless.

"In less than five seconds we went from normal operations to most of the state blacked out," Gonzales said recently.

"Alarms were ringing and our computers started flashing red. We didn't know if we were getting bad information or if this was a true blackout."

They soon found out. The Northeast blackout of August 14, 2003, was the biggest in North American history, and laid bare the vulnerabilities of an aging electric grid in the most populous region of the United States.

More than 50 million people from Ohio to Ontario would lose power, at an estimated cost of almost $10 billion. Coming just two years after September 11, the blackout reignited fears of what a targeted attack could achieve if, as an official report eventually concluded, a few untrimmed tree branches in Ohio could do that much damage.

Ten years on, electricity experts say a similar event is far less likely to happen in the US Northeast. Grids and power companies have overhauled their operations, and regulators now have the authority to punish any company that doesn't adhere to rules covering everything from maintenance schedules to how close tree branches can get to power lines.

Of course as Hurricane Sandy demonstrated last year, "We'll never be able to eliminate (blackouts) entirely," said Jay Apt, a former NASA astronaut and physicist and director of the Carnegie Mellon Electricity Industry Center. "The frequency of large and small blackouts has not really changed since we became good at keeping statistics on them 30 years ago."

And while grid operators learn lessons from every major blackout, they face diverse threats, including attacks on cybersecurity.

Mandatory standards

Eight months after the blackout, a joint US-Canada task force issued a report on its causes along with recommendations for the prevention of future cascading power outages.

The power industry and energy regulators have since spent hundreds of millions of dollars on new monitoring equipment and improved computer systems, in addition to adopting new standards and improving worker training and equipment maintenance.

To make sure power companies follow the rules, the 2005 Energy Policy Act gave the Federal Energy Regulatory Commission (FERC) the ability to impose fines of up to $1 million per violation per day for failure to comply with the standards.

In 2009, Florida Power and Light, the biggest electricity company in the state, paid a $25 million settlement over FERC allegations that one of its workers had caused a blackout the previous year by not following proper procedures.

"That change from voluntary to mandatory reliability standards is something we strongly supported since we were impacted by the lack of clear rules," said Gonzales, who is now the chief operating officer of the New York Independent System Operator (NYISO), which runs the state's power grid.

The task force determined that Ohio power company FirstEnergy Corp triggered the 2003 blackout because it didn't understand how vulnerable its system was.

A failure in FirstEnergy's computer system was compounded by human error after a technician called in to fix it forgot to turn back on a key application for monitoring the grid.

At the same time, the Midwest grid operator was experiencing its own computer problems. The two together were essentially flying blind, and did not detect the shutdown of several power lines that had been hit by wayward trees.

With no reaction from FirstEnergy or the grid operator, other lines quickly became overloaded. What followed was a power surge that grew large enough to jump state lines, first knocking out power in Michigan, rolling through Ontario and ultimately knocking New York offline. The whole event took little over an hour from when the first branch hit the power line in Ohio.

A spokesman for FirstEnergy, one of four investor-owned power companies in Ohio, said it has made efforts to strengthen the Ohio power grid. "FirstEnergy has added new facilities and equipment, enhanced operator training, and implemented new standards that are mandatory and enforceable in order to help enhance the reliability of the transmission grid," said Mark Durban.

Since 2003 Ohio has had numerous widespread blackouts caused by snow and thunder storms, but the number of outages caused by faulty equipment or human error has been falling, according to a company that tracks blackouts and their causes.

Eaton, a power services company, said in its annual report that Ohio had only 28 blackouts last year due to equipment failures or human error. While the state still ranks seventh in the United States for power outages, it was second only to California in 2009, when it had 42.

Sharper skills through simulations

Gonzales said that in 2003 there was no early warning that could have made a difference.

"We had no idea something was wrong with the power system in Ohio. It was a normal, warm summer day."

To help speed the recovery, the grid operator relied on just "one guy in a truck" navigating his way through congested traffic to reconnect the state to the wider grid at a substation just north of the New Jersey border, where they still had power.

New York utilities restored power to about two-thirds of homes and businesses by 8am Eastern time Friday, about 16 hours after the blackout started. Service was completely back after about 30 hours.

Gonzales says the same lack of information from neighboring grids would be improbable now. New technology allows the grid operator to take far more sophisticated measurements of its own system's power flows as well as neighboring systems', and faster than before.

Apt, at Carnegie Mellon, said training of grid controllers now includes frequent test runs to sharpen their skills in the event of a major disruption to the grid.

"We're doing a lot more simulation now," he said. "It's like what they say about airline pilots – it's years of boredom interspersed with moments of stark terror."

Cybersecurity challenges

While the grid is now seen as less susceptible to human error, malicious threats have risen.

Cyber intrusions and attempted attacks on grid computer systems have increased dramatically over the last decade, according to the US Department of Homeland Security, raising fears of both widespread power cuts and threats to nuclear facilities.

Gerry Cauley, president of the North American Electric Reliability Corp (NERC), the agency enforcing grid standards, said the power industry has made significant progress to address cybersecurity threats since he returned as president in 2010.

"I walked in the door and I felt a black hole of worry, risk and problems in terms of security. We were not in good shape at NERC or in the industry in terms of understanding what we should be doing and where we needed to focus," Cauley told an industry meeting earlier this year.

"We have been able to turn that around. It has become a way of life for us," Cauley said.

The power industry has put mandatory cybersecurity standards in place, including the requirement to share information quickly between grid operators, companies and regulators in the event of an attack. The US government is also looking at implementing its own rules.

The other big challenge is the changing way the United States generates power.

Since 2003 enough coal plants to power 15 million homes have shut down or converted to natural gas as their primary fuel source. That has left some grid operators questioning if they will ever face natural gas supply squeezes on the coldest days when they compete with energy companies providing gas to heat homes.

Renewables, like wind and solar power, also present some difficulty because they're less predictable. Grids and power companies are now experimenting with ways of cutting demand at times of peak consumption.

PJM Interconnection, which oversees a grid serving 61 million people in 13 parts of states from New Jersey to Illinois, uses so-called demand response programs, which compensate large companies and consumers for cutting back the power they use when the grid is under strain.

That can reduce electricity use for a short period of time by as much as 10,000 megawatts – the equivalent of enough electricity to power New York City on an average day.

PJM is also experimenting with using giant batteries, and programs that may enable owners of electric vehicles to sell power back even as they are charging their cars.

Chantal-Aimee Hendrzak, PJM executive director of system operations support, said the grid is looking at integrating a host of unconventional electric technologies.

"It does not have to be something new," she said. "Batteries, water heaters, flywheels, electric vehicles can all respond in less than a second to add (or remove) power. The more devices that can be used to balance the grid, the fewer big power plants have to be held in reserve."

-- Reuters