Blackout warnings came too late over telephone hot line
NEW YORK -- Warnings of trouble in the sprawling electric power grid ahead of the nation's worst blackout came too late, or not at all, over a telephone hot line network created to prevent widespread breakdowns, power officials and politicians said Monday.
Investigators were probing the role of the low-tech system of phones put in place to disseminate information between regional power groups to avert just such a crisis.
Failures in Ohio transmission lines prompted at least three conversations last Thursday between FirstEnergy Corp., the utility that owns them, and the industry group that manages transmission across much of the Midwest, said Mary Lynn Webster, a spokeswoman for the Midwest Independent Transmission System Operator.
Webster said the Midwest group initiated the first contact with FirstEnergy. The group is reviewing recordings of those conversations and computer logs and other data.
Those electricity transmission line failures, which began around 3:06 p.m. EDT, are a focus of the investigation and suspected as the blackout's possible starting point.
The failures were snowballing into systemwide disturbances by the time the Midwest group spoke with its counterpart in Pennsylvania, said Bob Hinkel, general manager of PJM Interconnection, which manages power across much of seven states and the District of Columbia.
"We know that as the specific events began to unfold we had some conversation with them about what we were observing in our system," Hinkel said. "The immediate response is you call the other operator."
FirstEnergy spokesman Todd Schneider said he could not provide any information about the conversations.
Premier Ernie Eves of Ontario complained that U.S. power managers did not notify their Canadian counterparts about the problems, either, as required under protocols developed after a 1965 blackout across much of the same region.
And many individual utility companies said they had little or no indication of problems in the system before their own facilities shut down.
A timeline of power problems compiled by the North American Electric Reliability Council, an industry group investigating the outage, showed power swings in the Eastern U.S. and Canada by 4:08 p.m. EDT and a series of line failures in Michigan starting about nine minutes later.
In Lansing, Mich., employees of the Board of Water and Light noticed irregularities on the grid, but nothing that seemed unusual. Spokesman John Strickler said any information about problems elsewhere would have helped the utility cope with the blackout, which reached its peak region-wide around 4:11 p.m. EDT.
"It looks like stuff started about an hour before this hit us. It would be nice if when the first system went down that there was an alert," he said. "We had no idea this was coming."
Independent system operators across the country are tied together by a system of telephone hotlines. The system was put in place after the 1965 blackout to provide individual operators with vital information about events occurring outside their jurisdictions. Each system is so complex that operators are dependent on their counterparts to share information.
"It's certainly crucial that those two parties are talking to each other," Hinkel said of ties between his group and the Midwest. "If you see changes but you don't know what's going on, the immediate response is you call the other operator."
Many phone lines are recorded. Those recordings, and logs of calls in and out of affected control rooms, will become key to the simultaneous investigations carried by industry groups, Congress and federal and state task forces.
"That's an integral part of this," said Stephen Allen, spokesman for the Northeast Power Coordinating Council, which is examining the failure in New York, New England and parts of Canada. "We will be looking at hardware, software and people."
The timing and content of the conversations preceding the blackout remain unclear, Hinkel and Webster said. Hinkel said the conversation between the Midwest and mid-Atlantic groups may have come seconds before the onset of the blackout.
"From the time when lines tripped in the PJM system and we lost some generation to the point in time when the collapse was all over, it all happened in roughly 11 seconds," he said. "In that period of time there's very little a human being is going to do to arrest that activity."
The Midwest power system recorded numerous voltage swings as early as midday Thursday, long before FirstEnergy's high-voltage lines failed south of Cleveland. Technicians were looking at the system reports "to determine what relationship if any (the swings) may have had with what occurred," said Webster of the Midwest power group.
The finding, if a connection can be shown, would support contentions by FirstEnergy that the system was in distress long before its lines "tripped" resulting in a reduction of power flowing through the system.
"What happened ... is much more complex than a few tripped power lines in our system," FirstEnergy spokesman Todd Schneider said. He said that the utility's data showed unusual conditions, including strange fluctuations of voltage, in the Midwest grid "as early as noon" Thursday, more than three hours before the FirstEnergy lines failed.