Shortly after 2:14 on the afternoon of August 14, 2003, the alarm system in FirstEnergy’s control room in Ohio stopped working. Its job was to sound and flash when a significant piece of equipment went from an acceptable condition to a problem. After that, no further alarms reached the consoles. The operators, investigators found, did not know they were working without them.
The operators watched a system that looked calm. At 2:27, a 345-kilovolt line tripped and reconnected. At 2:32, a neighboring utility, AEP, called to ask about it. FirstEnergy had no alarm and no log of the trip. According to the joint U.S.-Canada task force that investigated the blackout, the FirstEnergy operator dismissed the information as either not accurate or not relevant to his system.
Elsewhere in the company, the computers were saying something. At 2:41 the primary server hosting the alarm function failed, and the backup took over. An automatic page went to FirstEnergy’s IT staff. At 2:54 the backup failed too. Another page. The IT staff, the task force found, did not notify the control room operators.
At 3:08 the IT staff finished restarting the primary server. Its startup checks looked fine, and the staff believed they had restored it. The alarm system was still frozen. The IT staff did not check with the operators that it was working again.
The task force put it in one sentence: “for over an hour no one in FE’s control room grasped that their computer systems were not operating properly, even though FE’s Information Technology support staff knew of the problems and were working to solve them.” Investigators reasoned that the IT staff would have known a loss of both servers meant a loss of alarms. FirstEnergy told them its IT staff had been unaware until 3:42, when a control room operator told them about the alarm problem.
Meanwhile the operators talked with field crews, a nuclear plant, neighboring utilities and the regional reliability coordinators, and got pertinent information from each. They, the report says, “did not recognize the emerging problems from the clues offered.” Between 3:05 and 3:41, three of FirstEnergy’s 345-kilovolt lines came into contact with trees and tripped out.
A few minutes after 4:00, the blackout began. It reached an area of an estimated 50 million people in eight U.S. states and Ontario.
The task force grouped the causes in four. FirstEnergy and its regional reliability council had not understood the weak points of its system. FirstEnergy had not adequately managed tree growth along its lines. Regional coordinators had not given effective real-time help. And FirstEnergy had lacked situational awareness, including this finding: its “computer support staff and operations staff did not have effective internal communications procedures.”