California blackout

[topgroove]

Looks like the blame is gonna land on one poor bastard. Early reports are blaming a single utility worker for switching out a transmision capacitor bank. The line tripped about ten minutes after the cap bank was taken off line. I hope to god he was following a switching order. The blackout affected over five million customers in california,arizona, and mexico. Two states and a neighboring country.......oopsie.

[Pootnaigle]

Ummmmmmmmm I bleve he may just be OK kause of all the media coverage. But I spect he is gonna have a tight ass anyway.I'm purdy sure he would have been workin with a switchin order and if he followed it he can point the finger at whoever wrote it.Aint much freelancing stuff goes on in the transmisson side.And aint nobody in their right mind wants to open a capacitor anyways.

[topgroove]

I here ya poot. Right now there is a room full of suits figuring out how to word this thing. Ferc will be involved.looks like the cost of this thing will be over one hundred million dollars.

[CPOPE]

The blackout was concentrated in San Diego, and San Diego Gas & Electric, which supplies power to most of the region, said it was still examining how the failure had spread so widely. But the initial cause, it said, appeared to be human error in Arizona that took down a line that provides power to the San Diego area. A major nuclear power plant in San Onofre, about 50 miles north of San Diego, also shut down, causing more failures, officials said.

“On their own, these things would cause virtually no impact,” because there would be backup power, said Michael R. Niggli, the president of the utility company. “Having each of these things happen at one time is not something we have considered. That’s not the way the system is designed.”

Six million people lost power, and Mr. Niggli said he had seen nothing on that scale in his three decades at the company. The investigation could take weeks or even months, officials said.

Arizona’s largest utility, APS, said that a worker had been performing maintenance on a substation line when the failure happened, but that it was not clear whether that had been a direct cause.

“Operating and protection protocols typically would have isolated the resulting outage to the Yuma area,” the company said. “The reason that did not occur in this case will be the focal point of the investigation into the event, which already is under way.”

[loodvig]

I here ya poot. Right now there is a room full of suits figuring out how to word this thing. Ferc will be involved.looks like the cost of this thing will be over one hundred million dollars.

We used to call it 'a finger pointing committee'.

[topgroove]

I,m sure he went for his drug test by now. I wouldnt be suprised if they check his phone records, emails, hell they,re gonna know what he had for breakfast.

[Pootnaigle]

One thing to bear in mind is that if they do sumpin to this guy it is the same as an admission of guilt on their part.That could cost em big time.My bet is he skates.

[topgroove]

I hope so poot. I've seen errors on switching orders before.

[T-Man]

Transmission systems are usually based on a network configuration and there are more than one way to keep things going so stuff can be taken out and worked on and still have a feed that can handle the load. Having one person and one piece in control of that big of an area like a choke point would be more of a design flaw. It would be hard to make that, one dudes switching mistake. The guy in the field wouldn't know how all the load has been routed so someone dispatching would have had a full understanding of what could and couldn't be done.They may find a way to blame one person but I'm going to guess there is more to it than we can all sepculate on.

[Pootnaigle]

If its a switching error you can bet they will try and cover it up. If it aint, no matter how pissed off they are at this guy they simply cant risk doing anything to him.His may even be the only name in the whole workforce that top management knows but unless he screws up on a lesser scale he should be Ok.
Its kinda strange but from this day on he will be forever known as the guy that took out half the western world. It will follow him to the end of his days.Personally I hope he lives to see another day and passes on what he learned from all this to his buddies.

[topgroove]

To get a grip on the blackout, Lets take a look back at the August,2003 blankout in the NorthEast. My cheif and I were the only crew on property that day. It was around 4:20PM and we were getting ready to leave for the day, when all of the sudden we lost power at the service center. I remember thinking, Uh Oh .... this might be a terrorist attack!

Here's the sequence of events that happened that day;


Sequence of events
The following is the blackout's sequence of events on August 14, 2003[15] (http://en.wikipedia.org/wiki/Northeast_Blackout_of_2003#cite_note-14)[16] (http://en.wikipedia.org/wiki/Northeast_Blackout_of_2003#cite_note-15)[17] (http://en.wikipedia.org/wiki/Northeast_Blackout_of_2003#cite_note-16) (times in EDT (http://en.wikipedia.org/wiki/Eastern_Time_Zone)):

12:15 p.m. Incorrect telemetry (http://en.wikipedia.org/wiki/Telemetry) data renders inoperative the state estimator (http://en.wikipedia.org/wiki/State_observer), a power flow monitoring tool operated by the Indiana-based Midwest Independent Transmission System Operator (http://en.wikipedia.org/wiki/Midwest_Independent_Transmission_System_Operator) (MISO). An operator corrects the telemetry problem but forgets to restart the monitoring tool.
1:31 p.m. The Eastlake, Ohio (http://en.wikipedia.org/wiki/Eastlake,_Ohio) generating plant shuts down. The plant is owned by FirstEnergy (http://en.wikipedia.org/wiki/FirstEnergy), an Akron, Ohio (http://en.wikipedia.org/wiki/Akron,_Ohio)-based company that had experienced extensive recent maintenance problems.[specify (http://en.wikipedia.org/wiki/Wikipedia:Citing_sources)]
2:02 p.m. The first of several 345 kV (http://en.wikipedia.org/wiki/Kilovolt) overhead transmission lines (http://en.wikipedia.org/wiki/Overhead_transmission_line) in northeast Ohio fails due to contact with a tree in Walton Hills, Ohio (http://en.wikipedia.org/wiki/Walton_Hills,_Ohio).[18] (http://en.wikipedia.org/wiki/Northeast_Blackout_of_2003#cite_note-17)http://upload.wikimedia.org/wikipedia/commons/thumb/5/55/WMA_button2b.png/17px-WMA_button2b.png41°21′22″N 81°34′10″W / 41.35611°N 81.56944°W / 41.35611; -81.56944 (http://toolserver.org/~geohack/geohack.php?pagename=Northeast_Blackout_of_2003&params=41_21_22_N_81_34_10_W_)[citation needed (http://en.wikipedia.org/wiki/Wikipedia:Citation_needed)]
2:14 p.m. An alarm system fails at FirstEnergy's control room and is not repaired.
3:05 p.m. A 345 kV transmission line known as the Chamberlin-Harding line fails in Parma (http://en.wikipedia.org/wiki/Parma,_Ohio), south of Cleveland, due to a tree.
3:17 p.m. Voltage dips temporarily (http://en.wikipedia.org/wiki/Brownout_(electricity)) on the Ohio portion of the grid. Controllers take no action.
3:32 p.m. Power shifted by the first failure onto another 345 kV power line, the Hanna-Juniper interconnection, causes it to sag into a tree, bringing it offline as well. While MISO and FirstEnergy controllers concentrate on understanding the failures, they fail to inform system controllers in nearby states.
3:39 p.m. A FirstEnergy 138 kV line fails in northern Ohio (http://en.wikipedia.org/wiki/Ohio).[19] (http://en.wikipedia.org/wiki/Northeast_Blackout_of_2003#cite_note-18)
3:41 p.m. A circuit breaker (http://en.wikipedia.org/wiki/Circuit_breaker) connecting FirstEnergy's grid with that of American Electric Power (http://en.wikipedia.org/wiki/American_Electric_Power) is tripped (http://en.wikipedia.org/wiki/Circuit_breaker) as a 345 kV power line (Star-South Canton interconnection) and fifteen 138 kV lines fail in rapid succession in northern Ohio.
3:46 p.m. A fifth 345 kV line, the Tidd-Canton Central line, trips offline.
4:05:57 p.m. The Sammis-Star 345 kV line trips due to undervoltage and overcurrent interpreted as a short circuit. Later analysis suggests that the blackout could have been averted prior to this failure by cutting 1.5 GW of load in the Cleveland–Akron area.
4:06–4:08 p.m. A sustained power surge north toward Cleveland overloads three 138 kV lines.
4:09:02 p.m. Voltage sags deeply as Ohio draws 2 GW (http://en.wikipedia.org/wiki/Gigawatt) of power from Michigan, creating simultaneous undervoltage and overcurrent conditions as power attempts to flow in such a way as to rebalance the system's voltage.
4:10:34 p.m. Many transmission lines trip out, first in Michigan and then in Ohio, blocking the eastward flow of power around the south shore of Lake Erie (http://en.wikipedia.org/wiki/Lake_Erie) from Toledo, Ohio (http://en.wikipedia.org/wiki/Toledo,_Ohio), east through Erie, Pennsylvania (http://en.wikipedia.org/wiki/Erie,_Pennsylvania) and into the Buffalo, New York (http://en.wikipedia.org/wiki/Buffalo,_New_York) metropolitan area. Suddenly bereft of demand, generating stations go offline, creating a huge power deficit. In seconds, power surges in from the east, overloading east-coast power plants whose generators go offline as a protective measure, and the blackout is on.
4:10:37 p.m. The eastern and western Michigan power grids disconnect from each other. Two 345 kV lines in Michigan (http://en.wikipedia.org/wiki/Michigan) trip. A line that runs from Grand Ledge (http://en.wikipedia.org/wiki/Grand_Ledge,_Michigan) to Ann Arbor (http://en.wikipedia.org/wiki/Ann_Arbor,_Michigan) known as the Oneida-Majestic interconnection trips. A short time later, a line running from Bay City (http://en.wikipedia.org/wiki/Bay_City,_Michigan) south to Flint (http://en.wikipedia.org/wiki/Flint,_Michigan) in Consumers Energy (http://en.wikipedia.org/wiki/Consumers_Energy)'s system known as the Hampton-Thetford line also trips.
4:10:38 p.m. Cleveland separates from the Pennsylvania grid.
4:10:39 p.m. 3.7 GW power flows from the east along the north shore of Lake Erie, through Ontario to southern Michigan and northern Ohio, a flow more than ten times greater than the condition 30 seconds earlier, causing a voltage drop across the system.
4:10:40 p.m. Flow flips to 2 GW eastward from Michigan through Ontario (a net reversal of 5.7 GW of power), then reverses back westward again within a half second.
4:10:43 p.m. International connections between the United States and Canada begin failing.
4:10:45 p.m. Northwestern Ontario separates from the east when the Wawa-Marathon 230 kV line north of Lake Superior (http://en.wikipedia.org/wiki/Lake_Superior) disconnects. The first Ontario power plants go offline in response to the unstable voltage and current demand on the system.
4:10:46 p.m. New York separates from the New England grid.
4:10:50 p.m. Ontario separates from the western New York grid.
4:11:57 p.m. The Keith-Waterman, Bunce Creek-Scott 230 kV lines and the St. Clair (http://en.wikipedia.org/wiki/Saint_Clair,_Michigan)-Lambton (http://en.wikipedia.org/wiki/Lambton_County,_Ontario) #1 230 kV line and #2 345 kV line between Michigan and Ontario fail.
4:12:03 p.m. Windsor, Ontario (http://en.wikipedia.org/wiki/Windsor,_Ontario) and surrounding areas drop off the grid.
4:12:58 p.m. Northern New Jersey (http://en.wikipedia.org/wiki/New_Jersey) separates its power-grids from New York (http://en.wikipedia.org/wiki/New_York) and the Philadelphia (http://en.wikipedia.org/wiki/Philadelphia) area, causing a cascade of failing secondary generator plants along the Jersey coast and throughout the inland west.
4:13 p.m. End of cascading failure (http://en.wikipedia.org/wiki/Cascading_failure). 256 power plants are off-line, 85% of which went offline after the grid separations occurred, most due to the action of automatic protective controls.

[topgroove]

Daniel Froetscher, Vice President of Energy Delivery for APS, said: 'We’re struggling. We have to take a hard look at the system design and figure out exactly what happened. We don’t know the underlying causes.'
Early Thursday, APS officials noticed an issue with a series capacitor at the North Gila substation outside the city of Yuma, Mr Froetscher said.
The capacitor — a piece of equipment about the size of a small car that helps the utility manage voltage — wasn’t functioning properly.
APS personnel were dispatched to take it offline, the LA Times.



Read more: http://www.dailymail.co.uk/news/article-2036010/How-did-Power-bosses-baffled-engineer-caused-U-S-blackout.html#ixzz1XbFCl6JD