I was probably one of the first in Europe to hear about the US blackout. I’ve customers in Manhattan. One of them rang me in Belfast as soon as she determined not only her Upper West flat, but also her Chinatown server rack were both affected. She has a big UPS but no backup generator. It just isn’t feasible for a facility her size. I advised an immediate controlled shutdown.
It seems government officials were announcing “this wasn’t a terrorist incident” almost before people like my customer completed their emergency procedures. I found and still find this strange. It may well be true. It probably is true, but the haste to discount the possibility was unseemly.
Terrorism and sabotage are not necessarily spectacular events. It doesn’t take a bombing or an armed attack to bring down a power grid. In 1964 the East Coast power grid came down all by itself. It was due to a cascade of protective shutdowns after a “First Cause” failure. That may be the case this week as well, but we don’t know yet. A sudden reversal of power flow on the Lake Erie power loop occurred instants before the cascading failures began. That is sufficient information to tell us absolutely nothing.
Thus unarmed with facts, I will now sally forth into the vacuum of hard data and suggest some attack scenarios.
The Saboteur. Someone with appropriate knowledge may have penetrated a targeted power facility and simply thrown a few switches. A “mole” at a power station would be best, but power stations are not Fort Knoxen. A trained agent could probably get in and out of some “weakest link” facility somewhere without being detected.
Does anyone remember the incident of November 11, 2001 (see Charleston Daily Mail, “Guard Chases Men Near Power Plant”) during which a security guard slugged a person attempting to enter power plant grounds from the river?
The Hack. Someone could have cracked a power company control system and “adjusted” a few things. I once authored software systems for control of large building complexes. Most such systems have queues of time based actions. If an attacker penetrated several systems, they could insert minor events synchronized to milliseconds. Even the actual queue insertions could be handled by stealthy, pre-positioned “Trojan Horse” programs. An innocuous looking message could trigger the countdown sequence. The trigger could be sent from anywhere on the planet. Perhaps the Microsoft worm was a diversion.
Each event on it’s own would be insignificant, but the sum of all could be a big problem.
A trail might be left, but it would be difficult to uncover if there was a dispersed attack. If only one site were involved it would be much easier to find evidence both because the source of the First Cause would be pin-pointed and because the event itself would be out of the ordinary.
If the attackers were moderately good they would leave a trail only discoverable by computer forensics. The critical computer log entries would be gone unless printed on paper as they occured… or if they were intended to be found.
These scenarios are an intellectual exercise. Taking down a power grid is an annoyance but doesn’t accomplish anything in and of itself. There has been no “other” event connected to it. No claims of holding American infrastructure under threat. No major attack during the early blackout confusion. No operational movements and pre-setting of people or material… hmmm.
Just thought I’d keep y’all worryin’ over there!
Thank you, Dale.
Now, where did I put my diazepam?
I too have written environmental control software. The issues regarding security power infrastructure are well known, including the replacement of old solenoid switches with ones that are controlled by microprocessors.
It’s always hard to rule out sabotage from within … that’s true of software errors in any domain, including credit card processing, bank funds transfer or just some annoyed programmer who decides to “teach management a lesson” via whatever business system s/he was working on.
That said, as I understand the sequence of the cascade, your scenrios don’t fit. In particular, the reverse surge that kicked off the cascade seems both very localized and also very large. Not impossible to pull off but it would have taken a LOT of access to a LOT of switches, plus a very detailed knowledge of the system’s timing and overflow characteristics.
That kind of knowledge and expertise is unlikely to be in the hands of terrorists … it takes a lot of direct familiarity with the system to acquire it and know how to use it. It’s hard to prove a negative, of course, but unless an insider basically set the whole thing up. I don’t think this this was an attack of that sort.
If the switches clearly all went down at the same time, otoh, I’d consider your scenarios to be a little more possible ….
okay, I take it back. Should have done some research before posting …. SCADA design docs have been found in al Qaeda computers. Yikes!!
hat tip to Instapundit for the link.
Actually, I think it would be nearly impossible to intentionally initiate a cascade failure of this kind.
First, The very complexity of the power grid means that nobody really understands them well enough to to pinpoint one part of the system which will trigger the cascade failure
Second, such a point would have to escape the notice of all the people who’s job it is to notice such points.
Third, in distributed networks widespread failure requires a “Prefect Storm” confluence of events to make the entire network susceptible to cascade failure. In previous blackouts, the failure only spread because one specific switch or generator was offline at the exact moment the trigger failure occurred. A minor change in the networks state would have stopped the cascade. An attacker would therefore have to bide his time waiting for the exact state to arise before launching his attack.
Even an internal saboteur using a software attack would have to have access to real time information about the entire state of the grid itself. Even a lag of few minutes in data collection could cause the malevolent software to miss the time window to trigger the cascade. I doubt such real time information exists.
Although counter-intuitive, the near organic complexity of large networks like power grids makes them resilient to intentional disruption.
OK, you smart guys have convinced me that conditions called for to precipitate the recent outage are simply to complex to have been caused by (1) terrorists, (2) rogue power-grid technicians, or (3) naughty software. Fair enough.
So, who (or what) could have brought this about?
I’m not entirely convinced. I’m not trying to specifically argue it actually happened this time, because I’ve not heard of any of the sorts of events I’d expect. But then… it wouldn’t necessarily be advertised.
The sudden reversal of power flow indicates there was very suddenly “a power vacuum”. You could get this either from a sudden huge load or by a large power station going off line… or by a station going out of phase. *That* would cause some fireworks somewhere though, unless the de-phasing was very slight.
It’s also one of the things few people know about bringing power stations on the grid. The AC phase has to be matched up quite precisely at the instant it goes “online” or else all hell breaks lose.
I agree that though terrorist agency seems unlikely in this case the possibility must be investigated.
However, in addition to the comments above in this section, and many recent commentaries by people close to the electric utility industry noting that changing conditions have raised questions about how robust the transmission system is, consider the nature of the terrorist enemy.
If a Western country were waging war, seeking to undermine its opponent by destabilizing its power grid would certainly be seen as worth an investment in time and effort. But Islamist terrorists appear mostly interested in planning attacks designed to kill large numbers of people right away. That is the kind of people they are. Nothing about playing with the lights in Cleveland or forcing restauranteurs in New York to throw out fish satisfies any part of the Islamist agenda, least of all the bloodlust so central to it.
It was Gray Davis’s fault. 🙂
Seriously, from what I can see there are basically three factors that created one huge problem. One power grid, high usage, and one fault. Oops. This is the wonderfully efficient government enforced monopoly (in other places we’d say “nationalized”) electric system.
I agree with Zathras’ idea about a typical terrorist goal of instant kill of large numbers of people.
But, what about the anthrax scare shortly after 9-11?
The potential for progressively increasing numbers of very sick people over a span of days?, weeks?..
Naturally, sabotage should be considered. But a creaking, rickety power infrastructure is perhaps a more realistic possibility. In the western U.S., we suffered a multistate power outage several years before 9/11 that was due to power lines failing in Oregon.
Here’s some real data (and media speculation) re. the power outage; it does show some interesting anomalies in the hours before the outage hit: Blog linking to both uninformed punditry and hard data.
Why assume it is anybody’s “fault”. Any more than a stock-market crash, a much more common cascade phenomenon, is anyone’s fault. “Creaking, rickety power infrastructure” sounds like the language of collective perfectionism to me. In real life everything is creaking and ricketty: the bits work till they break and are replaced. It is called home-improvement, capitalism, or ecology, depending on what you are looking at.
A complex system that crashes once in 30 years looks pretty good to me. 50M of the worlds most comfortable people got a nasty surprise, but I’ve yet to hear of anyone seriously hurt.
You may all like to visit SpaceWeather.com
A powerfull sunspot roared into life a couple of days ago….this type of activity was the cause of the N.E. Grid failure a couple of decades ago.
Yow. I hadn’t hear that. Canada has special protection on power lines because far north in auroral areas you can get enormous voltages across the miles of power lines due to solar storms. However I’d expect there to have been really nice aurorae (of course it was day light); it is unusual for the effects to go south into the US (but not unheard of); and I’d expect the impact to be more widespread. Unless it was just a camel back breaking straw…
I’ll have to go read the link posted earlier.
At SpaceWeather.com scroll down to the links listings and click on “recent solar events”.
I cannot get the link script to work here….
I also found it strange how quickly we were informed that it WAS NOT the MSBlast worm. Even a company the size of MS would have trouble paying off the litigation from such an event.
I do feel there’s more to this than meets the eye.
I seriously doubt MSBlast could be used as part of the main thrust of such an attack. It could, at best, be a cover or a trigger event.
Most control systems are Unix based. The older ones are purpose built microprocessor controlled with proprietary protocols running on things ling RS422; many places still have racks of relays and manual controls I imagine. DFWAB as they say…
The sort of attack I envisioned was to gain access to an internal network, perhaps via office boxes running microsoft OS’s, then penetrate the Unix systems from inside and plant the trojan for later use.
I don’t think anyone runs power plants on MS OS’s. I surely hope not. [The thought of one of those Homer Simpson OS’s running a nuke plant is enough to make you want a case of bog rolls and a very deep mineshaft to crawl into.]
“Perhaps the Microsoft worm was a diversion.”
“I also found it strange how quickly we were informed that it WAS NOT the MSBlast worm.”
SCADA uses Windows, and its main form of communication uses RPC. MSBlast exploits RPC on Windows.
To quote Steve Gibson, “Each instance of the worm emits only 50 SYN packets per second, deliberately and significantly throttling each machine’s contribution to the attack.”
What if the Microsoft target was a diversion?
Mr. Herbert wrote:
“Creaking, rickety power infrastructure” sounds like the language of collective perfectionism to me
Apparently Mr. Herbert assumes the interstate power-delivery system in the States is a free-market enterprise. It is not. Unlike the individual states, who in large part have power delivery and generation systems that have been deregulated, the interstate power grid is creaky, rickety, over-regulated, and over-taxed (both in the fiscal as well as the duty-cycle sense).
In a non-competitive, regulated environment such as our power grid, how does Mr. Herbert suggest that we consumers apply market pressure? As far as I can tell, our only recourse is either more odious regulation at a federal level, or by criticism of the quality-of-service and demands that the utilities do better- which was the upshot of my original post.
And, Mr. Herbert, the power grid in our part of the country has failed far more often than once in thirty years. Power line failures have brought on regional blackouts several times in the last decade. Transmission line failures twelve-hundred miles from my home (over two thousand kilometers) that bring down my local power carrier strikes me as a system both creaking and rickety, indeed.
Lastly, we must consider sabotage. This is always a possibility, and it is foolish not to consider it.
I don’t think that the application of “consumer pressure” is how any market free or unfree works.
I’m not worried by the idea that things might work as well as they have to under all normal circumstances, but do break down occaisionally.
Well, you are certainly correct in that an unfree market is not influenced by “consumer pressure”- that’s what coercive legislation is about, and why state-mandated monopolies are so bad.
As for free markets, “consumer pressure” (in the form of purchases not made for inadequate products and services) is the very essence of a free market as far as the buyer is concerned.
That things break down also does not concern me greatly; what worries me is critical systems that fail catastrophically as the result of relatively small disturbances. But this is probably inevitable when a system grows by accretion and not by design.
That a small perturbations leads to big and unpredictable change is more a property of all sorts of systems, even quite small, simple-looking ones and being designed as a whole doesn’t necessarily help when dealing with complex sets of real-world stimuli. Even if it did, you’d have to redesign the whole every time you wanted to add a component, if you were determined to squeeze out the chaos.
I can’t prove it, but I have the impression that accretion copes better with chaos than design because it has more opportunity to fail in small ways before catastrophe, and more likelihood of redundancy.
I suspect that accretion in natural systems does produce more robust thingies, with natural selection for fitness weeding out the poor performers.
In human-designed systems (in particular “always-on 24/7” critical systems) that are not refitted or redesigned to accommodate changes to the operating environments’ requirements, there is never a real chance to test for the fittest solution. Although with good modeling and lots of CPU cycles, maybe you could test proposed solutions for fitness without resorting to physical implementation. But of this I am not convinced.