How Complex Systems Fail

Highlights from How Complex Systems Fail:

• Title) How Failure is Attributed to Proximate Cause
• 1) It is the presence of these hazards that drives the creation of defenses against hazard that characterize these systems
• 2) multiple layers of defense
• 3) Catastrophe requires multiple failures
• 4) Eradication of all latent failures is limited primarily by economic cost but also because it is difficult before the fact to see how such failures might contribute to an accident
• 5) complex systems run as broken systems
• 5) The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws
• 6) It is impossible to eliminate the potential for such catastrophic failure
• 7) Post-accident attribution accident to a ‘root cause’ is fundamentally wrong
• 7) There are multiple contributors to accidents
• 7) social, cultural need to blame specific, localized forces or events for outcomes
• 8) Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case
• 8)  It seems that practitioners “should have known”
• 9) dual roles
• 9) Outsiders rarely acknowledge the duality of this role
• 11) After an accident, practitioner actions may be regarded as ‘errors’ or ‘violations’
• 11) biased by hindsight and ignore the other driving forces, especially production pressure
• 13) Human expertise in complex systems is constantly changing
• 13) need to replace experts who leave
• 14) Change introduces new forms of failure
• 14) overt: open and observable
• 14) use of new technology
• 14) decrease the number of low consequence but high frequency failures
• 14) create opportunities for new, low frequency but high consequence failures
• 14) Not uncommonly, these new, rare catastrophes have even greater impact than those eliminated by the new technology
• 14) hard to see the contribution of technology to the failure
• What to do? Freeze a system from all and any change?
• 15) post-accident remedies usually increase the coupling and complexity
• 16) Safety is a characteristic of systems and not of their components
• 18) Failure free operations require experience with failure
• "Game day" and failure injection