Keystone pipeline has spilled substantially more oil, and more often, than indicated in the risk assessments the company provided to regulators. But why and how does a common risk assessment become unrealistic and describe such a rosy scenario?
What regulators saw in the pre-construction project risk assessment was reportedly the following: “A spill of more than 50 barrels will occur “not more than once every 7 to 11 years over the entire length of the pipeline in the United States.””
The predicted frequency would be 1/7 to 1/11 per year.
However, the reality is that since 2010, the Keystone pipeline leaked three times in the United States. That puts the frequency to 3/7 per year, roughly 3−4 times more than stated in the pre-construction risk assessment.
Let’s note that the phrase “more than 50 barrels” in the assessment does not declare an upper limit, but suggests a minor accident. Of the three leaks, two leaked about 400 barrels, and one 5,000 barrels.
Reasons for Unrealistic Risk Assessments
Sometimes, we see risk assessors who base their estimates on best-case scenario assessments, completely leaving out worst-case scenarios and even average-case scenarios. That corresponds to biasing and censoring issues.
However, biasing and censoring issues also occur with the common practice of looking at worst-case but “credible” accidents. We quote the word “credible” as we never see it defined in this type of assessment. The fallacy lies in an idea: if one designs a system to withstand all the individual worst-case “credible” accidents, then the system is inherently protected against those “credible” accidents. The fallacy has multiple aspects beyond not defining “credible;” for example, it does not cover uncertainties, interdependencies and common cause failures.
Those aspects, unfortunately, do characterize industrial and natural accidents.
How to Avoid the Pitfalls of Unrealistic Risk Assessments
Here are a few guidelines:
- Risk assessment methodologies should never ignore a scenario or impose a selection.
- The result of the analysis will filter out the irrelevant risks because the methodology is sound, not because of a priori selection and framing bias.
- Using third-party risk assessors is paramount. Keystone company reportedly followed the standard practice of hiring their own expert to provide regulators the risk assessment.
In the eye of the public, doubt of their motive will always remain. Indeed, these are blatant cases of conflict of interest and focusing bias.
Third-party Independent Risk Assessment
This is the reason why third-party risk assessment is paramount to bring credibility to a project. We have seen some technical review boards ask for them.
In the case of tailings dam risk, the United Nations Environmental Programme is unambiguous: Risk assessment should be performed by an external third party.
Their 2017 report “Mine Tailings Storage: Safety Is No Accident” states as follows:
- Independent expert risk assessments would reduce risk of dams’ failures by providing independent, third-party, unbiased expert oversight.
- Emphasis on the role and importance of monitoring and independent, third-party review throughout life cycle is of critical importance.
- Independent technical reviews should conduct and publish their reports through the life cycle of the infrastructure.
Framing Probabilities
This is why when we perform risk analysis, we spend a lot of time framing probabilities of mishaps. We like to be generally right rather than precisely wrong.
We also compare our probability estimates to benchmarks and oftentimes carry out benchmarking across industries. This enhances our credibility and transparency. An example of cross-industry risk tolerance benchmarking applied to hydrocarbons and other hazardous material rail transport is available in this study.
