Cognitive biases potentially affecting judgment of global risksForthcoming in Global Catastrophic Risks, eds. Nick Bostrom and Milan CirkovicDraft of August 31, 2006.Eliezer Yudkowsky([email protected])Singularity Institute for Artificial IntelligencePalo Alto, CAIntroduction 1All else being equal, not many people would prefer to destroy the world. Even facelesscorporations, meddling governments, reckless scientists, and other agents of doom, requirea world in which to achieve their goals of profit, order, tenure, or other villainies. If ourextinction proceeds slowly enough to allow a moment of horrified realization, the doers ofthe deed will likely be quite taken aback on realizing that they have actually destroyed theworld. Therefore I suggest that if the Earth is destroyed, it will probably be by mistake.The systematic experimental study of reproducible errors of human reasoning, and whatthese errors reveal about underlying mental processes, is known as the heuristics andbiases program in cognitive psychology. This program has made discoveries highlyrelevant to assessors of global catastrophic risks. Suppose you're worried about the risk ofSubstance P, an explosive of planet-wrecking potency which will detonate if exposed to astrong radio signal. Luckily there's a famous expert who discovered Substance P, spent thelast thirty years working with it, and knows it better than anyone else in the world. Youcall up the expert and ask how strong the radio signal has to be. The expert replies that thecritical threshold is probably around 4,000 terawatts. "Probably?" you query. "Can yougive me a 98% confidence interval?" "Sure," replies the expert. "I'm 99% confident thatthe critical threshold is above 500 terawatts, and 99% confident that the threshold is below80,000 terawatts." "What about 10 terawatts?" you ask. "Impossible," replies the expert.The above methodology for expert elicitation looks perfectly reasonable, the sort of thingany competent practitioner might do when faced with such a problem. Indeed, thismethodology was used in the Reactor Safety Study (Rasmussen 1975), now widelyregarded as the first major attempt at probabilistic risk assessment. But the student ofheuristics and biases will recognize at least two major mistakes in the method - not logicalflaws, but conditions extremely susceptible to human error.1I thank Michael Roy Ames, Nick Bostrom, Milan Cirkovic, Olie Lamb, Tamas Martinec, Robin LeePowell, Christian Rovner, and Michael Wilson for their comments, suggestions, and criticisms.Needless to say, any remaining errors in this paper are my own.-1-

The heuristics and biases program has uncovered results that may startle and dismay theunaccustomed scholar. Some readers, first encountering the experimental results citedhere, may sit up and say: "Is that really an experimental result? Are people really suchpoor guessers? Maybe the experiment was poorly designed, and the result would go awaywith such-and-such manipulation." Lacking the space for exposition, I can only plead withthe reader to consult the primary literature. The obvious manipulations have already beentried, and the results found to be robust.1: AvailabilitySuppose you randomly sample a word of three or more letters from an English text.Is it more likely that the word starts with an R ("rope"), or that R is its third letter("park")?A general principle underlying the heuristics-and-biases program is that human beings usemethods of thought - heuristics - which quickly return good approximate answers in manycases; but which also give rise to systematic errors called biases. An example of aheuristic is to judge the frequency or probability of an event by its availability, the easewith which examples of the event come to mind. R appears in the third-letter position ofmore English words than in the first-letter position, yet it is much easier to recall wordsthat begin with "R" than words whose third letter is "R". Thus, a majority of respondentsguess that words beginning with "R" are more frequent, when the reverse is the case.(Tversky and Kahneman 1973.)Biases implicit in the availability heuristic affect estimates of risk. A pioneering study byLichtenstein et. al. (1978) examined absolute and relative probability judgments of risk.People know in general terms which risks cause large numbers of deaths and which causefew deaths. However, asked to quantify risks more precisely, people severely overestimatethe frequency of rare causes of death, and severely underestimate the frequency ofcommon causes of death. Other repeated errors were also apparent: Accidents werejudged to cause as many deaths as disease. (Diseases cause about 16 times as many deathsas accidents.) Homicide was incorrectly judged a more frequent cause of death thandiabetes, or stomach cancer. A followup study by Combs and Slovic (1979) talliedreporting of deaths in two newspapers, and found that errors in probability judgmentscorrelated strongly (.85 and .89) with selective reporting in newspapers.People refuse to buy flood insurance even when it is heavily subsidized and priced farbelow an actuarially fair value. Kunreuther et. al. (1993) suggests underreaction to threatsof flooding may arise from "the inability of individuals to conceptualize floods that havenever occurred. Men on flood plains appear to be very much prisoners of theirexperience. Recently experienced floods appear to set an upward bound to the size of losswith which managers believe they ought to be concerned." Burton et. al. (1978) report thatwhen dams and levees are built, they reduce the frequency of floods, and thus apparentlycreate a false sense of security, leading to reduced precautions. While building dams-2-

decreases the frequency of floods, damage per flood is so much greater afterward that theaverage yearly damage increases.It seems that people do not extrapolate from experienced small hazards to a possibility oflarge risks; rather, the past experience of small hazards sets a perceived upper bound onrisks. A society well-protected against minor hazards will take no action against majorrisks (building on flood plains once the regular minor floods are eliminated). A societysubject to regular minor hazards will treat those minor hazards as an upper bound on thesize of the risks (guarding against regular minor floods but not occasional major floods).Risks of human extinction may tend to be underestimated since, obviously, humanity hasnever yet encountered an extinction event. 22: Hindsight biasHindsight bias is when subjects, after learning the eventual outcome, give a much higherestimate for the predictability of that outcome than subjects who predict the outcomewithout advance knowledge. Hindsight bias is sometimes called the I-knew-it-all-alongeffect.Fischhoff and Beyth (1975) presented students with historical accounts of unfamiliarincidents, such as a conflict between the Gurkhas and the British in 1814. Given theaccount as background knowledge, five groups of students were asked what they wouldhave predicted as the probability for each of four outcomes: British victory, Gurkhavictory, stalemate with a peace settlement, or stalemate with no peace settlement. Fourexperimental groups were respectively told that these four outcomes were the historicaloutcome. The fifth, control group was not told any historical outcome. In every case, agroup told an outcome assigned substantially higher probability to that outcome, than didany other group or the control group.Hindsight bias is important in legal cases, where a judge or jury must determine whether adefendant was legally negligent in failing to foresee a hazard (Sanchiro 2003). In anexperiment based on an actual legal case, Kamin and Rachlinski (1995) asked two groupsto estimate the probability of flood damage caused by blockage of a city-owneddrawbridge. The control group was told only the background information known to thecity when it decided not to hire a bridge watcher. The experimental group was given thisinformation, plus the fact that a flood had actually occurred. Instructions stated the citywas negligent if the foreseeable probability of flooding was greater than 10%. 76% of thecontrol group concluded the flood was so unlikely that no precautions were necessary;57% of the experimental group concluded the flood was so likely that failure to takeprecautions was legally negligent. A third experimental group was told the outcome and2Milan Cirkovic points out that the Toba supereruption ( 73,000 BCE) may count as a nearextinction event. The blast and subsequent winter killed off a supermajority of humankind; geneticevidence suggests there were only a few thousand survivors, perhaps less. (Ambrose 1998.) Notethat this event is not in our historical memory - it predates writing.-3-

also explicitly instructed to avoid hindsight bias, which made no difference: 56%concluded the city was legally negligent. Judges cannot simply instruct juries to avoidhindsight bias; that debiasing manipulation has no significant effect.Viewing history through the lens of hindsight, we vastly underestimate the cost ofpreventing catastrophe. In 1986, the space shuttle Challenger exploded for reasonseventually traced to an O-ring losing flexibility at low temperature. (Rogers et. al. 1986.)There were warning signs of a problem with the O-rings. But preventing the Challengerdisaster would have required, not attending to the problem with the O-rings, but attendingto every warning sign which seemed as severe as the O-ring problem, without benefit ofhindsight.3: Black SwansTaleb (2005) suggests that hindsight bias and availability bias bear primary responsibilityfor our failure to guard against what Taleb calls Black Swans. Black Swans are anespecially difficult version of the problem of the fat tails: sometimes most of the variancein a process comes from exceptionally rare, exceptionally huge events. Consider afinancial instrument that earns 10 with 98% probability, but loses 1000 with 2%probability; it's a poor net risk, but it looks like a steady winner. Taleb (2001) gives theexample of a trader whose strategy worked for six years without a single bad quarter,yielding close to 80 million - then lost 300 million in a single catastrophe.Another example is that of Long-Term Capital Management, a hedge fund whose foundersincluded two winners of the Nobel Prize in Economics. During the Asian currency crisisand Russian bond default of 1998, the markets behaved in a literally unprecedentedfashion, assigned a negligible probability by LTCM's historical model. As a result, LTCMbegan to lose 100 million per day, day after day. On a single day in 1998, LTCM lostmore than 500 million. (Taleb 2005.)The founders of LTCM later called the market conditions of 1998 a "ten-sigma event".But obviously it was not that improbable. Mistakenly believing that the past waspredictable, people conclude that the future is predictable. As Fischhoff (1982) puts it:When we attempt to understand past events, we implicitly test the hypotheses or rules weuse both to interpret and to anticipate the world around us. If, in hindsight, wesystematically underestimate the surprises that the past held and holds for us, we aresubjecting those hypotheses to inordinately weak tests and, presumably, finding littlereason to change them.The lesson of history is that swan happens. People are surprised by catastrophes lyingoutside their anticipation, beyond their historical probability distributions. Then why arewe so taken aback when Black Swans occur? Why did LTCM borrow leverage of 125billion against 4.72 billion of equity, almost ensuring that any Black Swan would destroythem?-4-

Because of hindsight bias, we learn overly specific lessons. After September 11th, the U.S.Federal Aviation Administration prohibited box-cutters on airplanes. The hindsight biasrendered the event too predictable in retrospect, permitting the angry victims to find it theresult of 'negligence' - such as intelligence agencies' failure to distinguish warnings of AlQaeda activity amid a thousand other warnings. We learned not to allow hijacked planesto overfly our cities. We did not learn the lesson: "Black Swans occur; do what you can toprepare for the unanticipated."Taleb (2005) writes:It is difficult to motivate people in the prevention of Black Swans. Prevention is not easilyperceived, measured, or rewarded; it is generally a silent and thankless activity. Justconsider that a costly measure is taken to stave off such an event. One can easily computethe costs while the results are hard to determine. How can one tell its effectiveness,whether the measure was successful or if it just coincided with no particular accident? .Job performance assessments in these matters are not just tricky, but may be biased infavor of the observed "acts of heroism". History books do not account for heroicpreventive measures.4: The conjunction fallacyLinda is 31 years old, single, outspoken, and very bright. She majored inphilosophy. As a student, she was deeply concerned with issues of discrimination andsocial justice, and also participated in anti-nuclear demonstrations.Rank the following statements from most probable to least probable: is a teacher in an elementary school.Linda works in a bookstore and takes Yoga classes.Linda is active in the feminist movement.Linda is a psychiatric social worker.Linda is a member of the League of Women Voters.Linda is a bank teller.Linda is an insurance salesperson.Linda is a bank teller and is active in the feminist movement.89% of 88 undergraduate subjects ranked (8) as more probable than (6). (Tversky andKahneman 1982.) Since the given description of Linda was chosen to be similar to afeminist and dissimilar to a bank teller, (8) is more representative of Linda's description.However, ranking (8) as more probable than (6) violates the conjunction rule ofprobability theory which st