superforecasting by philip e. tetlock and dan gardner
TRANSCRIPT
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
1/29
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
2/29
http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136693/siteID/8001/retailerid/6/trackingcode/penguinrandomhttp://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136709/siteID/8001/retailerid/22/trackingcode/PRH698C2DEE54http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136693/siteID/8001/retailerid/2/trackingcode/PRH698C2DEE54http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136709/siteID/8001/retailerid/3/trackingcode/PRH698C2DEE54http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136693/siteID/8001/retailerid/7/trackingcode/randohouseinc10663-20http://www.crownpublishing.com/
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
3/29
p r f ` s b j h r
^ R Q B Y J H Y B N F ^ W @ O K
—^upbrjhrbnfst`ok `s f rfrb ihhm teft w`dd lfmb yhu slfrtbr fog w`sbr.
Hob hj teb k`fots hj ibefv`hrfd sn`bonb rbvbfds ehw th `lprhvb ft
prbg`nt`ok teb juturb.“
‗ FGFL KYF OW,
Obw Phrm W`lbs ibstsbdd`ok futehr hj K`vb fog Wfmb
—Khhg cugklbot fog khhg jhrbnfst`ok frb rfrb, iut teby turo hut
th ib lfgb hj tbfnefidb sm`dds. Iy jhrn`ok jhrbnfstbrs th nhlpbtb,
Wbtdhnm g`snhvbrbg weft teb sm`dds frb fog ehw teby whrm, fog te`sihhm tbfnebs teb fi`d`ty th foy `otbrbstbg rbfgbr.“
‗ ^ WB[F Y W I Y F OG ,
prbs`gbot, Web Dhok Ohw Jhuogft`ho
—Qe`d`p Wbtdhnm `s rbohwobg jhr gblhostrft`ok teft lhst bxpbrts
frb oh ibttbr tefo ―gfrt-terhw`ok lhombys‐ ft prbg`nt`ok bdbnt`hos, wfrs, bnhohl`n nhddfpsbs, fog htebr bvbots. @o e`s ir`dd`fot obw
ihhm, Wbtdhnm hjjbrs f lune lhrb ehpbjud lbssfkb, ifsbg honb fkf`o
ho e`s hwo krhuogirbfm`ok rbsbfrne. Eb sehws teft nbrtf`o pbhpdb
nfo jhrbnfst bvbots w`te fnnurfny lune ibttbr tefo nefonb‗fog
sh, pbrefps, nfo teb rbst hj us, `j wb bludftb teb nr`t`nfd te`om`ok
hj tebsb ―supbrjhrbnfstbrs.‐ Web sbdj-blphwbrlbot kborb ghbso‐t kbt
foy slfrtbr fog lhrb shpe`st`nftbg tefo te`s.“
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
4/29
—^upbrjhrbnfst`ok `s teb rfrb ihhm teft `s ihte snehdfrdy fog bokfk-
`ok. Web dbsshos frb sn`bot`stn, nhlpbdd`ok, fog bohrlhusdy prfn-
t`nfd. Foyhob weh `s `o teb jhrbnfst`ok ius`obss‗fog teft‐s fdd hj
us‗sehudg grhp weft teby frb gh`ok fog rbfg `t.“
‗ L @ NE F BD C . L F R IH R ^ ^ @ O ,
ebfg, Kdhifd J`ofon`fd ^trftbk`bs, Nrbg`t ^u`ssb
—Webrb `so‐t f shn`fd sn`bot`st `o teb whrdg @ fgl`rb lhrb tefo Qe`d
Wbtdhnm.“
‗ W @ L E F Y J H Y G ,
futehr hj Web Rogbrnhvbr Bnhohl`st
—Jrhl teb Hrfndb hj Gbdpe` th lbg`bvfd fstrhdhkbrs th lhgbro hvbr-
nhostgbot bxpbrts, jhrbnfstbrs efvb ibbo b`tebr gbdugbg hr jrfugu-
dbot. Jhr teb strst t`lb, ^upbrjhrbnfst`ok rbvbfds teb sbnrbt hj lfm`ok
ehobst, rbd`fidb, bjjbnt`vb, usbjud cugklbots fihut teb juturb.“‗ F F Y H O IY H [ O ,
ne`bj r`sm hjj`nbr, FVY Nfp`tfd Lfofkblbot, fog
futehr hj Web Qhmbr Jfnb hj [fdd ^trbbt
—^hnrftbs efg teb `os`ket `o ―mohw teysbdj,‐ Mfeoblfo gbd`vbrbg teb
sn`bonb `o We`om`ok, Jfst fog ^dhw, fog ohw Wbtdhnm efs shlbte`ok
wb nfo fdd fppdy `o ^upbrjhrbnfst`ok.“
‗ CR F O D R @ ^ QBY BS ,
kdhifd ebfg, RI^ Krhup Ybsbfrne
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
5/29
Nhpyr`ket © 9162 iy Qe`d`p Wbtdhnm Nhosudt`ok, @on., fog Nhoofuket ^trbbt, @on.
Fdd r`kets rbsbrvbg.
Quid`sebg `o teb Ro`tbg ^tftbs iy Nrhwo Quid`sebrs,
fo `lpr`ot hj teb Nrhwo Quid`se`ok Krhup,
f g`v`s`ho hj Qboku`o Yfoghl Ehusb DDN, Obw Phrm.
www.nrhwopuid`se`ok.nhl
NYH[O fog teb Nrhwo nhdhpeho frb rbk`stbrbgtrfgblfrms hj Qboku`o Yfoghl Ehusb DDN.
D`irfry hj Nhokrbss Nftfdhk`ok-̀ o-Quid`nft`ho Gftf
]N@Q gftf\
@^IO 0;5-1-5176->::0->
b@^IO 0;5-1-5176->:;1-0
Qr`otbg `o teb Ro`tbg ^tftbs hj Flbr`nf
@ddustrft`hos iy Chb DbLhoo`br
Cfnmbt gbs`ko iy
9 7 : 5 61 0 ; 2 > 6
J`rst Bg`t`ho
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
6/29
6
Fo Hpt`l`st`n ^mbpt`n
[b frb fdd jhrbnfstbrs. [ebo wb te`om fihut nefok`ok chis, kbt-
t`ok lfrr`bg, iuy`ok f ehlb, lfm`ok fo `ovbstlbot, dfuone-
`ok f prhgunt, hr rbt`r`ok, wb gbn`gb ifsbg ho ehw wb bxpbnt tebjuturb w`dd uojhdg. Websb bxpbntft`hos frb jhrbnfsts. Hjtbo wb gh hur
hwo jhrbnfst`ok. Iut webo i`k bvbots efppbo‗lfrmbts nrfse, wfrs
dhhl, dbfgbrs trblidb‗ wb turo th teb bxpbrts, tehsb `o teb mohw.
[b dhhm th pbhpdb d`mb Whl Jr`bglfo.
@j yhu frb f [e`tb Ehusb stfjjbr, yhu l`ket stog e`l `o teb
Hvfd Hjstnb w`te teb prbs`gbot hj teb Ro`tbg ^tftbs, tfdm`ok fihut
teb L`ggdb Bfst. @j yhu frb f Jhrtuob 211 NBH, yhu l`ket spht e`l
`o Gfvhs, neftt`ok `o teb dhuokb w`te ebgkb juog i`dd`hof`rbs fog
^fug` pr`onbs. Fog ̀ j yhu gho‐t jrbqubot teb [e`tb Ehusb hr swfomy
^w`ss ehtbds, yhu nfo rbfg e`s Obw Phrm W`lbs nhdulos fog ibstsbdd-
`ok ihhms teft tbdd yhu weft‐s efppbo`ok ohw, wey, fog weft w`dd
nhlb obxt.6 L`dd`hos gh.
D`mb Whl Jr`bglfo, I`dd Jdfnm jhrbnfsts kdhifd bvbots. Iut tebrb`s f dht dbss gblfog jhr e`s `os`kets.
Jhr ybfrs, I`dd whrmbg jhr teb R^ Gbpfrtlbot hj Fkr`nudturb `o
Fr`zhof‗—pfrt p`nm-fog-sehvbd whrm, pfrt sprbfgsebbt“‗iut ohw
eb d`vbs `o Mbfroby, Obirfsmf. I`dd `s f oft`vb Nhroeusmbr. Eb krbw
up `o Lfg`sho, Obirfsmf, f jfrl thwo webrb e`s pfrbots hwobg fog
puid`sebg teb Lfg`sho ^tfr- Lf`d , f obwspfpbr w`te dhts hj sthr`bs
fihut dhnfd sphrts fog nhuoty jf`rs. Eb wfs f khhg stugbot `o e`ke
e d g e i e d j ` g j e
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
7/29
9 supbrjhrbnfst`ok
Fr`zhof. Eb wfs f`l`ok jhr f QeG `o lfte, iut eb rbfd`zbg `t wfs
ibyhog e`s fi`d`t`bs‗—@ efg ly ohsb ruiibg `o ly d`l`tft`hos“ `s
ehw eb puts `t‗fog eb grhppbg hut. @t wfso‐t wfstbg t`lb, ehwbvbr.Ndfssbs `o hro`tehdhky lfgb I`dd fo fv`g i`rg- wftnebr, fog ibnfusb
Fr`zhof `s f krbft pdfnb th sbb i`rgs, eb g`g stbdgwhrm pfrt-t`lb jhr
sn`bot`sts, tebo kht f chi w`te teb Gbpfrtlbot hj Fkr`nudturb fog
stfybg jhr f we`db.
I`dd `s stjty-stvb fog rbt`rbg, fdtehuke eb sfys `j shlbhob hjjbrbg
e`l f chi eb whudg nhos`gbr `t. ^h eb efs jrbb t`lb. Fog eb spbogs
shlb hj `t jhrbnfst`ok.I`dd efs foswbrbg rhukedy terbb euogrbg qubst`hos d`mb —[`dd
Yuss`f hjstn`fddy foobx fgg`t`hofd Rmrf`o`fo tbrr`thry `o teb obxt
terbb lhotes8“ fog —@o teb obxt ybfr, w`dd foy nhuotry w`tegrfw
jrhl teb burhzhob8“ Weby frb qubst`hos teft lfttbr. Fog teby‐rb
g`jstnudt. Nhrphrft`hos, ifoms, blifss`bs, fog `otbdd`kbonb fkbon`bs
strukkdb th foswbr sune qubst`hos fdd teb t`lb. —[`dd Ohrte Mhrbf
gbthoftb f oundbfr gbv`nb ibjhrb teb bog hj te`s ybfr8“ —Ehw lfoy
fgg`t`hofd nhuotr`bs w`dd rbphrt nfsbs hj teb Bihdf v`rus `o teb obxt
b`ket lhotes8“ —[`dd @og`f hr Irfz`d ibnhlb f pbrlfobot lblibr hj
teb RO ^bnur`ty Nhuon`d `o teb obxt twh ybfrs8“ ^hlb hj teb qubs-
t`hos frb ghwor`ket hisnurb, ft dbfst jhr lhst hj us. —[`dd OFWH
`ov`tb obw nhuotr`bs th ch`o teb Lblibrse`p Fnt`ho Qdfo (LFQ)
`o teb obxt o`ob lhotes8“ —[`dd teb Murg`stfo Ybk`hofd Khvbro-lbot ehdg f rbjbrbogul ho oft`hofd `ogbpbogbonb te`s ybfr8“ —@j
f oho-Ne`obsb tbdbnhlluo`nft`hos strl w`os f nhotrfnt th prhv`gb
@otbrobt sbrv`nbs `o teb ^efokef` Jrbb Wrfgb Shob `o teb obxt twh
ybfrs, w`dd Ne`obsb n`t`zbos efvb fnnbss th Jfnbihhm fog/hr Ww`ttbr8“
[ebo I`dd strst sbbs hob hj tebsb qubst`hos, eb lfy efvb oh ndub ehw
th foswbr `t. —[eft ho bfrte `s teb ^efokef` Jrbb Wrfgb Shob8“ eb
lfy te`om. Iut eb ghbs e`s ehlbwhrm. Eb kftebrs jfnts, ifdfonbs
d e` g d
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
8/29
Fo Hpt`l`st`n ^mbpt`n >
sefrb e`s tehukets ho NOO. Eb efs obvbr ibbo `ov`tbg th Gfvhs th
s`t ho f pfobd w`te Whl Jr`bglfo. Fog teft‐s uojhrtuoftb. Ibnfusb
I`dd Jdfnm `s f rblfrmfidb jhrbnfstbr. [b mohw teft ibnfusb bfne hobhj I`dd‐s prbg`nt`hos efs ibbo gftbg, rbnhrgbg, fog fssbssbg jhr fnnu-
rfny iy ̀ ogbpbogbot sn`bot`stn hisbrvbrs. E`s trfnm rbnhrg ̀ s bxnbddbot.
I`dd `s oht fdhob. Webrb frb tehusfogs hj htebrs foswbr`ok teb
sflb qubst`hos. Fdd frb vhduotbbrs. Lhst frbo‐t fs khhg fs I`dd, iut
fihut 9% frb. Weby `ondugb bok`obbrs fog dfwybrs, frt`sts fog sn`bo-
t`sts, [fdd ^trbbtbrs fog Lf`o ^trbbtbrs, prhjbsshrs fog stugbots. [b
w`dd lbbt lfoy hj tebl, `ondug`ok f lfteblft`n`fo, f stdllfmbr,fog shlb rbt`rbbs bfkbr th sefrb teb`r uogbrusbg tfdbots. @ nfdd tebl
supbrjhrbnfstbrs ibnfusb teft ̀ s weft teby frb. Ybd`fidb bv`gbonb prhvbs
`t. Bxpdf`o`ok wey teby‐rb sh khhg, fog ehw htebrs nfo dbfro th gh
weft teby gh, `s ly khfd `o te`s ihhm.
Ehw hur dhw-prhstdb supbrjhrbnfstbrs nhlpfrb w`te nbrbirfd nb-
dbir`t`bs d`mb Whl Jr`bglfo `s fo `otr`ku`ok qubst`ho, iut `t nfo‐t ib
foswbrbg ibnfusb teb fnnurfny hj Jr`bglfo‐s jhrbnfst`ok efs obvbr
ibbo r`khrhusdy tbstbg. Hj nhursb Jr`bglfo‐s jfos fog nr`t`ns efvb
hp`o`hos hob wfy hr teb htebr‗—eb of`dbg teb Frfi ^pr`ok“ hr —eb
snrbwbg up ho teb 911> `ovfs`ho hj @rfq“ hr —eb wfs prbsn`bot ho
OFWH bxpfos`ho.“ Iut tebrb frb oh efrg jfnts fihut Whl Jr`bg-
lfo‐s trfnm rbnhrg, cust bogdbss hp`o`hos‗fog hp`o`hos ho hp`o-
`hos.9
Fog teft ̀ s ius`obss fs usufd. Bvbry gfy, teb obws lbg`f gbd`vbrjhrbnfsts w`tehut rbphrt`ok, hr bvbo fsm`ok, ehw khhg teb jhrbnfst-
brs weh lfgb teb jhrbnfsts rbfddy frb. Bvbry gfy, nhrphrft`hos fog
khvbrolbots pfy jhr jhrbnfsts teft lfy ib prbsn`bot hr whrtedbss hr
shlbte`ok `o ibtwbbo. Fog bvbry gfy, fdd hj us‗dbfgbrs hj oft`hos,
nhrphrftb bxbnut`vbs, `ovbsthrs, fog vhtbrs‗lfmb nr`t`nfd gbn`s`hos
ho teb ifs`s hj jhrbnfsts wehsb qufd`ty `s uomohwo. Ifsbifdd lfo-
fkbrs whudgo‐t grbfl hj kbtt`ok hut teb nebnmihhm th e`rb f pdfybr
` e d ` j ` ` B j
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
9/29
7 supbrjhrbnfst`ok
th teb jhrbnfstbrs weh ebdp us lfmb gbn`s`hos teft lfttbr jfr lhrb
tefo foy ifsbifdd kflb, wb‐rb nhotbot th ib `kohrfot.>
@o teft d`ket, rbdy`ok ho I`dd Jdfnm‐s jhrbnfsts dhhms qu`tb rbfsho-fidb. @ogbbg, rbdy`ok ho teb jhrbnfsts hj lfoy rbfgbrs hj te`s ihhm
lfy prhvb qu`tb rbfshofidb, jhr `t turos hut teft jhrbnfst`ok `s oht f
—yhu efvb `t hr yhu gho‐t“ tfdbot. @t `s f sm`dd teft nfo ib nudt`vftbg.
We`s ihhm w`dd sehw yhu ehw.
WEB HOB FIHRW WEB NE@LQ
@ wfot th sph`d teb chmb, sh @‐dd k`vb fwfy teb puone d`ob3 teb fvbrfkb
bxpbrt wfs rhukedy fs fnnurftb fs f gfrt-terhw`ok ne`lpfozbb.
Phu‐vb prhifidy ebfrg teft hob ibjhrb. @t‐s jflhus‗`o shlb n`r-
ndbs, `ojflhus. @t efs phppbg up `o teb Obw Phrm W`lbs , teb [fdd
^trbbt Churofd , teb J`ofon`fd W`lbs , teb Bnhohl`st , fog htebr hutdbts
frhuog teb whrdg. @t khbs d`mb te`s3 F rbsbfrnebr kftebrbg f i`k krhup
hj bxpbrts‗fnfgbl`ns, puog`ts, fog teb d`mb‗th lfmb tehusfogs
hj prbg`nt`hos fihut teb bnhohly, sthnms, bdbnt`hos, wfrs, fog htebr
`ssubs hj teb gfy. W`lb pfssbg, fog webo teb rbsbfrnebr nebnmbg teb
fnnurfny hj teb prbg`nt`hos, eb jhuog teft teb fvbrfkb bxpbrt g`g
fihut fs wbdd fs rfoghl kubss`ok. Bxnbpt teft‐s oht teb puone d`ob
ibnfusb —rfoghl kubss`ok“ `so‐t juooy. Web puone d`ob `s fihut fgfrt-terhw`ok ne`lpfozbb. Ibnfusb ne`lpfozbbs frb juooy.
@ fl teft rbsbfrnebr fog jhr f we`db @ g`go‐t l`og teb chmb. Ly
stugy wfs teb lhst nhlprbebos`vb fssbsslbot hj bxpbrt cugklbot `o
teb sn`bot`stn d`tbrfturb. @t wfs f dhok sdhk teft thhm fihut twboty ybfrs,
jrhl 6057 th 9117, fog teb rbsudts wbrb jfr r`nebr fog lhrb nhostrun-
t`vb tefo teb puone d`ob sukkbstbg. Iut @ g`go‐t l`og teb chmb ibnfusb
`t rf`sbg fwfrbobss hj ly rbsbfrne (fog, ybs, sn`bot`sts sfvhr teb`r stj-
` j j ) F g @ dj e g g e dg —g
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
10/29
Fo Hpt`l`st`n ^mbpt`n 2
@ fdsh g`go‐t l`og ibnfusb teb chmb lfmbs f vfd`g ph`ot. Hpbo
foy obwspfpbr, wftne foy WZ obws sehw, fog yhu stog bxpbrts weh
jhrbnfst weft‐s nhl`ok. ^hlb frb nfut`hus. Lhrb frb ihdg fog nhost-gbot. F efogjud ndf`l th ib Hdylp`fo v`s`hofr`bs fidb th sbb gbnfgbs
`oth teb juturb. [`te jbw bxnbpt`hos, teby frb oht `o jrhot hj teb
nflbrfs ibnfusb teby phssbss foy prhvbo sm`dd ft jhrbnfst`ok. Fnnu-
rfny ̀ s sbdghl bvbo lbot`hobg. Hdg jhrbnfsts frb d`mb hdg obws‗shho
jhrkhttbo‗fog puog`ts frb fdlhst obvbr fsmbg th rbnhon`db weft
teby sf`g w`te weft fntufddy efppbobg. Web hob uogbo`fidb tfdbot
teft tfdm`ok ebfgs efvb `s teb`r sm`dd ft tbdd`ok f nhlpbdd`ok sthry w`tenhov`nt`ho, fog teft `s bohuke. Lfoy efvb ibnhlb wbfdtey pbggd`ok
jhrbnfst`ok hj uotbstbg vfdub th nhrphrftb bxbnut`vbs, khvbrolbot hj-
stn`fds, fog hrg`ofry pbhpdb weh whudg obvbr te`om hj swfddhw`ok
lbg`n`ob hj uomohwo bjstnfny fog sfjbty iut weh rhut`obdy pfy jhr
jhrbnfsts teft frb fs gui`hus fs bd`x`rs shdg jrhl teb ifnm hj f wfkho.
Websb pbhpdb‗fog teb`r nusthlbrs‗gbsbrvb f ougkb `o teb r`is. @
wfs efppy th sbb ly rbsbfrne usbg th k`vb t̀ th tebl.
Iut @ rbfd`zbg teft fs whrg hj ly whrm sprbfg, ̀ ts fppfrbot lbfo-
`ok wfs lutft`ok. [eft ly rbsbfrne efg sehwo wfs teft teb fvbrfkb
bxpbrt efg ghob d`ttdb ibttbr tefo kubss`ok ho lfoy hj teb phd`t`nfd
fog bnhohl`n qubst`hos @ efg phsbg. —Lfoy“ ghbs oht bqufd fdd. @t
wfs bfs`bst th ibft nefonb ho teb sehrtbst-rfokb qubst`hos teft hody
rbqu`rbg dhhm`ok hob ybfr hut, fog fnnurfny jbdd hjj teb jurtebr hut bx-pbrts tr`bg th jhrbnfst‗fpprhfne`ok teb gfrt-terhw`ok-ne`lpfozbb
dbvbd terbb th stvb ybfrs hut. Weft wfs fo `lphrtfot stog`ok. @t tbdds
us shlbte`ok fihut teb d`l`ts hj bxpbrt`sb `o f nhlpdbx whrdg‗fog
teb d`l`ts ho weft `t l`ket ib phss`idb jhr bvbo supbrjhrbnfstbrs th
fne`bvb. Iut fs `o teb ne`dgrbo‐s kflb hj —tbdbpehob,“ `o we`ne f
perfsb `s we`spbrbg th hob ne`dg weh pfssbs `t ho th fohtebr, fog
sh ho, fog bvbryhob `s sehnmbg ft teb bog th g`snhvbr ehw lune `t
e e g e d id g ` e dd
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
11/29
: supbrjhrbnfst`ok
bxpbrt jhrbnfsts frb usbdbss,“ we`ne `s ohosbosb. ^hlb vfr`ft`hos wbrb
bvbo nrugbr‗d`mb —bxpbrts mohw oh lhrb tefo ne`lpfozbbs.“ Ly
rbsbfrne efg ibnhlb f ifnmsthp rbjbrbonb jhr o`e`d`sts weh sbb tebjuturb fs `oebrbotdy uoprbg`ntfidb fog mohw-ohte`ok phpud`sts weh
`os`st ho prbnbg`ok —bxpbrt“ w`te —sh-nfddbg.“
^h @ t`rbg hj teb chmb. Ly rbsbfrne g`g oht supphrt tebsb lhrb
bxtrblb nhondus`hos, ohr g`g @ jbbd foy fjsto`ty jhr tebl. Whgfy, teft
`s fdd teb lhrb trub.
Webrb ̀ s pdboty hj rhhl th stfmb hut rbfshofidb phs`t`hos ibtwbbo
teb gbiuombrs fog teb gbjbogbrs hj bxpbrts fog teb`r jhrbnfsts. Hoteb hob efog, teb gbiuombrs efvb f ph`ot. Webrb frb sefgy pbggdbrs
hj qubst`hofidb `os`kets `o teb jhrbnfst`ok lfrmbtpdfnb. Webrb frb
fdsh d`l`ts th jhrbs`ket teft lfy cust oht ib surlhuotfidb. Hur gbs`rb
th rbfne `oth teb juturb w`dd fdwfys bxnbbg hur krfsp. Iut gbiuom-
brs kh thh jfr webo teby g`sl`ss fdd jhrbnfst`ok fs f jhhd‐s brrfog.
@ ibd`bvb `t `s phss`idb th sbb `oth teb juturb, ft dbfst `o shlb s`tuf-
t`hos fog th shlb bxtbot, fog teft foy ̀ otbdd`kbot, hpbo-l`ogbg, fog
efrgwhrm`ok pbrsho nfo nudt`vftb teb rbqu`s`tb sm`dds.
Nfdd lb fo —hpt`l`st`n smbpt`n.“
WEB ^MBQW@N
Wh uogbrstfog teb —smbpt`n“ efdj hj teft dfibd, nhos`gbr f yhuok Wu-
o`s`fo lfo puse`ok f whhgbo efognfrt dhfgbg w`te jru`ts fog vbk-
btfidbs ghwo f gusty rhfg th f lfrmbt `o teb Wuo`s`fo thwo hj ^`g`
Ihuz`g. [ebo teb lfo wfs terbb, e`s jftebr g`bg. Eb supphrts e`s
jfl`dy iy ihrrhw`ok lhoby th stdd e`s nfrt, ehp`ok th bfro bohuke
sbdd`ok teb prhgunb th pfy hjj teb gbit fog efvb f d`ttdb dbjt hvbr. @t‐s
teb sflb kr`og bvbry gfy. Iut te`s lhro`ok, teb phd`nb fpprhfne teb
g e ‐ ` m e` d i e e ` d g
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
12/29
Fo Hpt`l`st`n ^mbpt`n ;
eb efs oh lhoby. F phd`nbwhlfo sdfps e`l fog `osudts e`s gbfg jf-
tebr. Weby tfmb e`s snfdbs fog e`s nfrt. Web lfo khbs th f thwo hjstnb
th nhlpdf`o. Eb `s thdg teb hjstn`fd `s iusy `o f lbbt`ok. Eul`d`ftbg,jur`hus, phwbrdbss, teb lfo dbfvbs.
Eb rbturos w`te jubd. Huts`gb teb thwo hjstnb eb ghusbs e`lsbdj,
d`kets f lftne, fog iuros.
Hody teb nhondus`ho hj te`s sthry `s uousufd. Webrb frb nhuotdbss
phhr strbbt vboghrs `o Wuo`s`f fog fnrhss teb Frfi whrdg. Qhd`nb nhr-
rupt`ho `s r`jb, fog eul`d`ft`hos d`mb tehsb `o`ntbg ho te`s lfo frb
f gf`dy hnnurrbonb. Weby lfttbr th oh hob fs`gb jrhl teb phd`nb fogteb`r v`nt`ls.
Iut te`s pfrt`nudfr eul`d`ft`ho, ho Gbnblibr 6;, 9161, nfusbg
Lheflbg Ihufz`z`, fkbg twboty-s`x, th sbt e`lsbdj ho strb, fog
Ihufz`z`‐s sbdj-`llhdft`ho spfrmbg prhtbsts. Web phd`nb rbsphogbg
w`te typ`nfd irutfd`ty. Web prhtbsts sprbfg. Ehp`ok th fssufkb teb
puid`n, teb g`ntfthr hj Wuo`s`f, Qrbs`gbot S`ob bd- Fi`g`ob Ibo Fd`,
v`s`tbg Ihufz`z` `o teb ehsp`tfd.
Ihufz`z` g`bg ho Cfoufry 7, 9166. Web uorbst krbw. Ho Cfoufry
67, Ibo Fd` bg th f nusey bx`db `o ^fug` Frfi`f, bog`ok e`s twboty-
terbb-ybfr mdbpthnrfny.
Web Frfi whrdg wftnebg, stuoobg. Webo prhtbsts bruptbg `o
Bkypt, D`iyf, ^yr`f, Chrgfo, Muwf`t, fog Iferf`o. Fjtbr terbb gb-
nfgbs `o phwbr, teb Bkypt`fo g`ntfthr Ehso` Luifrfm wfs gr`vbojrhl hjstnb. Bdsbwebrb, prhtbsts swbddbg `oth rbibdd`hos, rbibdd`hos `oth
n`v`d wfrs. We`s wfs teb Frfi ^pr`ok‗fog `t stfrtbg w`te hob phhr
lfo, oh g`jjbrbot jrhl nhuotdbss htebrs, ib`ok efrfssbg iy phd`nb, fs
sh lfoy efvb ibbo, ibjhrb fog s`onb, w`te oh fppfrbot r`ppdb bjjbnts.
@t `s hob te`ok th dhhm ifnmwfrg fog smbtne f ofrrft`vb frn, fs @
g`g ebrb, nhoobnt`ok Lheflbg Ihufz`z` th fdd teb bvbots teft hwbg
hut hj e`s dhobdy prhtbst. Whl Jr`bglfo, d`mb lfoy bd`tb puog`ts, `s
m`dd g e j ` ` d d ` e L`ggd B
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
13/29
5 supbrjhrbnfst`ok
Obw Phrm W`lbs nhrrbsphogbot `o Dbifoho. Iut nhudg bvbo Whl
Jr`bglfo, `j eb efg ibbo prbsbot teft jftfd lhro`ok, efvb pbbrbg
`oth teb juturb fog jhrbsbbo teb sbdj-`llhdft`ho, teb uorbst, teb thp-pd`ok hj teb Wuo`s`fo g`ntfthr, fog fdd teft jhddhwbg8 Hj nhursb oht.
Oh hob nhudg. Lfyib, k`vbo ehw lune Jr`bglfo mobw fihut teb
rbk`ho, eb whudg efvb lusbg teft phvbrty fog uoblpdhylbot wbrb
e`ke, teb oulibr hj gbspbrftb yhuok pbhpdb wfs krhw`ok, nhrrup-
t`ho wfs rflpfot, rbprbss`ho wfs rbdbotdbss, fog tebrbjhrb Wuo`s`f
fog htebr Frfi nhuotr`bs wbrb phwgbr mbks wf`t`ok th idhw. Iut
fo hisbrvbr nhudg efvb grfwo bxfntdy teb sflb nhondus`ho teb ybfribjhrb. Fog teb ybfr ibjhrb teft. @ogbbg, yhu nhudg efvb sf`g teft
fihut Wuo`s`f, Bkypt, fog sbvbrfd htebr nhuotr`bs jhr gbnfgbs. Weby
lfy efvb ibbo phwgbr mbks iut teby obvbr idbw‗uot`d Gbnbl-
ibr 6;, 9161, webo teb phd`nb pusebg teft hob phhr lfo thh jfr.
@o 60;9 teb Flbr`nfo lbtbhrhdhk`st Bgwfrg Dhrboz wrhtb
f pfpbr w`te fo frrbst`ok t`tdb3 —Qrbg`ntfi`d`ty3 Ghbs teb Jdfp hj f
Iuttbry‐s [`oks `o Irfz`d ^bt Hjj f Whrofgh `o Wbxfs8“ F gbnfgb
bfrd`br, Dhrboz efg g`snhvbrbg iy fnn`gbot teft t`oy gftf botry vfr`f-
t`hos `o nhlputbr s`ludft`hos hj wbftebr pfttbros‗d`mb rbpdfn`ok
1.21:69; w`te 1.21:‗nhudg prhgunb grflft`nfddy g`jjbrbot dhok-
tbrl jhrbnfsts. @t wfs fo ̀ os`ket teft whudg ̀ osp`rb —nefhs tebhry“3 ̀ o
ohod`obfr systbls d`mb teb ftlhspebrb, bvbo slfdd nefokbs `o `o`t`fd
nhog`t`hos nfo luserhhl th bohrlhus prhphrt`hos. ^h, `o pr`on`pdb,f dhob iuttbry `o Irfz`d nhudg fp `ts w`oks fog sbt hjj f throfgh `o
Wbxfs‗bvbo tehuke swfrls hj htebr Irfz`d`fo iuttbr`bs nhudg fp
jrfot`nfddy teb`r wehdb d`vbs fog obvbr nfusb f oht`nbfidb kust f jbw
l`dbs fwfy. Hj nhursb Dhrboz g`go‐t lbfo teft teb iuttbry —nfusbs“
teb throfgh `o teb sflb sbosb teft @ nfusb f w`obkdfss th irbfm webo
@ e`t `t w`te f efllbr. Eb lbfot teft `j teft pfrt`nudfr iuttbry
efgo‐t fppbg `ts w`oks ft teft lhlbot, teb uojftehlfidy nhlpdbx
m j e ` ` g ` dg e i e g
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
14/29
Fo Hpt`l`st`n ^mbpt`n 0
Frfi ^pr`ok l`ket obvbr efvb efppbobg, ft dbfst oht webo fog fs `t
g`g, `j teb phd`nb efg cust dbt Lheflbg Ihufz`z` sbdd e`s jru`ts fog
vbkbtfidbs teft lhro`ok `o 9161.Bgwfrg Dhrboz se`jtbg sn`bot`stn hp`o`ho thwfrg teb v`bw teft
tebrb frb efrg d`l`ts ho prbg`ntfi`d`ty, f gbbpdy pe`dhshpe`nfd qubs-
t`ho.7 Jhr nbotur`bs, sn`bot`sts efg supphsbg teft krhw`ok mohwd-
bgkb lust dbfg th krbftbr prbg`ntfi`d`ty ibnfusb rbfd`ty wfs d`mb f
ndhnm‗fo fwbshlbdy i`k fog nhlpd`nftbg ndhnm iut st`dd f ndhnm‗
fog teb lhrb sn`bot`sts dbfrobg fihut `ts `oofrgs, ehw teb kbfrs
kr`og thkbtebr, ehw teb wb`kets fog spr`oks juont`ho, teb ibttbr tebynhudg nfpturb `ts hpbrft`hos w`te gbtbrl`o`st`n bquft`hos fog prbg`nt
weft t̀ whudg gh. @o 6567 teb Jrbone lfteblft`n`fo fog fstrhohlbr
Q`brrb-^`lho Dfpdfnb thhm te`s grbfl th `ts dhk`nfd bxtrblb3
[b lfy rbkfrg teb prbsbot stftb hj teb uo`vbrsb fs teb bjjbnt
hj `ts pfst fog teb nfusb hj `ts juturb. Fo `otbddbnt we`ne ft f
nbrtf`o lhlbot whudg mohw fdd jhrnbs teft sbt ofturb `o lh-
t`ho, fog fdd phs`t`hos hj fdd `tbls hj we`ne ofturb `s nhlphsbg,
`j te`s `otbddbnt wbrb fdsh vfst bohuke th suil`t tebsb gftf th
fofdys`s, `t whudg blirfnb `o f s`okdb jhrludf teb lhvblbots
hj teb krbftbst ihg`bs hj teb uo`vbrsb fog tehsb hj teb t`o`bst
fthl= jhr sune fo `otbddbnt ohte`ok whudg ib uonbrtf`o fog teb
juturb cust d`mb teb pfst whudg ib prbsbot ibjhrb `ts bybs.
Dfpdfnb nfddbg e`s `lfk`ofry bot`ty f —gblho.“ @j `t mobw bvbry-
te`ok fihut teb prbsbot, Dfpdfnb tehuket, t̀ nhudg prbg`nt bvbryte`ok
fihut teb juturb. @t whudg ib hlo`sn`bot.2
Dhrboz phurbg nhdg rf`owftbr ho teft grbfl. @j teb ndhnm syl-
ihd`zbs pbrjbnt Dfpdfnbfo prbg`ntfi`d`ty, `ts hpphs t̀b `s teb Dhrboz`fo
ndhug. E`ke snehhd sn`bonb tbdds us teft ndhugs jhrl webo wftbr
d g g ` d We` g ` d i
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
15/29
61 supbrjhrbnfst`ok
ho nhlpdbx jbbgifnm `otbrfnt`hos flhok grhpdbts. Wh nfpturb tebsb
`otbrfnt`hos, nhlputbr lhgbdbrs obbg bquft`hos teft frb e`kedy sbo-
s`t`vb th t`oy iuttbry-bjjbnt brrhrs `o gftf nhddbnt`ho. ^h bvbo `j wbdbfro fdd teft `s mohwfidb fihut ehw ndhugs jhrl, wb w`dd oht ib fidb
th prbg`nt teb sefpb f pfrt`nudfr ndhug w`dd tfmb. [b nfo hody wf`t
fog sbb. @o hob hj e`sthry‐s krbft `rho`bs, sn`bot`sts thgfy mohw vfstdy
lhrb tefo teb`r nhddbfkubs f nbotury fkh, fog phssbss vfstdy lhrb
gftf-nruone`ok phwbr, iut teby frb lune dbss nhostgbot `o teb prhs-
pbnts jhr pbrjbnt prbg`ntfi`d`ty.
We`s `s f i`k rbfsho jhr teb —smbpt`n“ efdj hj ly —hpt`l`st`n smbp-t`n“ stfonb. [b d`vb `o f whrdg webrb teb fnt`hos hj hob obfrdy phw-
brdbss lfo nfo efvb r`ppdb bjjbnts frhuog teb whrdg‗r`ppdbs teft
fjjbnt us fdd th vfry`ok gbkrbbs. F whlfo d`v`ok `o f Mfosfs N`ty
suiuri lfy te`om Wuo`s`f `s fohtebr pdfobt, fog ebr d`jb efs oh nho-
obnt`ho th `t, iut `j seb wbrb lfrr`bg th fo f`r jhrnb ofv`kfthr weh
`bs hut hj teb obfriy [e`tblfo F`r Jhrnb Ifsb, seb l`ket ib sur-
pr`sbg th dbfro teft hob hisnurb Wuo`s`fo‐s fnt`hos dbg th prhtbsts,
teft dbg th r`hts, teft dbg th teb thppd`ok hj f g`ntfthr, teft dbg th
prhtbsts `o D`iyf, teft dbg th f n`v`d wfr, teft dbg th teb 9169 OFWH
`otbrvbot`ho, teft dbg th ebr eusifog ghgk`ok fot`f`rnrfjt strb hvbr
Wr`phd`. Weft‐s fo bfs`dy trfnbfidb nhoobnt`ho. Hjtbo teb nhoobnt`hos
frb efrgbr th spht, iut teby frb fdd frhuog us, `o te`oks d`mb teb pr`nb
wb pfy ft teb kfs stft`ho hr teb dfyhjjs ghwo teb strbbt. @o f whrdg webrb f iuttbry `o Irfz`d nfo lfmb teb g`jjbrbonb ibtwbbo cust fo-
htebr suooy gfy `o Wbxfs fog f throfgh tbfr`ok terhuke f thwo, `t‐s
l`sku`gbg th te`om foyhob nfo sbb vbry jfr `oth teb juturb.:
WEB HQW@L@^W
I ` ` e` ` e d` ` g` i`d` g `
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
16/29
Fo Hpt`l`st`n ^mbpt`n 66
Nrfom up teb l`nrhsnhpb ho f gfy `o teb d`jb hj teft whlfo
d`v`ok `o f Mfosfs N`ty suiuri3 Ft :3>1 `o teb lhro`ok, seb grhps
pfpbrs `oth f ir`bjnfsb, kbts `o ebr nfr, gr`vbs ebr usufd rhutb th whrm,fog pfrms ghwothwo. Fs seb ghbs bvbry wbbmgfy lhro`ok, seb wfdms
pfst teb stftubs hj d`hos fog `oth teb Krbbm-`osp`rbg hjstnb iu`dg`ok
hj teb Mfosfs N`ty D`jb @osurfonb Nhlpfoy. Ft ebr gbsm, seb whrms
ho sprbfgsebbts jhr f we`db, pfrt`n`pftbs `o f nhojbrbonb nfdd ft 613>1,
spbogs f jbw l`outbs irhws`ok ho teb Flfzho wbis`tb, fog foswbrs
b-lf`ds uot`d 66321. Webo seb wfdms th f d`ttdb @tfd`fo rbstfurfot th
efvb duone w`te ebr s`stbr.We`s whlfo‐s d`jb `s `oubonbg iy lfoy uoprbg`ntfidb jfnthrs‗
jrhl teb dhttbry t`nmbt `o ebr pursb th teb Frfi ^pr`ok teft rbsudts `o
ebr eusifog y`ok l`ss`hos hvbr D`iyf th teb jfnt teft teb pr`nb hj
kfs cust wbot up stvb nbots f kfddho ibnfusb tebrb wfs f nhup `o shlb
nhuotry seb‐s obvbr ebfrg hj‗iut tebrb `s fs lune hr lhrb teft `s
qu`tb prbg`ntfidb. [ey g`g seb dbfvb ehlb ft :3>18 ^eb g`go‐t wfot
th kbt stunm `o ruse ehur. Hr th put teft fohtebr wfy, seb prbg`ntbg
teft trfjstn whudg ib lune ebfv`br dftbr‗fog seb wfs fdlhst nbr-
tf`ody r`ket ibnfusb ruse ehur `s e`kedy prbg`ntfidb. [ebo seb grhvb,
seb fot`n`pftbg htebr gr`vbrs‐ ibefv`hr nhostfotdy3 teby w`dd sthp ft
teb `otbrsbnt`ho webo teb d`ket `s rbg= teby w`dd stfy `o teb`r dfobs
fog s`kofd ibjhrb turo`ok. ^eb bxpbntbg teb pbhpdb weh sf`g teby
whudg ch`o teb 613>1 nhojbrbonb nfdd th gh sh, fog seb wfs r`ket.^eb frrfokbg th lbbt ebr s`stbr ft ohho ft teb rbstfurfot ibnfusb teb
rbstfurfot‐s phstbg ehurs `og`nftbg `t whudg hpbo tebo, fog phstbg
ehurs frb f rbd`fidb ku`gb.
[b lfmb luogfob prbg`nt`hos d`mb tebsb rhut`obdy, we`db htebrs
cust fs rhut`obdy lfmb prbg`nt`hos teft sefpb hur d`vbs. [ebo teb
whlfo turobg ebr nhlputbr ho, seb `onrbfsbg bdbntr`n`ty nhosulp-
t`ho ̀ o Mfosfs N`ty iy f i`t, fs bvbry htebr whrmbr ibb g`g teft lhro-
` g dd ` d e g g g e g
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
17/29
69 supbrjhrbnfst`ok
prhidbls ibnfusb bdbntr`n`ty prhgunbrs fot`n`pftb tebsb surkbs fog
vfry teb`r hutput fnnhrg`okdy. [ebo teb whlfo wbot th Flfzho,
teb wbis`tb e`ked`ketbg nbrtf`o prhgunts `t tehuket seb whudg d`mb, fjhrbnfst gbr`vbg jrhl ebr pfst purnefsbs fog irhws`ok fog teft hj
l`dd`hos hj htebrs. [b nhostfotdy bonhuotbr prbg`nt`vb hpbrft`hos d`mb
teft ho teb @otbrobt‗Khhkdb pbrshofd`zbs sbfrne rbsudts iy putt`ok
weft `t te`oms yhu w`dd stog lhst `otbrbst`ok ho thp‗iut teby hpbr-
ftb sh slhhtedy wb rfrbdy oht`nb. Fog tebo tebrb‐s teb whlfo‐s whrm-
pdfnb. Web Mfosfs N`ty D`jb @osurfonb Nhlpfoy `s `o teb ius`obss hj
jhrbnfst`ok g`sfi`d`ty fog gbfte, fog `t ghbs f khhg chi. Weft ghbso‐tlbfo `t mohws prbn`sbdy webo @ w`dd g`b, iut `t ghbs efvb f khhg `gbf
hj ehw dhok shlbhob hj ly fkb fog prhstdb‗sbx, ̀ onhlb, d`jbstydb‗`s
d`mbdy th d`vb. Mfosfs N`ty D`jb wfs jhuogbg `o 6502. @j `ts fntufr`bs
wbrbo‐t khhg jhrbnfstbrs, t̀ whudg efvb khob ifomrupt dhok fkh.
^h lune hj hur rbfd`ty `s te`s prbg`ntfidb, hr lhrb sh. @ cust
Khhkdbg thlhrrhw‐s suor`sb fog suosbt t`lbs jhr Mfosfs N`ty, L`s-
shur`, fog kht tebl ghwo th teb l`outb. Wehsb jhrbnfsts frb rbd`fidb,
webtebr teby frb jhr thlhrrhw, teb gfy fjtbr, hr stjty ybfrs jrhl ohw.
Web sflb `s trub hj t`gbs, bnd`psbs, fog pefsbs hj teb lhho. Fdd nfo
ib prbg`ntbg jrhl ndhnmd`mb sn`bot`stn dfws w`te bohuke prbn`s`ho th
sft`sjy Dfpdfnb‐s jhrbnfst`ok gblho.
Hj nhursb bfne hj tebsb phnmbts hj prbg`ntfi`d`ty nfo ib firuptdy
puonturbg. F khhg rbstfurfot `s vbry d`mbdy th hpbo `ts ghhrs webo `tsfys `t w`dd, iut `t lfy oht, jhr foy oulibr hj rbfshos, jrhl f lfo-
fkbr sdbbp`ok dftb, th strb, ifomruptny, pfogbl`n, oundbfr wfr, hr f
peys`ns bxpbr`lbot fnn`gbotfddy nrbft`ok f idfnm ehdb teft sunms up
teb shdfr systbl. Web sflb `s trub hj foyte`ok bdsb. Bvbo tehsb stjty-
ybfr suor`sb fog suosbt jhrbnfsts nhudg ib hjj shlbweft `j, shlb-
t`lb `o teb obxt stjty ybfrs, f lfss`vb spfnb rhnm iulps Bfrte hjj
`ts hri`t frhuog teb suo. Webrb frb oh nbrtf`ot`bs `o d`jb‗oht bvbo
g e g `j ` i i`d` e ` `
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
18/29
Fo Hpt`l`st`n ^mbpt`n 6>
f ndhug-nhlput`ok obtwhrm fog teb blbrkbonb hj f juturb shn`bty
sh puid`n-sp`r`tbg fog prhspbrhus teft teb stftb nfo ib juogbg w`te
nefr`tfidb ghoft`hos.^h `s rbfd`ty ndhnmd`mb hr ndhug-d`mb8 @s teb juturb prbg`ntfidb hr
oht8 Websb frb jfdsb g`nehthl`bs, teb strst hj lfoy wb w`dd bonhuotbr.
[b d`vb `o f whrdg hj ndhnms fog ndhugs fog f vfst culidb hj htebr
lbtfpehrs. Roprbg`ntfi`d`ty fog prbg`ntfi`d`ty nhbx`st uobfs`dy `o
teb `otr`nftbdy `otbrdhnm`ok systbls teft lfmb up hur ihg`bs, hur
shn`bt`bs, fog teb nhslhs. Ehw prbg`ntfidb shlbte`ok `s gbpbogs ho
weft wb frb try`ok th prbg`nt, ehw jfr `oth teb juturb, fog uogbr weft n`rnulstfonbs.
Dhhm ft Bgwfrg Dhrboz‐s stbdg. [bftebr jhrbnfsts frb typ`nfddy
qu`tb rbd`fidb, uogbr lhst nhog`t`hos, dhhm`ok f jbw gfys febfg,
iut teby ibnhlb `onrbfs`okdy dbss fnnurftb terbb, jhur, fog stvb
gfys hut. Lune ibyhog f wbbm, wb l`ket fs wbdd nhosudt teft gfrt-
terhw`ok ne`lpfozbb. ^h wb nfo‐t sfy teft wbftebr `s prbg`ntfidb
hr oht, hody teft wbftebr `s prbg`ntfidb th shlb bxtbot uogbr shlb
n`rnulstfonbs‗fog wb lust ib vbry nfrbjud webo wb try th ib lhrb
prbn`sb tefo teft. Wfmb shlbte`ok fs sbbl`okdy s`lpdb fs teb rbdft`ho-
se`p ibtwbbo t`lb fog prbg`ntfi`d`ty3 `t `s kbobrfddy trub teft teb jur-
tebr wb try th dhhm ̀ oth teb juturb, teb efrgbr ̀ t ̀ s th sbb. Iut tebrb nfo
ib prhdhokbg bxnbpt`hos th teb rudb. Qrbg`nt`ok teb nhot`ouft`ho hj f
dhok iudd lfrmbt ̀ o sthnms nfo prhvb prhsttfidb jhr lfoy ybfrs‗uot`d`t suggbody prhvbs th ib yhur uogh`ok. Fog prbg`nt`ok teft g`ohsfurs
whudg nhot`oub th prbs`gb ft teb thp hj teb jhhg nef`o wfs f sfjb ibt
jhr tbos hj l`dd`hos hj ybfrs‗uot`d fo fstbrh`g sbt hjj f nftfndysl
teft hpbobg up bnhdhk`nfd o`nebs jhr f t`oy lfllfd teft bvbotufddy
bvhdvbg `oth f spbn`bs teft tr`bs th prbg`nt teb juturb. Dfws hj peys`ns
fs`gb, tebrb frb oh uo`vbrsfd nhostfots, sh sbpfrft`ok teb prbg`ntfidb
jrhl teb uoprbg`ntfidb ̀ s g`jstnudt whrm. Webrb‐s oh wfy frhuog ̀ t.
L d ` m e i e We m d
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
19/29
67 supbrjhrbnfst`ok
wey wb mohw teft hob- fog twh-gfy jhrbnfsts frb typ`nfddy qu`tb fn-
nurftb we`db b`ket-gfy jhrbnfsts frb oht. [`te tebsb fofdysbs, lbtbh-
rhdhk`sts frb fidb th sefrpbo teb`r uogbrstfog`ok hj ehw wbftebr whrmsfog twbfm teb`r lhgbds. Webo teby try fkf`o. Jhrbnfst, lbfsurb, rb-
v`sb. Ybpbft. @t‐s f obvbr-bog`ok prhnbss hj `onrblbotfd `lprhvblbot
teft bxpdf`os wey wbftebr jhrbnfsts frb khhg fog sdhwdy kbtt`ok ibttbr.
Webrb lfy ib d`l`ts th sune `lprhvblbots, ehwbvbr, ibnfusb wbftebr
`s teb tbxtihhm `ddustrft`ho hj ohod`obfr`ty. Web jurtebr hut teb jhrb-
nfstbr tr`bs th dhhm, teb lhrb hpphrtuo`ty tebrb `s jhr nefhs th fp `ts
iuttbry w`oks fog idhw fwfy bxpbntft`hos. I`k dbfps `o nhlput`okphwbr fog nhot`oubg rbstoblbot hj jhrbnfst`ok lhgbds lfy ougkb teb
d`l`ts f d`ttdb jurtebr `oth teb juturb iut tehsb fgvfonbs krfgufddy kbt
efrgbr fog teb pfyhjjs ser`om thwfrg zbrh. Ehw khhg nfo `t kbt8 Oh
hob mohws. Iut mohw`ok teb nurrbot d`l`ts `s t̀sbdj f sunnbss.
@o sh lfoy htebr e`ke-stfmbs bogbfvhrs, jhrbnfstbrs frb krhp`ok
`o teb gfrm. Weby efvb oh `gbf ehw khhg teb`r jhrbnfsts frb `o teb
sehrt, lbg`ul, hr dhok tbrl‗fog oh `gbf ehw khhg teb`r jhrbnfsts
nhudg ibnhlb. Ft ibst, teby efvb vfkub euonebs. Weft‐s ibnfusb teb
jhrbnfst-lbfsurb-rbv`sb prhnbgurb hpbrftbs hody w`te`o teb rfrbstbg
nhostobs hj e`ke-tbne jhrbnfst`ok, sune fs teb whrm hj lfnrhbnhoh-
l`sts ft nbotrfd ifoms hr lfrmbt`ok fog stofon`fd prhjbss`hofds `o
i`k nhlpfo`bs hr hp`o`ho phdd fofdysts d`mb Oftb ^`dvbr.; Lhrb hjtbo
jhrbnfsts frb lfgb fog tebo . . . ohte`ok. Fnnurfny `s sbdghl gbtbr-l`obg fjtbr teb jfnt fog `s fdlhst obvbr ghob w`te sujstn`bot rbkudfr-
`ty fog r`khr teft nhondus`hos nfo ib grfwo. Web rbfsho8 Lhstdy ̀ t‐s f
gblfog-s`gb prhidbl3 Web nhosulbrs hj jhrbnfst`ok‗khvbrolbots,
ius`obss, fog teb puid`n‗gho‐t gblfog bv`gbonb hj fnnurfny. ^h
tebrb `s oh lbfsurblbot. [e`ne lbfos oh rbv`s`ho. Fog w`tehut
rbv`s`ho, tebrb nfo ib oh `lprhvblbot. @lfk`ob f whrdg `o we`ne
pbhpdb dhvb th ruo, iut teby efvb oh `gbf ehw jfst teb fvbrfkb pbrsho
e j e i dg i e
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
20/29
Fo Hpt`l`st`n ^mbpt`n 62
teb kuo `s strbg, bog `t fjtbr f spbn`stbg g`stfonb‗fog tebrb frb oh
`ogbpbogbot rfnb hjstn`fds fog t`lbmbbpbrs lbfsur`ok rbsudts. Ehw
d`mbdy `s `t teft ruoo`ok t`lbs frb `lprhv`ok `o te`s whrdg8 Oht vbry. Frb teb ibst ruoobrs ruoo`ok fs jfst fs eulfo ib`oks frb peys`nfddy
nfpfidb8 Fkf`o, prhifidy oht.
—@ efvb ibbo strunm iy ehw `lphrtfot lbfsurblbot `s th `l-
prhv`ok teb eulfo nhog`t`ho,“ I`dd Kftbs wrhtb. —Phu nfo fne`bvb
`onrbg`idb prhkrbss `j yhu sbt f ndbfr khfd fog stog f lbfsurb teft w`dd
gr`vb prhkrbss thwfrg teft khfd. . . . We`s lfy sbbl ifs`n, iut `t `s
flfz`ok ehw hjtbo `t `s oht ghob fog ehw efrg `t `s th kbt r`ket.“5 Eb`s r`ket fihut weft `t tfmbs th gr`vb prhkrbss, fog `t `s surpr`s`ok ehw
rfrbdy `t‐s ghob `o jhrbnfst`ok. Bvbo teft s`lpdb strst stbp‗sbtt`ok f
ndbfr khfd‗efso‐t ibbo tfmbo.
Phu l`ket te`om teb khfd hj jhrbnfst`ok `s th jhrbsbb teb juturb
fnnurftbdy, iut teft‐s hjtbo oht teb khfd, hr ft dbfst oht teb shdb khfd.
^hlbt`lbs jhrbnfsts frb lbfot th botbrtf`o. We`om hj NOIN‐s C`l
Nrflbr w`te e`s —ihhyfe!“ set`nm, hr Cheo LnDfuked`o, teb ehst
hj Web LnDfuked`o Krhup, ibddhw`ok ft e`s pfobd`sts th prbg`nt teb
d`mbd`ehhg hj fo bvbot —ho f snfdb jrhl zbrh th tbo, w`te zbrh rbprb-
sbot`ok zbrh phss`i`d`ty fog tbo rbprbsbot`ok nhlpdbtb lbtfpeys`nfd
nbrt`tugb!“ ^hlbt`lbs jhrbnfsts frb usbg th fgvfonb phd`t`nfd fkbo-
gfs fog kfdvfo`zb fnt`ho‗fs fnt`v`sts ehpb th gh webo teby wfro
hj dhhl`ok ehrrhrs uodbss wb nefokb hur wfys. Webrb `s fdsh grbss-th-`lprbss jhrbnfst`ok‗ we`ne `s weft ifoms gbd`vbr webo teby pfy
f jflhus puog`t th tbdd wbfdtey nd`bots fihut teb kdhifd bnhohly `o
9121. Fog shlb jhrbnfsts frb lbfot th nhljhrt‗iy fssur`ok teb
fug`bonb teft teb`r ibd`bjs frb nhrrbnt fog teb juturb w`dd uojhdg fs
bxpbntbg. Qfrt`sfos frb jhog hj tebsb jhrbnfsts. Weby frb teb nhko`-
t`vb bqu`vfdbot hj sd`pp`ok `oth f wfrl ifte.
We`s culidb hj khfds `s sbdghl fnmohwdbgkbg, we`ne lfmbs `t
g`jst d m` g g @ ‐
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
21/29
6: supbrjhrbnfst`ok
Fog ybt te`s stfkoft`ho `s f i`k rbfsho wey @ fl fo hpt`l`st`n
smbpt`n. [b mohw teft `o sh lune hj weft pbhpdb wfot th prbg`nt‗
phd`t`ns, bnhohl`ns, stofonb, ius`obss, tbneohdhky, gf`dy d`jb‗prbg`nt-fi`d`ty bx`sts, th shlb gbkrbb, `o shlb n`rnulstfonbs. Iut tebrb `s sh
lune bdsb wb gh oht mohw. Jhr sn`bot`sts, oht mohw`ok ̀ s bxn`t`ok. @t‐s
fo hpphrtuo`ty th g`snhvbr= teb lhrb teft ̀ s uomohwo, teb krbftbr teb
hpphrtuo`ty. Wefoms th teb jrfomdy qu`tb flfz`ok dfnm hj r`khr `o sh
lfoy jhrbnfst`ok ghlf`os, te`s hpphrtuo`ty `s eukb. Fog th sb`zb `t,
fdd wb efvb th gh ̀ s sbt f ndbfr khfd‗fnnurfny!‗fog kbt sbr`hus fihut
lbfsur`ok.@‐vb ibbo gh`ok teft jhr lune hj ly nfrbbr. Web rbsbfrne teft
prhgunbg teb gfrt-terhw`ok-ne`lpfozbb rbsudt wfs pefsb hob. Qefsb
twh stfrtbg `o teb sullbr hj 9166, webo ly rbsbfrne (fog d`jb)
pfrtobr Ifrifrf Lbddbrs fog @ dfuonebg teb Khhg Cugklbot Qrhcbnt
fog `ov`tbg vhduotbbrs th s`ko up fog jhrbnfst teb juturb. I`dd Jdfnm
rbsphogbg. ^h g`g f nhupdb hj tehusfog htebrs teft strst ybfr, fog
tehusfogs lhrb `o teb jhur ybfrs teft jhddhwbg. Nuludft`vbdy, lhrb
tefo twboty tehusfog `otbddbntufddy nur`hus dfypbhpdb tr`bg th stkurb
hut `j prhtbsts `o Yuss`f whudg sprbfg, teb pr`nb hj khdg whudg pdul-
lbt, teb O`mmb` whudg ndhsb fihvb 0,211, wfr whudg brupt ho teb
Mhrbfo pbo`osudf, fog lfoy htebr qubst`hos fihut nhlpdbx, nefd-
dbok`ok kdhifd `ssubs. Iy vfry`ok teb bxpbr`lbotfd nhog`t`hos, wb
nhudg kfukb we`ne jfnthrs `lprhvbg jhrbs`ket, iy ehw lune, hvbr we`ne t`lb jrflbs, fog ehw khhg jhrbnfsts nhudg ibnhlb `j ibst
prfnt`nbs wbrb dfybrbg ho bfne htebr. Df`g hut d`mb teft, `t shuogs
s`lpdb. @t wfso‐t. @t wfs f gblfog`ok prhkrfl teft thhm teb tfdbots
fog efrg whrm hj f ludt`g`sn`pd`ofry tbfl ifsbg ft teb Ro`vbrs`ty hj
Nfd`jhro`f, Ibrmbdby, fog teb Ro`vbrs t̀y hj Qboosydvfo`f.
I`k fs `t wfs, teb Khhg Cugklbot Qrhcbnt (KCQ) wfs hody pfrt hj
f lune dfrkbr rbsbfrne bjjhrt sphoshrbg iy teb @otbdd`kbonb Fgvfonbg
Y e Q c F ` ` (@FYQF) G ‐ i jj i e id g
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
22/29
Fo Hpt`l`st`n ^mbpt`n 6;
rbphrts th teb g`rbnthr hj Oft`hofd @otbdd`kbonb fog `ts chi `s th sup-
phrt gfr`ok rbsbfrne teft prhl`sbs th lfmb Flbr`nfo `otbdd`kbonb
ibttbr ft weft `t ghbs. Fog f i`k pfrt hj weft Flbr`nfo `otbdd`kbonbghbs `s jhrbnfst kdhifd phd`t`nfd fog bnhohl`n trbogs. Iy hob rhuke
bst`lftb, teb Ro`tbg ^tftbs efs twboty tehusfog `otbdd`kbonb fofdysts
fssbss`ok bvbryte`ok jrhl l`outb puzzdbs th lfchr bvbots sune fs
teb d`mbd`ehhg hj fo @srfbd` sobfm fttfnm ho @rfo`fo oundbfr jfn`d`-
t`bs hr teb gbpfrturb hj Krbbnb jrhl teb burhzhob.0 Ehw khhg `s fdd
te`s jhrbnfst`ok8 Weft `s oht bfs`dy foswbrbg ibnfusb teb `otbdd`kbonb
nhlluo`ty, d`mb sh lfoy lfchr prhgunbrs hj jhrbnfst`ok, efs obvbribbo mbbo ho spbog`ok lhoby th stkurb teft hut. Webrb frb vfr`hus
rbfshos jhr teft rbduntfonb, shlb lhrb rbspbntfidb tefo htebrs, iut
wb‐dd kbt `oth teft dftbr. [eft lfttbrs `s teft te`s jhrbnfst`ok `s nr`t`-
nfd th oft`hofd sbnur`ty fog ybt d`ttdb nfo ib sf`g w`te foy nhostgbonb
fihut ehw khhg `t `s, hr bvbo webtebr `t‐s fs khhg fs f ludt`i`dd`ho-
ghddfr hpbrft`ho w`te twboty tehusfog pbhpdb sehudg ib. Wh nefokb
teft, @FYQF nrbftbg f jhrbnfst`ok thuroflbot `o we`ne stvb sn`bo-
t`stn tbfls dbg iy thp rbsbfrnebrs `o teb stbdg whudg nhlpbtb th kbo-
brftb fnnurftb jhrbnfsts ho teb shrts hj thuke qubst`hos `otbdd`kbonb
fofdysts gbfd w`te bvbry gfy. Web Khhg Cugklbot Qrhcbnt wfs hob
hj tehsb stvb tbfls. Bfne tbfl whudg bjjbnt`vbdy ib `ts hwo rbsbfrne
prhcbnt, jrbb th `lprhv`sb weftbvbr lbtehgs `t tehuket whudg whrm,
iut rbqu`rbg th suil`t jhrbnfsts ft 0 f.l. bfstbro stfogfrg t`lbbvbry gfy jrhl ^bptblibr 9166 th Cuob 9162. Iy rbqu`r`ok tbfls
th jhrbnfst teb sflb qubst`hos ft teb sflb t`lb, teb thuroflbot nrb-
ftbg f dbvbd pdfy`ok stbdg‗fog f r`ne trhvb hj gftf fihut weft whrms,
ehw wbdd, fog webo. Hvbr jhur ybfrs, @FYQF phsbg obfrdy stvb euo-
grbg qubst`hos fihut whrdg fjjf`rs. W`lb jrflbs wbrb sehrtbr tefo `o
ly bfrd`br rbsbfrne, w`te teb vfst lfchr`ty hj jhrbnfsts bxtbog`ok hut
lhrb tefo hob lhote fog dbss tefo hob ybfr. @o fdd, wb kftebrbg hvbr
`dd` ` g` `g d c g i e j
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
23/29
65 supbrjhrbnfst`ok
wb ibft teb nhotrhd krhup iy ;5%. KCQ fdsh ibft `ts uo`vbrs`ty-
fjstd`ftbg nhlpbt`thrs, `ondug`ok teb Ro`vbrs`ty hj L`ne`kfo fog
L@W, iy ebjty lfrk`os, jrhl >1% th ;1%, fog bvbo hutpbrjhrlbgprhjbss`hofd `otbdd`kbonb fofdysts w`te fnnbss th ndfss`stbg gftf. Fjtbr
twh ybfrs, KCQ wfs gh`ok sh lune ibttbr tefo `ts fnfgbl`n nhlpbt`-
thrs teft @FYQF grhppbg teb htebr tbfls.61
@‐dd gbdvb `oth gbtf`ds dftbr, iut dbt‐s ohtb twh mby nhondus`hos teft
blbrkb jrhl te`s rbsbfrne. Hob, jhrbs`ket `s rbfd. ^hlb pbhpdb‗
pbhpdb d`mb I`dd Jdfnm‗efvb ̀ t ̀ o spfgbs. Weby frbo‐t kurus hr hrfndbs
w`te teb phwbr th pbbr gbnfgbs `oth teb juturb, iut teby gh efvb frbfd, lbfsurfidb sm`dd ft cugk`ok ehw e`ke-stfmbs bvbots frb d`mbdy th
uojhdg terbb lhotes, s`x lhotes, f ybfr, hr f ybfr fog f efdj `o fg-
vfonb. Web htebr nhondus`ho `s weft lfmbs tebsb supbrjhrbnfstbrs sh
khhg. @t‐s oht rbfddy weh teby frb. @t `s weft teby gh. Jhrbs`ket `so‐t f
lystbr`hus k`jt ibsthwbg ft i`rte. @t `s teb prhgunt hj pfrt`nudfr wfys
hj te`om`ok, hj kftebr`ok `ojhrlft`ho, hj upgft`ok ibd`bjs. Websb
efi`ts hj tehuket nfo ib dbfrobg fog nudt`vftbg iy foy `otbdd`kbot,
tehuketjud, gbtbrl`obg pbrsho. @t lfy oht bvbo ib fdd teft efrg th
kbt stfrtbg. Hob rbsudt teft pfrt`nudfrdy surpr`sbg lb wfs teb bjjbnt
hj f tuthr`fd nhvbr`ok shlb ifs`n nhonbpts teft wb‐dd bxpdhrb `o te`s
ihhm fog frb sullfr`zbg `o teb Wbo Nhllfoglbots fppbog`x. @t
thhm hody fihut s`xty l`outbs th rbfg fog ybt `t `lprhvbg fnnurfny
iy rhukedy 61% terhuke teb bot`rb thuroflbot ybfr. Pbs, 61% lfyshuog lhgbst, iut `t wfs fne`bvbg ft sh d`ttdb nhst. Fog obvbr jhrkbt
teft bvbo lhgbst `lprhvblbots `o jhrbs`ket lf`otf`obg hvbr t`lb
fgg up. @ sphmb fihut teft w`te Ffrho Irhwo, fo futehr, f [fdd
^trbbt vbtbrfo, fog teb ne`bj r`sm lfofkbr ft FVY Nfp`tfd Lfofkb-
lbot, f ebgkb juog w`te hvbr $611 i`dd`ho `o fssbts. —@t‐s sh efrg th
sbb ibnfusb `t‐s oht grflft`n,“ eb sf`g, iut `j `t `s sustf`obg —`t‐s teb
g`jjbrbonb ibtwbbo f nhos`stbot w`oobr weh‐s lfm`ok f d`v`ok, hr teb
e ‐ ` i m dd e ` “66 F dg d m d
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
24/29
Fo Hpt`l`st`n ^mbpt`n 60
wb`kets fog flftburs, seb sf`g, `s teft teb ebfvywb`kets mohw teb
g`jjbrbonb ibtwbbo f :1 ⁇ 71 ibt fog f 71 ⁇ :1 ibt.
Fog ybt, `j `t‐s phss`idb th `lprhvb jhrbs`ket s`lpdy iy lbfsur-`ok, fog `j teb rbwfrgs hj `lprhvbg jhrbs`ket frb suistfot`fd, wey
`so‐t lbfsur`ok stfogfrg prfnt`nb8 F i`k pfrt hj teb foswbr th teft
qubst`ho d`bs `o teb psynehdhky teft nhov`onbs us wb mohw te`oks
wb rbfddy gho‐t‗te`oks d`mb webtebr Whl Jr`bglfo `s fo fnnurftb
jhrbnfstbr hr oht. @‐dd bxpdhrb te`s psynehdhky `o nefptbr 9. Jhr nbo-
tur`bs, `t ehiidbg prhkrbss `o lbg`n`ob. [ebo peys`n`fos stofddy fn-
nbptbg teft teb`r bxpbr`bonb fog pbrnbpt`hos wbrb oht rbd`fidb lbfoshj gbtbrl`o`ok webtebr f trbftlbot whrms, teby turobg th sn`bot`stn
tbst`ok‗fog lbg`n`ob stofddy stfrtbg th lfmb rfp`g fgvfonbs. Web
sflb rbvhdut`ho obbgs th efppbo `o jhrbnfst`ok.
@t who‐t ib bfsy. Nefptbr > bxfl`obs weft `t tfmbs th tbst jhrb-
nfst`ok fs r`khrhusdy fs lhgbro lbg`n`ob tbsts trbftlbots. @t‐s f i`k-
kbr nefddbokb tefo `t lfy fppbfr. @o teb dftb 6051s @ whrmbg hut f
lbtehghdhky fog nhoguntbg weft wfs, ft teb t`lb, teb i`kkbst tbst hj
bxpbrt phd`t`nfd jhrbnfst`ok fnnurfny bvbr. Hob rbsudt, gbd`vbrbg lfoy
ybfrs dftbr, wfs teb puone d`ob teft ohw lfmbs lb squ`rl. Iut fo-
htebr g`snhvbry hj teft rbsbfrne g`go‐t rbnb`vb obfrdy fs lune fttbo-
t`ho bvbo tehuke `t wfs jfr lhrb `lphrtfot3 hob krhup hj bxpbrts
efg lhgbst iut rbfd jhrbs`ket. [eft lfgb teb g`jjbrbonb ibtwbbo teb
bxpbrts w`te jhrbs`ket fog tehsb weh wbrb sh ehpbdbss teby grfkkbgteb fvbrfkb ghwo th teb dbvbd hj f gfrt-terhw`ok ne`lp8 @t wfso‐t
shlb lyst`nfd k`jt hr fnnbss th `ojhrlft`ho htebrs g`go‐t efvb. Ohr
wfs `t foy pfrt`nudfr sbt hj ibd`bjs. @ogbbg, w`te`o f qu`tb w`gb rfokb
hj v`bws, weft teby tehuket g`go‐t lfttbr. @t wfs ehw teby tehuket.
@osp`rbg `o pfrt iy teft `os`ket, @FYQF nrbftbg `ts uoprbnb-
gbotbg jhrbnfst`ok thuroflbot. Nefptbr 7 `s teb sthry hj ehw teft
efppbobg‗fog teb g`snhvbry hj supbrjhrbnfstbrs. [ey frb teby sh
g We ` e e e 2 e e [e
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
25/29
91 supbrjhrbnfst`ok
l`ket suspbnt `t‐s `otbdd`kbonb teft lfmbs fdd teb g`jjbrbonb. @t‐s oht.
Weby‐rb fdsh rblfrmfidy oulbrftb. D`mb I`dd Jdfnm, lfoy efvb fg-
vfonbg gbkrbbs `o lfteblft`ns fog sn`bonb. ^h `s teb sbnrbt frnfoblfte8 Oh. Bvbo supbrjhrbnfstbrs weh frb nfrg-nfrry`ok lfteblft`-
n`fos rfrbdy usb lune lfte. Weby fdsh tbog th ib obwscuom`bs weh
stfy ho thp hj teb dftbst gbvbdhplbots fog rbkudfrdy upgftb teb`r jhrb-
nfsts, sh yhu l`ket ib tblptbg th fttr`iutb teb`r sunnbss th spbog`ok
bogdbss ehurs ho teb chi. Pbt teft thh whudg ib f l`stfmb.
^upbrjhrbnfst`ok ghbs rbqu`rb l`o`lul dbvbds hj `otbdd`kbonb,
oulbrfny, fog mohwdbgkb hj teb whrdg, iut foyhob weh rbfgs sbr`-hus ihhms fihut psynehdhk`nfd rbsbfrne prhifidy efs tehsb prbrbq-
u`s`tbs. ^h weft `s `t teft bdbvftbs jhrbnfst`ok th supbrjhrbnfst`ok8 Fs
w`te teb bxpbrts weh efg rbfd jhrbs`ket `o ly bfrd`br rbsbfrne, weft
lfttbrs lhst `s ehw teb jhrbnfstbr te`oms. @‐dd gbsnr`ib te`s `o gbtf`d,
iut irhfgdy spbfm`ok, supbrjhrbnfst`ok gblfogs te`om`ok teft `s
hpbo-l`ogbg, nfrbjud, nur`hus, fog‗fihvb fdd‗sbdj-nr`t`nfd. @t fdsh
gblfogs jhnus. Web m`og hj te`om`ok teft prhgunbs supbr`hr cugk-
lbot ghbs oht nhlb bjjhrtdbssdy. Hody teb gbtbrl`obg nfo gbd`vbr `t
rbfshofidy nhos`stbotdy, we`ne `s wey hur fofdysbs efvb nhos`stbotdy
jhuog nhll`tlbot th sbdj-̀ lprhvblbot th ib teb strhokbst prbg`nthr
hj pbrjhrlfonb.
@o teb stofd nefptbrs, @‐dd rbshdvb fo fppfrbot nhotrfg`nt`ho ib-
twbbo teb gblfogs hj khhg cugklbot fog bjjbnt`vb dbfgbrse`p,rbsphog th weft @ te`om frb teb twh strhokbst nefddbokbs th ly rb-
sbfrne, fog nhondugb‗fpprhpr`ftbdy jhr f ihhm fihut jhrbnfst`ok‗
w`te f nhos`gbrft`ho hj weft nhlbs obxt.
F JHYBNF^W FIHRW JHYBNF^W@OK
I i e` m e` ` dd e d d g g Fj dd
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
26/29
Fo Hpt`l`st`n ^mbpt`n 96
fdkhr`tels, fog I`k Gftf. Ft `ts nhrb, teb jhrbnfst`ok @ stugy `ovhdvbs
suicbnt`vb cugklbot3 `t `s pbhpdb te`om`ok fog gbn`g`ok, ohte`ok
lhrb. @so‐t teb t`lb jhr sune sdhppy kubsswhrm grfw`ok th f ndhsb8@o 6027, f ir`dd`fot psynehdhk`st, Qfud Lbbed wrhtb f slfdd ihhm
teft nfusbg f i`k st`r.69 @t rbv`bwbg twboty stug`bs sehw`ok teft wbdd-
`ojhrlbg bxpbrts prbg`nt`ok hutnhlbs‗ webtebr f stugbot whudg
sunnbbg `o nhddbkb hr f pfrhdbb whudg ib sbot ifnm th pr`sho‗ wbrb
oht fs fnnurftb fs s`lpdb fdkhr`tels teft fggbg up hicbnt`vb `og`nf-
thrs d`mb fi`d`ty tbst snhrbs fog rbnhrgs hj pfst nhogunt. Lbbed‐s ndf`l
upsbt lfoy bxpbrts, iut suisbqubot rbsbfrne‗ohw lhrb tefo twheuogrbg stug`bs‗efs sehwo teft `o lhst nfsbs stft`st`nfd fdkhr`tels
ibft suicbnt`vb cugklbot, fog `o teb efogjud hj stug`bs webrb teby
gho‐t, teby usufddy t`b. K`vbo teft fdkhr`tels frb qu`nm fog nebfp,
uod`mb suicbnt`vb cugklbot, f t`b supphrts us`ok teb fdkhr`tel. Web
ph`ot `s ohw `og`sputfidb3 webo yhu efvb f wbdd-vfd`gftbg stft`st`nfd
fdkhr`tel, usb `t .
We`s `os`ket wfs obvbr f terbft th teb rb`ko hj suicbnt`vb cugk-
lbot ibnfusb wb sh rfrbdy efvb wbdd-vfd`gftbg fdkhr`tels jhr teb
prhidbl ft efog. @t wfs cust `lprfnt`nfd jhr lfte th g`spdfnb pdf`o
hdg te`om`ok‗`o 6027 fog bvbo thgfy.
Iut spbntfnudfr fgvfonbs `o `ojhrlft`ho tbneohdhky sukkbst wb
frb fpprhfne`ok f e`sthr`nfd g`snhot`ou`ty `o eulfo`ty‐s rbdft`ho-
se`p w`te lfne`obs. @o 600; @IL‐s Gbbp Idub ibft nebss neflp`hoKfrry Mfspfrhv. Ohw, nhllbrn`fddy fvf`dfidb nebss prhkrfls nfo
ibft foy eulfo. @o 9166 @IL‐s [ftsho ibft Cbhpfrgy! neflp`hos
Mbo Cboo`oks fog Irfg Yuttbr. Weft wfs f vfstdy thukebr nhlput`ok
nefddbokb, iut [ftsho‐s bok`obbrs g`g `t. Whgfy, `t‐s oh dhokbr `lphs-
s`idb th `lfk`ob f jhrbnfst`ok nhlpbt`t`ho `o we`ne f supbrnhlputbr
trhuonbs supbrjhrbnfstbrs fog supbrpuog`ts fd`mb. Fjtbr teft efp-
pbos, tebrb w`dd st`dd ib eulfo jhrbnfstbrs, iut d`mb eulfo Cbhpfrgy!
`dd d e e j `
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
27/29
99 supbrjhrbnfst`ok
teft [ftsho nhudg bfs`dy stbdg f qubst`ho fihut teb prbsbot hr pfst
d`mb —[e`ne twh Yuss`fo dbfgbrs trfgbg chis `o teb dfst tbo ybfrs8“
Iut @ wfs nur`hus fihut e`s v`bws ho ehw dhok `t w`dd tfmb jhr [ft-sho hr hob hj `ts g`k`tfd gbsnbogfots th stbdg qubst`hos d`mb —[`dd twh
thp Yuss`fo dbfgbrs trfgb chis `o teb obxt tbo ybfrs8“
@o 60:2 teb phdylfte Ebribrt ^`lho tehuket wb wbrb hody
twboty ybfrs fwfy jrhl f whrdg `o we`ne lfne`obs nhudg gh —foy
whrm f lfo nfo gh,“ we`ne `s teb shrt hj of`vbdy hpt`l`st`n te`ok
pbhpdb sf`g ifnm tebo, fog hob rbfsho wey Jbrrunn`‗ weh efs
whrmbg `o frt`stn`fd `otbdd`kbonb jhr te`rty ybfrs‗`s lhrb nfut`husthgfy.6> Nhlput`ok ̀ s lfm`ok bohrlhus str`gbs, Jbrrunn` ohtbg. Web
fi`d`ty th spht pfttbros `s krhw`ok spbntfnudfrdy. Fog lfne`ob dbfro-
`ok, `o nhli`oft`ho w`te iurkbho`ok eulfo-lfne`ob `otbrfnt`hos
teft jbbg teb dbfro`ok prhnbss, prhl`sbs jfr lhrb juogflbotfd fg-
vfonbs th nhlb. —@t‐s kh`ok th ib hob hj tebsb bxphobot`fd nurvbs teft
wb‐rb m`og hj ft teb ihtthl hj ohw,“ Jbrrunn` sf`g.
Iut tebrb `s f vfst g`jjbrbonb ibtwbbo —[e`ne twh Yuss`fo dbfg-
brs trfgbg chis8“ fog —[`dd twh Yuss`fo dbfgbrs trfgb chis fkf`o8“
Web jhrlbr `s f e`sthr`nfd jfnt. Web nhlputbr nfo dhhm `t up. Web
dfttbr rbqu`rbs teb nhlputbr th lfmb fo `ojhrlbg kubss fihut teb
`otbot`hos hj Zdfg`l`r Qut`o, teb nefrfntbr hj Gl`tr` Lbgvbgbv,
fog teb nfusfd gyofl`ns hj Yuss`fo phd`t`ns, fog tebo `otbkrftb teft
`ojhrlft`ho `oth f cugklbot nfdd. Qbhpdb gh teft shrt hj te`ok fddteb t`lb, iut teft ghbso‐t lfmb `t bfsy. @t lbfos teb eulfo irf`o `s
whogrhus‗ibnfusb teb tfsm `s stfkkbr`okdy efrg. Bvbo w`te nhl-
putbrs lfm`ok kfddhp`ok fgvfonbs, teb shrt hj jhrbnfst`ok teft su-
pbrjhrbnfstbrs gh `s f dhok wfy hjj. Fog Jbrrunn` `so‐t surb wb w`dd
bvbr sbb f eulfo uogbr kdfss ft teb ^l`tesho`fo w`te f s`ko sfy`ok
—suicbnt`vb cugklbot.“
Lfne`obs lfy kbt ibttbr ft —l`l`nm`ok eulfo lbfo`ok,“ fog
e i i g` ` e i e ` i — e ‐ g`jj
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
28/29
Fo Hpt`l`st`n ^mbpt`n 9>
`ok,“ Jbrrunn` sf`g. Weft‐s f spfnb eulfo cugklbot w`dd fdwfys hn-
nupy.
@o jhrbnfst`ok, fs `o htebr stbdgs, wb w`dd nhot`oub th sbb eulfo cugklbot ib`ok g`spdfnbg‗th teb nhostbroft`ho hj we`tb-nhddfr
whrmbrs‗iut wb w`dd fdsh sbb lhrb fog lhrb syotebsbs, d`mb —jrbb-
stydb nebss,“ `o we`ne eulfos w`te nhlputbrs nhlpbtb fs tbfls,
teb eulfo grfw`ok ho teb nhlputbr‐s `og`sputfidb strboktes iut
fdsh hnnfs`hofddy hvbrr`g`ok teb nhlputbr. Web rbsudt `s f nhli`of-
t`ho teft nfo (shlbt`lbs) ibft ihte eulfos fog lfne`obs. Wh rb-
jrflb teb lfo-vbrsus-lfne`ob g`nehthly, nhli`oft`hos hj KfrryMfspfrhv fog Gbbp Idub lfy prhvb lhrb rhiust tefo purb-eulfo
hr purb-lfne`ob fpprhfnebs.
[eft Jbrrunn` ghbs sbb ibnhl`ok hishdbtb `s teb kuru lhgbd
teft lfmbs sh lfoy phd`ny gbiftbs sh pubr`db3 —@‐dd nhuotbr yhur Qfud
Mruklfo phdbl`n w`te ly O`fdd Jbrkusho nhuotbrphdbl`n, fog
rbiut yhur Whl Jr`bglfo hp-bg w`te ly Irbt ^tbpebos idhk.“ Jbr-
runn` sbbs d`ket ft teb bog hj te`s dhok gfrm tuoobd3 —@ te`om `t‐s kh`ok
th kbt strfokbr fog strfokbr“ jhr pbhpdb th d`stbo th teb fgv`nb hj
bxpbrts wehsb v`bws frb `ojhrlbg hody iy teb`r suicbnt`vb cugklbot.
Eulfo tehuket `s ibsbt iy psynehdhk`nfd p`tjfdds, f jfnt teft efs hody
ibnhlb w`gbdy rbnhko`zbg `o teb dfst gbnfgb hr twh. —^h weft @ wfot
`s teft eulfo bxpbrt pf`rbg w`te f nhlputbr th hvbrnhlb teb eulfo
nhko`t`vb d`l`tft`hos fog i`fsbs.“67
@j Jbrrunn` `s r`ket‗@ suspbnt eb `s‗ wb w`dd obbg th idbog
nhlputbr-ifsbg jhrbnfst`ok fog suicbnt`vb cugklbot `o teb juturb.
^h `t‐s t`lb wb kht sbr`hus fihut ihte.
-
8/20/2019 Superforecasting by Philip E. Tetlock and Dan Gardner
29/29
http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136693/siteID/8001/retailerid/6/trackingcode/penguinrandomhttp://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136709/siteID/8001/retailerid/22/trackingcode/PRH698C2DEE54http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136693/siteID/8001/retailerid/2/trackingcode/PRH698C2DEE54http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136709/siteID/8001/retailerid/3/trackingcode/PRH698C2DEE54http://links.penguinrandomhouse.com/type/affiliate/isbn/9780804136693/siteID/8001/retailerid/7/trackingcode/randohouseinc10663-20http://www.crownpublishing.com/