"let's leave that up to the computer" chris snijders
DESCRIPTION
"Let's leave that up to the computer" Chris Snijders. Prof. dr. C.C.P. Snijders Dept of Technology Management Technische Universiteit Eindhoven Postbus 513 5600 MB Eindhoven [email protected]. Overview. Case: Cook county hospital The science behind it - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/1.jpg)
"Let's leave that up to the computer"
Chris Snijders
Prof. dr. C.C.P. SnijdersDept of Technology Management
Technische Universiteit EindhovenPostbus 513
5600 MB [email protected]
![Page 2: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/2.jpg)
Chris Snijders
Overview
• Case: Cook county hospital• The science behind it• The computer as a decision
maker
![Page 3: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/3.jpg)
Chris Snijders
Case: Cook county hospital
Emergency Department
-250.000 patients per year
-many persons without insurance
-not enough rooms, overworked staff
-1996: Brendan Reilly director
(see Gladwell, 2005)
![Page 4: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/4.jpg)
Chris Snijders
Problem 1: acute chest pain
Diagnose through:
blood pressure, stethoscope: fluid in the lungs, how long have you been experiencing pain, how does it feel precisely, where does it hurt, does it always hurt or only when you exercise, have you had heart problems before, how about your cholesterol, do you have diabetes, let's look at your ECG, are there any heart problems in the family, do you use drugs, how old are you, are you in shape, do you smoke, do you drink, check appearance: stressed, overweight, ....
High risk : 8
Medium risk : 12
Go home30 p/day
![Page 5: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/5.jpg)
Chris Snijders
Reilly finds Goldman: obv 10,000 cases
Only 4 things matter
ECGBlood pressureFluid in your lungs"unstable angina"
![Page 6: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/6.jpg)
Chris Snijders
Great! So let's do that! Or not...
Implementation: … physicians protest …
A test: 20 cases were given to several physicians
Hardly any agreement between physicians!
![Page 7: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/7.jpg)
Chris Snijders
Reilly tests Goldman’s idea
vs
82% 95%
physician Goldman’s scheme
![Page 8: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/8.jpg)
Chris Snijders
A literature check …
Clinical versus statisticalprediction
For instance (zie Grove et al., 2000)
– Survival probabilities in medical procedures– Probability of recidivism– Probability of success of starting firms– Choice of job candidates– Diagnosing schizofrenia– Predicting school success– …
![Page 9: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/9.jpg)
Chris Snijders
The results …
Over 160 studies
When given the same info,the number of cases in whichthe expert wins = ??
![Page 10: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/10.jpg)
Chris Snijders
… and that is understandable
We emphasize the improbable’ (Stickler)
Confirmation bias (Edwards, Wason)
Hindsight bias (Fischhoff)
Cognitive dissonance (Festinger)
“Dealing with probabilities / Base rate neglect”(Bar-Hillel)
Mental sets (Redelmayer, Tversky)
Our memory fools us (Wagenaar)
![Page 11: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/11.jpg)
Chris Snijders
And there are more of these
"Mental Floating Frankfurters"
![Page 12: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/12.jpg)
Chris Snijders
Restriction 2: Memory
“Where were you, when …”
Shuttle Columbia Crew Lost Feb. 1, 2003
![Page 13: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/13.jpg)
Chris Snijders
Restriction 3: the “availability heuristic”
What is more likely, a plain crash or a car crash?
![Page 14: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/14.jpg)
Chris Snijders
Restriction 4: dealing with probabilities
Suppose: a manager has a good intuition in business:
– when a problem will arise: he gets a gut-feeling that something is wrong with probability 90%
– when no problem will arise: he gets a gut-feeling that something is wrong with probability 10%
On average, there is a problem in 5% of the transactions.
The manager starts a transaction, and he gets a gut-feeling that something might be wrong.
What is the probability that something is indeed wrong?
![Page 15: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/15.jpg)
Chris Snijders
Restriction 4: dealing with probabilities
A murder has been committed. The only evidence available is DNA, found at the murder scene. DNA-research shows a match with your DNA.
The probability that two persons are diagnosed as having the same DNA is about 1 in 100.000.
How likely is it that you are the murderer?
![Page 16: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/16.jpg)
Chris Snijders
Restriction 5: overconfidence
Trivial Pursuit: estimate how many questions you will know
Estimates are generally too high ... and this gets worse with expertise!
![Page 17: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/17.jpg)
Chris Snijders
Restriction 6:Finding non-existent patterns
![Page 18: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/18.jpg)
Chris Snijders
Restriction 7: the noble art of finding a broken leg
![Page 19: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/19.jpg)
Chris Snijders
Decision making =Store, retrieve, combine
![Page 20: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/20.jpg)
Chris Snijders
Our own experiments:purchasing managers
vs
![Page 21: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/21.jpg)
Chris Snijders
A database of purchasing transactions
• Collected at Utrecht and Eindhoven University
• Since 1995• >4000 purchasing transactions• >1500 firms• 300 characteristics per transaction
![Page 22: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/22.jpg)
Chris Snijders
The role of the purchasing manager
Purchasing transaction is to
be started
Purchasing transaction completed
BTW we chose economics, because it is relatively rare in this kind of research
![Page 23: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/23.jpg)
Chris Snijders
![Page 24: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/24.jpg)
Chris Snijders
Main idea• Purchasing professionals should be able to judge, for a given purchasing transaction, how likely it is that problems will arise.
• The test: we present purchasing professionals with a set of purchasing transactions. The professionals then judge how likely it is that problems occur.
• But … these transactions were sampled from our database of purchasing transactions. For all these transactions, we know which and how many problems occurred.
•We can therefore compare the predictions of the professionals with what actually happened.
![Page 25: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/25.jpg)
Chris Snijders
Our test
Case from database
grade
6.2
ManagerStudent
ComputerLay
grade
5.4
grade
6.2
grade
6.9
![Page 26: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/26.jpg)
Chris Snijders
Model-based prediction
• based on cross-validated “Multivariate Adaptive Regression Splines” (of order 1)
![Page 27: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/27.jpg)
Chris Snijders
A set of experiments1. Original test (91 participants)
2. I&L test 1. A call in the Dutch “Tijdschrift voor inkoop en logistiek” (72 participants; webversion)
3. I&L test 2. After publication of the results in I&L, we left the website version open (72 additional participants)
4. MC. MasterClass in Strategic Purchasing and Supply Management, Corsendonk, Belgium (13 participants)
5. Rep03. Replication of original experiment, including small extension (118 participants)
6. PC03. 249 PanelClix participants (opt-in panel in NL)
7. IK05. 148 participants (48 professionals) as part of a class assignment in Information Sciences
![Page 28: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/28.jpg)
Chris Snijders
![Page 29: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/29.jpg)
Chris Snijders
Extensions (1)• “This is not what I do in my regular job”
Ask before during and after test whether professionals feel they are doing a good job.
• “I would have done better if I would have had more information”
Use either 7 or 14 pieces of information.
• “I usually perform better under pressure”
Do it again - now there is pressure.Do it again - with a time constraint.
![Page 30: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/30.jpg)
Chris Snijders
Extensions (2)• Accountability
Have purchasing managers:- explain to the interviewer why they decide as they do- decide in groups of two
• How about differences between persons?
- younger professionals perform a bit better- those who “decide by instinct” perform a bit worse
![Page 31: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/31.jpg)
Chris Snijders
The risk of experience
… trust increases with experience,
… but the test score does not ...
experience experience
trust
score
![Page 32: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/32.jpg)
Chris Snijders
Seasoned vs young proExperienced• Relatively crude• Associating and fast• Comparison with experienced ‘prototypical cases’• Story telling But: • Overgeneralization• May seem efficient, but is not effective
Young pro• More ‘fine-tuning’• Deliberate and slow• Less experienced, and thereby less bothered by ‘prototypical cases’• More abstract• Less mistakes• Decisions may seem less efficient, but are relatively effective
"Recognition Primed
Decision making"
(Klein)
![Page 33: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/33.jpg)
Chris Snijders
Improving experts• Expert system 1 Averaging per case across
professionals. For each case, use the average score of the experts
• Expert system 2 Averaging within professionals, across cases: for each expert, calculate the implicit model they are using. Use that model to predict again (based on volume, length of relation, ability to judge price/quality)
Increase from 0.19 0.23, still < 0.32
No improvement.
![Page 34: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/34.jpg)
Chris Snijders
Improving experts (2)• Averaging professionals with the computer model (50%
pro, 50% model).
• Combining expert system 2 with the computer model (50% expert system 2, 50% model).
Increase from 0.19 0.27, still < 0.32.
Increase from 0.19 0.36 > 0.32 (p=0.024, 1-sided)
![Page 35: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/35.jpg)
Chris Snijders
Kinds of e-purchasing
• Decrease tedious paperwork• Viewing your partners’ supplies• Online market places [auctioning]• Structuring decisions
… but there is an opportunity for …
• ... letting the computer decide! [real business intelligence]
![Page 36: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/36.jpg)
Chris Snijders
For which kinds of decisions?
Purchasing to be done Purchasing done
How do I choose my purchasing team (how many persons, which levels, etc)?
How much time do I invest into writing a detailed list of demands?
How shall I choose my supplier?
What kind of contract shall I use? (tailor-made, standard, ...)
How much time and effort am I going to put into this transaction to make sure that it runs smooth?
Which other ways of trying to make sure that this transaction runs smooth shall I use?
….
1) Quantitative
2 ) Have data, or willingness to collect
3) Returning decisions
From those, choose the ones that are most important to you.
![Page 37: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/37.jpg)
Chris Snijders
Beware
So purchasing managers / experts are no longer necessary?
Other tasks
e-decisionsOther decisions
![Page 38: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/38.jpg)
Chris Snijders
Experts decide in different ways
– are more selective in their search for info– store information faster– use less information, and often this information is
combined in a non-linear way– have a more active pattern of contingent search:
they consider subsets of variables, different subsets in each case
– compare given information with the information in their knowledge base …
– … but choices are often based on over-generalization of specific cases
– use more “broken-leg cues”
![Page 39: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/39.jpg)
Chris Snijders
… but not better
“process performance paradox”:
in a large number of tasks, the experts decide in different ways, but not better than those with a minimal amount of training.
(cf. Camerer and Johnson, 1991)
![Page 40: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/40.jpg)
Chris Snijders
Do not take this to extremes...
![Page 41: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/41.jpg)
Chris Snijders
Models in, intuition out?
No. RPD model works well, for specific tasks and you need to properly train your intuition
– deliberate practice– compile extensive experience bank– obtain accurate and timely feedback– enrich experiences by reviewing prior
ones
![Page 42: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/42.jpg)
Chris Snijders
Gigerenzer: Fast and frugal• Humans use “fast and frugal” strategies (Simon: bounded
rationality, satisficing) or “Probabilistic Mental Models”
by employing simple decision-strategies individuals with limited knowledge are able to arrive at equally, or even more, correct predictions than individuals who have extensive knowledge
• Gigerenzer’s experiment
Which city has the most inhabitants?Arnhem Eindhoven
Try to see how such fast-and-frugal strategies can have a role in the experiments
![Page 43: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/43.jpg)
Chris Snijders
Example: #inhabitants
Predict which of 2 US cities has the most homeless people, based on
– vacancy rate– temperature– unemployment rate– poverty rate– public housing
[all binary variables, cut at the median]
![Page 44: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/44.jpg)
Chris Snijders
Strategies [when training set <> test set]
• Minimalist -• Take-the-best 63
• Dawes’ Rule 61• Multiple regression 60• Bayesian network 65
![Page 45: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/45.jpg)
Chris Snijders
Fast-and-frugal design:left - same - right
Additional factor: time pressure / 48 Purchasing pros, 50 students, 50 super-laypeople
![Page 46: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/46.jpg)
Chris Snijders
Results (1)
All cases Only extremes
![Page 47: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/47.jpg)
Chris Snijders
Results (2)
![Page 48: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/48.jpg)
Chris Snijders
Results (3)(only using the larger differences)
Perfect 100
Model 60Take-the-best 56Linear regression 43Unit regression (std) 43
Random 33
![Page 49: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/49.jpg)
Chris Snijders
Some conclusions
• In our managerial context, model-based decision making clearly outperforms professional decision making
• Good model scores can be achieved by fast-and-frugal strategies...
• ... but the humans in our experiment clearly did not use this kind of logic
![Page 50: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/50.jpg)
Chris Snijders
Implications
– one might be able to teach people to improve their judgment(s)
– in actual managerial practice, there might often be no need to develop fancy models to get a close to optimal improvement
– decision support by creating expert systems that mimic experts makes no sense in this context
– To get this implemented, you need two things: a Goldman and a Reilley
![Page 51: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/51.jpg)
Chris Snijders
Bottom line
Model based prediction can and should be used much more often than it is now.
![Page 52: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/52.jpg)
Chris Snijders
You have to get used to it ...
![Page 53: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/53.jpg)
Chris Snijders
Example: Bordeaux wines
![Page 54: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/54.jpg)
Chris Snijders
Try it yourself (1)
• http://islandia.law.yale.edu/ayres/predictionTools.htm
![Page 55: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/55.jpg)
Chris Snijders
Try it yourself (2)• Life expectancy• Likelihood of marital dissolution• Expected length of a child• Prices of airline-tickets• Risk for and survival probabilities for different
diseases (incl. cancer, diabetes, osteoperosis, etc)
• Probability of bankruptcy• ...
![Page 56: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/56.jpg)
Chris Snijders
Example: life expectancy
http://www.nmfn.com/tn/learnctr--lifeevents--longevity_game
![Page 57: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/57.jpg)
Chris Snijders
Example: breast cancer
![Page 58: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/58.jpg)
Chris Snijders
Example: marriage dissolution
![Page 59: "Let's leave that up to the computer" Chris Snijders](https://reader036.vdocuments.net/reader036/viewer/2022062305/56814871550346895db57aca/html5/thumbnails/59.jpg)
Chris Snijders
Example: bankruptcy