advanced decision architectures collaborative technology alliance a computational model of...
Post on 02-Jan-2016
217 Views
Preview:
TRANSCRIPT
Advanced Decision Architectures Collaborative Technology Alliance
A Computational Model of Naturalistic Decision Making and the Science of
Simulation
Walter WarwickAmy Santamaria
& many, many others
Advanced Decision Architectures Collaborative Technology Alliance
Overview
• The M&S big picture
• The work– Birth
• You can’t model this; that’s not a model
– Life• Where’s the data
– Quiet reflection• The science of simulation
Advanced Decision Architectures Collaborative Technology Alliance
The Big Picture
• An effort to improve human behavior representations for M&S but incorporating a better model of decision making– Better than: a “tactical” or probabilistic
decision– Allows new kinds of behavior to play inside of
task network models
• Not an exercise in theory validation– Though we’d like our work to illuminate theory
Advanced Decision Architectures Collaborative Technology Alliance
The Birth of the RPD Widget
• From a descriptive model to a theoretical model:– A clash of traditions– A lot of thrashing– The emergence of a cottage industry and an M&S land grab
• A new decision type (“RPD”) in the Micro Saint Sharp family of task network modeling tools
• Widget intended to capture:– Experience-based decision making via a multiple trace model of
memory and simple reinforcement routine– Recognitional decision making via similarity-based recall
mechanism that draws on *every* past experience– Expectancy generation and feedback—several different versions
implemented, rarely used and no clear indication that we can do anything interesting with it
Advanced Decision Architectures Collaborative Technology Alliance
Using the Widget
• To specify an RPD decision type, the modeler supplies:
• Cues that prompt recognition (map MSS variables into “subjective,” discrete cues)
• Alternative courses of action (usually given by the structure of the task network)
• Reinforcement (seat of the pants)• Set run-time properties and parameters (seat of the pants)
• This defines the structure of each “trace”—a individual decision making experience comprising the cue values at decision time, the action that was taken and the outcome (good or bad)
Advanced Decision Architectures Collaborative Technology Alliance
What You Get
• Four applications (validation studies); two flavors:– Categorization: Brunswik Faces and Weather
Prediction– Dynamic behavior: Prisoners’ Dilemma and
Dynamic Stocks and Flows
Advanced Decision Architectures Collaborative Technology Alliance
Brunswik Faces
Advanced Decision Architectures Collaborative Technology Alliance
The Results
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Pro
babi
lity
of R
espo
ndin
g C
ateg
ory
A
Human Data RPD Model ACT-R Model No Preference Regression Model
Advanced Decision Architectures Collaborative Technology Alliance
Weather Prediction
Pattern Cues (cards present)
Frequency
Probability of fine weather
1 1 2 3 19 0.895 2 1 2 4 9 0.778 3 1 2 26 0.923 4 1 3 4 9 0.222 5 1 3 12 0.833 6 1 4 6 0.500 7 1 19 0.895 8 2 3 4 19 0.105 9 2 3 6 0.500
10 2 4 12 0.167 11 2 9 0.556 12 3 4 26 0.077 13 3 9 0.444 14 4 19 0.105
Total 200 0.500
Advanced Decision Architectures Collaborative Technology Alliance
The Results
Advanced Decision Architectures Collaborative Technology Alliance
Prisoners’ Dilemma
Cooperate Defect
Cooperate (3,3) (4,0)
Defect (0,4) (1,1)
Player A
Player B
Advanced Decision Architectures Collaborative Technology Alliance
The Results
Advanced Decision Architectures Collaborative Technology Alliance
Dynamic Stocks and Flows
Advanced Decision Architectures Collaborative Technology Alliance
The Results
Advanced Decision Architectures Collaborative Technology Alliance
Some Interesting Comparisons
• Categorization– Isomorphic internal representation for different
tasks
• Dynamic Models– Very different internal representations for
similar tasks
• In general, fits are satisfying, but not very illuminating– Model vs modeler vs task vs ???
Advanced Decision Architectures Collaborative Technology Alliance
Developing a Science for Simulation
• Model comparison has roots in two traditions• The AI tradition
– Long tradition in AI of “tests” for general intelligence– Similarly, competition has emerged a means for establishing
benchmarks of performance– In both cases, the proof is in the pudding
• Success is the metric of performance
• The Hypothetico-Deductive tradition– Theories generate predictions; if the predictions are confirmed
by observation, the theory is confirmed– In this case, build a model and see if it predicts (retrodicts)
actual human performance– Experimental science 101
Advanced Decision Architectures Collaborative Technology Alliance
Conventional Wisdom
• AI competition + HD method = Model Comparison– Pick a task– Develop a bunch of models– See which ones make the best predictions
(given some measure of goodness-of-fit)– Declare a winner!
Advanced Decision Architectures Collaborative Technology Alliance
Familiar Concerns
• Concerns about fitting the data (does a good fit really confirm anything?)
• Concerns about simulating the task environment (have we made too many simplifying assumptions?)
• Concerns about models interacting with the task environment (is the model really performing the task?)
• Lots of valuable and important discussions here
Advanced Decision Architectures Collaborative Technology Alliance
A Deeper Concern
• The real focus in a model comparison shouldn’t be on the “winner” but on understanding how the various approaches are implemented– Good predictions are a minimum requirement
• The relationship between theory and model is not easily assessed– Often the most difficult part of the comparison– But the most important part
• Is there anything better than a qualitative assessment of reasonableness?
Advanced Decision Architectures Collaborative Technology Alliance
Toward a Science of Model Comparison
• A general problem here is that the history of computer simulation as experiment is not yet well understood– Cognitive models are just one application– Working at a strange intersection of theory and
engineering (cf. “Computer Science as Empirical Inquiry”)
• Absent a theory of the simulation as experiment, the best we can do is look at current and, we hope, best practices
top related