advanced decision architectures collaborative technology alliance a computational model of...

Advanced Decision Architectures Collaborative Technology Alliance

A Computational Model of Naturalistic Decision Making and the Science of

Simulation

Walter WarwickAmy Santamaria

& many, many others


Overview

• The M&S big picture

• The work– Birth

• You can’t model this; that’s not a model

– Life• Where’s the data

– Quiet reflection• The science of simulation


The Big Picture

• An effort to improve human behavior representations for M&S but incorporating a better model of decision making– Better than: a “tactical” or probabilistic

decision– Allows new kinds of behavior to play inside of

task network models

• Not an exercise in theory validation– Though we’d like our work to illuminate theory


The Birth of the RPD Widget

• From a descriptive model to a theoretical model:– A clash of traditions– A lot of thrashing– The emergence of a cottage industry and an M&S land grab

• A new decision type (“RPD”) in the Micro Saint Sharp family of task network modeling tools

• Widget intended to capture:– Experience-based decision making via a multiple trace model of

memory and simple reinforcement routine– Recognitional decision making via similarity-based recall

mechanism that draws on *every* past experience– Expectancy generation and feedback—several different versions

implemented, rarely used and no clear indication that we can do anything interesting with it


Using the Widget

• To specify an RPD decision type, the modeler supplies:

• Cues that prompt recognition (map MSS variables into “subjective,” discrete cues)

• Alternative courses of action (usually given by the structure of the task network)

• Reinforcement (seat of the pants)• Set run-time properties and parameters (seat of the pants)

• This defines the structure of each “trace”—a individual decision making experience comprising the cue values at decision time, the action that was taken and the outcome (good or bad)


What You Get

• Four applications (validation studies); two flavors:– Categorization: Brunswik Faces and Weather

Prediction– Dynamic behavior: Prisoners’ Dilemma and

Dynamic Stocks and Flows


Brunswik Faces


The Results

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Pro

babi

lity

of R

espo

ndin

g C

ateg

ory

A

Human Data RPD Model ACT-R Model No Preference Regression Model


Weather Prediction

Pattern Cues (cards present)

Frequency

Probability of fine weather

1 1 2 3 19 0.895 2 1 2 4 9 0.778 3 1 2 26 0.923 4 1 3 4 9 0.222 5 1 3 12 0.833 6 1 4 6 0.500 7 1 19 0.895 8 2 3 4 19 0.105 9 2 3 6 0.500

10 2 4 12 0.167 11 2 9 0.556 12 3 4 26 0.077 13 3 9 0.444 14 4 19 0.105

Total 200 0.500


The Results


Prisoners’ Dilemma

Cooperate Defect

Cooperate (3,3) (4,0)

Defect (0,4) (1,1)

Player A

Player B


The Results


Dynamic Stocks and Flows


The Results


Some Interesting Comparisons

• Categorization– Isomorphic internal representation for different

tasks

• Dynamic Models– Very different internal representations for

similar tasks

• In general, fits are satisfying, but not very illuminating– Model vs modeler vs task vs ???


Developing a Science for Simulation

• Model comparison has roots in two traditions• The AI tradition

– Long tradition in AI of “tests” for general intelligence– Similarly, competition has emerged a means for establishing

benchmarks of performance– In both cases, the proof is in the pudding

• Success is the metric of performance

• The Hypothetico-Deductive tradition– Theories generate predictions; if the predictions are confirmed

by observation, the theory is confirmed– In this case, build a model and see if it predicts (retrodicts)

actual human performance– Experimental science 101


Conventional Wisdom

• AI competition + HD method = Model Comparison– Pick a task– Develop a bunch of models– See which ones make the best predictions

(given some measure of goodness-of-fit)– Declare a winner!


Familiar Concerns

• Concerns about fitting the data (does a good fit really confirm anything?)

• Concerns about simulating the task environment (have we made too many simplifying assumptions?)

• Concerns about models interacting with the task environment (is the model really performing the task?)

• Lots of valuable and important discussions here


A Deeper Concern

• The real focus in a model comparison shouldn’t be on the “winner” but on understanding how the various approaches are implemented– Good predictions are a minimum requirement

• The relationship between theory and model is not easily assessed– Often the most difficult part of the comparison– But the most important part

• Is there anything better than a qualitative assessment of reasonableness?


Toward a Science of Model Comparison

• A general problem here is that the history of computer simulation as experiment is not yet well understood– Cognitive models are just one application– Working at a strange intersection of theory and

engineering (cf. “Computer Science as Empirical Inquiry”)

• Absent a theory of the simulation as experiment, the best we can do is look at current and, we hope, best practices

advanced decision architectures collaborative technology alliance a computational model of...

Documents