Dr. Bruce J. West

Chief Scientist

Mathematical & Information Science Directorate

Army Research Office

Bruce.j.west@us.army.mil

919-549-4257

UNCLASSIFIED / FOR OFFICIAL USE ONLY

UNCLASSIFIED / FOR OFFICIAL USE ONLY

Decision Making and Stochastic Delay

at

Workshop on Social Computing, Behavior Modeling and

Prediction

1 April 2008

• Discounted Utility Model & intertemporal choice• Anomalies from discounted utility theory

– irrationality– hyperbolic discounting

• Objective and subjective time– entropy and the direction of time

– time as a stochastic variable

• Individuality and paternalism– some experiments

– fit of theory to data

• Conclusions

Outline of talk

Decision Making and Delay

Discounted Utility Model (DUM)

uuuuUT

tTtt

t

11 ,...,,

• Discount factor δ compresses many mechanisms• mortality, uncertainty, time compression,…

• Accepted as both normative (how things should be) and descriptive (how things are)…..but was initially arbitrary Samuelson (1937).

• Exponential form implies time consistency (rationality)

Decision Making and Delay

Anomalies from DUM

• Time inconsistency– empirical discount factor is not constant

• over time

• across type of intertemporal choices

• Delay effect (hyperbolic discounting)• Interval effect (non-stationarity)• Sign effect (gains vs. loses)• Magnitude effect (small vs. large)• Direction effect• Sequence effects (ordered set vs. single)

Decision Making and Delay

Model comparison

• Exponential delay model– monotonic decrease in valuewith objective time– constant rate results in timeconsistency – rationality

• Hyperbolic delay model– decreasing rate results in time inconsistency – irrationality (preference reversal)

hyperbolic

exponential

exponential

hyperbolic

hyperbolic

exponential

Decision Making and Delay

Objective vs. subjective time

• Hyperbolic models– Objective time

• clockwork universe

• entropy and the direction of time

– Subjective time• unidirectional

• probability and statistics

• Motivate decision-making using

ensemble distributions– subjective time– stochastic delays

Decision Making and Delay

Delay and uncertainty• Decision-making models of intertemporal choice can be extended to incorporate probabilistic choice where

p is the probability of reward at time t and F is an unspecified function.

discrete continuous

• No reward before delay time t

• Delay-time probability density

)F(p)U(U(t) 0

dττ ψp

t)ΨU(t)=U(dtt

0

dt

tdt

Decision Making and Delay

F

Fr

'• Deterministic discount rate is replaced with a conditional

probability per unit time

• The ratio of the delay time distribution function to the survival probability density, integrates to

• The utility function in terms of subjective time is therefore

tt

ttttobt

trt

Pr1

lim)(0

tdttrt

0''exp

tdttrUtUtU

0''exp00

Stochastic rate

Decision Making and Delay

Example rate

• Rate of reward production suggested by hyperbolic distribution

• Probability of no reward before time t is

so that the utility function is inverse power law

• T measures response time and α measures irrationality

tr

rtr

1

0

1)(

tT

T

trtr

dtrt

r

r

t

1

0

1

01

0

1

1

'1

'exp

Trr

TtT

TUtU 0

1

;1

;)0()( = =

Decision Making and Delay

Experimental data

• Students (20) asked to make decisions about hypothetical

money to be received immediately or at a later time, concerning

the subjects themselves or another person not known to them. Takahashi, Physica A (2007).

self other

T=31α=0.28

T=1.85α=0.11

Decision Making and Delay

Implications from experiments

• The response times could describe paternalistic policy making government officials, where irrationality is enhanced.

• Irrationality is nowhere more significant than in the military where choices may determine whether others live or die.

Decision Making and Delay

Decision Making and Delay

• Nonlinear dynamic equation solved on the interval (0,1).

• define a delay-time distribution density

• assume a uniform distribution of initial conditions to obtain

• delay-time distribution density is non-Poisson, renewal and non-ergodic

dq

dtaq z

zazq 1

1

0 11

ddqqp 00 )(

1

;11

T

T

Measured discount rates

• Higher discount rates compared with controls* ( smaller T and α in stochastic intertemporal model)– smoking– excessive alcohol consumption– illicit drug use (cocaine, crack-cocaine and heroin)– pathological gambling– age– cognitive ability (negative correlation with intellectual

achievement)

• Consistent with neuroeconomic hypothesis that prefrontal cortex is essential for patient (forward looking) decision making.

* Chabris, Laibson & Schuldt, The New Palgrave Dictionary of Economics (2007).

Decision Making and Delay

Brain Activity

Sanfey, Loewenstein, McClure and Cohen, TRENDS in Cognitive Science (2006).

β network”: midbrain dopamine network; reward processing (ventral striatum V.Str. and medial prefrontal cortex mPFC.)

δ network: cognition; dorsolateral prefrontal cortex (diPFC) and right posterior parletal cortex (R.Par.)

Decision Making and Delay

More brain activity

15.0 5.0

5.0• Two discounting slopes

• < one year

• > one year

• Different parts of the brain light

up under fMRI• short-term

• long-term

Wittmann & Paulus (2007)

5.0

Decision Making and Delay

Decision Making and Delay

Conclusions and Speculations

• decision-making is not always rational

• irrationality in intertemporal choice models take a hyperbolic form

• inverse power laws or hyperbolic utility functions can be generated by stochastic delay times

• different parts of the brain control decisions associated with long and short delay times

• the complexity of the brain produces the subjective nature of biological time