adaptivelab talk1

Adaptive LabTalk Series

Electrical andComputer

Engineering

Outline

ProblemDefinition

System Model

Overall Model

Theory ofPlannedBehavior

CognitiveDissonance

Theory ofOverjustification

ControllerDesign

Assumptionsand InitialConditions

ControllerDesign: Stage1


SimulationResults

Conclusion

A Model-Based Feedback-Control Approach toBehaviour Modification ThroughReward-Induced Attitude Change

J.Ni, D. Kulic, and D. Davison

presented by: Noha El-Prince

April 16, 2013

Electrical and Computer Engineering Adaptive Lab Talk Series



Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

1 Outline

2 Problem Definition

3 System ModelOverall ModelTheory of Planned BehaviorCognitive DissonanceTheory of Overjustification

4 Controller DesignAssumptions and Initial ConditionsController Design: Stage1Controller Design: Stage2

5 Simulation Results

6 Conclusion




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Problem Definition

Trying to change the behavior of a person to a desiredbehavior.

The person may have either a negative/positive attitudetowards the desired behavior.




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Methodology

Model the internal cognitive psychological state of aperson.Design a controller based on the cognitive model.Goal: Tracking desired behavior via a sequence ofrewards.




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Overall System Model




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Theory of Planned Behavior

Aout[k] = Aout[k − 1] + ∆Aout[k − 1], (1)

∆Aout[k] = ∆ACDout [k] + ∆AOJ

out[k], (2)

Arew[k] = r1Arew[k − 1] + µ1(1− r1)R[k − 1], (3)

BI[k] = Aout[k] +Arew[k], (4)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Theory of Planned Behavior

B[k] =

Bd[k] if BI[k] ≥ Bd[k] and Aout[k] ≤ Bd[k]

Aout[k] if (BI[k] < Bd[k] and Aout[k] ≥ 0)

or Aout[k] > Bd[k]

0 otherwise.(5)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Cognitive Dissonance Theory (Block A)

A person’s behavior is inconsistent with one of hisattitudes ⇒ dissonance pressure

A person trying to reduce dissonance pressure by changingattitude/behavior

In our case : Inconsistency arises in 2 situations:

� The child declines the reward vs. value money� The child accepts the reward vs. feeling bored

How to quanitify dissonance pressure ?




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Quantifying Dissonance Pressure

Dissonance = ”%” of inconsistent cognitive pairs

PCDraw [k] =

Bsgn[k] Mincon[k]

Mincon[k]+Mcon[k]if Mincon[k] +Mcon[k] > 0

0 otherwise.

(6)

Bsgn[k] =

{+1 if B[k] ≥ Bd[k] or Aout[k] ≥ 0−1 otherwise.

(7)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Quantifying Dissonance Pressure - cont.

M1incon[k] =

{|Arew[k]| if sgn(Arew [k]) 6= Brel[k]

0 otherwise,(8)

M2incon[k] =

{|Aout[k]| if sgn(Aout[k]) 6= Bsgn[k]

0 otherwise,(9)

M1con[k] =

{|Arew[k]| if sgn(Arew [k]) = Brel[k]

0 otherwise,(10)

M2con[k] =

{|Aout[k]| if sgn(Aout[k]) = Bsgn[k]

0 otherwise,(11)

Mincon[k] =2∑

i=1

Miincon[k], Mcon[k] =

2∑i=1

Micon[k], (12)

Brel[k] =

{+1 if B[k] ≥ Bd[k]−1 otherwise.

(13)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Special Case: Attitude Reversal

Aout[k] small, R[k] small, Bd[k] is high ⇒ Child declinesthe reward

To reduce Diss. pressure: increase Aout OR “give up”jogging ⇒ Aout[k] <<<

r[k] =

+1 if Bd[k]−BI[k] > αrevAout[k], Aout[k] ≥ 0,

K1PCD[k] > 2Aout[k], and Arew[k] > 0,

−1 otherwise.

(14)

PCD[k] =

{(1− r2)PCD

raw [k] if r[k − 1] = 1

r2PCD[k − 1] + (1− r2)PCD

raw [k] otherwise.(15)

PrawCD




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Quantifying ∆Aout

Assume the change in Aout[k] is proportional to dissonancepressure, with proportionality constant K1 > 0:

∆ACDout [k] =

{−K1P

CD[k] if r[k] = 1

+K1PCD[k] otherwise.

(16)

PCD




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Overjustification Theory (Block B)

Overjustification Theory

when a reward is given to a person to do something thatshe/he already enjoys doing, such rewards arecounter-productive in that they reduce the intrinsic desire ofthe person towards that behavior.




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Overjustification Theory - cont.

Let Bt[k] = minimal attitude level to which theoverjustification effect can drive Aout[k].Assume Bt[k] is a constant fraction of Bd[k], i.e.,

Bt[k] = αBd·Bd[k], (17)

for some constant 0 < αBd< 1.

If Bt[k] > Aout[k] ⇒ overjustification pressure does notdecrease Aout, and the reverse is true i.e.

Arelout[k] = max{0, Aout[k]−Bt[k]}. (18)

where Arelout[k]: a relative attitude with respect to Bt[k]




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Overjustification Theory - cont.

Then the raw and filtered overjustification pressures, and the resulting change in intrinsic attitude, arecomputed just as in our previous work, but using Arel

out instead of Aout, as follows:

POJraw [k] =

Arelout[k]Arew[k] if Arel

out[k] > 0 and Arew[k] > 0and B[k] ≥ Bd[k]

0 otherwise,

(19)

POJ

[k] = r3POJ

[k − 1] + (1− r3)POJraw[k], (20)

∆AOJout[k] =

{−K2P

OJ [k] if K2POJ [k] ≤ Arel

out[k]

−Arelout[k] otherwise.

(21)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Controller Design: Assumptions

Mother Knows varoius plant parameters(µ1, r1, r2, r3, αrev, αBd, k1, k2)andA

∗0.

The child do not know the value of B∗d .

Bd[k + 1] is assigned to the child by end of day k.

i.c: PCD[0] = POJ [0] = Arew[0] = 0, Aout = A∗0.

Reward is not given everyday: N= Settling time

If impulsive reward applied at time 0, a transient(1− rk−1

2 ) appears.

Approach: wait for the transient to settle before applyingthe next impulsive reward.




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Controller Design: Stage1

BI[k + 1] ≥ Bd[k + 1].

⇓R[k] >>> enough to force B[k + 1] > 0. >>

⇓Bsgn[k + 1] = +1.

⇓PCDraw [k + 1] > 0. >>

⇓Goal: increase Aout from −ve to +ve.




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Controller Design: Stage1- cont.

BI[k] = Aout[k] +Arew[k]

= A0 + µ1R[k] ≥ Bd[k + 1]

R[k] =Bd[k + 1] + |A0|

µ1(22)

The associated dissonance pressure is:

PCDraw [k + 1] =

Bsgn[k + 1] · |Aout[k + 1]

|Aout[k + 1]|+Arew[k + 1]=

|A0||A0|+ µ1R[k]

.

(23)

Maximizing (23) subject to (22) results in Bd[k + 1] = 0 andR[k] = |A0|/µ1. For improved robustness:

Bd[k + 1] = 2ε (24)

R[k] =2Bd[k + 1] + |Aout[k]|

µ1=

2ε+ |Aout[k]|µ1

(25)

(26)Electrical and Computer Engineering Adaptive Lab Talk Series



Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion


Goal: (0 ≤ Aout[k] ≤ B∗d) for k = 0, N, 2N, 3N, . . .

Use sequence of reward impulses, each impulse appliedevery N days.

Inorder to raise Aout[k], give the child R[k]<<< enoughto be :

Rejected by the child ⇒ PCD < 0⇒ Aout ⇑ .

Avoid exciting the OVJ dynamics that makes Aout ⇓ .Avoid attitude reversal.

Q. What is the appropriate value of R[k] that guaranteeabove three conditions satisfied ?




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion





Rejected by the child ⇒ PCD < 0⇒ Aout ⇑ .Avoid exciting the OVJ dynamics that makes Aout ⇓ .

Avoid attitude reversal.





Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion





Rejected by the child ⇒ PCD < 0⇒ Aout ⇑ .Avoid exciting the OVJ dynamics that makes Aout ⇓ .Avoid attitude reversal.





Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Controller Design: Stage2 - cont.

To enforce the child to reject the reward R[k] force:

BI[k + 1] < Bd[k + 1]

Aout[k] +Arew[k] < Bd[k + 1]

Aout[k] + r1Arew[k − 1] + µ1(1− r1)R[k − 1] < Bd[k + 1]

R[k] <Bd[k + 1]− r1Arew[k]−Aout[k]

µ1(1− r1)

R[k] <Bd[k + 1]−Aout[k]

µ1(27)

Equation(27) gurantees child reject reward and OJ = 0.




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion


Attitude reversal is avoided on day k+1 if R[k] is chosen s.t :

Bd[k]−BI[k] ≤ αrevAout[k], Aout[k] ≥ 0

Bd[k] +Aout[k]−Arew[k] ≤ αrevAout[k]

Aout[k] + r1Arew[k − 1] + µ1(1− r1)R[k − 1] ≤ Bd[k + 1]

R[k] ≥ Bd[k + 1]− (αrev + 1)Aout[k]

µ1(28)

Equation(28) gurantees avoidance of attitude reversal.Q. How to keep R[k] at a reasonable level ?




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion


By introducing a controller tuning parameter β ∈ (0, 1), theaggressiveness of attitude increase can be adjusted:

Ad = βAout[k] + (1− β)(Aout[k] +K1(1− rN−12 )).

R[k] =Aout[k]

µ1

(K1(1− rN−1

2 )

Aout[k] +K1(1− rN−12 )−Ad

− 1

). (29)

To avoid driving the attitude higher than needed (i.e., beyondB∗

d), we add a saturator as follows:

Ad = min{B∗d , βAout[k] + (1− β)(Aout[k] +K1(1− rN−1

2 ))}.(30)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion


Get the value of Bd[k + 1] from the formulas of R[k] :

Bdmin[k] = Aout[k](K1(1− r2)N−1

Aout[k] +K1(1− r2)N−1 −Ad(31)

Bdmax[k] = Aout[k](K1(1− r2)N−1

Aout[k] +K1(1− r2)N−1 −Ad+ αrev

(32)

Bdmin[k] < Bd[k + 1] ≤ Bdmax[k]. (33)

Bd[k + 1] = γBdmin[k] + (1− γ)Bdmax[k]. (34)




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Simulation Results

0 5 10 15 20 25 30 350

50

100

150

Day number (k)

Behavio

r (m

ins)

Bd

*

B[k]

Bd[k]

Open−Loop Implementation

0 5 10 15 20 25 30 35

0

50

100

YESYES

YESYES NO

NO

NO

Day number (k)

Rew

ard

Offere

d (

$)

R[k]

0 5 10 15 20 25 30 35

−50

0

50

Day number (k)

Attitude (

min

s)

Aout

[k]

∆ Aout

CD[k]




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Simulation Results

0 5 10 15 20 25 30 350

50

100

150

Day number (k)

Behavio

r (m

ins)

Bd

*

B[k]

Bd[k]


0 5 10 15 20 25 30 35

0

50

100

YESYES

YESYES

NONO

NO

NO

Day number (k)

Rew

ard

Offere

d (

$)

R[k]

0 5 10 15 20 25 30 35

−50

0

50

Day number (k)

Attitude (

min

s)

Aout

[k]

∆ Aout

CD[k]




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Simulation Results

0 5 10 15 20 25 30 350

50

100

150

Day number (k)

Behavio

r (m

ins)

Bd

*

B[k]

Bd[k]


0 5 10 15 20 25 30 35

0

50

100

YESYES

YESYES

NO NO NO NO NO NONO

Day number (k)

Rew

ard

Offere

d (

$)

R[k]

0 5 10 15 20 25 30 35

−50

0

50

Day number (k)

Attitude (

min

s)

Aout

[k]

∆ Aout

CD[k]




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion

Conclusion and Future Work

A new model-based behavior-modification algorithm havebeen developed.

Pros:

No reward are required in the long term.

Good transient behavior (i.e. no overshoot).Flexible timing of the control scheme.

Cons:

The approach requires good knowledge of the plantparameters.

In case closed-loop implementation: A regularmeasurement of Aout is needed.Lacks experimental validation of the plant model.

Future work:

Online parameter estimation of plant parameters.

Experimental validation of plant model




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion



Pros:

No reward are required in the long term.Good transient behavior (i.e. no overshoot).

Flexible timing of the control scheme.

Cons:

The approach requires good knowledge of the plantparameters.In case closed-loop implementation: A regularmeasurement of Aout is needed.

Lacks experimental validation of the plant model.

Future work:

Online parameter estimation of plant parameters.Experimental validation of plant model




Engineering

Outline

ProblemDefinition

System Model

Overall Model


CognitiveDissonance


ControllerDesign




SimulationResults

Conclusion



Pros:

No reward are required in the long term.Good transient behavior (i.e. no overshoot).Flexible timing of the control scheme.

Cons:

The approach requires good knowledge of the plantparameters.In case closed-loop implementation: A regularmeasurement of Aout is needed.Lacks experimental validation of the plant model.

Future work:

Online parameter estimation of plant parameters.Experimental validation of plant model


adaptivelab talk1

Documents

desired behavior

aoutk bdkaoutk

aoutk bdk0

bik bdk

stage1controller design

cognitive model

value moneythe child

feeling boredelectrical