trials and tribulations
DESCRIPTION
Trials and Tribulations. Architectural Constraints on Modeling a Visuomotor Task within the Reinforcement Learning Paradigm. Subject of Investigation. How humans integrate visual object properties into their action policy when learning a novel visuomotor task. BubblePop ! - PowerPoint PPT PresentationTRANSCRIPT
TRIALS AND TRIBULATIONS
Architectural Constraints on Modeling a Visuomotor Task within the Reinforcement Learning Paradigm
SUBJECT OF INVESTIGATION
How humans integrate visual object properties into their action policy when learning a novel visuomotor task.
• BubblePop!
Problem: Too many possible questions…
Solution: Motivate behavioral research by looking at modeling difficulties.
• Nonobvious crossroads
APPROACH
Since the task has a scalar performance signal, model must utilize reinforcement learning.
• Temporal Difference Back Propagation
Start with an extremely simplified version of the task and add back the complexity once you have a successful model.
Analyze the representational and architectural constraints necessary for each model.
5x5 grid-world
4 possible actions• Up, down, left, right
1 unmoving target
Starting locations of target and agent randomly assigned
Fixed reward upon reaching target and a new target generated
Epoch ends after fixed number of steps
FIRST STEPS: DUMMY WORLD
DUMMY WORLD ARCHITECTURES
25 units for the grid 4 Actions
8 Hidden Layer
1
context
Expected Reward
(ego only)
The whole grid (allocentric), or agent centered (egocentric)
Current architectures learn each action independently.
‘Up’ is like ‘Down’, but different.
• It shifts the world
1 action, 4 different inputs• “In which rotation of the
world would you rather go ‘up’ in?”
BUILDING IN SYMMETRY
Scaled grid size up to 10x10• Not as unrealistic as one might think… (tile
coding)
Scaled number of targets• Difference from 1 to 2, but not from 2 to
many.
Confirmed ‘winning-est’ representation
Added memory
WORLD SCALING
Added a ‘ripeness’ dimension to target, and changed the reward function:If target.ripeness >.60
reward = 1;
Else
reward = -.66667;
NO LOW HANGI NG FRUI T:THE R I P ENESS P ROBL EM
How the problem occurs:
1. At a high temperature you move randomly.
2. The random pops net zero reward.
3. The temperature lowers and you ignore the target entirely.
ANN EAL ING AWAY THE CU RS E O F P ICKINESS
No feedback for almost ripe
So how could we anneal our ripeness criterion?
Anneal the amount you care about unripe pops.
Differentiate internal and extern reward functions
A PS YCHO L OGICAL LY PL AUS IBL E SO LUTION
FUTURE DIRECTIONS
Investigate how the type of ripeness difficulty impacts computational demands.
• Difficulty due to reward schedule vs. perceptual acuity vs. redundancy vs. conjunctive-ness vs. ease of prediction
How to handle the ‘Feature Binding ‘Problem’ in this context• Emergent binding through deep learning?
Just keep increasing complexity and see what problems crop up.
• If the model gets to human level performance without a hitch, then that’d be pretty good to.
SUMMARY& DISCUSSION
Egocentric representations pay off in this domain, even with the added memory cost.
• In any domain with a single agent?
Symmetries in the action space can be exploited to greatly expedite learning
• Could there be a general mechanism for detecting such symmetries?
Difficult reward functions might be learnt via annealing internal reward signals.
• How could we have this annealing emerge from the model?
QUESTIONS?