11/27/18

1

Learningcomplexvisualconcepts

detectinghands directionofgaze

Difficulttodetect,appearearly,importantforsubsequentlearningofagents,goals,interactions

11/27/18

2

Whyarehandsdifficulttodetect?

VanGogh Kirchnermultipleappearances

smallandinconspicuous

Selectivitytohandsappearsearlyininfancy

usingaheadcameratostudyvisualexperience

“overall…handswereinviewanddynamicallyactingonanobjectinover80%oftheframes”

Yoshida&Smith2008

Whatmakeshandslearnablebyhumans?

11/27/18

3

Motion,handas‘mover’(7monthsold)

Saxe,Carey(2006)Theperceptionofcausalityininfancy

Earlysensitivitytospecialmotiontypes• highsensitivitytomotioningeneral

(detectingmotion,motionsegmentation,tracking)

• specificsub-classesofmotion:self-motion,passive,and‘mover’

Aspecificmotionishighlyindicativeofhands

11/27/18

4

Detecting‘mover’eventsUllman,Harari,Dorfman (2012)

•moverevent=movingimageregioncausingastationaryregiontomoveorchangeaftercontact

• simpleandprimitive,priortoobjectsorfigure-groundsegmentation

•‘mover’asaninnateteachingsignalforlearningtheconceptofahand

Trainingvideos

Moviesofscenes,peoplemoving,manipulatingobjects,movinghands

‘Mover’eventsaredetectedinallmoviesandusedfortraining

11/27/18

5

‘Mover’eventsextractedfromvideos

Highfractionofdetectedmovereventsinvolvehands(90%recall,65%precision)

Extractmanyimagesofobjectsenteringcellsofmoverevents&useimagestotrainahandobjectclassifier

Handdetectioninstillimages

Trainedclassifiermainlydetectedhandsinobjectmanipulationscenes

11/27/18

6

ContinuedlearningTwodetectionalgorithms:

• Detecthandswithgreaterrangeofappearancebytracking

• Detectmorehandsbythebodycontext

Handbysurroundingcontext

faceshoulderupper-arm lower-armhand

Amano, Kezuka, Yamamoto, 2004Slaughter, Heron-Delaney, 2010Slaughter, Neary, 2011

11/27/18

7

appearance context

Gazedirection

• infantsfollowthegazeofothers• startsat3-6monthsandcontinuestodevelop• headorientationfirst,eyecueslater• importantindevelopmentofcommunication&language• modelingmainlyheaddirection

W.H.Wollaston (1824)OntheApparentDirectionofEyesinaPortrait

11/27/18

8

Gazecuesaresubtleandinconspicuous

movereventssupplytheteachingsignal

Gazeextraction

Training Testing

humansmodel

Usefacialappearancesandgazedirectionsderivedfrommovereventsasatrainingsettolearndirectionofgaze