social networks: analyzing social information in deep ... · social networks: analyzing social...

Social Networks:Analyzing Social Information in Deep Convolutional Neural Networks Trained for Face Identification

SocialTraits• Humansmakesocialtraitinferencesfromfacesreadily[1]andrapidly[2]• Traitinferencespredictimportantdecisions(e.g.,votingpreferences)[3,4]• Socialtraitscanbegeneratedfrommodelsoffacestructureandreflectance[5,6]

IdentityDescriptors

Introduction&Goals

Results

0.344

0.245

0.253

0.261

0.253

0.394

0.107

0.19

0.125

0.165

0.303Anxious

Artistic

Assertive

Careless

Efficient

Impulsive

Lazy

Quiet

Shy

Talkative

Warm

0.00 0.25 0.50 0.75 1.00Coefficient of Determination

WarmTalkative

ShyQuietLazy

ImpulsiveEfficientCarelessAssertiveArtisticAnxious

DCNNsmodeledafterprimatevisualcortex

• Networkusedinthisstudycontains6convolutionallayers,3fullyconnectedlayers[9]

• State-of-the-artperformanceonchallenging,unconstrainedIJB-Adataset[10]

ConnorJ.Parde1,YingHu1,CarlosCastillo2,SwamiSankaranarayanan2,andAliceJ.O’Toole11TheUniversityofTexasatDallas,2UniversityofMaryland

TalkativeEnergeticWarmShyQuiet

SympatheticAssertiveHelpfulArtistic

AnxiousTrusting

Soft-heartedEfficientThoroughCarelessImpulsiveReliableLazy

Collectedratingsfor18SocialTraits

11UniqueDimensions

Averagedhighlycorrelatedtraits:• talkative,energetic• warm,sympathetic,soft-

hearted,trusting,helpful,reliable

• efficient,thorough

SocialTraitRatings

Humanratingsofsocialtraitsforfaces• 280faceimages• 18traitsfromBigFiveFactorsofPersonality[9]• 20setsofratingsperface• responsesaveragedacrossparticipants

AcknowledgementsThisresearchisbaseduponworksupportedbytheOfficeoftheDirectorofNationalIntelligence(ODNI),IntelligenceAdvancedResearchProjectsActivity(IARPA),viaIARPAR&DContractNo.2014-14071600012.Theviewsandconclusionscontainedhereinarethoseoftheauthorsandshouldnotbeinterpretedasnecessarilyrepresentingtheofficialpoliciesorendorsements,eitherexpressedorimplied,oftheODNI,IARPA,ortheU.S.Government.TheU.S.GovernmentisauthorizedtoreproduceanddistributereprintsforGovernmentalpurposesnotwithstandinganycopyrightannotationthereon.

Goal1:Measuresimilaritybetweenhumanand

computertraitpredictionsmade

fromidentity-trainedDCNNs

Goal3:Predictindividual

socialtraitinferencesfromtop-levelDCNNfeatures

Participants:• n =80(60female)• Meanage=21

Stimuli:• 280images,194identities• 204female,76male• Caucasian• neutralexpression• Ratingscollectedforfront-facing images

• N xK “featurematrix”obtainedfromDCNN• N xM “traitmatrix”obtainedfromaveraged

participantresponses• Predicttraitmatrixfromfeaturematrixusing

linearregression

<K features>

<Nfaces>

<M traits>

<Nfaces>

Learne

dWeigh

tMatrix

W

W

• RemoveL features(n =140)withlowlearnedweights,re-trainmodel• Keeponlyfeaturesimportantfortraitprediction

• Columnsinthefinaltraitmatrixarecomputerpredictionsofcolumnsfromoriginaldatamatrix

<K- L features>

<Nfaces>

<M traits>

<Nfaces>

New

Weigh

tMatrix

W’ ’

W’

VerifyStructureofFaceTraitSpace(e.g.[5])

• principalcomponentanalysisofhumantraitratings• created“traitspace”

• 2significantprincipalcomponents:• 1st componentinterpretedasapproachability• 2nd componentinterpretedasdominance

• Predictionsmadefromnon-frontalDCNNfeatures

DCNNsforFaceIdentification• State-of-the-artforfaceidentification[7]andgeneralizeoverviewpoint,illumination,etc.• “Top-level”DCNNsfeaturesretainnon-identityinformation(e.g.,pose,imagequality)[8]• Doface-identificationfeaturesalso retainsocialtraitinformation?

DCNNsmodeledafterprimatevisualcortex• EarlylayersmodelV1-V4,finallayersmodelITcortex• Forfaceidentification,finalDCNNlayerstores

abstractidentitycode<- facerepresentation

Trait-ProfilePredictions

IndividualTraitPredictions

PredictSocialTraitInferences

• Similaritybetweenhuman-generatedandcomputer-predictedtraitvectorsmeasuredusingcosinedistance

• AccuracyofindividualtraitpredictionsmeasuredusingR2 betweenhuman-generatedandcomputer-predictedvalues

• Errorbetweenhumanratingsandpredictedtraits,plottedagainstanulldistribution• Alltraitspredictedsignificantlyabovechance• Blueline:α=0.002• Redline:predictedvalue

• Cosinesimilaritybetweenhuman-generatedtraitprofilesandcomputerpredictions:

Goal2:Measureaccuracyoftraitpredictions

usingDCNNfeaturesfromnon-frontal

images

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Talkative"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Warm"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Shy"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Quiet"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Assertive"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Artistic"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Anxious"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Efficient"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Careless"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Impulsive"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Lazy"

Trait-PredictionError

R2 BetweenHumanInferencesandComputerPredictions

• Differenttraitspredictedtodifferentextents

• Alltraitinferencespredictedabovechance

References[1]Bruce,V.,&Young,A.(1986).Understandingfacerecognition.Britishjournalofpsychology,77(3),305-327.[2]Bar,M.,Neta,M.,&Linz,H.(2006).Veryfirstimpressions.Emotion,6(2),269.[3]Todorov,A.,Mandisodza,A.N.,Goren,A.,&Hall,C.C.(2005).Inferencesofcompetencefromfacespredictelectionoutcomes.Science,308(5728),1623-1626.[4]Rule,N.O.Ambady,N.(2008).Thefaceofsuccess: Inferencesfromchiefexecutiveofficers'appearancepredictcompanyprofits.PsychologicalScience:AJournaloftheAmericanPsychologicalSociety/APS,19,109–111.[5]Oosterhof,N.N.,&Todorov,A.(2008).Thefunctionalbasisoffaceevaluation.ProceedingsoftheNationalAcademyofSciences,105(32),11087-11092.[6]Walker,M.,&Vetter,T.(2009).Portraitsmadetomeasure:Manipulatingsocialjudgmentsabout individualswithastatisticalfacemodel.JournalofVision,9(11),12-12.[7]Taigman,Y.,Yang,M.,Ranzato,M.A.,&Wolf,L.(2014).Deepface:Closingthegaptohuman-levelperformanceinfaceverification.InProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition (pp.1701-1708).[8]Parde,C.J.,Castillo,C.,Hill,M.Q.,Colon,Y.I.,Sankaranarayanan,S.,Chen,J.C.,&O’Toole,A.J.(2017,May).FaceandImageRepresentationinDeepCNNFeatures.InAutomaticFace&GestureRecognition(FG2017),201712thIEEEInternationalConferenceon (pp.673-680).IEEE.[9]Gosling,S.D.,Rentfrow,P.J.,&SwannJr,W.B.(2003).AverybriefmeasureoftheBig-Fivepersonalitydomains.JournalofResearchinpersonality,37(6),504-528.[10]Sankaranarayanan,S.,Alavi,A.,Castillo,C.D.,&Chellappa,R.(2016,September).Tripletprobabilisticembeddingforfaceverificationandclustering.InBiometricsTheory,ApplicationsandSystems(BTAS),2016IEEE8thInternationalConferenceon (pp.1-8).IEEE.

Conclusions

Humantraitinferencescanbepredictedfromthe

top-levelfeaturesofaDCNNtrainedforface

identification

Conclusion1

Traitinferencesassignedtofrontal

facescanbepredictedfromDCNNfeaturesgeneratedforbothfrontalandnon-frontalfaces

Conclusion2

Top-levelDCNNfeaturesforface

identificationretainrobusttrait

representation– eachindividualtraitpredictedabove

chance

Conclusion3

Nulldistribution:𝝰 =arccos(0.078)

UsingKfeatures:𝝰 =arccos(0.353)

UsingK– L features:𝝰 =arccos(0.533)

• DCNNrepresentationallowsforstate-of-the-artidentification• Notindependentofimageinformation,socialtraits