Upload File

social networks: analyzing social information in deep ... · social networks: analyzing social...

  • Home
  • Documents
  • Social Networks: Analyzing Social Information in Deep ... · Social Networks: Analyzing Social Information in Deep Convolutional Neural Networks Trained for Face Identification Social
1
Social Networks: Analyzing Social Information in Deep Convolutional Neural Networks Trained for Face Identification Social Traits • Humans make social trait inferences from faces readily [1] and rapidly [2] • Trait inferences predict important decisions (e.g., voting preferences) [3, 4] • Social traits can be generated from models of face structure and reflectance [5, 6] Identity Descriptors Introduction & Goals Results 0.344 0.245 0.253 0.261 0.253 0.394 0.107 0.19 0.125 0.165 0.303 0.00 0.25 0.50 0.75 Coefficient of Determination Warm Talkative Shy Quiet Lazy Impulsive Efficient Careless Assertive Artistic Anxious DCNNs modeled after primate visual cortex • Network used in this study contains 6 convolutional layers, 3 fully connected layers [9] • State-of-the-art performance on challenging, unconstrained IJB-A dataset [10] Connor J. Parde 1 , Ying Hu 1 , Carlos Castillo 2 , Swami Sankaranarayanan 2 , and Alice J. O’Toole 1 1 The University of Texas at Dallas, 2 University of Maryland Talkative Energetic Warm Shy Quiet Sympathetic Assertive Helpful Artistic Anxious Trusting Soft-hearted Efficient Thorough Careless Impulsive Reliable Lazy Collected ratings for 18 Social Traits 11 Unique Dimensions Averaged highly correlated traits: • talkative, energetic • warm, sympathetic, soft- hearted, trusting, helpful,reliable • efficient, thorough Social Trait Ratings Human ratings of social traits for faces • 280 face images • 18 traits from Big Five Factors of Personality [9] • 20 sets of ratings per face • responses averaged across participants Acknowledgements This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014- 14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representingthe official policies or endorsements,either expressedor implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Governmentis authorized to reproduce and distribute reprints for Governmentalpurposes notwithstandinganycopyright annotation thereon. Goal 1: Measure similarity between human and computer trait predictionsmade from identity-trained DCNNs Goal 3: Predict individual social trait inferences from top- level DCNN features Participants: • n = 80 (60 female) • Mean age = 21 Stimuli: • 280 images, 194 identities • 204 female, 76 male • Caucasian • neutral expression • Ratings collected for front- facing images • N x K “feature matrix” obtained from DCNN • N x M “trait matrix” obtained from averaged participant responses • Predict trait matrix from feature matrix using linear regression < K features > < N faces > < M traits > < N faces > Learned Weight Matrix W W • Remove L features (n = 140) with low learned weights, re-train model • Keep only features importantfor trait prediction • Columns in the final trait matrix are computer predictionsof columns from original data matrix < K-L features > < N faces > < M traits > < N faces > New Weight Matrix W ’ ’ W’ Verify Structure of Face Trait Space (e.g. [5]) • principal component analysis of human trait ratings • created “trait space” • 2 significant principal components: • 1 st component interpreted as approachability • 2 nd component interpreted as dominance • Predictions made from non-frontalDCNN features DCNNs for Face Identification • State-of-the-art for face identification [7] and generalize over viewpoint, illumination, etc. • “Top-level” DCNNs features retain non-identity information (e.g., pose, image quality) [8] • Do face-identification features also retain social trait information? DCNNs modeled after primate visual cortex • Early layers model V1-V4, final layers model IT cortex • For face identification, final DCNN layer stores abstract identity code <- face representation Trait-Profile Predictions Individual Trait Predictions Predict Social Trait Inferences • Similarity between human-generated and computer-predicted trait vectors measured using cosine distance • Accuracy of individual trait predictions measured using R 2 between human-generated and computer-predicted values • Error between human ratings and predicted traits, plotted against a null distribution • All traits predicted significantly above chance • Blue line: α = 0.002 • Red line: predicted value • Cosine similarity between human-generated trait profiles and computer predictions: Goal 2: Measure accuracy of trait predictions using DCNN features from non-frontal images 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Talkative" 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Warm" 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Shy" 0 10 20 30 40 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Quiet" 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Assertive" 0 10 20 30 40 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Artistic" 0 10 20 30 40 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Anxious" 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Efficient" 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Careless" 0 10 20 30 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Impulsive" 0 10 20 30 40 0.00 0.25 0.50 0.75 1.00 Prediction Error count "Lazy" Trait-Prediction Error R 2 Between Human Inferences and Computer Predictions • Different traits predicted to different extents • All trait inferences predicted above chance References [1] Bruce, V., & Young, A. (1986). Understanding face recognition. British journal of psychology, 77(3), 305-327. [2] Bar, M., Neta, M., & Linz, H. (2006). Very first impressions. Emotion, 6(2), 269. [3] Todorov, A., Mandisodza, A. N., Goren, A., & Hall, C. C. (2005). Inferences of competence from faces predict election outcomes. Science, 308(5728), 1623-1626. [4] Rule, N. O. Ambady, N. (2008). The face of success: Inferences from chief executive officers' appearance predict company profits. Psychological Science: A Journal of the American Psychological Society/APS, 19, 109–111. [5] Oosterhof, N. N., & Todorov, A. (2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences, 105(32), 11087-11092. [6] Walker, M., & Vetter, T. (2009). Portraits made to measure: Manipulating social judgments about individuals with a statistical face model. Journal of Vision, 9(11), 12-12. [7] Taigman, Y., Yang, M., Ranzato, M. A., & Wolf, L. (2014). Deepface: Closing the gap to human- level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1701-1708). [8] Parde, C. J., Castillo, C., Hill, M. Q., Colon, Y. I., Sankaranarayanan, S., Chen, J. C., & O’Toole, A. J. (2017, May). Face and Image Representation in Deep CNN Features. In Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on (pp. 673-680). IEEE. [9] Gosling, S. D., Rentfrow, P. J., & Swann Jr, W. B. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in personality, 37(6), 504-528. [10] Sankaranarayanan, S., Alavi, A., Castillo, C. D., & Chellappa, R. (2016, September). Triplet probabilistic embedding for face verification and clustering. In Biometrics Theory, Applications and Systems (BTAS), 2016 IEEE 8th International Conference on (pp. 1-8). IEEE. Conclusions Human trait inferences can be predicted from the top-level features of a DCNN trained for face identification Conclusion 1 Trait inferences assigned to frontal faces can be predicted from DCNN features generated for both frontal and non-frontal faces Conclusion 2 Top-level DCNN features for face identification retain robust trait representation – each individual trait predicted above chance Conclusion 3 Null distribution: =arccos( 0.078) Using K features: = arccos(0.353) Using K–L features: = arccos(0.533) • DCNN representation allows for state-of- the-art identification • Not independentof image information, social traits

Upload: others

Post on 13-Jul-2020

17 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Social Networks: Analyzing Social Information in Deep ... · Social Networks: Analyzing Social Information in Deep Convolutional Neural Networks Trained for Face Identification Social

Social Networks:Analyzing Social Information in Deep Convolutional Neural Networks Trained for Face Identification

SocialTraits• Humansmakesocialtraitinferencesfromfacesreadily[1]andrapidly[2]• Traitinferencespredictimportantdecisions(e.g.,votingpreferences)[3,4]• Socialtraitscanbegeneratedfrommodelsoffacestructureandreflectance[5,6]

IdentityDescriptors

Introduction&Goals

Results

0.344

0.245

0.253

0.261

0.253

0.394

0.107

0.19

0.125

0.165

0.303Anxious

Artistic

Assertive

Careless

Efficient

Impulsive

Lazy

Quiet

Shy

Talkative

Warm

0.00 0.25 0.50 0.75 1.00Coefficient of Determination

WarmTalkative

ShyQuietLazy

ImpulsiveEfficientCarelessAssertiveArtisticAnxious

DCNNsmodeledafterprimatevisualcortex

• Networkusedinthisstudycontains6convolutionallayers,3fullyconnectedlayers[9]

• State-of-the-artperformanceonchallenging,unconstrainedIJB-Adataset[10]

ConnorJ.Parde1,YingHu1,CarlosCastillo2,SwamiSankaranarayanan2,andAliceJ.O’Toole11TheUniversityofTexasatDallas,2UniversityofMaryland

TalkativeEnergeticWarmShyQuiet

SympatheticAssertiveHelpfulArtistic

AnxiousTrusting

Soft-heartedEfficientThoroughCarelessImpulsiveReliableLazy

Collectedratingsfor18SocialTraits

11UniqueDimensions

Averagedhighlycorrelatedtraits:• talkative,energetic• warm,sympathetic,soft-

hearted,trusting,helpful,reliable

• efficient,thorough

SocialTraitRatings

Humanratingsofsocialtraitsforfaces• 280faceimages• 18traitsfromBigFiveFactorsofPersonality[9]• 20setsofratingsperface• responsesaveragedacrossparticipants

AcknowledgementsThisresearchisbaseduponworksupportedbytheOfficeoftheDirectorofNationalIntelligence(ODNI),IntelligenceAdvancedResearchProjectsActivity(IARPA),viaIARPAR&DContractNo.2014-14071600012.Theviewsandconclusionscontainedhereinarethoseoftheauthorsandshouldnotbeinterpretedasnecessarilyrepresentingtheofficialpoliciesorendorsements,eitherexpressedorimplied,oftheODNI,IARPA,ortheU.S.Government.TheU.S.GovernmentisauthorizedtoreproduceanddistributereprintsforGovernmentalpurposesnotwithstandinganycopyrightannotationthereon.

Goal1:Measuresimilaritybetweenhumanand

computertraitpredictionsmade

fromidentity-trainedDCNNs

Goal3:Predictindividual

socialtraitinferencesfromtop-levelDCNNfeatures

Participants:• n =80(60female)• Meanage=21

Stimuli:• 280images,194identities• 204female,76male• Caucasian• neutralexpression• Ratingscollectedforfront-facing images

• N xK “featurematrix”obtainedfromDCNN• N xM “traitmatrix”obtainedfromaveraged

participantresponses• Predicttraitmatrixfromfeaturematrixusing

linearregression

<K features>

<Nfaces>

<M traits>

<Nfaces>

Learne

dWeigh

tMatrix

W

W

• RemoveL features(n =140)withlowlearnedweights,re-trainmodel• Keeponlyfeaturesimportantfortraitprediction

• Columnsinthefinaltraitmatrixarecomputerpredictionsofcolumnsfromoriginaldatamatrix

<K- L features>

<Nfaces>

<M traits>

<Nfaces>

New

Weigh

tMatrix

W’ ’

W’

VerifyStructureofFaceTraitSpace(e.g.[5])

• principalcomponentanalysisofhumantraitratings• created“traitspace”

• 2significantprincipalcomponents:• 1st componentinterpretedasapproachability• 2nd componentinterpretedasdominance

• Predictionsmadefromnon-frontalDCNNfeatures

DCNNsforFaceIdentification• State-of-the-artforfaceidentification[7]andgeneralizeoverviewpoint,illumination,etc.• “Top-level”DCNNsfeaturesretainnon-identityinformation(e.g.,pose,imagequality)[8]• Doface-identificationfeaturesalso retainsocialtraitinformation?

DCNNsmodeledafterprimatevisualcortex• EarlylayersmodelV1-V4,finallayersmodelITcortex• Forfaceidentification,finalDCNNlayerstores

abstractidentitycode<- facerepresentation

Trait-ProfilePredictions

IndividualTraitPredictions

PredictSocialTraitInferences

• Similaritybetweenhuman-generatedandcomputer-predictedtraitvectorsmeasuredusingcosinedistance

• AccuracyofindividualtraitpredictionsmeasuredusingR2 betweenhuman-generatedandcomputer-predictedvalues

• Errorbetweenhumanratingsandpredictedtraits,plottedagainstanulldistribution• Alltraitspredictedsignificantlyabovechance• Blueline:α=0.002• Redline:predictedvalue

• Cosinesimilaritybetweenhuman-generatedtraitprofilesandcomputerpredictions:

Goal2:Measureaccuracyoftraitpredictions

usingDCNNfeaturesfromnon-frontal

images

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Talkative"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Warm"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Shy"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Quiet"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Assertive"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Artistic"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Anxious"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Efficient"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Careless"

0

10

20

30

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Impulsive"

0

10

20

30

40

0.00 0.25 0.50 0.75 1.00Prediction Error

coun

t

"Lazy"

Trait-PredictionError

R2 BetweenHumanInferencesandComputerPredictions

• Differenttraitspredictedtodifferentextents

• Alltraitinferencespredictedabovechance

References[1]Bruce,V.,&Young,A.(1986).Understandingfacerecognition.Britishjournalofpsychology,77(3),305-327.[2]Bar,M.,Neta,M.,&Linz,H.(2006).Veryfirstimpressions.Emotion,6(2),269.[3]Todorov,A.,Mandisodza,A.N.,Goren,A.,&Hall,C.C.(2005).Inferencesofcompetencefromfacespredictelectionoutcomes.Science,308(5728),1623-1626.[4]Rule,N.O.Ambady,N.(2008).Thefaceofsuccess: Inferencesfromchiefexecutiveofficers'appearancepredictcompanyprofits.PsychologicalScience:AJournaloftheAmericanPsychologicalSociety/APS,19,109–111.[5]Oosterhof,N.N.,&Todorov,A.(2008).Thefunctionalbasisoffaceevaluation.ProceedingsoftheNationalAcademyofSciences,105(32),11087-11092.[6]Walker,M.,&Vetter,T.(2009).Portraitsmadetomeasure:Manipulatingsocialjudgmentsabout individualswithastatisticalfacemodel.JournalofVision,9(11),12-12.[7]Taigman,Y.,Yang,M.,Ranzato,M.A.,&Wolf,L.(2014).Deepface:Closingthegaptohuman-levelperformanceinfaceverification.InProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition (pp.1701-1708).[8]Parde,C.J.,Castillo,C.,Hill,M.Q.,Colon,Y.I.,Sankaranarayanan,S.,Chen,J.C.,&O’Toole,A.J.(2017,May).FaceandImageRepresentationinDeepCNNFeatures.InAutomaticFace&GestureRecognition(FG2017),201712thIEEEInternationalConferenceon (pp.673-680).IEEE.[9]Gosling,S.D.,Rentfrow,P.J.,&SwannJr,W.B.(2003).AverybriefmeasureoftheBig-Fivepersonalitydomains.JournalofResearchinpersonality,37(6),504-528.[10]Sankaranarayanan,S.,Alavi,A.,Castillo,C.D.,&Chellappa,R.(2016,September).Tripletprobabilisticembeddingforfaceverificationandclustering.InBiometricsTheory,ApplicationsandSystems(BTAS),2016IEEE8thInternationalConferenceon (pp.1-8).IEEE.

Conclusions

Humantraitinferencescanbepredictedfromthe

top-levelfeaturesofaDCNNtrainedforface

identification

Conclusion1

Traitinferencesassignedtofrontal

facescanbepredictedfromDCNNfeaturesgeneratedforbothfrontalandnon-frontalfaces

Conclusion2

Top-levelDCNNfeaturesforface

identificationretainrobusttrait

representation– eachindividualtraitpredictedabove

chance

Conclusion3

Nulldistribution:𝝰 =arccos(0.078)

UsingKfeatures:𝝰 =arccos(0.353)

UsingK– L features:𝝰 =arccos(0.533)

• DCNNrepresentationallowsforstate-of-the-artidentification• Notindependentofimageinformation,socialtraits

Analyzing Social Media Networks with NodeXL Derek Hansen, Cody Dunne, and Ben Shneiderman

Analyzing Affiliation Networks - BEBR Home affiliation...Analyzing Affiliation Networks Stephen P. Borgatti and Daniel S. Halgin . LINKS Center for Social Network Analysis . Gatton

Analyzing and Visualizing Social Networks

Analyzing the User Experience of Vertical Social Networks

Analyzing the User Experience of Vertical Social Networks: Nuzzel

Big Friend is Watching You: Analyzing Online Social ...conferences.sigcomm.org/sigcomm/2012/paper/wosn/p7.pdf · Big Friend is Watching You: Analyzing Online Social Networks Tracking

Modelling and Analyzing Multimodal Dyadic Interactions Using Social Networks

Analyzing social media networks with NodeXL - Chapter-07 Images

Analyzing TCPIP Networks with Wireshark

Generating and analyzing spatial social networks · Generating and analyzing spatial social networks Meysam Alizadeh1,2 • Claudio Cioffi-Revilla1,2 • Andrew Crooks1,2! Springer

Analyzing (Social Media) Networks with NodeXLben/papers/Smith2009Analyzing.pdf · Analyzing (Social Media) Networks with NodeXL Marc A. Smith1, Ben Shneiderman2, Natasa Milic-Frayling3,

Big Friend is Watching You: Analyzing Online Social Networks

Analyzing social media networks with NodeXL - Chapter-10 Images

Analyzing social media networks with NodeXL - Chapter- 08 images

Analyzing (Social Media) Networks with NodeXL - Pererperer.org/papers/adamPerer-NodeXL-CT2009.pdf · Analyzing (Social Media) Networks with NodeXL Marc A. Smith 1, Ben Shneiderman

Analyzing (Social Media) Networks with NodeXLben/papers/Smith2009Analyzing.pdf · 2015-10-05 · Analyzing (Social Media) Networks with NodeXL Marc A. Smith1, ... NetDraw that provide

SOCIAL NETWORKS AND ACADEMIC ACHIEVEMENT€¦ · Social networks and academic achievement: analyzing the peer -effect o n São Paulo Public School System / Renata Mayer Gukovas

Analyzing social media networks with NodeXL - Chapter-04 images

Analyzing (Social Media) Networks with NodeXL · Analyzing (Social Media) Networks with NodeXL Marc A. Smith 1, ... NetDraw that provide graphical interfaces, ... However, we see

[] analyzing spammers' social networks for fun and profit

Analyzing Social Media Networks with NodeXL Marc A. Smith

ANALYZING AND SECURING SOCIAL NETWORKS · ANALYZING AND SECURING SOCIAL NETWORKS Bhavani Thuraisingham • Satyen Abrol Raymond Heatherly • Murat Kantarcioglu Vaibhav Khadilkar

Analyzing social media networks with NodeXL - Chapter-13 Images

Analyzing Global Semantic Social Networks

MatLink: Enhanced Matrix Visualization for Analyzing Social Networks · 2018-01-04 · MatLink: Enhanced Matrix Visualization for Analyzing Social Networks Nathalie Henry 1,2 and

On analyzing user preference dynamics with temporal social ... · Online social networks, such as Facebook, Twitter, and music social networks, facilitate the building of social relations

Analyzing healthcare sequences in social networks

Analyzing social media networks with NodeXL - Chapter-03 images

Analyzing social media networks with NodeXL - Chapter- 12 images

Analyzing (Social Media) Networks with NodeXLeduarda/papers/ct09-smith.pdf · Analyzing (Social Media) Networks with NodeXL Marc A. Smith1, Ben Shneiderman2, Natasa Milic-Frayling3,

Analyzing the Terrorist Social Networks with Visualization Tools

Analyzing social media networks with NodeXL - Chapter-15 Images

Building and Analyzing Social Networks Insider Threat Analysis with Large Graphs

Generating and analyzing spatial social networkskrasnow.gmu.edu/.../Generating-and-analyzing-spatial-social-network… · Generating and analyzing spatial social networks ... and

Analyzing social media networks with NodeXL - Chapter-11 Images