Problem Confidence Preserving Machine (CPM)Facial Action Unit (AU) detection Diagram of CPM

Iterative CPM (iCPM)

Confidence Preserving Machine for Facial Action Unit DetectionJiabei Zeng1, Wen-Sheng Chu2, Fernando De la Torre2, Jeffrey F. Cohn2,3, and Zhang Xiong1

SANTIAGO, CHILE DECEMBER 11-18, 2015

AU12presence

AU12absence

?

ExperimentsSettingsSIFT descriptors on predetermined facial landmarks10-fold with disjoint training and test sets

Evaluate CPM componants

Comparisons

Standard supervised methods

trai

ning

sub

ject

ste

st s

ubje

ct

(a)

(b)

easy negative samples hard samples easy positive samples

test samplescon�dent classi�ers

QSS classi�er

Main idea

Training subjects

Test subject

classifiersconfident

(iterative CPM)easy test samples

PredictionTrain a QSS

classifier

Train confident classifiers

0 1 2 30

1

2

1 2 3 4 5

0

1

2

0 1 2 3 4 5-1

0

1

2

3

0 1 2 3 4 5

0

1

2

0 1 2 3 4 5

0

1

2

0 1 2 3 4 5-1

0

1

2

3

0 1 2 3 4 5-1

0

1

2

3

0 1 2 3 4 5

0

1

2

3

0 1 2 3 4-1

0

1

2

3

acc=0.97 acc=0.99 acc=1.00

(w− , w+) on the top of each other

w+w−w+

w−

wt wt

training data 1

training data 2

test data

upda

te e

asy

(har

d)

sam

ples

identify easy test samples identify easy test samples

add

easy

test

sam

ples

upda

te e

asy

(har

d)

sam

ples

retrain (w− , w+) retrain (w− , w+) train (w− , w+)

train wt train wt

iter#1 iter#2iter#0

Boosting × × × × × ×TW-SVM [6] × × × × ×Self-paced learning [7] × × × × × ×RO-SVM [8] × × × × × ×Co-training [9] × × × ×Lap-SVM [2] × × × × ×DAM [3] × ×CPMs

Multiple classifiers

Indentify easy/hard

Unlabeleddata

Different distribution

Smoothnessassumption

Non-parallel hyperplane

Progressivelabeling

Confident classifiers(w+, w-), or wy to confidently predict on positive and neg-ative samples, respectively

×

Previous work

Alternating algorithm for training confident classifiers

Single hyperplanePerform well on easy samples, e.g., high intensity AUs, frontal head pose or on particular subjectsPerform badly on hard samples, e.g. , subtle AUs

Easy-to-hard strategy

Address specific and known sources- E.g. subtle AUs, head pose, individual differences [1]Semi-supervised learning methods (SSL) [2]- Smoothness/Cluster/Manifold assumptions on rela-tionships between input and label space- Not suitable for AUTransfer learning methods [3,4,5]- Only individual differences- Absence of spatial-temporal smoothness- Less efficient than CPM

Quasi-semi-supervised (QSS) learningGeneralization on unseen test subjects

−1

1

−1

1

−1

1

100 200 300 400 frame 100 200 300 400 frame 100 200 300 400 frame

truthw/o Sw/ S

A person-specific classifier wt for a test subjectQuasi-semi-supervised (QSS) classifier

Objective (4)

Effectiveness of spatial-temporal smoothnessIdentify easy and hard samples (1)

easy: (w+, w-) have same predictionshard: (w+, w-) have different predictions

Objective (2)

Relabeling strategiesHolistic relabeling: Relabel all the hard samples

+-

hard datanegativepositive

noise instance

Localized relabeling: Relabel part of the hard samples

Comparison with alternative methods

(w+, w-) might not generalize well due to mismatch between training and test.iCPM jointly learns (w+, w-) and QSS.

iCPM on a synthetic example

DatasetsGFT [10]: 50 2-min spontaneous videos from 50 participants BP4D [11]: 328 spontaneous videos from 41 participants DISFA [12]: 27 spontaneous videos from 27 participants Metric

F1-fr

ame

454749515355

GFT

F1-e

vent

36

39

424548

4546

47

4849

SVM Conf CPM iCPM

DISFA2630

34

3842

BP4D4244

464850

57

56

55

Frame-based F1

Event-based F1

baseline methods (SVM, Adaboost), SSL (Laplacian SVM [2]), transfer learning methods (DAM [3], MDA [4], GFK [5])

[1] W.-S. Chu, et al. Selective transfer machine for personalized facial action unit detection. CVPR 2013.[2] S. Melacci, et al. Laplacian support vector machines trained in the primal, JMLR, 12:1149–1184, 2011.[3] Duan L, et al. Domain adaptation from multiple sources: A domain-dependent regularization approach, TNNLS, 23(3): 504-518, 2012.[4] B. Gong, et al. Geodesic flow kernel for unsupervised domain adaptation, CVPR 2012. [5] Q. Sun, et al. A two-stage weighting framework for multi-source domain adaptation, NIPS 2011.

[6] R. Khemchandani, et al. Twin support vector machines for pattern classification, TPAMI, 29:905-910, 2007.[7] M. P. Kumar, et al. Self-paced learning for latent variable models, NIPS 2010.[8] Y. Grandvalet, et al. Support vector machines with a reject option, NIPS 2009.[9] A. Blum, et al.Combining labeled and unlabeled data with co-training, CoLT 1998.

[10] J. F. Cohn, et al. Spontaneous facial expression in a small group can be automatically measur- ed: An initial demonstration, Behavior Research Methods, 42(4):1079–1086, 2010.[11] X. Zhang, et al. A high-resolution spontaneous 3d dynamic facial expression database, FG 2013.[12] S. Mavadati, et al. Disfa:A spontaneous facial action intensity database,TAC,4:151–160, 2013.

1 2 33

Easy samples

Hard samples

Data A Data B Holistic on A Holistic on B Localized on B

To get the person-specific classifier wt

Virtual labels for easy sampels from (w+, w-) Propagate virtual labels from easy to hard test samples under semi-supervided manner.

loss spatial-temporalsmoothness

valu

e of

index of ji i+1i-1i-2i-3 i+2 i+3i+4 i+5i-4i-5

spatialdisimilarity

excludepreservetemporalsimilarity

truth labels

relabel

MH096951,Supported in part by 61272350, 2013AA01A603