Face Recognition Using Face Patch Networks Chaochao Lu , Deli Zhao, Xiaoou Tang

Department of Information Engineering, The Chinese University of Hong Kong

1. Main Idea 2. Approach Overview 4. Comparison to Existing Measures

3. Approach Details 5. Tuning Parameters

6. Result on LFW Benchmark 7. Conclusion

𝒅𝑬

𝒅𝑹𝑷

𝒅𝑨𝑨′𝑬 > 𝒅𝑨𝑩

𝑬

𝒅𝑺𝑷

𝒅𝑨𝑨′𝑺𝑷 > 𝒅𝑨𝑩

𝑺𝑷

𝒅𝑨𝑨′𝑹𝑷 < 𝒅𝑨𝑩

𝑹𝑷

𝜱𝑨 = −𝟗𝟎∘

𝜱𝑨 = −𝟒𝟓∘

𝜱𝑨 = 𝟎∘ 𝜱𝑨 = +𝟒𝟓∘

𝜱𝑨 = +𝟗𝟎∘

𝜱𝑩 = +𝟗𝟎∘

𝜱𝑩 = +𝟒𝟓∘ 𝜱𝑩 = 𝟎∘

𝜱𝑩 = −𝟒𝟓∘

𝜱𝑩 = −𝟗𝟎∘

A B

A′

Motivation

Proposed Idea

1 2 3 4 50.7

0.75

0.8

0.85

0.9

10 20 30 40 500.8

0.82

0.84

0.86

0.88

30 40 50 60 700.844

0.846

0.848

0.85

0.852

0 0.5 1

0.7

0.8

0.9

Kin

recognition rate

recognition rate

recognition rate

recognition rate

rNN Kout α

(rNN=50, Kout=50, α=0.5) (Kin=4, Kout=50, α=0.5) (rNN=30, Kin=4, α=0.5) (Kin=4, Kout=40, rNN=30)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.40.5

0.6

0.7

0.8

0.9

1

Associate-Predict (90.57%)

Combined multishot (89.50%)

Ours (91.26%)

Single LE+holistic (81.22%)

Multiple LE+comp (84.45%)

combined PLDA (90.07%)

false positive rate

true positive rate

(a)

⋮

⋮

𝚿

𝓕𝟏

𝓕𝒕

𝓕𝑻

f111

f𝑖𝑗1

f𝑀1

f11𝑡

f𝑀𝑡

f11𝑇

f𝑖𝑗𝑇

f111 f11

𝑡 f11𝑇

f𝑖−1,𝑗−11 f𝑖−1,𝑗−1

𝑡 f𝑖−1,𝑗−1𝑇

f𝑖𝑗1 f𝑖𝑗

𝑡 f𝑖𝑗𝑇

f𝑖+1,𝑗+11 f𝑖+1,𝑗+1

𝑡 f𝑖+1,𝑗+1𝑇

f𝑀1 f𝑀

𝑡 f𝑀𝑇

𝑮𝟏𝟏𝒈𝒍𝒐𝒃𝒂𝒍

𝑮𝒊−𝟏,𝒋−𝟏𝒈𝒍𝒐𝒃𝒂𝒍

𝑮𝒊𝒋𝒈𝒍𝒐𝒃𝒂𝒍

𝑮𝒊+𝟏,𝒋+𝟏𝒈𝒍𝒐𝒃𝒂𝒍

𝑮𝑴𝒈𝒍𝒐𝒃𝒂𝒍

f𝑖𝑗𝑡

⋮

⋮

⋮ ⋮

⋮ ⋮

f𝑖𝑗𝑎

f𝑖𝑗𝑏

𝑮𝒂

𝑮𝒃

𝑮𝒂 ⋃𝑮𝒃 𝑠𝑜𝑢𝑡

(b) (d) (e) (f) Construct the Global Network Calculate the similarity

⋯ ⋯

⋯ ⋯

⋯ ⋯

⋯ ⋯

⋯ ⋯

f111 f11

𝑡 f11𝑇

f𝑖−1,𝑗−11 f𝑖−1,𝑗−1

𝑡 f𝑖−1,𝑗−1𝑇

f𝑖𝑗1 f𝑖𝑗

𝑡 f𝑖𝑗𝑇

f𝑀1 f𝑀

𝑡 f𝑀𝑇

⋮

⋮

⋮ ⋮

⋮ ⋮

𝑮𝒈𝒍𝒐𝒃𝒂𝒍 (c)

f𝑖+1,𝑗+11 f𝑖+1,𝑗+1

𝑡 f𝑖+1,𝑗+1𝑇

⋯ ⋯

⋯ ⋯

⋯ ⋯

⋯ ⋯

⋯ ⋯

Out-face Network

In-face Network

𝑆1𝑖𝑛

⋮ ⋮ ⋮

Face a

Obtain Patch Correspondence and Their

Overlapping Neighborhood

Construct KNN Graph based on Patch Features

(Showing Appearance)

Compute Patch Similarities using the

Random Patch Measure

𝑆𝑀𝑖𝑛

Face b

Patch p

Patch q

(a)

𝐺𝑝𝑎 ⋃ 𝐺𝑞

𝑏

(b) (c) (d)

In-face Network Out-face Network Adjacency Matrix

Fusion Method 𝒔𝑓𝑖𝑛𝑎𝑙 = 𝛼𝒔𝑖𝑛, (1 − 𝛼)𝑠𝑜𝑢𝑡

𝐏 𝑖, 𝑗 = exp(−𝑑𝑖𝑠𝑡 x𝑖 , x𝑗

2

𝜎2, if x𝑗 ∈ 𝒩𝑖

𝐾

0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

where 𝑑𝑖𝑠𝑡(x𝑖 , x𝑗) is the pairwise distance

between x𝑖 and x𝑗, 𝒩𝑖𝐾 is the set of KNNs of

x𝑖, and 𝜎2 =1

𝑛𝐾[ 𝑑𝑖𝑠𝑡(x𝑖 , x𝑗)

2x𝑗∈𝒩𝑖

𝐾𝑛𝑖=1 ].

To get the transition probability matrix, we perform 𝐏(i, j) ← 𝐏(i, j)/ 𝐏(i, j)𝒏

𝒋=𝟏 .

C𝐆𝑖𝑗𝑎 =

1

|𝐆𝑖𝑗𝑎 |𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑖𝑗

𝑎 )−1𝟏,

C𝐆𝑖𝑗𝑏 =

1

|𝐆𝑖𝑗𝑏 |𝟏𝑇(𝐈 − 𝑧𝐏

𝐆𝑖𝑗𝑏 )−1𝟏,

C𝐆𝑖𝑗𝑎∪𝐆𝑖𝑗

𝑏 = C𝐺𝑖𝑗 =1

|𝐺𝑖𝑗|𝟏𝑇(𝐈 − 𝑧𝐏𝐺𝑖𝑗)

−1𝟏,

𝑆𝑖𝑗𝑖𝑛 = Φ

𝐆𝑖𝑗𝑎∪𝐆𝑖𝑗

𝑏 = C𝐆𝑖𝑗𝑎∪𝐆𝑖𝑗

𝑏 − C𝐆𝑖𝑗𝑎 + C

𝐆𝑖𝑗𝑏 ,

𝒔𝑖𝑛 = 𝑆11𝑖𝑛, ⋯ , 𝑆𝑖𝑗

𝑖𝑛, ⋯ , 𝑆𝑀𝑖𝑛 .

C𝐆𝑎 =1

|𝐆𝑎|𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑎)

−1𝟏,

C𝐆𝑏 =1

|𝐆𝑏|𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑏)

−1𝟏,

C𝐆𝑎∪𝐆𝑏 =1

|𝐆𝑎 ∪ 𝐆𝑏|𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑎∪𝐆𝑏)

−1𝟏,

𝑠𝑜𝑢𝑡 = Φ𝐆𝑎∪𝐆𝑏 = C𝐆𝑎∪𝐆𝑏 − C𝐆𝑎 + C𝐆𝑏 .

Path Centrality: 𝐶𝐺 =1

𝑁𝟏𝑇(𝐈 − 𝑧𝐏)−1𝟏, where z <

1

ρ 𝐏 and ρ 𝐏 is the spectral radius of 𝐏.

Random Path Measure: Φ𝑮𝑖 ∪𝑮𝑗 = C𝑮𝑖 ∪𝑮𝑗 − (C𝑮𝑖 + C𝑮𝑗), is regarded as the similarity between

two networks 𝑮𝑖 and 𝑮𝑗.

Multi-PIE: this dataset contains face images from 337 subjects under 15 view points and 19 illumination conditions in four recording sessions.

LFW: this dataset contains 13,233 uncontrolled face images of 5,749 public figures of different ethnicity, gender, age, etc.

Random Path (RP) Measure Two types of networks on face data: the in-face

network and the out-face network Extensive experiments on the Multi-PIE and LFW

face databases validate our approach.

Illustration of the superiority of our random path (RP) measure over other measures (for example, Euclidean (E) measure and the shortest path (SP) measure): Due to the large intra-personal variations (e.g., pose, illumination, and expression), there may be underlying structures in face space (denoted by the red and blue clusters). For three face images A, B, and A’ of two

different persons, the distances are 𝑑𝐴𝐴′𝐸 > 𝑑𝐴𝐵

𝐸 and

𝑑𝐴𝐴′𝑆𝑃 > 𝑑𝐴𝐵

𝑆𝑃 if measured by Euclidean measure (solid

green line) and the shortest path measure (solid yellow line). In other words, A is more similar to B than to A’. Incorrect decisions are usually made because the intra-personal variation is much larger than the inter-personal variation. If we consider their underlying structures and compute their similarity by our random

path measure (dashed yellow line), we get 𝑑𝐴𝐴′𝑅𝑃 < 𝑑𝐴𝐵

𝑅𝑃.

The correct decision can be made.

Random Path (RP) Measure: it includes all paths of different lengths in the network, which enables it to capture more discriminative information in faces and significantly reduce the effect of noise and outliers.

In-face network: it captures the local information of face images. Out-face network: it captures the global information of face images.

f𝑀𝑇