Face Recognition Using Face Patch Networks Chaochao Lu , Deli Zhao, Xiaoou Tang
Department of Information Engineering, The Chinese University of Hong Kong
1. Main Idea 2. Approach Overview 4. Comparison to Existing Measures
3. Approach Details 5. Tuning Parameters
6. Result on LFW Benchmark 7. Conclusion
𝒅𝑬
𝒅𝑹𝑷
𝒅𝑨𝑨′𝑬 > 𝒅𝑨𝑩
𝑬
𝒅𝑺𝑷
𝒅𝑨𝑨′𝑺𝑷 > 𝒅𝑨𝑩
𝑺𝑷
𝒅𝑨𝑨′𝑹𝑷 < 𝒅𝑨𝑩
𝑹𝑷
𝜱𝑨 = −𝟗𝟎∘
𝜱𝑨 = −𝟒𝟓∘
𝜱𝑨 = 𝟎∘ 𝜱𝑨 = +𝟒𝟓∘
𝜱𝑨 = +𝟗𝟎∘
𝜱𝑩 = +𝟗𝟎∘
𝜱𝑩 = +𝟒𝟓∘ 𝜱𝑩 = 𝟎∘
𝜱𝑩 = −𝟒𝟓∘
𝜱𝑩 = −𝟗𝟎∘
A B
A′
Motivation
Proposed Idea
1 2 3 4 50.7
0.75
0.8
0.85
0.9
10 20 30 40 500.8
0.82
0.84
0.86
0.88
30 40 50 60 700.844
0.846
0.848
0.85
0.852
0 0.5 1
0.7
0.8
0.9
Kin
recognition rate
recognition rate
recognition rate
recognition rate
rNN Kout α
(rNN=50, Kout=50, α=0.5) (Kin=4, Kout=50, α=0.5) (rNN=30, Kin=4, α=0.5) (Kin=4, Kout=40, rNN=30)
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.40.5
0.6
0.7
0.8
0.9
1
Associate-Predict (90.57%)
Combined multishot (89.50%)
Ours (91.26%)
Single LE+holistic (81.22%)
Multiple LE+comp (84.45%)
combined PLDA (90.07%)
false positive rate
true positive rate
(a)
⋮
⋮
𝚿
𝓕𝟏
𝓕𝒕
𝓕𝑻
f111
f𝑖𝑗1
f𝑀1
f11𝑡
f𝑀𝑡
f11𝑇
f𝑖𝑗𝑇
f111 f11
𝑡 f11𝑇
f𝑖−1,𝑗−11 f𝑖−1,𝑗−1
𝑡 f𝑖−1,𝑗−1𝑇
f𝑖𝑗1 f𝑖𝑗
𝑡 f𝑖𝑗𝑇
f𝑖+1,𝑗+11 f𝑖+1,𝑗+1
𝑡 f𝑖+1,𝑗+1𝑇
f𝑀1 f𝑀
𝑡 f𝑀𝑇
𝑮𝟏𝟏𝒈𝒍𝒐𝒃𝒂𝒍
𝑮𝒊−𝟏,𝒋−𝟏𝒈𝒍𝒐𝒃𝒂𝒍
𝑮𝒊𝒋𝒈𝒍𝒐𝒃𝒂𝒍
𝑮𝒊+𝟏,𝒋+𝟏𝒈𝒍𝒐𝒃𝒂𝒍
𝑮𝑴𝒈𝒍𝒐𝒃𝒂𝒍
f𝑖𝑗𝑡
⋮
⋮
⋮ ⋮
⋮ ⋮
f𝑖𝑗𝑎
f𝑖𝑗𝑏
𝑮𝒂
𝑮𝒃
𝑮𝒂 ⋃𝑮𝒃 𝑠𝑜𝑢𝑡
(b) (d) (e) (f) Construct the Global Network Calculate the similarity
⋯ ⋯
⋯ ⋯
⋯ ⋯
⋯ ⋯
⋯ ⋯
f111 f11
𝑡 f11𝑇
f𝑖−1,𝑗−11 f𝑖−1,𝑗−1
𝑡 f𝑖−1,𝑗−1𝑇
f𝑖𝑗1 f𝑖𝑗
𝑡 f𝑖𝑗𝑇
f𝑀1 f𝑀
𝑡 f𝑀𝑇
⋮
⋮
⋮ ⋮
⋮ ⋮
𝑮𝒈𝒍𝒐𝒃𝒂𝒍 (c)
f𝑖+1,𝑗+11 f𝑖+1,𝑗+1
𝑡 f𝑖+1,𝑗+1𝑇
⋯ ⋯
⋯ ⋯
⋯ ⋯
⋯ ⋯
⋯ ⋯
Out-face Network
In-face Network
𝑆1𝑖𝑛
⋮ ⋮ ⋮
Face a
Obtain Patch Correspondence and Their
Overlapping Neighborhood
Construct KNN Graph based on Patch Features
(Showing Appearance)
Compute Patch Similarities using the
Random Patch Measure
𝑆𝑀𝑖𝑛
Face b
Patch p
Patch q
(a)
𝐺𝑝𝑎 ⋃ 𝐺𝑞
𝑏
(b) (c) (d)
In-face Network Out-face Network Adjacency Matrix
Fusion Method 𝒔𝑓𝑖𝑛𝑎𝑙 = 𝛼𝒔𝑖𝑛, (1 − 𝛼)𝑠𝑜𝑢𝑡
𝐏 𝑖, 𝑗 = exp(−𝑑𝑖𝑠𝑡 x𝑖 , x𝑗
2
𝜎2, if x𝑗 ∈ 𝒩𝑖
𝐾
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
where 𝑑𝑖𝑠𝑡(x𝑖 , x𝑗) is the pairwise distance
between x𝑖 and x𝑗, 𝒩𝑖𝐾 is the set of KNNs of
x𝑖, and 𝜎2 =1
𝑛𝐾[ 𝑑𝑖𝑠𝑡(x𝑖 , x𝑗)
2x𝑗∈𝒩𝑖
𝐾𝑛𝑖=1 ].
To get the transition probability matrix, we perform 𝐏(i, j) ← 𝐏(i, j)/ 𝐏(i, j)𝒏
𝒋=𝟏 .
C𝐆𝑖𝑗𝑎 =
1
|𝐆𝑖𝑗𝑎 |𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑖𝑗
𝑎 )−1𝟏,
C𝐆𝑖𝑗𝑏 =
1
|𝐆𝑖𝑗𝑏 |𝟏𝑇(𝐈 − 𝑧𝐏
𝐆𝑖𝑗𝑏 )−1𝟏,
C𝐆𝑖𝑗𝑎∪𝐆𝑖𝑗
𝑏 = C𝐺𝑖𝑗 =1
|𝐺𝑖𝑗|𝟏𝑇(𝐈 − 𝑧𝐏𝐺𝑖𝑗)
−1𝟏,
𝑆𝑖𝑗𝑖𝑛 = Φ
𝐆𝑖𝑗𝑎∪𝐆𝑖𝑗
𝑏 = C𝐆𝑖𝑗𝑎∪𝐆𝑖𝑗
𝑏 − C𝐆𝑖𝑗𝑎 + C
𝐆𝑖𝑗𝑏 ,
𝒔𝑖𝑛 = 𝑆11𝑖𝑛, ⋯ , 𝑆𝑖𝑗
𝑖𝑛, ⋯ , 𝑆𝑀𝑖𝑛 .
C𝐆𝑎 =1
|𝐆𝑎|𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑎)
−1𝟏,
C𝐆𝑏 =1
|𝐆𝑏|𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑏)
−1𝟏,
C𝐆𝑎∪𝐆𝑏 =1
|𝐆𝑎 ∪ 𝐆𝑏|𝟏𝑇(𝐈 − 𝑧𝐏𝐆𝑎∪𝐆𝑏)
−1𝟏,
𝑠𝑜𝑢𝑡 = Φ𝐆𝑎∪𝐆𝑏 = C𝐆𝑎∪𝐆𝑏 − C𝐆𝑎 + C𝐆𝑏 .
Path Centrality: 𝐶𝐺 =1
𝑁𝟏𝑇(𝐈 − 𝑧𝐏)−1𝟏, where z <
1
ρ 𝐏 and ρ 𝐏 is the spectral radius of 𝐏.
Random Path Measure: Φ𝑮𝑖 ∪𝑮𝑗 = C𝑮𝑖 ∪𝑮𝑗 − (C𝑮𝑖 + C𝑮𝑗), is regarded as the similarity between
two networks 𝑮𝑖 and 𝑮𝑗.
Multi-PIE: this dataset contains face images from 337 subjects under 15 view points and 19 illumination conditions in four recording sessions.
LFW: this dataset contains 13,233 uncontrolled face images of 5,749 public figures of different ethnicity, gender, age, etc.
Random Path (RP) Measure Two types of networks on face data: the in-face
network and the out-face network Extensive experiments on the Multi-PIE and LFW
face databases validate our approach.
Illustration of the superiority of our random path (RP) measure over other measures (for example, Euclidean (E) measure and the shortest path (SP) measure): Due to the large intra-personal variations (e.g., pose, illumination, and expression), there may be underlying structures in face space (denoted by the red and blue clusters). For three face images A, B, and A’ of two
different persons, the distances are 𝑑𝐴𝐴′𝐸 > 𝑑𝐴𝐵
𝐸 and
𝑑𝐴𝐴′𝑆𝑃 > 𝑑𝐴𝐵
𝑆𝑃 if measured by Euclidean measure (solid
green line) and the shortest path measure (solid yellow line). In other words, A is more similar to B than to A’. Incorrect decisions are usually made because the intra-personal variation is much larger than the inter-personal variation. If we consider their underlying structures and compute their similarity by our random
path measure (dashed yellow line), we get 𝑑𝐴𝐴′𝑅𝑃 < 𝑑𝐴𝐵
𝑅𝑃.
The correct decision can be made.
Random Path (RP) Measure: it includes all paths of different lengths in the network, which enables it to capture more discriminative information in faces and significantly reduce the effect of noise and outliers.
In-face network: it captures the local information of face images. Out-face network: it captures the global information of face images.
f𝑀𝑇