face recognition using face patch networks · pdf file face recognition using face patch...

Click here to load reader

Post on 08-Oct-2020

0 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Face Recognition Using Face Patch Networks Chaochao Lu , Deli Zhao, Xiaoou Tang

    Department of Information Engineering, The Chinese University of Hong Kong

    1. Main Idea 2. Approach Overview 4. Comparison to Existing Measures

    3. Approach Details 5. Tuning Parameters

    6. Result on LFW Benchmark 7. Conclusion

    ๐’…๐‘ฌ

    ๐’…๐‘น๐‘ท

    ๐’…๐‘จ๐‘จโ€ฒ ๐‘ฌ > ๐’…๐‘จ๐‘ฉ

    ๐‘ฌ

    ๐’…๐‘บ๐‘ท

    ๐’…๐‘จ๐‘จโ€ฒ ๐‘บ๐‘ท > ๐’…๐‘จ๐‘ฉ

    ๐‘บ๐‘ท

    ๐’…๐‘จ๐‘จโ€ฒ ๐‘น๐‘ท < ๐’…๐‘จ๐‘ฉ

    ๐‘น๐‘ท

    ๐œฑ๐‘จ = โˆ’๐Ÿ—๐ŸŽ โˆ˜

    ๐œฑ๐‘จ = โˆ’๐Ÿ’๐Ÿ“ โˆ˜

    ๐œฑ๐‘จ = ๐ŸŽ โˆ˜ ๐œฑ๐‘จ = +๐Ÿ’๐Ÿ“

    โˆ˜

    ๐œฑ๐‘จ = +๐Ÿ—๐ŸŽ โˆ˜

    ๐œฑ๐‘ฉ = +๐Ÿ—๐ŸŽ โˆ˜

    ๐œฑ๐‘ฉ = +๐Ÿ’๐Ÿ“ โˆ˜ ๐œฑ๐‘ฉ = ๐ŸŽ

    โˆ˜

    ๐œฑ๐‘ฉ = โˆ’๐Ÿ’๐Ÿ“ โˆ˜

    ๐œฑ๐‘ฉ = โˆ’๐Ÿ—๐ŸŽ โˆ˜

    A B

    Aโ€ฒ

    Motivation

    Proposed Idea

    1 2 3 4 5 0.7

    0.75

    0.8

    0.85

    0.9

    10 20 30 40 50 0.8

    0.82

    0.84

    0.86

    0.88

    30 40 50 60 70 0.844

    0.846

    0.848

    0.85

    0.852

    0 0.5 1

    0.7

    0.8

    0.9

    Kin

    re co gn it io n r at e

    re co gn it io n r at e

    re co gn it io n r at e

    re co gn it io n r at e

    rNN Kout ฮฑ

    (rNN=50, Kout=50, ฮฑ=0.5) (Kin=4, Kout=50, ฮฑ=0.5) (rNN=30, Kin=4, ฮฑ=0.5) (Kin=4, Kout=40, rNN=30)

    0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.5

    0.6

    0.7

    0.8

    0.9

    1

    Associate-Predict (90.57%)

    Combined multishot (89.50%)

    Ours (91.26%)

    Single LE+holistic (81.22%)

    Multiple LE+comp (84.45%)

    combined PLDA (90.07%)

    false positive rate

    tr u e p o si ti v e ra te

    (a)

    โ‹ฎ

    โ‹ฎ

    ๐šฟ

    ๐“•๐Ÿ

    ๐“•๐’•

    ๐“•๐‘ป

    f11 1

    f๐‘–๐‘— 1

    f๐‘€ 1

    f11 ๐‘ก

    f๐‘€ ๐‘ก

    f11 ๐‘‡

    f๐‘–๐‘— ๐‘‡

    f11 1 f11

    ๐‘ก f11 ๐‘‡

    f๐‘–โˆ’1,๐‘—โˆ’1 1 f๐‘–โˆ’1,๐‘—โˆ’1

    ๐‘ก f๐‘–โˆ’1,๐‘—โˆ’1 ๐‘‡

    f๐‘–๐‘— 1 f๐‘–๐‘—

    ๐‘ก f๐‘–๐‘— ๐‘‡

    f๐‘–+1,๐‘—+1 1 f๐‘–+1,๐‘—+1

    ๐‘ก f๐‘–+1,๐‘—+1 ๐‘‡

    f๐‘€ 1 f๐‘€

    ๐‘ก f๐‘€ ๐‘‡

    ๐‘ฎ๐Ÿ๐Ÿ ๐’ˆ๐’๐’๐’ƒ๐’‚๐’

    ๐‘ฎ๐’Šโˆ’๐Ÿ,๐’‹โˆ’๐Ÿ ๐’ˆ๐’๐’๐’ƒ๐’‚๐’

    ๐‘ฎ๐’Š๐’‹ ๐’ˆ๐’๐’๐’ƒ๐’‚๐’

    ๐‘ฎ๐’Š+๐Ÿ,๐’‹+๐Ÿ ๐’ˆ๐’๐’๐’ƒ๐’‚๐’

    ๐‘ฎ๐‘ด ๐’ˆ๐’๐’๐’ƒ๐’‚๐’

    f๐‘–๐‘— ๐‘ก

    โ‹ฎ

    โ‹ฎ

    โ‹ฎ โ‹ฎ

    โ‹ฎ โ‹ฎ

    f๐‘–๐‘— ๐‘Ž

    f๐‘–๐‘— ๐‘

    ๐‘ฎ๐’‚

    ๐‘ฎ๐’ƒ

    ๐‘ฎ๐’‚ โ‹ƒ๐‘ฎ๐’ƒ ๐‘ ๐‘œ๐‘ข๐‘ก

    (b) (d) (e) (f) Construct the Global Network Calculate the similarity

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    f11 1 f11

    ๐‘ก f11 ๐‘‡

    f๐‘–โˆ’1,๐‘—โˆ’1 1 f๐‘–โˆ’1,๐‘—โˆ’1

    ๐‘ก f๐‘–โˆ’1,๐‘—โˆ’1 ๐‘‡

    f๐‘–๐‘— 1 f๐‘–๐‘—

    ๐‘ก f๐‘–๐‘— ๐‘‡

    f๐‘€ 1 f๐‘€

    ๐‘ก f๐‘€ ๐‘‡

    โ‹ฎ

    โ‹ฎ

    โ‹ฎ โ‹ฎ

    โ‹ฎ โ‹ฎ

    ๐‘ฎ๐’ˆ๐’๐’๐’ƒ๐’‚๐’ (c)

    f๐‘–+1,๐‘—+1 1 f๐‘–+1,๐‘—+1

    ๐‘ก f๐‘–+1,๐‘—+1 ๐‘‡

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    โ‹ฏ โ‹ฏ

    Out-face Network

    In-face Network

    ๐‘†1 ๐‘–๐‘›

    โ‹ฎ โ‹ฎ โ‹ฎ

    Face a

    Obtain Patch Correspondence and Their

    Overlapping Neighborhood

    Construct KNN Graph based on Patch Features

    (Showing Appearance)

    Compute Patch Similarities using the

    Random Patch Measure

    ๐‘†๐‘€ ๐‘–๐‘›

    Face b

    Patch p

    Patch q

    (a)

    ๐บ๐‘ ๐‘Ž โ‹ƒ ๐บ๐‘ž

    ๐‘

    (b) (c) (d)

    In-face Network Out-face Network Adjacency Matrix

    Fusion Method ๐’”๐‘“๐‘–๐‘›๐‘Ž๐‘™ = ๐›ผ๐’”๐‘–๐‘›, (1 โˆ’ ๐›ผ)๐‘ ๐‘œ๐‘ข๐‘ก

    ๐ ๐‘–, ๐‘— = exp(โˆ’ ๐‘‘๐‘–๐‘ ๐‘ก x๐‘– , x๐‘—

    2

    ๐œŽ2 , if x๐‘— โˆˆ ๐’ฉ๐‘–

    ๐พ

    0, ๐‘œ๐‘กโ„Ž๐‘’๐‘Ÿ๐‘ค๐‘–๐‘ ๐‘’

    where ๐‘‘๐‘–๐‘ ๐‘ก(x๐‘– , x๐‘—) is the pairwise distance

    between x๐‘– and x๐‘—, ๐’ฉ๐‘– ๐พ is the set of KNNs of

    x๐‘–, and ๐œŽ 2 =

    1

    ๐‘›๐พ [ ๐‘‘๐‘–๐‘ ๐‘ก(x๐‘– , x๐‘—)

    2 x๐‘—โˆˆ๐’ฉ๐‘–

    ๐พ ๐‘› ๐‘–=1 ].

    To get the transition probability matrix, we perform ๐(i, j) โ† ๐(i, j)/ ๐(i, j)๐’๐’‹=๐Ÿ .

    C๐†๐‘–๐‘— ๐‘Ž =

    1

    |๐†๐‘–๐‘— ๐‘Ž | ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐๐†๐‘–๐‘—

    ๐‘Ž )โˆ’1๐Ÿ,

    C ๐†๐‘–๐‘— ๐‘ =

    1

    |๐†๐‘–๐‘— ๐‘ | ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐

    ๐†๐‘–๐‘— ๐‘ )โˆ’1๐Ÿ,

    C ๐†๐‘–๐‘— ๐‘Žโˆช๐†๐‘–๐‘—

    ๐‘ = C๐บ๐‘–๐‘— = 1

    |๐บ๐‘–๐‘—| ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐๐บ๐‘–๐‘—)

    โˆ’1๐Ÿ,

    ๐‘†๐‘–๐‘— ๐‘–๐‘› = ฮฆ

    ๐†๐‘–๐‘— ๐‘Žโˆช๐†๐‘–๐‘—

    ๐‘ = C๐†๐‘–๐‘— ๐‘Žโˆช๐†๐‘–๐‘—

    ๐‘ โˆ’ C๐†๐‘–๐‘— ๐‘Ž + C๐†๐‘–๐‘—

    ๐‘ ,

    ๐’”๐‘–๐‘› = ๐‘†11 ๐‘–๐‘›, โ‹ฏ , ๐‘†๐‘–๐‘—

    ๐‘–๐‘›, โ‹ฏ , ๐‘†๐‘€ ๐‘–๐‘› .

    C๐†๐‘Ž = 1

    |๐†๐‘Ž| ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐๐†๐‘Ž)

    โˆ’1๐Ÿ,

    C๐†๐‘ = 1

    |๐†๐‘| ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐๐†๐‘)

    โˆ’1๐Ÿ,

    C๐†๐‘Žโˆช๐†๐‘ = 1

    |๐†๐‘Ž โˆช ๐†๐‘| ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐๐†๐‘Žโˆช๐†๐‘)

    โˆ’1๐Ÿ,

    ๐‘ ๐‘œ๐‘ข๐‘ก = ฮฆ๐†๐‘Žโˆช๐†๐‘ = C๐†๐‘Žโˆช๐†๐‘ โˆ’ C๐†๐‘Ž + C๐†๐‘ .

    Path Centrality: ๐ถ๐บ = 1

    ๐‘ ๐Ÿ๐‘‡(๐ˆ โˆ’ ๐‘ง๐)โˆ’1๐Ÿ, where z <

    1

    ฯ ๐ and ฯ ๐ is the spectral radius of ๐.

    Random Path Measure: ฮฆ๐‘ฎ๐‘– โˆช๐‘ฎ๐‘— = C๐‘ฎ๐‘– โˆช๐‘ฎ๐‘— โˆ’ (C๐‘ฎ๐‘– + C๐‘ฎ๐‘—), is regarded as the similarity between two networks ๐‘ฎ๐‘– and ๐‘ฎ๐‘—.

    Multi-PIE: this dataset contains face images from 337 subjects under 15 view points and 19 illumination conditions in four recording sessions.

    LFW: this dataset contains 13,233 uncontrolled face images of 5,749 public figures of different ethnicity, gender, age, etc.

    ๏ƒ˜ Random Path (RP) Measure ๏ƒ˜ Two types of networks on face data: the in-face

    network and the out-face network ๏ƒ˜ Extensive experiments on the Multi-PIE and LFW

    face databases validate our approach.

    Illustration of the superiority of our random path (RP) measure over other measures (for example, Euclidean (E) measure and the shortest path (SP) measure): Due to the large intra-personal variations (e.g., pose, illumination, and expression), there may be underlying structures in face space (denoted by the red and blue clusters). For three face images A, B, and Aโ€™ of two

    different persons, the distances are ๐‘‘๐ด๐ดโ€ฒ ๐ธ > ๐‘‘๐ด๐ต

    ๐ธ and

    ๐‘‘๐ด๐ดโ€ฒ ๐‘†๐‘ƒ > ๐‘‘๐ด๐ต

    ๐‘†๐‘ƒ if measured by Euclidean measure (solid

    green line) and the shortest path measure (solid yellow line). In other words, A is more similar to B than to Aโ€™. Incorrect decisions are usually made because the intra- personal variation is much larger than the inter- personal variation. If we consider their underlying structures and compute their similarity by our random

    path measure (dashed yellow line), we get ๐‘‘๐ด๐ดโ€ฒ ๐‘…๐‘ƒ < ๐‘‘๐ด๐ต

    ๐‘…๐‘ƒ.

    The correct decision can be made.

    ๏ƒ˜ Random Path (RP) Measure: it includes all paths of different lengths in the network, which enables it to capture more discriminative information in faces and significantly reduce the effect of noise and outliers.

    ๏ƒ˜ In-face network: it captures the local information of face images. ๏ƒ˜ Out-face network: it captures the global information of face images.

    f๐‘€ ๐‘‡

View more