the quotient image: class-based recognition and synthesis under varying illumination t. riklin-raviv...
TRANSCRIPT
![Page 1: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/1.jpg)
The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination
T. Riklin-Raviv and A. Shashua
Institute of Computer Science Hebrew University
![Page 2: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/2.jpg)
Example: 3 basis images
The movie is created by simplylinearly combining the 3 basis images.
![Page 3: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/3.jpg)
Given a single image y =
And a database of other images of the same “class”
Image Synthesis:
We would like to generate new images from y simulatingchange of illumination.
BootstrapSetlight
ID
![Page 4: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/4.jpg)
Recognition
Given a database of images of the same “class”, under varying illumination conditions
and a novel image
Match between images of the same object.
![Page 5: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/5.jpg)
Definition: Ideal Class of Objects
The images produced by an ideal class of objects are
jT
i syxnyxyxI ),(),(),(
Where ),( yxi is the albedo (surface texture) of object i of the class
),( yxn is the normal direction (shape) of the surface, i.e., all objects have the same shape.
js is the point light source direction.
yx,
![Page 6: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/6.jpg)
Related WorkBasic result :Shashua 91,97
Application and related systems:Hallinan 94,
Kriegman et. al 96,98
Work on “class-based” synthesis and recognition of images-mostly with varying viewing positions:Basri 96, Bymer & Poggio 95, Freeman & Tenenbaum 97,Vetter & Blanz 98, Vetter, Jones & Poggio 97, 98Edelman 95, Atick, Griffin & Redlich 97.
Rendering under more general assumption: Dorsey et al. 93,94
Linear class : Vetter & Poggio 97,92
Additive error term : Sali & Ulman 98
Reflectance Ratio: Nayar & Bolle 96
![Page 7: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/7.jpg)
The Quotient image: Definition
Given images ss ay ,
of objects y and a respectively, under illumination S
a
yyQ
=
sNy Tys
sNa Tas
Thus yQ depends only on relative surface information
and is independent of illumination.
(pixel by pixel division)
![Page 8: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/8.jpg)
The Quotient image Method: Proposition
Let 321 ,, aaa 3 images of object a .
Let sy Image of object y illuminated by light s .
Moreover, the image space of y is spanned by
varying the coefficients.
321 ,, xxx that satisfy:
j
yjjs Qaxy )(
Then, there exist
![Page 9: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/9.jpg)
The Quotient image Method: Proof
321 ,, aaa Illuminated by: 321 ,, sss
j
jjsxssy Illuminated by:
j
yjjs Qaxy )(
snTa
a
y
![Page 10: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/10.jpg)
a1a2
a3
Pa
a1a2
a3
Q-Image
* =?N=1
j
jjsxs
s
s1
s2
s3
The Quotient image Method: N=1sy
s332211 axaxaxas
sa
sa
![Page 11: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/11.jpg)
Pa
a1a2
a3
* =
Q-ImageN=1
The Quotient image Method: N=1sa Synthesized Image
s
332211 azazazas
sa s
![Page 12: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/12.jpg)
The Quotient image Method: Conclusions
yQ one can generate sy and all other images of
the image space of y .
Given
Given sy and the coefficients jx that satisfies
j jjsxs then yQ readily follows
j jj
sy ax
yQ
In order to obtain the correct coefficients jx
a bootstrap set of more than one object is needed .
![Page 13: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/13.jpg)
The Quotient image Method: N>1
Original image
sy
s
xA1
1AP
xA2
2AP
xA3
3AP
syxA 11 syxA 22
syxA 33
N
isii yxAxf
1
2ˆ
2
1ˆ
3misAi
![Page 14: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/14.jpg)
The Quotient image Method: Theorem-1
The energy function
N
isii yxAxf
1
2ˆ
2
1ˆ
has a (global) minimum xx ˆ , if the albedo y
of object y is rationally spanned by the bootstrap set.
i.e if there exist coefficients N ,,1 such that
NN
Ny
11
221 ...
![Page 15: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/15.jpg)
N
isii yxAxf
1
2ˆ
2
1ˆ
iiis
i
Tii
ii
Ti vyAAAx
1
ˆ
sTi
N
rr
Tri yAAAv
1
1
The Quotient Image Method: Solving For X and i
Min,x
We also have: sTi
Ts
Tsi
i
yAxyyf
ˆ0
![Page 16: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/16.jpg)
for Ni 1 written explicitly
0
0
0
11
2211
1111
sTss
TN
TNNs
TN
T
sTT
NNsTT
sTT
NNsTss
TT
yyyAvyAv
yAvyAv
yAvyyyAv
The Quotient image Method: Solving For X and i
![Page 17: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/17.jpg)
The Algorithm
Given : NAA ,,1 a bootstrap set and a novel image sy
homogenous system of linear equations in N ,1
Scale such that i i N
Compute i iivx
Ax
yQ s
y Where A is the average of NAA ,,1
ynew QAzZy )( For all choices of z
Use the minimization function:
N
isii yxAxf
1
2ˆ
2
1ˆ Min
,xto generate
![Page 18: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/18.jpg)
Samples of few faces out of 9*200 faces images taken from T. Vetter databasewhich was mainly used as a bootstrap set and as a source for novel images inthe further demonstration.
Frontal faces : Collection of objects all have the same shape but differ in their surface texture (albedo)...
light
ID
![Page 19: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/19.jpg)
Synthesis from Single PictureAnd 10 faces from the bootstrap setunder 3 different light conditions
Synthesis from 3 pictures
Linear CombinationQuotient Method
![Page 20: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/20.jpg)
10 other faces
from the database, each under 3light conditions
Synthesis from 3 picturesSynthesis from Single imageand the bootstrap set
![Page 21: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/21.jpg)
Synthesis from 3 picturesSynthesis from Single imageand the bootstrap set
Bootstrap Set
Original image Quotient Image
![Page 22: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/22.jpg)
(0,+20) (+50,0)(-50,0)(0,-35)(0,0)
Original Images Compared to Q-Image Synthesized Images
1st Row: Original Images
2nd Row: Q-Image Synthesized Images
3rd Row: Exact Values of Light Direction: center, down, up, right, left
![Page 23: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/23.jpg)
Light Coefficient Comparison Ground Truth Vs. Q-Image Coefficients
1st Coefficient
2nd Coefficient
3rd Coefficient
![Page 24: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/24.jpg)
Using Different Database
Animation Using the database
3x3 images’ Database
The Quotient Image The original image
![Page 25: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/25.jpg)
The Original images
Q images , 1 object bootstrap set
Q images , 10 object bootstrap set
![Page 26: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/26.jpg)
Handling Color Images
RGB HSV Transformation
Original color image R G B
H S V
![Page 27: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/27.jpg)
Original Image Quotient Image
Synthesized Sequence
![Page 28: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/28.jpg)
Original Images Quotient Images
Synthesized Sequences
Monica and Bill Under a New Light
![Page 29: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/29.jpg)
Original Image Quotient Image
Synthesized Sequence
![Page 30: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/30.jpg)
Recognition under varying illumination
Each object in the database is represented by its quotient image only.The quotients can be made of images with varying illuminations.
The quotient images was generated out of N*3 (N=20) base images.
A=
B=
C=
. . .
. . .
N
Database generation
![Page 31: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/31.jpg)
Identification
Given a new image of an object appears in the data base under any lightcondition, it’s quotient is computed from A,B,C … (as was done in the database generation) . Then It is compared to the quotientsin the data base.
Other methods used for comparison
1. CorrelationDatabase: Each object is represented in the database by it’s image under
any/certain lightening condition. Identification: Correlation between the test image to the images stored in the
database.2. PCA Database: Applying PCA on the objects’ images + 3*20 additional images of 20 objects under 3 illumination (to compare conditions to the quotient method). Having eigen vectors,
each object is represented by it’s eigen vectors’ coefficients.Identification: Comparison (LSE) between the test image coefficients (generated the same way as the database) and the database.
![Page 32: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/32.jpg)
Recognition Results
0%
5%
10%
15%
20%
25%
30%
35%
sameill.
varyingill
Q methodCorrelation
Quotient method comparing to correlation
![Page 33: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/33.jpg)
Recognition Results - cont
0%
1%
2%
3%
4%
5%
6%
7%
same ill. varyingill
Q methodPCA 30 evPCA 50 ev
Quotient Method Vs. PCA
![Page 34: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew](https://reader035.vdocuments.net/reader035/viewer/2022062301/56649f145503460f94c296a5/html5/thumbnails/34.jpg)
The End