introduction of saliency map
DESCRIPTION
Introduction of Saliency Map. Presenter: Chien-Chi Chen Advisor: Jian-Jiun Ding. Outline. Introduction of saliency map Button-up approach L. Itti’s approach Frequency-tuned Multi-scale contrast Depth of field Spectral Residual approach Global contrast based Top-down approach - PowerPoint PPT PresentationTRANSCRIPT
Introduction of Saliency Map
Presenter: Chien-Chi ChenAdvisor: Jian-Jiun Ding
1
Outline• Introduction of saliency map• Button-up approach
– L. Itti’s approach– Frequency-tuned– Multi-scale contrast– Depth of field– Spectral Residual approach– Global contrast based
• Top-down approach– Context-aware
• Information maximum– Measuring visual saliency by site entropy rate
2
Outline• Introduction of saliency map• Button-up approach
– L. Itti’s approach– Frequency-tuned– Multi-scale contrast– Depth of field– Spectral Residual approach– Global contrast based
• Top-down approach– Context-aware
• Information maximum– Measuring visual saliency by site entropy rate
3
Introduction of saliency map
• Low-level(contrast)– Color– Orientation– Size– Motion– Depth
• High-level– People– Context
Important!
4Judd et
al, 2009
Low-level
With face
detection
Outline• Introduction of saliency map• Button-up approach
– L. Itti’s approach– Frequency-tuned– Multi-scale contrast– Depth of field– Spectral Residual approach– Global contrast based
• Top-down approach– Context-aware
• Information maximum– Measuring visual saliency by site entropy rate
5
L. Itti’s approach
• Architecture:Gaussian Pyramids
R,G,B,Y Gabor pyramids for = {0º, 45º, 90º, 135º}
L. Itti’s approach• Center-surround Difference• Achieve center-surround difference through across-scale difference
• Operated denoted by Interpolation to finer scale and point-to-point subtraction
• One pyramid for each channel: I(), R(), G(), B(), Y()where [0..8] is the scale
L. Itti’s approach• Center-surround Difference
– Intensity Feature Maps• I(c, s) = | I(c) I(s)|• c {2, 3, 4}• s = c + where {3, 4}• So I(2, 5) = | I(2) I(5)|
I(2, 6) = | I(2) I(6)| I(3, 6) = | I(3) I(6)| …
• 6 Feature Maps
L. Itti’s approach• Center-surround Difference
•Color Feature Maps
Red-Green and Yellow-Blue
Center-surround DifferenceOrientation Feature Maps
•
+R-G
+R-G+G-R
+G-R +B-Y
+Y-B
+Y-B
+B-Y
+B-Y
Same c and s as with intensity
),(),(),,( sOcOscO
RG(c, s) = | (R(c) - G(c)) (G(s) - R(s)) |BY(c, s) = | (B(c) - Y(c)) (Y(s) - B(s)) |
L. Itti’s approach• Normalization Operator• Promotes maps with few strong peaks• Surpresses maps with many comparable
peaks1. Normalization of map to range [0…M]2. Compute average m of all local maxima 3. Find the global maximum M4. Multiply the map by (M – m)2
L. Itti’s approach
Inhibition of return
Example of Operation:
Frequency-tuned
12
Image Average
Gaussian blur
L
I a
b
( , )hc
hc hc
hc
L
I x y a
b
( , ) ( , )hc
S x y I I x y
Multi-scale contrast
• Saliency algorithm
Image Saliency map
Multi-scale contrast
Center-surround histogram
Color spatial-distribution
ConditionalRandomField
13
Multi-scale contrast
Multi-scale contrast• Local summation of
laplacian pyramid
Center-surround histogram• Distance between histograms
of RGB color:
2
1 ( )( , ) || ( ) ( ) ||
L l lc
l x N xf x I I x I x
22 ( )1
( , )2 ( )
i is
s i is
R RR R
R R
* 2
( )( ) arg max ( ( ), ( ))s
R xR x R x R x
*
2 * *
{ | ( )}( , ) ( ( ), ( ))h xx s
x x R xf x I R x R x
14
Multi-scale contrast
• Color spatial-distribution
Image(RGB)
GMMDistance from pixel x to image center
The variance of Coordinate of pixel x and y
( , ) ( | ) (1 ( )) (1 ( ))s xc
f x I p c I V c D c
15
Multi-scale contrast
• Energy term:
• Saliency object:
1 ,( | ) ( , ) ( , , )
Kk k x x x
x k x xE A I F a I S a a I
( , ), 0( , )
1 ( , ), 1k x
k xk x
f x I aF a I
f x I a
• Pairwise feature:
,( , , ) | | exp( )x x x x x xS a a I a a d
, || ||, 2x x x xd I I L norm
2 1(2 || || )x xI I
16
Multi-scale contrast
• CRF:
• The derivative of the log-likelihood with respect to
1( | ) exp( ( | ))P A I E A I
Z
* arg max log ( | ; )n n
nP A I
k
17
Depth of field
• As the spread of single lens reflex camera, more and more low depth of field(DOF) images are captured.
• However, current saliency detection methods work poorly for the low DOF images.
18
Depth of field
• Algorithm:
19
Depth of field
• Classification: • Focal Point: In a low DOF image
DOGRectangle with the highest edge density, and center is initial focal point
2( , ) ( , )
d
S i j S i j Ae
• Composition Analysis:segmentation Region
1 2 3
r
i
A n d
A mr rS S e
20
Spectral Residual Approach
• First scaling image to 64x64.• Then we smoothed the saliency map with a
gaussian filter g(x) ( = 8).
21
Global contrast-based
• Histogram based contrast(Lab):
2( )O N 2( ) ( )O N O n
Quantization of Lab
Each channel to have 12 different value
312 1728
8522
Global contrast-based
• Region based contrast– Segment the Image– [Efficient graph-based image segmentation]
23
Outline• Introduction of saliency map• Button-up approach
– L. Itti’s approach– Frequency-tuned– Center-surround– Depth of field– Spectral Residual approach– Global contrast based
• Top-down approach– Context-aware
• Information maximum– Measuring visual saliency by site entropy rate
24
Context-Aware
• Goal: Convey the image content
25
Liu et al, 2007
Context-Aware
• Distance between a pair of patches:
( , )( , )
1 ( , )color i j
i jposition i j
d p pd p p
c d p p
salient
High j
Context-Aware
• Distance between a pair of patches:
High for K most similar
Saliency
k
rq K most similar patches at scale r1
11 exp ( , )
i j
Kr r ri
k
S d p qK
Context-Aware
• Salient at:– Multiple scales foreground– Few scales background
1
1 Mrr
i ir r
S SM
Scale 1 Scale 4
Context-Aware
• Foci =
• Include distance map
0.8iS
1 ( )focid i
X
iS
ˆ 1 ( )i i fociS S d i
Outline• Introduction of saliency map• Button-up approach
– L. Itti’s approach– Frequency-tuned– Center-surround– Depth of field– Spectral Residual approach– Global contrast based
• Top-down approach– Context-aware
• Information maximum– Measuring visual saliency by site entropy rate
30
Measuring visual saliency by site entropy rate
31
1
Measuring visual saliency by site entropy rate
32
A fully-connected graph representation is adopted for each
2
Sub-band graph representation
33
Sub-band graph representation
34
Measuring visual saliency by site entropy rate
35
A random walk is adopted on each sub-band graph. And Site entropy rate(SER) is measured the average information from a node to the other
3
The site entropy rate
•
•
36
ijij
ijj
P
, :, ,
2i
i i ij ijj i j j i
WW W
W
Conclusion
• Image processing is funny • Unusual in its neighborhood will correspond
to high saliency weight• Contrast is the key of saliency
37
Reference[1] R. Achanta, F. Estrada, P. Wils, and S. S¨usstrunk. Salient region detection and
segmentation. In ICVS, pages 66–75. Springer, 2008. 410, 412, 414[2] R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk. Frequency-tuned salient
region detection. In CVPR, pages 1597–1604, 2009. 409, 410, 412, 413, 414, 415[3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual attention for rapid
scene analysis. IEEE TPAMI, 20(11):1254–1259, 1998. 409, 410, 412, 414[4] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR,
pages 1–8, 2007. 410, 412, 413, 414[5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency detection. In
CVPR, 2010. 410, 412, 413, 414, 415[6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast based Salient
Region Detect. In CVPR, 2011 .[7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to detect a
salient object. IEEE TPAMI, 33(2):353–367, 2011. 410[8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by Site Entropy
Rate, In CVPR, 2010.
38