statistics of natural image categories - eecs at uc...
TRANSCRIPT
![Page 1: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/1.jpg)
Statistics of Natural ImageCategories
Authors: Antonio Torralba and Aude Oliva
Presented by:Sebastian Scherer
![Page 2: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/2.jpg)
2
Experiment
Please estimate the average depth from the camera viewpoint to all locations(pixels) in the next picture.
Next you will see a circle for 3s, then a picture for 1s.Concentrate on the circle.After that I will ask you about the average depth of the picture.
![Page 3: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/3.jpg)
3
![Page 4: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/4.jpg)
4
![Page 5: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/5.jpg)
5
What would you estimate the mean depth of this picture was?
![Page 6: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/6.jpg)
6
![Page 7: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/7.jpg)
7
![Page 8: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/8.jpg)
8
What is the mean depth of this picture?
![Page 9: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/9.jpg)
9
What is the mean depth of this picture?
![Page 10: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/10.jpg)
10
Problem
We want to determine global properties about an image.
However avoid• Explicit segmentation• No object recognition
Properties that are important for later stages of image processing are
• Scale• Type
![Page 11: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/11.jpg)
11
Outline
Global features: power spectra
Examples
Localized spectra
Applications of global features• Naturalness/Openness classification• Scene categorization• Object recognition• Depth estimation
![Page 12: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/12.jpg)
12
Power spectra
Decompose the image using a discrete Fourier transform:
The power spectrum is then given by the amplitude and phase
![Page 13: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/13.jpg)
13
Power spectrum plot
Horizontal
VerticalContour plotrepresentinga percentageof total energyof the spectrum
50%
80%
![Page 14: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/14.jpg)
14
Computing and visualizing a spectrum in Matlab•Computing the Spectrum (Matlab):
• Ifft = abs(fftshift(fft2(I,w,h)));
•Visualization:• imshow(log(Ifft)/max(max(log(Ifft))));• colormap(cool);
•(From Jonathan Huang's slides)
![Page 15: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/15.jpg)
15
Interesting fact: 1/f Spectra
Natural Image Spectra follow a power law!
As(θ) is called the Amplitude Scaling Factor
2-η(θ) is the Frequency Exponent. η clusters around 0 for natural images.
Any guesses on why this law holds?
![Page 16: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/16.jpg)
16
Use the spectra of images in order to categorize a scene
Why is this a good idea?• Texture varies with scene scale:
– Atmosphere is a low pass filter?– Sky– Mountains
– vs– Leaves– Objects
• Phase varies with environment:– Man-made– Nature
![Page 17: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/17.jpg)
17
Spectral signatures for different scenes
HorizonAll orientations
Buildings
![Page 18: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/18.jpg)
18
Scene scales
“The point of view that any given observer adopts on a specific scene is constrained by the volume of the scene.”
![Page 19: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/19.jpg)
19
What can one spectrum of an image not capture?
Generally we have a upright viewpoint.The horizon is towards the top.
![Page 20: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/20.jpg)
20
Non-stationary power spectra
Decompose the image using a DFT in local regions:
The localized power spectrum is then given by the amplitude and phase for a specific region
In their case: 8x8 spatial locations
![Page 21: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/21.jpg)
21
Non-stationary power spectra at different depth scales
Man-made
Natural
![Page 22: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/22.jpg)
22
What can we do with the power spectra?
Replicate perception of humans along different scales• Naturalness• Openness
Semantic categorization• Determine context of scene• Apply specialized methods after context is determined
Object recognition• Determine if an object exists in the scene• Only presence no location• Likely regions of objects
Depth estimation• Estimate the mean depth of the scene• Provides cue for object recognition
![Page 23: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/23.jpg)
23
Naturalness vs. Openness
Projection of images on thesecond and third principalcomponent.
Openness
Naturalness is represented bythe third principal component
![Page 24: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/24.jpg)
24
PCA
The power spectrumNormalization:
Perform PCA on the normalized power spectrum to get the spectral principal components (SPC).
![Page 25: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/25.jpg)
25
The first principal components of a set of images
Openness
Naturalness
![Page 26: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/26.jpg)
26
Semantic categorization
![Page 27: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/27.jpg)
27
Semantic categorization calculation
+
-
w
![Page 28: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/28.jpg)
28
Object recognition
![Page 29: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/29.jpg)
29
Object recognition – Algorithm
During training phase learn from a set of annotated pictures.O: object class, v_c: image statisticsBayes rule (without marginalization):
Equal prior is assumed:
Estimate with a mixture of Gaussians from
training set.
![Page 30: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/30.jpg)
30
Object recognition – Results 1
![Page 31: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/31.jpg)
31
Object recognition – Results 2
True positive rate True negative rate
![Page 32: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/32.jpg)
32
Adding spatial information
Split the picture in four equal regions.Learn a mixture of Gaussians in order to determine the region where one is most likely to find an object. Given the spectral feature at four location what is the most likely position of a face.
Attention will be on the most likely region to find a face.
![Page 33: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/33.jpg)
33
Regions of interest
90% of faces were within a region of 35% of the size of the image of the largest P(x|v_c)
![Page 34: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/34.jpg)
34
Gist
Use the global features as a prior on the location of objects in a object detection and localization algorithm. Since x is dependent on many factors only learn y and s.
where π(q) are the mixing weights, W is the regression matrix, µ are mean vectors, and Σ are covariance matrices for cluster q.
![Page 35: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/35.jpg)
35
Gist – Results
![Page 36: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/36.jpg)
36
Depth Estimation – Feature vector
Have a feature vector v.v' consists of the downsampled energy vector
k: wavelet index, x: location, M spatial resolutionFeature vector size: M^2 K
Apply PCA to reduce the dimensionality of v' to get v.v is a L-dimensional vector obtained by projecting v' on the first L principal components.
=> v is size L.
![Page 37: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/37.jpg)
37
Depth Estimation - Learning
Want to optimize this expression
D:depth, v: features, Nc: number of clusters, p(ci): cluster weight, g(v|ci): multivariate gaussian, g(D|v,ci):
Result is a mixture of linear regressions:
![Page 38: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/38.jpg)
38
Depth Estimation – Global features
![Page 39: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/39.jpg)
39
Depth Estimation – Localized features
![Page 40: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/40.jpg)
40
Scene category from depth
![Page 41: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/41.jpg)
41
Depth Estimation – Face Detection
Determine the size of an object as
Now have approximately the right scale for object detection:
![Page 42: Statistics of Natural Image Categories - EECS at UC Berkeleyefros/courses/LBMV07/presentations… · Statistics of Natural Image Categories Authors: Antonio Torralba and Aude Oliva](https://reader033.vdocuments.net/reader033/viewer/2022051806/5fff63f8f90a0b7ea4761f60/html5/thumbnails/42.jpg)
42
Discussion
Why do power spectra work so well?
Why is there such a large distinction between man-made and human? Are there possibly more distinct classes? Rural streets?
How do humans calculate mean depth when estimating the depth for training?