making machine understand beauty: a photography · disclaimer this presentation contain images from...
TRANSCRIPT
Making Machine Understand
Beauty: A Photography Perspective
Saimunur RahmanAdvanced Robotic Vision
© IEEE TMM (Lu et al., 2015)
Technical Sharing
Disclaimer
● This presentation contain images from various sources collected using Google Image search. If any of slides left source description, please feel free to consider Google Search as the source.
● This presentation contain materials and results which are private and confidential, and currently under publication and patent filing process. Please maintain the Vitrox honor code and do not disclose outside.
One picture is worth a thousand words.
~ chinese proverb
Beautiful Capture “Beauty is in the eye of the beholder”
A SUBJECTIVE CONJECTURE
● Everyone has different taste!
● Universal acceptance of beauty
○ Sunrise and sunset
○ Blue ocean, Mount Everest
Amatuer
Professional
7
Beauty affecting photographic factors
Beautiful Capture
IMAGE CAPTURING TECHNIQUES
a. http://www.brandon-schaefer.com/wp-content/uploads/2013/07/composition-03.jpgb. https://s-media-cache-ak0.pinimg.com/originals/4c/25/2d/4c252dcad46ae590e6298caf61e5fc5a.jpg
a
b
Rule of thirds Rule of thirds+POI
Normal image Lighting+effects
Lighting ContrastImage
Compos.
8
Vs.
Good Capture Bad Capture
Source: Wang, Yeqing, Yi Li and Fatih Murat Porikli. “Fine-tuning Convolutional Neural Networks for visual aesthetics.” ICPR (2016). 9
In photography, beauty = aesthetics !
10
1815-1825
Greek aisthētikós:
- aisthēt (ḗs) - a person who affects great love of art, music, poetry, etc.,
- ikos or -ic - occurs in nouns that represent a substantive use of adjectives
11Image source: http://www.dictionary.com/browse/aesthetic
12
Okay, What makes a higher
aesthetic image?
Image source: a. http://www.ideachampions.com/weblogs/Confused.jpg, b. https://expertfile.com/speakers/img/ExpertCroppedHomePage.png
Well, judged by experts based on common photo-graphic rules
a b13
“2” dominating factors for comps. rules
Lighting Contrast&
15
How to make machine understand the sense of beauty a.k.a. aesthetics?
Image Source: http://www.quickmeme.com/too-much-feel16
higher aesthetic image = more likes!
Facebook posts with images see 2.3X more engagement
Source: https://blog.hubspot.com/marketing/visual-content-marketing-strategy17
Possible applications
Image gallery
a. http://ezoui.com/gallery/img/03_Grid.pngb. https://s-media-cache-ak0.pinimg.com/736x/df/57/95/df5795c199c0d28d3a1eb476c349739a.jpgc. https://screenshots.en.sftcdn.net/en/scrn/79000/79347/video-thumbnails-maker-28.jpg
Video thumbnailMultimedia archives
a b c 18
Case study: EyeEm
19Video URL: https://player.vimeo.com/video/154364175
Outline of talk
● Learning for image aesthetics evaluations
● Recent progress on image aesthetics evaluations
● My work on image aesthetics evaluations
● Image popularity vs. image aesthetics
● Image quality vs. image aesthetics
● Demo for image aesthetics evaluations
● Sharing by two aesthetics evaluation researchers
20
Machine learning for aesthetics prediction
21
Image aesthetics ranking
Good Bad
Binary ranking Fuzzy ranking
22
[1 ………….. 0]
Common binary ranking pipeline
23
Higher ranked images
Lower ranked images
Feature Extraction
Train Classifier
Good/Bad
Feature Extraction
Test ImageTrained
Classifier
Training Phase
Testing Phase
Good/Bad
Common fuzzy ranking pipeline
24
Higher to lower ranked images
Feature Extraction
Train Regressor
[0 ... 1]
Feature Extraction
Test ImageTrained
Regressor
Training Phase
Testing Phase
[0 ... 1]
Aesthetics inference
25
Datasets
Image source: http://vision.stanford.edu/Datasets/collage_s.png
26
Photo.net DPChallenge AVA
~20k10 rating/imgRating: [0-7]
~17kBinary
ranking/img
~250k78-549 rating/img
Rating: [0-10]
27
Guidelines for judging images
it looks good
attracts/holds attention
has an interesting
composition
has great use of color
contains drama, humor, impact
(if sports) peak
moment, struggle
of athlete
Raking in Photo.net
Images were collected from numerous sources through Google search
Aesthetic Feature Extraction
28Image source: http://www.kdnuggets.com/wp-content/uploads/feature-extraction.jpg
29
Generic image features
Low level High level
Edge distribution Rule of third
Color histogram Golden ratio
Slide courtesy: Magzhan Kiranbay
Slide courtesy: Magzhan Kiranbay
Generic image feature types
31
Global Local
example: GIST example: SIFT
Global Features: GIST
32Image courtesy: Nguyen, Quang-Khue, Thi-Lan Le, and Ngoc-Hai Pham. "Leaf based plant identification system for android using surf features in combination with bag of words model and supervised learning." In Advanced Technologies for Communications (ATC), 2013 International Conference on, pp. 404-407. IEEE, 2013.
GIST Estimation
Local Feature: SIFT
33Image source: https://www.codeproject.com/KB/recipes/619039/SIFT.JPG
DoG in scales
Additional step for local features:
global feature transformation or holisticization
35
Input Image Feature extraction
Expected outcome
General expectation from a descriptor
Input Image Feature extraction … … …
… … …
Local feature outcome
Holistic transformer known as feature encoder such as Bag-of-Features, Sparse Coding, LLC etc.
Generic/Task-specific deep features
36
● Use popular pre-trained models on datasets such as ImageNet on aesthetics images● Fine-tune deep architecture weights with aesthetics images● Any intermediate till end of layers can be used as features!
Sample CNN architecture image source: https://upload.wikimedia.org/wikipedia/commons/6/63/Typical_cnn.png
Classifier/Regressor
Classification: if “y” is discrete/categorical variable
Regression: if “y” is real number/continuous
37
f:x y
Recent progress on Aesthetic Evaluations
38
39
Typical aestheticevaluation pipeline
Image reproduced from Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Image Aesthetic Assessment: An Experimental Survey." arXiv preprint arXiv:1610.00838 (2016).
Hand-engineered features
● Simple Image Features: color, brightness, contrast, texture, simplicity,composition geometry … [global features]
● Image Composition Features: salient regions, composition regions [global+local features]
● General-Purpose Features: SIFT, HOG, GIST etc. [mostly local features]
● Task-Specific Features: known image nature [global+local features]
40Source: Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Image Aesthetic Assessment: An Experimental Survey." arXiv preprint arXiv:1610.00838 (2016).
Deep learned features
● Generic Deep Features
○ Single stream architecture
○ Multi-stream architecture
● Pretrained/tuned features
● Mostly CNNs
41
Single stream architecture
Multi-stream architecture
Image reproduced from Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Image Aesthetic Assessment: An Experimental Survey." arXiv preprint arXiv:1610.00838 (2016).
Some results: DPChallenge (CUHK-PQ) dataset
42
CHUK-PQ Dataset: ~4.5k positive and ~13k negative images
Source: Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Image Aesthetic Assessment: An Experimental Survey." arXiv preprint arXiv:1610.00838 (2016).
Some results: AVA dataset
43
AVA Dataset: ~176k positive and ~74k negative images
Source: Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Image Aesthetic Assessment: An Experimental Survey." arXiv preprint arXiv:1610.00838 (2016).
CNN Transfer Learning for Image Aesthetics Evaluations
44
Slides removed due to publication issues!
Real-time Aesthetics Evaluation of South-Asian Selfies
45
Slides removed due to publication issues!
Image Popularityvs.
Image Aesthetics
46
Image popularity vs. aesthetics
47Image courtesy: Khosla, Aditya, Atish Das Sarma, and Raffay Hamid. "What makes an image popular?." In Proceedings of the 23rd international conference on World wide web, pp. 867-876. ACM, 2014.
What makes an image popular? (Khosla et al. 2014)
48Slide courtesy: Aditya Khosla
Image qualityvs.
Image Aesthetics
49
Image quality vs. image aesthetics
● Image quality measure (IQM) used for objective quality assessment
● Typically used for image restoration
○ Superresolution
○ De-blur
○ de-artifacts
● Usually have a reference image for comparison
● IQM is not designed to measure the subjective nature of human perceived aesthetic quality
○ May generate misleading results
50
Interesting Observation
51
Aesthetics Score: 0.23
Interesting Observation (2)
52
Aesthetics Score: 0.76
53
Thank you
54
Got Any Question?