ai-sketcher: a deep generative model for generating high ... · ai-sketcher - cnn-based autoencoder...
TRANSCRIPT
AI-Sketcher: A Deep Generative Model for Generating High Quality Sketches
Nan Cao, Xin Yan, Yang Shi, Chaoran Chen
Intelligent Big Data Visualization LabTongji University
BACKGROUND - Cave Painting
2
BACKGROUND - Children’s Drawings
3
BACKGROUND - Design Drafts
4
Can AI help designers create high quality sketches and boost their productivity and creativity?
5
AI-Sketcher
6
7
Reference Pipeline for AI-Supported Sketching
Stroke Encoding Learning Generating
1 2 3
8
[△x,△y,p1,p2,p3]
Start coordinates relative to the end of last stroke
one-hotcontinuous
continuousdrawing from the last point,
end of drawing a stroke
end of drawing a sketch
Stroke Encoding
Stroke Encoding
Learning
Generating
9——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
Learning (Sketch-RNN)
Stroke Encoding
Learning
Generating
10——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
Encoder: Bidirectional RNN (BRNN)
Learning (Sketch-RNN)
Random Sampling for Stroke Generation
Encoder: Bidirectional RNN (BRNN)
Stroke Encoding
Learning
Generating
11——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
Decoder: Autoregressive RNN
Random Sampling for Stroke Generation
Learning (Sketch-RNN)
Encoder: Bidirectional RNN (BRNN)
Stroke Encoding
Learning
Generating
12——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
Generating (Sketch-RNN)
Stroke Encoding
Learning
Generating
13——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
LIMITATIONS OF SKETCH-RNN
Low quality
2. dealing with multi-class situations1. generating sketches in one category (bird).
14
AI-SKETCHER
15
AI-SKETCHER
16
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
AI-SKETCHER
17
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
18
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
19
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
20
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
21
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
22
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
23
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
Encoder: Bidirectional RNN
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
24
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
Xs: the sequences of strokes.
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
25
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
c is a k-dimensional one-hot conditional vector with k indicates the number of conditions.
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
26
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
hc is further transformed into two vectors to capture the distributions of the training strokes:
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
27
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
A latent vector has been randomly sampled from the distributions for generating the next strokes:
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
28
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
For decoding zs is concatenated together with the image feature vector zr, the latent influence vector ad, the conditional vector c and the last stroke vector si:
z = [zs; zr; ad; c; si]
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
29
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
predict the probabilities of the relative position of the next drawing point:
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
30
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
31
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
32
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER - Influence Layer
33
AI-SKETCHER - Influence Layer
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
It considers all the previous hidden node values until the latest drawing step in the RNN encoder.
34
AI-SKETCHER - Influence Layer
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
The influence vector ad is a latent vector whose fields are sampled from these normal distributions:
35
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
36
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
37
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
AI-SKETCHER - CNN-based Autoencoder
38
AI-SKETCHER - CNN-based Autoencoder
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
Xr: the input raster image matrix.
39
• three convolutional layers with the stride size as 2.
• the other three layers with the stride size as 1.
AI-SKETCHER - CNN-based Autoencoder
Encoder:
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
40
AI-SKETCHER - CNN-based Autoencoder
Encoder:
• The last layer is a fully-connected neural network to produce the latent feature vector zr with 128 dimensions.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
41
AI-SKETCHER - CNN-based Autoencoder
Decoder:
• three deconvolutional layers with stride sizes equal to 2.
• the other three layers with stride sizes equal to 1.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
42
AI-SKETCHER - CNN-based Autoencoder
ReLU is used as the activation function in both convolutional and deconvolutional layers.
tanh is used as the activation function of the fully-connected neural network.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
43
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
44
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
45
AI-SKETCHER - Loss function
• Loss function is modified.
46
AI-SKETCHER - Loss function
• Loss function is modified.
47
AI-SKETCHER - Loss function
• Loss function is modified.
the reconstruction lossestimates the differences between the generated strokes and the training samples.
estimates the distribution differences between the generated strokes and the strokes in the training set modeled by the standard normal distribution.
48
AI-SKETCHER - Loss function
• Loss function is modified.
49
EVALUATION
We performed three experiments in purpose of validating the AI-Sketcher’s
• drawing quality
• capability of generating sketches from multiple classes
• generation diversity
50
EVALUATION - Dataset
The QuickDraw dataset contains over 50 million sketches in 75 object categories and originally used for training Sketch-RNN.
https://quickdraw.withgoogle.com/ 51
The FaceX dataset consists of 5 million sketches of both male's and female's facial expressions showing seven different types of emotions.
EVALUATION - Dataset
https://facex.idvxlab.com/ 52
EVALUATION - Dataset
The FaceX dataset consists of 5 million sketches of both male's and female's facial expressions showing seven different types of emotions.
https://facex.idvxlab.com/
53
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
54
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
55
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
56
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
57
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
58
EVALUATION - 1st Experiment
A within-subject user study
• 20 participants (10 females)
• The repeated measures one way ANOVA analysis showed that the generation quality of AI-Sketcher had an average rating of 3.9and was significantly better than that of the baseline models (with all p<.01).
Drawing quality
59
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
60
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
61
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
62
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
63
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
64
EVALUATION - 3rd Experiment
Generation Diversity
Experiments based on QuickDraw Dataset
In each set, the pairwised distances between sketches were calculated based on the perceptual hash.
The unpaired t-test showed that AI-Sketcher and Sketch-RNN had no significant difference.
65
POTENTIAL APPLICATION
66
CONCLUSION
67
We introduced AI-Sketcher, a hybrid deep
learning model to automatically generate high
quality sketch drawings .
Our model improves drawing quality by
• employing a CNN-based autoencoder to
capture the positional information.
• introducing an influence layer to more
precisely guide the generation of each stroke.
• provideding a conditional vector to support
multi-class sketch generation.
https://facex.idvxlab.com/
Thank You
Nan Cao, Xin Yan, Yang Shi, Chaoran Chen
Intelligent Big Data Visualization Lab
Tongji University
68