generative adversarial nets - cosy.sbg.ac.atheld/teaching/wiss_arbeiten/slides_19-2… · paper...

Generative Adversarial Nets

Schuiki Johannes, Kempnter Stefan

January 31, 2020

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 1 / 46

Disclaimer

The main aspects and ideas of this presentation are based on thepaper ”Generative Adversarial Networks” from Ian Goodfellow et al.

(2014) [2].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 2 / 46

Contents

1 Introduction

2 Prerequisites

3 Generative Adversarial Nets

4 Applications

5 Resources

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 3 / 46

Introduction

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 4 / 46

Introduction

This person does not exist!

Figure: Result of StyleGAN. Src.:https://www.thispersondoesnotexist.com/ [1]

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 5 / 46

Prerequisites

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 6 / 46

The Perceptron

f (x;w) = φ

(n∑

i=1

xiwi+b

)= y

�1

�1

�2�2 Σ

�3

�3

y

Figure: A perceptron with three inputs x1 to x3 and an activation function φ

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 7 / 46

Neural Networks / Multilayer Perceptron

f (x;w) = ? = y

�1

�2

ℎ13

�3

ℎ14

ℎ11

ℎ12

ℎ23

ℎ24

ℎ21

ℎ22

�

Figure: A multi-layer-perceptron with three inputs x1 to x3 and an activationfunction φ and hidden layers h

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 8 / 46

Forward Pass

h1i = φ

∑j

xjw1ji

�1

�2

ℎ13

�3

ℎ14

ℎ11

ℎ12

ℎ23

ℎ24

ℎ21

ℎ22

�

( )�� ( )�1�� ( )�2�� ( )�3�( )ℎ1� ( )ℎ2�

Figure: A multi-layer-perceptron with three inputs x1 to x3 and an activationfunction φ and hidden layers hSchuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 9 / 46

Forward Pass

h2i = φ

∑j

h1jw2ji

�1

�2

ℎ13

�3

ℎ14

ℎ11

ℎ12

ℎ23

ℎ24

ℎ21

ℎ22

�

( )�� ( )�1�� ( )�2�� ( )�3�( )ℎ1� ( )ℎ2�

Figure: A multi-layer-perceptron with three inputs x1 to x3 and an activationfunction φ and hidden layers hSchuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 10 / 46

Forward Pass

y = φ

∑j

h2jw3j

�1

�2

ℎ13

�3

ℎ14

ℎ11

ℎ12

ℎ23

ℎ24

ℎ21

ℎ22

�

( )�� ( )�1�� ( )�2�� ( )�3�( )ℎ1� ( )ℎ2�

Figure: A multi-layer-perceptron with three inputs x1 to x3 and an activationfunction φ and hidden layers hSchuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 11 / 46

Forward Pass

y = f (x;w) = φ

∑i

φ

∑j

φ

(∑k

xkw1kj

)w2ji

w3i

�1

�2

ℎ13

�3

ℎ14

ℎ11

ℎ12

ℎ23

ℎ24

ℎ21

ℎ22

�

( )�� ( )�1�� ( )�2�� ( )�3�( )ℎ1� ( )ℎ2�

Figure: A multi-layer-perceptron with three inputs x1 to x3 and an activationfunction φ and hidden layers hSchuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 12 / 46

Activation Function φ

Figure: Different activation functions and their derivatives.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 13 / 46

Forward Pass

f (x;w) = φ

∑i

φ

∑j

φ

(∑k

xkw1kj

)w2ji

w3i

f is differentiable!

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 14 / 46

Objective function (Cost/Loss)

We use a loss function (in this case Sum of Squared Differences) as aquality metric of the net.

L(x;w) =N∑

n=1

(f (x;w)− targetn)2

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 15 / 46

Gradient Descent

w(i+1) = w(i) − η∂L(w)

∂w

�∂

∂��

������� �����

������� �

���� ��������

� ���ℎ� ��

����

=�(�

;�

)

����. �����

Figure: Simplified illustration of stochastic gradient descent with one weight.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 16 / 46

Back propagation

All we need: Chain Rule

y = f (u) u = g(x)

dy

dx=

dy

du∗ dudx

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 17 / 46

Back propagation

∂loss

∂w1ij

�1

�2

ℎ13

�3

ℎ14

ℎ11

ℎ12

ℎ23

ℎ24

ℎ21

ℎ22

�

( )�� ( )�1�� ( )�2�� ( )�3�( )ℎ1� ( )ℎ2�

Figure: A multi-layer-perceptron with three inputs x1 to x3 and an activationfunction φ and hidden layers h

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 18 / 46

Back propagation

We remember:

L(x;w) =N∑

n=1

(f (x;w)− targetn)2

f (x;w) = φ(∑i

φ(∑j

φ(∑k

xkw1kj)︸ ︷︷ ︸h1

w2ji )

︸ ︷︷ ︸h2

w3i )

We calculate:

∂L(x;w)

∂w1ij=∂L(x;w)

∂f (x;w)...

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 19 / 46

Back propagation

We remember:

L(x;w) =N∑

n=1

(f (x;w)− targetn)2

f (x;w) = φ(∑i

φ(∑j

φ(∑k

xkw1kj)︸ ︷︷ ︸h1

w2ji )

︸ ︷︷ ︸h2

w3i )

We calculate:

∂L(x;w)

∂w1ij=∂L(x;w)

∂f (x;w)

∂f (x;w)

∂h2...

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 20 / 46

Back propagation

We remember:

L(x;w) =N∑

n=1

(f (x;w)− targetn)2

f (x;w) = φ(∑i

φ(∑j

φ(∑k

xkw1kj)︸ ︷︷ ︸h1

w2ji )

︸ ︷︷ ︸h2

w3i )

We calculate:

∂L(x;w)

∂w1ij=∂L(x;w)

∂f (x;w)

∂f (x;w)

∂h2

∂h2

∂h1...

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 21 / 46

Back propagation

We remember:

L(x;w) =N∑

n=1

(f (x;w)− targetn)2

f (x;w) = φ(∑i

φ(∑j

φ(∑k

xkw1kj)︸ ︷︷ ︸h1

w2ji )

︸ ︷︷ ︸h2

w3i )

We calculate:

∂L(x;w)

∂w1ij=∂L(x;w)

∂f (x;w)

∂f (x;w)

∂h2

∂h2

∂h1

∂h1

∂w1ij

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 22 / 46

Generative Adversarial Nets

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 23 / 46

Generative Models

Figure: The generative model approximates the true data distribution by mappingz pz to the data-space. Src.:https://openai.com/blog/generative-models/

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 24 / 46

The Generative Adversarial Framework

Generator

DescriminatorRandom Noise

Training Data

RealFake

Drawn from (�)��

Drawn from �����(�)

�(�)

�(�) �(�(�))or

�(�) >1

2

�(�) ≤1

2

Figure: The components of the generative adversarial framework as discribed in[2].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 25 / 46

Generative Adversarial Nets

Random Noise

Training Data

RealFake

Drawn from (�)��

Drawn from �����(�)

�(�, )��

�(�, )�� �(�(�, ))��or

�(�, ) >��

1

2

�(�, ) ≤��

1

2

Figure: The components of generative adversarial nets as discribed in [2].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 26 / 46

The objective function

A min max game!

minG

maxD

V (D,G ) = Ex∼pdata(x) [logD(x)] + Ez∼pz (z)[log(1− D(G (z))]

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 27 / 46

The objective function

Discriminator perspective

minG

maxD

V (D,G ) = Ex∼pdata(x) [logD(x)] + Ez∼pz (z)[log(1− D(G (z))]

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 28 / 46

The objective function

Generator perspective

minG

maxD

V (D,G ) = Ex∼pdata(x) [logD(x)]+Ez∼pz (z)[log(1− D(G (z))]

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 29 / 46

for number of training iterations dofor k steps do• Sample minibatch of m samples {z (1), ..., z (m)} from pz .• Sample minibatch of m samples {x (1), ..., x (m)} from pdata.• Update the discriminator by ascending its stochastic gradient:

∇θd1

m

m∑i=1

[logD(x (i)) + log(1− D(G (z (i))))

]end

• Sample minibatch of m samples {z (1), ..., z (m)} from pg .• Update the generator by descending its stochastic gradient:

∇θg1

m

m∑i=1

log(1− D(G (z (i))))

end

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 30 / 46

for number of training iterations dofor k steps do• Sample minibatch of m samples {z (1), ..., z (m)} from pz .• Sample minibatch of m samples {x (1), ..., x (m)} from pdata.• Update the discriminator by ascending its stochastic gradient:

∇θd1

m

m∑i=1

[logD(x (i)) + log(1− D(G (z (i))))

]end

• Sample minibatch of m samples {z (1), ..., z (m)} from pg .• Update the generator by descending its stochastic gradient:

∇θg1

m

m∑i=1

log(1− D(G (z (i))))

end

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 31 / 46

Visualization of the Training Process

Figure: A GAN trying to approximate the normal distribution. Black representsthe normal distribution. Green the distribution generated by the Generator andblue the discriminative distribution. Src.: [2]

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 32 / 46

Theoretical Results

Nice! But can you prove it?Yes No Maybe?!

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 33 / 46

Global Optimality

All proofs consider the non parametric setting. (Infinite capability ofthe nets.)

At any step the proofs consider an optimal discriminator.

Using the optimal discriminator they reformulate the criterion andshow that global minimum of the criterion is given for pg = pdata.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 34 / 46

Convergence of the Algorithm

The reformulated ”virtual” training criterion uses the Jenson-Shannondivergence.

The proofs argue that this divergence is convex for a fixed pdata andvariable pg .

Since it is already proved that the global minimum is at pg = pdatathe algorithm will converge towards it.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 35 / 46

The reality

In practice, adversarial nets represent a limited family of pg distri-butions via the function G (z ; θg ), and we optimize θg rather thanpg itself, so the proofs don’t apply.[2, p.7]

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 36 / 46

Advantages

Backpropagation can be used. No other methods and/orapproximations due to intractable gradients required.

Statistical: The generator never sees images from the original datadistribution. Only the gradients. → No ”memorization” possible

GANs can model sharp and degenerate distributions while othermodels need the distrubution to be ”smooth” to a certain extent.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 37 / 46

Disadvantages

There is no explicit representation of the distribution pg

The training of D and G has to be balanced

Immense amounts of data are required for training.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 38 / 46

Latent Space Arithmetic

The space where you draw a random input vector for your generatoris called ’Latent Space’

Some directions in this space have semantic meaning

Figure: man with glasses - man + woman = woman with glasses. Screenshotfrom [3].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 39 / 46

Change of Style

Figure: cycleGAN. Screenshot from [4].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 40 / 46

HD Image generation from semantic map

Figure: Screenshot from [5].Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 41 / 46

Text to Image

Figure: Screenshot from [6].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 42 / 46

Ethical Issues: Deepfake

Figure: Screenshot from Youtube [7].

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 43 / 46

Thank You!

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 44 / 46

Resources I

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila,“Analyzing and improving the image quality of stylegan,” ArXiv,vol. abs/1912.04958, 2019.

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial networks,”ArXiv, vol. abs/1406.2661, 2014.

P. Bojanowski, A. Joulin, D. Lopez-Paz, and A. Szlam, “Optimizing thelatent space of generative networks,” 07 2017.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-imagetranslation using cycle-consistent adversarial networks,” in Computer Vision(ICCV), 2017 IEEE International Conference on, 2017.

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 45 / 46

Resources II

T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro,“High-resolution image synthesis and semantic manipulation with conditionalgans,” in Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, 2018.

H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas,“Stackgan++: Realistic image synthesis with stacked generative adversarialnetworks,” IEEE Transactions on Pattern Analysis and Machine Intelligence,vol. PP, 10 2017.

“https://www.youtube.com/watch?v=vwrhrbb-1ig.”

Schuiki Johannes, Kempnter Stefan Generative Adversarial Nets January 31, 2020 46 / 46