Upload File

g d gan o is l stable attend our oral presentation!vaishnan/nips17_poster.pdf · j. goodfellow et...

  • Home
  • Documents
  • G D GAN O IS L STABLE Attend our Oral Presentation!vaishnan/nips17_poster.pdf · J. Goodfellow et al., Generative Adversarial Networks (NIPS 2014) An increasingly popular class of
1
GRADIENT DESCENT GAN OPTIMIZATION IS LOCALLY STABLE Vaishnavh Nagarajan, J. Zico Kolter We address a fundamental open question in GANs about the stability of the saddle point equilibrium, using tools from non-linear systems theory. TOOLBOX: NON-LINEAR SYSTEMS Proving GAN stability is hard because of concave- concave-ity arbitrarily close to equilibrium, even for linear models! Locally exponentially stable: if for any initialization sufficiently close to equilibrium, the system converges to equilibrium quickly. [1] ASSUMPTIONS GRADIENT NORM BASED REGULARIZATION To use non-linear systems tools, we update both parameters simultaneously (not alternately) and infinitesimally for the true (not empirical) objective. Some of our assumptions are strong, yet they shed light on what is required for stability. 1. 2. OUR EXTENSION: Eigenvalue can have zero real part as long as it corresponds to direction within a subspace of equilibria Alternative stability analysis: LASALLE’S INVARIANCE PRINCIPLE Informally, equilibrium is exponentially stable if there exists a non- negative energy function (“Lyapunov function”) that at any time either i) strictly decreases ii) remains constant only instantaneously, and is zero only at equilibrium. RESULTS THEOREM: Under assumptions 1, 2, 3 the GAN system is locally exponentially stable. The Jacobian at equilibrium is: 1.0 0.5 0.0 0.5 1.0 0.0 0.5 1.0 1.5 2.0 w 2 a WGAN, Η0.0 1.0 0.5 0.0 0.5 1.0 0.0 0.5 1.0 1.5 2.0 w 2 a WGAN, Η0.5 1.0 0.5 0.0 0.5 1.0 0.0 0.5 1.0 1.5 2.0 w 2 a WGAN, Η0.25 Decreasing, and only instantaneously non-decreasing energy function: Distance from equilibrium REFERENCES 1. H. K Khalil. Non-linear Systems. Prentice-Hall, New Jersey, 1996. 2. L. Metz, et al., Unrolled generative adversarial networks. (ICLR 2017) 3. M. Heusel et al., GANs trained by a TTUR converge to a local Nash equilibrium (NIPS 2017) 4. I. J. Goodfellow et al., Generative Adversarial Networks (NIPS 2014) An increasingly popular class of generative models — models that “understand” data to output new random data. Formally, GANs learn distribution over the data. GENERATIVE ADVERSARIAL NETWORKS (GANS) 3. Attend our Oral Presentation! WED DEC 6TH 11:05 -- 11:20 AM Hall C SIMULTANEOUS GRADIENT DESCENT WGAN CONCLUSION THEORETICAL ANALYSIS OF GAN DYNAMICS USING NON-LINEAR SYSTEMS: despite lack of concave-convexity, GANs are locally stable under right conditions — perhaps why they work so well in practice! Regularizer can provably enhance local stability. LINEARIZATION THEOREM: The equilibrium is locally exponentially stable iff Jacobian at equilibrium has eigenvalues with strictly negative real parts. (even though the discriminator is not trained to optimality!) We assume at equilibrium, these locally convex functions are locally strongly convex — implying a locally unique equilibrium. Relaxation: Zero eigenvalues must correspond to “flat” directions within a subspace of equilibria. strongly convex Key Lemma: Matrices of this form have eigenvalues with strictly negative real parts despite zero block! 1.0 0.5 0.0 0.5 1.0 0.0 0.5 1.0 1.5 2.0 w GAN dynamics for a small system learning a uniform distribution. Discriminator Generator 2. Provides foresight to the generator: When norm of discriminator’s gradient is minimized, discriminator cannot improve itself much to outdo generator . 3. Similar to 1-unrolled [2] approximation of the pure minimization problem. There exist simultaneous gradient descent WGAN systems which do not necessarily converge to the equilibrium. OPEN QUESTIONS •Analyze other objectives and optimization techniques: unrolled GANs, f-GANs … •Relaxing some assumptions: Non-realizability: when discriminator is not linear in parameters? Without strong curvature? Intuitively: slower convergence, but still stable… •Global convergence, at least for linear discriminator and generator! •When do equilibria satisfying our conditions exist? Other proofs [3,4] require discriminator to be trained more often — effectively, closer to optimality, bringing system closer to pure minimization. Global equilibrium is realizable. 1. Makes Jacobian “more stable”

Upload: others

Post on 18-Aug-2020

2 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: G D GAN O IS L STABLE Attend our Oral Presentation!vaishnan/nips17_poster.pdf · J. Goodfellow et al., Generative Adversarial Networks (NIPS 2014) An increasingly popular class of

GRADIENT DESCENT GAN OPTIMIZATION IS LOCALLY STABLE Vaishnavh Nagarajan, J. Zico Kolter

We address a fundamental open question in GANs about the stability of the saddle point equilibrium,

using tools from non-linear systems theory.

TOOLBOX: NON-LINEAR SYSTEMS

Proving GAN stability is hard because of concave-concave-ity arbitrarily close to equilibrium, even for linear models!

Locally exponentially stable: if for any initialization sufficiently close to equilibrium, the system converges to equilibrium quickly. [1]

ASSUMPTIONS

GRADIENT NORM BASED REGULARIZATION

To use non-linear systems tools, we update both

parameters simultaneously (not

alternately) and infinitesimally for the true

(not empirical) objective.

Some of our assumptions are strong, yet they shed light on what is required for stability.1. 2.

OUR EXTENSION: Eigenvalue can have zero real part as long as it corresponds to direction within a subspace of equilibria

Alternative stability analysis: LASALLE’S INVARIANCE PRINCIPLE Informally, equilibrium is exponentially stable if there exists a non-negative energy function (“Lyapunov function”) that at any time either i) strictly decreases ii) remains constant only instantaneously, and is zero only at equilibrium.

RESULTS

THEOREM: Under assumptions 1, 2, 3 the GAN system is locally exponentially stable.

The Jacobian at equilibrium is:

!1.0 !0.5 0.0 0.5 1.00.0

0.5

1.0

1.5

2.0

w2

a

WGAN, Η#0.0

!1.0 !0.5 0.0 0.5 1.00.0

0.5

1.0

1.5

2.0

w2

a

WGAN, Η#0.5

!1.0 !0.5 0.0 0.5 1.00.0

0.5

1.0

1.5

2.0

w2

a

WGAN, Η#0.25

Decreasing, and only instantaneously non-decreasing energy function: Distance from

equilibrium

REFERENCES 1. H. K Khalil. Non-linear Systems. Prentice-Hall, New Jersey, 1996. 2. L. Metz, et al., Unrolled generative adversarial networks. (ICLR 2017) 3. M. Heusel et al., GANs trained by a TTUR converge to a local Nash

equilibrium (NIPS 2017) 4. I. J. Goodfellow et al., Generative Adversarial Networks (NIPS 2014)

An increasingly popular class of generative models — models that “understand” data to output new random data. Formally, GANs learn distribution over the data.

GENERATIVE ADVERSARIAL NETWORKS (GANS)

3.

Attend our Oral Presentation! WED DEC 6TH 11:05 -- 11:20 AM

Hall C

SIMULTANEOUS GRADIENT DESCENT WGAN

CONCLUSION

THEORETICAL ANALYSIS OF GAN DYNAMICS USING NON-LINEAR SYSTEMS: despite lack of concave-convexity, GANs are locally stable under right conditions — perhaps why they work so well in practice! Regularizer can provably enhance local stability.

LINEARIZATION THEOREM: The equilibrium is locally exponentially stable iff Jacobian at equilibrium has eigenvalues with strictly negative real parts.

(even though the discriminator is not trained to optimality!)

We assume at equilibrium, these locally convex functions are locally strongly convex — implying a locally unique equilibrium.

Relaxation: Zero eigenvalues must correspond to “flat” directions within a subspace of equilibria.

strongly convex

Key Lemma: Matrices of this form have eigenvalues with strictly negative real parts despite zero block!

!1.0 !0.5 0.0 0.5 1.00.0

0.5

1.0

1.5

2.0

w2

a

GAN, Η#0.0

GAN dynamics for a small system learning a uniform distribution.

Discriminator

Gen

erat

or

2. Provides foresight to the generator: When norm of discriminator’s gradient is minimized, discriminator cannot improve itself much to outdo generator.

3. Similar to 1-unrolled [2] approximation of the pure minimization problem.

There exist simultaneous gradient descent WGAN systems which do not necessarily converge to the equilibrium.

OPEN QUESTIONS

•Analyze other objectives and optimization techniques: unrolled GANs, f-GANs …

•Relaxing some assumptions: Non-realizability: when discriminator is not linear in parameters? Without strong curvature? Intuitively: slower convergence, but still stable… •Global convergence, at least for linear discriminator and generator! •When do equilibria satisfying our conditions exist?

Other proofs [3,4] require discriminator to be trained more often — effectively, closer to optimality, bringing system closer to pure minimization.

Global equilibrium is realizable.

1. Makes Jacobian “more stable”

The Achilles’ heel of deep learning · Explaining and harnessing adversarial examples, Goodfellow et al., ICLR 2015. Adversarial examples are “blind spots”. Robustness vs accuracy

Generative adversarial networks...Generative adversarial nets. NeurIPS. Longer version Goodfellow, (2016). NIPS 2016 Tutorial: Generative Adversarial Networks. Extra Radford et al,

Generative Adversarial Nets - cosy.sbg.ac.atheld/teaching/wiss_arbeiten/slides_19-2… · paper "Generative Adversarial Networks" from Ian Goodfellow et al. (2014) [2]. Schuiki Johannes,

Reading: I. Goodfellow et al., Generative adversarial ...web.cecs.pdx.edu/~mm/MachineLearningFall2018/GANs.pdfGenerative Adversarial Networks Reading: I. Goodfellow et al., Generative

CS230 Deep Learningcs230.stanford.edu/projects_winter_2019/reports/15782825.pdf · Generative Adversarial Networks (GANs) [Goodfellow et al, 2014; Isola et al, 2017] and Variational

Generative Adversarial Networks (GANs) - Ian Goodfellow · 2020. 9. 24. · Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Presentation at Berkeley

Adversarial Examples - Ian Goodfello · 09/08/2015  · - “Explaining and Harnessing Adversarial Examples.” Goodfellow et al., ICLR 2014. - “Distributional Smoothing by Virtual

What's next for adversarial ML? - floriantramer.com · Joint work with Gili Rusak, Giancarlo Pellegrino and Dan Boneh ... dog cat bird (Goodfellow 2016) Adversarial Training Labeled

Generative Adversarial Networks (GANs) - Ian Goodfellow, OpenAI

Poster: SirenAttack: Generating Adversarial Audio for End ... Generating Adversarial... · [2]I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial

Generative Adversarial Networks (GANs) · where D is our discriminator, G is our generator, and and are their corresponding parameters. [ 2 ] E. A. Goodfellow, Ian, Generative Adversarial

Imperial College Londonspiral.imperial.ac.uk/bitstream/10044/1/61341/2/MedIA2018... · Web viewGenerative adversarial networks (GANs) (Goodfellow et al., 2014) have also been adopted

Adversarial Examples and Adversarial Training...(Goodfellow 2016) In this presentation • “Intriguing Properties of Neural Networks” Szegedy et al, 2013 • “Explaining and

Generative adversarial networksGenerative adversarial networks 1 Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio

Least Squares Generative Adversarial Networks...Generative Adversarial Networks (GANs) were pro-posed by Goodfellow et al. [6], who explained the the-ory of GANs learning based on

Adversarial Examples and Adversarial Training · (Goodfellow 2016) In this presentation • “Intriguing Properties of Neural Networks” Szegedy et al, 2013 • “Explaining and

Generative Adversarial Networks - Ian Goodfellow...Deep Learning Workshop, ICML 2015 --- Ian Goodfellow In today’s talk… • “Generative Adversarial Networks” Goodfellow et

Generative adversarial networks - University of Torontodtarlow/pos14/talks/goodfellow.pdfGenerative adversarial networks 1 Ian Goodfellow Jean ... • State-of-the-art general purpose

Unsupervised Learning – Generative Models (PixelRNNs, PixelCNNs) – Generative ... · 2019. 3. 28. · Generative Adversarial Networks Ian Goodfellow et al., “Generative Adversarial

Topic 10. Detecting Out-of-Distributionalinlab.kaist.ac.kr/resource/AI602_Lec06_Adversarial_examples.pdf · *source: Goodfellow et al., Explaining and Harnessing Adversarial Examples,

Generative Adversarial Networksmadhavan/courses/aml2020/lectures/... · 2020. 11. 19. · 1."Generative Adversarial Networks"(Nov 2020) by Goodfellow etal: Communications of the ACM,

Adversarial Examples and Adversarial Trainingc4dm.eecs.qmul.ac.uk/horse2016/HORSE2016_Goodfellow.pdf• “Explaining and Harnessing Adversarial Examples” Goodfellow et al 2014 •

Generating Adversarial Examples with Adversarial NetworksGenerative Adversarial Networks (GANs)Goodfellow et al. have achieved visually appealing results in both im-age generation

THE CURSE OF DIMENSIONALITY AND IMAGE RECOGNITION€¦ · Image Source: ‘Explaining and Harnessing Adversarial Examples’, ICLR 2015, Goodfellow, Shlens, Szegedy • Attack above

Adversarial Examples and Adversarial Training · Adversarial Examples and Adversarial Training Ian Goodfellow, OpenAI Research Scientist Guest lecture for CS 294-131, UC Berkeley,

MetricGAN: Generative Adversarial Networks based Black-box ...proceedings.mlr.press/v97/fu19b/fu19b.pdf · Generative adversarial networks (GANs) (Goodfellow et al., 2014) has shown

Adversarial DefenseRecap Goodfellow et al, 2015 Untargeted Attack Recap Younis et al, 2019 Targeted Attack Outline Adversarial Defense Defense Strategies Adversarial Training …

Collaborative Sampling in Generative Adversarial … › pdf › 1902.00813.pdfGenerative Adversarial Networks Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) consist

Coupled Generative Adversarial Networks29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain. arXiv:1606.07536v2 [cs.CV] 20 Sep 2016 We apply CoGAN

Adversarial Approaches to Bayesian Learning and Bayesian ... · Learning and Bayesian Approaches to Adversarial Robustness Ian Goodfellow, OpenAI Research Scientist NIPS 2016 Workshop

Generative Adversarial Networks (GANs) Adversarial Networks (GANs) Hossein Azizpour Most of the slides are courtesy of Dr. Ian Goodfellow (Research Scientist at OpenAI) and from his

Defending Against Adversarial Examples - Machine Learning and … · Ian Goodfellow, Staff Research Scientist, Google Brain NIPS 2017 Workshop on Machine Learning and Security (Goodfellow

Vulnerability of machine learning models to …petra/slides/hora2016.pdfWorks on adversarial examples III. Explaining and Harnessing Adversarial Examples 2015,Goodfellow et al. linear

Defending Against Adversarial Attackscbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Introduction to Adversarial Examples Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy:

Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial