Upload File

summary of several autoencoder models - github pages€¦ · adversarial autoencoders alireza...

  • Home
  • Documents
  • Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial
34
Summary Of Several Autoencoder models Presentor: Ji Gao Department of Computer Science, University of Virginia https://qdata.github.io/deep2Read/

Upload: others

Post on 20-May-2020

10 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

SummaryOfSeveralAutoencodermodels

Presentor:JiGao

DepartmentofComputerScience,UniversityofVirginiahttps://qdata.github.io/deep2Read/

Page 2: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

List

• AdversarialAutoencoders• PixelGAN Autoencoders• GeneratinganddesigningDNAwithdeepgenerativemodels• FeedbackGAN(FBGAN)forDNA:aNovelFeedback-LoopArchitectureforOptimizingProteinFunctions• AutoregressiveGenerativeAdversarialNetworks

Page 3: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

AdversarialautoencodersAlireza Makhzani, JonathonShlens,Navdeep Jaitly, IanGoodfellow,BrendanFrey

• Useadversariallearningintrainingautoencoders

Page 4: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Autoencoders

• Autoencoder

• Decoder=Generator:Startfromaprior(oftennormaldistribution),producesample

Page 5: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Autoencoders

WhyDecoderwork?• Anydistributionin𝑑 dimensioncanbegeneratedbyasufficientlycomplicatedfunctionon𝑑 normallydistributionalvariables.

Page 6: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Whynotdirectlyoptimizedecoder?

• Ifdirectlyoptimizedecoderviasampling,itwilltakeexponentiallynumberofsamples(Andalsoexponentiallyparameters)• Alotofthesamplingareuseless,foraX,weonlyneedthepartofzthatarelikelytoproduceX• FindmostlikelyztoproduceXcansavehugeamountoftimeandmaketheprocesstractable

Page 7: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

VariationalAutoencoder

𝐷#$(𝑄(𝑧)||𝑃(𝑧|𝑥)) = 𝐸.~0[log𝑄 𝑧 − log𝑃(𝑧|𝑥)]

= 𝐸.~0[log𝑄 𝑧 − log𝑃 𝑥 𝑧 − log𝑃 𝑧 + log𝑃(𝑥)]

Latentvariable𝑧~𝑃(𝑧) Ifwesample𝑄(𝑧) toapproximate𝑃(𝑥),wehave

log𝑃(𝑥) − 𝐷#$(𝑄(𝑧)||𝑃(𝑧|𝑥)) = 𝐸.~0[log𝑃 𝑧|𝑥 ] − 𝐷#$(𝑄(𝑧)||𝑃(𝑧))

Bayesian

Reasonabletolet𝑄(𝑧) conditionedonx.

Wehave:log𝑃(𝑥) ≥ 𝐸.~0[log𝑃 𝑧|𝑥 ] − 𝐷#$(𝑄(𝑧|𝑥)||𝑃(𝑧))

Variationalbound

Page 8: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

VariationalAutoencoderlog𝑃(𝑥) ≥ 𝐸.~0[log𝑃 𝑧|𝑥 ] − 𝐷#$(𝑄(𝑧|𝑥)||𝑃(𝑧))InVAE,welet𝑄 𝑧 𝑥 = 𝑁(𝑧|𝜇 𝑥;𝜃 , Σ(𝑥; 𝜃))

Inthiscase:

Samplexandz,wehave

Page 9: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Posterier

• GaussianPosterier

Page 10: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

VariationalAutoencoder

FromTutorialonVariationalAutoencoders https://arxiv.org/abs/1606.05908

Page 11: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Adversarialautoencoder

• VAEworkson

• 𝐷#$(𝑄 𝑧 𝑥 ||𝑃 𝑧 ) termcanbeoptimizedinadversarialtraining• Trainrepeatedlyintwosteps:1.Maximize𝐸.~0[log𝑃 𝑥|𝑧 ]2.Minimizethedistancebetween𝑄(𝑧|𝑥) and𝑃(𝑧)

log𝑃(𝑥) ≥ 𝐸.~0[log𝑃 𝑥|𝑧 ] − 𝐷#$(𝑄(𝑧|𝑥)||𝑃(𝑧))

Page 12: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Freedomofchoosingq()

• ComparetoVAE,inthisformitcanbeoptimizedusingseveraldifferentways:• 1.Deterministic:𝑞 𝑧 𝑥 isadeterministicfunctiononx• 2.Gaussianposterior:𝑄 𝑧 𝑥 = 𝑁 𝑧 𝜇 𝑥; 𝜃 , Σ 𝑥; 𝜃 similartoVAE.Canusethesamereparameterization• 3.Universalapproximator posterior,𝑞 𝑧 𝑥, 𝜂 = 𝛿(𝑧 − 𝑓(𝑥, 𝜂))

Page 13: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Adversarialautoencoderperformance

Page 14: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Loglikelihood

Page 15: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Supervisedlearning

• Fullysupervisedlearningtogeneratesamplesinaparticularway

Page 16: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Semi-supervisedlearning

• 2adversarialnets:Onewithcategoricaldata• Traininthreephases:• 1.Reconstructionphase• 2.Regularizationphase• 3.Semi-supervisedphase

Page 17: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

PixelGAN AutoencodersAlirezaMakhzani,BrendanFrey

Page 18: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

PixelGAN Autoencoders

• UsePixelCNN asthegenerativepath• PixelCNN conditionedonq(z|x)

Page 19: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Categoricalprior

Page 20: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Experiment

Page 21: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

GeneratinganddesigningDNAwithdeepgenerativemodelsNathanKilloran,LeoJ.Lee,AndrewDelong,DavidDuvenaud,BrendanJ.Frey

• 2017• ThreeapproachestogenerateDNAsequence:• 1.GAN• 2.Activationmaximization(DeepDream)• 3.Ajointof1and2

Page 22: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

GANondiscreteoutput

• DNAsequenceisdiscrete,similartoNLPtask• WGAN-GPcangeneratethesequenceinthedirectway:

LetGANdirectlyoutputone-hotcharacterembeddings fromalatentvectorwithoutanydiscretesamplingstep.Softmax directlypassedtocritic.

Page 23: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

GANonDNA

• UsesuchmethodonDNA:

Page 24: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

ActivationMaximization

• ThemethodisactuallyDeepDream:• Startfromsamplex,makeittargetatacertainpropertyt(output)• 𝑥 → 𝑥 + 𝜖∇E𝑡• Worksoncontinuouscase,soneedtorelaxdiscretesymbolsintocontinuouscase

Page 25: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Jointmethod

• UseGANtogeneratesample• Useactivationmaximizationtooptimizeasampletocertainproperties

Page 26: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Experiment:Motif

• Samplesequencestunedtohaveahighpredictorscore

Page 27: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Experiment

Page 28: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

FeedbackGAN(FBGAN)forDNA:aNovelFeedback-LoopArchitectureforOptimizingProteinFunctionsAnvita Gupta,JamesZou

• 2018• Target:DesignDNAautomaticallyfollowingsomeproperties

Page 29: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

FeedbackGAN

• (a)WGAN-GPasgenerator• (b)Analyzer:supposetobeanyfunction• Ratethegeneratedsamples• Markthetopsortedsamplesasrealsamples

• (c)Feedbackscheme• Sendthetopsortedsamplebacktothediscriminator

Page 30: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Evaluation

• Beforetraining,3.125%ofsequencesinitially followedthecorrectgenestructure• Aftertraining,77.08%ofsampledsequencescontainedthecorrectgenestructure

Page 31: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

AutoregressiveGenerativeAdversarialNetworksYasin Yazici,Kim-HuiYap,StefanWinkler

• ICLR18Workshop

Page 32: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

ARGAN

• ReplacediscriminatorintoaCNN+Autoregressivemodel• Motivation:anautoregressivemodelwouldmodelthefeaturedistributionbetterthanfullyconnectedlayers

Page 33: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

S-ARGANandC-ARGAN

Page 34: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial

Result


Generative Probabilistic Novelty Detection with ...vision.csee.wvu.edu/GPND.pdf · Generative Probabilistic Novelty Detection with Adversarial Autoencoders Stanislav Pidhorskyi, Ranya

Epileptic Seizure Forecasting With Generative Adversarial ...clustering,Gaussianmixturemodels,HiddenMarkovModels and autoencoders [13], [14]. Most of these unsupervised learning techniques

Structured Generative Adversarial Networks · Among various DGMs, variational autoencoders (VAEs) and generative adversarial networks (GANs) can be trained unsupervisedly to map a

Code-Bridged Classifier (CBC): A Low or Negative Overhead ... · autoencoders are employed to remove the adversarial pertur-bation and reconstruct a clean input. In [15] distillation

Reforming Generative Autoencoders via Goodness-of-Fit ...auai.org/uai2018/proceedings/papers/356.pdf · mechanisms, and include adversarial methods, variational methods, MCMC based

Ladder Variational Autoencoders

Sinkhorn AutoEncoders - GitHub Pages · of the data. Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) belong to the former class, by learning to transform noise into

Manifolds, Autoencoders and Generative Adversarial Networks€¦ · Manifolds, Autoencoders and Generative Adversarial Networks Volker Tresp Summer 2019 1File Size: 4MBPage Count:

Adversarial autoencoders for novelty detection · Adversarial Autoencoders for Novelty Detection: An adversarial autoencoder (AAE) is a proba-bilistic autoencoder that uses the recently

Unsupervised Representation Learning with Prior-Free and Adversarial …min.sjtu.edu.cn/files/papers/2018/Conference/2018-IICME... · 2019-08-22 · adversarial learning, autoencoders

Adversarial Defense based on Structure-to-Signal Autoencoders€¦ · Adversarial Defense based on Structure-to-Signal Autoencoders Joachim Folz Sebastian Palacio? Joern Hees Andreas

Learning 2018 Adversarial and Secure - TUM · deep learning systems using adversarial examples." arXiv preprint (2016). Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy

Generative Appearance Flow: A Hybrid Approach for Outdoor ... · success of generative adversarial networks (GANs) and autoencoders. While most work is focused on simply generating

Robust Anomaly Detection in Images using Adversarial Autoencoders · Robust Anomaly Detection in Images using Adversarial Autoencoders Laura Beggel 1;2( ), Michael Pfei er , and Bernd

Lecture 22: Adversarial Image, Adversarial Training, Variational Autoencoders… · 2018-11-08 · Adversarial Images Adversarial Training Autoencoder VAE GAN Conclusions Lecture

Adversarial Latent Autoencoders...possible with SyleGAN alone, while maintaining the same level of visual detail. Acknowledgments This material is based upon work supported by the

Reforming Generative Autoencoders via Goodness-of-Fit … · 2018-12-22 · The original generative adversarial network (GAN) (Goodfellow et al., 2014) sparked a surge in generative

PERSPECTIVES · design in computational chemistry and to explore the open- ended chemical space, autoencoders and generative adversarial networks have emerged as powerful tools to

Deep Generative Models: GANs and VAE · Deep convolutional generative adversarial networks Auxiliary classifier GANs Odena, A., Olah, C., & Shlens, J. (2016). Conditional image synthesis

Variational Autoencoders

Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the

PERCEPTUAL GENERATIVE AUTOENCODERS

rld,mnorouzi,shlens @google.com arXiv:1702.00783v1 [cs.CV ... · Google Brain {rld,mnorouzi,shlens}@google.com Abstract We present a pixel recursive super resolution model that synthesizes

THE CURSE OF DIMENSIONALITY AND IMAGE RECOGNITION€¦ · Image Source: ‘Explaining and Harnessing Adversarial Examples’, ICLR 2015, Goodfellow, Shlens, Szegedy • Attack above

autoencoders - ucbrise.github.io · adversarial models using the encodings. ... -Autoencoders work in the unsupervised setting, and since data is many orders of magnitude cheaper

LEARNING PRIORS FOR ADVERSARIAL AUTOENCODERS · Learning Priors for Adversarial Autoencoders Hui-Po Wang, Wei-Jan Ko and Wen-Hsiao Peng Department of Computer Science National Chiao

Sinkhorn AutoEncoders - UAIauai.org/uai2019/proceedings/papers/253.pdfof the data. Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) belong to the former class, by learning

Lecture 10: Autoencoders

Lecture 9: Unsupervised, Generative & Adversarial Networks · Lecture 9: Unsupervised, Generative & Adversarial Networks ... GENERATIVE & ADVERSARIAL NETWORKS - 20 o Autoencoders

A Two-Pronged Defense against Adversarial Exampleshchen/paper/meng2017-slides.pdfAdversarial examples p(x is panda) = 0.58 4 p(x is gibbon) = 0.99 [ICLR 15] Goodfellow, Shlens, and

Open-set Recognition with Adversarial Autoencoders · Open-set Recognition with Adversarial Autoencoders Ranya Almohsen, Stanislav Pidhorskyi, Gianfranco Doretto Introduction Problem

AutoEncoders& Kernels

Denoising Adversarial Autoencoders · Denoising Adversarial Autoencoders Antonia Creswell BICV Imperial College London Email: [email protected] Anil Anthony Bharath BICV Imperial College

Jon Shlens, Google Brain team Vijay Vasudevan, Irwan Bello, …rll.berkeley.edu/deeprlcoursesp17/docs/quoc_barret.pdf · 2017. 8. 20. · Vijay Vasudevan, Irwan Bello, Jon Shlens,

Autoencoders - Purdue University