arshadowgan: shadow generative adversarial network for...

8

ARShadowGAN: Shadow Generative Adversarial Network for Augmented Reality in Single Light Scenes Daquan Liu 1 , Chengjiang Long 2* , Hongpan Zhang 1 , Hanning Yu 1 , Xinzhi Dong 1 , Chunxia Xiao 1,3,4* 1 School of Computer Science, Wuhan University 2 Kitware Inc., Clifton Park, NY, USA 3 National Engineering Research Center For Multimedia Software, Wuhan University 4 Institute of Artificial Intelligence, Wuhan University [email protected], {daquanliu,zhanghp,fishaning,dongxz97,cxxiao}@whu.edu.cn ∗This work was co-supervised by Chengjiang Long and Chunxia Xiao. 1

Upload: others

Post on 25-Aug-2020

17 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

ARShadowGAN: Shadow Generative Adversarial Network for Augmented Reality in Single Light Scenes

Daquan Liu1, Chengjiang Long2*, Hongpan Zhang1, Hanning Yu1, Xinzhi Dong1, Chunxia Xiao1,3,4*

1School of Computer Science, Wuhan University

2Kitware Inc., Clifton Park, NY, USA

3National Engineering Research Center For Multimedia Software, Wuhan University

4Institute of Artificial Intelligence, Wuhan University

[email protected], {daquanliu,zhanghp,fishaning,dongxz97,cxxiao}@whu.edu.cn

∗This work was co-supervised by Chengjiang Long and Chunxia Xiao. 1

Page 2: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

With the physically based rendering theory:

• It requires an inverse rendering process which is very expensive and challenging in practice.

• What’s worse, any inaccurate estimation may result in unreasonable virtual shadows.

We propose a data-driven based ARShadowGAN to directly generate virtual shadows without inverse

rendering. With the powerful generation ability of GAN, it is much more efficient and easier to use.

How to Automatically GenerateVirtual Shadows for AR?

2

Page 3: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

Shadow-AR Dataset

• Shadow-AR dataset contains total 3,000 image quintuples

Input Image Input Mask Mask-Real Object Mask-Real Shadow Target Image

3

Page 4: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

ARShadowGAN

Image

Encoder

vShadow

Decoder

ResNetblock

additionDiscrim

inato

r

RealFake

Atten

tion

Encoder

32x32x512

rShadow

Decoder

Occluder

Decoder

16x16x512

16x16x512

Refinement

Pooling

concatenation

⊕

⊕

Ⓒ

Ⓒ

Ⓒ

Ⓒ

4

Page 5: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

Attention Visualization

0.0

1.0

Input Image Input Mask Attn-Real Shadow Attn-Real Object 5

Page 6: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

Experimental Results

Input Image Pix2Pix Pix2Pix-Res ShadowGAN Mask-ShadowGAN ARShadowGAN GT6

Page 7: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

Shadow Generation

Input Image Input Mask Attn-Real Shadow Attn-Real Object Output Image

7

Page 8: ARShadowGAN: Shadow Generative Adversarial Network for ...graphvision.whu.edu.cn/papers/liudaquan2020_pres.pdf · ARShadowGAN: Shadow Generative Adversarial Network for Augmented

Paper QR Code:

http://www.chengjianglong.com/publications/ARShadowGAN_CVPR.pdf

Thank you!

This work was partly supported by Key Technological Innovation Projects of Hubei Province (2018AAA062), Wuhan

Science and Technology Plan Project (No. 2017010201010109), the National Key Research and Development Program of

China (2017YFB1002600), the NSFC (No. 61672390, 61972298). The corresponding author is Chunxia Xiao.

8

Semi-Supervised Generative Adversarial Hashing for Image Retrievalopenaccess.thecvf.com › ...Wang_Semi-Supervised_Generative_Adver… · Semi-Supervised Generative Adversarial Hashing

Generative adversarial nets

Generative Adversarial Networks (GANs) - Sciencesconf.org · Generative Adversarial Networks (GANs) Anirban Mukhopadhyay TU Darmstadt, Germany. Introduction Generative vs. Discriminative

ADVERSARIAL EXAMPLES FOR GENERATIVE MODELS

Generative Adversarial Nets - Semantic Scholargenerative adversarial text to image synthesis. Scott Reed, ZeynepAkata. ... Conditional Sequence Generative Adversarial Nets[J]. arXiv

Lec: Generative Adversarial Networks

GENERATIVE ADVERSARIAL SELF-IMITATION LEARNING

Collaborative Sampling in Generative Adversarial … › pdf › 1902.00813.pdfGenerative Adversarial Networks Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) consist

Hypothetical Exploration on Generative Adversarial Networks · Convolutional Neural Networks, Generative Adversarial Net- works, Laplacian Pyramid of Adversarial Networks. I. INTRODUCTION

Recurrent Generative Adversarial Networks for Compressive ...on-demand.gputechconf.com/.../s8197-recurrent-generative-adversarial-neural-networks...Xing, and John Pauly, ``Deep generative

Shadow Detection With Conditional Generative …openaccess.thecvf.com/content_ICCV_2017/papers/Nguyen_Shadow... · Shadow Detection with Conditional Generative Adversarial Networks

Perceptual Generative Adversarial Networks for Small ...openaccess.thecvf.com/content...Generative_Adversarial_CVPR_2017… · Perceptual Generative Adversarial Networks for Small

A (conditional) Generative Adversarial Network

Adversarial Distillation of Bayesian Neural Network Posteriorsproceedings.mlr.press/v80/wang18i/wang18i.pdf · 2.3. Generative Adversarial Networks Generative Adversarial Networks

EmotiGAN: Emoji Art using Generative Adversarial Networkscs229.stanford.edu/proj2017/final-reports/5244346.pdfA. Generative Adversarial Networks A Generative Adversarial Network[4]

Generative Adversarial Networks (GANs)

AnomiGAN: Generative Adversarial Networks forpsb.stanford.edu/psb-online/proceedings/psb20/Bae.pdf2.4. Generative Adversarial Networks GANs25 are designed to complement other generative

ARShadowGAN: Shadow Generative Adversarial Network for …openaccess.thecvf.com/content_CVPR_2020/papers/Liu_AR... · 2020-06-29 · ARShadowGAN: Shadow Generative Adversarial Network

Lecture 19: Generative Adversarial Networksrgrosse/courses/csc321_2018/readings/L19 GANs.pdfLecture 19: Generative Adversarial Networks Roger Grosse 1 Introduction Generative modeling

Generative Adversarial Networks (part 2)deeplearning.cs.cmu.edu/document/slides/bstriner_gans_part_2.pdf · Unrolled Generative Adversarial Networks Unrolled Generative Adversarial

COMP9444 Neural Networks and Deep Learning 9b. Generative Adversarial … · 2020. 7. 25. · COMP9444 20T2 Generative Adversarial Networks 13 Generative Adversarial Networks GAN

Generative Adversarial Imitation Learning

Generative adversarial networks...Generative adversarial nets. NeurIPS. Longer version Goodfellow, (2016). NIPS 2016 Tutorial: Generative Adversarial Networks. Extra Radford et al,

Wasserstein Generative Adversarial Networksproceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf · Wasserstein Generative Adversarial Networks Martin Arjovsky1 Soumith Chintala2

Generative Adversarial Networks, and Applications Adversarial Networks, and Applications Outline: •Generative Models vs Discriminative Models (Background) •Generative Adversarial

Lecture 9: Unsupervised, Generative & Adversarial Networks · Lecture 9: Unsupervised, Generative & Adversarial Networks ... GENERATIVE & ADVERSARIAL NETWORKS - 20 o Autoencoders

Generative Adversarial Networkstat.snu.ac.kr/mcp/Lecture10_GAN.pdf · 2019-11-11 · Generative Adversarial Network Generative adversarial network: Setup Let y = 1 if the data is

Lecture 7: Generative Adversarial Networks

Triple Generative Adversarial Netspapers.nips.cc/paper/6997-triple-generative-adversarial-nets.pdf · the labels. To address the problems, we present triple generative adversarial

Discriminator-Free Generative Adversarial Attack

Generative Adversarial Networks - HAW Hamburgubicomp/projekte/master... · Generative Adversarial Networks ... Outline 1. Basic idea 2. Generative Adversarial ... com/2017/01/generative-adversarial-networks-hot-topic-machine-learning

Generative Adversarial Transformers - Stanford University

Unsupervised Generative Adversarial Alignment

Conditioning of Generative Adversarial Networks · 2019-01-31 · Generative Adversarial Networks are: •Generative Models •Parametric –Latent Vector •Differentiable –Allow

Image Representation Learning with Generative Adversarial ... · Image Representation Learning with Generative Adversarial Networks Keywords machine learning, generative models, image