real-time image style transfercs229.stanford.edu/proj2017/final-posters/5140602.pdf · image style...

Real-TimeImageStyleTransferZhaoyouWang,zhaoyou@stanford.eduZhaohengGuo,zhaoheng@stanford.edu

MotivationMotivationImage style transfer is a long-standing problem that seeks to transfer the style of a reference style image onto another input picture. Our project has implemented two recent proposed image style transfer algorithms based on convolutional neural networks.

ModelNeuralalgorithm:1. ContentLoss

2.StyleLoss

3.JointLossFunction

4.Output:

Realtimestyletransfer:TrainanImageTransformNetwork

InputImage ImageTransformNetwork

ContentImage

StyleImage

JointLoss

Two epochs over the 80k Microsoft COCO dataset with batch size 4 (resize to 256x256). We use a pre-trained VGG16 network for the loss calculation and an image transformation network with 3 convolution layers, 5 residual layers and 3 deconvolution layers.

Style

Baseline

Real-time

Results

TrainingDetails

References1. L. A. Gatys, A. S. Ecker and M. Bethge.

A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015.

2. J. Johnson, A. Alahi and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694–711. Springer, 2016.

3. D. Ulyanov, A. Vedaldi and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.

Llc(yc, y) =

1

2

�

ij

(F lij(yc) � F l

ij(y))2

Neuralalgorithmresultswithdifferentstyleweights.

Comparisonoftwomethods:

El(y, ys) =1

4N2l M

2l

!

ij

(Glij(y)−Gl

ij(ys))2

Ls(y, ys) =!

l

wlEl

Glij =

�

k

F likF l

jk

y = arg miny

LJ(yc, ys, y)

LJ

LJ(yc, ys, y) = �Lc(yc, y) + �Ls(ys, y)

W � = arg minW

LJ(yc, ys, fW (yc))

FuturePlans1. Addphotorealisticlossandsemantic

segmentationtoreducelocaldistortioninoriginalneuralalgorithm

2. Implementrealtimealgorithmforphotostyletransfer

3. Trainastyleextractionnetworktorealizearbitraryimagestyletransfer

DiscussionImportantdetailsforfinalresults:1. Inputimagesneedpreprocessing

(subtractmeanvalueandnormalize)tomakelossnetworkworkproperly.

2. Projectingthevaluesofgeneratedimagestorange[0,255]togetridofsomenoisyartifacts.

3. Duringthetrainingprocess,allowingtheimagetransformationnetworktoworkinrange[0,255]generatesmuchbetterresultsthanthan[0,1].

SophiaLu,sophialu@stanford.edu