InSituNet: Deep Image Synthesis for Parameter Space Exploration

of Ensemble Simulations

Wenbin He, Junpeng Wang, Hanqi Guo, Member, IEEE, Ko-Chih Wang, Han-Wei Shen, Member, IEEE,

Mukund Raj, Youssef S. G. Nashed, and Tom Peterka, Member, IEEE

Abstract— We propose InSituNet, a deep learning based surrogate model to support parameter space exploration for ensemble sim-ulations that are visualized in situ. In situ visualization, generating visualizations at simulation time, is becoming prevalent in handlinglarge-scale simulations because of the I/O and storage constraints. However, in situ visualization approaches limit the flexibility ofpost-hoc exploration because the raw simulation data are no longer available. Although multiple image-based approaches have beenproposed to mitigate this limitation, those approaches lack the ability to explore the simulation parameters. Our approach allows flexi-ble exploration of parameter space for large-scale ensemble simulations by taking advantage of the recent advances in deep learning.Specifically, we design InSituNet as a convolutional regression model to learn the mapping from the simulation and visualizationparameters to the visualization results. With the trained model, users can generate new images for different simulation parametersunder various visualization settings, which enables in-depth analysis of the underlying ensemble simulations. We demonstrate theeffectiveness of InSituNet in combustion, cosmology, and ocean simulations through quantitative and qualitative evaluations.

Index Terms—In situ visualization, ensemble visualization, parameter space exploration, deep learning, image synthesis.

1 INTRODUCTION

Ensemble simulations [64] have been playing an increasingly impor-tant role in various scientific and engineering disciplines, such as com-putational fluid dynamics, cosmology, and weather research. As thecomputational power of modern supercomputers continues to grow, en-semble simulations are more often conducted with a large number ofparameter settings in high spatial and/or temporal resolutions. Despitethe advances in accuracy and reliability of simulation results, how-ever, two challenges have emerged: (1) I/O bottleneck for the move-ment of the large-scale simulation data and (2) effective explorationand analysis of the simulation parameters. In situ visualization [5, 38],which generates visualization at simulation time and stores only thevisualization results (that are much smaller than the raw simulationdata [1, 2]) for post-hoc analysis, addresses the first challenge to someextent. However, it also limits the flexibility of post-hoc explorationand analysis, because the raw simulation data are no long available.

This study focuses on improving scientists’ ability in exploring thein situ visualization results of ensemble simulations and extendingtheir capability in investigating the influence of different simulationparameters. Several pioneering works have been proposed to facili-tate post-hoc exploration of in situ visualization results. For example,the Cinema framework [1, 2] visualized the simulation data from dif-ferent viewpoints in situ and collected images to support post-hoc ex-ploration. The volumetric depth images [21, 22] stored ray segmentswith composited color and opacity values to enable post-hoc explo-ration of arbitrary viewpoints for volume rendering. However, theseapproaches focus more on extending the capability to explore the vi-sual mapping parameters (e.g., transfer functions) and view parame-ters (e.g., view angles) and have little consideration of the simulationparameters, which are important in studying ensemble simulations.

Simulation parameter space exploration is not trivial, because therelationship between the simulation parameters and outputs is often

• Wenbin He, Junpeng Wang, Ko-Chih Wang, and Han-Wei Shen are with the

Department of Computer Science and Engineering, The Ohio State

University. E-mail: {he.495, wang.7665, wang.3182, shen.94}@osu.edu.

• Hanqi Guo, Mukund Raj, Youssef S. G. Nashed, and Tom Peterka are with

the Mathematics and Computer Science Division, Argonne National

Laboratory. E-mail: {hguo, mraj, ynashed, tpeterka}@anl.gov.

Manuscript received xx xxx. 201x; accepted xx xxx. 201x. Date of Publication

xx xxx. 201x; date of current version xx xxx. 201x. For information on

obtaining reprints of this article, please send e-mail to: reprints@ieee.org.

Digital Object Identifier: xx.xxxx/TVCG.201x.xxxxxxx

highly complex. The majority of existing simulation parameter spaceexploration approaches [14, 65] resorted to visualizing a set of sim-ulation parameters and outputs simultaneously and revealing the cor-respondence between the parameters and outputs through visual link-ings. However, these approaches often depend on the raw simulationdata that might not be available for large-scale ensemble simulations.Moreover, these approaches have limited ability in inferring simula-tion outputs with respect to new parameters. Hence, extra simulationshave to be conducted for new parameters, which cost enormous com-putational resources for most scientific simulations.

In this work, we propose InSituNet, a deep learning based surro-gate model to support parameter space exploration for ensemble sim-ulations that are visualized in situ. Our work is based on the observa-tion that images of high accuracy and fidelity can be generated withdeep neural networks for various image synthesis applications, suchas super-resolution [18, 32, 36], inpainting [46, 68], texture synthe-sis [24, 71], and rendering [6, 19]. Specifically, we train InSituNetto learn the end-to-end mapping from the simulation, visual mapping,and view parameters to visualization images. The trained model en-ables scientists to interactively explore synthesized visualization im-ages for different simulation parameters under various visualizationsettings without actually executing the expensive simulations. Our ap-proach consists of three major steps.

1. In situ training data collection from ensemble simulationsGiven ensemble simulations conducted with different simulationparameters, we visualize the generated simulation data in situwith various visual mapping and view parameters. The result-ing visualization images and the corresponding parameters arecollected and used for the offline training of InSituNet.

2. Offline training of InSituNet Given the parameters and imagepairs, we train InSituNet (i.e., a convolutional regression model)with cutting-edge deep learning techniques on image synthesisto map simulation, visual mapping, and view parameters to visu-alization images directly.

3. Interactive post-hoc exploration and analysis With thetrained InSituNet, we build an interactive visual interface that en-ables scientists to explore and analyze the simulation from twoperspectives: (1) inferring visualization results for arbitrary pa-rameter settings within the parameter space with InSituNet’s for-ward propagations and (2) analyzing the sensitivity of differentparameters with InSituNet’s backward propagations.

We demonstrate the effectiveness of the proposed approach in com-bustion, cosmology, and ocean simulations, and compare the predictedimages of InSituNet with the ground truth and alternative methods. In

addition, we evaluate the influence of different hyperparameters of In-SituNet (e.g., the choice of loss functions and the network architec-tures) and provide guidance in configuring the hyperparameters. Insummary, the contributions of this paper are threefold:

• A deep image synthesis model (i.e., InSituNet) that enables post-hoc parameter space exploration of ensemble simulations

• An interactive visual interface to explore and analyze the param-eters of ensemble simulations with the trained InSituNet

• A comprehensive study revealing the effects of different hyper-parameters of InSituNet and providing guidance for applying In-SituNet to other simulations

2 RELATED WORK

In this section, we review related work in image-based in situ visu-alization, parameter space exploration of ensemble simulations, anddeep learning for visualization.

2.1 Image-Based In Situ Visualization

Based on the output, in situ visualization can be categorized intoimage-based [1, 2], distribution-based [20], compression-based [17,35], and feature-based [12] approaches. We regard our work as animage-based approach, which visualizes simulation data in situ andstores images for post-hoc analysis. Tikhonova et al. [59–61] gener-ated images of multiple layers in situ to enable the adjustment of trans-fer functions in post-hoc analysis. Frey et al. [22] proposed volumetricdepth images, a compact representation of volumetric data that can berendered efficiently with arbitrary viewpoints. Fernandes et al. [21]later extended volumetric depth images to handle time-varying volu-metric data. Biedert and Garth [8] combined topology analysis andimage-based data representation to preserve flexibility for post-hoc ex-ploration and analysis. Ahrens et al. [1, 2] proposed Cinema, a frame-work that stores visualization images in situ and performs post-hocanalysis via exploration and composition of those images.

Compared with these approaches, our work supports not only theexploration of various visual mapping and view parameters but alsothe creation of visualizations under new simulation parameters withoutactually running the simulation.

2.2 Parameter Space Exploration

The existing parameter space exploration works for ensemble simula-tions can generally be reviewed from two perspectives, the adoptedvisualization techniques and the objective of parameter space explo-ration. Visualization techniques that designed for high-dimensionaldata are often borrowed to visualize the parameter space of ensem-ble simulations, as the simulation parameters are typically treated asmultidimensional vectors. These techniques include but are not lim-ited to: parallel coordinate plots [44, 65], radial plots [14–16], scat-ter plots [39, 45, 58], line charts [9], matrices [48], and glyphs [10].For the objectives of parameter space exploration, we believe the sixtasks sorted out by Sedlmair et al. [55] could best summarize the liter-ature, which are optimization [62], partitioning [7, 65], filtering [47],outliers [47, 49], uncertainty [9, 11], and sensitivity [9]. We refer theinterested readers to the work of Sedlmair et al. [55] for the detaileddefinition of each task, as well as the example visualization works.

The aforementioned parameter visualization techniques and analy-sis tasks mostly focus on a limited number of simulation inputs andoutputs collected from ensemble runs. In this paper, we train a surro-gate model to extend our study to arbitrary parameter settings withinthe parameter space, even if the simulations were not executed withthose settings. In addition, our approach is incorporated with in situ vi-sualization, which is widely used in large-scale ensemble simulations.

2.3 Deep Learning for Visualization

The visualization community has started to incorporate deep learningin visualization research. For example, Hong et al. [30] used longshort-term memory [29] to estimate access pattern for parallel parti-cle tracing. Han et al. [26] used autoencoders [52] to cluster stream-lines and streamsurfaces. Xie et al. [67] used neural network embed-dings to detect anomalous executions in high performance computing

applications. Berger et al. [6] proposed a deep learning approach toassist transfer function design using generative adversarial networks(GANs) [25], which is closely related to our approach. Specifically,we focus on parameter space exploration of ensemble simulations in-stead of transfer function design for volume rendering.

Our work is related to deep learning based image synthesis, whichhas been used in various applications, including super-resolution [18,32, 36], denoising [68, 70], inpainting [46, 68], texture synthesis [24,71], text-to-image synthesis [50], style transfer [23,32,72], and render-ing [6, 19]. We investigate and combine different state-of-the-art deeplearning techniques on image synthesis (e.g., perpetual losses [32, 36]and GANs [25]) to improve the quality of our image synthesis results.

3 OVERVIEW

Ensemble

Simulations

Image Database

(Training Data)

training

offline

Interactive

VisualizationInSituNet

exploring

parameterscreating image

database in situ

Fig. 1. Workflow of our approach. Ensemble simulations are conductedwith different simulation parameters on supercomputers, and visualiza-tion images are generated in situ for different visual mapping and viewparameters. The generated images and the parameters are collectedinto an image database. A deep image synthesis model (i.e., InSituNet)is then trained offline based on the collected data, which is later usedfor parameter space exploration through an interactive visual interface.

Figure 1 provides the workflow of our approach, which consists ofthree major components. First, given ensemble simulations conductedwith different simulation parameters, we visualize the generated simu-lation outputs in situ with different visual mapping and view parame-ters on supercomputers. The three groups of parameters—simulation,visual mapping, and view parameters—along with the correspondingvisualization results (i.e., images) are collected to constitute an imagedatabase (Section 4). Second, with the collected data pairs betweenparameters and the corresponding images, we train InSituNet to learnthe end-to-end mapping from the simulation inputs to the visualiza-tion outputs (Section 5). To improve the accuracy and fidelity of thegenerated images, we use and combine different state-of-the-art deeplearning techniques on image synthesis. Third, with the trained InSi-tuNet, we build an interactive visual interface (Section 6) to exploreand analyze the parameters from two aspects: (1) predicting visualiza-tion images interactively for arbitrary simulation, visual mapping, andview parameters within the parameter space and (2) investigating thesensitivity of different input parameters to the visualization results.

4 IN SITU TRAINING DATA COLLECTION

simulationparameters

simulation data

visualizationimages

run ensemblesimulations

visualize with selectedparameters in situ

visual mappingparameters

viewparameters

Fig. 2. Our in situ training data collection pipeline. Simulation data, gen-erated with different simulation parameters, are visualized in situ withdifferent visual mapping and view parameters. The in situ visualizationgenerates a large number of images, which are collected along with thecorresponding parameters for the training of InSituNet offline.

Figure 2 illustrates our in situ training data collection pipeline.Given ensemble simulations conducted with different simulation pa-rameters, we perform in situ visualization with a desired set of visualmapping parameters (e.g., isosurfaces extraction with a set of isoval-ues) and different view parameters (e.g., viewpoints). We denote an in-stance of simulation, visual mapping, and view parameters as Psim, Pvis,and Pview, respectively, which corresponds to a visualization image I.The parameters (highlighted in green in Figure 2) and the correspond-ing visualization images (highlighted in blue in Figure 2) constitutedata pairs, which will be stored and used to train InSituNet. InSituNet

learns a function F that maps the three groups of parameters to thecorresponding visualization image, which can be defined as

F(Psim,Pvis,Pview)→ I, (1)

so that it can predict visualization images for unseen parameters. Inthe following, we discuss the three groups of parameters in detail.

Simulation parameters Psim are represented as a vector with oneor more dimensions, and the value range of each dimension is definedby scientists. By sweeping the parameters within the defined ranges,ensemble simulations are conducted to generate the ensemble data.

Visual mapping parameters Pvis are predefined operations to vi-sualize the generated simulation data, such as pseudo-coloring withpredefined color schemes. Note that we limit the users’ ability in se-lecting arbitrary visual mappings to produce and store fewer images.

View parameters Pview are used to control the viewpoints that theimages are created from. In this work, we define the viewpoints bya camera rotating around the simulation data, which is controlled byazimuth θ ∈ [0,360] and elevation φ ∈ [−90,90]. For panning andzooming, we resort to image-based operations (i.e., panning and resiz-ing the images) as proposed in [2]). To train a deep learning modelthat can predict visualization images for arbitrary viewpoints, we sam-ple the azimuth and elevation and generate images from the sampledviewpoints. Based on our study, we found that taking 100 viewpointsfor each ensemble member is sufficient to train InSituNet.

With the specified values for the three groups of parameters, wegenerate the corresponding visualization images. Our work uses RGBimages compressed to the portable network graphics (PNG) format in-stead of more sophisticated image formats, such as volumetric depthimages [21, 22] or explorable images [59–61], for two reasons. First,the benefits of using those sophisticated image formats, such as sup-porting changing of viewpoints, can be achieved by InSituNet trainedon the RGB images. Second, RGB images are more generally applica-ble for various visualizations and more easily to be handled by neuralnetworks compared with those sophisticated image formats.

5 INSITUNET ARCHITECTURE AND TRAINING

input

parameters

Regressor

Discriminator adversarial loss

ground truth

prediction

Feature

Comparator

feature

reconstruction

loss

Training Inferencenew input

parameters

Trained

Regressor

prediction

Fig. 3. Overview of InSituNet, which is a convolutional regression modelthat predicts visualization images from input parameters. During train-ing, the regression model is trained based on the losses computed withthe assist of a pretrained feature comparator and a discriminator.

Figure 3 illustrates the training and inference pipelines of InSituNet.In the training stage, InSituNet consists of three subnetworks: a regres-sor, a feature comparator, and a discriminator. The regressor Rω is adeep neural network (defined by a set of weights ω) modeling the func-tion that maps input parameters to visualization images as defined inEquation 1. To train a regressor that can generate images of high fi-delity and accuracy, we introduced the feature comparator F and thediscriminator Dυ to compute losses by comparing the predicted andthe ground truth images. The feature comparator is a pretrained neu-ral network whose convolutional kernels are used to extract and com-pare image features (e.g., edges, shapes) between the predicted and theground truth images to obtain a feature reconstruction loss. The dis-criminator Dυ is a deep neural network whose weights υ are updatedduring training to estimate the divergence between the distributions ofthe predicted and the ground truth images. The divergence is knownas the adversarial loss [25], which is combined with the feature recon-struction loss to train Rω . In the inference stage, we need only thetrained Rω , which can predict visualization images for parameters thatare not in the training data. In the following, we discuss the networkarchitecture, the loss function, and the training process in detail.

5.1 Network Architecture

Three subnetworks are involved during training: the regressor Rω ,feature comparator F , and discriminator Dυ . The regressor Rω

and discriminator Dυ are two deep residual convolutional neural net-works [27] parameterized by the weights ω and υ , respectively. Thearchitectures of Rω and Dυ are designed by following the networkarchitecture proposed by [34, 41], because the scale of our image syn-thesis problem is similar to theirs. For the feature comparator F , weuse the pretrained VGG-19 model [57], which has been widely usedin many deep image synthesis approaches [31, 32, 36].

5.1.1 Regressor Rω

con

cat

(15

36

, 4×

4×

16

×k)

resh

ap

e

(k, 3

, 3, 3

)

tan

h

sim

ula

tio

n

pa

ram

ete

rs

(51

2)

(l)

(l, 5

12

)

(51

2, 5

12

)

(15

36

)

(4×

4×

16

×k)

(4, 4

, 16

×k)

(in

=1

6×k,

ou

t=1

6×k)

(in

=1

6×k,

ou

t=8

×k)

(in

=8

×k,

ou

t=8

×k)

(in

=8

×k,

ou

t=4

×k)

(in

=4

×k,

ou

t=2

×k)

(in

=2

×k,

ou

t=k)

(8, 8

, 16

×k)

(16

, 16

, 8×k)

(32

, 32

, 8×k)

(64

, 64

, 4×k)

(12

8, 1

28

, 2×k)

(25

6, 2

56

, k)

(25

6, 2

56

, 3)

ima

ge

ReLU

others

bacthnormalization

2D convolution

fully connected

residual block

(in

, ou

t, 3

, 3)

(ou

t, o

ut,

3, 3

)

up

sam

plin

g

(in

, ou

t, 1

, 1)

up

sam

plin

g

sum

input/output

vie

w

pa

ram

ete

rs

(51

2)

(n)

(n, 5

12

)

(51

2, 5

12

)

vis

ua

l ma

pp

ing

pa

ram

ete

rs

(51

2)

(m)

(m, 5

12

)

(51

2, 5

12

)

a

Fig. 4. Architecture of Rω , which encodes input parameters into a latentvector with fully connected layers and maps the latent vector into anoutput image with residual blocks. The size of Rω is defined by k, whichcontrols the number of convolutional kernels in the intermediate layers.

The architecture of Rω is shown in Figure 4, which takes the Psim,Pvis, and Pview as inputs and outputs a predicted image I. The threetypes of parameters are first fed into three groups of fully connectedlayers separately, and the outputs are then concatenated and fed intoanother fully connected layer to encode them into a latent vector. Notethat the parameters could also be concatenated first and then fed intofully connected layers. However, as each parameter is fully connectedwith all neurons in the next layer, more weights will be introducedin the network and the network size will increase. Next, the latentvector is reshaped into a low-resolution image, which is mapped to ahigh-resolution output image through residual blocks performing 2Dconvolutions and upsamplings. Following the commonly used archi-tecture [34, 41], we use the rectified linear unit (ReLU) activation func-tion [43] in all layers except the output layer. For the output layer, weuse the tanh function to normalize each pixel into [−1,1].

Note that we introduce a constant k in the network architecture tocontrol the number of convolutional kernels in the intermediate layers.The constant k is used to balance the expressive power and the size andtraining time of Rω to cope with datasets in different complexities.

Residual Blocks Rω consists of several residual blocks (Fig-ure 4a), which are proposed in [27] to improve the performance ofneural networks with increasing depth. We adopted the residual blockshere because Rω often needs to be very deep (i.e., more than 10 con-volutional layers) to synthesize images with high-resolutions. Insideeach residual block, the input image is first upsampled by using near-est neighbor upsampling. The upsampled image is then fed into twoconvolutional layers with kernel size 3×3. In the end, the originalinput image is added to the output, and the result is sent to the nextlayer. Batch normalizations are performed on the output of each con-volutional layer to stabilize the training. Note that if the resolution orthe channel number of the input image is not the same as the output,we perform the upsampling and convolution operations on the inputimage to transform it into the size of the output.

5.1.2 Discriminator Dυ

The architecture of Dυ is shown in Figure 5, which takes a pre-dicted/ground truth image and the corresponding parameters as inputsand produces a likelihood value indicating how likely the input imageis a ground truth image conditioning on the given parameters. With the

(25

6, 2

56

, 3)

(in

=3

, ou

t=k)

(in

=k,

ou

t=2

×k)

(in

=2

×k,

ou

t=4

×k)

(in

=4

×k,

ou

t=8

×k)

(in

=8

×k,

ou

t=8

×k)

(in

=8

×k,

ou

t=1

6×k)

(12

8, 1

28

, k)

(64

, 64

, 2×k)

(32

, 32

, 4×k)

(16

, 16

, 8×k)

(8, 8

, 8×k)

(4, 4

, 16

×k)

ima

ge

(in

=1

6×k,

ou

t=1

6×k)

glo

ba

l su

m

po

olin

g

(16

×k,

1)

(16

×k)

(1)

(1)

sum

sig

mo

id

(in, out, 3, 3)

(out, out, 3, 3)

(in, out, 1, 1)

average

pooling

sum

average

pooling

rea

l / f

ake

ReLU

others

2D convolution

fully connected

residual block

input/output(1

53

6, 1

6×k)

(16

×k)

do

t

con

cat

(51

2)

(l)

(l, 5

12

)

(51

2, 5

12

)

(15

36

)

(51

2)

(n)

(n, 5

12

)

(51

2, 5

12

)

(51

2)

(m)

(m, 5

12

)

(51

2, 5

12

)

a

b

sim

ula

tio

n

pa

ram

ete

rs

vie

w

pa

ram

ete

rsv

isu

al m

ap

pin

g

pa

ram

ete

rs

Fig. 5. Architecture of Dυ . Input parameters and the predicted/groundtruth image are transformed into latent vectors with fully connected lay-ers and residual blocks, respectively. The latent vectors are then incor-porated by using the projection-based method [42] to predict how likelythe image is a ground truth image conditioning on the given parameters.Similar to Rω , the size of Dυ is controlled by the constant k.

likelihood values, an adversarial loss can be defined to update Dυ andRω (details in Section 5.2.2). Similar to Rω , the three types of parame-ters are encoded into a latent vector in Dυ through fully connected lay-ers. Meanwhile, the input image is fed through several residual blocksto derive its intermediate representation that is a latent vector. Thelatent vectors are then incorporated to obtain the likelihood value con-ditioning on the three groups of parameters. ReLU activations are usedin all layers except the output layer, which instead uses the sigmoidfunction to derive a likelihood value within [0,1].

Residual blocks The architecture of the residual blocks in Dυ

(Figure 5a) is similar to that in Rω except that downsampling (averagepooling in this work) is performed instead of upsampling to transformimages into low-resolution representations and no batch normalizationis performed, because it often hurts the performance of Dυ [34, 37].

Projection-based condition incorporation We employed theprojection-based method [42] to incorporate the conditional informa-tion (i.e., the three groups of parameters) with the input image. Thismethod computes a dot product between the data to be incorporated,which in our work is the latent vector of the input parameters and thelatent vector of the image (Figure 5b). Compared with other condi-tion incorporation methods, such as vector concatenation [6, 40], theprojection-based method improves the quality of conditional imagesynthesis results, as demonstrated by Miyato and Koyama [42].

5.1.3 Feature Comparator F

con

v1

_1

relu

1_

1

po

ol1

con

v1

_2

relu

1_

2

con

v2

_2

relu

2_

2

con

v2

_1

relu

2_

1

po

ol2

2D convolution

ReLU

max pooling

input

feature maps

h

w

c channels

Fig. 6. Architecture of F (i.e., VGG-19 network), where each layer islabeled with its name. Feature maps are extracted through convolutionallayers (e.g., relu1 2) for feature-level comparisons.

To produce high quality image synthesis results, we also strive tominimize the feature-level difference between the generated and theground truth image by employing a commonly used feature compara-tor F , namely the pretrained VGG-19 model [57] shown in Figure 6.F is a convolutional neural network, and the convolutional kernels oneach layer have been pretrained to extract certain types of image fea-tures, such as edges and shapes. With it, we extract the features from a

generated image, as well as its corresponding ground truth image, andminimize the difference between those features to improve the qualityof the generated image (see details in Section 5.2.1). Specifically, weuse the layer relu1 2 to extract feature maps for comparison based ontwo observations. First, early layers such as the layer relu1 2 of theVGG-19 network focus on low-level features such as edges and basicshapes, which commonly exist in scientific visualization images. Sec-ond, through our experiments we found that artifacts are introducedinto the generated image by pooling layers (e.g., pool1 in Figure 6).Hence, we use the layer relu1 2 that is before the first pooling layer.

5.2 Loss Function

Given an image I generated using Rω and the corresponding groundtruth image I, a loss function L is defined by measuring the differencebetween them. Minimizing L can, therefore, be conducted by updatingthe parameters ω of Rω over an iterative training process. The moststraightforward choice for L is the average of the pixel wise distancebetween I and I, such as the mean squared error. As shown in ear-lier works [31, 32, 36], however, the average pixel wise distance oftenproduces over-smoothed images, lacking high-frequency features.

In this work, we define L by combining two advanced loss func-

tions: a feature reconstruction loss [32] LF,lf eat and an adversarial

loss [25] Ladv R, namely

L= LF,lf eat +λLadv R, (2)

where λ is the coefficient between them. LF,lf eat measures the difference

between features extracted from the feature comparator F using itsconvolutional layer l, whereas Ladv R quantifies how easily the discrim-inator Dυ can differentiate the generated images from real ones. Ascan be seen, minimizing Ladv R requires training Rω and Dυ togetherin an adversarial manner (i.e., the adversarial theory of GANs [25]).In order to train Dυ , an adversarial loss Ladv D is used.

5.2.1 Feature Reconstruction Loss

The feature reconstruction loss between image I and I is defined bymeasuring the difference between their extracted features [31, 32, 36].Specifically, for a given image I, our feature comparator F (the pre-trained VGG-19) is applied on I and extracts a set of feature maps,

denoted as F l(I). Here l indicates which layer the feature maps arefrom (e.g., the relu1 2 of F). The extracted feature maps can be con-sidered as a 3D matrix of dimension h×w×c, where h, w, and c arethe height, width, and number of channels, respectively, as shown inFigure 6. The feature reconstruction loss between I and I can, there-

fore, be defined as the pixel wise mean squared error between F l(I)

and F l(I). Extending this definition to a batch of images, the feature

reconstruction loss between I0:b−1 and I0:b−1 (b is the batch size) is

LF,lf eat =

1

hwcb

b−1

∑i=0

‖F l(Ii)−F l(Ii)‖22. (3)

Using the feature reconstruction loss enables our regressor to pro-duce images sharing similar feature maps with the correspondingground truth images, which lead to images with sharper features.

5.2.2 Adversarial Loss

In addition to the feature reconstruction loss described above, we addan adversarial loss Ladv R into the loss function. Unlike the featurereconstruction loss, which measures the difference between each pairof images, the adversarial loss focuses on identifying and minimizingthe divergence between two image distributions following the adver-sarial theory of GANs. Specifically, our discriminator Dυ is trainedalong with the regressor Rω to differentiate images generated by Rω

with ground truth images. As the regressor Rω becomes stronger overthe training, the discriminator Dυ is forced to identify more subtle dif-ferences between the generated images and the ground truth.

The adversarial loss can be used as complementary to the featurereconstruction loss for two reasons. First, the feature reconstruction

loss focuses on the average difference between images, and the adver-sarial loss focuses on local features that are the most important to dif-ferentiate the predicted and ground truth images. Second, the featurereconstruction loss compares the difference between each pair of thegenerated and ground truth images, and the adversarial loss measuresdivergence between two image distributions.

In this work, we use the standard adversarial loss presented in [25],which uses different loss functions for the generator and discriminator.For the generator (i.e., our regressor Rω ), the adversarial loss is

Ladv R =−1

b

b−1

∑i=0

logDυ (Ii), (4)

which reaches the minimum when the discriminator cannot differenti-ate the generated images from the ground truth images. This loss iscombined with the feature reconstruction loss to update our regressor(Equation 2). The adversarial loss of the discriminator is defined as

Ladv D =−1

b

b−1

∑i=0

(logDυ (Ii)+ log(1−Dυ (Ii))), (5)

which estimates the divergence between the distribution of the gener-ated images and the ground truth images.

5.3 Techniques to Stabilize Training

We use several techniques to stabilize the adversarial training of Rω

and Dυ . The instability of adversarial trainings is a well-known prob-lem [25], especially when the resolution of synthesized images ishigh [6]. The previous work [6] divided the training into two stagesfor stabilization. In the first stage, the opacity GAN that produces64×64 opacity images is trained, whereas the opacity-to-color transla-tion GAN is trained in the second stage to produce 256×256 color im-ages, conditioning on the 64×64 opacity images. In this work, we traina single pair of adversarial networks (i.e., Rω and Dυ ) that directlyproduces 256×256 color images with the help of recent techniques instabilizing the adversarial training, including the spectral normaliza-tion [41] and the two time-scale update rule (TTUR) [28].

5.3.1 Spectral Normalization

Spectral normalization [41] is used to mitigate the instability of thediscriminator, which is a major challenge in stabilizing the adversarialtraining. Spectral normalization is a weight normalization technique,which outperforms other weight normalization techniques in many im-age synthesis tasks as shown in [34]. Spectral normalization normal-izes the weight matrix of each layer based on the first singular value ofthe matrix. With spectral normalization, the discriminator is enforcedto be Lipschitz continuous, such that the discriminator is constrainedand stabilized to some extent. Spectral normalization is applied oneach layer of the discriminator without changing the network architec-ture; hence spectral normalization is not labeled in Figure 5.

5.3.2 Learning Rate

The learning rates of Rω and Dυ are critical for the stability of theadversarial training. This work uses the Adam optimizer [33] thatchanges the learning rate of each weight dynamically during trainingwith respect to the momentum of the weight gradients. In detail, thelearning rate in the Adam optimizer is controlled by three hyperparme-ters: the initial learning rate α , the first-order momentum β1, and thesecond-order momentum β2. To stabilize the training, a small α is of-

ten preferred; and we found that 5×10−5 stabilized the training in ourcases. In addition, we found that a bigger β1 often cripples the trainingand set β1 to 0 as suggested in [13,69]. Compared with β1, β2 has lessinfluence on the stability of the training, which is set to 0.999.

In previous works on training GANs, we found that people often up-date the discriminator more frequently than the generator, because theydo not want to update the generator based on a discriminator that is notstrong enough. Doing so, however, leads to a longer training time. Ourwork uses the same update frequency for the regressor and discrimina-tor but with different learning rates αD and αR (i.e., the TTUR tech-nique [28]). Based on the empirical results shown in [13, 69], we set

the learning rate of the discriminator to be 4 times that of the regressor,that is, αD = 2×10−4 and αR = 5×10−5.

5.4 Training Process

Algorithm 1 Training process of InSituNet.

Input: Training data includes parameters {Psim,Pvis,Pview}0:N−1 andthe corresponding images I0:N−1. Initial weights ω and υ of Rω

and Dυ , respectively. The feature comparator F .Output: Optimized weights ω and υ

1: Repeat:2: {Psim,Pvis,Pview}0:b−1 , I0:b−1 sampled from training data

3: I0:b−1← Rω ({Psim,Pvis,Pview}0:b−1)

4: υ ← Adam(∇υLadv D( I0:b−1, I0:b−1;υ) ,υ ,αD,β1,β2)5: ω ← Adam(∇ωL( I0:b−1, I0:b−1;ω) ,ω,αR,β1,β2)6: Until exit criterion is satisfied

The process of training our regressor and discriminator is shownin Algorithm 1. Given the training data collected in situ, namely, Npairs of paramters {Psim,Pvis,Pview}0:N−1 and the corresponding im-ages I0:N−1, we first initialize the network weights ω and υ using theorthogonal initialization [54]. Then, the discriminator and regressorare updated alternatively by using the stochastic gradient descent untilthe exit criterion is satisfied. The exit criterion used in this work is themaximum number of iterations, which is set to 125,000 because theloss converged in our cases after 125,000 iterations.

In each iteration, a batch of parameters {Psim,Pvis,Pview}0:b−1 andthe corresponding images I0:b−1 are sampled from the training data(line 2), where b is the batch size. Next, the current Rω takes{Psim,Pvis,Pview}0:b−1 as inputs and produces I0:b−1 (line 3). Accord-

ing to the loss Ladv D defined on I0:b−1 and I0:b−1 in Equation 5,the weights of the discriminator are updated (line 4). Similarly, theweights of the regressor are updated as well, according to the loss func-tion L (defined in Equations 2, 3, and 4), which is computed using thefeature comparator F and the updated discriminator Dυ (line 5). Whenupdating the weights υ and ω , the gradients ∇υ and ∇ω of the lossfunctions Ladv D and L are computed, respectively. With ∇υ and ∇ω ,the weights υ and ω are updated through two Adam optimizers usingthe learning rates discussed in the preceding section.

6 PARAMETER SPACE EXPLORATION WITH INSITUNET

Subregion Sensitivity of: BwsA Visualization ViewCompute Overall Sensitivity Curve

Simulation Parameters

BwsA

1 1.5 2 2.5 3 3.5 3.8

Visual Mapping Parameters

Isovalue of temperature:

15 20 25

View Parameters

theta

0 60 120 240 300 360160

phi

-90 -30 0 30 60 90-54

Parameters View

sensitiv

ity

300

350

400

450

500

550

600

a b

a1

b1

sensitivity0 80

salinity30 40

Fig. 7. Visual interface for parameter space exploration. (a) The threegroups of parameters: simulation, visual mapping, and view parameters.(b) The predicted visualization image and the sensitivity analysis result.

With the trained InSituNet, users can perform parameter space ex-ploration of ensemble simulations from two perspectives. First, withInSituNet’s forward propagations, users can interactively infer the vi-sualization results for arbitrary parameters within the parameter space.Second, using InSituNet’s backward propagations, users can investi-gate the sensitivity of different parameters and thus have better under-standing on parameter selections. To support the parameter space ex-ploration, we built an interactive visual interface as shown in Figure 7,which contains two views: Parameters View (Figure 7(a)) and Visual-ization View (Figure 7(b)). In the following, we explain how users canperform parameter space exploration with this visual interface.

Table 1. Datasets and timings: k controls the size of InSituNet to cope with datasets in different complexities; diversity [63] measures how diversethe generated images are; tsim, tvis, and ttr are timings for running ensemble simulations, visualizing data in situ, and training InSituNet, respectively;t f p and tbp are timings for a forward and backward propagation of the trained InSituNet, respectively.

SimulationPsim

PvisPview

k DiversitySize (GB) Performance

Name Number Name Number Raw Image Network tsim (hr) tvis (hr) ttr (hr) t f p (s) tbp (s)

SmallPoolFire Ck,C 4,000 pseudo-coloring with 5 color schemes N/A N/A 32 2.72 ≈25.0 0.43 0.06 1,420.0 6.45 16.40 0.031 0.19

Nyx OmM,OmB,h 500 volume rendering with a transfer function θ ,φ 100 48 1.72 ≈30.0 3.92 0.12 537.5 8.47 18.02 0.033 0.22

MPAS-Ocean BwsA 300 isosurface visualization with 3 isovalues θ ,φ 100 48 1.75 ≈300.0 3.46 0.15 229.5 10.73 18.13 0.033 0.23

6.1 Inference of Visualization Results

InSituNet is able to interactively infer the visualization results for anyuser-selected parameter values. As shown in Figure 7(a), the threegroups of input parameters for the InSituNet are visualized by usingdifferent GUI widgets. For the simulation and view parameters, be-cause their values are usually in continuous ranges, we visualize themusing slider bars whose ranges are clipped to the corresponding param-eters’ predefined value ranges. Users are able to select arbitrary param-eter values by interacting with those sliders. For the visual mappingparameters, users can switch among a set of predetermined optionsusing the ratio buttons, for example, selecting different isovalues forisosurface visualizations, as shown in Figure 7(a).

The selected values for the three groups of parameters are fed intothe trained InSituNet. Through a forward propagation of the network,which takes around 30 ms, the corresponding visualization image forthe given set of parameters is generated and visualized in the Visual-ization View, as shown in Figure 7(b).

6.2 Sensitivity Analysis on Simulation Parameters

Because InSituNet is differentiable, users can perform sensitivity anal-ysis for the simulation parameters using the network’s backward prop-agations. Specifically, users can compute the derivative of a scalarvalue derived from the generated image (e.g., L1 norm of the pixelvalues) with respect to a selected simulation parameter. The absolutevalue of the derivative can be treated as the sensitivity of the parame-ter, which indicates how much the generated image will change if theparameter gets changed. Note that the sensitivity analysis in this workis used to reflect the changes (with respect to the parameters) in theimage space rather than the data space. Inspired by [6], our analysisincludes overall sensitivity analysis and subregion sensitivity analysis.

In overall sensitivity analysis, we focus on analyzing the sensitivityof the entire image with respect to each simulation parameter acrossits value range. To this end, we sweep each parameter across its valuerange while fixing the values of other parameters. Images are then gen-erated from the selected parameter values and aggregated into a scalar(i.e., the L1 norm of the pixel values). The aggregated scalar values arethen back propagated through the InSituNet to obtain the sensitivity ofthe selected parameter values. In the end, a list of sensitivity valuesis returned for each parameter and visualized as a line chart on top ofthe slider bar corresponding to the parameter (Figure 7(a1)) to indicatehow sensitive the parameter is across its value range.

In subregion sensitivity analysis, we analyze the sensitivity of a se-lected parameter for different subregions of the generated image. Thisanalysis is done by partitioning the visualization image into blocks andcomputing the sensitive of the parameter for the L1 norm of the pixelvalues in each block. The computed sensitivity values are then colorcoded from white to red and overlaid on top of the visualization imageto indicate what regions are more sensitive with respect to the selectedparameter (red blocks in Figure 7(b1)).

7 RESULTS

We evaluated InSituNet using combustion, cosmology, and ocean sim-ulations (Section 7.1) from four aspects: (1) providing implementationdetails and analyzing performance (Section 7.2); (2) evaluating the in-fluence of different hyperparameters (Section 7.3); (3) comparing withalternative methods (Section 7.4); and (4) performing parameter spaceexploration and analysis with case studies (Section 7.5).

7.1 Ensemble Simulations

We evaluated the proposed approach using three ensemble simulations:SmallPoolFire [66], Nyx [4], and MPAS-Ocean [51]. They are sum-

marized in Table 1 (left) and detailed below.

SmallPoolFire is a 2D combustion simulation from the Open-FOAM simulation package [66]. We used it as a test case to eval-uate InSituNet by studying two parameters: a turbulence parameterCk∈[0.0925,0.0975] and a combustion parameter C∈[4.99,5.01]. Wesampled 4,000 parameter settings from the parameter space: 3,900 fortraining and 100 for testing. Images were generated for the tempera-ture field by using pseudo-coloring with five predefined color schemes.To study how diverse the generated images are, we use the methodproposed by Wang et al. [63], which measures the diversity as the re-ciprocal of the average structural similarity (SSIM) between every pairof images. The diversity of the images in this dataset is 2.72, whichmeans the average SSIM is smaller than 0.4.

Nyx is a cosmological simulation developed by Lawrence Berke-ley National Laboratory. Based on the scientists’ suggestion, we stud-ied three parameters: the total matter density (OmM ∈ [0.12,0.155]),the total density of baryons (OmB ∈ [0.0215,0.0235]), and the Hub-ble constant (h ∈ [0.55,0.85]). We sampled 500 parameter settingsfrom the parameter space: 400 for training and 100 for testing. Thesimulation was conducted with each parameter setting and generateda 256×256×256 volume representing the log density of the dark mat-ters. The volume was visualized in situ by using volume renderingwith a predefined transfer function of the wave colormap1 and from100 different viewpoints. The diversity of the generated images is 1.72.

MPAS-Ocean is a global ocean simulation developed by LosAlamos National Laboratory. Based on the domain scientists’ inter-est, we studied the parameter that controls the bulk wind stress ampli-fication (BwsA ∈ [1,4]). We generated 300 ensemble members withdifferent BwsA values. We used 270 of them for training and the restfor testing. The isosufaces of the temperature field (with isovalue={15,20, 25}) were extracted and visualized from 100 different viewpointsfor each ensemble member. The isosurfaces were colored based onsalinity, using the colormap suggested by Samsel et al. [53]. The di-versity of the generated images is 1.75.

7.2 Implementation and Performance

The proposed approach consists of three components: the in situ datacollection, the training of InSituNet, and the visual exploration andanalysis component. We discuss the implementation details and per-formance of the three components in the following.

The in situ visualization was implemented by using ParaView Cat-alyst2 following the Cinema framework [1, 2]. The simulations and insitu visualization were conducted on a supercomputer of 648 compu-tation nodes. Each node contains an Intel Xeon E5-2680 CPU with14 cores and 128 GB of main memory. We used 1, 28, and 128 pro-cesses, respectively, for the SmallPoolFire, Nyx, and MPAS-Oceansimulations. InSituNet was implemented in PyTorch3 and trained withan NVIDIA DGX-1 system, which contains 8 NVIDIA V100 GPUswith NVlink. The visual interface was implemented based on a webserver/client framework. The interface was implemented with D3.json the client side, and the images were generated from a Python server(with the assist of the trained InSituNet) and sent to the client for visu-alization. The visual exploration and analysis were tested on a desktopwith an Intel Core i7-4770 CPU and an NVIDIA 980Ti GPU.

The space and computation costs using the proposed approach forthe three different datasets are listed in Table 1 (right). The size ofInSituNet is less than 1% and 15% of the raw simulation data and the

1https://sciviscolor.org2https://www.paraview.org/in-situ3https://pytorch.org

Ny

xS

ma

llP

oo

lFir

eM

PAS

-Oce

an

Lmse LfeatGround Truth Ladv_R Lfeat +10 -2Ladv_R

log

de

nsi

ty9

12

.5sa

linit

y3

04

0te

mp

era

ture

30

01

85

0

Fig. 8. Qualitative comparison of InSituNet trained with different lossfunctions. Combining L f eat and Ladv R gives the results of high quality.

image data, respectively. In terms of data reduction, we also compareour approach with several data compression methods and the resultscan be found in the supplementary material. The training of InSituNetgenerally takes more than 10 hours, but the time is much less than ac-tually running the ensemble simulations with extra parameter settings.After training, a forward or backward propagation of InSituNet takesless than one second on a single NVIDIA 980Ti GPU.

7.3 Model Evaluation for Different Hyperparameters

We evaluated InSituNet trained with different hyperparameters (i.e.,loss functions, network architectures, and numbers of training sam-ples) qualitatively and quantitatively using the data that were excludedfrom the training to study two questions: (1) Is InSituNet able to gen-erate images that are close to the ground truth images? (2) How do thechoices of hyperparameters influence the training results?

For quantitative evaluations, we used four metrics that focus on dif-ferent aspects to compare the predicted images with the ground truthimages, including peak signal-to-noise ratio (PSNR), SSIM, earthmover’s distance (EMD) between color histograms [6], and Frechetinception distance (FID) [28].

PSNR measures the pixel-level difference between two images us-ing the aggregated mean squared error between image pixels. A higherPSNR indicates that the compared images are more similar pixel wise.

SSIM compares two images based on the regional aggregated statis-tical information (e.g., mean and standard deviation of small patches)between them. A higher SSIM means the compared images are moresimilar from a structural point of view.

EMD is used in [6] to quantify the distance between the color his-tograms of two images. A lower EMD means the compared imagesare more similar according to their color distributions.

FID approximates the distance between two distributions of images,which is widely used in recent image synthesis works [13, 69] as acomplementary to other metrics. A lower FID suggests the two imagecollections are more similar statistically.

7.3.1 Loss Functions

We evaluated InSituNet trained with different loss functions includingthe mean squared error Lmse, the feature reconstruction loss L f eat , theadversarial loss Ladv R, and the combination of L f eat and Ladv R.

Figure 8 compares the images generated by InSituNet trained withdifferent loss functions with the ground truth (The enlarged figurecan be found in the supplementary material). We can see that usingLmse often generates over-smoothed images lacking high-frequencyfeatures, whereas using L f eat can mitigate the problem to some ex-tent. Using Ladv R can generate images that are as sharp as the groundtruth, but the features are often not introduced in the desired positions.

Table 2. Quantitative evaluation of InSituNet trained with different lossfunctions. The model trained with the combination of L f eat and Ladv R

generates images with the best EMD and FID and only a slightly lowerPSNR and SSIM compared with the model trained with Lmse or L f eat .

Lmse L f eat Ladv R L f eat +10−2Ladv R

SmallPoolFire

PSNR 25.090 24.937 20.184 24.288

SSIM 0.9333 0.9390 0.8163 0.9006

EMD 0.0051 0.0064 0.0056 0.0037

FID 21.063 15.881 12.859 9.4747

Nyx

PSNR 31.893 29.055 24.592 29.366

SSIM 0.8684 0.8698 0.7081 0.8336

EMD 0.0037 0.0083 0.0064 0.0022

FID 60.825 54.670 24.036 6.2694

MPAS-Ocean

PSNR 26.944 26.267 17.099 24.791

SSIM 0.8908 0.8885 0.7055 0.8655

EMD 0.0025 0.0044 0.0036 0.0017

FID 115.74 120.37 28.927 21.395

By combining L f eat and Ladv R, we are able to generate images withsharp features, and those images are also similar to the ground truth.

Table 2 reports the quantitative results from using different lossfunctions. We found that using Lmse gives the best PSNR, becausethe network using Lmse is trained to minimize the mean squared error(i.e., maximize the PSNR). Using L f eat gives the best SSIM in somecases, because it focuses more on the structure of the images. However,using Lmse or L f eat often results in poor performance regarding EMDand FID. When training InSituNet with Ladv R, the FID value can beimproved, but the values of PSNR and SSIM drop a lot. By combin-ing L f eat and Ladv R, both EMD and FID improved a lot, though thePSNR and SSIM got slightly worse than using Lmse or L f eat .

(a) (b)

Fig. 9. Images generated by InSituNet trained with L f eat that uses dif-ferent layers after the first pooling layer of VGG-19: (a) relu2 1 and (b)relu3 1. Checkerboard artifacts are introduced.

For L f eat , using which layer of the pretrained VGG-19 to extractfeatures from images can affect the image synthesis results. Throughempirical studies, we found that using any layers after the first poolinglayer of VGG-19 will introduce undesired checkerboard artifacts, be-cause of the “inhomogeneous gradient update” of the pooling layer [3],as shown in Figure 9. Hence, we use the last layer right before the firstpooling layer, which is the layer relu1 2.

Table 3. Evaluating the weight λ of Ladv R: λ = 0.01 provides the resultsthat balance the PSNR, SSIM, EMD, and FID.

λ = 0.005 λ = 0.01 λ = 0.02 λ = 0.04

PSNR 30.043 29.366 29.040 27.232

SSIM 0.8619 0.8336 0.8253 0.7680

EMD 0.0041 0.0022 0.0023 0.0025

FID 21.267 6.2694 6.6819 9.8992

We also evaluated the influence of the weight λ for Ladv R whencombining it with L f eat (defined in Equation 2), and the results areshown in Table 3. We found that increasing λ over 0.01 cannot im-prove the accuracy of the generated images any further. In addition, asmall λ (i.e., 0.005) will hurt the image accuracy in terms of EMD andFID, although the value of PSNR and SSIM can be improved slightly.We thereby set λ to 0.01 to balance its effects on the four metrics.

7.3.2 Network Architectures

We evaluated InSituNet with different network architectures in termsof the accuracy of predicted images, the network size, and the trainingtime. As mentioned in Section 5.1, the architecture of our network iscontrolled by a constant k, which controls the number of convolutionalkernels in the intermediate layers. In this experiment, we evaluatedfour k values: 16, 32, 48, and 64.

Figure 10 shows the PSNR and EMD of images generated by InSi-tuNet with the four k values. We can see that InSituNet with larger

(a) (b)

PS

NR

SmallPoolFire Nyx MPAS-Ocean0

5

10

15

20

25

30

35

k=48

k=16k=32

k=64

EM

D

SmallPoolFire Nyx MPAS-Ocean0.000

0.002

0.004

0.006

0.008

0.010

k=48

k=16k=32

k=64

Fig. 10. Quantitative evaluation of different network architectures con-trolled by k with (a) PSNR and (b) EMD.

Table 4. Size and training time of different network architectures con-trolled by k for the Nyx dataset.

k = 16 k = 32 k = 48 k = 64

Network Size (MB) 26.4 67.4 125.2 199.6

Training Time (hr) 13.73 16.42 18.02 20.17

k values can generate more (or at least equally) accurate images, be-cause a larger k gives more expressive power to the neural network.On the other hand, training InSituNet with a larger k also costs moretime, and more storage will be needed to store the networks, as shownin Table 4 using the Nyx dataset as an example. Hence, to balance theaccuracy of the generated images and the cost from both computationand storage, we set k to 32, 48, and 48 for the SmallPoolFire, Nyx, andMPAS-Ocean, respectively.

7.3.3 Number of Ensemble Runs used for Training

Table 5. Evaluation of the number of ensemble runs used for training.

Simulation # Ensemble Runs PSNR SSIM EMD FID

SmallPoolFire

900 21.842 0.8714 0.0040 14.398

1900 23.192 0.9016 0.0036 11.732

2900 23.932 0.9018 0.0037 9.5813

3900 24.288 0.9006 0.0037 9.4747

Nyx

100 28.108 0.7951 0.0025 9.8818

200 29.404 0.8319 0.0022 6.5481

300 29.398 0.8326 0.0023 6.4239

400 29.366 0.8336 0.0022 6.2694

MPAS-Ocean

70 24.347 0.8554 0.0016 37.229

140 24.593 0.8607 0.0017 28.380

210 24.732 0.8643 0.0017 22.794

270 24.791 0.8655 0.0017 21.395

We compared InSituNet trained using different numbers of ensem-ble runs (Table 5) to study how many ensemble runs will be needed totrain a good model for the three simulations. We found that this num-ber is different in different simulations, depending on the complexityof the mapping between simulation parameters and visualization re-sults. Experiment results show that the accuracy of generated imagesbecomes stable when the number of ensemble runs is greater than2,900, 200, and 210 for the SmallPoolFire, Nyx, and MPAS-Oceansimulation, respectively. As a result, we used 3,900, 400, and 270 runsfrom the three simulations to train InSituNet for the rest of the study.

7.4 Comparison with Alternative Methods

We compared our method with two alternative methods including in-terpolating images from the training data that close to the target imageand the GAN-based volume rendering method (GAN-VR) [6] usingthe Nyx dataset. For the interpolation method, we sample g imagesfrom the training data whose parameter settings are the top g closest tothe parameter setting of the test image and interpolate the sampled im-ages using inverse distance weighting interpolation [56]. We evaluatedg from 1 to 5 and present the result of g = 3 in this section because itbalances the four metrics (More results are in the supplementary ma-terial). For GAN-VR, we incorporated the simulation parameters intoboth the opacity GAN and the opacity-to-color translation GAN andremoved the transfer function related parameters because we used afixed transfer function for this dataset. For InSituNet, we selected anetwork architecture whose size is not greater than the size of GAN-VR network, for a fair comparison.

Figure 11 compares the ground truth images with the images gen-erated by using interpolation, GAN-VR, and InSituNet. With the new

Ground Truth GAN-VR InSituNetInterpolation

Fig. 11. Comparison of the images generated using interpolation,GAN-VR, and InSituNet with the ground truth images.

Table 6. Quantitative comparison of images generated with interpolation,GAN-VR, and InSituNet.

Network Size PSNR SSIM EMD FID

Interpolation N/A 23.932 0.6985 0.0070 58.571

GAN-VR 80.5 MB 20.670 0.6274 0.0056 38.355

InSituNet 67.5 MB 28.471 0.8034 0.0023 9.0152

network architecture (e.g., the projection-based condition incorpora-tion method in Section 5.1), loss functions (e.g., the feature recon-struction loss in Section 5.2), and training strategies (e.g., the spectralnormalization in Section 5.3), InSituNet can generate results that bet-ter preserve features compared with the other two methods. The quan-titative comparisons between the three methods are shown in Table 6.InSituNet outperforms the other two methods in all four metrics.

7.5 Parameter Space Exploration

This section demonstrates the effectiveness of our deep image synthe-sis driven parameter space exploration through case studies on the Nyxand MPAS-Ocean simulations.

7.5.1 Case Study with the Nyx Simulation

OmM

0.12 0.127 0.141 0.148 0.1550.136

600

1000

1400

1800

2200

2600

3000

sensitiv

ity

OmB

0.0215 0.022 0.023 0.02350.02253

0200400600800

100012001400

sensitiv

ity

h

0.55 0.65 0.7 0.75 0.8 0.850.59

0

1000

2000

3000

4000

5000

sensitiv

ity

Simulation Parameters Predicted Image

Fig. 12. Parameter space exploration with the Nyx simulation. For theselected parameter values, the sensitivity of different parameters is es-timated and visualized as line charts on the left, whereas the predictedimage is visualized on the right.

Our first case study is focused on investigating the influence of dif-ferent simulation parameters (i.e., OmM, OmB, and h) on the Nyxsimulation. The explorations of different visualization settings can befound in our associated video.

Figure 12 shows a selected parameter setting with the predicted vi-sualization image. To understand the influence of each parameter, wecomputed the sensitivity of the three parameters with respect to theL1 norm of the predicted image, shown as the three line charts in Fig-ure 12. From the scale of the three charts (i.e., the values along thevertical axes), we see that parameter h is more sensitive to parameterOmM and parameter OmM is more sensitive to parameter OmB.

Focusing on the most sensitive parameter, namely, parameter h, weexplored how it affects the visual appearance of the predicted images.Figure 13 shows five images predicted by using five different h values,while parameter OmM and OmB are fixed at the values shown on thetwo corresponding slider bars in Figure 12. We first evaluate the ac-curacy of the sensitivity curve (blue curve in Figure 13) computed bybackpropagation with the central difference method. To this end, we

h0.6 0.80.56

010002000300040005000

0.64 0.7 0.76 0.83

sensitiv

ity

Fig. 13. Comparison of the visual appearance of the predicted imagesusing different h values to see the effect of this simulation parameter.

first regularly sample the simulation parameters along the curve (128samples are drawn) and then generate the visualization images withrespect to the sampled simulation parameters. The L1 norm of thegenerated images is then computed and used to compute the sensitivecurve using the central difference method. The result is shown as theorange curve in Figure 13. We can see that the sensitivity curves gener-ated with the two methods are similar. From the guidance provided bythe line chart in Figure 13, we see that parameter h is more sensitive inthe first half of its range (i.e., the left side of the dashed line). The threeimages generated using h values from this range demonstrate a biggervariance compared with the two images shown on the right (which areimages generated by using h values from the second half of its range).

7.5.2 Case Study with the MPAS-Ocean Simulation

θ=40 θ=60 θ=80 θ=100 θ=120 θ=140

temp=15

temp=20

temp=25

Fig. 14. Predicted images of the MPAS-Ocean dataset for different iso-surfaces and viewpoints, which reasonably reflect the change of viewprojections and shading effects.

BwsA=1.0 BwsA=2.5 BwsA=4.0

pre

dic

ted

ima

ge

sub

reg

ion

sen

siti

vit

y

salin

ity

30

40

sen

siti

vit

y

0

200

ab

Fig. 15. Forward prediction (top row) and backward subregion sensitivityanalysis (bottom row) for different BwsA. Regions that influenced byBwsA (i.e., regions a and b) are highlighted by the sensitivity map.

Our next case study explores different parameter settings for theMPAS-Ocean simulation and demonstrates the subregion sensitivityanalysis for the simulation parameter BwsA, which characterizes thebulk wind stress. Note that here we focus only on exploration andanalysis of new parameter settings, the comparison between the pre-dicted and ground truth images is discussed in the previous sections.

Figure 14 shows isosurface visualizations of the temperature fieldwith three different isovalues from six different viewpoints. The valueof parameter BwsA is fixed at 1 in this study. The images reasonablyreflect the change of view projections and shading effects. More explo-ration and visualization results can be found in our associated video.

Figure 15 shows the predicted images when using different BwsAvalues. All images are generated from the temperature field (iso-value=15) of MPAS-Ocean and from the same viewpoint. The first rowof images shows the result of forward inference with different BwsAvalues, whereas the second row of images overlays the subregion sensi-tivity maps onto the corresponding images of the first row. The labeledregions (i.e., Figure 15a, b) change the most when adjusting the valueof BwsA, and the subregion sensitivity maps on the second row echothese bigger changes, as indicated by the darker red color.

8 LIMITATIONS, DISCUSSION, AND FUTURE WORK

This section discusses several directions that we would like to explorein the future: (1) improving the flexibility in exploring arbitrary visualmapping parameters; (2) increasing the accuracy of predicted images;and (3) increasing the resolution of predicted images.

One limitation of our approach is that we restricted the users’ abil-ity in exploring arbitrary visual mapping parameters, for example, ex-hausting all possible transfer functions for volume rendering. Instead,we allow users to switch only among several predefined visual map-pings, for example, the three isovalues when exploring the MPAS-Ocean data. Theoretically, training a deep learning model to predictvisualization images for arbitrary simulation and visualization param-eters is feasible. However, it will require a large number of trainingimages to cover the joint space of all possible simulation and visualiza-tion parameters. For example, in order to train a model that can predictvolume rendering results of a single volume data for arbitrary transferfunctions, 200,000 training images are required, as shown in [6]. Con-sequently, the size of the training data may even exceed the size of theraw simulation data, which offsets the benefit of in situ visualization.Considering this issue, we would like to explore deep learning tech-niques that do not require a large number of training samples, such asone- or zero-shot learning, to improve the flexibility of exploration.

Similar to most other machine learning techniques, generating pre-diction results that are exactly the same as the ground truth is extraordi-nary difficult. By taking advantage of recent advances in deep learningfor image synthesis, the proposed approach has already outperformedother image synthesis based visualization techniques in terms of thefidelity and accuracy of the generated images (see the comparison inSection 7). However, we believe further improvement is still possible,and we would like explore other network architectures and/or otherloss functions to improve our deep image synthesis model.

Our network architecture limits the resolution of output images to256×256, which might not be sufficient for some high-resolution sim-ulation data. We believe that our network architecture has the potentialto generate images with higher resolutions by adding more residualblocks, and we will investigate this approach in the future.

9 CONCLUSION

In this work, we propose InSituNet, a deep learning based image syn-thesis model supporting the parameter space exploration of large-scaleensemble simulations visualized in situ. The model is trained to learnthe mapping from ensemble simulation parameters to visualizations ofthe corresponding simulation outputs, conditioned on different visu-alization settings (i.e., visual mapping and view parameters). With atrained InSituNet, users can generate visualizations of simulation out-puts with different simulation parameters without actually running theexpensive simulation, as well as synthesize new visualizations withdifferent visualization settings that are not used during the runs. Ad-ditionally, an interactive visual interface is developed to explore thespace of different parameters and investigate their sensitivity using thetrained InSituNet. Through both quantitative and qualitative evalua-tions, we validated the effectiveness of InSituNet in analyzing ensem-ble simulations that model different physical phenomena.

ACKNOWLEDGMENTS

This work was supported in part by US Department of Energy LosAlamos National Laboratory contract 47145 and UT-Battelle LLC con-tract 4000159447 program manager Laura Biven.

REFERENCES

[1] J. Ahrens, S. Jourdain, P. O’Leary, J. Patchett, D. H. Rogers, P. Fasel,

A. Bauer, M. Petersen, F. Samsel, and B. Boeckel. In situ MPAS-Ocean

image-based visualization. In Proceedings of the International Confer-

ence for High Performance Computing, Networking, Storage and Analy-

sis, Visualization & Data Analytics Showcase, 2014.

[2] J. Ahrens, S. Jourdain, P. O’Leary, J. Patchett, D. H. Rogers, and M. Pe-

tersen. An image-based approach to extreme scale in situ visualization

and analysis. In Proceedings of the International Conference for High

Performance Computing, Networking, Storage and Analysis, pp. 424–

434, 2014.

[3] A. Aitken, C. Ledig, L. Theis, J. Caballero, Z. Wang, and W. Shi.

Checkerboard artifact free sub-pixel convolution: A note on sub-pixel

convolution, resize convolution and convolution resize. arXiv preprint

arXiv:1707.02937, 2017.

[4] A. S. Almgren, J. B. Bell, M. J. Lijewski, Z. Lukic, and E. V. Andel.

Nyx: A massively parallel AMR code for computational cosmology. The

Astrophysical Journal, 765(1):39, 2013.

[5] A. C. Bauer, H. Abbasi, J. Ahrens, H. Childs, B. Geveci, S. Klasky,

K. Moreland, P. O’Leary, V. Vishwanath, B. Whitlock, and E. W. Bethel.

In situ methods, infrastructures, and applications on high performance

computing platforms. Computer Graphics Forum, 35(3):577–597, 2016.

[6] M. Berger, J. Li, and J. A. Levine. A generative model for volume ren-

dering. IEEE Transactions on Visualization and Computer Graphics,

25(4):1636–1650, 2019.

[7] S. Bergner, M. Sedlmair, T. Moller, S. N. Abdolyousefi, and A. Saad.

ParaGlide: Interactive parameter space partitioning for computer sim-

ulations. IEEE Transactions on Visualization and Computer Graphics,

19(9):1499–1512, 2013.

[8] T. Biedert and C. Garth. Contour tree depth images for large data visual-

ization. In Proceedings of Eurographics Symposium on Parallel Graphics

and Visualization, pp. 77–86, 2015.

[9] A. Biswas, G. Lin, X. Liu, and H.-W. Shen. Visualization of time-varying

weather ensembles across multiple resolutions. IEEE Transactions on

Visualization and Computer Graphics, 23(1):841–850, 2017.

[10] A. Bock, A. Pembroke, M. L. Mays, L. Rastaetter, T. Ropinski, and

A. Ynnerman. Visual verification of space weather ensemble simulations.

In Proceedings of 2015 IEEE Scientific Visualization Conference, pp. 17–

24, 2015.

[11] M. Booshehrian, T. Moller, R. M. Peterman, and T. Munzner. Vismon:

Facilitating analysis of trade-offs, uncertainty, and sensitivity in fisheries

management decision making. In Computer Graphics Forum, vol. 31, pp.

1235–1244. Wiley Online Library, 2012.

[12] P.-T. Bremer, G. H. Weber, J. Tierny, V. Pascucci, M. S. Day, and J. B.

Bell. Interactive exploration and analysis of large-scale simulations using

topology-based data segmentation. IEEE Transactions on Visualization

and Computer Graphics, 17(9):1307–1324, 2011.

[13] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for

high fidelity natural image synthesis. In Proceedings of International

Conference on Learning Representations, 2019.

[14] S. Bruckner and T. Moller. Result-driven exploration of simulation param-

eter spaces for visual effects design. IEEE Transactions on Visualization

and Computer Graphics, 16(6):1468–1476, 2010.

[15] H. Chen, S. Zhang, W. Chen, H. Mei, J. Zhang, A. Mercer, R. Liang,

and H. Qu. Uncertainty-aware multidimensional ensemble data visualiza-

tion and exploration. IEEE Transactions on Visualization and Computer

Graphics, 21(9):1072–1086, 2015.

[16] D. Coffey, C.-L. Lin, A. G. Erdman, and D. F. Keefe. Design by dragging:

An interface for creative forward and inverse design with simulation en-

sembles. IEEE Transactions on Visualization and Computer Graphics,

19(12):2783–2791, 2013.

[17] S. Di and F. Cappello. Fast error-bounded lossy HPC data compression

with SZ. In Proceedings of International Parallel and Distributed Pro-

cessing Symposium, pp. 730–739, 2016.

[18] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using

deep convolutional networks. IEEE Transactions on Pattern Analysis and

Machine Intelligence, 38(2):295–307, 2016.

[19] A. Dosovitskiy, J. T. Springenberg, and T. Brox. Learning to generate

chairs with convolutional neural networks. In Proceedings of 2015 IEEE

Conference on Computer Vision and Pattern Recognition, pp. 1538–1546,

2015.

[20] S. Dutta, C.-M. Chen, G. Heinlein, H.-W. Shen, and J.-P. Chen. In situ

distribution guided analysis and visualization of transonic jet engine sim-

ulations. IEEE Transactions on Visualization and Computer Graphics,

23(1):811–820, 2017.

[21] O. Fernandes, S. Frey, F. Sadlo, and T. Ertl. Space-time volumetric depth

images for in-situ visualization. In Proceedings of 2014 IEEE Symposium

on Large Data Analysis and Visualization, pp. 59–65, 2014.

[22] S. Frey, F. Sadlo, and T. Ertl. Explorable volumetric depth images from

raycasting. In Proceedings of 2013 XXVI Conference on Graphics, Pat-

terns and Images, pp. 123–130, 2013.

[23] L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic

style. arXiv preprint arXiv:1508.06576, 2015.

[24] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis using convo-

lutional neural networks. In Proceedings of Advances in Neural Informa-

tion Processing Systems, pp. 262–270, 2015.

[25] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,

S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In

Proceedings of Advances in Neural Information Processing Systems, pp.

2672–2680, 2014.

[26] J. Han, J. Tao, and C. Wang. FlowNet: A deep learning framework for

clustering and selection of streamlines and stream surfaces. IEEE Trans-

actions on Visualization and Computer Graphics, 2018, Early Access.

[27] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image

recognition. In Proceedings of 2016 IEEE Conference on Computer Vi-

sion and Pattern Recognition, pp. 770–778, 2016.

[28] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.

GANs trained by a two time-scale update rule converge to a local nash

equilibrium. In Proceedings of Advances in Neural Information Process-

ing Systems, pp. 6626–6637, 2017.

[29] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural

Computation, 9(8):1735–1780, 1997.

[30] F. Hong, J. Zhang, and X. Yuan. Access pattern learning with long short-

term memory for parallel particle tracing. In Proceedings of 2018 IEEE

Pacific Visualization Symposium, pp. 76–85, 2018.

[31] X. Hou, L. Shen, K. Sun, and G. Qiu. Deep feature consistent variational

autoencoder. In Proceedings of 2017 IEEE Winter Conference on Appli-

cations of Computer Vision, pp. 1133–1141, 2017.

[32] J. Johnson, A. Alahi, and F.-F. Li. Perceptual losses for real-time style

transfer and super-resolution. In Proceedings of European Conference on

Computer Vision, pp. 694–711, 2016.

[33] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In

Proceedings of International Conference on Learning Representations,

2015.

[34] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly. The GAN

landscape: Losses, architectures, regularization, and normalization. arXiv

preprint arXiv:1807.04720, 2018.

[35] S. Lakshminarasimhan, N. Shah, S. Ethier, S.-H. Ku, C. S. Chang,

S. Klasky, R. Latham, R. Ross, and N. F. Samatova. ISABELA for effec-

tive in situ compression of scientific data. Concurrency and Computation:

Practice and Experience, 25(4):524–540, 2013.

[36] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta,

A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic sin-

gle image super-resolution using a generative adversarial network. In

Proceedings of 2017 IEEE Conference on Computer Vision and Pattern

Recognition, pp. 105–114, 2017.

[37] M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are

GANs created equal? A large-scale study. In Proceedings of Advances in

Neural Information Processing Systems, pp. 700–709, 2018.

[38] K.-L. Ma. In situ visualization at extreme scale: Challenges and opportu-

nities. IEEE Computer Graphics and Applications, 29(6):14–19, 2009.

[39] K. Matkovic, D. Gracanin, B. Klarin, and H. Hauser. Interactive visual

analysis of complex scientific data as families of data surfaces. IEEE

Transactions on Visualization and Computer Graphics, 15(6):1351–1358,

2009.

[40] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv

preprint arXiv:1411.1784, 2014.

[41] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normaliza-

tion for generative adversarial networks. In Proceedings of International

Conference on Learning Representations, 2018.

[42] T. Miyato and M. Koyama. cGANs with projection discriminator. In

Proceedings of International Conference on Learning Representations,

2018.

[43] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltz-

mann machines. In Proceedings of International Conference on Machine

Learning, pp. 807–814, 2010.

[44] H. Obermaier, K. Bensema, and K. I. Joy. Visual trends analysis in time-

varying ensembles. IEEE Transactions on Visualization and Computer

Graphics, 22(10):2331–2342, 2016.

[45] D. Orban, D. F. Keefe, A. Biswas, J. Ahrens, and D. Rogers. Drag and

track: A direct manipulation interface for contextualizing data instances

within a continuous parameter space. IEEE Transactions on Visualization

and Computer Graphics, 25(1):256–266, 2019.

[46] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Con-

text encoders: Feature learning by inpainting. In Proceedings of 2016

IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–

2544, 2016.

[47] H. Piringer, W. Berger, and J. Krasser. Hypermoval: Interactive visual

validation of regression models for real-time simulation. In Computer

Graphics Forum, vol. 29, pp. 983–992. Wiley Online Library, 2010.

[48] J. Poco, A. Dasgupta, Y. Wei, W. Hargrove, C. R. Schwalm, D. N.

Huntzinger, R. Cook, E. Bertini, and C. T. Silva. Visual reconciliation

of alternative similarity spaces in climate modeling. IEEE Transactions

on Visualization and Computer Graphics, 20(12):1923–1932, 2014.

[49] K. Potter, A. Wilson, P.-T. Bremer, D. Williams, C. Doutriaux, V. Pas-

cucci, and C. R. Johnson. Ensemble-vis: A framework for the statistical

visualization of ensemble data. In 2009 IEEE International Conference

on Data Mining Workshops, pp. 233–240. IEEE, 2009.

[50] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Gener-

ative adversarial text to image synthesis. In Proceedings of International

Conference on Machine Learning, pp. 1060–1069, 2016.

[51] T. Ringler, M. Petersen, R. L. Higdon, D. Jacobsen, P. W. Jones, and

M. Maltrud. A multi-resolution approach to global ocean modeling.

Ocean Modelling, 69:211–232, 2013.

[52] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal

representations by error propagation. In D. E. Rumelhart, J. L. McClel-

land, and CORPORATE PDP Research Group, eds., Parallel Distributed

Processing: Explorations in the Microstructure of Cognition, Vol. 1, pp.

318–362. MIT Press, Cambridge, MA, USA, 1986.

[53] F. Samsel, M. Petersen, G. Abram, T. L. Turton, D. Rogers, and J. Ahrens.

Visualization of ocean currents and eddies in a high-resolution global

ocean-climate model. In Proceedings of the International Conference

for High Performance Computing, Networking, Storage and Analysis, Vi-

sualization & Data Analytics Showcase, 2015.

[54] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the

nonlinear dynamics of learning in deep linear neural networks. arXiv

preprint arXiv:1312.6120, 2013.

[55] M. Sedlmair, C. Heinzl, S. Bruckner, H. Piringer, and T. Moller. Visual

parameter space analysis: A conceptual framework. IEEE Transactions

on Visualization and Computer Graphics, 20(12):2161–2170, 2014.

[56] D. Shepard. A two-dimensional interpolation function for irregularly-

spaced data. In Proceedings of the 1968 23rd ACM National Conference,

pp. 517–524, 1968.

[57] K. Simonyan and A. Zisserman. Very deep convolutional networks for

large-scale image recognition. In Proceedings of International Confer-

ence on Learning Representations, 2015.

[58] R. Splechtna, K. Matkovic, D. Gracanin, M. Jelovic, and H. Hauser. Inter-

active visual steering of hierarchical simulation ensembles. In Proceed-

ings of 2015 IEEE Conference on Visual Analytics Science and Technol-

ogy, pp. 89–96, 2015.

[59] A. Tikhonova, C. D. Correa, and K.-L. Ma. Explorable images for visu-

alizing volume data. In Proceedings of 2010 IEEE Pacific Visualization

Symposium, pp. 177–184, 2010.

[60] A. Tikhonova, C. D. Correa, and K.-L. Ma. An exploratory technique for

coherent visualization of time-varying volume data. Computer Graphics

Forum, 29(3):783–792, 2010.

[61] A. Tikhonova, C. D. Correa, and K.-L. Ma. Visualization by proxy:

A novel framework for deferred interaction with volume data. IEEE

Transactions on Visualization and Computer Graphics, 16(6):1551–1559,

2010.

[62] T. Torsney-Weir, A. Saad, T. Moller, H.-C. Hege, B. Weber, J.-M. Ver-

bavatz, and S. Bergner. Tuner: Principled parameter finding for image

segmentation algorithms using visual response surface exploration. IEEE

Transactions on Visualization and Computer Graphics, 17(12):1892–

1901, 2011.

[63] J. Wang, L. Gou, H. Yang, and H.-W. Shen. Ganviz: A visual analyt-

ics approach to understand the adversarial game. IEEE Transactions on

Visualization and Computer Graphics, 24(6):1905–1917, 2018.

[64] J. Wang, S. Hazarika, C. Li, and H.-W. Shen. Visualization and visual

analysis of ensemble data: A survey. IEEE Transactions on Visualization

and Computer Graphics, 2018, Early Access.

[65] J. Wang, X. Liu, H.-W. Shen, and G. Lin. Multi-resolution climate ensem-

ble parameter analysis with nested parallel coordinates plots. IEEE Trans-

actions on Visualization and Computer Graphics, 23(1):81–90, 2017.

[66] H. G. Weller, G. Tabor, H. Jasak, and C. Fureby. A tensorial approach

to computational continuum mechanics using object-oriented techniques.

Computers in Physics, 12(6):620–631, 1998.

[67] C. Xie, W. Xu, and K. Mueller. A visual analytics framework for the

detection of anomalous call stack trees in high performance computing

applications. IEEE Transactions on Visualization and Computer Graph-

ics, 25(1):215–224, 2019.

[68] J. Xie, L. Xu, and E. Chen. Image denoising and inpainting with deep

neural networks. In Proceedings of Advances in Neural Information Pro-

cessing Systems, pp. 341–349, 2012.

[69] H. Zhang, I. J. Goodfellow, D. N. Metaxas, and A. Odena. Self-attention

generative adversarial networks. arXiv preprint arXiv:1805.08318, 2018.

[70] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a Gaussian

denoiser: Residual learning of deep CNN for image denoising. IEEE

Transactions on Image Processing, 26(7):3142–3155, 2017.

[71] Y. Zhou, Z. Zhu, X. Bai, D. Lischinski, D. Cohen-Or, and H. Huang. Non-

stationary texture synthesis by adversarial expansion. ACM Transactions

on Graphics, 37(4):49:1–49:13, 2018.

[72] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image

translation using cycle-consistent adversarial networks. In Proceedings

of 2017 International Conference on Computer Vision, pp. 2242–2251,

2017.