Visualization of spatiotemporal data inimmersive virtual reality

Lukas LundgrenJoshua Hughes Höglund

Jonas Giek

January 30, 2018

1

Abstract

Visualizing results of stochastic and discrete reaction-diffusion dy-namics within biochemical networks is a challenging task. Here wepresent virtual reality as an option to do this. We have developedan application that imports simulation data from reaction-diffusionsimulations into an interactive VR environment. To aid data visual-ization, tools for visualizing trajectories and molecule reactions wereimplemented. Simulation tools such as importing pre-made simulationfiles, time stepping, running simulations in the cloud and importingdifferent types of meshes were implemented in order to make our ap-plication useful for model exploration. The performance tests of ourapplication shows that the frame rate is kept above 90 frames per sec-ond (FPS) for 500 molecules and above but drops to below 60 FPSwhen approaching 1000 molecules. Time stepping rapidly becomes anissue for more than 300 molecules and would require substantial coderefactoring to make more efficient.

2

Contents1 Introduction 5

2 Background of underlying dynamics and solvers 6

3 Results 73.1 Exploring a model . . . . . . . . . . . . . . . . . . . . . . . . 7

3.1.1 In-game interface . . . . . . . . . . . . . . . . . . . . . 73.1.2 Analyzing model data . . . . . . . . . . . . . . . . . . 9

3.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2.1 Import time . . . . . . . . . . . . . . . . . . . . . . . . 113.2.2 Frame rate . . . . . . . . . . . . . . . . . . . . . . . . . 123.2.3 Time stepping . . . . . . . . . . . . . . . . . . . . . . . 13

4 Design and Implementation 134.1 Development tools . . . . . . . . . . . . . . . . . . . . . . . . 14

4.1.1 Unity . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.1.2 HTC Vive . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Simulation tools . . . . . . . . . . . . . . . . . . . . . . . . . . 164.2.1 Importing simulation files . . . . . . . . . . . . . . . . 164.2.2 Running simulations in the cloud . . . . . . . . . . . . 164.2.3 Time stepping . . . . . . . . . . . . . . . . . . . . . . . 174.2.4 Importing meshes . . . . . . . . . . . . . . . . . . . . . 174.2.5 Exporting our application . . . . . . . . . . . . . . . . 19

4.3 Visualization tools . . . . . . . . . . . . . . . . . . . . . . . . 194.3.1 Molecule reactions . . . . . . . . . . . . . . . . . . . . 194.3.2 Trajectories . . . . . . . . . . . . . . . . . . . . . . . . 194.3.3 Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.3.4 Particle information . . . . . . . . . . . . . . . . . . . 20

5 Discussion 215.1 Is cloud computing necessary? . . . . . . . . . . . . . . . . . . 215.2 Known issues . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.3 Improving performance . . . . . . . . . . . . . . . . . . . . . . 225.4 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6 Conclusion 24

3

7 Availability and implementation 247.1 Video link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.2 Download link . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

8 References 24

4

1 IntroductionThe concept of virtual reality (VR) has been around for a long time. Perhapsthe best early description of modern VR is by Stanley G Weinbaum’s 1935sci-fi novel Pygmalion’s Spectacles where a pair of spectacles immersed theuser in a fictional world [1].

Many attempts of VR innovations were conducted during the 20th cen-tury, mainly focusing on a non-interactive cinematic experience. Among thenotable contributions were Morten Heilig’s Sensorama and Telesphere Mask,the latter being a headset with stereoscopic 3D and wide vision[1].

VR has been used for scientific visualization even before the release oftodays consumer grade VR hardware products such as the Oculus Rift andthe HTC Vive. Examples of applications are archaeological data analysis,biological fluid flow (bioflow) visualization, brain white-matter analysis, andMars terrain exploration [2]. Today there are several choices in hardwarefor viable VR usage. While most VR software is developed for entertain-ment purposes, there are notable exceptions. The company Looker developsLookVR, a program used to display and interact with data in VR [3]. Vir-tualitics develops an application for observing and interacting with data inVR. Their platform allows multiple users to cooperate in the same scene anduses machine learning to help the user find patterns in their data [4].

Data often resides beyond two dimensions but exploring data in threedimensions is still in its infancy even though that is our innate way of ob-serving the world. An experiment compared different groups ability to accu-rately recreate maps of the Mars terrain by watching panoramic 2D picturesversus viewing the terrain in VR. Although the sample size of the experimentwas too small to draw statistically significant conclusions, the results werein favor of VR [5]. Another example where VR shows promise is for neu-ron tracing, where researchers from the University of Utah have developed avisualization tool using VR [6].

In this report we describe our attempt at visualizing simulation resultsfrom stochastic reaction-diffusion simulations in VR. We have created anapplication that imports 3D position data over multiple time steps into animmersive and interactive virtual reality environment using the game en-gine Unity. Our approach is to explore the use of VR as a tool for modelexploration.

VR offers the user enhanced field of view and interactivity in order toprocess more information. In stochastic simulations it is common that sta-

5

tistical aggregates are used to measure the properties of simulation results.Important qualitative information can be lost in these aggregates and hardfound bugs or anomalies can possibly be found more easily with VR. In ourapplication, no real world experimental data exists for comparison. Visu-alization using VR is therefore of value to see such a process in a close toreal-life way.

VR also offers a new way to popularize science giving the user a moreimmersive experience when looking at a simulation. A biological cell can bevisualized in VR by scaling it up and making it interactable. The user canview it from different perspectives, and different simulations can be comparedby placing the cells next to each other.

The layout of this report is as follows: Section 2 outlines the theoryunderlying the simulations. In section 3 we describe the results, in section 4the design and implementation of the application and in section 5 we discussour results, methods, known issues in the application and possible furtherdevelopment.

2 Background of underlying dynamics and solversThe simulations that we have visualized model reaction-diffusion dynamics.In our case these reactions take place within a single cell. In a reaction-diffusion system different species of molecules exist which act as productsand reactants of biochemical reactions. The molecules in the system movearound by diffusion and based on proximity there is a probability that theywill react [7]. One application of a simulation based on reaction-diffusiondynamics is to describe gene regulatory networks [8].

There are different computational approaches for simulating a reaction-diffusion system. The simulation files that we use come from a simulatorusing a hybrid approach, where the challenging molecule interactions arehandled by a more expensive microscopic model and the rest is handled by acheaper mesoscopic model. These models are the microscopic Smoluchowskimodel and the mesoscopic reaction-diffusion master equation. In the micro-scopic Smoluchowski model the molecules are modeled as hard spheres andtheir positions are tracked continuously in space. In the mesoscopic reaction-diffusion master equation molecules are point-particles and their positionsare tracked up to the resolution of a structured or unstructured grid approx-imating the domain. The simulator used tries to determine when to use the

6

microscopic and mesoscopic model leading to an efficient simulation which isalmost as accurate as a pure microscopic simulation [9].

3 ResultsIn the first part of the results each element involved with the simulations areexplored. The different tools for analyzing the model are described and illus-trated. In the second part the performance of the application is examined.

3.1 Exploring a model

3.1.1 In-game interface

The user begins just outside of the room shown in Figure 1a at position 1.The MenuScreen is located at position 2, the MeshSelector is at location3 and the MeanChart is at location 4. The user can teleport to the menuscreen and choose the number of time steps they want in their simulations,as shown in Figure 1b. When the number of time steps has been submitted,the menu screen looks as shown in Figure 1c. Now the user can choose amesh as in Figure 1d and open a simulation file as done in Figure 1c. If nomesh is selected the default mesh is a sphere. In figure 1e below the threedifferent types of meshes are shown: an E.coli, a Sphere and a Bumpy mesh.

7

(a) (b)

(c)(d)

(e)

Figure 1: (a) Overview of the simulation room. (b) Menu screen allowing theuser to chose the number of time steps. (c) Menu screen after time step is selected,allowing the user to start importing a simulation file and reset the program. (d)Screen for choosing mesh. (e) The meshes in order from left to right: E.coli, Sphereand Bumpy

The user also has the option to make new simulations by selecting thenumber of initial molecules and choosing either a new E.coli or a sphere sim-

8

ulation. In Figure 2a the user creates a new simulation on a sphere mesh.In this particular example 292 molecules are simulated. In Figure 2b theapplication is shown shortly after pressing the New Simulation button. Theapplication opens a command prompt and starts computing the new simula-tion. The command prompt itself displays information about the simulation.

(a) (b)

Figure 2: (a) The user creating a new simulation by selecting the number ofmolecules and mesh for the simulation. (b) A simulation being performed in thecloud.

3.1.2 Analyzing model data

When the simulations have been selected or created the user can arrange thecells by picking them up and moving them around. In Figure 3a the userpicks up a sphere cell and scales it up. In Figure 3b several trajectories areshown, illustrated by segments of straight lines going from red to yellow astime moves forward.

In Figure 3c the visualization of molecule reactions are shown. In the firstimage the CellWorld is shown the time step before the reactions take place.In the second image three reactions take place, visualized by red particleeffects. The third image shows the later part of the same reactions as thethe second image.

In Figure 3d the increase of the total number of molecules is shown whilethe number of S1-molecules decrease. Minimum and maximum values aredisplayed to the right of each chart and the title tells which species is shown.Figure 3e shows the screen displaying the mean plot of all visualized simula-tions. Currently the S12-molecules are plotted, increasing from 0 to 14.

9

(a) (b)

(c)

(d) (e)

Figure 3: (a) Interacting with the cell by grabbing it, scaling it down or up. (b)Trajectories shown for several different molecules in a Sphere cell. (c) On the lefta TimeWorld is shown, in the middle the next time step showing reactions takingplace at three different locations and to the right a short while after the reactionshave taken place. (d) Charts for the same simulation and time step. The left chartshows all molecules and the right chart shows S1 molecules. (e) Chart that displaysthe mean values of all the simulations.

10

3.2 Performance

To estimate the efficiency of the application some performance tests weremade. All tests were made on the hardware specified below in Table 1.

Aspect InformationProcessor model Intel(R) Core(TM) i7-7700 Kaby LakeProcessor speed 3.6 GHzNumber of processors 4Number of threads 8Memory Corsair 16GB DDR4 3200MhzOperating System Windows 10Graphics card Asus GTX 1080 ROG Strix 8GB

Table 1: Computer specifications.

3.2.1 Import time

The time required to import simulation files with different number of moleculesand time steps was studied. As is seen in Figure 4 below, the time increasesclose to linearly as the number of time steps increases. This is because thetime required to generate the molecules in view does not depend on thenumber of time steps.

Figure 4: Time required to import simulation files from the application as afunction of the number of time steps imported.

11

3.2.2 Frame rate

In Figures 5a, 5b and 5c the time per frame has been recorded for simulationswith 100, 500 and 1000 molecules, respectively. These plots were taken fromUnity’s profiler when running the built version of our application. The 345recorded frames give a good indication of how the frame rate varies when theuser looks at a time-static cell from different angles.

This gives an indication of the performance when visualizing a simulation.The frame rate should be maintained over 90 frames per second (FPS) for VRapplications to avoid nausea [10]. As seen in figure 5a this criteria is fulfilledwhen 100 molecules are visualized. We see in Figure 5b that this criteriais also fulfilled for 500 molecules with occasional drops below 90 FPS. For1000 molecules however 90 FPS cannot be maintained as seen in figure 5c.One thing to note is that this problem is avoided when the user scales downthe cell to about half its size. This is due to fewer triangles having to berendered.

The different colors on figures 5a, 5b and 5c correspond to different partsof the simulation process. The blue part corresponds to the time spentrunning scripts, the green part is rendering and yellow is time spent synchro-nizing the frame rate with the monitors refresh rate (VSync).

(a) (b)

(c)

Figure 5: Time per frame for a cell with (a) 100 molecules, (b) 500 molecules and(c) 1000 molecules.

12

3.2.3 Time stepping

The maximum time stepping rate that can be achieved without noticeableperformance issues depends on the number of molecules in the simulation.For up to 200 molecules the maximum rate is approximately 0.015s betweeneach time step, corresponding to ∼ 67 updates per second. For approxi-mately 300 molecules the update rate has to be decreased considerably toavoid performance issues. As the number of molecules increases above ap-proximately 400 fast time stepping becomes impossible without noticeableframe rate drops, eventually crashing the application. Simulations this bighave to be time stepped manually one time step at a time.

4 Design and ImplementationWe decided to divide the cell into different parts. The complete cell is called aCellWorld. Each CellWorld consists of a number of different time steps. Oneof these time steps is called a TimeWorld. The TimeWorlds are populated bymolecules, where the locations of these molecules are specified by simulationfiles. There also exists a GameObject called GameControl which acts as amanager over the rest of the application. It handles the time stepping andcreates new CellWorlds from existing files and in the cloud.

In Table 2 the overall layout of the application is illustrated, where thescripts for each GameObject and its corresponding function is seen.

13

Gameobject Script Function

GameControlTestMeshcreate Creates CellWorlds.

TimeSteps Filters visible molecule types andchecks which TimeWorlds that shouldbe active.

NewSimulation Handles new simulations in the cloud.

CellWorldMoleculeCreator Creates TimeWorlds by reading the

simulation files.DataWork Computes and updates data in chart

VRTK scripts Allows grabbing and scaling Cell-Worlds.

TimeWorld TimeWorld Creates trajectories and visuals atmolecule reaction locations.

Molecule Molecules Determines how Molecules look and al-lows selection of Molecules.

MenuScreen FolderReader Reads folder for .h5 files and displaysthem as buttons.

Table 2: Overview of the applications design

4.1 Development tools

4.1.1 Unity

Unity is a game engine developed for rendering 3D and 2D objects and gamedevelopment. Unity greatly simplifies the development process, mainly be-cause many low-level graphics optimization techniques are already built in aswell as support for easy implementation of VR applications. Unity supportsdevelopment in both JavaScript and C#, where we decided to use C#. WithUnity we created the 3D world the user resides within and all objects insideit. The base of all entities in Unity are the so called GameObjects. GameOb-jects can be two or three dimensional geometries or for example text objects,and is the base of everything created in our 3D world. Scripts are used tointeract with the GameObjects [11]. Unity also has so called Assets, whichcan be downloaded from the Asset Store. These can be for example softwareor 3D models created by the community that can either be downloaded for

14

free or for a fee. We made use of several different assets to simplify the de-velopment process. The assets we have used are the Virtual Reality Toolkit(VRTK), Sci-Fi Styled Modular Pack, MeshChartFree and TextMesh Pro.

The Virtual Reality Toolkit (VRTK) is a collection of C# scripts whichaids in developing VR products [12]. It was used to implement the camera’sand controller’s interaction with the scene. It helped in interaction withboth the cell and UI-elements. It was also used to implement teleportation,enabling easier movement around the scene.

Sci-Fi Styled Modular Pack was used for designing the scene. It has alarge set of 3D-model prefabs which enabled the building of a realistic set forthe scene.

MeshChartFree was used as a base for all charts in the scene. It’s an assetfor different types of charts. The visuals are customizable and the charts canbe updated at runtime when the input data changes.

TextMesh Pro substituted Unity’s built-in 3D-text. It offers more cus-tomization options and appears much sharper in a 3D setting than Unity’sstandard feature.

4.1.2 HTC Vive

The HTC Vive is a VR headset that utilizes tracking technology to immersethe user in a virtual and interactable world. Our project was designed to-wards the use of this particular headset, but also has limited mouse andkeyboard interaction for bug testing when running in Unity. VR headsetssuch as the Oculus Rift does not offer the same immersion since it does notutilize room scale tracking, and thus the HTC Vive is preferable when mov-ing around in the room. The HTC Vive has two controllers which the useruses to teleport and interact with the environment [13] .

Figure 6: The HTC Vive headset and controllers.

15

4.2 Simulation tools

4.2.1 Importing simulation files

The simulated molecules’ position together with a unique ID for each molecule,are stored in the .h5 file format. If we want to extract the positions at thefirst time step of molecules of species S1, then we have the following hierar-chy: /Trajectories/0/Type_S1/positions_0. For the unique IDs we have:/Trajectories/0/Type_S1/unique_ids_0. The C# language does not havenative support for reading and writing to a .h5-file. We were not able to findopen-source software for C# that would solve this problem in Unity. In-stead we can open an external process which executes a Python script, sincePython has libraries for dealing with .h5-files. The positions and IDs in the.h5-file were written to a .txt file, which is easily manipulated in C#. Thecontents are then read into one array for each of the different species and timesteps.The arrays are then put into two different ArrayLists, one for positiondata and one for IDs. The time steps are then looped through and the Time-Worlds are created. The ID-vectors are looped through for each TimeWorldand molecules are created at the locations specified by the position arrays.The TimeWorlds each contain arrays which include information about theposition and of its molecules. All of this is done in the MoleculeCreatorscript.

4.2.2 Running simulations in the cloud

The software lets the user launch new simulations from the VR environment.We hosted the simulator on the SNIC Science Cloud, which we accessed usingPuTTY. PuTTY is a free terminal emulator, serial console and network filetransfer application [14]. The component of PuTTY we used was plink, acommand line connection tool [14].

When the New simulation button is pressed in the 3D environment thescript NewSimul is executed, which then calls the script NewSimulation.This script opens up an external process using plink, which from a commandinstruction runs SSH to access the simulator and run a new simulation. Whenusing the command instructions from plink you have the ability to choosea PuTTY key on the local computer. We used this to be able to accessthe server together with a password. When the simulation is complete theplink.exe process is closed.

To download the simulation file from the server we use WinSCP. WinSCP

16

is a free open source software used for secure file transfers [15]. We openeda winscp.com process to retrieve the simulation files as soon as they werefinished. winscp.com can run WinSCP by using console commands.

4.2.3 Time stepping

Time stepping is done by activating the Molecules in the current time stepand deactivating the Molecules in the previous one. This is achieved by firstchecking which CellWorld is selected and finding the new TimeWorld. Ifthe time step we try to access does not exist (for example time t = −1),the closest existing time step is shown. Time stepping is done by pressingleft or right on the HTC Vive trackpad. The SelectingMolecules script iscontinuously listening for user input, and sends information to the TimeStepscript when the user presses the trackpad. The TimeStep script then tellsthe TimeWorld script to change which TimeWorld is to be shown.

We also have a system that only keeps 200 (this number can be changedfrom Unity) of the TimeWorlds populated by children Molecules. This is toincrease the maximum amount of imported timesteps that our program canhandle since Unity has a maximum limit of GameObjects. This means thata certain number of Molecules have to be instantiated and destroyed eachtime step as well.

By pressing up on the HTC Vive trackpad a movie is started meaningthat time stepping is done several times a second. The default value for thisis 25 time steps per second. This update rate can be changed from Unity aswell. By pressing down on the trackpad the movie is run in reverse.

4.2.4 Importing meshes

When running the simulation from the simulator one of the inputs is thecell geometry. This cell geometry is stored as an .h5 file and includes all thetetrahedrons building up the cell. The application needs the boundary ofthe cell geometry, which is stored as ’mesh/boundaryfacets’ in the .h5 file.This includes all the vertices of the triangles building up the cell boundary.The mesh file is opened with an external Python program which writes thevertices to a text-file for Unity to open.

We have a seperate scene in Unity used for importing meshes which iscalled LukasScene. In this scene there is a GameObject called MeshLoadingwhich has a script attached to it called ReadMeshFile. This script is responsi-

17

ble for reading the mesh and has two public variables. One is a GameObjectin the scene called ImportedMesh which will be the target for the importedmesh file. This means that the user can view if the mesh importing wasdone correctly by looking at the imported mesh in the scene. The secondpublic variable is the file name of the mesh that needs to be imported. Thisis specified through the Unity Editor.

The script ReadMeshFile starts the Python program reading all the ver-tices and then stores the points as an array of Vector3:s. The script sets thevertices of ImportedMesh to match the imported mesh’s vertices. Meshes inUnity need a normal vector for each of the triangles to illuminate the trian-gles correctly. Unity can usually do this automatically by calling a functioncalled RecalculateNormals(), but to do this the ordering of the vertices of thetriangles has to be correct so that the normals point out of the mesh insteadof into the mesh. Unfortunately the format of the mesh that is importeddoes not take this into account.

To circumvent this we calculated the normals ourselves using the crossproduct. For each of the triangles we calculated the normal vector andchanged the sign if the normal pointed into the mesh. It is important tonote that this approach only works for convex meshes, because it can’t easilybe determined if a vector points in or out of the mesh otherwise. If one wantsto import a non convex mesh the ordering of the vertices in the mesh hasto be correct before importing the file. Another approach is to provide thenormal vectors for each of the triangles in the .h5 file that is imported andload these vectors into Unity.

For the mesh to be saved as a mesh file in the Assets folder some linesin the script has to be uncommented. Three lines of code with the comment"Only active when importing meshes. Gives build errors otherwise!" have tobe uncommented. The reason these code lines are commented is that theyprevent Unity from creating an executable file. Another thing to consideris that Unity has a 65535 vertices limit to its meshes. A way to avoid thislimitation is to create multiple GameObjects that contain different parts ofthe mesh, but this hasn’t been implemented in our software. We recommendusing a lower resolution of the visualized mesh if this problem would arise.

The imported mesh will be added as an asset file in the Assets folder.The mesh can now be added to a GameObject. Use a pre-existing prefabs,SphereWorld for example, and create a copy of this. Change the mesh onthe prefab to the imported mesh and save this as a new prefab. Adjust theposition and rotation of the prefab to make sure the orientation is correct.

18

Once the new prefab has been created add a new button to the Meshselecting screen. Create a copy of one of the already existing buttons anddrag your prefab to the MeshButtonSelect script. Now everything is readyfor the user to select this mesh in the VR environment.

4.2.5 Exporting our application

Unity has an interface that allows for creating an executable file for theirgames. Using a built version instead of the in editor version greatly improvesperformance and file size but also enables our application to be used by auser without Unity. One thing to note when building our version is that thefolder Files has to be copied from the Assets folder in the project to theData folder in the built version for the application to work. It is also in theFiles folder that all the simulation files are kept which means that the userwill have to move their own simulation files to this folder.

4.3 Visualization tools

4.3.1 Molecule reactions

When Molecules react a small explosion appears at the location of the reac-tion. To find this location, we utilized the molecules’ unique IDs. In eachtime step, the set of molecule IDs are compared to the IDs of the previoustime step. If the sets are not equal, some molecule reaction has occurred andnew molecules have been created. The set of IDs in the current time step arethen looped through and compared to the IDs in the previous one. Whenthe new molecule is found, the position of this Molecule is extracted and anexplosion is displayed at this location. This is done in the TimeWorld afterthe time stepping.

4.3.2 Trajectories

Visualizations of trajectories are used to illustrate the path the molecule hastaken up to the shown point in time. The trajectories are visualized by usinga Line renderer, which is a GameObject used to draw straight lines in space.One Line renderer is used for each trajectory. The trajectory of a moleculeis calculated by iterating backwards in time from the current time step untilwe either reach the first time step, or when the ID of the selected moleculedoes not exist (hence, the time step where the molecule is created). While

19

iterating, the position of the molecule at that point in time is saved in anarray. When the iteration concludes, a straight line is drawn between eachpoint.

If a trajectory has been calculated previously and is to be updated, theprocedure above is unnecessarily complicated. If the time stepping is madebackwards in time, the most recent point in the trajectory is simply removed.If the time stepping is made forwards in time, a new point is simply addedto the existing trajectory.

When a molecule is selected, the SelectingMolecules scripts sends infor-mation about the molecule to the Molecules script, where the Line renderer iscreated and the code in TimeWorld for calculating the trajectories is called.The Line renderers and the corresponding molecules are then saved in arraysin the selected CellWorld.

4.3.3 Graphs

Each CellWorld has a graph attached to it. A script on the correspondingCellWorld called DataWork feeds the chart with data collected from eachTimeWorld object. DataWork also continuously listens for input to updatethe graphs.

A graph is represented by the GameObject ChartofParticles which haseight children and a script called LookAtCamera which makes the graphsconstantly be turned against the user. Five of those children are TextMeshPro-objects displaying title and axis names as well as minimum and maxi-mum values of the chart. The other three are Data, Frame and Dot. Datarepresents the plotted line, using a script called LineChart to calculate andrender the data provided by DataWork. Frame renders a background framefor the graph. Dot is a GameObject with the shape of a cube that movesalong the plotted path to indicate which time step the user is on. The Dataand Frame objects are contained in the MechChartFree asset.

4.3.4 Particle information

Information about the molecule can be displayed by toggling on and off aGameObject called ParticleInfo. It consist of an object called canvas, whichprojects a transparent plane at a set distance in front of the camera whichcontains the necessary information. A script called InfoDisplay disables andenables the canvas and displays the data it collects from the TimeWorld and

20

SelectingMolecule scripts.

5 Discussion

5.1 Is cloud computing necessary?

To run new simulations from the 3D-environment we utilize cloud computing.This means that CPU workload can be offloaded from the application. Thedrawback is that downloading the simulation results can be expensive. Forsmall simulations this is a considerable fraction of the total time, while forlarger simulations the relative cost is smaller.

The simulations could be run on the local computer instead of in thecloud to alleviate the problem for small simulations. This could often bedone without a performance penalty. Unity rarely utilizes all processingcapacity, and launching a simulation as an external process did not causethe frame rate to decrease. To achieve good performance we made frequentuse of coroutines. Coroutines is a way for Unity to execute processes in thebackground over multiple frames [16].

The public version of the application does not have cloud computingenabled by default for security reasons. By allowing simulations to run onthe local computer this feature could be enabled by default.

5.2 Known issues

There are a few known bugs in the current version of the application. Ifone tries to show more time steps than exist in a file, the graph above theCellWorld will not work correctly and the mean graph will be wrong as well.Some problems with the time stepping also occurs, showing multiple timesteps at the same time when going past the actual number of time steps.

There is an issue the first time one tries to switch between which typesof molecules that are shown. Instead of moving to the second type, it movesdirectly to the third type. After this first press, the switching works perfectly.This seems like a simple bug, but we have so far been unable to understandthe cause.

Selecting a CellWorld is done by pointing a laser at a molecule withinit. This means there is no way of selecting a CellWorld that currently doesnot display any molecules. A fix for this would to allow the user to select

21

CellWorld by pointing the laser at the Dot on the graph since it is alwaysvisible.

If the user imports a new simulation and decides to reset the programwhile the file is being imported an issue arises. When the user tries to importa new simulation file the importing will never finish.

5.3 Improving performance

The biggest performance issue has to do with how multiple time steps arehandled. The current approach is to create a TimeWorld for each timestepand activating and deactivating the children of these (the molecules) whentime stepping. In this way, when time stepping, all of the molecules of thecurrent time step have to be deactivated and all of the molecules of the nexttime step have to be activated.

The system only keeps 200 (this number can be changed from the UnityEditor) of the TimeWorlds populated by children molecules. This meansthat a maximum of 200 TimeWorlds have children molecules and all otherTimeWorlds are empty. This is to increase the maximum amount of im-ported time steps that our program can handle since Unity has a maximumlimit of GameObjects. The problem that arises from this is that, assumingmany imported time steps from a simulation with 100 molecules, during eachtime step around 100 molecules need to be instantiated in one of the emptyTimeWorlds and around 100 molecules destroyed in one of the populatedTimeWorlds.

The problem is that instantiation and destruction of GameObjects arevery expensive operations in Unity and should be avoided if possible. Activa-tion and deactivation of GameObjects is also expensive but not as expensiveas instantiation and destruction. We can maintain a pretty high update ratewhen time stepping a system of 100 molecules but with many more moleculesa high update rate cannot be maintained.

A better way to handle time stepping would have been to translate thealready existing molecules when time stepping. Only when a reaction occursshould instantiation and destruction of molecules occur. We think that thiswould be the most effective way of optimizing our program. However, itwould require substantial code refactoring.

There is also opportunity to improve FPS performance so that moremolecules can be visualized at once in a time-static cell. At the momentthe application is run in Ultra settings meaning that the graphics are ren-

22

dered for visuals rather than performance. One thing to consider is that themolecules could be rendered as tetrahedrons instead of spheres. This changemight improve performance since fewer triangles need to be rendered.

Improving import time for simulation files could be worthwhile if it turnsout that users import simulations with around 5000 time steps or more sincethe import time then takes several minutes. This could be improved byfinding a way to use a C# library to read .h5 files that works with Unity.This would likely streamline the importing process since less time will bespent reading and writing .txt files. We suspect that Unitys use of an older.NET framework might cause the library not to work with Unity. RichardFine from Unity has stated that in the future Unity will have improved C#functionality with native .NET 4.6 framework support [17]. At the time ofwriting (January 2018) the .NET 4.6 support is only experimental and hassome bugs. If the import approach is still to use Python scripts there isopportunity to parallelize reading and writing of .txt files.

5.4 Future work

To make this software more useful more work to generalize it should be done.This program is only able to read .h5-files structured with one simulation perfile. The visualization process is also made for exactly 10 types of molecules.Adding an interface for importing custom meshes without the user having tochange the source code would make it suitable for a larger audience as well.

At the moment the application only has two input options when creatinga new simulation, the number of molecules and what mesh to use. Addingmore input options would make this feature more useful. The option tochange attributes of different molecule species or reaction variables wouldbe straightforward to add. More challenging would be to allow the userto move molecules around from within the VR interface and then use thenew configuration as an initial condition for a new simulation. This wouldallow the user to make continued simulations based on a visualized cell whichhas had its interior molecules moved or deleted. This change would greatlyenhance our application as a tool for model exploration and would betterutilize the interactivity that VR provides.

The time stepping interaction can also be improved in several ways. Ifthe user wants to move to a certain point in time, they will have to movethrough all time steps up to that point. This could be solved by having aslider for selecting which time step to go to, or to allow the user to type in

23

the time step. The fast time stepping for simulations with many moleculescan also be improved by scaling the update rate according to the number ofmolecules, which would help prevent the application from crashing from toohigh work loads.

6 ConclusionWe have created an application that visualizes simulations modeling reaction-diffusion dynamics in immersive VR. Visualization tools such as trajectoriesand molecule reactions and simulation tools such as importing pre-made sim-ulation files, time stepping, running simulations in the cloud and importingdifferent types of meshes were implemented to make the application a usefultool for data visualization and model exploration. The application can befurther improved by importing the simulation data in C#, generalizing thecode to handle different molecule types, optimizing the time stepping and al-lowing the user to make continued simulations based on the visualized cell’scurrent characteristics.

7 Availability and implementation

7.1 Video link

A 15 minute long video tutorial and demonstration of our application:https://www.youtube.com/watch?v=RmIp86ZA24I

7.2 Download link

Download the application and source code here:goo.gl/u5Ja8k

8 References[1] V. R. Society, “History of virtual reality.” https://www.vrs.org.

uk/virtual-reality/history.html, 2017. [Online; accessed 11-January-2018].

24

[2] A. van Dam, D. H. Laidlaw, and R. M. Simpson, “Experiments in immer-sive virtual reality for scientific visualization,” Computers & Graphics,vol. 26, no. 4, pp. 535 – 555, 2002.

[3] Looker, “LookVR.” https://looker.com/platform/blocks/embedded/lookvr, 2018. [Online; accessed 11-January-2018].

[4] Virtualitics, “Virtualitics.” https://www.virtualitics.com/,2017. [Online; accessed 11-January-2018].

[5] C. Donalek, S. G. Djorgovski, A. Cioc, A. Wang, J. Zhang, E. Lawler,S. Yeh, A. Mahabal, M. Graham, A. Drake, S. Davidoff, J. S. Norris, andG. Longo, “Immersive and collaborative data visualization using virtualreality platforms,” in 2014 IEEE International Conference on Big Data(Big Data), pp. 609–614, Oct 2014.

[6] W. Usher, P. Klacansky, F. Federer, P.-T. Bremer, A. Knoll, J. Yarch,A. Angelucci, , and V. Pascucci, “A virtual reality visualization toolfor neuron tracing,” IEEE Transactions on Visualization and ComputerGraphics, pp. 994–1003, 2018.

[7] T. R. Sokolowski and P. Rein ten Wolde, “Spatial-Stochastic Simulationof Reaction-Diffusion Systems,” ArXiv e-prints, 2017.

[8] M. Sturrock, A. Hellander, S. Aldakheel, L. Petzold, and M. A. Chap-lain, “The role of dimerisation and nuclear transport in the Hes1 generegulatory network,” Bull. Math. Biol, 2014.

[9] S. Hellander, A. Hellander, and L. Petzold, “Mesoscopic-microscopicspatial stochastic simulation with automatic system partitioning,” J.Chem. Phys, 2017.

[10] “The importance of frame rates.” https://help.irisvr.com/hc/en-us/articles/215884547-The-Importance-of-Frame-Rates. [Online; accessed 16-January-2018].

[11] W. contributors, “Unity (game engine) — Wikipedia, the free ency-clopedia.” https://en.wikipedia.org/w/index.php?title=Unity_(game_engine)&oldid=819050758, 2018. [Online; ac-cessed 10-January-2018].

25

[12] “Welcome to VRTK.” https://vrtoolkit.readme.io/docs,2018. [Online; accessed 15-January-2018].

[13] W. contributors, “HTC Vive — Wikipedia, the free encyclope-dia.” https://en.wikipedia.org/w/index.php?title=HTC_Vive&oldid=819477784, 2018. [Online; accessed 10-January-2018].

[14] W. contributors, “Putty — Wikipedia, the free encyclopedia.”https://en.wikipedia.org/w/index.php?title=PuTTY&oldid=798126304, 2017. [Online; accessed 11-January-2018].

[15] “Introducing WinSCP,” 2017. [Online; accessed 11-January-2018].

[16] “Unity - Manual: Coroutines,” 2017. [Online; accessed 25-January-2018].

[17] “Unityscript’s long ride off into the sunset.” https://blogs.unity3d.com/2017/08/11/unityscripts-long-ride-off-into-the-sunset/. [online; accessed 18-January-2018].

26