Antara: An Interactive 3D Volume Rendering and Visualization

Framework

Pratik Kalshetti Parag Rahangdale Dinesh Jangra Manas BundeleChiranjoy Chattopadhyay

Indian Institute of Technology Jodhpur

December 12, 2018

Abstract

The goal of 3D visualization is to provide the user with an intuitive interface which enables him to explorethe 3D data in an interactive manner. The aim of the exploration is to identify and analyse anomalies orto give proof of the non-anomaly of the visualized organic structures. For 3D Medical Data, MagneticResonance Images (MRI) have been used. To create the 3D model, we used the Direct Volume Renderingtechnique. In the input 3D data, we have x, y and z coordinates and an intensity value for each voxel. The3D data is used by Volume Ray Casting to compute 2D projections from 3D volumetric data sets. In raycasting, a ray of light is made to pass through the volume data. The interaction of each voxel with this rayis used to assign RGB and alpha values for every voxel in the volume. As a result, we are able to generatethe 3D model of the region of interest using the 3D data. The 3D model is interactive, thus enabling us tovisualize the different layers of the 3D volume by adjusting the transfer function.

1 Introduction

In medical field, visualizing the internal structure of body is very important for proper medical diagnosis [7]. Toaid doctors in visualizing the internal body parts, scanning methods like MRI (Magnetic Resonance Imaging),X-ray and CT (Computed Tomography) scan are used. Output of such methods are generally one or moregrayscale 2D images[15]. It is difficult to visualize the internal structure with the help of these 2D images.

To solve this problem, we propose an interactive 3D volume rendering and visualization framework Antara(Sanskrit: Interior) from the data obtained using MRI scan as shown in Fig. 1. This has been achieved usingVolume Rendering [11]. Volume rendering is an important graphics and visualization technique. A renderedvolume can be used for displaying not only surfaces of a model but also the intricate details contained within.In order to achieve our goal, we chose Direct Volume Rendering (DVR) technique for 3D visualization. It helpsto generate an interactive 3D model which can be altered to show surfaces having different intensities[9]. DVRmethods generate images of a 3D volumetric data set without explicitly extracting geometric surfaces from thedata. These techniques use an optical model to map data values to optical properties, such as color and opacity.During rendering, these optical properties are accumulated along each viewing ray to form an image of the data.This helps in generating accurate images of the part being visualized [2].

2 Literature Survey

Most of the fundamental volume visualization algorithms can be divided into two categories: DVR algorithmsand Surface Fitting (SF) algorithms. SF algorithms fit (usually planar) surface primitives such as polygons orpatches to constant-value contour surfaces in volumetric data-sets. The user begins by choosing a particular

Figure 1: Extracting 3D bone from MRI of knee.

1

arX

iv:1

812.

0423

3v1

[cs

.GR

] 1

1 D

ec 2

018

threshold value and the geometric primitives are then automatically fit to the high contrast contours in thevolume that match the threshold. Cells whose corner values are all above the chosen threshold or all belowthe threshold are removed and thus have no effect on the final image. SF algorithms constitute few of thealgorithms like opaque cubes, marching cubes, dividing cubes and marching tetrahedra[4]. SF methods aretypically faster than DVR methods since SF methods only traverse the volume once to extract the surfaces.After extracting the surfaces, rendering hardware and other rendering methods can be used to quickly renderthe surface primitives each time the user changes a viewing or lighting parameter. Changing the SF thresholdvalue is time consuming because it requires that all of the cells be revisited to extract a new set of surfaceprimitives. Also, SF methods suffer from several problems such as occasional false positive and negative surfacepieces and incorrect handling of small features and branches in the data. Also, it only displays surfaces whichmeet a threshold density and will only show the surface that is located closest to the imaginary viewer.

DVR methods perform mapping of elements directly into the screen without using geometric primitives asan intermediate representation. DVR algorithms include approaches such as ray-casting, integration methods,splatting, and V-buffer rendering. In DVR method, transparency and colors are used to allow a better represen-tation of the volume to be depicted in a single image. DVR retains all of the volume data during rendering, andthus all the acquired voxels may contribute to each rendered image. Spatial gradients is computed prior to orduring rendering, and these gradients, as well as the relative contributions of each voxel, is adjusted to produceimages of surfaces. Thus volume rendering can produce images of surfaces. Therefore, if volume rendering ofmedical data sets is inexpensive and fast enough, surface rendering will no longer be utilized.

Ray-casting[12] is the popular technique of volume visualization for the production of high quality images.It conducts an image-order traversal of the image plane pixels, finding a color and opacity for each. The colortuple and opacity values are added to the pixel using a weighting formula and a step is taken along the ray tothe next re-sample point. In ray casting, rays continue in a straight line until the opacity encountered by theray sums to unity or the ray exits the rear of the volume. No shadows or reflections are generated in ray-casting.Because the magnitude of the gradient is usually a good indication of the strength of an iso-surface within a cell,the result is a large color contribution to the pixel when the ray encounters a new substance in the volume[8].Ray-casting is CPU-intensive, but the images show the entire data-set, not just a collection of thin surfaces asin SF. 3D models were created using Ray casting algorithm [10] which is a volume rendering technique. Volumerendering techniques are computationally intensive, whereas the ray tracing algorithm [18, 3, 1], is faster interms of running time and avoiding redundant computations.

The important objectives in visualizing medical data is high quality rendering and provide real time inter-action. This paper highlights the use of ray tracing for volume rendering in the medical domain for getting aninsight into the interior stucture of the organs of the body using DVR, thus producing high quality renderedimage focusing even minute anomalies in the bones and tissues. Also the real time interaction is achieved withthe recusive formulation of the rendering equation. Another novel highlight of the paper is the metric used foranalysing the result of the volume rendering. The idea is an extension of the image segmentation metric usedin 2D images, thus opening gates to use such metrics for 3D data.

Figure 2: Overall framework of Antara.

2

Figure 3: Optical model used for characterizing the optic properties of the volume.

3 System Overview

The overall approach comprises of the modules explained in Fig. 2. Representing a surface contained within avolumetric data set using geometric primitives can be useful in many applications, however, there are severalmain drawbacks to this approach [14]. First, geometric primitives can only approximate surfaces containedwithin the original data. Adequate approximations may require an excessive amount of geometric primitives.Therefore, a trade-off must be made between accuracy and space requirements. Second, since only a surfacerepresentation is used, much of the information contained within the data is lost during the rendering process[17]. For example, in CT scanned data useful information is contained not only on the surfaces, but within thedata as well. Also, amorphous phenomena, such as clouds, fog, and fire cannot be adequately represented usingsurfaces, and therefore must have a volumetric representation, and must be displayed using volume renderingtechniques.

3.1 Volumetric Function Interpolation

The volume grid V only defines the value of some measured property f(x, y, z) at discrete locations in space. Ifone requires the value of f(x, y, z) at an off-grid location (x, y, z), a process called interpolation must be employedto estimate the unknown value from the known grid samples V (x, y, z). We have used trilinear interpolation inthis case. It can be written as 3 stages of 7 linear interpolations, since the filter function is separable in higherdimensions. The first 4 linear interpolations are along x:

f(u, v0, 1, w0, 1) = (1− u)(f(0, v0, 1, w0, 1)) + uf(1, v0, 1, w0, 1) (1)

Using these results, 2 linear interpolations along y follow:

f(u, v, w0, 1) = (1− v)(f(u, 0, w0, 1)) + uf(u, 1, w0, 1) (2)

and, interpolation along z yields the result:

f(x, y, z) = f(u, v, w) = (1− w)(f(u, v, 0)) + wf(u, v, 1) (3)

Here the u, v, w are the distances (cell of size of 13 (empirically determined)) of the sample at (x, y, z) fromthe lower left rear voxel in the cell containing the sample point.

Optical models for direct volume rendering view the volume as a cloud of particles. Light from a source caneither be scattered or absorbed by particles. The particles in an actual cloud occlude incoming light, as well asadd their own glow as shown in Fig. 3.

3.2 Direct Volume Rendering

Volume rendering or DVR is the process of creating a 2D image directly from 3D volumetric data, hence itis often called direct volume rendering[5]. We have used an image-order technique. It includes the followingoperations:

1. Cast rays from the image pixels, sampling the grid at discrete locations along their paths.

2. Obtain the samples via interpolation.

3. The interpolated samples are further processed to simulate the light transport within a volumetric medium.

The fundamental element in DVR is the volume rendering equation. It is obtained by using an absorption andemission model for particles. The equation is formulated as partial differential equation which is then solved byOrdinary Differential Equation (ODE) solver. Using the Riemann approximation for the integral present in thesolution of the partial differential equation, we get the discretized version of the same. The discrete version ofthe rendering equation [11] at the distance D of the image plane is given by:

I(D) = I0

n∏i=1

(1− αi) +

n∑i=1

gi ×n∏

j=i+1

(1− αi) (4)

3

Figure 4: Illustration of the ray casting technique.

where I0 is the background light, αi and gi is the opacity and color value associated the sampled volume datapoints assuming n sampled data points.

Note that the first term in Eq. 4 captures absorption whereas the latter one focuses on the emission. Theabove equation requires redundant computation for each of the data points which can be resolved by usingrecursive implementation called the Front-to-Back Compositing which is defined in Eq. 5 and Eq. 6.

Cout = Cin + C(x)α(x)× (1− αin) (5)

This formulation makes it possible to implement the approach and perform the process in real time.

αout = αin + α(x)× (1− αin) (6)

Here, Cin and Cout denote the color value before and after any iteration and similarly αin and αout denote theopacity value before and after any iteration.

3.3 Raycasting

In nature a light source is always required to visualize any object. Raycasting algorithm is used to determine thecolor and opacity associated with each voxel present in volume data due to the light source[13]. It is achievedby appling following operation on each voxel [12]:

1. Cast a ray into the volume

2. Linearly interpolate data values from voxel corners

3. Convert data values to optical properties viz. color and opacity

4. Composite the optical properties

5. Return the final color to assign to the image plane pixel

The above steps are shown in Fig. 4.

3.4 Shading

We have explored various shading models and found Phong Illumination Model (see Eq. 7) best suitable forthe purpose of obtaining the color value g associated with each sampled data value. This is further emphasisedin Sec. 4.5. Here, the normal is computed using central difference method to find the gradient.

gi = C(xi)× Ia + C(xi)× Id × (N.L) + C(xi)× Is × (R.V )n (7)

where C(xi) is the color of the sample i; Ia, Id, Is are the light’s ambient, diffuse, and specular components, Nis the normal at sample i, V is the vector from sample point to eye, L is the light vector from the sample tolight source, R is the reflection vector and n is the shininess. Now once obtained the color value, the next taskis to obtain the opacity value which can be obtained from the opacity function.

4

3.5 Opacity Function

This stage computes the opacity value of different data points within the sampled volume. The user provides theopacity value with the help of a transfer function defined as f(x) = α(x)∀x ∈ [1, ..., n] where α(x) is the opacityvalue for all the data points with intensity value of x. The rationale behind providing this user interaction isto clearly view the region of interest by setting its opacity to a larger value. Finally, the compositing valuesobtained, thus forms the pixel values for the image plane which projects the 3D volume data onto a 2D plane.An illustration of this stage is shown in Fig. 5. Consider two objects in the volume data. The pyramid is bluein color and the sphere is red. When viewed from the bold face of the cuboid as shown, only the sphere isrendered on the image plane since the opacity value for red intensity data points is set to a high value whereasthat for blue data points is set to a low value.

3.6 Rendering Algorithm

The overall idea is depicted in Fig. 6. Consider a volume data consisting of four data points with intensityvalues being red(a), yellow(b), blue(c) and green(d). The blue and red points lie at same depth(z1) while greenand yellow lie at a different depth(z2) and the former depth being nearer to the screen. The red and the yellowdata point lie on the same ray along the camera to a pixel on the image plane. The opacity function shows theopacity values for each of the intensity values present in the volume data. Since the red and yellow data pointsare projected on the same pixel on the image, the final value at the pixel is a weighted combination of thesetwo values where the weights are determined by the opacity function which is modified by the user.

The steps have been illustrated below:

1. For full opacity (value = 1), intensities a, c and d are visible on the image plane. Intensity b has no effectbecause it is behind a (see Fig. 6(I)).

2. Zero opacity value for a and d hides them completely in the image plane and intensity b (see Fig. 6(II)).

3. Setting 0 < a, b < 1, a mixture of both the intensities is visible on the image plane (see Fig. 6(III)).

4. When a = 1, varying b has no significance as it is obstructed completely by a (see Fig. 6(IV)).

4 Experimental Results

4.1 Dataset

The dataset used for evaluation of the proposed approach is the volume data obtained via MRI and CT of 5human organs (brain, foot, knee, tooth and heart). The ground truth is created by manually segmenting thebone region from each of the slices(images) from the scan and then validated by a medical expert. Also we havescans of an engine and a car for proving the proposed algorithm’s generalization capability in other domains.

4.2 Qualitative Analysis

Antara’s performance on different datasets is shown in Fig. 7. The colors in the figure are chosen for bettervisualization. Changing the opacity function for different intensities varies the rendered object, thus giving aninsight into the different structures as per the material characteristics. In Fig. 7, the outer structure is obtainedby giving high opacity values to them and gradually decreasing this value for the outer structures helps invisualizing the inner structure. The obtained results can this aid in understanding the hidden structure andthus find flaws in it. This will help in better surgery as per the patient needs. The proposed method achievesthe objective of high accuracy required in the visualization of medical images, which can render non geometricalshapes with precision. The removal of tissues yields exact visualization of bones, which is possible due to theirrelative intensity difference.

Figure 5: Effect of varying opacity on rendered objects.

5

Figure 6: Illustration of Antara’s rendering process. (I)-(IV): effect of varying opacity values of different voxels.

Figure 7: Result on different datasets: (a) Brain CT, (b) Foot CT, (c) Knee MRI, (d) Tooth CT, (e) Heart CT.

4.3 Generalization Capability

Antara not only works well with medical data but also works on data from other domains. In Fig. 8 data fromautomobile sector is worked upon by the proposed algorithm and shows high quality result giving insight intointernal structures. This data includes a porche car scan and an engine volume data. This aids in detectingflaws in the internal structure of the machine. This shows that Antara has far reaching effects on differentdisciplines like city architecture plans, electrical systems and industrial machines.

6

Figure 8: Result on automobile volume data.

4.4 Quantitative Analysis

The qualitative analysis clearly depicted the efficacy of Antara in the previous section. Quantitative analysis,in terms of accuracy of the proposed algorithm, provides a better analysis of the technique. To evaluate theperformance of our segmentation approach, we have used the Dice similarity coefficient index as a measure toquantify the accuracy of the obtained results, where the ground truth has been generated manually and latervalidated by the doctor.

The adopted technique is a novel method for analysing the volume data. The genesis of this technique canbe traced back from the fact that dice coefficient is a technique widely used in validation of segmentation ofmedical images. Extending this idea, and with the thought that the volume data in MRI is a formed by slicewise images of a particular organ, the rendered image can then be considered as a result obtained by segmentingthe corresponding slice from input volume data(MRI slices). The Dice coefficient is given by:

d = 2 ∗ |Rseg ∩Rgt||Rseg|+ |Rgt|

(8)

where Rseg is the segmented result of Antara and Rgt is the manually segmented ground truth.

As shown in Fig. 9, the dice coefficient of a slice obtained from the rendered volume when compared withthe ground truth slice containing only the bone results into a value close to 1 for almost all the slices. Thus,this technique provides higher accuracy results which are essential in the medical domain. The average dicesimilarity coefficient index obtained over the entire data set used, was found to be 0.8176.

This value clearly proves the efficiency of the results generated by the proposed approach and hence makingit suitable for using in applications where high precision is required. This principle of measuring the result of a3D data using the methods available for 2D data is in itself an innovation in the analysis phase in the domainof computer graphics.

4.5 Effect of shading and shininess

Shading plays an important role in visualizing a scene. Human vision uses shading as a cue to form positionand depth. Total handling of light is very expensive and shading model gives a good approximation of whatwould really happen much less expensively. The two less expensive techniques (i.e. real time interactive) widelyused are Phong model and the Cel shading model. The result obtained by Phong model helps better visualizingthe data points that are occluded from the scanning direction as compared to the Cel model. As shown in Fig.10 the encircled region is not highlighted clearly in the Cel Model, while it is shaded properly to visualize itappropriately.

The importance of incorporating all the three lights viz. ambient, diffused and specular is clear from Fig. 11,where the rendered image by using various combination of these lights is shown. All these images proves thatnone of these lighting models when used alone or pairwise provide reallistic results, thus making it a necessityto use all the three lights together to provide high quality realistic result.

Shininess is the coefficient of specular reflectivity associated with the surface of an object being rendered.Appropriate coefficient needs to be chosen as per the application. All the results shown throughout have beengenerated with the value of 60 for the coefficient. The variation of shininess has been shown visually in Fig. 12.

The rendered image in Fig. 13 with all the lighting components included and a value of 60 for the shininesscoefficient gives an accurate result obtained in real time. The results are generated from the MRI scan ofthe knee when viewed from different orthogonal planes. The color is chosen for better visualization. The

7

Figure 9: Quantitative Evaluation.

Phong Model Cel Model

Figure 10: Comparison between Phong and Cel Model.

recommended value for shininess is 60 using the Phong Shading model with all the three light components forthe medical dataset.

5 Conclusions

Antara works on the basis of differences in the intensities of the components of the 3D data. It is a multi-purposeplatform for the analysis of various types of patient data. E.g., analysis of bones as well as tissues, detection oftumors, etc. Antara also works for non-medical domain also (e.g. automobiles). There is a dearth of such lowcost, open-source software available in the market for analyzing 3D models of medical images. Such softwarescan be used for image guided surgery and can be put to other applications in the medical image analysis field.In future, we are planning to do a organ or tissue specific rendering. Techniques such as [6] can be combinedwith Antara to let the doctors perform a virtual test surgery.

References

[1] Parker, Steven and Parker, Michael and Livnat, Yarden and Sloan, Peter-Pike and Hansen, Charles andShirley, Peter (2005) Interactive ray tracing for volume visualization In:SIGGRAPH

[2] Raman, Siva P and Horton, Karen M and Fishman, Elliot K (2013) MDCT evaluation of ureteral tumors: advantagesof 3D reconstruction and volume visualization In: American Journal of Roentgenology

8

Figure 11: Effect of shading on the rendered object.

Figure 12: Effect of varying shininess values on rendering.

[3] Levoy, Marc (1990) Efficient ray tracing of volume data In: ACM TOG

[4] Okuyan, Erhan and Gudukbay, Ugur Direct volume rendering of unstructured tetrahedral meshes using CUDA andOpenMP In: The Journal of Supercomputing

[5] Drebin, Robert A and Carpenter, Loren and Hanrahan, Pat Volume rendering In: ACM SCG

[6] Shrivastava, Prateek and Das, Sukhendu (2014) Particle Coding for Meshfree Cutting of Deformable Assets,In:ICVGIP

[7] Post, Frits H and Nielson, Gregory and Bonneau, Georges-Pierre (2012) Data visualization: the state of the art In:Springer Science & Business Media.

[8] Soler, Luc and Nicolau, Stephane and Pessaux, Patrick and Mutter, Didier and Marescaux, Jacques (2014) Real-time3D image reconstruction guidance in liver resection surgery In: Hepatobiliary surgery and nutrition.

[9] Hohne, Karl Heinz and Bomans, Michael and Pommert, Andreas and Riemer, Martin and Schiers, Carsten and Tiede,Ulf and Wiebecke, Gunnar (1990) 3D visualization of tomographic volume data using the generalized voxel model In:Visual Computing.

[10] Adeshina, AM and Hashim, Rathiah and Khalid, Noor Elaiza Abdul and Abidin, Siti ZZ (2012) Hardware-accelerated raycasting: Towards an effective brain MRI visualization In: arXiv:1211.5595.

[11] Kajiya, James T(1986) The rendering equation In: SIGGRAPH.

[12] Ray, Harvey and Pfister, Hanspeter and Silver, Deborah and Cook, Todd A (1999) Ray casting architectures forvolume visualization In: IEEE CGA.

[13] J. Meyer-Spradow and T. Ropinski and J. Mensmann and K. Hinrich (2009) Voreen: A Rapid-Prototyping Envi-ronment for Ray-Casting-Based Volume Visualizations In: IEEE T-VCG.

[14] Nelly Gordillo and Eduard Montseny and Pilar Sobrevill (2013) State of the art survey on {MRI} brain tumorsegmentatio In: Magnetic Resonance Imaging.

Axial Coronal Sagittal

Figure 13: Different Views of Knee.

9

[15] R. A. Robb and C. Barillo (1989) Interactive display and analysis of 3-D medical images In: IEEE T-MI.

[16] Levoy, Marc (1990) Efficient Ray Tracing of Volume Data In: ACM TOG.

[17] Elvins, T. Todd (1992) A Survey of Algorithms for Volume Visualization In: SIGGRAPH Comput. Graph.

[18] Hachisuka, Toshiya and Jarosz, Wojciech and Weistroffer, Richard Peter and Dale, Kevin and Humphreys, Gregand Zwicker, Matthias and Jensen, Henrik Wann (2008) Multidimensional adaptive sampling and reconstruction forray tracing In: ACM TOG.

10