interaction techniques in medical volume visualization bernhard preim
TRANSCRIPT
Interaction Techniques in Medical Volume Visualization
Bernhard Preim
Interaction Tasks and Techniques
Interaction Tasks• Exploration of original data• Manipulation of transfer functions• Multiplanar reformatting (MPR)
Bernhard Preim 2
Exploration of Original Data
Bernhard Preim 3
• “Browsing” through the slice data• Simple contrast and brightness
control via mouse movement (windowing also called ramp transfer function)
• Flexible definition of slices in a corresponding visualization
• Cine mode for animation impression
• Opening and closing of a legend in the viewerPatient information (name, date of birth, gender, Id)Image information (modality, voxel size, recording date)Option: more or less detailed legend
• Synchronized display of two data sets Example: Liver CT; first data set without contrast agent, second data set with CASynchronization related to windowing parameters and the displayed layer
• Selection of the viewing direction (coronary, sagittal, axial)
Bernhard Preim 4
Exploration of Original Data
Example for legends, data: Univ. Hospital Leipzig
Bernhard Preim 5
Exploration of Original Data
Change of contrast and brightness, data: Univ. Hospital Leipzig
Bernhard Preim 6
Exploration of Original Data
Moving of a cross line in communicated views (Peter Hastreiter, Uni Erlangen)
Bernhard Preim 7
Historical model:Drawings by Dürer
Exploration of Original Data
Transfer functions: Mapping of data onto presentation parameters (colors, gray values, transparency)
• Determine the visibility and perceptibility of structures• Parametrization of TFs is an essential interaction for the
exploration of volume data.
Challenges:• Exploration of data sets with unknown structures• Exploration of data sets with different structures of similar
intensity
Bernhard Preim 8
Transfer Functions
Three volume visualizations of one CT data set with different opacity transfer functions. No exliplicit classification is applied, leading to the problems with „teeth“
Bernhard Preim 9
Skin Bones Teeth
Transfer Functions
Requirements• Selection of predefined TFs (e.g. liver CT, lung CT)• Targeted search for suitable TFs• Correlation between adjustable parameters and characteristics
of the resulting images• Definition flexibility• Fast preview
Bernhard Preim 10
Transfer Functions
Typical transfer functions:• Windowing (ramp)• Bi-/trilevel windowing• Inverse windowing• Piecewise linear functions• Polynoms of higher degree/splines
Problem: No recognizable relation between TF characteristics and visualization
Bernhard Preim 11
Transfer Functions
Thorax CT data set, emphasis of skeletal structures
Bernhard Preim 12
Transfer Functions
Bernhard Preim 13
Thorax CT data set, emphasis of blood vessels
Transfer Functions
Representation and application of TFs• Discrete representation in lookup tables• Size: e.g. 4096 entries with 32 bit (8 bit each for RGB and
alpha)• Hardware support for Lookup tables
Problem: hardware dependency w.r.t. size of color tables
Bernhard Preim 14
Transfer Functions
Transfer Functions
Sophisticated concepts:
• Stochastic generation of TFs that may be selected by the user (multilevel iterative search), presentation as thumbnails (He et al. [1996], König et al. [2001])
• Image-based TF design (Fang et al. [1998])
• Enhanced TF
Integration of image processing filters (e.g. edge recognition)Local TFMultidimensional TF (illustration of derived data, e.g. gradient fields, Levoy [1988])
Bernhard Preim 15
Transfer Functions
Stochastic generation of TFs:
Iterative search process (He et al. [1996]):
1. Use of an initial TF library
2. "Mutation" of this function through a genetic algorithm (25 generations)3. Direct volume rendering (back then with VolVis 100x100 pixel, 10s)4. Subjective result analysis by the user
16Bernhard Preim
17Bernhard Preim
Source:
König et al. [2001]
Transfer Functions
Transfer Functions
Image-based TF design• Idea: Definition of the transfer function, image information
serve as context (Castro et al. [1998])
• Global histogram• Histogram along a layer• Histogram along a ray
Bernhard Preim 18
Transfer Functions
Histogram along the orange ray as context for TF specification
Bernhard Preim 19
Image: Dirk Bartz, Univ. Leipzig
eye ball (light)muscles (dark)
Transfer Functions
RGB Alpha and gray value Alpha TF (Courtesy of Peter Hastreiter, Univ. Erlangen)
Bernhard Preim 20
Transfer Functions
Composition of a TF as weighted sum of component functionsComponents may represent known materials, e.g. fat, bone, …
Parameters of component functions:
Sb, Sc - inner sampling points, Sa, Sd - outer sampling points
Bernhard Preim 21
Transfer Functions
Adaptation of a trapezoid template to the local histogram of a rectangular region.
Bernhard Preim 22
Source: Castro et al. [1998]
The Transfer Function Bake-Off, Data: Sheep heart (IEEE CG&Application 5/6 2001, Pfisterer)• Comparison of different TF specification techniques1. ISO rendering of the segmented raw data (sheep heart) 2. Trial&Error - (20 min) with VolumePro3. Without data model - ISO automatically selected according to the maximum gradient
magnitude4. 2D TF with data model – automatic distance map, semiautom. opacity, manual color map
Bernhard Preim 23
Transfer Functions
Source: Pfisterer [2001]
Transfer Functions
Data-based TechniquesSelection of a transfer function that emphasizes edges.Edge model:
Perfect intersection between 2 structures is "blurred" through an error function (point spread function of the data acquisition). Assumption: Blurring through an isotropic Gaussian function. → fits to CT data well
Source: Kindlman, Durkin [1998]
Bernhard Preim 24
Transfer Functions
Data-based Techniques: edge enhancementEdge criteria: strong gradient g, very small second derivative
h (zero crossing):
-h(v)p(v) =
g(v)
Data values along an edge, 1st and 2nd derivative
Bernhard Preim 25
Transfer FunctionsDetermination of g (v) and h (v) via average determination from all first and second derivatives of all voxels with value v.Internal representation:Histogram volume H:
x-axis → f (v)y-axis → f“(v)z-axis → f´(v)
Algorithm:1. Determine min. and max. valuesfor f‘‘(v) and the maximum for f´(v). Minimum for f´(v) is assumed to be 0.
2. Fill H, whereas the values are scaled such that min and max are depicted from f´ and f´´ to 0 and 256.
26Bernhard Preim
Transfer Functions
Data-based Techniques: edge enhancement• What can be determined from the histogram volume?
Edge positions w.r.t. the data• What can be entered by the user?
• A selection of the "peaks" that shall be depicted• Form of the depicted peaks via boundary emphasis
function (bef)• Typical forms of bef()
Bernhard Preim 27
Transfer Functions
Data-based Techniques: edge enhancement
Applied 2D opacity functionand volume rendering of the Visible Woman data set (TF automatically determined).
The small image indicates the 2DHistogram (intensity values vs. gradient magnitude)
Brightness indicates frequency of the combination.
Source: Kindlmann, Durkin (1998)
Bernhard Preim 28
Transfer Functions
Data-based Techniques: edge enhancementComparison of edge-enhancing direct volume rendering and iso-
surface rendering
Illustration of a spiny dendrite based on microscopy data
Source: Kindlmann and Durkin 1998
Bernhard Preim 29
Transfer Functions
Data-based Techniques: edge enhancement• Preconditions for successful application:
Existence of clear object boundariesHomogenous dataOnly little noise, no "outliers"Medicine: CT data (if CA is applied, it must be equally distributed)
Bernhard Preim 30
Transfer Functions
Data-based Techniques• Use of a once specified TF as reference• Goal: "Re-use" of an empirically specified TF
Application: targeted illustration of a structure in a modality (e.g. aneurysms in MR)Procedure:
Selection of a reference data set Dref and a TF Tref(v)Use of the normalized histograms of the data sets H(Dref )
and H(Dstudy)Non-linear transformation t of the intensity values of Dstudy, such that H (Dstudy) ~ H (t(Dref))Hence, Tstudy (v) = Tref (v)
Bernhard Preim 31
Transfer Functions
Data-based Techniques: reference TF• Determination of the similarity of the histograms
1. Idea: minimization of the histogram distances2. Better idea: use of the p-function by Kindlmann (considers also f‘(v) and f‘‘(v))In case of comparable data sets the p-values are similar to the histograms
Literature: Rezk-Salama et al., VMV [2000]
Bernhard Preim 32
Transfer Functions
Data-based Techniques: Reference TF
• Visualization of blood vessels in the brain with CT angiography, left: no adaptation, middle: illustration of the first idea (histogram transformation), right: adaptation of the p-function Source: Rezk-Salama et al. [2000]
Bernhard Preim 33
Transfer Functions
Multidimensional TFs• 1D TF: Map data onto opacity/colors• Multidimensional TFs: Use additionally derived information,
e.g. strength of the gradient or the second derivative
• Typical example: Adaptation of the opacity to the strength of the gradient, emphasis of data
intersections• Advantage: Additional degrees of freedom to generate
high-quality images• Disadvantage:High interaction costs
Bernhard Preim 34
Transfer Functions
Multidimensional TFs• Consideration of the 2nd derivative (1st derivative of a scalar
field → vector, 2nd derivative → matrix)
• Hessian Matrix:
• Criterion (scalar value) for the 2nd derivative: largest eigenvalue of the Hessian Matrix and strength of the 2nd derivative, respectively in direction to the gradient (instead of the Hessian Matrix)
Bernhard Preim 35
Transfer Functions
Multidimensional TFs• Gradient calculation usually via central differences
• Mapping of the gradient size to the opacity (gradient magnitude weighted transparency)
Bernhard Preim 36
Transfer Functions
Multidimensional TFs• Volume visualization with a
gradient-dependent TF for opacity, according to Levoy [1988]) (Visible Human CT data set)
Bernhard Preim 37
2D Transfer Functions
Starting point for a simple specification: gradient intensity histograms. Filtering is important. Goal: accentuation of intersections.
Bernhard Preim 38
Source: Stölzl [2004]
2D Transfer Functions
Bernhard Preim 39
Dense tissue and bone parts with additional gradient emphasis (green marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]
2D Transfer Functions
Bernhard Preim 40
More dense soft tissue (yellow marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]
2D Transfer Functions
Bernhard Preim 41
Regions with high gradients are visualized (red marking)Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004]
2D Transfer Functions
Edge detector as input to define arcs
Bernhard Preim 42
Source: Stölzl [2004]
2D Transfer FunctionsLocal TFs• Motivation: Often, global TFs enable no sufficient
differentiation• Another kind of 2D TFs with the 2nd dimension being the
object id• Example: Division of a lookup table into 4 segments for 4
different illustrations
Caution: Interpolation beyond segment borders is not allowed!
Bernhard Preim 43
Source: Rezk-Salama [2002]
2D Transfer Functions
Blood vessels in the lung lobes are displayed with separate local TFs.
Bernhard Preim 44
2D Transfer Functions
Template-based specification of 2D TFs1D templates:
2D templates:
Bernhard Preim 45
Source: Tappenbeck [2006]
Source: Castro et al. [1998]
2D Transfer Functions
Template-based specification of 2D TFs:• Simplification of the interaction is even more important than in
the 1D case• Discretization in a Lookup table• Sufficient size required, at least 256x256• Applicable to arbitrary 2D domains (intensity: gradient
strength, intensity: distance to a target structure, …)
Bernhard Preim 46
2D Transfer Functions
Representation of 2D TFs in a rectilinear grid as basis for discretization in an LUT
Bernhard Preim 47
Source: Tappenbeck [2006]
Distance-dependent TFs:• Additional entries:
Segmented target structure (tagged volume)Distance transformation w.r.t. the target
• Use of an editor for 2D TF• Sample applications:
Fade-in of safety margins around tumorsOpacity control in case of large organs
Bernhard Preim 48
2D Transfer Functions
2D Transfer Functions
Target structure: lung surfaceSelection of interesting structures by intensity and distances
Bernhard Preim 49
Source: Tappenbeck [2006]
Resulting Volume Visualization (quite useless example; but appropriate illustration of the concept). Distance to the lung is used to assign different colors and opacity values.
Bernhard Preim 50
Source: Tappenbeck [2006]
2D Transfer Functions
Useful example: Emphasis of blood vessels in certain distances around a tumor via distance-dependent TFs
Bernhard Preim 51
Source: Tappenbeck [2006]
2D Transfer Functions
Size-Based Transfer Functions
Bernhard Preim 52
Left: With a 1D TF vessels are emphasized. Right: A size-based TF is adjusted to emphasize larger structures. Intensity and size are combined and yield a clear view to an aneurysm (From: Correa, 2008).
Size-based TFs are designed to emphasize features of a certain size.Basis: (Continuous) multi-scale representation of the dataset defined by a diffusion process or wavelets.
Size-Based Transfer Functions
Bernhard Preim 53
Left: With a 1D TF vessels and bones of the hand cannot be discriminated. Middle: Size is mapped to color to distinguish vessels and bone. Right: A size-based TF is adjusted to emphasize smaller structures making veins visible (From: Correa, 2008).
Size-based TFs were re-used and refined by others, e.g. [Wesarg, 2010]
Stroke-Based Transfer Functions
Bernhard Preim 54
TFs are defined based on user-drawn strokes. The blue stroke indicates what should be displayed and the green stroke what should be omitted (left). The histogramm of the strokes is analyzed (middle) and a TF is defined that separates the regions indicated by the strokes (From: Ropinski, 2008).
Advanced Concepts
Increased GPU support and better performance enables more advanced concepts
Data are analyzed, derived features are used for manipulating the TF
A more goal-directed process is supported with real-time feedback to allow also trial-and-error
Shape-Based Transfer Functions
• Objects are roughly segmented.• Objects are decomposed based on a skeletonization• Segments are merged with sophisticated techniques to
avoid that too small segments are classified• Subobjects are assessed w.r.t.
Tubiness (e.g. vessels, nerves have high values) Surfaceness Blobbyness (e.g. organs have high values)
Shape-Based Transfer Functions
Subobjects are represented by a point in the barycenter of shapes
A transfer function assigns opacity values based on this shape information
Elongated objects (red dots) are emphasized(From: Praßni, 2009)
Shape-Based Transfer Functions
Shape-Based TFs may emphasize roughly spherical pathologies, such as polyps in the colon and aneurysms. (The close-ups show renderings with conventional TFs) (From: Praßni, 2009)
Shape-Based Transfer Functions
Discussion:•Results of shape classification is available via an easy-to-use interface•Related to earlier work from Sato et al. (2000) where similar features are analyzed in a multi-scale approach on a per-voxel basis•No complex histograms need to be analyzed•Combination with size-based TFs would enable more fine-grained control
Visibility-Driven Transfer Functions
A visibility histogramm indicates for each intensity value how visible the voxel‘s with this value are depending on the current viewpoint and the current opacity transfer function
A fast GPU-based histogram computation is essential to update visibility histograms after rotation (Correa and Ma, 2009)
Computation of the visibility histogram(From: Correa and Ma, 2009)
Visibility-Driven Transfer Functions
The intensity histogramme and the visibility resulting from the current TF. Left: The opacity TF makes skin and flesh visibile thus hiding bones despite of the high opacity values (low magenta peak on bones). Right: By chosing low opacity values for skin and flesh they remain noticeable
but now bones are clearly visible (From: Correa and Ma, 2009)
Multiplanar Reformatting
MPR illustration of an MRT data set of the head.Left: The cutting plane indicates which slice is cut out from the original data.
Bernhard Preim 62
Application for vessel diagnostics • MPR is automatically oriented
orthogonal to the vessel centerline • Integrated view of cross section and
3D visualization
Bernhard Preim 63
Multiplanar Reformatting
Local MPR:Rotation around a locally interesting structure (tumor, vessel centerline)
Bernhard Preim 64
Multiplanar Reformatting
Anatomical Reformatting:
Idea: Use of segmentation results for a reformatting during which layers with a constant
distance to an anatomical structure (e.g. an organ surface) arise.
Procedure: - Lines of the data set are shifted against each other in such a way that voxels on the surface are located in a layer vertically to the viewing direction.
- The originally curved slices are viewed in layers → organ-specific coordinate system.
Feature: Anatomically reformatted layers show only voxels with same distances to the organ boundary.
Bernhard Preim 65
Multiplanar Reformatting
Bernhard Preim 66
Anatomical reformatting based on a lung lobe segmentation.Round lesions in the mediastinum.Source: Dicken et al., BVM 2003, Data: Prof. Günther (RWTH Aachen)
Multiplanar Reformatting
Summary
• Suitable interaction techniques are crucial for the practical application of medical visualization techniques.
• On the one hand, the interaction should be simple and clear. On the other hand, it should be flexible enough.
• Presets, or automatically adapting presets, are often a good basis.
• Transfer functions: Image galleries and gradient-dependent TFs are the standard.
Bernhard Preim 67
Literature
S. Castro, A. König, H. Löffelmann, E. Gröller, Transfer Function Specification for the Visualization of Medical Data, Technical Report, Institute of Computer Graphics and Algorithms, Vienna University of Technology, 1998,
C. Correa, K. L. Ma. Size-based transfer functions: A new volume exploration technique. IEEE Transactions on Visualization and Computer Graphics, 14 (6), 1380-1387, 2008
V. Dicken, B. Wein, H. Schubert et al. Projektionsansichten zur Vereinfachung der Diagnose von multiplen Lungenrundherden in CT-Thorax-Aufnahmen, Bildverarbeitung für die Medizin, Springer, Reihe Informatik aktuell, März 2003
S. Fang, T. Biddlecome, and M. Tuceryan. Image-Based Transfer Function Design for Data Exploration in Volume Visualization. In Proc. IEEE Visualization, 1998
T. He, L. Hong, A. Kaufman, and H. Pfister, Generation of Transfer Functions with Stochastic Search Techniques, in Proc. of IEEE Visualization, 1996
G. Kindlmann and J. W. Durkin. Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering, In Proc. of IEEE Symposium on Volume Visualization, 1998
Bernhard Preim 68
LiteratureM. Levoy, Display Surfaces from Volume Data, IEEE Computer Graphics and Applications, 25 (1988)H. Pfister, W. E. Lorensen, C. L. Bajaj, G. L. Kindlmann, W. J. Schroeder, L. Sobierajski Avila, K. Martin,
R. Machiraju, J. Lee: The Transfer Function Bake-Off. IEEE Computer Graphics and Applications 21(3): 16-22 (2001)
J. S. Praßni, T. Ropinski, J. Mensmann, K. Hinrichs. Shape-based transfer functions for volume visualization. Proc. of IEEE PacificVis, 2010, 9-16
C. Rezk-Salama, Peter Hastreiter, J. Scherer, G. Greiner: Automatic adjustment of transfer functions for 3d volume rendering. In Proc. of Vision, Modelling and Visualization, S. 357-364, 2000.
C. Rezk-Salama, Volume Rendering Techniques for General Purpose Graphics Hardware, Dissertation, Philipp-Alexander Universität Erlangen-Nürnberg
T. Ropinski, J. Praßni, F. Steinicke, K. H. Hinrichs: Stroke-Based Transfer Function Design. Proc. of Volume Graphics 2008: 41-48
Y. Sato, C.-F. Westin, A. Bhalerao, S. Nakajima, N. Shiraga, S. Tamura. Tissue classification based on 3D local intensity structures for volume rendering. IEEE Trans. Vis. Graph., 6(2):160–180, 2000.
D. Stölzel. Entwurf gradientenabhängiger 2D-Transferfunktionen für die medizinische Volumenvisualisierung. Master's thesis, Dept. of Computer Science, Univ. MD, 2004.
• Tappenbeck, B. Preim, V. Dicken. Distance-Based Transfer Function Design: Specification Methods and Applications. In Simulation und Visualisierung, pages 259-274, 2006
S. Wesarg, M. Kirschner, M. F. Khan: 2D Histogram based volume visualization: combining intensity and size of anatomical structures. Int. J. Computer Assisted Radiology and Surgery 5(6): 655-666 (2010)
Bernhard Preim 69