Stereoscopic Video Overlay with Deformable Registration
Balazs VagvolgyiProf. Gregory Hager
CISST ERC
Dr. David Yuh, M.D.Department of Surgery
Johns Hopkins University
The CASA Project
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Today’s Surgical Assistant: A Simple Information Channel
The CASA Project
Stereo surface tracking
Stereo tool tracking
Virtual fixtures with
da Vinci Robot
Task graph execution system
HMM-based Intent Recognition
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Information Fusion with
da Vinci Display
Ultrasound
Capabilities of a Context-Aware Surgical Assistant (CASA)
Tissue Classification
PreoperativeImagery
The CASA Project
Stereo surface tracking
Stereo tool tracking
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Information Fusion with
da Vinci Display
Developing a Context-Aware Surgical Assistant (CASA)
PreoperativeImagery
Information Overlay
• Problem setting:– Given pre-operative scan
data from a suitable imagingmodality
– Video sequence from a stereo endoscope
• Add value– Overlay underlying anatomy on the stereo video
stream (x-ray vision)
– Include annotations or other information tied to imagery
Key Problem: Nonrigid registration of organ surface to data
Inputs: What Do We Know?
1. Pre-operative 3D model- most probably volumetric- only a portion of it will be visible on the endoscope- anatomy will be deformed during the surgical procedure
2. Camera system properties can be measured- optical & stereo calibration- local brightness/contrast/color response
3. Stereo image stream- 3D surface can be reconstructed- texture information
4. A guesstimate of model–endoscope 3D relationship- We can guess where to start searching [i.e. patient position]
Outputs: What Do We Generate?
1. Position of 3D model registered to stereo image
2. Model deformed to the current shape of anatomy
3. Rendering a synthetic 3D view on the stereo stream
4. Everything done real-time
Original Image Stereo Data Deformed Mesh
2D 3D
All this in a flow chart
Stereo imagepre-processing
Building andoptimizing
disparity map
DeformableRegistration to
3D surface
3D texturetracking
Recognizingdeformations
optical parameters
stereo video stream
Imageoverlay
disparity
3D data
image data
parameters3D model
Classical Stereo Vision: The Problem
• Blocks of each image are compared using SAD
• Optimization for each block independently on entire depth range
+ Very fast implementation (GPU)
¬ Lousy results
Small Vision Systemfrom Videre Design
(w/o structured light):
• Input images downsized to several scale levels (½, ¼, …)• Each scale processed with the same algorithm
– Propagate coarse search results to the finer scale
+ Quality of disparity map is better + Even faster than single scale computation¬ Requires
structured light
Solution #1: Lighting and Multi-Scale
SVL implementation(using structured light):
• Solve a (spatially) global optimization with regularization
– O(D) = min SAD(D) + Smooth(D)
• GLOBAL optimum found in polynomial time
Solution #2: Dynamic Programming
1. Defining the recursive cost function
2. Memoization
3. Finding lowest cost path, which is the disparity map (DM in red)
SmoothnessError
Solution #2: Dynamic Programming
Dynamic Programming on Images
• Minor issue: previous approach applies to scanline
• Approximate DP applied to entire image
- 3D disparity space (D):
- Cost function (C):
- Memoization (P):
Dynamic Programming: Results
Dynamic Programming: In Vivo Results
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm.
Raw Disparity Map Textured 3D Model
Surface to 3D Model Registration
• Inputs:– point cloud from the stereo surface modeler– point cloud generated from a model or volume image
• Outputs:- transformation to register the 3D model to the 3D surface
QuickTime™ and a decompressor
are needed to see this picture.
Results: Rigid Registration
Complete system (stereoplus registration) operatesat 5 frames/second
QuickTime™ and a decompressor
are needed to see this picture.
Current algorithm usesIPC with modificationsto account for occlusionsdue to viewpoint (z-buffer)
From Rigid to Deformable
• Calculate residual errors in z direction
• Define a spring-mass system
• Perform local gradient descent
Deformable Registration Results
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Final registration error of < 1mm exceptfor the area where the tool enters the image
Coming in CASA
The Language of Surgery
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Tool Tracking
Tissue Surface Classification
Thank you!
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.