stereoscopic video overlay with deformable registration balazs vagvolgyi prof. gregory hager cisst...
TRANSCRIPT
![Page 1: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/1.jpg)
Stereoscopic Video Overlay with Deformable Registration
Balazs VagvolgyiProf. Gregory Hager
CISST ERC
Dr. David Yuh, M.D.Department of Surgery
Johns Hopkins University
![Page 2: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/2.jpg)
The CASA Project
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Today’s Surgical Assistant: A Simple Information Channel
![Page 3: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/3.jpg)
The CASA Project
Stereo surface tracking
Stereo tool tracking
Virtual fixtures with
da Vinci Robot
Task graph execution system
HMM-based Intent Recognition
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Information Fusion with
da Vinci Display
Ultrasound
Capabilities of a Context-Aware Surgical Assistant (CASA)
Tissue Classification
PreoperativeImagery
![Page 4: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/4.jpg)
The CASA Project
Stereo surface tracking
Stereo tool tracking
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Information Fusion with
da Vinci Display
Developing a Context-Aware Surgical Assistant (CASA)
PreoperativeImagery
![Page 5: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/5.jpg)
Information Overlay
• Problem setting:– Given pre-operative scan
data from a suitable imagingmodality
– Video sequence from a stereo endoscope
• Add value– Overlay underlying anatomy on the stereo video
stream (x-ray vision)
– Include annotations or other information tied to imagery
Key Problem: Nonrigid registration of organ surface to data
![Page 6: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/6.jpg)
Inputs: What Do We Know?
1. Pre-operative 3D model- most probably volumetric- only a portion of it will be visible on the endoscope- anatomy will be deformed during the surgical procedure
2. Camera system properties can be measured- optical & stereo calibration- local brightness/contrast/color response
3. Stereo image stream- 3D surface can be reconstructed- texture information
4. A guesstimate of model–endoscope 3D relationship- We can guess where to start searching [i.e. patient position]
![Page 7: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/7.jpg)
Outputs: What Do We Generate?
1. Position of 3D model registered to stereo image
2. Model deformed to the current shape of anatomy
3. Rendering a synthetic 3D view on the stereo stream
4. Everything done real-time
Original Image Stereo Data Deformed Mesh
![Page 8: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/8.jpg)
2D 3D
All this in a flow chart
Stereo imagepre-processing
Building andoptimizing
disparity map
DeformableRegistration to
3D surface
3D texturetracking
Recognizingdeformations
optical parameters
stereo video stream
Imageoverlay
disparity
3D data
image data
parameters3D model
![Page 9: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/9.jpg)
Classical Stereo Vision: The Problem
• Blocks of each image are compared using SAD
• Optimization for each block independently on entire depth range
+ Very fast implementation (GPU)
¬ Lousy results
Small Vision Systemfrom Videre Design
(w/o structured light):
![Page 10: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/10.jpg)
• Input images downsized to several scale levels (½, ¼, …)• Each scale processed with the same algorithm
– Propagate coarse search results to the finer scale
+ Quality of disparity map is better + Even faster than single scale computation¬ Requires
structured light
Solution #1: Lighting and Multi-Scale
SVL implementation(using structured light):
![Page 11: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/11.jpg)
• Solve a (spatially) global optimization with regularization
– O(D) = min SAD(D) + Smooth(D)
• GLOBAL optimum found in polynomial time
Solution #2: Dynamic Programming
![Page 12: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/12.jpg)
1. Defining the recursive cost function
2. Memoization
3. Finding lowest cost path, which is the disparity map (DM in red)
SmoothnessError
Solution #2: Dynamic Programming
![Page 13: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/13.jpg)
Dynamic Programming on Images
• Minor issue: previous approach applies to scanline
• Approximate DP applied to entire image
- 3D disparity space (D):
- Cost function (C):
- Memoization (P):
![Page 14: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/14.jpg)
Dynamic Programming: Results
![Page 15: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/15.jpg)
Dynamic Programming: In Vivo Results
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm.
Raw Disparity Map Textured 3D Model
![Page 16: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/16.jpg)
Surface to 3D Model Registration
• Inputs:– point cloud from the stereo surface modeler– point cloud generated from a model or volume image
• Outputs:- transformation to register the 3D model to the 3D surface
![Page 17: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/17.jpg)
QuickTime™ and a decompressor
are needed to see this picture.
Results: Rigid Registration
Complete system (stereoplus registration) operatesat 5 frames/second
QuickTime™ and a decompressor
are needed to see this picture.
Current algorithm usesIPC with modificationsto account for occlusionsdue to viewpoint (z-buffer)
![Page 18: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/18.jpg)
From Rigid to Deformable
• Calculate residual errors in z direction
• Define a spring-mass system
• Perform local gradient descent
![Page 19: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/19.jpg)
Deformable Registration Results
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Final registration error of < 1mm exceptfor the area where the tool enters the image
![Page 20: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/20.jpg)
Coming in CASA
The Language of Surgery
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Tool Tracking
Tissue Surface Classification
![Page 21: Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns](https://reader031.vdocuments.net/reader031/viewer/2022032308/56649f435503460f94c643b4/html5/thumbnails/21.jpg)
Thank you!
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.