head-mounted display visualization system for surgical
TRANSCRIPT
1
Head-mounted display visualization system for surgical operations
Supervisor: Prof. Elena De Momi
Co-supervisors: Dr. Hang Su,
Candidate: Debora Bonvino, 920622
Dr. Salih Ertug Ovur
HAND’S MOVEMENT
Introduction
Minimally Invasive Surgery – Robot Assisted Surgery
Debora Bonvino
MINIMALLY INVASIVE SURGERY
ROBOT-ASSISTED SURGERY
OPERATOR SIDE PATIENT SIDEVIDEO STREAM
ENDOSCOPESURGICAL
INSTRUMENTS
VIDEO
2
CONSOLE MANIPULATORS
Debora Bonvino
Visualization systems
Robot - Assisted SurgeryMinimally Invasive Surgery
• Camera held by assistant
• 2D visualization on an external monitor
• 3D visualization on the console
• 2D external monitor for first assistant
Limited field of viewLack of depth perception
Hard eye-hand coordination
Limited field of viewHard eye-hand coordination
Uncomfortable head movements
State of the art
[G. Chen et al. ,2009] [C.D Van’t Hullenaar et al. ,2017]
3
State of the art
Debora Bonvino
[I.M. Sauer et al. , 2017][S. Drouin et al. , 2015] [L. Qian et al. , 2018]
HIDDEN STRUCTURE RECOGNITION PRE – INTRA OPERATIVE FUSION AR DEVICE – ROBOT INTEGRATION
Head-Mounted Display (HMD) - Augmented Reality (AR) 4
Move towards more versatile and intuitive solutions
• Case – Dependent Systems• Hard eye-hand coordination• Uncomfortable head movements
Aim
Debora Bonvino
Aim of the work
Implementation of a novel human-machine interface for AR visualization system and teleoperation using an AR device
AR DEVICE –DA VINCI
RESEARCH KIT (dVRK)
INTEGRATION
AUGMENTED REALITY
DEVELOPMENT • User-friendly interface• More ergonomic system• Coordination improvement• Accuracy improvement
GRAPHIC INTERFACE
SETUP
5
Methods
Debora Bonvino
System architecture
Video streamManipulators’ position
Surgeon’s movement
OPERATOR SIDEPATIENT SIDE
AUGMENTED SCENARIO
COMMUNICATION CHANNELTELEOPERATION GRAPHIC INTERFACE
6
Head-Mounted Display
joystick
Endoscope
Manipulator
Debora Bonvino
Methods
Graphic Interface Setup
GRAPHIC INTERFACE
SETUP
7
Left imagecompressed
Video split
Camera stream
Methods
Debora Bonvino
UNITY ENVIRONMENTIMPLEMENTATION
Right imagecompressed
Graphic Interface Setup
VIDEO STREAM
LEFT
10
80
1920
RIGHT
8
Methods
Debora Bonvino
UNITY ENVIRONMENTIMPLEMENTATION
Graphic Interface Setup
VIDEO STREAM
[P. Zhao et al. , 2010]
PHYSIOLOGICAL BINOCULAR VISION BASED
9
Methods
Debora Bonvino
UNITY ENVIRONMENTIMPLEMENTATION
Graphic Interface Setup
VIDEO STREAM
[P. Zhao et al. , 2010]
PHYSIOLOGICAL BINOCULAR VISION BASED
9
Methods
Debora Bonvino
AR Device - dVRK Integration
AR DEVICE –DA VINCI
RESEARCH KIT (dVRK)
INTEGRATION
AR VISION SYSTEM SETUP
WORLD – Laboratory Reference SystemECM – Endoscope Control ManipulatorHMD – Head-Mounted DisplayPSM – Patient Side Manipulator
10
Methods
Debora Bonvino
AR Device - dVRK Integration
JOYSTICK
HMD
ECM
PSM
ECM – Endoscope Control ManipulatorHMD – Head-Mounted DisplayPSM – Patient Side Manipulator
11
OPERATOR SIDEPATIENT SIDE
Methods
Debora Bonvino
HMD POSITION
AR Device - dVRK Integration
WORLD
WORLD – Laboratory Reference SystemECM – Endoscope Control ManipulatorHMD – Head-Mounted DisplayPSM – Patient Side Manipulator
12
Methods
Debora Bonvino
JOYSTICK– PSM POSITION
HMD POSITION
AR Device - dVRK Integration
WORLD
Joystick
WORLD – Laboratory Reference SystemECM – Endoscope Control ManipulatorHMD – Head-Mounted DisplayPSM – Patient Side ManipulatorVIRTUAL – Unity Reference System
12
Methods
Debora Bonvino
JOYSTICKBUTTON STATE
CONOTROLLERS’ POSITION
HMD’s POSITION
ENABLE/DISABLE
TRACKPAD TRIGGER GRIP
12AR Device - dVRK Integration
Debora Bonvino
Augmented Reality Development
AR DEVICE –DA VINCI
RESEARCH KIT (dVRK)
INTEGRATION
AR VISION SYSTEM SETUP
REAL SCENARIO
VIRTUAL INFORMATION
Methods
AUGMENTED REALITY
DEVELOPMENT
13
Methods
Debora Bonvino
INSTRUMENTSLOCALIZATOR
TARGET LOCALIZATOR
TARGET HIGHLIGHTER
SUPPORT FOR ORIENTATION
BETTER TARGET LOCALIZATION
SUPPORT FOR NAVIGATION
14Augmented Reality Development
Methods
Debora Bonvino
Test and Evaluation
TASK DESIGN EXPERIMENTAL PROTOCOL
15
Methods
Debora Bonvino
Task Design – Pick and Place
TASK 1 -CHESSBOARD
TASK 3 –RING INSERTION
TASK 2 –HEIGHT POSITIONING
STANDARD CONFIGURATION AR CONFIGURATION
16
Methods
Debora Bonvino
Experimental Protocol
10 non-medical participants
PROTOCOL
TASK 1: Chessboard
TASK 3: Ring insertion
STANDARD CONFIGURATION
No augmentation
3 repetitions
AR CONFIGURATION
Augmented guidance
3 repetitions
TASK 2: Height positioning
SUCCESSFUL SCORE COMPLETION TIME PERFORMANCE INDEX
min: 0max: 3
S𝒖,𝒌
S = total number of successful attempts u = specific user k = specific task repetition
𝑻𝒖,𝒌 [𝒔𝒆𝒄]
u = specific user k = specific task repetition
P𝒖,𝒌 =T𝒖,𝒌S𝒖,𝒌
T = completion timeS = successful scoreu = specific user k = specific task repetition
PERFORMANCE METRICS QUALITATIVE ANALYSIS
[H. Mnyusiwalla et al. ,2020][H. Mnyusiwalla et al. ,2020] [R. Campo et al. ,2010]
17
Results
Debora Bonvino
Successful Score 18
1,41,5
1,3
2,16667
2,033
2,16667
TASK 1 TASK 2 TASK 3
2.5
2
1.5
1
0.5
0
1.4
2.17
1.5
2.03
1.3
2.17
STANDARD AR
30.00390 0.00402
AV
ERA
GE
SUC
CES
SFU
L SC
OR
E
* Wilcoxon signed-rank test shown significative statistical differences (𝛼 = 0.05)
STATISTICAL ANALYSIS
BASIC ARWilcoxon signed-rank
test
STANDARDCONFIGURATION
ARCONFIGURATION
STATISTICAL ANALYSIS
ARWilcoxon signed-rank
test
TASK 1 < 0.05
TASK 2
TASK 3 < 0.05
p - values
Results
Debora Bonvino
Completion Time
STATISTICAL ANALYSIS
BASIC ARWilcoxon signed-rank
test
STANDARDCONFIGURATION
ARCONFIGURATION
TASK 1 TASK 2 TASK 3
STATISTICAL ANALYSIS
ARWilcoxon signed-rank
test
* Wilcoxon signed-rank test shown significative statistical differences (𝛼 = 0.05)
STD AR STD AR STD AR
STD = Standard configuration
19
TASK 1 < 0.05
TASK 2
TASK 3 < 0.05
p - values
CO
MP
LETI
ON
TIM
E
Results
Debora Bonvino
Performance Index
TASK 1 TASK 2 TASK 3
* Wilcoxon signed-rank test shown significative statistical differences (𝛼 = 0.05)
STD = Standard configuration
STD AR STD AR STD AR
20
0.00976STATISTICAL ANALYSIS
BASIC ARWilcoxon signed-rank
test
STANDARDCONFIGURATION
ARCONFIGURATION
STATISTICAL ANALYSIS
ARWilcoxon signed-rank
test
TASK 1 < 0.05
TASK 2 < 0.05
TASK 3 < 0.05
p - values
PER
FOR
MA
NC
E IN
DEX
0.03711
Conclusions
Debora Bonvino
Conclusions
User-friendly interface
Results achieved suggest that the system developed provides:
Orientation improvement
Efficiency improvement
21
Conclusions
Debora Bonvino
Future works
MORE COMPLEXSURGICAL TASKS
VISUAL FEEDBACK
22
ANATOMICAL 3D MODELS REGISTRATION
Methods
Debora Bonvino
Bridge the Windows system and Linux one:
Communication Channel
ROS#RosBridge_suite
23
Results
Debora Bonvino
Latency
HAND MOTION PSM MOTION
45 ms
30 Hz 15 msDELAY
HEAD MOTIONECM MOTION
90 Hz
VIDEO STREAM
DELAY
28
Methods
Debora Bonvino
HTC VIVE PRO
HARDWARE AND ACCESSORIES
• Head-Mounted display
• Joysticks
• Base stations
• Vive Tracker
• Wireless adapter
DEVICE CONFIGURATION
24
Results
Debora Bonvino
Qualitative analysis 35
AGE: 26.5 ± 1.3601
GENDER: 60% Female40% Male
SUBJECTS FEATURES:
DOMINANT HAND
What is your level of experience with virtual reality system?
How much do you rate your level of experience with video games?
Results
Debora Bonvino
Qualitative analysis 36
QUESTIONNAIRE:
I think I would need support from a technician to use this interface
STA
ND
AR
DA
R
Results
Debora Bonvino
Qualitative analysis 37
I still had/needed to learn a lot before becoming familiar with this interface.
AR
STA
ND
AR
D
QUESTIONNAIRE:
Results
Debora Bonvino
Qualitative analysis 38
I felt comfortable using this interface
AR
STA
ND
AR
D
QUESTIONNAIRE:
Methods
Debora Bonvino
Reference Systems
world
world
virtual
25
Methods
Debora Bonvino
HMD – JOYSTICK TRANSFORMATION
x
x
z
z
y
y
Controllers’ position
Joystick
26
Methods
Debora Bonvino
PSM TRANSFORMATION
VIRTUAL PSM
PSMs’ position
x
z
y
27