multi-sensory augmentation for simulation-based training

Multi-sensory Augmentation for Simulation-Based Training in Robot-Assisted Surgery

Chiara Galli de Paratesi

December 15, 2020

Master Thesis in Biomedical Engineering

Supervisor Academic TutorsElena De Momi Andrea Mariani

Academic Year 2019-20

Guido Caccianiga

Nome Cognome, assoc.prof. ABC Dept.

Introduction

Robot-Assisted Minimally Invasive Surgery (RAMIS)

Chiara Galli de Paratesi 2

da Vinci Surgical System (Intuitive Surgical Inc.)

Need for practical skills training

Complexity of surgery as sensorimotor task

Need to deal with new control modalities

Mas

ter

sid

e

Pat

ien

tsi

de


Introduction

Virtual Reality Simulation-Based Training

Training

in the Operating Room

High risk for patients.

Time consuming.

Requires significant human resources.

Does not provide a standardized

means of assessment.

Risk-free.

Versatile.

Possibility to extract performance

parameters.

Possibility to integrate with multi-

sensory augmentation modalities.


Training

in Virtual Reality simulation

3

dV Trainer (Mimic Technologies Inc.)


Introduction

Training Augmentation

AUGMENTATION

Guidance:Assists through the idealmovement to be learnt

Feedback:Explicitly shows the

user’s error



Introduction

Training Augmentation

AUGMENTATION


Vision AuditoryHaptic

• Hand-eye coordination• Context awareness• Depth perception

• Force learning• Manipulation• Tactile discrimination

• Time-related and repetitivemovements

5

[Zhu, 2020]


State of the Art

Augmentation Modalities

OTH

ER F

IELD

SR

AM

IS

[Huang, 2006]

[Malpani, 2017] [Brown, 2017]

[Yoon, 2017] [Sigrist, 2015]


Vision AuditoryHaptic

6

No firm consensus on the most suited

sensory modality for surgical skills acquisition


Research Objective

Subsequent combination of Augmented Guidance and Feedback delivered during the same training session.

1. Acquire procedural steps of the task 2. Refine movements through error-correction

AIM 1

Guidance and Feedback investigated in the sensory domains of visual, haptic and visuo-haptic.

Integration of a Virtual Reality task with augmentation modalities to optimize the acquisition of fundamental skills in robotic surgery.

" The subsequent combination of Guidance and Feedback leads to an optimization of motor learning. "

Hypothesis 1

" The sensory modality has an impact on the efficacy of the augmentation. "

Hypothesis 2


AIM 2

7


Materials and Methods

User Study Outline

3. Augmentations

1. Experimental setup

4. Acquisition Protocol

5. Performance Metrics

2. Virtual Reality Task




Experimental Setup

Hardware: Master console of

da Vinci Research Kit (dVRK)

Software: Assisted Tele-operation

with Augmented Reality (ATAR)


https://github.com/jhu-dvrk/sawIntuitiveResearchKit/wiki https://github.com/neemoh/ATAR/[Enayati, 2017]



Virtual Reality Task

Ring displacements are

used to generate the visual

and haptic feedback and to

evaluate the accuracy

performance

ENHANCED NEEDLE DRIVING


PICK-UP

PRE-ROTATION

INSERTION

TRANSFER

EXTRACTION

Hand-eye coordination

Steady hand

Accuracy

[Caccianiga, 2020]

10



Visual Augmentation: Guidance

Ghost gripper: for

position and orientation

during Pick-up and

Transfer

Reference frames: for

univocal Pre-rotation

Blu circle: for ideal

trajectory during

Insertion and Extraction




Visual Augmentation: Feedback


The color and number of

visual aids turned on

indicate the ring

displacement intensity

The position along the

visual aids crown

indicate the ring

displacement direction

[Caccianiga, 2020]

12


Force and torque perceived on the manipulator guiding to the desired pose


Haptic Augmentation: Guidance

𝑻𝒅: [ 𝒑𝒅, 𝒓𝒅 ]: desired manipulator pose

𝑻𝒄: [ 𝒑𝒄, 𝒓𝒄 ] : current manipulator pose

𝒗𝒄 : manipulator linear velocity

𝝎𝒄: manipulator angular velocity

𝐾𝑡, 𝐾𝑟: elastic coefficients (180 N/m, 0.15 N/m)

𝐷𝑡 , 𝐷𝑟: viscosity coefficients (15 N*s/m, 0.01 N*s/m)

VISCO-ELASTIC METHOD

𝒇 = 𝐾𝑡 ∗ 𝒑𝒅 − 𝒑𝒄 − 𝐷𝑡 ∗ 𝒗𝒄𝝉 = 𝐾𝑟 ∗ (𝒓𝒅 − 𝒓𝒄) − 𝐷𝑟 ∗ 𝝎𝒄


𝒇

𝝉

[ 𝒑𝒅, 𝒓𝒅 ]

𝒇,𝝉 ∈ ℝ𝟑: force and torque

[ 𝒑𝒄, 𝒓𝒄 ]



Haptic Augmentation: Feedback

𝒅𝒊: distance between equilibrium and current

pose of each ring

𝒗𝒄 : manipulator linear velocity

𝐾𝑡: elastic coefficient (360 N/m)

𝐷𝑡 : viscosity coefficient (40 N*s/m)

VISCO-ELASTIC METHOD

𝒇 =

𝒊=𝟏

𝟑

𝒅𝒊 ∗ 𝐾𝑡 − 𝐷𝑡 ∗ 𝒗𝒄

𝒅𝒊

𝒇 ∈ ℝ𝟑: force

Resistive force perceived on the manipulator resulting from ring displacement




Acquisition Protocol

32non-medical participants

4 groups

C : no augmentation

V : visual augmentation

H : haptic augmentation

VH : visual and haptic augmentation

cBASELINE cTRAINING cEVALUATION

c5 rep c c c5 rep10 rep

B ET1 T2

caugmented

guidancec

augmented

feedbackcno augmentation

PROTOCOL

cno augmentation

10 rep


rep = repetitions



Performance Metrics

PHASE COMPLETION TIME

RING DISPLACEMENT

𝑇𝑢,𝑘 = 𝑁 ∗ 𝑓𝑠𝑎𝑚𝑝𝑙𝑒

𝑃 = total number of completed repetitions for the specific stage, for each group rep = number of stage repetitions𝑁𝑢𝑠𝑒𝑟𝑠 = number of users of each group

𝑀𝑢,𝑘 = σ𝑖=13 σ𝑛 𝑑𝑖(𝑛)

𝑑𝑖 = distance between equilibrium and current pose of each ringi = spans the number of ringsn = spans the number of time samplesu = specific userk = specific task repetition

N= total number of samples𝑓𝑠𝑎𝑚𝑝𝑙𝑒 = sampling frequency

u = specific userk = specific task repetition

𝑃% =𝑃

𝑟𝑒𝑝 ∗ 𝑁𝑢𝑠𝑒𝑟𝑠* 100

𝑑𝑖



Results and Discussion

Phase Completion

Insertion phase Transfer phase Extraction phase

Ph

ase

Co

mp

leti

on

(𝑃%

)

Task Repetition

Ph

ase

Co

mp

leti

on

(𝑃%

)

Ph

ase

Co

mp

leti

on

(𝑃%

)

Task Repetition Task Repetition

B T1a T1b T2a T2b E B T1a T1b T2a T2b E B T1a T1b T2a T2b ET1a T1a T1aE E E


Steep improvement in the Phase Completioncapability after the introduction of Guidance

At evaluation: the experimental groups performed better with respect to control.

17


Tim

e (s

)Results and Discussion

Time


Statistical analysis within group: Wilcoxonsigned-rank test. Statistical analysis betweengroups: Wilcoxon rank-sum test .

Only the VH group outperformedcontrol at evaluationVisual clues might involve highercognitive elaboration.

18

bas

elin

e

evaluation

p values


Results and Discussion

Accuracy: Ring Displacement


Statistical analysis within group: Wilcoxonsigned-rank test. Statistical analysis betweengroups: Wilcoxon rank-sum test .

T1a

p C V H VH

C 0.023 <0.001 0.007 <0.001

V 0.879 0.008 0.328 0.038

H 0.505 0.442 0.008 0.798

VH 0.798 0.574 0.13 0.008

At evaluation: all experimental groups outperformed control.Almost no statistically significantdifferences between experimentalgroups.

Most relevant improvement: after the introduction of Augmented Guidance

19

bas

elin

e

evaluation

p values


Conclusions


• The introduction of Guidance led to a steep increase in the Phase Completioncapability and in the Time-Accuracy of the task.

• The subsequent application of Feedback caused a further, but not comparable, improvement acriss all performance metrics.

The subsequent combination of Augmented Guidance and Feedback leads to an optimization of motor learning.

• The most relevant difference between sensory domains was related to Time: haptic clues might involve a faster processing of the information with respect to the visual ones.

The potentiality of the haptic augmentation in the acquisition of practical skills wasderived: it might involve a lower cognitive elaboration.

AIM 1

AIM 2

20


Future Developments


Wider study population

Longer study protocol

More complex surgical tasks

Retention tests

Different combination of Guidance and Feedback

21


Any question?

Thank you for your kind attention!


<[email protected]>

22

* Equally contributed to the work

C. Galli de Paratesi*, G. Caccianiga*, A. Mariani*, A. Menciassi, E. De Momi, "Multisensory Guidance and Feedback for Simulation-based Training in Robot-Assisted Surgery: a Preliminary Comparison of Visual, Haptic, and Visuo-Haptic. " Submitted to RA-L and ICRA 2021.

Nome Cognome, assoc.prof. ABC Dept. 23


Subjective Metrics


How much do you think the presence of the augmentation increased the workload of the task?

Perc

enta

geo

f re

spo

nse

s

Perc

enta

geo

f re

spo

nse

s

Perc

enta

geo

f re

spo

nse

s



Subjective Metrics




Trajectory Error Metric


Used to validate the Ring Displacement Metric (developed specifically for this task)

𝑇𝑘 =

𝑗=1

200

𝑛𝑜𝑟𝑚(𝑛𝑐𝑗 − 𝑛𝑑𝑗)

𝑇𝑘 ∶ Euclidean norm of distance between current and desired needle points duringInsertion, Transfer and Extraction

k = task repetitionj = spans the number of data points𝑛𝑐𝑗 = pose along the current needle mesh of point j

𝑛𝑑𝑗 = pose along the desired needle mesh of point j

Insertion Extraction:Transfer:

𝑐 = 0.3757 𝑐 = 0.8875 𝑐 = 0.7517

Positive correlation between the two metrics



Time by phase: Insertion


bas

elin

e

evaluation

p values



Time by phase: Transfer


bas

elin

e

evaluation

p values



Time by phase: Extraction


bas

elin

e

evaluation

p values



Ring Displacement by phase: Insertion


bas

elin

e

evaluation

p values



Ring Displacement by phase: Transfer


bas

elin

e

evaluation

p values



Ring Displacement by phase: Extraction


bas

elin

e

evaluation

p values

multi-sensory augmentation for simulation-based training

Documents