1

Lakmal Seneviratne

Professor of Mechatronics Professor of Mechanical Eng.

Kings College London Khalifa Univeristy, Abu Dhabi.

Tactile Interactions During Robot Assisted

Surgical Interventions

2

Overview

1. Surgical Robotics – Soft Robots

2. Tactile Interactions - Learning

3. MRI Compatible Force Sensing

4. Haptic Interfaces

5. In Hand Manipulations

3

King’s College London

4

Khalifa University

• Established in 2009

• Vision - To be a leading international

center of higher education and research

5

1. Surgical Robotics – Soft Robots

6

Surgical Robotics - Da Vinci Surgical System

www.intuitivesurgical.com

7

Surgical Robotics – Robotic Catheterisation

Hansen Medical: Robot-steered catheterization

tool for cardiac ablation procedures.

8

• Tools enter the body through narrow openings and manipulate soft organs that can move, deform, or change in stiffness.

• Teleoperated

• Restricted access (through Trocar ports),

minimal haptic feedback,

rigid robot tools,

confined space,

safety-critical, real-time, bio-compatible, sterilized tools, MRI compatible.

MIS

10

STIFF-FLOP - Inspiration

11

STIFF-FLOP

• Biological inspiration taken from octopus

• Soft, highly redundant manipulation device,

Embedded distributed sensing

cognitive development and intelligent control

Learning and cognitive reasoning - learn from physical interactions with environment,

12

STIFF-FLOP

•STIFF-Flop Concept

13

Octopus-like robot arm

•EU Project

•OCTOPUS

14

2. Tactile Interfaces and Learning

15

Tissue Palpation

Ex-Vivo Test Rig for Indentation

Ex-vivo indentation tests on bovine liver

- Measure Force Vs Displacement characteristics

16

Bovine Liver Static indentation (6mm probe)

Dual Maxwell Model for Palpation

Talal M. Al-ja'afreh, Yahya H. Zweiri, Lakmal D. Seneviratne, and Kaspar Althoefer, A New Soft Tissue Indentation Model for Estimating

Force-Displacement' Characteristics using Circular Indenters. Proc IMechE, 2008.

17

Rolling Palpations

Fz (Normal Force)

Fx (Drawbar Pull)

Force/Torque Sensor

Horizontal Motion

Vertical

Motion

6-DOF robotic

Manipulator

WheelTissue

Ex-Vivo Test Rig (Rolling Device)

18

Rolling Palpations – Soft Tissue

•Test results on pork kidney embedded with simulated tumor

•Kidney sample with an embedded nodule

Liu, H, Noonan, D. P., Challacombe, B. J., Dasgupta, P., Seneviratne, L. D., Althoefer*, K, Rolling Mechanical Imaging for Tissue Abnormality Localization During Minimally Invasive Surgery, IEEE Transactions on Biomedical Engineering, 2010.

19

Rolling imaging from a silicone phantom embedded with 9 nodules

Rolling Imaging

20

Rolling Images

21

Rolling Indentation of Human Prostates

•2D

•3D

•Phantom Omni

•Rolling Probe

22

FE Model for RI

23

•23

K. Sangpradit, H. Liu, P. Dasgupta, K. Althoefer, and L. Seneviratne, “Finite Element Modelling of Rolling Indentation” IEEE

Transactions on Biomedical Engineering, 2011.

• 3 nodules, 10 mm diameter

- Arruda-Boyce model. FE model parameters from unixial tests.

• FE results are in good agreement with the corresponding experimental data, RMS range 0.02-0.15 %

Material μ ,shear Modulus (kPa)

λm, Locking stretch

Mass Density (kg/m3)

Type of

mesh

Rubber (N1) 73.40 1.01 1000 CPS4R

Silicone (RTV6166 gel)

4.98 1.05 980

CPS4R

24

3. Force Sensing for Surgical Applications

25

Fibre Optic Uni-Axial Force Sensor

P. Puangmali, H. Liu, K. Althoefer, and L. D. Seneviratne, “Optical Fibre Sensor for Soft Tissue Investigation during Minimally Invasive Surgery”, ICRA 2008.

26

Fibre Optic 3 Axis Force Sensor

3-Axis Force Sensor

P Puangmali, H Liu, L D Seneviratne, P Dasgupta, K Althoefer. Miniature 3-Axis Distal Force Sensor for Minimally Invasive Surgical

Palpation. IEEE/ASME Transactions on Mechatronics. 2011,

27

The miniaturized sensor (11 mm diameter) consists of a force sensor and four displacement sensors

Fibre optic technique is applied , MRI-compatible

Force sensor

Displacement sensor

Fibre Optic Stiffness Sensor

Panagiotis Polygerinos, Lakmal D. Seneviratne, and Kaspar Althoefer, Triaxial Catheter-Tip Force Sensor for MRI-Guided Cardiac Procedures,

IEEE/ASME Transactions on Mechatronics. 2012

28

Calibration results of fibre-optic

force sensor

Force Sensing

(a) (b)

(c)

Indentation Depth sensing (different Orientations)

Silicone

Mini FID θz

Fibre Optic Stiffness Sensor

Hongbin Liu, Jichun Li, Xiaojing Song, Lakmal Seneviratne, Kaspar Althoefer. "Rolling Indentation Probe for Tissue Abnormality

Identification during Minimally Invasive Surgery", IEEE Transactions Robotics. 2011.

.

29

Fibre Optic Force Sensing for Cardiac Catheters

Wei Yao, Tobias Schaeffter, L Seneviratne. K Althoefer,

MR-compatible Catheter for Cardiac Catheterization,

ASME Journal of Medical Devices, 2012

30

3-Axis Catheter Force Sensor

Fibre-optic catheter-tip force sensor.

12Fr catheter-tip integrated with tri-axial force sensor.

Panagiotis Polygerinos, Asghar Ataollahi, Tobias Schaeffter, Reza Razavi, Lakmal D. Seneviratne, and Kaspar Althoefer.

MRI-Compatible Intensity-Modulated Force Sensor for Cardiac Catheterization Procedures. IEEE Transactions on Biomedical

Engineering, 58 (3), pp. 721-726. 2011.

31

• Constant pressure on 9mm

Sphere.

• Displacement of the ball

indicates a change in stiffness

of the surface tissue.

Airflow Force Sensor

32

Indika Wanninayake, Lakmal Seneviratne, Kaspar Althoefer, Novel Indentation Depth Measuring System for Stiffness Characterization in Soft Tissue Palpation. IEEE ICRA 2012

– Direct stiffness indication

– Tuneable force range (depend on the inlet air pressure)

– Friction free rolling over the tissue

– Array of Tactile Element (sphere diameter – 4mm)

– Sensitivity -0.008, diameter 18mm

Air Flow Tactile Probe

33

•33

Air Flow Tactile Probe

34

The location of the 3 nodules on in the silicone phantom correspond to

peaks.

Airflow Force Sensor – Experimental Evaluation

35

Airflow Force Sensor

• Low friction sensor motion

• Simple design with potential for miniaturisation

• Can be built from MR-compatible materials

36

Pseudo-Haptic Feedback

Force matrices of different indentation depths

Palpation input device

Robot arm

Rolling probe Force

sensor

Soft tissue

•Real soft tissue

•Virtual soft tissue

•Tissue

properties

•Haptic

feedback

•RMIS system

• Rolling

indentation probe

•Tissue

properties

37

Palpation simulation system with touch pad as input device

38

Palpation with Pseudo-Haptic Feedback

Cursor Real position

Tumor

Virtual force

v

Soft tissue

x

y

z

39

Pseudo-Haptic Feedback

• (a) cursor speed (b) cursor size

Min Li, Lakmal Seneviratne, Kaspar Althoefer. Tissue Stiffness Simulation and Abnormality

Localization using Pseudo-Haptic Feedback. IEEE ICRA 2012

40

HANDLE – EU FP7 IP UPMC (France), Shadow (UK), UC3M (Spain), FCTUC (Portugal), KCL (UK) ORU (Sweden), UHAM (Germany), CEA (France), IST (Portugal)

Project Objectives

• Characterization of object affordances • Learning and imitation of human strategies in handling tasks • Improving skills through 'babbling' • Autonomous in-hand dextrous manipulation

41

Learning Through Touch

Hongbin Liu, Lakmal Seneviratne, Kaspar Althoefer. Real-Time Local Contact Shape and

Pose Classification using a Tactile Array Sensor. IEEE ICRA 2012

42

Learning Through Touch

Right angle edge

λ1=121.9

λ2=22.5

λ3=7.5

flat surface

λ1=123.6

λ2=48.8

λ3=3.7

concave surface

(r = 45 mm)

λ1=123.6

λ2=53.3

λ3=7.5

sphere

(r = 6 mm)

λ1=28.7

λ2=18.1

λ3=15.6

vertex (corner)

λ1=30.1

λ2=8.45

λ3=6.23

cylinder

(r = 6 mm)

λ1=50.7

λ2=20.6

λ3=20.5

square

(a = 8 mm)

λ1=35.1

λ2=16.7

λ3=15.7

ring

(router= 10 mm,

rinner= 4 mm )

λ1=42.2

λ2=36.3

λ3=37.7

e1

e3

e3

e2

e1

e2

e3

e1

e2 e3

e1

e2 e3

e1

e1

e2 e3

e1

e2

e3

e2

e3

e1 e2

2

43

Challenges

Human-Robot system – Human in loop to deal with uncertainties. Monitoring, error recovery

Perception – Multi-modal (Tactile, Vision, etc)

Grasping and Manipulations

Tactile Interactions – Control, learn

Soft systems

44

Ribeiro