Gaze Controlled Robotic Camera

System

Anuj Awasthi

Anand Sivadasan

Veeral Patel

Outline Background Significance Problem Statement Concept Methodology Specific aims Budget Project Participation Time Frame

Background

• Laparoscopic robotic surgery• Eye tracker application

Visual mouse

• Human factors Computer vision based control Face mouse Voice control

Requirements in Laparoscopic Surgery

Maintain the surgical point of interest in the centre of the image.

Provide the required magnification of the area. Produce and maintain a horizontal image of the point of

interest. Perform the preceding actions automatically, although

they can be modulated by the surgeon.

Visual Mouse Application

Obtaining the horizontal and vertical coordinates with the eye tracker

The technique of live streaming of the horizontal and vertical coordinates

Interfacing of eye tracker and computer

Eye Tracker System

Human Factors Computer vision based control of robotic camera

Camera control based on computer vision tracking of the surgical tools. Image processing used to differentiate surgical tool of interest from

surroundings. No input required from the surgeon.

Disadvantages

Surgeon’s area of interest not taken into consideration Assumes surgical area to be surgeon’s area of interest always Surgeon ends up looking at corners of the screen often

Human FactorsFace Mouse control for robotic camera Image based system Tracks facial features of surgeon real time Controls camera based on pitch, yaw and roll of

surgeon’s face

Disadvantages Constant face movements causes strain Difficult to keep pace with movement of tools

Human Factors Voice control of Robotic camera Uses voice and pedal controls Uses voice recognition techniques Set of voice commands the camera Disadvantages Considerable burden on surgeon Difficult to perform dual inputs

Significance

Reduction in work load on surgeon Accuracy of surgical tasks Impact on surgical time Hands Free Control

Problem Statement

“To develop a camera control system which reduces the work load on the surgeon without compromising on the quality of surgeon’s video display ”

Concept Gaze based robotic camera

Acquire gaze of the surgeon with eye tracker.

Camera manipulation using eye tracker data interfaced with robot controls

Robotic Hardware

A small wireless 320 X 240 resolution camera with an inbuilt transmitter

A Receiver Set Two Servomotors (HS 422) Links Usbor Servo Controller Pivot Post Gripper Washer, Set of Clamps, Bolts, Nuts Eye tracker System

MethodologyOperation site Surgeon Site

Surgical Site

Server System (HOST Computer) Usbor Servo Controller

Visual C++ 6.0 Coding Servo Motors Robot Arm End Effecter

Inverse Kinematics to be followed Wireless Camera

AAA Battery supplied Receiver

Surgeon’s Site

Dedicated system (Client ) Image Acquisition through Internet

Streaming Video Live Motion JPEG System

Image Processing Intel’s Open CV Library Improve Brightness and Contrast

Eye tracker System

Fuzzy Based Control

PupilCluster 1

Cluster 2 Cluster 3

Cluster 4

Cluster 5 Cluster 6

Fuzzy C-Means Algorithm

Point of Gaze keeps fluctuating. Entire Eye tracker screen supposed to be

divided into clusters. Fuzzy C-Means Algorithm used. Degree of belongingness of the point of gaze

to a cluster is supposed to be the Degree of Membership of the fuzzy function

Point of Gaze co-ordinates assumed to be same as co-ordinates of cluster centers.

Specific Aims

To cover the surgical area with camera. To obtain the point of gaze of the surgeon

with eye tracker. To control the robotic camera based on the

point of gaze coordinates. To facilitate Surgeon’s view.

Budget

Project Participation

Robot Assembling : Veeral, Anuj & Anand Inverse kinematics : Anuj & Anand Software for Kinematics Control: Anuj &

Veeral Interfacing Eye tracker and Robot : Anuj,

Veeral & Anand Eye Tracker Output : Anand & Veeral

Time Frame

Task Time Duration

Conceptualization Jan 10th-Jan 25th

Literature Review Jan 26th-Feb 10th

Ordering Hardware Feb 15th

Proposal Writing Feb 15th-Feb 27th

Robot Assembling Feb 28th-March 5th

Software Development March 5th-March 25th

Final Report Writing March 25th-April 10th

Testing April 10th-April 20th

References

M. Farid,F. Murtagh,J.L. Starck.” Computer Display Control and Interaction using Eye-Gaze". School of Computer Science ,Belfast,UK.

Atsushi Nishikawa “Face Mouse : A Novel Human-Machine Interface for Controlling the Position of a Laparoscope” IEEE Transactions on Robotics and Automation,Vol. 19,No. 5 ,October 2003.

Murtagh F. ”Eye Gaze Tracking System-Visual Mouse Application Development”,3rd Year Training Report ,E.N.P.S. Engineering Degree, March-August 2001.

Reference (Contd..)

M.E. Allaf. “Laparoscopic Visual Field – Voice vs. foot Pedal interfaces for control of AESOP Robot "Surgical Endoscopy.Feb 1998.

A. Casals,J. Amat,E. Laporte. ”Automatic Guidance of an Assistant Robot in Laparoscopic Surgery” International Conference on Robotics and Automation, IEEE 1996.

R. Hurteau,S. DeSantis “Laparoscopic Surgery Assisted by a Robotic Cameraman:Concept and Experimental Results”IEEE 1994.

References (Contd….)

George P. Mylonas,Danail Satyanov. ”Gaze Contingent Soft tissue Deformation Tracking for Minimally Invasive Robotic Surgery” MICCAI 2005, LNCS 3749, pp. 843 – 850, 2005.

Shamsi T. Iqbal ,Brian P. Bailey. “Using Eye-Gaze Patterns to Identify User tasks”GHC04,2004

THANK YOU!!!!!!

QUESTIONS???????