recent advances in robot-assisted surgery: soft …real.mtak.hu/101974/1/5 17_saci2019.pdfrecent...

Recent Advances in Robot-Assisted Surgery: SoftTissue Contact Identification

Tamas D. Nagy∗ and Tamas Haidegger∗†∗Antal Bejczy Center for Intelligent Robotics, Obuda University, Budapest, Hungary

†Austrian Center for Medical Innovation and Technology (ACMIT), Wiener Neustadt, AustriaEmail: {tamas.daniel.nagy, tamas.haidegger}@irob.uni-obuda.hu

Abstract—Robot-Assisted Minimally Invasive Surgery(RAMIS) is becoming standard-of-care in western medicine.RAMIS offers better patient outcome compared to traditionalopen surgery, however, the surgeons’ ability to identify thetissues with the sense of touch is missing from most roboticsystems. Regarding haptic feedback, the most promisingdiagnostic technique is probably palpation; a physical contactexamination method through which information can be gatheredabout the underlying structures by gently pressing with thefingers. In open surgery, palpation is widely used to identifyblood vessels, tendons or even tumors; and the knowledge onthe exact location of such elements is often crucial with respectto the outcome of the intervention. This paper presents a reviewof the actual research directions in the field of palpation inRAMIS.

Index Terms—Palpation, Surgical Robotics, Haptic Feedback,Robot-Assisted Minimally Invasive Surgery.

I. INTRODUCTION

The advancements of the last few decades reshaped the faceof surgical interventions radically. The technique of MinimallyInvasive Surgery (MIS) started a revolution, when operatingthrough small incisions using so-called laparoscopic instru-ments, while the visual feedback is provided by endoscopiccameras. This technique offered a number of benefits, such asfaster recovery or lower risk of complications, and so becamea standard in clinical practice across specialities. MIS also pre-sented new challenges to the surgeons, like the limited rangeof motion or operating in cumbersome body positions. Robot-Assisted Minimally Invasive Surgery (RAMIS) appeared toease these difficulties; the surgeon is able to operate in acomfortable, seated position at the master console, while theirmotion is copied by the slave, or patient side instruments.These teleoperated systems—of whom probably the mostfamous is the da Vinci (Intuitive Surgical Inc., Sunnyvale,CA)—offer enhanced vision and dexterity alongside superiorergonomy [1], [2].

In the case of traditional, open surgical practice, manual pal-pation is frequently used to gain information about the deeper,non-visible layers of soft tissue. This way the surgeon canidentify anatomies with different stiffness to the surroundings,like nerves, blood vessels and tumors as well, since canceroustissue is usually harder than its environment. In MIS, tissuestiffness is commonly investigated using a procedure calledinstrument palpation—the tissue is palpated by a long instru-ment through the trocar, however this technique is less accurate

and less sensitive than manual palpation [3]. Unfortunately,most of the current RAMIS systems still lack the ability offorce sensing and haptic feedback, thus instrument palpationis infeasible.

II. METHODS

In this review, we followed the Preferred Reporting Itemsfor Systematic Review and Meta-Analysis (PRISMA) [4]. Tofind relevant publications in the field of haptics and palpationin RAMIS, the databases PubMed and Google Scholar wereused. Since this paper focuses mainly on palpation in RAMIS,the area of haptic feedback is only briefly addressed. Dur-ing the search procedure the keywords ’palpation’, ’surgery’,’stiffness’, ’feedback’, ’sensor’ and ’autonomous’ were usedtogether. After the analysis according to PRISMA (Fig. 1.),35 relevant studies remained from the area of palpation, thoseare organized into Table I, with the following columns:

• ’Force sensing method’: force sensing techniques aredivided to thee categories: direct, indirect and vision-based

• ’Platform’: the robotic platform the project based on;• ’Key aspect of the study’: brief description of the results;• ’Form of evaluation’: the method used to validate the

results;• ’Year’: the year the paper was published;• ’Ref.’: reference to the paper;The spatial distribution of RAMIS palpation research is

overviewed on a map (Fig. 2.), alongside the current da VinciResearch Kit (DVRK) locations [5].

III. HAPTIC FEEDBACK

In the human body, at least six types of receptors are reliablefor haptic sensation. Basically, all of these are measuring forceinduced deformations, and can be divided into two groups;tactile and kinaesthetic sensors. Tactile receptors are sensibleto higher frequencies, and located in the skin, with varyingdensity all over the body, e.g., the skin on the fingers is quiterich in those. In contrast, kinaesthetic receptors are locatedmostly in muscles, joints and tendons, and are sensible in alower dynamic range [6].

Haptic feedback could be useful in a number of mannersduring RAMIS interventions. However, neither the da Vinci,neither the vast majority of other commercialized RAMISsystems possesses this function. Surgeons might benefit from

SACI 2019 • IEEE 13th International Symposium on Applied Computational Intelligence and Informatics • May 29-31 • Timişoara, Romania

978-1-7281-0686-1/19/$31.00 ©2019 IEEE 000099

Fig. 1. Flow chart of the selection process according to thePreferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement.

Fig. 2. Geographical distribution of institutes working onrobotic palpation worldwide (marked with red pins), comparedto the DVRK locations (blue pins).

feeling the tension of the thread during suturing, the grippingforce during tissue manipulation, and also collecting infor-mation via palpation [7]. The following sections present thedifferent aspects of haptic feedback in RAMIS.

A. Clinical aspects

In general, safety is a critical aspect of RAMIS, and hapticfeedback has the potential to offer improvements in thisfield. Patient safety could be enhanced e.g., by preventing thesurgeon to enter prohibited regions and thus damage sensitivetissues, like blood vessels and nerves, or simply limiting themaximum amount of force the surgeon can apply. Beyondsafety, haptics could support the decision making as well,by letting the surgeon know or feel the amount of externalor grip force applied. Furthermore, providing palpation and

tactile information to the surgeon would also offer diagnosticalbenefits [8]–[10]. In RAMIS, surgical training is exceptionallyimportant, and haptic feedback has even more advantages fornovices [11]. Utilizing haptic feedback during the training(e.g., manipulating in a simulated environment) the traineescan link the applied forces to vision, thus later, working on asystem without haptic feedback, they will be able to estimatethose forces based on vision solely.

On the patient side, the most basic requirements towardssensorized instruments providing haptic feedback are steriliz-ability and bio-compatibility. The cleaning of the instrumentsis commonly performed by autoclave, and most of the avail-able force sensors would not endure such high temperatures.Important to note that the size of the instruments and thetrocars is limited as well. Furthermore, due to the clinicalusage, the reliability, accuracy and the cost of such instrumentsare important parameters as well.

B. Technical aspects

Forces and tactile information can be presented to the sur-geon in a number of ways. Probably the most straightforwardis to display the used forces on the endoscopic camera screenwith different colours; tissue properties or tactile informationcan also be visualized using augmented reality overlay, or ina separate diagram. Haptic information can be presented tothe surgeon through the master controllers by force feedbackcontrol method as well [7], [12]–[20]. Moreover, there are anumber of methods to display fine tactile information, e.g.,air flow, vibration or deformation of the skin [14], [21]–[25].Displaying the force applied by the jaws of the grippers hasalso a number of advantages, fastening tissue manipulation,using the minimal force required [26]–[30].

Evidently, all of the different haptic displays require dif-ferent sensors at the patient side instruments. There is awide palette of usable force sensing techniques is RAMIS,those are presented in details below, in the palpation section.Moreover, advanced tactile sensorization is currently underintensive research, e.g. using force sensor arrays, based onpressure measurement or optical phenomena [15], [16], [23],[31]–[33].

IV. PALPATION

Palpation is a medical technique, where the physician ex-amines tissues by the sense of touch, to gain informationon the underlying structures. This method is widely usedin interventional medicine for diagnostic purposes. In thetraditional, open surgical practice, palpation is commonly usedfor the identification of tissues (e.g., to find damage) findcertain organs (e.g., blood vessels) or lesions (e.g., tumors). Itis probably the most promising aspect of soft tissue contactidentification to be implemented in RAMIS, since technicallyit is relatively easy to achieve, yet highly beneficial for thesurgeon. However, the addition of force sensing capability toRAMIS instruments is problematic due to the limited spaceand sterilization requirements.

T. D. Nagy and T. Haidegger • Recent Advances in Robot-Assisted Surgery: Soft Tissue Contact Identification

000100

Force sensing Form of# method Platform Key aspect of the study evaluation Year Ref.1 direct other Tumor identification using kinaesthetic feedback ex vivo 2008 [34]2 direct da Vinci Tissue property estimation and visual overlay in RAMIS ex vivo 2009 [35]3 direct other Prostate stiffness mapping using tactile sensor ex vivo 2011 [36]4 direct other Utrasound elastography in prostatectomy ex-vivo 2012 [37]5 direct other MRI-compatible piezoelectric palpation sensor in vitro 2012 [32]6 direct Phantom Premium Haptic feedback and augmented reality in RAMIS in vitro 2012 [38]7 direct Phantom Premium Identification of hard inclusions using machine learning in vitro 2013 [39]8 direct other Endoscopic stiffness probe for soft tissue identification in vitro 2014 [40]9 direct DVRK Deflection-based single-use palpation probe in vitro 2015 [41]10 direct da Vinci Force sensing on the back of EndoWrist instruments in vitro 2015 [42]11 direct DVRK Autonomous tumor loc. with Gaussian Process adaptive sampl. in vitro 2016 [43]12 direct DVRK Anutonomous tumor palpation and resection in vitro 2016 [44]13 direct other Soft robotic skin for autonomous palpation in vitro 2017 [25]14 direct DVRK Force-controlled exploration to update geom. f deformed env. in vitro 2017 [45]15 direct other Hard inclusion identification using rolling indentation probe in vitro 2017 [46]16 direct other Soft robotic probe with stiffness control for palpation in vitro 2017 [47]17 direct Phantom Omni Prostate cancer localization using rolling indentation probe ex vivo 2017 [48]18 direct da Vinci Force sensor integrated into da Vinci instrument shaft in vitro 2017 [49]19 direct other Inexpensive triaxial force sensor for MIS in vitro 2017 [50]20 direct da Vinci Forceps with triaxial force sensing abilities sensored validation 2018 [51]21 direct DVRK Autonomous search and augmented reality overlay for tumors in vitro 2018 [52]22 indirect other Wheeled palpation probe using FBG sensory ex-vivo 2008 [53]23 indirect other Tactile sensing system based on an expandable balloon ex vivo 2010 [31]24 indirect other Abnormality identification using rolling indentation probe in vitro 2011 [54]25 indirect da Vinci FBG force sensor system for RAMIS sensored validation 2011 [55]26 indirect other Force sensing micro-forceps using FBG sensory in-vivo 2012 [56]27 indirect other Wireless palpation probe in vivo 2014 [57]28 indirect other Softness measurement using acoustic sensory in vitro 2015 [58]29 indirect da Vinci Force sensing from the EndoWrist back end in vitro 2015 [42]30 indirect DVRK Sensorizing da Vinci instrument shaft using FBG sensors in vitro 2016 [59]31 indirect DVRK External force estimation from motor currents on DVRK sensored validation 2017 [60]32 vision other Haptic feedback based on visual cues in silico 2015 [61]33 vision other Vision based force estimation ex vivo 2017 [62]34 vision other Video-tactile pneumatic probe to estimate tissue stiffness in vitro 2017 [33]35 vision da Vinci Force prediction with visual inf. using machine learning ex vivo 2018 [63]

TABLE I. The overview of RAMIS palpation literature. The platforms mentioned: da Vinci (da Vinci Surgical System, IntuitiveSurgical Inc., Sunnyvale, CA), DVRK (da Vinci Research Kit [5]), Phantom Premium and Phantom Omni (3D Systems, RockHill, South Carolina).

The stiffness mapping of a soft tissue area (Fig. 3) canbe organized into two basic groups alongside the utilizedsampling methods; discrete and continuous. During discretemapping, certain points of the tissue are compressed witha palpation probe. This can be performed in a rectangulargrid, but methods using adaptive sampling density, based onthe local stiffness are also exist [43]. The tissue can also bemapped continuously, without lifting the probe; this requireslow friction on the surface or a wheeled palpation probe, andadvanced control methods [48], [54].

The pressure required for the palpation of soft tissuesvaries; it highly depends on the examined tissue, and on thesize and shape of the probe tip. Important to note that theabsolute pressure—and so the stiffness—values are usuallyless important, rather the stiffness changes in the local areaare significant. Due to this fact, the absolute pressure valuesare often not calculated, but the stiffness is expressed e.g.,simply in the form of applied force.

Similarly, indentation depth is also highly dependent of theexamined tissue and evaluation method. In general, given thedimension of a RAMIS palpation probe tip, the applied force is

Fig. 3. The stiffness pseudo-colour map (left) and contour map(right) of a silicone phantom with hard inclusions (units: mm).The 9 inclusions can be clearly identified on the stiffness mapgained by palpation [54].

usually somewhere between 0.1–5 N, and the indentation depthis about 2–8 mm. It is true in general that lower forces areused to examine tissue properties, while higher forces enablestructure identification [19], [39], [41], [54].

One of the key aspects of palpation is that how the ap-plied force is measured. In the following sections, different


000101

implementations on RAMIS palpation are presented, organizedalongside the type of utilized sensor modality.

A. Palpation based on direct force sensing

The most straightforward way to measure the applied forceis based on elastic deformation, so called direct force sensing.Common force sensors, like strain gauges or spring-basedapproaches utilize this principle. The biggest advantage ofdirect force sensing is the easy implementation and low cost,that enables the development of disposable probes as well.However, the placement of those sensors is problematic, dueto the space limitations and requirements of sterilization andcleaning. Some of the force sensors are placed directly onthe tooltip of the instrument, along the shaft or even into thedriving chain [25], [32], [34]–[52], [59].

One of the first results in RAMIS palpation was presentedby Yamamoto et al. [35]. In this study a force sensor was placeunder the silicone phantom, that was palpated by the da Vinciin teleoperation to identify tissue properties. Another exampleof the direct approach is presented by a group at UC Berkeley[41]. In this work an inexpensive palpation probe is proposedfor the da Vinci. The design is based on the displacementmeasurement of the spring-attached probe tip (Fig. 4). It wasshown that this probe is capable of localizing subcutaneousvessel-like structures of a silicone phantom with high accuracy.

B. Palpation based on indirect force sensing

Indirect force sensing techniques are also applicable forpalpation. The methods do not require electrical componentsto be inserted into the patient’s body, since the applied force isestimated using external sensory. We can find examples in theliterature based on acoustic reflection, pressure of the probe’smedium or even the motor currents of the robot arms [31],[42], [57], [58], [60]. The drawback of this technique incontrast to the direct force sensing is its complexity and signalprocessing and computation requirements.

Fig. 4. A disposable palpation probe, mountable on a daVinci instrument. This spring-based palpation probe was usedfor locating subcutaneuos blood vessel in phantom environ-ment [41].

In the work of [58], we can find a solution for indirect forcesensing. The developed sensor responds to contact positionand force at deformable cavity in the sensor probe by usingacoustic reflection (Fig. 5).

For RAMIS palpation, probably the most promising forcesensors are the Fiber Bragg Grating (FBG) force sensors.FBG sensors are based on the strain-induced shift of Braggwavelength; a particular wavelength of light that is reflectedby the optical fiber, while all other wavelengths are transmit-ted [64]. These optical fiber sensors possess small physicalsize and are also tolerate the high temperature during au-toclave sterilization, furthermore immune to electromagneticinterference. The mentioned features make FBG force sensorsan optimal candidate for RAMIS palpation probes, as alreadyintegrated into RAMIS instruments by various research groups(Fig. 6) [53]–[56], [59].

C. Palpation with vision-based force estimation

The applied force during RAMIS can also be estimatedbased on the endoscopic camera images. The major benefit ofthis technique is that no additional device needs to be addedto the operating room setup. Furthermore, bio-compatibilityand sterilization are solved already in the case of RAMISendoscopes. This technique involves several different types ofimage processing methods, such as feature extraction, filteringof light reflections, and also can be approached using neuralnetworks, but the crucial parts are usually the reconstructionof the tissue surface and handling the inhomogenity in thetissue. Based on the detected deformations of the targetedtissue surface, the applied force values can be estimated [33],[61]–[63], [65]. Despite the mentioned benefits of this tech-nique, the implementation is extremely complex, and usuallycomputationally intense.

One of the most robust solutions for vision-based forcesensing was proposed by Aviles et al. [62]. The first step is torecover a 3D deformable structure by extracting the geometryof motion of the hearts surface (Fig. 7). Then, a deep neuralnetwork, derives the relationship between the visual-geometricinformation and the applied force. This solution provided highaccuracy results, with an average root-mean square error of0.02 N.

Fig. 5. Acoustic reflection based indirect force sensing. Theblue curve represents input wave from speaker, red curve thetotal wave measured by microphone. The phase delay of themeasured wave relates to the magnitude and position of thedeformation [58].


000102

Fig. 6. FBG displacement sensor for RAMIS, developed byLiu et al. [54].

Fig. 7. a) Based on the deformation of the reconstructedsurface the applied force can be estimated using deep learningtechnique. b) The deformation of the occluded regions arerecovered from the palpation of other locations [62].

V. CONCLUSION AND DISCUSSION

In this study, the current research directions in RAMISpalpation were presented. As of today, there are no com-mercially available solutions, nevertheless, the last few yearsshowed a significant activity in the field. 35 relevant paperswere identified and organized alongside the used sensorymethods: 21 utilized direct measurement of the force-induceddeformation, 10 measured the contact forces indirectly, viasome medium or a driving chain, and in 4 of the processedworks the applied force was estimated by enhanced imageprocessing algorithms.

It is generally true that the direct force sensing methods areeasier to implement, but are more problematic in terms of re-quired physical space or cleaning abilities. In contrast, indirectand vision-based methods do not rely on electric componentsinserted into the patient’s body, however, generally are morecomplex. Amongst the mentioned technologies, FBG forcesensory promises the biggest chance of breakthrough in this

field; FBG-based probes can be small enough physically, moreresistant to the high temperature requirements of sterilization,and are also less sensitive to electromagnetic disturbancescompared to other force sensors.

ACKNOWLEDGMENT

This work was partially supported by ACMIT (AustrianCenter for Medical Innovation and Technology), which isfunded within the scope of the COMET (Competence Centersfor Excellent Technologies) program of the Austrian Govern-ment. T. D. Nagy and T. Haidegger are supported through theNew National Excellence Program of the Ministry of HumanCapacities. T. Haidegger is a Bolyai Fellow of the HungarianAcademy of Sciences.

REFERENCES

[1] A. Takacs, D. A. Nagy, I. J. Rudas, and T. Haidegger, “Origins of Sur-gical Robotics: From Space to the Operating Room,” Acta PolytechnicaHungarica, vol. 13, no. 1, pp. 13–30, 2016.

[2] L. Marton, Z. Szanto, T. Haidegger, P. Galambos, and J. Kovecses,“Internet - based Bilateral Teleoperation Using a Revised Time - DomainPassivity Controller,” Acta Polytechnica Hungarica, pp. 27–45, 2017.

[3] M. Ottermo, O. Stavdahl, and T. Johansen, “Palpation instrument foraugmented minimally invasive surgery,” in Proc. of the 2004 IEEE/RSJInternational Conference on Intelligent Robots and Systems (IROS),vol. 4. Sendai, Japan: IEEE, 2004, pp. 3960–3964.

[4] D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, and T. P. Group,“Preferred Reporting Items for Systematic Reviews and Meta-Analyses:The PRISMA Statement,” PLOS Medicine, vol. 6, no. 7, Jul. 2009.

[5] P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, andS. P. DiMaio, “An open-source research kit for the da Vinci R© SurgicalSystem,” in Proc. of the IEEE International Conference on Robotics andAutomation, Hong Kong, 2014, pp. 6434–6439.

[6] T. A. Kern, “Biological Basics of Haptic Perception,” in EngineeringHaptic Devices: A Beginner’s Guide for Engineers. Berlin, Heidelberg:Springer Berlin Heidelberg, 2009, pp. 35–58.

[7] A. Takacs, L. Kovacs, I. Rudas, R.-E. Precup, and T. Haidegger, “Modelsfor force control in telesurgical robot systems,” Acta PolytechnicaHungarica, vol. 12, pp. 95–114, 2015.

[8] M. Tavakoli, A. Aziminejad, R. V. Patel, and M. Moallem, “Multi-sensory force/deformation cues for stiffness characterization in soft-tissue palpation,” in Proc. of the Annual International Conference of theIEEE Engineering in Medicine and Biology Society. IEEE Engineeringin Medicine and Biology Society. Annual Conference, vol. 1, 2006, pp.837–840.

[9] M. Moradi Dalvand, B. Shirinzadeh, S. Nahavandi, and J. Smith, “Ef-fects of realistic force feedback in a robotic assisted minimally invasivesurgery system,” Minimally invasive therapy & allied technologies:MITAT: official journal of the Society for Minimally Invasive Therapy,vol. 23, no. 3, pp. 127–135, Jun. 2014.

[10] S. Laufer, C. M. Pugh, and B. D. Van Veen, “Modeling Touch and Pal-pation Using Autoregressive Models,” IEEE Transactions on BiomedicalEngineering, vol. 65, no. 7, pp. 1585–1594, Jul. 2018.

[11] O. A. J. van der Meijden and M. P. Schijven, “The value of hapticfeedback in conventional and robot-assisted minimal invasive surgeryand virtual reality training: A current review,” Surgical Endoscopy,vol. 23, no. 6, pp. 1180–1190, Jun. 2009.

[12] K. Salisbury and A. Bejczy, “Kinesthetic coupling between operator andremote manipulator,” NASA, Technical Report, Jan. 1980.

[13] Y. Kuroda, M. Nakao, T. Kuroda, H. Oyama, and M. Komori, “Interac-tion model between elastic objects for haptic feedback considering col-lisions of soft tissue,” Computer Methods and Programs in Biomedicine,vol. 80, no. 3, pp. 216–224, Dec. 2005.

[14] Z. F. Quek, W. Provancher, and A. Okamura, “Evaluation of SkinDeformation Tactile Feedback for Teleoperated Surgical Tasks,” IEEETransactions on Haptics, 2018.

[15] M. I. Tiwana, S. J. Redmond, and N. H. Lovell, “A review of tactilesensing technologies with applications in biomedical engineering,” Sen-sors and Actuators, vol. 179, pp. 17–31, Jun. 2012.


000103

[16] P. S. Girao, P. M. P. Ramos, O. Postolache, and J. Miguel Dias Pereira,“Tactile sensors for robotic applications,” Measurement, vol. 46, no. 3,pp. 1257–1271, Apr. 2013.

[17] J. Konstantinova, A. Jiang, K. Althoefer, P. Dasgupta, andT. Nanayakkara, “Implementation of Tactile Sensing for Palpationin Robot-Assisted Minimally Invasive Surgery: A Review,” IEEESensors Journal, vol. 14, no. 8, pp. 2490–2501, Aug. 2014.

[18] M. Moradi Dalvand, B. Shirinzadeh, A. H. Shamdani, J. Smith, andY. Zhong, “An actuated force feedback-enabled laparoscopic instru-ment for robotic-assisted surgery,” The international journal of medicalrobotics and computer assisted surgery: MRCAS, vol. 10, no. 1, pp.11–21, Mar. 2014.

[19] A. Takacs, I. J. Rudas, and T. Haidegger, “Surface deformation andreaction force estimation of liver tissue based on a novel nonlinearmass–spring–damper viscoelastic model,” Medical & Biological Engi-neering & Computing, vol. 54, no. 10, pp. 1553–1562, Oct. 2016.

[20] A. Torabi, M. Khadem, K. Zareinia, G. Sutherland, and M. Tavakoli,“Application of a Redundant Haptic Interface in Enhancing Soft-TissueStiffness Discrimination,” IEEE Robotics and Automation Letters, 2019.

[21] P. Puangmali, K. Althoefer, L. D. Seneviratne, D. Murphy, and P. Das-gupta, “State-of-the-Art in Force and Tactile Sensing for MinimallyInvasive Surgery,” IEEE Sensors Journal, vol. 8, no. 4, pp. 371–381,Apr. 2008.

[22] M. Bianchi, J. C. Gwilliam, A. Degirmenci, and A. M. Okamura,“Characterization of an air jet haptic lump display,” in Proc. of theAnnual International Conference of the IEEE Engineering in Medicineand Biology Society, vol. 2011, 2011, pp. 3467–3470.

[23] M. Li, S. Luo, and G. Xu, “A tactile sensing and feedback system fortumor localization,” in Proc. of the 13th International Conference onUbiquitous Robots and Ambient Intelligence (URAI), Xian, China, Aug.2016, pp. 259–262.

[24] C. Pacchierotti, D. Prattichizzo, and K. J. Kuchenbecker, “CutaneousFeedback of Fingertip Deformation and Vibration for Palpation inRobotic Surgery,” IEEE Transactions on Biomedical Engineering,vol. 63, no. 2, pp. 278–287, Feb. 2016.

[25] F. Campisano, S. Ozel, A. Ramakrishnan, A. Dwivedi, N. Gkotsis, C. D.Onal, and P. Valdastri, “Towards a soft robotic skin for autonomoustissue palpation,” in Proc. of the 2017 IEEE International Conferenceon Robotics and Automation (ICRA), May 2017, pp. 6150–6155.

[26] N. T. Burkhard, M. R. Cutkosky, and J. R. Steger, “Slip Sensingfor Intelligent, Improved Grasping and Retraction in Robot-AssistedSurgery,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 4148–4155, Oct. 2018.

[27] U. Kim, D.-Y. Seok, Y. B. Kim, D.-H. Lee, and H. R. Choi, “Devel-opment of a grasping force-feedback user interface for surgical robotsystem,” in Proc. of the 2016 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS). Daejeon, South Korea: IEEE,Oct. 2016, pp. 845–850.

[28] U. Kim, Y. B. Kim, J. So, D.-Y. Seok, and H. R. Choi, “SensorizedSurgical Forceps for Robotic-Assisted Minimally Invasive Surgery,”IEEE Transactions on Industrial Electronics, vol. 65, no. 12, pp. 9604–9613, Dec. 2018.

[29] C. Lee, Y. H. Park, C. Yoon, S. Noh, C. Lee, Y. Kim, H. C. Kim, H. H.Kim, and S. Kim, “A grip force model for the da Vinci end-effectorto predict a compensation force,” Medical & Biological Engineering &Computing, vol. 53, no. 3, pp. 253–261, Mar. 2015.

[30] J. J. O’Neill, T. K. Stephens, and T. M. Kowalewski, “Evaluationof Torque Measurement Surrogates as Applied to Grip Torque andJaw Angle Estimation of Robotic Surgical Tools,” IEEE Robotics andAutomation Letters, vol. 3, no. 4, pp. 3027–3034, Oct. 2018.

[31] Y. Tanaka, Q. Yu, K. Doumoto, A. Sano, Y. Hayashi, M. Fujii, Y. Kajita,M. Mizuno, T. Wakabayashi, and H. Fujimoto, “Development of a real-time tactile sensing system for brain tumor diagnosis,” InternationalJournal of Computer Assisted Radiology and Surgery, vol. 5, no. 4, pp.359–367, Jul. 2010.

[32] A. Hamed, K. Masamune, Z. T. H. Tse, M. Lamperth, and T. Dohi,“Magnetic resonance imaging-compatible tactile sensing device basedon a piezoelectric array,” Proceedings of the Institution of MechanicalEngineers. Part H, Journal of Engineering in Medicine, vol. 226, no. 7,pp. 565–575, Jul. 2012.

[33] M. M. Gubenko, A. V. Morozov, A. N. Lyubicheva, I. G. Goryacheva,M. Z. Dosaev, M.-S. Ju, C.-H. Yeh, and F.-C. Su, “Video-tactilepneumatic sensor for soft tissue elastic modulus estimation,” BiomedicalEngineering Online, vol. 16, no. 1, Aug. 2017.

[34] G. L. McCreery, A. L. Trejos, M. D. Naish, R. V. Patel, and R. A.Malthaner, “Feasibility of locating tumours in lung via kinaestheticfeedback,” The international journal of medical robotics + computerassisted surgery: MRCAS, vol. 4, no. 1, pp. 58–68, Mar. 2008.

[35] T. Yamamoto, B. Vagvolgyi, K. Balaji, L. L. Whitcomb, and A. M.Okamura, “Tissue property estimation and graphical display for teleop-erated robot-assisted surgery,” in Proc. of the 2009 IEEE InternationalConference on Robotics and Automation (ICRA), Kobe, May 2009, pp.4239–4245.

[36] Q. Peng, S. Omata, D. M. Peehl, and C. E. Constantinou, “Stiffnessmapping prostate biopsy samples using a tactile sensor,” in Proc. of theAnnual International Conference of the IEEE Engineering in Medicineand Biology Society. IEEE Engineering in Medicine and Biology Society.Annual Conference, vol. 2011, 2011, pp. 8515–8518.

[37] I. N. Fleming, C. Kut, K. J. Macura, L.-M. Su, H. Rivaz, C. M.Schneider, U. Hamper, T. Lotan, R. Taylor, G. Hager, and E. Boctor,“Ultrasound elastography as a tool for imaging guidance during prosta-tectomy: Initial experience,” Medical Science Monitor: InternationalMedical Journal of Experimental and Clinical Research, vol. 18, no. 11,pp. 635–642, Nov. 2012.

[38] T. Yamamoto, N. Abolhassani, S. Jung, A. M. Okamura, and T. N.Judkins, “Augmented reality and haptic interfaces for robot-assistedsurgery: Augmented reality and haptic interfaces for robot-assistedsurgery,” The International Journal of Medical Robotics and ComputerAssisted Surgery, vol. 8, no. 1, pp. 45–56, Mar. 2012.

[39] K. A. Nichols and A. M. Okamura, “Autonomous robotic palpation:Machine learning techniques to identify hard inclusions in soft tissues,”in Proc. of the 2013 IEEE International Conference on Robotics andAutomation, Karlsruhe, 2013, pp. 4384–4389.

[40] A. Faragasso, A. Stilli, J. Bimbo, Y. Noh, H. Liu, T. Nanayakkara,P. Dasgupta, H. A. Wurdemann, and K. Althoefer, “Endoscopic add-on stiffness probe for real-time soft surface characterisation in MIS,”Conference proceedings: ... Annual International Conference of theIEEE Engineering in Medicine and Biology Society. IEEE Engineeringin Medicine and Biology Society. Annual Conference, vol. 2014, pp.6517–6520, 2014.

[41] S. McKinley, A. Garg, S. Sen, R. Kapadia, A. Murali, K. Nichols,S. Lim, S. Patil, P. Abbeel, A. M. Okamura, and K. Goldberg, “A single-use haptic palpation probe for locating subcutaneous blood vessels inrobot-assisted minimally invasive surgery,” Aug. 2015, pp. 1151–1158.

[42] T. K. Stephens, Z. C. Meier, R. M. Sweet, and T. M. Kowalewski,“Tissue Identification Through Back End Sensing on da Vinci EndoWristSurgical Tool 1,” Journal of Medical Devices, vol. 9, no. 3, Sep. 2015.

[43] A. Garg, S. Sen, R. Kapadia, Y. Jen, S. McKinley, L. Miller, andK. Goldberg, “Tumor localization using automated palpation with Gaus-sian Process Adaptive Sampling,” in Proc. of the 2016 IEEE Interna-tional Conference on Automation Science and Engineering (CASE), FortWorth, 2016, pp. 194–200.

[44] S. McKinley, A. Garg, S. Sen, D. V. Gealy, J. McKinley, Y. Jen, andK. Goldberg, “Autonomous Multilateral Surgical Tumor Resection withInterchangeable Instrument Mounts and Fluid Injection Device,” 2016.

[45] L. Wang, Z. Chen, P. Chalasani, R. M. Yasin, P. Kazanzides, R. H.Taylor, and N. Simaan, “Force-Controlled Exploration for Updating Vir-tual Fixture Geometry in Model-Mediated Telemanipulation,” Journal ofMechanisms and Robotics, vol. 9, no. 2, 2017.

[46] M. Li, J. Konstantinova, G. Xu, B. He, V. Aminzadeh, J. Xie, H. Wur-demann, and K. Althoefer, “Evaluation of stiffness feedback for hardnodule identification on a phantom silicone model,” PLOS ONE, vol. 12,no. 3, Mar. 2017.

[47] N. Sornkarn and T. Nanayakkara, “Can a Soft Robotic Probe UseStiffness Control Like a Human Finger to Improve Efficacy of HapticPerception?” IEEE Transactions on Haptics, vol. 10, no. 2, pp. 183–195,Apr-Jun 2017.

[48] J. Li, H. Liu, M. Brown, P. Kumar, B. J. Challacombe, A. Chandra,G. Rottenberg, L. D. Seneviratne, K. Althoefer, and P. Dasgupta, “Exvivo study of prostate cancer localization using rolling mechanicalimaging towards minimally invasive surgery,” Medical Engineering &Physics, vol. 43, pp. 112–117, May 2017.

[49] G. A. Fontanelli, L. R. Buonocore, F. Ficuciello, L. Villani, and B. Si-ciliano, “A novel force sensing integrated into the trocar for minimallyinvasive robotic surgery,” in Proc. of the 2017 IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS), Sep. 2017, pp.131–136.


000104

[50] L. Li, B. Yu, C. Yang, P. Vagdargi, R. A. Srivatsan, and H. Choset,“Development of an inexpensive tri-axial force sensor for minimally in-vasive surgery,” in Proc. of the 2017 IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS), Vancouver, BC, Sep. 2017,pp. 906–913.

[51] L. Yu, Y. Yan, X. Yu, and Y. Xia, “Design and Realization of ForcepsWith 3-D Force Sensing Capability for Robot-Assisted Surgical System,”IEEE Sensors Journal, vol. 18, no. 21, pp. 8924–8932, Nov. 2018.

[52] N. Zevallos, A. Srivatsan Rangaprasad, H. Salman, L. Li, J. Qian,S. Saxena, M. Xu, K. Patath, and H. Choset, “A Real-time AugmentedReality Surgical System for Overlaying Stiffness Information,” in Proc.of the Robotics: Science and Systems, Jun. 2018.

[53] P. Puangmali, H. Liu, K. Althoefer, and L. D. Seneviratne, “Optical fibersensor for soft tissue investigation during minimally invasive surgery,”in Proc. of the 2008 IEEE International Conference on Robotics andAutomation. Pasadena, CA, USA: IEEE, May 2008, pp. 2934–2939.

[54] H. Liu, P. Puangmali, D. Zbyszewski, O. Elhage, P. Dasgupta, J. S. Dai,L. Seneviratne, and K. Althoefer, “An indentation depth-force sensingwheeled probe for abnormality identification during minimally invasivesurgery,” Proceedings of the Institution of Mechanical Engineers. PartH, Journal of Engineering in Medicine, vol. 224, no. 6, pp. 751–763,2010.

[55] H. Song, H. Kim, J. Jeong, and J. Lee, “Development of FBG sensorsystem for force-feedback in minimally invasive robotic surgery,” inProc. of the 2011 Fifth International Conference on Sensing Technology,Nov. 2011, pp. 16–20.

[56] X. He, M. A. Balicki, J. U. Kang, P. L. Gehlbach, J. T. Handa, R. H.Taylor, and I. I. Iordachita, “Force sensing micro-forceps with integratedfiber Bragg grating for vitreoretinal surgery,” in Proc. of the OpticalFibers and Sensors for Medical Diagnostics and Treatment ApplicationsXII, vol. 8218. International Society for Optics and Photonics, Jan.2012.

[57] M. Beccani, C. Di Natali, L. J. Sliker, J. A. Schoen, M. E. Rentschler,and P. Valdastri, “Wireless tissue palpation for intraoperative detectionof lumps in the soft tissue,” IEEE Transactions on Bio-medical Engi-neering, vol. 61, no. 2, pp. 353–361, Feb. 2014.

[58] T. Fukuda, Y. Tanaka, M. Fujiwara, and A. Sano, “Softness measurementby forceps-type tactile sensor using acoustic reflection,” in Proc. ofthe 2015 IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS), Hamburg, Germany, Sep. 2015, pp. 3791–3796.

[59] K. S. Shahzada, A. Yurkewich, R. Xu, and R. V. Patel, “Sensorizationof a surgical robotic instrument for force sensing,” in Proc. of the SPIEBiOS, I. Gannot, Ed., San Francisco, California, United States, Mar.2016.

[60] H. Sang, J. Yun, R. Monfaredi, E. Wilson, H. Fooladi, and K. Cleary,“External force estimation and implementation in robotically assistedminimally invasive surgery,” The International Journal of MedicalRobotics and Computer Assisted Surgery, vol. 13, no. 2, Jun. 2017.

[61] M. Li, J. Konstantinova, E. L. Secco, A. Jiang, H. Liu, T. Nanayakkara,L. D. Seneviratne, P. Dasgupta, K. Althoefer, and H. A. Wurdemann,“Using visual cues to enhance haptic feedback for palpation on virtualmodel of soft tissue,” Medical & Biological Engineering & Computing,vol. 53, no. 11, pp. 1177–1186, Nov. 2015.

[62] A. I. Aviles, S. M. Alsaleh, J. K. Hahn, and A. Casals, “Towards Retriev-ing Force Feedback in Robotic-Assisted Surgery: A Supervised Neuro-Recurrent-Vision Approach,” IEEE Transactions on Haptics, vol. 10,no. 3, pp. 431–443, Jul. 2017.

[63] C. Gao, X. Liu, M. Peven, M. Unberath, and A. Reiter, “Learning toSee Forces: Surgical Force Prediction with RGB-Point Cloud TemporalConvolutional Networks,” in Proc. of the ISIC 2018: OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clin-ical Image-Based Procedures, and Skin Image Analysis, Jul. 2018, pp.118–127.

[64] K. Hill and G. Meltz, “Fiber Bragg grating technology fundamentalsand overview,” Journal of Lightwave Technology, vol. 15, no. 8, pp.1263–1276, Aug./1997.

[65] A. I. Karoly, R. Fuller, and P. Galambos, “Unsupervised Clustering forDeep Learning: A tutorial survey,” Acta Polytechnica Hungarica, vol. 15,no. 8, pp. 29–53, 2018.


000105


000106

recent advances in robot-assisted surgery: soft …real.mtak.hu/101974/1/5 17_saci2019.pdfrecent...

Documents