Unsupervised Learning of Surgical Smoke Removal from Simulation

L. Chen1, W. Tang1, and N. W. John 21Department of Creative Technology, Bournemouth University

2School of Computer Science, University of Chester{chenl, wtang}@bournemouth.ac.uk, nigel.john@chester.ac.uk

INTRODUCTION

The surgical smoke produced during minimally inva-sive surgery can not only reduce the visibility of thesurgeons, but also severally affect the performance ofimage processing algorithms used for image guidedsurgery such as image tracking, segmentation, de-tection and retrieval. Besides from physical smokeevacuation devices, many research works [3] [1] [4] [6][7] address this issue by using vision-based methodsto filter out the smoke and try to recover the clearimages. More recently, end-to-end deep learning ap-proaches [2] have been introduced to solve the de-hazing and de-smoking problems. However, it is ex-tremely difficult to collect large amounts of data forthe effective learning of the implicit de-smoking func-tion, especially for surgical scenes. In this paper, wepropose a computational framework for unsupervisedlearning of smoke removal from rendering smoke onlaparoscopic video. Compared to conventional im-age processing approaches, our proposed frameworkis able to remove local smoke and recover more realis-tic tissue colour but will not affect the areas withoutsmoke. Although trained on synthetic images, theexperimental results show that our network is ableto effectively remove smoke on laparoscopic imageswith real surgical smoke.

MATERIALS AND METHODS

Render Smoke

CNN

Loss

Fig. 1: Overview of our framework for unsupervisedlearning of smoke removal

With the recent development of Convolutional Neu-ral Networks (CNNs), machines can solve many com-puter vision problems when provided with large-scaleground-truth data for supervised learning. However,the acquisition of ground truth is an expensive, time-consuming and labour-intensive task. Especially forsurgical scenes, the amount of data and the accu-racy of data must be satisfied to ensure acceptableresults for clinical use. To address this issue, we pro-posed an unsupervised framework for learning smokeremoval. As can be seen from Figure 1, our frame-

work is composed of a render engine for synthesizingsmoked images from laparoscopic videos and a CNNfor the end-to-end training of de-smoking.

RandomSmoke

Synthetic Image

Fig. 2: The random smoke is rendered on the back-ground of laparoscopic images to synthesize smoke im-age.

Smoke SynthesisWe use Blender 1 – an open source 3D creationsoftware for the synthesis of smoke image for train-ing. As shown in Figure 2, the real laparoscopicimages from Hamlyn Centre Laparoscopic / Endo-scopic Video Datasets 2 [8] are used as backgroundimages, and the smoke is rendered with randomlyintensity, density and location on the background tosimulate the real smoke. The variation of smoke ren-der ensures that our network will not over-fitting tocertain smoke intensity, density and location. Withthe help of powerful render engine, we are able tosynthesize unlimited amount of realistic images withthe presence of simulated surgical smoke. More im-portantly, this process is done automatically withoutany human intervention and cost.End-to-end Learning of Smoke RemovalAs smoke removal is a pixel-wise task, we adopt afully convolutional encoder-decoder network to gen-erate the same size de-smoked image with the inputimage. Since the smoke removal task needs to pre-serve most of the details from the input images, fol-lowing the U-Net structure [5], skip connections areimplemented for directly transferring high-level in-formation to the bottom of the network to preventthe loss of high quality details.

1https://www.blender.org/2http://hamlyn.doc.ic.ac.uk/vision/

Synthetic Result Real Result

Fig. 3: The result for our smoke removal framework.Left: Synthetic smoke images and results; Right: Realsurgical smoke images and results

The loss function for the network is the L1 error oforiginal image and de-smoked image from synthe-sized smoke image.

L =∑xy

|Ioriginal(x, y)− Idesmoked(x, y)| (1)

RESULTS

Our smoke removal network was implemented onTensorflow platform and trained on a workstationwith a single NVidia Titan X GPU. We used anin-vivo partial nephrectomy procedure dataset fromHamlyn Centre Laparoscopic / Endoscopic VideoDatasets [8] that contains 20,000 stereo image pairsfrom 11 video sequences, in which 7,553 images with-out surgical smoke are manually picked as ground-truth dataset. We randomly rendered smoke on these7,553 images to construct our training dataset. Theimages are scaled to 256x128 to accelerate trainingand inference. After training, our model only takes12ms to remove smoke for a 256x128 image.For testing, we run our trained network on laparo-scopic images with real surgical smoke, the resultsare shown in Figure 3. The qualitative visual inspec-tion of the results shows that our proposed frame-work can effectively remove real surgical smoke andrecover correct tissue colours. The code used for datasynthesis and training can be found at here 3.

DISCUSSION

In this work, we present a novel unsupervised deeplearning framework for automatic surgical smoke re-moval during minimally invasive surgery. Our frame-work only needs laparoscopic images as input, and3D render engine is used to randomly render smokeon these images to synthesize dataset for training.The preliminary results show that although our net-work is fully training on synthesize dataset, it hasthe ability to remove real surgical smoke during mini-mally invasive surgery at real-time speed. This solves

3https://github.com/melights/cnn desmoke

the problem that deep learning need large amountsof hand-labelled ground truth for training, especiallyfor medical datasets that profession knowledge isneeded for labelling data. In future work, it is in-teresting to see whether training networks from syn-thetic dataset can be extended to many other taskssuch as laparoscope tracking, surgical instrumentsdetection and tissue/organ segmentation, which willovercome the shortage of medical ground-truth dataand greatly benefit the deep learning technology tobe used in surgical scenes.

REFERENCES

[1] A. Baid, A. Kotwal, R. Bhalodia, S. N. Merchant,and S. P. Awate. Joint desmoking, specularityremoval, and denoising of laparoscopy images viagraphical models and bayesian inference. In IEEEISBI, apr 2017.

[2] B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao. De-hazeNet: An end-to-end system for single imagehaze removal. IEEE Transactions on Image Pro-cessing, 25(11):5187–5198, nov 2016.

[3] A. Kotwal, R. Bhalodia, and S. P. Awate. Jointdesmoking and denoising of laparoscopy images.In IEEE ISBI, apr 2016.

[4] X. Luo, A. J. McLeod, S. E. Pautler, C. M.Schlachta, and T. M. Peters. Vision-based surgi-cal field defogging. IEEE Transactions on Medi-cal Imaging, 36(10):2021–2030, oct 2017.

[5] O. Ronneberger, P. Fischer, and T. Brox. U-net:Convolutional networks for biomedical image seg-mentation. In MICCAI, 2015.

[6] K. Tchaka, V. M. Pawar, and D. Stoyanov. Chro-maticity based smoke removal in endoscopic im-ages. In SPIE Medical Imaging, feb 2017.

[7] C. Wang, F. Alaya Cheikh, M. Kaaniche, andO. J. Elle. A Smoke Removal Method for La-paroscopic Images. ArXiv e-prints, Mar. 2018.

[8] M. Ye, E. Johns, A. Handa, L. Zhang, P. Pratt,and G.-Z. Yang. Self-supervised siamese learn-ing on stereo image pairs for depth estimation inrobotic surgery. In Hamlyn Symposium on Med-ical Robotics, 2017.