tobias huber, tom wunderling, markus paschold, hauke lang ...isg · an impact on information and...
TRANSCRIPT
TobiasHuber,TomWunderling,MarkusPaschold,
HaukeLang,WernerKneist,ChristianHansen
Highly-ImmersiveVirtualRealityLaparoscopySimulation:DevelopmentandFutureAspects Pre-print version TobiasHuber,MarkusPaschold,HaukeLang,WernerKneist
DepartmentofGeneral,VisceralandTransplantSurgery,
UniversityMedicineoftheJohannesGutenberg-UniversityMainz,Germany
TomWunderling,ChristianHansen
FacultyofComputerScience,
Otto-von-GuerickeUniversityMagdeburg,Germany
Thisisapre-printofanarticlepublishedintheInternationalJournalofComputerAssisted
RadiologyandSurgery.Thefinalauthenticatedversionisavailableonlineat:
https://doi.org/10.1007/s11548-017-1686-2
2
Abstract
PurposeVirtualReality(VR)applicationswithhead-mounteddisplays(HMDs)havehad
animpactoninformationandmultimediatechnologies.Thecurrentworkaimedto
describetheprocessofdevelopingahighly-immersiveVRsimulationforlaparoscopic
surgery.
MethodsWecombinedaVRlaparoscopysimulator(LapSim)andaVRHMDtocreatea
user-friendlyVRsimulationscenario.Continuousclinicalfeedbackwasanessentialaspect
ofthedevelopmentprocess.WecreatedanartificialVR(AVR)scenariobyintegratingthe
simulatorvideooutputwithVRgamecomponentsoffiguresandequipmentinan
operatingroom.Wealsocreatedahighly-immersiveVRsurrounding(IVR)byintegrating
thesimulatorvideooutputwitha360°videoofastandardlaparoscopyscenariointhe
department'soperatingroom.
ResultsClinicalfeedbackledtooptimizationofthevisualization,synchronization,and
resolutionofthevirtualoperatingrooms(inboththeIVRandtheAVR).Preliminarytesting
resultsrevealedthatindividualsexperiencedahighdegreeeofexhilarationandpresence,
withrareeventsofmotionsickness.Thetechnicalperformanceshowednosignificant
differencecomparedtothatachievedwiththestandardLapSim.
ConclusionOurresultsprovidedaproof-of-conceptforthetechnicalfeasibilityofan
customhighlyimmersiveVR-HMDsetup.Futuretechnicalresearchisneededtoimprove
thevisualization,immersion,andcapabilityofinteractingwithinthevirtualscenario.
Keywords:SurgicalTraining,VirtualReality,LaparoscopicSurgery,Human-Computer-
Interaction,Visualization
3
1Introduction
VirtualReality (VR) iscurrentlyused insurgical training to improvepsychomotor
skills that are important for laparoscopic surgery, suchashand-eye coordination, spatial
orientation,andmanipulationsinthepresenceofafulcrumeffect[1].Nonetheless,theuse
ofVRsimulatorsislimitedindailyclinicalroutine,duetoalackingrealismoftasks,abstract
graphicdesign,andtheawarenessofparticipantstobeinatrainingenvironment[2,3].The
varietyoftaskshasbeenenlargedsincethedevelopmentofVRlaparoscopysimulations,and
currently,abstracttrainingtasksandproceduraloperationsareavailableinVR.Inaddition,
graphic design of simulation tasks and virtual tissue interaction in VR laparoscopy
simulationhaveimproved.However,usersarecontinuouslyawarethattheyareinatraining
environmentandnotarealsurgicalsituation.Realisticsurroundingsthatincreasetheuser’s
senseofpresenceduringaVRsimulationareonlypossiblewhenperformingteamtraining
sessions, which require more time, infrastructure, and human resources [4]. Recent
advancementsininformationandmultimediatechnologyhavemadeitpossibletodevelop
VR applications in combination with head mounted displays (HMDs). These combined
applicationsarecurrentlyusedintheentertainmentindustry,inthemilitary,andinaviation
training. Medical applications include VR-HMD psychological interventions for
posttraumaticstressdisorders,phobias,cognitiverehabilitationandandpaintreatmentfor
burn victims [5-8].Additionally, VRwill enhance the validity of clinical, behavioural and
affective and social neurosciences due to more realistic test scenarios [9]. The rise of
commerciallyavailableVR-HMDsoverthelastyearhasledtothedevelopmentofavirtual
operating room environment to increase the attractiveness and the degree of presence
duringVRlaparoscopysimulations.Thepresentworkaimedtodescribethedevelopment
processofanimmersiveVRlaparoscopysimulationsetup.Thegoalwastocombineexisting
technologies to create a user-friendly simulation scenario with high immersion and
presence. Continuous clinical evaluations and feedback from laparoscopic surgeons
comprisedanessentialpartofthedevelopmentprocess.
4
2RelatedWork
ThecurrentuseofVRinmedicalfieldsismainlyduetothevisualpossibilitiesitoffers,
whichareveryhelpfulforprocessessuchaseducation,simulation,planning,navigation,and
even rehabilitation [10]. VR simulations in surgery are used to teach technical skills,
behavioralskills,andentireprocedurestotraineesandpracticingsurgeonsworldwide[11].
In VR laparoscopy trainers, users perform surgical tasks with standard laparoscopy
instruments [12]. Studies on VR laparoscopy simulations have concluded that the skill
acquisition is equivalent to that acquiredwith laparoscopy box trainer simulations, and
theseskillscanbetransferredtotheoperatingroom(OR)[13,14].Also,abriefpre-surgical
VRwarm-upcanimproveperformanceintheOR[15].
Technical and visual improvements have influenced VR simulators. Three-
dimensionaldisplayswithpolarizationor shutter technologieshavebeen integratedand
investigated with differing results [16,17]. On the other hand, recent developments in
surgery, such as robotic operations, have led to the development of VR robotic surgery
simulators[18].VisualizationforaVRroboticsurgerysimulationcanbeseenasavariantof
anHMDsimulation,performedontheroboticconsole;however,theORsurroundingshave
not yet been considered in this context. A recent study described a VR application that
combinedaVRlaparoscopysimulationandalow-immersionHMD,withonlya45°fieldof
view, to produce a virtual scenario, which consisted of a peg transfer task in a plain,
computer-generatedroom[19].
The latest generation HMDs, which began shipping in 2016, feature several
technological advancements realized in recent years. High pixel density displays from
smartphoneswereusedinearlyprototypestoreducethe‘screen-door’effect.Thesedisplays
operatewith a high refresh rate to reduce latency,which causesmotion sickness.Novel
methods and custom sensors were developed to improve positional tracking [20].
Asynchronousre-projectionwasintroducedtoreducelatencyevenfurther.Thistechnique
introducessmallchangesontoapreviouslyrenderedframe,accordingtothemostrecent
positional tracking data, which lowers the computational requirements of the graphics
processingunit(GPU)[21].Othersoftwaresolutionswererequiredtocorrectfordistortion
andchromaticaberrationsthatarisefromthelenses.Thesesolutionswereneededtomap
theHMDtoawiderfieldofviewandcreateamorecomfortablepointoffocus.
5
Fig. 1: Custom IVR setupwith LapSimSimulatorwith the4D joysticks (Simball, G-coder
Systems,Sweden);thegogglesaretheHTCVive(HMD);andtheheadphonesprovidesound.
Aconventionalmonitorisnotneededinthissetup.
3MaterialsandMethods
VirtualRealityLaparoscopySimulationSystem
The basis of our custom setup was a VR laparoscopy simulator without haptic
feedback(LapSim),purchasedfromSurgicalScienceAB,Goethenburg,Sweden.Itconsisted
ofa27-inchLCDmonitor(AOCInternational,Taiwan),akeyboardandmouse,aWindows7
PC,andSimball™4Djoysticks,withadoublefoot-switch(G-coderSystems).Allhardware
components were mounted on a rolling, height-adjustable array, and they were readily
accessible,duetotheopendesignofthechassis.ForinteractionswiththeVRenvironment,
the simulator provided Simball 4D joysticks (Fig. 1 andFig. 2). Their laser-marked ball
joints,withthreedegreesoffreedom,allowedreal-timecalculationsoftheexact3Dangular
position.Theinputdevicesincludedagrasperinstrumentontheleftandrightsides,anda
camera instrument in the center. During our tests, the camera was not used in any
6
laparoscopic task.Thecomputer featuredproprietarysoftware fromSurgicalScienceAB,
version2015,whichranonWindows7.Thissoftwareallowedtheuser toperformbasic
trainingtasks,likepegtransferorpatterncutting,butalsocomplex,morerealisticscenarios,
like a cholecystectomy or appendectomy simulation. The software logged the user's
performancebyrecordingasetoftask-specificparameters.Thisallowedtheassessmentof
execution quality for each performed task; thus, the individual's improvement in
psychomotorskillscouldbemonitoredovertime.
CustomVirtualRealityHeadMountedDisplaySystem
TwoVR-HMDsolutions fromtheconsumermarketwereconsidered inouraimto
extendtheexistingLapSimexperiencewithadditionalhardwareandcontent:TheOculus
RiftCV1andtheHTCVive.Comparedtotheconventionalbuilt-inmonitor,HMDsprovidea
widerfieldofview.Withthecombinedheadtrackingandstereoscopicdeptheffects,theuser
isimmersedinanall-aroundvisualexperiencethatismuchclosertorealhumanvisionthan
the2DdisplayoftheLapSim.ThehighrefreshrateandlowlatencyoftheOLEDdisplayson
theHMDwereimportantinachievingminimumsimulatorsickness.WechosetheHTCVive
HMDforthisstudy,becausecomparedtotheOculusRiftCV1, it featuredaslightlylarger
fieldofviewof110°comparedto101°fortheOculusRiftCV1andalargetrackingareaof
4.6 by 4.6 meters, which allowed highly-immersive, room-scale VR [22]. A separate
computerwasnecessarytodrivetheHTCViveHMDwithoutinterferingwiththeexisting
LapSimsoftwareandhardware(Fig.2c).AVR-readylaptop(MSIGT72VR-6RE16H51)was
chosenoveracustom-builtdesktopPC forbettermobility.TheMSI laptopwasequipped
withanIntelCorei7processor(6700HQ,16GBRAM),aNvidiaGTX1070graphicscard(8GB
VRAM),andtheportsnecessaryforconnectingtheHTCVivelinkbox.
VideoandAudioSignalTransfer
Tointegratethevideooutputpreviouslydisplayedonthe2DLapSimmonitor(Fig.
2a) intoavirtualenvironmentcreatedwith theseparateHTCViveHMDsystem,a frame
grabberwasused(Fig.2d).ThisUSB3.0HDvideocapturedevice(StartechUSB3HDCAP;
Startech, Northampton, UK) received the HDMI output from the LapSim PC at 1080p
resolutionand60fps,andsent itovertheUSBtothe laptopviatheHTTPlive-streaming
protocol,HLS.AnHDMIsplitter(Fig.2e)wasinsertedupstreamtoallowthevideosignalto
7
beroutedbacktotheLapSimsystemforsimultaneousdisplayonthebuilt-inmonitor.The
simultaneous display allowed other developers to follow the laparoscopy simulation, as
usual,onthe2Dmonitor.Thus,theotherdeveloperscouldprovidetechnicaladministration
andcontroltheLapSimsoftwarewiththekeyboardandmouse,whiletheuserwaswearing
theHMD. Audio feedback generated by the LapSim software (e.g.,whenworkingwith a
virtualelectronicdevice)wastransferredviaananalogcablefromtheLapSimPCheadphone
jacktotheMSIlaptopmicrophonejack(Fig.2f).Thefinalaudiomix-downwasperformed
on the laptop. The user wearing the HMD listened to the audio output with stereo
headphonesconnectedtotheHTCViveHMD.
Fig.2:ComponentdiagramoftheVRlaparoscopysimulationsystemandourcustomizedVR
HMDsystem.a: conventional2Dscreen,b: surgical simulator, c:VR-readyMSI laptop,d:
HDMItoUSBframegrabber,e:HDMIsignalsplitter,f:stereoaudiocable,g:localnetwork
connection(LAN),h:Simballdataloggerauxiliaryprogram.
8
CreationofVirtualRealitySurroundings
The cross-platform game engine, Unity™ (Version 5.4.2), was used to create the
virtualsurroundingsbyintegratingthevirtualmonitorwiththeSimballjoystickmovements.
Thelivevideostreamfromthescreengrabberwasdisplayedonaplane,whichservedasa
virtualmonitor,with theUnity engine, “WebCamTexture class”. The original streamwas
rendered at the standard resolution of 1920´1080 pixels. To optimize performance, the
texturewasdown-sampledto960´540pixelspriortorendering.SoundfromtheLapSim
systemwasintegratedbyplacinganAudioSourceobjectinthescenethatcapturedtheinput
fromthelaptop’smicrophonejack.Theremainingvirtualspacewasfilled,viadraganddrop,
withmodelsofmedicaldevices,furniture,props,andanimatedcharactermodelsofmedical
personnel and the patient. The models could be moved around to create a setting that
mimickedtheappearanceofanORduringlaparoscopicsurgery.
Analternativewaytocreateavirtualsurroundingwastointegratetheplaybackfrom
a360°video.Tocreatethiseffect,thevideoclipcouldbemappedontoaspherewiththe
MovieTexturefeatureinUnity.TheSteamVRcamerarigwasthenplacedinthecenterofthe
sphereandscaleddowntoaverysmallsizetoadjustforthefixedperspectiveofthe360°
videorecording.
ClinicianFeedback
Four members of the surgical department were continuously involved in the
developmentprocess.AllfourmembershadpreviousexperiencewiththeVRlaparoscopy
simulator.
TestingPhase
As part of the first clinical pilot investigations, other members of the surgical
departmentperformedlaparoscopictasksonthesimulator.Hypothetically,adifferencein
performanceduetodistractionsintheimmersivesetupmaybepossible.Furthermore,the
degreeofimmersionintoavirtualworld,theattentiontotheenvironmentandexhilaration
of the participant are important aspects to optimize the surrounding.We thus used the
validated questionnaire by Nichols et al. to quantify these aspects [23]. Furthermore,
negativepsychopathologicalaspectsandvegetativesideeffectshavebeendescribedinVR
andhaveleadtothedescriptionofa“codeofconduct”regardVRresearch[24].Thuswe
9
investgatedparticipants’heart rate (stress level) andmotion sicknesswith thevalidated
motionsicknessscalebyKeshavarzetal.[25].
The presented technical data is a comparison of immersive and regular VR
simulation. This was investigated with the consectuvie performance of three tasks (peg
transfer,finedissection,andcholecystectomy)afteraninitialwarm-upphaseinregularVR
mode.Theselectedtasksrepresentdifferentaspectsoflaparoscopicsurgery,navigational
maneuver,finepreparationandproceduralaspects.Taskswerealwaysperformedinregular
modefirstandAVRsecondandintheabovementionedorder.Simulatormetricshavebeen
analysedusingthetotalz-scoreofeachtask.Additionally,metricshavebeengroupedinto
categoriestime,handlingeconomicsanderrorsaspreviouslydescribed[15].
Fig. 3: Screenshot of the artificial virtual reality (AVR) operating room. The virtual
environmentwasmodeled, based on a 3Dmodel kit from Vertigo Games, Rotterdam. A
virtualinstrumentallowsinteractionwiththeLapSimsimulator.Avirtualmonitor(center)
showsthelaparoscopysimulator'sgraphicoutput.
10
4Results
A general overview of the customVR setupwe developed is shown inFig. 2. All
software and hardware components and connections proved to be generally technically
feasible.Thesetupwasuserfriendly,easytoassemble,andhighlymobile.
ArtificialVirtualOperatingRoom
We firstdevelopedanartificial virtual reality (AVR)OR.OurAVRwasbasedona
commerciallyavailablemodular3Dassetkit,whichcontainedafullyequippedsurgicalOR
(VertigoGames,Rotterdam,Netherlands).Itfeaturedanimated3Dcharactersandseveral
modelsofsurgicaldevices,props,andfurniture.Thefeedbackfromtheclinicaldevelopment
teamledtorepositioningcertaincomponentsinthevirtualenvironment.Thepositionofthe
virtualdisplaywasadaptedtocorrespondtothesettinginourclinic.Thepositionsofthe
trocars, sterile equipment, and surrounding teammembers in the AVRwere changed to
ensurethepositionwascomfortablefortheuserandreflectedtheclinicalsetting(Fig.3).
Additionally,thesizeofthevirtualmonitorwasincreasedtoimprovevisibilityoffinedetails,
but at the same time, it remained realistic. A very small delay between the joystick
movementsandthecorrespondingmovementsintheVRsurgicalmonitorwasachievedon
the virtual display. Even after the AVR was optimized, clinical feedback from the
development team revealed that they remained aware of the fact that they were in an
artificialenvironmenttheentiretime,whichledtoalowdegreeofpresence.Thislimitation
stimulatedtheinitiativetoimprovethesurroundingsandcreateamorerealisticsimulation.
IntegrationofLaparoscopyInstruments
At first, the fourdevelopers remarked that the instruments in frontof themwere
missingduringtheAVRsimulation.TheSimballinputdevicesprovidedahapticsensationto
the user,whichwas similar to the sensation of holding real surgical instruments. These
instrumentsweretheonlymeansbywhichtheusercouldphysicallyinteractwiththevirtual
world.Therefore,itwasimportantthatvirtualrepresentationsoftheseinstrumentscould
beseenwhenwearingtheHMD.ThisvisualizationoftheinstrumentsinVRenabledtheuser
tolocateandgrabthedeviceswhilewearingtheHMD.Toachievethiseffect,theinputdata
fromtheSimballshadtobetranslatedintomovementsthatwerethenappliedtothevirtual
11
objects represented by the shapes of the simulated instruments. Because the virtual
environmentwasgeneratedintheVRsystem,andnottheLapSimsystem,theSimballsinput
datahadtobeinterceptedandsentfromonecomputertotheother.Anauxiliaryprogram,
theSimballDataLogger(Fig.2h),whichraninthebackgroundontheLapSimPC,recorded
thedatawithoutinterferingwiththeLapSimsoftware.Eeroutedthedatastreamtopass
throughalocalareanetwork,whichsentittothelaptopthatrantheVRsystem(Fig.3).The
movementsofthevirtualinstrumentswereveryaccurate,accordingtotheusers.Thedouble
foot-switchwasnotvisualizedwithinthevirtualenvironment.However,thisdidnotseem
tobeaproblemfortheclinicianssincethefootswitchisoftennotvisibleintherealoperating
roomaswellduetothesurgicalcovers.
Highly-ImmersiveVirtualOperatingRoom
AnalternativeVRenvironment,thehighlyimmersiveVR(IVR)OR,wascreatedwith
a360°camera(SamsungGear360,SamsungAG,Seoul,Korea).Sphericalvideosequences
wererecordedinsidearealsurgicalORattheUniversityHospitalMainz,Germany.Forthe
video,asurgicalstaffre-enactedthesituationofareallaparoscopicsurgery.Actualmembers
of the departments acted as the patient, scrub nurses, laparoscopy assistant, and
anesthesiologist (Fig.4).TheORwas first recordedwithout a scripteddialogue.Later, a
second scenariowas createdwith interactions between the fictional characters (e.g., the
scrubnurses).Thesecondscenarioincludedsounds,actions,andconversationsthatwere
typicalduringastandardlaparoscopicprocedure.
Thetestingclinicianswerehighlyexhilaratedinresponsetothepresencetheyfeltin
theOR,particularlyduringthesecondscenario.Thesensationofpresencewasevenmore
exhilaratingwhenperformingproceduralsimulationsettings,suchasacholecystectomy,on
the LapSim. Nonetheless, the display resolution was limited in the AVR and the IVR,
particularlyintaskslikefinedissection,wherebloodvesselsmustbedifferentiated.Despite
manychangesofsettings,thislimitationwasmentionedbyallparticipants.Thatfindingled
totheconclusionthattheresolutionwaslackingwiththeHMD.Down-sizingtheinputsignal
to960´540inUnitydidnotimpactthevisualqualityoftheimagesignificantly,becausethe
virtualmonitoronlycoveredpartoftheuser'sfieldofview.ThenativeresolutionoftheHTC
ViveHMDwasinsufficienttodisplaytheoriginalimageinfullHDresolutionatthatsize.
12
Fig.4:Screenshotofthe360°highlyimmersivevirtualreality(IVR)operatingroom(OR).A
sphericalvideowasgeneratedwithaSamsungGear360°cameraintheORattheUniversity
HospitalMainz,Germany.Thevirtualmonitorinthecentershowsthegraphicoutputfrom
thelaparoscopysimulator(LapSim).
TestingPhase
In a previously published pilot study, we found no significant difference in
performancebetweentheregularVRlaparoscopyandIVR.Theparticipatingsurgicalstaff
washighlyexhilaratedandindicatedahighlevelofpresence[26].Inadifferentapproach,
we previously investigated potential vegetative side effects. Nausea was present in this
previous investigation in 10% of participants (2/20). These two female surgeons had a
historyofmotionsickness.AlthoughheartrateswereelevatedduringIVRsimulations,the
elevationswerenotstatisticallysignificant[27].
ThecurrentanalysisoftechnicalperformancecomparedregularVRlaparoscopyand
AVRof16participants.Weobservednostatisticaldifferencesinthetotalz-scoresorinthe
categorizedz-scoresforhandlingeconomics,errors,andtime.Thesetechnicalresultsare
displayedinTable1.
13
Table1:Calculatedz-scoresfortasksperformedbyparticipants(n=16)
VRLSession AVRSession Taskparameters Median(IQR) Median(IQR) P*
Cholecystectomy Total 2.76(-1.18;5.40) 1.07(-2.18;6.14) 0.918Time 0.39(-0.60;0.74) 0.28(-0.86;0.85) 0.836Economics 2.47(-3.00;4.45) 0.93(-0.74;4.72) 0.836Errors 0.80(-1.48;2.33) 0.97(-1.77;2.03) 0.756 FineDissection Total 0.35(-1.58;3.10) 0.39(-3.53;3.35) 0.918Time 0.12(-0.71;0.57) -0.12(-0.59;0.86) 0.959Economics 1.22(-1.19;2.16) 0.12(-1.43;1.63) 1.000Errors -0.19(-1.57;1.64) -0.18(-1.23;1.49) 0.756 PegTransfer Total 0.59(-1.73;2.06) 1.39(-1.33;2.67) 0.756Time 0.19(-0.62;0.60) 0.28(-0.22;0.72) 0.796Economics 0.23(-0.24;0.79) 0.78(-1.29;1.24) 0.756Errors -0.61(-0.61;1.01) 0.27(-0.60;0.71) 0.679 IQR:interquartilerange;VRL:virtualrealitylaparoscopysimulation;AVR:artificialvirtualrealityoperatingroom*Wilcoxon-Signed-Rank-Test5Discussion
Previously,Bowmanetal.[28]discussedtheimportantaspectsofimmersion,andthe
sensation of presence in VR. These aspects require hardware components that provide
optimalrefreshrates,framerates,displaysizes,anddisplayresolutions.Inthecurrentstudy,
ourVRsetupincludedanHMDwithpotentiallythebesthardwarecomponentscommercially
available.Nevertheless, theresolutionintheHMDwaslimitedforveryfinepreparations,
anditmustbeoptimizedtodeterminethesweetspotwiththebestcombinationofasmooth
frame rate and video playback, an acceptable virtual monitor size, and good image
resolution.Accordingtothe“reality-virtualitycontinuum”[29],thecombinationoftheVR
laparoscopysimulationandavirtualORinthecurrentsetuprepresentedacombinationof
twovirtuallygeneratedworlds.TheVRlaparoscopysimulatorisusuallyperceivedascloser
14
toreality thanthenewlydevelopedsetup,becausethesimulation ispresentedona two-
dimensionalmonitorinanout-of-contextenvironment.Theonlyconnectionlefttorealityis
the haptic user interface (Simball joysticks) which makes the virtual display of the
instruments evenmore important.However, interaction inAVRand IVR is currentlynot
possible and should be improved by further technical advances. An overview of the
limitations in the setupwe developed is given inTable 2,with corresponding technical
solutionsandratingsofclinicalimportanceandtechnicalviability.Handtracking(e.g.,via
stereocameras)mayincreasespatialawarenessandproprioception,andthisfeaturemay
function as an interface for changing the simulator settings in VR. Furthermore, several
issuesremainwithHMDtechnologythataredisadvantagestoenhancingtheimmersion,and
intheseaspects,thetraditionalmonitoroutshinesthenewHMDs.Forexample,althoughthe
selectedHMDpossessedhighpixeldensity(above450ppi),atcloseviewingdistances,the
so-calledscreen-dooreffectwhichisdefinedasthevisibilityofinter-pixelspaceswasquite
noticeable [30]. In addition, from an ergonomic standpoint, the roughly half a kilogram
weight of theHMD can becomeuncomfortablewith prolonged use, and thisweight also
hindersimmersion.However,newadvancementsmaygiverisetohigher-resolution,lighter-
weight, and possibly wireless HMDs; thus, both the image quality and weight might be
reduced in future applications to increase user-friendliness. Furthermore, the cost and
complexitycouldbereducedbycombiningthetwocomputersetupsintoasinglecomputer
system. We demonstrated the technical feasibilities of both the AVR, produced with a
computer-generatedOR,andtheIVR,producedwiththe360°sphericalvideosequence.
Despitethementionedtechnicallimitations,thecurrentpilotstudy,whichcompared
theAVRtoregularVRlaparoscopy,showedthatparticipantstechnicalperformancewasnot
different toregularVR.Still,anon-randomizedstudydesign isa limitationregarding the
interpretationofthecurrentresults.CochranereviewscomparingregularboxtrainerstoVR
laparoscopyshowedequivalenceinthosetrainingmethods[14].Thecurrentresultsshow
no significant difference in performance compared to regular VR surgical simulation
techniques. This was consistent with our previous investigation, where we compared
regularVRlaparoscopyandIVR.Inthatstudy,althoughwefoundnodifferenceintechnical
performance, the questionnaires revealed that the users experienced a high degree of
presenceandexhilarationandaratherlowrateofmotionsicknesswiththeIVR[26,27].The
exhilarationexperiencedinVRlaparoscopycombinedwiththeHMDscenarioswasakeyaim
of the current approach, and it is likely to increase the attractiveness ofVR laparoscopy
15
simulation.Furtherstudieswithalargercohortmustbeperformedtoevaluatethegeneral
influenceofAVRandIVRonlaparoscopysimulations.
However,futureresearchshouldplacethegreatestemphasisoninteractionswithin
theVRenvironment,accordingto theuser’sperformanceontheLapSim.Here, itmaybe
possibletousetechnicaldatafromthesimulatororaspeechrecognitionfeaturetotrigger
differentenvironmentalscenarios,dependingonthesituation.Immersionmaybeimproved
byreplacingstaticvideopartswithhighresolution360°imagesorbyconstructingvirtual
environments with multiple 360° videos. Currently, these improvements might be
technicallydemanding;however,itwouldnotbedifficulttoimplementbinauralrecordings
ofORsoundsto increasepresenceinthegeneratedworld.Thedegreeof immersionmay
generallybehigher in thecomputer-generatedAVR than in the IVR,becauseAVRallows
movements about the virtual room (roomscale VR). However, the fact that the OR
surroundingsandtheactingindividualswerefamiliartoallparticipantsmayhaveincreased
thedegreeofpresenceintheIVR.Ontheotherhand,inIVR,movementinthevirtualroom
is currently impossible, due to the static video recording. Movement in IVR might be
achievedby fusingmultiple360° camera recordingsof avirtual environment to createa
photorealistictypeofAVR.Thelowdegreeofmotionsicknessachievedmightbeexplained
by the fact that all movements were controlled by the participants, and not by the VR
environment.TheabilitytosimulatecameranavigationwasnotusedinthisfirstVRsetup.
Apotential goalof futuredevelopmentsmightbe to integrateanassistant thatperforms
camera navigation; however, this integrationmight also affectmotion sickness. Another
promisingimprovementduetohighly-immersiveVRapplicationcouldbethesimulationof
stress training in the OR as part of the surgeon’s learning curve. Further research
additionallyneedstofocusonthesimulationofinteractivescenarios.IVRmaybeauseful
tooltosupportthisandinterfacesshouldbedevelopedforsurgicalsimulationsoftwarethat
triggeraspectsinavirtualsurrounding.
Inconclusion,wehavepresentedthetechnicalandclinicaldevelopmentofahighly
immersiveVRlaparoscopysimulationsetup.Thisnewgenerationofsimulationwillenable
clinicalstudiestoevaluatetheimpactofVRforsurgicaltraining.Furthertechnicaladvances
areneededtoimprovevisualizationandinteractivity.Clinicalanalysesshouldfocusonthe
influenceofAVRandIVRonlaparoscopytraining.
16
Table2:OverviewofcurrentlimitationsofthecustomimmersiveVirtualRealitySimulationandpossiblesolutionsforfutureresearch,includingratingsoftheirclinicalimportanceandtechnicalviability
Currentlimitation PossiblesolutionClinical
importanceTechnicalviability
Monoaudiorecordingof360°camera
Binauralrecordingofintraoperativesounds
3 3
Missingvisibilityofuser’shandmotion
Handtrackingwithdatagloves 1 2
Missinghaptic/tactilefeedbackfromobjects(patient,table,steriledrapes,otherpersons…)
Useofappropriatehapticfeedbackdevice
3 1
Flatspherewithmissingdepthoffield
Three-dimensional360°cameravideo
3 2
Static360°videoofsurroundings(noroom-scaleVRinIVR)
Multiple360°videostocreateaphotorealisticVRoperatingroomtoenableroom-scaleVR
2 2
NoreactionofsurroundingstoVRsimulatordata(e.g.,mistakes)
RecordingofsimulatordatathatcanbeusedtotriggerdifferentscenariosTextrecognitionofsimulatorcommands
1 3
InteractionwithVRsimulator(administrative)
Connectdataglovestosteersimulationsoftware
1 2
Cameranavigationnotincluded
Performvirtualsurgeriesasateamofsurgeons,andincludeacameranavigator,withtwoVRheadsets
2 1
Simpleoperatingroomscenario
Recorddifferentscenarios(e.g.,stresstraining)
1 3
Surroundingsfamiliartotheparticipants
Recorddifferentoperatingrooms 2 3
Rangeofclinicalimportanceratings:1=highimportance,3=lowimportanceRangeoftechnicalviabilityratings:1=difficult,3=feasible
17
Acknowledgments
TheauthorsthankS.Rohde,B.Golla,andM.Kosta(UniversityofMagdeburg)fortechnical
support. Additionally, we appreciate the support from and discussions with H. Hecht,
DepartmentofGeneralExperimentalPsychology,JohannesGutenbergUniversity,Mainz.
EthicalApproval
Forthistypeofstudy,formalconsentwasnotrequired.Thisstudydidnotincludepatients
oranimals.
Funding
Financial support for the laparoscopy simulatorwas provided by themedical education
project,“MAICUM”,fromtheMedicalCenteroftheJohannesGutenbergUniversityofMainz.
Fundingfortheadditionalimmersivevirtualrealityhardwarewasprovidedbyintramural
fundingfromtheMedicalCenteroftheJohannesGutenbergUniversityofMainz,Germany,
and educational intramural funding by the Otto-von-Guericke University Magdeburg,
Germany.
ConflictofInterest
TheauthorsTH,TW,MP,HL,WKandCHdeclarenoconflictsof interestor financial ties
relatedtothisstudy.
Informedconsent
Thisarticledoesnotcontainpatientdata.
18
References
1. AggarwalR,GrantcharovTP,EriksenJR,BlirupD,KristiansenVB,Funch-JensenP,
DarziA(2006)Anevidence-basedvirtualrealitytrainingprogramfornovice
laparoscopicsurgeons.AnnSurg244(2):310-314.
doi:10.1097/01.sla.0000218094.92650.44
2. vanDongenKW,vanderWalWA,RinkesIH,SchijvenMP,BroedersIA(2008)Virtual
realitytrainingforendoscopicsurgery:voluntaryorobligatory?SurgEndosc22
(3):664-667.doi:10.1007/s00464-007-9456-9
3. HuberT,KirschniakA,JohanninkJ(2017)SurveyofTraininginLaparoscopicSkillsin
Germany.ZentralblChir142(1):67-71.doi:10.1055/s-0042-116327
4. EarleD,BettiD,ScalaE(2017)Developmentofarapidresponseplanfor
intraoperativeemergencies:theCirculate,Scrub,andTechnicalAssistanceTeam.AmJ
Surg213(1):181-186.doi:10.1016/j.amjsurg.2016.01.047
5. ChiricoA,YadenDB,RivaG,GaggioliA(2016)ThePotentialofVirtualRealityforthe
InvestigationofAwe.FrontPsychol7:1766.doi:10.3389/fpsyg.2016.01766
6. SchmittYS,HoffmanHG,BloughDK,PattersonDR,JensenMP,SoltaniM,CarrougherGJ,
NakamuraD,ShararSR(2011)Arandomized,controlledtrialofimmersivevirtual
realityanalgesia,duringphysicaltherapyforpediatricburns.Burns37(1):61-68.
doi:10.1016/j.burns.2010.07.007
7. RizzoA,HartholtA,GrimaniM,LeedsA,LiewerM(2014)Virtualrealityexposure
therapyforcombat-relatedposttraumaticstressdisorder.Computer47(7):31-37
8. LaverKE,GeorgeS,ThomasS,DeutschJE,CrottyM(2015)Virtualrealityforstroke
rehabilitation.CochraneDatabaseSystRev(2):Cd008349.
doi:10.1002/14651858.CD008349.pub3
9. ParsonsTD(2015)VirtualRealityforEnhancedEcologicalValidityandExperimental
ControlintheClinical,AffectiveandSocialNeurosciences.Frontiersinhuman
neuroscience9:660.doi:10.3389/fnhum.2015.00660
10. RivaG(2003)Applicationsofvirtualenvironmentsinmedicine.MethodsInfMed42
(5):524-534
11. OlaskyJ,SankaranarayananG,SeymourNE,MageeJH,EnquobahrieA,LinMC,
19
AggarwalR,BruntLM,SchwaitzbergSD,CaoCG,DeS,JonesDB(2015)Identifying
OpportunitiesforVirtualRealitySimulationinSurgicalEducation:AReviewofthe
ProceedingsfromtheInnovation,Design,andEmergingAlliancesinSurgery(IDEAS)
Conference:VRSurgery.SurgInnov22(5):514-521.doi:10.1177/1553350615583559
12. PascholdM,SchroderM,KauffDW,GorbauchT,HerzerM,LangH,KneistW(2011)
Virtualrealitylaparoscopy:whichpotentialtraineestartswithahigherproficiency
level?IntJComputAssistRadiolSurg6(5):653-662.doi:10.1007/s11548-010-0542-4
13. LarsenCR,SoerensenJL,GrantcharovTP,DalsgaardT,SchouenborgL,OttosenC,
SchroederTV,OttesenBS(2009)Effectofvirtualrealitytrainingonlaparoscopic
surgery:randomisedcontrolledtrial.BMJ338:b1802.doi:10.1136/bmj.b1802
14. NagendranM,GurusamyKS,AggarwalR,LoizidouM,DavidsonBR(2013)Virtual
realitytrainingforsurgicaltraineesinlaparoscopicsurgery.CochraneDatabaseSyst
Rev(8):CD006575.doi:10.1002/14651858.CD006575.pub3
15. PascholdM,HuberT,KauffDW,BuchheimK,LangH,KneistW(2014)Preconditioning
inlaparoscopicsurgery--resultsofavirtualrealitypilotstudy.LangenbecksArchSurg
399(7):889-895.doi:10.1007/s00423-014-1224-4
16. KneistW,HuberT,PascholdM,LangH(2016)[3DVirtualRealityLaparoscopic
SimulationinSurgicalEducation-ResultsofaPilotStudy].ZentralblChir141(3):297-
301.doi:10.1055/s-0033-1350609
17. SorensenSM,SavranMM,KongeL,BjerrumF(2016)Three-dimensionalversustwo-
dimensionalvisioninlaparoscopy:asystematicreview.SurgEndosc30(1):11-23.
doi:10.1007/s00464-015-4189-7
18. TillouX,CollonS,Martin-FrancoisS,DoerflerA(2016)RoboticSurgerySimulator:
ElementstoBuildaTrainingProgram.JSurgEduc73(5):870-878.
doi:10.1016/j.jsurg.2016.04.008
19. SankaranarayananG,LiB,ManserK,JonesSB,JonesDB,SchwaitzbergS,CaoCG,DeS
(2016)Faceandconstructvalidationofanextgenerationvirtualreality(Gen2-VR)
surgicalsimulator.SurgEndosc30(3):979-985.doi:10.1007/s00464-015-4278-7
20. LaValleSM,YershovaA,KatsevM,AntonovMHeadtrackingfortheOculusRift.In:
RoboticsandAutomation(ICRA),2014IEEEInternationalConferenceon,2014.IEEE,
pp187-194
20
21. EvangelakosD,MaraMExtendedTimeWarplatencycompensationforvirtualreality.
In:Proceedingsofthe20thACMSIGGRAPHSymposiumonInteractive3DGraphicsand
Games,2016.ACM,pp193-194
22. CoburnJQ,FreemanI,SalmonJL(2017)AReviewoftheCapabilitiesofCurrentLow-
CostVirtualRealityTechnologyandItsPotentialtoEnhancetheDesignProcess.
JournalofComputingandInformationScienceinEngineering17(3):031013
23. NicholsS,HaldaneC,WilsonJR(2000)Measurementofpresenceanditsconsequences
invirtualenvironments.InternationalJournalofHuman-ComputerStudies52(3):471-
491.doi:http://dx.doi.org/10.1006/ijhc.1999.0343
24. MadaryM,MetzingerTK(2016)RealVirtuality:ACodeofEthicalConduct.
RecommendationsforGoodScientificPracticeandtheConsumersofVR-Technology.
FrontiersinRoboticsandAI3(3).doi:10.3389/frobt.2016.00003
25. KeshavarzB,HechtH(2011)ValidatinganEfficientMethodtoQuantifyMotion
Sickness.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety53
(4):415-426.doi:10.1177/0018720811403736
26. HuberT,PascholdM,HansenC,WunderlingT,LangH,KneistW(2017)New
dimensionsinsurgicaltraining:immersivevirtualrealitylaparoscopicsimulation
exhilaratessurgicalstaff.SurgEndosc.doi:10.1007/s00464-017-5500-6
27. HuberT,PascholdM,HansenC,LangH,KneistW(2017)["Iwilldolaparoscopy
somewhereelse":Total,highlyimmersivevirtualrealitywithoutsideeffects?].Der
Chirurg.doi:10.1007/s00104-017-0465-5
28. BowmanDA,McMahanRP(2007)VirtualReality:HowMuchImmersionIsEnough?
Computer40(7):36-43.doi:10.1109/MC.2007.257
29. MilgramP,TakemuraH,UtsumiA,KishinoFAugmentedreality:aclassofdisplayson
thereality-virtualitycontinuum.In,1995.pp282-292
30. ChoJ-m,KimY-d,JungSH,ShinH,KimTScreenDoorEffectMitigationandIts
QuantitativeEvaluationinVRDisplay.In:SIDSymposium2017.vol1.WileyOnline
Library,pp1154-1156