Immersive Virtual Characters for Immersive Virtual Characters for Educating Medical Communication SkillsEducating Medical Communication Skills

T. Bernard, C. Oxendine, D. S. Lind, P. WagnerT. Bernard, C. Oxendine, D. S. Lind, P. WagnerDept of Surgical Oncology (Medical College of Georgia)Dept of Surgical Oncology (Medical College of Georgia)

K. Johnsen, A. Raij, R. Dickerson, R. Wells, B. LokK. Johnsen, A. Raij, R. Dickerson, R. Wells, B. LokDept of Dept of Comp and Info Science and Eng (College of Eng)Comp and Info Science and Eng (College of Eng)

M. Cohen, A. StevensM. Cohen, A. StevensDept of Surgery (College of Med)Dept of Surgery (College of Med)J. Cendan, M. Duerson, R. PaulyJ. Cendan, M. Duerson, R. PaulyDept of Community Health and Family Med (College of Med) Dept of Community Health and Family Med (College of Med) R. FerdigR. FerdigCollege of EducationCollege of Education

Previous Work: Technology for Previous Work: Technology for Medical Communication SkillsMedical Communication Skills

Traditional approaches [Bearman 2003]Traditional approaches [Bearman 2003]– Narrative Narrative – Problem SolvingProblem Solving

Human Patient Simulator [Meurs 1997]Human Patient Simulator [Meurs 1997]– Mechanical simulationMechanical simulation– Motors and actuators for I/OMotors and actuators for I/O

JUST VR [Ponder 2002]JUST VR [Ponder 2002]– Immersive approach (Stereo Projection)Immersive approach (Stereo Projection)– Trains students to react to emergency Trains students to react to emergency

situationssituations– Uses human instructor to control systemUses human instructor to control system

Virtual Standardized Patient [Hubal Virtual Standardized Patient [Hubal 2000]2000]– Commercial Desktop application by RTICommercial Desktop application by RTI– Speech RecognitionSpeech Recognition– Natural Language EngineNatural Language Engine

Project DescriptionProject Description

Simulate a Simulate a standardized standardized patientpatient encounter encounter Allow repeated interaction with Allow repeated interaction with an an Immersive, interactive Immersive, interactive virtual patient virtual patient in a constrained in a constrained scenarioscenarioVirtual patient, DIANAVirtual patient, DIANAVirtual instructor, VIC Virtual instructor, VIC To address issues with SPsTo address issues with SPs– Experience diversityExperience diversity– Quality ControlQuality Control– FeedbackFeedback

Communication skillsCommunication skillsNo physical diagnosisNo physical diagnosisInterpersonal SimulatorInterpersonal Simulator

Play VideoPlay Video

Things to look for: interaction modalitiesThings to look for: interaction modalities

SystemSystem

Low CostLow Cost– < $8,000(USD)< $8,000(USD)

COTSCOTSComponentsComponents

Potential:Potential:– Every HospitalEvery Hospital

Natural Interaction InputNatural Interaction Input

No Keyboard, No MouseNo Keyboard, No MouseSpeech RecognitionSpeech Recognition– Dragon Naturally Speaking 8 Dragon Naturally Speaking 8

ProPro– Accuracy 90+% with 10 Accuracy 90+% with 10

minutes trainingminutes training– 70% match to database70% match to database

Track Communication CuesTrack Communication Cues– Non-VerbalNon-Verbal

Track head gazeTrack head gazeTrack left handTrack left handTrack body leanTrack body lean

– VerbalVerbalInflectionInflectionJargonJargon

Gesture RecognitionGesture Recognition– Pointing, handshakePointing, handshake

Natural Interaction OutputNatural Interaction Output

DIANA and VIC look DIANA and VIC look at userat userLife-size charactersLife-size charactersAnimationAnimation– Hand gesturesHand gestures– Head movementHead movement

Perspective-Correct Perspective-Correct RenderingRenderingWhy this worksWhy this works– Does not rely on Does not rely on

complete sentencescomplete sentences– Constrained scenarioConstrained scenario– Students trained on Students trained on

specific questionsspecific questions

Eight StudiesEight Studies

20042004– AprilApril: Project initiated: Project initiated– AugustAugust: Prototype (n=7) UF: Prototype (n=7) UF– OctoberOctober: Experts (n=3) UF: Experts (n=3) UF– DecemberDecember: Pilot Test (n=10) UF: Pilot Test (n=10) UF

20052005– JuneJune: Two Institutions (n=16) UF/MCG: Two Institutions (n=16) UF/MCG– JulyJuly: VP vs SP (n=16, n=8) UF/MCG: VP vs SP (n=16, n=8) UF/MCG– OctoberOctober: Cultural Bias (n=16) MCG: Cultural Bias (n=16) MCG– OctoberOctober: Class Integration (n=33) UF: Class Integration (n=33) UF

nn = 101 = 101Testing CentersTesting Centers– Harrell Center at UFHarrell Center at UF– Medical College of GeorgiaMedical College of Georgia– Diana was in Exam Room #3 (video)Diana was in Exam Room #3 (video)

Eight StudiesEight Studies

VP VP ≈? SP≈? SPHow is experiencing an interpersonal scenario with a virtual How is experiencing an interpersonal scenario with a virtual person similar to – and different from – experiencing an person similar to – and different from – experiencing an interpersonal scenario with a real person?interpersonal scenario with a real person?

Clearly differentClearly different– But… in what important ways?But… in what important ways?

Patient-doctor interview provides a constrained scenarioPatient-doctor interview provides a constrained scenario

The study asks:The study asks:– Are post-encounter impressions similar?Are post-encounter impressions similar?– Are empathy and other emotions and attitudes similarly expressed?Are empathy and other emotions and attitudes similarly expressed?– Which social constructs are followed?Which social constructs are followed?

These questions must be explored to:These questions must be explored to:– Determine the extent to which interpersonal scenarios can be simulated with Determine the extent to which interpersonal scenarios can be simulated with

virtual humansvirtual humans– Identify how component technologies need to improve to enable effective Identify how component technologies need to improve to enable effective

interpersonal virtual human systemsinterpersonal virtual human systems

Overall PerformanceOverall Performance

Similar ExperiencesSimilar Experiences– Same % of participants asked key questionsSame % of participants asked key questions– Out of 12: 6.3 Out of 12: 6.3 ± 1.7, 5.5 ± 2.1 (± 1.7, 5.5 ± 2.1 (αα = 0.37) = 0.37)– Same % of participants passed the scenarioSame % of participants passed the scenario– SP = 50%, VP = 36%SP = 50%, VP = 36%

But…But…– Some critical items were not asked at the same frequencySome critical items were not asked at the same frequency– Sexual activity (SP = 0.88, VP = 0.44)Sexual activity (SP = 0.88, VP = 0.44)– Nausea (SP=0.88, VP=0.25)Nausea (SP=0.88, VP=0.25)– Related to virtual patient expressivenessRelated to virtual patient expressiveness

Behavioral MeasuresBehavioral Measures

EmpathyEmpathy– Empathetic moment – “I’m scared, can you help Empathetic moment – “I’m scared, can you help

me?”me?”– Expressed the same % and same # of times (SP Expressed the same % and same # of times (SP

= 2.2, VP = 1.3)= 2.2, VP = 1.3)

But…But…– Appears less genuine (very robotic)Appears less genuine (very robotic)– Conversation flow is “rapid-fire”Conversation flow is “rapid-fire”

Confirmatory phrases statistically different (SP = 20, VP Confirmatory phrases statistically different (SP = 20, VP = 3.5)= 3.5)

Lessons LearnedLessons Learned

Overall experiences similarOverall experiences similar– Questions askedQuestions asked– Global measuresGlobal measures

Education goals metEducation goals met– Students rated educational merits similarlyStudents rated educational merits similarly– Students rated difficulty similarlyStudents rated difficulty similarly

Global measures of realism do not workGlobal measures of realism do not work– Battery of specific measures more accurateBattery of specific measures more accurate

Practice tool Practice tool in addition in addition to SPsto SPsDiana rated a Diana rated a 6.36(.85)6.36(.85) on a 1 to 10 scale (Average on a 1 to 10 scale (Average is a is a 7.477.47 for real SP’s) for real SP’s)VR works out of the lab VR works out of the lab

PapersPapersComputer Science

– Raij,et al., “Interpersonal Scenarios: Virtual≈Real?”, IEEE Virtual Reality 2006– Johnsen, et al., “Evolving an Immersive Medical Communication Skills Trainer”, Journal

on Presence: Teleoperators and Virtual Environments– Dickerson, et al., “Virtual Patients: Assessment of Synthesized Versus Recorded

Speech”, Medicine Meets Virtual Reality 14– Johnsen, et al., “Experiences in Using Immersive Virtual Characters to Educate Medical

Communication Skills”, IEEE Virtual Reality 2005– Dickerson, et. al., “Evaluating a Script-Based Approach to Simulating Patient-Doctor

Interaction”, SCS 2005 Int’l Conf on HCI Adv for Modeling and SimMedicine

– Stevens, et al., “The Use of Virtual Patients to Teach Medical Students History Taking and Communication Skills“, American Journal of Surgery

– Lind and Lok, “The Role of Virtual Patients in Medical Education: Teaching Tool Versus Technological Trend”, FOCUS on Surgical Education

– Stevens, et al., “Implementing a Virtual Patient into the Medical School Curriculum at the University of Florida”, Southern Group on Education Affairs 2006 (pres.)

– Cohen, et al., “Do Health Professions Students Respond Empathetically to a Virtual Patient?”, Southern Group on Education Affairs 2006 (pres.)

– Bernard, et al. “A Multiinstitutional Pilot Study to Evaluate the Use of Virtual Patients to Teach Health Professions Students History-Taking and Communication Skills”, Society of Medical Simulation 2006

– Stevens, et al., “The Use of Virtual Patients to Teach Medical Communication Skills”, Assoc of Surg Educ 2005, Southern Group on Educational Affairs 2005 (pres.)

Current WorkCurrent Work

Incorporation into the Incorporation into the classroomclassroom

UF Essentials of Patient UF Essentials of Patient Care Care – n=34 students this yearn=34 students this year– n=135 next yearn=135 next year

MCG Patient MCG Patient CommunicationCommunication– n=200 each yearn=200 each year

StudyStudy– Real Speech vs. Synethsized Real Speech vs. Synethsized

Speech Speech – Immersive vs. Non-immersiveImmersive vs. Non-immersive

Do people carry real world biases into the virtual world?

Do people carry real world biases into the virtual world?

The Use of a Virtual Scenario to Teach Communication Skills for Geriatric Patients

Patient DiversityPatient Diversity

Do students treat Do students treat different virtual patients different virtual patients similarly?similarly?

MeasuresMeasures

Eye gazeEye gaze

Body LeanBody Lean

Vocal InflectionVocal Inflection

TimeTime

InterruptionsInterruptions

Communication SkillsCommunication Skills

Fundamental to clinical practice.Fundamental to clinical practice. Patient & physician satisfaction.Patient & physician satisfaction.

Patient understanding & adherence.Patient understanding & adherence.

Healthcare outcomes.Healthcare outcomes.

Malpractice litigation.Malpractice litigation.

Learned skill & experiential Learned skill & experiential learning alone is insufficient.learning alone is insufficient.

Virtual Objective Structured ExamVirtual Objective Structured Exam(VOSCE)(VOSCE)

NBME

Curriculum Curriculum integrationintegration

UF 1st year Medical Students

n = 135

MCG1st year Medical Students

n = 200

Virtual Patient Experience(Early 1st semester)

EPC courses(Semesters 2-4)

ECM courses(Semesters 2-4)

EPC courses(Semester 1)

ECM courses(Semester 1)

Virtual Patient TeamsVirtual Patient Teams

T. Bernard, C. Oxendine, D. S. Lind, P. WagnerT. Bernard, C. Oxendine, D. S. Lind, P. WagnerDept of Surgical Oncology (Medical College of Georgia)Dept of Surgical Oncology (Medical College of Georgia)

K. Johnsen, A. Raij, R. Dickerson, R. Wells, B. LokK. Johnsen, A. Raij, R. Dickerson, R. Wells, B. LokDept of Dept of Comp and Info Science and Eng (College of Eng)Comp and Info Science and Eng (College of Eng)

M. Cohen, A. StevensM. Cohen, A. StevensDept of Surgery (College of Med)Dept of Surgery (College of Med)J. Cendan, M. Duerson, R. PaulyJ. Cendan, M. Duerson, R. PaulyDept of Community Health and Family Med (College of Med) Dept of Community Health and Family Med (College of Med) R. FerdigR. FerdigCollege of EducationCollege of Education