1

Technology Assisted Learning

Division

Training Pediatric Examination Interaction Skills

Robert Hubal

Technology Assisted Learning Division

RTI International

2

Synthetic Characters for Interaction Skills Training & Assessment

Need for characters to respond appropriately: To user’s natural language input. Behave as if they are happy, sad, confused,

schizophrenic, etc. (Insert your favorite adjective.) Response involves:

– Choice of utterance.– Facial expression.– Gesture and whole-body movement.– Updates to cognitive, emotional models.– Application flow.

No attempt (yet) to involve user input via head-mounted display, gaze tracking, physiologic indicators, vocal affect.

3

Series of Interaction Skills Training Applications

Law enforcement: Encounters with persons

with mental illness.

Deception detection.

Assess behavior of at-risk populations.

Medical: Taking patient history.

Interacting with children.

Trauma patient assessment.

Military: Maintenance assistance.

Chemical agent attack triage.

Self aid / buddy aid.

Civilian: Conducting surveys.

Customer service.

Informed consent.

4

Specific Application: Virtual Pediatric Standardized Patient

Audience – medical students, pediatrics rotation.

Intent is to provide practice using verbal skills to engage patients. Implement strategies considered successful for

calming or eliciting information from pediatric patients.

Why? Because use of real pediatric SP’s is not always ethical, reliable, or even possible.

Synthetic characters – 4 year old girl, 9 year old boy, adolescent female.

5

Scenarios & Outcomes

Girl: Check ears with otoscope within 5 minutes. Girl’s ear may have inflammation, affecting behavior. Strategies such as giving a toy, demonstrating on a teddy bear,

having her sit on her mother’s lap will calm her down. Obviously need a mother in the room.

Boy: Check breathing with stethoscope (front & back) within 2 minutes. Boy may be asthmatic. Boy will be cooperative.

Teen: Conduct an adolescent social history. Teen will provide different answers depending on presence of

parent. A father may or may not be in the room.

6

Tools Used to Gather Modeling Data

Models involved: Cognitive.

Emotional.

Linguistic.

Gestural.

Subject-matter expert guidance and feedback.

Videotapes: Of medical students,

pediatricians interacting with kids.

Of kids in natural settings. Used in existing intervention

programs. Direct observation (of users

of simulations). Motion capture. Literature on:

Normal behavior characteristics (gesture, facial expression, language, conversational pragmatics).

Children’s behavior.

7

Cognitive Modeling

What is known about the world: What is the intent of the scenario (e.g., user has to

check ears with otoscope).

Reason about social roles and conventions: What can be stated or asked at any point in the

dialog.

How it gets stated or asked.

Application-specific parameters and initial conditions drive how synthetic characters express knowledge.

8

Emotion Modeling

Maintain continuous base emotion variables anchored by modifiable descriptors.

Map emotional states onto base variables and enable updating: Emotion mapping: AFRAID (FEAR >= SOMEWHAT_HIGH)

Emotion setting: APPEASE (ANGER –= VOLATILITY × MEDIUM)

Separate personality (stable; e.g., VOLATILITY) from mood (dynamic; e.g., FEAR).

Updates based on student’s input: Format (command, request, statement). Lexical analysis (politeness, degree of compromise, detachment). Semantic content (topic relevance, sensitivity).

9

Linguistic Modeling

Need to interpret inputs: Commands, queries or requests, statements of

appreciation or understanding, threats or insults.

Appropriateness to particular synthetic character.

Need to generate appropriate responses: Reply, question, challenge, deny, zone out.

Different responses based on cognitive, emotional states.

Pragmatics of conversation are dependent on synthetic character’s particular role.

10

Gestural Modeling

Whole body, head, arms/hands, facial expression (using emotional state mappings).

Gestures: Most verbal utterances accompanied by gestures. Most representational, rest beats, idle motions. Children’s gestures generally larger than adults’.

Posture: Sustained glance and a smile invites contact. Head bent and shoulders down typical of unease,

sadness. Heads up for deference to authority. Eyes, mouth wide open, torso back indicates fear.

11

Demo

12

Evaluation

Participants: pediatric experts at annual COMSEP meeting, Colorado medical students.

Measures: participants’ responses to characters, logged usage data, post-usage ratings.

Results: Scenarios address important pediatric competencies at which on

average only half of medical students are competent at graduation.

Response time and overall conversation realistic, scenarios somewhat comparable to real world situations.

Synthetic characters helpful and allow for more experiential learning.

Participants enjoyed using the simulations and felt they were likely to learn with synthetic characters during their career.

13

What Next?

Interaction between behavior engine and physiology model. Use a multiple-compartment (e.g., brain, heart, liver) transport

architecture. Represents physiological functions and pharmacological actions

(e.g., drugs, inhaled & chemical agents, tracers/dyes) and interactions.

Facilitating integration of gestures within an utterance: Beats, nods and shakes, leaning forward, autonomic movements. Governing potentially interfering gestures.

Obtain user information based on: Vocal affect. Facial expression. Biometric measures.