Assessment of IPE to Move Beyond Attitudes: Featured tools and case studies

from the field Thursday, January 8

Moderator & Presenter

Sarah Shrader, PharmD

Clinical Associate Professor, University of Kansas, School of Pharmacy

Presenters

Erin Abu-Rish Blakeney, PhD, RN Faculty,

University of Washington, School of Nursing

Seattle, WA

Doug Brock, PhD Associate Professor

Department of Family Medicine and MEDEX Northwest,

University of Washington-School of Medicine

Seattle, WA

LuAnn Wilkerson, EdD Professor of Medicine and Senior Associate Dean for

Medical Education, David Geffen School of Medicine,

University of California, Los Angeles

Assessment of IPE to Move Beyond Attitudes: Featured Tools and Case

Studies from the Field

Part 1

January 8, 2015

Sarah Shrader, PharmD, FCCP, BCPS, CDE (moderator)

Assessment Beyond Attitudes • Intensifying interest to assess IPE beyond attitudinal

measures

• Quantitative, qualitative, and mix-methods approach

• Cochrane Review Updated 2013: “To improve the quality of evidence relating to IPE and patient outcomes or healthcare process outcomes, the following three gaps will need to be filled: first, studies that assess the effectiveness of IPE interventions compared to separate, profession-specific interventions; second, RCT, CBA or ITS studies with qualitative strands examining processes relating to the IPE and practice changes; third, cost-benefit analyses.”

Reeves S, Perrier L, Goldman J, Freeth D, Zwarenstein M. Interprofessional education: effects on professional practice and healthcare outcomes (update). Cochrane Database of Systematic Reviews 2013, Issue 3. Art. No.: CD002213. DOI:10.1002/14651858.CD002213.pub3.

Evaluation Outcomes Measured

Abu-Rish, E. et al. Current Trends in Interprofessional Education of Health Sciences Students: A Literature Review. 2012;26: 444-51.

Assessment Methods

Abu-Rish, E. et al. Current Trends in Interprofessional Education of Health Sciences Students: A Literature Review. 2012;26: 444-51.

A Point to Ponder

• Interprofessional Activity/Program Evaluation VS. • Interprofessional Learner Assessment

Level 1a: Reaction • Learners’ views on the learning experience and its interprofessional

nature Level 2a: Modification of attitudes/perception • Changes in reciprocal attitudes between participant groups.

Changes in perception or attitude towards the value and/or use of team approaches to caring for a specific patient/client group.

Level 2b: Acquisition of knowledge and/or skills • Including knowledge and skills linked to interprofessional

collaboration. Level 3: Behavioral change • Identifies individuals’ transfer of interprofessional learning to their

practice setting and their changed professional practice Level 4a: Change in organizational practice • Wider changes in the organization and delivery of care Level 4b: Benefits to patients/clients • Improvements in health or well-being of patients/clients

Kirkpatrick/Barr’s Evaluation Framework

From Barr, Koppel, Reeves, Hammick, Freeth. Effective Interprofessional Education Argument, Assumption & Evidence. 2005 Malden, MA: Blackwell Publishing,Ltd.

Miller’s Pyramid of Assessment

Additional Resources Canadian Interprofessional Health Collaborative. An inventory of quantitative tools measuring interprofessional education and collaborative practice outcomes. 2012 Aug. Available from http://www.chd.ubc.ca/news/featured-inventory-quantitative-tools-measure-interprofessional-education-and-collaborative-pra. National Center for Interprofessional Practice and Education Measurement Instruments https://nexusipe.org/measurement-instruments

Additional Resources Valentine MA, Nembhard IM, Edmondson AC. Measuring teamwork in health care settings: A Review of Survey Instruments [Internet]. Harvard Business School. 2011 May. Report No.:11-116. Available from: http://hbswk.hbs.edu/item/6727.html. Measuring the Impact of Interprofessional Education on Collaborative Practice and Patient Outcomes: A Consensus Study. October 7, 2014. Available from: https://www.iom.edu/Activities/Global/MeasuringtheImpactofInterprofessionalEducation/2014-OCT-07.aspx

Additional Resources

Blue A, Chesluk B, Conforti L, Holmboe E. Assessment and Evaluation in Interprofessional Education: Lessons Learned from a Multi-Methods Study. • Article accepted for publication in the Journal

of Allied Health • Webinar from 2014 posted on National Center

Resource Exchange (click link above).

ASSESSMENT OF IPE TO MOVE BEYOND ATTITUDES: VALIDATION OF THE PERFORMANCE ASSESSMENT FOR INTERPROFESSIONAL COMMUNICATION AND TEAMWORK (PACT)

Doug Brock, PhD Erin Abu-Rish Blakeney, PhD, RN Chia-Ju Chiu, PhD, PT

Acknowledgements • This work was completed as Dr. Chia-Ju Chiu’s dissertation, to develop widely applicable tools for assessing team performance.

• The work was supported by the University of Washington Macy Project Evaluation Workgroup.

• The work was funded in part by a grant from the Josiah Macy Foundation.

Background

• University of Washington IPE

• Training pre-licensure students

• Focus on Kirkpatrick/Barr acquisition of skills / knowledge

• Assessment of team performance, not individuals

• Emphasis on simulation

• Josiah Macy Foundation Grant

• TeamSTEPPS® National Training program (2009)

Conceptual Framework– TeamSTEPPS®

Aims 1) Develop psychometrically sound measures to

assess teamwork and communication skills

2) Collect data, summarize and present construct validity evidence for the PACT tools

Context—The Macy Project • Three simulated acute care cases aimed at improving team communication skills • Dyspnea in a Hospitalized Patient • Teenager with Asthma • Postoperative patient with tachycardia

• Medical school capstone included medical, nursing, pharmacy, & PA students • Pilot study—49 IP students • Validation study—306 IP students

Development of the PACT

PACT Toolset Validation Methods • Tools included in validation study

• PACT-Novice – Peer or untrained observer scale • PACT-Expert – Faculty or expert scale • PACT-Video – Trained raters

• Assessed reliability and validity

PACT—Expert

PACT—Novice

PACT—Video

Construct Validity Evidence 1) Content 2) Substantive 3) Structural 4) Generalizability 5) External 6) Consequential * Messick, 1995

Reliability Evidence

Version Number of

Valid Records

Number of Items

ICC F Cronbach’s

Alpha

PACT-Novice

904 5 .85 6.46** .85

PACT-Expert

85 13 .76 4.09** .79

PACT-Video

112 26 .90 9.63** .91

ICC(1,k) was calculated with one-way random effects model where rater effects are random. ICC for Average Measures was reported. ** F is significant p < .001

Content: Relevance & Representativeness • Scenarios, training, & tools were developed incorporating TeamSTEPPS® framework, constructs & terminology:

1. Team Structure,

2. Leadership,

3. Situation Monitoring,

4. Mutual Support,

5. Communication.

• All 5 TeamSTEPPS teamwork constructs were represented in global quality ratings in all tools • PACT-Expert & PACT-Video included behavioral markers

Substantive: PACT-Novice & PACT-Expert Consistency in Responses & Processes • PACT-Novice scoring system was introduced to the raters during didactic session.

• Scenarios unfolded under faculty guidance.

• Coaching of clinical skills ensured focusing on communication and teamwork skills.

• Administered immediately following simulated scenarios & before the debriefing.

Substantive: Consistency in Responses & Processes PACT-Video • PACT-Expert raters were trained through PACT-Video rater training program & were experts in TeamSTEPPS®.

• PACT-Video raters received rater training program before rating the videos.

• Strict rating protocol was developed to ensure the consistency of rating processes.

Structural: Scales reflect theoretical target

• Internal consistencies of the subscales • Exploratory factor analysis

Structural—PACT Expert Behavioral Marker

Rotated Factor*

Communication Leadership & Team Structure

Mutual Support

Hand-off &

S-Bar 10. Repeat back instructions or clarifications. .673

5. Applies the STEP process. .627 .355

13. Demonstrates closed-loop communication. .597

12. Demonstrates SBAR. .544 .520

9. Verbalize their activities aloud. .524

2. Refer to protocols and checklists. .362

3. Conducts briefs/huddles/debriefs. .326 .565

1. Recognize a leader .769 4. Empowers team members to speak freely and ask questions.

.658

6. Patient/Family is included. .363

8. Use Two Challenge rule or CUS. .889

7. Ask for assistance prior to task overload. .353

11. Hands off to another team member. .597 Extraction Method: Principal Axis Factoring. Rotation Method: Varimax with Kaiser Normalization.

*Only values > 0.3 are displayed.

Conclusions • Validity is a continuing and iterative process of establishing evidence to support specific uses & interpretations of the tool.

• Preliminary construct validity evidence for PACT was demonstrated.

Next Steps • Broad dissemination of PACT toolset

• American Educational Research Association (AERA) 2015 • Additional publications

• Strengthen validity: • Replication/Different study designs to enhance generalizability • Build external validity through associations with external

measures • Explore the consequences/impact—consequential validity—of

team assessment

• Explore use with practicing clinical teams

Best Uses for PACT Toolset • TeamSTEPPS-aligned situations/scenarios

• Assess team performance (not individual)

• PACT-Novice: During teaching/training • Observers • Debriefing

• PACT-Video: Research tool for understanding team behavior and performance.

PACT Toolset Resources • Link to PACT tool set: http://collaborate.uw.edu/educators-toolkit/tools-for-evaluation/performance-assessment-of-communication-and-teamwork-pact-too

• Link to “Simulation Team Training Toolkit”: http://www.collaborate.uw.edu/educators-toolkit/interprofessional-simulation-team-training-faculty-toolkit/simulation-team-trainin

Creating Innovative Tools for Assessing Learners’ Interprofessional

Competencies LuAnn Wilkerson, EdD

With funding from the Josiah Macy Jr. Foundation And a team of co-investigators from Nursing and Medicine

Assessing Interprofessional Competencies

IPEC Core Competencies

Assessment Tools Values/ Ethics

Roles/ Responsibilities

Interprofessional Communication

Teams/ Teamwork

OSCE stations X X X

Direct Observation X X

Knowledge Test X X X

Video Case Exam X X X

Implicit Association Test X X

Multi-Source Evaluation X X

Assessing Interprofessional Competencies

IPEC Core Competencies

Assessment Tools Values/ Ethics

Roles/ Responsibilities

Interprofessional Communication

Teams/ Teamwork

OSCE stations X X X

Direct Observation X X

Knowledge Test X X X

Video Case Exam X X X

Implicit Association Test X X

Multi-Source Evaluation X X

OSCE Station

Roles/Responsibilities Interprofessional communication

Teams/Teamwork

40

• Setting: Outpatient heart failure follow up clinic • Complaint: need for medication change • Roles: NP and senior medical student

– Standardized NP and senior medical student OR – Standardized senior medical student and NP student

• Can be used as a simulation or an OSCE

OSCE Station

Roles/Responsibilities Interprofessional communication

Teams/Teamwork

• Setting: Outpatient Family Practice Clinic • Complaint: sudden change in vision • Roles: NP student and medical student • Checklist focus:

– Skills in handoffs – Collaborative decision making – Understanding scope of practice

Interprofessional Collaborator Assessment Rubric – Individual performance

TeamSTEPPS Questionnaire – team performance

eWalk© Direct Observation Tool Interprofessional Communication

Teams/Teamwork

eWalk® (Media-X Systems Inc., Ottawa, Canada) uses a tablet database to collect, store, analyze, and report observational data

eWalk® (Media-X Systems, Inc, Ottawa, Canada) allows to collect, store, analyze, and report data based on real time observation.

Knowledge Test

Roles/responsibilities Interprofessional communication

Teams/teamwork

• Item bank for test construction – 49 questions keyworded to 37 discreet objectives from the

Core Competencies for Interprofessional Collaborative Practice

• Vignette-based • Multiple choice format • 7-item collaborative practice case study

– 20 questions on system-based practice and policy

• Answer key

• Goal: to synthesize feedback from team members with whom a student has worked for a period of time

– Student submits the names of 6 team members – Online survey tool (REDcap) is used to contact and remind invitees to respond

• Survey uses 6 Interprofessional Collaborator Assessment Rubric categories – Communication (7 items) – Collaboration (4 items) – Roles and Responsibilities (7 items) – Collaborative Patient-Family Centered Approach (4 items) – Team Functioning (5 items) – Conflict Management/Resolution (4 items)

• Invitee selects which behaviors he/she has observed and indicates level of the described behavior demonstrated over time

– Developing -- Occasionally demonstrates – Competent -- Frequently demonstrates – Mastery -- Consistently demonstrates

Multi-source Assessment

Interprofessional Communication Teamwork

Interprofessional issues addressed in The Patient or the Protocol: • Shared decision

making • Teamwork • Conflict of interest • Paternalism • Palliative care • Roles and

responsibilities • Clinical trials and

informed consent • Patient safety

Zaption® (Stigler JW, Zaption Inc.,2013 ) combines video with question/response database https://www.zaption.com

Video Case Exam

Values/Ethics Roles/Responsibilities

Teams/Teamwork

Zaption® can add to any online video: • multiple lines of text • Images • drawings done with

the mouse • essay questions with

multiple-line text box • questions requiring a

number for response • questions requiring

drawing the response • multiple choice

questions

47

• The IAT was developed by Tony Greenwald and Mahzarin Banaji in 1994 to study attitudes and beliefs that are “normally outside of conscious control.” – Measures the strength of associations between concepts and

evaluations or stereotypes about others. – In the early 2000’s, Nosek, Greenwalk and Banaji developed a

demonstration website to study intergroup attitudes and stereotypes. – Growing evidence that these unintended biases can be changed. – https://implicit.harvard.edu

Banaji & Greenwald, The Blind Spot: Hidden Biases of Good People, 2013.

Implicit Association Test

Roles/responsibilities Values/ethics

• Goal: help students evaluate their implicit assumptions about the roles of nurses and physicians in the health care system.

• The concepts and evaluative terms for the IAT were chosen based on focus groups held with UCLA medical students and advanced practice nursing students.

• Project Implicit created the interprofessional IAT.

Formative Evaluation Evaluation Questions Methods

1. How do the results of the student needs assessment inform tool design, content and learning outcomes?

Focus groups with Advanced Practice nursing students and 3rd year medical students

2. What are the major competency gaps for medical and APN students and do they differ by students’ clinical level or profession?

Same as above

3. How useful are the assessment tools when used formatively?

Pilot testing 6 tools

Implementation Evaluation Evaluation Questions Methods 1. Usability: How easily do students find the IPE assessment tools to complete? How can the tools and implementation be improved?

Develop usability evaluation for each tool

2. What improvements can be made in the faculty and preceptor implementation guides for the assessment tools to better prepare for developing and/or observing IP competencies?

Pilot test each tool

3. How can the implementation processes be improved or refined to optimize the product?

Pilot test each tool

4. What are the psychometric properties of each of the assessment tools? Are revisions needed to improve reliability and validity?

Analyze pilot data

Outcome Study Methods Evaluation Questions Cohort study

1. To what extent do students’ performance on the new assessment tools vary based on prior experience in IP settings, prior academic performance, age, gender, health profession, or mean ratings of their SBH small group instructors?

2. To what extent does classroom team composition of System-Based Healthcare course nursing and medical students moderate results on the assessment measures? 3. Do medical students trained in IP groups differ from those trained in medical-student-only groups on the OSCE stations? 4. What are the strengths and major threats to validity related to the study design?

Assessing Interprofessional Competencies

IPEC Core Competencies

Assessment Tools Values/ Ethics

Roles/ Responsibilities

Interprofessional Communication

Teams/ Teamwork

OSCE stations X X X

Direct Observation X X

Knowledge Test X X X

Video Case Exam X X X

Implicit Association Test X X

Multi-Source Evaluation X X

54

Questions?

Save the Date

Assessment of IPE to Move Beyond Attitudes: Featured tools and case studies from

the field

Presented by Amy Blue, PhD,

Vernon Curran, Phd and

Sara Shrader, PharmD, FCCP, BCPS, CDE

Thursday, February 12

10:00 – 11:00 A.M. Central Time

Thank you!

Please visit us at www.aihc-us.org for more information on future AIHC Webinars.