pharmacy curriculum outcomes assessment (pcoa)

40
Pharmacy Curriculum Outcomes Assessment (PCOA) Moderated by: Abby A. Kahaleh, BPharm, MS, PhD, MPH February 16, 2016

Upload: irene-collins

Post on 18-Jan-2018

230 views

Category:

Documents


0 download

DESCRIPTION

Contact Information Abby A. Kahaleh, BPharm, MS, PhD, MPH Curriculum SIG Chair 847-330-4537 (Phone) [email protected]

TRANSCRIPT

Page 1: Pharmacy Curriculum Outcomes Assessment (PCOA)

Pharmacy Curriculum Outcomes Assessment (PCOA)

Moderated by:Abby A. Kahaleh, BPharm, MS, PhD, MPHFebruary 16, 2016

Page 2: Pharmacy Curriculum Outcomes Assessment (PCOA)

Contact Information

Abby A. Kahaleh, BPharm, MS, PhD, MPH

Curriculum SIG Chair

847-330-4537 (Phone)

[email protected]

Page 3: Pharmacy Curriculum Outcomes Assessment (PCOA)

Presenters

1. Elizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of AssessmentUniversity of Houston College of Pharmacy

2. Tara Jenkins, RPh, PhD Associate Dean of Academic AffairsUniversity of Houston College of Pharmacy

3. Justine Gortney, Pharm.D., BCPSClinical Assistant ProfessorWayne State University College of Pharmacy

Page 4: Pharmacy Curriculum Outcomes Assessment (PCOA)

Program Description According to NABP, the Pharmacy Curriculum Outcomes

Assessment (PCOA) is an effective assessment tool for colleges and schools of pharmacy to use for evaluating student performance in the curricula. PCOA facilitates a review of a pharmacy student’s performance from year to year in relation to the entire curriculum and compares the results to national data. The purpose of this program is to provide participants with expertise from administrators and

faculty who used PCOA in their institutions.  

Page 5: Pharmacy Curriculum Outcomes Assessment (PCOA)

Objectives

Discuss the benefits of using PCOA as a non-high stakes APPE-readiness exam

Discuss the benefits of using PCOA as a high-stakes APPE-readiness exam

Compare and contrast different techniques of setting minimal competency when using PCOA

Design a remediation plan for students who do not meet minimal competency on PCOA

Page 6: Pharmacy Curriculum Outcomes Assessment (PCOA)

Presenter IJustine S. Gortney, Pharm.D., BCPSDirector of Assessment, Division of PharmacyClinical Assistant ProfessorWayne State University,Eugene Applebaum College of Pharmacy and Health Sciences

Page 7: Pharmacy Curriculum Outcomes Assessment (PCOA)

Pharmacy Institution’s Stakes Attached to Student Performance

N=38Gortney JS, Bray B, Salinitri F. Am J Pharm Ed 2015; 79(9) Article 137.

5%8%

26%

61%

High -impacts pro-gressionMedium-course gradeLow-development planNone-no negative impact

Page 8: Pharmacy Curriculum Outcomes Assessment (PCOA)

Where We House PCOA: Professionalism Curriculum

Event P1 Year P2 Year P3 Year P4 Year

Professional Community

Service

Student Curriculum Student Student

Professional Meeting

Student Curriculum Student Student

Professional Event

Student Curriculum Student Student

Summative Evaluation

---- PCOA PCOA ----

Page 9: Pharmacy Curriculum Outcomes Assessment (PCOA)

Wayne State Operational Timeline

Education about Process

PCOA Thursday &

Friday

Results Receipt and Distribution

Students- August,

November, December Mid

January Distributed to Students and Faculty in April

Remediation (Student

Development)

Evaluated by SEC March-April

Advisor meetings May; Plans due in late May/Early June

Rollout: P2s 2013, P3s 2014

Page 10: Pharmacy Curriculum Outcomes Assessment (PCOA)

With Low Stakes, How Do We Incentivize Students? Social Element- Breakfast the day of exam High Performance on PCOA - Rewards

– Recognition at graduation for top two– Letter of recognition from the Dean’s Office for

top 10%– Overall examination scores above the college

mean will be rewarded with 1% bonus on your final PPS grade

Poor performance: Student development plan

Page 11: Pharmacy Curriculum Outcomes Assessment (PCOA)

WSU Results Distribution PlansStudent Class Meetings Faculty Meeting Rollout

• Review overall class results• Review how to interpret PCOA

scorecard• Distribute individual student

scorecards• Distribute individual student

development plans• P3 only: Review performance

change from P2 and reflection

• Provide comparison of the following:

• Overall school results• Comparison with national• Comparison within class and

across classes• Review timelines of student

development plans

Publish results on dashboard: http://www.cphs.wayne.edu/pharmd/strategic_assessment_2015/pcoa_data_2015_webupdate.pdf

Page 12: Pharmacy Curriculum Outcomes Assessment (PCOA)

Example of WSU P2 to P3 scorecard

Name P2 P3

Change in Score P2 to

P3

Assess the magnitude of the change in

PCOA

Confidence Rank

Overall_SS 343 369 26  -Area_1_SS 377 418 41  -Area_2_SS 338 357 19  -Area_3_SS 348 329 -19  -Area_4_SS 330 380 50  -

Overall Percentile 58 58 0  -Area 1 Percentile 66 85 19  -Area 2 Percentile 54 52 -2  -Area 3 Percentile 56 33 -23  -

Area 4 Percentile 54 53 -1 Confidence

RankPhysiology 70 70 0 7

Biochemistry 44 38 -6 6

Page 13: Pharmacy Curriculum Outcomes Assessment (PCOA)

Past: Student Development Plans / ThresholdsYear 2013 P2a 2014 P2a 2015 P2a 2014 P3b

(2013 P2)2015 P3 b

(2014 P2)Basic Bio A

B- - E

FG

L

Pharm Sci AB

I PQ

H NO

SBAS ABCD

JKLM

RS

AB

I

Clin Sci - - TUV

-

aTwo SD below WSU meanbSame as P2 2013 except Clin Sci (current P3 2 SD below P3 mean)

Page 14: Pharmacy Curriculum Outcomes Assessment (PCOA)

WSU Student Development Plan 2013-2015 (Student Driven)

Self assessment of the content area– How have you studied this content in the past?– What are your current study habits?– What do you perceive as a limitation to studying or

understanding the specific content at hand? Plan for improvement

– How will you proceed forward to improve your knowledge and skills in these areas?

• Generate a personal action plan that is practical and implementable

• Describe the timeframe you propose to do this over– What resources will you use to do this?

Self-reassessment of the content area after plan for improvement implemented

– What specific methods will you use to assess your progress? (eg. online quizzes in Access Pharmacy)

Page 15: Pharmacy Curriculum Outcomes Assessment (PCOA)

WSU Student Development Plans Modified for 2016 Cut-points modified

– Agreed 1 SD below 3yr NABP composite based on curriculum map for P2s & overall score P3s

• P2s for Basic Biomed (282) and PharmSci (280)

• P3s Overall Score (303) Revised Student Development Plan

– Development/rollout being finalized– Faculty and content-structured – Timed during 6 weeks of first APPE block

Page 16: Pharmacy Curriculum Outcomes Assessment (PCOA)

How Have We Used This Data in WSU Curricular Assessment

Page 17: Pharmacy Curriculum Outcomes Assessment (PCOA)

Limitations We Have to Date Using as low stakes; student motivation Most of our data analysis to this point has been in P2s

PCOA with P1 GPA: r=0.50, p<0.01 (N=143) PCOA with P2 GPA: r=0.53, p<0.01 (N=142) Early curriculum, science-based classes showed low-

moderate but significant correlations with PCOA scores Lacking NAPLEX correlation data due to date of exam

Does not appear on surface to relate to risk for APPE rotation failure; APPE grading scale lacking variability

Page 18: Pharmacy Curriculum Outcomes Assessment (PCOA)

At WSU, How Do We Feel this Has Benefited our Students and Program

Provided a starting block for summative evaluations Provided positive feedback for students regarding their

knowledge Provided specific assessment data to track our students’

knowledge and benchmark against others– Used specifically during curricular renewal process

• Part of a triangulation of data driving “renewal”• Plan on using as a consistency check after “renewal”

– ACPE self-study data– Higher Learning Commission (HLC) outcomes

measurement

Page 19: Pharmacy Curriculum Outcomes Assessment (PCOA)

Presenters II and IIIElizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of Assessment University of Houston College of Pharmacy Tara Jenkins, RPh, PhD Associate Dean of Academic Affairs University of Houston College of Pharmacy

Page 20: Pharmacy Curriculum Outcomes Assessment (PCOA)

Utilizing the PCOA as a High Stakes Exam

Page 21: Pharmacy Curriculum Outcomes Assessment (PCOA)

Objectives UHCOP High-Stakes Exam History PCOA Pilot Setting Cut-off Scores Remediation Questions

Page 22: Pharmacy Curriculum Outcomes Assessment (PCOA)

PCOA Project Team Elizabeth A. Coyle, Pharm.D., FCCM, BCPS

Catherine L. Hatfield, Pharm.D.

Gerida Brown

Tara L. Jenkins, RPh, Ph.D.

UHCOP Faculty

Page 23: Pharmacy Curriculum Outcomes Assessment (PCOA)

Milemarker Homegrown cumulative didactic exam Started as high stakes for P3’s in 2000

– Included questions from all 3 years of didactic curriculum

– Written by faculty– 200 case-based questions– Questions Angoffed – Mean of Angoff score from 200

questions had to be met for minimal competency

Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.

Page 24: Pharmacy Curriculum Outcomes Assessment (PCOA)

Milemarker P3’s had to meet minimum competency

on Milemarker III to advance to APPEs– Average pass rate from 2000-2012 was

99% P1 & P2’s also took Milemarkers I & II

– Formative with remediation and points towards Milemarker III

New cases & validated questions were hard to accumulate

Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.

Page 25: Pharmacy Curriculum Outcomes Assessment (PCOA)

PCOA Pilot

January of 2013 P3 students

– All required to take– Formative information only

P3 students still had to take Milemarker III in April 2013

Page 26: Pharmacy Curriculum Outcomes Assessment (PCOA)

Exam Comparisons

MILEMARKER III PCOA Overall Questions: 200 Questions 220 (20 “test” questions) Biomedical Sciences: 10.5% (21 questions) 16% (32 questions) Clinical Sciences: 61% (122 questions) 32% (64 questions) Pharmaceutical Sciences: 13.5% (27 questions) 30% (60 questions) Social/Behavior/Admin: 15% (30 questions) 22% (44 questions)

Page 27: Pharmacy Curriculum Outcomes Assessment (PCOA)

Results Comparison

Overall Average (% Correct): 78.9% UH 73%/ National 67% Biomedical Sciences (% Correct): 82.9% UH 73%/National 66% Clinical Sciences (%Correct): 80.4% UH 76%/National 73% Pharmaceutical Sciences (%Correct): 68.9% UH 67%/ National 57% Social/Behavior/Admin (% Correct): 76.5% UH 77%/ National 71%

Milemarker III PCOA

Page 28: Pharmacy Curriculum Outcomes Assessment (PCOA)

PCOA Benefits External measurement and benchmark of

our students’ performance in pharmacy curricula

Psychometrically sound questions written by content experts from colleges of pharmacy around the country

Ability to trend results longitudinally

Does not require faculty time to analyze and write capstone questions

Page 29: Pharmacy Curriculum Outcomes Assessment (PCOA)

Identify Cut off Score For P3 PCOA

Page 30: Pharmacy Curriculum Outcomes Assessment (PCOA)

Which Scores to Use? Percentile rank — the percentage of scores in a

distribution that is equal to or lower than the measured score.

Raw score — Total number of raw points a test taker receives based on the number of questions answered correctly; typically, for example, 20 correct answers means a raw score of 20

Scaled score — Scores that have been mathematically transformed from one set of numbers (i.e., the raw score) to another set of numbers in order to make them comparable in some way — for example, across different editions, or “forms,” of the same test

X Tuan and R Michel. R & D Connections No 16. Sept. 2011

Page 31: Pharmacy Curriculum Outcomes Assessment (PCOA)

National Scaled Scores700

361

Nat

iona

l Sca

led

Scor

e (n

orm

ed re

fere

nce

sam

ple)

357 (scaled normed referenced score change)

0Year

1 2 73 4 5 6 8

50%ile 50%ile50%ile 50%ile

50%ile 50%ile

50%ile

50%ile

Page 32: Pharmacy Curriculum Outcomes Assessment (PCOA)

UHCOP Cutoff Score* NSS vs. 10% below NSS vs. 1 or 2 STD

* PCOA Cutoff score for P3’s is 1 STD from the NSS scale score. The STD will be calculated from the PCOA student scores each year.

Page 33: Pharmacy Curriculum Outcomes Assessment (PCOA)

3 Year Comparisons

P3 Class (N)

Class Percentile

National Scaled Score

Class Ave Scaled Score

Standard Deviation

Passing Score

Number of Remediating

after 1st PCOA

Number (%) < NSS

Number ≥ 90th percentile

Number ≤ 20th percentile

P3 Class 2013 (N=109)

75th 361 405 41 N/A (pilot)

N/A (pilot) 14 (12.8%) 28 1

P3 Class 2014 (N=113)

56th 361 388 41 325 14* 45 (39.8%) 10 11

P3 Class 2015 (N=109)

65th 357 388 47 310 2 23 (21.1%) 22 3

* If we would have used 1 STD cutoff, 11/113 would have remediated

Page 34: Pharmacy Curriculum Outcomes Assessment (PCOA)

High Stakes Exams & Academic Performance

Coyle EA, Hatfield CL, Cottreau JM, Brown G, Smesny A. Poster (July 2013) AACP Annual Meeting, Chicago, IL

Page 35: Pharmacy Curriculum Outcomes Assessment (PCOA)

Remediation What kind of remediation do you use?

– Retest PCOA?– Homegrown exam?– Tailored remediation?

How do remediated students perform on rotations?

How often do you remediate?

Page 36: Pharmacy Curriculum Outcomes Assessment (PCOA)

• Meet minimum competency go on to APPE in May

PCOA Jan/Feb ALL P3's

• Meet minimum competency go on to APPE in May.

• Do not meet minimum competency, receive I in IPPE II.

• Follow a remediation plan/take a remediation course once classes end in May.

Remediation 1 March/April PCOA

• Meet minimum competency start APPE's in August .

• Do not meet minimum competency, follow remediation plan/take remediation course.

Remediation 2July PCOA*

Remediation 3Sep/Oct PCOA

PCOA Remediation Policy for P3’s

• Meet minimum competency start APPE's in January.

• Do not meet minimum competency, follow remediation plan/take remediation course.

After 3rd remediation repeat until student meets 6 year program limit. After 5th remediation I in IPPE II becomes a U.

* Not in PCOA normal testing windows

Page 37: Pharmacy Curriculum Outcomes Assessment (PCOA)

Conclusions Adapting PCOA as a high stakes

assessment was an easy transition at UHCOP

Minimum competency should be set off of the scaled score and not percentile or raw score

A clear remediation process is imperative to close the loop for high stakes exams

Page 38: Pharmacy Curriculum Outcomes Assessment (PCOA)

Q&A

Page 39: Pharmacy Curriculum Outcomes Assessment (PCOA)

Discussion Questions

1. Compare and contrast the PCOA exam with other assessment instruments 2. What are the advantages/disadvantages of various remediation plans? 3. Describe examples of successful student/faculty development plans4. What are the steps for closing the assessment feedback loop based on the PCOA results?

Page 40: Pharmacy Curriculum Outcomes Assessment (PCOA)

Thank You!