using examsoft data to prepare for and ease the accreditation process
TRANSCRIPT
1
Use of ExamSoft Data to Prepare For and Ease the
Accreditation ProcessKimberly K. Daugherty, PharmD, BCPS
Ashley Castleberry, PharmD, MAEdLeah Simpson, EdD, MPA
Polling Question: Who is in the audience?• Please select the category that best fits you:
1. Assistant/Associate Dean2. Director of Assessment3. Chair of Curriculum and/or Assessment Committee4. Faculty member5. Staff Member
Session Learning Outcomes• Discuss how the autonomy supportive feedback system can be used
by students and faculty to enhance student learning and metacognitive development
• Demonstrate how longitudinal assessment data run from ExamSoft can be used to provide students with assessment report cards showing their progress across a curriculum
• Model how ExamSoft longitudinal data can be aggregated for use in accreditation reports
Autonomy and ExamSoft• Self-Determination Theory (SDT) states that all humans have three
basic psychological needs: autonomy, competence, and relatedness (Deci & Ryan, 1985, 1994, 2000)
• Autonomy in learning has been linked to greater student satisfaction, persistence, and deeper learning (Seiver & Troja, 2014; Wielenga-Meijer, Taris, Wigboldus & Kompier, 2012)
Autonomy-Support• By meeting students’ intrinsic desires, autonomy-support has been
shown to increase student satisfaction, persistence, creativity, and curiosity (Patrick & Williams, 2009; Seiver & Troja, 2014)
• Perception of autonomy-support leads to greater school achievement (Mih & Mih, 2013)
• Feedback can be autonomy-supportive when it Feeds Up, Feeds Back, and Feeds Forward (Hattie & Timperley, 2007)
University of KentuckyStudent End of Exam Report Cards
Feed Up, Feed Back, and Feed Forward with ExamSoft Reports
Students LOVE these reports!
Does your institution use end of exam reports?
1. Yes2. No
University of Arkansas for Medical Sciences
Longitudinal Reports-Students-Course
-Curriculum
Reports > Longitudinal Analysis
Student and Course Reports
Student and Course Reports
64.17%
Student and Course Reports
How Longitudinal Reports Fit Autonomy Support Model• Feed Up
• Provide students with data needed to formulate goals in specified areas
• Feed Back• Provides students with data on achievement of their goals
• Feed Forward• Students meet with faculty to discuss data and make plans for improvement
How Longitudinal Reports Fit Autonomy Support Model• Programmatic Assessment
• Data from courses provide aggregate data on course effectiveness• Data from the curriculum provide aggregate data on curriculum effectiveness
Curriculum ReportsBloom’s Taxonomy
Curriculum ReportsDidactic Topics
Does your institution use longitudinal reporting at the course level?
1. Yes2. No
Sullivan UniversityComprehensive Student Report Cards
SUCOP Comprehensive Student Report Card• Background on Sullivan University College of Pharmacy
• 3-year year around program• Quarter system
• Comprehensive Student Report Cards• Reason for implementation: to provide students and their advisors
information on overall GPA, progression on pharmacy knowledge and skills, and progression on program outcome attainment
• Data on report cards is aggregated from multiple sources• E-value-experiential software• ExamSoft-didactic testing software• Campus Vue-Student Information System• CHARM: OSCE data
Student Report Card • Required didactic elements: data is rolled
up from exam question results tagged in ExamSoft
• Program outcome elements: data rolled up from exam question results tagged in ExamSoft
• Report cards are compiled after the Fall and Spring quarters are completed for the first and second professional year classes
Required Didactic Elements
Assessment BS PS SABS CS
ExamSoft Student Average ** 63.92 77.73 77.55 76.41
Number of Questions 888 1748 674 1291
Class Average Class SD 75.29 12.07 86.27 11.43 85.69 17.34 84.98 13.86
PCOA Aug 2015 Student Averag 248 277 263 246
SUCOP Avg-National Avg 328-350 324-334 327-339 318-326
Program Outcomes
Community IPPE*** 3 3.00 3.00 3 NA 3.00 3.00
Hospital IPPE*** 3 3 3 NA 3 3 NA
ExamSoft Student Average** 75.87 78.52 71.4 82.67 76.37 81.63 67.65
# of Questions 3063 933 222 70 125 686 34 Class Average 85.46 83.66 85.55 84.39 82.92 89.06 84.88 Class SD 6.41 9.06 9.41 12.69 8.61 11.6 14.96
Assessment Areas 1.1 2.1 2.2 2.3 2.4 3.1 3.2
How Report Cards Fit Autonomy Support Model• Feed Up
• Provide students with data needed to be able to formulate goals in each of the specified areas
• Feed Back• Provides students with data on achievement of their goals
• Feed Forward• Students meet with their advisors to discuss the data and make plans for
improvement
• Programmatic assessment• Data from the individual report cards is “rolled-up” to provide aggregate data on
element attainment
How data can be combined with other data to tell accreditation story PY1 Didactic PY1 and PY2 Didactic
EO # Questions Mean SD # Questions Mean SD 1.1 1820 84.5% 0.1 4896 85.9% 0.1 2.1 300 78.5% 0.1 809 86.7% 0.1 2.2 270 86.5% 0.1 270 86.6% 0.1 2.3 46 77.9% 0.2 98 85.8% 0.1 2.4 14 78.2% 0.1 212 85.7% 0.1 3.1 293 90.2% 0.1 871 84.3% 0.1 3.2 34 93.0% 0.1 62 92.7% 0.1 3.3 90 85.1% 0.2 97 81.4% 0.1 3.4 8 93.7% 0.1 9 89.6% 0.1 3.5 3 77.0% 0.2 7 90.2% 0.1 3.6 149 87.0% 0.1 176 90.4% 0.1 4.1 25 87.2% 0.1 25 92.7% 0.1 4.2 5 76.1% 0.2 6 87.9% 0.1 4.3 NA NA NA NA NA NA 4.4 79 85.9% 0.1 577 87.8% 0.1
JCPP-PPCP % of Exam Mean SD % of Exam Mean SD
C NA NA NA 2.6% 36.0% 0.2 H NA NA NA 55.7% 80.1% 0.1 A NA NA NA 17% 93.8% 0.1 R NA NA NA 19% 80.3% 0.2 M NA NA NA 5% 61.5% 0.2
Blocks 1 and 2* Blocks 3 and 4** EO Mean SD Mean SD 1.1 3.20 0.58 3.63 1.6 2.1 3.07 0.73 4.04 1.19 2.2 2.91 1.32 3.84 1.54 2.3 NA NA 3.21 2.04 2.4 2.64 1.5 3.41 1.82 3.1 3.35 0.51 4.18 0.86 3.2 2.98 0.8 3.64 1.6 3.3 NA NA 2.84 2.11 3.4 3.72 0.47 4.45 0.73 3.5 NA NA 3.69 1.62 3.6 2.97 0.73 3.98 1.29 4.1 3.61 0.61 4.37 0.73 4.2 3.29 0.58 4.01 1.32 4.3 NA NA 3.12 1.93 4.4 3.40 0.52 4.53 0.66
JCPP-PCPP Mean SD Mean SD
C NA NA 4.12 1.21 H NA NA 4.11 1.17 A NA NA 4.02 1.14 R NA NA 4.02 1.17 M NA NA 3.92 1.25
Blocks 1 and 2: 4-point Likert scale (0=NA-4=Exceed); Blocks 3 and 4: 5-point Likert scale (0=NA-1=Beyond entry level)
Does your institution use longitudinal reporting at the program level?
1. Yes2. No
QUESTIONS