programmatic actions upon assessment results · programmatic actions upon assessment results: a...
Post on 06-Jan-2020
6 Views
Preview:
TRANSCRIPT
Programmatic actions upon assessment results
A case study of breadth and depth in higher education
Yao HillAssessment Office
Hill, Y. Z. (2016, September). Programmatic actions upon assessment results: A case study of breadth and depth in higher education [PowerPoint slides]. Paper presented at the Hawaii Educational Research Association Annual Conference, Honolulu, HI.
Program Assessment Definition
An on-going process designed to monitor and improve student learning. Faculty: a) develop explicit statements of what students should learn (i.e., student learning outcomes); b) verify that the program is designed to foster this learning (alignment); c) collect data/evidence that indicate student attainment (assessment results); d) use these data to improve student learning(close the loop). (Allen, M., 2008)
Learning Outcomes
LearningOpportuni-
ties
Collect & Evaluate Evidence
Interpret Results
Use Results
Assessment cycle
238degree
programs need to demonstrate using assessment for improvement
Program has increasingly reported assessment use
51%
70%
2011 (N=230) 2012 (N=233) 2013 (N=232) 2014 (N=238) 2015 (N=238)
How and how better use assessment process and results?
Research Foci
Breadth
DepthFactors
Communicate for more actions
Data: 2015 Annual Program Assessment Reports
Sample: 151 reports from 164 programs that claimed use of results
Method:Thematic Analysis + Expert JudgementMade sure our coding is trustworthy: 79% interrater agreement on 28 randomly selected programs
Made sure excellent cases were truly excellent: 87% intra-rater agreement or major categories agreement on 47 programs
Strategic Thematic Coding to Promote Use
Stability
Ready to useSpecificity
Used existing major categories
ACPRS
Assessment tools and procedures
Course changes
Program curriculum and policy
Resources and personnel
Student out-of-course experiences
Two other often neglected options
Celebration
No change needed
CN
37%
Making changes to assessment tools & processes is the most frequently reported use
Course changes
Student out-of-course experience
Program curriculum and policy
No change needed
Celebration
Resources/Personnel
Assessment processes and proceduresA
Learning Outcomes
13%
Curriculum Map8%
Collect & Evaluate Evidence
20%
Interpret Results
Use Results
Most changes to assessment are tools and processes when collecting &evaluating evidence
Learning Outcomes
13%
Curriculum Map8%
Collect & Evaluate Evidence
20%
Interpret Results
Use ResultsEvidence/Instrument
Collection Procedure
Rubric
Evaluation Procedure
Standards/Benchmark
22%
37%
Course and program curriculum policy changes are often related & represent common programmatic actions.
48% changed course or program
Course changes
Student out-of-course experience
Program curriculum and policy
No change needed
Celebration
Resources/Personnel
Assessment processes and procedures
C
P
Changes in assignment are in line with recommendations from higher ed leaders
2%
2%
2%
8%
8%
10%
15%
Course combination/deletion
Course title & description
General education designation
Classroom activities/materials
New Course(s)
Assignment (Signature assignment 3%)
Content coverage/general
Small manageable changes can make a meaningful difference.
2%
2%
2%
8%
8%
10%
15%
Course combination/deletion
Course title & description
General education designation
Classroom activities/materials
New Course(s)
Assignment (Signature assignment 3%)
Content coverage/general
Programs identified need to define clear learning pathway
1%2%2%
2%3%3%
3%4%
5%
Increased course repeat limitsInstitutioned course-taking sequenceIncreased course selection flexibility
Examined curriculum coherenceServed professional accreditationCreated/Restructured curriculumChanged graduation requirement
Increased sections/frequencyAdded pre-requisite(s)/co-requisite(s)
Programs seriously examined alignment between curriculum and outcomes
1%2%2%
2%3%3%
3%4%
5%
Increased course repeat limitsInstitutioned course-taking sequenceIncreased course selection flexibility
Examined curriculum coherenceServed professional accreditationCreated/Restructured curriculumChanged graduation requirement
Increased sections/frequencyAdded pre-requisite(s)/co-requisite(s)
Programs increased opportunities for timely graduation
1%2%2%
2%3%3%
3%4%
5%
Increased course repeat limitsInstitutioned course-taking sequenceIncreased course selection flexibility
Examined curriculum coherenceServed professional accreditationCreated/Restructured curriculumChanged graduation requirement
Increased sections/frequencyAdded pre-requisite(s)/co-requisite(s)
23%
Meaningful co-curricular activities enhance learning in both undergraduate and graduate students
Course changes
Student out-of-course experience
Program curriculum and policy
No change needed
Celebration
Resources/Personnel
Assessment processes and procedures
S
There are multiple ways to support students outside of the classroom.
3%
6%
8%
9%
Financial/Career support
Resource materials (handbooks, website)
Cocurricular activities (workshops)
Academic advising/mentoring
14%
At a time when resource is scare, assessment results provide persuasive arguments in requesting and getting it.
Course changes
Student out-of-course experience
Program curriculum and policy
No change needed
Celebration
Resources/Personnel
Assessment processes and procedures
R
Consider revising the job description, too.
1%
1%
2%
2%
2%
3%
7%
Acquired a new lab/facility
Reassigned teaching duty
Acquired new material/technology
Increased faculty collaboration
Provided professional development
Acquired student assistant
Acquired new faculty/faculty time
When faculty share, they are likely to learn from each other and improve.
1%
1%
2%
2%
2%
3%
7%
Acquired a new lab/facility
Reassigned teaching duty
Acquired new material/technology
Increased faculty collaboration
Provided professional development
Acquired student assistant
Acquired new faculty/faculty time
Breadth Depth
Excellent, 47
Good, 39Trying, 36
Minimum, 42 20% (47 out of 238) of the UH Mānoaprograms demonstrated excellent use of results
Excellent Use• Actions are clearly aligned
with results.• Careful deliberation of
reasons behind actions.
Good Use• Actions are meaningful and
reasonable.• Some clues that actions are
based on assessment results.
Trying to Use• Some programmatic actions
with no or very weak alignment with results
Minimal Use• Results indicate no actions
needed
Active and full faculty participation is closely associated with good or excellent use
79%
62%
17%10%
Excellent Good Trying Minimum
Next task: Communicate & Advocate
A C P R S
C N
Deliver use packages
- Statistical skills- Research methods- Written communication
Give people options for low-hanging fruit
A
CR
S
Develop a rubric 1.5 hr
Faculty share OC assignment 1.5 hr
Share & assignment instructions 1.5 hr
Advise students to take 2 method courses
Show case excellent programs
Fall 16 Workshops
Univ. Recognition
Implications for the field of evaluation?
Do we know how clients have the used the results?
Implications for the field of evaluation?
Do we know how our clients have the used the results?
Implications for the field of evaluation?
Have we used that information to help other clients?
Implications for the field of evaluation?
How can we show case our clients to build community of practice?
Image Source
• http://cdn2.bigcommerce.com/server2400/vxsr4nt/products/53974/images/24986/red_silver_gray_blue_hot_pink_corrugated_cardboard_shipping_mailers_boxes__23557.1402412707.1280.1280.jpg?c=2
top related