Programmatic actions upon assessment results

A case study of breadth and depth in higher education

Yao HillAssessment Office

Hill, Y. Z. (2016, September). Programmatic actions upon assessment results: A case study of breadth and depth in higher education [PowerPoint slides]. Paper presented at the Hawaii Educational Research Association Annual Conference, Honolulu, HI.

Program Assessment Definition

An on-going process designed to monitor and improve student learning. Faculty: a) develop explicit statements of what students should learn (i.e., student learning outcomes); b) verify that the program is designed to foster this learning (alignment); c) collect data/evidence that indicate student attainment (assessment results); d) use these data to improve student learning(close the loop). (Allen, M., 2008)

Presenter

Presentation Notes

Assessment office was established in 2008 based on faculty senate recommendation. It reports to both faculty senate and AVCAA. There are two faculty specialists currently in the office to carry out the mission of the office.

Learning Outcomes

LearningOpportuni-

ties

Collect & Evaluate Evidence

Interpret Results

Use Results

Assessment cycle

238degree

programs need to demonstrate using assessment for improvement

Program has increasingly reported assessment use

51%

70%

2011 (N=230) 2012 (N=233) 2013 (N=232) 2014 (N=238) 2015 (N=238)

How and how better use assessment process and results?

Research Foci

Breadth

DepthFactors

Communicate for more actions

Data: 2015 Annual Program Assessment Reports

Sample: 151 reports from 164 programs that claimed use of results

Method:Thematic Analysis + Expert JudgementMade sure our coding is trustworthy: 79% interrater agreement on 28 randomly selected programs

Made sure excellent cases were truly excellent: 87% intra-rater agreement or major categories agreement on 47 programs

Strategic Thematic Coding to Promote Use

Stability

Ready to useSpecificity

Used existing major categories

ACPRS

Assessment tools and procedures

Course changes

Program curriculum and policy

Resources and personnel

Student out-of-course experiences

Two other often neglected options

Celebration

No change needed

CN

37%

Making changes to assessment tools & processes is the most frequently reported use

Course changes

Student out-of-course experience

Program curriculum and policy

No change needed

Celebration

Resources/Personnel

Assessment processes and proceduresA

Learning Outcomes

13%

Curriculum Map8%

Collect & Evaluate Evidence

20%

Interpret Results

Use Results

Most changes to assessment are tools and processes when collecting &evaluating evidence

Learning Outcomes

13%

Curriculum Map8%

Collect & Evaluate Evidence

20%

Interpret Results

Use ResultsEvidence/Instrument

Collection Procedure

Rubric

Evaluation Procedure

Standards/Benchmark

22%

37%

Course and program curriculum policy changes are often related & represent common programmatic actions.

48% changed course or program

Course changes

Student out-of-course experience

Program curriculum and policy

No change needed

Celebration

Resources/Personnel

Assessment processes and procedures

C

P

Changes in assignment are in line with recommendations from higher ed leaders

2%

2%

2%

8%

8%

10%

15%

Course combination/deletion

Course title & description

General education designation

Classroom activities/materials

New Course(s)

Assignment (Signature assignment 3%)

Content coverage/general

Small manageable changes can make a meaningful difference.

2%

2%

2%

8%

8%

10%

15%

Course combination/deletion

Course title & description

General education designation

Classroom activities/materials

New Course(s)

Assignment (Signature assignment 3%)

Content coverage/general

Programs identified need to define clear learning pathway

1%2%2%

2%3%3%

3%4%

5%

Increased course repeat limitsInstitutioned course-taking sequenceIncreased course selection flexibility

Examined curriculum coherenceServed professional accreditationCreated/Restructured curriculumChanged graduation requirement

Increased sections/frequencyAdded pre-requisite(s)/co-requisite(s)

Programs seriously examined alignment between curriculum and outcomes

1%2%2%

2%3%3%

3%4%

5%

Increased course repeat limitsInstitutioned course-taking sequenceIncreased course selection flexibility

Examined curriculum coherenceServed professional accreditationCreated/Restructured curriculumChanged graduation requirement

Increased sections/frequencyAdded pre-requisite(s)/co-requisite(s)

Programs increased opportunities for timely graduation

1%2%2%

2%3%3%

3%4%

5%

Increased course repeat limitsInstitutioned course-taking sequenceIncreased course selection flexibility

Examined curriculum coherenceServed professional accreditationCreated/Restructured curriculumChanged graduation requirement

Increased sections/frequencyAdded pre-requisite(s)/co-requisite(s)

23%

Meaningful co-curricular activities enhance learning in both undergraduate and graduate students

Course changes

Student out-of-course experience

Program curriculum and policy

No change needed

Celebration

Resources/Personnel

Assessment processes and procedures

S

There are multiple ways to support students outside of the classroom.

3%

6%

8%

9%

Financial/Career support

Resource materials (handbooks, website)

Cocurricular activities (workshops)

Academic advising/mentoring

14%

At a time when resource is scare, assessment results provide persuasive arguments in requesting and getting it.

Course changes

Student out-of-course experience

Program curriculum and policy

No change needed

Celebration

Resources/Personnel

Assessment processes and procedures

R

Consider revising the job description, too.

1%

1%

2%

2%

2%

3%

7%

Acquired a new lab/facility

Reassigned teaching duty

Acquired new material/technology

Increased faculty collaboration

Provided professional development

Acquired student assistant

Acquired new faculty/faculty time

When faculty share, they are likely to learn from each other and improve.

1%

1%

2%

2%

2%

3%

7%

Acquired a new lab/facility

Reassigned teaching duty

Acquired new material/technology

Increased faculty collaboration

Provided professional development

Acquired student assistant

Acquired new faculty/faculty time

Breadth Depth

Excellent, 47

Good, 39Trying, 36

Minimum, 42 20% (47 out of 238) of the UH Mānoaprograms demonstrated excellent use of results

Excellent Use• Actions are clearly aligned

with results.• Careful deliberation of

reasons behind actions.

Good Use• Actions are meaningful and

reasonable.• Some clues that actions are

based on assessment results.

Trying to Use• Some programmatic actions

with no or very weak alignment with results

Minimal Use• Results indicate no actions

needed

Active and full faculty participation is closely associated with good or excellent use

79%

62%

17%10%

Excellent Good Trying Minimum

Next task: Communicate & Advocate

A C P R S

C N

Deliver use packages

- Statistical skills- Research methods- Written communication

Give people options for low-hanging fruit

A

CR

S

Develop a rubric 1.5 hr

Faculty share OC assignment 1.5 hr

Share & assignment instructions 1.5 hr

Advise students to take 2 method courses

Show case excellent programs

Fall 16 Workshops

Univ. Recognition

Implications for the field of evaluation?

Do we know how clients have the used the results?

Implications for the field of evaluation?

Do we know how our clients have the used the results?

Implications for the field of evaluation?

Have we used that information to help other clients?

Implications for the field of evaluation?

How can we show case our clients to build community of practice?

Image Source

• http://cdn2.bigcommerce.com/server2400/vxsr4nt/products/53974/images/24986/red_silver_gray_blue_hot_pink_corrugated_cardboard_shipping_mailers_boxes__23557.1402412707.1280.1280.jpg?c=2