What’s Going On? Five Year Reviews and Assessment: Sharing Experiences so Far

Ruth Guzley, Mike Ward and Bill Loker, AURA

Five Year Program Reviews:The Broader Context …

Were routine, episodic efforts of little utility, much futility

What do we want to know about our programs? What are we trying to accomplish? How well are we meeting our goals? Academic success, good use of

resources

Assessment: Doing the Right Thing for the Right Reasons

The Intentional Curriculum: Do individual courses add up to a program?

The Reflective Department: Are we working together to provide a quality educational experience for our students?

The Culture of Evidence: How do we know the answers to the above questions?

Assessment can provide these answers …

The Assessment Juggernaut

WASC reaccredidation … educational effectiveness reviews …

http://www.wascsenior.org/wasc/Doc_Lib/2001%20Handbook.pdf

Cornerstones … CSU system-wide accountability effort focused on quality of Baccalaureate programs …

http://www.csuchico.edu/prs/AccountabilityReport_2002.doc

Educational effectiveness and other stakeholders … students, parents, legislators, etc

Aligning Five Year Review with Assessment Demands …

New Five Year Review reflects assessment concerns

Transforms FYR from an episodic time sink into a cumulative exercise in evidence gathering to improve educational outcomes and satisfy stakeholder accountability

demands Assessment only one component of

FYR

From the FYR template …

Articulating a Collective Vision: Mission and Program goalsCriteria for Review

1.1 The program has a clear and published mission statement that is appropriate for higher education and consonant with the mission and strategic priorities of the university.

1.3 The program periodically reviews and revises the mission statement as appropriate. The review process involves appropriate stakeholders.

Program Performance (Hint: Insert mission statement here.)

From the FYR template … (cont’d)

Organizing For Learning: Curriculum

Criteria for Review

2.2 The program has specified its expected learning outcomes and they have been widely shared among its members, including faculty, students, staff, and … external stakeholders.

2.5 The program has established processes for assessing student learning outcomes and for assuring that students are achieving core competencies for completion of the program.

Program Performance … (Hint: you don’t want a blank here.)

From the FYR template … (cont’d)

Organizing For Learning: Faculty Resources

Criteria for Review

3.1 Faculty recruitment, incentives and evaluation practices are aligned with … program missions and educational objectives.

3.3 The deployment of faculty reflects the mission and program goals …

Program Performance … (Hint: Need mission, goals and outcomes statements to answer these questions.)

The FYR is consequential

3. Final Review and Approval of Continuous Approval Plan

“The Review Team – consisting of an Office of the Provost representative, College Dean, Department Chair, Program Director, and an undergraduate student – will make one of four recommendations to the Provost ….

1. Recommend continuation …2. Recommend continuation , but cite specific

concerns …3. Recommend the program remain under

continuing review ….4. Recommend suspension ….

(Hint: Outcome #4 not so good …)

The Right Thing, Right Reasons, & Get a Jump on FYR Aligning Mission, Goals, Student Learning

Outcomes and Curriculum is Good Practice Best Practice engages in a process of

continuous assessment & improvement SLO assessment is a necessary component

of FYR … you don’t want blanks! Departments do things other than teach,

but excellence in undergraduate education is our #1 strategic priority …

Assessment of SLOs: evidence of success

Toward A Comprehensive Assessment Plan:

Program Mission Goals Student Learning Outcomes (SLOs) Course Alignment Matrices

An Outcomes / Course Alignment Matrix

Course SLO 1 SLO 2 SLO 3 SLO 4

151 I I

177 P I

201 P I

277 P P

340 M - oral presentation

M

477 P - term paper

Stage 1 – Ensuring a Cohesive CurriculumIn which courses are particular outcomes introduced (I), practiced (P), mastered (M)?

Assessment of SLOs

Direct Assessment Methods: Published Tests Local Exams Embedded Assignments & Activities Portfolios

Indirect Assessment Methods: Surveys Interviews Focus Groups

Assessment of SLOs

Direct Assessment Methods: Examine Student Work …

Indirect Assessment Methods: Examine Student / Alumni Perceptions

…

Effective Assessment Programs Use Multiple Measures … Triangulate

Using Rubrics to “Objectively” Evaluate Student Work …

Explicit Scoring for Student Work: Holistic Scoring

Analytical Rubrics

Program Faculty Should Discuss, Debate, Develop Some Consensus on What Methods and Measures (Rubrics) to Use …

Evaluation and Use of Data for Continuous Improvement

Assessment Data Must be Evaluated Systematically, Periodically Recommend Annual Collection / Assessment

/ Action Plan Modify Curriculum and Pedagogy to Improve

Student Learning … Typical Actions:

Modify Content, Sequencing of Course(s) Modify Pedagogical Approaches

Follow-up Focus Group or Survey to Triangulate …

Moving Toward Continuous Improvement:

The Entire Process (Goals, Outcomes, Assessment Measures) Must be Evaluated Periodically Recommend Five Year Re-evaluation to

synchronize with FYR process Are we measuring reasonable work samples? Is our process meaningful? Are we better

informed about what our students are learning or not?

Is our process sustainable? Can we be more efficient and still have valid measures?

The Assessment Plan Design Process … Wherever you begin developing your

Assessment Program will NOT be the ideal place to start …

You Must Begin Somewhere …

Iteration will save you ... Continuous Refinement of Methods Consensus and Team Building Alignment of Program Resources with

Program Goals and Faculty Values …

Case Study: Mechanical Engineering Program Mission

Mechanical Engineering Program Mission Statement

The Mechanical Engineering Program has the primary mission of providing a high-quality undergraduate engineering education with

1.A curriculum that is firmly grounded in engineering fundamentals2.A faculty that provides superior teaching and mentoring both in and out of the classroom3.A faculty whose focus is undergraduate education4.Class sizes that encourage student participation5.Project experiences that build on fundamentals and develop team skills6.Facilities and equipment that are readily accessible7.An environment that is conducive to learning and encourages students from different genders and backgrounds

Note: Not Goals or Outcomes for Student Learning !

Mechanical Engineering Program graduates demonstrate the ability to:

A. Apply knowledge of math, science, and mechanical engineering to formulate and solve problems.

B. Conduct experiments, and analyze and interpret the resultant data.C. Design mechanical components and systems to meet desired

needs, and design experiments to evaluate system performance with respect to specifications.

D. Use current mechanical engineering tools.E. Communicate technical matters effectively in oral, written, and

graphical form.F. Function effectively as members of multi-disciplinary teams.G. Understand their professional and ethical responsibilities, the

impact of their activities on society and the environment, and appreciate contemporary issues facing society.

H. Use information resources and recognize the importance of continued learning.

Mechanical Engineering Student Learning Outcomes

Mechanical Engineering Embedded Assessment Matrix:

SLOs: A B C D E F G H

Courses where student work is measured

Embedded Assessment of Outcome C in ME 138: Design mechanical components and systems to meet desired needs …

•Student grades on design projects are based on assessment of 6 design projects: 3 individual, 3 group

•A standardized rubric is used to score each design

•The performance metric:•A cumulative grade of C or better on design projects must be attained to demonstrate the ability to design mechanical components and systems (competence).

Other Measures of Mechanical Engineering Outcome C (Triangulation!)

•Senior Exit Survey Question: How well prepared are you to design a component or system to meet desired needs?

•Mean Responses (4.29 Spring 2002, 4.0 Spring 2003)

•Scale: 5=Very Well Prepared – 1=Very Unprepared

•A mean response of 4.0 is considered acceptable to faculty

Outcome C: Design mechanical components and systems to meet desired needs …