Developing an Assessment Plan

Owens Community College Assessment Day WorkshopsNovember 13-14, 2009

Anne Fulkerson, Ph.D.Institutional Research

Agenda

What is assessment? Why assess? Types of assessment Basic steps in the assessment planning process

Identifying outcomes Defining measures Evaluating measures Other important considerations

OCC assessment plan template & rubric Sample assessment plan How IR can help

What is Assessment?

“Assessment is the ongoing process of:

Establishing clear, measurable expected outcomes of student learning.

Ensuring that students have sufficient opportunities to achieve those outcomes.

Systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches our expectations.

Using the resulting information to understand and improve student learning.”

Suskie (2004), pg. 3

Why Assess?

Demonstrate effectiveness Improve student learning

Measure Learning

Adapted from Suskie (2004) and Maki (2004)

Provide Learning Opportunities

Establish Learning Outcomes

Implement Change

Types of Assessment

Formative—assessment that takes place while learning is in progress (or while a new program is being developed) in order to provide feedback for improvement

Summative—assessment that takes place at the end of a course or program to document that learning has occurred or to make judgments about the efficacy of a program

Establish Mission Statement

Identify Student Learning Outcomes

Student Learning Outcomes: Specifiable activities, products, skills, abilities, behaviors, attitudes, or pieces of knowledge that students attain as a result of their involvement in a particular set of educational experiences.

Represent the most important competencies that all students should possess

Do not reflect ALL learning that might occur Often aligned with accrediting body standards No magic number, but keep them manageable

Expressing Student Learning Outcomes

Too vague:Students will demonstrate information literacy skills

Too specific:Students will be able to use institutional online services toretrieve information

Better:Students will locate information and evaluate it criticallyfor its validity and appropriateness

Expressing Student Learning Outcomes

Fuzzy Terms: Action Words:

KnowUnderstandBecome aware ofAppreciateThink criticallyDemonstrate knowledgeLearnWrite proficiently

RememberIdentifyPerformCreateDefineSummarizeExplainDiscussDescribeSolveFind/LocateAnalyze

EvaluateApplyDebate/ArgueSynthesizeIntegrateResearchChooseConstructDesignDevelopOrganizeUse

Bloom’s Taxonomy (Revised)

Based on an adaptation of Anderson & Krathwohl (2001)http://uwf.edu/cutla/assessstudent.cfm

Thin

king

Ski

lls

Not Measurable:

Recognize a need for lifelong learning and plan for personal andprofessional growth

Demonstrate an historical knowledge of the symphonic, string orchestra, and chamberensemble repertoire

Sample Student Learning Outcomes

Measurable:

Describe and adopt a plan for ongoing professionaldevelopment and lifelong learning

Describe the historical development of the symphonic, string orchestra, and chamberensemble repertoire

http://uwf.edu/cutla/assessstudent.cfm

Identify Opportunities for Students to Achieve Each Learning Outcome

Course / Activity / ExperienceLearning Outcome

1.

2.

3.

4.

5.

Define Measures for Each Outcome

Types of Measures:

Quantitative—numeric (e.g., test scores, placement rates, GPA, structured surveys)

Qualitative—described by words rather than numbers (e.g., interviews, focus groups, observations, rubrics)

Direct—require students to display knowledge or skill (e.g., tests, performances, assignments)

Indirect—second-hand evidence (e.g., surveys, course evaluations, journal reflections)

Word of Caution

Don’t measure everything, just because you can Pick the best measures Keep it manageable

Determine How Measures will be Evaluated Criterion-based benchmarks—compares

student performance with a pre-established standard. Norm-referenced benchmarks—compares

student performance with a standardized norm or a group of peers.

Value-added approach—compares student performance at two points in time to see if they have improved.

Longitudinal/historical approach—compares current students with prior students.

Other Things to Consider

What resources are needed?

Who’s responsible for collecting and analyzing the data?

Build support through participation in the planning process

Assessment plans are perpetual drafts

OCC Assessment Plan Templatehttps://intranet.owens.edu/committees/outcomes/index.html

Program Mission: Program Vision:

Program Level Student Learning Outcomes Program Student Learning Outcomes # 1: Where taught? Where

Where measured?

Direct measure(s)

Indirect measure(s)

Resources needed

Person(s) responsible

Schedule Benchmark(s)

1. 2.

1. 2.

1. 2.

1. 2.

Program SLO # 2. Where taught?

Where measured?

Direct measure(s)

Indirect measurer(s)

Resources needed

Person(s) responsible

Schedule Benchmark(s)

1. 2.

1. 2.

1. 2.

1. 2.

Program SLO # 3 Where taught? Where

measured? Direct measure(s)

Indirect measurer(s)

Resources needed

Person(s) responsible

Schedule Benchmark(s)

1. 2.

1. 2.

1. 2.

1. 2.

Program SLO # 4 Where taught? Where

measured? Direct measurer(s)

Indirect measurer(s)

Resources needed

Person(s) responsible

Schedule Benchmarks(s)

1. 2.

1. 2.

1. 2.

1. 2.

Program SLO # 5 Where taught? Where

measured? Direct measurer(s)

Indirect measurer(s)

Resources needed

Person(s) responsible

Schedule Benchmark(s)

1. 2.

1. 2.

1. 2.

1. 2.

OCC Assessment Plan & Report Rubrichttps://intranet.owens.edu/committees/outcomes/index.html

Elements (1) Developing Established Exemplary Program goals and intended student learning outcomes (Appropriateness)

Lack of or incomplete inclusion of some established criteria.

Describe how the goals & SLO align with College &/or School mission & vision. Describe how the goals & SLO link to accreditation standards. if app Describe how the goals & SLO link to industry and/or academic standards.

Provides evidence supporting the statements in “Established” category.

Evidence is appropriate for the measurement of student learning (Evidence)

Lack of or incomplete inclusion of some established criteria.

Evidence that the measurement instrument is aligned with goals and/or SLO. Benchmarks in place and evaluated regularly Artifacts when utilized are evaluated with rubrics, when appropriate. At least one direct & one indirect measure as appropriate.

Normed evidence Provide information on validity & reliability of instrumentation. Rubrics are constructed and/or agreed upon by most stakeholders. Multiple direct & indirect measures.

Individuals responsible for delivering the program engage in systematic analysis of evidence. (Analysis of Evidence & Shared Responsibility)

Lack of or incomplete inclusion of some established criteria.

Describe how the data was analyzed. Provide evidence that appropriate stakeholders were involved in the analysis of data. Describe how the data was used to make decisions about program improvement. Describe action steps needed to improve data collection and/or program. Provide a summary of the analysis with references to data.

Provide evidence that the previous plan of action was implemented. Provide evidence that the previous action plan was assessed. Provide results of the implementation of the action plan.

Results are shared with essential stakeholders

(Shared Responsibility)

Lack of or incomplete inclusion of some established criteria.

Evidence that the report was shared with all faculty and advisory committee members where applicable.

Evidence that the report was shared with additional stakeholder groups.

Sample Assessment Plans

Owens— https://www.owens.edu/portrait/index.html

College of Charleston, SC— http://spinner.cofc.edu/~oap/docs.html

Supplemental Instruction

How IR Can Help

Data support and analysis Assessment consulting

Survey development Rubric development Identifying appropriate metrics Identifying existing data sources

References

Maki, P. (2004). Assessing for Learning: Building a Sustainable Commitment Across the Institution. Stylus Publishing: Herndon, VA.

Palomba, C. & Banta, T.W. (1999). Assessment Essentials. Planning, Implementing, and Improving Assessment in Higher Education. John Wiley & Sons: San Francisco, CA.

Suskie, L. (2004). Assessing Student Learning: A Common Sense Guide. Jossey-Bass: San Francisco, CA.

Owens Community College, Student Learning Assessment Committee (2009). Building your SLAC assessment plan: Instructions for the SLAC assessment plan template version #1: intranet.owens.edu/committees/outcomes/index.html.

University of West Florida, Center for University Teaching, Learning, & Assessment. http://uwf.edu/cutla/Assessres.cfm.