Assessing and Improving Student Outcomes

Presentation for TENNAIR Conference

August 7, 2008

Gwen P. Aldridge, Ph.D.Director of Assessment

J. Nevin Robbins, Ph.D.Executive Director for Planning and Analysis

Presentation to include:

• Importance of SLOs• Defining Learning Objectives• Selecting Measures & Setting Benchmarks• Comparing Outcomes with Intended Objectives• Using Results to Improve Learning• Online Documentation – Southwest’s System• SLO Hints

Current Focus on Importance of Student Learning Outcomes

This focus represents a convergence of two major trends in higher education: the assessment movement and the accountability movement.

Assessment and Accountability

• Assessment has evolved from efforts to improve the quality of teaching and learning by improving the quality of educational outcomes

• Accountability has evolved from the efforts of state legislatures to prove institutional effectiveness in making higher education more cost effective

Tennessee Academic Audits for Programs• Criterion 1.1: The faculty completed an honest analysis of their process for developing learning objectives for the program, considering measurability, clarity, and what students need to know.

• Criterion 1.2 : The faculty have documented or proposed a process for developing learning objectives that are based on realistic and appropriate evidence.

• Criterion 1.3: The faculty have documented or proposed

specific plans to take best practices and appropriate benchmarks into account in the analysis of learning objectives.

Good Assessment Practice

• First - assess what is most important• Realize that anything that can be taught or

learned can be assessed• Apply assessment at the course, program, and

institutional levels• Organize every course and every program

around clearly defined learning goals and objectives, explicit assessment methods, and measurable outcomes.

Program Outcomes

• Are needed for internal assessment of continuing improvement

• Are needed for external assessment of institutional effectiveness for accountability purposes

• Are useful in communicating program results for purposes of marketing

• Insure that the program is more than just a cluster of courses

The Purpose of Student Outcomes Assessment Effort

Academic Program Improved

Accreditation Requirements Satisfied

Effectiveness Demonstrated

To tell students what is expected of them and thus…

help students learn more effectively

make it clear what students can gain

help instructors design their materials more effectively

help instructors select the appropriate teaching strategy

assist in setting examinations based on the material

developed

ensure that appropriate assessment strategies are employed

What SLOs can accomplish:

SACS Compliance Standard 3.3 Institutional Effectiveness

• 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas (Institutional Effectiveness):

• 3.3.1.1 educational programs, to include student learning outcomes

• 3.3.1.2 administrative support services

• 3.3.1.3 educational support services

• 3.3.1.4 research within its educational mission, if appropriate

• 3.3.1.5 community/public service within its educational mission, if appropriate

Simply… the 4 Column Model:

• Learning outcome

• Method of assessment

• Assessment results

• Use of results

Assessment Learning Cycle

From SACS-COC Resource Manual for Review Teams

“The expectation is that the institution will engage in ongoing planning and evaluation to ensure that, for each academic program, the institution develops and assesses student learning outcomes.”

Note the “each” in the expectations – it’s not “some”.

From SACS-COC Resource Manual for Review Teams - continued

“Program and learning outcomes are grounded in the faculty’s knowledge of the content and coherence of the discipline as well as in the learning process and reflect expectations for performance consistent with the level of the program and the mission of the institution.”

From SACS-COC Resource Manual for Review Teams - continued

Sample Documentation for this requirement:Representative examples of program and learning

outcomes for each educational programDescriptions of methods for evaluating student

achievement of these outcomesReports of the results of the evaluation, examples

of how the results have been used for program improvement, and examples of how methods of evaluation have been improved over time

From SACS-COC Resource Manual for Review Teams - continued

“At appropriate intervals, program and learning outcomes and evaluation methods are evaluated and revised as appropriate.”

From SACS-COC Resource Manual for Review Teams - continued

“Program and learning outcomes specify the knowledge, skills, values, and attitudes students are expected to attain in courses or in a program.”

“Methods for evaluating the extent to which students achieve these outcomes are appropriate to the nature of the discipline and consistent over time to enable the institution to evaluate cohorts of students who complete courses or a program.”

Unit/Program Intended

Outcomes

Whatdo you want

students to know think or do when

they have completed the

program?

Intended Method of Measurement

and Level of Performance

Howwill you measure? What measurement

activity will demonstrate that

students accomplished the

outcome, and what overall level of performance do you wish to set?

Assessment/Evaluation

Results

Later...

What was the score?

Was performance up to the level you set?

Give Results.

Use of Results

Improvement

Either

Make a change and state that change

Or state

“No action necessary.”

UNIT ANNUAL OUTCOME OBJECTIVES EDUCATIONAL PROGRAMS

Column 1: Defining Learning Objectives

The dream begins with a teacher who believes in you, who tugs and pushes and leads you to the next plateau, sometimes poking you with a sharp stick called "truth." 

• ~Dan Rather

Promising practices in Defining Learning

ObjectivesMake certain that Learning Objectives

Are specific to the program

Are focused on what is critical to the program

Are descriptive of the what a student should gain as a result of their

completion of the program

Are clear and understandable

Are written to a level of specificity while allowing a certain amount of

interpretation leeway for faculty

Use action verbs

Are realistic given the typical student in the program

Are assessable

Is outcome singular?

Singular – Statement of One Outcome

Don’t bundle multiple outcomes in a single statement. This leads to problems in formulating means of assessment.

When writing SLOs, the focus should be on observable outcomes and an “action verb” can

provide that focus.SLOs usually begin with something like:

By the end of the secondary education program,

students will be able to design curriculum and

appropriate instruction.

Design is the “action verb” in this example.

Level Action Verbs Outcome Example

Knowledge Recite, List By the end of the chemistry program, students will be able to list all of the elements on the Periodic Table.

Comprehension Translate, interpret, predict, generalize, identify examples

By the end of the French program, students will be able to translate a paragraph of text from English to French.

Application Apply, rewrite By the end of the BIS program, students will be able to apply basic Web development skills.

Analysis Analyze, dissect, resolve, solve, diagnose, investigate

By the end of the special education program, students will be able to diagnose learning disabilities in K-12 settings.

Synthesis Create, synthesize, write By the end of the art program, students will be able to create at least 12 original works in their medium.

Evaluation Evaluate, judge, rate, appraise

By the end of the music education program, students will be able to judge student performances.

Level Action Verbs Outcome ExamplePerception Chooses, describes, detects,

differentiates, relates, selectsBy the end of the music theatre program, students will be able to relate types of music to particular dance steps.

Set Begins, displays, explains, moves, proceeds, reacts, responds, shows

By the end of the physical education program, students will be able to show the proper stance for batting a ball.

Guided, Complex, Mechanical Response

Assembles, builds, constructs, dissects, fastens, fixes, grinds, heats, measures mends, mixes, sketches

By the end of the aviation maintenance program, students will be able to draw sketches of aircraft repairs and alterations.

Adaptation Adapts, alters, changes, rearranges, reorganizes, revises, varies

By the end of the industrial education program, students will be able to adapt their lessons on woodworking skills for disabled students

Origination Arranges, combines, composes, constructs, creates, designs, originates

By the end of the dance program, students will be able to create a dance step.

Statements of Educational Outcomes should answer WHAT?

What students should:

Know (cognitive)Think (affective)Do (behavioral)

when they have completed a degree program.

How many SLOs are enough?

A program may have only a few SLOs or a long list of

them. A total of five to seven outcomes is typical for a

program. Most career-technical programs have standardized student competency

profiles and instructors may select the learning outcomes from these. An

instructor should not list each competency, but should include the ones

deemed most important.

_______ (who or what will be changed)

__________ (when will the changes take place)

__________ (how much change will there be)

Goals are necessary for improvement because when we shoot at nothing, we usually hit it.

Time-Based

Measurable

Specific

Evaluating SLO Statements

1. Are they reasonable given student ability?

2. Are they clear?

3. Is accomplishment measurable?

4. Is outcome singular (not a bundle of outcomes)?

5. Do they focus on what students can know, think,

or do as a result of completing the program?

A Citation for One Institution Regarding Learning Outcomes: 3.4.1 Non-Compliance: The On-Site Committee should seek evidence that the college establishes and evaluates student

learning outcomes for each of its educational programs.

• Comments from the Off-site team: The execution of the student learning outcome system varied from effective and accurate in the description of student learning outcomes and their assessment to ineffectively broad in defining the student learning outcomes and assessments not keyed to the desired outcomes. For example, certain programs effectively and descriptively focused on what students could do, know or believe as a result of completing their programs and also used assessment tools directly addressing the expected outcomes. Other programs presented groups of student learning outcomes from courses making up the program or from a course sub-section as one learning outcome.

Identify the problem here:

Unit: Commercial Truck Driving

• The students will demonstrate the ability to safely operate commercial trucks.

Outcome: Problem

Funeral Service Technology: The student will be able to explain the techniques and theory of all elements of funeral service.

General Agriculture: Students will demonstrate knowledge of photosynthesis, plant anatomy, cellular functions, classifications and genetics.

Commercial Truck Driving: The students will demonstrate the ability to safely operate commercial trucks.

Business and Office Technology: Students will demonstrate the ability to keyboard alphanumeric material at specified speed and accuracy levels.

Outcome: Problem

Funeral Service Technology: The student will be able to explain the techniques and theory of all elements of funeral service.

•Ineffectively Broad

General Agriculture: Students will demonstrate knowledge of photosynthesis, plant anatomy, cellular functions, classifications and genetics.

•Groups of Learning Outcomes

Commercial Truck Driving: The students will demonstrate the ability to safely operate commercial trucks.

•Ineffectively Broad

Business and Office Technology: Students will demonstrate the ability to keyboard alphanumeric material at specified speed and accuracy levels.

•Acceptable

Column 2: Selecting Measures & Setting Benchmarks

• Are Assessments Keyed to the Desired Outcome?• Are Assessments Reasonable Given Student Ability?

• “Fledgling skiers make the most progress when they are pushed outside their comfort zone, but not so far that they are scared off the slopes altogether.” a ski instructor

We’re #1!!!Mississippi - #1 in Poverty, #1 in Obesity, #1 in Teenage Pregnancies……

Assessment of the Outcomes

Each SLO should have its own assessment.

At the end of the assessment, an instructor should know which

SLOs were achieved, and which were not achieved. If goal related

questions are used on a final exam to assess more than one outcome,

an item analysis should be done.

For example, 80% of students achieved Outcome 1, 60% of students

achieved Outcome 2, 75% of students achieved Outcome #3…

Without such individual assessment of outcomes, the

instructor fails to know in the end what was

accomplished and thus what individual changes in

instruction should be made in improving Student

Learning in the program. The ultimate purpose of

writing and assessing SLOs is Program

Improvement.

Some acceptable measures to evaluate SLOs:

• standardized tests with questions tied to specific SLOs

• locally developed tests with SLO goal-related questions identified (use item analysis to judge performance on each individual outcome)

• analysis of theses, portfolios, recitals, speeches • student performance in lab activity

Assessments…Are they reasonable given student ability?

Make sure criteria (bench marks) for success and expected student achievements are not set at levels far beyond the ability of most students.

We should STRETCH, but not STRAIN.

Individual Students Criteria/Intended Educational

Outcomes Average

SLO Student 1 Student 2 Student 3 Student 4 Student 5

Cell Biology 4 9 5 7 9 6.8

Metabolism 3 9 7 8 9 7.2

Reproduction 9 9 7 9 9 8.6

Evolution

Taxonomy

2

7

9

9

4

7

4

8

4

9

4.6

8.0

TOTAL 25 45 30 36 40

Individual Student Grade F A D C B

Individual StudentGrading and Educational Outcomes Assessment

Class Grades or G.P.A. as a Means of Assessment

Why not just use the course grades as assessment? With course grades, we have assessment of individuals, not programs Also, the objectivity of the evaluator is questioned Finally, an instructor is not provided with information that allows that instructor to assess student accomplishment on the individual SLOs

(Generally grades are not accepted as a mean of assessment for SLOs).

Individual Section Averages Criteria/Intended Educational

Outcomes Average

SLO Section 1 Section 2 Section 3 Section 4 Section 5

Cell Biology 3 7 5 6 8 5.8

Metabolism 4 8 8 7 7 6.8

Reproduction 8 9 8 8 9 8.4

Evolution

Taxonomy

3

6

7

9

4

7

3

8

3

7

4.0

7.4

Biology I ClassesGrading and Educational Outcomes Assessment

Citation for NWCC from SACS-COC3.4.1 Non-Compliance Recommendation: The On-Site Committee should seek evidence that the college establishes and evaluates student learning outcomes for each of its educational programs.Comments from the Off-site team: …One program listed all student learning outcomes from each course, but used class management and course requirement standards as assessment means. In assessing learning outcomes, most programs used a retention indicator instead of a learning outcome indicator, that of course completion, or a measurement tool that did not address the desired outcome. In those cases a “C” grade in a course or a comprehensive exam grade without targeting portions of the exam relevant to the learning outcome were presented as proof of achieving the outcome.

After completing the pottery course, 75% of students will demonstrate mastery of this technique by making a C or better in the course.

Students will demonstrate mastery of this techniques by making a C or better on the final exam.

After competing the pottery course, 75% of students will demonstrate mastery of this technique by receiving a rating of “good” or better on pieces produced based on class critique and the grading of formed pieces. Emphasis will be on use of knowledge attained, consistency in building, and creativity.

Art: Students will demonstrate mastery of hand building pottery techniques by creating forms with pinch pots, coiled symmetrical cylinders, and crafting coil and slab combinations.

After completing the pottery course, 75% of students will demonstrate mastery of this technique by making a C or better in the course.

No!

Students will demonstrate mastery of this techniques by making a C or better on the final exam.

No!

After competing the pottery course, 75% of students will demonstrate mastery of this technique by receiving a rating of “good” or better on pieces produced based on class critique and the grading of formed pieces. Emphasis will be on use of knowledge attained, consistency in building, and creativity.

Okay

Art: Students will demonstrate mastery of hand building pottery techniques by creating forms with pinch pots, coiled symmetrical cylinders, and crafting coil and slab combinations.

Outcome: Agriculture Technology

Assessment

1. Student will demonstrate the ability to disassemble and reassemble clutch packs from the 16 speed power shift transmission.

90% of all students completing the program will demonstrate skills by completing student competencies.

2. Student will demonstrate an ability to repair and replace component parts of a machine including eccentrics, shafts, bearings, fasteners and o-rings.

3. Students will demonstrate the ability to service the fuel system in internal combustion engines.

With “Assessments not keyed to desired outcomes…”

Column 3: Comparing Outcomes with Intended Objectives

At some point, you just have to decide that you are not going to be afraid of the data.

It may not be pretty… But you can use it.

Column 4: Using Results to Improve Learning

How might we use the results?

We may decide to…

Change the related SLO

Adjust the Program/Unit Activities

Modify the Means of Assessment

Who Kissed the Mirror?

Assessment results are used for

Improving Educational

Programs…. Thus Column 4, our

“Use of Results”.

Ever tried. Ever failed. Try again. Fail again.

Fail better. Samuel Beckett

The End Product of Assessment Activities

Statement of “use of results”• The Use of Results should be stated in

past tense - an action that has already taken place to improve the program.

• But, what if the improvement won’t take place until the following year (semester)?

You may say, “A decision was made to begin…….. in the upcoming semester.” Make the decision now.

Unit/Program Intended

Outcome

Welding:

Students will be able to perform welds for all

position fillet and groove welding within a limited thickness range

of plain carbon steel material.

Intended Method of Measurement

and Level of Performance

80% of students in Gas Tungsten Arc Welding will perform all welds

and welding operations identified in the Student

Competency Profile.

Assessment/Evaluation

Results

75% of students successfully completed this competency profile.

Use of Results

Improvement

A decision was made to give students more hands-on in this skill and to spend more time on questions and answers.

UNIT ANNUAL OUTCOME OBJECTIVES EDUCATIONAL PROGRAMS

A Yearly Implementation Cycle

Identify Expected Outcomes & Identify Means of

Assessment: (Columns 1 & 2) early Fall

Carry Out Means of Assessment during the school year

Report Assessment and Use of Results (Columns 3 & 4)

Either in May at the end of the school year or the following

August

We are continually faced with a series of great opportunities

brilliantly disguised as insoluble problems…John Gardner

SLO Hints:For SLOs to accomplish anything,

The administration at a college must take steps to promote inquiry and alleviate fear. Instructors must be assured that assessment results of the SLOs will be in no way tied to their own evaluations.

Lessons Learned…Some SLO Hints

• There are some unacceptable words that sometimes jump into the assessment column that just won’t work (retention, course grade of C or better, comprehensive exam grade)

Lessons learned…• The expectations of Peer Review Teams are

as varied as the individuals who comprise those teams. You can never predict in advance what a team will wish to focus on, or how knowledgeable team members will be on the topics being reviewed – but you can be sure of one thing, and that is that SLOs are currently a hot topic.

Celebrate Success

Your college is a haven for excellent student learning.

Embrace assessment, learn tomeasure student learningcommunicate it, and celebrate

it!

Gwen P. Aldridge, Ph.D.Director of Assessment

J. Nevin Robbins, Ph.D.Executive Director for Planning and Analysis

Southwest TN Community CollegeMemphis, TN

Office: 901-333-5257

gpaldridge@southwest.tn.edu nrobbins@southwest.tn.edu