1 understanding student learning outcomes and assessment bob barr andrew lamanque rob johnstone...
TRANSCRIPT
1
Understanding Student Learning Outcomes and
Assessment
Bob BarrAndrew LaManque
Rob Johnstone
Adapted from material prepared by Fred Trapp, Long Beach CC,
the RP Group, and James O.Nichols, University of Mississippi
Breakout Session, District Opening Day, September 22, 2005
2
Session Outcomes
Describe a multi-step process for conducting learning outcomes assessment and name the purpose of each step.
Distinguish among levels of assessment work.
3
Session Outcomes
Critique a student learning outcome statement.
Describe two means of assessment and match them to an outcome statement.
4
What Is Student Learning Outcomes Assessment?
An ongoing process of systematically, gathering, analyzing, and interpreting evidence aimed at understanding and improving student learning.
Involves making learning expectations explicit and public.
Involves setting appropriate criteria and high standards for learning quality.
5
Why Do Assessment?
The Reason – Improve student learning and institutional support services.
The Effort – Continuous reflection on the teaching-learning process
An Occasion to Our Display Work – Regional or program accreditation.
6
Accrediting Commission for Community & Junior Colleges
(ACCJC)
Standard II: Student Learning Programs and Services
“The institution engages in ongoing, systematic evaluation and integrated planning to assure currency and measure achievement of its stated student learning outcomes for courses, degrees, certificates, and general education and vocational programs.”
James Nichols, Department Guide & Record Book 7
The Assessment Process
Student Learning
Assessment Activities
3. Means of Assessment
and Criteria for Success
2. Intended Learning
Outcomes
5. Use of Results
4. Assessment Results
1. Institutional Mission & overall institutional learning outcomes
8
Assessment Process and Key Questions
1. Link of GE and program to mission & institutional outcomes; course to program outcomes.
Where does the ______ program fit into the mission and learning outcomes of the college?
2. Specify intended learning outcomesWhat learning comes of completing the course,
program, or degree?
9
Assessment Process and Key Questions
3. Identify means of assessment and criteria for success
How shall we discover what students have learned? How good is good enough?
4. Assemble the results and make meaning from them
How well did they do relative to expectations?
10
Assessment Process and Key Questions
5. Using the resultsWhat shall we do with these results?
Document, document, documentSuggestion…
Department/program plan Program review
11
Levels of SLO Assessment Work
College-wide General education or core
competencies Instructional Program or pre-major
or Student Service Program Course or sections of a commonly
taught course
12
Step 1. Linking to Mission -Expanded Statement of
Purpose
Mission of the college Broad statements of defining purposes and
outcomes GE, program, course purpose statements
Links the GE package and program to the mission of the college
Links the course to GE package or program goals Cornerstone of self-assessment (& self-
study) How well are you accomplishing your purposes?
13
Step 1. Linking to Mission -Linking the Three Levels
College or GE Outcome Students will be able to speak and write
effectively. Academic Program Outcome (Psychology)
Students can speak and write effectively using the principles and concepts of the discipline.
Course Outcome (Psych Measurement) Students can prepare a written summary and
interpretation of standardized test results.
14
Traditional Achievement Outcomes Success rates Retention or persistence rates Number and rates of graduates or transfers
Student Learning Outcomes What faculty intend students to know
(cognitive), think or believe (affective) or be able to do (behavioral) when they have completed
Degree (general education) Field of concentration (program) Course
Two Types of Educational Outcomes
(Institutional/Program Performance)
15
Step 2. Specify Intended Learning Outcomes
Ideally, the curriculum and courses are designed by working backwards from the ultimate outcomes.
What roles will program graduates hold and what must they be able to do “out there” in the real world or in upper division work?
How will the outcomes be demonstrated “in here?”
What skills and concepts need to be learned?
16
Step 2. Specify Intended Learning Outcomes
Helpful hints Use concrete, action verbs
See Bloom’s Taxonomy of Educational Objectives Anticipate the “deliverable”
What is it that students must do to convince you that they are where you want them to be with respect to learning?
Enabling vs. terminal Target key concepts or change points in the
program Major aspects of learning central to the discipline
17
Example: Specify Intended GE Learning Outcomes
Be exposed to good writing examples. (in need of repair)
Upon completion of a degree, the student will be able to…
Write effectively
Compose a range of written products designed for different audiences and purposes.
18
Example: Specify Intended Psychology Program Learning
Outcomes
Hear and read about the scientific method of inquiry as used in the social sciences. (in need of repair)
Upon completion of the program, the student will be able to…
Use critical thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes..
James Nichols, Department Guide & Record Book 19
Expanded Statement of Institutional
Purpose
Identity Statement:
Your Community College is an open-admission, community-based, comprehensive college.
Mission Statement:
Serve persons of all ages in preparing for entry and careers in a variety of fields, such as automotive technology.
Program Intended Educational Outcomes:1. Graduates of the Automotive Technology Program will be successfully employed in the field. (achievement outcome, not a SLO)
2. Graduates of the Automotive Technology Program will be technically proficient.
3. Employers of Automotive Technology Program graduates will rate them competent based on the education received in the program and would employ others.
Example: Auto Tech Program Intended Student Learning
Outcomes
20
Step 2. Specify Intended Student Services Program Learning
Outcomes
Know how to be a more prepared student. (in need of repair)
Upon completion of the EOP&S summer transition program, the student will be able to…
Develop and use a personal schedule that incorporates regular study time, regular study location and study groups to match their class schedule.
21
Characteristics of Intended Student Learning Outcomes
Student-focused, not teacher focused Focus on learning resulting from an
activity, not the activity itself Focus on important, non-trivial
aspects of learning Focus on skills & abilities central to
the discipline, based on professional standards of excellence
22
Characteristics of Intended Student Learning Outcomes
General enough to capture important learning but clear and specific enough to be measurable
Focus on aspects of learning that will develop and endure but that can be assessed in some form now
23
Aligning Outcomes,Teaching-Learning and Assessment
Outcome statements
Methods of teaching/learning
Assumptions about teaching/learning
Methodsof assessing
Assumptions about methods
What do you expect students to know, understand, be able to do as a result of your teaching?
What methods develop/ foster your primary outcomes?
What assumptions underlie your methods?
What methods do you believe provide evidence of desired student outcomes?
What assumptions underlie your belief that each method will elicit students’ demonstration of your desired outcomes?
Step 3. Some Means of Assessment
Embedded Narrative Assignments Case study Portfolio (paper or electronic) Capstone project Senior exit essay
Performances Critical situation scenario Debate Demonstration Oral presentation Online monitoring
Tests Standardized test Homegrown test In class objective test In class essay
Student Centered Exit interview Focus group Self-rating of progress
Archival Library use, on-line activity Transcript analysis Transfer records Stakeholders Perspective
Alumni, employee, employer surveys
Experience of faculty who inherit our students
We’ve always done this, but now systematically to improve student learning rather than just to assess the student and assign a grade.
25
Step 3. Means of Assessment Levels of assessment
College (General Education, Core Abilities)
Program (Deg, cert, or course sequence) Course
Types of evidence Direct Indirect and supportive
26
Step 3. Means of Assessment Direct evidence of student learning
outcomes (think: in the aggregate) Passage of license, certification or subject
area exams Results of general education exams Capstone projects or performances Employer or internship supervisor ratings of
student performance Class assignments, projects, etc. Departmental exam results But, NOT grades.
27
Step 3. Means of Assessment Indirect evidence of student
learning outcomes (think: in the aggregate) Student self-reflective reports Surveys of perception
Alumni, graduating or current students Focus group interviews
28
Step 3. Means of Assessment
“Targets” for Assessment Cognitive outcomes (knowledge,
thinking disciplines) Attitudinal outcomes (values, beliefs) Behavioral/performance outcomes
(use of knowledge or values) or Knowledge, Skills, and Abilities
(KSAs)
29
Grading is not Assessment Evaluation of individual students =
assessment Focus is individual not groups of students A summative, not formative act Objectivity of single evaluator vs. group Generally not accepted as direct evidence Uses of the grading process
Agreed upon course exam or part of exam Row and column model for assignments
30
Grading and SLO Assessment(Embedded Assignment)
Criteria Tim Jane Mary Joe Dave Average
Spelling 3 4 1 2 3 2.6
Grammar 2 5 3 2 5 3.4
Punctuation 4 5 2 3 4 3.6
Structure 3 2 3 5 3 3.8
Total 13 17 10 12 15
Student Grade C A D C B
Total down the column for individual grading. Analyze across the row for assessment of intended outcomes from the group.
James Nichols
31
Step 3. Means of Assessment - Course Exams
Agreed upon course exam questions Monitor student responses to
identify topics of weakness Item analysis to discern what groups
are having difficulty with selected items
Revisit teaching approach to the topic
32
Step 3. Means of Assessment - Embedded Assignments
Embedded narrative class assignments Evaluated for the grade mark Later a sample is re-inspected to
Assess program accomplishment against expected outcomes
Explore the learning process Agreed upon performance criteria
The fruits of campus dialogue Expert judgment of the faculty
Inter rater reliability
33
Step 3. Means of Assessment -Rubrics
Key Questions What essential elements must be
present in student work to ensure high quality?
What does student work look like at each level of mastery?
34
Step 3. Means of Assessment -Elements of A Useful Rubric
Dimensions/criteria of rating (the rows) Levels of mastery (the columns) e.g.,
Beginner Developed Accomplished
Commentary describing features of work at each level of mastery (the cells) Build from strengths and weaknesses teachers
see in student work over the years
35
Step 3. Means of Assessment - Portfolios
A collection of student work Selective or all inclusive Displays what students can do Best if student is required to reflect upon
and evaluate their own work and learning Caution in program use
Require the same work product from all students at pre-determined points in time.
36
Step 3. Means of Assessment
Attitudinal Learning Surveys of students, graduates,
alumni Affirmation of accomplishments Indirect measure
Exam or task to explore attitudes and apply values
37
Step 3. Means of Assessment
Behavioral and Performance Student Learning Outcomes Specific motor skills Computer applications use Writing talent Computational skills Speaking ability
38
Step 3. Means of Assessment
Behavioral and Performance Case studies Internships Simulations Projects Classroom work in a capstone
course
39
Step 3. Means of Assessment - Final Performances
Most frequently used means Portfolio review Public performance Juried competition
Drawn from “embedded” assignments that require synthesis and application of learning
40
Step 3. Criteria for Success
Identify the criteria before assessment is done
How well “ought” the completers (on average) perform?
Role of dialogue, professional judgment and collective responsibility
41
Step 3. Criteria for Success Primary student performance areas
Minimum overall score, rating, response expected if program is functioning at an acceptable level
Sub areas of student performance More detailed minimum sub-scale or
performance which faculty need to review to ascertain likely cause and make improvement
42
Program Intended Educational Outcomes:
2. Graduates of the Automotive Technology Program will be technically proficient.
3. Employers of the Automotive Technology Program graduates will be rate them competent based on the education received in the program.
Means of Program Assessment and Criteria for Success:
2a. At the close of their final term, 90% of the graduates will be able to identify and correct within a given period of time all of the mechanical problems in five test cars that have been "prepared" for the students by Automotive Technology Program faculty. No single automotive malfunction will fail to be identified and corrected by more than 20% of students.
2b. 80% of Automotive Technology Program graduates will pass the National Automotive Test. On no subscale will students average missing 30% or more of the items.
3. 80% of the respondents to an Employer Survey conducted every 3 years by the college will respond that they find graduates competent and would employ others.
Example: Criteria for Success - Automotive Technology
Program
43
Step 4. Assessing, Assembling Results and Making Meaning of
Them
Assessing and Assembling Results Logistics of doing assessment Recording results
Making meaning The creative tension between “ought”
and “is”
44
Means of Program Assessment and Criteria for Success:
2a. At the close of their final term, 90% of the graduates will be able to identify and correct within a given period of time all of the mechanical problems in five test cars that have been "prepared" for the students by Automotive Technology Program faculty. No single automotive malfunction will fail to be identified and corrected by more than 20% of students.
2b. 80% of Automotive Technology Program graduates will pass the National Automotive Test. On no subscale will students average missing 30% or more of the items.
Summary of Data Collected:
2a. 79% overall success rate. Electrical system malfunction undetected by 34% of students.
2b. Pass rate on National Automotive Test was 83%; however, on “hydraulic theory” subscale students missed an average of 34% of questions.
Example: Recording ResultsGraduates of the Automotive Technology Program will be technically proficient.
45
Step 5. Documenting and Using the Results - “Closing the Loop”
Fitting the response to the results Faculty adjust what we teach Faculty adjust how we teach Faculty adjust the means of
assessment Faculty adjust where we put the bar
of success Document
see J. Nichols, Department Guide & Record Book 46
Summary of Data Collected:
2a. 79% overall success rate. Electrical system malfunction undetected by 34% of students.
2b. Pass rate on National Automotive Test was 83%; however, on “hydraulic theory” subscale students missed an average of 34% of questions.
3. 90% reported willingness to employ graduates, but only 50% of body shops.
Use of Results:
2a. Expanded electrical trouble-shooting component of AT 202 to include automotive electrical systems.
2b. Modified means of teaching hydraulic theory during AT 102 (Basic Auto Systems).
3. Added body shop representative to Advisory Committee and are reviewing curriculum to determine if separate program is needed.
Example: Auto Tech Program(Using Results & Documenting Those
Actions)
47
The Bottom Line, A Call To Learning
A college is an organization composed of learners
(End)
James Nichols, Department Guide & Record Book 48
Expanded Statement of Institutional
Purpose
Mission Statement:
Your Community College is an open-admission, community-based, comprehensive college.
Goal Statement:
Serve persons of allages in preparing for job entry and careers in automotive technology.
Program Intended Educational Outcomes:
1. Graduates of the Automotive Technology Program will be successfully employed in the field.(achievement, not Student Learning Outcome)
2. Graduates of the Automotive Technology Program will be technically proficient.
3. Employers of the Automotive Technology Program graduates will rate them competent based on the education received in the program.
Means of Program Assessment and Criteria for Success:
1a. 50% of the responding graduates of the Automotive Technology Program will report employment in the field on the Graduating Student Survey administered at the time of program completion.1b. 80% of the graduates of the Automotive Technology Program will report employment in the field on the recent Alumni Survey administered one year after graduation.
2a. At the close of their final term, 90% of the graduates will be able to identify and correct within a given period of time all of the mechanical problems in five test cars that have been "prepared" for the students by Automotive Technology Program faculty. No single automotive malfunction will fail to be identified and corrected by more than 20% of students.2b. 80% of Automotive Technology Program graduates will pass the National Automotive Test. On no subscale will students average missing 30% or more of the items.
3. 80% of the respondents to an Employer Survey conducted every 3 years by the college will respond that they find the graduates competent and would employ others .
Summary of Data Collected:
1a. 73% reported employment.
1b. 81% reported employment one year after graduation.
2a. 79% overall success rate. Electrical system malfunction undetected by 34% of students.2b. Pass rate on National Automotive Test was 83%; however, on “hydraulic theory” subscale students missed an average of 34% of questions.
3. 90% reported willingness to employ graduates, but only 50% of body shops.
Use of Results:
1a. Revised criteria for success to 70%.
1b. No action necessary at this time, however, will continue to monitor.
2a. Expanded electrical trouble-shooting component of AT 202 to include automotive electrical systems.
2b. Modified means of teaching hydraulic theory during AT 102 (Basic Auto Systems).
3. Added body shop representative to Advisory Committee and are reviewing curriculum to determine if separate program is needed.
Automotive Technology Program
James Nichols 49
Expanded Statement of Institutional
Purpose
Mission Statement: … an open-admission, community-based, comprehensive college designed to provide inexpensive, quality educational opportunities (transfer, career/technical and continuing education)…
Goal Statement:Serve traditional students seeking the first two years of instruction leading to a bachelor’s degree
Program Intended Educational Outcomes:
1. Students transferring will find courses taken fully accepted as prerequisites for junior and senior level courses at four-year colleges. (not a SLO)
2. After one year of adjustment to the four-year college, the grades of students transferring will be similar to those of students who initially enrolled at four-year colleges.
3. Graduates transferring to a four-year college as a full-time student will complete degrees at almost the same rate as those students originally enrolling at the four-year college. (not a SLO)
Means of Program Assessment and Criteria for Success:
1. Each year one of the college’s six academic departments will contact their counterparts at the three four-year colleges where most students transfer and all courses designed to support the transfer will be found fully accepted as prerequisites by the four-year institutions.
2. Analysis of data concerning the grades of students transferring to four-year colleges will indicate that the differences between the average of transfer students’ GPAs and that of native students is statistically insignificant one year after enrollment at the four-year college.
3. Analysis of data received from each of Your CC three primary transfer student destinations will indicate that the difference in the average number of semesters to the BA degree completion of full-time transfer students from Your CC and students originally enrolling in the four-year college is statistically insignificant.
Summary of Data Collected:
1. Problem noted in introductory accounting where microcomputer technology is now used at two primary transfer institutions.
2. Overall GPA of transfers found comparable to native students, but significantly less in math classes.
3. Degree completing time of Your CC transfer students found to be virtually identical to native students.
Use of Results:
1. Expansion of microcomputer use in introductory accounting problems at Your CC is underway.
2. Math 107 (college algebra) is being strengthened to better relate with calculus at four-year institutions..
3. No change necessary.
Transfer Program, Your CC(CC with no majors and few transfers)