Assessment UpdateAssessment Update2007-082007-08

Presented by Darby KaikkonenPresented by Darby Kaikkonen

Director of Institutional ResearchDirector of Institutional Research

BackgroundBackground

July 2007- Received notice that College would July 2007- Received notice that College would have a focused interim visit in Fall 2008 from have a focused interim visit in Fall 2008 from Northwest Commission (accrediting body) on Northwest Commission (accrediting body) on standards pertaining to assessmentstandards pertaining to assessment

December 2007- College adopted new strategic December 2007- College adopted new strategic plan which included strategy: “Implement plan which included strategy: “Implement systematic academic assessment that evaluates systematic academic assessment that evaluates student learning outcomes in major program student learning outcomes in major program areas and in general education.” areas and in general education.”

Progress in Assessment Process:Progress in Assessment Process:2007-082007-08

Creation of assessment plansCreation of assessment plans General education (College Wide Ability) General education (College Wide Ability)

assessment: Critical Thinkingassessment: Critical Thinking Linking assessment, strategic planning, Linking assessment, strategic planning,

and budgetand budget

Assessment PlansAssessment Plans

ProcessProcess

All departments submitted assessment All departments submitted assessment reports AND assessment plans in reports AND assessment plans in December 2007December 2007

ARC reviewed assessment plans and ARC reviewed assessment plans and provided feedback for improvement to provided feedback for improvement to departmentsdepartments

Assessment plans implemented Spring Assessment plans implemented Spring quarter 2008quarter 2008

Critical Thinking AssessmentCritical Thinking Assessment

BackgroundBackground

Each course that includes critical thinking Each course that includes critical thinking as part of a course outline would include as part of a course outline would include an assessment of that ability on their an assessment of that ability on their 2007-09 assessment plan2007-09 assessment plan

Interdisciplinary task force developed a Interdisciplinary task force developed a common rubric to use for measuring common rubric to use for measuring critical thinking in all types of coursescritical thinking in all types of courses

First quarter of data collection designed to First quarter of data collection designed to be formative and set baselinesbe formative and set baselines

Pilot Test and Reliability AnalysisPilot Test and Reliability Analysis

Pilot Test:Pilot Test: Used for editing purposes and ease of Used for editing purposes and ease of

understandingunderstanding Reliability Analysis:Reliability Analysis:

Tested how consistently evaluators measured Tested how consistently evaluators measured the same work the same work Level of agreement measured using Cronbach’s Level of agreement measured using Cronbach’s Alpha reliability statisticAlpha reliability statisticAcceptable level is .700 or above, Critical Acceptable level is .700 or above, Critical thinking rubric scored .893thinking rubric scored .893

Results from Benchmark QuarterResults from Benchmark Quarter

Nearly 2,000 records of student work from Nearly 2,000 records of student work from 29 different disciplines were evaluated 29 different disciplines were evaluated using the rubricusing the rubric

Student Work Samples by Division:Student Work Samples by Division:Division

975 48.8 48.8 48.8

876 43.9 43.9 92.7

85 4.3 4.3 96.9

61 3.1 3.1 100.0

1997 100.0 100.0

Natural and Applied Sciences

Social Science and Business

Humanities/Communication

Applied Technology

Total

Valid

Frequency Percent Valid PercentCumulative

Percent

Overall Mean Scores by DimensionOverall Mean Scores by Dimension

931 1 4 2.65 1.105

971 1 4 2.50 1.028

354 1 4 2.78 .898

994 1 4 2.79 .970

870 1 4 2.59 1.074

229

Creates, integrates, andevaluates ideas, concepts,and/or information

Analyzes and evaluatesproblems and/or solvesproblems using multipleprocesses

Examines attitudes, values,and assumptions andassesses their implications

Integrates experience,reason, and evidence tomake meaningfulconclusion and judgments

Justifies key results andprocedures, explainsassumptions and reasons

Valid N (listwise)

N Minimum Maximum Mean Std. Deviation

Dimension Mean Scores by Course Level Dimension Mean Scores by Course Level CategoriesCategories

Course Level Category   Dimension A Dimension B Dimension C Dimension D Dimension E

  Overall N N Mean N Mean N Mean N Mean N Mean

Pre College 186 186 2.10 0 * 0 * 0 * 186 2.06

100-120 743 338 2.87 391 2.62 198 2.99 396 2.72 290 2.56

121-140 11 61 2.20 95 2.63 0 * 49 3.27 56 3.00

141-160 102 10 2.50 18 2.89 10 2.70 102 2.55 38 2.63

161-180 146 8 3.50 8 3.38 8 3.63 139 3.24 8 3.38

201-220 467 237 2.73 347 2.19 110 2.35 207 2.57 208 2.80

221-240 58 58 2.71 58 2.69 0 * 0 * 58 2.95

241 and above 185 33 3.42 54 2.83 28 2.79 101 2.93 26 2.92

Key Findings from Baseline Quarter:Key Findings from Baseline Quarter: Dimension C called “Examines attitudes, values, and assumptions Dimension C called “Examines attitudes, values, and assumptions

and assesses their implications in a variety of contexts” was and assesses their implications in a variety of contexts” was evaluated significantly less than each of the other four dimensionsevaluated significantly less than each of the other four dimensions

The overall means scores for each dimension were very similar to The overall means scores for each dimension were very similar to one another, and fell somewhere between a “beginning” and a one another, and fell somewhere between a “beginning” and a “usually or largely” level of demonstration“usually or largely” level of demonstration

There is evidence that students in highest course levels There is evidence that students in highest course levels demonstrate greater critical thinking skills than students in Pre demonstrate greater critical thinking skills than students in Pre college and beginning college level courses; however, not all college and beginning college level courses; however, not all dimensions of critical thinking are significantly positively correlated dimensions of critical thinking are significantly positively correlated with course level. Students who were evaluated at the end of a with course level. Students who were evaluated at the end of a series of progressive courses focusing on critical thinking skills series of progressive courses focusing on critical thinking skills tended to score higher than their counterpartstended to score higher than their counterparts

The scores of each critical thinking dimension are very highly The scores of each critical thinking dimension are very highly correlated with one another, so much so that the scores could be correlated with one another, so much so that the scores could be combined into a single index that represents the overall concept of combined into a single index that represents the overall concept of critical thinking. This means that a student was likely to receive the critical thinking. This means that a student was likely to receive the same or very close to the same score on each dimension of the same or very close to the same score on each dimension of the rubric rubric

See full report on the intranet:See full report on the intranet:“Links” “Institutional Research” “College “Links” “Institutional Research” “College

Wide Ability Assessment: Critical Wide Ability Assessment: Critical Thinking Preliminary Report”Thinking Preliminary Report”

Linking Assessment, Linking Assessment, Strategic Planning, and Strategic Planning, and

BudgetBudget

Institutional Assessment LoopInstitutional Assessment Loop

Conceptual Level

Operational Level

Measurement Level

College Mission and Goals

Philosophy Statements

Student Learning Outcomes

Program Outcomes

Assessment and Results

2007-2010 Strategic Plan2007-2010 Strategic PlanGoal:Goal: South Puget Sound Community College is student-centered. South Puget Sound Community College is student-centered.

Objective:Objective: Offer dynamic and balanced educational programs in a global context. Offer dynamic and balanced educational programs in a global context. Strategy: Support and recognize curricular innovation.Strategy: Support and recognize curricular innovation. Strategy:Strategy: Deliver programs that anticipate and respond to stakeholder needs.Deliver programs that anticipate and respond to stakeholder needs. Strategy:Strategy: Respond to the increasing demand for online and hybrid offerings. Respond to the increasing demand for online and hybrid offerings. Strategy:Strategy: Implement systematic academic assessment that evaluates student learning outcomes in major program Implement systematic academic assessment that evaluates student learning outcomes in major program areas and in general education areas and in general education ObjectiveObjective: Provide comprehensive and effective services to students: Provide comprehensive and effective services to students .. Strategy:Strategy: Develop and implement a one-stop philosophy for services to students.Develop and implement a one-stop philosophy for services to students. Strategy:Strategy: Develop and implement a strategic enrollment management plan. Develop and implement a strategic enrollment management plan. Goal:Goal: South Puget Sound Community College is effective, efficient, and fiscally accountable. South Puget Sound Community College is effective, efficient, and fiscally accountable. Objective:Objective: Develop and employ technology that reflects and supports instructional, internal, and external Develop and employ technology that reflects and supports instructional, internal, and external community needscommunity needs

Strategy:Strategy: Implement and support a learning management system.Implement and support a learning management system. Strategy:Strategy: Focus on maintaining and replacing instructional and institutional equipment to meet the needs of Focus on maintaining and replacing instructional and institutional equipment to meet the needs of tomorrow’s learnerstomorrow’s learners Objective:Objective: Actively create and cultivate connections and partnerships with the community. Actively create and cultivate connections and partnerships with the community. Strategy:Strategy: Address community and business environmental changes through dynamic planning. Address community and business environmental changes through dynamic planning. Objective:Objective: Integrate assessment, planning, and resource allocation at the institutional and program levels. Integrate assessment, planning, and resource allocation at the institutional and program levels. StrategyStrategy: : Implement a systematic instructional program review process that integrates academic assessment and Implement a systematic instructional program review process that integrates academic assessment and planning and influences resource allocation. planning and influences resource allocation. Strategy:Strategy: Systematically assess and improve processes and procedures (institutional effectiveness plan).Systematically assess and improve processes and procedures (institutional effectiveness plan). Strategy:Strategy: Diversify college funding sources. Diversify college funding sources. SStrategytrategy: : Develop and implement a marketing plan. Develop and implement a marketing plan. ObjectiveObjective: Maintain high standards for recruiting, retaining, hiring, and evaluating employees.: Maintain high standards for recruiting, retaining, hiring, and evaluating employees. Strategy:Strategy: Recruit and retain faculty and staff that reflect the college’s diverse population and commitment toRecruit and retain faculty and staff that reflect the college’s diverse population and commitment to excellenceexcellence

Strategy:Strategy: Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a diverse, multi-cultural society. diverse, multi-cultural society.

Strategy:Strategy: Enhance leadership and professional development opportunities.Enhance leadership and professional development opportunities. Strategy:Strategy: Design and implement a staffing model that supports efficient and accessible operations. Design and implement a staffing model that supports efficient and accessible operations. Objective:Objective: Create a sustainable physical environment. Create a sustainable physical environment. Strategy:Strategy: Develop and implement a responsible master plan. Develop and implement a responsible master plan. Strategy:Strategy: Assess current practices and reduce college consumption.Assess current practices and reduce college consumption.

2007-2010 Strategic Plan- Budget Priorities2007-2010 Strategic Plan- Budget PrioritiesGoal:Goal: South Puget Sound Community College is student-centered. South Puget Sound Community College is student-centered.

Objective:Objective: Offer dynamic and balanced educational programs in a global context. Offer dynamic and balanced educational programs in a global context. Strategy: Support and recognize curricular innovation.Strategy: Support and recognize curricular innovation. Strategy:Strategy: Deliver programs that anticipate and respond to stakeholder needs.Deliver programs that anticipate and respond to stakeholder needs. Strategy:Strategy: Respond to the increasing demand for online and hybrid offerings. Respond to the increasing demand for online and hybrid offerings. Strategy:Strategy: Implement systematic academic assessment that evaluates student learning outcomes in major program Implement systematic academic assessment that evaluates student learning outcomes in major program areas and in general education areas and in general education ObjectiveObjective: Provide comprehensive and effective services to students: Provide comprehensive and effective services to students .. Strategy:Strategy: Develop and implement a one-stop philosophy for services to students.Develop and implement a one-stop philosophy for services to students. Strategy:Strategy: Develop and implement a strategic enrollment management plan. Develop and implement a strategic enrollment management plan. Goal:Goal: South Puget Sound Community College is effective, efficient, and fiscally accountable. South Puget Sound Community College is effective, efficient, and fiscally accountable. Objective:Objective: Develop and employ technology that reflects and supports instructional, internal, and external Develop and employ technology that reflects and supports instructional, internal, and external community needscommunity needs

Strategy:Strategy: Implement and support a learning management system.Implement and support a learning management system. Strategy:Strategy: Focus on maintaining and replacing instructional and institutional equipment to meet the needs of Focus on maintaining and replacing instructional and institutional equipment to meet the needs of tomorrow’s learnerstomorrow’s learners Objective:Objective: Actively create and cultivate connections and partnerships with the community. Actively create and cultivate connections and partnerships with the community. Strategy:Strategy: Address community and business environmental changes through dynamic planning. Address community and business environmental changes through dynamic planning. Objective:Objective: Integrate assessment, planning, and resource allocation at the institutional and program levels. Integrate assessment, planning, and resource allocation at the institutional and program levels. StrategyStrategy: : Implement a systematic instructional program review process that integrates academic assessment and Implement a systematic instructional program review process that integrates academic assessment and planning and influences resource allocation. planning and influences resource allocation. Strategy:Strategy: Systematically assess and improve processes and procedures (institutional effectiveness plan).Systematically assess and improve processes and procedures (institutional effectiveness plan). Strategy:Strategy: Diversify college funding sources. Diversify college funding sources. SStrategytrategy: : Develop and implement a marketing plan. Develop and implement a marketing plan. ObjectiveObjective: Maintain high standards for recruiting, retaining, hiring, and evaluating employees.: Maintain high standards for recruiting, retaining, hiring, and evaluating employees. Strategy:Strategy: Recruit and retain faculty and staff that reflect the college’s diverse population and commitment toRecruit and retain faculty and staff that reflect the college’s diverse population and commitment to excellenceexcellence

Strategy:Strategy: Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a diverse, multi-cultural society. diverse, multi-cultural society.

Strategy:Strategy: Enhance leadership and professional development opportunities.Enhance leadership and professional development opportunities. Strategy:Strategy: Design and implement a staffing model that supports efficient and accessible operations. Design and implement a staffing model that supports efficient and accessible operations. Objective:Objective: Create a sustainable physical environment. Create a sustainable physical environment. Strategy:Strategy: Develop and implement a responsible master plan. Develop and implement a responsible master plan. Strategy:Strategy: Assess current practices and reduce college consumption.Assess current practices and reduce college consumption.

2008-09 Equipment Request2008-09 Equipment Request How does your proposal address budget priorities?

(30 points max)

Develop and implement the strategic enrollment management plan* Identify our enrollment problem and implement plan to address. * Support the enrollment goals identified in the 2007-10 SEM plan. * Improve retention - identify why students leave and create strategies to address. * Develop a research plan and build capacity to obtain and use data.

Design and implement a staffing model that supports efficient and effective operations. * Assess new and vacant positions to align with institutional priorities and needs.

Develop and implement marketing plan. Respond to increasing demand for online and hybrid offerings. Maintain and replace equipment to meet needs of today's learning

*Create a replacement plan for equipment - computers, maintenance equipment, carpeting, instructional equipment, printers, copiers and software.

Implement systematic academic assessment that evaluates student learning outcomes in major program areas and in general education.

Implement a systematic instructional program process that integrates academic assessment, planning and influences resource allocations.

2008-09 Equipment Request2008-09 Equipment Request

How does your proposal address budget priorities?

(30 points max) Clear ties to priorities with specific measurable

outcomes The proposal addresses findings based on

assessment results The proposal supports an activity in a 2007-09

assessment plan

Automotive Department Automotive Department ExampleExample

Outcome from Automotive Outcome from Automotive Assessment PlanAssessment Plan

Outcome 1 Measurable Criteria Measurement ToolTime Frame

A program defined outcome is to have employment ready student and graduates of the Automotive Technology Program demonstrate hands-on competency in the 8 ASE task categories: engine repair, automatic transmission & transaxle, manual drive train & axles, suspension & steering, brakes, electrical & electronics, heating and air conditioning, and engine performance.**This outcome is required annually by NATEF for program accreditation**

Students will achieve the following lab completion scores required by NATEF: 95% of the priority 1 tasks, 80% of the priority 2 tasks, and 50% of the priority 3 tasks, at the completion proficiency level of at least 3 on a 1 to 4 scale

Use a task tracking program, completion data, and weekly feedback reports will be provided to students on completed tasks that meet the measured criteria. Quarterly reports will be kept for comparison with goal criteria

Every AUTO course for:2007-08 academic year and 2008-09 academic year

Automotive Equipment RequestAutomotive Equipment Request

Connecting Goals, Strategic Plan, and Connecting Goals, Strategic Plan, and BudgetBudget

College Goal (Student Centered)College Goal (Student Centered)

Strategic Priority (Academic Assessment)Strategic Priority (Academic Assessment)

Budget Decision (Connect to Assessment Plan)Budget Decision (Connect to Assessment Plan)==

Ability to analyze student learning in order to make Ability to analyze student learning in order to make judgments about effectiveness of curriculumjudgments about effectiveness of curriculum

ACCOMPLISHES GOALACCOMPLISHES GOAL

Next Steps…Next Steps…

Assessment Plan UpdatesAssessment Plan Updates

Logan model assessment report is NOT Logan model assessment report is NOT due this falldue this fall

ARC requests that each department ARC requests that each department complete an assessment update formcomplete an assessment update form

Form will be due to your division dean no Form will be due to your division dean no later than October 31, 2008later than October 31, 2008

Assessment Update FormAssessment Update Form2007-09 Assessment Plan:

Mid-Cycle Report - Fall 2008Please complete the following nine items and return them to your division dean no later than October 31, 2008. 1. Name of person submitting report: 2. Department:3. Has your department begun collecting assessment data?

a) Yesb) No

4. If your department has not begun collecting data, in which quarter will the collection begin? a) Fall 2008b) Winter 2009c) Spring 2009

5. Has your department analyzed any assessment data and compiled results?a) Yesb) No

6. If yes, please briefly explain any changes that will be implemented within this assessment cycle as a result of your analysis. If you have already entered your results into a Logan Model grid, feel free to attach it to this report.7. Please briefly describe any anticipated budget needs that your assessments may have identified.8. Please briefly describe any assistance from the Office of Institutional Research that you may need to complete your assessment plan. 9. Any other questions or concerns?

Questions?Questions?