+ from infancy to adolescence: growing assessment from the grassroots larry worster, director of...

+

From Infancy to Adolescence: Growing Assessment from the Grassroots

Larry Worster, Director of Student Services Technology and Assessment

Derrick Haynes, Director of Student Academic Success Center

Metropolitan State College of Denver

+Goals of this Workshop

The student (participant) will understand Metro State’s process for combating assessment resistance.

The student will understand the teaching model for creating change.

The student will be able to layer activities to create small wins.

The student will create rubrics for evaluating assessment activities.

+Infancy: The Discussion

Change in institutional leadership (2003) New Board of Trustees New President New Interim VP of Student Services

Imposed Mandate for Student Services (2004-2005) Director of Research and Development Meetings with directors Discussions Lists of possible outcomes created by

departments

+Infancy: The Framework

Development of Framework (2005-2006) New VPSS New Assistant Vice President for assessment Division-wide Assessment Committee Learning Reconsidered 1 and 2 Outline of MSCD Assessment Handbook Assessment Reports generated (2006) Conceptual Framework

+

+

+

+2006 Report

+2006 Reports: Excerpt

Actual Results: Career Services Student satisfaction survey post 2006 evaluations did not

adequately measure information/skills questions to reflect learning outcomes. It did measure if students were pleased with services.

Improvements Based on Results Improve evaluation forms questions to assess if students

are learning career management skills. Resources needed

Need better questions to assess accurately how well students are learning career management skills.

+Adolescence: Model for Change (2006-2008) Director of Student Services Technology (Oct

2006) VPSS and AVP leave the college (Oct 2006) Interim VPSS appointed (Oct 2006) Director of Technology becomes Director of

Student Services Assessment and Technology He STUDIES! Learning Reconsidered 1 and 2 He reads everything!

+Small Wins

The massive scale on which social problems are conceived precludes innovative action because dysfunctional levels of arousal are induced.

Reformulation of social issues as mere problems allows for a strategy of small wins wherein a series of concrete, complete outcomes of moderate importance build a pattern that attracts allies and deters opponents.

Ironically, people often can't solve problems unless they think they aren't problems.

Weick, Karl E., The Theory of Small Wins (American Psychologist, 1984: 39/1)

+StudentVoice

StudentVoice creates surveys for administration through web and/or PDA.

Data collection and tabulation is automatic.

StudentVoice provides models for survey development.

StudentVoice consultants provide feedback and suggestions for improvement of surveys.

Analysis by filtering, cross-tabulation, and question selection is easy.

Exporting of data and graphs is a feature of the StudentVoice dashboard.

+Model for Change

Change

Practicality

Organic Simplicity

Saturation

Change

Saturation

Organic Growth

Practicality

Simplicity

+Simplicity

Moving from conceptual activities to simple, achievable tasks Write a learning objective

Work individually with directors to develop assessment knowledge Explaining assessment using the relationship

metaphor

Use StudentVoice to teach directors how to create quality surveys. Use StudentVoice expertise to adjust question

language and Likert scales.

+Practicality

Use Learning Reconsidered to categorize student learning objectives.

Host small workshops with directors to write student learning objectives.

Use actual survey projects as case studies to demonstrate how to assess

Share objectives, plans, and results in a private website

+Organic Growth

Carefully select and use of the division assessment committee for peer review

Use champions to model assessment activities and showcase the assessment process.

Highlight the growth of assessment activities within the division in meetings

Provide visibility for activities using different voices to advocate for assessment

Integrate assessment plans and reports in the annual report

+Saturation

Work to saturate the division with leaders who participate in and value assessment activities

Make assessment a part of the job and not an additional burden

Teach the most resistant staff how to use StudentVoice on their own to create surveys

+Change Methodology: The Teaching Model

Teach Professionals about assessment simply and practically, creating organic growth and moving toward saturation Concepts Activities Interpretations Impacts

Use Small Wins to Create Incremental Progress through the process of layering of activity

+Layering of Activities: Learning Objectives

Spring 2007

Construct Learning Objectives with Learning Reconsidered Categories and Sub-Categories One-on-one sessions Small Workshops

Voluntary Announced by Email Maximum of Six Participants Use one person as a case study All participate

Principle: “The student will . . . do or know . . . something”

+Annual Report: Assessment Report

June 2007

Department Reports on the status of assessment activities included as a part of the annual report Include a minimum of one learning objective. What methodology was or will be used to measure? What did analysis show? How have the results been shared with professionals and students? What changes or adjustments in program activities or assessment

methodologies have occurred as a result of the analysis of the assessment?

Principle: Report something, no matter how small. Don’t invent!

+Peer Review of Reports

Summer 2007 Peer Review by Student Services Assessment

Committee Reports and peer review documents on assessment

committee website password protected to create a non-threatening environment

Reports also reviewed by Institutional Assessment Committee with teams of student services and academic affairs staff and faculty

Principle: Use peers to make suggestions for improvement of learning objectives, new learning objectives, ways of analyzing, etc.

+Website

+Large Survey Administration

Fall 2007-Spring 2008 Large Survey Administration

StudentVoice Benchmark Surveys: Campus Recreation, Orientation, Career Services, Profile of the American College Student

NSSE Admissions, Financial Aid, Registrar, Career

Services Principle: Send each student no more than one

unsolicited, general survey. Collect large data samples as a starting point.

+Small Survey Administration

Spring-Fall 2007 Small Survey Administration Use one-on-one sessions to develop survey

questions Use StudentVoice assessment counselors to

improve questions and Likert scales Administer surveys at point of service

Open web administration, PDA, Transcribe paper surveys into SV interface

Administer surveys by email distribution Principle: Create surveys for each department to

address at least one learning objective

+Assessment Plans

January 2008 Require Assessment Plans

Student learning objective or learning objectives:

How will the department track student activity? How will the objective(s) be assessed? How the assessment will be analyzed?

Principle: Expectation of that creating objectives and surveys will result in understanding the role of planning in assessment development

+Assessment Plan Rubrics

March 2008 Develop a set of rubrics by examining SS Assessment Plans for

good practices Student Services Assessment Committee Three questions

What are the criteria that may be evaluated for an aspect of the plan Learning objectives, measure, analytical method, plan for

improvement What is the gold standard for each performance indicator? What is the coal standard for each performance indicator?

Rubrics will be used to evaluate plan in 2008 Reports. Principle: Rubrics clearly communicate administrative expectations.

+Assessment Workshop

Workshop Required for one member of each department

Administer division-wide assessment first

Design curriculum to address divisional weaknesses

Develop website tutorials to support workshop activities

Assess workshop

Write Director of Assessment Assessment Report (modeling)

+Weaknesses

Using Bloom’s Taxonomy for constructing learning objectives

Survey Development Skills Writing good questions Fighting ambiguity Spotting confounded questions

StudentVoice Skills Filtering of responses Cross-tabulation Creating views Creating graphs Exporting data

+Too Many Weaknesses for This Year

Goals for 2008-2009 Workshops: How to Build Proposals for Program Change or Emphasis Tying proposals for program change to assessments Building effective rationales into program changes Building assessment into program changes

+Bloom’s Taxonomy

Bloom’s Verbs for Use in Constructing Student Learning Objectives

1.00Knowledge

Define RepeatRecordListRecallNameRelateUnderline

2.00Comprehension

TranslateRestateDiscussDescribeRecognizeExplainExpressIdentifyLocateReportTellReview

3.00Application

Interpret ApplyEmployUse DemonstrateDramatizePracticeIllustrateOperateScheduleShopSketch

4.00Analysis

DistinguishAnalyzeDifferentiateAppraiseCalculateExperimentTestCompareContrastCriticizeDiagramInspectQuestionRelateDebateInventorySolveExamineCategorize

5.00Synthesis

ComposePlanProposeDesignFormulateArrangeAssembleCollectConstructCreateSet upOrganizeManagePrepare

6.00Evaluation

JudgeAppraiseRateCompareValueReviseScoreSelectChooseAssessEstimatemeasure

+Assessment of Assessment Objectives

+Website Tutorials

Bloom’s Verbs for Use in Constructing Student Learning Objectives

+Report Rubrics

Rubrics for the Assessment Report announced

Assessment Plan Rubrics will be used for 2008 Plans

Assessment Plan and Report Rubrics will be used for 2009 Plans and Reports

Rubrics will be adjusted by the assessment committee after each cycle

+Workshop Outcomes

+Rubrics

Student Learning Objectives: Scope

a: Scope

5 Department has learning objectives developed for all aspects of its program.

4 Department has learning objectives developed for two or more aspects of its program.

3 Department has defined a learning objective for an aspect of its program.

2 Department has an objective stated, however it is not stated as a learning objective.

1 Department has no learning true learning objectives defined.

Comments:

+Rubric Workshop Exercise

Pick an aspect of the Report Student Learning Objectives Measures Analysis Proposals for Program Change

Create Criteria

Create Performance Indicator Sentences Create the Gold Standard Create the Coal Standard Fill in to create a five-point scale

+Thank you!

Larry Worster, Director of Student Services Technology and Assessment

Derrick Haynes, Director of Student Academic Success Center

Metropolitan State College of DenverAll materials available at:

http://www.mscd.edu/~ssac

worster@mscd.edu

haynesd@mscd.edu