Using CAS Standards and Review Process for Improving Student Learning and Program Outcomes

Patricia CarrettaGeorge Mason University

CAS Officer-at-LargeNACE Representative to CAS

NASPA 2008 International Assessment & Retention Conference

Purpose

CAS Standards and Review ProcessIntegrating CAS into assessment of

programs and services and student learning outcomes

Role and Use of CAS Standards:

1. Student Affairs / Student Services at the division level?

2. Student Affairs / Student Services at the departmental or program level?

3. Other?4. Have looked at the CAS Standards for at

least one functional area?5. Have participated in a department/program

review process using CAS standards?6. Have used CAS standards for other

purposes?

Participant Outcomes

Define the role of CAS self-assessment / review process within a comprehensive assessment plan

List the seven step process of conducting a CAS review Appreciate the value of engaging a variety of

institutional stakeholders in the review process Strategize how to conduct a comprehensive

assessment of programs, services and student learning using CAS Standards

Formulate an assessment plan Understand the importance of assessing and reporting

on input data as well as student learning and development outcomes as a strategies for improving learning and program quality

About CAS:

The mission of the Council for the Advancement of Standards in Higher Education (CAS) is to promote the improvement of programs and services to enhance the quality of student learning and development.

CAS is a consortium of professional associations who work collaboratively to develop and promulgate standards and guidelines and to encourage self-assessment.

CAS Resources

CAS Professional Standards, 2006 editionSelf-Assessment GuidesCAS Statement of Shared Ethical

PrinciplesCAS Characteristics of Individual

ExcellenceFrameworks for Assessing Learning and

Development Outcomes

CAS Guiding Principles

Students and Their InstitutionsDiversity and MulticulturalismOrganization, Leadership, and Human

ResourcesHealth Engendering EnvironmentsEthical Considerations

Organization of CAS Standards

Mission Program Leadership Human Resources Ethics Legal

Responsibilities Equity and Access Diversity

Organization and Management

Campus and External Relations

Financial Resources Technology Facilities and

Equipment Assessment and

Evaluation

Student Learning & Development

Must promote student learning and development outcomes

Must identify relevant and desirable student learning & development outcomes

Must assess outcomes Must provide evidence of their impact on

student learning and development Must articulate how they contribute to or

support student learning and development

CAS: Student Learning & Development Outcomes

Knowledge acquisition, construction, integration and application

Cognitive complexityIntrapersonal developmentInterpersonal competenceHumanitarianism & civic engagementPractical competence

Student Learning & Development Outcomes

Consistent with the institutional mission, programs and services must identify relevant and desirable student learning and development outcomes from among the six domains. When creating opportunities for student learning and development, programs and services must explore possibilities for collaboration with faculty members and other colleagues.

Programs and services must assess relevant and desirable student learning and development outcomes and provide evidence of their impact on student learning and development. Programs and services must articulate how they contribute to or support student learning and development in the domains not specifically assessed.

Essential Elementsin the CAS Approach

Culture that values involvement of all its members

Quality indicatorsUse of standards and guidelinesWillingness/capacity:

To examine itself and assemble resultsTo report and use the results

Uses of CAS Standards

Credibility, accountability, improvement: Program & service improvement; measures of quality

and effectiveness; measures of impact on learning Design of new programs & services Restructuring / reorganization Change in leadership Institutional self-studies Preparation for accreditation

Staff development Academic preparation

CAS: Standards and Guidelines

STANDARD (BOLD TYPE; AUXILARY VERB MUST)To support the institution’s mission and goals, Career services must provide faculty and staff and administrative units with information, guidance, and support on career development and employment issues and linkages with the broader community.

GUIDELINE (LIGHT-FACED TYPE; VERB SHOULD)Career Services should support faculty and staff and administrative units by: Identifying and disseminating information on employment

trends, top employing organizations, and co-op and internship sponsors.

Comprehensive Assessment Plan

Includes:Assessment of programs and servicesAssessment of student outcomes,

including student learning and development

Program Evaluation

Is the program or service functioning effectively to achieve its mission?

What evidence is available to support the determination?

How is evidence used to make program decisions?

Student Outcomes Assessment

What is the effect of our work on students?

How are they different as a result of interacting with our programs and services?

How do we demonstrate their learning?How do we know? Measure?

The CAS SAG: A Self-Assessment Guide

Translates CAS standards into an effective workbook / evaluative format

Promotes program self-assessment and development

Informs on program strengths and weaknesses

Supports professional staff development Leads to enhanced programs and services

that benefit student learning and development

CAS Self-Assessment Process

1. Establish and prepare the Self-Assessment Team

2. Initiate the Self Study3. Compile and review Documentary and

Evaluative Evidence4. Identify Discrepancies5. Determine appropriate Corrective Action6. Recommend Steps for Program

Enhancement7. Prepare an Action Plan

The Self-Assessment Team

Appoint and Train the Team: Size of team Composition: internal, external Coordinator/Leader Establish team ground rules Discuss meaning of each standard Establish team’s inter-rater reliability Encourage team discussion; expect

disagreements; commit to consensual resolution

Team Actions

Decide whether to include guidelines or other measures that go beyond the standards

Gather and analyze quantitative and qualitative data

Individuals rate each and every criterion measure

Obtain additional documentary evidence if required to make an informed team decision

Complete the assessment, ratings & action plan worksheets

Evaluative EvidenceSTUDENT RECRUITMENT MATERIALS:

Brochures & other program information Participation policies & procedures

PROGRAM DOCUMENTS: Mission statements; program purpose & philosophy statements Catalogs & related materials Staff & student manuals; policies & procedures statements

ADMINISTRATIVE DOCUMENTS: Organization charts; student & staff profiles Financial resource statements & budgets Annual reports

STAFF ACTIVITY REPORTS: Curriculum vitae & resumes; professional activity Service to other programs, departments, or community

STUDENT ACTIVITY REPORTS: Portfolios, developmental transcripts, resumes Reports of student service

RESEARCH & EVALUATION DATA: Needs assessments & self-studies Program evaluation; graduate & follow-up studies

Quantitative Data

Summary of objective responses on a questionnaire or program evaluation

Statistics about use by students and various population subgroups

Needs assessments, follow-up studies, and self-study reports

Institutional research reports and fact books Comparative or benchmark data at local,

regional or national level

Qualitative Data

Focus group informationWritten summaries of responses to

open-ended questions in interviews and on evaluations

Client satisfaction surveys, self-reports, and written comments, both solicited and unsolicited

Using the SAGs

SAGs offer a ready format for evaluationIn most instances, there are multiple

criterion statements for each standardEach criterion measure focuses on a

particular aspect of the standard, allowing raters to express more detailed and specific judgments

Criterion Measure Rating Scale

ND 1 2 3 4 NRNot Done Not met Minimally Well Fully Not Rated

Met Met Met

Using this scale, consider each criterion statement and decide…•The extent to which each criterion measure has been met by the program or service

ND 1 2 3 4 NRNot Done Not met Minimally Well Fully Not Rated

Met Met Met

Assessment Criteria ExamplePart 6: FINANCIAL RESOURCESCounseling Services (CS) must have adequate funding to accomplish its mission and goals. Funding priorities

must be determined within the context of the stated mission, goals, objectives, and comprehensive analysis of the needs and capabilities of students and the availability of internal or external resources.

CS must demonstrate fiscal responsibility and cost effectiveness consistent with institutional protocols.

ND 1 2 3 4 NR Not Done Not Met Minimally Met Well Met Fully Met Not Rated

PART 6. FINANCIAL RESOURCES (Criterion Measures) Rating Scale NOTES6.1 The program has adequate funding to accomplish its mission and goals. ND 1 2 3 4 NR

6.2 Funding priorities are determined within the context of program mission, student needs, and available fiscal resources. ND 1 2 3 4 NR

6.3 The program demonstrates fiscal responsibility and cost effectiveness consistent with institutional protocols. ND 1 2 3 4 NR

Part 6: Financial Resources Overview Questions

A. What is the funding strategy for the program?

B. What evidence exists to confirm fiscal responsibility and cost-effectiveness?

CAS: Work Form AAssessment, Ratings, and Significant Items

CASWork Form A

Assessment, Ratings, and Significant ItemsINSTRUCTIONS:This work form should be completed following individual ratings of the participants. For each of the 13 Parts, identify (circle) the criterion measure item number(s) in the column labeled for which there is a substantial rating discrepancy (two or more ratings apart). Items not circled should reflect consensus among judges that practice in that area is satisfactory. Items where judgment variance occurs need to be discussed thoroughly by team members. Follow this action by determining which practices (criterion measures) can be designated as “excellent” or “unsatisfactory” and record them in the Step One column. In Step Two, list the items requiring follow-up action including any criterion measure rated as being unsatisfactory by any reviewer.

Step One

Part Items Excellent Unsatisfactory

1. Mission 1.1a1.4

1.1b1.5

1.1c1.6

1.2 1.3

2. Program 2.1 2.4.3 2.4.8 2.4.132.6 2.8d2.11

2.2 2.4.4 2.4.9 2.4.142.7 2.8e2.12

2.32.4.52.4.102.4.152.8a2.8f2.13

2.4.12.4.62.4.112.4.162.8b2.9

2.4.22.4.72.4.122.52.8c2.10

CAS: Work Form A continued

Step Two: List item number(s) for each Part determined to merit follow-up and describe the practice weaknesses that require attention

1.

2.

3.

4.

5.

CAS Work Form B: Follow Up Actions

INSTRUCTIONS:The purpose of this work form is to begin the planning for action to be taken on practices judged to merit follow-up (See Step 1, Work Form A). In Step Three, transfer short descriptions of the practices requiring follow-up and detail these items using the table format provided.

Step Three: Describe the current practice that requires change and actions to initiate the change

Practice Description

Corrective Action Sought

Task Assigned To

Timeline Due Dates

CAS Work Form C:Summary Action Plan

Step Four: This form concludes the self-assessment process and calls for action to be taken as a consequence of study results. Write a brief action plan statement in the spaces below for each Part in which action is required.

Part 1: Mission

Part 2: Program

Part 3: Leadership

Part 4: Human Resources

Part 5: Ethics

Closing the loop

Purpose & scope of study is defined Self-study team is trained Self-study is conducted Evaluative evidence is collected Discrepancies are identified Appropriate actions are determined Special actions for program enhancement are

recommended Action plan is developed and communicated

Student Outcomes Assessment

What is the effect of our work on students?

How are they different as a result of interacting with our programs and services?

What have students learned?How have they developed?

Why Focus on Student Learning?

AccountabilityAlignment with institution’s missionConsistent with language of the

academyQuality improvementMore effective learning Better defined and more effective

planning and strategy development

Defining Learning Outcomes

“Outcomes usually identify growth in some dimension of knowing, being or doing.”

Susan Komives & Sarah Schoper, “Developing Learning Outcomes,” Learning Reconsidered 2 (2006)

Knowledge (cognitive)Attitude (affective)Skill (psychomotor)

CAS: Student Learning & Development Outcomes

Knowledge acquisition, construction, integration and application

Cognitive complexityIntrapersonal developmentInterpersonal competenceHumanitarianism & civic engagementPractical competence

Plan for Assessing Student Learning Outcomes

Review/revise mission Identify major services, programs, activities Specify desired student outcomes Identify information needed to show student

learning Identify existing and new data needed Develop process for measuring outcomes Identify ways to use findings to

change/improve

Writing Learning Outcomes

Identify words/phrases that describe what an ideal program participant or college graduate would be able to:Know – Be – Do … as a result of

engaging with your program, activity or service.

What would he/she learn from the experience?

Student Outcomes MatrixMission: The Multicultural Research and Resource Center (MRRC) consolidates and disseminates research on intercultural and cross-cultural inclusion, and collaborates with academicians to develop curriculum based in related theory. Additionally, the Center assists students with research and practice regarding multiculturalism in higher education; and administers an international model of leadership development training university-wide and in partnership with other entities in the regional, national and international community.

CAS Student Outcome Domain: Humanitarianism and Civic Engagement

Major Services, Programs, Activities

Dimensions of Outcome Domains:Understanding and appreciation of cultural and human differences

Social responsibility

Global Perspective

Sense of civic responsibility

•Classroom presentations•Curriculum/faculty development•Campus speakers•Applied research on issues affecting multiculturalism, community and handling controversial issues•Building Bridges partnership•NCBI training

Desired outcomes:Understands one’s own identity and culture; seeks involvement with people different from oneself; articulates the advantages and impact of a diverse society; identifies systematic barriers to equality and inclusiveness, then advocates and justifies means for dismantling them; in interactions with others, exhibits respect and preserves the dignity of others

Desired outcomes:Recognizes social systems and their influence on people; Appropriately challenges the unfair or uncivil behavior of individuals or groups; Participates in service/volunteer activities that achieve reciprocity; Articulates the values and principles involved in personal decision making; Affirms and values the worth of individuals and communities

Desired outcomes:Understands and analyzes the interconnectedness of societies worldwide; Demonstrates effective stewardship of human, economic, and environmental resources

Desired outcomes:Demonstrates consideration of the welfare of others in decision making; Critiques decisions and learns from mistakes; Engages in principled dissent; understands and participates in relevant governance systems

Writing Learning Outcomes

Describe the learner – characteristics of users/participants of programs and services. What knowledge, skills, experiences do they start with?

Determine acceptable levels of performance

Consider how you would provide evidence of achievement

Ways to measure learning outcomes

Surveys, questionnaires, self-report (indirect) Observations Interviews Focus groups Reflective journals Performance evaluations Pre-test/post-test Portfolios Capstone projects Review of project/performance by peers and

internal/external reviewers

Tips, suggestions

Identify all your program outcomes Rank outcomes based on importance

Activity outcome Program outcome Institutional or division outcome

Identify where learning opportunities are occurring (mapping)

Design and implement assessment cycle – one outcome at a time

Remember:

Start smallStart doableStart narrowStart targetedStart!

One program, one workshop, one desired outcome, one measured tool

Additional CAS Resources

www.cas.eduStandards book

Prologue & Context with “How to” information

FALDOs – book and CD formats as package

SAGs – CD or individually