learning from peer evaluators - amazon s3...nuria m. cuevas, ph.d. ([email protected]) and alexei...

2016 SACSCOC Institute on Quality Enhancement and Accreditation Concurrent Session – Learning from Peer Evaluators: Commonly Cited Issues with Outcomes Assessment in Academic | Student Support Services

(CS 3.3.1.3) and Administrative Units (CS 3.3.1.2)

Nuria M. Cuevas, Ph.D. ([email protected]) and Alexei G. Matveev, Ph.D. ([email protected]) July 18, 2016 || Grapevine, TX

1

Learning from Peer Evaluators: Commonly Cited Issues with Outcomes Assessment in Academic & Student Support Services (CS 3.3.1.3)

and Administrative Support Services (CS 3.3.1.2)

Nuria M. Cuevas, Ph.D. and Alexei G. Matveev, Ph.D.

Southern Association of Colleges and Schools Commission on Colleges (SACSCOC)

1

SESSION OUTLINE

I. Background

II. CS 3.3.1 : Structure | Compliance Components

III. CS 3.3.1.2-3: Key Non-Compliance Factors

IV. Summary and Q & A

Handout | Exercise

2

I. BACKGROUND

3




2

CS 3.3.1.2-3

“The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: …

3.3.1.2 administrative support services

3.3.1.3 academic and student support services…”

(Principles of Accreditation, 2012)

4

5

0%

20%

40%

60%

80%

100%

Off-Site On-Site C&R

CS 3.3.1.2: Reaffirmation Review - (% of institutions found to be in non-compliance) ||

PRELIMINARY DATA

2011 2012 2013 2014 2015

6

0%

20%

40%

60%

80%

100%

Off-Site On-Site C&R

CS 3.3.1.3: Reaffirmation Review - (% of institutions found to be in non-compliance) ||

PRELIMINARY DATA

2011 2012 2013 2014 2015




3

Discussion Outline

• Intent and structure of CS 3.3.1

• Common non-compliance factors identified by peer evaluators for CS 3.3.1.2-3

– Content analysis of ~400 2011-2015 peer evaluation committee reports

• Illustrative quotes (handout)

• Selected pointers

7

II. Comprehensive Standard (CS) 3.3.1: Structure

8

CS 3.3.1

• Who? – Institution

• Action? – Identifies, assesses, and provides evidence

• Identifies what? – Expected program outcomes

• Assesses what? – Extent of achievement

• Provides evidence of what? – Improvement based on analysis of the results

9




4

Compliance Components

Embedded in the wording of the Principles – (and frequently signaled by numbers, commas, and the use of

compound modifiers),

the compliance components are

– the discrete elements that must be addressed for each requirement and standard.

• Resource Manual, Handbook for Institutions Seeking

Reaffirmation – Guide, NOT a checklist

10

CS 3.3.1.2-3

“The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: …

3.3.1.2 administrative support services

3.3.1.3 academic and student support services…”

(Principles of Accreditation, 2012)

11

12

Identification of Expected Outcomes

Data Collection

Use of Assessment Findings for Unit | Service Improvement

Identify appropriate ways to measure these outcomes

Evaluate what the results mean




5

13

IN

ST

IT

UT

IO

NA

L M

IS

SIO

N

Perfo

rm

an

ce

Sta

nd

ard

s

Ach

iev

em

en

t

Da

ta

Meaningful?

Specific?

Measurable?

Expected

Program

OUTCOME #

Pro

gram

M

issio

n

Da

ta

Co

llectio

n

Meth

od

olo

gy

Im

pro

vem

en

ts

Plans

Actions

An

aly

sis &

Ju

dg

men

t

III. CS 3.3.1.2-3: Commonly Cited Issues by Peer Evaluators

14

Guiding Research Question

• “THINK LIKE A PEER EVALUATOR”

–How do peer evaluators think when reviewing institutional claims and evidence of outcomes assessment in support services?

• On what factors do peer evaluators tend to focus when providing evaluative feedback to institutions on CS 3.3.1.2-3?

SNAPSHOT of collective peer evaluators’ reasoning: Dynamic and emerging interpretations and applications within the framework established by the Principles

15




6

Research Project Overview

Qualitative Content Analysis (work-in-progress)

• 2011–2015 reaffirmation classes

o Primarily off-site reaffirmation committee reports

– Findings of non-compliance

• Key non-compliance factors

16

Common Non-Compliance Factors Preliminary Findings

17

A. Sample

B. Statements of Expected Outcomes

C. Data Collection

D. Use of Results

G. Presentation

F. P

atte

rn

E. A

lignm

ent

A. Sample (if sampling is used)

Key Non-compliance Aspects

• Non-representative sample of unit assessment reports [cross-referenced with F. Pattern]

o Lack of a clear definition of a “student support service” or “administrative support service” unit

– Inconsistent listing [cross-referenced with G. Presentation]

• No rationale for the sample

• Unit assessment reports are not reflective of student population

o No indication that online and off-campus students are included in assessments (if applicable)

18




7

A. Sample Exercise / Selected Pointers

19

B. Statements of Expected Outcomes Key Non-compliance Aspects

• Statements of expected outcomes are not articulated

- Not reflective of the scope of unit activities

• Outcomes are not focused

• Statements of expected outcomes reflect unit activities rather than impact of these activities

20

B. Statements of Expected Outcomes Exercise / Selected Pointers

21




8

C. Data Collection Key Non-compliance Aspects

• Reference points (quantitative or qualitative expected

performance levels, standards, benchmarks…) are not pre-set

• Data are not presented

• Assessments are not conducted

22

C. Data Collection Exercise / Selected Pointers

23

D. Use of Results Key Non-compliance Aspects

• Improvement plans and actions

–Not listed

–Not specific

–Not enacted • Not documented [cross-referenced with G.

Presentation]

24




9

D. Use of Results Exercise / Selected Pointers

25

E. Alignment Key Non-compliance Aspects

• Logical (direct or indirect) relationship between - Statements of expected outcomes and data collection

methodology [cross-referenced with C. Data Collection]

- Assessment results/analysis and improvement plans [cross-referenced with D. Use of Results]

• Overall coherence of assessment systems and processes [cross-referenced with F. Pattern and G. Presentation]

26

E. Alignment Exercise / Selected Pointers ers

27




10

F. Institutional Pattern Key Non-compliance Aspects

• Lack of systemic implementation - Across the units

• Lack of systematic implementation - Over time

• Lack of consistent implementation - Dramatic “hills and valleys” in quality

28

F. Pattern Exercise / Selected Pointers

29

G. Presentation Key Non-compliance Aspects

• Links are not working

• Narrative is not analytic / coherent

• Supporting evidence is poorly organized

30




11

G. Presentation Exercise / Selected Pointers

31

Common Non-Compliance Factors Preliminary Findings

32

A. Sample

B. Statements of Expected Outcomes

C. Data Collection

D. Use of Results

G. Presentation

F. P

atte

rn

E. A

lignm

ent

Emerging Overarching Theme

• MATURITY of institutional effectiveness (IE) systems, processes, and data

–Sufficient information used as a basis for sound decision making

33




12

IV. SUMMARY AND Q&A

34

Peer Evaluators: Overall Themes

35

All compliance components

Alignment or logical linkages

Sufficient description and

analysis

Evidence of systemic, systematic, and

consistent implementation

Overall Pointers

36

• Active leadership support and engagement

• Consistent/ongoing attention to assessment systems and processes

• Staff assessment expertise and professional development

• Effective and meaningful reporting




13

Selected Resources: General • SACSCOC evaluator online training modules

– www.sacscoc.org/WebBasedTrainingModules.asp

• Publications

– Banta, T.W., & Palomba, C.A. (2015). Assessment essentials: Planning, implementing, and improving assessment in higher education, 2nd ed. San Francisco: Jossey-Bass.

– Middaugh, M.F. (2010). Planning and assessment in higher education: Demonstrating institutional effectiveness. San Francisco: Jossey-Bass.

– Suskie, L. (2014). Five dimensions of quality: A common sense guide to accreditation and accountability. San Francisco: Jossey-Bass. 37

Selected Resources: Administrative Support Services

– Middaugh, M.F. (2010). Measuring administrative effectiveness. In: Planning and assessment in higher education: Demonstrating institutional effectiveness, pp. 157-172. San Francisco: Jossey-Bass.

– Miller, B.A. (2007). Assessing organizational performance in higher education. San Francisco: Jossey-Bass.

– Nichols, J.O. & Nichols, K. (2005). The department's guide to assessment implementation in administrative and educational support units. New York: Agathon Press.

38

Selected Resources: Student Affairs

– Bingham, R.P., Bureau, D., & Duncan, A.G. (Eds.) (2015). Leading assessment for student success: Ten tenets that change culture and practice in student affairs. Sterling, VA: Stylus

– Bresciani, M. J., & Hickmott, J. (2010). Demonstrating student success: A practical guide to outcomes-based assessment of learning and development in student affairs. Sterling, VA: Stylus.

– Henning, G.W., & Roberts, D. (2016). Student affairs assessment: Theory to practice. Sterling, VA: Stylus.

– Schuh, J.H., Biddix, J.P., Dean, L.A., & Kinzie, J. (2016). Assessment in student affairs, 2nd ed. San Francisco: Jossey-Bass. 39




14

Selected Resources: Library / Learning Resources

– Gilchrist, D., & Oakleaf, M. (2012). An essential partner: The librarian’s role in student learning assessment (NILOA Occasional Paper No. 14). Urbana, IL: University of Illinois and Indiana University, NILOA.

– Hernon, P., & Dugan, R.E. (2002). An action plan for outcomes assessment in your library. Chicago: American Library Association.

40

41

learning from peer evaluators - amazon s3...nuria m. cuevas, ph.d. ([email protected]) and alexei...

Documents