EVAL 6000: Foundations of Evaluation

Carl D. Westine & Dr. Chris L. S. Coryn December 2, 2010

Agenda

• The Program Evaluation Standards (45 min)– Overview of JCSEE and the standards– The “new” 30 standards– 2nd Edition vs. 3rd Edition– Research on overlaps across standards– Activity on 3rd Edition

• Sufflebeam and Coryn 2010 (10 min)• Stufflebeam 2001 (10 min)• Break (15 min)• Activity: Hard Won Lessons (60 min)

The JCSEE

• The Joint Committee for Standards on Educational Evaluation was created in 1975, and currently oversees the maintenance and updates to the Program Evaluation Standards

• Made up of representatives from 12-17 professional societies

• WMU folks involved in the JCSEE: Daniel Stufflebeam, James Sanders, Arlen Gullickson

• Currently Daniela Schröter is becoming more involved with the revision process (our point of contact)

What are Standards

• JCSEE defined an evaluation standard as a “principle commonly agreed to by experts in the conduct and use of evaluation, that when implemented will lead to greater evaluation quality” (JCSEE, 2010, p. 292)

• Focus on North America but other countries have been ada/opting these

• Other standards exist (GAO, Guiding Principles, etc.)

• For educational programs but… education is relied upon by everything

Early Standards

• The first edition was published in 1981.– Contained 30 standards– Defined the groups (Utility,

Feasibility, Propriety, and Accuracy)• Spearheaded by Dan Stufflebeam

(WMU) and a collection of education and evaluation experts

Second Edition Standards

• The second edition was published in 1994– Contained 30 standards– Maintained same group structure

• Achieved status of being accredited as official standards though the American National Standards Institute (ANSI)

• Update led by James Sanders (WMU)

Current Happenings

• Transition from the 2nd Edition to the 3rd Edition

• Third Edition is being published, and copies can be ordered through the JCSEE website:

www.jcsee.org• Don Yarrborough at the University of

Iowa oversees the JCSEE and the development of the newest standards

The Latest Edition

• 3rd Edition• Took more than 5 years to develop and

finalize (several delays in publishing)• The next update has already begun

– JCSEE continuously takes suggestions and feedback from users

– Goal to update standards more frequently

Organization of the PES Book

• Applying the Standards• Functional Table of Standards

– Shows which standards are relevant at points along the evaluation continuum

• Standards– Group Overview/Scenario– Standard Statements– Rationale/Clarification (Overview)– Implementing (Guidelines)– Hazards (Common Errors)– Application(s)– Documentation

The Program Evaluation Standards

• The Program Evaluation Standards for educational evaluations

• Should be used for:– Guiding an evaluation effort (formative)– Assessing the quality of educational

programs (summative)– Assessing the quality of an evaluation of

an educational program (metaevaluation)

– A tool to help policy makers understand evaluations

– [Research on evaluation] (in my own opinion)

The Program Evaluation Standards

• There are now 5 categories totaling 30 individual standards

• Designed to ensure that an evaluation will…– Utility (7 -> 8): “… serve the information needs of

intended users.”– Feasibility (3 -> 4): “… be realistic, prudent, diplomatic,

and frugal.”– Propriety (8 -> 7): “… be conducted legally, ethically,

and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.”

– Accuracy (12 -> 8): “… reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.”

– Evaluation Accountability (0 -> 3): be well documented and held subject to internal and external evaluation (JCSEE, 2010).

Utility Standards

• U1 Evaluator Credibility Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context.

• U2 Attention to Stakeholders Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation.

• U3 Negotiated Purposes Evaluation purposes should be identified and continually negotiated based on the needs of stakeholders.

• U4 Explicit Values Evaluations should clarify and specify the individual and cultural values underpinning purposes, processes, and judgments.

Utility Standards (cont.)

• U5 Relevant Information Evaluation information should serve the identified and emergent needs of stakeholders.

• U6 Meaningful Processes and Products Evaluations should construct activities, descriptions, and judgments in ways that encourage participants to rediscover, reinterpret, or revise their understandings and behaviors.

• U7 Timely and Appropriate Communicating and Reporting Evaluations should attend to the continuing information needs of their multiple audiences.

• U8 Concern for Consequences and Influence Evaluations should promote responsible and adaptive use while guarding against unintended negative consequences and misuse.

Feasibility Standards

• F1 Project Management Evaluations should use effective project management strategies.

• F2 Practical Procedures Evaluation procedures should be practical and responsive to the way the program operates.

• F3 Contextual Viability Evaluations should recognize, monitor, and balance the cultural and political interests and needs of individuals and groups.

• F4 Resource Use Evaluations should use resources effectively and efficiently.

Propriety Standards

• P1 Responsive and Inclusive Orientation Evaluations should be responsive to stakeholders and their communities.

• P2 Formal Agreements Evaluation agreements should be negotiated to make obligations explicit and take into account the needs, expectations, and cultural contexts of clients and other stakeholders.

• P3 Human Rights and Respect Evaluations should be designed and conducted to protect human and legal rights and maintain the dignity of participants and other stakeholders.

• P4 Clarity and Fairness Evaluations should be understandable and fair in addressing stakeholder needs and purposes.

Propriety Standards (cont.)

• P5 Transparency and Disclosure Evaluations should provide complete descriptions of findings, limitations, and conclusions to all stakeholders, unless doing so would violate legal and propriety obligations.

• P6 Conflicts of Interests Evaluations should openly and honestly identify and address real or perceived conflicts of interests that may compromise the evaluation.

• P7 Fiscal Responsibility Evaluations should account for all expended resources and comply with sound fiscal procedures and processes.

Accuracy Standards

• A1 Justified Conclusions and Decisions Evaluation conclusions and decisions should be explicitly justified in the cultures and contexts where they have consequences.

• A2 Valid Information Evaluation information should serve the intended purposes and support valid interpretations.

• A3 Reliable Information Evaluation procedures should yield sufficiently dependable and consistent information for the intended uses.

• A4 Explicit Program and Context Descriptions Evaluations should document programs and their contexts with appropriate detail and scope for the evaluation purposes.

Accuracy Standards (cont.)

• A5 Information Management Evaluations should employ systematic information collection, review, verification, and storage methods.

• A6 Sound Designs and Analyses Evaluations should employ technically adequate designs and analyses that are appropriate for the evaluation purposes.

• A7 Explicit Evaluation Reasoning Evaluation reasoning leading from information and analyses to findings, interpretations, conclusions, and judgments should be clearly and completely documented.

• A8 Communication and Reporting Evaluation communications should have adequate scope and guard against misconceptions, biases, distortions, and errors.

Evaluation Accountability Standards

• E1 Evaluation Documentation Evaluations should fully document their negotiated purposes and implemented designs, procedures, data, and outcomes.

• E2 Internal Metaevaluation Evaluators should use these and other applicable standards to examine the accountability of the evaluation design, procedures employed, information collected, and outcomes.

• E3 External Metaevaluation Program evaluation sponsors, clients, evaluators, and other stakeholders should encourage the conduct of external metaevaluations using these and other applicable standards.

Utility (Old vs. New)

2nd Edition (OLD) 3rd Edition (NEW)

U1 Stakeholder Identification U1 Evaluator Credibility

U2 Evaluator Credibility U2 Attention to Stakeholders

U3 Information Scope and Selection U3 Negotiated Purposes

U4 Values Identification U4 Explicit Values

U5 Report Clarity U5 Relevant Information

U6 Report Timeliness and Dissemination

U6 Meaningful Processes and Products

U7 Evaluation Impact U7 Timely and Appropriate Communicating and Reporting

U8 Concern for Consequences and Influences

-1+2=+1

Feasibility (Old vs. New)

2nd Edition (OLD) 3rd Edition (NEW)

F1 Practical Procedures F1 Project Management

F2 Political Viability F2 Practical Procedures

F3 Cost Effectiveness F3 Contextual Viability

F4 Resource Use

+1

Propriety (Old vs. New)

2nd Edition (OLD) 3rd Edition (NEW)

P1 Service Orientation P1 Responsive and Inclusive Orientation

P2 Formal Agreements P2 Formal Agreements

P3 Rights of Human Subjects P3 Human Rights and Respect

P4 Human Interactions P4 Clarity and Fairness

P5 Complete and Fair Assessment P5 Transparency and Disclosure

P6 Disclosure of Findings P6 Conflicts of Interest

P7 Conflict of Interest P7 Fiscal Responsibility

P8 Fiscal Responsibility

-1

Accuracy (Old vs. New)

2nd Edition (OLD) 3rd Edition (NEW)

A1 Program Documentation A1 Justified Conclusions and Decisions

A2 Context Analysis A2 Valid Information

A3 Described Purposes and Procedures A3 Reliable Information

A4 Defensible Information Sources A4 Explicit Program and Context Descriptions

A5 Valid Information A5 Information Management

A6 Reliable Information A6 Sound Designs and Analysis

A7 Systematic Information A7 Explicit Evaluation Reasoning

A8 Analysis of Quantitative Information A8 Communication and Reporting

A9 Analysis of Qualitative Information

A10 Justified Conclusions

A11 Impartial Reporting

A12 Metaevaluation

-3+1=-2

Evaluation Accountability (Old vs. New)

2nd Edition (OLD) 3rd Edition (NEW)

A3 Described Purposes and Procedures

EA1 Evaluation Documentation

A12 Metaevaluation EA2 Internal Metaevaluation

EA3 External Metaevaluation

+1

Significant Changes

• Managing focus (Process Use)– Project– Information

• Less emphasis on “actionable” statements

• Evaluation Accountability– New standard group– Transparency/documentation– Metaevaluation

Significant Changes (cont.)

• Combining of Standards– Quant/Qual into Design– Human Rights/Interactions into Respect– Clarity/Timeliness/Dissemination into

Appropriate Communication • Expansion of the drawn out

examples/scenarios

Using the Standards

• Three principles to guide the use of the Program Evaluation Standards

1. Standards require adaptive, responsive, and mindful use

– The user must discover how to apply them in each specific situation

2. Order of the Standard groups does not matter (contrary to what others say: Utilization-Focused, Accuracy)

3. In depth knowledge is required– Don’t just read the individual standards

Overlaps Across the Standards

• Functional Table of Standards begins the process of identifying standards that are related to each other a specific points in the evaluation process– How should we identify overlaps

(dependency relationships) among the standards?

– Could this enhance metaevaluation efficiency?

– Do specific standards have more significance than others?

Activity

• Four Groups: Utility, Feasibility/Evaluation Accountablity, Propriety, and Accuracy.

• Identify the keywords in the standard statements.

Example: U1 Evaluator Credibility “Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context.”