Unit 1 Module 4

Explain: Roles of the Evaluator

Introduction to Educational Evaluation5100-652Dr. Kristin Koskey

Multiple Personalities

2

“Detective” “Designer” “Judge”“Honest broker”

“Creative consultant”

“Organizational analyst”

“Leader” “Devil’s advocate” “Change agent”“Counselor” “Follower” “Communicator”“Mediator” “Technical geek” “Diplomat”

“Negotiator” “Problem solver” “Public relations agent”

Grubbs, 2009; Mark, 2002; Patton, 2007; Skolits et al., 2009

3

Pre-Evaluation Stage

• Role of the detective Learn about the context Conduct an evaluability assessment

• Role of the negotiator Reach an agreement on the evaluation

• Role of the designer Develop the evaluation plan

4

Pre-Evaluation Stage: Role of the Detective

• Learn about the context1. What is to be evaluated? 2. Who is the client? 3. Why was this program initiated?4. What are the program goals? 5. What activities are believed to lead to these intended

goals?6. What is the theory behind these program goals7. What is the political climate surrounding the

evaluation? 8. Who are the most relevant stakeholders?9. Who are the beneficiaries of the program? 10. Has this program been previously evaluated? If so, what

have different stakeholders observed as a result of the program?

11. What data already exist on this program?

5

Pre-Evaluation Stage: Role of the Detective Continued

• Determine the evaluability of the project Identify whether the program model is clear.

Check the alignment between the program activities and program model

Determine if the program goals are achievable

Determine if the stakeholders information needs are feasible

Assess whether the stakeholders and you agree on the evaluation goals and uses

6

Pre-Evaluation Stage: Role of Detective Continued

• When and when not to get involved

7

Green Light

Program goals are well defined

Program objectives are feasible to achieve given the program activities and timeframe

Relevant data can be collected or obtained at a reasonable cost (cost in terms of money and time)

It is clear that the evaluation results will be used for decision-making and in what ways

Sufficient funds are available to provide for a quality evaluation

Pre-Evaluation Stage: Role of Detective Continued

• When to and when not to get involved

8

Red Light

Program goals are ill-defined The request for the evaluation is too premature for the stage of the program

Program objectives are not feasible to achieve

A conflict of interest exists

Data needed to achieve the evaluation goals will not be available or feasible to collect in the allotted timeframe or budget

Key stakeholders cannot reach a consensus on the direction of the program and/or evaluation

Evaluation would provide trivial information and the results would not be used for decision making

Intended users of the results is not clear

Pre-Evaluation Stage: Role of Negotiator

• Reach an agreement on:– Evaluation to be conducted– Goals of the evaluation– How it will be conducted– Timelines– Budget– Who will do what and when– Use of evaluation findings

9

Pre-Evaluation Stage: Role of Designer

Evaluation model(s) based on the evaluation goals

Research design MethodologyData sources to be collected

• Formative and summative Strategies for data analysis and reporting

Budget

10

Active-Evaluation Stage

• Role of the diplomat Build good working relationships and trust with the stakeholders

• Role of the researcher Implement the evaluation plan

• Role of the judge Program strengths and weaknesses Value, merit, worth of whatever is being evaluated

• Role of the reporter Determine audience(s) Communicate relevant evaluation results Provide recommendations

11

Post-Evaluation Stage

• Role of the advocator Educate stakeholders how to use the evaluation results

Promote stakeholders use of the evaluation results

• Role of the reflector Engage in self-reflection as a professional

• What worked well and what didn’t work well?• Is there anything I could have done differently?• What skills should I strengthen?

12

References

• American Evaluation Association (2009). Guiding principles for evaluators. American Journal of Evaluation, 30(1), 5-6.

• Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed). Boston, MA: Pearson.

• Grubbs, S. T. (2009). An evaluation of an alternative teacher certification program: A matter of trust. American Journal of Evaluation, 30(4), 581-586. 

• Mark, M. M. (2002). Toward better understanding of alternative evaluator roles. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 17-36). Greenwich, CT: Information Age Publishing.

• Patton, M. Q. (2007). Utlization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage

• Skolitis, G. J., Morrow, J. A., & Burr, E. M. (2009). Reconceptualizing evaluator roles. American Journal of Evaluation, 30(3), 275-295. doi: 10.1177/1098214009338872

13