Chapter 11: Managing for Quality and

Use模块 11：

质量及使用导向的管理

22

Introduction 简介

• Managing the Design Matrix• 管理设计矩阵• Managing an Evaluation• 管理评价• Managing Effectively• 有效管理• Assessing Quality of an Evaluation• 对评价质量进行评估• Using Evaluation Results• 使用评价结果

33

Evaluation Design Matrix评价设计矩阵

• Key to success in evaluation is in planning• 评价成功的关键因素的规划• Evaluation Design Matrix, links descriptive, normative,

and impact evaluation questions• 评价设计矩阵联系描述性、规范性以及影响力评价问题• Helps to systematically map out the evaluation• 协助系统地描绘出评价的全貌• Focuses the attention on each of the major components• 注意力集中在每个重要的成分上

44

About the Design Matrix关于设计矩阵

• Takes time to fill in since you do not have all the information available at start

• 不用着急填写，因为一开始你没有所有有用的信息• It is an iterative process• 这是一个重复叠加的过程• Can change the format of the design matrix to

fit your needs• 可以改变设计矩阵的模式来适应你的需要

55

Guide for Using the Evaluation Design Matrix如何使用评价设计矩阵

• For each question, complete all columns• 完成每个问题的所有表栏• One row for each question 一个问题一行

– if you have two main questions, the matrix should have two rows

– 如果你有 2 个主要问题，矩阵应该有 2 行– each row specifies how you plan to design your

evaluation for each question– 每一行都具体说明你是如何计划设计每个问题的评价

• Comment column may help keep track of unresolved issues and/or concerns

• 评论栏能帮助你同没有解决的问题或焦点保持跟进

66

Two Useful Tools两个有用的工具

• Evaluation Plans and Operations Checklist (Stufflebeam)

• 评价计划以及操作检查单（ Stufflebeam）• Checklist for Program Evaluation Planning

(McNamara)• 项目评价计划的核查单（麦克那马拉）• You may want to adapt these to fit your needs• 你可以更改这些以适应你的需求

77

Reviewing Your Design检查你的设计

• Make sure all of the pieces connect and will give you the best chance of obtaining the data necessary to answer the evaluation questions

• 保证所有部分之间都有连接，而且可以让你尽可能获得必要的数据来回答评价问题

88

Pre-testing事前测试

• Pre-testing is essential 事前测试很重要– if survey, have several people use it in a real setting– 如果是调查，安排几个人在真实的场景中使用– if focus group, conduct as if real– 如果是专题讨论组，就像真的一样来进行测试– 进行测试 De-brief respondents, find out what worked, what

did not• 仔细检查回复，找出什么管用，什么不能

– ask how they would fix the questions– 问他们将如何回答问题

• Have experts review your plans• 让专家来检查你的计划• Have cold reader review surveys for clarity, grammar, and

typing• 就明晰性、语法以及拼写进行咨询

99

Front-End Planning从始至终的计划

• Pay attention to:• 注意：

– timing and time management– 时间的设置和安排– selection of actors and resources involved in the evaluation– 对要素的选择以及评价涉及到的资源– role of program logic and program theory– 项目逻辑和项目理论的作用– design of the study– 研究的设计– question to what extent data that are needed can be made

available– 询问能够得到多少必要数据

1010

Timing of the Evaluation评价的时间安排

• For an evaluation that emphasizes use, time it so your findings are available when decisions are being made or actions being taken

•对于强调使用性的评价，对它进行时间安排，这样在你做出决定或者采取行动时，就可以有发现作参考了。

1111

Questions to Help Timing有助于时间设置的问题

• What decision, if any, is the evaluation finding expected to influence? 评价的发现能影响什么样的决策？

• When will decisions be made? By whom? When, then, must the evaluations findings be presented to be timely and influential?

• 决策什么时候做出？由谁做出？之后，评价的发现什么时候被展示出来才是及时以及有影响力的？

• What is the history and context of the decision-making process?

• 决策过程的历史及背景如何？• What other factors will affect the decision-making?• 其他什么因素能影响决策？• How much influence do you expect the evaluation to have —

realistically? • 实事求是来说，你希望评价具有什么样的影响力？(continued on next slide) 续下页

1212

Questions to Help Timing (cont.)有助于时间设置的问题（续）

• To what extent has the outcome of the decision already been determined?

• 决策的结果多大程度上已经被决定了？• What data and findings are needed to support decision

making?• 需要什么样的信息和发现来支持决策过程？• What needs to be done to achieve that level of

influence?• 达到这样的影响力需要做到什么？• How will we know afterward if the evaluation was used

as intended?• 我们之后如何知道是否评价的使用是同预期相同的？

1313

The Evaluation Team评价团队

• Who 谁– client (commissioner of the evaluation)– 客户（评价的委托人）– Stakeholders– 利益相关者– other evaluators– 其他评价者– Participants– 参与者

• How 如何– Terms of Reference– 任务大纲

1414

Value of Terms of Reference (TOR)任务大纲（ TOR）的价值

• Terms of Reference (TOR) describe the overall evaluation and establish the initial agreements prior to the work plan

• 任务大纲描述整个评价过程，并且建立工作计划开始之前的最初协议

• TOR process: • 任务大纲的过程：

– ensures all stakeholders are included in the discussion and decision-making

– 保证所有的利益相关者都被融入到了讨论以及决策过程– establishes the basic guidelines so everyone

understands expectations and context of the evaluation

– 建立最基本的指导方针，这样每个人都知道评价的内容和预期

1515

OECD, DAC Glossary DefinitionOECD, DAC 的词汇定义

• Terms of Reference (TOR) are a written documentation that present:

• 任务大纲是一个书面文本，它展示了：– the purpose and scope of the evaluation– 评价的目的及范围– the methods to be used– 使用的方法– the standard against which performance is to be assessed

or analyses are to be conducted– 评估表现或者进行分析的标准– the resources and time allocated– 需要分配的时间和资源– reporting requirements– 报告要求

1616

TOR Typically Include:任务大纲包括：

• Title: short and descriptive• 标题：简短的，描述性的• Project or program description• 计划或项目描述• Reasons for the evaluation and expectations• 评价的原因及期望• Scope and focus of the evaluation: the issues to be

addressed and questions to be answered• 评价的范围以及重点：需要谈到的和需要回答的问题• Stakeholder involvement: who will be involved, defined

responsibilities, and accountability process• 利益相关者的介入：谁会介入，定义责任以及负责过程(continued on next slide)续下页

1717

TOR Typically Include: (cont.)任务大纲通常包括： （续）

• Evaluation process: what will be done• 评价过程：需要做什么• Deliverables: typically an evaluation work plan, interim

report, final report and presentations• 可提交的：通常是评价工作计划、中期报告、最终报告和展示• Evaluator qualifications: education, experience, skills

and abilities required• 评价者的资质：教育、经验、技能以及需要的能力• Cost projection based on activities, time, number of

people, professional fees, travel, and any other related costs

• 基于活动、时间、人数、专业费用以及其他相关费用智商的成本预计

1818

Guidelines for Writing TOR任务大纲的写作技巧

• State clearly the objectives of the evaluation• 说清楚评价的目的

– identify the stakeholders of the evaluation– 说明评价的利益相关者– the products expected from the evaluation– 评价预期产生的产品– how the products are to be used– 产品如何使用– the specific issues to be addressed– 要谈到的具体问题– the methodology– 方法论– arrangements for the evaluation– 评价的准备(continued on next slide)续下页

1919

Guidelines for Writing TOR (cont.)

任务大纲的写作技巧• Do not simply state the objectives in technical or

process terms. Be clear on how the evaluation is expected to help the organization

• 不要仅仅用专业术语或过程来描述目的，清楚说明评价如何能帮助组织结构

• Focus on key issues to be addressed by the evaluation• 集中注意力于评价的主要问题• Avoid too many issues. It is better to have an evaluation

that examines a few issues in-depth rather than one that looks into a broad range of issues superficially

• 避免太多问题。集中注意力集中的深入的检查几个问题比肤浅的涉及很多的问题要好

2020

TOR Checklists任务大纲的检查单

• Evaluation Contracts Checklist (Stufflebeam)• 评价合同检查单（ Stufflebeam）• Checklist for Developing and Evaluating

Evaluation Budgets (Horn)• 发展及评估评价预算的检查单（ Horn）• Key Evaluation Checklist (Scriven)• 主要的评价检查单（ Scriven）

2121

Contracting Evaluations聘用他人进行评价

• If human resources are not available in your organization, you may need to hire one or more other people to assist

• 如果你的机构人员不足，你需要雇佣其他人员进行协助

• Can be brought in for the entire study or only parts

• 可以是全程参与，也可以是部分参与

2222

Two Main Steps for Hiring Contractor雇佣合同工的两个步骤

1 Developing a request for proposal (RFP) 完成一个征求建议书（ RFP）2 Use a selection panel to choose a contract

evaluator 组成一个遴选委员会来挑选合同制评价者

2323

Items for Contracting合同的条款

• Purposes of the evaluation 评价目的• Background and context for the study 研究背景• Key information requirements 主要的信息要求• Evaluation objectives 评价目标• Deliverables required 要求的提交内容• Time frame 时间表• Criteria for tender selection 细致选择的标准• Contract details for the project manager 项目经理的合同细节

• Deadline for the proposal 议定书的截止日期• Budget and other resources 预算以及其他资源

2424

Selection Process for Contracting承包的选择过程

• Select a panel comprising people with:• 遴选委员会的成员须具备：

– evaluation knowledge and experience– 评价知识和经验– knowledge of the program area– 项目领域的知识– knowledge of the culture– 文化方面的知识– ownership of the findings and their uses– 成果及其使用的所有者

• Have the panel select the proposal using the criteria in the RFP

• 委员会利用 RFP的要求来选择议定书• Keep a record of the selection process• 对筛选过程进行记录

2525

Criteria for Selecting the Contractor选择承包方的标准

• What is the contractor’s record of accomplishment?• 承包方取得了什么样的成就？• Has the RFP been adequately addressed?• RFP是否得到了充分的重视？• Is there a detailed explanation of implementation?• 是否有对执行过程的详细解释？• What is the communicating and reporting strategy?• 沟通策略和汇报策略是什么？• Is there evidence of competencies?• 是否有能胜任的证据？• What is the cost — is it specified in detail?• 成本是什么？是否有详细说明？

2626

Responsibilities of Purchaser, Once Hired一旦被雇佣，采购方的责任

• Keeping goals and objectives clear• 保证目标明确• Maintaining ownership of the study• 保持研究的所有权• Monitoring the work and providing timely feedback• 监测工作，提供适时反馈• Decision-making — in good time• 在合适时做出决策• If changes are required to the contract, being open to

negotiation with the contractor• 如果合同需要改动，应该同承包方公开谈论

2727

Managing Roles and

Responsibilities 管理角色以及责任

• Clearly define and agree to who is responsible for what and when• 明确定义并就何人就什么以及何时负责达成一致• Who needs roles and responsibilities defined:• 谁需要定义角色和责任：

– evaluation managers– 评价经理– Evaluators– 评价者– Client– 客户– providers of information (stakeholders)– 信息提供者（利益相关者）– consumers (impactees)– 消费者（受影响者）

2828

Evaluation Manager Roles评价经理的角色

• Helping evaluators do their work 帮助评价者工作– clarifying the TOR for the team– 帮助团队明确任务大纲– answering questions– 回答问题– checking on status of responsibilities– 检查责任的归属及状态– asking if they need additional resources– 询问他们是否需要额外的资源– helping team learn– 帮助团队学习– facilitating team meetings– 主持团队会议

• Choosing staff 挑选成员• Reviewing results and report 检查结果以及报告

2929

Evaluation Manager Responsibilities项目经理的责任

• Detailed responsibilities in three areas:• 在三个领域的责任细则：

– Preparation– 准备– Implementation– 执行– follow-up– 跟进

3030

Evaluators Skills and Abilities评价者的技巧和能力

• Expertise in the specific subject matter• 在相关领域的专业能力• Knowledge of key development issues especially those

relating to the main goals or the ability to see the “big picture”

• 重点发展领域的知识，尤其是那些跟主要目标以及需要见到全貌的能力有关的知识

• Familiarity with organization’s business and the way such business is conducted

• 对组织业务以及组织业务开展方式的熟悉• Evaluation skills in design, data collection, data analysis,

and preparing reports• 设计中的评价技巧、数据收集、数据分析以及准备汇报• Skills in the use of information technology• 信息技术中使用的技巧

Essential Competencies for Program Evaluators (ECPE)项目评价者的主要资质（ ECPE）

• See Table 12.1 in text• 参见 表 12.1• Includes these major areas:• 包括以下主要领域：

– professional practice 专业经验– systematic inquiry 系统咨询– situational analysis 情景分析– project management 项目管理– interpersonal competence 人际技能

3131

3232

Team Leader团队领导

• If multiple evaluators are working on an evaluation, may have a team leader

• 如果多个评价者在从事同一评价，需要一个团队领导• Responsibilities• 责任：

– supervise team members and manage day-to-day process– 监督团队成员，并管理日常事务– act as mediator if there are dissenting voices within the

evaluation team– 如果评价团队中出现不同意见，领导应能进行仲裁– may write the report– 可以撰写报告– make sure the team satisfies the TOR– 保证团队对于任务大纲是满意的

3333

Client客户

• The person who officially requests the evaluation

• 正式要求进行评价的人• Best if this is the same person to whom

the person reports• 最好与汇报人是同一个人

3434

Managing Projects项目管理

• Managing time• 时间管理• Managing scope• 范畴管理• Managing people• 人员管理• Managing money• 资金管理• Managing resources• 资源管理

3535

Project Management Process项目管理过程

Initiating

启动

Initiating

启动

Planning

计划

Planning

计划

Executing

执行

Executing

执行

Controlling

控制

Controlling

控制

Closing

结束

Closing

结束

Skip to Managing People

3636

Phase I: Initiating第一阶段：启动

• Projects begin by confirming there is a need for the project

• 该项目的需求得到确认之时，即是该项目的启动之时• Three Actions to be undertaken in this phase• 这个阶段需要进行三个行动

– Demonstrate project need and feasibility– 表明项目的需要和可行性– Obtain project authorization– 获得项目授权– Obtain authorization for the phase– 获得这个阶段的授权

3737

Phase II: Planning第二阶段：计划

• Managers spend much of their time planning• 管理人员需要花很多时间来计划• 13 actions in this phase• 这个阶段有 13个行动• Some steps are optional, depends on the size

and scope of the project• 一些步骤是可选的，这取决于项目的大小以及范畴

3838

Actions for Planning项目行动

• Describe the project scope• 描述项目范畴• Define and sequence project activities• 定义次序以及项目活动• Estimate durations for activities and resources required• 估计活动的持续时间以及需要的资源• Develop a project schedule• 完成一个项目计划表• Estimate costs• 估计成本• Build a budget and spending plan• 完成预算以及预算使用计划(continued on next slide)续下页

3939

Actions for Planning (cont.)

项目行动 ( 续 )

• Create a formal quality plan• 制定一个正式的质量计划• Create a formal project communication plan• 制定一个正式的项目沟通计划• Organize and acquire staff• 组织及获取人员• Identify risks and plan to respond• 确认风险以及应对计划• Plan for and acquire outside resources• 计划以及获得外在的资源• Organize the project plan• 组织项目计划• Close out the project planning phase• 完成项目计划阶段

4040

Phase III: Executing第三阶段：执行

• Manager makes sure the tasks are being completed

• 管理人员保证任务完成• One action to be undertaken:• 一个行动应该完成：

– Execute project activities– 执行项目活动

4141

Phase IV: Controlling第四阶段：控制

• Manager keeps track of people, activities, money, and scope

• 经理对人员、活动、经费以及范畴保持沟通• One action to be undertaken• 完成一个行动

– Control project activities– 控制项目活动

4242

Phase V: Closing第五阶段：收尾

• Manager finalizes activities, archives information• 管理人员完成活动，对信息整理归档• One action to be undertaken• 一个行动需要完成

– Close out project activities– 完成项目活动

4343

Role of Evaluation Manager评价经理的角色

• Clearly describe the desired end project• 清晰的描述理想的完成项目• Describe what it was that you liked about the relevant previous efforts• 描述你认为有关前期努力的优点在哪里• Involve the evaluator(s) in the planning• 使评价人员参与到计划中• Monitor the progress of the evaluation• 监测评价的过程• Establish a timeline• 建立时间表• Motivate the evaluator’s to produce their best• 激励评价人员发挥他们的最佳水平• Avoid micro-managing the evaluator(s)• 避免对评价人员的管理过度• Thank the evaluator(s) for their work• 感谢评价人员付出的努力

4444

Managing People人员管理

• As a manager, you are managing people, NOT evaluations!

• 作为经理，你管理的是人，不是评价！• People are complicated. They are not machines. Their

behavior will change from day to day.• 人是复杂的，他们不是机器，他们的行为会随时间改变• The manager is responsible for everything that happens

within the manager’s scope of authority.• 经理须对在其职权范围内所发生的一切负责

Working with Stakeholders同利益相关者合作

• Building trust• 建立信任• Involving Directly Affected Stakeholders• 纳入直接受影响的利益相关者• Seeking Feedback• 寻找反馈• Involving the Voiceless• 纳入那些不发表意见的人• Involving the Opposition• 纳入反对意见

4545

4646

Managing Tasks任务管理

• May be easier than managing people• 比人员管理容易• Task map • 任务图

– everyone’s assignments– 每个人的任务– start and completion dates– 开始以及完成时间

• Gantt chart • 甘特图表

– chart showing interrelationships of projects and schedules

– 说明项目以及日程表之间关系的表格

4747

Example of Task Map任务图的例子

Task 任务

Name 名字

Start Date开始时间

Date due截至日期

Review prior meetings会议开始前检查

Linda 7/01 7/31

Schedule meeting with stakeholders同利益相关者排定会议

Ed 7/15 7/31

Conduct stakeholder meetings举行利益相关者会议

Linda and Ed 8/1 8/15

Design the evaluation设计评价

Ray 7/1 8/31

Develop data collection instruments信息收集工具的开发

Ray 8/1 8/31

4848

Example of Gantt Chart甘特图表的例子

Task/ 任务 Month/ 月份1 2 3 4 5 6 7

Review检查

Meetings会议

Design设计

Implement执行

----- ----- -----

4949

Scope of Work工作范畴

• Similar to TOR, they covers these tasks:• 同任务大纲类似，他们包括以下任务：

– identify what is to be evaluated– 确认评价的对象– provide a brief background on the intervention– 对于干预提供简短的背景– identify existing data– 确认已有的信息– state the purpose of the evaluation along with audience and use– 从评价目的、对象以及使用开始– identify questions 确认问题– specify the methods 确认方法– discuss the composition of the team – 讨论团队的组成– sets out the schedule and budget– 制订日程以及预算

Go ahead to Assessing Quality

5050

A Good Evaluation:一个好的评价：

• Meets stakeholder needs and requirements• 达到利益相关者的需要和要求• Is relevant and realistic in scope• 范畴上相关且实际• Uses appropriate methods• 使用合适的方法• Produces reliable, accurate, and valid data• 产生可靠的、精确的以及有效的数据• Includes appropriate and accurate analysis of data• 包括合适、精确的数据分析• Presents impartial conclusions• 展示不偏不颇的结论• Conveys results clearly — in oral or written form• 清楚地说明结果——以口头和书面形式• Meets professional standards• 达到专业标准

5151

Kusek and Rist’s Six Characteristics of Quality EvaluationsKusek 和 Rist的质量评价的 6 个特征• Impartiality

• 公平• Usefulness• 实用• Technical adequacy• 技术适当• Stakeholder involvement• 利益相关者的介入• Feedback and dissemination• 反馈与成果散播• Value for money• 物有所值

5252

Checklists for Quality质量的核查单

• The Key Evaluation Checklist (Scriven)• 主要的评价核查单（ Scriven）• Program Evaluations Meta-evaluation Checklist

(Stufflebeam)• 项目评价的广义评价核查单（ Stufflebeam）• Utilization Focused Evaluation Checklist (Patton)• 关注于利用的评价核查单（ Patton）• Deliberative Democratic Evaluation Checklist (House &

Lowe)• 审慎而民主的评价核查单（ House & Lowe）• Guiding Principles Checklist (Stufflebeam)• 核查单的指导原则（ Stufflebeam）

5353

Meta-evaluator广义评价员

• A person with evaluation expertise who is not involved in conducting the evaluation, but who you can use as a sounding board, advisor, and helpful critic

• 一个具备评价技能的人，他不会参与评价的实施过程，但是他能被用作一个出色的咨询对象，并提供有益的批评

5454

Quick Meta-evaluation迅速的广义评价

• Consider getting a “rapid assessment” meta-evaluator to quickly look over your evaluation plan (or report)

• 考虑获得一个“快速测评”的广义评价员，以便能迅速察看你的评价计划（或报告）

• Offer to act as a meta-evaluator/review for someone else

• 主动要求充当别人的广义评价员

5555

Kusek and Rist: Kinds of Information from Evaluations Kusek 和 Rist：评价中的各种信息

• Strategy: are the right things being done?• 战略：正确的事情做了吗？• Operations: are things being done right?• 运行：事情做对了吗？• Learning: are there better ways?• 教训：是否有更好的方法？

5656

Kusek and Rist: Pragmatic Uses for Evaluations Kusek 和 Rist：评价的实际应用

• Help make resource allocation decisions• 帮助资源配置的决策• Help rethink the causes of a problem• 帮助重新思考问题的原因• Identify emerging issues• 确认出现的问题• Support decision making on competing or best

alternatives• 支持有竞争力的或最好的替代计划的决策• Support public sector reform and innovations• 支持公共领域的改革和革新• Build consensus on the causes of a problem and how to

respond• 就问题的原因进行统计，并研究如何应对

5757

Patton: Uses for Findings from EvaluationsPatton: 来自评价的发现的利用

• Judge merit or worth•评判优点或价值• Improve programs•改进计划• Generate knowledge• 产生知识

5858

Patton: Primary Uses of Evaluation Logic and ProcessesPatton: 评价逻辑和过程的主要用途

• Enhanced sharing understandings• 增进理解分享• Supporting and reinforcing the program

intervention• 支持及加强计划干预• Increasing engagement• 不断增加投入

self-determination, and ownership• 自决以及所有权

5959

Suggestions for Ways to Improve Use of Evaluations评价用法改进的几点建议

• Gain support from the top• 获得来自高层的支持• Involve the stakeholders at the top, bottom, and sides• 让上层、底层以及周边的利益相关者融入进来• Make the evaluation a part of the institution• 使评价成为机构的一部分• Plan your evaluations• 规划你的评价• Consider your timing: timing is everything• 考虑你的时间：时间就是一切• Communication is important• 沟通很重要

6060

Influence and Effects of Evaluation评价的影响和效果

• How do you bring the information you learned to the attention of the decision-makers?

• 如何使你获得的知识得到决策者的重视？• How can you be in the right place at the right time?• 你如何在正确的时间出现在正确的地方？• Brief the decision makers on the evaluation results at

the meeting where decisions will be made• 在决策会议上对决策者简要汇报评价结果• Ensure that decision makers have a package that

summarizes the results, their implications, and recommendations if any

• 保证决策者可获得总结结果、启示以及可能的建议的一揽子建议

6161

Carol Weiss: To Get Use –Identify the Evaluation Users Carol Weiss: 利用——确认评价的使用者

• Who are the primary intended users of the evaluation?• 评价的主要预期使用者是谁？• Who are the target audiences of the evaluation?• 评价的目标受众是谁？• Which groups or individuals are most likely to be affected by the

evaluation?• 哪个群体或者个体最容易被评价的结果影响到？• Whose actions and/or decisions will be influenced by their engagement

with the evaluation process or evaluation findings?• 谁的行为或决定会因为他们对评价过程或评价发现的介入而受到影响？• How can the intended users of the evaluation be involved?• 如何能使得评价的预期使用者介入？• What challenges/barriers might you face in identifying and involving

users? How to overcome them?• 在确认并融入利用者的过程中，你会遇到什么样的挑战和障碍？如何来克服这些困难？

6262

Foster Evaluation Thinking培养评价思考

• Increased clarity, specificity, and focus• 提升确切性、具体性和重点• Being systematic and making assumptions explicit• 系统性地作出明确的假设• Translating program concepts, ideas, and goals into operational

plans• 将计划的概念、想法以及目标变为可操作的计划• Distinguishing inputs and processes from outputs• 区分投入以及产出过程• Valuing empirical evidence• 重视实证证据• Separating statements of fact from interpretations and judgments• 把事实陈述和解释判断分开