innovation network-evaluation plan workbook
TRANSCRIPT
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
1/25
Evaluation Plan Workbook
Page 0 of 23
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
2/25
Evaluation Plan Workbook
Page 1 of 23
Evaluat ion Plan WorkbookT abl e o f Co n te n ts
P a g e
Introduction
How to Use this Workbook ......................................................................2
What is Evaluation?....................................................................................3
Proving vs. Improving: A Brief History of Evaluation..........................3
Our Approach..............................................................................................3Evaluation Principles..................................................................................4
Why Evaluate?.............................................................................................5
Developing an Evaluation Plan ............................................................................6
Implementation and Outcomes: One Without the Other?....................7
Evaluating Implementation.......................................................................8
What You Did: Activities and Outputs........................................8
How Well You It: Additional Evaluation Questions .................9
Evaluating Outcomes .................................................................................11
What Difference Did You Make? ................................................11
Indicators..........................................................................................11
Elements of a Strong Indicator Statement ...................................13
Indicator Examples .........................................................................14Setting Indicator Targets................................................................15
Multiple Indicators .........................................................................15
Direct v. Indirect Indicators...........................................................17
Data Collection Preview.........................................................................................18
Methods ................................................................................................18
Ease of Data Collection...............................................................................20
Review Your Plan ................................................................................................21
Appendices: Evaluation Templates
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
3/25
Evaluation Plan Workbook
Page 2 of 23
In t ro du ct i o n
How to Use this Workbook
Welcome to Innovation Networks Evaluation Plan Workbook, which offers an
introduction to the concepts and processes of planning a program evaluation. We hope
that after following this workbook, you will understand evaluation as a tool for
empowerment. You will learn how evaluation can help your organization be more
effective, and you will be able to develop plans to evaluate both the implementation andoutcomes of your programs.
This is the second workbook in a series. We recommend that before using this
workbook, you go through the Logic Model Workbook.
You may use this book in whatever way suits you best:
As a stand-alone guide to help create an evaluation plan for a program As an extra resource for users of the online Evaluation Plan Builder (see the Point
K Learning Center at www.innonet.org)
As a supplement to an in-person training on evaluation planning.
This checklist icon appears at points in the workbook at which you should take an
action, or record something gather information, take action, write something in your
template, or enter something into your online Logic Model Builder.
You can create your evaluation plan online using the Evaluation Plan Builder in
Innovation Networks Point K Learning Center, our suite of online planning and
evaluation tools and resources at www.innonet.org. This online tool walks you through
the evaluation planning process; allows you to use an existing logic model as a
framework for your evaluation plan; saves your work so you can come back to it later;
and share work with colleagues to review and critique.
For those of you who prefer to work on paper, an evaluation plan template is located at
the end of this workbook. You may want to make several copies of the template, to
allow for adjustments and updates to your evaluation plan over time.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
4/25
Evaluation Plan Workbook
Page 3 of 23
One of the negative connotations often associated with evaluation is that it is something
done to people. One is evaluated. Participatory evaluation, in contrast, is a process
controlled by the people in the program or community. It is something they undertake as
a formal, reflective process for their own development and empowerment.
M. Patton, Qualitative Evaluation Methods1
Wh at i s Eval u at i o n ?Evaluation is the systematic collection of information about a program that enables
stakeholders to better understand the program, improve its effectiveness, and/or makedecisions about future programming.
P ro vi n g vs . Im p ro vi n g : A Br i e f Hi s to ry o f Eval u at i o n
Evaluation has not always beenand still is not alwaysviewed as a tool to help those
involved with a program to better understand and improve it.
Historically, evaluation focused onproving whether a program worked, rather than on
improving it to be more successful. This focus on proof has meant that objective,
external evaluators conducted the evaluations. Research designs used rigorous scientific
standards, using control or comparison groups to assess causation. Evaluations occurred
at the end of a project and focused only on whether the program was a success or
failure; it did not seek to learn what contributed to or hindered success. Finally, this
type of evaluation often disengaged program staff and others from the evaluation
process; these stakeholders rarely learned answers to their questions about a programand rarely received information to help them improve the program.
Ou r Ap p ro ach
We believe evaluation can be a form of empowerment. Participatory evaluation
empowers an organization to define its own success, to pose its own evaluation
questions, and to involve stakeholders and constituents in the process. Rather thanbeing imposed from the outside, evaluation can help program stakeholders identify
what a program is expected to accomplish (and when), thereby making sure everyones
expectations for the program are aligned. By looking systematically at what goes into a
program, what the program is doing and producing, and what the program is achieving,
this evaluation approach enables program stakeholders both to be accountable for
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
5/25
Evaluation Plan Workbook
Page 4 of 23
Eval u at i o n P r i n c i p l e s 2
We believe that evaluation is most effective when it:
Links to program planning and delivery. Evaluation should inform planningand implementation.Evaluation shouldnt be done only if you have some extra
time or only when you are required to do it. Rather, evaluation is a process
integral to a programs effectiveness.
Involves the participation of stakeholders. Those affected by the results of anevaluation have a right to be involved in the process. Participation will help them
understand and inform the evaluations purpose. Participation will also promote
stakeholder contribution to, and acceptance of, the evaluation results. This
increases the likely use of the evaluation results for program improvement.
Supports an organizations capacity to learn and reflect. Evaluation is not anend in itself; it should be a part of an organizations core management processes,
so it can contribute to ongoing learning.
Respects the community served by the program. Evaluation needs to berespectful of constituents and judicious in what is asked of them. Evaluation
should not be something that is done to program participants and others
affected by or associated with the program. Rather, it should draw on their
knowledge and experience to produce information that will help improve
programs and better meet the needs of the community.
Enables the collection of the most information with the least effort. Youcantand dont need toevaluate everything! Focus on what you need toknow. What are the critical pieces of information you and your stakeholders
need to know to remain accountable and to improve your program?
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
6/25
Evaluation Plan Workbook
Page 5 of 23
Wh y Eval u ate ?
Conducting a well-conceived and implemented evaluation will be in your interest. Itwill help you:
Understand and improve your program. Even the best-run programs are notalways complete successes. Every program can improve; the information
collected in an evaluation can provide guidance for program improvement. As
you incorporate evaluation into your ongoing work, you will gain useful
information and become a learning organization one that is constantly
gathering information, changing, and improving.
Test the theory underlying your program. The systematic data you collect aboutyour programs short-, intermediate and long-term achievements as well as its
implementation helps you to understand whether (and under what conditions)
the hypotheses underlying your program are accurate, or whether they need to
be modified.
Tell your programs story. The data collected through evaluation can providecompelling information to help you describe what your program is doing and
achieving. Evaluation results provide a strong framework for making your
programs case before stakeholders, funders, and policy-makers.
Be accountable. Evaluation helps you demonstrate responsible stewardship offunding dollars.
Inform the field. Nonprofits that have evaluated and refined their programs canshare credible results with the broader nonprofit community. A community thatcan share results can be more effective.
Support fundraising efforts. A clear understanding of your programwhatyou did well, and precisely how you accomplished your outcomeshelps you
raise additional funds to continue your work and expand or replicate your
efforts.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
7/25
Evaluation Plan Workbook
Page 6 of 23
De ve l o p i n g an Eval u at i o n P l an
Evaluation planning identifies and organizes questions you have about your program
and plots a route to get answers. Most questions that organizations probe through
evaluation are in three categories:
What did we do? How well did we do it? What difference did our program make? (What changes occurred because of our
program?)
Your programs logic model will form the foundation of your evaluation plan. As you
look at your logic model, you will find questions about your program that you hope to
answer. The purpose of evaluation planning is to identify these questions and plan a
route to finding the answers.
Two major forms of evaluation help answer these questions.
1) Implementation Evaluation: Are you performing the services or activities asplanned? Are you reaching the intended target population? Are you reaching the
intended number of participants? Is it leading to the products you expected?
How do the participants perceive these services and activities? These questions
are about implementation.
2) Outcomes Evaluation: Is your target audience experiencing the changes inknowledge attitudes behaviors or awareness that you sought? What are the
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
8/25
Evaluation Plan Workbook
Page 7 of 23
Im p l e m e n tat i o n an d Ou tco m e s : On e wi th o u t th e Oth e r?
We believe that an effective evaluation should answer both types of questions. You can
do one without the other, but you will not learn as much as if you conduct both types.
In the old days, nonprofit evaluation focused on documenting and reporting on
program activities. Nonprofits and others then assumed that if it implemented program
activities as planned, desired results would occur for the individuals, families,
organizations, or communities they served. In general, the nonprofit community focused
on reporting on implementation to the exclusion of outcomes. In recent years, the
pendulum has swung in the opposite direction: nonprofits are under pressure tomeasure and report outcomes, with little emphasis on implementation.
The evaluation framework we use incorporates both outcomes and implementation.
Programs are complex. You need to understand the degree to which you accomplished
your desired outcomes. You also want to learn what aspects of your program
contributed to those achievements and what barriers exist to your program getting its
ideal results.
This workbook will:
Lead you through the development of a plan to evaluate your programsimplementation.
Help you create a plan to evaluate your programs outcomes.It doesnt matter which you do first if you would prefer to start with outcomes, please
do so.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
9/25
Evaluation Plan Workbook
Page 8 of 23
Eval u at i n g I m p l e m e n tat i o n:
Wh at Di d Yo u Do ? Ho w We l l Di d Yo u Do I t ?
Activities Outputs &
Implementation
Questions
Data Collection
Method
(How to Measure)
Data Collection Effort
(have, low, med, high)
OutputsActivity Group
Questions
OutputsActivity Group
Questions
The illustration above outlines the components of your implementation evaluation plan.
Wh at Yo u Di d : Act i v i t i e s & Ou tp u ts
Your implementation evaluation plan starts with identification of the activities of your
program. The activities are the actions that the program takes to achieve desired
outcomes. If your program entails many activities, you may have organized these
activities into activity categoriesclosely related groups of activities in your program.
Your outputs are the tangible products of your programs activities. Outputs are also the
evidence of your activities. In implementation evaluation, outputs are the items you will
actually measure to evaluate your activities. Measuring outputs answers the question:What Did We Do? This is often the easiest and most direct process in evaluation.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
10/25
Evaluation Plan Workbook
Page 9 of 23
Build Your Evaluation Plan: Information from the Logic Model
(Visit the Point K Learning Center at www.innonet.org to download our Logic Model
Workbook or use the online Logic Model Builder.)
If you are working on paper:o Take your activity categories and individual activities from your logic
model and place them in the Activities column in your Implementation
Plan template.
o Take your outputs from the logic model, and place them in the Outputsboxes in your plan.
o Review your outputs and determine if you want to track any additionalproducts from your activities. Include these in the outputs boxes.
If you are working online at Point K, this information will pre-fill from yourLogic Model into your Evaluation Plan.
Ho w We l l Yo u Di d I t : Addi t i o n al Eval u at i o n Qu e s t i o n s
Documenting activities and associated outputs tells us what you did. However, that
isnt sufficient for evaluation purposes (after all, you already had your activities and
outputs identified in your logic model). The purpose of implementation evaluation is to
understand how well you did it.
The next step is to identify other questions you have about your activities and theiroutputs. What information will help you better understand the implementation of your
program? The following are examples of the types of questions you might consider:
Participation: Did the targeted audience participate in the activities as expected?Why? Were some individuals over- or under-represented? Why?
Quality: Were the services/materials you provided perceived as valuable by theintended audience? Were they appropriate? How did others in the field view
their quality?
Satisfaction: Did those affected by your programs services approve of them?Why? Who was most/least satisfied?
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
11/25
Evaluation Plan Workbook
Page 10 of 23
Ch o o s i n g Qu e s t i o n s
Brainstorm additional questions you may want to answer regarding the
implementation of your program. Use Participation/Quality/Satisfaction/
Context only as a guide; there is no need to have one of each question for all
activities, or limit your questions to these categories. Keep your list targeted to
those need-to-know questions that will have the biggest impact on program
improvement.
The answers to these questions can offer rich feedback for program
improvement. They move beyond a simple inventory of outputs and activities,
often probing the viewpoint of those you are serving. Still, they are related to
your programs implementation rather than outcomes; they address what you did,
not what changes occurredbecause of your program.
Build Your Evaluation Plan: Insert your questions into the Questions box
in your evaluation plan template, or on the Implementation Questions tab of
the online Evaluation Plan Builder.
Implementation evaluation offers important information about what you did and
how well you did it. The lessons you learn can serve as benchmarks for progress
against your original program plan.
Perhaps you arent conducting the activities as planned; or You are conducting those activities, but they are not leading to the
products/outputs you intended, or
They did lead to the intended outputs, but the quality or satisfaction levelsare not what you had hoped.
This information can help you determine if you need to adjust your plan, change
activities, or reconsider your theoretical assumptions. Evaluating your
implementation can provide a feedback loop in the midst of your effort, before
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
12/25
Evaluation Plan Workbook
Page 11 of 23
Eval u at i n g Ou tco m e s : Wh at Di f f e re n ce Di d Yo u Make ?
These days its no longer acceptable for nonprofits to assume that good
intentions, good-faith effort, or even exemplary program implementation will
result in the desired outcomes for those we serve. It is important to spend time
developing a plan to measure the achievement of outcomes.
In your logic model, you identified your desired outcomesthe changes you
expect to see as a result of your work. Outcomes are frequently expressed aschanges in knowledge, skill, attitudes, behavior, motivation, decisions, policies,
and conditions. They occur among individuals, communities, organizations, or
systems.
In di cato rs
In order to evaluate how successfully you have achieved your outcomes, you
will need to determine indicators for your outcomes.
An indicator is the evidence or information that will tell you whether your
program is achieving its intended outcomes. Indicators are measurable and
observable characteristics. They answer the question: How will we know
change occurred?
We often state outcomes as abstract concepts or ambitions. Indicators are the
measurement of outcomes. They are specific characteristics or behaviors that
provide tangible information about those concepts or ambitions. Often, one
outcome will have more than one indicator. When you develop your indicators,
it may be helpful to ask: What does the outcome look like when it occurs? How
will I know if it has happened? What will I be able to see?
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
13/25
Evaluation Plan Workbook
Page 12 of 23
An indicator should be:
Meaningful: The indicator presents information that is important to keystakeholders of the program. Keep in mind that different people can have
different perceptions about what defines success for a program.
Reaching consensus among key stakeholders regarding what success
looks like is essential to ensuring buy-in to your evaluation results.
Direct: The indicator or combination of indicators captures enough of theessential components of the outcome to represent the outcome. Severalindicators can be necessary to measure an outcome adequately. However,
there is no standard for the number of indicators to use. While multiple
indicators are often necessary, more than three or four may mean that the
outcome is too complex and should be better defined. An indicator must
also reflect the same type of change as the outcome. For example, if an
outcome is about a change in attitude or opinion, the indicator should notreflect a behavior change.
Useful: The information provided by this indicator can be put topractical use for program improvement.
Practical to Collect: The data for the indicator shouldnt be a burden tocollect. Consider whether you can collect data about your indicator in a
timely manner and at reasonable cost. Sometimes an indicator meets the
other criteria described above, but the effort to collect it would be too
burdensome. Our evaluation template offers you an opportunity to note
the level of effort involved.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
14/25
Evaluation Plan Workbook
Page 13 of 23
El e m e n ts o f a S t ro n g In di cato r S ta te m e n t
To assist in evaluation, a strong indicator statement should include these four
elements:
How much. Identify the amount of change among your target populationthat would indicate a successful level of achievement. This sets the target
for your work; base this on an understanding of your baseline and a level
of change that is reasonable for your program.
Who. Specify the target population you will measure. What. Describe the condition, behavior, or characteristic that you will
measure.
When. Note the timeframe in which this change should occur.For example, an outcome of an economic empowerment training program may
be that participants institute improved money management practices. One
indicator for that outcome would be the statement:
75% of participants open a free bank checking account within six monthsof beginning the program.
Assume that you know, through an intake process, that most of a programs
participants manage their money through expensive financial services such as
payday loan businesses. If 75% of the participants open a free bank checking
account within six months of beginning the program, you might view that as
ONE indicator of improved money management practices.
Breaking down the statement into the elements above, you see:
How Much = 75%
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
15/25
Evaluation Plan Workbook
Page 14 of 23
The following are examples of outcomes and indicators.
Outcomes IndicatorsNew mothers increase their
knowledge of child development.
75% of new mothers in the program satisfactorily complete
a short survey about child development at the end of the
course
Target audiences increase
knowledge about the signs of child
abuse and neglect
50% of community focus group members can identify the
signs of child abuse and neglect six months after education
campaign ends
Residents feel neighborhood is asafer place for children
60% of neighborhood residents report in one year that theybelieve the neighborhood is safer for children than it was
one year before.
Diversified program resources In one year, each of four funding sources (public, private
corporation, individual donors, foundations) comprise at
least 15% and not more than 55% of program income
Increased cultural competency of
legal aid attorneys
80% of legal aid attorneys self-report learning about cultural
issues at end of workshop
Within 6 months, 60% of clients of trained attorneys indicateattorney was knowledgeable about cultural issues
Increased legislators awareness of
policy options
In six months, 45% of surveyed legislators indicate
awareness of different policy options
Students (K-6) demonstrate
knowledge about opera
By the end of class:
-65% of children can identify the story of the opera being
performed.
-65% can describe 3 of the characters in the opera.
-50% can identify the type of voice (soprano, alto, tenor,bass) of the character singing a part
Youth have increased knowledge
about the consequences of long-
term ATOD use/abuse.
At end of course
-90% of participants report that they gained knowledge
about the risks/harms associated with ATOD use
-80% report that it is important not to use alcohol or other
drugs.
While developing your indicators, you may realize that your outcomes are
unclear or ambiguous. This process offers an opportunity to reconsider or
further clarify your outcomes.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
16/25
Evaluation Plan Workbook
Page 15 of 23
S e t t i n g In di cato r T arge ts
An indicator indicates that your program is achieving its intended outcomes,
which are changes you want to see in individuals, groups, communities or
systems. To be meaningful, you should identify an amount of change that you
believe demonstrates successful achievement of the related outcome.
Setting targets may seem challenging. No one wants to be held accountable for
unrealistic expectations. At the same time, we all want programs to lead to realchange.
How do you choose a realistic target? Consider these guideposts:
What is your baseline? What does your experience tell you? What are standards in the field? What do stakeholders expect?
Perhaps your program is new or you have no baseline information because you
have never gathered this information before. In that case, your first evaluation
may provide baseline information; in future years, you will want to create
realistic targets to show progress.
Mu l t i p l e In di cato rs
Some outcomes have just one closely related indicator. If an outcome is to
increase the awareness among your members of a new service your organization
provides, your sole indicator may be that 80% of your members report awareness
of new service within six months. A smoking reduction programs outcome is forparticipants to stop smoking. In this case, the indicator will be a percentage of
participants who stop smoking after participating in the program. In that case,
only one indicator will be necessary.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
17/25
Evaluation Plan Workbook
Page 16 of 23
--A percentage of participants who meet criteria in mock interviews at end of
training course
--A percentage of participants who develop quality resumes within 1 month of
completing course
--A percentage of participants who can identify three key sources of job listings
by end of the training course
When developing your logic model, you identified a chain of outcomes. This
chain is useful in both clarifying your theory of change and in setting realisticexpectations. The chain is also important from an evaluation standpoint
knowing what you are achieving. Each link in the chain has corresponding
indicators.
Short-TermOutcomes
LEARNING: The knowledgeparents and guardians gainfrom the literature & PSAs.
Increasedunderstanding amongtargeted parents of theimportance ofchildhood immunization
Increased knowledgeamong targeted parentsof where to go to havetheir childrenimmunized
Indicators:
47% of targeted parents
showed increasedknowledge aboutimmunization after theprogram
39% of participating parentscould identify an accessible
IntermediateOutcomes
BEHAVIOR: The actionsparents & guardians take asa result of that knowledge.
Increased number oftargeted parents whotake their children tobe immunized
Indicator:
26% increase in parents
taking their children forimmunizations in the targetarea in the year following
the program
Long-TermOutcomes
CONDITION: The conditionsthat change as a result ofthose actions.
Increased number ofchildren of targetedparents who continueto receive up to dateimmunizations
Healthier children
Indicators:
86% of children in the
target area who received 2-month DTaP immunizationsalso received 4-month DTaPimmunizations
68% of children in thetarget area who received 2-
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
18/25
Evaluation Plan Workbook
Page 17 of 23
Di re c t ve rs u s In di re c t In di cato rs
Ensure that your indicators relate directly to the outcome you are evaluating, and are
evidence of the same type of change. Below are some examples of direct versus indirect
indicators for the sample outcomes.
OUTCOMES INDICATORS
Participating new mothers have their
children immunized (behavior)
Indirect: #/% of participating new mothers
who are aware of importance of immunization
(awareness)
Direct: #/% of children of participating
mothers who are up-to-date in immunizations
within 1 year (behavior)
Participating children understand
principles of good sportsmanship
(knowledge)
Indirect: #/% of children who participate on
teams after finishing program (unrelated
behavior)Direct: #/% of participating youth who are
able to identify five good sportsmanship
behaviors by the end of the season
(knowledge); #/% of fights and arguments
among student athletes decreases each year of
program (behavior, but shows knowledge in
action)Targeted teens increase knowledge of
certain environmental health hazards
(knowledge)
Indirect: #/% of students who receive brochure
on topic during first 6 months of program
(output)
Direct: #/% of targeted students who can
identify 3 health hazards at end of first year of
program (knowledge)
Build Your Evaluation Plan: Insert your Indicators into the Indicators box in your
evaluation plan template, or on the Indicators/Data Collection tab of the online
Evaluation Plan Builder.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
19/25
Evaluation Plan Workbook
Page 18 of 23
Data Co l l e c t i o n P re v i e w
Of all the parts of evaluation, data collection can be the most daunting. Many of
us believe we need to be statisticians or evaluation professionals to engage in
quality data collection, but that simply isnt true.
This workbook will introduce you to the basic concepts of data collection, to help
you complete your evaluation plans. [A separate data collection workbook will
be available soon at the Point K Learning Center.]
Data Collection Methods:
Whats the best way to gather the information you need?
So far, you have identified what you want to evaluate and what you will
measure. In implementation evaluation, these are activities and their related
outputs and additional questions. In outcome evaluation, these are program
outcomes and their related indicators.
Now you will consider methods to collect the data. Outputs, implementation
questions, and indicators are what you will measure; data collection methods are
how you will measure these.
The goal in data collection is to minimize the number of collection instruments
you use and maximize the amount of information you collect from each one!
When choosing the best data collection method to obtain the information you
need, consider the following:
Which methods will be least disruptive to your program and to those yourserve?
Which methods can you afford and implement well? Which methods are best suited to obtain information from your sources
(considering cultural appropriateness and other contextual issues)?
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
20/25
Evaluation Plan Workbook
Page 19 of 23
The most common data collection strategies fall into the following broad categories.
1. Review documentsAnalysis of printed material including program records, research reports, census
data, health records, budgets. Document review is a common method of
collecting data about activities and outputs for implementation evaluation.
2. ObserveObserve situations, behaviors and activities in a formalized and systematic way,
usually using observational checklists and trained observers. This is a goodmethod to use in settings where experiencing actual events or settings (rather
than hearing about them) is an important part of the evaluation.
3. Talk to peopleCollect verbal responses from participants and other stakeholders through
interviews (in-person or phone) or focus groups. This method is helpful when it
is important to hear complex or highly individual thoughts of a certain group ofindividuals.
4. Collect written responses from peopleCollect written responses through surveys (in-person, e-mail, online, mail,
phone), tests, or journals/logs. Except in the case of journals, this method is often
used when you need a lot of information from a large number of people or when
it is important that identical information be available from all respondents.
5. Other methodsReview pictorial/multi-media data in photographs, audiotapes, compact discs,
visual artwork. Conduct expert or peer reviews in which professionals in the
field with specific expertise assess a set of activities or products. Use a case study,
an intensive investigation of one unit to use for learning purposes, often as an
exemplar or model to be avoided.
Brainstorm:Consider data collection methods for each item you will
measure.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
21/25
Evaluation Plan Workbook
Page 20 of 23
Eas e o f Data Co l l e c t i o n
When you do identify data collection methods, the next step is to identify the
level of effort required to collect the data. Consider the cost and time required to
create new data collection tools. Also, consider the cost involved in actually
collecting and analyzing the data. Some methods are more expensive than others.
For example, interviews take more time (and therefore resources) than surveys.
For many methods youll need database software. For each data collection
method you identify, consider whether you already have a tool in place that youcould use (such as an intake survey). If not, think about the amount of effort
required to create and use the new data collection tool. Assess whether it will
require a low, medium, or high level of effort.
Build Your Evaluation Plan:Assess the level of effort required to collect the
data using the methods you have identified so far. Use the scale: have (for thosemethods your organization already has available); low; medium; or high.
Enter your data collection methods and their corresponding levels of effort into
your evaluation plan template or the online Evaluation Plan Builder.
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
22/25
Evaluation Plan Workbook
Page 21 of 23
R e vi e w Yo u r P l an
Together, your implementation and outcome templates form one evaluation
plan. Take time to review your plan so far to make sure it would lead to a
worthwhile evaluation.
Here are some tips for reviewing your plan:
For implementation evaluation, have you identified additional questionsthat will help you improve the quality of the activities you are
conducting?
Have you narrowed that list of questions to those few need-to-knowtopics?
Have you identified indicators for all your outcomes? Has any outcomerequired more than three indicators? If so, consider whether that outcome
needs to be re-defined or separated into several intended changes. Do most or all of your indicator statements include the key elements of
who, how much, what and by when?
Have you identified reasonable targets for you indicators?
If you have any questions about program planning or evaluation, or are interested in our
in-person services, please visit our website, www.innonet.org or contact us at:
In n o vat i o n Ne two rk , In c .
1625 K S t . , NW, 11 t h Fl o o r
Was h i n gto n , DC 20006
( 202) 728- 0727
i n f o @ i n n o n e t .o rgwww. i n n o n e t .o rg
Copyright Policy
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
23/25
Implementation Evaluation Plan
Innovation Network, Inc.
www.innonet.org [email protected]
Activities Outputs & Implementation Questions Data Collection
Method
Data Collection
Effort(Have, Low, Medium,
High)
OutputsProgram Component
Questions
Program Component Outputs
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
24/25
Implementation Evaluation Plan
Innovation Network, Inc.
www.innonet.org [email protected]
Activities Outputs & Implementation Questions Data Collection
Method
Data Collection
Effort(Have, Low, Medium,
High)
Questions
-
7/28/2019 Innovation Network-Evaluation Plan Workbook
25/25
Outcomes Evaluation Plan
Innovation Network, Inc.
www.innonet.org [email protected]
Outcomes Indicators Data Collection
Method
Data Collection
Effort(Have, Low, Medium,
High)