building an evaluation system

7/27/2019 Building an Evaluation System

1/32

Building a Strategic Learning and

Evaluation System or Your Organization

Hallie Preskill, PhD

Katelyn Mack, MS


2/32

Discovering Better Ways to Solve Social Problems

FSG is a nonproit consulting irm specializing in strategy, evaluation, and research.

Our international teams work across all sectors by partnering with corporations,

oundations, school systems, nonproits, and governments in every region o the globe.

Our goal is to help companies and organizationsindividually and collectivelyachieve

greater social change.

Working with many o the worlds leading corporations, nonproit organizations, and

charitable oundations, FSG has completed more than 400 consulting engagements

around the world, produced dozens o research reports, published inluential articles in

Harvard Business Review, Stanord Social Innovation Review, Foundation Review, and

theAmerican Journal o Evaluation, and has been eatured in The New York Times, Wall

Street Journal, Economist, Financial Times, BusinessWeek, Fast Company, Forbes, and

on NPR, amongst others.

Learn more about FSG at www.sg.org
http://www.fsg.org/http://www.fsg.org/


3/32

3

Table o Contents

Overview 4

Evaluation Vision 7

Strategy and Focus 10

Monitoring and Evaluation Activities 15

Supportive Environment 21

Learning Culture and Practices 26

Conclusion 29

Reerences 30

Building a Strategic Learning and

Evaluation System or Your Organization


4/32

Building a Strategic Learning and Evaluation System for Your Organization4

The Urgent Need to Align Evaluation and Strategy

Evaluation in many o todays organizations is ad hoc, not aligned to strategy, underunded,

and underused. We have ound this to be true regardless o sector, type or size o organization,

or whether they are or proit, philanthropic, or nonproit organizations. Oen times, this

situation has led to monitoring and evaluation eorts being perceived as adding little value to

organization decision makers, and a perception that monitoring and evaluation eorts are not

worth their cost.

At the same time, evaluation as a tool or strategiclearning is gaining traction within the philanthropic ield.

This ocus on learning is generating interest in evaluation

methods and approaches that provide the insights needed

to inorm strategic decisions. Though concerns about the

value o evaluation continue, oundations and nonproits

are increasing their investments in evaluation and are

experimenting with new approaches expecting that they

will help increase their eectiveness and impact.

Further evidence o this continued commitment to evaluation was ound in a recent study

conducted by the Center or Evaluation Innovation or the Evaluation Roundtable (The Centeror Evaluation Innovation, 2013). Responses rom more than 40 oundation evaluation sta via

a survey and interviews indicated that:

The evaluation unction and investments continue to expand. Moreover, the number o

FTEs dedicated to evaluation also appears to be increasing slightly (over 2009).

Evaluation continues to play a role in shaping oundation strategy and its use during all

phases o the strategy lie cycle (beginning, middle, end) is improving.

With calls or evaluation to inorm learning and a desire to move rom outputs to outcomes

and results, there is a tremendous opportunity and need to increase the value o evaluation,

to connect it with strategy, and to elevate it to a meaningul and critically important unctionwithin an organization.

Strategic learning is the useo data and insights rom a

variety o inormation-gathering

approaches including

evaluation to inorm decision

making about strategy.

(Coman and Beer, 2011)

The Ideal Intersection o Strategy and Evaluation

Learningor Social Impact

EvaluationStrategy

Overview


5/32

5

The Challenge: Disconnection

In many organizations, the meaningulness and usability o evaluation inormation has been

limited because o its disconnection rom strategic and organizational-level decision making.

Even in the most well-intentioned organizations with leaders who believe in evaluations ability

to support individual, group, and organizational learning evaluative thinking and practice are

loosely aligned, ragmented, and siloed. Through our practice, we have observed:

Most evaluations ocus on program- or initiative-level questions only; they are not designed

to answer important strategic questions within and across programs.

There is a lack o alignment between the kind o monitoring and evaluation data being

collected and the needs o program and organizational leaders in making strategic and

organizational decisions.

What gets evaluated is mostly ad hoc and relects the needs o particular individuals or

departments, rather than the organization as a whole.

Evaluation budgets are a mystery; there is rarely a dedicated organizational-level budget line

item or monitoring, evaluation, and learning work; as a result, this work is underunded.

There are ew internal mechanisms or capturing, storing, accessing, and sharing learnings

rom evaluation eorts.

There are ew processes, systems, and opportunities or learning rom and about evaluation,

which limits the ability o organization members to make sense o indings and to translate

them into action.

As a result, organizations spend signiicant amounts o time and eort developing a compelling

strategy, deining goals, and articulating a convincing theory o change, without putting into

place the inrastructure and support needed to monitor and evaluate whether or not the strategy

is actually working. I organizations were to establish and maintain systems and processes that

support eective monitoring, evaluation, and learning practices, they would be more eective in:

Understanding and tracking their strategys eects, inluence, and impacts;

Ensuring the collection and use o meaningul and useul grantee and other stakeholder

inormation;

Facilitating and supporting individual, group, and organizational learning;

Providing insights into the eectiveness and efciencies o the organizations core activities(e.g., communications); and

Inorming the ield about key learnings rom their work.

The current state o evaluation in the philanthropic and nonproit sectors points to the need

or a more strategic approach to evaluation. In this guide, we address the question: How can

organizations be more systematic, coordinated, and intentional about what to evaluate,

when, why, with whom, and with what resources?

Overview


6/32


A Strategic Learning and Evaluation System

SupportiveEnvironment

EvaluationVision

Strategyand Focus

Monitoringand EvaluationActivities

LearningCultureandPrac

tices

Developing anEvaluation Vision

Principlesand Values,Stakeholders

Strategiesand Tactics

Systems Map andTheory o Change

Strategic EvaluationQuestions

Outcome Mapsand Indicators

Grantee Reportingand EvaluationCapacity

Program, Initiative,and Organization-Focused Evaluations

Communicatingand UsingEvaluation Findings

Leadership

Human Resources

Financial Resources

IT and KnowledgeManagementSystems

The Opportunity: Strategic Learning and Evaluation Systems

I learning and evaluation eorts are to inorm an organizations decision making practices,

then there needs to be a comprehensive strategy or evaluation. A strategic approach to

evaluation requires a clear vision or evaluation; a culture that osters individual, group,

and organizational learning; a compelling and cogent strategy; coordinated evaluation and

learning activities; and a supportive environment. When ully implemented, these elements

work together to ensure that learning and evaluation activities relect and eed into the

organizations latest thinking. In this context, evaluation can help answer the most pressing

questions o leadership and sta. The result is what FSG calls a Strategic Learning and

Evaluation System (also known by the acronym SLES).

This brie describes each SLES component and how an organization can achieve signiicant

advances in the timeliness, relevance, credibility, and useulness o evaluation practice.


7/32

7

DEVELOPING AN EVALUATION VI SION

To develop an evaluation vision, organization members should consider the ollowing

questions in craing a 24 sentence vision statement:

What role might learning and evaluation play in the organization?

What value will learning and evaluation add to the organization and other stakeholders?

How might evaluation contribute to strategic decision making?

The Caliornia Endowment (TCE), or example, uses its evaluation vision to describe its

commitment to using evaluation (among other activities) or learning within the oundation

and among its partners and grantees.

The Caliornia Endowment is committed to promoting learning among sta, grantees,

partners, and the ield through evaluation, research, and convenings that inorm

strategic decision making, build evaluation capacity, oster adaptive change, and help

TCE and its partners achieve impact in Building Healthy Communities.

An evaluation vision relects the values that the organization has or learning and evaluation, and

communicates evaluations role in strategic and organizational decision making. Organizations that

clearly articulate the principles and values underlying their approach to evaluation ind themselves

more willing to use evaluation to build knowledge, inorm action, and hold stakeholders accountable.

Evaluation VisionPrinciples and Values, Stakeholders


8/32


The International Gay and Lesbian Human Rights Commission (IGLHRC) is an international

organization dedicated to human rights advocacy on behal o people who experience

discrimination or abuse on the basis o their actual or perceived sexual orientation, genderidentity, or expression. The organization ocuses on two strategic goals: 1) Capacity building or

the documentation o human rights violations, and 2) Global advocacy. Their vision or learning

and evaluation is:

IGLHRC strives to engage in collaborative and learning-oriented monitoring and

evaluation that guides internal decision making and inorms conversations with

external stakeholders about IGLHRCs inluence, eects, and impact.

A clear vision or evaluation signals to sta and the ield what evaluation is going to be usedor and the values on which evaluation practice is based. Organizations may emphasize

learning, collaboration, impact, accountability, organizational eectiveness, and a host o

other purposes or evaluation in their vision statement. A list o core values or evaluation

transparency, authenticity, commitment to impact, among others usually accompanies a

vision statement. Values provide a guiding ramework or making decisions about what to

evaluate and the way in which it will be undertaken.

For example, the Issues Afecting Women Programme o the Oak Foundation developed its

evaluation and learning system to relect the ollowing values:

We approach evaluation in a way that is:

Collaborative we aim to work with our partners and other actors in the ield to

contribute to the inormation we seek to collect, to determine the best ways to go

about doing so, and to share in the indings;

Designed to generate actionable and on-going learnings we strive to only gather

inormation that can guide our strategies and decisions; and we view evaluation as

an ongoing process, not a one-time event at the end o a grant, project, or strategy;

Flexible and adaptive to the many dynamic contexts we work in we work with

grantees o all shapes and sizes, working in contexts that are in constant lux; we

thus do not subscribe to a rigid one-size-its-all approach to evaluation; and

Mindul o our partners time and resources we appreciate how precious time and

resources are and thus strive to engage in evaluation that is not an onerous side-

activity but rather strengthens the internal learning and capacities o our partners.


9/32

9Evaluation Vision

When it is not clear why evaluation is important, how it will be used, and who stands to beneit

most, a mismatch between the expectations o unders and grantees can occur. Failure to

communicate a clear, simple evaluation vision can result in conusion and misunderstanding,and in the some cases, an unproductive skepticism o an organizations intentions. Funders

may see evaluation as a way to improve perormance, while grantees see unders interest in

evaluation as an attempt to cut o or scale back their investment. This situation highlights the

importance o communicating the evaluation vision with internal and external stakeholders

in a consistent and authentic way to minimize any misconceptions about how evaluation is

meant to be conducted and used.

What value will learning and evaluation add to your organization?

What role can learning and evaluation play in your organization?

How might evaluation contribute to strategic decision making?

Who needs to be inormed about the evaluation vision, and whatare eective ways or communicating with these stakeholders?

How do we want to use what is learned rom our evaluation work?

Questions to Consider


10/32


A SLES exists to support strategic decision making, both at the program and organizationlevels. Thereore, it is critically important that an organization not only have, but be able toarticulate and agree on the strategies and tactics it is using to achieve its goals. In developinga SLES, an organization reviews and assesses the coherence o the current strategy orstrategies being employed. Without a clear understanding o the organizations strategy,evaluation activities easily become scattershot and disconnected making it difcult to turnevaluation indings into insights or strategic learning and change.

Taking a more strategic approach to evaluation requires an organization to a) understand thelarger context in which its work is occurring, and b) relect on and discuss the assumptionsembedded in its strategy and how they believe change occurs. This conversation highlightspotential conlicts or misunderstandings about expected outcomes, and clariies the issueo contribution vs. attribution. We have ound that engaging stakeholders in developing asystems map helps acilitate this dialogue. The activity typically involves providing lipchartpaper and markers to groups o 3-4 and inviting participants to draw a map o the systemin which their work lives (e.g., college readiness system, substance abuse and prevention

STRATEGIES AND TACTICS

SYSTEMS MAP AND THEORY OF CHANGE

This element o a Strategic Learning and Evaluation System ensures that there are agreed upon

strategies to guide organization and program level decisions, that staf understand the larger

system in which the work is happening, and that the majority o evaluation activities are guided by

a set o strategic evaluation questions.

Strategy and Focus


11/32

11Strategy and Focus

IGLHRC believes that building capacity to document human rights violations and engaging in advocacy at

regional and global intergovernmental ora constitute the most efcient way to contribute to building stronger

LGBT rights movements that will help LGBT people around the world live richer, uller, and ulilling lives.

Greater level o LGBT rightsinstitutionalized and operationalizedat the UN and regional ora

Allied movements are committedto LGBT rights issues

Efective LGBT work is led by diverse,representative, community basedgroups, individuals, and organizations

State and non-state and privateactors are held accountable orSOGI related violations

Improved cultural and social normsor LGBT people

LGBT communities more efectivelymeet the broad needs o LGBT peopleliving in their country and region

Build the capacity o organizations

and leaders in the South to document

human rights violations

Collaborate with partners on

the ground to document human

rights violations

Lead efective LGBT engagement

at international, regional, and

national human rights bodies

Support activists in the South in

their advocacy eforts

LGBT people around the world are

still discriminated against

Activists in the South need to build

their capacity to document human

rights violations

Lack o organizations ocused on

LGBT human rights documentation

capacity building

Increased need to understand

what works in advancing the LGBT

human rights movement and

demonstrate impact

Outcomes

Activities

TheProblem

system, news and inormation system), and what organizations and/or actors, play a role in,inluence, or work towards the same goals. Each group shares their map, which is ollowed by

a large group discussion that ocuses on commonalities and dierences, aulty assumptions,new insights, and implications or evaluation.

The photo at right shows the results o such an activitywith the Northwest Area Foundations work as they weredeveloping a strategic learning and evaluation system. Whatis particularly interesting is how dierent their maps are;participants highlighted many important issues and questionsthat later became part o several SLES components.

It is also important or the organization to develop a high level theory o change that depicts theorganizations understanding o the problem, how it plans to address or solve the problem, and the

long-term change it is hoping to aect. While there are many ways to illustrate a theory o change,we have ound it best to keep it airly simple. Its utility is in its ability to connect the organizationsstrategy to the change(s) it seeks.

For example, the International Gay and Lesbian Human Rights Commission developed theollowing theory o change that rames and guides their work:


12/32


Once the organizations strategy and tactics have been agreed upon and are explicit, and

there is a broader understanding o the system in which the organization is working, then its

time to develop a set o high level strategic evaluation questions that will ocus and guide

the majority o the organizations evaluation and learning activities. These questions are

the overarching questions that serve as guideposts or understanding what an organization

is achieving, in what ways, and with what kinds o resources. The vast majority o uture

evaluations should be grounded in and help inorm the answers to the strategic evaluation

questions. Some strategic evaluation questions may be inward looking; or example, In what

ways are our organizational structures and processes supporting (or hindering) grantees

ability to implement innovative strategies to improve educational attainment? Otherquestions may ocus on the process, outcomes, and impact o the work, while others could

address the ways in which the ield is inluencing the work and vice versa.

Good strategic learning and evaluation questions:

Frame the scope and boundaries o evaluation activities

Are grounded in the programs theory o change

Relect a variety o key stakeholders inormation needs

Are those that matter most or decision-making and action

Can be answered through the collection and analysis o data

It is important to develop a set o questions that will guide the organizations evaluation

activities or the next 13 years. Oentimes, these questions address process, impact,

organization, and ield-level outcomes. The ollowing is a general set o questions that could be

tailored to the speciic oci and issue areas o the organizations mission, values, and strategy:

Process:

In what ways are the organizations statewide, regional, and local eorts working together

to inluence policy?

How can the organization strengthen its relationships with and impact on grantees?

To what extent, and in what ways, is progress being made toward accomplishing the

organizations multi-year goals?

What actors are inluencing the extent to which progress is being made?

STRATEGIC EVALUATION QUESTIONS


13/32

13

Outcomes and Impact:

To what extent and in what ways have the organizations investments led to changes in

local and state policies and practices?

Which models are proving to be most eective?

To what extent are the organizations strategic approaches eective relative to

alternative approaches?

What dierence has our work made in the last ten years?

Organization:

How have the organizations structures and processes helped or hindered progress toward

addressing its goals?

To what extent and in what ways is the organization eectively communicating its strategy,values, and vision within the organization?

How has the organization increased its organizational capacity to respond to an increasingly

challenging ield?

Field:

To what extent is the organization contributing to the ields knowledge? To what extent is

it catalyzing action?

What role is the organization playing relative to others working on these issues?

To what extent and in what ways is the organization transorming the ield?

The tailored set o questions would look something like the ollowing, as illustrated by those

developed by the King Baudouin Foundation in Brussels, Belgium. Here are a sample o their

questions speciically ocused on their process and outcomes:

1. Supporting capacity-building actions

How have our capacity-building eorts helped the grantees and partners

o those eorts?

To what extent are the grantees and partners satisied with ourcapacity-building eorts?

2. Raising knowledge (in a speciic target population)

To what extent has the amount o knowledge increased?

To what extent has the quality o knowledge increased?

To what extent has interest in the topic increased?

Strategy and Focus


14/32


3. Forming opinions/attitudes

To what extent are opinion leaders engaging with our content/events?

What % o opinions/attitudes in our target group have we changed? How?

To what extent and how have opinions/attitudes in the political arena or the

popular media shied?

To what extent and how have shis in opinion/attitudes led to shis in action?

4. Initializing policy actions

Has policy in act changed?

How are policy makers engaging with our content/events? How many?

How are stakeholders, beyond the direct stakeholders o our project,

embracing and advocating or the policy recommendations weve made?

To what extent are members o parliament making public statements about

the issues were promoting?

What are our organizations strategies or achieving its goals? Is there astrategic plan where these are made explicit? How current is this plan?

How aligned are the various departments/programs with ourorganizations strategies?

What is the system in which our work is taking place? Where is thereenergy in the system? Where are there gaps? Where is our organization inthe system? What implications are there?

What is our organizations or program areas theory o change? How isthis theory o change relected in our strategy? Is the theory o changearticulated in written orm?

Who needs to be involved in developing program or project level theorieso change and outcome maps i more work is needed?

What are the questions our organization would like to have answeredthrough its learning and evaluation system within the next 13 years?(These should be tied closely to the organizations strategies)



15/32

15

OUTCOME MAPS AND INDICATORS

It has become expected and common practice or program sta to develop outcome maps

or logic models or their various programs.1 Quite simply, an outcome map makes explicit,

in one place, the activities, resources, outputs, and short, interim, and long-term outcomes

o a program or initiative. When designed collaboratively, an outcome map clariies the

assumptions underlying the program in addition to expectations or what the program

would look like i it were successul. Outcome maps oen describe the relationships between

activities and outcomes and anticipated or predicted pathways to success. The undamental

value o outcome maps is that they provide an opportunity or sta and senior leaders (and

sometimes grantees) to negotiate understandings about the true purpose and hoped or

outcomes o a strategy overall, as well as one or more programs. Even though outcome mapshave their limitations (e.g., they are linear, they assume the program is based on a model

that has been tested and the expected outcomes are known or can be predicted, and they

dont capture systems variables), they are an invaluable resource or 1) helping organizations

make explicit their assumptions and understandings about the issue and the organizations

solution, and 2) designing and implementing monitoring and evaluation activities.

A major element o a Strategic Learning and Evaluation System is determining what to

evaluate, when, and at what level, and how to communicate and use the evaluation indings or

learning, adaptation, and change. This includes developing one or more program, initiative, and/

or organization-level outcome maps, considering what to ask or rom grantees and how oen,

determining which programs to evaluate, and how to communicate key learnings with multiple

stakeholders (intended users o the key indings).

1 We use the term outcome map in this document since we have ound that the nonproit and philanthropic ields have mixed experiences and eelings

about logic models. We are more interested in what an outcome map or logic model ofers staf than in what it is called.

Monitoring andEvaluation Activities


16/32


GRANTEE REPORTI NG AND EVALUATION CAPACITY (RELEVANT TO FUNDERS)

PROGRAM, INI TIATIVE, AND O RGANIZATION-FOCUSED EVALUATIONS

Nearly every oundation asks its grantees to provide some kind o perormance or accountability

inormation once or twice a year during the grant period in the orm o a narrative report,

program documents, or survey. However, the inormation collected is oen not aligned to either

the programs or organizations strategy, and given oundation stas multiple responsibilities

and workload, grantee reports may not be read or used as much as might be desired.

Developing a SLES provides an opportunity to connect grantees work with the oundations

and ensures that grantees are asked to collect inormation on topics that matter to them,

and also to the oundation those related to the organizations strategy and tactics. This

component o the SLES addresses the extent to which grantees work relects: The desired outcomes embedded in the program and organization-level outcome maps;

The degree to which grantee-reported inormation is aligned with the organizations

strategic evaluation questions;

The ways in which grantee inormation is reported and used; and

The grantees capacity to collect the requested inormation.

Insights rom working on this component oen highlight the need or making some

adjustments to what is being asked or rom grantees, how inormation is being collected, and

potential areas in which to support grantee evaluation capacity.

Evaluation planning provides a structure and set o processes or determining the scope,

timing, and purpose o various evaluation eorts, so that evaluations are strategic, intentional,

and learning-oriented. Without an overarching evaluation plan that includes all or most

programs and initiatives, evaluation activities become ragmented and increasingly isolated

rom strategic and programmatic decision making. This ragmentation also makes it difcult to

use evaluation indings or strategic learning.

Program- or initiative-level evaluations within nonproits and oundations relect many

dierent types, approaches, sizes, and purposes. They are oen conducted by a third-party

evaluator who can bring the necessary time, skills, and expertise to the evaluation work. Since

it is unnecessary to evaluate all programs at any given time, a SLES guides sta in prioritizing

what kinds o inormation they need in the near term and rom whom (which programs,

activities, organizations).


17/32

17Monitoring and Evaluation Activities

Organization-level evaluations might ocus on the organizations capacity to do its work (e.g.,

evaluating the eectiveness o its structures, processes, and unctions), the ields perceptions

o the organizations work, or the next phase o the organizations strategy. These evaluationsmay be conducted with internal personnel, or they might warrant hiring a third-party evaluator,

depending on the need or an outside perspective and the stas availability.

Each evaluation should have a set o its own key evaluation questions, and these are

explicitly linked to at least one Strategic Evaluation Question that the leadership, board, or

other stakeholders want to know more about. Once the questions have been determined,

an organization can decide what types o resources it might need to carry out an evaluation

(depending on the breadth, depth, and scope o the evaluation).

Dierent Types o Inquiry Address Dierent Kinds o Inormation Needs

Research

Developmental

Evaluation

Process Evaluation

(Formative)

Impact Evaluation

(Summative)

Seeks answers to questions in order to generate new knowledgeand/or understanding

Supports innovation by bringing data to inorm and guide ongoingdecision-making as part o the development process

Tracks adherence to accountability requirements; Assesses a programor initiative by quantitatively measuring key indicators o progress

Explores how a program or initiative is making progress towardsits goals in terms o outputs and short-term outcomes

Determines the long-term efects o a program or initiative

Monitoring

and Perormance

Measurement


18/32


COMMUNICATING AND USING EVALUATION FINDINGS

When it comes to evaluation approaches, there are several choices, but choosing the right

one depends on the stage at which the program is in its lie cycle and the purpose o the

evaluation. For example, monitoring and perormance measurement is commonly usedby oundations to track grantees perormance as part o a grantee reporting process.

Developmental evaluation may be used to understand and inorm the development o an

innovative and experimental program or initiative (Preskill and Beer, 2012). Strategic reviews

can inorm an organization about the progress and/or impact o a particular strategy over

time, or cluster evaluations can explore and determine the inluence, eects, or impacts o a

group o programs or grantees. Clarity around the purpose o evaluations will help determine

which orms o evaluative inquiry are most appropriate or getting the inormation needed to

inorm the Strategic Evaluation Questions. The purpose and scope o the evaluation will, in

part, determine the design and data collection methods used in implementing the evaluation.

Improving the mechanisms or reporting and communicating evaluation inormation helps

ensure that high quality, relevant, and timely evaluations are ully relected upon, discussed,

and ultimately used. Many organizations invest great resources in careully designing

a methodologically rigorous evaluation while giving little thought to the ways in which

evaluation indings will be shared with and used by various stakeholders. This leaves many

evaluation reports unread and unused, rather than actively inorming decisions that are being

made about key strategies and tactics. One o the ways to address this is by developing a planor communicating evaluation indings and insights, both internally and externally.

For example, the Communities Foundation o Texas (CFT) determined that it wanted to share

learnings rom its evaluation work with both internal and external audiences and, as a result,

developed the ollowing communicating and reporting plan to guide these eorts. Whats

important to note is the variety o strategies the oundation uses to share its insights and

indings with others.


19/32

19Monitoring and Evaluation Activities

CFT Trustees

CFT Leadership

Grantees

Media

Other Funders

Policymakers

CFT Grantmaking and Program Staf

Nonproits / Partners

CFT All Staf

Donors

Discussion

Memo/Email

ProgressReport

Dashboard

Presentation

SocialMedia

Webinar

Newsletter

Report/WhitePaper

Video

Convening

Spectrum o Communities Foundation o Texas (CFT) Communications Approaches

Fewer Resources More Resources


20/32


Are there theories o change and outcome maps or each o our program

areas? To what extent are these complete, relective o reality, and used? Taking into account the Strategic Evaluation Questions, which programs/

portolios would beneit most rom evaluation within the next year or two?

What kind o evaluation would be most appropriate or the program, givenwhere it is in its lie cycle perormance measurement, developmental,ormative, or summative (or retrospective lookback)?

What is our organizations perspective on what constitutes relevant,credible, and useul inormation? To what extent does our organizationneed/want numbers and/or stories?

How are evaluation indings communicated and reported internallyand externally?

What communication and reporting strategies are working particularly well?What strategies are not achieving the desired impact?

A sound communications plan requires identiying who can use and learn rom the evaluation,

what the key messages o the evaluation are, and howthe intended audiences will best

receive that inormation. For example, a board member will likely be interested in dierentkinds o inormation and expect it in a dierent ormat than a program ofcer or nonproit

sta member. The architects o an evaluation must plan and budget or communication eorts

appropriately, and communications sta must also be aware o and support these activities.



21/32

21

Evaluation unctions most efectively when it is well supported by a strong inrastructure, which

includes supportive leadership and human, inancial, and technological resources. The most

critical is an organizations people sta who are assigned responsibility to oversee and/or carry out

evaluation activities. Financial and technological resources, such as knowledge management systems,

are also critical to developing a ully unctioning SLES. While designated evaluation sta may have

direct control over certain evaluation and learning activities, it is oen the case that the inrastructure

needed to support the design, implementation, communication, and use o evaluation indings is

the responsibility o other departments such as Inormation Technology (IT) and communications.

Thereore, the support o organizational leadership and coordination across departments is essential

to ensure that resources or evaluation are adequate and made available at the right time.

LEADERSHIP

For a strategic learning and evaluation system to work or evaluation to be integrated

into how the organization does its work and accomplishes its goals it is critical that

organizational leaders believe that evaluation is a meaningul and important activity. As such,

they must consistently: a) communicate the importance o evaluation or decision making, b)

encourage organization members to engage in asking questions, relection, and dialogue, and

c) provide the necessary time, personnel, and inancial resources to ensure that evaluationscan be conducted. Leaders who support evaluation:

Actively engage in, and encourage others to think evaluatively;

Provide resources (inancial, time, personnel) or conducting relevant, credible, timely, and

useul evaluations;

Supportive Environment


22/32


Use indings to learn, make decisions, and take action;

Clearly articulate stas roles and responsibilities regarding how, when, and by whom

evaluations are to be conducted; and

Provide reward and recognition systems that value stas engagement in learning and

evaluation activities.

How does our organizations leadership think about and value evaluation?

What can be done to help our leaders understand their role as championso learning and evaluation?

Who might be internal champions or leaders o evaluation within ourorganization?



23/32

23Supportive Environment

How are current evaluations budgeted? What percent o eachprograms budget is set aside or learning and evaluation activities?

Howdoesourorganizationdeterminehowmuchisallocatedto

any one evaluation?

Howadequatearethecurrentevaluationunds?Whatelseisneededto ensure an ongoing ability to conduct learning, monitoring, andevaluation activities?

FINANCIAL RESOURCES

While it always takes a committed set o leaders to elevate evaluation and its use within an

organization, adequate inancial and human resources dier by organization size, annual

expenditures, as well as the evaluations purpose and role. On average, oundations tend to

spend much less on evaluation than the oen recommended 510% o grant making; in act,

most spend less than 1% (Patrizi Associates, 2010). A true commitment to evaluation is backed

by a consistent allocation o inancial resources earmarked or evaluative purposes on an

annual basis. These resources must include the ull array o evaluation activities including

investing in various kinds o evaluations (e.g., developmental, ormative, summative, strategic

reviews) as well as learning processes, such as retreats and sta, and communications

costs. Providing adequate resources to learning and evaluation eorts is also a signal o theleaderships support o using evaluative inquiry to acilitate organizational learning.



24/32



To what extent does our organization have sufcient human resourcesto support ongoing evaluation and learning processes? What elsemight be needed?

What are our internal stas roles and responsibilities or learning andevaluation? What role might external evaluators play?

Are there internal sta who would like to be more involved in theorganizations learning and evaluation work? I yes, what can be done

to support their increased participation?

What kinds o evaluation capacity building activities would bebeneicial to sta?

HUMAN RESOURCES

Successul implementation o a strategic learning and evaluation system also requires

deliberate investment o time and sta resources to manage, design, and conduct the needed

evaluations, as well as to plan or and acilitate learning rom the indings. Some organizations

rely almost entirely on external evaluators, whereas others depend primarily on internal sta

to oversee and conduct monitoring and evaluation activities. The development o a SLES helps

the organization determine what kinds o sta resources are needed to support its learning,

monitoring, and evaluation work.


25/32

25


INFORMATION TECHNOLOGY AND KNOWLEDGE MANAGEMENT

Oen overlooked in many organizations is the need or technological processes and resources

that are necessary to support not only data collection and analysis but also the ability to

capture, organize, and access monitoring and evaluation inormation or continuous learning

and improvement. In many organizations, inormation is stored in multiple locations email

servers, websites (public and/or internal), personal hard drives, and in print. Access is either

limited or challenging, creating challenges to using evaluation indings in meaningul ways.

An eective knowledge management and database system enables multiple inputs

rom multiple programs so that data can be captured, stored, summarized, and used by

organization members when they need inormation. Knowledge management systems

can range rom simple shared drives where reports are posted online to cloud-based

platorms with complex unctionality. To create a useul and useable system or knowledge

management, an organization should consider the primary audience or the system and its

needs, the existing culture around data sharing, and the alignment with work processes, as

well as the time available to input and use the inormation in the system.

What kinds o technology/soware currently support the collection andanalysis o grantee and other orms o evaluation data? How eective/useulare these technologies?

Does our organization have a knowledge management system? Is it used?What would need to happen to enhance the current knowledge managementsystem so that it can support the learning and evaluation system?

What kinds o learning and evaluation resources would be meaningul toorganization members? How can these be made available to those who needand want them?

Supportive Environment


26/32


An organization may design an elegant and comprehensive evaluation system as described

in this brie, but i it lacks a learning culture, the system will have little to no eect in making

evaluation strategic and useul. Evaluation, at its core, is a means or learning, adapting,

and changing. Organizations need to support a culture o risk taking, trust, tolerance or

ailure, and curiosity i they are truly going to beneit rom evaluation eorts. In other words,

organizations need to embrace and support a learning culture: An environment that supports

and encourages the collective discovery, sharing, and application o knowledge (Gill, 2010).

Good evaluations afrm and challenge what we think we know, provide evidence on the

inluence, eects, and impact o our work, and give us the conidence to make decisions andto take action. Supporting this view, Grantmakers or Eective Organizations has written,

Evaluation is a core learning practice. It provides the content o learning as

grantmakers and their grantees explore the results o their work and how to

improve their perormanceEvaluation, o course, is not the only way in which

organizations learn. But grantmakers must think evaluatively about their work and

have access to the inormation, eedback, and data that only evaluation can deliver.

(GEO, 2009, p. 6).

An important way to enhance the relevance and use o evaluation activities is to embed

learning processes into the ways in which the organization gathers, analyzes, understands,discusses, and uses evaluation indings (Preskill and Torres, 1999). This creates a dynamic

environment where learning takes place among multiple people, in various roles, and with

dierent levels o responsibility or applying evaluation knowledge in their work. In many

organizations, organizational culture is established at the top. While leadership plays a

major role in setting the tone or learning, all sta are considered responsible or supporting

evaluation and learning in their own ways.

Learning Cultureand Practices


27/32

27

Learning requires consistent engagement in 5 key learning processes.

1. Engaging in Relection creating space, slowing down, paying attention, creating new

patterns o thinking, creating alternative interpretations, creating new theories o action.

2. Engaging to Dialogue participants working together toward common understanding, inding

common ground, re-examining all positions, admitting that others thinking can improve on

ones own, searching or strengths and value in others positions, listening to understand.

3. Asking Questions seeking clariication, probing assumptions, reasons, and evidence,

illuminating viewpoints and perspectives, probing implications and consequences,

questioning the questions (Socratic questions).

4. Identiying and Challenging Values, Assumptions, and Belies asking questions (testing

assumptions), suracing mental models, seeking evidence, understanding inerences.

5. Seeking Feedback asking or and providing eedback on experiences, assumptions,

perceptions, and actions.

These learning processes are the key ingredients to creating a healthy and productive

learning culture. Opportunities or embedding learning rom monitoring and evaluation

activities might include:

Hostinglearningconveningsonvariousevaluationtopicsandindingswithgrantees

and/or partners.

Holdingbrownbaglunchestopresentanddiscusslearningsromanevaluationortoshare

learnings rom engagement in daily work.

Addingtimetoprogramstafmeetingstorelecton,discuss,andlearnaboutvarious

evaluation topics.

Conductingaeractionreviews(AAR),whicharestructuredreviewsordebrieingsor

analyzing what happened, why it happened, and howit can be done better. An AAR is

distinct rom a debrieing because it begins with a clear comparison o intended vs. actual

results achieved.

Engagingprogramstafina12dayquarterlyorsemi-annualretreattodiscussthe

implementation and learnings rom individual evaluations or evaluation-related issues and

challenges across evaluations.

UsingAppreciativeInquirytoilluminateefective,successul,andmotivating

evaluation experiences.2

Hostingroundtableresearchdiscussions.

Buildingandusingaknowledgemanagementsystemtoserveasaresourceor

evaluation practices and key learnings.

AdministeringtheReadinessorLearningandEvaluation(ROLE)survey(availableat

www.sg.org) and acilitating conversations around the indings.

2 Appreciative Inquiry is a process that engages a group o people in searching or what a program, organization, situation, etc. looks like when it is at

its best; when something is working and is providing new insights, excitement, and a uture that is collectively desired. Within an evaluation context,

Appreciative Inquiry can build on what is succeeding to urther reine, develop, and adapt new learnings, processes, and goals.

Learning Culture and Practices


28/32


Towhatextentdoesourorganizationhaveacultureolearningandevaluation, leaders who champion learning and evaluation, and thesystems and structures in place to support learning and evaluation?

Towhatextentandhowareevaluationindingscurrentlysharedwithinour organization? Outside the organization?

Whatkindsolearningactivitiescurrentlyexistwithinourorganization?How eective are these?

Whatelseisneededtoencourageandsustainlearning?

Whatsystemsandstructuresdowehavethatcurrentlysupportsta learning?


For example, the Blue Shield o Caliornia Foundation is committed to being a learning

organization and to using evaluation as a means or acilitating team and organizational

learning. To this end, they are making signiicant eorts to integrate discussions based onthe ollowing questions into sta meetings and their eorts to build a culture o learning.

InwhatwaysistheFoundationevolvingintoalearningorganization?

TowhatextenthastheFoundationgrownintakingrisksandlearningrommistakes?

Howconsistentarethemessagesaboutlearningwithactionstakenbythe

organizations leadership?

WhathavebeenthemostefectivetoolsorincreasinglearningacrossFoundationstaf?

Howhavestafintegratedtimeorrelectionanddialogueintotheirdailywork

practices?

Whatassetsdowehavetobuildon,andwhatdoweneedtochangesothatBSCF

can become more o a learning organization?

In the end, an authentic commitment to learning means recognizing the value o learning at the

individual, group, and organizational levels. It means having learning leaders who champion and

model learning; it means developing, nurturing, and sustaining a culture that supports the

ive learning processes, in addition to taking risks, and trusting one another; it means rewarding

and recognizing sta or engaging in learning and evaluation processes; and it means making

learning a priority through the expectation and provision o time and space to allow it to happen.


29/32

29

Conclusion

So, What Now?

When all o the components o a SLES are in place intentionally connected, interdependent,

and dynamic an organization is able to more eectively use evaluation indings to develop

and reine its strategy and, ultimately, achieve its long-term impact. For readers who wish

to develop a strategic learning and evaluation system, we recommend starting with an

assessment o what is currently in place and possible gaps or each o the SLES components,

as well as the alignment o current practices (e.g., to what extent and how are evaluation

practices aligned with strategic decision making needs?). Once it is clear what is needed to

build out the SLES, work can begin to develop or reine each component and to develop a road

map or action plan or implementing the newly rereshed, aligned, and strategic system oevaluation activities. A strategic learning and evaluation system not only provides guidance

or making important evaluation decisions, but it also gives the organization conidence that

evaluation resources are being used efciently and eectively. In the end, a SLES provides the

organization with an evaluation strategy that ultimately increases the value o evaluation or

the organization.


30/32


Coman J. and Beer T. (2011). Evaluation to Support Strategic Learning: Principles and Practices.

Center or Evaluation Innovation, Washington, DC.

Gill S.J. (2010). Developing a Learning Culture in Nonproit Organizations. Thousand Oaks, CA:

SAGE Publications.

Grantmakers or Eective Organizations (GEO) (2009). Evaluation in Philanthropy: Perspectives rom the Field.

Washington, DC.

Patrizi Associates. (2010). Use o Evaluative Inormation in Foundations: Benchmarking Data. Evaluation

Roundtable. Available at: http://evaluationroundtable.org/publications.html.

Preskill H., Beer T. (2012). Evaluating Social Innovation. FSG and Center or Evaluation Innovation.

Available at www.sg.org and www.evaluationinnovation.org

Preskill H., Torres R.T. (1999). Evaluative Inquiry or Learning in Organizations. Thousand Oaks, CA:

SAGE Publications.

The Center or Evaluation Innovation (2013). Evaluation in Foundations: 2012 Benchmarking Data.

Available at http://www.evaluationinnovation.org

Reerences


31/32

Acknowledgements

We wish to thank the ollowing organizations or their commitment to ensuring that all o

their monitoring, evaluation, and learning eorts are strategic. We greatly appreciate their

generosity in allowing us to share some o the ruits o their labor:

BlueShieldoCaliorniaFoundation

CommunitiesFoundationoTexas

InternationalGayandLesbianHumanRightsCommission

KingBaudouinFoundation

NorthwestAreaFoundation

OakFoundation

TheCaliorniaEndowment

Authors

Hallie Preskill, PhD, Managing Director

[email protected]

Katelyn Mack, MS, Consultant

[email protected]

This report was irst published on August 20, 2013.

All statements and conclusions, unless speciically attributed to another source, are those

o the authors and do not necessarily relect those o the other organizations or reerences

noted in this report.

Building a Strategic Learning and Evaluation System or Your Organization

by FSG is licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License.

Permissions beyond the scope o this license may be available at www.sg.org.


32/32

FSG

FSG is a nonproit consulting irm specializing in strategy, evaluation, and research,

ounded in 2000 as Foundation Strategy Group. Today, FSG works across sectors in

every region o the worldpartnering with oundations, corporations, nonproits, and

governmentsto develop more eective solutions to the worlds most challenging issues.

building an evaluation system

Documents