evaluating community organizing (2)

7/29/2019 Evaluating Community Organizing (2)

1/16

Grassroots Actionand Learning for Social Change:

2010 Center for Evaluation Innovation. All rights reserved.

May not be reproduced whole or in part without written permission from Center for Evaluation Innovation.

Evaluating CommunityOrganizing

CatherineCrystal Foster

Justin Louie

Blueprint Research

& Design, Inc.

March 2010


2/16

Grassroots Action and Learning or Social Change: Evaluating Community Organizing 1

Catherine Crystal Foster and Justin Louie

Blueprint Research & Design, Inc.

March 2010

Evaluating Community Organizing

Grassroots Actionand Learning for Social Change:

Community

organizing shares

characteristics with

policy advocacy,

but it differs in

important ways.

Approaches for

evaluating the two

also differ.

In recent years, the practice o community organizing has matured and evolved,

gaining visibility as a vibrant and potent orce or social change. This has coincided

with a growing ocus on systemic reorms throughout the social sector, reorms

that eective community organizing can help advance. As this work has gained moreprominence, and its potential has been more broadly understood, interest in assess-

ing the contributions and impact o organizing activities has surged.

This interest is part o a larger trend seeking metrics to drive program improvement

and innovation in the social sector. Growing numbers o unders, evaluators, and

nonprots are striving to develop meaningul program assessments, particularly or

systemic reorms such as organizing, advocacy, and other policy change work.

While it was once common to hear questions about whether advocacy eorts

could be meaningully evaluated, those have long since given way to discussions

o how to take advocacy evaluation to the next level. Development o rameworks

and an array o tools to measure progress along the multiple dimensions o policychange and advocacy have emerged. And now, ater implementing and vetting these

approaches, practitioners have been adapting them to address the particular needs

and circumstances o community organizing, as distinct rom other orms o social

change or advocacy work.

Community organizing or social change shares many characteristics with policy

advocacy, but it diers in signicant ways, and the approaches to evaluating the two

also dier. As evaluators, we have partnered with organizers, advocates, and their

unders over the last ve years, and we have seen these dierences rst-hand. Since

developing our initial ramework or assessing advocacy and policy change activities

in 2005,1

we have worked closely with seasoned and emerging community organizerson a range o evaluation projects, covering multiple issue areas in a variety o

locations. We have provided technical assistance and tools to organizers, helped

systematize their existing assessment processes, and led external evaluation eorts

in partnership with them. Throughout our work, we have continued to learn rom

organizers and rene our own thinking and processes.

1 Guthrie, K., Louie, J., David, T., & Crystal Foster, C. (2005). The challenge of assessing policy and advocacyactivities: Strategies for a prospective evaluation approach. San Francisco, CA: Blueprint Research &Design.


3/16


Community

organizing

catalyzes the power

of individuals to

work collectively

for the changes

they want in their

communities. It

places a strong

emphasis on

developingcommunity leaders.

This brie describes our vision o community organizing evaluation. It is grounded

in a set o principles that have emerged rom our experiences with organizers, as

well as the collective wisdom o others working in this eld. We have learned that

evaluation o organizing should be:

Participatory, rather than purely third-party

Prospective (contemporaneous and orward-looking), rather than retrospective

Learning-based, rather than part o a pass-or-ail mentality

Real-time, rather than delayed and academic

Respectul o the culture o organizing

Attentive to leadership development as well as policy wins

Focused more on evidence than proo.

We begin the brie with an examination o the similarities and dierences between

advocacy and organizingdierences that shape how our approach to evaluationo organizing diers rom evaluation o advocacy. We then present a ramework or

evaluating organizing, and discuss some o the lessons we have learned through the

ongoing process o implementation.

What is Community Organizing?

Community organizing is a democratically-governed, values-driven process thatcatalyzes the power o individuals to work collectively to make the changesthey want to see in their communities. Community organizers honor and develop

the leadership potential in everyday people by helping them identiy problems and

solutions, and then by supporting them as they take action to make those solutionsa reality. In so doing, organizing challenges the existing power structure.

Relationships lie at the heart o organizing, and the one-to-one relational

conversation between an organizer and a community member is the building block

o organizing. As those community members participate in social change work, build

skills, and take on responsibilities, they become leaders within the organizing

group. Developing these leaders and building the base o leaders and other

community members is an ongoing ocus o community organizing.

A given community organizing group may emerge on its own and remain

independent, or be associated with one o several national organizing networks,

such as the PICO National Network, Association o Community Organizations or

Reorm Now (ACORN), Industrial Areas Foundation (IAF), Direct Action and Resource

Training Center (DART), Gamaliel Foundation, and others. Some are aith-based, and

others are institutionally- or place-based. Organizing may also coalesce around an

issue, like immigration, health, or education; or around populations, such as youth,

or a specic ethnic group.


4/16


Organizing Shares Qualities with Policy Advocacy

Like policy advocacy, community organizing eorts requently require longtimerames to come to ruition. The process o growing and mobilizing a base,building power, shiting attitudes, and changing policies or practices does not

happen overnight. Holding those in power accountable ollowing change also takes

time. The work o organizing, like advocacy, is complex and iterative. Strategies

shit, oten rapidly and multiple times, based on the external environment and otheractors. Ultimate outcomes may not be readily quantiable. The work is requently

collaborative, which brings with it benets and challenges. And organizing

campaigns, like advocacy campaigns, oten spring rom a sense o crisis and cause

the participants to approach the work with a high level o emotional intensity.

But Organizing is Distinct rom Advocacy

Organizing and advocacy dier, however, at a core level. Community organizingis emphatically bottom-up. It is the community members who select theissues, proer the solutions, and drive strategy and execution. Most advocacy is

undamentally top-down, even i the work is authentically undertaken on behalo community members. Advocates speak or others, while organizers inspire

community leaderseveryday peopleto speak or themselves. Tellingly, the

so-called Iron Rule o organizing is, Never do or people what they can do or

themselves.

Organizers and leaders also believe that community members can be experts, and

that expertise is not the sole domain o policy proessionals. A low-income mother

with little ormal education can be an expert on local educational needs just like a

senior think tank ellow, through her own experience or by conducting community-

led action research in her neighborhood school.

The leader-ocused lens also points to another dierence rom advocacy. In

organizing, leadership development is a central concern and a key outcome in

addition to policy change objectives. This has major implications or priorities and

goals. It makes capacity development look dierent in organizing than in advocacy,

since the capacities to attract and develop leaders are a top priority in organizing.

Finally, certain logistical aspects o organizing dier rom advocacy in a signicant

way. Organizers operate in a predominantly oral culture, in contrast to the more

archived, written culture o advocacy. Organizing oten places a premium on process

and ritual, particularly as it concerns base-building and direct actions. In addition,

organizing takes place in a more diuse setting: in homes, churches, schools, or

community venues, rather than in a central oce or the corridors o the state house.

Evaluation o Organizing Presents Unique Challenges

In many respects, the similarities between advocacy and organizing presentsimilar evaluation challenges. But or organizing, evaluation requires additionalconsiderations that refect the particular qualities o the work. Most important, the

bottom-up nature o organizingdriven by the community, not by organizational

managers or external proessionalscreates a whole new set o complexities. This


5/16


orientation collides with the inherently top-down nature o traditional third-party

evaluation, in which outside experts ask the questions, set the terms, and make

judgments. As we have noted, organizers have a undamentally dierent view rom

advocates not only o how decisions are made and priorities are set but also in

where expertise resides. That aects how organizers view evaluation generally, and

what role they see or themselves and their leaders in that process.

I the community-dened, bottom-up goals or the work do not align completelywith a unders goals, an evaluator measuring against those goals aces the dicult

task o navigating between the two. When the work unolds as part o a multi-site

initiative in which multiple communities have been unded to work on an issue,

those complications are compounded. Since the goals, strategies, and tactics o

organizing bubble up in ways that are highly context-specic, multi-site evaluation

o an organizing eort is particularly hard. It is quite dicult to standardize

methodologies and roll up results when the work and processes are driven by the

needs and approaches o each community.

As well, the intense ocus on leadership development in organizing, and the

emphasis on process within some schools o organizing lead to identicationo interim benchmarks and goals that oten dier rom those in an advocacy

campaign targeting similar policy change objectives. Organizing requires additional

benchmarks and goals related to the processes o growing leadership and power,

and organizers may prioritize them dierently rom advocates.

Finally, certain practical considerations implicit in organizing work can impact

evaluation. Many organizers value refection quite highly, and incorporate it in

their work more explicitly than some advocates. This is particularly true or aith-

based organizers. As a result, evaluation may be more about systematizing inormal

refection and helping to ocus it more on impact than process, not about teaching

the value o it. Yet, while they do refect regularly, organizers have very little timeor ormal evaluation and the rigorous, uniorm, and documented processes o data

collection and analysis that ormal evaluation can imply. They pride themselves on

never being in the oce, instead spending their time in the community. Leaders who

carry out the work are community members who may have entirely separate day

jobs, making systematic evaluation ar more challenging than when partnering with

advocates working in a more traditional oce environment.

How an Evaluator Can Address the Unique Challenges o EvaluatingOrganizing

When conronted with these complicating actors, how can an evaluator

respond? In our work, we have started by setting an overall approach to

the evaluation that refects key attributes o community organizing. First and

oremost, a community organizing evaluation is most eective when it involves

the participants in an ongoing and meaningul way, and rames the evaluation as a

learning opportunity or everyone. Embracing the bottom-up approach o organizing

and the organizers highly democratic notion o expertise leads to an evaluation

that leverages the strengths o organizing rather than ghting its values. This is a

stark contrast to accountability-ocused third-party analysis. We have ound that

the participatory approach is the most eective way to capture useul and reliable

Organizing requires

benchmarks

to capture the

process of growing

leadership and

power.


6/16


inormation, interpret that inormation, and ensure that data collected are actually

used to improve the work. Intentional relationship building with organizers and

leaders is crucial to success in such an evaluation.

The dynamic nature o most organizing work and the intensity o organizing

campaigns make real-time data collection and eedback particularly important.

Organizers need eedback today to move their campaigns orward tomorrow.

They are not as interested in academic examinations o long-past work or detailedcompilations o proo. Experienced organizers know what power looks like and

they know where they want to go. What they oten welcome is a process or more

systematically examining the multitude o interrelated orces and activities that will

take them there.

Evaluation eedback or organizers becomes most useul when it is oriented

toward learning rather than judgment or a pass-or-ail accountability. A ocus on

inormation or learning and growth has particular resonance or organizers who

devote themselves to developing individual leaders and the process o building

power. Faith-based organizers and those adopting an action-refection method o

organizing naturally gravitate toward a more ormative, developmental approach toevaluation. 2

Given this receptivity, and our interest in promoting evaluation or organizational

learning, we have ound it productive to help organizers build their own internal

evaluation capacities. This enables them to work in greater partnership with us and

to carry on authentic assessment activities in the long-term, both with and without

external evaluation assistance.

We have also ound that in contrast to program or advocacy evaluation, organizing

evaluation must measure and credit achievement o leadership development

and capacity development goals as much as policy goals, since that mirrors the

organizers work.

The evaluation also works best when it accommodates organizing style and culture.

This requires attention to the language and philosophical underpinnings o a given

organizing groups approach, recognition o the volunteer-based stang structure,

and the more oral culture implicit in organizing work.

Basic Scaolding or Evaluation o Community Organizing

Understanding the dierences between community organizing evaluation andadvocacy evaluation at the conceptual level we have described is crucial, buthow does an evaluator use that understanding to implement a real evaluation onthe ground?

2 Formative evaluation, in contrast to summative evaluation, assesses a program or activities whilethey are forming or unfolding. It looks at the context, inputs, and processes of implementation to helpimprove and refine the program or activities. Developmental evaluation, a term coined by MichaelQuinn Patton, is a team effort that supports program, project, personnel and/or organizational devel-opment through a process of continuous improvement and adaptation. The evaluator in a develop-mental evaluation facilitates data-based decision-making by the team.

Evaluation for

organizers is most

useful when it is

oriented toward

learning rather

than judgment

or a pass-or-fail

accountability.


7/16


We have ound in our work that to structure an evaluation o community organizing

that yields timely inormation that organizers and their unders can use, we need to

start by erecting some basic scaolding with the organizers. There is little new about

this part o the process, particularly to those amiliar with advocacy evaluation, but

it is important to establish what the supporting beams look like. As we describe, it is

the next stages o construction that add the particular approaches and perspective

that eective organizing evaluation requires.

To begin, in partnership with organizers:

Articulate a theory o change,3 by whatever name and in whatever orm is most

accessible to the organizers and their leaders, making it clear that the theory

does not need to be linear or static.

Articulate evaluation questions, thinking collaboratively about what organizers,

leaders, unders, and other stakeholders want to learn.

Dene benchmarks and indicators, paying particular attention to mid-term

benchmarks and to the range o outcomes that are important to reaching

organizing goals, recognizing that they may shit over time.

Establish multiple data collection methods and collect data by involving

evaluators, organizers, and leaders in the process.

Provide real-time eedback, to the extent easible.

Document progress toward benchmarks and goals, and re-evaluate and rene the

goals and benchmarks as strategies shit.

Engage with unders on ndings and processes.

Next, based on what has been learned:

Organizers rene their strategy.

Organizers and evaluators rene evaluation practices and maintain on-going

evaluation.

Building on the ScaoldingWhat We Have Learned

T he basic elements o community organizing evaluation are relativelystraightorward, but how does an evaluator build on them in practice to create alearning partnership that eels authentic to organizers and yields useul results?

Categorize Organizing Goals Systematically

We have learned that when determining evaluation questions, setting benchmarks,

and selecting data collection methods, it helps to categorize the work in a way that

incorporates the values and orientation o organizing. One useul ramework has

3 An organizers theory of change describes, in broad terms, the ultimate goals of the organizing effortand the logical chain of activities and interim outcomes that the organizers believe will get them tothose goals.


8/16


been developed by the Alliance or Justice (AFJ).4 We adapted the AFJ rame, using the

ollowing core components o organizing:

Participation and membership (the base): This includes, but is not limited to,

the numbers o members o an organizing group, their attendance at organizing

events, and the extent to which they represent the community being organized.

It encompasses their engagement in issue and strategy selection and their

identication with the goals o the organizing group.

Constituent leadership and power: Flowing rom participation, real leadership

is the distinguishing hallmark o community organizing, as opposed to advocacy

or other orms o social change work. As members gain skills, take ownership

o change processes, and begin to lead others in their community, they become

leaders. They then use that leadership capacity to gain power and make change.

Organizational power: The collective power o a community organizing group is

what gives it the ability to name problems, demand specic solutions, and hold

those in power accountable. It gives the organization visibility and leverage.

Organizing wins: Wins are the most valued currency in community

organizing. When organizers and leaders succeed in a campaign, show power

in a demonstrable way, or get what they seek in an ask, they chalk up a win.

A win may not be as obvious as passage o a desired policy. It may also be a

key step along the way, the prevention o a less-desired outcome, or a shit in

relationships.

Meaningul impact ollowing wins: Implementation and accountability ater a

win are as important as the win itsel. Having the power to realize the promise o

a win is critical in organizing.

Organizational capacity: As with any successul social change entity, acommunity organizing group requires structures, resources, knowledge, and

vision to achieve its goals. Given the way community organizing operates, the

ability to orge and strengthen relationships, create strategic alliances, and

manage a base are vital capacities in community organizing.

Refection and innovation: An organizing groups process or sel-assessment,

renement o strategy, and adaptability in a rapidly shiting environment are key

to separating skilled organizing rom mere activism (direct action that may not

be grounded in strategy or oriented toward sustained and eective ollow-up or

realignment o power relationships). Clearly, this aspect o organizing has strong

resonance in the evaluation context.

4 This Core Components of Community Organizing framework and additional useful information onevaluating organizing can be found at AFJs Resources for Evaluating Community Organizing website,http://www.afj.org/for-nonprofits-foundations/reco/.

When developing

the evaluation, the

core components of

organizing can be a

guide.


9/16


Crat Evaluation Questions Meaningul to Organizers

We have engaged both unders and grantees in the development o evaluationquestions at the very start. This keeps the evaluation relevant and useul, andpromotes greater buy-in and participation by the organizers. As we continuously

look to organizers or cooperation in data collection and or engagement on

interpretations and ndings, it has been quite helpul that many o the questions we

are asking are the ones they want answered.

Evaluation questions are highly individualized to the needs o the evaluation audience

and sponsors. We ound, however, that in our organizing evaluations, they generally

all within the core organizing component categories listed above. By reerencing

the components when developing questions, it may be easier to see what is missing

and to surace any unintended biases in proposed areas o inquiry. Training attention

on organizational capacity and constituent leadership and power, or example, can

balance what might otherwise be a more exclusive ocus on policy wins.

Set Clear Interim Benchmarks, and Be Willing to Reine Them

Data collection and analysis may seem like the hardest work o an evaluation.Yet setting meaningul, sucient, and achievable interim benchmarks that areexpected to occur between activities and longer-term social change goals can oten

be a major struggle in organizing work. With varying degrees o success, we have

encouraged hard thinking about interim benchmarks ocused more on substantive

outcomes. We have ound it can be helpul to look explicitly at the ull range o

organizing components and the ways in which they interrelate when setting

meaningul interim benchmarks. Similarly, we have relied on regular, in-depth

dialogue with organizers about the way progress within each component develops

and what it looks like, since organizing benchmarks rarely remain static over time.

Reerring back to the relatively high-level components o organizing we set orthearlier adds a consistency to the work when the shiting environment in which

organizers operate causes strategies and tactics to evolve or quickly change.

The table on page 15 illustrates examples o some o the benchmarks we have used

and the corresponding data collection methods used to track progress toward those

benchmarks.

Use Data Collection Methods that Fit with Organizing Culture

We have used a wide array o data collection methods in our organizingevaluations. We ound that we obtain the most useul inormation, with themost productive organizer participation, by using methods that t the style and

existing practices o the organizing group and are tailored to capture the dierent

components o organizing. When possible, we support organizers in creating their

own data collection tools.

For example, we worked with a multi-issue, aith-based community organizing

group in Caliornia on evaluating their eorts to expand access to health care in

their county. They wanted to look deeply at the strength and breadth o the local

organizing committees o their member congregations (aspects o Participation


10/16


and Membership and Constituent Leadership and Power). These committees

were the core teams working to move the organizing groups health access agenda.

The organizers led an inclusive, thorough process to select the key participation

measures they wished to track. They then worked with a consultant to customize

ree sotware (Zoho) that would allow each organizer to track their leaders by

committee (even when out in the eld, using their iPhones), and to provide clear

charts and graphs that highlighted progress and areas or growth.

Looking at a very dierent aspect o growth, we helped education organizers in

Pennsylvania think about how to document their progress in building external

champions. These champions are key indicators o development o organizational

power in order to leverage wins. The organizers created an online survey tool that they

update regularly, which tracks relationships and interactions with specic targets over

time. We can reer to the results as we assess progress, while the organizers can use

the data to rene their own strategies and activities. We have shared the templates

or this and other organizer data collection tools with other organizing groups in the

multi-site collaborative in which the Pennsylvania organizers participate. We plan to

post them on a private website so that organizers across sites can adapt each others

methods (taking care not to publicize matters that reveal too much about strategy inways that could jeopardize the organizing work).

One other methodology that we have ound particularly productive across almost

all aspects o organizing work is what we call the critical incident debrie.5

Immediately ater a win, loss, or infection point in an organizing drive, we conduct a

series o 360-degree interviews with individuals and groups both inside and outside

o the campaign, including opponents, allies, observers, and decision makers. We

analyze what took place, what went well, what could be improved, what growth

occurred, and what can be learned as the work moves orward. We write up our

ndings in a narrative or the organizers to use internally and share with their

unders. Our ndings are generally revealing, sometimes raw, sometimes validating,and close enough in time to be actionable. In one example, our debrie ollowing a

coalitions painul legislative loss in Colorado helped provide insight on leadership

and strategy that guided the coalition in a renewed eort to move the bill during the

next legislative cycle.

Develop Buy-In and Trust to Ensure Successul Implementation

Successul implementation o an organizing evaluation hinges on the relationshipbetween the organizers and the evaluators, and on the extent to which theevaluation meets the organizers where they are.

Our evaluation eorts have sparked the most learning and have worked best when

we have had internal evaluation champions within the organizing group. The

champions help ensure engaged participation, which we rely upon or data collection,

interpretation, and use. For example, we have worked with executive directors

o organizing groups whose enthusiasm or evaluation has created a culture o

5 Our methodology was inspired by the Intense Period Debrief focus group protocol used byInnovation Network. http://www.innonet.org/?section_id=101&content_id=581. See also Bagnell Stuart,J. (2007). Necessity leads to innovative evaluation approach and practice. The Evaluation Exchange,13(1), 10-11.

Evaluations tend

to work best when

internal evaluation

champions areengaged.


11/16


learning and inquiry within their organizations. This has led them to inuse data and

documentation into the organizations refection practices and to institutionalize

evaluation. An organizing group in Denver, or example, used one o our debrieng

narratives to call a meeting with their unders to start a dialogue on some o the

provocative questions we raised. This led to a productive exchange and noteworthy

shit in power relationships and credibility or the organizing grantees. Our work with

a Caliornia group gave the executive director and a key sta person the opportunity

to develop and implement a new suite o planning, tracking, and refection protocols

that the organizers now use routinely.

But we have also seen that it is important that the evaluation is widely held within an

organization, not merely by the champion. Absent broader buy-in, evaluators may not

receive consistent, complete, and sustainable fows o inormation, particularly given

how busy organizers can be and the usual rate o turnover in organizing groups.

Our evaluation champions all were naturally attuned to learning, but our ability

to cultivate them as evaluation champions depended on our eorts to develop

relationships with them. This relationship-building takes time, and requires evaluators

to work to earn the organizers trust. Honesty, preservation o the condentialityo strategic inormation, using the unique language o organizing, and showing

understanding o the groups organizing philosophy all make a real dierence.

Similarly, demonstrated interest in digging deep into the policy and political issues

the organizers are conronting, and understanding their political environment and

the players in it, yields major dividends. For some organizers, that means having the

kind o one-to-one relational meetings that organizers have with their leaders. For

others, it means poring over blog posts and news accounts to get a detailed grounding

in the political realities o a campaign so that we can speak more as partners and use

organizers time respectully. Ultimately, the relationship and the evaluation need to

have enough value to the organizers to justiy the expense o time and eort. And the

relationship needs to be strong enough that evaluators can maintain the ongoing datacollection and eedback activities that prospective, real-time evaluation requires, even

when the eedback delivered may not be what the organizer wants to hear.

Beyond the basics o interpersonal relationship-building, in our work with organizers

we have secured additional stipends or the organizers to assist with local data

collection. While the stipends do not ully compensate the organizers or the time they

spend on evaluation, the act that we sought the stipends is a gesture o respect and

helps solidiy our partnership.

We have also ound it important to respect the culture and existing organizational

systems and approaches o our organizing partners. In some cases, that means using

more rudimentary data collection approaches than desired, or relying on existing

systems. For others, it means waiting until the organization can more organically

arrive at systems that have meaning to them and are sustainable. For example, we

created an online media tracking tool or one organization, which they modied and

piloted. They determined it was not a process they could maintain, nor was it yielding

the inormation they really needed. We gave them the space to create what they

needed and they now are ocusing on tracking the growth o their relationships with

journalists in targeted markets rather than logging coverage. They have ound this

useul and so have we.

When developing

data collection

systems, it is

important to respect

the culture of

organizing groups

and their existing

evaluation capacity

or systems.


12/16


Set Achievable Expectations to Maximize Learning and Sustainability

For organizations that were building internal evaluation mechanisms rather thanrelying on external evaluation, we have encouraged them to go slow and startsmall. That may require ocusing on a single aspect o the work. The organization

that examined its local organizing committees using Zoho sotware, or example,

was initially going to evaluate its health policy work more directly. Instead, the

leadership development work provided a comortable and helpul entry point.Starting with activities that oer useul, timely insights into ongoing work has

been most eective in our organizing evaluations to date. In act, despite our

recommendations, some organizers with whom we worked have been so excited

about implementing new tracking and evaluation systems that they have orged

ahead, only to become overwhelmed part-way through. Our eagerness to see these

organizations embrace evaluation probably inhibited us rom orcing the issue and

requiring them to scale down their expectations and not bite o more than their

organization could handle. As approaches have been tested, they have learned and

retooled the approaches to work or them in the longer term.

Use Evaluation Results to Create Impact, But Recognize theChallenges o Getting Results in Real-Time

We rmly believe that evaluation has value only i the results are used. Wehave been gratied by the extent to which organizers and their unders havelearned rom our evaluation eorts and acted on what they have learned. We have

seen that ownership o the data and willingness to learn (even rom hard truths)

can lead to powerul impact. The Colorado organizers who have used evaluation to

rene campaign strategy and explore issues with unders in a new way are just some

examples. One coalition o organizing partners in Chicago is using evaluation data

on a win to make certain that implementation realizes its ull potential. Another

Chicago organizer coalition is learning rom our debrieng on a win to ensure thateorts to repeat the success are executed most eectively. Several organizers

elsewhere are rening their base-building eorts ater examining areas or growth

revealed through data.

Using results well, however, is not always easy. Timeliness matters, and providing

real-time or close-in-time data is oten challenging. Collecting enough data rom

a sucient number and range o sources is not quick, particularly when the

inormation sources are external to the organizing eort. Organizers assisting with

data collection may be too busy to collect data regularly enough or it to be useul.

While we attempt to provide eedback as rapidly as possible, we have realized over

time that we needed to be realistic with organizers about how long it can take to getsucient and accurate inormation back to them. Nevertheless, we continue to try

to make our eedback happen as quickly as possible.

It can also be dicult to propagate the learning in ongoing, ast-paced, and sensitive

campaigns. Similarly, the politics and complex communications paths in coalitions

and networks can slow down or complicate inormation sharing, particularly when

there are questions o attribution or responsibility. We have ound it helpul to be

as patient, ocused, and transparent as possible and to provide our analysis in an

accessible way. Liting up examples rom peer organizing groups has also eectively


13/16


motivated organizers to ocus on and use evaluation themselves.

Clariy What Success Means

Dening and measuring success plays a critical role in evaluation, whether thatmeans a policy win, the achievement o learning goals, or achieving growth inorganizational capacity. Through our work in a variety o social change contexts

and with a range o unders, it has become clear how important it is to suraceassumptions about what success looks like rom the perspective o an organizer,

versus an advocate, versus a under (and even among dierent types o unders).

As we have noted, success or organizers encompasses leadership development

and positive shits in power dynamics as well as desired policy change. Success or

others may be more limited. In addition, our evaluation work with organizers and

advocates has reinorced that or many organizers who work in a values-based

context, how you win is oten as important as winning itsel. Passing a bill may be an

important goal, but it may be equally important or the individual leaders pushing

or that passage to have their role and identity ramed in a particular way as the bill

is considered, or or the compromises to evince a particular power alignment. Thismay not be as true or some allied advocates or unders. Evaluation o an organizing

campaign needs to account or such nuances when measuring success.

Moreover, unders o organizing may dier about the role that organizing can and

should play in a policy success, depending on whether they come at the grant as

issue-oriented unders or as community or civic engagement unders. Issue-based

unders (health care, education, environment, etc.) may see organizing as simply one

tool to be used in reaching a policy change or social change goal, while community

or civic engagement unders may see organizing as the central component o

the work and an end in and o itsel. When raming the evaluation questions and

analyzing the results, evaluators must ensure that there is clarity on the undersorientation and how unders articulate success.

Conclusion

We have learned rom our work with organizers that expertise comes in manyorms and rom a variety o perspectives. This brie presents our perspectiveon evaluating community organizing based on the evaluations we have conducted

to date. We are still learning, and there is much we would like to develop urther.

How can we provide more useul, usable tools that t with organizing systems and

culture? How can we nd ways to get valid eedback to organizers more quickly?

How can we balance the needs and expectations o unders, organizers, and

community leaders? We see this brie as part o a continuing conversation on how to

evaluate community organizing in a way that benets all involved. We look orward

to more discussions that validate, challenge, or rene our thinking, and we welcome

new voices to the conversation.

As we continue our evaluation eorts, we will work to uncover and communicate

new lessons or the eld, and bring the voices o organizers into the conversation. As

part o our recent evaluation capacity- building eorts with community organizing

grantees o The Caliornia Endowment, we developed a series o videos in which

Success for

organizers

encompasses

leadership

development and

positive shifts in

power dynamics

as well as desired

policy change.


14/16


organizers discuss their experiences developing and conducting evaluations o their

eorts. These can be ound at http://www.blueprintrd.com/do-it-yoursel-community-

organizing-evaluation. We invite you to take a look and to join our conversation.

About the Authors

Catherine Crystal Foster is an independent strategist, advocate, and evaluator

with over 20 years experience in the nonprot community. She serves a strategicpartner o the San Francisco-based philanthropy strategy rm Blueprint Research

& Design, where she co-leads evaluations o policy, advocacy, and community

organizing initiatives. Catherine also partners with nonprot organizations and

unders to advance their policy and advocacy activities. These partnerships are

grounded in Catherines experience directing child advocacy and welare policy

initiatives beore opening her consulting practice. She has also served on numerous

nonprot boards o directors and is an active participant in civic aairs in her home

city o Palo Alto, Caliornia. Catherine can be reached at [email protected].

Justin Louie is a senior analyst at Blueprint Research & Design, and has spent the

last 13 years developing and conducting evaluations or nonprots, or-protcorporations, oundations and government agencies. He leads Blueprints practice

area in organizational learning and evaluation. Early on in his career he was a

community and youth organizer ocused on HIV prevention, and he has served

on the boards o a number o grassroots, state, and national policy and political

organizations. He has degrees in sociology (BA) and public policy (MPP) rom the

University o Caliornia, Berkeley. Justin can be reached at [email protected].

About Blueprint Research & Design, Inc.

Blueprint Research & Design helps grantmaking oundations, individual andamily donors, and philanthropic networks achieve their missions. Blueprint

believes that philanthropy is a critical part o our global society and that research

and systems thinking help individual organizations best achieve their goals. Services

include program strategy and design; organizational learning and evaluation;

and philanthropic industry analysis. Blueprint has been a leader in the advocacy

evaluation eld, and with support rom The Caliornia Endowment, produced a

seminal report on the eld in 2005 titled The Challenge of Assessing Policy and

Advocacy Activities: Strategies for a Prospective Evaluation Approach. That report

can be ound at www.blueprintrd.com.

About the Center or Evaluation Innovation

The Center or Evaluation Innovation is pushing evaluation practice in newdirections and into new arenas. The Center specializes in areas that are hard tomeasure and where resh thinking and new approaches are required. This includes

advocacy and policy change, communications, and systems change eorts. The

Center works with other organizations to develop and then share new ideas and

solutions to evaluation challenges through research, communications, training

development, and convening. www.evaluationinnovation.org.

This brief was developed

with generous supportfrom The California

Endowment, with

special thanks to Astrid

Hendricks and Gigi

Barsoum.

Acknowledgements


15/16


About Advocacy Evaluation Advances

In 2009, 120 advocates, evaluators and unders gathered at The CaliorniaEndowments Center or Healthy Communities or two days o thought-provokingpresentations and discussions on recent advocacy evaluation advances. The

convening, sponsored by The Caliornia Endowment with support rom The Atlantic

Philanthropies and Annie E. Casey Foundation, ocused on real-lie experiences

with advocacy evaluation and what has been learned rom testing dierent toolsand approaches in this emerging eld over the last several years. It also ocused on

challenges that still must be addressed, and identied priorities or the eld moving

orward. Some o the content and ndings eatured in this brie were presented and

discussed during the convening. To access other convening and presenter resources,

visit theAdvocacy Evaluation Advances web page at www.calendow.org/article.

aspx?id=3774.


16/16

1 This is a form of participatory monitoring and evaluation developed by Rick Davies and Jess Dart that involves the participants collectionand discussion of stories about the most significant changes resulting from a program or action. See http://www.mande.co.uk/docs/MSCGuide.pdf.

Sample Benchmarks and Data Collection Methods or the Core Components o Organizing

O Compo Bcmk D Coc Mod

PartiCiPatiOn anDMeMBershiP

COnstituent leaDershiPanD POwer

OrganizatiOnal POwer

Organizing wins

Meaningful iMPaCtfOllOwing wins

OrganizatiOnalCaPaCity

refleCtiOn anDinnOvatiOn

Changes in numbers, demographicsor location o members

Changes in attendance (numbers, types

o events, who attends)

Changes in attitudes, skills, andknowledge

Changes in sel-esteem and sel-eicacy

Changes in stature within communityor among decision makers

Development o relationships withdecision makers, media, and inluentialigures

Changes in stature within communityor among decision makers

Changes in membership

Changes in turnout to events

Policy wins

Policy wins

Shits in norms or content o debate

Holding the line against negativeactions

Implementation o policies

Changes in practices

Public accountability or action orinaction

Sustained shits in norms or contento debate

Impact on community

Changes staing

Changes in inrastructure

Changes in skills

Changes in resources

Building on and systematizing internalprocesses

Inusing data and documentation intorelection

Use o data in reinement o strategyor tactics

Membership tracking (includingdemographic and geographic ino)

Attendance tracking

Tracking elements o leadership growth

Organizer check-ins and debries

Documenting 1-to-1s

Journaling/portolios

Focus groups

Power analysis

Relationship/champion tracking

Base-building/mobilization tracking

Media tracking

Policy developments tracking

Interviews

Critical incident debries or case studies

Policy tracking

Collection o archival documents

Media tracking


Policy implementation tracking

Community indicators tracking

Action research (accountability surveys,interviews, ocus groups)

Interviews


Organizational capacity assessments

Most Signiicant Change1

Interviews and check-in calls ormeetings

Interviews and check-in calls ormeetings

Collection o assessment documents orexamination o systems

evaluating community organizing (2)

Documents