integrating evaluative inquiry into the organizational … · 2017. 5. 23. · la revue canadienne...

99LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME

Abstract:

Résumé:

Corresponding author: B. Cousins, Faculty of Education, University of Ottawa,145 Jean Jacques Lussier, Ottawa, ON Canada, K1N 6N5;

The Canadian Journal of Program Evaluation Vol. 19 No. 2 Pages 99–141ISSN 0834-1516 Copyright © 2004 Canadian Evaluation Society

INTEGRATING EVALUATIVE INQUIRY INTOTHE ORGANIZATIONAL CULTURE: A REVIEWAND SYNTHESIS OF THE KNOWLEDGE BASE

J. Bradley CousinsSwee C. GohShannon ClarkUniversity of OttawaOttawa, Ontario

Linda E. LeeProactive Information Service Inc.Winnipeg, Manitoba

The purpose of this article is to explore, through an extensivereview and integration of recent scholarly literature, the con-ceptual interconnections and linkages among developments inthe domains of evaluation utilization, evaluation capacity build-ing, and organizational learning. Our goal is to describe andcritique the current state of the knowledge base concerning thegeneral problem of integrating evaluation into the organiza-tional culture. We located and reviewed 36 recent empirical stud-ies and used them to elaborate a conceptual framework thatwas partially based on prior work. Methodologically, our resultsshow that research in this area is underdeveloped. Substantive-ly, they show that organizational readiness for evaluation maybe favourably influenced through direct evaluation capacitybuilding (ECB) initiatives and indirectly through doing and us-ing evaluation. We discuss these results in terms of an agendafor ongoing research and implications for practice.

Cet article a pour but d’explorer, grâce à un examen poussé et àl’intégration de la littérature académique récente, les intercon-nexions et liens conceptuels entre les faits nouveaux dans lesdomaines de l’utilisation de l’évaluation, du renforcement descapacités d’évaluation, et de l’apprentissage organisationnel.L’objectif est de décrire et de critiquer l’état actuel de la base deconnaissances en ce qui a trait au problème général de l’inté-gration de l’évaluation à la culture organisationnelle. Nous avonsidentifié et examiné 36 études empiriques récentes utilisées pour

THE CANADIAN JOURNAL OF PROGRAM EVALUATION100

élaborer un cadre conceptuel partiellement fondé sur des tra-vaux antérieurs. Du point de vue méthodologique, les résultatsdémontrent que la recherche dans ce domaine est encore sous-développée. Du point de vue «validation», ils démontrent quel’état de préparation de l’organisation à l’évaluation peut êtrefavorablement influencée directement par des initiatives de ren-forcement des capacités d’évaluation (RCE) et indirectement parl’exécution et l’utilisation d’évaluations. La discussion proposeun plan pour la recherche continue et identifie les implicationspour la pratique.

The concept of evaluation utilization has been of inter-est to evaluation scholars and practitioners for quite some time. Thisparticular domain of interest has undergone considerable evolutionand development due in no small measure to the pioneering effortsof such noted theorists and researchers as Weiss (1981), Alkin,Daillak, and White (1979), and Patton et al. (1977). Perhaps amongthe most dramatic early developments in the conceptualization ofthe utilization construct was the recognition that its use need notbe limited to directly observable, instrumental consequences of evalu-ation findings and recommendations; conceptual development orlearning about the object being evaluated was identified as a legiti-mate if not pervasive consequence of evaluation activity. Indeed,such learning might be conceived to be an instance of “capacity build-ing,” a topic of considerable interest within evaluation circles as webegin the new millennium.

In response to a call from the president of the American EvaluationAssociation (AEA), Milstein and Cotton (2000) opened up the poten-tial for dialogue among professional evaluators about concepts rel-evant to “evaluation capacity building” (ECB) by providing athoughtful framework for consideration. They define evaluation ca-pacity as “the ability to conduct an effective evaluation (i.e., one thatmeets accepted standards of the discipline)” (pp. 1–2). They thenproceed to differentiate ECB from other kinds of capacity buildingsuch as the ability of individuals, organizations, or communities toachieve broad social or organizational goals. Presumably the kindsof instrumental and conceptual consequences of evaluation men-tioned above would fall into this latter category. To the extent thatevaluation helps individuals, organizations, or communities under-stand phenomena of interest or take appropriate actions in order toachieve valued goals, it would be considered a capacity building ex-ercise, but one that is differentiated from ECB per se. In their words,“while undeniably important, this form of capacity is different from


the capacity needed to conduct effective evaluation activities in thefirst place” (Milstein & Cotton, p. 2).

A recent volume of New Directions for Evaluation by Compton, Baizer-man, and Stockdill (2002) takes this perspective on ECB one step fur-ther. These authors frame ECB not only in terms of the ability to doquality evaluation but also to use it within the organizational context.Specifically, they define the term, albeit unparsimoniously, as:

A context-dependent intentional action system of guidedprocesses and practices for bringing about and sustain-ing a state of affairs in which quality program evalua-tion and its appropriate uses are ordinary and ongoingpractices within and/or between one or more organiza-tions/programs/sites. (Stockdill, Baizerman, & Compton,2002, p. 8, emphasis added)

In this conceptual definition the consequences of evaluation, in termsof its appropriate uses, are thrown into the mix. We would arguethat such an extension is laudable and that a sharp distinction be-tween ECB and capacity building associated with the consequences(i.e., uses) of evaluation is unwarranted and perhaps counterpro-ductive. Moreover, the integration of evaluation into the culture oforganizations has, in our view, as much to do with the consequencesof evaluation as it does the development of skills and knowledge ofevaluation logic and methods.

From our point of view, there is much to be learned from an integra-tion of what we know about evaluation utilization and ECB. Wechoose to frame evaluation as an organizational learning system. Inother words, evaluative inquiry, as part and parcel of organizationalculture and operations, has the potential to serve as a potent meansfor organizations to develop their organizational learning capacity(OLC). Through doing evaluation and developing the capacity to doit, organizations become more adroit in constructing shared repre-sentations of knowledge and structures, predisposed to generate newknowledge, inclined to capture and interpret external information,and apt to question basic assumptions about the organization, itsgoals, and strategies for achieving them. Such consequences, in theparlance of evaluation utilization, represent group or organizationlevel “process use.”

The purpose of this article is to explore, through an extensive re-view and integration of recent scholarly literature, the conceptual


interconnections and linkages among developments in the domainsof evaluation utilization, evaluation capacity building, and organi-zational learning. We argue that the capacity to do effective evalua-tion is inextricably linked to the kinds of capacity building outcomesof evaluation that we might generally associate with evaluation uti-lization. Further, we propose that ECB is tightly connected to or-ganizational development and capacity building to the extent thatit contributes to an organization’s propensity to learn. A product ofour review and integration is a conceptual framework that linksevaluation activities and consequences with organizational capac-ity building. We begin by introducing this framework and thenproceed to systematically review and integrate recent empirical lit-erature using it as an organizing mechanism. Finally, we develop aresearch agenda around the problem of integrating evaluative in-quiry into the organizational culture.

CONCEPTUAL FRAMEWORK

Using some of our prior conceptual work (Cousins, 1996, 2003; Goh,2000; Goh & Richards, 1997) as a platform, we developed the theo-retical framework that appears in Figure 1 through reading, crit-iquing, and synthesizing recent scholarly literature linkingevaluation to organizational development and learning. The frame-work is a conceptual representation of the key variables of interest(spheres) and relationships among them (arrows). In this sectionwe describe the framework, which then serves as an advance organ-izer for the review of empirical literature to follow.

The spheres of interest are roughly divided into those associatedwith the organization itself and those associated with evaluationactivities and consequences situated within the organizational con-text. Organizational consequences are shown to depend on an or-ganization’s capacity to learn, which in turn is sustained by variousorganizational support structures. One such support structurewould be evaluative inquiry that leads to various consequences in-cluding the use of evaluation findings. We considered evaluativeconsequences to be organizational consequences in this context.Through the use of evaluative inquiry and experiences in usingevaluation, the organization develops its evaluation capacity. Suchcapacity becomes integrated with OLC, manifest in the develop-ment of the cultural variable “organizational readiness for evalua-tion.” We now consider the individual components of the frameworkmore directly.


Figure 1Conceptual Framework of Evaluative Inquiry as an Organizational Learning System

Organizational Dimensions

Organizational learning is a psychosocial construct that has gar-nered considerable attention from organization theorists, educationaladministrators, and other corporate managers over the past decade(Cousins, 1996; Fiol & Lyles, 1985; Garvin, 1993; Goh & Richards,1997; Huber, 1991; Leithwood, Aitken, & Jantzi, 2001). It is wellknown that organizations differ quite enormously in their ability tolearn and adapt to rapidly changing environments (Fiol & Lyles,1985; Senge, 1990), but what is not readily understood is how tofoster the development of significant and sustainable OLC (Garvin,1993).

Organizational Consequences

Fundamental to conceptions of organizational learning is the devel-opment among organization members of shared mental representa-tions or understandings of the organization and how it operates.

EVALUATIVEINQUIRY

ParticipationMethod choice

Internal/externalPurpose

ECB

ORGANIZATIONALPARAMETERS

EVALUATIONCAPACITY

EVALUATIONCONSEQUENCES

ORGANIZATIONALCONSEQUENCES

ORGANIZATIONALSUPPORT

STRUCTURES

Evaluation Evaluation

knowledge and skilllogic

Organizationalreadiness

for evaluation

Communication structures

Reward systems Professionaldevelopment

Knowledgeproduction

Processuse

Use of findings:•Instrumental•Conceptual•Symbolic

Shared representation

Question basic assumptions

Problem solving anddecision making

Goalattainment

Mission

ORGANIZATIONALLEARNINGCAPACITY

Leadership Teamwork

Cultureof experi-mentationKnowledge transfer


Most theorists agree that organizational learning cannot happen inthe absence of individual learning by organization members. Thismulti-dimensional construct ranges from low-level, first-order, orsingle-loop learning, where change is incremental, to high-level, sec-ond-order, or double-loop learning where fundamental assumptionsabout the organization and its operation are brought to the surface,questioned, and ultimately altered (Fiol & Lyles, 1985; Huber, 1991;Lant & Mezias, 1992; Lundberg, 1989).

Organizational Learning Capacity

The allure of understanding organizations as learning systems ispowerful, but as Garvin (1993) suggests, it is imperative that wemove on to clearer guidelines for practice and operational advice asto how to build a learning organization. To this end Goh (2000) syn-thesized and integrated the theoretical literature and argues thatlearning organizations have the following core strategic buildingblocks:

• Mission and vision: clarity and employee support of themission, strategy, and espoused values of the organization;

• Leadership: leadership that is perceived as empoweringemployees, encouraging an experimenting culture, andshowing strong commitment to the organization;

• Experimentation: a strong culture of experimentation thatis rewarded and supported at all levels in the organization;

• Transfer of knowledge: the ability of an organization totransfer knowledge within and from outside the organiza-tion and to learn from failures;

• Teamwork and cooperation: an emphasis on teamwork andgroup problem-solving as the mode of operation and for de-veloping innovative ideas.

Although they are presented as separate dimensions, Goh (2000)proposed that these building blocks are interdependent and mutu-ally supportive conditions in a learning organization; in essence theydefine an organization’s learning capacity. Such capacity determinesthe extent to which organizational consequences occur.

Organizational Support Structures

OLC building blocks are also said to depend on organizational struc-tures and supports such as low job formalization and the acquisi-tion of appropriate knowledge and skills by organization members


(Goh, 2000). Reward systems, in the form of formal and informalincentive mechanisms, also represent structural support for OLC,as do various communication structures within the organization,which serve to foster the horizontal and vertical flow of knowledgeand information. Professional development activities, formal andinformal, represent yet another organizational support structure.

Evaluation Dimensions

We define evaluation as systematic inquiry leading to judgementsabout program (or organization) merit, worth, and significance, andsupport for program (or organizational) decision making. Judgementimplies making comparisons between systematically gathered data(observation) and some standard or basis for comparison (anotherprogram, performance at a previous point in time, an external bench-mark). We include in our definition such activities as “needs assess-ment,” systematic inquiry into identified needs to which programs(or organizations) are intended to respond, and “monitoring” sys-tematic observation that does not necessarily imply judgement butnevertheless provides support for decision making.

Evaluative Inquiry

Evaluation purposes range from hard-nosed judgement-orientedsummative exercises to formative, improvement-oriented ones in-tended solely to provide support for decision making. These evalua-tions can be conducted by members or groups external to theorganization, by those internal to it, or by a blend of the two. Itvaries in the extent to which it involves non-evaluator stakeholders(i.e., those not trained as evaluators), ranging from exclusive use assources of data and information to deep and full partnerships be-tween evaluators and non-evaluator stakeholders in carrying outthe evaluation (Cousins & Whitmore, 1998). Multiple methods canbe employed in evaluation, ranging from quantitative to qualitativeand often involving a mix of the two. In some cases, method choiceimplies deeper epistemological considerations, but it is importantto distinguish between method and methodology (Kushner, 2002).

We consider ECB to be a special case of evaluative inquiry. We con-cur with the definition of ECB put forward by Stockdill et al. (2002)(see above), but are of the view that actions associated with devel-oping infrastructure, processes, and procedures that work to developevaluation capacity represent interventions that have a distinct


evaluative component (Preskill & Torres, 1999). And so, evaluativeinquiry can lead to various forms of utilization or more directly andintentionally to building evaluation capacity.

Evaluative inquiry, within our framework, is conceived to be an or-ganizational support structure — not unlike communication struc-tures, professional development structures, and the like — with thepotential to foster organizational learning. It is enhanced and dis-tinguished in Figure 1 because it provides the central focus for ourpaper. We are interested to know how evaluative inquiry can be in-tegrated into the organizational culture. It is important to note thatevaluative inquiry is a structure that can in and of itself be sup-ported by other organizational support structures, such as trainingand professional development, resources, and infrastructure.

Evaluation Consequences

Ultimately, evaluation leads to knowledge production, the validity,credibility, sophistication, timeliness, and relevance of which dependon the evaluation processes in place. It also leads to forms of the useof the knowledge produced. Conventional forms are captured underthe heading “use of findings,” including instrumental uses (supportfor discrete decisions), conceptual uses (enlightenment, learning),and symbolic uses (persuasion, compliance) (Shulha & Cousins,1997). But “process use,” a term coined by Patton, is an additionaldiscernable consequence of stakeholder participation in, or knowl-edge of, evaluation activities (Cousins, 1996; Patton, 1997, 1998;Preskill & Caracelli, 1997; Shulha & Cousins, 1997). By virtue oftheir proximity to the evaluation, stakeholders may develop in waysthat are quite independent of the findings or substantive knowledgeemerging from the inquiry. For example, participation has the po-tential to lead to the development of research skills and the capac-ity for self-critique, self-determination, and systematic inquiry atthe level of individual stakeholders. There may also be collectiveeffects at the group, team, or organizational level. Process use ofthis sort is of central interest to us in this article. We see it as amore indirect mode of evaluative inquiry leading to the developmentof evaluation capacity.

As pointed out above, we consider evaluation consequences to be aspecial case of organizational consequences. Instrumental, concep-tual, and symbolic uses of evaluative findings can be framed as ele-ments of organizational decision making, problem solving, andlearning.


Evaluation Capacity

Following Stockdill et al. (2002), evaluation capacity is representedby the organizational processes and practices that are in place tomake a quality evaluation and its uses routine. To the extent towhich evaluation capacity is present within the organization, mem-bers of the organization develop their knowledge of evaluation logicand methods. They also develop their skills in actually doing evalu-ation, including those associated with planning; instrument devel-opment; data collection, processing, and analysis; interpretation; andreporting and follow-up. In essence, through sustained evaluativeinquiry and, in particular, continued and routine use of evaluationfindings and processes, evaluation capacity becomes integratedwithin the organizational culture. It becomes what might be labeled“organizational readiness for evaluation,” a term originally coinedby Mayer (1975, cited by Seiden, 2000) in relation to Davis andSalasin’s (1975/1983) A-VICTORY model of organizational develop-ment and change.

Many interventions have been proposed as ways to enhance OLCwithin organizations, including strategic planning, employee train-ing initiatives, focused organizational development and school im-provement projects, and the installation of management informationsystems. Evaluative inquiry is an intervention that holds much prom-ise in this regard. The literature suggests that organizational readi-ness for evaluation is a cultural dimension that becomes part of OLC.In theory, if evaluation becomes integrated into the ongoing activi-ties within an organization, it may become a learning system thatfosters the development of shared values and understanding amongorganization members. Several evaluation theorists have recognizedthe potential for evaluation to foster such collective developmentaleffects (Cousins, 1996; Cousins & Earl, 1992; Forss, Cracknell, &Samset, 1994; Owen & Lambert, 1995; Patton, 1999; Preskill, 1994;Preskill & Torres, 1999; Torres, Preskill, & Piontek, 1996).

REVIEW AND INTEGRATION OF CURRENT EMPIRICALLITERATURE

In essence this article represents a form of meta-analysis of em-pirical research in the area. Described below are the sampling cri-teria, characteristics of the sample, and the method used to analyzefindings across studies. We then turn to an integration of the find-ings.


Sampling Criteria

Despite early interest in organizational readiness for evaluation,the construct appears to have moved off the radar screen of evalua-tion scholars until recently (Seiden, 2000). With burgeoning inter-est in evaluation’s relationship to organizational learning, the termhas once again come into favour among researchers. For this rea-son, we sampled available empirical research from within the pastfive years only. We implemented computer searches of standard databases — Educational Resources Information Clearinghouse (ERIC),Psychlit, Sociofile — to identify recent published work. Keywordsincluded “evaluative inquiry,” “organizational learning,” “organiza-tional readiness for evaluation,” and “evaluation utilization.” Wesampled only published research subject to peer review or disserta-tion/thesis committee review. From an initial sample, we trackeddown other works through bibliographic follow-up. We also scannedrecent issues of well-known evaluation journals (e.g., American Jour-nal of Evaluation, Canadian Journal of Program Evaluation, Evalu-ation, and Evaluation and Program Planning).

The resulting sample is therefore multidisciplinary with empiricalstudies reported in many different evaluation practice and organi-zational contexts. To be included, studies had to be empirical, thatis, based on systematic observation of evaluation or organizationalpractice. Organizational learning studies that were located had toinclude an evaluative inquiry element in order to be retained.

Sample Descriptive Characteristics

Ultimately we identified a sample of 36 studies, which are describedin Table 1. Fifteen of the studies are what we might loosely callempirical (e.g., Boaz, 2002; King, 2002; VanderPlaat, Samson, &Raven, 2001). That is to say, they are based on reflective accounts ofthe author (usually an evaluator) with regard to one or more spe-cific case examples. These narratives are based on observation andinterpretation of lived experiences with evaluation, yet authors donot specify methods for capturing their observations nor other rele-vant sources of evidence supporting the case. While rich in detailand insight, it is not possible for the analyst to judge the credibilityor trustworthiness of the account on the basis of conventional can-ons for social inquiry. These papers are narratives implicitly in ac-cordance with constructivist principles of inquiry and attendantrelativist epistemological assumptions.


Tabl

e 1

Desc

riptiv

e Ch

arac

teris

tics

of E

mpi

rical

Stu

dies

Stud

ySa

mpl

eCo

ntex

tM

etho

dsPu

rpos

e / R

esea

rch

Mai

n Fi

ndin

gsQu

est io

n(s)

Bal th

asar

&Ri

eder

(200

0)

Biot

t & C

ook

(200

0)

Boaz

(200

2)

Bran

don

& H

iga

(200

4)

Eval

uato

rs,

prog

ram

dire

ctor

san

d fe

dera

l of f i

ceof

ene

rgy

deci

sion

mak

ers

from

4 o

utof

40

eval

uatio

ns

Not a

pplic

able

Not a

pplic

able

41 s

choo

lad

min

istra

tors

and

teac

hers

in 1

7sc

hool

s

Seco

nd p

hase

of a

larg

erth

ree-

stud

y pr

ojec

t tha

tin

volv

ed th

e ev

alua

t ion

of th

e Sw

iss

Ener

gy20

00 p

rogr

am

Eval

uatio

n of

the

natio

nal E

arly

Yea

rsEx

celle

nce

Cent

res

Pilo

tPr

ogra

mm

e (E

YECP

P) in

the

UK

The

eval

uatio

n of

the

Bette

r Gov

ernm

ent f

orOl

der P

eopl

e (B

GOP)

inth

e UK

Scho

ol-b

ased

trai

ning

initi

ativ

e to

enh

ance

eval

uatio

n kn

owle

dge

and

skill

s in

the

USA

Met

a-ev

alua

t ion:

8in

terv

iews

plu

sdo

cum

ent r

evie

w in

each

of 4

cas

e st

udie

s

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

pers

pect

ive

ofev

alua

tor

Refle

ctiv

e ca

se s

tudy

:W

ritte

n fro

m p

oint

of

view

of t

he e

valu

ator

Long

itudi

nal f

ield

stud

y: q

uest

ionn

aire

san

d re

trosp

ectiv

ein

terv

iews

.

-The

form

s of

lear

ning

thro

ugh

eval

uat io

n-T

he c

ondi

t ions

of l

earn

ing

thro

ugh

eval

uat io

n

To e

xplo

re c

ross

cut

ting

eval

uatio

n is

sues

To e

xam

ine

how

evid

ence

-ba

sed

polic

y m

akin

gpr

esen

ts c

halle

nges

for

rese

arch

ers

Chan

ge in

atti

tude

s to

ward

eval

uatio

n; c

hang

e in

conf

iden

ce to

do

eval

ua-

tion;

per

cept

ions

of

eval

uatio

n ca

pabi

litie

s

Dif fe

rent

kin

ds o

f lea

rnin

g in

eac

h ca

se s

tudy

:di

rect

—wh

ere

feed

back

info

led

to th

e re

com

men

-da

t ions

of t

he e

valu

atio

ns b

eing

use

d, in

dire

ct—

new

appr

ecia

t ion

of th

e si

tuat

ion

is o

bser

ved

and

argu

men

ts c

an in

f luen

ce th

e co

mm

unic

atio

n of

polit

ical

issu

es. I

t is

nece

ssar

y to

dis

tingu

ish

the

type

s of

lear

ning

and

effe

ct o

f eva

luat

ions

at t

hele

vels

of o

pera

tion,

stru

ctur

e, a

nd s

trate

gy.

As th

e EY

ECPP

pro

ject

pro

gres

sed

ther

e we

rein

crea

sed

tens

ions

bet

ween

per

form

ance

man

age-

men

t and

par

ticip

ator

y ev

alua

tion.

Diff

icul

t to

dolo

cally

giv

en li

mite

d re

sour

ces

and

pres

sure

; key

to a

wor

thwh

ile ro

le fo

r loc

al e

valu

ator

s is

in th

eir

cont

ribut

ion

to le

arni

ng.

Rese

arch

ers

can

be m

ore

effe

ctiv

e in

max

imizi

ngut

iliza

tion

for p

olic

y an

d pr

actic

e if

they

go

beyo

nd th

e tra

ditio

nal r

oles

of d

eliv

erin

g re

ports

and

pres

entin

g fin

ding

s to

fund

ers.

The

like

lihoo

dof

use

for p

olic

y de

pend

ent o

n th

e fo

rm in

whi

chits

find

ings

are

mad

e av

aila

ble.

No im

prov

emen

t in

attit

udes

. Sel

f-con

fiden

ce a

sev

alua

tors

impr

oved

. Ass

essm

ent o

f cap

abili

ties

as e

valu

ator

s fa

vour

able

. (con

tinue

d on

nex

t pag

e)

THE CANADIAN JOURNAL OF PROGRAM EVALUATION110Ta

ble

1 (c

ontin

ued)

Stud

ySa

mpl

eCo

ntex

tM

etho

dsPu

rpos

e / R

esea

rch

Mai

n Fi

ndin

gsQu

est io

n(s)

Chac

on-

Mos

coso

et a

l .(2

002)

Com

pton

et a

l .(2

001)

Com

pton

et a

l.(2

002)

Conn

or &

Tanj

asiri

(199

9)

The

Hom

eles

sCh

i ld C

are

Serv

ices

Orga

niza

t ion

inSe

vil le

, Spa

in

Not a

ppl ic

able

Not a

pplic

able

2 pr

ojec

t dire

ctor

sin

Col

orad

oHe

alth

yCo

mm

uniti

esIn

itiat

ive

(CHC

I)

Form

ativ

e ev

alua

t ion

with

hom

eles

s ch

i ld c

are

cent

ers

in S

evi l l

e, S

pain

Impl

emen

tat io

n of

Col la

bora

t ive

Eval

uat io

nFe

llows

Pro

ject

(CEF

P),

invo

lvin

g Am

eric

anCa

ncer

Soc

iety

, gra

duat

est

uden

ts, a

nd fa

culty

inth

e US

A

ECB

effo

rts w

ithin

the

Amer

ican

Can

cer S

ocie

tysi

nce

1995

; nat

ionw

ide,

com

mun

ity-b

ased

,vo

lunt

eer o

rgan

izatio

n in

the

USA

Two

out o

f 29

com

mun

ities

that

are

impl

emen

ting

the

Colo

rado

Hea

lthy

Com

mun

ities

Initi

ativ

e(C

HCI)

in th

e US

A

Ref le

ctiv

e ca

se s

tudy

:wr

i t ten

from

poi

nt o

fvi

ew o

f eva

luat

ors

Ref le

ctiv

e ca

se s

tudy

:wr

i t ten

from

the

pers

pect

ives

of t

hose

invo

lved

in th

e de

sign

and

impl

emen

tatio

n of

the

CEFP

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

the

pers

pect

ives

of 4

ECB

prac

titio

ners

invo

lved

from

the

begi

nnin

g in

1995

Case

stu

dies

:pa

rtici

pant

obse

rvat

ions

of

eval

uato

rs;

disc

ussi

ons

with

proj

ect d

irect

ors

To d

escr

ibe

exam

ple

ofm

utua

l cat

alyt

ic m

odel

of

form

ativ

e ev

alua

t ion

To d

escr

ibe

CEFP

and

iden

t i fy

less

ons

lear

ned

afte

r firs

t 3 y

ears

of

impl

emen

tatio

n

To te

ll th

e st

ory

of E

CB a

tth

e AC

S th

roug

h lo

okin

g at

cont

ext,

hist

ory

of E

CB,

prac

tice

prin

cipl

es a

ndth

eir o

pera

tiona

lizat

ion,

asse

ssm

ent o

f ECB

, and

less

ons

lear

ned

To e

xam

ine

cros

s-cu

tting

eval

uatio

n is

sues

Mod

el is

impl

emen

ted

succ

essf

ully

; eva

luat

ion

woul

d no

t hav

e be

en a

s se

nsi t i

ve w

i thou

t the

act iv

e co

oper

atio

n of

loca

l pra

cti t i

oner

s an

dre

pres

enta

t ives

at t

he c

entre

; eva

luat

or k

ey to

proc

ess.

Less

ons

set t i

ng u

p CE

FP: a

dmin

istra

t ive

and

tech

supp

ort ;

annu

al t r

aini

ng, o

ppor

tuni

t ies

for c

onsu

l-ta

tion,

out

side

fund

ing,

eva

luat

ion

expe

rtise

,cl

arity

re: e

valu

atio

n. Im

plem

enta

tion

less

ons:

com

mon

fram

ewor

k; tr

aini

ng, p

rogr

am e

valu

atio

nex

perti

se, s

tude

nt e

ntry

ski

lls, t

ensi

ons

betw

een

repo

rt an

d th

esis

requ

irem

ents

, tim

e in

tens

ive,

ince

ntiv

es.

ECB

trigg

ered

by

chan

ges

in th

e la

rger

con

text

,es

peci

ally

refo

rm in

pro

gram

dev

elop

men

t.Va

rious

stra

tegi

es to

enh

ance

ECB

, but

mos

tef

fect

ive

is e

valu

atio

n fe

llows

hip

colla

bora

tive.

Seve

ral p

rinci

ples

gui

de E

CB, a

mon

g th

em is

eval

uate

eva

luat

ion

unit;

mos

tly in

form

al to

dat

ebu

t som

e fo

rmal

asp

ects

; tra

inin

g st

aff i

n us

e of

eval

uatio

n pr

oces

ses

and

prod

ucts

.

Citiz

en in

volv

emen

t fos

tere

d ow

ners

hip

of th

epr

ojec

t and

sug

gest

s th

at a

ctua

l use

will

be

mor

elik

ely.

Impo

rtanc

e of

a b

road

-bas

ed a

dvis

ory

coun

cil t

hat k

ept t

he p

roje

ct in

touc

h wi

th it

sco

mm

unity

. New

par

tner

ship

s we

re fo

rmed

.Ev

alua

tion

capa

city

in th

e co

mm

unity

dev

elop

ed.


Maj

or in

f luen

ce o

f fun

ding

age

ncy

expe

ctat

ions

,av

aila

bil i t

y of

eva

luat

ion

reso

urce

s, le

ader

ship

,an

d st

aff ,

and

avai

labi

l i ty

of e

valu

atio

n to

ols

and

tech

nolo

gy. S

tage

mod

el o

f EC:

com

plia

nce,

inve

stm

ent ,

adva

ncem

ent .

Mus

t tai

lor E

CB to

stag

e.

Cont

inua

l inv

olve

men

t of g

rant

ees

and

prog

ram

and

eval

uat io

n ex

perts

impo

rtant

to m

akin

ggu

idan

ce a

nd te

chni

cal a

ssis

tanc

e sy

stem

s m

ore

user

-frie

ndly

and

cos

t-ef fe

ctiv

e.

The

mod

el p

rovi

ded

a co

ncep

tual

fram

ewor

k fo

rth

e en

tire

eval

uatio

n pr

oces

s. E

nsur

ed c

lose

scru

tiny

of th

e co

ntex

tual

fact

ors;

bro

ught

the

need

s of

the

eval

uatio

n in

to fo

cus.

Bal

ance

dev

alua

tion

prod

uced

that

cou

ld s

peak

to th

e ne

eds

of d

ispa

rate

aud

ienc

es.

Eval

uatio

n re

quire

men

t was

initi

ally

per

ceiv

ed a

sbu

rden

som

e. M

any

alum

ni s

choo

ls s

usta

ined

the

prog

ram

’s ev

alua

tion

lega

cy e

vide

nced

thro

ugh

data

-driv

en g

oal-r

elat

ed in

quiry

, sys

tem

atic

refle

ctio

n an

d pl

anni

ng, a

nd th

e pr

esen

ce o

f an

eval

uatio

n-m

inde

d cu

lture

.

Prog

ress

revi

ewed

on

4 go

als

for 2

yea

rs o

f ECB

work

: (1)

dev

elop

sta

ff co

mm

itmen

t and

ski

lls in

prog

ram

eva

luat

ion

and

its u

se—

parti

al; (

2) b

uild

an in

frast

ruct

ure

for d

ata

colle

ctio

n, a

naly

sis,

and

pres

enta

tion—

parti

al; (

3) fa

cilit

ate

exis

ting

scho

olim

prov

emen

t pro

cess

—so

me;

(4) c

reat

e ne

twor

k

To d

escr

ibe

bel ie

fs a

ndat

t i tud

es to

ward

eva

lua-

t ion;

to id

ent i f

y fa

ctor

sin

f luen

cing

eva

luat

ion

capa

city

To a

sses

s th

e ef

f icac

y of

eval

uat io

n ca

paci

ty-

bui ld

ing

ini t i

ativ

e

To d

escr

ibe

the

appl

icat

ion

of th

e 10

-sta

ge m

odel

of

educ

atio

nal i

nnov

atio

nwi

thin

the

cont

ext o

ftra

nsfo

rmat

ion

To p

rovi

de e

vide

nce

ofho

w th

e pr

oces

ses

and

prac

tices

sur

roun

ding

the

use

of d

ata

for w

ise

deci

sion

-mak

ing

beco

me

intri

nsic

ally

der

ived

To d

escr

ibe

expe

rienc

e an

dex

plor

e is

sues

aris

ing

indi

stric

t’s e

ffort

to e

nhan

ceev

alua

tion

capa

city

; how

the

work

was

orig

inal

lyco

ncei

ved

and

how

and

Qual

i tat iv

e f ie

ld s

tudy

:co

nten

t ana

lysi

s of

audi

otap

ed in

terv

iew

trans

crip

ts

Case

stu

dy: c

onte

ntan

alys

is o

f tec

hnic

alas

sist

ance

requ

ests

that

cam

e in

dur

ing

one-

year

per

iod

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

pers

pect

ive

ofev

alua

tor

Expl

orat

ory

case

stud

y: s

emi-s

truct

ured

inte

rvie

ws

Refle

ctiv

e ca

se s

tudy

:re

trosp

ectiv

eob

serv

atio

ns o

f aut

hor,

a un

iver

sity

pro

fess

orhi

red

for 2

yea

rs a

s an

inte

rnal

eva

luat

or

8 si

tes

in w

hich

Cen

ters

for D

isea

se C

ontro

lfu

nded

HIV

pre

vent

ion

prog

ram

s; u

sual

ly 5

or

fewe

r ful

l -tim

eem

ploy

ees

in th

e US

A

Desc

ript io

n an

d an

alys

isof

eva

luat

ion

train

ing

and

tech

nica

l ass

ista

nce

ini t i

ativ

e in

the

cont

ext o

fHI

V pr

even

tion

prog

ram

sin

the

USA

Eval

uatio

n of

Lea

rnin

g in

Engl

ish

for A

cade

mic

Purp

oses

(LEA

P) in

Sout

h Af

rica

Mid

dle

and

seco

ndar

ysc

hool

s in

volv

ed in

MSI

P in

Can

ada

Larg

e ur

ban

scho

oldi

stric

t in

Min

neso

ta th

atis

com

mitt

ed to

enha

ncin

g ev

alua

tion

func

tion

and

capa

city

.Au

thor

hire

d as

inte

rnal

8 co

mm

uni ty

-ba

sed

orga

niza

-t io

ns, 9

hea

l thde

partm

ents

, 28

tech

nica

l ass

ist-

ance

pro

vide

rs

58 fu

nded

hea

l thde

partm

ent

pers

onne

l

Not a

pplic

able

6 pr

inci

pals

from

alum

ni M

anito

baSc

hool

Impr

ove-

men

t Pro

gram

(MSI

P) s

choo

ls

Not a

pplic

able

Gibb

s et

al .

(200

2)

Gil l i

am e

t al .

(200

3)

Jaco

bs (2

000)

Katz

et a

l.(2

002)

King

(200

2)

(con

tinue

d on

nex

t pag

e)


ble

1 (c

ontin

ued)

Stud

ySa

mpl

eCo

ntex

tM

etho

dsPu

rpos

e / R

esea

rch

Mai

n Fi

ndin

gsQu

est io

n(s)

of p

eopl

e to

rout

inize

inqu

iry—

limi te

d. L

esso

nsle

arne

d: e

ach

proj

ect a

n in

stru

ctio

nal o

ppor

tuni

ty;

dang

er o

f cre

atin

g m

ore

dem

and

than

can

han

dle;

part i

cipa

tory

app

roac

h ef

fect

ive.

Know

ledg

e m

anag

emen

t and

org

aniza

t iona

lm

emor

y ar

e re

curs

ive

soci

al s

yste

ms.

Sof

tkn

owle

dge

alon

g wi

th h

ard

test

sco

re d

ata

allo

wed

orga

niza

tiona

l lea

rnin

g to

occ

ur. I

mpo

rtanc

e of

wisd

om in

lear

ning

.

Eval

uatio

n ev

olve

d fro

m p

artic

ipat

ory

appr

oach

with

dep

ende

ncy

on e

valu

ator

, to

empo

werm

ent

self-

dire

cted

eva

luat

ion,

to c

apac

ity-b

uild

ing

focu

s dr

iven

by

teac

her e

mpo

werm

ent.

Inte

rnal

and

exte

rnal

acc

ount

abili

ty a

re p

ossi

ble.

Dem

and

is th

e m

ain

prer

equi

site

; syn

ergi

es w

ithot

her c

apac

ity-b

uild

ing

work

is e

ssen

tial;

tailo

rEC

B ac

cord

ing

to c

ount

ry c

ircum

stan

ces;

con

duct

base

line

diag

nosi

s. S

tage

s of

dev

elop

men

tde

scrib

ed: b

ank

is in

crea

sing

ly c

omm

itted

to E

CBas

a w

orth

while

act

ivity

and

it is

bei

ng a

ccor

ded

are

ason

ably

hig

h pr

iorit

y.

why

i t ch

ange

d

The

deve

lopm

ent a

nd a

p-pl

icat

ion

of o

rgan

izat io

nal

mem

ory

as a

n an

alyt

ic c

on-

stru

ct fo

r und

erst

andi

ngca

se d

ata;

the

utili

ty o

fm

emor

y as

a c

onst

ruct

help

ful i

n un

ders

tand

ing

orga

niza

tiona

l out

com

esan

d pe

rform

ance

s

To d

escr

ibe

evol

utio

n of

the

role

of e

valu

atio

n in

capa

city

bui

ldin

g fo

r the

Man

itoba

Sch

ool I

mpr

ove-

men

t Pro

gram

To te

ll th

e st

orie

s of

ECB

less

ons

lear

ned

in th

eBb

nk’s

expe

rienc

e wi

thgo

vern

men

ts in

div

erse

and

diffi

cult

envi

ronm

ents

, and

how

ECB

is b

ecom

ing

rec-

ogni

zed

and

valu

ed w

ithin

the

bank

itse

lf

Case

stu

dy: n

arra

t ive

anal

ysis

by

auth

orus

ing

inte

rvie

ws,

obse

rvat

ions

, and

docu

men

ts

Case

stu

dy: p

artic

ipan

tob

serv

atio

n; e

xcer

pts

from

prio

r dat

aco

llect

ion/

repo

rts

Refle

ctiv

e ca

se s

tudy

:de

velo

ped

from

the

pers

pect

ive

of a

ban

kEC

B co

ordi

nato

r

eval

uato

r for

2-y

ear

perio

d in

the

USA

Unde

rsta

ndin

gre

mem

berin

g as

an

orga

niza

t iona

l lea

rnin

gco

nstru

ct in

sch

ools

inth

e US

A

Eval

uatio

n su

ppor

t for

fund

ed s

choo

lim

prov

emen

t pro

gram

inM

anito

ba, C

anad

a

Wor

ld B

ank

expe

rienc

esin

dev

elop

ing

ECB

indi

vers

e an

d di

fficu

ltgo

vern

men

tal c

onte

xts

over

a 2

0-ye

ar p

erio

d

Adm

inis

trato

rs,

teac

hers

, and

stud

ents

in th

e US

city

sch

ool d

istri

ct

Eval

uato

r ref

lec-

tions

on

eval

uatio

nro

le in

mul

ti-ye

ar,

mul

ti-si

te C

anad

ian

scho

ol im

prov

e-m

ent i

nitia

tive

Not a

pplic

able

Krus

e (2

003)

Lee

(199

9)

Mac

kay

(200

2)


Mos

t ly in

form

al le

arni

ng ta

king

pla

ce. O

L oc

curs

mor

e wi

th o

wner

/man

ager

s. W

ork-

base

d le

arni

ngwa

s in

cide

ntal

and

any

kno

wled

ge g

aine

d wa

son

ly u

sed

in th

e sh

ort t

erm

. Firm

s th

at e

mpl

oyed

inte

ntio

nal l

earn

ing

stra

tegi

es w

ere

slow

er a

ndm

ore

del ib

erat

e, b

ut w

ere

mor

e ef

fect

ive

atin

f luen

cing

the

dire

ctio

n an

d de

l iver

y of

a f i

rm’s

prog

ram

s.

Impe

tus

cam

e fro

m s

enio

r adm

inis

trat io

n. S

ixpr

inci

ples

gui

ded

the

mul

t i -ye

ar, m

ult i -

face

ted

ECB

ini t i

ativ

e. S

ucce

ssfu

l ECB

but

not

inve

sti-

gate

d sy

stem

atic

ally

; no

effe

cts

on o

rgan

izatio

nsp

ecifi

ed. A

dmin

role

in c

reat

ing

trust

ing

envi

ron-

men

t piv

otal

; mus

t dea

l with

def

ence

s of

re-

sour

ces,

righ

teou

snes

s, a

nd re

sist

ance

.

The

work

grou

p m

et it

s ob

ject

ive

and

muc

h m

ore.

The

scho

ol is

now

in th

e pr

oces

s of

set

ting

addi

tiona

l cur

ricul

um g

oals

, and

dat

a ar

e no

wbe

ing

colle

cted

on

the

indi

cato

rs th

at w

ere

form

edin

to c

ase

conf

eren

ce te

am c

heck

lists

.

Colla

bora

tive

eval

uatio

n ca

n re

sult

in im

prov

edqu

ality

of e

valu

atio

ns. S

truct

ural

requ

irem

ents

were

impo

rtant

fact

ors.

Tool

kit a

ppea

rs to

be

a su

cces

sful

met

hod

for

teac

hing

man

ager

s to

dev

elop

logi

c m

odel

s an

dim

plem

ent e

valu

atio

n te

chni

ques

; som

e su

gges

-tio

ns fo

r im

prov

emen

t.

To f i

nd o

ut th

e ty

pe a

ndin

cide

nces

of O

L an

d th

elo

cus

of O

L in

thei

r fi rm

s;to

exa

min

e m

ore

clos

ely

the

OL in

som

e f i r

ms;

toco

mpa

re a

nd c

ontra

stco

mpa

nies

that

are

sim

i lar

in s

ize in

term

s of

OL

Why

stre

ngth

enin

g ev

alua

-t io

n ca

paci

ty b

ecam

e an

expl

ici t

prio

ri ty;

prin

cipl

esan

d pr

oced

ures

; effe

cts

ofca

paci

ty b

uild

ing;

less

ons

lear

ned

Wha

t spe

cific

eva

luat

orro

les,

phi

loso

phie

s, a

ndin

terp

erso

nal d

ynam

ics

hold

the

pote

ntia

l to

fost

erpr

oces

s in

fluen

ce?

Can

colla

bora

tive

eval

ua-

tion

and

netw

orki

ngim

prov

e th

e qu

ality

of s

elf-

eval

uatio

n?

To d

escr

ibe a

Too

lkit

initi

a-tiv

e des

igne

d to

hel

p pu

blic

heal

th m

anag

ers i

n On

tario

impr

ove t

heir

know

ledg

ean

d sk

ills i

n lo

gica

l mod

elde

velo

pmen

t and

pro

gram

eval

uatio

n

Fiel

d st

udy:

mul

t i -m

etho

d st

aged

stu

dyin

volv

ing

tele

phon

esu

rvey

, int

ervi

ews,

case

stu

dies

Ref le

ctiv

e ca

se s

tudy

:re

port

is in

tegr

atio

n of

inte

rnal

and

ext

erna

lvi

ewpo

ints

of

resp

ectiv

e au

thor

s/pa

rtici

pant

s

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

pers

pect

ive

ofev

alua

tor

Long

itudi

nal c

ase

stud

y: re

ports

from

eval

uatio

n fa

irpr

esen

tatio

ns g

iven

by

eval

uatio

n co

ntra

ctor

sin

2 c

onse

cutiv

e ye

ars

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

the

poin

tof

vie

w of

eva

luat

ors

and

ECB

prov

ider

s

A re

port

of a

stu

dy o

for

gani

zat io

nal l

earn

ing

in th

e sm

all b

usin

ess

sect

ion

of th

e UK

econ

omy

Mam

mot

h EC

B ef

fort

inla

rge

nat io

nal U

SApu

blic

hea

l th p

reve

ntio

nan

d co

ntro

l org

aniza

tion

CDPC

ove

r mul

ti-ye

arpe

riod.

Man

y pa

rtner

orga

niza

tions

A US

A sc

hool

that

prov

ides

inst

ruct

ion

toph

ysic

ally

and

med

ical

lydi

sabl

ed c

hild

ren

from

Kto

12

Eval

uatio

n Vo

ices

(EV)

appr

oach

was

impl

e-m

ente

d in

the

cont

ext o

fea

rly c

hild

hood

edu

ca-

tion

prog

ram

eva

luat

ion

in th

e US

A

An e

valu

atio

n ca

paci

ty-

build

ing

initi

ativ

e in

the

Onta

rio, C

anad

a pu

blic

heal

th s

yste

m

6000

org

aniza

-t io

ns fr

om y

ello

wpa

ges;

600

sm

all

busi

ness

man

ag-

ers;

60

mat

ched

case

stu

dies

Not a

ppl ic

able

Not a

pplic

able

Prog

ram

pra

cti-

tione

rs p

artic

ipat

-in

g in

eva

luat

ion

netw

orki

ng a

ctiv

i-tie

s

Not a

pplic

able

Mat

lay

(200

0)

Mils

tein

et a

l .(2

002)

Mor

abito

(200

2)

O’Su

lliva

n &

D’Ag

ostin

o(2

002)

Porte

ous

et a

l.(1

999)

(con

tinue

d on

nex

t pag

e)


ble

1 (c

ontin

ued)

Stud

ySa

mpl

eCo

ntex

tM

etho

dsPu

rpos

e / R

esea

rch

Mai

n Fi

ndin

gsQu

est io

n(s)

Five

mai

n ca

tego

ries

of in

f luen

ce o

n pr

oces

s us

e:fa

cil i t

atio

n of

eva

luat

ion

proc

esse

s; m

anag

emen

tsu

ppor

t ; ad

viso

ry g

roup

mem

ber c

hara

cter

ist ic

s;co

mm

unic

atio

n at

tribu

tes;

org

aniza

t ion

char

acte

r-is

t ics.

Exa

mpl

es o

f pro

cess

use

iden

t i fie

d: le

arn-

ing

abou

t eva

luat

ion;

abo

ut th

e pr

ogra

m; o

vera

l lev

alua

t ion

expe

rienc

e.

Sign

ifica

nt o

rgan

izatio

nal c

onse

quen

ces

ob-

serv

ed, e

spec

ially

cap

acity

bui

ldin

g. C

ondi

tions

for p

artic

ipat

ory

eval

uatio

n we

re fa

vour

able

.

Prog

ram

term

inat

ed o

n th

e ba

sis

of n

on-e

valu

a-tiv

e in

form

atio

n; s

ome

deve

lopm

ent o

f sha

red

unde

rsta

ndin

g, e

ngag

emen

t, se

lf-de

term

inat

ion.

Envi

ronm

enta

l fac

tors

affe

ctin

g us

e: ru

mou

rs o

f am

erge

r with

ano

ther

com

pany

had

pro

foun

d ef

fect

.

Wid

er s

choo

l com

mun

ities

mus

t be

invo

lved

indi

scus

sion

s of

SIP

from

com

mitm

ent t

hrou

ghim

plem

enta

tion.

Fac

tors

that

influ

ence

the

deve

l-op

men

t of p

rofe

ssio

nal c

omm

uniti

es: l

eade

rshi

pst

yle

of p

rinci

pal,

past

eve

nts,

pol

itics

of a

lloca

t-in

g re

sour

ces,

org

aniza

tion

of s

choo

ls, a

nd

Wha

t did

adv

isor

y gr

oup

mem

bers

lear

n fro

min

volv

emen

t in

eval

ua-

t ion?

How

did

they

lear

n?W

hat f

acto

rs s

uppo

rted

orhi

nder

ed le

arni

ng?

To in

vest

igat

e th

e or

gani

-za

tiona

l con

sequ

ence

s of

inte

rnal

par

ticip

ator

y ev

al-

uatio

n in

a C

anad

ian

na-

tiona

l tra

inin

gor

gani

zatio

n

Use

and

non-

use

of th

eev

alua

tion

resu

lts, w

ithpa

rticu

lar e

mph

asis

on

proc

ess

use;

fact

ors

con-

tribu

ting

to u

se a

nd n

on-

use

of th

is p

rivat

e se

ctor

eval

uatio

n

How

do s

choo

l im

prov

e-m

ent p

rogr

ams

fost

er th

ede

velo

pmen

t of p

rofe

s-si

onal

com

mun

ities

? W

hat

orga

niza

tiona

l fac

tors

supp

ort a

nd/o

r im

pede

the

Case

stu

dy o

f two

eval

uat io

ns; i

nter

view

s

Long

itudi

nal c

ase

stud

y; p

artic

ipan

tob

serv

atio

n;in

terv

iews

, foc

usgr

oup

Desc

riptiv

e ca

se s

tudy

:sa

tisfa

ctio

n in

terv

iews

,su

rvey

s

Colle

ctiv

e ca

se s

tudy

of th

ree

scho

ols:

cons

truct

ivis

tor

ient

atio

n, s

choo

l-ba

sed

obse

rvat

ion,

conf

eren

ces

and

Year

ly e

valu

atio

ns o

f the

Tel l

a Fr

iend

pro

gram

of

the

ACS

in th

e US

A

Cana

dian

nat

iona

ltra

inin

g pr

ogra

m re

form

initi

ativ

e

Eval

uatio

n of

a s

ales

prog

ram

with

in a

busi

ness

ser

vice

sor

gani

zatio

n (X

YZCo

rpor

atio

n) in

the

USA

USA

dist

rict-w

ide

scho

olim

prov

emen

t ini

tiativ

e

Pres

kil l

et a

l .(2

003)

Robi

nson

&Co

usin

s (i2

004)

Russ

-Eft

et a

l.(2

002)

Scrib

ner e

t al.

(199

9)

16 in

terv

iews

wi th

advi

sory

gro

upm

embe

rs a

nd s

en-

ior a

dmin

istra

tors

of A

mer

ican

Can

-ce

r Soc

iety

(ACS

)

9 Ca

nadi

an k

eyin

form

ant n

atio

nal

prog

ram

pra

cti-

tione

rs

23 s

ales

peo

ple,

10 im

plem

enta

tion

spec

ialis

ts, a

nd 9

cons

ulta

nts;

40

repr

esen

tativ

es o

fcl

ient

com

pani

es

3 ru

ral m

iddl

esc

hool

s: s

choo

lad

min

and

lead

er-

ship

team

mem

-be

rs a

nd o

ther

teac

hers

, N =

35


teac

her w

ork.

The

cond

i t ion

s fo

r org

aniza

t iona

l lea

rnin

g ar

e th

esa

me

as th

e co

ndi t i

ons

that

are

ass

ocia

ted

with

the

3 sc

hool

lead

ersh

ip v

aria

bles

: prin

cipa

ltra

nsfo

rmat

iona

l lea

ders

hip,

act

ivel

y in

volv

edad

min

istra

t ive

team

s, a

nd d

istri

bute

d le

ader

ship

.Or

gani

zat io

nal l

earn

ing

is a

lso

a si

gni f i

cant

med

iato

r of p

rinci

pal a

nd le

ader

ship

team

ef fe

cts

on te

ache

rs’ w

ork

and

on s

tude

nt o

utco

mes

.

CSLA

bui

lds

prog

ram

and

eva

luat

ion

expe

rtise

by

crea

ting

a fra

mew

ork

for p

rofe

ssio

nal d

evel

op-

men

t tha

t can

be

used

in a

flex

ible

man

ner;

help

sid

entif

y re

sour

ces;

com

plem

ents

ext

erna

l eva

lua-

tion

by b

uild

ing

loca

l cap

acity

; use

s da

ta to

enco

urag

e st

aff t

o co

ntem

plat

e ho

w th

eir v

alue

shi

nder

or c

ontri

bute

to a

pro

gram

’s su

cces

s

Curri

culu

m fo

rum

: 8 is

sues

add

ress

ed b

ut m

ost

were

adm

inis

trativ

e; li

ttle

lear

ning

took

pla

ce.

Soci

al ju

stic

e &

equ

ity fo

rum

: inq

uiry

and

act

ion

were

inte

gral

, fee

dbac

k so

licite

d fro

m s

tude

nts.

Teac

hers

’ vie

ws o

n di

vers

ity a

nd d

iscr

imin

atio

nwe

re w

illin

gly

ques

tione

d bu

t not

vie

ws a

bout

curri

culu

m.

Need

s as

sess

men

t use

ful f

or fo

cusi

ng te

chsu

ppor

t, in

form

ing

refo

rm o

f sta

te p

ropo

sal

guid

elin

es. P

artic

ipan

t con

fiden

ce v

arie

d ac

ross

four

ski

ll ca

tego

ries;

mor

e pr

ogre

ss o

n pr

elim

i-

deve

lopm

ent o

f pro

fes-

sion

al c

omm

uni t i

es?

To e

xam

ine

the

natu

re o

for

gani

zat io

nal l

earn

ing

and

the

lead

ersh

ip p

ract

ices

and

proc

esse

s th

at fo

ster

orga

niza

t iona

l lea

rnin

g in

Aust

ral ia

n hi

gh s

choo

ls

To in

trodu

ce a

nd d

escr

ibe

the

appl

icat

ion

of C

SLA

impl

emen

tatio

n th

roug

hev

alua

tion

of s

choo

l-wid

edi

scip

line

plan

Wha

t kin

ds o

f iss

ues

are

addr

esse

d an

d de

cisi

ons

mad

e in

a s

choo

l tha

t has

striv

ed to

dem

ocra

tize

deci

sion

-mak

ing

and

beco

me

a le

arni

ng o

rgan

i-za

tion?

To d

ocum

ent s

ucce

ssfu

lin

crea

se in

age

ncy

capa

c-ity

as

a co

nseq

uenc

e of

ECB

initi

ativ

e; to

iden

tify

A pa

th m

odel

is u

sed

tote

st th

e re

lat io

nshi

psbe

twee

n sc

hool

-leve

lfa

ctor

s an

d sc

hool

outc

ome

mea

sure

s in

term

s of

stu

dent

s’pa

rt ici

pat io

n in

and

enga

gem

ent w

i thsc

hool

in A

ustra

lia

Impl

emen

tatio

n of

cont

inuo

us s

yste

m-

leve

l ass

essm

ent

(CSL

A) in

con

text

of

scho

ol-le

vel s

taff

capa

city

bui

ldin

g in

the

USA

Scho

ol w

ith e

stab

lishe

dst

ruct

ures

for s

hare

dde

cisi

on-m

akin

g an

dco

llect

ive

lead

ersh

ip;

char

acte

ristic

s of

lear

ning

org

aniza

tion

inAu

stra

lia

Stat

e-fu

nded

eva

luat

ion

supp

ort u

nit

com

mis

sion

ed to

Sil in

s et

al .

(200

2)

Smith

&Fr

eem

an (2

002)

Stev

enso

n(2

001)

Stev

enso

n et

al.

(200

2)

Teac

hers

and

stud

ents

from

96

seco

ndar

y sc

hool

sfro

m S

outh

Aust

ral ia

and

Tasm

ania

, N =

2503

(tea

cher

s)an

d N

= 35

00(s

tude

nts)

Not a

pplic

able

One

seco

ndar

ysc

hool

inAu

stra

lia; 1

/3 o

fte

achi

ng s

taff

14 c

omm

unity

-ba

sed

agen

cies

resp

onsi

ble

for

eval

uatin

g th

eir

proj

ects

, doc

umen

tan

d ar

t i fac

t col

lect

ion

Fiel

d st

udy,

2 p

hase

s:(a

) sur

vey

data

was

col le

cted

usi

ng tw

oqu

est io

nnai

res;

(b)

path

ana

lysi

sex

amin

ed th

e na

ture

and

stre

ngth

of

inte

rrela

t ions

hips

betw

een

varia

bles

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

pers

pect

ive

of m

odel

deve

lope

rs/

impl

emen

ters

Case

stu

dy: o

bser

ved

mee

tings

, for

mal

inte

rvie

ws, i

nfor

mal

conv

ersa

tions

,co

llect

ion

and

anal

ysis

of re

leva

nt d

ocum

ents

Long

itudi

nal c

ase

stud

y: e

valu

atio

npr

opos

als,

con

fiden

cera

tings

re: m

ultip

le

(con

tinue

d on

nex

t pag

e)


ble

1 (c

ontin

ued)

Stud

ySa

mpl

eCo

ntex

tM

etho

dsPu

rpos

e / R

esea

rch

Mai

n Fi

ndin

gsQu

est io

n(s)

nary

ste

ps a

nd d

esig

n th

an o

n an

alys

is a

ndut

i l iza

t ion.

Tec

hnic

al a

ssis

tanc

e es

sent

ial .

Exte

rnal

refo

rm v

iewe

d as

ext

rinsi

c m

otiv

ator

,te

ache

rs b

ecom

e in

trins

ical

ly m

otiv

ated

. Maj

orro

le fo

r lea

ders

hip,

stru

ctur

e fo

r cap

acity

bui

ldin

g,co

ntin

uous

flow

of i

nfor

mat

ion

and

com

mun

ica-

tion.

Cul

ture

of c

ontin

uous

dat

a us

e ov

er ti

me.

Elem

ents

of r

efor

m e

nviro

nmen

t im

porta

nt.

An e

mpo

werm

ent e

valu

atio

n wi

th it

s em

phas

is o

nse

lf-as

sess

men

t, fa

cilit

atio

n, a

nd c

apac

ity b

uild

-in

g ca

n m

eet t

he c

halle

nges

of d

iver

sity

yet

guar

ante

e th

at th

e ev

alua

tion

will

stre

ngth

en th

ein

divi

dual

loca

l pro

gram

s.

Mut

ual d

epen

denc

e be

twee

n sh

ared

vis

ion

and

lead

ersh

ip; b

etwe

en H

R de

velo

pmen

t, te

am le

arn-

ing,

and

per

sona

l mas

tery

; bet

ween

mut

ual m

od-

els/

info

rmat

ion

and

anal

ysis

/pro

cess

impr

ove-

men

t; be

twee

n sy

stem

s th

inki

ng/c

usto

mer

focu

s/st

rate

gic

plan

ning

; org

aniza

tiona

l per

form

ance

.

own

prog

ram

s

46 U

S sc

hool

ad-

min

istra

tors

,te

ache

rs, s

taff,

pare

nts,

and

stu

-de

nts

plus

12

dis-

trict

adm

inis

trato

rsin

Edi

son

Proj

ect

scho

ol

Not a

pplic

able

5 ca

seco

rpor

atio

ns

task

s, c

heck

l ist o

fev

alua

t ion

step

sco

mpl

eted

Long

i tudi

nal c

ase

stud

y: in

terv

iew

and

obse

rvat

ion

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

poi

nt o

fvi

ew o

f eva

luat

ors

Mul

tiple

cas

e st

udy:

sem

i-stru

ctur

edin

terv

iews

mai

n ar

eas

of re

sist

ance

How

do th

e pr

oces

ses

and

prac

t ices

sur

roun

ding

the

use

of d

ata

for d

ecis

ion-

mak

ing

cont

ribut

e to

acu

lture

of c

ontin

uous

impr

ovem

ent?

To il

lust

rate

the

utili

ty a

ndfe

asib

ility

of e

mpo

werm

ent

eval

uatio

n in

a c

ompr

ehen

-si

ve s

tate

-fund

ed p

rogr

am

Wha

t are

the

mut

ual

depe

nden

cies

bet

ween

Seng

e’s le

arni

ng o

rgan

iza-

tion

chan

ge s

trate

gies

and

the

MBQ

NA fr

amew

ork?

incr

ease

eva

luat

ion

capa

city

bui

ldin

g: n

eeds

asse

ssm

ent ,

mod

ell in

g,te

ch a

ssis

tanc

e, t r

aini

ngin

the

USA

Edis

on P

roje

ct s

choo

l ,m

ajor

USA

refo

rmin

itiat

ive

fund

ed b

yfo

unda

tion

desi

gned

tocr

eate

inno

vativ

esc

hool

s th

at o

pera

tewi

thin

cur

rent

pub

licex

pend

iture

s

Larg

e st

ate-

wide

Toba

cco

Cont

rol

Prog

ram

, spo

nsor

ed b

yth

e Ca

lifor

nia

Stat

e De

pt.

of H

ealth

in U

SA

Aust

ralia

n co

rpor

atio

nsim

plem

entin

g TQ

M in

cont

ext o

f Mal

colm

Bald

ridge

Nat

iona

lQu

ality

Awa

rd (M

BNQA

)

Suth

erla

nd(2

004)

Tang

et a

l.(2

002)

Terz

iovs

ki e

t al.

(200

0)


Eval

uat io

n of

BPR

had

the

fol lo

wing

ben

efi ts

:fa

cil i t

ated

the

chan

ge p

roce

ss a

nd le

d to

bet

ter

OL, l

ed to

pro

cess

inno

vat io

n su

ch a

s th

ein

trodu

ctio

n of

impr

oved

tech

nolo

gica

l cap

abi l i

-t ie

s in

the

com

pani

es.

Use

take

s pl

ace

in s

ocia

l con

text

; 4 ty

pes

ofar

gum

ents

abo

ut e

valu

ativ

e re

sul ts

; man

yex

ampl

es o

f pos

t hoc

legi

t imizi

ng a

ctio

ns;

eval

uatio

n in

cite

s ar

gum

ent a

nd c

an re

sult

inre

ason

ed d

ebat

e; w

ritte

n re

ports

offe

r com

mon

plat

form

for d

ialo

gue

and

argu

men

tatio

n.

Empl

oyin

g a

mor

e in

clus

ive

notio

n of

par

tici-

pant

s ch

ange

s th

e un

ders

tand

ing

of th

e te

rm“p

artic

ipat

ory

rese

arch

.” Pr

ogra

m p

artic

ipan

ts,

staf

f, go

vern

men

t fun

ders

, com

mun

ity p

artn

ers,

man

ager

s, a

nd e

valu

ator

s al

l hav

e a

capa

city

toem

powe

r and

be

empo

were

d. T

he s

ucce

ss o

f aso

cial

inte

rven

tion

is b

ette

r jud

ged

by w

hat

happ

ens

in th

e ov

eral

l env

ironm

ent t

han

by w

hat

happ

ens

to in

divi

dual

s an

d or

gani

zatio

ns in

the

colle

ctiv

e.

Is th

ere

a l in

k be

twee

nev

alua

t ion

act iv

i t ies

that

are

part

of a

BPR

exe

rcis

ean

d or

gani

zat io

nal l

earn

ing

and

inno

vat io

n?

To c

ompa

re h

ow a

genc

yev

alua

t ions

con

duct

ed,

obje

ctiv

es, a

nd ro

leev

alua

tion

play

ed in

over

all f

ram

ewor

k of

man

agem

ent d

evel

opm

ent

How

the

conc

epts

of

parti

cipa

tion,

em

powe

r-m

ent,

and

soci

al c

hang

ear

e co

nstru

cted

Orga

niza

t ions

inco

nstru

ctio

n in

dust

ry in

3 Eu

rope

an c

ount

ries

that

und

erwe

nt a

busi

ness

pro

cess

re-

engi

neer

ing

exer

cise

(BPR

)

Gove

rnm

ent a

genc

ies

inFi

nlan

d in

div

erse

pol

icy

sect

ors

Expe

rienc

es w

ith th

eAt

lant

ic C

omm

unity

Actio

n Pr

ogra

m fo

rCh

ildre

n (C

APC)

regi

onal

eva

luat

ion

inCa

nada

Desc

ript iv

e ca

sest

udie

s: th

e da

ta fr

omse

mi-s

truct

ured

inte

rvie

ws fo

l low

apa

t tern

of a

naly

sis

Met

a-ev

alua

t ion:

sem

i-stru

ctur

edin

terv

iews

Refle

ctiv

e ca

se s

tudy

:wr

itten

from

pers

pect

ive

ofev

alua

tors

3 ca

seco

rpor

atio

ns

27 a

genc

y an

dsu

perv

isin

gde

partm

ent

pers

onne

l

Not a

pplic

able

Vako

la, M

.(2

000)

Valo

virta

(200

2)

Vand

erPl

aat e

tal

. (20

01)


The remaining studies were relatively evenly distributed over fourdesign types. Five were qualitative case studies (e.g., Balthasar &Reider, 2000; Katz, Sutherland & Earl, 2002), and an additionalfive were comparative case studies (e.g., Connor & Tanjasiri, 1999;Scribner, Cockrell, Cockrell, & Valentine, 1999) Six studies werebased on longitudinal data collection (e.g., O’Sullivan & D’Agostino,2002; Robinson & Cousins, 2004; Sutherland, 2003) or multi-methodfield studies (e.g., Matlay, 2000). Many of the studies are NorthAmerican, taking place in the U.S.A. (17) and Canada (6), but sev-eral are from other jurisdictions including the U.K. (Biott & Cook,2000; Boaz, 2002; Matlay, 2000), Australia (Silins, Mulford, &Zarins, 2002; Stevenson, 2001; Terziovski, Howell, Sohal, &Morrison, 2000), Spain (Chacon-Moscoso, Anguera-Argilaga, Perez-Gil, & Holgado-Tello, 2002), Finland (Valovirta, 2002), Switzerland(Balthasar & Rieder, 2000), and South Africa (Jacobs, 2000). Twostudies (Mackay, 2002; Vakola, 2000) involved data collection frommultiple countries.

Method

Having identified and obtained the articles in our sample, we di-vided them among the authors for analysis and summarization. Eachanalyst produced a one-page summary for each of the allotted arti-cles. The summary was structured by the column headings appear-ing in Table 1. In summarizing the articles, analysts looked forevidence and verbatim quotations that illustrated relationshipsamong the variables of interest. Verbatim quotations were capturedin bold with appropriate page numbers. All summaries were com-piled into a single electronic file ordered alphabetically by author.The file was then converted to Folio Views (4.3) and coded accord-ing to the main variables of interest by one of the analysts. In work-ing through the files, the analyst developed the conceptualframework showing relationships among the variables (see Figure1). This figure and descriptive text were then circulated among theother authors for input and verification. The analyst made somemodifications to the framework and proceeded to integrate the find-ings as presented below. Available to the analyst were copies of theoriginal studies as well as the coded summaries.

All in all, we judge the knowledge base linking evaluative

integrating evaluative inquiry into the organizational … · 2017. 5. 23. · la revue canadienne...

Documents