program theory evaluation

7/25/2019 Program Theory Evaluation

1/9

The historical developmentof

program theory evaluation,

current variations in theory and practice, and pressingissues are discussed.

Program Theory Evaluation: Practice,Promise, and Problems

Patricia]. Rogers, Anthony Petrosino, TracyA.

Huebner;Timothy A. Hacsi

For over thirty years now, many evaluators have recommended makingexplicit the underlying assumptions about howprograms are expected towork-theprogram theory-and then using this theory to guide the eval-uation. In this chapter, we provide an overview ofprogram theory evalu-ation (PTE), based on our search to find what was value-added toevaluations that used this approach. We found fewer clear-cut, full-blownexamples in practice than expected, bu t we found many interesting vari-

ations of PTE in practice and much to recommend it. And elements ofPTE, whether the evaluators use the terminology or not, are being usedin a wide range of areas of concern to evaluators. Based on this review, inthis chapter we discuss the practice, promise, and problems of PTE.

What Is ProgramTheory Evaluation?

Because this volume is intended to demonstrate the diversity ofpractice,we have used a broad definition of program theory evaluation. We consider

i t to have two essential components, one conceptual and one empirical.

PTE consists of an explicit theory or model of how the program causes theintended or observed outcomes and an evaluation that is at least partlyguided by this model. This definition, though deliberately broad, doesexcIude some versions of evaluation that have the word theory attached to

them. It does not cover all six types of theory-driven evaluation defined byChen (1990)but only the type he refers to as intervening mechanism eval-uation. It does not include evaluations that explicate the theory behind a

NEW D~K ECTIUNSoit EVALUATION, no. 87,Fall 2000Jossey Bass 5


2/9


3/9


4/9


5/9


6/9


7/9


8/9

12 PROGRAM THEORY IN EVALUATION

to harsh empirical tests can be less attractive. Practical difficulties aboundas well. When PTE is implemented at a small project, staff may not have thetime or skills to collect and analyze data in ways that either test theprogramtheory orprovide useful information to guide decisions and action. Ifpro-gram theory is used to develop accountability systems, there is a real risk ofgoal displacement,wherein staff seek to achieve targets and stated objectivesat the cost of achieving the ultimate goal or sustainability of the program(Winston, 1991).

Conclusion

In this chapter, we have outlined the range of activity that can be consid-eredprogram theory evaluation and have identified major issues in its the-ory and practice. These are discussed in more detail by the other chaptersin this volume.

References

Bennett, C. Up the Hierarchy.Joumal off i tension, 1975, 13 2), 7-12.Bickman, L. (ed.). Using Program Theory in Evaluation.New Directions for Program Eval-

Bickman,L. (ed.). Advances in Program Theory.New Directions for Program Evaluation,

Bickman, L., and others. Long-Term Outcomes to Family Caregiver Empowerment.

Chen, H. T. Theory-Driven Evaluation. Thousand Oaks, Calif.: Sage, 1990.Funnell, S. Program Logic:An Adaptable Tool. Evaluation News and Comment, 1997,

Funnell, S. and Mograby,A. Evaluating Employment Programs Deliveredby Commu-nity Organisations. Proceedings of the Annual Conference of the Australasian Evalua-tion Society, 1995,2, 531-552.

Hatry, H., van Houten, T., Plantz, M ., and Greenway, M. T. Measuring Program Out-comes: A Practical Approach. Alexandria, Va.: United Way of America, 1996.

Homel, R. Random Breath Testing in New South Wales: The Evaluation of a Success-ful Social Experiment.National Evaluation Conference 1990, Proceedings, vol. 1.Aus-tralasian Evaluation Society, 1990.

Lenne, B., and Cleland, H. Describing Program Logic. Program Evaluation Bulletin1987, no. 2. Public Service Board of New South Wales, 1987.

Lipsey, M. W. Theory as Method: Small Theories of Treatments. In L. Sechrest and A.Scott (eds.), Understanding Causes a n d Generalizing About Them.New Directions forProgram Evaluation, no. 57. San Francisco:JosseyBass, 1993.

Lipsey, M. W., and Pollard, J. Driving Toward Theory in Program Evalualion: MoreModels to Choose From. Evaluation and Program Planning, 1989,12,317-328.

Madaus, G., Stufflebeam,D., and Scriven,M. Evaluation Models: Evaluation ofEducationaland Social Programs. Nonvell, Mass.: Kluwer, 1983.

Milne, C. Outcomes Hierarchies and Program Logic as Conceptual Tools: Five CaseStudies. Paper presented at the international conference of the Australasian Evalua-tion Society, Brisbane, 1993.

Patton, M. Q. Utilization-Focused Evaluation. (3rd. ed.) Thousand Oaks, Calif.: Sage,1996.

Pawson,

R., and Tilley, N. Realistic Evaluation. London: Sage, 1995.

uation, no. 33. San Francisco: Jossey-Bass, 1987.

no. 47. San Francisco: Jossey-Bass, 1990.

Journal of Child and Family Studies, 1998, 7 3), 269-282.

6 1), 5-17.


9/9

PROGRAM THEORY EVALUATION: PRACTICE, PROMISE, AND PROBLEMS 13

Petrosino, A. J. Answering the Why Question in Evaluation: The Causal-ModelApproach. Canadian Journal of Program Evaluation, 2000, IS ,1-24.

Rogers, P. J. Program Theory Evaluation: Not Whether Programs Work But Why. InG. Madaus, D. Stufflebeam, and T. Kelleher eds.), Evaluation Models: Evaluation ofEdzdcutional and Social Programs. Nonuell, Mass.: Kluwer, forthcoming.

Rossi,P. H., Freeman, H., and Lipsey, M. W. Program Evaluation: A Systematic Approach.Thousand Oaks, Calif.: Sage, 1999.

Schon,D. A. Theory-of-Action Evaluation. Paper presented to the Harvard EvaluationTask Force, Apr. 1997.

Stewart, K., Cotton, R., Duckett, M., and Meleady,K. The New South Wales ProgramLogic Model: The Experience of the AIDS Bureau, New South Wales Department ofHealth. Proceedings of the Annual Conference of the Australasian Evaluation Society,

Suchman, E. A. Evaluative Research: Principles and Practice in Public Service and SocialAction Programs.New York: Russell Sage Foundation, 1967.

Ward, V. M., Maine, D., McCarthy,J., and Kamara,A. AStrategy for the EvaluationofActivities to Reduce Mortality in Developing Countries. Evaluation Review, 1994,18,

Weiss, C. H. Evaluation Research: Methods of Assessing Program Effectiveness. Englewood

Cliffs,N.J.: Prentice Hall, 1972.Weiss, C. H. Nothing As Practical As Good Theory: Exploring Theory-Based Evalua-

tion for Comprehensive Community Initiatives for Children and Families. In J. P.Connell,A. C. Kubisch, L, B. Schorr, and C H. Weiss (eds.),New Approaches to Eval-uating Community Initiatives: Concepts, Methods and Contexls. Washington, D.C.: AspenInstitute, 1995.

Weiss,C. H. How Can Theory-Based Evaluation Make Greater Headway?Evaluation

Review, 1997 21 501-524.Weiss, C. H. Evaluation: Methodsfor Studying Programs and Policies. (2nd ed.) Engle-

wood Cliffs,N.J.: Prentice Hall, 1998.Wholey, J. S. Evaluability Assessment: Developing Program Theory. In L. Bickman

(ed.), Using Program Theory in EvaZuation. New Directions for Program Evaluation,no. 33. San Francisco: Jossey-Bass, 1987.

Winston,J A. Linking Evaluation and Performance Management. Paper presented atthe annual conference of the Australasian Evaluation Society, Adelaide, 1991.

1990,2,315-322.

438-457.

PATHCUJ.ROGERS is director of the Programfor Public Sector Evaluation in theFaculty Applied Science, Royal Melbourne Institute of Technology,Australia.

ANTHONY P TROSINO is researchfellow at the Centerfor Evaluation, Initiativesfor Children Program,American Academy ofArts und Sciences, and researchassociate at the Haward Graduate School of Education.

TRACYA. HUEBNER is coordinatorfor comprehensive school reform at WestEd.

TIMOTHYA. HACSr is research fellow at the Harvurd Childrens Initiutive andteaches history at the Harvard Extension School.

program theory evaluation

Documents