Can Democratic Evaluation be Scientific?

Ann OomsFrances Lawrenz

University of MinnesotaCollege of Education and Human Development

Department of Educational Psychology

Overview of Presentation

What is Democratic Evaluation? History of Democratic Evaluation. What is Scientific Evaluation? Two examples of Democratic

Scientific Evaluation Efforts. Conclusion.

Democratic Evaluation: defined

“Government by the people” Exercised directly or through

elected representatives Based on principles of social

equality and respect for the individual

Democratic Evaluation: History

MacDonald (1973, 1977): democratic evaluation

Bryk (1983): stakeholder-based evaluation

Fetterman (1994): empowerment oriented evaluation

Democratic Evaluation: History Floc’hlay and Plottu (1998): Model

for the Operationalization of Democratic Evaluation Empowerment Evaluation Participatory Evaluation Multi-criteria Evaluation

Counterpower exercised by those who do not agree

Democratic Evaluation: History

House and Howe (1999): Model of Democratic Deliberative Evaluation Inclusion Dialogue Deliberation Impact:

Better informed decision-making parties A thoughtful and deliberated population

Democratic Evaluation: History

Patton (2002): think evaluatively suggests including a methodological

dialogue to maintain methodological quality.

Democratic Evaluation: our definition If an evaluation is democratic:

All interests are represented Procedures for controlling any

imbalances of power All groups participate seriously and

authentically in meaningful ways Groups participate in appropriate ways Reflective deliberation about findings

and implications

Scientific Evaluation: defined

No Child Left Behind Act of 2001 U.S. National Research Council’s

Committee of Scientific Principles for Education Research

Scientific Evaluation: our definition

Scientific Evaluations are: Evaluations that use experimental and comparison groups or some other sort of quasi-experimental design that can demonstrate causality

Democratic, Scientific Evaluation: Two Examples

Collaboratives for Excellence in Teacher Preparation (CETP) Program Frances Lawrenz

AlphaSmart Evaluation Ann Ooms

CETP

Collaboratives for Excellence in Teacher Preparation evaluation Frances Lawrenz

CETP: Democratic?

There were power imbalances Evaluation was required by the

funder and was conducted for the funder

There was no full participation in the CETP evaluation

There were limited decisions about how results were to be used

CETP: Scientific? The pre post assessments of

faculty instructional approaches Comparison of existing qualities

of institutions, classrooms, teachers, etc. to pre determined standards

A random selection of participants

Comparison of students in matched classes

CETP: Scientific?

None of these meet the “gold standard” of randomly assigned experimental and comparison groups but they do have some elements of experimental design and causality

Also through the negotiation process, additional “non scientific” data were also included

AlphaSmart Evaluation

AlphaSmart Evaluation Ann Ooms

AlphaSmart: Democratic?

Not all stakeholders were represented. For example, there were no principals, parents or students in the inquiry group

There were minor imbalances of power in the group: the teachers had more power

AlphaSmart: Democratic?

The members of the inquiry group were involved seriously and authentically and they participated in ways which matched their skills

There was reflective discussion about the findings and whether or not the AlphaSmarts should be used

AlphaSmart: Scientific?

There was no comparison group Students were not pre tested: the

study was a post test only However, there were pre and post

interviews and regular meetings with the teachers which might provide some indication of causality

AlphaSmart: Scientific?

The students’ retrospective opinions were gathered

Additionally the evaluators and the teachers observed the students as they used the AlphaSmarts and were able to form their own opinions of the success or failure of the technology

Conclusion of two examples

As in past work, producing democratic evaluations appears to be quite difficult

The two examples did NOT accomplish a full democratic scientific evaluation

Limitations of Scientific Evaluation

Scientific Evaluation narrows the meaning of evaluation to achievement of specific outcomes

Scientific Evaluation deemphasizes the importance of understanding the process and meaning

Limitations of Scientific Evaluation

Scientific Evaluation does not accommodate for methodological dialogs

Scientific Evaluation precludes the participation of anyone who has different philosophical perspectives

Conclusion

Scientific Evaluation seems only possible if everyone involved has the same philosophical perspective.

Democratic, Scientific Evaluation: a wonderful but inaccessible dream?

Contact Information

Ann Ooms: ooms0001@umn.edu

Frances Lawrenz: lawrenz@umn.edu