ELAINE SEYMOURETHNOGRAPHY & EVALUATION RESEARCHUNIVERSITY OF COLORADO AT BOULDER

Scaling up Innovation:Why Theories of Change

Matter

Cite as: Seymour, E., (2010). Scaling up Innovation: Why Theories of Change Matter. Presented at the Workshop on Disseminating CCLI Innovations: Arlington, VA, February 18-19, 2010.Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)

What’s the problem?

The momentum for STEM education reform has slowed, even stalled.

Research-grounded teaching methods have not yet been adopted by the majority of faculty in STEM departments in universities and colleges.

What are the reasons for this?

1. Dissemination Issues

Research on how people learn best, Findings from project evaluations, and Proven or promising practices, are not accessible in a coherent,

comprehensive, searchable, regularly updated guide to what’s available and where it can be found

Some of what’s out there is very good, and ready for adoption, but is available in a partial and scattered form.

Some consequences

Proposal writers (and reviewers) may be unaware of the successes or limitations of earlier or concurrent work, risking repetition and duplication of effort.

Potential adopters may not find what’s already available.

Skeptics may doubt the existence of proven practices

Meaningful progress is dependent on the collective memory of funders’ program designers.

2. A related issue: research without development

The NSF has found it difficult to extend its historic mission to stimulate “discovery” by taking a proactive role in organizing and fostering uptake of what is already known.

The upshot:The STEM education reform effort has lacked

funder-initiated coherent, sustained, and nationally-applied strategies to build what has been learned and created.

 

So, what’s needed?

1. Commissioned syntheses of research, evaluation findings, faculty-tested curricula, pedagogies, materials, learning assessment tools and professional development methods that are updated annually and available on an accessible, searchable, national database (the NDL?).

2. A coordinated, sustained, and appropriately-funded effort to stimulate and support the adoption of effective education practices in departments and institutions nationwide.

3. A decline in the perceived value of teaching that limits its significance for

reward and tenure

Teaching seen as a far less important part of the faculty role than research.

This is structurally reinforced because institutions are increasingly dependent on funding raised by research grants.

Professional rewards are focused on faculty research productivity: faculty feel pressured, lack choice.

A focus on improving their teaching can risky

Other consequences:

The professional development of STEM graduate students as future faculty teachers is inadequate and does not reflect best practices

STEM faculty do not encourage their STEM majors to choose careers in K-12 mathematics and science teaching  

A serious, longstanding, and growing shortfall of discipline-qualified mathematics and science teachers in middle and high schools.

4. Pervasiveness of a flawed theory of change:

“If new resources and practices for teaching and learning are shown to have value for improving learning process and performance,

then individual faculty, departments, and their institutions are likely to adopt and institutionalize these improvements

without sustained agency support.” (This theory also underwrites the common

five-year limit on funding.)

The research evidence: proving is necessary but not sufficient for uptake of good practices

Despite considerable efforts to prove the educational value of research-grounded approaches to learning

(via presentation and publication of research and evaluation findings, workshops and demonstrations),

peers and departments have been highly resistant to the implications of such evidence for their own practice.

Researchers of this phenomenon have noted:

Scientists respond differently to the outcomes of experiments when they are undertaken in education rather then their own discipline.

The opinion of respected research colleagues about teaching and learning methods counts for more than evidence from educational experiments offered by colleagues.

(Thus, the support from radicalized seniors is critical in securing change.)

5. Flawed theories of change undermine any program or project’s

chances of success

“A theory of change is a predictive assumption about the relationship between desired changes and the actions that may produce those changes.” (Connolly & Seymour, 2009)

“If I do x, then, for these reasons, I expect y to occur.”

It is essentially a wager.

Theories of change matter because they drive our choice of actions, and shape their

outcomes

They are more often embedded than explicit: Why?

We are apt to jump from identifying a problem to choosing ways to ameliorate it.

We often fail to articulate the reasons why we expect our strategies for educational change, diffusion, or institutionalization, to achieve the desired results.

If we are unaware of the theory of change on which we are operating:

We cannot discuss or determine whether we are using appropriate strategies.

If our activities are less successful than we anticipated, it harder to see what went wrong.

We may blame our strategies, or how we implemented them.

But we should first consider the predictive assumptions on which our strategies were based:

This applies equally to our choices of disseminative strategies

One powerful way to improve the chances that our activities will succeed is to take the time to:

lay out our our choices of action, discuss why we think they are likely to succeed, discard whatever strategies we can’t justify. Doing this: makes us consider evidence, situation realities, andalternative causes of the problems we want to

address, avoid strategies that “just feel right.” exposes which of our predictive assumptions don’t

hold up.

Cite as: Seymour, E., (2010). Scaling up Innovation: Why Theories of Change Matter. Presented at the Workshop on Disseminating CCLI Innovations: Arlington, VA, February 18-19, 2010.

Unless otherwise specified this work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License (creativecommons.org/licenses/by-nc-sa/3.0/us/)