Dr. Robert Curtis, Director, IMSSDr. Susan Tucker, Evaluation & Development Associates LLC

Defining Effective STEM Teaching and Learning

Through a Logic Model Cycle

Agenda

① Setting the Context

② Logic Model Theories of Action

③ Logic Models Applied to MSP work

④ Emerging Questions and Learnings about Logic Models for our MSP

Integrated Middle School Science Project Goals① Increase quality, quantity, and diversity of

middle school science teachers by providing PD and coaching support so teachers improve science content knowledge, pedagogy, inquiry, and leadership

② Create online PD anchored in inquiry-based science instructional aligned with the Foundational Level General Science Credential, state and national science standards, & lesson study

③ Develop partnership infrastructure to support systemic change through a coherent focus on building science instructional leadership capacity.

Definition of Effective STEM Teaching• integrates science and engineering

practices, cross cutting ideas and content knowledge.

• focus and support inquiries and science practices while interacting with students

• orchestrate discourse among students about scientific ideas

• challenge students to accept and share responsibility for their own learning

• recognize and respond to student diversity and encourage all students to participate fully in science learning

STEM Learning: Scientific and Engineering Practices (NRC Framework for K-12 Science Education)1. Asking questions (for science) and defining problems (for engineering)

2. Developing and using models

3. Planning and carrying out

investigations

4. Analyzing and interpreting data

Scientific and Engineering Practices5. Using mathematics, information and computer technology, and computational thinking

6. Constructing explanations (for science) and designing solutions (for engineering)

7. Engaging in argument from evidence

8. Obtaining, evaluating, and communicating information

Logic Model Theory of ActionThe logic model can serve as the basis for program evaluation, research and project management plans, involving stakeholders in logic modeling process so that buy-in and “ownership” occurs, which is central to the success of the subsequent evaluation and research efforts and use of results ( Torres, Hopson & Casey, 2008  )   

Logic Model Uses & Benefits

• Facilitates conversation to negotiate common language & goals

• Represent program and theories graphically

• Align with budget to articulate & review priorities

• Shows how learning, action, & outcomes related

• Delineates & aligns indicators of achievement

• Enables a common framework and road map

Think-Pair-Share:

Expectations• Have you had experience

using LM?

• If so, how have you used them?

• At this point, what do you want to get out of this session?

IMSS examples of LM• Define effective STEM teaching and

learning

• Refine and evaluate project goals outcomes and activities across various project components

• Inform and coordinate program management, research and evaluation

• Increase coherence across all partners

• Build capacities of all partners for inquiry-based teaching and learning and evaluation

Logic Model as Road Map

• Where are we going?

• How will you get there?

• What will tell you that you’ve arrived?

12

Generic logic model

»

Inputs Outputs Outcomes Outcomes Outcomes

Activities

Participation

» »»»

A diagram of the theory of how a program is supposed to work

A graphic depiction of relationships between activities and results

Strategy Impact

C O N T E X T

Source: University of Wisconsin – Extension

http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

EVALUATION: check and verify

What do you want to know? How will you know it?

PLANNING: start with the end in mind

University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

Logic Model Process applied to NSF MSP• Started with 5 year program logic model and then logic models developed by various project working groups.

• Focused on building capacity of partners as well as some subgroups that cut across partners to use logic modeling as a strategy to reveal and negotiate core assumptions.

• Individual logic models serve as a tool for inquiry into the various project components and groups.

• The process of modeling and the product of negotiated models serve multiple purposes: a project management tool, evaluation tool and reflective tool for each of our groups and the project as a whole.

INPUTS OUTPUTS OUTCOMES

Program investments

Activities Participation Short Medium

What we invest

What we do

Who we reach

Long-term

Linear vs iterative projects

Feedback loops and multi-dimensions

University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

What results

Starting is not always pretty

Phasing in LM① during proposal writing: negotiated overall LM for

grant & then generated annual LM’s

② During year 1-2: created LM’s for core components of grant: PD, coaching, research, preservice, afterschool program

③ Using constant comparison of LM revealed 7 core assumptions

④ Evaluator and project management team aligned core components by gap analysis & identified priorities

⑤ Evaluator facilitated quarterly data retreats to align data of R&E with LM plans

Project Overview

Project Overview

Project Overview

Project Overview

Project Overview

Project Overview Logic Map: Year 1

Project Overview Logic Map: Year 2

Coaching Logic Model: Year 2

Research Question 2Logic Map Year 2: Leadership Team, Faculty, Coaches, CoPs &

Evaluator

27

Think-Pair-Share:How could you use a logic model?

• What purpose(s): Evaluation, planning, management, collaboration?

• What groups could use LM and how?• What project complexities do you need to

consider:– Number of partners?– Diversity of partners?– Years of collaborating prior to onset of grant?– Number of skilled group facilitators across partnership?– Base of trust across partners?

Logic Model Outcomes• Process helped each group identify clearer goals,

theories of action and actionable outcomes.

• Focus priorities each year, and serve as a management tool to self-monitor progress.

• Process helped each group identify clearer goals, theories of action and actionable outcomes

• Network across groups to increase coherence across partners; build capacity and share learning's about effective STEM teaching

• Has helped examine distribution of leadership and build towards systemic instructional change in all partner organizations.

Emerging Questions @ LM• What is needed to facilitate LM across the

diversity of our partnership?

• How does LM capacity evolve over time across various partners and subgroups?

• How does LM contribute to sustainable capacity building of various stakeholders regarding project planning, implementation and evaluation use?

• How does LM contribute to conceptualizing partnership?

Learnings• Process of LM may be more important in the

short term given shifts in first 1-2 years

• Collaborative structures in second year enable more focused & aligned questions and methods for evaluation, research and program across partners

• Ecological framework reflects the partnership’s evolution of core assumptions

• Authentic partnership requires critical reflection about unexamined assumptions & trust

Resources and References

• Torres, R. T., Hopson, R. K., & Casey, J. (2008). Case Study of LM Use: Building STEM Education and Evaluation Capacity Through Research on LM Use. AERA, Denver, Co.

• UW-Cooperative Extension: Program Development and Evaluation. http:// www. uwex. edu/ ces/ pdande/

• RAND Working Paper, Math Science Partnership of Southwest Pennsylvannia, Year Three Evaluation Report, 2006

Project Overview Logic Map: Year 3

Project Overview Logic Map: Year 4

Project Overview Logic Map: Year 5

Research Question 2Logic Map Year 2: IHE Partners

Research Question 2Logic Map Year 2: Evaluate Grant Partners

EXAMPLE: Do you really have a partnership?

What strategies have helped explore assumptions in your project?

• partnerships are where partners are closely bound to one another, share common goals, share liability and risk, and have a mutual interest in adapting behavior to one another in the pursuit of shared objectives.

Think-Pair-Share:

Questioning Assumptions

Research says…• organizations tend to require

compelling environmental pressures to engage in such potentially invasive relationships (Aldrich, 1999)

Assumption• EXAMPLE: partnerships are where

partners are closely bound to one another, share common goals, share liability and risk, and have a mutual interest in adapting behavior to one another in the pursuit of shared objectives.

Core Assumptions to examine:• Continuous collective improvement

• Culturally responsive practice

• Building capacity for CoP

• Networked Learning

• Distributed Leadership

• Authentic Partnership

• Data enhanced systemic instructional change

© 2011 IMSS. All Rights Reserved.