using curriculum to change how teachers teach science … · teachers teach science and students...

Using Curriculum to Change How Teachers Teach Science and Students

Learn Science A Paper Set Prepared by

Susan M. Kowalski, Janet Carlson, Pamela Van Scotter, Brooke N. Bourdélat-Parks, Stephen R. Getty, Betty

Stennett, and Paul Beardsley NARST Annual Conference

Indianapolis, IN 25 March 2012

Flow of the Paper Set

•  A Research-based Development Process •  Features of the Curriculum •  Teacher Practice Associated with

Curriculum •  Student Learning Associated with the

Curriculum

The Challenge – 1

•  Middle school is a critical time for –  Inspiring student interest in science – Establishing foundational understandings in science (Hanson, 2004; Beghetto, 2007)

•  To inspire interest and promote understanding, we need – Effective teaching practices – Coherent, rigorous, focused instructional materials

The Challenge – 2

•  Preponderance of materials are “fragmented” (AAAS, 2001; Schmidt, 2001)

•  Two-thirds of middle school science teachers have a degree in a field other than science (Fulp, 2002)

•  Teachers teaching out-of-field rely heavily on materials (Ball & Feiman-Nemser, 1988)

•  Problems are most severe in low-income schools

The Challenge – 3

•  Teachers need access to materials that support development of teacher content knowledge and pedagogical content knowledge.

•  Achievement gaps by demographic subgroups persist nationwide

•  The U.S. requires a scientifically literate citizenry to face global challenges

The Challenge – 4

•  Science instruction must evolve to incorporate current knowledge about learning.

•  Rigor, coherence, and focus of middle school science curricula need to be improved.

Our Perspective

Effective curricula can be a valuable means to improve student interest

and achievement in science.

Taking Science to School, NRC, 2007

Curriculum (Matters)

Research

Design Development

Our Work to Address the Challenge

•  Develop instructional materials that are beneficial to both teachers and students

•  Part of a US Department of Education IES-funded Goal 2 development and innovation (Grant # R305A080422)

•  Materials iteratively developed and tested (pilot study plus two feasibility tests)

•  Materials include seven key features

Theoretical Framework for Approach to Learning

•  Our curriculum development process draws on years of research on cognitive development and how humans learn – Vygotsky (1962) and Piaget (1975) – Summary of recent findings well-articulated

more recently in the meta-analysis How People Learn (Bransford, Brown, & Cocking, 2000).

Theoretical Framework for Approach to Learning

•  Key findings summarized in How People Learn include the following: –  Students come to class with their own conceptions

about how the natural world works. –  Students need a strong foundation of knowledge upon

which to build their understanding of new ideas. –  Students benefit from a metacognitive approach to

learning where they are responsible for monitoring their own progress.

Features of the Curriculum Materials For students 1.  Rigorous, coherent, and focused 2.  The BSCS 5E Instructional Model™ 3.  Comprehensive assessment package 4.  Metacognitive strategies 5.  Literacy strategies 6.  Collaborative learning For teachers 7.  Highly educative

Scope & Sequence

6th Grade Form and Func0on/

Evolu0on and equilibrium

7th Grade Constancy and Change / Evolu0on and equilibrium

8th Grade Systems and Subsystems /

Energy and Ma@er

Con

tent

Sta

ndar

ds Science as

inquiry •  Science as a way of

knowing •  Science as a way of

knowing

•  Science as a way of knowing

•  Develop explanations using evidence

•  Communicate scientific procedures and explanations

Core concepts (physical)

•  Properties and changes in properties of matter

•  Motions and forces •  Integrating chapter

•  Energy in energy out •  Transfer of energy

Core concepts (life)

•  Structure and function in living systems (cells)

•  Regulation and behavior (structure and function)

•  Reproduction and heredity

•  Diversity and adaptations of organisms

•  Inheritance and evolution of behavior

•  Living energy •  Structure and function in

living systems •  Regulation and behavior •  Populations and ecosystems

Core concepts (Earth-‐space)

•  Structure of Earth systems

•  Integrating chapter

•  Earth’s history •  Integrating chapter

•  Energy in earth systems •  Earth in the solar system

Mul0-‐disciplinary Unit – Science & Society

•  Natural hazards •  Risks and benefits •  Abilities of

technological design

•  Science and technology in society

•  Understandings about science and technology

•  Energy and your body •  Personal health •  Body systems

8th Grade Systems and Subsystems /Energy and Ma@er

Science as Inquiry •  Science as a way of knowing •  Develop explanations using evidence •  Communicate scientific procedures and

explanations Physical Science •  Energy in energy out

•  Transfer of energy Life Science •  Living energy

•  Structure and function in living systems •  Regulation and behavior •  Populations and ecosystems

Earth-space Science •  Energy in earth systems •  Earth in the solar system

Science & Society •  Energy and your body •  Personal health •  Body systems

Development Process

•  Development team included scientists, teachers, and science educators

•  Held a design conference – Shared conceptual flow graphics for units and

chapters •  BSCS 5E Instructional Model™ guided the

development – Engage, Explore, Explain, Elaborate, and

Evaluate

Theoretical Framework – Curriculum

•  Four strands from Taking Science to School (NRC, 2007) guided our process. This report recommends that students 1.  know, use, and interpret scientific explanations of

the natural world, 2.  generate and evaluate scientific evidence and

explanations, 3.  understand the nature and development of scientific

knowledge, and 4.  participate productively in scientific practices and

discourse.

Development Process •  Use the Understanding by Design

approach (Wiggins and McTighe, 2005)

– Backward Design •  Stage 1: What do we want students to learn? •  Stage 2: What will serve as evidence that they

have learned? •  Stage 3: What should the learning sequence be?

– Begin the development with the Evaluate – Then develop the Engage, Explore, Explain,

and Elaborate for each chapter •  Iterative process

Review and Test

•  Internal reviews were conducted by the team, and external reviews were conducted by content experts.

•  First test of feasibility included 25 teachers – Selected for diversity across

•  Geographic location •  Urban, Suburban, Rural •  Student population

Formative Stage of Research

•  Collected formative data – Student pre-posttests – Teacher surveys – Classroom observations

•  External reviews of teacher and student materials by experts – Used rubrics designed to evaluate the extent

to which the materials reflected each of the intended features

Iterative Revision Process

•  The team revised the materials as necessary to attend to the external reviews of the materials, the observation data, and the student learning results following the first feasibility test.

•  The team used the same backward design process and internal review process.

•  These iterative steps resulted in 6 iterations of curriculum revision.

Second Test of Feasibility

•  Tested materials around the US in second test of feasibility – 24 field test teachers – Approximately 2000 students – Attended to diversity – Hosted a three-day orientation for teachers

•  Presenting results from this test

Characteristics of Participating Teachers & Students and their Schools

•  12 states •  17% urban; 25% rural; 58% suburban

districts •  13% private schools; 87% public •  2-100% of students from under-

represented groups (average: 50%) •  2-37 years of teaching experience

(average: 14 yrs)

Methodology – Mixed Methods

•  Student data –  modified Attitudes

Toward Science Inventory (Weinburgh & Steele, 2000)

–  Pre and Posttests (content and confidence)

–  Sample student notebooks

•  Curriculum data –  external expert reviews

of features

•  Teacher data –  Usability of each feature –  Observations

• RTOP (Sawada et al., 2002) • BSCS Teacher Fidelity of Implementation Rubric (BSCS, 2009)

• BSCS Student Fidelity of Implementation rubric (BSCS, 2009)

• Collaboratives for Excellence in Teaching Preparation (CETP) 5-minute observation rubric (Lawrenz et al., 2007)

Ongoing Research

•  Digital version of the program will be available at no monetary charge for use during the 2012-13 school year.

•  Go to elearn.bscs.org to register and see the curriculum.

What is the Evidence of Usability and Feasibility of the Curriculum Features?

Features of the Curriculum Materials For students 1.  Rigorous, coherent, and focused 2.  The BSCS 5E Instructional Model™ 3.  Comprehensive assessment package 4.  Metacognitive strategies 5.  Literacy strategies 6.  Collaborative learning For teachers 7.  Highly educative

Data Collection

Key Feature External Review

Classroom Observations

Teacher Surveys

Rigorous, coherent, and focused X X

The BSCS 5E Instructional Model X X

Comprehensive assessment package X X

Metacognition strategies X X Literacy strategies X X X Collaborative learning X X X

Dominant theme of inquiry Unifying concepts of energy and systems Chapter Organizers help students and teachers see where they are in the learning process by making connections between ideas explicit

Feature 1: Enhancing Rigor, Coherence, and Focus

Feature 1: Enhancing Rigor, Coherence, and Focus

Teacher Surveys External reviewers (combined results)

Not at all coherent

Low coherence

Moderate coherence

High coherence

Not at all coherent

Somewhat coherent

Coherent Highly coherent

Feature 2: The BSCS 5E Instructional Model™

Engage

Explore

Explain Elaborate

Evaluate


External reviewers

0% 25% 50% 75% 100%

Use of 5Es

Data points represent the 4 units


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PS1

PS2

PS3

PS4

PS5

PS6

LS1

LS1a

LS2

LS2a

LS3

LS4

LS5

LS5a

ES1

ES2

ES3

ES3a

ES4

ES4a

ES5

ES5a

ES6

ES7

ES7a

SS1

SS1a

SS2

SS2a

SS3

SS3a

SS4

SS5

SS5a

SS6

SS6a

Percen

t of P

ossible Po

ints

Teacher ID

Teacher Fidelity of Implementa0on to BSCS 5E Instruc0onal Model™


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PS1

PS2

PS3

PS4

PS5

PS6

LS1

LS1a

LS2

LS2a

LS3

LS4

LS5

LS5a

ES1

ES2

ES3

ES3a

ES4

ES4a

ES5

ES5a

ES6

ES7

ES7a

SS1

SS1a

SS2

SS2a

SS3

SS3a

SS4

SS5

SS5a

SS6

SS6a

Percen

t of P

ossible Po

ints

Teacher ID

Student Fidelity of Implementa0on to BSCS 5E Instruc0onal Model™

Feature 3: Comprehensive Assessment Package

Feature 3: Comprehensive Assessment Package

Teachers surveys External reviewers

Not at all effective

Somewhat effective

Effective Highly effective

Not incorporated

Very well incorporated

Incorporated

92% of elements

Feature 4: Metacognitive Strategies

Feature 4: Metacognitive Strategies

Teacher surveys •  In a few cases, classroom observations

showed that students tended to reflect back on thinking to the extent that teachers encouraged them to do so.


Somewhat effective


Feature 5: Literacy Strategies

Feature 5: Literacy Strategies

Teacher surveys

External reviews Results for almost all elements of Science and Society unit


Somewhat effective



Somewhat effective

Highly effective

Feature 6: Collaborative Learning

•  CCCR Strategy – Consider-Contribute-Consult-Revise Strategy

•  Use and reflect on teamwork skills


Teacher surveys

External reviews


Somewhat effective


Element not present

Element present

Element present and

clearly supported


Classroom observations Use of collaborative learning

When collaborative learning was seen, other elements rated…

Feature not evident

Feature evident

Feature not evident

Feature evident

Discussion

Based on teacher surveys, external reviews, and classroom observations •  Each of the 6 key features related to students

were found to be usable and feasible for students and teachers

•  In several chapters, revisions were made after the second field test to further strengthen the key elements

What do the Data Say about Teacher Practice Associated with Curriculum?

Theoretical Framework for Approach to Teaching

Teachers play an interpretive role in bringing curricula to life for their students

– Select elements for inclusion – Emphasize or deemphasize curricular

elements

Theoretical Framework

A complex “teacher-curriculum relationship” exists (Remillard, 2005)

– Contextually based – Dependent on both the teacher and the

curriculum – Tightly interconnected with other teacher

practices

Theoretical Framework

“Curriculum materials could contribute to professional practice if they were created

with closer attention to processes of curricular enactment.”

(Ball & Cohen, 1996, p. 7, emphasis in original)

Theoretical Framework •  To support teachers, curricular materials

should be educative. That is, they should be designed to promote teacher learning as well as student learning.

•  Nine design heuristics of educative science materials guided the development of teacher support materials (Davis & Krajcik, 2005).

•  Nine heuristics articulate roles for materials in three major areas (total of 24 elements) – PCK for Science Topics – PCK for Scientific Inquiry – Subject Matter Knowledge

Educative Materials Heuristics and Elements of Curriculum Materials

Educative Materials Major Area

Element in Materials

PCK for Science Topics

Information on common student conceptions Information on pedagogical strategies Explanations and instructions for process and procedure steps Suggestions for specific opportunities for individual and group assessment Outcomes and Indicators of Success Samples of student work



Elements in Materials

PCK for Scientific Inquiry

Descriptions of the nature of each “E” and rationale for the instructional model Extensive use of “focus questions” within units Multiple activities for students to ask and answer their own questions with support for teachers on how to guide students toward appropriate questions

Rationale for the importance of having students design their own investigations Multiple opportunities for students to use an explanation template and guidance to teachers for fading scaffolds Multiple opportunities for students to collaborate with guidance for both teachers and students on how to share ideas successfully.



Element in Materials

Subject Matter Knowledge

Comprehensive teacher background information on science content Teacher background materials extend beyond the level of understanding required by students Teacher background materials illustrate the relationships between key ideas and everyday phenomena

Example Design Heuristic 4 (includes 2 elements)

Instructional materials should support teachers in anticipating, understanding, and dealing with students’ ideas about science

– Curriculum materials should help teachers recognize the importance of students’ ideas and help teachers identify likely student ideas within a topic

– Curriculum materials should help teachers gain insight into how they might be able to deal with the ideas in their teaching, for example, by giving suggestions of thought experiments likely to promote the development of more scientific ideas.

Design Heuristic 4 in Science and Society Unit, Chapter 5

•  Chapter rated “comprehensive and thorough” by an external evaluator

•  Evidence: –  In the Engage phase, the Process and Procedure

section (p. 12) highlights a particular idea that students might have: “Some may say that they don’t want to know anything about diabetes.” The TE provides a suggestion for how to address this idea: “Encourage them to think of questions a person who just found out they had diabetes might want to know.”

Data Collection in Service of Iterative Development

•  External reviews of teacher materials – Review rubric aligned with nine heuristics – Each of 24 elements scored on a scale

•  Little or none •  Some •  Comprehensive and thorough


•  Teacher surveys – Administered after every activity –  Included items related to both teacher support

materials and student materials –  Included extensive comments on enactment –  Included extensive comments on any changes

teachers made and why (sequence, omissions, augmentations, etc.)


•  Teacher Observations – RTOP (Sawada et al., 2002) – BSCS Fidelity of Implementation Rubric

(BSCS, 2009) •  Student Tests (Pre and Post)

Results First, a caveat… The data presented here resulted from a development study. Because the purpose of the data collection was to inform revisions to the materials, the data are not of sufficient scope to make broad generalizations. For example, there was no comparison group in this study; therefore, we cannot make claims about the benefit of these materials over others.

Results

•  Successful incorporation of MOST educative features:

•  Two elements out of 24 needed enhancement: –  The curriculum materials should help teachers adapt

and use approaches for collecting and analyzing data across multiple topic areas even when the data being collected seem fairly different (e.g. plant growth as opposed to weather conditions).

–  Curriculum materials should help teachers recognize the importance of having students design their own investigations.

Examples of Changes to Materials Based on Review

Collecting and Analyzing Data Teacher’s Guide helps teachers adapt and use vector notation as a means of collecting and analyzing data by 1.  introducing the technique as students

investigate planetary velocities around a central star and

2.  showing teachers how to adapt the technique as students investigate the relative strengths of gravitational forces of attraction between planetary objects.

Examples of Changes to Materials Based on Review

Importance of Having Students Design Investigations

•  Revised a Teacher’s Guide for helping students design investigations. The introductory paragraphs of this guide provide extensive rationale for the importance of providing experiment design opportunities to students.

Results

•  Evidence of teacher use of reform-based practice: – Mean RTOP score (100 possible)

•  This study: 63.3 •  Nationally for MS science: 50 (Sawada et al., 2002) •  Finding aligns with that of another BSCS study:

–  HS materials –  Random assignment –  External researchers conducting observations –  BSCS teachers had RTOP scores > 2 standard deviations

above control teachers

Results

•  Mean Fidelity of Implementation (FoI) score: 88% •  Significant correlation between FoI and RTOP:

r = .423; p = .040 ↑Fidelity ~ ↑RTOP

•  Association with student learning??? –  Hierarchical linear modeling (students nested within

teachers) –  FoI neared significance (p = .056) in predicting mean

student posttest (adjusted by pretest and student demographics)

–  RTOP non-significant (p = .456)

Results in short…

Use of the materials is positively

associated with reform-based teaching practices and is also associated with

improved student achievement.

What is the Evidence of Student Learning?

Proficiencies of Science

Students should be able to •  know, use, and interpret scientific

explanations of the natural world; •  generate and evaluate scientific evidence

and explanations; •  understand the nature and development of

scientific knowledge; and •  Participate productively in scientific

practices and discourse (NRC, 2007)

Methods

•  Pre- and Posttests – Content Questions – Confidence Questions

•  Student Notebooks •  Classroom Observations

– CETP 5-minute observation protocol (Lawrenz et al., 2007)

– Fidelity of Implementation of the BSCS 5E Instructional Model: Students in the Classroom [Student FoI] (BSCS, 2009)

Strand 1: Know, use, and interpret scientific information

Content Questions Unit Content Gain

Mean Difference

(SE)

p Effect Size (95% CI)

Earth Science 5.23 (.19) .001 1.12 (1.01 to 1.23)

Physical Science

3.65 (.24) .001 .78 (0.66 to 0.91)

Life Science 3.84 (.25) .001 .65 (0.51 to 0.79)

Science and Society

5.93 (.23) .001 1.28 (1.14 to 1.43)

Achievement Gains by Demographic Group

Subject Student Group N p Value

Effect Size

Earth Science

Gender Boys 329 < .001 1.12

Girls 380 < .001 1.23

Race/Ethnicity White/Asian 330 < .001 1.19

Underrepresented 379 < .001 1.46

ELL Status English Native 631 < .001 1.31

ELL 78 < .001 1.31

FRL Status No FRL 415 < .001 1.32

FRL 294 < .001 1.46



Effect Size

Life Science

Gender Boys 182 < .001 .813

Girls 225 < .001 1.03

Race/Ethnicity White/Asian 266 < .001 .971

Underrepresented 141 < .001 .855

ELL Status English Native 346 < .001 .797

ELL 61 < .001 .990

FRL Status No FRL 335 < .001 .885

FRL 72 < .001 .957



Effect Size

Physical Science

Gender Boys 269 < .001 .943

Girls 248 < .001 1.43




ELL 98 < .001 1.28


FRL 190 < .001 1.43



Effect Size

Science and Society

Gender Boys 199 < .001 1.31

Girls 257 < .001 1.53




ELL 141 < .001 1.48


FRL 222 < .001 1.62

Putting Achievement Gains in Context

Lynch and colleagues (2005) conducted a study comparing highly rated reform materials (Chemistry That Applies, State of Michigan, 1993) to “business as usual.”

Group N p Value Effect Size Chemistry that Applies

1087 p < .001 .81

Business as Usual

809 p < .001 .49

Strand 1: Know, use, and interpret scientific information

Confidence Questions Unit Confidence Gain

Mean Difference (SE)

p Effect Size (95% CI)

Earth Science 10.8 (.52) .001 .77 (0.66 to 0.88)

Physical Science

3.10 (.50) .001 .27 (0.14 to 0.41)

Life Science 11.5 (.95) .001 .77 (0.49 to 1.04)

Science and Society

10.93 (.62) .001 .78 (0.65 to 0.92)

Example of Change in Content and Confidence

Q. Suppose you can measure the total amount of energy in different feeding levels. Which level would have the LEAST amount of energy?

A. Plants and other producers B. Herbivores that eat plants C. Carnivores that eat herbivores D. Secondary carnivores that eat other carnivores

How confident are you that you answered the question correctly?

Changes in Understanding and Confidence Frequency of selecting “A” (Misconception)

Frequency of selecting “D” (Correct Answer)

Which answer had the highest confidence rating?

Pretest 99 79 Misconception (answer A)

Posttest 40 152 Correct answer (answer D)

Generate and Evaluate Scientific Evidence and Explanations

Science Notebooks Score of “2” indicates that responses were often accurate but incomplete.

Element of Explanation

(N = 75)

Mean SD

Claim 2.23 .89

Evidence 2.03 .64

Reasoning 1.80 .68

Context for Understanding Scientific Evidence and Explanation Scores

•  Ruiz-Primo and colleagues (2010) scored students’ explanations in notebooks

Completeness Percent of Students

Complete explanations with claim, evidence, and reasoning

18.1%

Provide only claim and evidence 12.0%

Provide only claim 40.3% Provide only data 9.7%

Understanding the Nature of Scientific Evidence and Explanations

•  Science as Inquiry Assessment – Significant gains from pretest to posttest

(p < .001) – Effect size d = 0.1

•  Science notebooks – Students were able to generate accurate but

incomplete explanations from evidence

Productive Participation in Scientific Discourse

•  Classroom Observations: CETP 5-minute observation protocol – What cognitive demands were placed on students

during each 5-minute segment of time? •  Passively receiving information? •  Applying knowledge? •  Generating Explanations?

– To what extent were students “on task”? •  Majority of students on task > 90% of class time.

> 60% of class time

Productive Participation in Scientific Discourse

•  Student Fidelity of Implementation – Participation closely aligned with developers’

intentions – Example: materials provided clear protocols

for sharing work with other students and revising work based on that feedback

Implications—Development

•  We offer a model of research-based curriculum development.

•  We offer a model of how to incorporate key features of curricula.

Implications—Student Learning

Research-based curricula can play an important role supporting student science proficiency

– Students can learn challenging content; – Students can generate scientific explanations; – Students can understand how scientific

knowledge develops; and – Students can participate productively in

scientific discourse.

Implications—Teachers

•  Curriculum materials can be beneficial for both teachers and students.

•  Educative materials may enhance teacher practice and student learning.

Dissemination Model

•  Ongoing R&D model for materials through online curriculum dissemination beginning Fall 2012

•  Materials available free of monetary charge in exchange for student pre/post data and teacher usability data

•  elearn.bscs.org

Future Research

•  Efficacy trial to increase our confidence in making causal claims between use of the instructional materials and their effects on teacher practice.

•  Further development work (for grades 6-7)

Thanks to the Team!

•  Janet Carlson (PI) •  Pam Van Scotter (Co-PI) •  Susan M. Kowalski (project lead--

research) •  Brooke N. Bourdelát-Parks (developer) •  Betty Stennett (project lead--development) •  Stephen R. Getty (developer) •  Paul Beardsley (developer)

Thanks!

Research supported by the Institute for Education Sciences (IES)

Grant Number R305A080422

This presentation and the associated paper will be available at

www.bscs.org/sessions Monday afternoon

using curriculum to change how teachers teach science … · teachers teach science and students...

Documents