the evaluation of mathematics and science partnerships program a quasi experimental design study...

The Evaluation of Mathematics and Science Partnerships Program

A Quasi Experimental Design Study

Abdallah Bendada, Title II Directorhttp://www.dpi.wi.gov/cal/t2bgrant.html

MSP Regional ConferenceFebruary 14-15, 2008

• Mathematics and Science Partnerships Grant

• Evaluation Model: Quasi-experimental design

• Sampling

• Instruments

• Outcomes

• Conclusions

• Future Studies

Program Design

What is the true impact of the Mathematics and Science Partnerships Grant on student

achievement in Wisconsin?

Evaluation Model

1. Participants reaction (Surveys, logs, …)

2. Participants Learning (Assessment, tests, ..)

3. Organization support for changes (administration )

4. Participants use of new knowledge (Observation, performance)

5. Student learning outcomes (Assessment, tests, …)

All = Gold

2 to 5 = Silver

2 and 5= Copper

Anyone = Tin

R O X OR O O

R is the program Group and R the Comparison

Group

Time

O refers to the testing

X is the treatment

Vertical alignment of O shows the pre- and post- testing time

Design

Sampling

• Program group: 75 middle school mathematics teachers from high-need LEAs. pre-selected

• Only 69 teachers participated completed the program

• Comparison group: 75 middle school teachers from similar

LEAs. Matched to the program group.

• Only 71 teachers completed the program

• Program group: 5 students from each participating teacher randomly assigned. (325)

• Comparison group: 5 students from each participating teacher randomly assigned. (325)

Instruments

1- Teachers

The Test of Mathematics Knowledge for Teaching School Mathematics also known as Learning Mathematics for Teaching (LMT) developed by Deborah Ball and Hyman Bass at the University of Michigan as an NSF project, were used to assess the teacher in both the pre- and post-test.

2- Students

The Measures of Academic Progress (MAP) developed by the Northwest Evaluation Association (NWEA) in mathematics has been used to assess student achievement .

More than 138 school districts in Wisconsin use MAPs in mathematics and reading to monitor the progress of student achievement.

X Xz

SD

z

2

( )1

1xx

k X k Xr

k k

1 2

2 21 2( ) / 2

X Xd

SD SD

Statistical Formulas

Score Gain Differences between Program and Comparison Groups

-2

0

2

4

6

8

10

12

14

16

1 2 3 4

Groups

Sco

re G

ain

s

Program Teachers

Comparison StudentsComparison Teachers

Program Students

Program Teachers' Gains

-2.00

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

0 20 40 60 80

Teacher

Z-S

core

Teacher: Score Distribution

0

0.1

0.2

0.3

0.4

0.5

-3 -2 -1 0 1 2 3 4

Score

Freq

uenc

y

ProgramComparison

Student Gains

-5

0

5

10

15

20

0 10 20 30 40 50 60 70 80

Student Teams

Raw

Sco

re G

ain

Program

Comparison

Z-Score True Gains

-2.00-1.50-1.00-0.500.000.501.001.502.002.503.00

-2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Teacher Gain

Stu

den

t Gai

n

Students: Score Distribution

0.00

0.20

0.40

0.60

0.80

1.00

1.20

-2.00 -1.00 0.00 1.00 2.00 3.00

Score

Comparison

Program

Raw Score

Program Group Comparison Group

Teachers Students Teachers Students

Pre Post Gain Pre Post Gain Pre Post Gain Pre Post Gain

mean 8 18 10 12 23 11 8 8 0 9 9 0

SD 1.97 2.79 1.90 3.64 1.81 3.30 1.06 1.22 1.00 1.69 1.46 1.33

min 3 12 6 6 18 3 6 4 -4 5 5 -2

Max 13 24 15 19 26 18 11 11 1 14 12 4

d 6.44 4.32 --------- --------------

Participants 67 335 67 335

Raw score Analysis

Conclusions

• Positive Impact in mathematics,

• Program teacher gained more content knowledge,

• The program impact is on teachers with lower content knowledge,

• Program students achieved higher,

• The program impact is on students with lower content knowledge,

• Effective evaluation designs are necessary for impact programs, and

• Variables such as the socioeconomic group, ethnic group, and disability should be factored in the evaluation design

Disclaimer

The instructional practices and assessments discussed or shown

in these presentations are not intended as an endorsement by

the U.S. Department of Education