measuring changes in teachers’ mathematics content knowledge dr. amy germuth compass consulting...

Measuring Changes in Teachers’ Mathematics

Content Knowledge

Dr. Amy Germuth

Compass Consulting Group, LLC

2

Overview of Session

Introductions

Importance of Measuring Teacher Content Knowledge

Measuring Teacher Content Knowledge

Discussion

3


US ED MSP Program Office Perspective

The Government Performance Results Act (GPRA) A law passed in 1993 that requires federally funded

agencies to develop and implement an accountability system based on performance measurement.


4

Requires all federally funded program to: Outline long-term and annual performance goals that

include outcomes, Develop indicators to assess performance goals, Collect and analyze data on the indicators, and Report progress toward achieving performance goals based

on GPRA data and data from grantees’ evaluations.

MSP GPRA indicator related to content knowledge The percentage of MSP teachers who significantly increase

their content knowledge, as reflected in project-level pre- and post-assessment


5

GPRA and grantee evaluation findings are used by Congress to inform appropriations – they want concrete evidence of results

More uniform and higher quality data will enable the MSP Program office to more accurately and convincingly assess progress toward MSP performance goals

This is challenging for measuring changes in teachers’ content knowledge


6

MSP Grantee Perspective

Has the potential to affect funding of the program at federal level

US ED annual reporting requirement

One of the more direct variables for individual partnerships to measure and demonstrate program outcomes


7

Because of the variability in the activities projects are implementing to improve teacher content knowledge the use of one instrument is not a viable solution

Some instruments are better than others Search for or develop an instrument that:

Is aligned with the content of your MSP professional development activity;

Has been piloted with teachers similar to those participating in your program;



8

Provides evidence of reliability or the extent to which an instrument yields consistent, stable, and uniform results over repeated administrations under the same conditions each time

Provides evidence of validity or the extent to which 1) the instrument measures the skills it sets out to measure and 2) the inferences and actions made on the basis of the scores are appropriate and accurate


Figure obtained from the website: http://www.socialresearchmethods.net/KB/rel&val.htm

Discussion Question: What instruments have grantees been using? What are the properties of those instruments?

9

Potential Tests Learning Mathematics for Teaching (LMT) Project – University of

Michigan Includes several surveys that measures teachers’ content knowledge for

teaching mathematics in : Number and Operations(K-6, 6-8), Patterns, Functions, and Algebra (K-6, 6-8), and Geometry (3-8)

http://sitemaker.umich.edu/lmt/about

Diagnostic Teacher Assessments in Mathematics and Science (D-TAMS) - University of Louisville Six versions of four different pre-post assessments designed to assess

change in teacher content knowldge are available: Number / Computation, Geometry / Measurement, Probability / Statistics, and Algebraic Ideas.

http://louisville.edu/edu/mathscience/diagnostic.html

Knowing Mathematics for Teaching Algebra (KAT) – Michigan State University Instruments not yet available http://www.msu.edu/~/kat/index.htm

10

Resources

Overview of reliability and validity http://writing.colostate.edu/guides/research/relval/index.cfm http://www.socialresearchmethods.net/KB/ http://www.socialresearchmethods.net/tutorial/Colosi/lcolosi2.htm Crocker, L. & Algina, J. (1986). Introduction to classical and modern

test theory. Belmont, CA: Wadsworth. Shultz, K. S. & Whitney, D. W. (2005). Measurement theory in

action: Case studies and exercises. Thousand Oaks, CA: Sage.

Information on established instruments http://www.apa.org/science/faq-findtests.html http://www.unl.edu/buros/


11

Information on developing instruments Crocker, L. & Algina, J. (1986). Introduction to classical and modern

test theory. Belmont, CA: Wadsworth. Haladyna, T. M. (1999). Developing and validating multiple-choice

test items (2nd ed.). Mahwah, NJ: Erlbaum. Osterlind, S. J. (1998). Constructing test items: Multiple-choice,

constructed response, performance, and other formats. Boston, MA: Kluwer.

Shultz, K. S. & Whitney, D. W. (2005). Measurement theory in action: Case studies and exercises. Thousand Oaks, CA: Sage.


12

When To Administer Your Test

Administer the pretest immediately prior to the MSP activity or college course

Administer the posttest after the MSP activity and follow-up are complete, but within the same school year as the initial MSP activity


13

How to Determine Whether Teachers Have Made Significant Gains

To report for GPRA, US ED needs a way to tell whether the observed gains in teacher content knowledge between the pretest and the posttest are significant

US ED is proposing that projects use a statistical test called a dependent t-test to determine whether teachers have made significant gains


14

Proposed process The federal program office would supply grantees with an Excel

spreadsheet with embedded formulas to do the needed calculations

Grantees would enter into the spreadsheet the pretest scores and posttest scores for the teachers they test

Grantees would use one spreadsheet per test The spreadsheet would calculate the needed statistics and

produce a report for grantees showing the total number of teachers and the number who showed significant gains

Grantees would report this information to US ED through the Annual Performance Report

US ED would aggregate the information from all grantees and use it for GPRA reporting


15

Discussion

Experiences developing content knowledge instruments

Challenges encountered in measuring changes in content knowledge

Questions about measuring changes in content knowledge

measuring changes in teachers’ mathematics content knowledge dr. amy germuth compass consulting...

Documents