project on educator effectiveness & quality chancellor summit september 27, 2011 cynthia...

Project on Educator Effectiveness & Quality

Chancellor SummitSeptember 27, 2011

Cynthia Osborne, Ph.D.

Accountability System for Educator Preparation Programs (SB 174)

Standard 1: Certification Exam Passing Rates

Standard 2: Principal Survey of 1st Year Teacher; Graduates’ Survey of Preparedness

Standard 3: Teachers’ Influence on Growth in Student Achievement in First Three Years Post Certification

Standard 4: Quality of the Field Supervision

September 27, 2011 2

Purpose of PEEQ

Tasked by TEA to design a metric to determine EPP’s graduates’ influences on student achievement

Metric is required due to:o SB 174 student achievement standard (#3)o State longitudinal data system grant (SLDS II)o School improvement grant (SIG)o State fiscal sustainability fund (SFSF)

Will focus only on 1st through 3rd year teachers Objective is to provide feedback to EPPs to improve teaching and

student performance – not just for accountability


SB 174 and PEEQ


SB 174 Standard III

Student Performance Growth (VAM)

Teacher Observation

Principal Assessment

Teacher's Impact

School or Grade-Level Performance

Standard IV

Standard III

Standard II

Standard I

Please note that the weights of SB174 and the weights of Standard III have not been determined.

What Predicts Effective Teaching?

Years of experience: teachers show 2/3 of all improvement in first 5 years

General intelligence: selectivity of the college, college entrance exams, teacher licensing exams, and IQ tests

Certification in subject area (especially math): masters degree is not predictive

Rigorous observations of teachers in classroom Most variation in teacher effectiveness is not explained by

“teacher inputs” – focus has shifted to “student outcomes”


Measuring Teacher Effectiveness in Texas

Our current system, PDAS, does not provide information to the state or EPPs on effective teachers

Over 98% of teachers are rated proficient, thus EPPs have no indicators to improve programming

NCLB standards (highly effective) are inconsistent with PDAS findings; students are not all proficient

Goal is to improve the quality of the pipeline of new teachers in the state by providing EPPs with information on how their graduates are influencing student achievement

September 27, 2011

Measuring Teacher Effectiveness Across the Nation

States and districts are increasingly using student performance to measure teacher effectiveness

Have moved from a system of measuring “teacher inputs” to measuring “student outcomes”

31 states have initiated plans

States are moving quickly from plan to implementation; LEAs move more slowly

Wide variety exists in components measured and instruments used

Few states use teacher effectiveness for EPP accreditation – although momentum is gaining

September 27, 2011

An Effective Metric Should Be:

Comprehensive: Includes multiple sources; not just test scores Useful: Leads to improvement in EPPs and teaching Valid & Reliable: Reflects student achievement and considers

variations in populations of students; Problems with measurement error, test scaling, and data quality must be minimized

Integrated: EPPs held accountable for training teachers to do what teachers will be held accountable for doing

Transparent: Clear how components are measured and weighted


Decisions PEEQ Must Make

What components should be included in a comprehensive metric to determine a teacher’s effect on student achievement? How should we weight the components?

What is the most valid and reliable method to measure student performance using test scores?


A Comprehensive Metric Should Include:

Growth in Student Performance on TAKS (VAM) Observations of Teachers in Classroom

o Classroom Environmento Curriculum and Instruction

Principals’ Assessment of Influence on Achievement Teacher’s Impact on Student Achievement School or Grade Level Growth in Performance


Weighting of Components Will Be Determined Empirically


30%

20%

50%

TAP Model

Student Performance (Value-Added Measure)

School Performance (Value-Added Measure)

Skills, Knowledge, & Responsibilities

50%

40%

5% 5%

IMPACT - DCStudent Performance (Value-Added Measure)

Observations

School Value-Added

Commitment to Community

50%50%

LouisianaStudent Per-formance (Value-Added Measure)

TBD

35%

35%

15%

15%

MemphisStudent Performance (Value-Added Measure)

Observations

Stakeholder Perceptions

Content Knowledge

40%

30%

30%

Hillsborough

Student Performance (Value-Added Measure)

Principal Observation

Mentor/Peer Teacher Observation

PEEQ’s Metric


Student Performance Growth (VAM)

Teacher Observation


Teacher's Impact

School or Grade-Level Performance

Please note that the weights have not been determined.

Measuring Student Performance of EPPs’ Graduates

Common approacheso Percent of students passing standardized testo Percent increase in students passing test

Strengthso Straight forward and transparento Common standard and expectation of knowledge

Limitationso Does not account for a student’s prior knowledge or growtho May lead to focus on teaching to test



Value Added Modelso Goal is to predict what a student should score and compare to

actual performance – “teacher effectiveness” is the differenceo Various methods used to calculateo Require a large enough sample to create comparisonso Need sufficient data to accurately predict student performanceo Attribution of students to teachers is crucialo Tests must be vertically aligned to show growtho PEEQ will use a VAM, but not at individual teacher level, yet



Value Added Models (Continued) Strengths

o Account for past performance of students and student - and school-level characteristics that may affect teaching and student performance

o Emphasis is on growth in student performanceo Teachers’ past effectiveness is highly predictive of future

Limitationso Difficult to understand calculations (less transparent)o Measurement error may bias estimates


Observations of Teachers

Must be mindful of the scale:o 75,000 beginning teachers (in first 3 years)o 1,400 school districts

Principals are required to observe beginning teachers and fill out a survey on each teacher (ASEP)

This survey provides the platform to gather information on teachers’ classroom environment and instruction (elements linked to student achievement)


Classroom EnvironmentTo what extent is this beginning teacher able to . . . (4 pt scale)

Effectively implement discipline-management procedures? Communicate clear expectations for achievement and behavior

that promote and encourage self-discipline and self-directed learning?

Provide support to achieve a positive, equitable, and engaging learning environment?

Build and maintain positive rapport with students? Build and maintain positive rapport and two-way communication

with students’ families?


InstructionTo what extent is this beginning teacher able to . . . (4 pt scale)

implement varied instruction that integrates critical thinking, inquiry, and problem solving?

respond to the needs of students by being flexible in instructional approach and differentiating instruction?

use the results of formative assessment data to guide instruction? engage and motivate students through learner-centered instruction? integrate effective modeling, questioning, and self-reflection (self-assessment)

strategies into instruction? assume various roles in the instructional process (e.g. instructor, facilitator,

audience)? set clear learning goals and align instruction with standards-based content? provide quality and timely feedback to students?



“How would you rate this teacher’s influence on student achievement?”

10 point scale: unacceptable to exceptional


Information to EPPs for Standard #3

EPPs will get a total score, and disaggregated information on graduates

Information to EPPs will include:o Overall effectivenesso Aggregate scores by each metric componento Scores by:• Certification program• Type/location of school• Grade level/subject matter


Feedback from Districts/Teachers

Want better indicators of effectiveness; have concerns about the uses of the metric

Claim that quality of field supervision distinguishes EPPs

Encourage a stronger dialog and more interaction between districts and EPPs to align teaching and practice


Feedback from EPPs

Deans want better indicators of effectiveness of graduates

Concerned about variation in support teachers receive across districts

Concerned that information has not previously been available; would like information for two years prior to accountability

No consensus on what new teachers should know relative to experienced teachers


We want your input!

We have a statistical advisory group and invite each EPP to designate a member.

We conduct quarterly stakeholder meetings and invite all stakeholders to participate.

We update our website regularly with new research briefs: www.utexas.edu/lbj/peeq


Expected Timeline

Principal Survey (Standard #2) administered in summer 2011

Data linking teachers and students available fall 2011

Pilot Metric available spring 2012

Pilot Metric presented to SBEC June 2012


project on educator effectiveness & quality chancellor summit september 27, 2011 cynthia...

Documents

student achievement

student performance

student outcomesseptember

student achievementmetric

student achievementseptember

teacher effectivenesshave

weights of standard

training teachers