You’ve Got a Great Teacher Education Program so Why

Doesn’t Anyone Know About it?

Mark GirodMark GirodChair, Teacher EducationChair, Teacher Education

Western Oregon UniversityWestern Oregon University

AASCU Academic Affairs Summer MeetingAASCU Academic Affairs Summer MeetingFriday, July 29th, 2011

Portland, Oregon

11

Strengths and challenges in your teacher preparation

programs• Identify 2-3 strengths of the teacher Identify 2-3 strengths of the teacher

preparation programs on your campuspreparation programs on your campus

• Identify 2-3 challenges of your programsIdentify 2-3 challenges of your programs

• Can you identify the essential challenges in Can you identify the essential challenges in teacher preparation today?teacher preparation today?

• Bottom lineBottom line: we need evidence driven : we need evidence driven programs, with shared outcomes, programs, with shared outcomes, systematic programs of research and systematic programs of research and structured dissemination plansstructured dissemination plans

Identifying an outcome measure: articulating the

inference chainTo get from teacher education to impact on pupil’s learning requires a chain To get from teacher education to impact on pupil’s learning requires a chain

of evidence with several critical links: empirical evidence demonstrating of evidence with several critical links: empirical evidence demonstrating the link between teacher preparation programs and teacher candidates’ the link between teacher preparation programs and teacher candidates’ learning, empirical evidence demonstrating the link between teacher learning, empirical evidence demonstrating the link between teacher candidates’ learning and their practices in actual classrooms, and candidates’ learning and their practices in actual classrooms, and empirical evidence demonstrating the link between graduates’ practices empirical evidence demonstrating the link between graduates’ practices and what and how much their pupils learn. Individually, each of these and what and how much their pupils learn. Individually, each of these links is complex and challenging to estimate. When they are combined, links is complex and challenging to estimate. When they are combined, the challenges are multiplied (p. 303).the challenges are multiplied (p. 303).

Cochran-Smith, M. (2005). Studying teacher education: What we know and need to know. Cochran-Smith, M. (2005). Studying teacher education: What we know and need to know. Journal of Teacher Education, 56Journal of Teacher Education, 56 , 301-306., 301-306.

Multi-leveled model…

This is a complex and multilayered inference chain that requires This is a complex and multilayered inference chain that requires examination of data at multiple levels including:examination of data at multiple levels including:

• Candidate level variables (GPA, prior coursework, exam scores…)Candidate level variables (GPA, prior coursework, exam scores…)• We have much of this information about our candidates – but is it in a form we can use?We have much of this information about our candidates – but is it in a form we can use?

• Program level variables (structure, courses, fieldwork, outcomes…)Program level variables (structure, courses, fieldwork, outcomes…)• What is the nature of these variables? Are these grouping variables? Independent What is the nature of these variables? Are these grouping variables? Independent

variables?variables?

• Candidate learning (performance on anchor assignments…)Candidate learning (performance on anchor assignments…)• Variability? How do we rate scaffolding in our failure free systems?Variability? How do we rate scaffolding in our failure free systems?

• Candidate practices in classrooms (observations of teaching, Candidate practices in classrooms (observations of teaching, strategies…)strategies…)

• TWS? Observations? Data quality? Both dependent and mediating? Aggregate?TWS? Observations? Data quality? Both dependent and mediating? Aggregate?

• Contextual factors (classroom, school, community variables…)Contextual factors (classroom, school, community variables…)• Most salient? And are we measuring the right ones?Most salient? And are we measuring the right ones?

• P-12 learning (kind and complexity of outcomes targeted and met...)P-12 learning (kind and complexity of outcomes targeted and met...)• As evidenced by (assessment)? Aggregation? Definitions? Judgment?As evidenced by (assessment)? Aggregation? Definitions? Judgment?

Necessary conditions…

At least five conditions seem to be needed if a teacher preparation At least five conditions seem to be needed if a teacher preparation program is going to serve as context for research:program is going to serve as context for research:

1.1. Persons responsible for the management and operation of the Persons responsible for the management and operation of the program must be inclined toward experimentationprogram must be inclined toward experimentation

2.2. Persons responsible for the management and operation of the Persons responsible for the management and operation of the program must view it as subject to continuous change, and view a program must view it as subject to continuous change, and view a systematically designed program of research on its effectiveness as a systematically designed program of research on its effectiveness as a major data source for its change.major data source for its change.

3.3. Data of a quality that will support trustworthy research must be Data of a quality that will support trustworthy research must be collected as a normal part of program operation.collected as a normal part of program operation.

4.4. Sophisticated data management, storage, retrieval and display Sophisticated data management, storage, retrieval and display capabilities must be available.capabilities must be available.

5.5. There must be an advisory structure to insure the research that is There must be an advisory structure to insure the research that is pursued has value to people in the program as well as to the pursued has value to people in the program as well as to the profession at large.profession at large.

How to run the system…

Need to establish a conceptual model linking the inference chain:Need to establish a conceptual model linking the inference chain:

• P-12 learning is the product of…P-12 learning is the product of…

• Relationship between teacher/curriculum/learner…Relationship between teacher/curriculum/learner…

• Nested in contexts…Nested in contexts…

Kinds of studies needed:Kinds of studies needed:

• Policy oriented – cost/benefits, program evaluationPolicy oriented – cost/benefits, program evaluation

• Practice oriented – follow-up studies, short and long-term effectPractice oriented – follow-up studies, short and long-term effect

• Basic research – hypothesis testing, hypothesis generatingBasic research – hypothesis testing, hypothesis generating

• Quality assurance – form use, rating patterns, inter-rater Quality assurance – form use, rating patterns, inter-rater reliabilityreliability

• Data display – development of procedures to display outcomesData display – development of procedures to display outcomes

• Instrumentation – refinement of instrumentsInstrumentation – refinement of instruments

And the results…

• Teacher preparation programs as contexts for researchTeacher preparation programs as contexts for research

• Empirically validated systems of teacher preparationEmpirically validated systems of teacher preparation

• Model building and theory building that would lead to an Model building and theory building that would lead to an empirically validated field of teacher preparationempirically validated field of teacher preparation

• Policy and practices informed by evidence (best case scenario…)Policy and practices informed by evidence (best case scenario…)

• Better teachers and more and better P-12 learningBetter teachers and more and better P-12 learning

• The argument suggests we have too much to lose to The argument suggests we have too much to lose to not not attempt attempt to become an empirically driven business… and TWS is uniquely to become an empirically driven business… and TWS is uniquely well positioned to help move us ahead if we can improve the well positioned to help move us ahead if we can improve the quality of the data and our ability to aggregatequality of the data and our ability to aggregate

• 1975…1975…

So what barriers have kept us from getting there?

DiscussionDiscussion::

• What are some of the barriers to data driven teacher What are some of the barriers to data driven teacher preparation programs that connect to P-12 student learning?preparation programs that connect to P-12 student learning?

• What will be the consequences if we don’t get there?What will be the consequences if we don’t get there?

• How can we move toward this?How can we move toward this?

What is a Teacher Work Sample?

A Teacher Work Sample is an authentic performance assessment completed in a real-world setting that demonstrates a candidate’s ability to assess, plan, instruct, and reflect in a standards-based educational system and impact student learning in a positive manner.

99

The revolution of standards-based schooling…

The standards set for learning in today’s schools define the successive bars to be reached by students as they progress in their learning, and standards-linked assessments indicate where students stand at a particular point in time with respect to a particular bar, but it is each student that needs to reach each bar and the main job of teachers is to help each student in each classroom make steady progress toward each bar that lies immediately ahead.

(Del Schalock, 2006)

1010

Use for teacher licensure

• Since 1989, in Oregon, each teacher candidate has been required to successfully implement two teacher work samples prior to being awarded initial licensure.

• A teacher work sample is an empirically validated performance assessment that includes:• A setting description• Pre-assessment• Learning outcomes• Lesson plans• Post-assessment• Analysis of learning data• Reflective analysis

1111

Principles…

• An instructional program needs to be aligned with and supportive of what candidates are asked to do, including the documentation and reporting that is required in completing a work sample.

• School contexts that model and are supportive of what candidates are asked to do need to be available for practicum and student teaching placements.

• A supervision, evaluation, and feedback system needs to be in place that provides guided practice in applying and carrying out the tasks teacher work sampling demands of candidates.

1212

Principles…

• Judgments about a candidate’s effectiveness as a teacher need to take into account the gains in learning made by every student taught.

• Documentation of a candidate’s effectiveness as a teacher needs to be accompanied by observations of practice and descriptions of context, as well as evidence of learning gains by students.

• Multiple lines of evidence need to be considered in reaching a recommendation for licensure, only some of which come through teacher work sampling.

1313

Principles…

• Multiple reviewers of evidence need to be involved in preparing a recommendation for a license to teach, only some of whom represent a teacher education faculty.

• Evidence needs to be assembled and reported by a teacher education faculty on the confidence that can be placed in all lines of evidence collected through teacher work sampling that inform a licensing decision (the reliability and validity of information used).

• A conceptual map is needed to help inform and give meaning to candidates regarding the way in which the previous 8 principles inform the TWSM.

1414

1515

Undergirding skills

Candidates prepare products or components of a work sample as evidence of their developing skillfulness. These skills, when employed with acumen, facilitate connections between teaching and learning.

It is connecting teaching and learning that we value… and we judge it by examining the products of the work sample. Therefore we must distinguish between skills and products.

See handout distinguishing between the undergirding skills and the products developed in a teacher work sample.

1616

Weaving a tapestry

When a candidate successfully weaves together the underlying skills of the teacher work sample with real children, in real 21st century schools, we argue that they have maximized opportunities for P-12 student learning.

The teacher work sample stands as evidence of this connecting teaching and learning.

1717

TWS Multi-purposing

Use of TWS results by programs Disaggregated data

Candidates’ ability to teach to state and national Candidates’ ability to teach to state and national standards standards

Candidates’ ability to enact best practices in content Candidates’ ability to enact best practices in content based pedagogy linked to national professional standards based pedagogy linked to national professional standards

Candidates’ ability to impact student learningCandidates’ ability to impact student learning

Aggregated data Program accountabilityProgram accountability Program improvementProgram improvement Context for researchContext for research

See handout on empirical foundations and supports for TWSSee handout on empirical foundations and supports for TWS1818

Optimism about TWSM

Reasonableness: hard to argue that we should not value P-12 student learning

Feasibility: can be used in any context, at all levels, and is practically familiar

Serves multiple purposes: both a pedagogical model as well as an assessment tool

Empirically validated: TEP-II, a longitudinal study of teachers used TWS and explained more than 40% of the variance in student learning*

Schalock, H.D., Schalock, M., & Girod, G. (1997). Teacher work sample methodology as used at Western Oregon State College. In J. Millman (Ed.), Grading teachers, grading schools. Thousand Oaks, CA: Corwin.

*For more information on TEP-II see http://www.tr.wou.edu/tep/products.html

1919

Trustworthiness

Validation of TWSM extended at Idaho State University and Western Kentucky University

• Dependable rating system and rater training• Bias free rating• Validation of frequency, criticality, necessity,

and representativeness of TWS actions to actual practices

• Linked TWS performance to levels of P-12 learning

Denner, P., Norman, A., Salzman, S., Pankratz, R., & Evans, S. (2004). The Renaissance Partnership teacher work sample: Evidence supporting score generalizability, validity, and quality of student learning assessment. In E. Guyton and J. Rainer Dangel (Eds.), Research linking teacher preparation and student performance. Dubuque, Iowa: Kendall.

2020

Systems-level approach

Teacher work sampling as a compass:• Keeps programs focused on P-12 student

learning• Keeps faculty focused on teaching skills,

knowledge, and dispositions that help to maximize P-12 student learning

• Keeps institutions focused on university teaching as a valued outcome

• Codifies a systems-level commitment to “connecting teaching and learning”

2121

Teacher development…

Students responded to the prompt:Did the process of completing a teacher work sample help you

think differently about teaching and learning? If so, how? If not, why not?

Analyzing my teaching using the work sample really helped me understand the importance of making adaptations and modifications as needed. It made me realize the importance of alignment between context, instruction, adaptations, and assessment. I don't think I really understood this before.

2222

Teacher development

And from another student:

The teacher work sample has helped me develop a sense of necessity when it comes to being sensitive and attentive to details – especially the needs and abilities of my students. It helped me recognize the importance of paying attention to pre-assessment results when designing instruction. I can modify instruction to better work with my students' needs.

2323

NCATE’s focus on student learning

A centerpiece of the NCATE performance-based system is collecting and aggregating data to show that candidates have the knowledge and skills to teach effectively so that students learn, a requirement that directly impacts approximately two-thirds of all new teacher graduates nationally.

(Westat, 2006)

2424

TEAC’s focus on student learning

Although less explicit, TEAC also references P-12 student learning as found in Quality Principle 1.3 Teaching Skill which states that teachers must “act on their knowledge in a caring and professional manner that would lead to appropriate levels of achievement for all their pupils.”

NOTE: Less than one year ago, NCATE and TEAC consolidated and teacher preparation accreditation is now under a new organization called Council for the Accreditation of Education Preparation (CAEP)

2525

Art or science?

• Teacher education lacks a unifying theory founded on empirically defensible assertions

• If developing a science of teacher education is possible it includes mapping a complex set of interactions, nested in multiple levels, resting on shifting contextual milieu

2626

TWS well positioned

• Given attention to each stage of Cochran-Smith’s inference chain, and sensitivity to multiple levels of context… teacher work sampling is methodologically well positioned to more systematically explore empirical connections between preparation, practices, and P-12 student learning

• Though VAM is powerful… it is not yet particularly useful for informing the work of teacher education beyond giving us hope that what we do matters!

2727

Future directions

Maturation of TWSM:• Ongoing instrument validation predictive

validity• Dimensionality of undergirding constructs• Codifying scoring procedures• Codifying non-negotiables at multiple levels• Contextualizing methodology• Aggregation of TWS information• Contributing to a scholarship of teaching

education• Linking to data warehouses cross-validation• Etc… 2828

Research Questions…

Descriptive studyDescriptive study::

How does the learning of P-12 students vary, as evidenced by data How does the learning of P-12 students vary, as evidenced by data from the TWS, across:from the TWS, across:

• Kind and complexity of outcomes pursued and metKind and complexity of outcomes pursued and met

• Individuals and groups of students (ethnicity and instructional Individuals and groups of students (ethnicity and instructional program)program)

• Variations in preparation experiences - planned variation studiesVariations in preparation experiences - planned variation studies

• Variations in context, candidate level variables, program level Variations in context, candidate level variables, program level variables…variables…

• Variations in instructional strategies, assessment strategies, and Variations in instructional strategies, assessment strategies, and efforts to differentiateefforts to differentiate

Initial observations…

• More variance within groups than between groups - for More variance within groups than between groups - for ethnicity and academic programethnicity and academic program

• Between program variance illustrates pedagogical Between program variance illustrates pedagogical differences differences notnot program quality program quality

• Need for a web-based, data-entry portal that aggregates Need for a web-based, data-entry portal that aggregates P-12 learning gain dataP-12 learning gain data

• Will require within-program conversations about common Will require within-program conversations about common language and expectationslanguage and expectations

• Will afford programs the ability to link the evidentiary Will afford programs the ability to link the evidentiary chain and become data-driven settings for the chain and become data-driven settings for the improvement of teacher education… at the program levelimprovement of teacher education… at the program level

P-12 student impact

Oregon Collaborative Research Initiative:Through examination of 98 teacher work samples

representing the learning of 2,400 children:• Overwhelmingly, students were focused on low-level,

cognitive outcomes• Only small, non-significant differences for student-level

effects (race/ethnicity, academic program)• Only small, non-significant differences for school-level

effects (SES, urban/rural, size)

3232

Learning Gain Aggregator…

Tension in balancing pedagogical goals with Tension in balancing pedagogical goals with evaluation/research goalsevaluation/research goals

Replace existing efforts don’t add work to students or Replace existing efforts don’t add work to students or facultyfaculty

Getting instructors on boardGetting instructors on board

Using FileMaker Pro for a web-accessible databaseUsing FileMaker Pro for a web-accessible database

Candidate data…

Learning outcome data…

Outcomes by student data…

Reporting… cuts by fields…

And a real example…

And one more real example…

What will we be able to say and do?

• Descriptive studies of the kind and complexity of outcomes Descriptive studies of the kind and complexity of outcomes pursued and met by: pursued and met by: (1) candidates with particular qualities (1) candidates with particular qualities (2) in particular kinds of programs (2) in particular kinds of programs (3) in particular kinds of placement settings (3) in particular kinds of placement settings (4) working with particular kinds of P-12 children (4) working with particular kinds of P-12 children

Ex: What do the learning profiles look like for candidate seeking initial Ex: What do the learning profiles look like for candidate seeking initial licensure in science, working in urban schools, with African American licensure in science, working in urban schools, with African American girls, receiving special education services, when pursuing performance girls, receiving special education services, when pursuing performance outcomes for learning?!!outcomes for learning?!!

• Planned variation studies: Programs become laboratories for Planned variation studies: Programs become laboratories for experimentation with a fixed dependent variableexperimentation with a fixed dependent variable

Challenges and opportunities…and this is the last slide!

• Getting faculty to agree to a common vision of their workGetting faculty to agree to a common vision of their work

• Having faculty leadership to establish proof of conceptHaving faculty leadership to establish proof of concept

• Protecting time and resources to get thereProtecting time and resources to get there

• Gives faculty a defensible outcome variable for all researchGives faculty a defensible outcome variable for all research

• Provides evidence on program effectivenessProvides evidence on program effectiveness

• Moves teacher preparation aheadMoves teacher preparation ahead

Mark Girod – girodm@wou.edu

Western Oregon University