the teacher instructional practices and processes system...

The Teacher Instructional Practices and Processes System:

An Overview and Implementation Guidelines

Edward Seidman Mahjabeen Raza Sharon Kim

Professor of Applied Psychology Project Co-Director TIPPS Project Co-Director TIPPS

Global TIES for Children

Transforming Intervention Effectiveness and Scale

Why Classrooms?



3

School vs. Classroom Settings

Because schools are where children and youth spend

many of their waking hours, they complement families as

a critical loci of learning as well as adaptive and

maladaptive development

Most research indicates that classrooms account for

greater proportion of variance in achievement than

schools

Classrooms are also a key source for children’s psychosocial

development



4

Importance of what goes on inside classrooms

Recent years have witnessed increasing attention to the

proximal interactions youth have inside their classrooms

with adults and other students

It focuses on several different construct domains and

related measurement operations:

Teacher instructional, support and management practices

Via observational measurement

Classroom and norms (e.g. climate and expectations)

Via survey measures

Student-student interactions

Via social networks



5

Ways of gathering observational data on

classroom

Structural/environmental scan

e.g. Are there chairs, blackboards, books, etc.?

Time-on-task

Checklists

Systematic behavioral observations of quality

Generic vs. subject-specific



6

Eliminate Conceptual Overlap

Consolidation of similar concepts and removal of extraneous ones

Examine relevance in low and middle income countries

Intervention Programs

Integration of concepts from education programs geared towards emergency, chronic crisis, and early reconstruction

Cultural Anchoring

Focus groups of teachers & administrators

Adaptation of culturally appropriate language

Principles:

Minimization of subjectivity

Granular stated dimensions for more specific feedback

Use of concrete examples

Crosswalk of Tools

CLASS: Classroom Assessment Scoring System(USA)

FFT: Danielson Framework for Teaching(USA)

IEQ: Improving Education Quality(INTERNATIONAL)



7

Systematic Classroom Observations

Time and frequency measures

Checklists

What they

are not:

Behaviorally anchored indicators tied to core processes and practices in the classroom

What they are:

Fill a gap that exists in understanding what goes on in a classroom

What they do:



8

Tw o - p r o n g e d G o a l o f T I P P S

Rigorous research instrument to study

mediating processes

Feedback device to teachers



9

Extant Methods

of Quality Rating:

Strengths

&

Weaknesses

• Easy to

use

7-point • Complex

• Much

room for

subjectivity

•Too few

points for

quality

• Less

nuance

than 5-

or 7-

point

Strengths Weaknesses

5-point

• End- &

mid-points

clearly

defined

• Nuanced

3-point

4-point• No

middle

default



10

Four point scale (A, B, C, D)

6

2.##Teacher#takes#pupils’#ideas#and#interests#into#consideration#for#class#activities#and#assignments.

Importance+of+Concept:!Teachers!who!are!flexible!towards!incorporating!pupils’!interests,!points8of8view,!and!expressions!into!the!lesson!demonstrate!a!commitment!to!their!learning!and!understanding.!An!effective!teacher!incorporates!pupil!suggestions!and!tailors!the!lesson!them.

Indicators!of!a!teacher#incorporating#pupil#ideas#and#interests!may!include:

● Teacher!encourages!pupils!to!express!their!understanding!of!the!material!they!are!learning

● Teacher!builds!upon!ideas!and!experiences!that!pupils!share!during!a!lesson!through!adapting!lesson!content!or!teaching!practice

● Teacher!places!pupil!learning!needs!at!the!center!of!teaching!by!assigning!activities!geared!toward!their!interests

Category#A#Examples Category#B#Examples Category#C#Examples Category#D#Examples

The!teacher!does!not!incorporate!pupil!interests!or!suggestions!into!the!lesson.!

The!teacher,!at!times,!may!allow!pupils!to!express!interests!or!suggestions!about!the!lesson!but!lesson!may!still!highly!teacher!driven.!

The!teacher!encourages!pupils!to!express!and!discuss!their!interests!or!suggestions,!yet,!incorporation!into!the!lesson!is!only!intermittent.!

Teacher!regularly!allows!for!pupil!expression!and!tailors!the!lesson!in!a!manner!that!incorporates!pupil!interest.!

In!a!lesson!about!farm!animals,!the!teacher!lists!a!variety!of!animals!(cow,!

goat,!chicken,!etc.)!in!a!lecture!format.

In!a!lesson!about!farm!animals,!the!teacher!says,!“Raise!your!hand!if!you!have!a!chicken!at!home.”!!Pupils!raise!their!hands,!but!the!teacher!then!moves!on!with!the!lesson.

In!a!lesson!about!farm!animals,!a!pupil!asks,!“We!have!a!chicken.!Is!that!a!farm!animal?”!!Teacher!responds,!“Yes,!that!is!also!a!farm!animal...”!!The!teacher!then!moves!on!with!the!lecture.

In!a!lesson!about!farm!animals,!a!pupil!asks,!“We!have!a!chicken.!Is!that!a!farm!animal?”!!Teacher!responds,!“Very!good.!That!is!also!a!farm!animal.!And!can!you!tell!us!what!you!could!do!with!a!chicken!on!a!farm?”



12

Innovations of scaling and how we got

there

Observer Sheet Restructuring

• Content did not change much

• Training and visual structure changes enabled

raters to do this much more reliably and

much less subjectively



13

Do not use without authors’ permission

6

2.''Teacher'uses'students’'ideas'and'interests'to'inform'activities'and'assignments.

Importance+of+Concept:+An(effective(teacher(includes(student(suggestions(and(ideas(and(even(adjusts(the(lesson(to(them.(Teachers(who(are(flexible(towards(incorporating(students’(interests,(pointsUofUview,(and(expressions(into(the(lesson(demonstrate(a(commitment(to(their(learning(and(understanding.(The+lesson+is+student@driven+rather+than+teacher@centered.

Indicators(of(a(teacher'incorporating'student'ideas'and'interests(may(include:

● Teacher(encourages(students(to(express(their(understanding(of(the(material(they(are(learning

● Teacher(builds(upon(ideas(and(experiences(that(students(share(during(a(lesson(through(adapting(lesson(content(or(teaching(practice

● Teacher(places(student(learning(needs(at(the(center(of(teaching(by(assigning(activities(geared(toward(their(interests

Column'A Column'B

Teacher'does'not'use'students’'ideas'and'interests'to'inform'for'class'

activities'and'assignments.

Teacher'uses'students’'ideas'and'interests'to'inform'class'activities'and'

assignments.

Very'Accurate Somewhat'Accurate Somewhat'Accurate Very'Accurate

(

The(teacher(does(not(incorporate(student(interests(or(suggestions(into(the(lesson.((

(

The(teacher,(at(times,(may(allow(students(to(express(interests(or(suggestions(about(the(lesson(but(lesson(may(still(highly(teacher(driven.((

(

The(teacher(encourages(students(to(express(and(discuss(their(interests(or(suggestions,(yet(incorporation(into(the(lesson(is(intermittent.(

(

Teacher(regularly(allows(for(student(expression(and(tailors(the(lesson(in(a(manner(that(incorporates(student(interest.(

Teacher(introduces(the(topic(of(mammals(in(a(biology(lesson(by(writing(it(on(the(board.((Teacher(proceeds(to(

lecture(on(mammals(for(the(entire(lesson.((Students(only(speak(when(questioned(directly.(

Teacher(introduces(the(topic(of(mammals(of(mammals(in(a(biology(lesson(by(writing(it(on(the(board.(Teacher(proceeds(to(lecture(on(

mammals,(occasionally(stopping(and(posing(a(few(questions(to(the(students;(sometimes(students(raise(questions(that(teacher(answers(and(moves(along.((((

Teacher(begins(a(lesson(on(mammals(by(asking(students(for(examples(of(mammals(to(write(on(the(board.((The(

teacher(uses(some(student(examples(but(mostly(sticks(with(his/her(own(or(the(book(examples(of(different(types(of(mammals.((

Teacher(begins(a(lesson(on(mammals(by(asking(students(for(examples(of(mammals(to(write(on(the(board.((The(teacher(uses(

student(examples(to(get(students(to(think(about(the(different(types(of(mammals.(

(

( (

Dichotomous Rating Format

(two dichotomous choices)



15

TIPPS: Concepts

Dichotomous Rating

Format

Step One:

A or B

Step Two:Very Accurate or

Somewhat Accurate



16

Closer Look at Deeper Concept

Teacher uses students’ ideas and interests to inform

class activities and assignments.

Importance of Concept: An effective teacher includes student

suggestions and ideas and even adjust the lesson to them. Teachers

who are flexible towards incorporating students’ interests, points-of-

view, and expressions into the lesson demonstrate a commitment

to their learning and understanding. The lesson is student-centered

rather than teacher-driven.



17

Closer Look at Concept Manifestations

Do not use without authors’ permission

6

2.''Teacher'uses'students’'ideas'and'interests'to'inform'activities'and'assignments.

Importance+of+Concept:+An(effective(teacher(includes(student(suggestions(and(ideas(and(even(adjusts(the(lesson(to(them.(Teachers(who(are(flexible(towards(incorporating(students’(interests,(pointsUofUview,(and(expressions(into(the(lesson(demonstrate(a(commitment(to(their(learning(and(understanding.(The+lesson+is+student@driven+rather+than+teacher@centered.

Indicators(of(a(teacher'incorporating'student'ideas'and'interests(may(include:

● Teacher(encourages(students(to(express(their(understanding(of(the(material(they(are(learning

● Teacher(builds(upon(ideas(and(experiences(that(students(share(during(a(lesson(through(adapting(lesson(content(or(teaching(practice

● Teacher(places(student(learning(needs(at(the(center(of(teaching(by(assigning(activities(geared(toward(their(interests

Column'A Column'B

Teacher'does'not'use'students’'ideas'and'interests'to'inform'for'class'activities'and'assignments.

Teacher'uses'students’'ideas'and'interests'to'inform'class'activities'and'

assignments.

Very'Accurate Somewhat'Accurate Somewhat'Accurate Very'Accurate

(

The(teacher(does(not(incorporate(student(interests(or(suggestions(into(the(lesson.((

(

The(teacher,(at(times,(may(allow(students(to(express(interests(or(suggestions(about(the(lesson(but(lesson(may(still(highly(teacher(driven.((

(

The(teacher(encourages(students(to(express(and(discuss(their(interests(or(suggestions,(yet(incorporation(into(the(lesson(is(intermittent.(

(

Teacher(regularly(allows(for(student(expression(and(tailors(the(lesson(in(a(manner(that(incorporates(student(interest.(

Teacher(introduces(the(topic(of(mammals(in(a(biology(lesson(by(writing(it(on(the(board.((Teacher(proceeds(to(

lecture(on(mammals(for(the(entire(lesson.((Students(only(speak(when(questioned(directly.(

Teacher(introduces(the(topic(of(mammals(of(mammals(in(a(biology(lesson(by(writing(it(on(the(board.(Teacher(proceeds(to(lecture(on(

mammals,(occasionally(stopping(and(posing(a(few(questions(to(the(students;(sometimes(students(raise(questions(that(teacher(answers(and(moves(along.((((

Teacher(begins(a(lesson(on(mammals(by(asking(students(for(examples(of(mammals(to(write(on(the(board.((The(

teacher(uses(some(student(examples(but(mostly(sticks(with(his/her(own(or(the(book(examples(of(different(types(of(mammals.((

Teacher(begins(a(lesson(on(mammals(by(asking(students(for(examples(of(mammals(to(write(on(the(board.((The(teacher(uses(

student(examples(to(get(students(to(think(about(the(different(types(of(mammals.(

(

( (

Guiding Rationale for Categorization:

•Examples illustrate the degree

•Examples and indicators ultimately relate back

to the broader concept

•Goal is to observe what is happening and

remove preconceived notions about

classroom



19

Focus groups & manual

review by various local

education stakeholders

But most adaptations

were linguistic

Continuous feedback

from local observers

Real-life examples from

videotaped observations

and observer notes for

concrete, contextually

relevant manual

examples

Cultural Adaptations



The Art of Observation

The Art of Observation



Training to observe, not judge a classroom

21

Cultural (and linguistic) understanding of classrooms they will be

observing

Able to parse through the concurrently occurring processes in the

classroom to rate a particular dimension

Clarity on behavioral manifestations of different dimensions

Able to rate consistently over time (with refresher sessions, of course)

...but we work hardest on one rule



22

In other words, you are a

camera… not Photoshop

Contextual Lens Don’t focus only on the teacher (or only on the students)

Importance of taking notes Focus is strictly on behavior

Providing a “play-by-play” of the observation period

Behaviors come first, the manual comes second



23

Training and Certification to Observe

Calibration Certification

Three tries, three criteria

Practicing on Segments

Large group, paired and individually

TIPPS Training

Dimension-wise understanding, bias awareness

All observers complete mandatory

observation training and must clear

calibration criteria to be able to

conduct observations.

Calibration Criteria:

1. > 60% perfect match with master codes

2. > 90% within-1 match with master codes

3. < 16% dichotomous mismatch with master codes



24

Ugandan Secondary Classrooms

Secondary grades (7th and 8th)

Language of instruction: English

Public schools Rural and Urban

Live observations Classrooms

22 Observers, trained and met criteria to be certified

Calibration Certification

Three tries, three criteria

Practicing on Segments

Large group, paired and individually

TIPPS Training

Dimension-wise understanding, bias awareness

Observed 742

classrooms

Pairs of observers

per classroom

Sub-set of 60

classrooms

videotaped

Currently being

observed in fully

crossed design

(parallel structure to

US study)



25

Are these concepts reliably assessed?

Some items have little or no variance (we eliminated from

further analyses)

Why little or no variance?

Poorly constructed item?

Difficult construct to detect in a classroom

Cultural mismatch?

Clearly missing are item(s) on management of misbehavior

We are working on improving items for inclusiveness and

creating others for management



26

Uganda: Live Inter-rater Reliability at BaselineRaw

Agreement Cohen’s Kappa

Kappa

Weighted

Krippendorf ’s

Alpha

Item Median Median Median Median

1 0.95 0.77 0.93 0.91

2 0.80 0.69 0.83 0.84

3 0.80 0.67 0.73 0.78

4 0.82 0.54 0.67 0.61

5 0.74 0.53 0.73 0.71

6 0.70 0.59 0.78 0.79

7 0.80 0.70 0.74 0.82

8 0.73 0.56 0.72 0.70

9 0.78 0.65 0.75 0.80

10 0.86 0.65 0.75 0.76

11 0.72 0.54 0.67 0.64

12 0.88 0.63 0.65 0.64

13 0.80 0.62 0.74 0.74

14 0.97 0.46 0.53 0.58

15 0.94 0.47 0.47 0.58

18 0.79 0.52 0.55 0.53



27

Inter-rater Reliability Updated

AC1 AC1

Item Baseline Endline

1 0.96 0.97

2 0.74 0.90

3 0.77 0.87

4 0.78 0.85

5 0.77 0.86

6 0.75 0.82

7 0.76 0.90

8 0.72 0.84

9 0.76 0.89

10 0.79 0.86

11 0.73 0.81

12 0.90 0.93

13 0.76 0.86

14 0.95 0.99

15 0.92 0.98

18 0.80 0.87



28

What the data is telling us so far

• Use of student ideas/interests & Making connections to

everyday life experiences

• These are two related, low-performing items for the Ugandan

teachers. Confirmed by ministry that these are areas in which

teachers struggle in general in Uganda. Not much to address

these topics in the teacher training curriculum.

• Use of instructional materials

• Conversely, this was a moderately well-performing area for

most of the teachers in our sample. Confirmed that this made

sense given teacher training curriculums place much emphasis

on the use of materials.

• Specific feedback

• Also an area that is lacking, as observed by ministry.



31

ECD vs. Primary vs. Secondary Versions

Developmentally appropriate classroom processes and practices:

Teacher explaining how to work together

Teacher helping them think aloud their reasoning

Important role of Free Play

Gender and diverse learner responsiveness

Developmentally appropriate descriptions of behavior and examples:

5 year olds do engage in reasoning, it just looks different from 15 year olds

Misbehavior managed differently with 4 year olds vs 14 year olds

Safety and hygiene need more emphasis for young children



32

Simply a research tool?

Feedback as an intervention

(Where we hope to go)

• Direct feedback to teachers… How?

• Profiles – see Peter’s talk

• Via technology

• Pakistan

• Feedback in context of supportive peer group

• Teacher Learning Circles

• DRC

• Feedback in conjunction with a coach/mentor

• Uganda

the teacher instructional practices and processes system...

Documents