Supporting Implementation of Evidence-Based

Practices in the Early Childhood Classroom

Through Performance-Based Coaching

Patricia Snyder, University of Florida

Lise Fox, University of South Florida

Mary Louise Hemmeter, Vanderbilt University

Coaching

• Create a supportive environment for learning

• Jointly examine and reflect on current practices

• Help to apply new skills

• Within a supportive context, give specific feedback & problem-solve challenging situations

• Goal: Assist person to attain competence and confidence to engage in action, self-reflection, self correction; use new skills and strategies fluently; and use or adapt new skills and strategies in other situations, as appropriate

(Flaherty, 1999; Kinlaw, 1999; Rush & Shelden, 2006)

Performance-Based Coaching

• Practices are made specific and explicit for

teachers and coaches

• Incremental application of practices with

individualized implementation support

(structured around goal setting and action

planning)

• Structured feedback protocol

– Reflective

– Supportive

– Corrective

Coaching for Implementation

ID Focus Evidence-based practice

Practitioner needs in relation to competence

and confidence about practice

Learning about the

“practice or intervention”

Active and passive methods

Multiple exemplars

Concrete, specific

Observe Needs assessment

Set goals and develop action plan

Coach Rapport and collaboration

Active methods

Performance feedback

Observe Needs assessment

Revise goals and action plan

Measure implementation Fidelity of implementation measures

Measure outcomes Practitioner/”setting” measures

Child measures

Supporting Implementation of

Evidence-Based Practices

Evaluating the Potential Efficacy of the Teaching Pyramid Model

Helping early childhood practitioners implement promotion,

prevention, and intervention strategies that support young

children's social-emotional competence and prevent

challenging behavior

Institute of Education Sciences:

R324A07212- Examining the potential efficacy of a classroom-wide model for promoting

social-emotional development and addressing challenging behavior

Helping early childhood

practitioners implement

embedded instruction for

preschool children with

disabilities

Institute of Education Sciences:

R324A070008 - Impact of professional

development on preschool teachers’

use of embedded-instruction practices

Evaluating the Potential Efficacy of Embedded Instruction for Early Learning

Two Potential Efficacy Trials Using

Performance-Based Coaching

Teaching Pyramid intervention = IV

PD to learn about and support

implementation of TP practices

PD provided in TPI condition:

• Workshops (19.5 hr),

implementation guides,

materials, on-site coaching

PD Intervention = IV

Content focus = Embedded

Instruction

PD intervention conditions:

• Workshops (14.9 hr),

implementation guides,

materials, on-site coaching

• Workshops, implementation

guides, materials, web-based

self coaching

• BAU PD

Practices in Each Study

Unpacking Coaching

To make statements about the impact of performance-based coaching, we need

• Coaching framework

• Define components of framework

• Identify phases of coaching

• Coaching protocol for each component

• Defined coaching strategies

• Adherence data

• Dosage or exposure data

• Information about “practice” content of coaching sessions

• Criteria for practice implementation fidelity

Coaching Framework Components

Embedded Instruction for

Early Learning Teaching Pyramid

Phases of Coaching • Early Sessions

– Entry and building rapport

– Observation

– Needs assessment and initial goal setting

– Debrief and Feedback • Reflection

• Supportive feedback only*

• Latter Sessions – Continue building rapport

– Observation

– Goal setting

– Debrief and feedback • Reflection

• Supportive and corrective feedback**

• Final Session – Progress to date and remaining goals

– Strategies for sustaining

– Additional resources

– Evaluation of coach and coaching processes

Protocol for Goal Setting and

Action Planning • “Discrepancy” needs assessment

• Where am I now with respect to practice?

• Where do I want to be?

• What are logical next steps related to

implementation of practice?

• Identify initial goals and develop action

plan

• Recursive process after initial goals and

action plan are implemented

Protocol for Observation

• Observe

• Interact (if agreed to ahead)

• Make Observation Notes

• Videotape*

Protocol for Debrief and Feedback

• Open meeting

• Summarize observation and encourage reflection

• Supportive performance feedback

• Corrective performance feedback

• Targeted support

• Support for planned actions (goal setting, if needed)

• End meeting

Define Coaching Strategies

• Observing

• Problem-solving discussion

• Goal setting/planning

• Reflective conversation

• Role-play

• Performance feedback

(verbal)

• Performance feedback

(graphic)

• Side-by-side gestural

support

• Side-by-side verbal support

• Modeling

• Videotape teacher

• Video demonstration

• Review teacher’s video

• Graphing

• Provide materials or

resources

• Environmental

arrangements

• Other help in classroom

Gathering Adherence, Dose,

Strategy, and Coaching

Content Data

Coaching Logs:

Observation - Early Sessions

Coaching Logs:

Observation – Latter Sessions

Coaching Logs:

Debrief and Feedback Example

Coaching Log:

Record Time and Strategies Used

Coaching

Adherence:

Example from

Fidelity Checklist

Example from Embedded Instruction for Early Learning

Select Findings

Dose: On-Site

Length of coaching = 12-16 weeks

Mean number of sessions = 13

(range 7 to 17)

Mean session duration (min):

Early sessions

• Observation = 144 (R = 85-205)

• Debrief = 44 (R = 10-135)

Sessions 3 and Beyond

• Observation = 105 (R= 30 – 305)

• Debrief = 38 (R = 15- 105)

Length of coaching = 13-16 weeks

No of sessions = 16 (13 sessions

for 1 teacher, all others 16

sessions)

Mean session duration:

Early sessions

• Observation = 75 min (SD = 25)

• Debrief = 40 min (SD = 12)

Latter sessions

• Observation = 72 min (SD = 14)

• Debrief = 39 (SD = 12)

Dose: Self Coaching

Embedded: Procedural Fidelity

Coaching Orientation

(n = 12)

Early

(n = 24)

Latter

(n = 65)

Email

(n = 76)

Final

(n = 12)

All

Sessions

(n = 189)

Coach report

% coaching log

indicators

M (SD)

98.6

(2.1)

96.7

(3.7)

98.1

(2.7)

98.5

(3.7)

100.0 98.2

(3.2)

No. of sessions

with second

observer

4 5 15 25 4 53

Second observer

% coaching log

indicators

M (SD)

100 91.8

(9.2)

95.7

(3.4)

96.3

(4.9)

97.9

(4.2)

96.1

(5)

25

TPOT:

Procedural Fidelity: Coaching

Early Sessions

(n = 40)

Sessions 3 and

Beyond

(n = 219)

Email

(n = 204)

Closing

Sessions

(n = 10)

All Sessions

(n = 473)

Number of

sessions with

fidelity

(percent)

14

(35)

73

(33.33)

78

(38.24)

3

(30)

168

(35.51)

Mean

percentage of

procedural

fidelity

(range)

79.2

(45.45-100)

88.84

(50-100)

88.62

(37.5-100)

89.74

(76.92-100)

86.6

(37.5-100)

26

Embedded :

Coaching Strategies - Observation

Embedded:

Coaching Strategies Debrief

28

TPOT: Strategies used in Early and Latter Coaching Observations

TPOT: Debriefing Strategies used in Early and Latter Coaching Sessions

Select Teacher and Child

Outcome Data

Evaluating the Potential Efficacy of Embedded

Instruction for Early Learning

Teacher Implementation of

What to Teach: Quality of

Learning Targets

Teacher Implementation of How to

Teach: Learning Trials

Rate of Trials

Rate of Complete Trials

Cohen’s d

effect sizes:

On-site

coach vs.

control =

1.41

Self-coach

vs. control =

1.51

Cohen’s d

effect sizes:

On-site

coach vs.

control =

1.35

On-site-

coach vs.

self = 1.23

Cohen’s d

effect sizes:

On-site

coach vs.

control =

2.20

On-site-

coach vs.

self = 1.91

Child Outcome Data

Social Validity Data:

PD Interventions

Limitations and Implications

• Limitations – A priori power analyses based on alpha .20

– Small n potential efficacy trial – but promise

– Standardized and decontextualized child outcome measures

– Metrics used to evaluate “dosage” of self-coaching

– Need to explore mediators of PD-related effects

• Findings and Implications – High-quality workshops and either variant of coaching appear

sufficient for improving quality of learning targets

– On-site coaching using systematic performance-based protocol

improved rate and accuracy of embedded instruction learning

trials when compared to PD intervention with self-coaching or

BAU PD

– Different implementation supports for different components of

embedded instruction

– Social validity data strong, particularly for workshops plus on-

site coaching

Select Teacher and Child

Outcome Data

Evaluating the Potential Efficacy of the Teaching

Pyramid Model

Figure 1. Mean Teaching Pyramid Observation Tool

Total number of TPOT indicators = 108.

Wave 4 [t(40.03)=6.80, p<.001, Cohen’s d=2.6)

0

10

20

30

40

50

60

70

80

Wave 1 Wave 2 Wave 3 Wave 4

Intervention

Control

0

5

10

15

20

25

Wave 1 Wave 2 Wave 3 Wave 4

Intervention

Control

Figure 2. Mean frequency of positive social

interactions during 60 min observation session across

waves for Cohort 1 target children

Teacher Implementation Child Social Interactions

SSIS Child Outcomes

Adjusted Means Cohen’s d

Effect Size Intervention Control

Target Children

Social 88.6 84 .41

Problem Behavior 108.7 115.5 -.52*

Non Target

Children

Social 103.8 96.4 .46*

Problem Behavior 95.2 99 -.29

Implementation Moderates/Mediates Social Skills

• Limitations

– SSIS is teacher report

– No measure of pre-academic skills/school readiness

– TPOT does not measure delivery of intensive

interventions to an individual child

– TPOT does not measure density or “dosage” of

implementation

• Implications

– Effectiveness of the professional development

approach

– Fidelity of implementation of TPOT practices is

related to improved child social and behavioral

outcomes

Limitations and Implications

Implications Related to

Performance-Based Coaching • Specify coaching framework for supporting implementation of

multi-component interventions

• Define components of coaching framework and coaching strategies

• Specify and define practice/intervention that is focus for coaching

• Ensure fidelity of implementation of coaching framework components and associated strategies

• Record and examine dosage and fidelity of implementation of coaching

• Record fidelity of implementation of practice/intervention by practitioner who is coached

Acknowledgments

Embedded Instruction for Early Learning Tools for Teachers (TfT)

Evaluating the Efficacy of the Teaching Pyramid Model: Using the TPOT

• Alana Schnitz, Vanderbilt University

• Kathleen Artman, The Ohio State University

• Kiersten Kinder, Vanderbilt University

• Denise Binder, University of South Florida

• Shelley Clarke, University of South Florida

• Kelly Feeney-Kettler, Vanderbilt University

• Jessie Morris Adams, Vanderbilt University

• Tara McLaughlin, University of Florida

• Alice Kaye Emery, University of Florida

• Crista Scott, University of Washington

• Lindsey Lawrence, University of Washington

• Sara Lubar, University of Wisconsin-Milwaukee