· web viewshort activities in the course that explore what participants already know about...

An Evaluation of the Massachusetts Pilot Teachers’ SEI Endorsement Course

Erin Haynes, American Institutes for Research

Diana August, American Institutes for Research

Christine Paulsen, Concord Evaluation Group

DECEMBER 2012

An Evaluation of the Massachusetts Pilot Teachers’ SEI Endorsement Course

December 19, 2012

Erin HaynesResearcher, American Institutes for Research

Diane AugustManaging Researcher, American Institutes for Research

Christine PaulsenPrincipal Research Scientist, Concord Evaluation Group

1000 Thomas Jefferson StreetWashington, DC 20007Phone: (202) 403-5000 Fax: (202) 403-5001

Copyright © 2012 American Institutes for Research. All rights reserved.3280_12/19

Contents

Executive Summary.........................................................................................................................1Description of Evaluation............................................................................................................1Research Questions......................................................................................................................1Recommendations........................................................................................................................2

Introduction......................................................................................................................................8Research Questions........................................................................................................................10

Quality of Delivery....................................................................................................................10Fidelity of Implementation........................................................................................................10Coherence..................................................................................................................................10Teacher Outcomes.....................................................................................................................10

Research Context...........................................................................................................................11Data Collection Methods...............................................................................................................13

Activity 1: Course Observations................................................................................................13Activity 2: Instructor Interviews................................................................................................14Activity 3: Teacher Surveys......................................................................................................14

Findings and Recommendations....................................................................................................16Quality of Delivery....................................................................................................................16Fidelity of Implementation........................................................................................................30Coherence..................................................................................................................................31Teacher Outcomes.....................................................................................................................34

Next Steps......................................................................................................................................38References......................................................................................................................................39Appendix A: Data Collection Instruments....................................................................................43

Face-to-Face Session Observation Protocol..............................................................................43Online Session Observation Protocol........................................................................................51Instructor Interview Protocol.....................................................................................................57Participant Focus Group Protocol..............................................................................................60Participant Survey......................................................................................................................61

Appendix B: Participant Survey Results.......................................................................................72

Tables

Table 1. RETELL Pilot Sessions...................................................................................................12

Table 2. Research Questions Addressed by Each Data Collection Activity.................................13

Table 3. RETELL Pilot Session Observations..............................................................................14

Table 4. Usefulness Ratings for Readings Assigned in the RETELL Course...............................18

Table 5. Topics for Which RETELL Participants Reported Greater Understanding...................35

Table 6. Topics in Which RETELL Participants Need Additional Information..........................35

Figures Figure 1. RETELL Course Participants’ Responses to Survey Questions About Course Readings........................................................................................................................................17

Figure 2. RETELL Course Participants’ Responses to Survey Questions About Instructor Quality...........................................................................................................................................21

Figure 3. RETELL Course Participants’ Responses to Survey Questions About Online Sessions..........................................................................................................................................22

Figure 4. RETELL Course Participants’ Responses to Survey Question About Course Pace.....25

Figure 5. Average Time Spent on Activities During RETELL Course Face-to-Face Sessions....26

Figure 6. RETELL Course Participants’ Responses to Survey Questions About Time Allocation......................................................................................................................................27

Figure 7. RETELL Course Participants’ Response to Survey Question About Course Instructions....................................................................................................................................28

Figure 8. RETELL Course Participants’ Response to Survey Question About Time Allocated to Discussions of Classroom Strategy Application.......................................................33

Figure 9. RETELL Course Participants’ Response to Survey Question About Usefulness of Online Illustrations of Strategy Application..................................................................................34

Executive Summary

Description of Evaluation

In order to improve instruction for English language learners (ELLs) in Massachusetts, the Massachusetts Department of Elementary and Secondary Education (ESE) has introduced the Rethinking Equity and Teaching of English Language Learners (RETELL) initiative. Under this initiative, all core academic teachers in the state will be required to complete comprehensive professional development (PD) in sheltered English instruction (SEI) methods by July 1, 2016. For simplicity in this report, this PD will be referred to as the RETELL course pilot, although the course will be called the Teacher SEI Endorsement Course in the future.

Implementation of the full program will begin across the state in February 2013. In order to ensure that this program is ready for complete rollout by February, a pilot of the PD was implemented during fall 2012, ending in November. The American Institutes for Research (AIR), in collaboration with Concord Evaluation Group (CEG), has completed an evaluation of the pilot to provide feedback on the quality of the RETELL PD and RETELL instructors’ fidelity to course materials in time to make any necessary changes to the program before the February rollout. A preliminary report of the findings of the evaluation was submitted to ESE in November. This document constitutes a full report of the RETELL pilot evaluation.

Research Questions

The RETELL pilot evaluation report responds to nine research questions, grouped into four critical categories for success of the PD program: quality of delivery, fidelity of implementation, coherence, and teacher outcomes. The research questions are as follows.

Quality of Delivery

1. What is the overall quality of delivery of the online and face-to-face courses, in terms of content, process, and context, as well as pace, allocation of time, and clarity of PD instruction? How can the overall quality of delivery of the online and face-to-face courses be improved?

2. In what ways could the support for RETELL course instructors be improved to enhance their instruction?

3. What ongoing feedback loops can be established to ensure the quality of the RETELL instruction and course delivery?

Fidelity of Implementation

4. What is the level of implementation of RETELL components during the online and face-to-face courses? How can the level of implementation of RETELL components during the online and face-to-face courses be improved?

5. What ongoing feedback loops can be established to ensure the fidelity of the RETELL instruction and course delivery?

American Institutes for Research MA SEI Endorsement Pilot Evaluation—1

Coherence

6. What is the level of coherence of the RETELL sessions?

7. What is the level of coherence of the course with what teachers experience in their educational contexts?

Teacher Outcomes

8. How could the RETELL course better contribute to key understandings of teachers of ELL students in regular classroom settings, in terms of both the content of the course and the course delivery?

9. What additional information do teachers need to be successful in the RETELL course?

Recommendations

The following is the complete list of recommendations made in this report in each of the four identified critical areas: quality of delivery, fidelity of implementation, coherence, and teacher outcomes.

Quality of Delivery

Quality of Content

Goals, objectives, and expected outcomes appear in the course syllabi and presentation slides. Participants seem capable of assimilating the information from course materials. Therefore, no change is recommended.

No changes are recommended to the topics covered in the course syllabi. If changes are made to the syllabi for other purposes (e.g., to ensure that participants have more opportunities to engage with specific strategies), make sure that key areas of ELL instruction (see description and sources in full report) are retained.

Teacher buy-in is essential in order to motivate teachers to adopt practice, especially when the PD does not already align with the beliefs of the teacher (Kubitskey & Fishman, 2005). Because some recipients of the RETELL PD may not be convinced that they can successfully impart content to students who are not proficient in English, consider adding a case study or video to the first session of the course that illustrates how sheltering instruction can help ELLs access content at the same time as they acquire English language skills.

To reduce printing burdens for teachers, provide a CD or printed copies of the required readings.

Reduce the reading load by carefully selecting readings that are the most relevant to the successful implementation of SEI and that are clearly written. Consider readings that provide practical advice about helping ELLs meet high content standards.

Quality of Process

The pilot instructors expressed knowledge of the course participants’ backgrounds. This knowledge is important to the quality of process, and instructors of the main course should learn about their participants’ backgrounds and contexts. Consider including some short activities in the course that explore what participants already know about various


topics and what they would like to learn. Provide sufficient flexibility for instructors to be responsive to participants’ needs.

Identify instructional strategies that are aligned with each course topic. Have PD instructors model these strategies using elements of model lessons and give participants an opportunity to apply what they have learned in a new context. Consider providing participants with a rubric to record each strategy they use, with information about the classroom context, how the strategy worked, and recommendations for adjustments.

Ask participants to briefly summarize what they learned at the end of each session, either as a whole class, in small groups or partners, or as a warm-up during a subsequent online session. As part of their summary, ask them to link the course materials to specific experiences or examples from their own educational contexts, or describe methods they have used related to the session.

Pilot instructors appeared to be skilled at leading collaborative discussions and engaging participants. Ensure that instructors in future RETELL courses are prepared to use good pedagogical practice to facilitate collaboration and encourage participation.

Consider developing professional learning communities (PLCs) as part of the RETELL course so that collaboration among participants can extend beyond the duration of the course. A review of relevant research on PLCs by Vescio et al. (2008) finds that well-developed PLCs have positive effects for both teacher practice and student outcomes.

Consider methods for increasing discussions and interactions online. Some options include the following:

o Establish weekly “office hours” during which the instructor is online and available for chatting (consider making attendance at some office hours a course requirement).

o Create a blog or wiki in which participants can report on the implementation of strategies or share ideas for teaching that came to them as a consequence of a reading or session.

o Group participants and provide tasks that require them to check in with each other regularly, either in person or via the online platform.

o Set up a collaborative project online to which all participants are required to contribute. Ideally, such a project would be best facilitated by the online platform (such as posting videos of instruction and eliciting constructive feedback or conducting an analysis of student work using graphic tools; see Little et al., 2003).

Encourage instructors to actively monitor online activity and send reminders and prompts to participants as needed.

Quality of Context

It will be essential to select highly qualified instructors for a course that is not heavily scripted. Unless activities are added and scripted, only RETELL instructors who are prepared to develop and incorporate their own activities should be selected.

Consider developing and providing answer guides for questions posed in the course materials.


Pace of Course

Carefully monitor the semester-long course to make sure that the pace is not too fast. For example, the schedule should allow reasonable time between sessions for participants to complete readings and assignments, and to test the methods and instructional strategies they are learning.

Allocation of Time in Course

Because participants found course time allocations appropriate for lectures and in-class interactions with colleagues (including small-group work and whole-class discussion), maintain these activities at their current levels.

In order to address instructor concerns and participant feelings that not enough time was devoted to strategies, consider replacing some of the small-group discussions with small-group strategy practice; whole-class discussions also could be used to discuss how theory is realized in practice and how to apply instructional strategies successfully.

Replace significant out-of-class assignments with activities that require collaboration with colleagues, particularly on the practice of strategies. Be sure to incorporate enough time between assignments to allow participants to engage in these activities meaningfully.

To address time allocation during the self-paced, self-directed online sessions, consider creating an activity that helps participants understand how they learn best. Provide tips for time management as part of the introduction to the online sessions.

Clarity of Instruction

Consider including some short activities in both the face-to-face and the online portions of the course that explore participants’ conceptions and misconceptions to ensure that they are addressed.

For the online sessions, provide a single list of required readings, materials, and activities for each session with hyperlinks to each.

Include summaries of readings and information about possible questions or concerns in online materials.

Provide a list of assignments and readings ahead of the course so that participants have additional time to complete the work.

Support for Course Instructors

Make sure that course classrooms are easily accessible, can be accessed ahead of each session for setup, and have Internet access. If possible, designate a person at each site to ensure that these logistics are taken care of.

Provide participant contact information to instructors as early as possible (ideally, at least two weeks before the first session).

Establish a clear policy about material distribution and provide it to all instructors. Include information about which materials and services ESE provides and which it does not.


Establishment of Feedback Loops

Provide information to instructors about which aspects of the RETELL course will need to be consistent across the state, such as participant grading or evaluation information.

Consider providing a training session for RETELL course instructors during which they can practice elements of the PD with each other before delivering it to course participants.

Establish a consistent feedback system for all RETELL PD courses. The feedback system might consist of an observation by an ESE official, formal participant surveys (e.g., short surveys delivered via Survey Monkey after each class), or an outside evaluation of teacher products (e.g., lesson plans) or outcomes.


Strategically group participants with similar backgrounds and educational contexts into the same courses so that instructors can focus on their needs efficiently, increasing the likelihood of covering all course content. As an added advantage, grouping participants from similar educational contexts can facilitate the formation of PLCs (see, e.g., Newmann & Associates, 1996).

Check online sessions to ensure that they cover all topics and assignments in the syllabus, especially if there have been changes to the sessions.

Provide instructors with information about how long each activity should take.

Facilitate regular meetings among instructors (by region and/or participant type—e.g., secondary STEM) throughout the course semester to discuss RETELL delivery and fidelity to the syllabus. These meetings would also be a good opportunity for ESE to collect feedback from instructors about recommended changes.

Make necessary changes to the course between, not during, rollout semesters.

Coherence

Level of Coherence of the RETELL Sessions

Revise course materials to provide explicit connections between topics covered in each session, including revising the online materials to include explicit connections to previous and upcoming sessions.

Explain to instructors the importance of communicating expectations for the online course during the face-to-face sessions, even if there is only a little time to do so.

Consider implementing the following instructor-recommended changes:

o Incorporate session 4 into session 5.

o Consolidate information about academic language into a single session.

o Incorporate information about the Common Core State Standards throughout the sessions rather than introducing it in session 15.

o The topic of assessment should come at the end of the course, after reading and writing have been covered.


o Provide more grade-level differentiation in Module C [within the secondary courses].

Level of Coherence With What Teachers Experience in Their Educational Contexts

Connections to participants’ educational contexts are crucial. Ensure that RETELL course instructors are as prepared as the pilot instructors to connect material to real-world contexts, use stories to link theory to practice, and lead participants in discussions of how strategies would work in their educational contexts.

Consider incorporating the practice of instructional strategies into small-group work and whole-class discussions in order to increase its impact.

Update the online materials to make better connections to participants’ educational contexts. In each session, include

o Stories or videos that model practical strategies

o Discussions of possible challenges that teachers might face in applying strategies

o Prompts that encourage participants to post and discuss examples of how strategies would work in their educational contexts

Teacher Outcomes

Conduct a brief survey at the outset of each course to determine areas in which participants are already well versed and areas where they need additional instruction.

Consider providing (additional) instruction in the following areas, as determined by participant needs:

o Creating lessons and practical strategies for the classroom

o Effective practice with ELLs who do not speak Spanish

o Standardized assessments and ELLs

o ELLs with special needs

o Grade-specific strategies

o Supporting language development versus helping students access content

o Students with interrupted formal education (SIFE)

o Using data

Fishman et al. (2003) propose an iterative PD research design model, in which feedback about teacher outcomes is incorporated into future PD sessions; a similar model has been proposed by Kubitskey & Fishman (2007). The current evaluation gathers self-reported data from teachers, but it also is important to measure teachers’ skills and knowledge. The following formative and summative evaluation activities are recommended both to provide teachers with information about their progress in the course and to improve future iterations of the course:

o Develop and incorporate protocols for assessing participants’ plans for and reports of how they implemented course strategies.

o Develop and conduct pre- and post-course assessments of participants’ knowledge about ELLs and ELL instruction.


o Include a classroom observation component in the course. Teachers might observe each other and provide peer-to-peer feedback; a school or district ESL coordinator or coach might observe participants and provide feedback as part of the RETELL course; or RETELL course instructors might observe each participant teaching. Observations are recommended for teachers in high- and medium-incidence schools; they are not recommended if participants do not have ELLs in their classrooms at the time of the RETELL PD.

Make sure that instructors can provide timely feedback to participants using the Moodle online platform. For instructors who are unfamiliar with Moodle, offer training on how to use it effectively.


[This page intentionally left blank.]


IntroductionEnglish language learner (ELL) enrollment in PK–12 grew nationwide by more than 51 percent between the 1998–99 school year and the 2008–09 school year, while the growth of total student enrollment increased by only 7 percent (NCELA, 2011). This growth trend is reflected in Massachusetts, where the proportion of ELLs in public schools has grown 57 percent since 2000, while enrollment of English-proficient students has remained steady; by 2021, it is estimated that 20 percent of all Massachusetts students will be ELLs (Massachusetts Department of Elementary and Secondary Education [ESE], 2009; ESE, 2012). Furthermore, the number of districts in Massachusetts enrolling at least one ELL has nearly doubled between 2000 and 2011; in 2011, 324 of the state’s 400 districts had at least one ELL, and 63 districts had 100 or more ELLs (ESE, 2012). In light of this growth, all Massachusetts educators can expect to work with ELLs in their careers.

However, there is a significant achievement gap between ELLs and their English-proficient peers in the state. Results from the Massachusetts Comprehensive Assessment System (MCAS) for English language arts and mathematics in Grades 4, 7, 8, and 10 reveal that in both subject areas at all grade levels, the proportion of ELLs scoring at Warning (the lowest rating for Grades 4‒8) or Failing (the lowest rating for Grade 10) was significantly higher than the proportion of English-proficient students receiving those ratings (Owens, 2010). Further, ELLs in Massachusetts are more likely than English-proficient students to repeat a grade and were 25 percent more likely to be suspended, according to 2008 data, up from 16 percent more likely in 2006 (Owens, 2010).

In order to improve instruction for ELLs in Massachusetts and ultimately improve their achievement, ESE has introduced the Rethinking Equity and Teaching of English Language Learners (RETELL) initiative. Under this initiative, all core academic teachers in the state will be required to complete comprehensive professional development (PD) in sheltered English instruction (SEI) methods by July 1, 2016.1 This PD will incorporate 16 online and face-to-face sessions during the course of an academic semester. The PD’s main purpose is to prepare teachers to shelter their instruction by increasing their knowledge of ELL demographics and cultural backgrounds, second-language acquisition theory, literacy, English language development standards and assessments, and effective practices in ELL instruction.

Implementation of the full program will begin across the state in February 2013. In order to ensure that this program is ready for complete rollout by February, a pilot of the PD was implemented at five sites for 8.5–13.5 weeks during fall 2012 (the five sites were Springfield, Boston, Worcester, Somerville, and Lowell). The pilot was completed at all five sites by the end of November. The American Institutes for Research (AIR), in collaboration with Concord Evaluation Group (CEG), completed an evaluation of the pilot at four of the sites (Boston, Worcester, Somerville, and Lowell).2 The purpose of the evaluation was to provide feedback on the quality of the RETELL PD and RETELL instructors’ fidelity to course materials in time to make any necessary changes to the program before the February rollout. A preliminary report of the findings of the evaluation was submitted to ESE in November. This document constitutes a full report of the RETELL pilot evaluation.

1 For simplicity in this report, this PD will be referred to as the RETELL course pilot, although the course will be called the Teacher SEI Endorsement Course in the future.2 Note that the Springfield site PD is not evaluated as part of this report because it received its PD earlier than the other sites, and because it served as a test case for the pilot PD materials. One course observation was conducted in Springfield, but the results of that observation were provided to ESE separately and are not included here.


Research Questions

The RETELL pilot evaluation report responds to nine research questions, grouped into four critical categories for success of the PD program: quality of delivery, fidelity of implementation, coherence, and teacher outcomes. These categories will be described in greater detail in the Findings and Recommendations section. The research questions are provided here:

Quality of Delivery

1. What is the overall quality of delivery of the online and face-to-face courses, in terms of content, process, and context, as well as pace, allocation of time, and clarity of PD instruction? How can the overall quality of delivery of the online and face-to-face courses be improved?

2. In what ways could the support for RETELL course instructors be improved to enhance their instruction?

3. What ongoing feedback loops can be established to ensure the quality of the RETELL instruction and course delivery?


4. What is the level of implementation of RETELL components during the online and face-to-face courses? How can the level of implementation of RETELL components during the online and face-to-face courses be improved?

5. What ongoing feedback loops can be established to ensure the fidelity of the RETELL instruction and course delivery?

Coherence

6. What is the level of coherence of the RETELL sessions?

7. What is the level of coherence of the course with what teachers experience in their educational contexts?

Teacher Outcomes

8. How could the RETELL course better contribute to key understandings of teachers of ELL students in regular classroom settings, in terms of both the content of the course and the course delivery?

9. What additional information do teachers need to be successful in the RETELL course?


Research Context

Data to address the nine research questions were collected in four pilot districts: Boston, Worcester, Somerville, and Lowell. Seven instructors provided PD at these sites; each site had two instructors, but one instructor taught at two of the sites (Boston and Worcester). These instructors were selected for their expertise in English as a second language (ESL) and experience delivering PD to educators in Massachusetts. Three of the instructors reported delivering the SEI category trainings, which were the previous PD effort aimed at preparing teachers to work with ELLs. All the instructors had at least a master’s degree in ESL or a related field, and all had provided PD to teachers in Massachusetts districts.

Instructors spent one week with the developers of the RETELL course curriculum in the summer before the pilot implementation to review the course content. This week was structured as a working group, during which the instructors gave input and feedback about the course. One instructor for the four evaluation sites also taught the course at Springfield, beginning two months prior to the other sites. She made some changes to the course materials on the basis of her experiences, in consultation with ESE. Some of these changes were provided to the other instructors during the delivery period at the other sites. The RETELL pilot course was not scripted, so all the instructors had latitude in its delivery, and they all provided their own activities. One instructor said, “We were instructed to just go ahead of them [the trainees] and pave the sidewalk as they walked on it.”

The instructors delivered 16 sessions, divided among three modules. The first module develops participants’ knowledge of ELLs, including ELL demographics, the diversity within populations of ELLs, and their educational context. It emphasizes the importance of integrating students’ background knowledge and skills into instruction, while building the foundational skills they need to succeed in a U.S. academic setting. The second module introduces the theoretical principles of second-language acquisition and their impact on the instruction of second-language learners. The third module builds on the second module principles with a focus on teaching ELLs academic language and enabling them to master content. This module illustrates how curriculum can be scaffolded for ELLs at different levels of English-language proficiency and for different content areas.

Ten of the courses were delivered face-to-face, including the first and last sessions, and six of the sessions were delivered online via the Moodle platform. All face-to-face sessions were three hours long, except the first and last sessions, which were two hours each. (Table 1 provides information about each session’s delivery type and length, by module.) PD participants were expected to complete two or three readings for each session, including the online sessions. They also were asked to complete several assignments, which fell into four categories: journal writing (nine entries); critical analyses of challenges in teaching ELLs (five analyses); adaptations of lesson plans for ELLs (11 lesson plans); and short formal papers (three papers). The course was delivered over 8.5–10 weeks at each site (it was delivered over 13.5 weeks in Springfield, but that site is not included in the evaluation).


Table 1. RETELL Pilot Sessions

Module Session Type Length

A1 Face-to-face 2 hours2 Online varies3 Face-to-face 3 hours

B4 Online varies5 Face-to-face 3 hours

C

6 Face-to-face 3 hours7 Online varies8 Online varies9 Face-to-face 3 hours10 Online varies11 Face-to-face 3 hours12 Online varies13 Face-to-face 3 hours14 Face-to-face 3 hours15 Face-to-face 3 hours16 Face-to-face 2 hours

Participants were content area teachers in the respective districts. The Boston course was intended for elementary teachers, Worcester and Somerville were for secondary English language arts/humanities teachers, and the Lowell course was for secondary STEM teachers. A total of 103 participants were enrolled in the four sites (25 in Boston, 28 in Lowell, 24 in Somerville, and 26 in Worcester), but participation dwindled at all four sites during the course.


Data Collection Methods

Research activities to address the nine research questions included course observations, interviews with instructors, and participant surveys. Each of these research activities provided information for a subset of the research questions, as presented in Table 2. More detailed descriptions of the activities follow.

Table 2. Research Questions Addressed by Each Data Collection Activity

Data Collection Activity

Research Questions Addressed

Course observations 1, 4, 6Instructor interviews 2, 3, 5Participant surveys 1, 7, 8, 9

Activity 1: Course Observations

Fourteen course observations were completed for this evaluation. Nine were completed during face-to-face sessions, and five during online sessions. The sites, sessions, and types are presented in Table 3. Sessions were selected for observation across the four sites with the goal of maximizing the variety of sessions, session types (online or face-to-face), and participant type (elementary teachers, secondary English language arts/humanities teachers, or secondary STEM teachers); as a secondary consideration, sessions were selected that did not have an observer from ESE in order to make sure that all pilot sessions were observed. Sessions earlier in the sequence occurred before the evaluation period and could not be observed. Likewise, Session 16 could not be observed at any of the sites because observations had to be completed by November 16 in order to finalize analyses and provide timely feedback to ESE. The final online and face-to-face observation protocols are in Appendix A.


Table 3. RETELL Pilot Session Observations

Site Date Session Type TeachersBoston October 13 5 Face-to-face Elementary

October 13 6 Face-to-face ElementaryOctober 19 8 Online ElementaryNovember 3 13 Face-to-face ElementaryNovember 3 14 Face-to-face Elementary

Worcester October 9 6 Face-to-face ELA/humanitiesOctober 16 9 Face-to-face ELA/humanities

Lowell November 15 2 Online STEMOctober 12 7 Online STEMNovember 7 14 Face-to-face STEMNovember 14 15 Face-to-face STEM

Somerville October 23 10 Online ELA/humanitiesOctober 26 11 Face-to-face ELA/humanitiesOctober 30 12 Online ELA/humanities

Activity 2: Instructor Interviews

Interviews were completed by phone with six of the seven RETELL pilot instructors, including the two co-instructors at the Lowell site, the two co-instructors at the Somerville site, the primary instructor at the Boston and Worcester sites (the same person), and the co-instructor at the Worcester site. Interviews were designed to elicit the instructors’ perspectives about the structure and quality of content of the RETELL PD, information about the support they needed and the feedback they received, and their perspectives on the outcomes of the PD for participants. The instructor interview protocol is in Appendix A.

Activity 3: Teacher Surveys

A survey was adapted from a midcourse survey that had been designed and administered by ESE. During the first week of the evaluation period, volunteers from the Springfield PD site were asked to form a focus group to discuss the survey questions. A total of five participants were asked about each survey question in order to ensure that the questions were valid measures of the constructs (quality of delivery, course coherence, and teacher outcomes).3 The survey was revised to reflect their comments before it was administered to all sites. The focus group protocol and final survey are included in Appendix A.

Participants at all four evaluation sites were asked to complete the survey during the week of November 12, during their second-to-last session. A total of 57 participants completed the survey, including 18 participants from Somerville, 16 participants from Lowell, and a combined total of 23 participants from Boston and Worcester (these two sites had the same instructor, who

3 Participants from the Springfield site formed the focus group because they had already completed most of the training early in the evaluation period.


combined the completed surveys, so no differentiation is possible). Compiled survey results are in Appendix B.


Findings and Recommendations

This section presents the findings of the evaluation and provides recommendations to improve the RETELL course. The findings and recommendations are grouped by category: quality of delivery, fidelity of implementation, coherence, and teacher outcomes.

Quality of Delivery

Quality of delivery, examined through research questions 1, 2, and 3, encompasses several factors, including content, process, context, pace, allocation of time, and clarity of PD instruction. The first three factors are described in the draft Massachusetts Standards for Professional Development (SPD). Also included in the analysis of quality of delivery are ways in which the support for RETELL course instructors could be improved to enhance their PD delivery and feedback loops that could be established to ensure the quality of the PD instruction and course delivery.

Quality of Content

Quality of content is examined through research question 1. According to the Massachusetts SPD, high-quality PD has clear goals and objectives that are relevant to participant outcomes and aligned with state priorities. RETELL course goals and objectives were included in the syllabi and presentation slides, so despite the fact that instructors at all four sites were rarely observed articulating goals, objectives, or expected outcomes, a large majority (86 percent) of participants at all four sites indicated that instructors always or frequently made this information available, presumably as part of course materials.

It is beyond the scope of this evaluation to examine the alignment of the RETELL course goals with state priorities. An analysis of the course syllabi reveals, however, that the RETELL course materials cover critical aspects of ELL instruction, and they address key areas of concern for Massachusetts educators (such as serving subgroups of ELLs), as reported in Perez-Selles, Cazabon, & Mello (2011). For example, the first module develops participants’ knowledge of ELLs, including ELL demographics, the diversity within populations of ELLs, and their educational context. It emphasizes the importance of integrating students’ background knowledge and skills into instruction, while building the foundational skills they need to succeed in a U.S. academic setting (cf. Lee, Maerten-Rivera, Penfield, LeRoy, & Secada, 2008). The second module introduces the theoretical principles of second-language acquisition and their impact on the instruction of second-language learners. The third module builds on the second module principles with a focus on teaching ELLs academic language and enabling them to master content as specified in the Common Core State Standards (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010). The third module illustrates how curriculum can be scaffolded for ELLs at different levels of English language proficiency and for different content areas (cf. Saunders, 1999; Saunders & Goldenberg, 1999). The last module is of particular importance because, according to the syllabi, it emphasizes strategies that have been found to be effective with ELLs in high-quality research studies, such as providing multicomponent linguistic support for ELLs, including explicit vocabulary instruction, peer assistance, and the use of visual support (Carlo et al., 2004; Franken & Haslett, 1999; Vaughn et al., 2009; see also sources cited in Gersten et al., 2007), as well as providing differentiated instruction to capitalize on ELLs’ strengths (cf. August & Shanahan, 2010).


The RETELL course instructors were generally pleased with the scope of the curriculum and the division into three modules. One instructor, however, expressed concern over course buy-in; she did not feel that the course sufficiently established why SEI is important for all participants. Overall, participants in the PD were not observed disputing the factual material included in the course or explicitly questioning the usefulness of the course. They were able to complete activities correctly (though online activities often were completed late). It was sometimes unclear, however, whether participants had completed the readings. Though they sometimes referred to readings during face-to-face discussions or online comments, they almost never asked or posted clarifying questions about the main ideas of the reading selections. In all sessions, participants were observed complaining that the reading load was too high and that they did not have money to print or make copies of the articles. Forty-nine percent of the participants who completed surveys reported that they did not find the readings easy to understand. In addition, some of the instructors expressed the opinion that the focus on the reading and theory was very heavy, to the detriment of other activities like modeling strategies and experiential activities for the participants.

Nonetheless, in surveys almost all participants agreed (77 percent) or strongly agreed (14 percent) that the readings were related to course content, and 80 percent agreed or strongly agreed that the readings helped them better understand SEI. The majority also agreed (67 percent) or strongly agreed (6 percent) that the readings were applicable to instruction of ELLs. See Figure 1.

Figure 1. RETELL Course Participants’ Responses to Survey Questions About Course Readings

Strongly disagree Disagree Agree Strongly agree0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

5% 4%

77%

14%9% 11%

67%

13%

2%

26%

67%

6%

Readings related to course content Readings led to better understanding of SEIReadings were applicable to instruction of ELLs


This apparent disparity in participants’ reactions to the readings (on the one hand they found the readings difficult and thought there were too many, but on the other hand they found the readings useful) is likely due to the diversity of articles that they were asked to read. In order to explore this issue further, participants were asked to rate individual course readings as either problematic or useful. In recognition of the fact that they may not recall or have a reaction to some readings, participants were asked to rate only the readings they found problematic or useful. A score was assigned to each reading by subtracting the number of problematic ratings from the number of useful ratings, presented as a fraction of the total number of readings. The score for each reading is presented in Table 4, also provided as a percentage. A percentage above 50 percent indicates that if a reading elicited a response, the response was more heavily positive. A percentage below 50 percent, on the other hand, indicates that if a reading elicited a response, the response was more heavily negative.4

Table 4. Usefulness Ratings for Readings Assigned in the RETELL Course

Session Reading Useful/Total %

3 Harper & de Jong (2004) 33/39 85%2 DeCapua & Marshall (2011) 29/37 78%3 Colorin Colorado (2007) 27/37 73%2 Robertson & Lafond (2008) 23/33 70%7 Silbold (2011) 26/38 68%1 ESE (2012) 21/31 68%6 Calderon et al. (2005) 24/38 63%

14 DelliCarpini (2009) 20/32 63%8 de Jong & Harper (2005) 20/32 63%

12 Gottlieb (2006) 23/37 62%11 Misco & Castaneda (2008) 16/26 62%3 Dilg (2003) 20/34 59%

12 Abedi (2002) 19/33 58%15 WIDA (2012) 16/30 53%9 Himmel (2012) 15/31 48%

15 Echevarria et al. (2004) 14/30 47%14 Brisk et al. (2007) 14/32 44%10 Coleman & Goldenberg (2010) 12/30 40%7 Carlo et al. (2005) 12/30 40%8 Echevarria & Graves (2011) 12/30 40%5 Peregoy & Boyle (2008) 13/33 39%9 Varela (2010) 9/25 36%

4 No interpretation is available for the number of responses that a reading elicited. Participants may have failed to rate readings for a number of reasons, including that the reason was not memorable, that they did not read it, or that they read it, but found it neither useful nor problematic. These options were not included in the survey in order to reduce the burden of participating in the research.


Session Reading Useful/Total %

5 Finegan (2004) 11/35 31%13 Singhal (2005) 9/29 31%6 Lems et al. (2010) 9/29 31%

13 Cohen (2007) 9/31 29%10 Gottlieb et al. (2009) 9/33 27%4 Brown (2000) 8/32 25%4 Crawford (2004) 0/32 0%4 Meyer (2009) –1/31 –3%

11 Schleppegrell (2001) –8/30 –27%

The following are recommendations to improve the quality of the RETELL course content on the basis of the findings just described:

Goals, objectives, and expected outcomes appear in the course syllabi and presentation slides. Participants seem capable of assimilating the information from course materials. Therefore, no change is recommended.

No changes are recommended to the topics covered in the course syllabi. If changes are made to the syllabi for other purposes (e.g., to ensure that participants have more opportunities to engage with specific strategies), make sure that key areas of ELL instruction (see description and sources discussed earlier) are retained.

Teacher buy-in is essential in order to motivate teachers to adopt practice, especially when the PD does not already align with the beliefs of the teacher (Kubitskey & Fishman, 2005). Because some recipients of the RETELL PD may not be convinced that they can successfully impart content to students who are not proficient in English, consider adding a case study or video to the first session of the course that illustrates how sheltering instruction can help ELLs access content at the same time as they acquire English language skills.

To reduce printing burdens for teachers, provide a CD or printed copies of the required readings.

Reduce the reading load by carefully selecting readings that are the most relevant to the successful implementation of SEI and that are clearly written. Consider readings that provide practical advice about helping ELLs meet high content standards.

Quality of Process

Quality of process is examined through research question 1. According to the Massachusetts SPD, high-quality PD promotes collaboration among educators and provides opportunities for participants to apply what they are learning to their own educational context. An important first step in helping participants tie their coursework to their own contexts is learning what those contexts are. Indeed, during interviews, instructors expressed knowledge of participants’ backgrounds and their general knowledge about ELL instruction. But the instructors were almost never observed asking participants what they would like to learn about a topic, or what they


already knew. (It is possible that these conversations happened during sessions that were not being observed.)

The Massachusetts SPD also states that high-quality PD models good pedagogical practices, including being problem-centered, capitalizing on learners’ experiences, and allowing choice and self-direction. It is especially important to link theory presented in PD to practice (e.g., Darling-Hammond & McLaughlin, 1995; Garet, Porter, Desimone, Birman, & Yook, 2001; Kubitskey, 2006; Kubitskey & Fishman, 2006; Penuel, Fishman, Yamaguchi, & Gallagher, 2007). Most observed sessions, however—with only a few exceptions—did not focus heavily on strategies. Those sessions that did, spent little time reviewing them in depth. Though participants sometimes asked or posted questions about how something could be used in the classroom, or made suggestions for applying ideas in the classroom, they seldom were given clear steps they could apply to improve their skills. The instructors made similar observations; some expressed a desire for more time to model strategies and discuss practical strategies rather than focusing on theory. One stated, “I remember one of the courses on oral linguistics.… It was so overwhelming and so technical and unnecessary.… I think heavy theory about linguistics is just a lot for teachers. They’re practitioners and they’re practicing right then. You do have to wrap it up with research and theory to convince people along with explaining it to people but I also think if it’s way above people’s heads, not their field specifically, and it’s hard.”

It also was observed that instructors did not have sufficient time to give overviews when introducing new topics and activities, or summaries at the end of discussions, presentations, and activities (though overviews and summaries were almost always provided in online sessions). Overviews and summaries could have been a good place to effectively link theory to practice during PD sessions.

On the other hand, many examples of good pedagogical practices were observed. At all sites, instructors frequently asked whether there were questions and then waited for participants to respond. They used phrases like “good point,” “I can understand your interpretation of that,” “great question,” and “thanks for bringing that up.” They also frequently asked participants to “think about this,” or “what would you do if…?” et cetera. In face-to-face sessions, participants appeared actively engaged during presentations (though there were occasional instances of people checking e-mail or engaged in side conversations). Most participants reported that instructors facilitated discussions and responded to questions well or very well (82 percent and 86 percent, respectively). See Figure 2.


Figure 2. RETELL Course Participants’ Responses to Survey Questions About Instructor Quality

Poorly Adequately Well Very well Did not observe0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

2%

16%

40%42%

0%2%

12%

42%44%

0%

How well the instructor facilitated discussionsHow well the instructor responded to questions

In addition, in face-to-face sessions, instructors frequently promoted collaboration. During whole-group discussions, participants answered questions from other participants and commented on each other’s statements. Participants largely felt that interactions with colleagues during face-to-face sessions were pretty relevant (30 percent) or very relevant (58 percent). Their feelings about interactions are supported by research studies by Garet et al. (2001) and Kubitskey & Fishman (2007), who find that activities that encourage professional communication enhance teachers’ knowledge, skills, and instructional practices. Other research indicates that collaboration is important both in terms of implementation of new skills (Bryk & Schneider, 2002) and in terms of giving educators an opportunity to discuss and resolve issues of practice and initiate broader systemic changes in their schools (Frank, Zhao, & Borman, 2004; Lieberman & McLaughlin, 1992; Little, 1993; Penuel et al., 2007).

The online portion of the course, however, was less engaging and collaborative. Though participants had the opportunity to post questions and answer other participants’ questions or comment on what they had said, few took this opportunity. (There was one exception in which a discussion forum had multiple responses and some participants had an online dialogue going.) Some participants were observed to be highly engaged in posting online, but many did not post anything, even to fulfill a session requirement. In some online sessions, the instructors did not participate in the discussion forums by commenting on any posts, nor did they utilize the news feature. One instructor expressed frustration with using the online platform to generate discussion. She had difficulty using it and also was disappointed in the quality of posts that participants were generating. Overall, participants felt that interactions with colleagues during online sessions were not relevant (43 percent) or only a little relevant (27 percent), and most participants felt that the online portion of the course was not relevant (25 percent) or only a little relevant (39 percent). Only 9 percent of participants found it very relevant. See Figure 3.


Figure 3. RETELL Course Participants’ Responses to Survey Questions About Online Sessions

Not relevant A little relevant Pretty relevant Very relevant Did not occur0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

43%

27%

20%

5% 5%

25%

39%

28%

9%

0%

Relevance of online interactions with RETELL participantsRelevance of the online portion of RETELL

The following are recommendations to improve the quality of the RETELL course process on the basis of the findings just described:

The pilot instructors expressed knowledge of the course participants’ backgrounds. This knowledge is important to the quality of process, and instructors of the main course should learn about their participants’ backgrounds and contexts. Consider including some short activities in the course that explore what participants already know about various topics and what they would like to learn. Provide sufficient flexibility for instructors to be responsive to participants’ needs.

Identify instructional strategies that are aligned with each course topic. Have PD instructors model these strategies using elements of model lessons and give participants an opportunity to apply what they have learned in a new context. Consider providing participants with a rubric to record each strategy they use, with information about the classroom context, how the strategy worked, and recommendations for adjustments.


Ask participants to briefly summarize what they learned at the end of each session, either as a whole class, in small groups or partners, or as a warm-up during a subsequent online session. As part of their summary, ask them to link the course materials to specific experiences or examples from their own educational contexts, or describe methods they have used related to the session.

Pilot instructors appeared to be skilled at leading collaborative discussions and engaging participants. Ensure that instructors in future RETELL courses are prepared to use good pedagogical practice to facilitate collaboration and encourage participation.

Consider developing professional learning communities (PLCs) as part of the RETELL course so that collaboration among participants can extend beyond the duration of the course. A review of relevant research on PLCs by Vescio et al. (2008) finds that well-developed PLCs have positive effects for both teacher practice and student outcomes.

Consider methods for increasing discussions and interactions online. Some options include the following:

o Establish weekly “office hours” during which the instructor is online and available for chatting (consider making attendance at some office hours a course requirement).

o Create a blog or wiki in which participants can report on the implementation of strategies or share ideas for teaching that came to them as a consequence of a reading or session.

o Group participants and provide tasks that require them to check in with each other regularly, either in person or via the online platform.

o Set up a collaborative project online to which all participants are required to contribute. Ideally, such a project would be best facilitated by the online platform (such as posting videos of instruction and eliciting constructive feedback or conducting an analysis of student work using graphic tools; see Little et al., 2003).

Encourage instructors to actively monitor online activity and send reminders and prompts to participants as needed.

Quality of Context

Quality of context is examined through research question 1. According to the Massachusetts SPD, high-quality PD is facilitated by knowledgeable instructors and makes use of relevant resources. Kubitskey & Fishman (2007) find that in order for participant discussions to be fruitful arenas for deep conversations, it is important to have a well-trained local PD provider. For the RETELL pilot course, every participant across all four evaluation sites either agreed (35 percent) or strongly agreed (65 percent) with the statement that their instructor was knowledgeable about SEI. As additional evidence of the instructors’ knowledge, it was observed that the instructors answered all the questions posed in the course materials, though no answer guide was provided.


The instructors reported that they had a great deal of flexibility in the delivery of the course. For example, the course was not heavily scripted, and so they determined which activities and even which strategies to model. One instructor noted that the lack of scripting would make this a very difficult course to deliver without an advanced degree and extensive teaching experience: “I just don’t see honestly how someone without a Ph.D. or really advanced work in linguistics can take this and run with it. So I think that the idea is nice to let people have their own input into it, but I also think that you have to really have a strong command of the content to be able to do that. I mean this is after seven years of teaching college courses, developing courses…, so I just don’t know if they [ESE] have enough people to do that.”

The following are recommendations to improve the quality of the RETELL course context on the basis of the findings just described:

It will be essential to select highly qualified instructors for a course that is not heavily scripted. Unless activities are added and scripted, only RETELL instructors who are prepared to develop and incorporate their own activities should be selected.

Consider developing and providing answer guides for questions posed in the course materials.

Pace of Course

The pace of the course is examined through research question 1. Participants need adequate time to assimilate information that they are learning, but the pace of the PD should be brisk enough to maintain participants’ energy and to avoid boredom. During observations, the instructors appeared to be very skilled at gauging participants’ need to speed up or slow down. But the pace of the sessions was always very fast, with lots of content and little time for reflection. Indeed, instructors generally viewed the pace as much too fast, describing it as “overwhelming” and “problematic.” Several described the pace as “ridiculous.” One instructor noted that the course would work if each session were delivered every two weeks, giving participants time to apply what they’re learning in their classrooms in order to successfully complete course assignments. Participants agreed that the pace of the RETELL course was problematic; a large majority of participants thought that the pace of the course was a little too fast (40 percent) or much too fast (45 percent). Only 15 percent thought that the pace was just right, and no one found it too slow. See Figure 4.


Figure 4. RETELL Course Participants’ Responses to Survey Question About Course Pace

Much too slow A little too slow Just right A little too fast Much too fast0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

0% 0%

15%

40%

45%

Pace of the RETELL course

Note that plans are already in place to deliver the RETELL course over 16 weeks (one semester) rather than the 8.5–10 weeks of the pilot courses. ESE would prefer to not extend it any longer because of potential problems with implementation, such as scheduling conflicts.

The following recommendation to improve the pace of the RETELL course is on the basis of the findings just described:

Carefully monitor the semester-long course to make sure that the pace is not too fast. For example, the schedule should allow reasonable time between sessions for participants to complete readings and assignments, and to test the methods and instructional strategies they are learning.

Allocation of Time in Course

Related to course pacing, the allocation of the time spent in the course can have important implications for learning. For example, adults need time to reflect on what they are learning in terms of their own classroom experiences (cf. Kubitskey, 2006; Penuel et al., 2007). It also is important that they have adequate opportunities for collaboration with peers, through whole-class discussions, small-group work, and interactions outside session time. The allocation of time in the RETELL course is examined through research question 1.

On average, time during observed face-to-face sessions was allocated as presented in Figure 5, with small-group work and whole-class discussions taking up the majority of class time.


(Participant questions and discussions of how to apply strategies happened throughout the sessions, but not continuously.)

Figure 5. Average Time Spent on Activities During RETELL Course Face-to-Face Sessions

Instructor presentation/lectures

Small-group work

Individual work

Discussions of strategy application

Practice applying strategies

Whole-class discussion

Participant questions

Summarizing and review

0 20 40 60 80 100 120 140 160 180 200

22

58

9[throughout]

24

50[throughout]

7

Average Minutes

Most participants expressed the opinion that the right amount of time was spent on instructor lectures (71 percent), small-group work (77 percent), whole-class discussions (84 percent), interactions with colleagues (79 percent), individual work in class (53 percent), and videos (63 percent). They thought that too little time was spent on discussions of how to apply strategies (55 percent) and practice applying strategies (61 percent), and too much time was allocated to homework (58 percent) and reading (57 percent). See Figure 6.


Figure 6. RETELL Course Participants’ Responses to Survey Questions About Time Allocation

Instructo

r lectu

ring

Small

-group work

Whole-cla

ss disc

ussion

Interacti

on with

collea

gues

Individual w

ork (in

class)

Videos

Discussi

on of strat

egy ap

plication

Practice

applyin

g stra

tegies

Homework

Reading

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

20% 21%

9%

19%

44%38%

55%61%

33%

23%

71%77%

84%79%

53%

63%

41% 39%

9%

20%

9%

2%7%

2% 4%0%

4%0%

58% 57%

Too little time Right amount Too much time

For online sessions, participants reported devoting the most time (on average) to assignments (60 percent more than one hour) and reading (52 percent more than one hour). They devoted the least time to interactions with the instructor (90 percent less than one hour), learning how strategies can be applied in the classroom (69 percent less than one hour) and reading the slide presentations (68 percent less than one hour).

Almost all the instructors expressed frustration at the allocation of time in the course. They were concerned that they were trying to cover too much material, without time to focus on strategies and classroom applications. One instructor recommended condensing the slide presentations into very short overviews of the theory, devoting the rest of the session to practicing strategies.

The following are recommendations to improve the allocation of time in the RETELL course on the basis of the findings just described:

Because participants found course time allocations appropriate for lectures and in-class interactions with colleagues (including small-group work and whole-class discussion), maintain these activities at their current levels.


In order to address instructor concerns and participant feelings that not enough time was devoted to strategies, consider replacing some of the small-group discussions with small-group strategy practice; whole-class discussions also could be used to discuss how theory is realized in practice and how to apply instructional strategies successfully.

Replace significant out-of-class assignments with activities that require collaboration with colleagues, particularly on the practice of strategies. Be sure to incorporate enough time between assignments to allow participants to engage in these activities meaningfully.

To address time allocation during the self-paced, self-directed online sessions, consider creating an activity that helps participants understand how they learn best. Provide tips for time management as part of the introduction to the online sessions.

Clarity of Instruction

It is crucial that instruction be delivered in a manner that participants can easily understand. Clear definitions and descriptions, particularly of instructional strategies, and information about how to use the instructional strategies for teaching are important factors in influencing teachers’ instruction (Kubitskey & Fishman, 2007). Clarity of instruction in the RETELL course is examined through research question 1.

Overall, participants felt that instructors gave instructions well (44 percent) or very well (28 percent). See Figure 7. In the survey, 70 percent of participants reported rarely or never being unable to receive clarification from the instructor about something that confused them. This can be interpreted to mean that a large majority of participants received clarification when they needed it. In observations of face-to-face sessions, instructors at all four sites always responded directly to questions and comments, offering additional instruction or clarification when needed. In all sites except Worcester, they identified prior conceptions and misconceptions (one instructor made conceptions and misconceptions a topic for an activity and discussion). It was always easy to hear the instructor and see the presentation during face-to-face sessions.

Figure 7. RETELL Course Participants’ Response to Survey Question About Course Instructions

Poorly Adequately Well Very well Did not observe0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

7%

20%

44%

28%

0%

How well the instructor gave instructions


For the online sessions, the evaluators observed that the presentation slides were extremely easy to navigate, but online course materials were less so. Furthermore, in online sessions, instruction was never observed to address feedback from previous sessions nor to identify prior conceptions and misconceptions. In all cases but one, online instruction did not summarize the readings or provide information about possible questions or concerns.

Finally, instructors noted that the assignments changed frequently throughout the course, which was stressful for participants. They also noted that some participants needed extra time for the readings. One instructor stated, “I’ve had people in class say, ‘I have ADHD’…just like with our ELLs, you have to be able to differentiate and meet them three weeks ahead of time. If they don’t read it till the night before, well that’s their choice but some people like to have it [in advance].”

The following are recommendations to improve clarity of instruction in the RETELL course on the basis of the findings just described:

Consider including some short activities in both the face-to-face and the online portions of the course that explore participants’ conceptions and misconceptions to ensure that they are addressed.

For the online sessions, provide a single list of required readings, materials, and activities for each session with hyperlinks to each.

Include summaries of readings and information about possible questions or concerns in online materials.

Provide a list of assignments and readings ahead of the course so that participants have additional time to complete the work.

Support for Course Instructors

In addition to examining how the RETELL course was delivered, it is important to examine how instructors were supported in their delivery of the PD. Support for RETELL course instructors is examined through research question 2. During delivery of the pilot courses, some of the instructors experienced logistical issues, like lack of Internet access, locked bathrooms, inability to access the classroom ahead of a session to set up, and being placed in classrooms where people were unable to eat during three-hour evening sessions. Other logistical issues included making sure participants had access to Moodle and having participants’ e-mail addresses in advance. In addition, instructors reported that ESE was good about making copies for instructors if they requested them in advance. It sometimes, however, proved difficult to obtain other materials, like chart paper and markers.

The following are recommendations to better support RETELL course instructors:

Make sure that course classrooms are easily accessible, can be accessed ahead of each session for setup, and have Internet access. If possible, designate a person at each site to ensure that these logistics are taken care of.

Provide participant contact information to instructors as early as possible (ideally, at least two weeks before the first session).


Establish a clear policy about material distribution and provide it to all instructors. Include information about which materials and services ESE provides and which it does not.

Establishment of Feedback Loops

Feedback is an important aspect of course quality. The establishment of feedback loops to ensure course quality is examined through research question 3. Instructors sought feedback from participants, both informally (talking to participants after class) and formally (surveys). They also sought feedback from each other. In terms of ensuring course quality in the future, most instructors expressed the need for more scripted instruction so that the RETELL message is consistent across all courses. In addition, they wanted uniform information about grading. One instructor also suggested that they should have the opportunity to practice the trainings with other instructors to see what it should look like in practice before attempting to deliver it.

The following recommendations to establish feedback loops to ensure RETELL course quality are based on the findings just described:

Provide information to instructors about which aspects of the RETELL course will need to be consistent across the state, such as participant grading or evaluation information.

Consider providing a training session for instructors during which they can practice elements of the PD with each other before delivering it to RETELL course participants.

Establish a consistent feedback system for all RETELL PD courses. The feedback system might consist of an observation by an ESE official, formal participant surveys (e.g., short surveys delivered via Survey Monkey after each class), or an outside evaluation of teacher products (e.g., lesson plans) or outcomes.


During full-scale rollout, the strength of the RETELL PD will depend on the fidelity of its implementation. Course observations were used to evaluate the level of implementation during face-to-face and online sessions (research question 4) and instructor interviews examined how feedback loops could be established to ensure greater fidelity of implementation (research question 5). Note that because the course is still in its pilot stage, evaluation of fidelity of implementation was largely limited to alignment with the planned syllabus.

In every observed face-to-face session, instructors’ goals and objectives were the same as those stated in the syllabi, and at least one of the required readings was explicitly incorporated into the activities at each session. Participants were required to complete the homework in nearly every session. Most face-to-face sessions, however, did not cover all the content. Some sessions did not cover even half the content in the syllabi. The failure to cover all the content appeared to be caused by differences in participant backgrounds and educational contexts; instructors had to modify content, discussions, and activities to accommodate everyone, slowing the pace of each session.

Online sessions exhibit almost universal fidelity to the syllabi, with a few exceptions. In session 7, the Halliday assignment in the syllabus was eliminated. In session 10, the assignments in the syllabus for session 10 were included in the slides but were marked as assignments for session


11. And finally, in session 12 the slides did not address ELL assessment through MCAS accommodation, as specified in the syllabus.

Instructors reported using a clock or stopwatch to ensure that there was sufficient time for each activity in the materials that were provided. They also met regularly with the other RETELL pilot course instructors to discuss course delivery in order to ensure fidelity to the syllabi and course materials. They reported difficulty with fidelity, however, because the course was continuously changing while they were delivering it.

The following recommendations for fidelity of RETELL course implementation are based on the findings just described:

Strategically group participants with similar backgrounds and educational contexts into the same courses so that instructors can focus on their needs efficiently, increasing the likelihood of covering all course content. As an added advantage, grouping participants from similar educational contexts can facilitate the formation of PLCs (see, e.g., Newmann & Associates, 1996).

Check online sessions to ensure that they cover all topics and assignments in the syllabus, especially if there have been changes to the sessions.

Provide instructors with information about how long each activity should take.

Facilitate regular meetings among instructors (by region and/or participant type—e.g., secondary STEM) throughout the course semester to discuss RETELL delivery and fidelity to the syllabus. These meetings would also be a good opportunity for ESE to collect feedback from instructors about recommended changes.

Make necessary changes to the course between, not during, rollout semesters.

Coherence

A national study by Garet et al. (2001) found that coherence was one of the most important aspects of effective PD; PD that connects to teachers’ previous knowledge and training is more likely to have an impact on teacher practice. In this evaluation, two types of coherence are examined. The first is the level of coherence of the RETELL sessions (research question 6), or how well the sessions build on one another. The second is the level of coherence of the course with what teachers experience in their educational contexts (research question 7).

Level of Coherence of the RETELL Sessions

Because the RETELL course is a hybrid course with nearly a third of the sessions online, it is important that the sessions explicitly build on each other such that participants can recognize the relevance of what they are learning outside the face-to-face time. In addition, each session should help teachers augment their overall knowledge of effective instruction for ELLs by building on what was learned about the topic before. The level of coherence of the RETELL course is addressed in research question 6 by examining the ways in which explicit connections between sessions are made.

Instructors at all four sites were only occasionally observed giving a brief overview of the previous session and explaining how the current session built on what was learned before. Only


one instructor was observed connecting the material in a face-to-face session with material from a previous online session. They reported that there was insufficient time to make connections across sessions, in the interest of covering as much of each session’s content as possible. The majority (65 percent) of participants, however, reported that their instructor made a clear connection between the online and face-to-face content, so perhaps this occurred more often at sessions that were not observed for this evaluation. Observed online sessions did not explicitly build on previous sessions or provide explanations of how the online content would link to face-to-face sessions (except in one occasion when the instructor added comments to this effect).

The following recommendations to increase the level of coherence of the RETELL sessions are based on the findings just described:

Revise course materials to provide explicit connections between topics covered in each session, including revising the online materials to include explicit connections to previous and upcoming sessions.

Explain to instructors the importance of communicating expectations for the online course during the face-to-face sessions, even if there is only a little time to do so.

In addition, instructors recommended the following changes to increase coherence of the RETELL course:

Incorporate session 4 into session 5.

Consolidate information about academic language into a single session.

Incorporate information about the Common Core State Standards throughout the sessions rather than introducing it in session 15.

The topic of assessment should come at the end of the course, after reading and writing have been covered.

Provide more grade-level differentiation in Module C [within the secondary courses].

Level of Coherence With What Teachers Experience in Their Educational Contexts

Garet et al. (2001) noted the importance of ensuring that what teachers experience in PD be applicable to their own educational contexts. This type of coherence is examined through research question 7. The RETELL pilot course instructors also seemed to be aware of the importance of making connections to participants’ contexts; instructors at all sites used stories to link theory to practice, discussed problems that might occur with classroom application of strategies and how one might address them, and led participants in discussions of how a strategy would or would not work in their classroom. The majority of participants at all sites found it pretty useful (34 percent) or extremely useful (45 percent) to discuss application of strategies; no participant responded that it was not useful at all. Instructors were not, however, able to engage in these activities as often as they felt was necessary, and participants appeared to agree; overall, more participants felt that too little time was allocated to discussions of how to apply strategies to classroom instruction (55 percent) than felt that the right amount of time was allocated (41 percent). This difference was greatest in Boston and Worcester, where 68 percent of participants felt that too little time was allocated to discussion of strategy application. See Figure 8.


Figure 8. RETELL Course Participants’ Response to Survey Question About Time Allocated to Discussions of Classroom Strategy Application

Somerville Lowell Boston/Worcester Total0%

10%

20%

30%

40%

50%

60%

70%

80%

44%50%

68%

55%

44%50%

32%

41%

11%

0% 0%4%

Too little time Right amount Too much time

In addition, few examples of connections to classroom practice were evident in the online sessions. Though participants sometimes posted examples of how something would or would not work in their classrooms, the online instruction seldom discussed problems that might occur with classroom application and only sometimes incorporated stories or videos to link theory to practice. Most participants (70 percent) reported that less than an hour was devoted to illustration of how strategies could be applied to the classroom in the online sessions, but all reported that such illustrations were at least a little useful, and most found them pretty useful (29 percent) or extremely useful (38 percent). See Figure 9.


Figure 9. RETELL Course Participants’ Response to Survey Question About Usefulness of Online Illustrations of Strategy Application

Not useful at all A little useful Pretty useful Extremely useful Unknown0%

5%

10%

15%

20%

25%

30%

35%

40%

0%

29% 29%

38%

5%

Usefulness of online illustrations of strategy application

The following recommendations to increase the level of coherence of the RETELL sessions with what teachers experience in their educational contexts are based on the findings just described:

Connections to participants’ educational contexts are crucial. Ensure that RETELL course instructors are as prepared as the pilot instructors to connect material to real-world contexts, use stories to link theory to practice, and lead participants in discussions of how strategies would work in their educational contexts.

Consider incorporating the practice of instructional strategies into small-group work and whole-class discussions in order to increase its impact.

Update the online materials to make better connections to participants’ educational contexts. In each session, include

o Stories or videos that model practical strategies

o Discussions of possible challenges that teachers might face in applying strategies

o Prompts that encourage participants to post and discuss examples of how strategies would work in their educational contexts

Teacher Outcomes

The purpose of the RETELL initiative is to improve instruction for ELLs in Massachusetts by increasing mainstream teachers’ knowledge about ELLs and informing their instructional practices. This evaluation therefore examined how the course could better contribute to teachers’ key understandings of ELL students in regular classroom settings through research question 8 and additional information that teachers need to be successful with ELLs through research question 9.


The majority of participants in all sites agreed or strongly agreed that the course enhanced what they already knew about SEI (88 percent overall), but 61 percent also reported that some topics repeated what they already knew about SEI. The majority of participants reported that the course had contributed a fair amount or a great deal to their understanding in the areas shown in Table 5. It is impossible to determine whether the course contributed less to participants’ understanding in some areas because of their previous knowledge, or because the course did not adequately cover these areas.

Table 5. Topics for Which RETELL Participants Reported Greater Understanding

SEI TopicPercentage for Whom Course

Increased Understanding

Vocabulary instruction for ELLs 82%How ELLs acquire a second language 81%How to meet the needs of ELLs in the classroom 78%Social and cultural issues that affect ELLs 78%How to teach academic language to ELLs 78%How to apply SEI to reading and writing instruction 72%How to address the needs of diverse ELL populations

68%

How to differentiate instruction for ELLs 64%How to assess ELLs in the content areas 64%

As part of the survey, participants were also asked to list areas in which they felt they needed additional information to be successful with ELLs. These areas are given in Table 6.

Table 6. Topics in Which RETELL Participants Need Additional Information

SEI TopicCreating lessons and practical strategies for the classroomEffective practice with ELLs who do not speak Spanish Standardized assessments and ELLsELLs with special needsGrade-specific strategiesSupporting language developmentStudents with interrupted formal education (SIFE)

In addition to the topics in Table 6, one instructor felt that teachers needed more information about the difference between helping ELLs develop language and helping them access content. Other instructors expressed the opinion that there needed to be additional grade-level differentiation, and one felt that participants needed to be more immersed in data.


Of course, additional information that teachers need to be successful will vary from course to course, and teachers and instructors may not be aware of some of these needs without specific participant feedback structures. For example, the RETELL pilot course instructors reported several anecdotes of individual participants applying particular strategies or knowledge with their ELL students, but one instructor pointed out that it is impossible to tie the RETELL instruction to specific outcomes because there is no evidence to assess the outcomes. She explained that though there is a monitoring tool for participants’ lesson plans early in the course, there is no such tool for revisiting the plans at the end. She also suggested adding tools to get feedback about how participants are implementing strategies. Another instructor recommended a pre- and post-assessment of the content of the course.

The following recommendations to enhance the teacher outcomes of the RETELL course are based on the findings just described:

Conduct a brief survey at the outset of each course to determine areas in which participants are already well versed and areas where they need additional instruction.

Consider providing (additional) instruction in the following areas, as determined by participant needs:

o Creating lessons and practical strategies for the classroom

o Effective practice with ELLs who do not speak Spanish

o Standardized assessments and ELLs

o ELLs with special needs

o Grade-specific strategies

o Supporting language development versus helping students access content

o Students with interrupted formal education (SIFE)

o Using data

Fishman et al. (2003) propose an iterative PD research design model, in which feedback about teacher outcomes is incorporated into future PD sessions; a similar model has been proposed by Kubitskey & Fishman (2007). The current evaluation gathers self-reported data from teachers, but it also is important to measure teachers’ skills and knowledge. The following formative and summative evaluation activities are recommended both to provide teachers with information about their progress in the course and to improve future iterations of the course:

o Develop and incorporate protocols for assessing participants’ plans for and reports of how they implemented course strategies.

o Develop and conduct pre- and post-course assessments of participants’ knowledge about ELLs and ELL instruction.

o Include a classroom observation component in the course. Teachers might observe each other and provide peer-to-peer feedback; a school or district ESL coordinator or coach might observe participants and provide feedback as part of the RETELL course; or RETELL course instructors might observe each participant teaching. Observations are recommended for teachers in high- and medium-incidence schools;


they are not recommended if participants do not have ELLs in their classrooms at the time of the RETELL PD.

Make sure that instructors can provide timely feedback to participants using the Moodle online platform. For instructors who are unfamiliar with Moodle, offer training on how to use it effectively.


Next Steps

ESE plans to roll out the first phase of the RETELL course in February 2013. Before that time, many revisions will be made to the course materials on the basis of what has been learned from the pilot course.

This evaluation has made 44 recommendations in four major categories (quality of delivery, fidelity of implementation, coherence, and teacher outcomes) for improvement of the quality of the course. As a final recommendation, if these or other changes are implemented, it will be important to monitor their effects on the success and quality of the course. It also will be important to explore how the course functions when it is rolled out to a much larger group of teachers across many more districts and schools. Of especial interest is the online portion of the course. If it can be successfully revised to increase participant engagement and collaboration, it will be an important tool for ensuring that teachers across the state are prepared to provide effective instruction for the ELLs in their classes.


References

August, D., & Shanahan, T. (2010). Effective English literacy instruction for English learners. In F. Ong (Ed.) (with V. Aguila), Improving education for English learners: Research-based approaches (pp. 209–237). Sacramento, CA: CDE Press.

Bryk, A. S., & Schneider, B. (2002). Trust in schools: A core resource for improvement. New York: Russell Sage.

Carlo, M. S., August, D., McLaughlin, B., Snow, C. E., Dressler, C., Lippman, D., et al. (2004). Closing the gap: Addressing the vocabulary needs of English-language learners in bilingual and mainstream classrooms. Reading Research Quarterly, 39(2), 188–215.

Darling-Hammond, L., & McLaughlin, M. W. (1995). Policies that support professional development in an era of reform. Phi Delta Kappan, 76(8), 597–604.

Fishman, B. J, Marx, R. W., Best, S., & Tal, R. T. (2003). Linking teacher and student learning to improve professional development in systemic reform. Teaching and Teacher Education, 19(6), 643–658.

Frank, K. A., Zhao, Y., & Borman, K. (2004). Social capital and the diffusion of innovations within organizations: The case of computer technology in schools. Sociology of Education, 77(2), 148–171. doi: 10.1177/003804070407700203

Franken, M., & Haslett, S. J. (1999). Quantifying the effect of peer interaction on second language students’ written argument texts. New Zealand Journal of Educational Studies, 34(2), 281–293.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yook, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945.

Gersten, R., Baker, S. K., Shanahan, T., Linan-Thompson, S., Collins, P., & Scarcella, R. (2007). Effective literacy and English language instruction for English learners in the elementary grades: A practice guide (NCEE 2007–4011). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/practice_guides/20074011.pdf

Kubitskey, B. (2006). Extended professional development for systemic reform. Unpublished doctoral dissertation, University of Michigan, Ann Arbor.

Kubitskey, B., & Fishman, B. (2005, April). Untangling the relationship(s) between professional development, practice, student learning, and teacher learning. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Canada.


http://ies.ed.gov/ncee/wwc/pdf/practice_guides/20074011.pdf

Kubitskey, B., & Fishman, B. J. (2006). A role for professional development in sustainability: Linking the written curriculum to enactment. In S. A. Barab, K. E. Hay, Songer, N. B., & D. T. Hickey (Eds.), Making a difference: Proceedings of the seventh International Conference of the Learning Sciences (Vol. 1, pp. 363-369). Mahwah, NJ: Lawrence Erlbaum.

Kubitskey, B., & Fishman, B. J. (2007, April). A design for using long-term face-to-face workshops to support systemic reform. Paper presented at the American Educational Research Association, Chicago, IL.

Lee, O., Maerten-Rivera, J., Penfield, R. D., LeRoy, K., & Secada, W. G. (2008). Science achievement of English language learners in urban elementary schools: Results of a first-year professional development intervention. Journal of Research in Science Teaching, 45(1), 31–52.

Lieberman, A., & McLaughlin, M. W. (1992). Networks for educational change: Powerful and problematic. Phi Delta Kappan, 73(9), 673–677.

Little, J. W. (1993). Teachers’ professional development in a climate of educational reform. Educational Evaluation and Policy Analysis, 15(2), 129–151.

Little, J. W., Gearhart, M., Curry, M., & Kafka, J. (2003). Looking at student work for teacher learning, teacher community, and school reform. Phi Delta Kappan, 85(3), 184–192.

Massachusetts Department of Elementary and Secondary Education. (2009). Report to the legislature: English language acquisition professional development. Malden, MA: Author.

Massachusetts Department of Elementary and Secondary Education. (2012, May). The state of the state: A report on English language learners in Massachusetts. Paper presented at the Massachusetts Association of Teachers of Speakers of Other Languages Conference, Framingham, MA.

National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs. (2011). The growing number of English learner students. Washington, DC: Author. Retrieved from http://www.ncela.gwu.edu/files/uploads/9/growingLEP_0809.pdf

National Governors Association Center for Best Practices & the Council of Chief State School Officers. (2010). Common core state standards for English language arts. Washington, DC: Authors.

Newmann, F. M., & Associates. (1996). Authentic achievement: Restructuring schools for intellectual quality. San Francisco, CA: Jossey-Bass.

Owens, A. (2010). In the aftermath of Question 2: Students with limited English proficiency in Massachusetts. Boston: Massachusetts Office for Refugees and Immigrants. Retrieved from www.doe.mass.edu/research/reports/0610ori.doc


http://www.ncela.gwu.edu/files/uploads/9/growingLEP_0809.pdf

Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher, L. P. (2007). What makes professional development effective? Strategies that foster curriculum implementation. American Educational Research Journal, 44(4), 921–958.

Perez-Selles, M., Cazabon, M., & Mello, D. (2011). Needs assessment report: Sheltered English immersion (SEI) category trainings, Massachusetts Department of Elementary and Secondary Education. Woburn, MA: WestEd. Retrieved from http://www.doe.mass.edu/retell/1011SEIPD_NeedsAssess.pdf

Saunders, W. M. (1999). Improving literacy achievement for English learners in transitional bilingual programs. Educational Research and Evaluation, 5(4), 345–381.

Saunders, W. M., & Goldenberg, C. (1999). Effects of instructional conversations and literature logs on limited- and fluent-English-proficient students’ story comprehension and thematic understanding. Elementary School Journal, 99(4), 277–301.

Vaughn, S., Martinez, L. R., Linan-Thompson, S., Reutebuch, C. K., Carlson, C. D., & Francis, D. J. (2009). Enhancing social studies vocabulary and comprehension for seventh-grade English language learners: Findings from two experimental studies. Journal of Research on Educational Effectiveness, 2(4), 297–324.

Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80-91.


http://www.doe.mass.edu/retell/1011SEIPD_NeedsAssess.pdf

[This page intentionally left blank.]


Appendix A: Data Collection Instruments

Face-to-Face Session Observation Protocol

RETELL In-Class Observation ProtocolDate Location

Instructor Name

Observer

Observation Start Time

# of Participants (15 min in)

Observation End Time

Topic

Complete this section immediately after your observation. Rate each item below.

Frequency Quality Notes

Never SometimesAlmost Always

Poor Acceptable Excellent N/A

Sess

ion

Intr

oduc

tion

Instructor articulates goal of session and the effect the session should have on participants’ practice.

Instructor asks what participants would like to learn/know about the topic (or how much they already knew)

Instructor adjusts schedule to address these topics

American Institutes for Research MA SEI Endorsement Pilot Evaluation —43

Frequency Quality NotesIn

stru

ction

Instructor can be heard in the back of the room

Instructor responds directly to questions and comments

Instructor identifies prior conceptions and misconceptions about content

Instructor appears knowledgeable about content

Instructor offers additional instruction and/or clarification when a participant asks a question

Instructor gives a brief overview of the previous session and explains how the current session will build on what was learned.

If the course follows an online session, instructor gives an overview of the online component and summary of what should have been learned

If the course follows an online session, instructor summarizes the readings and asks if there are questions or concerns

Instructor uses clues from participants to speed up, slow down, or reengage

If the course follows an online session, instructor explains how the online content links to the content to be covered in the face-to-face session

Instructor gives overviews (when introducing topic, idea, activity)

Instructor gives summaries (at the end of a discussion, presentation, activity)


Frequency Quality NotesQ

ualit

y of

Cur

ricul

um Stru

gglin

g

Most participants complete the activities correctly (as the instructor intended)

Multiple participants ask for clarification on the same topic

Push

ing

Back

Participants suggest that the factual material included in the syllabus (the course content) is flawed in some manner (e.g. “I don’t think this is the case.” “In my experience, this usually works differently.” “I don’t see how that would be useful!”) Participants question the usefulness of the course (e.g. “This is good information but I’m not sure it’s applicable to me/us/the real world.” “I don’t know. I think the old category training was sufficient.”)

Peda

gogy

/Tra

nsfe

rrin

g Id

eas t

o Pr

actic

e

Instructor discusses problems that might occur with classroom application and addresses them

Instructor promotes collaboration among participants

Instructor connects content to real-world contexts

Instructor uses stories to link theory to practice

Engagement with important ideas relevant to classroom practice

Participants discussed an example of how something would/would not work in their classroom

Participants asked clarifying questions about how something could be used in the classroom

Participants offered suggestions on how ideas could be applied in the classroom



Participants answered questions from other participants or commented on other participants’ questions/comments on classroom application

Acce

ssib

ility

of M

ater

ial

Read

ings

Most participants completed the readings for the session

Participants reference readings during discussions

Participants state reading selections were useful or helpful

Participants ask clarifying questions about what the main ideas of reading selections were

Activ

ities When introducing activities, instructor states

how the activity is connected to course goals

Participants seem to have trouble following the instructions for activities

Voca

bula

ry

Participants readily adopt the phrases used in the course to describe phenomena or ideas (e.g. “culturally responsive” or “affective filter”)

Pow

erPo

int Participants are engaged during PowerPoint

presentations (see Overall Engagement for how to measure this) Participants ask questions during or after PowerPoint presentations Participants can see the presentation clearly



Cont

ent

Participants demonstrate some familiarity with much (30-40%) of the course content (e.g. if participants have heard of “culturally responsive practice” or “affective filter” they can understand what it is because they have witnessed or learned about it before--albeit by a different name)

Ove

rall

Enga

gem

ent

Most participants look at the instructor as he/she is speaking

Most participants nod their heads in agreement as the instructor is speaking

Most participants take notes as the instructor is speaking

When instructor asks general questions to gauge understanding (e.g. “Is this clear?” “Are there any questions?”), most participants nod or answer verbally

Participants turn to the appropriate pages/sections of the instructional materials when instructed to do so

Most participants are sitting upright in their seats

Most participants are checking email, voicemail, texting, or surfing the internet for content unrelated to the course

Participants ask clarifying questions

Paci

ng

Participants are flipping through later sections of the materials ahead of the instructor

Participants are still reading earlier sections of materials (or discussing with another participant) although the instructor has moved on



Course activities are distributed evenly throughout session.

Clas

sroo

m C

ultu

re

Participants and instructor challenged each other’s ideas

Instructor asks if there are questions and waits for participants to respond

Instructor uses phrases like “good point,” “I can understand your interpretation of that,” “great question,” “thanks for bringing that up,” or others

Instructor asks participants to “think about this,” or “what would you do if…,” “what’s a good way to handle…,” “why do you think that is,” or “what else would you like to know?” or others

Nex

t Ste

ps Participants presented with clear next steps they could apply to continue to improve their skills

Participants provided with list of resources (websites, contact info, etc.) to help implement what was learned in the classroom

Fide

lity

of

Impl

emen

tatio

n

Session covers all topics listed in the syllabus.

Instructor’s goals and objectives are the same as those stated in syllabus.

Instructor discusses and/or incorporates all required readings for session.

Participants are required to complete all homework for session as described in the syllabus.


Complete this section during your observation. When you observe a listed action, note the start and end for each occurrence (as many as needed). You do not need to describe the activity itself unless it is of particular importance, in which case, you may use the notes section.

Start End StartEnd

Time Start End Start End Start End

Allo

catio

n of

Tim

e

Instructor presenting/lecturing

Small group work/discussions

Individual work Discussions of how to apply strategies being learned to classroom instruction Practice applying strategies to classroom instruction

Whole class discussion




Complete this section during your observation. List examples of each that you observed. You may use the notes section to describe each in more detail.

NotesAc

cess

ibili

ty

Ah-ha moments

Demonstrating understanding of key concepts and ideas during course

Notes


Online Session Observation Protocol

RETELL Online Observation ProtocolInstructor Name Session #

Observer

Observation Start Time (if applicable)

# of Participants Average # of Posts per Participant

Observation End Time (if applicable)

Topic

Complete this section immediately after your observation. Rate each item below.

Occurrence Quality Notes

Yes No N/A Poor Acceptable Excellen

t N/A

Inst

ructi

on

The goal of session and the effect the session should have on participants’ practice are provided A brief overview of the previous session and an explanation of how the current session will build on what was learned are provided Instruction explicitly addresses feedback from previous sessions Instruction identifies prior conceptions and misconceptions about content

Instructor’s paper exhibits knowledge about content

Instructor uses the online platform effectively Instructor offers additional instruction and/or clarification when a participant posts a question Instructor’s paper summarizes the readings and provides information about possible questions or concerns


Occurrence Quality Notes

Yes No N/A Poor Acceptable Excellen

t N/A Explanations of how the online content links to the content to be covered in the face-to-face sessions are provided Overviews are provided (when introducing topic, idea, activity) Summaries are provided (at the end of a discussion, presentation, activity)


Never SometimesAlmost Always


Qua

lity

of C

urric

ulum Stru

gglin

g

Participant posts/comments are irrelevant to the stated requirements.

Push

ing

Back

Participant comments suggest that the factual material included in the syllabus (the course content) is flawed in some manner (e.g. “I don’t think this is the case.” “In my experience, this usually works differently.” “I don’t see how that would be useful!”) Participant comments question the usefulness of the course (e.g. “This is good information but I’m not sure it’s applicable to me/us/the real world.” “I don’t know. I think the old category training was sufficient.”)

Peda

gogy

/Tr

ansf

errin

g Instruction discusses problems that might occur with classroom application and addresses them

Instruction connects content to real-world contexts

Instruction incorporates stories/vides to link theory to practice


Frequency Quality Notes Never Sometimes

Almost Always


Idea

s to

Prac

tice

Engagement with important ideas relevant to classroom practice

Participants posted an example of how something would/would not work in their classroom

Participants posted clarifying questions about how something could be used in the classroom

Participants posted suggestions on how ideas could be applied in the classroom

Participants answered questions from other participants or commented on other participants’ questions/comments on classroom application

Acce

ssib

ility

of M

ater

ial Re

adin

gs

Most participants completed the readings for the session

Participants reference readings in their comments Participants posted clarifying questions about what the main ideas of reading selections were

Activ

ities Activities are introduced with information about how

the activity is connected to course goals

Participants complete the activities correctly (as the instructor intended)

Voca

bula

ry

Participants readily adopt the phrases used in the course in their comments (e.g. “culturally responsive” or “affective filter”)

Mat

eria

ls

The PowerPoint slides are easy to follow. Course materials are easy to navigate.


Frequency Quality Notes Never Sometimes

Almost Always


Parti

cipa

tion Participants have an opportunity to ask clarifying

questions

Participants have an opportunity to response to discussion questions

Ove

rall

Enga

gem

ent

Participants use online tools to indicate participation

Participants complete session requirements by posting online and responding to other participants’ posts

Paci

ng

Participants can cover all materials in the allotted session time.

Course activities are distributed evenly through session topics.

Clas

sroo

m

Cultu

re

Participants and instructor challenged each other’s ideas in posts

Comment prompts encourage participants to “think about this,” or “what would you do if…,” “what’s a good way to handle…,” “why do you think that is,” or “what else would you like to know?” or others


Occurrence Quality Notes Yes No N/A Poor Acceptable Excellent N/A

Nex

t Ste

ps Participants presented with clear next steps they could apply to continue to improve their skills

Participants provided with list of resources (websites, contact info, etc.) to help implement what was learned in the classroom

Fide

lity

of

Impl

emen

tatio

n

Session covers all topics listed in the syllabus.

Session goals and objectives are the same as those stated in syllabus.

Session discusses and/or incorporates all required readings for session.

Participants are required to complete all homework for session as described in the syllabus.


Complete this section as you are completing the online session activities. When you observe a listed action, note the duration, if applicable. You do not need to describe the activity itself unless it is of particular importance, in which case, you may use the notes section.

Duration Duration Duration Duration Duration

Allo

catio

n of

Tim

e

PowerPoint presentation

Participant discussions

Individual work

Practice applying strategies to classroom instruction



Notes

Acce

ssib

ility

Ah-ha moments

Demonstrating understanding of key concepts and ideas during course

Notes


Instructor Interview Protocol

_____________________________________________________________________District: Interviewer:Interviewee(s): Date/Time:

Background Information 1. Please briefly describe your background as a PD provider. Prompts: What is your ESL background? Have you previously provided PD on ELLs?

2. Could you please describe your role in RETELL? When did you first become involved as an instructor?

3. Were you involved in the development of the RETELL course content? [If yes] In what capacity? [If no] What process did you use to become familiar with it?

4. What is different in the RETELL course from PD you have offered before? What is similar?

Structure and Content of the RETELL PD5. How much leverage do you have in how you structure your RETELL course? Prompts: Is it heavily scripted? Are you expected to fill in a lot of the information?

6. How is the pace of the RETELL course? Prompts: Is it easy or difficult to cover all of the topics in the syllabus in each session? Are there topics you’ve had to skip, and if so, how have you chosen which topics to skip? Do the topics seem evenly distributed throughout the sessions?

7. How is the sequence of the RETELL topics?Prompts: Are the topics sequenced in a way that makes sense to you? If not, how would you rearrange them? Have you re-sequenced any of the topics? Is there too much, too little, or adequate repetition of important information?

8. What is the content focus of the RETELL PD?Examples: linguistic training particular to English language acquisition; explicit curricular and instructional strategies/best practices for teaching English to ELLs; cultural or community sensitivity/culturally responsive pedagogy

9. Could you describe the content in more detail? Examples/prompts: Is there training with models or approaches that have been shown to be effective with ELLs (e.g., modeling, using graphic organizers, contextualizing, using multiple modes of communication, using SIOP)? What instructional strategies are emphasized as important for ELLs’ acquisition of subject matter? Is data-based assessment of learner needs covered? Differentiated instruction? Are materials of relevance to ELLs incorporated?

10. What steps, if any, do you take to ensure the quality of the RETELL course you deliver?


11. What steps, if any, do you take to ensure that the RETELL sessions are delivered as they were designed?

12. From your perspective, which aspects of the RETELL content do teachers struggle with most? What is easiest for them?

13. From your perspective, what are [name of district] teachers’ greatest needs when it comes to PD on ELLs? Why?

Prompt: What is the capacity of teachers in the district to address the needs of ELLs? What are the challenges that teachers in the district face in working with ELLs?

14. Does the RETELL PD meet these teachers’ needs? Why or why not?

15. What has been the most successful aspect of delivering the RETELL material?

16. What has been the most challenging aspect of delivering this material?

Instructor Experiences17. Have you encountered any challenges with course logistics? If yes, please describe.Examples: internet access, timing, availability of the room, size of the room, communication with ESE, communication with participants

18. What kinds of support have you received in delivering the RETELL sessions? How has it helped you, if at all?

19. Do you feel you need additional support from ESE or from the district? [If yes] What kind of support do you most need?

20. What kinds of support are/would be most useful to you as a RETELL instructor?Prompt: Are there particular types of support that would improve the quality or fidelity of the course?

21. Do you receive formal or informal feedback as a RETELL instructor? [If yes] What kind of feedback do you receive, and how useful is it?

22. What kinds of feedback are/would be most useful to you as a RETELL instructor? Prompt: Are there particular types of feedback that would improve the quality or fidelity of the course?

Project Outcomes23. Have there been any observable effects of the RETELL PD on the teachers? Please

describe what you have observed.

24. [If effects are noted:] What factors or conditions do you think supported these changes?

25. How would you assess the effectiveness of the participants who have completed the program to work with ELLs?


Wrap-up26. Is there anything else you would like to tell me that might help ESE improve RETELL

and increase its impact?

Those are all the questions I have. Thank you very much for your time!


Participant Focus Group Protocol

This focus group will gather information about the teacher survey. Members of the focus group will each fill out the survey, and discuss each item as they answer it.

1. Say: “Thank you for agreeing to participate in this focus group. We are trying to gather information about our teacher survey. Specifically, we’re trying to find out if the questions give us the information we’re looking for. I have given you each a survey. Please answer the first item.”

2. When all participants have finished the first item, say, “I’m going to ask you a couple of questions about the first item.” Ask the following questions:

What did you think this item was asking you to do? Was there anything you found challenging about providing an answer for this item?

If yes, What was challenging, and why? Were there any other options that should have been included? Were there any options that didn’t make sense to include? Why?

3. Repeat for each multiple choice item. For each open ended question, ask, What did you think this item was asking you to do? Was there anything you found challenging about providing an answer for this item?

If yes, What was challenging, and why? Do you think this could be posed as a multiple choice question?

If yes, What would the options be?

If no, Why not?


Participant Survey

PART I: Face-to-Face Portion of the RETELL Course This first set of questions asks about your face-to-face time in class with an instructor.

1. Select the one rating that best reflects your opinion of the pace of the course.

Much too slow A little too slow Just right A little too fast Much too fast

Comments:

2. Select the rating that best reflects how well time was allocated to each of the activities listed below. Select only one rating for each activity.

Too Little Time The Right Amount of Time

Too Much Time

Instructor presenting/lecturing ○ ○ ○Small group work/discussions ○ ○ ○Discussion around how to apply strategies being learned to classroom instruction ○ ○ ○Practice during the course applying strategies to classroom instruction ○ ○ ○Whole class discussion ○ ○ ○Interacting with colleagues ○ ○ ○Individual work (in class) ○ ○ ○Individual work (homework) ○ ○ ○Reading ○ ○ ○Videos ○ ○ ○Comments:

3. Select the rating that best reflects how useful each of the activities listed below was. Select only one rating for each activity.


Not Useful At All

A Little Useful

Pretty Useful Extremely Useful

Unknown

Instructor presenting/lecturing ○ ○ ○ ○ ○Small group work/discussions ○ ○ ○ ○ ○Discussion around how to apply strategies being learned to classroom instruction ○ ○ ○ ○ ○Practice during the course applying strategies to classroom instruction ○ ○ ○ ○ ○Whole class discussion ○ ○ ○ ○ ○Interacting with colleagues ○ ○ ○ ○ ○Individual work (in class) ○ ○ ○ ○ ○Individual work (homework) ○ ○ ○ ○ ○Reading ○ ○ ○ ○ ○Videos ○ ○ ○ ○ ○Comments:


PART II: Online Portion of the RETELL CourseThis set of questions asks you about your experience with the online portion of the RETELL course.

4. Select the average amount of time each of the following activities took you while completing each online session. Select only one time range for each activity.

< 30 Min. per Session

30-60 Min. per Session



> 120 Min. per Session

PowerPoint presentations ○ ○ ○ ○ ○Activities ○ ○ ○ ○ ○Assignments ○ ○ ○ ○ ○Interactions with the instructor ○ ○ ○ ○ ○Illustration of how strategies can be applied to the classroom

○ ○ ○ ○ ○Reading ○ ○ ○ ○ ○Posting comments and responses online ○ ○ ○ ○ ○Comments:

5. Select one rating that best reflects how useful each of the following activities was. Select only one rating for each activity.

Not Useful At All

A Little Useful

Pretty Useful Extremely Useful

Unknown

PowerPoint presentations ○ ○ ○ ○ ○Activities ○ ○ ○ ○ ○Assignments ○ ○ ○ ○ ○Interactions with the instructor ○ ○ ○ ○ ○Illustration of how strategies can be applied to the classroom

○ ○ ○ ○ ○Reading ○ ○ ○ ○ ○Posting comments and responses online ○ ○ ○ ○ ○Comments:

6. Select one rating that best reflects how strongly you agree or disagree with each of the statements listed below. Select only one rating for each statement.


Strongly Disagree

Disagree Agree Strongly Agree

Not Applicable

The PowerPoint presentations were easy to follow. ○ ○ ○ ○ ○I felt as engaged in the online content as I was in the face-to-face session. ○ ○ ○ ○ ○I found it easy to navigate the course materials. ○ ○ ○ ○ ○I used the online platform to communicate with other participants when not required. ○ ○ ○ ○ ○The instructor made a clear connection between the online and face-to-face content. ○ ○ ○ ○ ○Comments:

7. Please tell us what changes you think would make the online portion of the RETELL course more useful for educators. Please be as candid as possible.


PART III: Instructor, Course Activities, and Classroom CultureThis set of questions asks you to rate the instructor, materials, and classroom culture of the RETELL course.

8. Select the rating that best reflects how strongly you agree or disagree with each of the following statements. Select only one rating for each statement.

Strongly Disagree


It was easy to see the instructor’s presentation. ○ ○ ○ ○The room was clean. ○ ○ ○ ○The room was comfortable. ○ ○ ○ ○The organization of the room facilitated collaboration with my colleagues. ○ ○ ○ ○Comments:

9. Select the rating that best reflects how often the lead instructor did each of the activities listed below. Select only one rating for each activity.

Never Rarely Sometimes Frequently Not Applicable

Slowed down or moved on based upon the mood in the room ○ ○ ○ ○ ○Demonstrated excitement about the content ○ ○ ○ ○ ○Engaged participants to ask questions and/or share expertise ○ ○ ○ ○ ○Checked for understanding ○ ○ ○ ○ ○Posted course goals and objectives ○ ○ ○ ○ ○Comments:


10. Select the rating that best reflects how well the instructor did each of the following. Select only one rating for each activity.

Poorly Adequately Well Very well Did Not Observe

Summarized readings ○ ○ ○ ○ ○Gave instructions ○ ○ ○ ○ ○Facilitated discussions ○ ○ ○ ○ ○Responded to questions ○ ○ ○ ○ ○Handled difference of opinion ○ ○ ○ ○ ○Linked instructional materials (e.g., readings) to the main ideas being presented ○ ○ ○ ○ ○Comments:

11. Select the rating that best reflects whether and how often each of the following occurred. Select only one rating for each statement.

Never Rarely Sometimes Frequently Not Applicable

I did not receive clarification from the instructor about something that confused me. ○ ○ ○ ○ ○I was uncertain about what to do during an activity after instructions were given. ○ ○ ○ ○ ○I did not get the opportunity to share my comments. ○ ○ ○ ○ ○I did not feel the main ideas from a reading were clear, even after discussing it in class. ○ ○ ○ ○ ○My questions to the instructor were not answered thoroughly. ○ ○ ○ ○ ○Comments:

12. What could improve the instruction/pedagogy of the course?


PART IV: The RETELL ReadingsThis set of questions asks you specifically about the course required readings.

13. Select the rating that best reflects how strongly you agree or disagree with each of the following statements. Select only one rating for each statement.

Strongly Disagree


In general, the readings were too academic. ○ ○ ○ ○In general, the readings were related to the course content. ○ ○ ○ ○In general, the readings were easy to understand. ○ ○ ○ ○In general, the readings were applicable to my instruction of English language learners. ○ ○ ○ ○In general, the readings helped me to better understand sheltered English immersion. ○ ○ ○ ○

14. The following is a list of all required course readings. Please mark only the readings you found particularly useful or problematic. It is not necessary to mark every reading.

Problematic Useful

Session 1: Massachusetts Department of Elementary and Secondary Education. (2012). User guide for DART detail: English language learners.

○ ○

Session 2: DeCapua, A., & Marshall. H. W. (2011). Reaching ELLs at risk: Instruction for students with limited or interrupted formal education.

○ ○

Session 2: Robertson, K. & Lafond, S. (2008). How to support ELL Students with Interrupted Formal Education.

○ ○

Session 3: Colorin Colorado. (2007). How to Reach Out to Parents of ELLs. ○ ○Session 3: Dilg, M. (2003). From home to school and home again. ○ ○Session 3: Harper, C. & de Jong, E. (2004). Misconceptions about teaching English language learners.

○ ○

Session 4: Brown, H. (2000). Section on “Communicative Competence.” In Principles of language learning and teaching

○ ○

Session 4: Crawford, J. (2004). Chapter 8: Basic Research in Language Acquisition.

○ ○

Session 4: Meyer, C. (2009). The study of language. ○ ○Session 5: Peregoy, S. F. & Boyle, O. F. (2008). Second language acquisition. ○ ○Session 5: Finegan, E. (2004). Acquiring first and second languages. ○ ○Session 6: Calderon, M., August, D., Cheung, A., Durán, D., & Madden, N. (2005). Bringing words to life in classrooms with English language learners.

○ ○

Session 6: Lems, K., Miller, L. D., & Soro, T. M. (2010). ELL oracy: Listening comprehension and oral language development.

○ ○

Session 7: Carlo, M. S., August, D., & Snow, C. (2005). Sustained vocabulary-learning strategy instruction for English Language Learners.

○ ○


Problematic Useful

Session 7: Silbold, C. (2011) Building English language learners’ academic vocabulary: Strategies & tips.

○ ○

Session 8: de Jong, E. J. & Harper, C. A. (2005). Preparing mainstream teachers for English language learners: Is being a good teacher good enough?

○ ○

Session 8: Echevarria, J. & Graves, A. (2011). Sheltered instruction in the content areas.

○ ○

Session 9: Himmel, J. (2012). Language objectives: The key to effective content area instruction for ELs.

○ ○

Session 9: Varela, E. (2010). Mainstreaming ELLs into grade-level classes. ○ ○Session 10: Coleman, R. & Goldenberg, C. (2010). What does research say about effective practices for ELLs?

○ ○

Session 10: Gottlieb, M., Katz, A. & Ernst-Slavit, G. (2009). Rethinking the language of school.

○ ○

Session 11: Misco, T. & Castaneda, M.E. (2009). Now what should I do for ELLs? : Reconceptualizing social studies curriculum design for ELL education.

○ ○

Session 11: Schleppegrell, M. J. (2001). Linguistic features of the language of schooling.

○ ○

Session 12: Abedi, J. (2002). Assessment and accommodation of English language learners: Issues, concerns, and recommendations.

○ ○

Session 12: Gottlieb, M. (2006). Classroom assessment. ○ ○

Session 13: Cohen, J. (2007). A case study of a high school English language learner and his reading.

○ ○

Session 13: Singhal, M. (2005). Process, practice, and pedagogy. ○ ○Session 14: Brisk, M.E., Horan, D.A., & McDonald, E. (2007). A scaffolded approach to learning to write.

○ ○

Session 14: DelliCarpini, M. (2009). Writing in the ESL classroom: Confessions of a guilty teacher.

○ ○

Session 15: Echevarria, J., Vogt, M., & Short, D. J. (2004). Introducing sheltered instruction.

○ ○

Session 15: World-Class Instructional Design and Assessment (WIDA). (2012). Focus on differentiation part 1.

○ ○

Comments:


PART V: All Aspects of the RETELL CourseThis final set of questions asks you to consider the course as a whole.

15. Select one rating that best reflects how strongly you agree or disagree with each of the statements listed below. Select only one rating for each statement.

Strongly Disagree


My instructor is knowledgeable about sheltered English immersion. ○ ○ ○ ○I was taught skills that I can apply directly to in my own classroom. ○ ○ ○ ○I was given opportunities to practice instructional strategies. ○ ○ ○ ○The content of this course enhances what I already know about sheltered English immersion. ○ ○ ○ ○The content of this course repeats what I’ve already learned about sheltered English immersion. ○ ○ ○ ○This course was a good use of my time. ○ ○ ○ ○Comments:

16. Select the rating that most accurately reflects how relevant you found each of the following aspects of the course for your instruction of English language learners. Select only one rating for each statement.

Not Relevant

A Little Relevant

Pretty Relevant

Very Relevant

Did not occur

The instructor’s face-to-face presentations ○ ○ ○ ○ ○Interactions with colleagues during the face-to-face portion of the course ○ ○ ○ ○ ○The online portion of the course ○ ○ ○ ○ ○Posting comments and responses online ○ ○ ○ ○ ○Interactions with colleagues during the online portion of the course (e.g., e-mails, comments) ○ ○ ○ ○ ○Online interactions with the instructor apart from the online presentations ○ ○ ○ ○ ○Comments:


17. The following is a list of all course sessions. Please mark only the sessions you found particularly useful for providing instruction to English language learners. It is not necessary to mark every session.

Particularly Useful for ELL Instruction

Session 1: Examining Data and Policies Relevant to English Language Learners ○Session 2: Diversity within English Language Learner Populations ○Session 3: Cultural and Social Aspects of Teaching in the SEI Classroom and School ○Session 4: The Role of Language and Linguistics in Instructing ELLs ○Session 5: Second Language Acquisition in the SEI Classroom ○Session 6: Oral Language, Listening Comprehension and Vocabulary Development in the SEI Classroom ○Session 7: The Role of Vocabulary Development in Unlocking Content ○Session 8: Sheltering Content for ELLs in the Classroom I ○Session 9: Sheltering Content for ELLs in the Classroom II ○Session 10: Characteristics of Academic Language ○Session 11: Specialized Academic Content Language ○Session 12: Assessment of ELLs in the SEI Classroom ○Session 13: Literacy: Reading in the SEI Classroom ○Session 14: Literacy: Writing in the SEI Classroom ○Session 15: The Benefits of Standards-Based Teaching for ELLs ○Session 16 (if completed): Effective Practices for the SEI Classroom ○Comments:


18. Select the rating that best reflects how much you think the course so far has contributed to your understanding of each of the following topics. Select only one rating for each statement.

Not At All A Little A Fair Amount A Great Deal Topic not

addressed

How to meet the needs of ELLs in the classroom ○ ○ ○ ○ ○Vocabulary instruction for ELLs ○ ○ ○ ○ ○How ELLs acquire a second language ○ ○ ○ ○ ○The social and cultural issues that affect ELL students ○ ○ ○ ○ ○How to apply sheltered English immersion to reading and writing instruction in the classroom

○ ○ ○ ○ ○

How to differentiate instruction when working with ELLs at various English proficiency levels ○ ○ ○ ○ ○How to address the needs of diverse ELL populations ○ ○ ○ ○ ○How to assess ELLs in the content areas ○ ○ ○ ○ ○How to teach academic language to ELLs ○ ○ ○ ○ ○Comments:

19. Given your expertise as a teacher, of the topics presented so far are there any that should have been delved into deeper even at the expense of skipping other topics? If so, please briefly explain why.

20. What did you like most about this course? What would you have liked to do more of?

21. Is there anything else you would like to share with us to help us improve the RETELL course?


Appendix B: Participant Survey ResultsPART 1: FACE-TO-FACE PORTION OF THE RETELL COURSE 1. Select the one rating that best reflects your opinion of the pace of the course.

Location nMuch too Slow

A little too slow

Just Right

A little too fast

Much too fast

Somerville 18 0% 0% 11% 50% 39%Lowell 15 0% 0% 0% 40% 60%Boston/Worcester 22 0% 0% 27% 32% 41%

Total 55 0% 0% 15% 40% 45%

2a. Time allocation to Instructor presenting/lecturing

Location nToo little time

Right amount

Too much time

Somerville 18 33% 61% 6%Lowell 15 0% 93% 7%Boston/Worcester 23 22% 65% 13%

Total 56 20% 71% 9%

2b. Time allocation to small group work/discussions


Right amount

Too much time

Somerville 18 33% 67% 0% Lowell 16 13% 88% 0%Boston/Worcester 23 17% 78% 4%

Total 57 21% 77% 2%2c. Time allocation to discussion around how the apply strategies being learned to classroom instruction


Right amount

Too much time


Total 56 55% 41% 4%2d. Time allocation to practice applying strategies to classroom instruction


Right amount

Too much time


Total 56 61% 39% 0%2e. Time allocation to whole class discussion.


Right amount

Too much time


Total 57 9% 84% 7%


2f. Time allocation to interacting with colleagues.


Right amount

Too much time


Total 57 19% 79% 2%2g. Time allocation to individual work (in class).


Right amount

Too much time


Total 55 44% 53% 4%

2h. Time allocation to individual work (homework).


Right amount

Too much time


Total 57 33% 9% 58%2i. Time allocation to reading.


Right amount

Too much time


Total 56 23% 20% 57%2j. Time allocation to videos.


Right amount

Too much time


Total 48 38% 63% 0%3a. Usefulness of instructor presenting/lecturing.

Location nNot useful at all

A little useful

Pretty useful

Extremely useful Unknown


Total 54 2% 13% 39% 46% 0%


3b. Usefulness of small group work/discussions.


A little useful

Pretty useful



Total 57 0% 18% 46% 37% 0%3c. Usefulness of discussion around how to apply strategies being learned to classroom instruction.


A little useful

Pretty useful



Total 56 0% 18% 34% 45% 4%3d. Usefulness of practice during the course applying strategies to classroom instruction.


A little useful

Pretty useful



Total 56 5% 20% 30% 41% 4%3e. Usefulness of whole class discussion.


A little useful

Pretty useful



Total 56 2% 18% 55% 25% 0%3f. Usefulness of interacting with colleagues.


A little useful

Pretty useful



Total 57 0% 16% 44% 40% 0%3g. Usefulness of individual work (in class).


A little useful

Pretty useful



Total 56 9% 27% 41% 16% 7%


3h. Usefulness of individual work (homework).


A little useful

Pretty useful



Total 54 11% 48% 33% 6% 2%3i. Usefulness of reading.


A little useful

Pretty useful



Total 56 4% 30% 55% 11% 0%3j. Usefulness of videos.


A little useful

Pretty useful



Total 50 10% 22% 44% 14% 10%

PART 2: ONLINE PORTION OF THE RETELL COURSE4a. Average amount of time for PowerPoint presentations

Location n <30 min. 30-60 min.60-90 min.

90-120 min. >120 min.


Total 56 29% 39% 18% 5% 9%4b. Average amount of time for activities.


90-120 min. >120 min.


Total 55 7% 29% 22% 18% 24%4c. Average amount of time for assignments.


90-120 min. >120 min.


Total 55 2% 15% 24% 24% 36%


4d. Average amount of time for interactions with the instructor.


90-120 min. >120 min.


Total 49 76% 14% 4% 2% 4%4e. Average amount of time for illustration of how strategies can be applied to the classroom.


90-120 min. >120 min.


Total 49 33% 37% 16% 8% 6%4f. Average amount of time for reading.


90-120 min. >120 min.


Total 56 5% 9% 34% 27% 25%4g. Average amount of time for posting comments and responses online.


90-120 min. >120 min.


Total 56 11% 43% 23% 9% 14%5a. Usefulness of PowerPoint presentations


A little useful

Pretty useful



Total 57 12% 25% 42% 21% 0%5b. Usefulness of activities.


A little useful

Pretty useful



Total 54 6% 24% 48% 20% 2%5c. Usefulness of assignments.


A little useful

Pretty useful



Total 56 7% 39% 41% 13% 0%


5d. Usefulness of interactions with the instructor.


A little useful

Pretty useful



Total 56 9% 7% 41% 41% 2%5e. Usefulness of illustration of how strategies can be applied to the classroom.


A little useful

Pretty useful



Total 56 0% 29% 29% 38% 5%5f. Usefulness of reading.


A little useful

Pretty useful



Total 57 7% 26% 54% 12% 0%5g. Usefulness of posting comments and responses online.


A little useful

Pretty useful



Total 56 32% 39% 21% 5% 2%6a. The PowerPoint presentations were easy to follow.

Location nStrongly disagree Disagree Agree

Strongly agree N/A


Total 57 5% 12% 56% 26% 0%6b. I felt as engaged in the online content as I was in the face-to-face session.


Strongly agree N/A


Total 57 39% 47% 12% 2% 0%6c. I found it easy to navigate the course materials.


Strongly agree N/A


Total 57 37% 46% 14% 4% 0%


6d. I used the online platform to communicate with other participants when not required.


Strongly agree N/A


Total 56 50% 38% 7% 0% 5%6e. The instructor made a clear connection between the online and face-to-face content.


Strongly agree N/A


Total 57 11% 25% 54% 11% 0%

PART 3: INSTRUCTOR, COURSE ACTIVITIES, AND CLASSROOM CULTURE8a. It was easy to see the instructor’s presentation.


Strongly agree

Somerville 18 0% 6% 50% 44%Lowell 15 0% 0% 73% 27%Boston/Worcester 23 0% 0% 43% 57%

Total 56 0% 2% 54% 45%8b. The room was clean.


Strongly agree


Total 56 2% 0% 50% 48%8c. The room was comfortable.


Strongly agree


Total 57 2% 14% 53% 32%8d. The organization of the room facilitated collaboration with my colleagues.


Strongly agree


Total 57 0% 5% 53% 42%


9a. How often the instructor slowed down or moved on based upon the mood in the room.

Location n Never RarelySometimes Frequently N/A


Total 57 2% 7% 32% 60% 0%


9b. How often the instructor demonstrated excitement about the content.



Total 57 0% 2% 5% 93% 0%9c. How often the instructor engaged participants to ask questions and/or share expertise.



Total 57 0% 4% 14% 82% 0%9d. How often the instructor checked for understanding.



Total 57 0% 4% 25% 72% 0%9e. How often the instructor posted course goals and objectives.



Total 57 4% 7% 11% 75% 4%10a. How well the instructor summarized readings.

Location n Poorly Adequately Well Very WellDid not observe


Total 56 4% 30% 43% 21% 2%10b. How well the instructor gave instructions.



Total 54 7% 20% 44% 28% 0%10c. How well the instructor facilitated discussions.



Total 57 2% 16% 40% 42% 0%


10d. How well the instructor responded to questions.



Total 57 2% 12% 42% 44% 0%10e. How well the instructor handled difference of opinion.



Total 57 4% 5% 30% 56% 5%10f. How well the instructor linked instructional materials (e.g., readings) to the main ideas being presented.



Total 57 2% 23% 23% 53% 0%11a. I did not receive clarification from the instructor about something that confused me.



Total 57 42% 28% 23% 2% 5%11b. I was uncertain about what to do during an activity after instructions were given.



Total 57 9% 28% 42% 19% 2%

11c. I did not get the opportunity to share my comments.



Total 57 56% 33% 4% 2% 5%


11d. I did not feel the main ideas from a reading were clear, even after discussing it in class.



Total 56 30% 34% 25% 7% 4%11e. My questions to the instructor were not answered thoroughly.



Total 56 59% 23% 13% 2% 4%

PART 4: THE RETELL READINGS 13a. In general, the readings were too academic.


Strongly agree


Total 55 7% 36% 40% 16%13b. In general, the readings were related to the course content.


Strongly agree


Total 56 5% 4% 77% 14%13c. In general, the readings were easy to understand.


Strongly agree


Total 55 11% 38% 45% 5%

13d. In general, the readings were applicable to my instruction of English language learners.


Strongly agree


Total 54 2% 26% 67% 6%


13e. In general, the readings helped me to better understand sheltered English immersion.

Location nStrongly disagree Disagree

Agree

Strongly agree


Total 55 9% 11%67% 13%

14. Usefulness of readingsReading n % Problematic % UsefulS1. ESE (2012) 31 16% 84%S2. DeCapua & Marshall (2011) 37 11% 89%S2. Robertson & Lafond (2008) 33 15% 85%S3. Colorin Colorado (2007) 37 14% 86%S3. Dilg (2003) 34 21% 79%S3. Harper & de Jong (2004) 39 8% 92%S4. Brown (2000) 32 38% 63%S4. Crawford (2004) 32 50% 50%S4. Meyer (2009) 31 52% 48%S5. Peregoy & Boyle (2008) 33 30% 70%S5. Finegan (2004) 35 34% 66%S6. Calderon et al. (2005) 38 18% 82%S6. Lems et al. (2010) 29 34% 66%S7. Carlo et al. (2005) 30 30% 70%S7. Silbold (2011) 38 16% 84%S8. de Jong & Harper (2005) 32 19% 81%S8. Echevarria & Graves (2011) 30 30% 70%S9. Himmel (2012) 31 26% 74%S9. Varela (2010) 25 32% 68%S10. Coleman & Goldenberg (2010) 30 30% 70%S10. Gottlieb et al. (2009) 33 36% 64%S11. Misco & Castaneda (2008) 26 19% 81%S11. Schleppegrell (2001) 30 63% 37%S12. Abedi (2002) 33 21% 79%S12. Gottlieb (2006) 37 19% 81%S13. Cohen (2007) 31 35% 65%S13. Singhal (2005) 29 34% 66%S14. Brisk et al. (2007) 32 28% 72%S14. DelliCarpini (2009) 32 19% 81%S15. Echevarria et al. (2004) 30 27% 73%S15. WIDA (2012) 30 23% 77%


PART 5: ALL ASPECTS OF THE RETELL COURSE 15a. My instructor is knowledgeable about sheltered English immersion.


Strongly agree


Total 57 0% 0% 35% 65%15b. I was taught skills that I can apply directly to my own classroom.


Strongly agree


Total 54 2% 11% 52% 35%

15c. I was given opportunities to practice instructional strategies.


Strongly agree


Total 56 4% 30% 45% 21%15d. The content of this course enhances what I already know about sheltered English immersion.


Strongly agree


Total 57 2% 11% 65% 23%15e. The content of this course repeats what I’ve already learned about sheltered English immersion.


Strongly agree


Total 54 6% 33% 39% 22%15f. This course was a good use of my time.


Strongly agree


Total 56 11% 27% 41% 21%


16a. Relevance of the instructor’s face-to-face presentations for instruction of ELLs.

Location nNot relevant

A little relevant

Pretty relevant

Very relevant

Did not occur


Total 57 0% 5% 39% 56% 0%16b. Relevance of interactions with colleagues during the face-to-face portion of the course for instruction of ELLs.


A little relevant

Pretty relevant

Very relevant

Did not occur


Total 57 2% 11% 30% 58% 0%16c. Relevance of the online portion of the course for instruction of ELLs.


A little relevant

Pretty relevant

Very relevant

Did not occur


Total 57 25% 39% 28% 9% 0%16d. Relevance of posting comments and responses online for instruction of ELLs.


A little relevant

Pretty relevant

Very relevant

Did not occur


Total 57 33% 40% 16% 11% 0%16e. Relevance of interactions with colleagues during the online portion of the course for instruction of ELLs.


A little relevant

Pretty relevant

Very relevant

Did not occur


Total 56 43% 27% 20% 5% 5%

16f. Relevance of online interactions with the instructor apart from the online presentations for instruction of ELLs.


A little relevant

Pretty relevant

Very relevant

Did not occur


Total 56 25% 18% 21% 13% 23%


17. Usefulness of sessions for ELL instruction.Somervill

e LowellBoston/

Worcester

Useful Useful UsefulTotal

Session 1 10 4 12 26Session 2 5 4 11 20Session 3 12 9 17 38Session 4 6 3 14 23Session 5 10 11 14 35Session 6 11 9 13 33Session 7 13 11 16 40Session 8 7 11 10 28Session 9 9 9 8 26Session 10 9 9 11 29Session 11 8 3 9 20Session 12 13 6 13 32Session 13 15 9 13 37Session 14 13 10 15 38Session 15 5 4 8 17Session 16 3 2 5 10

18a. The course contributed to my understanding of how to meet the needs of ELLs in the classroom.

Location n Not at all A littleA fair amount

A great deal

Topic not addressed


Total 55 4% 18% 38% 40% 0%18b. The course contributed to my understanding of vocabulary instruction for ELLs.


A great deal

Topic not addressed


Total 55 2% 16% 24% 58% 0%


18c. The course contributed to my understanding of how ELLs acquire a second language.


A great deal

Topic not addressed


Total 55 5% 13% 45% 36% 0%18d. The course contributed to my understanding of the social and cultural issues that affect ELL students.


A great deal

Topic not addressed


Total 55 5% 16% 36% 42% 0%18e. The course contributed to my understanding of how to apply SEI to reading and writing instruction in the classroom.


A great deal

Topic not addressed


Total 54 2% 26% 35% 37% 0%18f. The course contributed to my understanding of how to differentiate instruction when working with ELLs at various English proficiency levels.


A great deal

Topic not addressed


Total 55 5% 31% 31% 33% 0%18g. The course contributed to my understanding of how to address the needs of diverse ELL populations.


A great deal

Topic not addressed


Total 55 7% 25% 35% 33% 0%18h. The course contributed to my understanding of how to assess ELLs in the content areas.


A great deal

Topic not addressed


Total 56 7% 29% 34% 30% 0%18i. The course contributed to my understanding of how to teach academic language to ELLs.


A great deal

Topic not addressed


Total 55 0% 22% 45% 33% 0%


1000 Thomas Jefferson Street NWWashington, DC 20007-3835202.403.5000 | TTY: 877.334.3499www.air.org

· web viewshort activities in the course that explore what participants already know about...

Documents