k–3 formative assessment enhanced assessment grant: final … · 2019. 8. 15. · the field test...

K–3 Formative Assessment Enhanced Assessment Grant: Final Project Report September 2017

September 29, 2017

Prepared for: Cindy Bagwell North Carolina Department of Public Instruction Contract No. NC10130204 Prepared by: SRI International Suzanne Raber Sara Thayer Megan Cox Kathleen Hebbeler Traci Kutaka Suggested citation: Raber, S., Thayer, S., Cox, M., Hebbeler, K., & Kutaka, T. (2017). K–3 Formative Assessment Enhanced Assessment Grant: Final Project Report. Menlo Park, CA. SRI International. Acknowledgements We would like to thank the dedicated teachers and administrators who participated in the cognitive lab, pilot, and field test phases of this research as well as the state consortium members who provided their guidance to the development and enhancement of high quality formative assessment in early elementary classrooms: Amy, Dan and Steve in North Carolina; Lauren, Nicol, Pat and Leila in Arizona; Penny, Jennifer and Kimberly in Iowa; Jaci, Lee Anne and Sue in Maine; and Judi, Michelle and Diane in Rhode Island. We would especially like to thank Dr. Cindy Bagwell for her steadfast and inspiring leadership of this project.

This document was developed under a grant from the U.S. Department of Education. However, the contents do not necessarily represent the policy of the Department and you should not assume endorsement by the U.S. Federal Government or the North Carolina State Board of Education.

EAG K-3 Formative Assessment: Final Report September 2017 iii

Contents Executive Summary ................................................................................................................... ix

Background ............................................................................................................................ ix

Findings .................................................................................................................................. ix

Chapter 1: Introduction ................................................................................................................ 1

Project background and overview ............................................................................................ 1

Overview of the assessment .................................................................................................... 2

Initial ECD phases ................................................................................................................... 4

ECD codesign teams ........................................................................................................... 4

Cognitive labs ...................................................................................................................... 5

Pilot test ............................................................................................................................... 5

Field test design ...................................................................................................................... 6

Report organization ................................................................................................................. 8

Chapter 2: Validity of assessment content ................................................................................ 10

Are the skill levels of the construct progressions in the proper sequence? ............................ 10

Teacher survey results ....................................................................................................... 10

Distribution of children’s placements on progressions ........................................................ 11

Psychometric models ......................................................................................................... 14

Do the performance levels accurately reflect what is expected at entry to kindergarten? ....... 17

Is the assessment sensitive to children’s growth and development over time? ...................... 24

Teacher survey results ....................................................................................................... 24

Fall to winter change in placements on the progressions ................................................... 25

Placement on the progressions across grade levels .......................................................... 29

Does the assessment provide information that is useful for instruction for grades K–3? ........ 31

Does the assessment provide information useful for instruction for different subgroups, such as children with disabilities and English learners? ................................................................. 34

Children with disabilities ..................................................................................................... 34

English learners ................................................................................................................. 35

Do the assessment results vary for other subgroups (gender, race)? .................................... 35

Gender ............................................................................................................................... 36

Race/Ethnicity .................................................................................................................... 36

Chapter 3: Implementation of assessment process ................................................................... 38

Can teachers learn to implement the assessment with fidelity? ............................................. 38

EAG K-3 Formative Assessment: Final Report September 2017 iv

Implementation of the recommended assessment process ................................................ 39

Teacher confidence in status summary .............................................................................. 41

Teacher reliability in coding status summaries ................................................................... 42

Incorporating the assessment into classroom routines .......................................................... 46

Chapter 4: Implementation supports for teachers ...................................................................... 49

Professional development ..................................................................................................... 49

Materials and ongoing supports ............................................................................................. 51

Materials ............................................................................................................................ 51

Ongoing supports ............................................................................................................... 51

Teachers’ perceptions of the supports ............................................................................... 52

Technology ............................................................................................................................ 54

Chapter 5: Broader implementation supports ............................................................................ 56

Professional development for administrators ......................................................................... 56

Building-level infrastructure ................................................................................................... 58

Classroom conditions ............................................................................................................ 59

Technology available to support the assessment ............................................................... 59

Teacher training and preparation to implement formative assessment ............................... 60

Class time spent in different learning arrangements ........................................................... 60

Chapter 6: Using the K–3 Formative Assessment with all children ............................................ 62

Disability perspective ............................................................................................................. 62

Overview guidance and general recommendations ............................................................ 63

Construct-specific guidance ............................................................................................... 64

English learner perspective ................................................................................................... 67

Race/culture perspective ....................................................................................................... 68

Chapter 7: Summary, revisions, and recommendations ............................................................ 70

Overall assessment findings .................................................................................................. 70

Findings by Construct ............................................................................................................ 73

Language Development & Communication Domain ........................................................... 76

Physical/Motor Development Domain ................................................................................ 77

Cognitive Development Domain ......................................................................................... 77

Social-Emotional Development and Approaches to Learning Domains .............................. 78

Recommended grade levels for each construct ..................................................................... 79

Revisions/recommendations for assessment materials ......................................................... 80

EAG K-3 Formative Assessment: Final Report September 2017 v

Clarify the meaning of the construct ................................................................................... 83

Clarify progression skills (examples, watch fors, assessment means) ................................ 83

Conduct additional research on some constructs ............................................................... 83

Revisions to training materials and recommendations ........................................................... 84

Revisions to teacher and administrator training materials .................................................. 84

Recommendations regarding training ................................................................................. 84

Conclusion ............................................................................................................................ 85

References ............................................................................................................................... 87

Appendices ............................................................................................................................... 89

EAG K-3 Formative Assessment: Final Report September 2017 vi

Exhibits Exhibit Page

1.1. EAG K–3 assessment blueprint ....................................................................................... 3

1.2. Field test sample, by state ............................................................................................... 7

1.3. Field test sample, by grade .............................................................................................. 7

1.4. Demographics of children assessed in field test (Fall) ..................................................... 8

2.1. Unimodal distribution with smooth tails .......................................................................... 12

2.2. Unimodal, smooth tails, skewed to the higher end of the progression ............................ 12

2.3. Unimodal, smooth tails, skewed to the lower end of the progression ............................. 13

2.4. Unimodal, uneven tails .................................................................................................. 13

2.5. Example of Increasing mean “School Readiness” scores for each level of the progression: Emotion Regulation Strategies (Fall Kindergarten) .................................... 15

2.6. Example of skill levels too close together: Following Directions ..................................... 16

2.7. Teacher survey responses for Book Orientation: Percentage of teachers indicating each level ...................................................................................................... 20

2.8. Teacher survey responses and child assessment results for Book Orientation: Percentage of teachers indicating each level ................................................................. 21

2.9. Teacher responses, fall K data and performance levels for Book Orientation: Percentage of teachers indicating each level ................................................................. 22

2.10. Teacher responses and resulting performance levels at kindergarten entry .................. 23

2.11. Teacher ratings of how well construct progressions captured children’s growth ............ 25

2.12. Example comparing fall and winter progression placements: Grip & Manipulation (percentage of children) ................................................................................................. 26

2.13. Examples of fall to winter progression placements of children in kindergarten (percentage of children) ................................................................................................. 27

2.14. Percentage of children at maximum level and grade levels with opportunity for growth for each progression ..................................................................................... 28

2.15. Percent of children not at highest level in fall who were placed at a higher level in winter for each progression ............................................................................... 30

2.16. Example comparing kindergarten and grade 1 progression placements: Percentage of children placed on each level of the progression for Following Directions .................. 31

2.17. Teachers’ rating of constructs as “very” or “somewhat” helpful to instruction ................. 32

2.18. Teachers’ report of benefits of implementing the K–3 Formative Assessment ............... 33

3.1. Percentage of children with status summaries, multiple pieces of evidence, and learning statuses .................................................................................................... 41

EAG K-3 Formative Assessment: Final Report September 2017 vii

3.2. Percentage of teachers confident in progression placement (n = 130) ........................... 42

3.3. Evidence specificity by construct ................................................................................... 45

3.4. Evidence analysis: Percentage of evidence specific enough to code, coded status summary consistent with the teacher status summary, and coded status summary matched the teacher’s exactly ....................................................................................... 46

3.5. Percentage of teachers who reported they were unable to assess constructs during everyday instruction: Most challenging constructs .............................................. 47

3.6. Percentage of teachers who reported each challenge experienced in implementing the assessment ....................................................................................... 48

4.1. Aspects of training teachers new to assessment found most helpful ............................. 50

4.2. Sample page from Teacher Site showing construct materials ........................................ 52

4.3. Percentage of teachers rating materials as very or somewhat helpful ........................... 53

4.4. Percentage of teachers reporting supports provided and suggestions for supports needed for assessment implementation ........................................................................ 54

5.1. Time spent in different classroom learning arrangements .............................................. 61

6.1. Findings from review for each construct by disability ..................................................... 65

7.1. Summary of construct performance for each of the validity claims ................................. 75

7.2. Summary of construct-specific revisions and recommendations .................................... 81

EAG K-3 Formative Assessment: Final Report September 2017 ix

Executive Summary

Background

In 2011, North Carolina’s Department of Public Instruction began work on a formative assessment to be administered at kindergarten entry and throughout kindergarten into grades 1–3 with the primary purpose of informing instruction. In September 2013, North Carolina received an Enhanced Assessment Grant from the U.S. Department of Education to lead a 10-state consortium in the enhancement of this assessment. SRI worked with North Carolina and the other states in the Consortium, along with two other partners, the Build Initiative and Child Trends, to enhance this assessment.

The assessment, known as the K–3 Formative Assessment, consists of 18 construct

progressions in five domains of learning and development. Teachers gather information to

identify the child’s skill level on each progression through their everyday classroom activities.

The teacher captures evidence of what a child can do through notes, work samples, photos, or

videos and then uses this evidence to identify the child’s skill level on the progression. The

teacher also enters both the evidence and the skill level for each construct into a technology

platform. Additional assessment materials available to teachers include background information

on the construct and performance descriptors, examples for each skill level of the progression,

and guidelines on how to assess each construct.

SRI used evidence-centered design (ECD), a highly regarded research-based approach to

assessment development, to enhance the progressions and the assessment materials. The four

phases of the ECD process included review and revisions of the content of the assessment by

ECD codesign teams; cognitive laboratories with teachers to try out the assessment materials

with small numbers of children; a pilot test in K–3 classrooms in four states; and a field test with

additional K–3 classrooms in the same four states.

This report summarizes the results of the field test, which was designed to examine the validity

of the information provided by the assessment and the supports required to implement the

assessment with fidelity. The assessment was field tested in the 2016–17 school year in

Arizona, Iowa, Maine, and Rhode Island with 141 teachers and 1,312 students in kindergarten

through grade 3.

Findings The field test demonstrated that the K–3 Formative Assessment can be implemented by

teachers as intended, provided that teachers have adequate training in the assessment content

and the formative assessment process and have access to functioning technology that supports

EAG K-3 Formative Assessment: Final Report September 2017 x

the assessment. Most of the construct progressions performed well as measures of children’s

knowledge, skills, and abilities, both in terms of the sequencing of the skill levels on the

progression and their sensitivity to growth and development over time. Some of the

progressions presented some challenges for teachers. The analysis of the evidence

documented in the technology platform showed that the reliability with which teachers coded a

skill level for a child varied by construct. Teacher feedback reflected this variation, in that fewer

teachers felt confident about where they placed children on the progressions that had lower

reliability. Teachers also found some constructs more difficult than others to incorporate into

their daily classroom routines. Of particular importance for a formative assessment was the

finding that across constructs, between 20 and 50 percent of teachers did not find a given

construct useful in informing instruction. This may be a characteristic of the assessment that

needs to be improved or it could indicate a need for more professional development around how

to use the results of a formative assessment to inform instruction.

Overall, we were encouraged by the performance of most of the constructs in a field test setting

where most teachers had limited experience with the constructs and formative assessment

process. As teachers continue to work with the K–3 Formative Assessment, we recommend

additional research and professional development on the construct progressions that presented

some issues with their content and proved challenging to implement.

EAG K-3 Formative Assessment: Final Report September 2017 1

Chapter 1: Introduction This is the final report on the K–3 Formative Assessment, which the U.S. Department of Education funded under an Enhanced Assessment Grant in September 2013 to enhance an assessment developed by the North Carolina Department of Public Instruction under its Race to the Top Early Learning Challenge Grant. This report covers the work completed by SRI Education, a division of SRI International, using evidence-centered design.

Project background and overview North Carolina began the original work in 2011 to develop a kindergarten entry assessment

under a Race to the Top Early Learning Challenge Grant (NC K−3 Assessment Think Tank,

2013). North Carolina developed a formative assessment to be administered at kindergarten

entry and throughout kindergarten into grades 1–3 with the primary purpose of informing

instruction. In September 2013 North Carolina received an Enhanced Assessment Grant from

the U.S. Department of Education to lead a 10-state consortium in the further development and

enhancement of this K–3 assessment. The other states were Arizona, Delaware, Iowa, Maine,

North Dakota, Oregon, and Rhode Island, with the District of Columbia and South Carolina

participating as collaborating partners. North Carolina engaged three research partners to

support the enhanced assessment:

• SRI Education to further develop and enhance the assessment

• Child Trends to address longer term implementation of the assessment

• Build Initiative to support stakeholder and family engagement efforts.

SRI used evidence-centered design (ECD), a highly regarded research-based approach to

assessment development (Mislevy & Haertel, 2006), to enhance this assessment. The ECD

process was conducted in four phases over the 4-year project as follows:

1. ECD codesign teams reviewed and revised the assessment materials for each of the

constructs in the assessment.

2. Cognitive laboratories were conducted with teachers to try out the assessment materials

with small numbers of children, and the assessment and materials were revised based on

the findings.

3. The assessment was piloted in K–3 classrooms in four states, and the assessment and

materials were revised based on those findings.

4. The revised assessment was field-tested with additional K–3 classrooms in four states.


The major findings from the first three phases of ECD are summarized below and detailed in

Appendices A–C. This report concentrates on the findings from the field test of the K–3

Formative Assessment, which was conducted during the 2016–17 school year.

Overview of the assessment The application of ECD to the development of learning progressions under the Enhanced

Assessment Grant resulted in 18 construct progressions across five early learning and

development domains, ranging from early developing constructs such as fine motor Grip &

Manipulation to more complex cognitive skills such as Problem Solving (Exhibit 1.1). The

constructs included traditional academic skills like Object Counting and Writing and more

difficult to measure skills like Perseverance. Progressions varied in their recommended grade

range (kindergarten, K–1, K–2, or K–3) and number of skill levels on the progression (e.g., 3 for

the Crossing Midline progression, 13 for Mathematical Patterns).


Exhibit 1.1. EAG K–3 assessment blueprint

Construct Field Test Grades No. of Skills in Progression Assessment Means1 Associated Standards2

Hand Dominance K 3 O,S ELS

Grip & Manipulation K-1 4 O,S ELS

Crossing Midline K-1 3 O,S,T ELS

Gross Motor Development K-3 9 O,S,T ELS

Emotion Expression K-3 7 O ELS

Emotion Regulation Strategies K-3 5 O ELS

Emotional Literacy K-3 9 O,S ELS

Book Orientation K 4 O,S ELS

Print Awareness K-1 7 O,S ELS, CCSS

Letter Naming K-1 7 O,S ELS, CCSS

Following Directions K-3 6 O,S ELS

Writing K-3 7 O,S ELS, CCSS

Reading Comprehension K-3 8 O,S ELS, CCSS

Vocabulary K-3 8 O,S ELS, CCSS

Perseverance K-3 8 O ELS

Object Counting K-2 10 O,S,T ELS, CCSS

Mathematical Patterns K-3 13 O,S ELS, CCSS

Problem Solving K-3 10 O,S ELS

1 Refers to how the construct was assessed: Observation, Situation, Task 2 ELS = Early Learning Standards; CCSS = Common Core State Standards


Initial ECD phases During the first year of the enhancement project, a study was conducted to determine the

alignment of the constructs North Carolina had developed with the consortium states’ early

learning and development standards (Scott-Little, Kagan, Sumrall, & Fox, 2014). Members of

the consortium also identified priority areas for the possible development of constructs to add to

those developed by North Carolina: algebraic thinking and operations, the scientific inquiry

process, and behavior regulation. Project resources were sufficient to develop one additional

construct; Mathematical Patterns was developed as a measure of algebraic thinking.

ECD codesign teams

The North Carolina assessment materials for 15 constructs were submitted to SRI for further

development and enhancement by ECD codesign teams. For each construct, a codesign team

was created of experts in early childhood assessment, ECD, and the respective domain.

Supporting each team were one or two of the North Carolina developers of the original

construct. An SRI technician documented each team’s assessment using the web-based

Principled Assessment Designs for Inquiry system. Some of the constructs were designed to be

assessed at K entry and others were designed for grades K–3. The enhancement process was

to extend other constructs upwards to grade 3 if appropriate.

National domain experts reviewed the construct progressions and suggested revisions. Drawing

on that review, the codesign teams refined the progressions, developed performance

descriptors and examples for each skill on the progression, and refined or developed any

situations and tasks needed to assess a particular construct. This process resulted in the

development of the following materials to support the assessment of each construct:

• A Construct Progression document explaining the background, rationale for assessing,

the overarching understandings, and the specific skills in each progression with

performance descriptors and examples of what each skill looks like in the classroom.

• An Assessment Means Form, essentially an administrator’s manual, covering what a

teacher needs to know to assess a particular construct, including how to observe for the

skills and more structured situations and tasks to pinpoint specific skills.

• Single and Multiple Child Documentation Forms that enable the teacher to record

evidence of where a child falls on each progression, with brief reminders of behaviors to

watch for that differentiate each skill on the progression.

More information on the results of the ECD codesign team work is in Appendix A.


Cognitive labs

The assessment materials the codesign teams developed were tested by 122 K–3 teachers in

five states (Arizona, Iowa, Maine, North Carolina, and Rhode Island). The teachers were trained

on the K–3 formative assessment process and then asked to assess three to five children with

the materials for up to four different constructs during spring and fall 2015.3 The teachers

provided construct-specific feedback on the assessment materials through online surveys and

follow-up interviews. Their feedback was used to further refine the progressions and

accompanying assessment materials for the 2015–16 pilot test. The key findings from the

cognitive labs and the changes made in response were as follows:

• Teachers found the social-emotional constructs most difficult to assess. Recommended

changes were made in the progressions for Emotion Expression, Emotion Regulation

Strategies, and Emotional Literacy.

• Teachers reported some confusion about the Reading Comprehension and Writing

progressions because they contrasted with how reading and writing are assessed in more

traditional assessments. Clarifications were made to the related assessment materials.

More information on the cognitive labs is in Appendix B.

Pilot test

The full assessment was piloted with approximately 80 K–3 teachers and 800 children in four

states (Arizona, Iowa, Maine, Rhode Island) during the 2015–16 school year. Each teacher

assessed a sample of 10 children on all 17 constructs. Mathematical Patterns was not included

because it was developed after the pilot.

Teachers’ use of the assessment was supported by a technology platform. The teachers

entered evidence (such as anecdotal notes, photographs, or videos) from their observations for

each construct and child in the sample. They then determined where on the progression each

piece of evidence fell (preliminary learning status) and, across multiple pieces of evidence over

multiple observations, where to place the child on the progression (status summary). All the

assessment data (evidence, learning statuses, status summaries) could be entered into the

platform for easy review, reporting, and analysis. This process was further supported by an

application that allowed teachers to upload directly to the platform evidence (their notes, photos,

and videos) from a tablet or smart phone to assist in selecting a status summary.

3 The North Carolina staff conducted cognitive labs for Problem Solving, the last construct North Carolina developed, in spring 2016.


Teachers provided feedback on the manageability and usability of the assessment to inform

instruction through online surveys, focus groups, and interviews. They appreciated the

assessment’s focus on the whole child and coverage of multiple domains of learning and

development. They also appreciated being able to store and retrieve the evidence they collected

in the technology platform. Teachers’ other feedback and the response to it were as follows:

• The assessment process was time consuming, and documenting and interpreting multiple

pieces of evidence to determine where a child fell on each progression was cumbersome.

Consequently, the field test training was modified to further demonstrate how formative

assessment observations can fit into daily instructional routines.

• The social-emotional constructs were particularly difficult to assess. Accordingly, the

Emotional Literacy progression was simplified, the assessment materials for Emotion

Expression and Emotional Regulation Strategies were clarified, and more guidance on how

to assess these constructs was added.

• The Vocabulary construct was challenging to assess. Thus, two progression skill levels that

were difficult to observe in the classroom were combined (using vocabulary in different

contexts) and the assessment materials clarified to emphasize that the construct is about

vocabulary concepts rather than vocabulary word lists.

More information on the pilot test is in Appendix C.

Field test design The field test was designed to answer two overarching research questions:

1. Does the K–3 Formative Assessment provide valid information about what children know

and can do?

2. What supports are required to allow the assessment to be implemented with fidelity and

produce valid information?

Both questions address validity in that teachers who cannot implement the assessment well are

not likely to be able to use it to produce valid information about what children know and can do.

The field test sample comprised 141 K–3 teachers and 1,312 children in four states: Arizona,

Iowa, Maine and Rhode Island (Exhibit 1.2).


Exhibit 1.2. Field test sample, by state

State Number of Teachers Number of Children

Arizona 43 399

Iowa 10 101

Maine 50 450

Rhode Island 38 362

Total 141 1,312

Grade K–3 teachers in the four pilot states were asked to assess 10 children in their classroom

on those constructs assigned to their respective grade levels (see Exhibit 1.1), once in the fall

and again in the winter. The sample of teachers and children and the number of constructs

assessed by grade level are shown in Exhibit 1.3.

Exhibit 1.3. Field test sample, by grade

Grade Number of Constructs Number of Teachers Number of Children

Kindergarten 18 89 850

Grade 1 16 22 198

Grade 2 12 15 126

Grade 3 11 15 138

Total 141 1,312

Because the field test states were generally most interested in using this assessment at

kindergarten entry, they were able to recruit more teachers at that grade level. Consequently,

the sample contains far more kindergarten teachers, and the findings reflect the kindergarten

experience more strongly than the other grade levels. In addition, the sample of children

assessed was 68 percent White (Exhibit 1.4), with relatively small numbers of other racial/ethnic

groups, which limited the ability to examine the research questions for these subgroups.


Exhibit 1.4. Demographics of children assessed in field test (Fall)

Subgroups Number Percent

Gender (female) 660 50.3

English learner 102 7.8

Has IEP 113 8.6

Race (non-White) 239 18.2

Asian/Pacific Islander 17 1.3

Black 54 4.1

Hispanic 126 9.6

White 894 68.1

Multi/other 43 3.3

Unknown 178 13.6

Total 1,312

As during the cognitive labs and pilot, teacher feedback in the field test was gathered through

online surveys, focus groups, and interviews. In addition to determine performance levels,

kindergarten teachers were surveyed on where they would expect children entering

kindergarten to fall on each of the progressions, based on their experience assessing the

constructs with kindergarten children in the fall. All survey and interview protocols are in

Appendix D. The child assessment data were collected through the technology platform that

supported teachers’ use of the assessment.

Report organization The report chapters present the findings regarding the two research questions addressed in the

field test. The first research question, Does the K–3 Formative Assessment provide valid

information about what children know and can do, was broken into subquestions that

represented validity claims about the assessment content (Chapter 2) and assessment process

(Chapter 3). The second research question, What supports are required to allow the

assessment to be implemented with fidelity and produce valid information, was addressed

through subquestions about implementation supports for teachers (Chapter 4) and broader

system-level supports (Chapter 5). Chapter 6 summarizes the findings and recommendations of

expert reviews from disability, English learner, and racial/cultural perspectives, and describes


the changes that were made to the assessment based on the reviews. Chapter 7 summarizes

the findings and presents recommendations for possible revisions and future development

research regarding the assessment materials and related training materials. Finally, the report

appendices provide more detail on all four phases of the ECD assessment enhancement

process.


Chapter 2: Validity of assessment content Validity was addressed through a series of research questions that represented validity claims about the assessment content. The questions concerned the sequencing of the skills that make up the progression, performance levels at kindergarten entry, sensitivity of the assessment to change over time, and usefulness of the assessment in informing instruction for children in grades K–3, including children with disabilities and English learners.

Are the skill levels of the construct progressions in the proper sequence? Each progression is intended to be a set of skills that increase in complexity or difficulty with

each step. The sequencing of the skills in the progression was researched and intentionally

addressed as part of the development process. Educators in North Carolina developed the

construct progressions with input from domain experts, curriculum specialists, developmental

psychologists, and teachers, under the guidance of Margaret Heritage, an expert in formative

assessment. Before consideration by the ECD codesign teams, each construct progression had

been reviewed by a nationally known expert in that area of early childhood development to

verify that the progression incorporated the appropriate skill levels in the proper sequence (see

table of domain experts in Appendix E). The experts also had recommended changes as

appropriate. During the codesign process, the order of the skills was reviewed and validated as

we developed performance descriptors and classroom examples for each skill level.

In general, the field test findings indicated that the skill levels for each progression were in the

proper sequence for each grade. Further research is needed to examine some skill levels on

some progressions to determine whether they are sufficiently different to have value for

informing instruction.

The field test provided three kinds of empirical evidence for the sequencing of the skills:

• Teacher survey results

• The distribution of children’s placements on each progression

• An analysis of the child assessment data that produced psychometric models which

examined the ordinality of the skills on each progression.

Teacher survey results

After teachers had used the assessment in the fall and again in the winter, they answered a

question as part of an online survey that asked, Did the order of the levels seem right for all of

the progressions you assessed (that is, the construct progressed from least difficult to most

difficult skill)?


Across all constructs, 87 percent of teachers reported that they believed the levels of the

progressions they assessed were in the correct order. This varied by construct, with 92 to 99

percent of the teachers reporting the skill levels were in the proper sequence for any one

individual progression. The construct with the largest percentage of teachers uncertain about

the order of the skill levels was Gross Motor at 8 percent. Additional information from both the

pilot test and the field test suggested that teachers questioned the relative placements of

galloping (Skills F and G) and skipping (Skill I) for the Gross Motor construct.

Distribution of children’s placements on progressions

If the skills on a progression are in the proper sequence from less to more difficult, certain

patterns should be seen in the distributions of the children’s placements on the progression. For

constructs that children would be expected to master in kindergarten, the mode of the

distribution should be skewed to the highest skill level on the progression. For constructs that

are designed to extend well into grade 3, the distribution should extend across the skill levels.

For all of the distributions, we would expect to see relatively smooth curves indicating no

uneven patterns of selection for any particular level of the progression. We also would expect

the distribution of placements on the progression to shift from lower to higher between fall and

winter and as the grade levels increase.

We observed four types of distributions in progression placements.

1. Unimodal (i.e., one peak) with smooth tails on either side—Three progressions: Emotional

Literacy, Emotion Regulation Strategies, Writing

2. Unimodal, smooth tails, skewed to the higher end of the progression—Six progressions:

Book Orientation, Crossing Midline, Following Directions, Grip & Manipulation, Hand

Dominance, Letter Naming

3. Unimodal, smooth tails, skewed to the lower end of the progression—One progression:

Vocabulary

4. Unimodal with uneven tail(s)—Eight progressions: Emotion Expression, Gross Motor, Math

Patterns, Object Counting, Print Awareness, Perseverance, Problem Solving, Reading

Comprehension

Examples of each type of distribution, taken from the fall administration with kindergarten

children, are presented in Exhibits 2.1–2.4. The distributions of all fall kindergarten placements

(the status summary selected by the teacher) on each progression are in Appendix D, as are

the complete assessment results for both fall and winter administrations in grades K–3.


Exhibit 2.1. Unimodal distribution with smooth tails

Exhibit 2.2. Unimodal, smooth tails, skewed to the higher end of the progression

1%5%

9%

18%

37%

12%

6% 6% 5%1%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E F G H I

Frequency Distribution of Skill Levels on Emotional Literacy - Grade K Fall (n = 803)

0%5%

22%

30%

42%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D

Frequency Distribution of Skill Levels on Grip & Manipulation - Grade K Fall (n = 833)


Exhibit 2.3. Unimodal, smooth tails, skewed to the lower end of the progression

Exhibit 2.4. Unimodal, uneven tails

Although any of the four distribution types are possible when skills are in the correct order, the

uneven tails shown in Exhibit 2.4 for Object Counting suggest that some skill levels on the eight

progressions with this pattern may be very similar in difficulty making it hard for teachers to

differentiate them in these progressions. We conducted more rigorous psychometric analyses to

examine this issue further.

2%

13%

29%34%

11%8%

2% 2% 1%0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E F G H

Frequency Distribution of Skill Levels on Vocabulary - Grade K - Fall (n = 796)

0%3% 4%

6% 4% 5%

29%

21%

10%6%

12%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E F G H I J

Frequency Distribution of Skill Levels on Object Counting - Grade K Fall (n = 826)


Psychometric models

To further validate whether the skill levels in the 18 construct progressions were in the proper

sequence, the research team applied classical and modern test theory methods: exploratory

factor analysis (EFA) and Rasch modeling. EFA was applied to examine unidimensionality, that

is, whether the constructs when taken together were measuring a single, unidimensional latent

trait, such as school or learning readiness. When applied to the K–3 fall data, EFA indicated that

up to three latent factors may have been present among the constructs. Although fit statistics

were within the acceptable range for one-, two, and three-factor solutions, the one-factor

solution is considered more parsimonious in service of substantiating the claim that all

progressions were in the correct order. The Cronbach’s alpha for one overall “school readiness”

construct was .84, exceeding the social science standard of .80; this justified generating a total

score for modeling purposes. A total score was calculated so we could examine whether

children placed at a higher skill level for any construct progression (e.g., Skill C on Object

Counting) had higher mean total scores for “school readiness” than their peers placed at a lower

skill level (e.g., Skill A on Object Counting). Performance across constructs was calculated by

converting the skill levels on each progression to a numerical scale (Emerging = 1, A = 2, B = 3,

etc.) and summing across the progressions. for a total score was calculated solely for the

purposes of statistical modeling since a total score was not part of the design nor intended use

of the assessment.

A pattern of increasingly higher scores across the progression would support the claim that the

levels on the progressions are in the correct order. We examined the total score means of

children placed at each skill level on the progression for the 754 kindergarten children who were

assessed in the fall on all 18 progressions. Across the constructs we observed the same

general pattern: The mean total scores of children increased at each subsequent level on the

progression. This can be seen, for example, in the distribution of total mean scores for Emotion

Regulation Strategies in Exhibit 2.5.


Exhibit 2.5. Example of Increasing mean “School Readiness” scores for each level of the progression: Emotion Regulation Strategies (Fall Kindergarten)

There were, however, some exceptions to this pattern, mostly typically when the sample was

not large enough at any given skill on the progression to establish a valid mean (e.g., fewer than

10 at any one skill level) (Linacare, 2004; Wright & Linacare, 1992). Small sample numbers

were most often concentrated at the lower or higher ends of some progression where

kindergarten children were unlikely to be placed. We cannot reach any conclusions about the

sequencing for these skill levels on these progressions because the small number of children

means the findings are unreliable for these skill levels. We also observed essentially equivalent

mean total scores for some adjacent skills with larger numbers of children, suggesting that for

these progressions the skills may be very close together or difficult for teachers to differentiate,

as is evident in the distribution of total mean scores for Following Directions (Exhibit 2.6). For

this progression, the mean score for Level B is higher than for Level C which is not the expected

pattern. Note that we reach any conclusions about the “Emerging” level because it is based on

only 5 children.

7379

88 92102 105

0

20

40

60

80

100

120

Emerging(n = 19)

A(n = 112)

B(n = 139)

C(n = 209)

D(n = 195)

E(n = 80)

Mean Total Score School Readiness


Exhibit 2.6. Example of skill levels too close together: Following Directions

The distributions of total mean scores for each progression are in Appendix D. These

distributions suggest the constructs fall into three groups.

1. Properly sequenced skills at all levels of the progression (as shown by increasing total score

means with each skill level)—Ten progressions: Emotion Regulation Strategies, Emotional

Literacy, Grip & Manipulation, Hand Dominance, Crossing Midline, Letter Naming,

Perseverance, Print Awareness, Reading Comprehension, Writing

2. Unsure about the sequence of a few skills because of insufficient observations to validate

the sequence—Two progressions: Book Orientation, Gross Motor

3. Some skills may be too close to differentiate a child’s level (essentially equivalent total score

means at neighboring skill levels)—Seven progressions: Emotion Expression, Following

Directions, Gross Motor, Mathematical Patterns, Object Counting, Problem Solving,

Vocabulary

To further explore the possible dis-ordinality of progressions in the last group, a partial credit

Rasch model was applied to the fall K–3 assessment data to examine with greater rigor whether

steps in each progression were truly ordinal and sufficiently discriminated skill levels. Rasch

modeling is a psychometric approach for analyzing categorical data, such as answers to

questions on a reading assessment or, in this case, skill levels on a learning progression, as a

function of (a) the child’s abilities and (b) the skill difficulty.

71 7081 78

8792

103

0

20

40

60

80

100

120

Emerging(n = 5)

A(n = 36)

B(n = 35)

C(n = 83)

D(n = 125)

E(n = 126)

F(n = 344)

Mean Total Score School Readiness


We used an item response theory (IRT) framework to examine the ordering of the skill levels for

each of the 18 construct progressions based on the one-factor unidimensional solution

previously described. We applied the partial credit model (PCM) as a logistic measurement

model for polytomous items or progressions with an ordinal sequence (Embretson & Reise,

2000; Masters, 1982).

Factor scores were generated based on the PCM, using the full sample of 1312 K-3 children

assessed in the fall. In general, the IRT factor scores increased with higher skill levels on each

progression, lending support to the claim that the skills are in the correct order. We did,

however, encounter the same issues described previously when examining the total mean

scores for K children. There were insufficient numbers of observations (less than 10 children) at

the lowest skill level (Emerging) for several progressions (Book Orientation, Crossing Midline,

Emotion Expression, Following Directions Letter Naming, Object Counting, and Print

Awareness) to verify the ordinality of the Emerging level (Linacre, 2004; Wright & Linacre,

1992). The small number of observations was also a problem for verifying the ordinality of the

first three skills of the Gross Motor progression (Emerging, Skills A and B). Of greater concern

for the assessment are the two cases when the factor scores were equal or reversed at two

adjacent skill levels on the progression, suggesting that these skills are too close together to be

considered sequenced for instructional utility: Object Counting - Skills D and E, and

Mathematical Patterns - Skills B and C. The IRT factor scores for each construct progression

are presented in Appendix D, along with the previously discussed total mean scores.

Do the performance levels accurately reflect what is expected at entry to kindergarten? The K–3 Formative Assessment was designed to be used at kindergarten entry to inform

teachers’ instruction from kindergarten through grade 3. Kindergarten teachers often asked

where they should expect their students to fall on these progressions at the beginning of the

school year. Identifying where children are expected to be provides teachers with information

about which children are and are not on track. It also can provide administrators and

policymakers with a snapshot of children’s skills at kindergarten entry vis-à-vis expectations.

Parents, too, are interested in knowing where their child is relative to expected performance. For

the field test, we designed and implemented a method for establishing expected performance

levels for entry to kindergarten.


The research team used a mixed-methods approach and data from three different perspectives

to set performance levels at kindergarten entry:

• Teacher survey data regarding performance expectations on each progression

• Progression placement data for children in the fall of kindergarten

• National domain experts’ review of these data in light of their knowledge of kindergarten

entry skills for the construct progression they helped refine.

The performance levels were thus based on years of teacher experience with multiple

kindergarten classrooms and domain experts’ knowledge of current developmental literature on

each construct. We were able to establish performance levels for all the constructs, indicating

that the content of the assessment is appropriate for kindergarten children. We used an

accepted process for establishing performance levels based on teacher experience and domain

expertise that produced credible performance levels consistent with the actual performance of

the kindergarten children in the field test. We therefore conclude that the performance levels are

an accurate reflection of what is expected of children at kindergarten entry.

The research team used a standard-setting method (American Educational Research

Association, American Psychological Association, & National Council on Measurement in

Education, 2014; Bejar, 2008; Hambleton, 2000; Livingston & Zieky, 1982) that had four steps:

1. Identify who will determine performance levels at kindergarten (kindergarten teachers and

domain experts).

2. Define the performance descriptors teachers and domain experts will use to choose levels

on the progressions.

3. Collect and analyze judgments from teachers and domain experts.

4. Determine placement of performance levels.

See Appendix E for a list of the experts who reviewed the performance level data for each

construct.

Teacher surveys and domain expert questionnaires

After completing the collection of the field test assessment data in the fall and again in the

winter, all 89 kindergarten field test teachers were asked to complete a survey on which they

were to place a typical student on each construct progression at kindergarten entry based on

their experience assessing the constructs and teaching kindergarten. The teachers were asked

to identify two levels for each progression:


1. Select the level on the progression that represents the skill level of an average or typical

incoming kindergarten child in the first week of school; and

2. Mark the highest level on the progression where a child is so far below the skill level of a

typical kindergarten child that he or she may experience significant difficulties in the

classroom in this construct.

For the national experts, we created performance descriptors to help them determine which skill

levels on the progression would be included in different performance levels at the beginning of

kindergarten. Each descriptor was conceptualized based on instructional purposes. The

performance descriptors were the following:

• Performance level 1 (Developing): Students in this category have acquired some, but not all,

of the knowledge skills and abilities necessary to meet expectations or learning goals at

kindergarten entry. More support may be needed to help students achieve their learning

goals during the first few weeks of kindergarten.

• Performance level 2 (Demonstrating): Students in this category have acquired the

knowledge skills and abilities necessary to meet expectations or learning goals at

kindergarten entry.

Teacher survey responses were combined with frequency distributions for the fall 2016

kindergarten student placements on each progression in charts for easy comparison. Domain

experts were asked to review the charts, respond questions about where children’s knowledge

and skills fall on the progression at the beginning of kindergarten, and provide a written rationale

for their recommendations. The teacher survey and expert questionnaire are in Appendix D.

Teacher Results

Ninety-one percent of kindergarten field test teachers (n = 81) responded to the performance

level survey and 82 percent (n = 71) provided sufficient data for the analysis.

Teacher responses to the first instruction for each construct (select the level on the progression

that represents the skill level of an average or typical incoming kindergarten child in the first

week of school), varied by the perceived difficulty of the construct. For constructs that the

assessment data indicated were mastered by most children in kindergarten (Book Orientation

and Hand Dominance), teachers placed typical learners near the top of the progression. For

example, 49 percent of teachers responded that typical children would most likely be at skill

level C (opens book from front cover) on the Book Orientation progression, with 26 percent


responding that typical children would enter kindergarten with skills at the highest level of the

progression, level D (turns pages in order one at a time). A similar pattern emerged for Hand

Dominance and Crossing Midline, with teachers responding that typical children would exhibit

skills at the higher end of the progression.

For progressions that were shown to extend from kindergarten through grade 3, teachers’

responses followed expected patterns and were clustered toward the lower end of each

progression. For instance, when asked at which level they would expect a typical kindergarten

student to be placed on for Writing, 92 percent of teachers responded with skill level A or B.

When asked the same question for Mathematical Patterns, 69 percent responded that a typical

incoming kindergarten student would exhibit skills at level A or lower, and 90 percent responded

with level C or lower.

Teachers’ responses to the second instruction for each construct (select the level where a child

is so far below the skill level of a typical kindergartener that he or she may experience

significant difficulties in the classroom) followed expected patterns, with teachers choosing skills

an average of 1.9 (SD = .56) levels lower on each progression. Exhibit 2.7 illustrates teacher

responses to the first and second instruction for Book Orientation. Charts in Appendix D provide

these results for each progression.

Exhibit 2.7. Teacher survey responses for Book Orientation: Percentage of teachers indicating each level

0%3%

21%

50%

26%

51%

37%

10%

1% 0%0%

10%

20%

30%

40%

50%

60%

Below A A B C DBook Orientation (n = 68 Teachers)

Typical Far Below


Teachers’ responses were then combined with kindergarten assessment data collected in fall

2016 to compare where teachers expected children’s knowledge and skills at kindergarten entry

and where children’s knowledge and skills were during the fall data collection period. Because

the fall data collection for the field test occurred between 2 and 12 weeks after school began,

we expected that the teachers’ response to the performance setting for kindergarten entry would

be lower than actual student performance, which was the pattern in the data. Exhibit 2.8

illustrates this pattern for Book Orientation.

Exhibit 2.8. Teacher survey responses and child assessment results for Book Orientation: Percentage of teachers indicating each level

Domain expert review

Domain experts largely agreed with where the majority of teachers placed expectations at

kindergarten entry. On 14 constructs, overlap was apparent between where teachers reported

they would place an average incoming kindergarten student and the kindergarten entry

performance level experts recommended. For the other four constructs, experts noted concerns

that teachers were underestimating skills at kindergarten entry. Each expert provided a rationale

for the placement decision that included references to literature on kindergarten entry skills

when available. For these four constructs, we based the performance levels on those the

experts recommended. Exhibit 2.9 illustrates the final performance levels compared with

teacher responses for Book Orientation. Exhibit 2.10 presents the performance levels for all 18

constructs, along with the percentage of kindergarten teachers who identified each skill level as

0% 3%

21%

50%

26%

51%

37%

10%1% 0%1% 0% 3%

11%

86%

0%

20%

40%

60%

80%

100%

Below A A B C DBook Orientation

Teacher survey:Typical Teacher survey: Far Below Child Assessment


“typical” for that construct. Based on the performance levels, we calculated the percentage of

children who would have been identified as “developing” or “demonstrating” on each construct

based on their fall progression placements (Appendix D).

Exhibit 2.9. Teacher responses, fall K data and performance levels for Book Orientation: Percentage of teachers indicating each level

0% 3%

21%

50%

26%

51%

37%

10%1% 0%1% 0% 3%

11%

86%

0%

20%

40%

60%

80%

100%

Below A A B C D


Developing Demonstrating


Exhibit 2.10. Teacher responses and resulting performance levels at kindergarten entry

Book Orientation 3% 21% 50% 26%

Crossing Midline 1% 12% 64% 23%

Emotional Expression 1% 38% 20% 25% 10% 4% 1%

Emotion Regulation Strategies

51% 35% 14%

Emotional Literacy 26% 16% 38% 16% 1% 3%

Following Directions 17% 33% 32% 14% 1% 1%

Grip & Manipulation 28% 61% 10% 1%

Gross Motor 4% 7% 10% 22% 19% 20% 12% 4% 1%

Hand Dominance 10% 70% 20%

Letter Naming 7% 20% 3% 26% 40% 3% 1%

Math Patterns 4% 65% 7% 13% 6% 4%

Object Counting 25% 41% 13% 4% 6% 9% 1% 1%

Print Awareness 23% 40% 20% 14% 1% 1%

Perseverance 1% 33% 35% 17% 12% 1%

Problem Solving 1% 35% 42% 16% 6%

Reading Comprehension 4% 19% 46% 29% 1%

Vocabulary 1% 45% 48% 4% 1%

Writing 14% 78% 6% 1%

SKILL LEVELS: Below A A B C D E F G H I J K L M

Note. Teacher responses are percentage of teachers who indicated that skill level as typical for a child entering kindergarten.



Is the assessment sensitive to children’s growth and development over time? The field test findings showed changes in children’s progression placement over time and

indicated that most of the constructs in the assessment were sensitive to children’s growth and

development within a school year from fall to winter. For a few constructs, many children were

placed at the highest levels of the progression in the fall, so there was little opportunity for

growth beyond kindergarten entry. Although there were smaller numbers of children assessed in

grades 1–3, their placement was consistent with the hypothesis that children in the upper

grades would be placed at higher progression levels than children in the lower grades on those

progressions with levels that extended beyond kindergarten.

To answer this question, we examined the field test data from three sources:

• Teacher survey results

• Student placement on the progressions from fall to winter

• Student placement on the progressions across grade levels.

Teacher survey results

In an online survey administered after they had used the assessment for the second time in the

winter, teachers were asked:

• For each of the constructs you administered, indicate how well the progression was able to

capture children’s progress and development over time: very well, somewhat well, not well

or not at all well.

Across all constructs, 84 to 99 percent of teachers reported that the constructs they assessed

captured children's progress and development over time (either “very well” or “somewhat well”).

Teachers were less confident that the social-emotional constructs and Vocabulary captured

children’s progress over time. For these constructs, fewer teachers selected “very well” or

“somewhat well” than for the other progressions, and far fewer selected “very well”

(Exhibit 2.11).


Exhibit 2.11. Teacher ratings of how well construct progressions captured children’s growth

Fall to winter change in placements on the progressions

Teachers’ placement of children on the progressions provided further support for the validity

claim that the construct progressions are sensitive to student growth over time. This was evident

in the

• Shifts to higher levels in the distribution of progression placements from fall to winter on

each construct

• Higher placements of the same children tested in the fall and again in the winter for each

construct and grade level.

62%

40%

60%

50%

63%

52%

64%

47%

69%

62%

62%

64%

62%

30%

28%

29%

59%

67%

30%

44%

30%

38%

33%

39%

28%

39%

24%

37%

30%

33%

34%

55%

56%

55%

37%

30%

8%

16%

10%

12%

4%

9%

8%

14%

7%

9%

3%

4%

15%

16%

16%

4%

2%

0% 20% 40% 60% 80% 100%

Writing

Vocabulary

Reading Comprehension

Problem Solving

Print Awareness

Perseverance

Object Counting

Mathematical Patterns

Letter Naming

Hand Dominance

Gross Motor

Grip & Manipulation

Following Directions

Emotional Literacy

Emotion Regulation

Emotion Expression

Crossing-Midline

Book Orientation

Very well Somewhat well Not well


These patterns also provided evidence for the claim that the levels are sequenced correctly. If

the sequencing had not been correct, we would not see movement across the progression with

time.

Comparing the fall and winter assessment results for each construct across grade levels, we

found that the distribution of placements shifted from lower placements in the fall to higher

placements in the winter for all the progressions. This pattern can be seen, for example, in the

distribution of fall and winter placements for Grip & Manipulation for kindergarteners

(Exhibit 2.12).

Exhibit 2.12. Example comparing fall and winter progression placements: Grip & Manipulation (percentage of children)

The strongest evidence to address this question came from comparing individual change for

those children who were assessed in the fall and again in the winter. Of the 1,312 children

assessed in the fall, 1,278 also were assessed in the winter. We examined the differences

between the fall and winter means among children at each grade level by converting the

progression placements to numerical values (where Emerging = 1, A = 2, B = 3, C = 4, etc.) and

applying t tests for differences in the means. Winter means were significantly higher than the fall

means for all constructs and grade levels assessed with two exceptions: Perseverance in

grade 2 and Emotion Regulation Strategies in grade 3. The results comparing fall and winter

placement (tables in Appendix D) supported the claim that the 18 constructs are sensitive to

growth.

0%5%

22%30%

42%

0% 1% 5%12%

82%

0%

20%

40%

60%

80%

100%

Emerging A B C D

Grip and Manipulation - Grade K

Fall (n = 850) Winter (n = 785)


Across grade levels, children were generally placed at the same or higher levels on the

progressions in the winter. On some constructs children had limited opportunity to show growth

within grade level because many were already near the top at fall within that grade level.

Comparing the placements of children tested in the fall of kindergarten with their placements in

the winter, we found that children most often were at either a higher skill level in the winter or at

the same skill level if a child had already reached the highest level on the progression in the fall.

On three constructs, the opportunity to show growth in kindergarten was limited because most

children had reached the highest level on the progression in the fall: Book Orientation

(86 percent), Hand Dominance (74 percent), and Crossing Midline (71 percent). On four

constructs, at least 70 percent of children had higher placements on the progression in the

winter than the fall: Mathematical Patterns (76 percent), Object Counting (73 percent), Reading

Comprehension (73 percent), and Writing (70 percent) (Exhibit 2.13).

Exhibit 2.13. Examples of fall to winter progression placements of children in kindergarten (percentage of children)

13

19

19

70

73

73

76

86

71

74

5

11

6

4

26

12

12

19

4

2

4

11

4

5

0 20 40 60 80 100

Book Orientation (n = 763)

Crossing Midline (n = 773)

Hand Dominance (n = 777)

Writing (n = 770)

Reading Comprehension (n = 731)

Object Counting (n = 773)

Math Patterns (n = 733)

Higher Same, at the Max Same Lower

Constructs showing most opportunity for growth

Constructs showing least opportunity for growth


To determine the recommended grade levels for a construct, we identified the grade levels at

which a substantial majority of the children (70 percent) were placed at the highest level on the

progression in the fall. These constructs were classified as showing opportunity for growth at the

grade levels below where this occurred. The percentage of children at the maximum level in the

fall along with the grade levels with opportunity for growth for each construct are summarized in

Exhibit 2.14 with more detail provided in Appendix D.

Exhibit 2.14. Percentage of children at maximum level and grade levels with opportunity for growth for each progression

Field Test Percentage of Children at

Maximum Level in Fall Opportunity for Growth

Book Orientation (A-D) K 86 na na na Below K

Hand Dominance (A-C) K 74 na na na Below K

Crossing Midline (A-C) K-1 71 71 na na Below K

Grip & Manipulation (A-D) K-1 42 59 na na K-1

Letter Naming (A-G) K-1 31 65 na na K-1

Print Awareness (A-G) K-1 41 94 na na K

Object Counting (A-J) K-2 11 43 77 na K-1

Emotion Express (A-G) K-3 6 16 26 26 K-3

Emotion Regulation Strategies (A-E) K-3 6 19 33 42 K-3

Emotional Literacy (A-I) K-3 1 2 23 21 K-3

Following Directions (A-F) K-3 42 45 61 44 K-3

Gross Motor (A-I) K-3 42 43 83 84 K-1

Math Patterns (A-M) K-3 0 0 0 8 K-3

Perseverance (A-H) K-3 7 12 7 13 K-3

Problem Solving (A-J) K-3 0 1 10 1 K-3

Reading Comp (A-H) K-3 5 21 30 33 K-3

Vocabulary (A-H) K-3 0 4 10 0 K-3

Writing (A-G) K-3 0 0 2 8 K-3

na = not assessed


Six progressions did not show opportunities to measure student growth through grade 3:

• For three progressions—Book Orientation, Hand Dominance, and Crossing Midline—more

than 70 percent of the children tested in the fall of kindergarten were placed at the highest

skill level on the progression, so these constructs have limited instructional utility for most

children beyond K entry.

• Print Awareness was field-tested in grades K and 1, but it may be most appropriate for

informing instruction in kindergarten only, with 94 percent of children already placed at the

highest/maximum level on the progression in the fall of grade 1.

• Object Counting was field-tested in grades K–2, but it may be most useful for informing

instruction in grades K–1, with 77 percent of children already placed at the highest level on

the progression in the fall of grade 2.

• Gross Motor was field-tested in grades K–3 but appears to be providing useful information

only in grades K–1, with 83 and 84 percent of children already placed at the highest level on

the progression in the fall of grades 2 and 3, respectively.

We further examined the extent to which the progressions provided opportunities for growth by

looking only at those children who were not at the highest level of the progression in the fall

(Exhibit 2.15). Across nearly all progressions at all grade levels, most of these children were

placed at a higher level in the winter.

Placement on the progressions across grade levels

Another way to examine growth is by looking at the placements on the progressions across

grade levels. For constructs with opportunities for growth into the next grade level, we observed

a shift in the distributions from one grade level to the next. The kindergarten to grade 1 shift

pattern can be seen, for example, by comparing the distributions of kindergarten and grade 1

fall placements for Following Directions, with more children at the upper end of the progression

in grade 1 than in kindergarten (Exhibit 2.16).


Exhibit 2.15. Percent of children not at highest level in fall who were placed at a higher level in winter for each progression

Field Test Percentage of Progression Placements Higher in Winter

(for Children Not at Maximum Level in Fall)

Book Orientation (A-D) K 91 na na na

Hand Dominance (A-C) K 75 na na na

Crossing Midline (A-C) K-1 65 57 na na

Grip & Manipulation (A-D) K-1 85 59 na na

Letter Naming (A-G) K-1 82 69 na na

Print Awareness (A-G) K-1 93 82* na na

Object Counting (A-J) K-2 82 75 77* na

Emotion Express (A-G) K-3 66 70 45 70

Emotion Regulation Strategies (A-E) K-3 60 64 48 78

Emotional Literacy (A-I) K-3 64 70 64 46

Following Directions (A-F) K-3 66 46 51 77

Gross Motor (A-I) K-3 70 77* 74* 75*

Math Patterns (A-M) K-3 76 74 43 71

Perseverance (A-H) K-3 60 73 50 73

Problem Solving (A-J) K-3 66 76 66 82

Reading Comp (A-H) K-3 76 47 74 61

Vocabulary (A-H) K-3 68 57 67 50

Writing (A-G) K-3 70 60 42 46

na = not assessed; *n < 30 children


Exhibit 2.16. Example comparing kindergarten and grade 1 progression placements: Percentage of children placed on each level of the progression for Following Directions

Comparing the distribution and means for constructs assessed at multiple grades, we found the

expected variation in distributions by grade level. For the constructs for which opportunity for

growth extended into the next grade level, the distributions and the means moved to higher

levels on the progression as the grade level increased. Some exceptions from grade 2 to

grade 3 were noted where sample numbers were smaller (126 for grade 2 and 138 for grade 3).

These differences between grade level mean placements (kindergarten to grade 1, grade 1 to

grade 2, grade 2 to grade 3) were statistically significant (p < .05). The summary of results for

grades K–3 placements is in Appendix D.

Although higher mean scores at each successive grade level were found for most constructs, a

few did not show the expected pattern between some grade: Crossing Midline (between K

and 1), Grip & Manipulation (K–1), Emotion Regulation Strategies (grades 1–2), Following

Directions (K–1, grades 2–3), Perseverance (grades 2–3), Problem Solving (grades 2–3), and

Vocabulary (grades 2–3).

Does the assessment provide information that is useful for instruction for grades K–3? We asked teachers how they used information from the assessment in the surveys at the end of

the fall and winter administrations and in focus groups at the end of the field test. The majority of

teachers reported using the information for instruction and that it was useful, although they

1%4% 5%

11%16% 18%

45%

0%3%

6% 8%12% 13%

58%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E FFollowing Directions (Fall)

Grade K (n = 827) Grade 1 (n = 194)


found the assessment more useful for some conditions and constructs. A small number of

teachers felt the assessment did not provide new or relevant information outside what they were

already observing or assessing in the classroom.

After the fall administration, 86 percent of teachers reported using information from the K–3

Assessment to inform instruction. Over half the teachers (54 percent) used the information “a

great deal” or at least “somewhat.” The most helpful constructs (those that 75 percent or more

of teachers who assessed it indicated the construct was helpful) were Writing, Problem Solving,

Object Counting, Reading Comprehension, Print Awareness, and Following Directions

(Exhibit 2.17).

Exhibit 2.17. Teachers’ rating of constructs as “very” or “somewhat” helpful to instruction

When asked about the benefits of implementing the assessment, about half the teachers

(53 percent) reported that they developed a deeper understanding of the whole child and the

child’s developmental needs (Exhibit 2.18). Teachers frequently commented in surveys and

interviews that the information was especially useful in identifying struggling students and in

grouping students who needed additional support.

51%

60%

63%

64%

64%

65%

66%

68%

68%

70%

73%

73%

75%

75%

76%

77%

79%

81%

Gross Motor

Hand Dominance

Crossing Midline

Emotion Expression

Emotional Literacy

Emotion Regulation

Book Orientation

Grip & Manipulation


Vocabulary

Letter Naming

Perseverance


Print Awareness


Object Counting

Problem-Solving

Writing

Percentage of Teachers (n = 132)


Exhibit 2.18. Teachers’ report of benefits of implementing the K–3 Formative Assessment

After the winter administration, 86 percent of the teachers reported that they used information

from the assessment to inform instruction, and 68 percent found the assessment a helpful

resource for informing instruction in their classrooms.

The survey questions and focus groups also explored what kinds of difficulties teachers

encountered in using the assessment to inform instruction. One was that the data from the

assessment were redundant with information teachers already had collected. Most teachers

(81 percent) reported that they were able to use information from other assessments to place

children on progressions in the K–3 assessment, which suggests they were indeed already

collecting some of the information included in the assessment. The K–3 Formative Assessment

is intended to complement other assessments, and we encouraged teachers to use data from

other assessments in placing children on the progressions. Still, it is possible that there is too

much overlap among assessments for some constructs for teachers to appreciate the value of

these progressions in informing their instruction. Other impediments to instructional utility the

teachers reported were that the assessment was not aligned well enough with state standards

or district expectations, that they did not find it necessary to formally document what they

already knew about their students, and that the information from the assessment was not

relevant later in the school year.

23%

27%

27%

29%

30%

33%

52%

53%

No real benefits

Being better prepared to plan instruction that isdevelopmentally appropriate

Increased knowledge of how technology can be used fordata collection and for guiding instruction

Being better equipped to respond to individual studentneeds

Having data all in one place to review for instructionalplanning

Having readily available data to share with support staff,administrators, and families

Deeper understanding of individual child’s skills & needs

Better understanding of the whole child



Does the assessment provide information useful for instruction for different subgroups, such as children with disabilities and English learners? Most of the teachers who used the assessment with either children with disabilities or English

learners reported that they were able to assess children in these two subgroups and that the

assessment provided information useful for their instruction. Examining the progression

placements for these two subgroups, we found differences for children with disabilities for most

of the constructs but few placement differences for English learners compared with other

children in the class.

Children with disabilities

Children with disabilities comprised 8.6 percent of the field test sample overall (n = 113).

Teachers generally placed children with individualized education programs (IEPs) lower on each

construct progression than children without IEPs in both fall and winter administrations and

across grade levels. We examined the statistical significance of these subgroup differences in

progression placements in kindergarten, where 68 children had IEPs. Differences were

significant (most at p < .0001) for all constructs except Hand Dominance, Book Orientation

(winter only), and Mathematical Patterns (fall only), with children with IEPs placed lower on the

progression than children without IEPs. These results are presented in Appendix D.

Interpreting the differences across constructs is difficult because we do not know the

composition of the sample of children with IEPs, the nature of their disabilities, or whether

teachers made the accommodations indicated in their IEPs in implementing the assessment.

For two of the three constructs that did not consistently show group differences, Hand

Dominance and Book Orientation, many children were placed at the highest level of the

progression in the fall for Hand Dominance and by the winter for Book Orientation. For

Mathematical Patterns, children with IEPs were placed lower on the progression in the fall than

other children, but the mean difference did not quite reach significance (p = .06).

Slightly more than half (55 percent, n = 73) of the teachers implemented the assessment with a

child with an IEP. These teachers assessed on average two to three children with disabilities.

This was the same as the average number of children with IEPs (two to three) enrolled in

classes of 19 to 20 children in the field test. A majority of these teachers (57 percent) reported

that the assessment was no more difficult to implement with children with disabilities, with 32

percent reporting that it was a little more difficult. More important, 81 percent of these teachers

thought that the assessment provided information that was somewhat or very useful for

informing that child’s instruction.


English learners

English learners (ELs) comprised 7.8 percent of the field test sample overall (n = 102). There

were no consistent differences in how teachers placed ELs on each construct progression when

compared with non-ELs across both fall and winter administrations and across grade levels. We

examined the statistical significance of the differences between ELs and other children in

progression placements in kindergarten, where there were 73 EL students. Differences were

significant at p < .05 for six constructs. ELs were placed higher on the progression than non-ELs

for Book Orientation (fall), Crossing Midline (winter), Emotion Regulation Strategies (fall and

winter), Emotional Literacy (fall), and Hand Dominance (winter). ELs were placed lower on the

progression for Print Awareness (fall). These results are in Appendix D.

These differences across constructs are difficult to interpret without knowing more about the

composition of the EL sample, their experience with English, and the extent to which teachers

were assisted by a native language speaker in observing the skills that required communicating

or understanding the child’s first language. The absence of differences in progression

placements on constructs that rely most heavily on language (e.g., Letter Naming, Following

Directions, Reading Comprehension, Writing, and Vocabulary) suggests that the teachers found

ways to assess these constructs in the child’s native language or the children had sufficient

English skills so the teacher could assess them in English.

One quarter of the teachers (n = 33) implemented the assessment with English learners. These

teachers assessed on average three to four children who were learning English in school. On

average, there were two ELs enrolled in classes of 19–20 children in the field test. Most of these

teachers (76 percent) reported that the assessment was not more difficult to implement with

ELs, whereas 24 percent reported that it was a little more difficult. More important, 82 percent of

these teachers thought that the assessment provided information that was at least somewhat or

very useful for informing the instruction of ELs.

Do the assessment results vary for other subgroups (gender, race)? To further explore whether the K–3 Formative Assessment provides information useful for

instruction for all children, we examined the assessment results for gender and racial bias. We

would not expect to see any differences in where teachers placed children on these

progressions based on the race or gender of the child. However, since some studies of early

childhood development and early academic skills have shown gender and racial differences at

school entry and into the early elementary grades, we examined our data accordingly.


Gender

We compared the mean differences between boys and girls on the progressions and found no

gender differences in the fall of kindergarten for constructs in the cognitive domain (Object

Counting, Mathematical Patterns, Problem Solving) or for Reading Comprehension and the

physical constructs without a motor component (Hand Dominance and Crossing Midline).

Teachers placed girls higher than boys on constructs in the social-emotional domain and the

other constructs in the language and motor domains in the fall of kindergarten. Mean differences

on these progressions were statistically significant at p < .05 or higher. There were no

statistically significant differences at first grade, but differences reemerged in second and third

grade, favoring girls in Emotion Expression, Emotion Regulation Strategies, Following

Directions, Perseverance, Vocabulary, and Writing. Only one difference favored boys, Object

Counting in the fall of first grade. The findings for construct performance by gender are in

Appendix D. Note that the samples were considerably smaller at the higher grade levels and

therefore less likely to yield statistically significant differences. It is impossible to discern from

the field test information whether these differences reflect actual differences between boys and

girls or whether teachers bring a bias to their observation of some of the constructs.

Race/Ethnicity

As noted in Chapter 1, the sample of children assessed was 68 percent White, 18 percent non-

White, and 14 percent of unknown race/ethnicity. These data are based on information that

teachers provided in the technology platform for each child. The relatively small numbers of

non-White children limit our ability to examine racial/ethnic differences.

Despite the small sample size of non-White children, statistically significant differences were

found in the mean placements of White and non-White children for various constructs at

different grade levels and administrations (fall and winter). Some of the differences favored

White children, primarily in the language and communication domain: Hand Dominance (grade

K, fall), Letter Naming (grade 1), Object Counting (grades K and 2, fall), Perseverance (grade K,

fall), Print Awareness (grade K, fall), Reading Comprehension (grades K, 1, and 2, fall), and

Vocabulary (grade K, fall). Other differences favored non-White children, primarily in the social-

emotional and motor domains: Crossing Midline (grade 1, fall), Emotion Expression (grade 2,

winter), Emotion Regulation Strategies (grades 2 and 3, winter), Emotional Literacy (grade 2,

winter), Grip & Manipulation (grade 1, winter), and Gross Motor (grade 1, fall). Some constructs

showed differences favoring one group at one grade level and the other at a different grade

levels: Following Directions, Problem Solving, and Writing. The findings for construct


performance by race/ethnicity are in Appendix D. Given the relatively small number of non-

White children in the field test sample, especially in grades 1–3, we recommend further

research with a larger, more diverse sample before drawing any conclusions about how these

constructs are performing as measures for children from different racial/ethnic backgrounds.


Chapter 3: Implementation of assessment process A series of the research questions represented validity claims about important aspects of the assessment process: implementing the assessment with fidelity, including reliably coding status summaries; teacher confidence in their status summary ratings; and incorporating the assessment into classroom routines.

Can teachers learn to implement the assessment with fidelity? There is no fixed set of standardized practices for teachers to follow in implementing the K–3

Formative Assessment but rather a set of general guidelines for how they are to collect, record,

and interpret information about the children in their classroom. The field test results indicated

that the K–3 Formative Assessment could be implemented with fidelity, although some

constructs were more challenging to implement than others. The challenges in implementing

constructs were evident to a small extent in teachers’ selection of which constructs to assess

and to a larger extent in how faithfully teachers followed the assessment process. Variation was

apparent across constructs in teachers’ confidence in their status summary ratings and in the

reliability of their status summary coding. Nevertheless, the majority of teachers were able to

incorporate the assessment of most constructs into their classroom practice, and many teachers

were able to implement the assessment as intended. This is encouraging given the limited

training and experience the teachers had with the K–3 Formative Assessment before the field

test.

To produce valid data, an assessment must be implemented with fidelity to the assessment

process as designed. The K–3 Formative Assessment is based on teachers’ observations of

child behaviors in routine classroom activities and placement of each child on the skill levels on

each construct progression. Teachers were trained to collect multiple pieces of evidence of a

child’s skill level on each construct, determine the point on the progression (preliminary learning

status) indicated by each piece of evidence, and then interpret across these multiple pieces of

evidences to arrive at the child’s placement on the progression (status summary) at a given

time. Teachers were encouraged to gather at least two pieces of evidence for each child and

construct and enough evidence that they were confident in the child’s placement on the

progression. To determine whether teachers can learn to implement this assessment with

fidelity, we examined three indicators of their implementation:

• Implementation of the recommended assessment process

• Confidence in determining a child’s learning status

• Reliability in coding a child’s status summary.


Implementation of the recommended assessment process

The information entered into the technology platform suggests that teachers generally followed

the recommended assessment process with some variation by construct. Using the technology

platform for inferring how teachers implemented the assessment has limitations, however. A

lack of evidence entered into the technology platform could mean that a teacher did not collect

evidence, which would be a deviation from how the assessment was to be administered, or it

could mean that the teacher chose not to enter evidence, which is acceptable. Teachers

reported that collecting multiple pieces of evidence for each child and construct was time

consuming and that the process of entering the information onto the platform was cumbersome.

Some teachers also reported that they collected and used evidence to determine status

summary ratings but did not enter the evidence into the platform, further confirming the

importance of recognizing the limitations in reaching conclusions based on the contents of the

platform. Nevertheless, it is one of source of information about how teachers implemented the

assessment.

From the information in the platform, we learned the following about teachers’ adherence to

these steps in the assessment process:

• Assigned child a status summary (required). Teachers assigned a status summary for at

least one construct for 1,312 children in the fall and 1,229 children in the winter. Across

constructs, 93 to 98 percent of the children in the field test sample were assessed in the fall,

96 to 100 percent in the winter. Teachers assessed nearly all the constructs they were

asked to assess at their grade level with all the children they were asked to assess. The

constructs least frequently assessed were Emotional Literacy (93 percent of the children),

Emotional Regulation Strategies (94 percent), and Mathematical Patterns (94 percent), but

even these constructs were assessed for nearly all K–3 children in the sample.

• Documented multiple pieces of evidence (required but might not be in the platform).

Teachers did not consistently document more than one piece of evidence for determining a

child’s placement on a progression in the technology platform. Across constructs, 57 to 71

percent of the status summaries were based on multiple pieces of evidence in the fall, and

fewer (40 to 68 percent) were in the winter. Constructs least often documented with multiple

pieces of evidence were Emotional Literacy and Book Orientation (57 percent of the children

with status summaries) in the fall and Book Orientation (41 percent), Crossing Midline (40

percent), and Gross Motor (42 percent) in the winter.


• Assigned evidence a preliminary learning status (optional). Teachers were encouraged

to assign each piece of evidence a preliminary learning status to help them interpret multiple

pieces of evidence and determine the child’s status summary. Teachers followed this

optional step for 88 percent of all the evidence records entered on the platform. When

learning status was tabulated by construct and child, some variation was observed. Learning

status was assigned least often for Vocabulary (89%) and most often for Emotion

Regulation Strategies (99%), both during the winter administration.

These data are summarized in Exhibit 3.1. In both the fall and winter, teachers assigned a

status summary to nearly all constructs for all children who were assessed, and most of the

children were assigned the optional learning status as well.


Exhibit 3.1. Percentage of children with status summaries, multiple pieces of evidence, and learning statuses

Field Test

Percentage of Children Assigned a Status

Summary

Percentage of Children Whose

Status Summaries Based on Multiple Evidences

Percentage of Children Whose Evidence Included

a Learning Status

Book Orientation K 98 98 57 41 96 91

Crossing Midline K-1 97 100 62 40 97 94

Emotion Expression K-3 96 98 60 61 94 94

Emotion Regulation Strategies K-3 94 99 59 64 91 99

Emotional Literacy K-3 93 96 57 65 90 92

Following Directions K-3 98 100 66 58 94 92

Grip & Manipulation K-1 98 100 65 47 95 92

Gross Motor K-3 98 98 58 42 92 92

Hand Dominance K 98 100 71 49 98 91

Letter Naming K-1 98 100 64 45 95 90

Math Patterns K-3 94 98 62 64 95 91

Object Counting K-2 97 100 58 58 95 93

Perseverance K-3 96 99 66 62 94 92

Print Awareness K-1 98 98 61 54 92 91

Problem Solving K-3 95 98 60 63 90 92

Reading Comp K-3 95 100 64 59 95 91

Vocabulary K-3 95 98 60 62 92 89

Writing K-3 98 99 66 68 92 91

Teacher confidence in status summary

During the training, teachers often asked how much evidence was enough to support a child’s

placement on the progression (status summary). They understood that multiple pieces of

evidence were important but wanted to know exactly how many they should collect. There is no

exact number; rather, we suggested teachers collect enough evidence to be confident in the

child’s placement.

At the end of the winter assessment, we asked teachers to rate their confidence in having

accurately placed children on each of the progressions (Exhibit 3.2). Across constructs, 71 to 99


percent of teachers were at least somewhat or very confident in their ability to accurately place

children on each of the progressions they assessed. Their confidence varied by construct, with

teachers least confident in the accuracy of their placements on the three social-emotional

progressions: Emotional Literacy (75 percent), Emotion Regulation Strategies (71 percent), and

Emotion Expression (74 percent).

Exhibit 3.2. Percentage of teachers confident in progression placement (n = 130)

Teacher reliability in coding status summaries

We examined the evidence teachers provided in the platform from the fall data collection to

address two questions:

• What are the qualities of the evidence teachers entered into the platform (i.e., to what extent

is the evidence construct relevant, progression skill specific, and child specific)?

66%

42%

63%

43%

81%

53%

81%

52%

87%

87%

73%

76%

76%

23%

19%

21%

73%

86%

29%

41%

30%

47%

16%

36%

14%

36%

8%

10%

21%

16%

22%

52%

52%

53%

20%

13%

5%

17%

7%

11%

3%

11%

4%

12%

5%

3%

6%

8%

25%

29%

26%

7%

0% 20% 40% 60% 80% 100%

Writing

Vocabulary


Problem Solving

Print Awareness

Perseverance

Object Counting


Letter Naming

Hand Dominance

Gross Motor

Grip & Manipulation


Emotional Literacy

Emotion Regulation

Emotion Expression

Crossing-Midline

Book Orientation

Very confident Somewhat confident Not confident


• When the evidence is sufficiently specific, how often would a teacher and outside rater

assign a similar level to the child’s skills (the same status summary rating) based on the

evidence documented?

The methodology used for this evidence analysis followed important standards in assessment

development (American Psychological Association, American Educational Research

Association & National Council on Measurement in Education, 2014), and is detailed in

Appendix D. The evidence analysis entailed selecting a sample of records for each construct,

which were then coded by trained reviewers using a structured coding protocol.

• Sampling. Teachers entered 80,000 evidence records into the platform across the fall and

winter field test periods. From an exploratory analysis of teacher evidence collected during

the 2015–16 pilot test, we calculated the number of records needed to adequately examine

the reliability of coding the evidence and status summaries for each construct. This resulted

in drawing a smaller sample for the constructs with higher interrater agreement in the pilot

(e.g., Book Orientation, Grip & Manipulation) and a larger sample for the more complex

constructs that had shown lower interrater agreements in the pilot (e.g., Problem Solving,

Perseverance). The sample was further stratified by state and teacher to ensure that it was

representative of the field test participants.

• Coding protocol. Teachers’ evidence consisted of anecdotal notes, photographs, and

videos. From previous reviews of the evidence collected in North Carolina’s pilot of selected

constructs (Ferrara & Lambert, 2015), we learned that teacher evidence varies in specificity

and that some evidence may not be sufficiently specific to code. Consequently, we

developed a coding scheme that first examined each piece of evidence for three aspects of

specificity: construct relevance (Was the evidence related to the content of this construct?),

skill specificity (Was the evidence related to a skill level on the progression?), and child

specificity (Was the evidence specific to this child?). If the body evidence for the construct

was specific to skills on the progression and to the child sampled, the reviewer continued to

examine whether the evidence supported the status summary endorsed by the teacher. The

reviewer (1) coded a status summary, based on that evidence; (2) compared her status

summary with the teacher’s; and (3) determined whether the teacher’s status summary code

was consistent with the evidence documented. The detailed coding protocol and rubric are

in Appendix D.

• Reviewer training. The reviewers were project staff familiar with the assessment materials.

To ensure consistency among them, a 10 percent sample for each construct was coded by


two reviewers before the full sample was coded. Cohen kappa estimates ranged from .85 to

1.00, indicating almost perfect to perfect agreement among reviewers. Reviewers discussed

any disagreements to reach consensus and clarify coding rules to be used in the remaining

90 percent of the sample. Reviewer interrater reliability coefficients are displayed in

Appendix D.

Most of the evidence teachers provided in the platform was construct relevant, ranging from 70

percent for Perseverance to 99 percent for Vocabulary. A smaller proportion was found to be

specific to skills on the progression or specific to the child being observed; Across constructs,

the percent of evidence determined to be specific to skills on the progression ranged from 33

percent for Perseverance to 87 percent for Writing. Besides Perseverance, the percentage of

evidence found to be skill specific fell below 50 percent for several constructs: Emotional

Literacy (41 percent) and Problem Solving, Reading Comprehension, and Emotion Regulation

Strategies (49 percent each). The variation in the extent to which the evidence was specific to

the child being observed ranged from slightly over 50 percent for Emotional Literacy and

Problem Solving to more than 75 percent for Book Orientation, Emotion Expression, Hand

Dominance, and Grip & Manipulation.

Exhibit 3.3 presents the results for all constructs. Note that teachers were documenting

evidence for their own use in placing children on the progressions. They were trained on the

importance of high-quality documentation but also encouraged to find shortcuts in providing

documentation that would be most helpful for them.


Exhibit 3.3. Evidence specificity by construct

Percent of Evidence that was . . .

Book Orientation 36 90 74 75

Crossing Midline 38 95 68 68

Emotion Expression 32 78 60 76

Emotional Literacy 55 83 41 51

Emotional Regulation Strategies 55 71 49 58

Following Directions 35 90 84 59

Grip & Manipulation 37 95 79 78

Gross Motor 35 88 74 66

Hand Dominance 37 93 76 76

Letter Naming 32 88 73 69

Math Patterns 56 84 80 64

Object Counting 41 75 72 55

Perseverance 60 70 33 60

Print Awareness 35 96 73 69

Problem Solving 53 84 49 51

Reading Comprehension 66 94 49 58

Vocabulary 65 99 80 67

Writing 67 98 87 73

When reviewers determined the evidence was specific enough to code a status summary, they

arrived at one based on the evidence. Reviewers then determined whether the teacher’s status

summary was consistent with the evidence documented and whether it matched the reviewer’s

status summary. Constructs varied in the proportions of evidence that were specific enough to

determine a status summary, ranging from 33 percent for Perseverance to more than

80 percent for Following Directions, Grip & Manipulation, Hand Dominance, and Writing. The

consistency of the evidence with the teacher’s status summary ranged from below 40 percent

(Perseverance, Emotion Regulation Strategies, Emotional Literacy) to more than 70 percent

(Crossing Midline, Grip & Manipulation, Hand Dominance, and Writing). Exact agreement


between the reviewer’s and teacher’s status summary ratings ranged from less than 30 percent

(Emotional Literacy, Perseverance, Emotion Regulation Strategies) to more than 70 percent

(Crossing Midline, Hand Dominance, and Writing). Exhibit 3.4 presents these findings.

Exhibit 3.4. Evidence analysis: Percentage of evidence specific enough to code, coded status summary consistent with the teacher status summary, and coded status summary matched the teacher’s exactly

Note. The total length of the bar indicates the percentage of evidence for each construct that was specific enough to code. The length of the green and blue bars is the percentage where the teacher’s summary was consistent with the evidence. The blue bar is the percentage of codes where the reviewer’s code and the teacher’s matched exactly.

Incorporating the assessment into classroom routines The K–3 Formative Assessment was designed to be manageable and implemented as part of a

teacher’s daily classroom routine. We examined this by surveying and interviewing teachers

who participated in the field test. After implementing the assessment in the fall and again in the

0% 20% 40% 60% 80% 100%

Writing (N = 67)

Vocabulary (N = 65)

Reading Comprehension (N = 66)

Problem Solving (N = 53)

Print Awareness (N = 35)

Perserverance (N = 60)

Object Counting (N = 41)

Math Patterns (N = 56)

Letter Naming (N = 32)

Hand Dominance (N = 37)

Gross Motor (N = 35)

Grip & Manipulation (N = 37)

Following Directions (N = 35)

Emotional Regulation (N = 55)

Emotional Literacy (N = 55)

Emotion Expression (N = 32)

Crossing Midline (N = 38)

Book Orientation (N = 36)

Coder and teacher SS matched Codes consistent with teacher evidence Specific enough to code


winter, most teachers indicated they were able to assess the constructs assigned to their grade

level, although some were not able to easily work certain constructs into their instructional

routines.

Most teachers were able to assess their assigned constructs, assessing 93 to 100 percent of

the children on the constructs designated for their grade level. Teachers also reported that they

were not able to assess some constructs as part of everyday instruction. Gross Motor and

Mathematical Patterns were particularly challenging to incorporate into daily classroom routines

(Exhibit 3.5). Teachers reported that Gross Motor was difficult to assess because of limited

interactions with children during recess and/or physical education classes and limited physical

activity outdoors in winter. Mathematical Patterns was also difficult to assess during instruction,

teachers telling us that it is not something they typically teach.

Exhibit 3.5. Percentage of teachers who reported they were unable to assess constructs during everyday instruction: Most challenging constructs

We asked teachers what they thought about the manageability of this assessment for other

teachers at their grade level, and 51 percent thought that other teachers would be able to

incorporate the assessment into their classroom routines. Those who said teachers would not

be able to reported concerns that teachers would need more time for planning, collecting the

data, and implementing with a whole classroom.

When asked about challenges in implementing this assessment, teachers consistently cited

time and lack of support in the classroom as the primary issues. More than half the teachers

22%

22%

23%

24%

46%

54%

Vocabulary

Emotion Expression

Emotional Literacy

Emotion Regulation


Gross Motor

n = 133 teachers


(57 percent) reported not having enough support in the classroom to collect evidence. About

half (54 percent) of the teachers in the sample had paid paraprofessionals or aides in their

classrooms. Another challenge teachers reported was a lack of alignment with state standards.

As noted, Mathematical Patterns was problematic for teachers. Although not formally introduced

in the Common Core State Standards until grade 3, the foundational activities for Mathematical

Patterns appear as early as kindergarten, but this was not obvious to the teachers. Constructs

for nonacademic skills are not directly represented in the standards, so teachers might not have

considered them a priority or their responsibility. The most frequent challenges reported by

teachers are shown in Exhibit 3.6.

Exhibit 3.6. Percentage of teachers who reported each challenge experienced in implementing the assessment

Except for a few constructs, the data suggest the assessment can be incorporated into

classroom routines.

13%

19%

28%

33%

35%

57%

61%

Lack of support/buy-in from school administrators

Not enough information about assessment content

Technical difficulties accessing the platform or app

Not enough staff in the school were trained

Lack of alignment with state standards

Not enough support in classroom to collect evidence

Not enough time to implement

n = 132 teachers


Chapter 4: Implementation supports for teachers This chapter addresses the supports teachers require to implement the assessment with fidelity and produce valid information. Field test findings related to the following supports are discussed: professional development, ongoing assistance, assessment materials, and technology.

Professional development Teachers participating in the field test received 2 full days of training by SRI staff and staff

representing the technology platform vendor. Staff from North Carolina’s K–3 Formative

Assessment project provided additional support. The training covered formative assessment,

the assessment content (the constructs), the assessment process, the technology platform, and

field test research procedures. Teachers new to the assessment received the full 2 days of

training, whereas teachers returning from the pilot study received an abbreviated refresher

training (1–1.5 days). Drawing on our experience in training pilot test teachers to use this

assessment, we concentrated the field test training on two topics:

• Assessment content. Because adequately covering all 18 constructs in 2 days is difficult,

the training incorporated different constructs as examples of how to implement the

assessment throughout the training and provided an overview of the most salient aspects of

each domain. Teachers were encouraged to deepen their knowledge with online construct

modules before assessing each construct. One module was provided for each construct.

Teachers also were provided with a suggested administration schedule with the constructs

grouped into clusters that would be easier to administer together to help manage the

number of constructs.

• Assessment process. Teachers who had participated in the pilot appreciated the

opportunities to practice the assessment process, so the field test training incorporated self-

paced activities that teachers could move through individually or in small groups to reinforce

steps in the formative assessment process.

To answer the research question concerning the nature and quality of the initial professional

development provided to teachers, we surveyed teachers approximately 3–4 weeks after they

completed the training and had started to use the assessment materials; 145 teachers

responded. This survey also included questions about their background and experience. Focus

groups conducted at the end of the field test also addressed the training. Copies of the survey

and focus group protocol are in Appendix D, as well as a table describing the teachers who

participated in the field test.


Both the new and returning teachers responded positively to the survey questions about the

training, indicating that as a result of the training, they felt prepared to implement the

assessment. The majority of teachers (68 percent) felt they were “very” or at least “adequately”

prepared to implement the assessment after the field test training. Three-fourths of the new

teachers (75 percent) and more than half of the returning teachers (53 percent) felt that the

demonstration of the technology platform was a helpful component of the training. Other

aspects of training that new teachers found especially helpful were the overview of the construct

progression structure (77 percent), overviews of the five domains (69 percent), and

demonstration of Teacher Site (64 percent). In addition, the activities supporting the five steps in

the formative assessment process were rated as helpful by 37 to 52 percent of teachers

(Exhibit 4.1).

Exhibit 4.1. Aspects of training teachers new to assessment found most helpful

Teachers liked the interactive, hands-on nature of the training but suggested improvements

included additional “refresher” training, particularly on the use of the technology. Teachers

mentioned in focus group interviews and on surveys that the training provided a lot of

information—perhaps too much information—to be properly digested in a two-day session and

they would have benefited from an opportunity to revisit the technology after they had started

using it.

Three-quarters of the teachers (75 percent) clustered the administration of the constructs when

implementing the assessment for the first time in the fall. Most teachers who clustered the

37%

37%

39%

45%

50%

50%

52%

64%

69%

75%

77%

Q&A panel with returning pilot teachers

Practice "Getting a plan"

Practice "Systematically using data"

Practice "Summarizing the status"

Overview of formative assessment

Practice "Assigning a learning status"

Practice "Looking for and gathering evidence"

Demonstration of the Teacher Site

Overviews of five domains

Demonstration of technology platform

Overview of construct structure



constructs found it a helpful (39 percent) or at least somewhat helpful (57 percent) strategy for

managing the new content. Several teachers commented that it was more useful to plan and

customize the construct groupings around what they were teaching at that time rather than use

any of the suggested clustering options.

Materials and ongoing supports To address the question about what ongoing PD and other supports are needed to implement

the assessment well, we examined the responses of teachers who were surveyed at the end of

the fall and winter administrations (n = 135 and 133 teachers, respectively), as well as their

comments in the focus groups conducted at the end of the field test (n = 111 teachers). We

begin with a description of the materials and supports provided to teachers.

Materials

Each teacher received a K–3 Formative Assessment binder containing overview guidance on

how to implement the assessment and assessment materials for each of the 18 constructs.

• Construct progression document with performance descriptors and classroom examples

for each skill

• Assessment means form describing how to assess the construct, including situations and

tasks as needed to supplement observations

• Single and multiple child documentation forms for teachers to record their evidence,

associated preliminary learning statuses, and status summaries; these forms also included

“watch fors” for each skill on the progression, brief descriptions of the behaviors that

differentiate each skill from the next.

Ongoing supports

Ongoing professional development and other supports were available after the field test

trainings.

• Teachers were given access to the website Teacher Site that had all these construct-

specific materials, as well as a narrated training module for each construct and the resource

materials used at the field test trainings. A page from this website, showing where to find

materials for each of the 18 constructs, is presented in Exhibit 4.2


Exhibit 4.2. Sample page from Teacher Site showing construct materials

• Follow-up webinars were held for interested teachers early during the fall administration and

then either at the end of the fall data collection or at the beginning of the winter data

collection window. This included review and questions about the technology platform and

evidence application.

• Staff members from the technology platform vendor were available via email or help desk to

answer teachers’ questions about the platform or evidence application.

• The field test data collection was monitored, and teachers were offered help via email if they

appeared to have questions about how to document their assessment activities, based on

what they had entered into the platform.

Teachers’ perceptions of the supports

Teachers were asked in surveys and focus group interviews about the quality and helpfulness of

the materials and resources provided and what additional supports would be useful in


implementing the assessment beyond the field test. Most of the 134 teachers who responded to

the survey found the materials helpful (Exhibit 4.3). Specifically, they found the construct

progressions (96 percent), technology platform (90 percent), Teacher Site, (88 percent), K–3

Evidence Application (85 percent), and the Assessment Means Form (78 percent) to be the

most helpful materials and resources in implementing the assessment. Most teachers also

reported these assessment materials to be of high or moderate quality (90–94 percent).

Exhibit 4.3. Percentage of teachers rating materials as very or somewhat helpful

Only 44 percent of teachers (n = 59) reported that they watched at least one of the construct

modules, which were available on the Teacher Site and recommended for review after the

training as part of a self-study preparation for assessing each construct. Teachers most

frequently watched the modules for constructs perceived as more difficult to assess, such as the

social-emotional constructs and Mathematical Patterns. Nearly all teachers who viewed the

construct modules found them helpful (ranging from 94 to 100 percent across constructs).

Teachers who did not watch the modules either felt sufficiently prepared from the training,

materials, or from participating in the pilot (49 percent) or did not have time to watch the

modules (39 percent).

When asked whether their school had provided support for the implementation of the

assessment, 56 percent of teachers reported at least some kind of support (Exhibit 4.4).

Teachers’ suggested supports for future successful implementation of the assessment included

increased time allocated for planning (77%) and having team meetings or other opportunities to

48%

50%

63%

78%

85%

88%

90%

96%

Single Child Form

K-Entry Checklist

Multiple Children Form

Assessment Means Form

K-3 Evidence App

K-3 Teacher Site

K-3 Technology Platform

Construct Progression



discuss the assessment with colleagues (35%). In interviews and open-ended survey

responses, teachers also noted that they would benefit from more practice with the technology,

additional examples of activities they can use to assess each construct, more practice coding

real examples of evidence, and examples of how other teachers fit the assessment into

everyday instructional activities.

Exhibit 4.4. Percentage of teachers reporting supports provided and suggestions for supports needed for assessment implementation

Technology Teachers’ use of the assessment was supported by the technology platform and by the use of

tablets or smart phones. Many field test teachers were provided with tablets funded through the

grant to facilitate gathering evidence (especially videos and photos). All the assessment data

were recorded on the platform for easy review and reporting by teachers.

To address the research question concerning the role of technology in supporting fidelity of

implementation and instructional utility, we asked teachers about their use of the technology

platform and the mobile evidence application, both in surveys and during the focus groups. The

majority of teachers found the technology platform (78 percent) and mobile application (71

percent) helpful in implementing the assessment. In focus groups, teachers noted that the

technology was critical for making the assessment more manageable by expediting the

uploading of data (using the application), monitoring their data collection process, monitoring

student progress, and digitally organizing large quantities of information. Some teachers

2%

16%

18%

35%

77%

21%

3%

15%

24%

12%

Access to computer/laptop in classroom

Improved Internet access/connectivity

Professional learning communities

Team meetings to discuss

Increased time allocated for planning

Supports Received Supports Suggested



reported challenges with using the technology, including technical difficulties accessing the

technology platform or application (26 percent), not enough practice with the technology

platform or application before using it in the classroom (13 percent), and problems using the

technology platform or application. Staff members from the tech platform vendor were available

to help teachers with the technology, but they were not able to resolve district and school-level

issues (e.g., availability of tablets, lack of connectivity or bandwidth).


Chapter 5: Broader implementation supports Broader supports, beyond the teacher supports discussed in chapter 4, are required to implement the K–3 Formative Assessment with fidelity and produce valid information. The field test examined three broader system-level supports: professional development for administrators, building-level infrastructure, and classroom conditions. The findings are summarized here, with more detail provided in a related report by Child Trends.

Implementation of the K–3 Formative Assessment requires that teachers understand the

contents of the assessment and successfully manage the collection and entering of information.

We hypothesized and the pilot test results confirmed that successful implementation would

benefit from multiple types of support. These could include supports directly related to the

assessment (e.g., the provision of materials and technology), as well as broader supports that

facilitate the formative assessment process (e.g., provision of time, grade-level teams, and

classroom conditions). Administrators can play an active role in supporting successful

implementation by creating an environment conducive to the formative assessment process.

The field test addressed three research questions about broader implementation supports:

• What were the nature and quality of the initial professional development provided to administrators and others?

• What building-level infrastructure is seen as needed to support teachers in implementing

the assessment with fidelity and to use it to guide instruction?

• What are the classroom conditions that support or serve as barriers or facilitators to

implementing the formative assessment well (to fidelity) and using it to guide instruction?

To address these questions, Child Trends surveyed the administrators of teachers participating

in the field test near the end of the child assessment data collection in March−April 2017. In

addition, SRI included questions in teacher surveys and focus groups about the kinds of

building-level supports and classroom conditions that could sustain successful implementation

of this assessment. The factors that teachers reported were important to their assessment

implementation efforts are presented in the Child Trends report (Gebhart, Pratik, & Lin, 2017)

and summarized here.

Professional development for administrators Because we hypothesized that administrators play a key role in supporting teachers in the

formative assessment process, we developed and provided training for administrators of

teachers participating in the field test. This was either direct in-person training or virtual follow-


up webinars or training modules developed for building- and district-level administrators. This

training covered four topics, each in a separate online module:

• Module 1: What Does This Assessment Look Like?

• Module 2: Project Purpose and Background

• Module 3: Data Security and the IRB (institutional review board)

• Module 4: Supporting Your Teachers

Module 4 included suggestions on how administrators could support their teachers, such as

providing them with opportunities for reflection and collaboration with other teachers, discussing

the assessment in professional learning communities or staff meetings, and providing planning

time for teachers to work individually or meet with others implementing the assessment within

and across grades. These suggestions were summarized and shared with administrators as

part of the training (Appendix D).

Child Trends surveyed 73 principals and district-level administrators near the end of the field

test (March−April 2017) about the training they had received, their involvement in the K–3

Formative Assessment field test, resources provided to teachers, and facilitators and barriers to

assessment implementation. A total of 29 administrators (22 school principals and 7 district-

level administrators) responded to the survey. This was approximately 40 percent of the

administrators with teachers participating in the field test. Among the survey respondents, 19

reported that they had received training: 18 attended the in-person training, 1 also participated

in a follow-up webinar, and 1 viewed the administrator modules in lieu of the in-person training.

Most administrators reported benefiting from these trainings (e.g., 89 percent reported it as

“very” or at least “somewhat” helpful), and they also recommended additional follow-up trainings

throughout assessment implementation.

Not surprisingly, administrators who did not participate in any training reported that it was

difficult for them to support the implementation of this assessment. The in-person training was

held directly before school started so those administrators who attended had little time to

communicate with teachers about the assessment process or make adjustments in planning or

provision of supports before the field test began. Some administrators had difficulty seeing the

benefit of implementing the K–3 Formative Assessment in their schools, which may have

contributed to less than ideal support or buy-in for the assessment during the field test. The

administrators’ failure to see the benefit of the assessment did not create a problem for the

teachers, however. Relative to other challenges, only 13 percent of the field test teachers


specifically cited lack of buy-in or support from administration as a challenge in implementing

this assessment.

On the other hand, teachers did report that administrator support would have improved the

assessment process. As reported in Chapter 4, when asked what kinds of supports would be

needed to implement the assessment beyond the field test, many teachers in both the survey

and focus group interviews identified the kinds of supports that require administrator action;

among them were

• Providing additional planning time (77 percent)

• Providing additional staffing support in the classroom (57 percent)

• Convening team meetings (35 percent) or professional learning communities (18 percent) to

discuss the assessment

• Training additional staff in the building (e.g., administrators and teachers) (33 percent)

• Providing better access to technology (i.e., better Internet connectivity and provision of

tablets) (28 percent).

One teacher made the distinction between moral support and substantive support, noting that

her administrator was very supportive in letting her attend training and implement the

assessment in her classroom but had not provided any of the substantive supports that would

have been useful.

Administrators might be more willing and able to support teachers in using the assessment if

they had a better understanding of how the assessment can inform instruction. A small portion

(14 percent) of teachers reported that they did not use information from the assessment to

inform their instruction, some explaining that they did not know how to use the information

gleaned from the assessment. Supporting the teachers in learning how to use the data was

outside the scope of the field test, but it is likely that both teachers and administrators would

experience more benefits from implementing the assessment if teachers were better prepared

to use assessment results in their instruction.

Building-level infrastructure The administrator training materials identified several effective practices that support the

formative assessment process, including:


• Creating a positive classroom culture (ideally an environment with less need for discipline,

less time spent on behavior and more time supporting positive relationships with teachers

and peers)

• Use of multiple instructional formats with opportunities for active learning, including use of

whole group, small group, centers, situations, pairs, and individual learning

• Engagement in purposeful and intentional planning and decision-making, including

collaboration with peers to discuss evidence, the assessment process, and planning.

Teachers confirmed the message provided to administrators in the training; more than three-

quarters (77 percent) recommended that additional time for planning (e.g., how to incorporate

the assessment into instruction) would be helpful in successfully implementing the assessment

beyond the field test. When asked what kind of supports they had received, 44 percent of

teachers reported that they did not receive any of the supports listed to assist with

administration of the K–3 Formative Assessment. Among those who reported receiving support,

the most common were allowing teachers to hold team meetings to discuss the assessment (24

percent) and providing access to a computer or laptop in the classroom (21 percent).

Administrators identified additional administrator training and a desire to incorporate other staff,

such as school psychologists or occupational therapists, into the assessment process. Teachers

concurred that training more school staff and having them assist in collecting data for the

assessment would be beneficial.

Classroom conditions Teachers implemented the assessment under different classroom conditions. These conditions

varied in ways including

• The extent to which teachers had technology available to support the assessment

• Teachers’ own previous training and preparation to implement formative assessment

• How teachers allocated class time across different learning arrangements.

Technology available to support the assessment

Teachers identified technology as a critical component of assessment implementation. All

teachers reported having Internet access in their classrooms, and 77 percent reported having

“good” internet access. Almost all teachers (95 percent) had access to a computer, and 87

percent had a tablet or other mobile device that they could use in administering the assessment.

Although all teachers had access to the Internet, some reported during focus groups that


intermittent and slow Internet speed delayed uploading evidence. Some teachers chose to

upload evidence at home or opted not to upload audio, video, and/or image files because of the

Internet issues at the school. Teachers reported in the pilot and again in the field test that

access to reliable technology, including a tablet in the classroom to facilitate data collection and

a high-quality Internet connection to expedite uploading data collected via tablets and

smartphones, was essential to implementing the assessment.

Teacher training and preparation to implement formative assessment

Half the field test teachers reported having participated in training on formative assessment

before the field test, averaging about 15 hours of professional development with a range of 2 to

60 hours. More than 75 percent of teachers reported that they were already using formative

assessment in their classroom before this project. This information would suggest that most

teachers should have been comfortable with the formative assessment process, but the

teachers’ desire for more training with the K–3 Formative Assessment process suggests they

were not familiar with this type of formative assessment.

Class time spent in different learning arrangements

Successful implementation of this assessment requires that teachers be able to observe both

academic and nonacademic constructs in a variety of classroom settings and activities.

Teachers who have students spend time in a variety of arrangements (e.g., working in small

groups, individually, with peers) should find it easier to observe their students and capture the

skills on the assessment progressions. Teachers surveyed during the field test reported

considerable variation in how children in their classrooms spent their time:

• Working with peers (under teacher direction). Approximately half the teachers (51 percent)

reported that children spent 1 hour per day working with peers, and another one-quarter of

teachers reported that children spent 2 hours working with peers.

• Working independently. About half the teachers (48 percent) reported children spent 1

hour working independently, and another one-quarter of teachers reported that children

spent 2 hours working independently.

• Working in small groups (with teacher). About 42 percent of teachers reported they spent

an hour per day working in small groups with students. Few teachers (14 percent) reported

spending 2 or more hours in small group instruction.


The amount of classroom time teachers spent in the different learning arrangements is shown in

Exhibit 5.1. These data suggest that teachers used a variety of classroom arrangements, which

should have provided ample opportunity for them to observe the constructs in the assessment.

Exhibit 5.1. Time spent in different classroom learning arrangements

22%

24%

17%

43%

27%

48%

38%

51%

42%

46%

26%

26%

24%

12%

24%

5%

12%

8%

0% 20% 40% 60% 80% 100%

Working independently

Working on individual tasks under teacherdirection

Working with peers under teacher direction

Working in small groups with teacher

Teacher lecture/large group discussion


Half hour or less About 1 hour About 2 hours 3-4 hrs or more


Chapter 6: Using the K–3 Formative Assessment with all children This chapter summarizes the findings and recommendations of expert reviews from three perspectives: disability, dual language, and racial/cultural. We also summarize the resulting revisions to the assessment materials to improve teacher use with all children, including children with disabilities, English learners, and children from diverse racial and cultural backgrounds.

During the field test, experts reviewed the K–3 Formative Assessment for its use with different

populations of children:

• From an ability-diverse perspective, for use with children with disabilities

• From a language perspective, for use with English learners

• From a racial and cultural perspective, for use with children from diverse racial and cultural

backgrounds.

This chapter summarizes the findings and recommendations from these three reviews and

describes the changes made to the assessment based on the reviews.

Disability perspective Because the assessment is based on observing children during typical classroom routines,

multiple opportunities are available for any child to demonstrate competence. This alleviates the

problems that assessments with structured tasks and rigid instructions present for assessing

many children with disabilities. Furthermore, as part of the administration of the K–3 Formative

Assessment, teachers are instructed to use any instructional adaptations or accommodations on

a child’s IEP. As presented in chapter 2, most teachers reported that the assessment was no

more difficult to use for children with disabilities and that the information was useful for their

instruction. We did find, however, that teachers generally placed children with IEPs lower on

each construct progression than children without IEPs in both fall and winter administrations and

across grade levels.

During the ECD codesign process, each construct progression and the associated assessment

means were reviewed and adjusted if needed based on considerations for children with

disabilities. A section of the overview guidance was developed to address using the assessment

with children with disabilities, and specific guidelines were written for several constructs.

In addition, to ensure that we had fully considered the disability perspective, the assessment

materials were reviewed by Dr. Kathleen Hebbeler, a well-known expert in the assessment of

children with disabilities. Dr. Hebbeler was part of the SRI research team. She was involved

only in the enhancement of one construct (Object Counting) so was in a position to conduct an


objective review. She made several recommendations, some of which we were able to

incorporate into the final version of the assessment materials. Other recommendations will

require additional research, assessment design work, and field testing, some with children with

low-incidence handicaps. Some of these recommendations pertain to the assessment overall

and are summarized below under “Overview guidance.” Other recommendations more specific

to individual constructs are presented in Exhibit 6.1.

Overview guidance and general recommendations

The Assessment Overview and Directions That Apply to All Constructs, which is an overview

document with guidelines for use with all constructs, is one of the strengths of the assessment

with regard to children with disabilities. This guidance makes clear that

• The assessment is based on authentic interaction.

• Alternative means of communication are acceptable.

• Supports can be offered to the child.

• Accommodations are acknowledged and acceptable, along with flexible

administration/collection of evidence (e.g., positioning support, various times of the day).

Using other feedback in Dr. Hebbeler’s review, we added the following features throughout the

assessment that should make it easier for teachers to understand how to implement this

assessment with all children and the accommodations or exceptions that may be noted:

• Elaborated on guidance and added notes to the Assessment Means Form for each

construct, briefly explaining what considerations would be made when assessing this

construct for children with disabilities. This includes clarifying that the various situations

developed to help a teacher pinpoint a child’s skill level should be used as appropriate given

the child’s disabilities.

• Added examples of children with different types of disabilities throughout the Construct Progression Documents to help teachers see children with disabilities reflected in the

assessment materials.

• Added a space on the Single Child Documentation Forms for teachers to indicate when

the assessment of a construct is not appropriate for that child based on his/her disability to

explain why the teacher has not assessed the construct. We recommend that there also be

a place where teachers can indicate such information in a neutral or straightforward way on

the technology platform used to support this assessment.


Construct-specific guidance

In her review, Dr. Hebbeler noted that some constructs are not appropriate for children with

some disabilities. This is not a problem for the assessment as long as teachers have a way to

indicate this and are not required to place every child on a progression. An example of such a

progression is Gross Motor, which would not be appropriate for children with severe mobility

impairments. The review also included recommendations for adaptions or additional

development work for some constructs with some specific disabilities (for example, the need to

develop a construct analogous to Letter Naming for children learning to use Braille). Some of

these recommendations will require assessment development work and further research

specifically for children with low-incidence disabilities These recommendations are summarized

by disability groups in Exhibit 6.1.


Exhibit 6.1. Findings from review for each construct by disability

Construct Disability/Impairment More Research Notes

Hand Dominance Not appropriate for children with limited use of their arms or

hands.

Grip & Manipulation

Not appropriate for children with limited use of their arms or hands; situation not appropriate for children with limited vision.

Crossing Midline

Not appropriate for children with limited use of their arms; requires more research on how to assess children with visual impairments.

Gross Motor Not appropriate for children who cannot walk or with limited motor skills.

Emotion Expression Use of this progression likely to be more challenging for

special education professionals trained to focus on behavior.


Progression’s emphasis on articulating strategies rather than showing appropriate behavior disadvantages children with limited language.

Emotional Literacy

Need more research on relationship between emotional literacy & language disabilities; teachers will need alternative ways to assess children with limited communication skills.

Book Orientation Need an appropriate adaption for children with visual

impairment or insufficient motor skills to hold a book.

Print Awareness

Needs an adaptation for a child with visual impairment; may require more research to determine use for children reading Braille.

Letter Naming Need an adaptation for children with visual impairments; requires an expert in working with students with visual impairments to identify the appropriate skill levels.


Construct Disability/Impairment More Research Notes

Following Directions As a measure of receptive language, this progression is

heavily dependent on a child’s language skills.

Writing

Need to clarify that alternative ways of writing are acceptable for children who do not have the motor capability to draw pictures or write letters; also needs to be adapted for children with visual impairments.

Reading Comprehen-sion

Requires more research: assessing children learning to read Braille; development of comprehension strategies in children with intellectual disabilities; how children can communicate these strategies.

Vocabulary

Some skill levels dependent on the production of language; need to develop a different progression for children who sign since steps involve word structure based on spoken or written English.

Perseverance Requires teacher be able to identify activities that are moderately or very (but appropriately) difficult for children with intellectual disabilities.

Object Counting Adaptations needed for children who are not able to move

objects on their own.


Teacher must consider that each situation requires different skills that would be challenging for children with different impairments (hearing, auditory processing, motor, and visual).

Problem Solving

Heavily dependent on a child’s verbal skills; requires teacher to identify appropriate problems for children relative to their intellectual abilities; more research needed for how to interpret unexpected approaches.


English learner perspective The observation-based nature of this assessment may lend itself more readily to learning about

what young speakers of languages other than English know and can do in much the same way

as it does for children with disabilities. As noted in chapter 2, most teachers reported that the

assessment was no more difficult to use for English learners and that the information was useful

for their instruction. We found no consistent differences in how teachers placed ELs on each

construct progression compared with non-ELs across both fall and winter administrations and

across grade levels.

Many of the constructs, however, require that the teacher communicate with the child to

understand where the child falls on the progression. The overview that accompanies the K–3

Formative Assessment instructs teachers “to communicate with the child in a language . . . the

child understands and can respond in. This could be a language other than English (which could

involve an interpreter),” but it is impossible to know how feasible it is for teachers to do this.

As part of the field test, the Build Initiative arranged for Dr. Linda Espinosa, a national expert in

the teaching and assessment of English learners, to review the assessment materials. Key

points from Dr. Espinosa’s review were the following:

• Appropriate use of this assessment requires considerable professional development for

teachers, including how to work with dual language learners (DLLs) and ELs.

• Best assessment practice includes having an adult who speaks the child’s language collect

the observation data (evidence), especially for constructs that rely more heavily on language

and communication.

• The examples in the construct progression and assessment means documents should

include descriptions of children with multiple languages who are acquiring English and what

assessing these children is like.

• The appropriateness of this tool for DLLs/ELLs depends on the state’s policy regarding

English language acquisition and dual language development (which should be reflected in

the state’s comprehensive assessment system).

Although some recommendations related to the content of the assessment materials were

outside the scope of the project, we included several of the recommendations from Dr.

Espinosa’s review in the final assessment and training materials:

• Communicating with the child in a language the child understands and can respond in (part

of the Assessment Overview and Directions That Apply to All Constructs)


• Including additional instruction in the Assessment Overview and Directions That Apply to All

Constructs that speak to the need to involve an adult who speaks the child’s language in

collecting observational data, especially for those constructs that rely more heavily on

language and communication

• Adding examples to the Construct Progression and Assessment Means Forms of

children who are acquiring English so teachers can see what it looks like to assess them

• Reinforcing these points in the teacher professional development materials (training slides).

Race/culture perspective The use of teacher observation as the method of data collection in the K–3 Formative

Assessment means that each child is being seen through the race and cultural lens that the

teacher brings to the classroom. Any implicit biases that teachers have will be reflected

throughout their interactions with children and in the specific instructional activities they use,

how they engage with the children, and in where they place children on the progression. Implicit

bias refers to the attitudes people have toward other people or the application of stereotypes

without conscious knowledge. The goal was to make sure that the assessment content does not

incorporate any racial or cultural bias; creating good content is a necessary but not sufficient

condition for addressing this issue in the implementation of the assessment. Some teachers

also may benefit from additional training in culturally responsive instruction, which will improve

their interaction as well as their use of the assessment with children from different racial or

cultural backgrounds.

As part of the field test the Build Initiative arranged for Dr. Tanisha Felder, a national expert in

racial and cultural equity in schools, to review of the assessment materials. Here are key points

from Dr. Felder’s review:

• Teachers should see children from diverse racial and cultural backgrounds in the construct

progression examples of what the skills look like in the classroom. The names of the

children in these examples should reflect this diversity, be current and balanced by gender,

and convey positive meanings.

• Teachers should create ways that the child is authentically engaged in the activities being

observed in order to get a good measure of the child’s skill level on a particular progression.

Relatability and relevance of activities may vary depending on the children’s race and

cultural background. This can best be addressed through professional development for

teachers in culturally responsive instruction.


• Teachers must be aware of and consider cultural norms in their evaluation of a child’s

behavior, especially with the social-emotional constructs where the expression and

regulation of emotions may vary somewhat by culture. Cultural norms also may come into

play when teachers are giving children directions for a task. This recommendation needs to

be implemented with the acknowledgement that the cultural norms in the context of this

assessment are the classroom norms. This recommendation also should be addressed

through professional development for teachers in culturally responsive instruction.

Following Dr. Felder’s recommendations, the SRI research team made some changes to the

assessment materials.

• Changed the names of children used in the examples that appear in the Construct Progression Documents and Assessment Means Forms so that they reflect more racial

and cultural diversity and so that they are balanced by gender, in present-day use, and

convey positive connotations

• Added text in the Assessment Overview and Directions That Apply to All Constructs to

remind teachers that they can modify the situation or task directions as appropriate to reflect

the racial and cultural norms of the children in their classroom

• Reminded teachers to use activities or tasks that are likely to be engaging for the child given

his or her background and individual interests and to modify tasks if they are not engaging

• Reinforced these points in the teacher professional development materials (training slides).

Because the assessment is based on teacher observations of authentic behaviors in daily

classroom routines, teachers will need to be skilled in culturally responsive instruction, which

might require additional professional development so they can both instruct and assess a child’s

skill level in culturally responsive ways.


Chapter 7: Summary, revisions, and recommendations This chapter summarizes the findings from the field test including how each of the construct progressions performed vis a vis the validity claims. It concludes with a summary of the revisions to the assessment and training materials and recommendations for future research.

Overall assessment findings In chapters 2 through 5 we answered two overarching research questions posed in the field test

regarding the validity claims for the K–3 Formative Assessment:

1. Does the K–3 Formative Assessment provide valid information about what children know

and can do?

2. What supports are required to allow the assessment to be implemented with fidelity and

produce valid information?

To answer the first question, we addressed a series of subquestions that reflect the validity

claims for the K–3 Formative Assessment. Below we summarize the findings for each

subquestion followed by a more detailed, construct-specific examination of the validity claims.

• Are the skill levels in the construct progressions in the proper sequence? Based on

the opinions of experts and teachers and the assessment data collected during the field test,

the answer is yes, the skills in these construct progressions appear to be in the right order.

However, psychometric analyses indicated that two skill levels on two progressions (Object

Counting and Mathematical Patterns) were ranked very close to each other and thus

including both skill levels as unique steps in a progression may offer little instructional utility.

A limitation of the field test was that the sample was too small (i.e., fewer than 10 children)

to verify the sequence of skill levels at the beginning or end of some progressions.

• Do the performance levels accurately reflect what is expected at entry to kindergarten? Using expert and teacher judgment and the fall assessment data, we

established performance levels in each progression that reflect what is expected of children

at kindergarten entry. For four constructs, experts and teachers disagreed about where

entering kindergarteners should be, and experts commented that kindergarten teachers tend

to underestimate children’s abilities for these constructs. The field test data were somewhat

limited for verifying the performance levels, in that the levels were established for entry to

kindergarten and the field test data were collected over the first two months of school, and,

furthermore, most of the field teachers were new to the assessment. We recommend


additional validation of the performance levels with assessment data collected in the first few

weeks of school by teachers who are experienced with the assessment.

• Is the assessment sensitive to children’s growth and development over time? Teachers reported and the assessment data confirmed that the constructs captured an

increase in children’s skill level over time, both within the school year and from one grade

level to the next. On several constructs, children were placed at the top of the progression in

the fall of kindergarten or first grade, indicating that most children had no opportunity for

growth. Therefore, these constructs offer limited instructional utility beyond these points, but

teachers may still find them useful for monitoring the learning of children who struggle in

these areas.

• Does the assessment provide information that is useful for instruction for grades K through 3? The answer to this question varied by construct and to some extent by grade

level. A majority of the teachers reported instructional utility for all of the constructs for all

children, including children with disabilities and English learners. However, even though

Writing was the construct with the largest number of teachers endorsing its instructional

utility, nearly a fifth (19 percent) of the teachers reported it was not useful. Likewise, about

one-third of the teachers did not find the constructs in the physical/motor development and

the social-emotional development domains useful for informing instruction; teachers

indicated that they did not see the relationship to the curriculum. We recommend further

research on these constructs and the supporting materials to determine what would be

needed to improve their instructional utility for teachers.

• Can teachers learn to implement the assessment with fidelity, including reliably coding a child’s learning status? The evidence teachers documented on the technology

platform indicated that teachers were able to collect multiple pieces of evidence, assign a

preliminary learning status to the evidence, and interpret that information to reach a status

summary placement on the progression that was consistent with the evidence. The field test

assessed reliability by comparing teachers’ status summaries with ratings of an external

reviewer, and for most constructs there was high agreement. The three constructs where the

agreement was less than 60 percent were Emotional Literacy, Emotion Regulation

Strategies, and Object Counting, suggesting the data from these constructs might be

somewhat less reliable. Teachers also reported some challenges with implementation. For

example, they indicated that the data collection process was time-consuming and

documenting evidence on the platform cumbersome. These issues could hinder teachers’

ability to implement the assessment as intended.


• Can teachers confidently determine a child’s learning status? Teachers were confident

in their status summary placements for most constructs. Teachers were less sure about

determining children’s learning status on the social-emotional constructs than on the other

constructs.

• Can teachers incorporate the administration of the assessment into classroom routines? Teachers found that the constructs most directly related to their instruction and to

other assessments, such as those in the language development and communication and

cognitive development domains, were easier to assess and fit into their daily routines. Some

teachers struggled with constructs that they considered outside their typical classroom

activities such as Gross Motor, Mathematical Patterns, and the social-emotional constructs.

To answer the second research question about the implementation supports required to

implement the assessment with fidelity and provide valid information for instruction, we used

data from the teacher surveys, teacher focus groups, and administrator surveys.

• What was the nature and quality of the initial professional development provided to teachers? The field test teachers received two days of face-to-face training that covered the

content and process of the assessment and use of the technology. Follow-up webinars and

online modules on each construct were available to provide more in-depth information.

Teachers generally liked the training provided for the field test and felt it prepared them to

implement the assessment. They recommended more time be devoted to reviewing the

technology and implementation ideas from their colleagues once they began using the

assessment.

• What ongoing professional development and other supports are needed to implement the assessment well? The field test yielded limited information to answer this question

because the full range of ideal supports for the formative assessment was not represented.

Ideal implementation conditions would include, for example, committed administrators,

sufficient planning time, and opportunities for teachers to work together in implementing the

assessment and using the results. Because these supports did not exist in most of the sites,

the field test was not able to explore their effects on implementation. In surveys and

interviews, however, administrators and teachers reported that these kinds of supports

would be helpful to implement the assessment.

• Are there changes to the assessment and training materials that would make the assessment more manageable? Teacher feedback, input from experts from disability,

language and race/culture perspectives, and analysis of the assessment data suggested a


need for minor changes to the assessment materials. We have made revisions to the

assessment and training materials based on this input. Specifically, we revised examples

provided in the construct progression documents, clarified progression skill levels in the

assessment means forms, and repackaged the training materials to more fully cover the

most critical topics for implementation of the assessment.

• What was the nature and quality of the initial professional development provided to administrators and others? Training was provided to administrators either in person or

virtually through follow-up webinars or training modules that were developed for building-

and district-level administrators. According to their survey responses, fewer than half of the administrators associated with the field test teachers either attended an in-person training or

took advantage of the other training opportunities. The survey respondents found the

training beneficial and recommended more follow-up training opportunities once teachers

begin implementing the assessment.

• What building-level infrastructure is seen as needed to support teachers in implementing the assessment with fidelity and to use it to guide instruction? Field test

findings indicated that teachers need more time to plan for and implement the assessment,

including collecting evidences and loading the information onto the platform; opportunities to

hold team meetings to discuss the assessment; and a computer or laptop in the classroom.

• What are the classroom conditions that facilitate or serve as barriers to implementing the formative assessment well (to fidelity) and using it to guide instruction? Teachers

need access to the Internet in the classroom to enter or upload the data onto the technology

platform. Although all teachers had Internet access, not all of them had reliable access,

which was a barrier. Teachers reported using a variety of learning arrangements in their

classrooms including small groups, working with peers, and working independently. Having

a variety of arrangements should provide sufficient opportunities for teachers to observe

what is being captured in the construct progressions.

Findings by Construct To summarize the differences in findings across the constructs, we created a table with the

findings for each test of each validity claim for each construct (Appendix D, Exhibit D.7.1). Some

of the claims had multiple tests so we further summarized the evidence for each claim (see

Exhibit 7.1). We color-coded the evidence to indicate how strongly it supported each claim for

the construct, with darker colors indicating stronger support. Looking across the matrix, we can


see that some constructs performed quite well for all of the validity claims while others

presented a more mixed picture.

• The constructs with the strongest evidence were in the language development and

communication and physical/motor development domains: Writing, Book Orientation, Print

Awareness, Letter Naming, Reading Comprehension, Following Directions, Hand

Dominance, Grip & Manipulation, and Crossing Midline.

• The evidence was not as strong for constructs in the social-emotional development and

cognitive development domains: Emotion Regulation Strategies, Emotion Expression,

Emotional Literacy, Mathematical Patterns, Problem Solving, and Object Counting

• In addition, the evidence patterns for Gross Motor, Vocabulary, and Perseverance suggest

the need for additional discussion and follow up.

Below we discuss the findings for each construct within their domain and offer some

recommendations for revisions and additional professional development or research.


Exhibit 7.1. Summary of construct performance for each of the validity claims

*Approaches to learning


Language Development & Communication Domain

The seven constructs in this domain comprise preliteracy skills frequently assessed in preschool

and kindergarten (Book Orientation, Print Awareness, and Letter Naming), more traditional

elementary school subjects (Reading Comprehension, Writing, and Vocabulary), and one

measure of receptive language that is important to success in any elementary classroom

(Following Directions). Collectively, these constructs were well supported by most of the

evidence for most of the validity claims. These constructs have a strong research base and are

prominent in elementary curricula and assessments. The strong research base meant there was

much to draw on in developing the constructs, and their frequent inclusion in standards,

curricula, and assessments meant teachers were familiar with them. It is not surprising that

these progressions performed well with regard to sequencing of levels, opportunity for growth,

teacher confidence, and fit within classroom routines. They also fared better than constructs in

the other domains in terms of their perceived instructional utility and the extent to which

teachers could reliably determine progression placements based on the evidence they

documented.

Three language development and communication constructs would benefit from additional

research and development:

• Book Orientation was found to have limited instructional utility because most children were

placed at the highest level on the progression at kindergarten entry. This also meant the

construct was not sensitive to growth. A future decision is whether Book Orientation belongs

in an assessment designed for kindergarten through Grade 3.

• Following Directions was one of the constructs where the national experts and

kindergarten teachers did not agree on what entering kindergarteners should be able to do.

The performance levels were set based on expert recommendations, which resulted in only

45 percent of the field test sample assessed in the fall of kindergarten categorized as

“demonstrating” the expected skill level, whereas the percentages for most of the other

constructs were 90 percent or higher. Further research is needed to understand the

differences in how experts and teachers interpret the levels of this construct.

• Vocabulary measures a child’s conceptual understanding of how words work in our

language (e.g., synonyms, multi-meaning words, word structure, nuanced language) rather

than whether children have mastered the appropriate words at each grade level. Some field

test data suggested that teachers may be more familiar with the latter conceptualization of

vocabulary. Compared with other constructs, fewer teachers thought this construct captured


growth well, the summary status was less reliable, and teachers found the construct difficult

to assess. Together these findings suggest a lack of teacher understanding of the construct.

Teachers would benefit from more professional development on this construct, and more

research could strengthen its effectiveness in the assessment.

Physical/Motor Development Domain

Three of the four constructs in this domain — Hand Dominance, Grip Manipulation, and

Crossing Midline—were supported by the evidence for most of the validly claims. The exception

was Gross Motor which teachers did not see as relevant to what they teach, nor did its

assessment fit into their instructional routines. Instructional utility was a challenge for all of the

motor constructs; fewer teachers saw the constructs in this domain as useful for instruction

compared with constructs in the other domains. Another issue was opportunity for growth. More

than 70 percent of entering kindergarteners were at the top level of the progression on Hand

Dominance and Crossing Midline. These constructs might be most appropriately used as part of

a screening measure to identify children who are struggling in these areas. Teachers would then

need additional support to know how to help children in this area, to refer them for additional

evaluation, or both. The Gross Motor construct would benefit from additional research to

validate the sequencing of some of the skill levels. Also, few children were placed at the lower

end of this progression in the field test which meant that these skill levels are more appropriate

for pre-kindergarten children and that there were very little data at this end of the progression to

examine the ordering of the levels. More research is needed at the higher end of the

progression because teachers had difficulty identifying galloping and distinguishing it from

skipping.

Cognitive Development Domain

The three constructs in this domain (Object Counting, Mathematical Patterns and Problem

Solving) did not fare as well as the other constructs with regard to the validly claims. Teachers

were confident in their progression placements for these constructs and in the ability of these

progressions to capture growth and development over time, which was borne out by the

assessment data. However, the psychometric analyses suggested some minor issues with

sequencing of two of the skill levels in Object Counting and Mathematical Patterns. The

reliability of the status summary for Object Counting was lower than that of most of the other

constructs. The teachers and experts disagreed on performance expectations at kindergarten

entry for Mathematical Patterns and Problem Solving. The teachers were less able to fit


assessing Mathematical Patterns and Problem Solving into their classroom routines than other

constructs.

Object Counting appears to need more research on the structure of the progression.

Mathematical Patterns needs more research on structure and also presented problems for

teachers because they did not see it in their standards or curriculum. This might be resolved by

additional professional development or by the development of additional mathematical

progressions that are more aligned with the curriculum. Teachers recognized Problem Solving

as useful, but they may not have understood it well and it was challenging for them to assess.

More research and professional development on this construct may be needed to help teachers

collect information as part of everyday classroom routines.

Social-Emotional Development and Approaches to Learning Domains

For purposes of the construct summary, we have grouped Perseverance, the one measure of

the Approaches to Learning domain, with the three social-emotional constructs (Emotion

Expression, Emotion Regulation Strategies, and Emotional Literacy). The data on the validity

claims present a mixed picture for these four constructs suggesting a need for more work. The

findings support the order of the skills in these progressions and indicate that they are sensitive

to growth over time. Teachers agreed with experts on expectations for K entry performance.

Teachers reported during the pilot and again during the field test that the social-emotional

constructs were difficult to assess and about a third of the teachers did not see their

instructional relevance. The evidence that teachers provided to document where they placed

children on these progressions indicated that some teachers were not able to differentiate the

three constructs and did not understand what each was intended to measure. Although teachers

found the guidance for assessing these constructs helpful, teachers still reported that these

constructs were the hardest to assess, and the least useful in informing instruction, and they

yielded the lowest percentages of interrater agreement. On the other hand, some teachers in

the field test were very excited to see the social-emotional constructs included in the

assessment because they recognized the importance of these skills.

In contrast to the language development and communication domain, these domains are based

on a rapidly expanding but much smaller and newer body of research. The field is still

considering what to call some of the constructs within these domains and to how to differentiate

them from one another. These domains are well represented in standards for early childhood

across states but they are rarely included in K–12 standards. Accordingly, the kindergarten

through grade 3 teachers in the field test were less familiar with the content of these


progressions and the implications of the content for what they do in the classroom. Both the

conceptual issues and the absence of standards in these areas presented problems for the

constructs in these two domains. All four constructs would benefit from additional research on

the progression to clarify the content and make it easier for teachers to embed their assessment

into classroom activities. Without an educational context across preservice training and school

systems that recognizes and supports the importance of the social-emotional domain, it is

difficult to see how more professional development focused solely on the assessment will

address the entirety of the challenges in this domain, but clearly more professional development

is needed.

Based on the field test findings, we make the following recommendations:

• The focus on social-emotional development in the early elementary grades is bigger than

this assessment. It requires that teacher preparation programs and schools increase their

recognition of the importance of social-emotional development to learning and that teachers

can and should help children acquire skills and strategies in this domain. Teachers need to

be knowledgeable about social-emotional development and how they can support children

in acquiring skills in this domain. The professional development around these constructs

should continue to stress the contribution of social-emotional development to the child’s

success in school and how teachers can observe these skills in their students.

• These constructs would benefit from further development that incorporates both the

research and teachers’ perspectives in identifying skills that can be more easily observed as

part of classroom routines.

• Consider basing Emotion Regulation Strategies on observable behavior and do not require

the teacher to learn or infer what is going on in the child’s mind regarding the strategies he

or she employs.

Recommended grade levels for each construct Not all of the progressions were intended to be used from entry to kindergarten through grade 3,

but data on child performance were needed to identify precisely which constructs were suited

for which grade levels. The field test provided information to identify the appropriate grade levels

for each construct. We determined the grade levels in which to administer each construct for the

field test based on the results of the 2015–16 pilot study. When the data were unclear we

included higher grade levels to make sure we had tested the construct’s instructional utility at all

potentially relevant grade levels. As shown in Exhibit 1.1, this resulted in field testing all 18


constructs in kindergarten, 16 constructs in grade 1, 12 constructs in grade 2, and 11 constructs

in grade 3.

Based on the percentage of children in a grade level placed at the highest level of the

progression in the fall (see Exhibit 2.14), we made recommendations for the grade levels at

which each of the progressions has the greatest instructional utility. These are shown in

Exhibit 7.2. Any of the constructs would be appropriate for a child at any grade level who is not

on the highest level of the progression and therefore in need of additional support around the

skills measured by the construct.

Revisions/recommendations for assessment materials Based on the field test findings, we made some minor changes to the assessment materials to

further clarify some constructs for teachers. The review of evidence teachers documented as

part of placing children on progressions provided some insight into what teachers did not

understand about some skill levels on a particular progression. We also have recommended

additional research where changes in the progression itself should be considered. Where

possible, we identified the specific issue with constructs in need of more research. Exhibit 7.2

shows the construct progressions for which we made revisions to the assessment materials

based on the field test findings, the constructs for which additional professional development

may be needed, and the constructs for which we are recommending additional research. For the

few progressions that did not have good evidence across the set of validity claims, we

recommend that these progressions be used with caution until the additional research is

conducted.


Exhibit 7.2. Summary of construct-specific revisions and recommendations

Domain/Construct New

Recommended Grade Levels

Assessment Materials Clarified

Additional Professional Development

Recommended

Additional Research on Progression

Recommended

Additional Information

Hand Dominance (K) K entry Most children were already at top of progression in fall of K.

Grip & Manipulation (K–1)

Crossing Midline (K–1) K entry Most children were already at top of progression in fall of K.

Gross Motor (K–3) K-1 only Teachers continue to confuse galloping (Skills F and G) and skipping (Skill I); consider collapsing these skills or further clarify the difference in PD. Need more data to validate early steps in the progression that are more appropriate for pre-K.

Emotion Expression (K–3)

Teachers documented evidence for one construct that better fit another S-E construct and so may not understand the differences among the three constructs; we have provided examples and guidance to help teachers better distinguish the constructs. Emotion Regulation Strategies. It is difficult to observe children using strategies unless they verbalize them; consider refocusing progression skills more to self-regulation behaviors that can be observed, rather than on strategies.

Emotion Regulation Strategies (K–3)

Emotional Literacy (K–3)

Book Orientation (K) K entry Most children were already at top of progression in fall of K.

Print Awareness (K–1) K only Most children were already at top of progression in fall or winter of K.

Letter Naming (K–1) Clarified that children can identify letters by either naming them or selecting letters named for them to demonstrate letter naming.

Following Directions (K–3) Additional research needed to examine difference between K

teachers and experts on expected performance levels at K entry.

Writing (K–3) Clarified that teacher can establish purpose of writing (Skill D) by nature of assignment & student’s response.


Domain/Construct New

Recommended Grade Levels

Assessment Materials Clarified

Additional Professional Development

Recommended

Additional Research on Progression

Recommended

Additional Information

Reading Comprehension (K–3)

Changed name to Reading Comprehension Strategies. Evidence documented suggests additional PD may be needed to emphasize that children monitor for meaning as they read, as indicated by their response to miscues and strategies used.

Vocabulary (K–3)

Changed name to Vocabulary Concepts. Clarified difference between use of synonyms for concrete words (Skill C) and use of more nuanced language with subtle differences (Skills G and H). Multiple indicators suggested that teachers had difficulty assessing this progression so more PD may be needed.

Perseverance (K–3)

Skills E–H involve identifying activities which are moderately or very (but appropriately) difficult for child, which requires detailed knowledge across areas of the curriculum as to the precise difficulty level of a task for each child. Need to provide teachers more guidance on how to identify what is an appropriately challenging task for each child.

Object Counting (K–2) K–1 only Skills B, C and E are conceptually connected (consistently counts 4–6 objects) and need to be differentiated from Skill D (cardinality); lower reliability suggests additional teacher PD may also be needed.

Mathematical Patterns (K–3)

Need to do more to help teachers understand differences between sequential and growing patterns and between recursive and functional rules; may need research into other early mathematics progressions that are more aligned with K–3 curriculum.

Problem Solving (K–3) Need to provide teachers more guidance on how to identify what is an appropriately novel and challenging problem for each child.


Clarify the meaning of the construct

Based on teacher questions and comments during the field test and the nature of the evidence

they documented on the technology platform, we recommend that the names of two constructs

be modified to clarify the essence of the constructs:

• Reading Comprehension. Some teachers seemed to assess this construct assuming it

was a conventional measure of reading comprehension in which children read passages

and respond to questions about what they have read. Instead, this construct addresses the

child’s ability to monitor for meaning and apply comprehension strategies when reading. The

first few steps on this progression (Skills A through C) are pre-reading skills so the emphasis

on comprehension strategies is not as important as it is in the later steps. Starting at Skill D,

the child is reading and the focus of the construct is on comprehension strategies.

Consequently, we recommend that the construct be renamed Reading Comprehension Strategies and have changed the name accordingly in the assessment materials.

• Vocabulary. Similarly, some teachers seemed to assume that the Vocabulary construct was

about the word lists and word walls so common in early elementary classrooms. The skills

on the progression address the child's conceptual understanding and use of words. To help

teachers focus on the concepts represented by the different skill levels in this progression,

we recommend that the construct be renamed Vocabulary Concepts and have changed

the name accordingly in the assessment materials.

Clarify progression skills (examples, watch fors, assessment means)

In some of the progressions, there are examples of a particular skill that did not appear to be

well understood by the teachers based on their comments or the evidence they used to

document placements on the progression. Rather than change the skill itself, which had been

vetted by experts in both child development and assessment, we made clarifications in the

assessment materials: examples to better illustrate the skill in the construct progression and

clarifications regarding how to assess skills. We have made the revisions listed in Exhibit 7.2.

Conduct additional research on some constructs

As summarized earlier in this chapter and in chapters 2 and 3, the validity evidence for some of

the constructs was not as strong as we would have liked. We recommend additional

development work for some constructs including reconsideration of the construct itself or some

of the steps in the progressions for constructs in the social-emotional development and

cognitive development domains (see Exhibit 7.2).


Revisions to training materials and recommendations We edited the training materials to make them more appropriate for general use. In addition, we

made some changes based on the field test findings.

Revisions to teacher and administrator training materials

Materials have been edited to remove references to the field test and to the technology platform

vendor who supported the field test. Professional development materials were repackaged into

thematic units so that the materials can be more easily adapted for use as states and districts

implement the assessment. We did not change the contents of the individual progressions. The

materials for training and ongoing support continue to include:

• Teacher training materials comprised of presentation slides, handouts, and hands-on

activities that cover the following topics: an overview of formative assessment and the K–3

Formative Assessment; an overview of the five domains covered in the assessment; a

review of the K–3 formative assessment process; an activity guide and related handouts to

accompany the review of the K–3 formative assessment process; and a guide for

customizing the training to state and/or local needs.

• Online modules for each construct, intended for self-study as teachers prepare to assess

each construct.

• Administrator training materials, including presentation slides and online modules that cover

the formative assessment process, data security, and supporting teachers.

Final teacher and administrator materials were reviewed and edited to ensure that field test and

state-specific content were not included. For example, references to state data collection

calendars that were part of the field test were removed. Training materials were also reviewed

and updated for clarity and consistency with changes made to assessment materials.

Recommendations regarding training

Based on teacher feedback regarding the training and technology, we recommend the following:

• Integrate additional practice opportunities with the technology into the training. Teachers mentioned at several points in their responses to survey and interview questions a

desire to have more practice with the technology. Technology practice could be easily

integrated with the training activities that are part of the K–3 Formative Assessment training

materials. For example, teachers could practice entering observation notes into a

technology platform while observing video content.


• Provide time between training and implementation for self-study and follow-up training to occur. Teachers mentioned “information overload” in terms of the content

provided during the training which was not surprising given that a lot of information was

presented in a short time. State and district staff responsible for developing and customizing

the training schedules and training content should consider ways in which additional support

can be provided to teachers both before and after the formal assessment window begins

without placing too much burden on teachers over a short time period. The online training

modules and the follow-up trainings did not appear to be useful to teachers as fewer than

half took advantage of them.

• Regional, district, or school staff responsible for supporting technology use in the schools should attend the technology portion of the training to understand the technology needs associated with the K–3 Formative Assessment. Intermittent or slow

access to the Internet and lack of available computers and/or tablets added to the time

teachers spent on collecting, uploading, and managing observation data. Mitigation of the

technology issues would reduce teacher burden.

Conclusion The field test demonstrated that the K–3 Formative Assessment can be implemented by

teachers as intended, provided that teachers have adequate training in the assessment content

and the formative assessment process, and have access to functioning technology that

supports the assessment. Most of the construct progressions performed well as measures of

children’s knowledge, skills, and abilities, both in terms of their sequencing and their sensitivity

to growth and development over time. The analysis of the evidence documented in the

technology platform showed that the constructs varied in how reliably teachers coded a level for

the child. Teacher feedback reflected this variation, in that fewer teachers felt confident about

where they placed children on the progressions of constructs that had lower reliability. Teachers

found some constructs more difficult than others to incorporate into their daily classroom

routines. Of particular importance for a formative assessment was that across constructs,

between 20 and 50 percent of teachers did not find a given construct useful in informing

instruction. This may be a characteristic of the assessment that needs to be improved or it could

indicate a need for more professional development around how to use the results of a formative

assessment to inform instruction. Overall, we are encouraged by the performance of most of the

constructs in a field test setting where most teachers had limited experience with the constructs

and formative assessment process. As teachers continue to work with the K–3 Formative


Assessment, we recommend additional research and professional development on the

construct progressions that presented some issues with their content and proved challenging to

implement. These constructs are those in the social-emotional development and cognitive

development domains, along with Perseverance, Vocabulary, and Gross Motor.


References American Educational Research Association, American Psychological Association, & National

Council on Measurement in Education. (2014). Standards for educational and

psychological testing. American Educational Research Association.

Bejar, I. I. (2008). Standard setting: What is it? Why is it important. R&D Connections, 7, 1–6.

Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Taylor & Francis

Group: London.

Ferrara, A. M., & Lambert, R. G. (2015). Findings from the 2014 North Carolina Kindergarten

Entry Formative Assessment pilot. CEME Technical Report No. 2015-16. Charlotte, NC:

The Center for Educational Measurement and Evaluation.

Gebhart, T., Partika, A., & Lin, V. (2017). Implications for long-term implementation: Findings

from the K-3 Formative Assessment Consortium field test data collection. Bethesda, MD:

Child Trends.

Hambleton, R. K. (2000). Response to Hays et al. and McHorney and Cohen: Emergence of

item response modeling in instrument development and data analysis. Medical Care,

38(9), II–60.

Linacre, J. M. (2004). Optimizing rating scale category effectiveness. In E. V. Smith Jr., & R. M.

Smith (Eds.), Introduction to Rasch measurement (pp. 258–278). Maple Grove, MN:

Journal of Applied Measurement Press.

Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of

performance on educational and occupational tests. Princeton, NJ: Educational Testing

Service.

Masters, G.M. (June 1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), pp.

149-174.

Mislevy, R. J., & Haertel, G. (2006). Implications of evidence-centered design for educational

testing. Educational Measurement: Issues and Practice, 25(4), 6–20.

North Carolina K-3 Assessment Think Tank (2013). Assessment for learning and development

in K-3: A report by the state assessment think tank. Raleigh, NC: Author.


Scott-Little, C., Kagan, S., Sumrall, T., & Fox, E., (2014). Common early learning and

development standards analysis for the North Carolina EAG Consortium – Summary

report. Greensboro, NC: Build Initiative.

Wright, B. D., & Linacre, J. M. (1992). Combining and splitting categories. Rasch Measurement

Transactions, 6(3), 233–235.


Appendices Appendix A ............................................................................................................................. A-1

Appendix B ............................................................................................................................. B-1

Appendix C ............................................................................................................................. C-1

Appendix D ............................................................................................................................. D-1

Appendix E ............................................................................................................................. E-1

Appendix A Evidence-Centered Design (ECD) Codesign Teams: Enhancements to the Construct Progressions Exhibit A.1. Changes made through the ECD co-design process in the NC construct progressions

EAG K-3 Formative Assessment: Final Report September 2017 A-1

Exhibit A.1. Changes made through the ECD co-design process in the NC construct progressions

Domain Original NC Constructs EAG K-3 Constructs Changes Made through ECD Co-Design Process

Engagement in Self-Selected Activities

Perseverance

• Combined aspects of engagement in self-selected activities and perseverance in assigned activities into one construct

• Removed references to strategies to not confuse task strategies with strategies to stay engaged

Perseverance in Assigned Activities

Object Counting Object Counting • Expanded the number of skills from 8 to 10 • Introduced the number of objects counted as a factor in

skill difficulty level (4-6, 6-10, 18-20)

Problem Solving* Problem Solving*

• Reduced number of skills from 13 to 10 • Combined steps that involve solving and attempting to

solve a problem to focus on how children problem solve rather than successful problem solution

• Eliminated step that involves child generating sophisticated problems for others to solve as not typically observed in elementary classrooms

Mathematical Patterns • Added construct to measure algebraic thinking

(recommended by Executive Committee) and developed progression

Book Orientation & Print Awareness

Book Orientation • Separated into two distinct constructs

Print Awareness

Letter Naming Letter Naming

• Shortened and simplified progression; reordered some steps

• Defined skills in terms of letters known (e.g., in name) rather than numbers of letters

• Clarified that child can identify letters by either naming the letter or selecting the letter when asked

Following Directions Following Directions • Simplified and shortened progression from 11 to 6 steps • Removed emphasis on compliance tasks and positional

words

Writing Writing

• Removed skill (imitating conventions) that does not have clear developmental sequence

• Pulled out demonstration of writing with purpose and audience as a separate skill


Domain Original NC Constructs EAG K-3 Constructs Changes Made through ECD Co-Design Process

Reading Comprehension: Monitoring Meaning Reading Comprehension

• Shortened progression from 9 to 8 steps • Dropped last step requiring child to use optimal self-

correction strategy as this would be difficult for teacher to judge

School-Related Vocabulary Vocabulary

• Simplified and shortened progression from 13 to 8 steps to remove levels within a skill (e.g., begins to use, sometimes uses, consistently uses)

• Removed need for teacher to observe use of words in multiple contexts which is difficult to track

Gross Motor Gross Motor • Extended progression beyond basic locomotion (walking, running, stairs) to include hopping, galloping, and skipping

Fine Motor Hand Dominance • Separated into 2 distinct constructs

• Simplified and shortened progression for grip & manipulation, combining last two skills as indistinguishable Grip & Manipulation

Midline Crossing Midline • No substantive changes other than name to clarify construct

Emotion Regulation

Emotion Regulation Strategies • Split into two distinct constructs to capture dual themes of

expressing and managing emotions • After split, gaps between skills were too large, so additional

skills were added to even out granularity between skills. • Named “Emotion Regulation Strategies” to focus teacher’s

attention on the strategies children can learn to help regulate their emotions

Emotion Expression

Emotional Literacy Emotional Literacy • Combined skills to reduce complexity of progression and

eliminate concerns about developmental sequence • Extended progression to be more inclusive of K-3rd grades.

*The Problem Solving construct as developed by NC and enhanced for the EAG Consortium includes skills that tap two domains: Cognitive Development and Approaches to Learning.


Appendix B Cognitive Labs Internal Report

EAG K-3 Formative Assessment: Final Report September 2017 B-1

1

Cognitive Labs Report

Prepared by SRI International

July 29, 2016


Table of Contents Section 1: What Was Done as Part of Cognitive Labs? 5

Teacher Participation 5 Table 1. Number of Teachers Who Participated 5 Table 2. Number and Percentage of Teachers Per Grade Level 5

Survey Completion 6 Table 3. Average Number of Surveys Signed Up for and Completed (in Spring and Fall 2015 Periods) 6

Table 4. Number of Surveys Completed (Partially or Fully) 7 Section 2: What Was Learned from Cognitive Labs? 8

Most Children Could Be Placed 8 Table 5. Number of Child Assessments and Number and Percentage in Which Children Could and Could Not Be Placed 8

Table 6. Number of Teachers Who Selected Each Reason Why Children Couldn’t Be Placed 9

Table 7. Number and Percentage of Teachers Who Selected Positive or Negative Confidence Levels Related to Placing Children 9

Most Teachers Found Progressions Clear and Relevant 9 Table 8. Number Teachers Who Did or Did Not Agree with Statements about Clarity and Relevance of Progression Components 10

Most Teachers Found Progressions Helpful for Planning Instruction and Next Steps for Children 10

Table 9. Number of Teachers Who Did or Did Not Agree with Statements about Helpfulness of Constructs and Whether Children Could Be Placed

10

Most Teachers Found Progressions Relevant to Different Grade Levels 11 Table 10. Number of Teachers Who Did or Did Not Agree with Statements about Whether K Children and Children at Their Grade Level Could Be Placed

11

Most Teachers Found Reading Comprehension, Writing, and Vocabulary Constructs Aligned with Their Requirements 11

Table 11. Number of Teachers Who Did or Did Not Agree with Statement about Alignment of Progressions with Requirements 12

Most Teachers Found They Could Gather Sufficient Info from Routine Activities and Had Suggested Materials 12

Table 12. Number of Teachers Who Did or Did Not Agree with Statements about Routine Activities and Suggested Materials 12

Not All Teachers Used Situations, But Those Who Did Found Them Effective 13

Table 13. Number of Teachers Who Did or Did Not Use Situations 13 Table 14. Of Teachers Who Did Not Use Situations, Number and Percentage Who Selected Each Reason for Not Using Situations 14


Table 15. Of Teachers Who Used Situations, Number and Percentage Who Found Them Effective 14

Not All Teachers Used Tasks, But Those Who Did Found Them Effective 14 Table 16. Number of Teachers Who Did or Did Not Use Tasks 15 Table 17. Of Teachers Who Did Not Use Tasks, Number Who Selected Each Reason for Not Using Tasks 15

Table 18. Of Teachers Who Used Tasks, Number Who Found Them Effective 15

Teachers Provided Information on the Lowest and Highest Placement of Children That They Assessed 16

Figure 1. Book Orientation 16 Figure 2. Crossing Midline 16 Figure 3. Emotion Expression 17 Figure 4. Emotion Regulation 17 Figure 5. Emotional Literacy 18 Figure 6. Following Directions 18 Figure 7. Grip & Manipulation 19 Figure 8. Gross Motor 19 Figure 9. Hand Dominance 20 Figure 10. Letter Naming 20 Figure 11. Object Counting 21 Figure 12. Perseverance 21 Figure 13. Print Awareness 22 Figure 14. Problem Solving 22 Figure 15. Reading Comprehension 23 Figure 16. Vocabulary 23 Figure 17. Writing 24

Teachers Had Specific Feedback About Constructs 25 Book Orientation 25 Crossing Midline 25 Across Emotion Constructs 25 Emotion Expression 26 Emotion Regulation 26 Emotional Literacy 26 Following Directions 27 Grip & Manipulation 27 Gross Motor 28 Hand Dominance 28 Letter Naming 28 Object Counting 29 Perseverance 29 Print Awareness 29 Problem Solving 29 Reading Comprehension 30


Vocabulary 30 Writing 31

Section 3: What Changes Were Made in Response to Cognitive Labs Feedback (As Well as Pilot Feedback)? 32

Most Constructs Remained the Same 32 Some Constructs Had Changes to the Assessment Materials to Clarify Specific Points 32

Emotion Expression 32 Emotion Regulation 32 Following Directions 32 Gross Motor 32 Problem Solving 32 Reading Comprehension 33 Writing 33

Two Constructs Had Changes to the Progression 33 Emotional Literacy 33 Vocabulary 33


Section 1: What Was Done as Part of Cognitive Labs?

Teacher Participation

The purpose of cognitive labs was to give teachers an opportunity to try out individual constructs materials with children and provide construct-specific feedback to SRI. Cognitive Labs occurred in three time periods. The first round of Cognitive Labs occurred in the spring of 2015, the second round occurred in the fall of 2015, and the third round occurred in the spring of 2016. In the spring of 2015, 61 teachers in four states participated. In the fall of 2015, 56 teachers in four states participated. In the spring of 2016, four teachers in one state participated. See Table 1 for a state-by-state breakdown of participants. Table 1. Number of Teachers Who Participated

State Total Spring 2015 Fall 2015 Spring 2016 Arizona 26* 10 18 0

Iowa 14** 10 7 0 Maine 34 7 27 0

Rhode Island 38 34 4 0 North Carolina 4 0 0 4

Total 116 61 56 4 * There were two teachers in both spring and fall cohorts. ** There were three teachers in both spring and fall cohorts.

Of the 116 teachers who participated in cognitive labs, 37.93% were kindergarten teachers, 27.68% were first grade teachers, 14.29% were second grade teachers, 11.61% were third grade teachers, and 9.82% were multi-grade teachers. One teacher’s grade level is not known. See Table 2 for the number of teachers at each grade level. Table 2. Number and Percentage of Teachers Per Grade Level

Grade Number of teachers

Percentage of teachers

K 44 37.93% 1 31 27.68% 2 16 14.29% 3 13 11.61%

Multiage 11 9.82% Unknown 1 0.89%

Total 116


Survey Completion

In the spring and fall 2015 cognitive labs, teachers were typically given a choice of constructs to try out. If a teacher did not select a construct, one was assigned. Teachers were then given three weeks to use the construct materials (progression, assessment means form). In the spring 2016 cognitive labs, teachers were recruited to give feedback only on the problem solving construct materials. Across the spring and fall 2015 cognitive labs, teachers signed up for an average of 2.89 surveys and completed an average of 1.91 surveys (see Table 3).

Across all three cognitive labs periods, the teachers completed a total of 267 surveys, distributed unevenly across constructs (see Table 4). The constructs surveys were taken in the following order of frequency: grip & manipulation, emotion expression, emotion regulation, reading comprehension, following directions, gross motor, vocabulary, writing, crossing midline, perseverance, emotional literacy, letter naming, book orientation, print awareness, object counting, hand dominance, and problem solving. Table 3. Average Number of Surveys Signed Up for and Completed (in Spring and Fall 2015 Periods)

Surveys signed up for Surveys completed

Mean 2.89 1.91 Range 1-4 0-4


Table 4. Number of Surveys Completed (Partially or Fully)

Construct Number of surveys partially or fully completed Spring 2015 Fall 2015 Spring 2016 Total

Book orientation* 7 0 0 7 Crossing midline 1 11 0 12 Emotion expression** 24 9 0 33 Emotion regulation** 24 3 0 27 Emotional literacy 0 10 0 10 Following directions 0 19 0 19 Grip & manipulation 43 0 0 43 Gross motor 0 19 0 19 Hand dominance 3 2 0 5 Letter naming 2 7 0 9 Object counting 6 0 0 6 Perseverance 0 11 0 11 Print awareness* 7 0 0 7 Problem solving 0 0 4 4 Reading comprehension 0 20 0 20 Vocabulary 0 18 0 18 Writing 0 17 0 17 Total 117 146 4 267

* Book orientation and print awareness constructs were combined into one survey in the spring but were separated for the purposes of analysis. ** Emotion expression and emotion regulation constructs were combined into one survey in the spring but were separated for the purposes of analysis.


Section 2: What Was Learned from Cognitive Labs? Most Children Could Be Placed

In the cognitive labs, teachers were asked to try out each construct with 3-5 children.

Teachers generally followed that guideline. Across constructs, there were 982 child assessments. Because teachers did not have to obtain child consent, they tried out each construct with whomever they desired. Some children could have been assessed for multiple constructs, so the number of unique children assessed is not known. The majority of these child assessments allowed for the placement of children on the progression (97.61%). See Table 5 for a breakdown by grade.

When a child assessment did not result in a child being placed on the progression, teachers reported why they could not place the child. Teachers could choose from one or more of the five following reasons: (1) child was above the highest skill level, (2) child was below the lowest skill level, (3) the teacher was not able to collect enough data, (4) the data collected was too inconsistent for the teacher to decide on a placement, and (5) other. The frequency with which each reason was reported is presented in Table 6 and is broken down by grade level.

Teachers also reported how confidence they felt placing children on the progression. They chose between very confident, fairly confident, confident for some children, and not confident. The frequency with which positive or negative confidence levels were selected is reported in Table 7 and is broken down by grade level.

Table 5. Number of Child Assessments and Number and Percentage in Which Children Could and Could Not Be Placed

Number of child assessments

Assessments in which children could be

placed

Assessments in which children could not be

placed Across grades 982 961 (97.61%) 24 (2.39%)

K 363 363 (97.84%) 8 (2.16%) 1 305 296 (98.71%) 4 (1.29%) 2 124 124 (96.12%) 5 (3.88%) 3 128 128 (95.52%) 6 (4.48%)

Multi 62 50 (98.41%) 1 (1.59%)


Table 6. Number of Teachers Who Selected Each Reason Why Children Couldn’t Be Placed

Above

highest skill level

Below lowest skill

level

Not able to collect

enough data

Data too inconsistent

to decide Other

Across grades 4 3 2 4 3 K 2 1 1 2 1 1 0 0 0 2 2 2 1 0 1 0 0 3 1 1 0 0 0

Multi 0 1 0 0 0 Table 7. Number and Percentage of Teachers Who Selected Positive or Negative Confidence Levels Related to Placing Children

Fairly or Very Confident

Confident for some or not confident

Across grades 235 (89.69%) 27 (10.31%) K 93 (87.74%) 13 (12.26%) 1 65 (89.04%) 8 (10.96%) 2 30 (96.77%) 1 (3.23%) 3 35 (94.59%) 2 (5.41%)

Multi 12 (80.00%) 3 (20.00%) Most Teachers Found Progressions Clear and Relevant

Teachers answered several questions about the clarity, helpfulness, and relevance of the construct progressions components. Across construct progressions, teachers reported that the wording was clear (98.08%), the understandings were helpful (99.45%), and the performance descriptors and examples were helpful (98.47%). Regarding the performance descriptors and examples, a smaller percentage of teachers found them relevant (89.66%). See Table 8 for a breakdown by grade.


Table 8. Number of Teachers Who Did or Did Not Agree with Statements about Clarity and Relevance of Progression Components

Clear wording Helpful understanding(s)*

Helpful performance

descriptors and examples

Relevant performance

descriptors and examples

Agree Disagree Agree Disagree Agree Disagree Agree Disagree Across grades 256 5 181 1 257 4 234 27

K 104 2 64 0 105 1 98 8 1 72 1 49 0 71 2 65 8 2 24 1 30 1 26 5 22 9 3 37 0 32 0 37 0 32 5

Multi 14 0 12 0 14 0 13 1 * There were no understandings for the motor constructs. Most Teachers Found Progressions Helpful for Planning Instruction and Next Steps for Children

Teachers reported whether they found the construct progressions helpful for planning instruction. 83.91% of teachers reported that they were. For some constructs, teachers also reported whether the construct progressions were helpful for planning next steps for children, and 90.28% reported that they were. See Table 9 for a breakdown by grade. Table 9. Number of Teachers Who Did or Did Not Agree with Statements about Helpfulness of Constructs for Planning Instruction and Next Steps

Helpful for planning instruction

Helpful for planning next steps

Agree Disagree Agree Disagree Across grades 219 42 130 14

K 90 16 14 0 1 59 14 55 6 2 21 5 21 0 3 34 3 31 2

Multi 14 0 9 1


Most Teachers Found Progressions Relevant to Different Grade Levels

Teachers reported whether they thought most kindergarten children could be placed, and 99.53% agreed. For some constructs, teachers also reported whether children at the teacher’s particular grade level could be placed, and 97.22% agreed. See Table 10 for a breakdown by grade. Table 10. Number of Teachers Who Did or Did Not Agree with Statements about Whether K Children and Children at Their Grade Level Could Be Placed

K children could be placed

My grade level could be placed


K 102 1 14 0 1 57 0 61 0 2 24 2 9 3 3 16 0 31 2

Multi 12 0 10 0 Most Teachers Found Reading Comprehension, Writing, and Vocabulary Constructs Aligned with Their Requirements

Teachers were queried as to whether they felt the reading comprehension, writing, and vocabulary constructs were aligned with what the teachers were required to teach. For these constructs, 81.13% of teachers reported that they were. See Table 11 for a breakdown by grade.


Table 11. Number of Teachers Who Did or Did Not Agree with Statement about Alignment of Progressions with Requirements

Aligned with my requirements*

Agree Disagree Across grades 43 10

K 1 0 1 15 4 2 28 3 3 15 2

Multi 3 1 * This question was asked only in the surveys for reading comprehension, writing, and vocabulary. Most Teachers Found They Could Gather Sufficient Info from Routine Activities and Had Suggested Materials

Teachers reported whether they could gather enough information from routine activities, and 92.34% of teachers said they could. For some constructs, teachers reported whether they had the suggested materials, and 96.97% of teachers did. See Table 12 for a breakdown by grade. Table 12. Number of Teachers Who Did or Did Not Agree with Statements about Routine Activities and Suggested Materials

Enough info from routine activities

Had suggested materials


K 98 8 64 2 1 66 7 63 0 2 24 1 0 0 3 36 1 29 3

Multi 13 1 12 0


Not All Teachers Used Situations, But Those Who Did Found Them Effective

Situations were provided for some of the constructs. Eleven constructs had one Situation that was used in Cognitive Labs (Book Orientation, Crossing Midline, Following Directions, Grip & Manipulation, Gross Motor, Hand Dominance, Letter Naming, Object Counting, Print Awareness, Reading Comprehension, Writing); two constructs had three Situations (Emotional Literacy and Vocabulary); and three constructs had no Situations (Emotion Expression, Emotion Regulation, and Perseverance). Teachers reported whether they used the situation(s) (which was not required). Not all teachers used situations (37.55%). See Table 13 for a breakdown by grade.

Teachers who did not use the situation(s) reported whey they did not. The reasons varied, but the majority of times, teachers did not use the situation(s) because they were able to collect enough information through observation. See Table 14 for more information. The teachers who did use the situation(s) reported that they found them appropriate (98.91%) and easy to implement (96.74%). They also found the evidence from the situation(s) helpful (98.91%). See Table 15 for a breakdown by grade. Table 13. Number of Teachers Who Did or Did Not Use Situations

Used Situation Yes No

Across grades 92 153

K 30 45 1 31 46 2 12 20 3 15 27

Multi 4 15


Table 14. Of Teachers Who Did Not Use Situations, Number Who Selected Each Reason for Not Using Situations

Why wasn’t used

Able to obtain enough info

through observation

Not a good activity

Confusing or hard to use

Not appropriate

for this grade level

Other

Across grades 123 7 1 6 19

K 41 2 0 0 3 1 38 2 0 1 5 2 13 1 1 0 5 3 20 0 0 4 4

Multi 011 7 0 1 2 Table 15. Of Teachers Who Used Situations, Number Who Found Them Effective

Situation appropriate

Situation easy to implement

Situation evidence helpful

Agree Disagree Agree Disagree Agree Disagree Across grades 91 1 89 3 91 1

K 30 0 30 0 29 1 1 31 0 30 1 31 0 2 12 0 11 1 12 0 3 15 0 14 1 15 0

Multi 3 1 4 0 4 0 Not All Teachers Used Tasks, But Those Who Did Found Them Effective

Two constructs had a task that was used in Cognitive Labs (Crossing Midline and Gross Motor). Teachers reported whether they used the task (which was not required). Not all teachers used the task (32.26%). See Table 16 for a breakdown by grade.

Teachers who did not use the task reported whey they did not. The reasons varied, but the majority of times, teachers did not use the task because they were able to collect enough information through observation. See Table 17 for more information. The teachers who did use the task reported that they found them appropriate (100%) and easy to implement (100%). They also found the evidence from the situation(s) helpful (100%). See Table 18 for a breakdown by grade.


Table 16. Number of Teachers Who Did or Did Not Use Tasks

Used Task Yes No

Across grades 10 21

K 2 2 1 6 14 2 1 2 3 1 1

Multi n/a n/a Table 17. Of Teachers Who Did Not Use Tasks, Number Who Selected Each Reason for Not Using Tasks

Why wasn’t used

Able to obtain enough info

through observation

Not a good activity

Confusing or hard to use

Not appropriate

for this grade level

Other

Across grades 18 2 0 0 2

K 1 0 0 0 1 1 13 2 0 0 0 2 2 0 0 0 0 3 2 0 0 0 1

Multi n/a n/a n/a n/a n/a Table 18. Of Teachers Who Used Tasks, Number Who Found Them Effective

Task appropriate Task easy to implement

Task evidence helpful

Agree Disagree Agree Disagree Agree Disagree Across grades 10 0 10 0 10 0

K 2 0 2 0 2 0 1 6 0 6 0 6 0 2 1 0 1 0 1 0 3 1 0 1 0 1 0

Multi n/a n/a n/a n/a n/a n/a


Teachers Provided Information on the Lowest and Highest Placement of Children That They Assessed

Teachers reported the lowest and highest skill level at which they placed the children that they assessed. These data are reported in Figures 1-17, broken down by grade, with the number of teachers at each grade level reported in parentheses. Figure 1. Book Orientation

Figure 2. Crossing Midline


Figure 3. Emotion Expression

Figure 4. Emotion Regulation


Figure 5. Emotional Literacy

Figure 6. Following Directions


Figure 7. Grip & Manipulation

Figure 8. Gross Motor


Figure 9. Hand Dominance

Figure 10. Letter Naming


Figure 11. Object Counting

Figure 12. Perseverance


Figure 13. Print Awareness

Figure 14. Problem Solving


Figure 15. Reading Comprehension

Figure 16. Vocabulary


Figure 17. Writing


Teachers Had Specific Feedback About Constructs Teachers provided specific feedback about constructs on the cognitive lab surveys. Sample items of feedback are presented for each construct.

Book Orientation

• How used this to inform instruction: In some ways. The information I am gathering is information that I would have gathered even without this pilot. i.e. book handling skills, print awareness, etc.

• I felt as if both of these professions were too easy for my classroom and although I only "assessed" 3 students, all of my students would fall of the highest profession. Especially at this time of the year, all kindergarteners should know how to easily handle their reading books. If a child cannot do this it is usually handled on a one-to-one basis. Again, I think this would be great information to obtain in a preschool setting, not so much a middle/end of the year kindergarten assessment.

• Examples were relevant although this progression was way to easy for all of the students in my class.

Crossing Midline

• Easy to follow, understandable, teacher friendly. • Examples helpful--used for activity ideas.

Across Emotion Constructs

• I think this is very helpful. I do believe that the special needs department would be glad that these assessments were done because the information provided helps to show interventions and observations that may have been done to fix the problem the student is having with emotion in class.

• I continue to be impressed with the clear and concise language used in the materials. I was especially impressed with the Emotion Regulation language as this is such a critical area in a kindergarten classroom that can negatively or positively impact everything the teacher is trying to accomplish on a daily basis. With today's society being mainly that of both parents working, I have found that many children enter my classroom ill equipped to handle their emotions due to a lack of one-on-one time. The strategies on the construct progression allow for even a brand new teacher to understand how to handle to varied emotions of a kindergarten student who can vary in age from 4-6 years old and therefore be in very different places emotionally.

• It might be helpful to provide teachers with a list of possible strategies that the students might be using (Using Strategies) I also feel that you can not determine where a a child is placed based off of one observation. That child may be having a


bad day or a situation is a frustrating one. You need a variety of situations to get a full understanding of the child's regulation of emotions (Appropriate Expression of Emotions)

Emotion Expression

• I selected three different students with different emotional needs and I was surprised that they all fell on level A. That provided me with a lot of information.

• I have never changed my lesson plans if a students behavior was not where it should be. I have excluded students from fun things.

• Again, as I thought for the fine motor construct, it is a great resource for those few children whose behaviors need further classroom supports, but way too overhwelming to conceive of doing for the whole class.

• Made me realize I need more tools to help kids work on self-regulation. • Would be good to do at the beginning of the year with check-in activities with kids.

Emotion Regulation

• I continue to be impressed with the clear and concise language used in the materials. I was especially impressed with the Emotion Regulation language as this is such a critical area in a kindergarten classroom that can negatively or positively impact everything the teacher is trying to accomplish on a daily basis. With today's society being mainly that of both parents working, I have found that many children enter my classroom ill equipped to handle their emotions due to a lack of one-on-one time. The strategies on the construct progression allow for even a brand new teacher to understand how to handle to varied emotions of a kindergarten student who can vary in age from 4-6 years old and therefore be in very different places emotionally.

• For most students, this information is known in teacher's head based upon everyday observations and interactions and does not need to be placed on paper, just adds more work. Good resource for RTI purposes or parent conferences when difficulties arise.

Emotional Literacy

• I did not look at the situations ahead of time, but as I went back through to offer the feedback on this survey, the situations are exactly how I taught and assessed. The situations are spot on to how I feel most teachers would approach this topic.

• There is a lot of information for this construct. By this, I mean, I was surprised by the amount of skill levels. I was a little overwhelmed at first when I read through it.



• I learned a lot about one little boy. Prior to these observations, I thought he was going to have a very difficult time following a 2 or 3 step direction. However, he was able to follow them all without support which surprised me.

• I teach grade 3 and I feel that at this age my students should be able to complete the following directions construct. I feel most of the constructs would be great and very beneficial to to students in grades 1& 2, but by grade 3 these constructs should be the norm.

• This was great because I did not know how students really only listen to the last thing out of my mouth for following directions.

• The Following Directions progression really helped me to take a closer look at some other students in my class. I realized that some students that I thought could easily follow multi-step directions could not.

Grip & Manipulation

• The descriptors were helpful because they used real day-to-day scenarios that we use in our classroom (ex: holding a paintbrush with two fingers using precise strokes)

• The majority of students come to school able to manipulate pencils, scissors, etc... the few that do not, are provided with instruction on proper use and interventions when needed to correct grips. This is so individualized having a standardized progression is not necessarily needed.

• The information gathered through this construct was useful, but the anecdotal notes on this did not help me. At the beginning of the year, I observe students and work with them through their struggles…

• I think that the information that could be gleaned from this assessment is valuable and vital information, however, if school districts are unwilling to allow time in the classroom (especially kindergarten) when students are freely choosing activities, and those that would help exhibit the constructs, then it is going to be extremely difficult to implement. In my classroom, I am working with a small group or individuals during reading and writing time, so it is possible at times to observe students writing with a pencil, or coloring, but not for an extended period of time. Also, we rarely cut, and never paint.

• Can one child score more than one "Skill" score? I feel the rubric isn't complete yet. For example: a child might have an amazing pencil grip but struggle with scissors.

• If Kindergarten teachers would follow this it would help the first grade teachers so much. It it a harder to start this in 1st grade. They all ready have their habits and it is harder to change.

• The Construct Progression is so clearly and concisely written that even a new teacher could easily follow it. There can be such a large and varied skill base with fine motor skills in kindergarten students. Even with the higher academically


developed students can be low on their skill level for fine motor tasks. I have found it both difficult and challenging to break students of poor fine motor skill habits that they have entered kindergarten with. I found this to be very valuable to my teaching and I look forward to having the skill construct to use at the start of my next school year as it will be a valuable tool to use when discussing a child's fine motor progression with parents.

• It was easy to observe through our daily activities. I like that it doesn't take time away from our teaching time like other assessments do.

• I do feel that more training on the various grasps would have been helpful. I did use the materials (pictures & descriptions) given, but I was uncertain as to the proper name for each hold of the pencil as some of my students had different holds than the pictures. I did not feel 100% accurate at giving a clear label.

Gross Motor

• Most of my third grade students are too advanced for this construct • I needed to provide specific situations in order to observe skipping and galloping. I

could easily assess running and walking while at recess • I think it would also be beneficial to have other motor skills included in the "skills

documentation lists", like bouncing or catching a ball. Many young children have trouble with this skill, unless they are involved in sports outside of school.

• Liked being able to see where my kids were physically. • Like that it's about the whole child. • A lot of kids didn't know what gallop meant.

Hand Dominance

• I feel like this construct is established before they come to kindergarten, there is not a big need for direct instruction. The majority of my kids are already on level C, the couple on level B needed a couple reminders to stabilize, and the one friend on A is being evaluated for special needs.

• students at this age are still trying to decide their dominate hand • Probably more beneficial to prek.

Letter Naming

• Letter naming is such a huge part of the kindergarten curriculum and kinders are assessed to excess on this particular skill. This was quick and easy to apply to the construct since it is something done frequently already.

• We work with letter naming from day 1 in Kindergarten so this is something that we would plan for without knowledge of progression.

• Excited because this assessment is really developmental.


Object Counting

• Using to inform instruction: For example ,a student was able to sequence and count, but after looking at the numbers observation, the student still had difficulty with conservation/cardinality and was given some more opportunities to practice those skills as not to have a gap later on.

• I would like to see the content/progression become a little more in depth for 3rd grade core skills.

• It is good for the beginning and middle of Kindergarten. At the end of Kindergarten I found that my students were at the end of object counting.

• The literacy and math assessments were very repetitive with what I already do. They did not offer me new information on my students.

Perseverance

• I would know how to do this without the progressions • I observed the children during 3 different activities. One was an easy math fact

paper, the second was a moderately challenging reading activi ty, and the third a very challenging math activity. It was interesting to note that my more advanced students did better on the challenging activity than the rote math facts, scoring at least one skill level better. My less advanced students did about the same on alll three activities. It was a very interesting construct to use. I was over all pleased with how well most of my students did. I found the "watch for" notes on the documentation form very helpful. Thank you for the opportunity to work with these constucts. I have found them to be very valuable and I will add them to my "tool box."

• Perseverance became a classroom goal. Life long lessons. • Got a lot out of this. Will play in the back of my mind when I work with students.

Print Awareness

• I felt as if both of these professions were too easy for my classroom and although I only "assessed" 3 students, all of my students would fall of the highest profession. Especially at this time of the year, all kindergarteners should know how to easily handle their reading books. If a child cannot do this it is usually handled on a one-to-one basis. Again, I think this would be great information to obtain in a preschool setting, not so much a middle/end of the year kindergarten assessment.

Problem Solving

• Skills I and J could use more delineation between the two. Other domains/progressions

are integral parts of the problem solving process. I understand that the K-3 assessment process is intended to look at the whole child and covers perseverance and


collaboration in other components but I felt they needed to be addressed within the problem-solving section as well.

• One of the biggest focuses in my classroom as been to teach my children to become problem solvers. What I have noticed through this process is that a child's ability to problem solve varies greatly from one academic problem to the next depending on many variables such as interest, content, mood, engagement, etc. I collected evidence in 3 different content settings - writing, math and a STEM station. One of my students varied tremendously on the progression because he dislikes anything that requires writing so therefore he puts forth very little effort and shuts down easily. He is middle of the road in math so he placed a little higher but he a very scientific mind and is a kinesthetic learner so he placed the highest on the building a bridge/ raft activity. It almost seems as you need a problem solving construct for each academic area.

• Sometimes it was hard to try to figure out what kids were thinking when they weren’t verbalizing it themselves


• I utilize our curriculums' reading assessments to plan my instruction • I really enjoyed working with this activity. After I did the 6 for the study, I put the

rest of the class on the scale. It was easy to use and I was able to easily put students on the progression. The difference between the skill levels was well defined and skills needed to move up the progression was easier than the writing one. I had the students read a story orally and tested them and then had them read and answer question after silently reading a similar story. I was interested to find that some did better orally and some did better silently. Good activity!

• This is a great formative assessment tool for measuring how students are doing with reading strategies that are being taught in the classroom and where they are at in terms of using them to improve understanding of reading independently. It also gave great insight into which strategies are most effective for children of this age.

• I feel that I gather the same if not more information through what I am already doing.

• I am the Title 1 teacher at my school, so most of my work is related to reading fluency and comprehension. I found the progression easy to use and accurate as to what I see in student's reading development.

Vocabulary

• It's hard to connect the curriculum we are expected to teach to the progressions. I think it's a different way to look at assessment. We currently assess if a child understands the vocabulary we teach, not necessarily where they are on a progression.

• Vocabulary instruction is something I am already focusing on in my instruction. The skills don't always match what my students can do or are working on. I also have


other forms of assessment that I am already doing that provide more information and data.

• It was helpful to see where the children fell in terms of understanding of vocabulary; however feel like I need more info. to actually decide what to teach next, where it fits in to CCSS, etc.

• Most 3rd grade students are too advanced for this progression. • Had to set up specific situations, word sorts, writing to measure this progression

[Regarding gathering info from routine activities] • I like the clarity of the progression and found the details of the progression

(understanding, skills, performance descriptors and examples) they really helped to fine tune what I know about a child's language and vocabulary and helped me place them accurately (I think!) on the progression.

• this weeks words were related to a story that was a play... rehearsal, lines, characters etc. I used the word lines for the writing situation.. (get in a line, draw a line and learn your lines) The construct made me more aware of making sure I use the weekly vocabulary words in teaching and having the children use them..not just telling them what they mean!

• This was a helpful construct tool. I was able to share the information with parents at conference time. It gave parents some insight as to what would be helpful at home when giving directions to their child. Also, by making thoughtful observations on where some students were on the scale led to more thoughtful direction planning and implementing.

Writing

• In 3rd grade we write a lot of narrative and informational text and I felt like the performance descriptors weren't all that relevant to the genre of writing we do.

• Our curriculum, Common Core, expects all children to be able to do everything on the progression at all grade levels - i.e. revise own writing.

• I feel like the expectations for my students are more complex than what this construct outlines.

• Writing instruction isn't linear like this - I have students who can revise, but not be genre specific. Also, a student could be very purposeful in their writing, but not be able to form words.

• I liked the organization of using the progression. It blended well with the writing project and conferencing that I was doing with this second grade group. It didn't feel strange or like extra work. As we use a Common Core aligned writing progression, I did find a couple of things that were slightly different, but not so much so that it was a distraction.

• More information on how to move students to the next level would be great. As I have a large class,(22) most of the work except students selection was done in small group.


• It was very difficult not to do spelling issues. We have been taught to take a moment to do a lesson taken from their writing. This gives a one-on-one time to teach that individual student.


Section 3: What Changes Were Made in Response to Cognitive Labs Feedback (As Well as Pilot Feedback)?

Feedback from cognitive labs and the pilot was used to evaluate the construct

progressions and assessment materials to determine if changes were needed. Most constructs remained the same (N = 8), some constructs had changes to the assessment materials to clarify specific points (N = 7), and two constructs had changes to the progression. Most Constructs Remained the Same

The following construct progressions and assessment materials remained the same (except for minor formatting changes): book orientation, crossing midline, grip & manipulation, hand dominance, letter naming, object counting, perseverance, and print awareness. Some Constructs Had Changes to the Assessment Materials to Clarify Specific Points

The following construct progressions had changes to the assessment materials based on feedback from teachers: emotion expression, emotion regulation, following directions, gross motor, problem solving, reading comprehension, and writing. The changes made are presented for each construct.

Emotion expression. Additional background information was added to the progression document. The assessment means form was modified to include additional observation instructions related to emotion-eliciting events.

Emotion regulation. Additional background information was added to the progression document. The assessment means form was modified to include additional information on children's use of emotion regulation strategies, additional examples of strategies and information about strategies, expanded teacher instructions and information around "hard-to-assess" children, and revised skill-level teacher instructions.

Following directions. The assessment means form was modified to include additional information about eliciting a child’s attention before providing a direction, to avoid the need to repeat directions, since that is considered a support.

Gross motor. The assessment means form was modified to include additional guidance on what to do when a child can skip but not gallop. Additional guidance was also added about modeling skills when children do not know what terms such as “gallop” mean.

Problem solving. The assessment means form was modified to clarify the need to observe across content areas to determine consistent performance; importance of asking open-ended, non-leading questions; and scope of academic problem.


Reading comprehension. The assessment means form was modified by revising the "Specific Observation Instructions" to include information about skills being cumulative and using text slightly above the child's reading level. Writing. Additional background information was added to the progression document. A minor grammatical error in Skill G was corrected in all documents.

Two Constructs Had Changes to the Progressions

The following construct progressions had changes to the construct progression and assessment materials based on feedback from teachers: emotional literacy and vocabulary. The changes made are presented for each construct.

Emotional literacy. The skills were reduced and revised (from 14 to 9 skills), which necessitated also changing performance descriptors and examples. Additional background information was added to the progression document. The assessment means form was modified to include additional information about placing children on the progression. Vocabulary. Skills B and C were combined which also necessitated changing performance descriptors and examples. One of the examples for Skill D (new Skill C) was revised.


Appendix C Pilot Test Internal Report EAG K-3 Formative Assessment Pilot Study Report

EAG K-3 Formative Assessment: Final Report September 2017 C-1

EAG K-3 FORMATIVE ASSESSMENT PILOT STUDY REPORT

K-3 Formative Assessment: 2015-16 Pilot Study

Sara Thayer and Suzanne Raber

SRI Education

Author Note

This brief is an internal report of pilot activities conducted during the 2015-16 school year as

part of the Enhanced Assessment Grant awarded to the North Carolina Department of Public

Instruction and a consortium of ten states.



Executive Summary The full K-3 Formative Assessment was piloted with approximately 80 K–3 teachers and 800

children in four states (Arizona, Iowa, Maine, Rhode Island) during the 2015–16 school year.

Each teacher assessed a sample of 10 children on all 17 constructs.1 Teachers’ use of the

assessment was supported by a technology platform. The teachers entered evidence (such as

anecdotal notes, photographs, or videos) from their observations for each construct and child in

the sample. They then determined where on the progression each piece of evidence fell

(preliminary learning status) and, across multiple pieces of evidence over multiple observations,

where to place the child on the progression (status summary). All the assessment data

(evidence, learning statuses, status summaries) could be entered into the platform for easy

review, reporting, and analysis. This process was further supported by an application that

allowed teachers to upload directly to the platform evidence (their notes, photos, and videos)

from a tablet or smart phone to assist in selecting a status summary.

Teachers provided feedback on the manageability and usability of the assessment to inform

instruction through online surveys, focus groups, and interviews. They appreciated the

assessment’s focus on the whole child and coverage of multiple domains of learning and

development. They also appreciated being able to store and retrieve the evidence they collected

in the technology platform. Teachers’ other feedback and the response to it were as follows:






Emotional Literacy progression was simplified, the assessment materials for Emotion

Expression and Emotional Regulation Strategies were clarified, and more guidance on how

to assess these constructs was added.





1 Mathematical Patterns was not included because it was developed during the pilot year.



K-3 Formative Assessment: 2015-16 Pilot Study The pilot study was designed to gather information about the K-3 Formative Assessment from

teachers in four states (Arizona, Iowa, Maine and Rhode Island) who used the full complement

of the 17 constructs developed thus far to assess 10 children in each of their classes. Input

from the teachers on the training, the content of the assessment and it use and manageability,

was then used to make modifications in the assessment content and process for a larger scale

field test conducted in the 2016-17 school year.

Methodology Each of the four consortium member states that had volunteered to be part of the pilot study

recruited K-3 teachers who were interested in using formative assessment in their classrooms,

at K entry and then throughout the school year in grades K-3. Kindergarten teachers were

recruited to participate in the fall 2015 pilot study and then additional K-3 teachers were asked

to pilot the assessment in the winter/spring of the 2015-16 school year. Pilot teachers

participated in a 2-day training prior to the fall and winter data collection periods, conducted by

staff from SRI Education who led the assessment enhancement efforts and staff from the

technology vendor who supported the assessment.

The full K-3 Formative Assessment was piloted with approximately 80 K–3 teachers and 800

children in four states during the 2015–16 school year. Each teacher assessed a sample of 10

children on all 17 constructs.2 The fall pilot sample is described in Exhibits 1 and 3, the winter

pilot sample in Exhibit 2.

Teachers’ use of the assessment was supported by a technology platform. The teachers

entered evidence (such as anecdotal notes, photographs, or videos) from their observations for

each construct and child in the sample. They then determined where on the progression each

piece of evidence fell (preliminary learning status) and, across multiple pieces of evidence over

multiple observations, where to place the child on the progression (status summary). All the

assessment data (evidence, learning statuses, status summaries) was entered into the platform

for easy review, reporting, and analysis. This process was further supported by an application

that allowed teachers to upload directly to the platform evidence (their notes, photos, and

videos) from a tablet or smart phone to assist in selecting a status summary.

2 Mathematical Patterns was not included because it was developed after the pilot



Teachers provided feedback on the content, manageability and usability of the assessment to

inform instruction through online surveys, focus groups, and interviews. Surveys were sent to

teachers after the training and at the end of the fall and winter data collection periods. Teachers

also participated in focus groups or individual interviews at the end of the fall and winter data

collection windows.

Findings Findings are presented in four areas:

• Teacher feedback on the training and professional development they received;

• Implementation of the assessment;

• Manageability of the assessment materials and process; and

• Overall usability of the assessment materials and process

Training/professional Development Teacher Background and Education. In both the fall and winter samples, the majority of

teachers were white and female (Table #). Over half of the teachers in both the fall and winter

sample had at least a master’s degree and over three-fourths had earned the highest certificate

available in their state. Teachers had about 14 years of teaching experience, and over one-

quarter of teachers had experience teaching preschool.

Preparation for implementing the K-3 Assessment. Teachers received in-person training on

the formative assessment process, content of the assessment (i.e., construct progressions and

assessment means), and opportunities for practicing with the assessment, including how to use

the technology as part of the process. Embedded in the training were construct modules with

video content for “live” coding. Training materials provided to the teachers (i.e., reference

binders and handouts) as well as additional resources developed over the course of the pilot

were made available through a website. The majority of teachers felt very prepared or

adequately prepared to implement the assessment in both the fall (81%) and winter (86%).

Aspects of the fall training that were of particular help in preparing teachers to use the

assessment were overviews of the construct progressions (72%), assistance provided by the

trainers in organizing or “clustering” the constructs (53%), demonstration of the technology

(53%), and practice observing and documenting evidence (41%).

Based on feedback from the fall, changes were made to the winter training, including less time

spent on background information on formative assessment and the vision for the assessment,

and more hands-on opportunities to practice using the assessment materials. The background



and overview of the assessment were instead covered in pre-training modules rather than the

in-person training. Other content added to the training covered characteristics of high-quality

evidence, tips for documenting evidence, and fitting the assessment into the instructional day.

After the winter training, over 50% of teachers found the majority of the practice activities to be

helpful (i.e., planning, summarizing the status, looking for and gathering evidence, and

assigning learning statuses). Teachers in the fall sample who returned for the winter pilot were

provided the opportunity to attend the winter training and had access to the revised training

materials.

Implementation of Assessment Process We were able to learn a great deal about how teachers implemented the assessment from what

they documented on the technology platform. The teachers entered evidence (such as

anecdotal notes, photographs, or videos) from their observations for each construct and child in

the sample. They then determined where on the progression each piece of evidence fell

(preliminary learning status) and, across multiple pieces of evidence over multiple observations,

where to place the child on the progression (status summary). All the assessment data

(evidence, learning statuses, status summaries) could be entered into the platform for easy

review, reporting, and analysis. This process was further supported by an application that

allowed teachers to upload directly to the platform evidence (their notes, photos, and videos)

from a tablet or smart phone to assist in selecting a status summary.

Based on the evidences and status summaries documented on the platform, teachers were able

to assess most of the constructs and followed the process with some fidelity, documenting

multiple pieces of evidence, assigning preliminary learning status and determining a status

summary for each child and each construct. This varied by construct with teachers more likely

to complete the assessment of those constructs introduced in the earlier clusters and less likely

to complete the assessment of constructs they found more challenging. For example, in the fall

pilot less than 70 percent of the children in the sample were assessed on the social-emotion

constructs, Vocabulary and Perseverance.

From the information in the platform, we learned the following about teachers’ adherence to the

steps in the assessment process:

• Assigned child a status summary (required). Teachers assigned a status summary for at

least one construct for 372 children in the fall. Across constructs, 57 to 93 percent of the

children in the pilot sample were assessed in the fall. The constructs least frequently



assessed were Emotional Literacy (57 percent of the children), Emotional Regulation

Strategies (61 percent), Emotion Expression (63%), Vocabulary (64%) and Perseverance

(67%).

• Documented multiple pieces of evidence (required but might not be in the platform).

Teachers did not consistently document more than one piece of evidence for determining a

child’s placement on a progression in the technology platform. Across constructs, 39 to 78

percent of the status summaries were based on multiple pieces of evidence in the fall.

Constructs least often documented with multiple pieces of evidence were Reading

Comprehension (39%), Perseverance (40%), Vocabulary (43%), and Object Counting

(44%).

• Assigned evidence a preliminary learning status (optional). Teachers were encouraged

to assign each piece of evidence a preliminary learning status to help them interpret multiple

pieces of evidence and determine the child’s status summary. Teachers followed this

optional step for the majority of all the evidence records entered on the platform. When

learning status was tabulated by construct and child, some construct variation was

observed. Learning status was assigned least often for Emotional Literacy (49%), Emotion

Regulation Strategies (50%) and Vocabulary (51%).

• Looking across the three steps in the assessment process, we observed that teachers were

least likely to collect multiple pieces of evidence, assign the evidence with a preliminary

learning status and determine a status summary for the three constructs in the social-

emotional domain and Vocabulary. Based on this finding and the survey findings from

teachers described below, we recommended additional changes and supports for these

constructs.

Manageability of the Assessment A goal of the fall and winter pilot studies was to understand the manageability of implementing

the assessment in kindergarten, first, second, and third grades, with an emphasis on

kindergarten in the fall. To determine manageability of the assessment, teachers responded to

survey and interview questions about how teachers incorporated the assessment into their

instructional practice, their perceived difficulty in observing and understanding constructs, the

compatibility of the K-3 Assessment with other assessments administered, use of technology to

support the implementation of the assessment, and the challenges and support they received.

Teachers were provided multiple opportunities to reflect on implementation of the assessment

both during and after the assessment.



Use of Other Assessments. The intent of the K-3 Formative Assessment is to complement

other assessments a teacher might use as part of a balanced comprehensive assessment

system. Thus, it was expected and confirmed that teachers would be able to use information

from other assessments to inform the K-3 Assessment, which in turn should make administering

the assessment more manageable. Over 85 percent of teachers in the fall and winter (85% and

88%, respectively) reported that they used information from other assessments as evidence for

the K-3 Assessment. These other assessments included district-mandated assessments and

teacher-created assessments in reading and math, running records and writing prompts, and

formal benchmark tests (e.g., DIBELS, FAST, and Fountas and Pinnell’s Benchmark

Assessment System). Teachers often mentioned using information from other assessments for

the literacy and mathematics constructs, including Letter Naming, Reading Comprehension,

Writing, and Object Counting.

Difficulty Assessing Constructs. Teachers reported that some constructs were more difficult

to assess than others, namely, the constructs in the social-emotional domain, whereas the more

“Kindergarten” oriented constructs tended to be nominated as difficult less than the other

constructs or not at all.

• In both the fall and winter, teachers reported that the social-emotional constructs were the

most difficult to assess. When asked to indicate any construct they found to be difficult at the

end of the winter pilot, 76% of teachers reported at least one emotion construct was difficult,

and 42% of teachers reported all three emotion constructs as being difficult. Some teachers

noted inconsistencies in children’s behavior from day to day as a reason for the social-

emotional constructs being difficult. Two insightful comments from teachers:

– All of the emotional pieces were difficult because my students are 5 and 6…and

their emotions are SO drastically changing from minute to minute sometimes. It's

exhausting to try and keep up with and to try to "analyze" or figure things out enough

to place them on the chart was difficult.

– Without the time for play to work through these emotions and discuss and such a

high academic rigor expectation, it is harder to document throughout the day.

• At least 25 percent of teachers also reported Vocabulary and Perseverance as being difficult

to assess. Some teachers felt that the Perseverance construct was more open to

interpretation, which made it difficult to assess,



– I feel this was challenging because I can give my students difficult tasks, but I never

felt clear if it was super challenging for them and I am not going to give them

something they will fail at to see if they will stick with it.”

For Vocabulary, teachers reported that it was difficult to look for during the day,

• Unless given specific situations, it was hard to catch and listen to students using “school

related” language.

• Across constructs, teachers’ difficulties in assessing constructs often stemmed from inability

to observe the construct during the day and/or the construct took too much time to assess.

• Constructs not nominated as being difficult to assess by any teachers included Book

Orientation, Hand Dominance, Letter Naming, Grip and Manipulation, Print Awareness,

Object Counting and Writing.

Incorporating Assessment into Instructional Activities. The assessment is designed such

that constructs can be observed during typical classroom activities, however 27 percent of fall

teachers and 38 percent of winter teachers found one or more constructs did not fit into daily

instruction.

• Some fall (21%) and winter (13%) teachers found Crossing Midline to not fit into daily

instruction.

• Both fall (18%) and winter (27%) teachers found Gross Motor to be difficult to assess during

daily instruction.

• Some teachers also found that the emotion constructs did not fit into daily instruction (18%

in the fall for Emotion Expression, Emotion Regulation strategies, and Emotional Literacy).

Fewer teachers in the winter reported the Emotion Expression, Emotion Regulation

Strategies, and Emotional Literacy constructs to not fit into daily instruction (7%, 9%, and

11%, respectively).

• Notably, teacher found some constructs easier to fit into daily instruction in the winter. This

may be attributed to increased comfort with the assessment, and more focused training in

the winter, particularly for constructs reported as difficult to assess in the fall (e.g., social-

emotional constructs).

Use of Materials. Teachers were provided multiple resources to support the constructs,

information about the constructs and documentation forms, and provided feedback on the ease

of use and helpfulness of each resource:



• The Construct Progression document provided a rationale and background information

about the construct, skills on the progression for which to observe the children, performance

descriptors for each skill, and examples for each skill. Over half of the teachers in the fall

and winter reported the construct progressions as easy to use (72% and 66%) and/or helpful

(67% and 64%) during implementation.

• The Assessment Means Form also included the construct skills, performance descriptors

and selected examples, as well as information on how to assess a child for that construct.

For example, the Assessment Means Forms contain sections on key terminology used in the

Construct Progression, potential settings and materials, general teacher instructions, skill-

specific teacher instructions, and situations and tasks one may construct in order to better

elicit the information needed. Over one-quarter of teachers found the assessment means

forms easy to use in the fall and winter (25% and 31%), and even more teachers also found

them helpful (40% and 35%).

• Construct Modules were online tutorials for each construct that provided an overview of the

construct, detailed explanation of the construct progression skills, and videos illustrating

children at different levels on the progression. Over one-third of teachers in the fall and

winter found the construct modules easy to use (38% and 45%), and even more teachers

found them helpful (40% and 47%).

• The single child and multiple children Documentation Forms were designed as an

alternative to the tech platform for teachers to document evidence and determine status

summaries for a single child over time, or multiple children on one form. About one third of

teachers in the fall and winter found the single child forms easy to use (38% and 31%)

and/or helpful (33% and 33%). Just under half of the teachers in the fall and winter found the

multiple children documentation form to be easy to use (47% and 50%) and helpful (43%

and 45%).

• The Teacher Site was a web-based repository for electronic copies of all construct

progressions, assessment means forms, documentation forms, modules; PowerPoints and

other training materials; calendars and timelines for completion of the pilot; and an optional

discussion board for each state and construct. Few teachers found this teacher resource

website easy to use (9% in the fall and 16% in the winter). More teachers found the

Teacher Site helpful (20% in the fall, and 24% in the winter).



Overall the materials and resources were well received, particularly those materials that were

critical to implementing the assessment (i.e., Construct Progressions). Few teachers reported

any of the materials as being difficult to use.

Use and Perceptions of Technology. The technology platform and K-3 evidence application

were designed to provide a more expedient way of collecting and uploading evidence; organize

the evidence collected during their observations; and help teachers summarize, track, and

report information. Thus, the use of technology in this assessment was critical to making the

assessment process more manageable and less burdensome by providing alternatives to

traditional note-taking and record keeping. All teachers in the fall sample and nearly all teachers

in the winter sample (95%) reported having access to a computer in the classroom, and nearly

80% reported having access to a tablet or mobile device at school. All teachers had access to

the internet, however some teachers reported that their internet access was “spotty” in the fall

(14%) and the winter (21%).

Overall, both the technology platform and the K-3 evidence application for mobile devices and

tablets were reported to be easy to use and helpful.3

• Fifty-six percent of teachers in the fall and 60% of teachers in the winter found the

technology platform to be easy to use.

• Teachers described the platform overall as being “user-friendly” and easy to use during

interviews, though several teachers acknowledged that using the technology platform was a

steep learning curve at first.

• The one aspect of the technology platform that appealed to teachers the most was the

differing ways in which information could be viewed. For example, teachers liked being able

to view the constructs and examples on the platform as they were assigning learning and

status summaries, rather than having to refer back to hard copy or other soft copy materials.

Teachers also appreciated the different ways in which information could be accessed, such

as viewing status summaries by the class or by individual children. Teachers reported using

the individual and class reports for viewing progress in data collection, either for completion

by child and/or by the amount of data collected across constructs.

• More teachers found the technology platform helpful in the winter (66%) than in the fall

(41%). The increase in teachers reporting the technology as easy and helpful from fall to

3 Some states and districts provided tablets for teachers to use, although these devices were not always available at the beginning of data collection. Teachers had the option of using personal devices if none were provided.



winter is likely due to increased use (for returning teachers), improvements in training for the

winter pilot, additional support materials (e.g., application tutorials), and the fact that by

winter more teachers had been provided with tablets.

Teachers’ views about the K-3 mobile application were less straightforward than feedback on

the technology platform.

• In the fall, only about one-quarter of teachers found the application easy or helpful (28% and

25%, respectively). In the winter, more teachers found the application to be helpful than

easy, with increases in both ease of use and helpfulness from fall to winter (48% and 55%,

respectively).

• Teachers who liked the application reported in interviews that they found it made the

assessment process easier and less time consuming. Aspects of the application that were

particularly helpful were seeing the skills on the application, using the device to take and

upload videos and pictures, and using the speech-to-text feature.

• Teachers who did not report favorably on the application either did not use the application

due to issues with Wi-Fi in their school or classroom, prohibition of mobile devices in the

classroom, or because of problems they experienced while using the application that did not

get resolved. As mentioned, some teachers did not have consistent access to the internet,

and had to upload evidence and work onto the platform outside of school.

• Teachers also mentioned that it was at times hard to use “in the moment” because of

privacy features, such as having to re-enter the password every time the application

opened, “The password we always had to enter was time consuming…Took forever to get in

and [I] lost opportunities for collecting evidence.”

In open ended survey and interview questions regarding the technology, teachers identified

three primary ways in which the technology platform and K-3 application could be improved:

• Provide additional training, either on the platform, the application, or as a booster training

once data collection was underway;

• Organize information on the platform so information, particularly student evidence, is easy to

locate; and

• Provide easier access into the mobile application, including bypassing the password feature

when phone or tablet has briefly been in stand-by mode.



Challenges. The fall pilot was the first opportunity for any participating teacher to implement the

full assessment, and for some teachers, the winter pilot was their first opportunity to work with

the assessment materials. It was expected that there would be both challenges to implementing

the assessment.

• Asked about challenges part-way through the data collection, more than half of the teachers

identified that time to implement the assessment was the biggest challenge in both the fall

(63%) and winter (54%). Teachers also identified time spent implementing the assessment

as the greatest challenge at the conclusion of the winter pilot (39%). Teachers specified that

it was difficult to find time to observe the constructs and to upload data to the technology

platform, particularly during the work day.

• Additional challenges identified at the conclusion of the winter pilot included not enough

support in the classroom (42%), not enough staff in the school trained on the K-3

assessment (38%), not enough time to practice using the technology (33%), and perceived

lack of alignment with state standards (33%).

Despite challenges with implementing the assessment, teachers found certain aspects of the

assessment that facilitated it implementation, or discovered their own ways to make the process

more manageable.

• When asked what went well when implementing the assessment at the conclusion of the

winter pilot, 40 percent of teachers again mentioned use of the application on a tablet or

mobile device. One teacher commented, “The K-3 app was a wonderful and useful tool to

use. It allowed me to capture organic situations that were happening in my classroom. I

found by using this tool I did not have to set up any situations to capture evidence.”

• Another positive aspect of the pilot that teachers mentioned was the availability of and

support of project staff (9%).4

• Teachers also reported during interviews that they found ways to manage the

implementation of the assessment such as creating their own checklists for monitoring data

collection, planning which constructs would be observed in a day, and being flexible in their

approach to collecting data (i.e., capitalizing on opportunities to observe a construct).

• Finally, some teachers were able to draw on other staff to support the collection of data.

About 20 percent of teachers in the fall and winter reported working with other educators to

4 Each state had an assigned “research liaison” and a dedicated technology platform/application contact who were available to answer questions and provide one-to-one support.



help collect data from the assessment. And nearly 50 percent or more teachers in the fall

and winter (64% and 48%) reported having support staff in the classroom, which suggests

that support staff may be have been an under-utilized potential resource.

Usability of Overall Assessment Teachers reported that the assessment helped them understand the student skills on each

progression, where their students fell and their next steps in learning.

• The majority of teachers in the fall and winter reported that they were able to use information

from the assessment (72% and 60%, respectively).

• Of the teachers who used the information from the assessment, more than half in the fall

and winter reported that they used the information to identify areas of individual or group

needs or to generally inform instruction (61% and 69%, respectively). One teacher noted,

“[The information from the assessment] is helping to group students for differentiated work

and also helping to hone in on specific skills and create a dialogue with the specialists in the

building more knowledgeable in those areas.”

• Teachers most often described using data from the Language and Communication Domain,

the Cognitive Domain, and the Physical/Motor Domain. One teacher provided an example of

working with a student on crossing midline, “Working with a specific student who was

struggling in all academic areas, I assessed for crossing the midline and found that he could

NOT cross the midline even with assistance. This shows that the two parts of his brain are

not working in sync. Once we increased the crossing midline activities, he started to make

progress elsewhere.”

• Teachers who did not use information from the assessment indicated that they already knew

this information about their children, the information did not align with what they were

teaching, or the information was not as relevant at the time of assessment.

Use of the Assessment with Special Populations. More teachers were able to implement the

assessment with children with developmental delays or disabilities than English Language

Learners, though the majority that did implement the assessment with these populations found

the assessment to be useful or could be useful, either with or without additional modifications or

supports.

• More than half of the teachers were able to implement the assessment with children with

developmental delays or disabilities in the fall (63%) and winter (52%). The majority of these

teachers felt that the skills were or would be effective in capturing their abilities in the fall



and winter (92%a and 100%, respectively). Teachers noted that the kinds of supports that

would be helpful for working with these students would be more assistance from other staff

(e.g., OT, PT, team members), more time spent with the students, and additional guidance

for working with these students.

• Less than half of the teachers were able to implement the assessment with English

Language Learners in the fall (38%) and winter (13%). The majority of these teachers felt

that the skills were or would be effective in capturing their abilities in the fall and winter (75%

and 100%, respectively). Teachers noted that the supports that would be helpful might

include additional guidance for working with these students (particularly assessment in the

child’s native language) and use of visual materials.

Overall Perspectives on Using the Assessment. The majority of teachers in the pilot study

found the K-3 Formative Assessment to be a useful resource for instruction (70% and 61% of

teachers in the fall and winter, respectively). Teachers often cited the assessment as providing

a new or different perspective on students, providing a look at the whole-child, helping to drive

instruction, and providing information for speaking with parents. One teacher commented, “I've

really been able to get an in-depth picture of my kids and the ability of sharing that with the

parents has been great. There was overwhelming support from the parents and it's allowed me

to teach in the way that I know to be good for kids.”

There was less consistency in teacher’s responses for why the assessment was useful for their

instruction. Some teachers that did not find the K-3 Assessment as a resource for instruction

noted that much of the information they already knew at the time of the assessment, suggesting

a mismatch in timing of the pilot data collection windows and where they were in their course of

instruction. Other teachers felt that some of the information was already collected using other

instruments and thus duplicative in nature. Some teachers reported that the information could

have been useful with more guidance about how to use the information to support instruction or

had the assessment been conducted with all of the students in their classes.

Recommendations The overall objective for the pilot was to gather teacher perspectives on the assessment content

and process as well as the materials and training activities in order to inform the 2016-17 field

test. To this end, here is a summary of the teachers’ feedback and the recommendations for

changes that were implemented in the field test.








Emotional Literacy progression was shortened and simplified, the assessment materials for

Emotion Expression and Emotional Regulation Strategies were clarified, and more guidance

on how to assess these constructs was developed (see Assessing the Social - Emotional

Constructs: Overview).





• The progression placements for each construct at each grade level were reviewed and

decisions made to include additional grade levels where appropriate in the field test to

ensure that we collected the full range of data necessary to validate each progression.



Exhibits

Exhibit 1. Pilot Study Fall Sample (All K)

State N of Teachers N of Children Arizona 11 131 Iowa 8 88 Maine 8 89 Rhode Island 6 64 Total 33 372

Exhibit 2. Pilot Study Winter Sample (K-3)

Grade N of Teachers N of Children Kindergarten 37 352 Grade 1 13 114 Grade 2 8 67 Grade 3 7 51 Total 65 584

Exhibit 3. Pilot Study Fall Sample: Child Characteristics

Characteristics N of Children % of Children Gender (Female) 159 52.4% English Learner 28 7.5% Has IEP 30 8.1% Race/Ethnicity Asian/Pacific Islander 5 1.3% Black 9 2.4% Hispanic 56 15.1% White 256 68.8% Other 12 3.2% Unknown 34 9.1%


Appendix D Field Test: Supporting Exhibits Exhibit D.1.1. Teacher survey: Background and training

Exhibit D.1.2. Teacher survey: End of fall

Exhibit D.1.3. Teacher survey: End of field test

Exhibit D.1.4. Teacher focus group protocol

Exhibit D.2.1. Fall K progression placement distributions by distribution type

Exhibit D.2.2. Assessment results for 18 constructs by grade level and administration (fall,

winter)

Exhibit D.2.3 School readiness scores by construct and skill level: Mean total scores (K only)

and mean IRT factor scores (K-3)

Exhibit D.2.4. Teacher survey: Kindergarten entry performance levels

Exhibit D.2.5. Kindergarten entry performance levels: Sample domain expert questionnaire

Exhibit D.2.6. Final K entry performance levels compared to teacher survey responses and K

fall assessment data

Exhibit D.2.7. Percentage of kindergarten students developing and demonstrating expected

skill levels in fall based on expert recommended performance levels

Exhibit D.2.8. Mean progression placements for children assessed in fall and winter

Exhibit D.2.9. Mean progression placements for children assessed in fall by grade level

Exhibit D.2.10. Mean progression placements for children assessed in winter by grade level

Exhibit D.2.11. Percent of kindergarten children at higher, same, or lower levels on progression

from fall to winter administrations

Exhibit D.2.12. Percent of grade 1 children at higher, same, or lower levels on progression from

fall to winter administrations



EAG K-3 Formative Assessment: Final Report September 2017 D-1



Exhibit D.2.15. Progression placements for kindergarten children with and without IEPs

Exhibit D.2.16. Progression placements for kindergarten ELLs and non-ELLs

Exhibit D.2.17. Gender differences in construct performance by grade and test administration

Exhibit D.2.18. Racial/ethnic differences* in construct performance by grade and test

administration

Exhibit D.3.1. Coding protocol for evidence analysis

Exhibit D.3.2. Internal reliability coefficients for evidence analysis

Exhibit D.4.1. A profile of teachers participating in the field test

Exhibit D.5.1. Training guidance for administrators

Exhibit D.7.1. Construct performance for each validity claim


Exhibit D.1.1. Teacher survey: Background and training

K-3 Formative Assessment: Background & Post-Training Teacher Survey 2016–17

Section 1: General Background

1. What grade level are you teaching this year? � Kindergarten � 1st Grade � 2nd Grade � 3rd Grade � Mixed/Multi-Grade (please identify grades): ________ � Other (please describe): ____________________

If you are teaching kindergarten, 1b. Do you teach full- or half-day classes?

� Full Day � Half Day

2. How many hours per day does your class normally meet? (Enter a whole number only.) Hours: ________

3. Including this year, how many years have you been teaching? (Enter a whole number only, leave as "0" if this is your first year teaching.) Years: ________

4. Counting this school year, how many years have you taught each of the following grades and programs? (Enter the number of years to the nearest half-year. Please include part-time teaching. Leave the entry blank if you have never taught the grade or program listed.) a. Preschool or Head Start: ________ b. Kindergarten: (including Transitional/Readiness Kindergarten and Transitional/ pre-1st

grade): ________ c. First grade: ________ d. Second grade: ________ e. Third grade: ________ f. Fourth grade or higher: ________ g. English as a Second Language (ESL) program: ________ h. Bilingual education program: ________ i. Special education program: ________ j. Physical education program: ________ k. Art or music program: ________


5. Which of the following best describes the teaching certificate you currently hold in THIS state? (Mark one response.) � Regular or standard state certificate or advanced professional certificate � Certificate issued after satisfying all requirements except the completion of a

probationary period � Certificate that requires some additional coursework, student teaching, or passage of a

test � Certificate issued to persons who must complete a certification program in order to

continue � I do not hold any of the above certifications in THIS state.

If you hold a teaching certificate in THIS state: 5b. In what areas are you certified? (Select all that apply.)

� Elementary education � Early childhood education � Special education � English as a Second Language (ESL) or instruction for English language learners or

bilingual education � Other (please specify): ____________________

6. Do you have paid paraprofessionals/aides or volunteers working with children in your classroom? � Yes � No

If you selected yes to question 6: 6b. How many hours a week do different types of paid paraprofessionals/aides and/or

volunteers usually work directly with children on instructional tasks either in your classroom or in a pullout setting? (Enter a whole number of hours per week. Leave the entry as blank if none.) a. General paraprofessionals/aides: ________ b. Special education paraprofessionals/aides: ________ c. ESL or bilingual education paraprofessionals/aides: ________ d. Volunteers (e.g., parents, high school students, community members): ________

7. Do you have a computer in your classroom? � Yes � No

8. Do you have access to a tablet/mobile device at school? � Yes � No


9. Do you have internet access in your classroom? � Yes, good access � Yes, spotty access � No

10. Do you have access to internet somewhere in your school? � Yes, good access � Yes, spotty access � No

11. Have you had professional development on formative assessment (outside of this project)? � Yes � No

If you selected yes to question 11: 11b. Approximately how many hours? (Enter a whole number only.)

_______

12. Were you already using formative assessment in your classroom before this project? � Yes � No

Section 2: Demographics

13. What is your gender? � Female � Male

14. Are you Hispanic/Latino? � Yes � No

15. Which best describes your race? (Mark one or more responses to indicate what you consider yourself to be.) � American Indian or Alaska Native � Asian � Black or African American � Native Hawaiian or Other Pacific Islander � White


16. Which languages other than English do you speak fluently? (Select all that apply.) � No language other than English � Spanish � French � Arabic or Amharic � Vietnamese � A Chinese language � Japanese � Korean � Other language (please specify): ____________________

17. What is the highest level of education you have completed? � High school diploma or GED � Associate's degree � Bachelor's degree � At least one year of course work beyond a Bachelor's but not a graduate degree � Master's degree � Education specialist or professional diploma based on at least one year of course work

past a Master's degree level � Doctorate � Other (please specify): ____________________

Section 3: Post-Training Questions

18. How prepared do you feel to implement the assessment after the field test training? � Very prepared � Adequately prepared � Fairly prepared � Not prepared at all

19. Did you participate in the K-3 Formative Assessment Pilot in 2015–16? � Yes, in the Fall K Pilot and/or Winter K-3 Pilot � No, the Field Test is my first time using this assessment

If you selected yes to question 19: 19b. What aspect(s) of the training have been helpful when using the assessment? (Select all

that apply.)

� Updates to the K-3 Formative Assessment � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________


If you selected yes to question 19: 19c. Were there any aspects of the training not helpful? (Select all that apply.)

� Updates to the K-3 Formative Assessment � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________

If you selected no to question 19: 19d. What aspect(s) of the training have been helpful to you when using the assessment?

(Select all that apply.)

� Overviews of five domains � Overview of construct structure � Overview of formative assessment � Magnifying glass advance organizer � Practice “Getting a plan” � Practice “Looking for and gathering evidence” � Practice “Assigning a learning status” � Practice “Summarizing the status” � Practice “Systematically using data” � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________

If you selected no to question 19: 19e. Were any aspects of the training not helpful? (Select all that apply.)

� Overviews of five domains � Overview of construct structure � Overview of formative assessment � Magnifying glass advance organizer � Practice “Getting a plan” � Practice “Looking for and gathering evidence” � Practice “Assigning a learning status” � Practice “Summarizing the status” � Practice “Systematically using data” � Action planning � Question and answer panel with returning pilot teacher(s) � Demonstration of technology platform � Demonstration of the Teacher Site � Asking questions of SRI and NCDPI staff � Other (please specify): ____________________


20. What else would have been helpful to include in the training that would support your use of the assessment? (e.g., additional information or resources)

___________________________________________________________________

Thank you for your participation in this survey!


Exhibit D.1.2. Teacher survey: End of fall

K-3 Formative Assessment: Fall Teacher Survey 2016-17

Section 1: Assessment Materials

1. Please rate your ease of use for each of the following assessment materials/resources: (Select one response for each row.)

Very easy to use

Easy to use

Difficult/ confusing to

use

Very difficult/ confusing to

use

Did not use this material/

resource

Construct Progression □ □ □ □ □

Assessment Means Form □ □ □ □ □

Single Child Documentation Form □ □ □ □ □

Multiple Children Documentation Form □ □ □ □ □

K-Entry Checklist □ □ □ □ □

Online Construct Modules □ □ □ □ □

K-3 Technology Platform □ □ □ □ □

K-3 Evidence App □ □ □ □ □

K-3 Teacher Site □ □ □ □ □


2. How helpful have you found each of the following materials/resources in implementing the assessment so far? (Select one response for each row.)

Very helpful

Somewhat helpful

Not helpful

Not at all helpful

Did not use this material/

resource

Construct Progression □ □ □ □ □

Assessment Means Form □ □ □ □ □

Single Child Documentation Form □ □ □ □ □

Multiple Children Documentation Form □ □ □ □ □

K-Entry Checklist □ □ □ □ □

K-3 Technology Platform □ □ □ □ □

K-3 Evidence App □ □ □ □ □

K-3 Teacher Site □ □ □ □ □

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Construct Progression helpful? ________________________________________________________________________________________________________________________

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Assessment Means Form helpful? ________________________________________________________________________________________________________________________

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Single Child Documentation Form helpful? ________________________________________________________________________________________________________________________


If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the Multiple Children Documentation Form helpful? ________________________________________________________________________________________________________________________

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-Entry Checklist helpful? ________________________________________________________________________________________________________________________

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-3 Technology Platform helpful? ________________________________________________________________________________________________________________________

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-3 Evidence App helpful? ________________________________________________________________________________________________________________________

If you answered “not helpful” or “not at all helpful” to question 2: Why wasn't the K-3 Teacher Site helpful? ________________________________________________________________________________________________________________________


3. Did you watch any of the online construct modules? � Yes � No

If you answered “yes” to question 3: Which construct modules did you watch? (Select all that apply.)

� Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies’ � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary

If you answered “yes” to question 3: For each construct module watched, please rate its helpfulness. (Select one response for each row.)

Very helpful

Somewhat helpful

Not helpful

Not at all helpful

Book Orientation □ □ □ □

Crossing Midline □ □ □ □

Emotion Expression □ □ □ □

Emotion Regulation Strategies □ □ □ □

Emotional Literacy □ □ □ □ Fine Motor: Grip and Manipulation □ □ □ □

Fine Motor: Hand Dominance □ □ □ □

Following Directions □ □ □ □

Gross Motor □ □ □ □

Letter Naming □ □ □ □

Mathematical Patterns □ □ □ □


Very helpful

Somewhat helpful

Not helpful

Not at all helpful

Object Counting □ □ □ □

Perseverance □ □ □ □

Print Awareness □ □ □ □

Problem Solving □ □ □ □

Reading Comprehension □ □ □ □

Writing □ □ □ □

Vocabulary □ □ □ □

If you answered “no” to question 3: Why didn’t you watch any of the modules? ________________________________________________________________________________________________________________________


Section 2: Assessment Content and Process

4. Did you cluster the constructs when implementing the K-3 Assessment in your classroom this fall? � Yes � No

If you answered “no” to question 4: Why not? ________________________________________________________________________________________________________________________

If you answered “yes” to question 4: How helpful has the clustering been in implementing the assessment?

� Very helpful � Somewhat helpful � Not helpful � Not at all helpful


5. Please rate your ease in assessing each of the following constructs: (Select one response for each row.)

Did not assess

Very easy to assess

Easy to assess

Difficult to assess

Very difficult to assess

Book Orientation □ □ □ □ □

Crossing Midline □ □ □ □ □

Emotion Expression □ □ □ □ □

Emotion Regulation Strategies □ □ □ □ □

Emotional Literacy □ □ □ □ □

Fine Motor: Grip and Manipulation □ □ □ □ □

Fine Motor: Hand Dominance □ □ □ □ □

Following Directions □ □ □ □ □

Gross Motor □ □ □ □ □

Letter Naming □ □ □ □ □

Mathematical Patterns □ □ □ □ □

Object Counting □ □ □ □ □

Perseverance □ □ □ □ □

Print Awareness □ □ □ □ □

Problem Solving □ □ □ □ □

Reading Comprehension □ □ □ □ □

Writing □ □ □ □ □

Vocabulary □ □ □ □ □

If you answered “difficult to assess” or “very difficult to assess” for Book Orientation in question 5: Why was the Book Orientation construct difficult to assess? (Select all that apply.)

� The content of the construct was too hard to understand. � The progression was too long. � It was not applicable to my students. � There wasn't enough time to assess the skills. � Other (Please specify): ____________________


If you answered “difficult to assess” or “very difficult to assess” for Crossing Midline in question 5: Why was the Crossing Midline construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Emotion Expression in question 5: Why was the Emotion Expression construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Emotion Regulation Strategies in question 5: Why was the Emotion Regulation Strategies construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Emotional Literacy in question 5: Why was the Emotional Literacy construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Fine Motor: Grip and Manipulation in question 5: Why was the Fine Motor: Grip and Manipulation construct difficult to assess? (Select all that apply.)



If you answered “difficult to assess” or “very difficult to assess” for Fine Motor: Hand Dominance in question 5: Why was the Fine Motor: Hand Dominance construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Following Directions in question 5: Why was the Following Directions construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Gross Motor in question 5: Why was the Gross Motor construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Letter Naming in question 5: Why was the Letter Naming construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Mathematical Patterns in question 5: Why was the Mathematical Patterns construct difficult to assess? (Select all that apply.)



If you answered “difficult to assess” or “very difficult to assess” for Object Counting in question 5: Why was the Object Counting construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Perseverance in question 5: Why was the Perseverance construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Print Awareness in question 5: Why was the Print Awareness construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Problem Solving in question 5: Why was the Problem Solving construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Reading Comprehension in question 5: Why was the Reading Comprehension construct difficult to assess? (Select all that apply.)



If you answered “difficult to assess” or “very difficult to assess” for Writing question 5: Why was the Writing construct difficult to assess? (Select all that apply.)


If you answered “difficult to assess” or “very difficult to assess” for Vocabulary in question 5: Why was the Vocabulary construct difficult to assess? (Select all that apply.)


Section 3: Current Assessment Practices

6. How many other assessments are you required to use this school year? (Enter a whole number only, leave as “0” if none.) Number of assessments:

_______

If you have 1 or more assessments this school year: What domains/areas of development do these assessments capture? (Select all that apply.)

� Literacy/Reading � Math � Writing � Health � Other (Please specify): ____________________

If you have 1 or more assessments this school year: Consider all the time you spend assessing one child with all other required assessments. What is your estimate of total time per child? (Enter a whole number of minutes only, leave as “0” if none.)

Number of minutes: _______

7. Did you use data/information from other assessments as evidence for determining learning status(es) on any of the construct progressions in this assessment? � Yes � No


If you answered “yes” to question 7: What other assessments did you use to inform learning status(es) on the K-3 Formative Assessment? ________________________________________________________________________________________________________________________

8. To what extent are you using information from the K-3 Formative Assessment (e.g., evidence collected, learning status) to inform your instruction? � A great deal � Somewhat � A little � Not at all

9. What supports would be helpful to successfully use this assessment? (Select all that apply.) � Professional Learning Communities (PLCs) � Time allocated for planning � Team meetings to discuss � Good internet access � Access to computer/laptop in classroom � Other (Please specify): ____________________


Section 4: Tell us about your class

10. In a typical day, how much time does a child in your class spend in the following types of activities? (Select one response for each row.)

No time

Half hour or less

About one hour

About two hours

About three hours

Four hours or more

Working independently □ □ □ □ □ □

Working on individual tasks under teacher direction

□ □ □ □ □ □

Working with peers under teacher direction

□ □ □ □ □ □

Working in small groups with teacher

□ □ □ □ □ □

Teacher lecture with large group and/or large group discussion led by teacher

□ □ □ □ □ □

11. During a typical day, how much time per day would you estimate that you spend on classroom discipline and handling disruptive behavior? � Less than 15 minutes a day � 15 minutes to less than 30 minutes a day � 30 minutes to less than 45 minutes a day � 45 minutes to less than 1 hour a day � 1 hour to less than 2 hours a day � 2 hours or more a day

12. At this point in the school year, how would you rate the behavior of the children in your class? � Group misbehaves very frequently and is almost always difficult to handle. � Group misbehaves frequently and is often difficult to handle. � Group misbehaves occasionally. � Group behaves well. � Group behaves exceptionally well.


13. How often are the following materials or resources used in your class? (Select one response for each row.)

Not available Never

Once a month or

less

Two or three times

a month

Once or twice a week

Three or four times

a week Daily

Art materials □ □ □ □ □ □ □

Musical Instruments □ □ □ □ □ □ □

Costumes for creative dramatics/theater

□ □ □ □ □ □ □

Cooking or food related items □ □ □ □ □ □ □

Science equipment (for example, magnifying glass, scales, thermometers)

□ □ □ □ □ □ □



Exhibit D.1.3. Teacher survey: End of field test

K-3 Formative Assessment Final 2016-17 Teacher Survey

Section 1: Assessment Process and Materials

1. Which of the constructs did you administer? (Select all that apply.) � Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary

2. How did you collect evidence for this assessment? (Select all that apply) � Observation of classroom activities � Situation � Task � Other (please specify): ____________________

3. Did you use any of the (optional) Situations available on the assessment means form? � Yes � No


If you answered “yes” to question 3 Please rate the usefulness of the Situations you used. (Select one response for each row.)

Did not use

Very useful

Somewhat useful

Not useful

Not at all useful

a. Book Orientation – Help Me Read this Book □ □ □ □ □

b. Print Awareness – Help Me Read this Book □ □ □ □ □

c. Crossing Midline – All About Shapes □ □ □ □ □ d. Emotional Literacy – How Does the

Character Feel? □ □ □ □ □

e. Emotional Literacy – Why Do You Feel…? □ □ □ □ □ f. Emotional Literacy – Feelings in Social

Context □ □ □ □ □

g. Following Directions – Drawing Animals □ □ □ □ □ h. Grip and Manipulation – Friendship Wreath

Activity □ □ □ □ □ i. Hand Dominance – Friendship Wreath

Activity □ □ □ □ □

j. Gross Motor – Simon Says □ □ □ □ □

k. Letter Naming – Letter Naming Games □ □ □ □ □ l. Mathematical Patterns – Pattern

Calisthenics □ □ □ □ □

m. Mathematical Patterns – Pattern Game □ □ □ □ □ n. Object Counting – Hey! Let’s Go to the

Store! □ □ □ □ □

o. Problem Solving – Deserted Island □ □ □ □ □

p. Problem Solving – Building a Bridge □ □ □ □ □ q. Reading Comprehension – Individual

Reading Time □ □ □ □ □

r. Vocabulary – Group Discussion □ □ □ □ □

s. Vocabulary – Writing □ □ □ □ □

t. Vocabulary – Word Sort □ □ □ □ □

u. Writing – Free Writing Day □ □ □ □ □


4. Did you use any of the (optional) Tasks available on the assessment means form? � Yes � No

If you answered “yes” to question 4: Please rate the usefulness of the Tasks you used:

Did not use

Very useful

Somewhat useful

Not useful

Not at all useful

a. Crossing Midline Task □ □ □ □ □

b. Gross Motor Task □ □ □ □ □

c. Object Counting Task □ □ □ □ □

5. What did you use to document the evidence you collected? (Select all that apply.) � K-3 Technology platform � K-3 Evidence App � Single Child Documentation Form � Multiple Children Documentation Form � K-Entry Construct Checklist � Self-created form � Other (please specify): ____________________

6. Please rate the quality of the following assessment materials/resources:

High quality

Moderate quality

Low quality

Very low quality

a. Construct Progression Forms □ □ □ □

b. Assessment Means Forms □ □ □ □

c. Construct Modules □ □ □ □

d. K-3 Teacher Site □ □ □ □

e. K-3 Evidence App □ □ □ □

f. K-3 Technology Platform □ □ □ □

7. Thinking of all the constructs you assessed, what proportion of them were you able to assess as part of your everyday classroom activities? � All of them � Most of them � Some of them � None of them


If you selected “most of them” or “some of them” to question 7: Of the constructs you administered, which were you not able to assess as part of your everyday classroom activities? (Select all that apply.)

� Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary

8. Did the order of the levels seem right for all of the progressions you assessed (that is, the construct progressed from least difficult to most difficult skill)? � Yes � No


If you answered “no” to question 8: Please identify the progressions for which the levels were not in the right order: (Select all that apply.)

� Book Orientation � Crossing Midline � Emotion Expression � Emotion Regulation Strategies � Emotional Literacy � Fine Motor: Grip and Manipulation � Fine Motor: Hand Dominance � Following Directions � Gross Motor � Letter Naming � Mathematical Patterns � Object Counting � Perseverance � Print Awareness � Problem Solving � Reading Comprehension � Writing � Vocabulary


9. For each of the constructs you administered, indicate how well the progression was able to capture children’s progress and development over time: (Select one response for each row.)

Very well

Somewhat well

Not well

Not at all well





Emotional Literacy □ □ □ □

Fine Motor: Grip and Manipulation □ □ □ □














10. For each of the constructs you administered, please rate your level of confidence in being able to accurately place children on each of the progressions you assessed: (Select one response for each row.)

Very confident

Somewhat confident

Not confident

Not at all confident




















11. For each of the constructs you administered, how helpful was each of these constructs to your instruction? (Select one response for each row.)

Very helpful

Somewhat helpful

Not helpful

Not at all helpful



















If you answered “not helpful” or “not at all helpful” to constructs in question 11: Why wasn’t this construct helpful to your instruction: Book Orientation ________________________________________________________________________________________________________________________

Crossing Midline ________________________________________________________________________________________________________________________


Emotion Expression ________________________________________________________________________________________________________________________

Emotion Regulation Strategies ________________________________________________________________________________________________________________________

Emotional Literacy ________________________________________________________________________________________________________________________

Fine Motor: Grip and Manipulation ________________________________________________________________________________________________________________________

Fine Motor: Hand Dominance ________________________________________________________________________________________________________________________

Following Directions ________________________________________________________________________________________________________________________

Gross Motor ________________________________________________________________________________________________________________________

Letter Naming ________________________________________________________________________________________________________________________

Mathematical Patterns ________________________________________________________________________________________________________________________


Object Counting ____________________________________________________________

____________________________________________________________

Perseverance ________________________________________________________________________________________________________________________

Print Awareness ________________________________________________________________________________________________________________________

Problem Solving ________________________________________________________________________________________________________________________

Reading Comprehension ________________________________________________________________________________________________________________________

Writing ________________________________________________________________________________________________________________________

Vocabulary ________________________________________________________________________________________________________________________

12. During the K-3 assessment process, did you implement the assessment with any children with IEPs or 504 Plans? � Yes � No

If you answered “yes” to question 12: How many? (Enter a whole number only.) _______


If you answered “yes” to question 12: How difficult was it for you to assess this child or these children? Compared to using the assessment with the other children in the class, it was...

� Much more difficult � A little more difficult � Not any more difficult

If you answered “yes” to question 12: How useful was the information for informing his/her/their instruction?

� Very useful � Somewhat useful � Not useful � Not at all useful

13. During the K-3 assessment process, did you implement the assessment with any children who were English language learners? � Yes � No

If you answered “yes” to question 13: How many? (Enter a whole number only.) _______

If you answered “yes” to question 13: How difficult was it for you to assess this child or these children? Compared to using the assessment with the other children in the class, it was...

� Much more difficult � A little more difficult � Not any more difficult

If you answered “yes” to question 13: How useful was the information for informing his/her/their instruction?

� Very useful � Somewhat useful � Not useful � Not at all useful

14. Do you think that most teachers teaching at your grade level would be able to incorporate this assessment into their classroom routines? � Yes � No


If you answered “no” to question 14: Why not? ____________________________________________________________

____________________________________________________________

Section 2: Implementation Supports

15. What ways, if any, have you communicated or collaborated with other professionals about the K-3 Formative Assessment? (Check all that apply) � Professional Learning Communities (PLCs) � Grade-level meetings (e.g., K meeting or 3rd grade meeting) � Teacher-principal meetings � Informal discussions with other teachers � Other (please specify): ____________________ � I did not communicate or collaborate with others about this assessment.

16. For each of the communication and collaboration strategies you used, please tell us how helpful it was in supporting the implementation of the assessment. (Select one response for each strategy.)

Very helpful

Somewhat helpful

Not helpful

Not sure

Professional Learning Communities (PLCs) □ □ □ □ Grade-level meetings (e.g., K meeting or 3rd grade meeting) □ □ □ □

Teacher-principal meetings □ □ □ □

Informal discussions with other teachers □ □ □ □

Other (please specify): ______________ □ □ □ □

17. Did any other school staff provide data/evidence for you during the assessment process? (Select all that apply). � Paraprofessional or aide � Co-teacher � Special education teacher � Art, music or PE teacher � Therapist (speech, physical therapy, etc.) � Other (please specify): ____________________ � No other school staff provided data or evidence during the assessment process.


18. Did your school/administration provide you any of the following specifically to support the administration of the K-3 Assessment? (Select all that apply). � PLCs (professional learning community) � Increased time allocated for planning � Team meetings to discuss � Improved internet access/connectivity � Access to computer/laptop in classroom � Other (please specify): ____________________ � No, the school/administration provided none of this to support administration of the

assessments.

19. Would any of these supports have been helpful to you? (Select all that apply). � PLCs (professional learning community) � Increased time allocated for planning � Team meetings to discuss � Improved internet access/connectivity � Access to computer/laptop in classroom � Other (please specify): ____________________

20. Other than the supports you’ve already mentioned, what else would be helpful to support implementation of the K-3 Formative Assessment in the future?

___________________________________________________________________

Section 3: Opinions and Final Reflections

21. Based on your experience with the K-3 assessment process this school year, what challenges did you face in implementing this assessment? (Select all that apply) � Not enough information about the content of the assessment (e.g., uncertainty about the

purpose or use of the assessment, individual constructs) � Not enough information about expectations for implementing the assessment (e.g., the

cluster scheduling or dates for uploading evidence) � Not enough staff in the school were trained (e.g., it would have been helpful for

administrators to be trained; it would have been helpful to have more teachers in my school using the assessment).

� Not enough support in the classroom to collect the evidence (e.g., it would have been helpful for aides to help collect evidence)

� Lack of support and/or buy-in from my school administrators � Not enough time to implement � Lack of alignment with state standards � Not enough practice with the technology platform or app prior to using it in the classroom � Technical difficulties accessing the technology platform or app (e.g., accessing the

internet, downloading the app, internal server errors) � Problems using the technology platform or app (e.g., understanding how to use, locating

items on the platform/app, lack of support from technology provider) � Other (please specify): ____________________ � No challenges


22. What benefits have you experienced in relation to implementing the new assessment in your classroom? (Select all that apply) � Having deeper understanding of individual child’s skills and developmental needs � Having better understanding of the whole child, of development in multiple domains � Being better equipped to respond to individual student needs � Being better prepared to plan instruction that is developmentally appropriate � Having data all in one place to review for instructional planning � Having readily available data to share with support staff, administrators, and families � Having increased knowledge of how technology can be used for data collection and for

guiding instruction � Other (please specify): ____________________ � No real benefits

23. Overall, how helpful has the K-3 Formative Assessment been as a resource for informing instruction in your classroom? � Very helpful � Somewhat helpful � Not helpful � Not at all helpful

If you answered “not helpful” or “not at all helpful” to question 23: Why wasn’t this assessment helpful for informing your instruction? ____________________________________________________________

____________________________________________________________

Section 4: Tell Us About Your Class

24. As of today’s date... (Enter whole numbers only. Leave as zero if none.)

_______

25. How many children are currently enrolled in your class? _______

26. How many children in your class have a diagnosed disability? _______

27. How many English language learners (ELL) do you have in your class?

_______



Rev. 11/09/16

Exhibit D.1.4. Teacher focus group protocol

K-3 Formative Assessment Teacher Focus Group/Interview Protocol Winter/Spring 2017

Assessment Process 1. Let’s begin by talking about how you collected information on children for the assessment.

a. What did you do?

b. What worked well?

c. What didn’t work?

d. How could you improve your data collection process for the future?

2. Tell us about your planning process for including the assessment in your regular classroom

routine.

Technology Platform 3. What did you think about the technology (e.g., tech platform, evidence app)?

a. What did you like about it?

b. What didn’t you like about it?

4. Did you use the reports available on the tech platform?

a. (If yes/for those who did), what did you think about the reports? Were they useful?

b. Which of the reports, if any, were helpful for implementing this assessment and/or for

informing instruction?

Assessment and Instruction 5. Let’s talk about using the information from the assessment.

a. Were you able to use the information (e.g., evidence collected, learning status) to inform

your instruction?

b. How? Can you give us an example?

6. How would you envision sharing the information from this assessment with parents?

7. Do you feel the K-3 formative assessment construct progressions and learning statuses

provided you with meaningful information about your students’ abilities?

a. Can you share some examples?


Rev. 11/09/16

8. What other assessments did you use this school year?

a. Did you use other assessments in conjunction with this assessment?

i. (If so/for those who did), what did you use?ii. (If so/for those who did), how did you use it?

b. Possible probes

iii. Were they aligned with the K-3 formative assessment?iv. Did you use them as evidence for the K-3 assessment?

Implementation Supports 9. What extra resources and support, if any, did you receive for implementing the K-3

Assessment from your school, district or state?

Probe for:

a. Additional teachers/paraprofessionals in the classroom

b. Time allocated for planning

c. Internet access

d. Access to computer/laptop in classroom

10. Were there any other types of formal or informal help or assistance that you received or

provided to others that we have not yet discussed?

Probe for:

a. Help collecting evidences from other teachers/paraprofessionals

b. Professional Learning Communities (PLCs)

c. Grade-level meetings

d. Teacher-principal meeting

e. Informal discussions with other teachers

11. Is there anything that the leadership either in your school or district could have done to

make your experience with the assessment more successful?


Rev. 11/09/16

Opinions and Final Reflections

12. Now that you have gone through the entire assessment process, are there any changes to

the training or the materials that would help a teacher more effectively implement this

assessment?

Probe for:

a. Additional training

b. Additional materials or resources

c. Changes in the construct progressions

13. Did the implementation of the assessment positively or negatively affect other activities in

your classroom?

a. How so?

14. Is there anything else you would like to share about your experience implementing this

assessment?


Exhibit D.2.1. Fall K progression placement distributions by distribution type

Unimodal distribution with smooth tails

2%

15%18%

28% 26%

10%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E

Emotion Regulation Strategies - Grade K Fall (n = 787)

1%5%

9%

18%

37%

12%6% 6% 5%

1%0%

10%

20%

30%

40%

50%

60%


Emotional Literacy - Grade K Fall (n = 803)

1% 3%

27%

51%

13%

4%0% 0%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E F GWriting - Grade K (n = 826)


Unimodal distribution with smooth tail, skewed to higher end

1% 0% 3%11%

86%

0%

20%

40%

60%

80%

100%

Emerging A B C D

Book Orientation - Grade K Fall (n = 837)

0%5%

20%

75%

0%

20%

40%

60%

80%

100%

Emerging A B C

Crossing Midline - Grade K Fall (n = 829)

1%4% 5%

11%16% 18%

45%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E FFollowing Directions - Fall Grade K (n = 827)


Unimodal distribution with smooth tail, skewed to higher end (continued)

0%5%

22%

30%

42%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D

Grip & Manipulation: Grade K Fall (n = 833)

2%

23%

75%

0%

20%

40%

60%

80%

100%

Emerging A B C

Hand Dominance - Grade K Fall (n = 835)

0% 1% 1% 1%

9%

28% 29% 31%

0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E F G

Letter Naming - Grade K Fall (n = 835)


Unimodal distribution with smooth tail, skewed to lower end

Unimodal distribution with uneven tail(s)

2%

13%

29%34%

11%8%

2% 2% 1%0%

10%

20%

30%

40%

50%

60%


Vocabulary - Grade K - Fall (n = 796)

0%

12%16% 15%

20%24%

7% 7%

0%

10%

20%

30%

40%

50%

60%


Emotion Expression Grade K Fall (n = 805)

0% 0% 0% 1%6% 3%

14% 16%13%

46%

0%

10%

20%

30%

40%

50%

60%


Gross Motor - Grade K Fall (n = 832)


Unimodal with uneven tail(s) (continued)

4%9%

3%

18%

28%

18%

10%6% 3%

0% 0% 0% 0% 0%0%

10%

20%

30%

40%

50%

60%

Emerging A B C D E F G H I J K L M

Mathematical Patterns - Grade K Fall (n = 797)

0% 3% 4%6% 4% 5%

29%

21%

10%6%

12%

0%

10%

20%

30%

40%

50%

60%


Object Counting - Grade K Fall (n = 826)

1%

7%13%

10%

24%

11%14% 12%

9%

0%

10%

20%

30%

40%

50%

60%


Perseverance - Grade K Fall (n = 795)


Unimodal with uneven tail(s) (continued)

0%4%

10%5%

12%16%

11%

41%

0%

10%

20%

30%

40%

50%

60%


Print Awareness - Grade K Fall (n = 843)

1%6%

14%20%

28%

11%

3%10%

3% 2% 1%0%

10%

20%

30%

40%

50%

60%


Problem Solving - Grade K Fall (n = 797)

2%8%

16%

26%

11%8% 9%

12%8%

0%

10%

20%

30%

40%

50%

60%


Reading Comprehension - Grade K Fall (n = 784)


Exhibit D.2.2. Assessment results for 18 constructs by grade level and administration (fall, winter)

Construct, Grade Level and Administration

Children Assessed

Most Frequent Skill Level

Range of Skill Level

Placements Numeric Average

Standard Deviation

Book Orientation (A-D)

K-Fall 837 D-86% Emerging-D 4.8 0.5 K-Winter 733 D-98% Emerging-D 5.0 0.2

Hand Dominance (A-C) K-Fall 835 C-75% A-C 3.7 0.5 K-Winter 783 C-93% A-C 3.9 0.3

Crossing Midline (A-C) K-Fall 829 C-75% Emerging-C 3.7 0.6 K-Winter 782 C-87% Emerging-C 3.8 0.4 1-Fall 184 C-77% Emerging-C 3.7 0.6 1-Winter 193 C-85% A-C 3.8 0.4

Grip & Manipulation (A-D) K-Fall 833 D-42% A-D 4.1 0.9 K-Winter 784 D-82% A-D 4.8 0.6 1-Fall 197 D-64% A-D 4.5 0.8 1-Winter 193 D-79% A-D 4.7 0.7

Letter Naming (A-G) K-Fall 835 G-31% Emerging-G 6.7 1.2 K-Winter 783 G-73% B-G 7.6 0.8 1-Fall 196 G-66% E-G 7.6 0.7 1-Winter 192 G-84% E-G 7.8 0.5

Print Awareness (A-G) K-Fall 843 G-41% Emerging-G 6.3 1.9 K-Winter 782 G-84% A-G 7.7 0.9 1-Fall 185 G-94% E-G 7.9 0.3 1-Winter 181 G-99% E-G 8.0 0.2

Object Counting (A-J) K-Fall 826 F-29% Emerging-J 7.4 2.2 K-Winter 782 J-41% Emerging-J 9.4 1.9 1-Fall 185 J-46% A-J 9.7 1.7 1-Winter 193 J-66% A-J 10.3 1.2 2-Fall 126 J-78% C-J 10.6 0.9 2-Winter 115 J-93% D-J 10.9 0.7

Emotion Expression (A-G) K-Fall 805 E-24% Emerging-G 4.7 1.7 K-Winter 763 E-21% Emerging-G 5.8 1.8 1-Fall 195 E-19% A-G 5.4 1.9 1-Winter 193 G-39% A-G 6.5 1.6



Children Assessed




Standard Deviation

2-Fall 126 G-32% A-G 6.4 1.6 2-Winter 115 G-39% A-G 6.7 1.5 3-Fall 138 E-31% A-G 6.6 1.3 3-Winter 135 G-55% A-G 7.3 1.2

Emotion Regulation Strategies (A-E)

K-Fall 787 C-28% Emerging-E 3.9 1.3 K-Winter 773 E-21% Emerging-E 4.5 1.3 1-Fall 197 C-28% Emerging-E 4.5 1.2 1-Winter 194 E-48% A-E 5.1 1.1 2-Fall 126 E-35% Emerging-E 4.7 1.3 2-Winter 115 E-48% Emerging-E 5.0 1.2 3-Fall 127 E-31% A-E 5.2 1.0 3-Winter 135 E-56% Emerging-E 5.3 1.0

Emotional Literacy (A-I) K-Fall 803 D-37% Emerging-I 5.2 1.8 K-Winter 768 D-27% Emerging-I 6.5 2.1 1-Fall 164 D-22% A-I 6.2 2.0 1-Winter 162 H-22% B-I 7.6 2.0 2-Fall 126 H-29% A-I 7.8 2.0 2-Winter 115 I-43% C-I 8.8 1.5 3-Fall 126 H-28% A-I 8.1 1.8 3-Winter 135 I-53% B-I 9.2 1.3

Following Directions (A-F) K-Fall 827 F-45% Emerging-F 5.7 1.5 K-Winter 782 F-67% A-F 6.3 1.1 1-Fall 194 F-58% A-F 6.0 1.4 1-Winter 193 F-56% A-F 6.2 1.2 2-Fall 126 F-62% B-F 6.3 1.1 2-Winter 115 F-74% B-F 6.5 0.9 3-Fall 138 F-49% A-F 5.8 1.5 3-Winter 135 F-73% C-F 6.6 0.8

Gross Motor (A-I) K-Fall 832 I-46% Emerging-I 8.6 1.7 K-Winter 782 I-68% A-I 9.3 1.3 1-Fall 185 I-51% A-I 8.8 1.6 1-Winter 186 I-69% C-I 9.5 0.9 2-Fall 126 I-84% A-I 9.7 1.0 2-Winter 115 I-95% A-I 9.9 0.8 3-Fall 138 I-83% A-I 9.7 0.9 3-Winter 123 I-93% F-I 9.9 0.4



Children Assessed




Standard Deviation

Mathematical Patterns (A-M)

K-Fall 797 D-28% Emerging-H 5.0 1.8 K-Winter 770 F-26% Emerging-M 6.9 1.9 1-Fall 184 G-20% Emerging-K 7.3 2.1 1-Winter 193 H-37% C-L 8.7 1.7 2-Fall 126 H-26% A-M 9.0 2.5 2-Winter 115 I-31% B-M 9.6 2.6 3-Fall 127 J-20% B-L 9.0 3.0 3-Winter 124 J-27% Emerging-M 10.4 2.7

Perseverance (A-H) K-Fall 795 D-24% Emerging-H 5.5 2.1 K-Winter 770 H-22% Emerging-H 6.5 2.1 1-Fall 194 D-25% Emerging-H 5.9 1.9 1-Winter 193 G-27% B-H 7.2 1.7 2-Fall 126 G-45% A-H 7.4 1.7 2-Winter 115 G-40% B-H 7.7 1.4 3-Fall 138 G-27% A-H 6.8 1.9 3-Winter 135 H-45% A-H 7.7 1.6

Problem Solving (A-J) K-Fall 797 D-28% Emerging-J 5.0 2.0 K-Winter 766 D-28% Emerging-J 6.4 2.4 1-Fall 181 D-27% Emerging-J 6.1 1.8 1-Winter 193 G-28% A-J 7.7 2.1 2-Fall 126 G-27% A-J 7.9 2.1 2-Winter 115 G-28% C-J 8.9 1.8 3-Fall 138 G-33% B-J 7.4 1.8 3-Winter 127 I-35% A-J 8.9 1.6

Reading Comprehension (A-H)

K-Fall 784 C-26% Emerging-H 5.0 2.2 K-Winter 781 G, H-22% Emerging-H 6.7 2.0 1-Fall 196 H-27% C-H 7.3 1.5 1-Winter 193 H-34% A-H 7.6 1.4 2-Fall 126 H-32% B-H 7.4 1.5 2-Winter 115 H-57% B-H 8.1 1.4 3-Fall 138 H-37% Emerging-H 7.6 1.7 3-Winter 134 H-54% D-H 8.3 1.0

Vocabulary (A-H) K-Fall 796 C-34% Emerging-H 3.8 1.4 K-Winter 772 E-20% Emerging-H 5.2 2.0 1-Fall 184 B-23% Emerging-H 4.9 1.9



Children Assessed




Standard Deviation

1-Winter 184 D-23% A-H 5.9 1.9 2-Fall 126 F-40% B-H 6.4 1.6 2-Winter 115 F-30% B-H 7.5 1.4 3-Fall 137 F-31% Emerging-H 6.2 1.7 3-Winter 135 F-41% A-H 6.9 1.5

Writing (A-G) K-Fall 826 C-51% Emerging-E 3.9 0.9 K-Winter 780 C-32% Emerging-G 5.1 1.3 1-Fall 195 D-39% Emerging-G 5.3 1.2 1-Winter 193 F-35% Emerging-G 6.0 1.1 2-Fall 126 E-42% Emerging-G 6.0 1.1 2-Winter 115 F-55% A-G 6.6 1.0 3-Fall 138 F-35% B-G 6.5 1.2 3-Winter 134 F-54% Emerging-G 6.9 1.0


Exhibit D.2.3 School readiness scores by construct and skill level: Mean total scores (K only) and mean IRT factor scores (K-3)

Book Orientation Total Scores (K only) IRT Factor Scores (K-3)

Emerging 5 61.2 5 -1.43 A 2 76.0 2 -0.91 B 21 75.1 22 -0.97 C 79 76.6 89 -0.88 D 647 95.7 750 -0.19

Crossing Midline Total Scores (K only) IRT Factor Scores (K-3)

Emerging 3 58.7 4 -1.22 A 34 75.7 52 -0.76 B 149 82.8 201 -0.54 C 568 96.7 766 -0.08

Emotion Expression Total Scores (K only) IRT Factor Scores (K-3)

Emerging 4 53.0 4 -1.84 A 90 77.1 119 -0.76 B 124 87.2 160 -0.44 C 110 87.2 159 -0.38 D 153 92.5 202 -0.23 E 178 100.7 295 0.17 F 49 103.1 154 0.57 G 46 115.4 170 0.80

Emotion Regulation Strategies Total Scores (K only) IRT Factor Scores (K-3)

Emerging 19 73.4 23 -0.92 A 112 78.8 138 -0.70 B 139 88.2 192 -0.33 C 209 92.4 325 -0.15 D 195 101.5 314 0.20 E 80 105.2 245 0.68


Emotional Literacy Total Scores (K only) IRT Factor Scores (K-3)

Emerging 7 66.0 7 -1.34 A 36 69.7 42 -1.07 B 65 78.9 79 -0.75 C 133 88.9 177 -0.42 D 279 92.7 364 -0.26 E 90 96.2 137 -0.08 F 45 103.5 94 0.35 G 49 104.7 111 0.38 H 40 114.5 133 0.80 I 10 120.1 76 1.07

Following Directions Total Scores (K only) IRT Factor Scores (K-3)

Emerging 5 71.2 5 -1.15 A 36 70.3 48 -0.97 B 35 80.6 57 -0.58 C 83 78.3 135 -0.51 D 125 86.8 178 -0.33 E 126 91.9 230 -0.03 F 344 102.8 632 0.35

Grip and Manipulation Total Scores (K only) IRT Factor Scores (K-3)

Emerging 0 0 A 39 75.9 50 -0.86 B 168 84.4 210 -0.57 C 234 93.5 299 -0.23 D 313 99.0 488 0.04

Gross Motor Total Scores (K only) IRT Factor Scores (K-3)

Emerging 3 70.7 3 -1.11 A 3 64.3 7 -0.95 B 1 64.0 1 -1.32 C 9 65.4 11 -1.19 D 34 73.9 51 -0.81 E 21 73.6 40 -0.61 F 110 84.2 154 -0.45 G 122 92.0 154 -0.31 H 94 94.7 160 -0.02 I 357 99.4 700 0.31


Hand Dominance Total Scores (K only) IRT Factor Scores (K-3)

Emerging 0 0 A 12 77.7 13 -0.78 B 176 82.4 199 -0.69 C 566 96.4 653 -0.17

Letter Naming Total Scores (K only) IRT Factor Scores (K-3)

Emerging 3 53.0 3 -1.73 A 8 56.1 8 -1.63 B 10 67.4 11 -1.25 C 6 72.0 7 -1.09 D 69 80.2 78 -0.78 E 207 86.8 249 -0.50 F 220 94.3 288 -0.24 G 231 104.1 397 0.21

Mathematical Patterns Total Scores (K only) IRT Factor Scores (K-3)

Emerging 28 64.1 31 -1.33 A 74 83.3 77 -0.66 B 26 83.2 38 -0.49 C 139 87.1 163 -0.49 D 198 93.0 274 -0.25 E 144 97.9 160 -0.18 F 79 102.1 141 0.17 G 43 103.5 108 0.31 H 23 118.6 109 0.72 I 0 0 17 0.76 J 0 0 65 1.08 K 0 0 38 1.27 L 0 0 12 1.37 M 0 0 1 1.38

Object Counting Total Scores (K only) IRT Factor Scores (K-3)

Emerging 3 44.3 3 -2.11 A 23 65.2 25 -1.25 B 27 75.6 31 -0.94 C 51 77.7 54 -0.85 D 34 83.1 37 -0.66 E 40 80.6 50 -0.74 F 215 91.7 247 -0.38 G 155 98.0 187 -0.14 H 66 99.2 136 0.09 I 48 104.7 94 0.16 J 92 107.0 281 0.57


Perseverance Total Scores (K only) IRT Factor Scores (K-3)

Emerging 9 54.3 10 -1.59 A 56 73.6 66 -0.93 B 103 81.8 128 -0.61 C 72 87.3 121 -0.41 D 181 90.1 261 -0.27 E 71 96.8 139 0.05 F 99 101.0 164 0.18 G 94 104.7 211 0.56 H 69 111.0 153 0.79

Print Awareness Total Scores (K only) IRT Factor Scores (K-3)

Emerging 1 78.0 1 -0.85 A 30 74.0 30 -0.99 B 84 75.8 84 -0.93 C 39 80.5 40 -0.78 D 93 85.2 104 -0.61 E 130 89.7 143 -0.46 F 74 96.4 106 -0.26 G 303 103.9 530 0.15

Problem Solving Total Scores (K only) IRT Factor Scores (K-3)

Emerging 10 54.9 11 -1.54 A 50 74.4 53 -0.94 B 109 79.4 128 -0.76 C 153 87.6 181 -0.49 D 205 95.4 320 -0.13 E 83 100.2 154 0.09 F 27 105.0 53 0.38 G 73 106.5 195 0.55 H 20 115.2 79 0.94 I 16 115.1 38 1.01 J 8 130.5 30 1.30

Reading Comprehension Total Scores (K only) IRT Factor Scores (K-3)

Emerging 18 57.2 20 -1.49 A 59 77.2 60 -0.88 B 121 83.3 130 -0.65 C 197 88.8 218 -0.49 D 87 94.2 145 -0.09 E 58 97.4 104 -0.04 F 66 101.1 145 0.24 G 95 110.0 218 0.52 H 53 111.1 204 0.77


Vocabulary Total Scores (K only) IRT Factor Scores (K-3)

Emerging 10 57.5 14 -1.35 A 101 76.6 117 -0.83 B 217 90.3 297 -0.34 C 257 94.7 321 -0.22 D 78 101.1 139 0.09 E 59 106.6 143 0.40 F 16 113.0 118 0.94 G 12 108.6 67 0.95 H 4 116.8 27 1.08

Writing Total Scores (K only) IRT Factor Scores (K-3)

Emerging 10 59.4 12 -1.36 A 21 74.5 21 -0.97 B 211 82.5 240 -0.68 C 384 95.6 479 -0.21 D 98 104.2 213 0.13 E 30 116.9 162 0.68 F 0 0 126 0.96 G 0 0 32 1.34


Exhibit D.2.4. Teacher survey: Kindergarten entry performance levels

K Teacher Survey 2017 Kindergarten Entry Expectations Welcome! The purpose of this survey is to collect teacher input on expectations for children’s skills and abilities at kindergarten entry. You are being asked to provide the SRI team with valuable information on your expectations for children when they begin kindergarten. The information will help K teachers know where to start the assessment at the beginning of the school year and better describe and report where children are at kindergarten entry.

We recognize that incoming kindergarten children show a range of skills so we are interested in what you consider are the typical, or average range of expected skills for children entering the kindergarten classroom and the lower end of the range for expected skills at kindergarten entry.

Directions:

On the following screens, you will be presented with the 18 construct progressions and asked to mark two points on each progression:

Typical: For the first point, select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.

Far below: For the second point, mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.

For example,

• If you think Level D is the skill level of a typical child at kindergarten entry, select the button next to Skill D in the column labeled "Typical".

• If you think Level B is the skill level of a child who is so far below the skill level of the typical child at kindergarten entry that she is likely to experience some difficulties at kindergarten entry in this area, then you would mark B in the column labeled "Far Below". In this example, a child at skill level A would also be likely to experience difficulties but we want you to mark the highest level at which children are likely to experience some difficulty at kindergarten entry.

• If you think that the “likely to struggle” level is lower than Level A, mark that choice for the column labeled "Far Below".

It is important that you answer each question. Your responses will help us generate reporting structures and guidance for teachers using this tool in the future.

If you need additional information (e.g., examples) on any of the progressions, feel free to consult the construct progression documents in your binder or on the Teacher Site. And if you have any questions, please contact [email protected] or [email protected].


mailto:[email protected]


Typical - Select the level on the progression that represents the skill level of an average or typical incoming kindergarten child in the first week of school.

Far below: Mark the highest level on the progression where a child is so far below the skill level of a typical kindergarten child that he or she may experience significant difficulties in the classroom in this construct.

Domain: Language Development and Communication Construct: Book Orientation

Typical Far Below Skills: Book Orientation

□ □ “May experience significant difficulties in the classroom” is lower than Level A.

□ □ A. Holds or otherwise demonstrates awareness that a book is an object with

pages

□ □ B. Eventually holds the book upright (so pictures and text are right side up,

irrespective of front to back orientation)

□ □ C. Holds the book upright, opens it from the front cover, and turns pages (not

always one by one)

□ □ D. Holds the book upright, and starting at the front, turns the pages in order, one

page at a time

Construct: Print Awareness

Typical Far Below Skills: Print Awareness


□ □ A. Attends only to pictures and ignores text.

□ □ B. Indicates the general area of text and/or where we read words (making a

distinction between pictures and text).

□ □ C. Indicates where to begin reading and that lines of text are read from top to

bottom (where reading begins, top to bottom directionality).

□ □ D. Indicates that lines of text are read from left to right (e.g., tracking text) (left to

right directionality).

□ □ E. Indicates that lines of text are read from left to right; and at the end of each

line, returns to the beginning of the next line of text (e.g., tracking text) (return sweep).

□ □ F. Distinguishes between a letter and a word on a page of text (excluding the

words a, A, and I).

□ □ G. Indicates one word on the page for each word read aloud (concept of word).




Construct: Letter Naming

Typical Far Below Skills: Letter Naming


□ □ A. Distinguishes letters from pictures, shapes, or numerals

□ □ B. Distinguishes own first name from other names or other words (e.g., on

folders, name charts, among other words or names)

□ □ C. Locates, talks about, or asks questions about letters in one’s environment

□ □ D. Identifies (names or selects) one or more letters in own first name

□ □ E. Identifies (names or selects) some upper case or lowercase letters (in

additional to letters found in own first name)

□ □ F. Identifies (names or selects) each of the 26 letters in some form (may be a

combination of uppercase and lowercase)

□ □ G. Identifies (names or selects) all 52 letters in uppercase and lowercase form.

Construct: Following Directions

Typical Far Below Skills: Following Directions


□ □ A. Completes one-step directions when support is provided in the form of

visuals, gestures, or modeling.

□ □ B. Completes one-step directions (without support provided).

□ □ C. Partially completes two-step directions (without support provided) by

completing only one step or completing the steps out of order.

□ □ D. Completes two-step directions in the correct order (without support

provided).

□ □ E. Partially completes three-step directions (without support provided) by

completing two steps, or completing the steps out of order.

□ □ F. Completes three-step directions in the correct order (without support

provided).




Construct: Writing

Typical Far Below Skills: Writing


□ □ A. Dictates ideas, information, or stories to have them written down.

□ □ B. Uses drawings, scribbles or letter-like forms to express ideas, information or

stories.

□ □ C. Writes letters, words, or phrases to label drawings or express ideas,

information or stories.

□ □ D. Writes, demonstrating an understanding of purpose or audience or both.

□ □ E. Writes two or more related ideas, pieces of information, or events.

□ □ F. Writes using a genre-specific organizational structure.

□ □ G. Revises own genre-specific writing to provide clarity to the reader.

Construct: Reading Comprehension

Typical Far Below Skills: Reading Comprehension


□ □ A. Listens to a story or informational text (with or without pictures) and

demonstrates an understanding of the text.

□ □ B. Views pictures or images and demonstrates an understanding of the story or

information presented.

□ □ C. Pretends to read a familiar book, retelling the story line or information

presented in the text.

□ □ D. Reads without noticing when the reading does not make sense.

□ □ E. Monitors reading, noticing that the reading does not make sense, without

attempting to repair meaning.

□ □ F. Monitors reading, noticing that the reading does not make sense, and

attempts unsuccessfully to repair meaning.

□ □ G. Monitors reading and attempts to repair meaning by asking for and using

comprehension strategies.

□ □ H. Monitors reading successfully using self-selected comprehension strategies

to maintain and improve meaning.




Construct: Vocabulary

Typical Far Below Skills: Vocabulary


□ □ A. Connects a word to its related concept.

□ □ B. Uses school-related words appropriately in one or more contexts.

□ □ C. Uses multiple words that are closely related in meaning to describe a single

school-related concept.

□ □ D. Determines the appropriate meanings of multiple-meaning words by using

context.

□ □ E. Uses at least two meanings of multiple-meaning words.

□ □ F. Determines the meaning of unfamiliar words by using known words and

knowledge of word structure (e.g., roots, affixes, compound words)

□ □ G. Recognizes both literal meanings and non-literal meanings (e.g., language

that is nuanced, more precise or descriptive, has shades of meaning) of words and phrases.

□ □ H. Uses non-literal meanings of words and phrases (e.g., language that is

nuanced, more precise or descriptive, has shades of meaning)

Domain: Cognitive Development Construct: Object Counting

Typical Far Below Skills: Object Counting


□ □ A. Displays early counting behavior with 4-6 objects arranged in a line (i.e.,

says or indicates some number words while pointing to the objects but does not count all of the objects correctly).

□ □ B. Consistently counts 4-6 objects in a line correctly.

□ □ C. Consistently counts 4-6 objects in a scattered arrangement correctly.

□ □ D. Knows the last number word used while counting is the total quantity and

that the value of a collection of objects does not change unless objects are added or removed.

□ □ E. Consistently counts out 4-6 objects from a set of more than 10 objects.

□ □ F. Consistently counts out 8-10 objects from a set of more than 10 objects.

□ □ G. Correctly counts 18-20 objects in a scattered arrangement.

□ □ H. Uses beginning strategies (i.e., counting again from one or by repeating the

cardinal number in the original set and then counting on) to find the new total when one object is added to a set of 6-10 objects.

□ □ I. Procedures the correct number (without pause) when one object is added to

a set of 6-10 objects.

□ □ J. Produces the correct number automatically (without pause) when two

objects are added to a set of 6-10 objects.




Construct: Mathematical Patterns

Typical Far Below Skills: Mathematical Patterns


□ □ A. Duplicates sequential AB patterns with the same materials.

□ □ B. Duplicates sequential three-element patterns (e.g., ABB, ABC) with the

same materials.

□ □ C. Extends sequential AB patterns at least one unit.

□ □ D. Extends sequential three-element patterns (e.g., ABB, ABC) at least one

unit.

□ □ E. Duplicates sequential patterns (e.g., AB, ABC, AABB) using materials

different from those used in the model pattern (pattern abstraction).

□ □ F. Identifies repeating unit in sequential patterns.

□ □ G. Extends by at least one step or determines the missing step in spatial or

numerical growing patterns.

□ □ H. Communicates a recursive rule governing the next step in spatial or

numerical growing patterns.

□ □ I. Creates or enters data in a t-chart to document the relationship between the

ordinal position of a step in a growing pattern (i.e., first, second, third) and an important feature of the step.

□ □ J. Applies the relationship between the two variables in a t-chart to extend a

numerical growing pattern by at least one step.

□ □ K. Communicates a one-operation functional rule governing spatial or

numerical growing patterns and uses it to determine a far step. Note: a “far step” is more than 5 steps from the last represented step in the pattern.

□ □ L. Communicates a two-operation functional rule governing spatial or

numerical growing patterns and uses it to determine a far-step. Note: a “far step” is more than 5 steps from the last represented step in the patterns.

□ □ M. Creates an equation that symbolizes a functional rule governing a spatial or

numerical growing pattern.




Construct: Problem Solving

Typical Far Below Skills: Problem Solving


□ □ A. Acknowledges that a problem exists without attempting to solve it (this may

manifest as a child moving away from the problem).

□ □ B. Attempts to solve a problem by mimicking the motions and procedures of

others or seeking general support from others very early in the problem solving process (e.g., how do I do this? I don’t know what to do).

□ □ C. Attempts to solve a problem using random trial and error.

□ □ D. Solves or attempts to solve a familiar problem using procedures learned in

previous problem solving experiences.

□ □ E. Solves or attempts to solve a novel problem using procedures learned in

previous problem solving experiences without demonstrating knowledge of why the procedure is or is not successful

□ □ F. States a hypothesis about how to solve a novel problem, using both

concepts and procedures.

□ □ G. Solves or attempts to solve a novel problem by connecting concepts and

using familiar procedures.

□ □ H. Generates and explains an alternate problem solving approach (including

when an approach is not working).

□ □ I. Generates and explains multiple approaches for solving a problem.

□ □ J. Provides justification for why a chosen self- or peer-generated problem

solving approach might be the most efficient one for solving a problem.




Domain: Approaches to Learning

Construct: Perseverance

Typical Far Below Skills: Perseverance


□ □ A. Begins to persevere in familiar and/or interesting classroom activities but

quickly discontinues involvement in the activity after one or very few attempts.

□ □ B. Perseveres in familiar and/or interesting classroom activities, but

discontinues involvement after persisting through trials for a short period of the allotted time.

□ □ C. Perseveres in familiar and/or classroom activities and works through most

trials, but discontinues persisting before the end of the allotted time.

□ □ D. Perseveres in familiar and/or interesting classroom activities throughout the

activity, persisting through trials throughout the allotted time.

□ □ E. Perseveres in classroom activities that are novel and/or moderately difficult

for the child, flagging in effort at points of challenge in the activity.

□ □ F. Perseveres within the allotted time in classroom activities that are novel

and/or moderately difficult for the child sometimes persisting and maintaining effort at points of challenge in the activity.

□ □ G. Perseveres throughout the allotted time in classroom activities that are novel

and/or moderately difficult for the child, persisting and maintaining effort at points of challenge in the activity.

□ □ H. Perseveres throughout the allotted time in classroom activities that are very

difficult for the child, persisting and maintaining effort at points of challenge in the activity.

Domain: Social-Emotional Development

Construct: Emotion Expression

Typical Far Below Skills: Emotion Expression


□ □ A. Acts, talks or reacts to an experience or stimulus without demonstrating

awareness of the reactions of others around them.

□ □ B. Shows awareness that his or her own actions, words, and reactions to an

experience or stimulus has an impact on others.

□ □ C. Makes choices to express emotions in ways that get their needs met.

□ □ D. Begins to express emotions in ways that support building relationships.

□ □ E. Generally expresses emotions in ways that support building relationships.

□ □ F. Reflects about emotions and the consequences of actions with support from

the teacher.

□ □ G. Independently reflects about emotions and the consequences of actions.




Construct: Emotional Literacy

Typical Far Below Skills: Emotional Literacy


□ □ A. Expresses verbally or behaviorally a range of emotions.

□ □ B. Shows an awareness of his or her verbal and/or behavioral expression of

emotions.

□ □ C. Identifies emotions in self.

□ □ D. Identifies emotions in others.

□ □ E. Indicates a response to an event can include one or more emotions,

including conflicting emotions, within themselves or another person.

□ □ F. Indicates the same event can cause different people to experience different

emotions.

□ □ G. Indicates knowledge of how the social context relates to his/her own and

others’ emotion experience.

□ □ H. Interprets the emotional experience of others using the social context (e.g.,

provides justification that includes social information and inferences about others’ intentions).

□ □ I. Predicts the emotional experience of other using the social context (e.g.,

provides justification that includes social information and inferences about others’ intentions).

Construct: Emotion Regulation Strategies

Typical Far Below Skills: Emotion Regulation Strategies


□ □ A. When offered strategies for regulating the expression of emotions, begins to

use the strategies offered with continual support.

□ □ B. When offered strategies for regulating the expression of emotions,

consistently uses the strategies offered with minimal support.

□ □ C. Sometimes uses learned strategies independently to regulate the

expression of emotions during routine activities.

□ □ D. Consistently uses learned strategies independently to regulate the

expression of emotions during routine activities.

□ □ E. Independently uses learned strategies for regulating emotions during

complex contextual transitions (e.g., exposed to multiple stimuli simultaneously), to accomplish a different or new type of task, because of interruptions, or because of changes in the daily routines.




Domain: Health and Physical Development

Construct: Hand Dominance

Typical Far Below Skills: Hand Dominance


□ □ A. Uses no established dominance for lead/dominant hand (switching still

continues).

□ □ B. Uses established dominant hand.

□ □ C. Manipulates with dominant hand with assistance from other hand.

Construct: Grip and Manipulation

Typical Far Below Skills: Grip and Manipulation


□ □ A. Uses early fine motor skills (e.g., fisted grip, palmar grasp, or early scissor

grip) to hold and/or manipulate items, with whole arm movement.

□ □ B. Uses a more refined grip (e.g., using thumb and finger [pincer grip] or tripod

grip) to hold and/or manipulate objects with whole arm movement and increased stability from the shoulder.

□ □ C. Uses refined wrist and finger movement, beginning to transfer control of

movement from the shoulder to the elbow.

□ □ D. Uses hands with minimal elbow movement and primary control from wrist

and fingers.

Construct: Crossing Midline

Typical Far Below Skills: Crossing Midline


□ □ A. Movement is isolated to one side of the midline (the invisible line running

from our head to our toes, dividing the body into left and right halves).

□ □ B. Inconsistently crosses midline.

□ □ C. Consistently crosses midline.




Construct: Gross Motor

Typical Far Below Skills: Gross Motor


□ □ A. Walks on a flat foot or walks on the toes.

□ □ B. Walks by distributing weight from heel to toe WITHOUT arms and legs

moving in opposition.

□ □ C. Walks by distributing weight from heel to toe WITH arms and legs moving in

opposition.

□ □ D. Maintains balance and control when running.

□ □ E. Maneuvers around objects and people while running

□ □ F. Emerging galloping skills.

□ □ G. Gallops smoothly.

□ □ H. Hops forward smoothly on one foot.

□ □ I. Skips smoothly.

If you have additional comments about this survey in general or any of the progressions specifically, please provide feedback below. ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Thank you for completing this survey!

Your responses will be aggregated with others to inform how the K-3 Formative Assessment Consortium sets performance expectations for children at the beginning of kindergarten. If you have any questions, please contact [email protected] or [email protected].




Exhibit D.2.5. Kindergarten entry performance levels: Sample domain expert questionnaire

Instructions – [CONSTRUCT NAME] Please review the progression and construct charts in this packet, one progression at a time,

and respond to the questions. We ask that you provide a rationale for your responses that

includes your expert opinion as well as any references you used to formulate your response.

Learning/Construct Progression

To refresh your memory, we have included a copy of the construct progression in the email you

received with this packet. It describes the background and rationale for the progression as well

as the overarching understandings, skills in the progression, performance descriptors and

examples. We recommend you review the progression before you respond to the questions

below.

Charts

The first chart (Teacher Survey) illustrates where teachers would expect K children to fall on the

progression given their expectations in the first week of school. There are two categories/bars

on this chart: the blue bars represent the skill level of an average or typical incoming K child in

the first week of school, a child who is meeting classroom expectations and therefore may not

need additional support from the teacher outside of regular interactions. The orange bars

represent the skill level of a child who may need additional supports in order to meet classroom

expectations at the beginning of kindergarten.

The second chart (Child Assessment) illustrates actual child placements on the progression

from the Fall 2016 field test. These data were collected in the first 8 to 10 weeks of kindergarten

and represent where teachers actually placed children on the progression. Note that these

placements were based on observations that typically occurred after the first week of school.

Performance Level Descriptors

The research team has defined two performance levels that may help teachers use the

assessment and report aggregate information for groups of students. Because the purpose of

the K-3 Formative Assessment is to inform instruction, the performance levels have been

defined related to instructional expectations for the class. Please reference these descriptions to

inform your responses to the questions below.

Developing – child does not yet exhibit the knowledge and skills expected of children at

entry to kindergarten and may need additional support or assistance to fully participate in

kindergarten curriculum activities and instruction.


Demonstrating – child exhibits the knowledge and skills expected of children at entry to

kindergarten to fully participate in kindergarten curriculum activities and instruction.

Mathematical Patterns Teacher Survey Summary

When K teachers were asked “Indicate the skill level for a typical or average student in the first

week of kindergarten,” the majority of teachers (65%) responded skill level A as typical for K

entry. (blue bars)

When teachers were asked “indicate the skill level at which you would consider an incoming

kindergarten student far below typical”, 86% of responses for this construct were below skill

level A. (green bars)

Child Assessment Field Test Data

During the first 8 – 10 weeks of Kindergarten, teachers varied in their placement of children on

the progression with a majority of children (59%) landing between skills C and E.

4%

65%

7%13%

6% 4%

86%

13%0% 0% 0% 0%

0%

20%

40%

60%

80%

100%

BelowA

A B C D E F G H I J K L M

Mathematical Patterns - Teacher Survey

Typical Far Below

3%

9%

3%

16%

26%

17%

9%5%

3%0% 0% 0% 0% 0%

7%

0%

5%

10%

15%

20%

25%

30%

Emerging A B C D E F G H I J K L M Not assessed

Mathematical Patterns - Child Assessment


Your Activity Based on your knowledge of child development coupled with the data above, please respond to

the following questions for this construct. Provide a rationale with references where necessary

to inform your response. You may respond directly within this document.

1. In your opinion, which levels on the progression would you place into the “Demonstrating”

performance level?

Demonstrating – child exhibits the knowledge and skills expected of children at entry to

kindergarten to fully participate in kindergarten curriculum activities and instruction.

Please provide your rationale.

2. In your opinion, which levels on the progression would you place into the “Developing”

performance level?

Developing – child does not yet exhibit the knowledge and skills expected of children at

entry to kindergarten and may need additional support or assistance to fully participate in

kindergarten curriculum activities and instruction.

Please provide your rationale.

3. In your opinion are there levels on the progression where the performance levels overlap?

a. If so, which ones?

b. Why?


Exhibit D.2.6. Final K entry performance levels compared to teacher survey responses and K fall assessment data

3%

21%

50%

26%

51%

37%

10%1%1% 3%

11%

86%

0%

20%

40%

60%

80%

100%

Below A A B C D

Book Orientation



1%12%

64%

23%

46%53%

1%5%

20%

75%

0%

20%

40%

60%

80%

100%

Below A A B C

Crossing Midline

Teacher Survey:Typical Teacher survey:Far Below Child Assessment


1%

38%

20%25%

10%4% 1%

64%

33%

3%12%

16% 15%20% 24%

7% 7%

0%

20%

40%

60%

80%

100%

Below A A B C D E F G

Emotion Expression


51%

35%

14%

78%

22%

2%15% 18%

28% 26%

10%

0%

20%

40%

60%

80%

100%

Below A A B C D E



26%16%

38%

16%

1% 3%

59%

33%

6% 1%1% 5% 9%18%

37%

12%6% 6% 5% 1%

0%

20%

40%

60%

80%

100%

Below A A B C D E F G H I

Emotional Literacy



17%

33% 32%

14%

1% 1%

49% 46%

4%1% 4% 5%11%

16% 18%

45%

0%

20%

40%

60%

80%

100%

Below A A B C D E F



28%

61%

10%1%

57%

43%

5%

22%30%

42%

0%

20%

40%

60%

80%

100%

Below A A B C D

Grip & Manipulation



4% 7% 10%22% 19% 20%

12%4% 1%

19%

48%

17%

4% 7%1% 3%1% 6% 3%

14% 16% 13%

46%

0%

20%

40%

60%

80%

100%

Below A A B C D E F G H I

Gross Motor


10%

70%

20%

43%

57%

2%

23%

75%

0%

20%

40%

60%

80%

100%

Below A A B C

Hand Dominance


7%

20%

3%

26%

40%

3% 1%

49%

36%

10%1% 3%1% 1% 1%

9%

28% 29% 31%

0%

20%

40%

60%

80%

100%


Letter Naming



4%

65%

7%13%

6% 4%

87%

13%4%

9%3%

18%28%

18%10% 6% 3%

0%

20%

40%

60%

80%

100%

BelowA

A B C D E F G H I J K L M



25%

41%

13%4% 6% 9%

1% 1%

68%

30%

1%3% 4% 6% 4% 5%

29%21%

10% 6%12%

0%

20%

40%

60%

80%

100%

Below A A B C D E F G H I J

Object Counting



1%

33% 35%

17%12%

1%

66%

31%

3%1%7%

13% 10%

24%

11% 14% 12% 9%

0%

20%

40%

60%

80%

100%

Below A A B C D E F G H

Perseverance

Typical Far Below Child Assessment

23%

40%

20%14%

1% 1%

67%

29%

4%4%10%

5%12%

16%11%

41%

0%

20%

40%

60%

80%

100%


Print Awareness


1%

35%42%

16%6%

68%

31%

1%1%6%

14%20%

28%

11%3%

10%3% 2% 1%

0%

20%

40%

60%

80%

100%

Below A A B C D E F G H I J

Problem Solving



4%

19%

46%

29%

1%

75%

25%

2%8%

16%26%

11% 8% 9% 12% 8%

0%

20%

40%

60%

80%

100%




1%

45% 48%

4% 1%

82%

18%

2%13%

29% 34%

11% 8%2% 2% 1%

0%

20%

40%

60%

80%

100%


Vocabulary


14%

78%

6% 1%

69%

28%

3%1% 3%

27%

51%

13%4%

0%

20%

40%

60%

80%

100%


Writing



Exhibit D.2.7. Percentage of kindergarten students developing and demonstrating expected skill levels in fall based on expert recommended performance levelsa

K Children

Assessed in Fall K Entry Performance Levels

Book Orientation 837 3 97

Crossing Midline 829 6 94

Emotion Expression 805 12 88

Emotion Regulation Strategies 787 2 98

Emotional Literacy 803 6 94

Following Directions 827 55 45

Grip & Manipulation 833 5 95

Gross Motor 832 8 92

Hand Dominance 835 2 98

Letter Naming 835 3 97

Mathematical Patterns 797 34 66

Object Counting 826 17 83

Perseverance 795 1 99

Print Awareness 843 4 96

Problem Solving 797 42 58

Reading Comprehension 784 10 90

Vocabulary 796 2 98

Writing 826 4 96 aPerformance levels for K entry were set as of the first week in school. Children were actually assessed any time during the first 12 weeks of school in the fall. Actual performance should in most cases exceed expected performance levels.


Exhibit D.2.8. Mean progression placements for children assessed in fall and winter

Children

Assessed Fall Winter Difference Standard Deviation Probability

Book Orientation (A-D) Grade K 763 4.8 5.0 0.17 0.53 .0001

Hand Dominance (A-C) Grade K 777 3.7 3.9 0.18 0.47 .0001

Crossing Midline (A-C) Grade K 773 3.7 3.8 0.17 0.55 .0001 Grade 1 182 3.7 3.8 0.13 0.52 .001

Grip & Manipulation (A-D)

Grade K 776 4.1 4.8 0.64 0.83 .0001 Grade 1 193 4.5 4.7 0.21 0.80 .001

Letter Naming (A-G) Grade K 782 6.7 7.6 0.87 1.04 .0001 Grade 1 192 7.6 7.8 0.23 0.51 .0001

Print Awareness (A-G) Grade K 780 6.3 7.7 1.37 1.71 .0001 Grade 1 181 7.9 8.0 0.06 0.25 .005

Object Counting (A-J) Grade K 773 7.4 9.4 1.95 1.95 .0001 Grade 1 182 9.7 10.3 0.64 1.22 .0001 Grade 2 115 10.6 10.9 0.23 0.59 .0001

Emotion Expression (A-G)

Grade K 744 4.7 5.8 1.08 1.58 .0001 Grade 1 193 5.4 6.5 1.10 1.56 .0001 Grade 2 115 6.4 6.7 0.29 1.51 .05 Grade 3 134 6.6 7.3 0.64 1.03 .0001


Grade K 735 3.9 4.5 0.61 1.22 .0001 Grade 1 194 4.5 5.1 0.60 1.22 .0001 Grade 2 115 4.7 5.0 0.24 1.10 .05 Grade 3 124 5.2 5.3 0.12 0.95 NS

Emotional Literacy (A-I) Grade K 747 5.2 6.5 1.30 1.98 .0001 Grade 1 162 6.2 7.6 1.38 1.58 .0001 Grade 2 115 7.8 8.8 0.87 1.59 .0001 Grade 3 124 8.1 9.2 1.08 1.37 .0001


Children

Assessed Fall Winter Difference Standard Deviation Probability

Following Directions (A-F)


Gross Motor (A-I) Grade K 779 8.6 9.3 0.70 1.33 .0001 Grade 1 175 8.8 9.5 0.59 1.29 .0001 Grade 2 115 9.7 9.9 0.22 0.69 .001 Grade 3 123 9.7 9.9 0.20 0.83 .01

Mathematical Patterns (A-M)


Perseverance (A-H) Grade K 742 5.5 6.5 1.00 1.97 .0001 Grade 1 192 5.9 7.2 1.30 1.59 .0001 Grade 2 115 7.4 7.7 0.24 1.42 NS Grade 3 135 6.8 7.7 0.88 1.71 .0001

Problem Solving (A-J) Grade K 729 5.0 6.4 1.32 2.14 .0001 Grade 1 180 6.1 7.7 1.57 1.87 .0001 Grade 2 115 7.9 8.9 1.18 1.58 .0001 Grade 3 127 7.4 8.9 1.49 1.52 .0001



Vocabulary (A-H) Grade K 743 3.8 5.2 1.46 2.01 .0001 Grade 1 183 4.9 5.9 1.02 1.70 .0001 Grade 2 115 6.4 7.5 1.11 1.28 .0001 Grade 3 135 6.2 6.9 0.66 1.36 .0001

Writing (A-G) Grade K 770 3.9 5.1 1.21 1.22 .0001 Grade 1 193 5.3 6.0 0.69 0.95 .0001 Grade 2 115 6.0 6.6 0.53 1.02 .0001 Grade 3 134 6.5 6.9 0.37 0.97 .0001


Exhibit D.2.9. Mean progression placements for children assessed in fall by grade level

Constructs Children

Assessed Numeric Average Difference Probability

Crossing Midline (A-C)

Grade K 829 3.7 Grade 1 184 3.7 +0.03 NS

Grip & Manipulation (A-D) Grade K 833 4.1 Grade 1 197 4.5 +0.37 .0001

Letter Naming (A-G) Grade K 835 6.7 Grade 1 196 7.6 +0.86 .0001

Print Awareness (A-G) Grade K 843 6.3 Grade 1 185 7.9 +1.65 .0001

Object Counting (A-J) Grade K 826 7.4 Grade 1 185 9.7 +2.28 .0001 Grade 2 126 10.6 +0.90 .0001

Emotion Expression (A-G) Grade K 805 4.7 Grade 1 195 5.4 +0.70 .0001 Grade 2 126 6.4 +0.99 .0001 Grade 3 138 6.6 +0.22 NS


Grade K 787 3.9 Grade 1 197 4.5 +0.55 .0001 Grade 2 126 4.7 +0.24 NS Grade 3 127 5.2 +0.49 .005

Emotional Literacy (A-I)

Grade K 803 5.2

Grade 1 164 6.2 +1.06 .0001 Grade 2 126 7.8 +1.60 .0001 Grade 3 126 8.1 +0.24 NS

Following Directions (A-F) Grade K 827 5.7 Grade 1 194 6 +0.29 .05 Grade 2 126 6.3 +0.28 NS Grade 3 138 5.8 -0.45 .01


Constructs Children


Gross Motor (A-I)

Grade K 832 8.6 Grade 1 185 8.8 +0.20 NS Grade 2 126 9.7 +0.89 .0001 Grade 3 138 9.7 +0.03 NS

Mathematical Patterns (A-M) Grade K 797 5.0 Grade 1 184 7.3 +2.33 .0001 Grade 2 126 9.0 +1.61 .0001 Grade 3 127 9.0 +0.06 NS

Perseverance (A-H) Grade K 795 5.5 Grade 1 194 5.9 +0.41 .05 Grade 2 126 7.4 +1.54 .0001 Grade 3 138 6.8 -0.61 .01

Problem Solving (A-J) Grade K 797 5 Grade 1 181 6.1 +1.10 .0001 Grade 2 126 7.9 +1.75 .0001 Grade 3 138 7.4 -0.50 .05

Reading Comprehension (A-H) Grade K 784 5.0 Grade 1 196 7.3 +2.25 .0001 Grade 2 126 7.4 +0.13 NS Grade 3 138 7.6 +0.23 NS

Vocabulary (A-H) Grade K 796 3.8 Grade 1 184 4.9 +1.07 .0001 Grade 2 126 6.4 +1.54 .0001 Grade 3 137 6.2 -0.19 NS

Writing (A-G) Grade K 826 3.9 Grade 1 195 5.3 +1.48 .0001 Grade 2 126 6.0 +0.63 .0001 Grade 3 138 6.5 +0.49 .001


Exhibit D.2.10. Mean progression placements for children assessed in winter by grade level

Constructs Children



Grade K 782 3.8 Grade 1 193 3.8 0.00 NS

Grip & Manipulation (A-D) Grade K 784 4.8 Grade 1 193 4.7 -0.07 NS

Letter Naming (A-G) Grade K 783 7.6 Grade 1 192 7.8 +0.21 .001

Print Awareness (A-G) Grade K 782 7.7 Grade 1 181 8.0 +0.30 .0001

Object Counting (A-J) Grade K 782 9.4 Grade 1 193 10.3 +0.90 .0001 Grade 2 115 10.9 +0.54 .0001

Emotion Expression (A-G) Grade K 763 5.8 Grade 1 193 6.5 +0.69 .0001 Grade 2 115 6.7 +0.21 NS Grade 3 135 7.3 +0.53 .005


Grade K 773 4.5 Grade 1 194 5.1 +0.56 .0001 Grade 2 115 5.0 -0.09 NS Grade 3 135 5.3 +0.30 .05

Emotional Literacy (A-I) Grade K 768 6.5 Grade 1 162 7.6 +1.13 .0001 Grade 2 115 8.8 +1.18 .0001 Grade 3 135 9.2 +0.39 .05

Following Directions (A-F) Grade K 782 6.3 Grade 1 193 6.2 -0.16 NS Grade 2 115 6.5 +0.33 .05 Grade 3 135 6.6 +0.07 NS


Constructs Children


Gross Motor (A-I)

Grade K 782 9.3 Grade 1 186 9.5 +0.21 .05 Grade 2 115 9.9 +0.41 .0001 Grade 3 123 9.9 +0.03 NS

Mathematical Patterns (A-M) Grade K 770 6.9 Grade 1 193 8.7 +1.78 .0001 Grade 2 115 9.6 +0.83 .005 Grade 3 124 10.4 +0.85 .05

Perseverance (A-H) Grade K 770 6.5 Grade 1 193 7.2 +0.71 .0001 Grade 2 115 7.7 +0.48 .001 Grade 3 135 7.7 +0.06 NS

Problem Solving (A-J) Grade K 766 6.4 Grade 1 193 7.7 +1.30 .0001 Grade 2 115 8.9 +1.20 .0001 Grade 3 127 8.9 -0.04 NS

Reading Comprehension (A-H) Grade K 781 6.7 Grade 1 193 7.6 +0.89 .0001 Grade 2 115 8.1 +0.49 .005 Grade 3 134 8.3 +0.18 NS

Vocabulary (A-H) Grade K 772 5.2 Grade 1 184 5.9 +0.69 .0001 Grade 2 115 7.5 +1.59 .0001 Grade 3 135 6.9 -0.62 .001

Writing (A-G) Grade K 780 5.1 Grade 1 193 6.0 +0.96 .0001 Grade 2 115 6.6 +0.61 .0001 Grade 3 134 6.9 +0.22 NS


Exhibit D.2.11. Percent of kindergarten children at higher, same, or lower levels on progression from fall to winter administrations

We looked at those children who were assessed on each construct in the fall and again in the winter. Ns ranged from 729 to 782 depending on the construct. This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green) and the children who were placed lower on the progression in the winter (teal).

70

68

73

65

55

56

73

76

57

19

40

49

38

56

64

62

19

13

5

41

7

11

31

74

42

42

42

6

6

71

86

26

23

12

21

2

23

12

19

10

4

12

6

11

22

23

23

6

4

9

11

14

2

14

4

5

2

2

6

3

9

16

12

9

4

0 10 20 30 40 50 60 70 80 90 100

Writing (N=770)

Vocabulary (N=743)

Reading Comprehension (N=731)

Problem Solving (N=729)

Print Awareness (N=780)

Perseverance (N=742)

Object Counting (N=773)

Math Patterns (N=733)

Letter Naming (N=782)

Hand Dominance N=777)

Gross Motor (N=779)

Grip & Manipulation (N=776)

Following Directions (N=773)

Emotion Regulation (N=735)

Emotional Literacy (N=747)

Emotion Expression (N=744)

Crossing Midline (N=773)

Book Orientation (N=763)

Fall to Winter Progression Placements: Kindergarten



Exhibit D.2.12. Percent of grade 1 children at higher, same, or lower levels on progression from fall to winter administrations

We looked at those children who were assessed on each construct in the fall and again in the winter. Ns ranged from 162 to 193 depending on the construct. This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green) and the children who were placed lower on the progression in the winter (teal). Note that Book Orientation and Hand Dominance were not assessed in Grade 1.

60

55

37

76

5

65

43

74

24

44

24

25

52

69

59

16

4

21

94

12

43

65

43

59

45

19

2

16

71

30

25

24

11

15

8

17

8

3

8

10

15

20

17

7

10

16

17

13

9

7

9

3

10

9

19

14

9

9

5

0 10 20 30 40 50 60 70 80 90 100

Writing (N=193)

Vocabulary (N=183)



Print Awareness (N=181)




Letter Naming (N=192)

Gross Motor (N=175)

Grip & Manipulation (N=193)





Crossing Midline (N=182)

Fall to Winter Progression Placements: Grade 1




We looked at those children who were assessed on each construct in the fall and again in the winter. 115 2nd graders were assessed on the 12 constructs listed. This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green), and the children who were placed lower on the progression in the winter (teal).

42

60

52

59

46

17

43

12

20

32

49

33

2

10

30

10

7

77

83

61

33

23

26

54

26

10

21

25

3

47

4

11

20

17

23

3

3

9

10

22

2

10

8

15

11

18

0 10 20 30 40 50 60 70 80 90 100

Writing (N=115)

Vocabulary (N=115)






Gross Motor (N=115)









We looked at those children who were assessed on each construct in the fall and again in the winter. Third graders were assessed on the 11 constructs listed; Ns ranged from 123 to 135.This chart shows the % of children who had a higher placement on the progression in the winter (blue), those that were already at the maximum skill level in the fall (green), those who were at the same level in the winter (light green), and the children who were placed lower on the progression in the winter (teal).

43

50

41

81

63

65

12

43

27

61

52

8

33

13

8

84

44

42

21

25

34

40

19

10

9

23

3

8

15

14

15

15

10

7

8

15

3

4

16

4

7

0 10 20 30 40 50 60 70 80 90 100

Writing (N=134)

Vocabulary (N=134)





Gross Motor (N=123)








Exhibit D.2.15 Progression placements for kindergarten children with and without IEPs

Construct and IEP Status Fall Winter N Mean SD p < N Mean SD p < Book Orientation (A-D)

IEP 67 4.6 0.82 .05

65 5.0 0.17 NS

No IEP 770 4.8 0.51 708 5.0 0.19


IEP 68 3.5 0.68 .05

65 3.7 0.55 .05

No IEP 761 3.7 0.58 717 3.9 0.42

Emotion Expression (A-G)

IEP 65 3.9 1.62 .0001

63 4.7 1.92 .0001

No IEP 740 4.8 1.71 700 5.9 1.74 Emotion Regulation Strategies (A-E)

IEP 66 3.2 1.33 .0001

64 3.9 1.18 .0001

No IEP 721 4.0 1.26 709 4.6 1.27

Emotional Literacy (A-I)

IEP 65 4.5 1.61 .01

64 5.2 1.80 .0001

No IEP 738 5.2 1.76 704 6.6 2.10

Following Directions (A-F)

IEP 67 4.8 1.69 .0001

65 5.4 1.50 .0001

No IEP 760 5.8 1.45 717 6.4 1.06

Grip & Manipulation (A-D)

IEP 67 3.6 0.95 .0001

65 4.3 0.85 .0001

No IEP 766 4.1 0.91 719 4.8 0.53

Gross Motor (A-I)

IEP 67 7.8 1.79 .001

65 8.4 1.63 .0001

No IEP 765 8.6 1.71 717 9.3 1.28

Hand Dominance (A-C)

IEP 68 3.7 0.50 NS

65 3.8 0.44 NS

No IEP 767 3.7 0.47 718 3.9 0.26

Letter Naming (A-G)

IEP 67 6.0 1.63 .0001

66 7.0 1.05 .0001

No IEP 768 6.8 1.17 717 7.7 0.69 Mathematical Patterns (A-M)

IEP 65 4.6 1.95 NS

65 6.1 1.94 .0001

No IEP 732 5.1 1.83 705 7.0 1.83

Object Counting (A-J)

IEP 67 6.4 2.14 .0001

66 8.1 2.40 .0001

No IEP 759 7.5 2.23 716 9.5 1.75

Perseverance (A-H)

IEP 65 4.3 2.29 .0001 64 4.8 2.12 .0001


Construct and IEP Status Fall Winter N Mean SD p < N Mean SD p <

No IEP 730 5.6 2.04 707 6.6 2.02

Print Awareness (A-G) IEP 67 5.2 2.01

.0001 66 7.0 1.48

.001 No IEP 776 6.4 1.83 716 7.7 0.79

Problem Solving (A-J) IEP 65 4.0 1.67

.0001 65 4.9 2.33

.0001 No IEP 732 5.1 1.98 701 6.6 2.39


IEP 66 4.3 2.08 .01

65 5.6 2.21 .0001

No IEP 718 5.1 2.17 716 6.8 1.89

Vocabulary (A-H) IEP 65 3.4 1.40

.05 64 4.2 1.95

.0001 No IEP 731 3.8 1.36 708 5.3 1.96

Writing (A-G) IEP 67 3.3 0.90

.0001 65 4.1 1.25

.0001 No IEP 759 3.9 0.85 715 5.1 1.24


Exhibit D.2.16. Progression placements for kindergarten ELLs and non-ELLs

Construct and ELL Status Fall Winter N Mean SD p < N Mean SD p <

Book Orientation (A-D) ELL 72 4.9 0.26

.05 67 5.0 0.12

NS Non-ELL 765 4.8 0.56 706 5.0 0.19

Crossing Midline (A-C) ELL 72 3.8 0.45

NS 66 4.0 0.21

.05 Non-ELL 757 3.7 0.60 716 3.8 0.02

Emotion Expression (A-G) ELL 72 4.9 1.68

NS 67 6.2 1.25

NS Non-ELL 733 4.7 1.72 696 5.8 1.82

Emotion Regulation (A-E) ELL 72 4.3 1.09

.01 67 5.1 0.88

.001 Non-ELL 715 3.9 1.29 706 4.5 1.30

Emotional Literacy (A-I) ELL 72 5.7 1.88

.05 66 6.9 1.98

NS Non-ELL 731 5.1 1.74 702 6.4 2.12

Following Directions (A-F) ELL 72 5.9 1.35

NS 67 6.4 1.00

NS Non-ELL 755 5.7 1.51 715 6.3 1.15

Grip & Manipulation (A-D) ELL 72 3.6 0.95

NS 67 4.8 0.45

NS Non-ELL 761 4.3 0.87 717 4.8 0.58

Gross Motor (A-I) ELL 73 8.8 1.30

NS 66 9.4 1.05

NS Non-ELL 759 8.5 1.77 716 9.2 1.36

Hand Dominance (A-C) ELL 73 3.7 0.50

NS 67 4.0 0

.05 Non-ELL 762 3.7 0.47 716 3.9 0.29

Letter Naming (A-G) ELL 72 6.5 1.30

NS 67 7.7 0.67

NS Non-ELL 763 6.7 1.23 716 7.6 0.76

Math Patterns (A-M) ELL 71 4.9 1.93

NS 66 7.0 1.87

NS Non-ELL 726 5.0 1.83 704 6.9 1.66

Object Counting (A-J) ELL 72 7.0 2.26

NS 66 9.5 1.73

NS Non-ELL 754 7.5 1.98 716 9.4 1.86

Perseverance (A-H) ELL 72 5.0 1.97

NS 66 6.5 1.50

NS Non-ELL 723 5.5 2.10 705 6.5 2.14


Construct and ELL Status Fall Winter N Mean SD p < N Mean SD p <

Print Awareness (A-G) ELL 72 5.5 2.15

.01 66 7.7 0.70

NS Non-ELL 771 6.4 1.83 716 7.7 0.90

Problem Solving (A-J) ELL 72 4.9 2.43

NS 66 6.2 1.94

NS Non-ELL 725 5.0 1.93 700 6.4 2.47

Reading Comp (A-H) ELL 71 4.9 2.00

NS 66 6.6 1.79

NS Non-ELL 713 5.0 2.19 715 6.7 1.97

Vocabulary (A-H) ELL 72 3.7 1.09

NS 66 5.2 1.46

NS Non-ELL 724 3.8 1.40 706 5.2 2.02

Writing (A-G) ELL 71 4.0 1.06

NS 67 5.3 1.07

NS Non-ELL 755 3.8 0.85 713 5.0 1.29


Exhibit D.2.17. Gender differences* in construct performance by grade and test administration

Construct Kindergarten Grade 1 Grade 2 Grade 3 Fall Winter Fall Winter Fall Winter Fall Winter

Ns (850) (785) (198) (194) (126) (115) (138) (135) Book Orientation Girls NS na na na na na na

Crossing Midline NS NS NS NS na na na na

Emotion Expression Girls Girls NS NS NS NS Girls NS

Emotion Regulation Strategies Girls NS NS NS Girls Girls Girls Girls

Emotional Literacy Girls NS NS NS NS NS NS NS

Following Directions Girls NS NS NS NS NS Girls Girls

Grip & Manipulation Girls Girls NS NS na na na na

Gross Motor Girls Girls NS NS NS NS NS NS

Hand Dominance NS NS na na na na na na

Letter Naming Girls Girls NS NS na na na na

Mathematical Patterns NS NS NS NS NS NS NS NS

Object Counting NS NS Boys NS NS NS na na

Perseverance Girls Girls NS NS NS NS Girls NS

Print Awareness Girls NS NS NS na na na na

Problem Solving NS NS NS NS NS NS NS NS

Reading Comprehension NS Girls NS NS NS NS NS NS

Vocabulary Girls NS NS NS NS NS Girls NS

Writing Girls Girls NS NS NS NS Girls Girls

*Gender listed was placed higher on the progression. Mean differences were statistically significant atthe p < .05 level or lower. NS = non-significant or no differences. na = construct not assessed.


Exhibit D.2.18. Racial/ethnic differences* in construct performance by grade and test administration

Construct Kindergarten Grade 1 Grade 2 Grade 3 Fall Winter Fall Winter Fall Winter Fall Winter

Ns (739) (680) (164) (160) (125) (114) (105) (105)

Book Orientation NS NS na na na na na na

Crossing Midline NS NS Non-White NS na na na na

Emotion Expression NS NS NS NS NS Non-White NS NS

Emotion Regulation NS NS NS NS NS Non-White NS Non-

White

Emotional Literacy NS NS NS NS NS Non-White NS NS

Following Directions White NS NS NS Non-White

Non-White NS NS

Grip & Manipulation NS NS NS Non-White na na na n

Gross Motor NS NS Non-White NS NS NS NS NS

Hand Dominance White NS na na na na na na

Letter Naming NS NS White White na na na na

Math Patterns NS NS NS NS NS NS NS NS

Object Counting White NS NS NS White NS na na

Perseverance White NS NS NS NS NS NS NS

Print Awareness White NS NS NS na na na na

Problem Solving White White White NS NS NS NS Non-White

Reading Comp White NS White NS White NS NS NS

Vocabulary White NS NS NS NS NS NS NS

Writing White NS NS Non-white NS NS NS NS

*Race/ethnicity group listed (White or Non-white) was placed significantly higher on the progression.Mean differences were statistically significant at the p < .05 level or lower. NS = non-significant or no differences. na = construct not assessed


Exhibit D.3.1. Coding protocol for evidence analysis

Evidence Analysis Protocol 1. This protocol should be completed once agreement has been reached between the primary

and secondary coder for each construct.

2. Enter the construct abbreviation for the constructs to be coded in the designated coding

worksheet.

3. Copy and paste the child ID and state

from the sample spreadsheet provided

into the appropriate columns.

4. Locate all available evidence for the

assigned construct and child in the file

exported from the technology platform.

5. Code the data based on the rules listed

below.

6. For assigned students, record if any of

the pieces of evidence for a particular child and construct are a video or photo.

Note. If the piece of evidence is a photo or a video, go to the field test data on the K-3

technology platform to view that particular record.

7. Determine whether the Evidence for each student’s Status Summary is specific enough to

endorse a Status Summary (can where the child is on the progression be determined based

on just this piece of evidence?). This coding is conducted at three levels: (a) construct

relevance, (b) skill specificity, and (c) child specificity.

8. Coding Rules

When examining the evidence, the following rules apply to ensure consistency.

a. Each piece of evidence should be analyzed for the assigned construct only, regardless

of its value for use in determining Status Summary for other constructs.

b. Each piece of evidence should be taken at face value. Do not infer any conditions to the

teacher comments. This includes any inferences about the activity or what may have

occurred prior to or after the activity listed as evidence.

Constructs Language/

Communication Approaches to Learning BO Pe PA Motor LN HD RC CM Wr G&M Vo GM FD Cognitive

Social-Emotional OC EL PS EE MP

ERS


Coding descriptions and guiding questions for coding each piece of evidence

Levels Description Code Construct Relevant

When reviewing evidence for a child, are there one or more pieces of evidence that are relevant to the construct?

The evidence may not be specific to a child or skill, but may provide a description of activities or group skills the teacher observed that are related to the construct.

For example, “Asked class to walk over to cubbies, pick up crayons and notebooks and bring them back to their assigned seats” (Following Directions) or “children wrote stories about their vacation” (Writing). The evidence does not need to include child information to be construct relevant.

Number of pieces of evidence that are relevant to the construct

Skill-Specific When looking at all evidence for an assigned child, does the evidence describe key skills or behaviors found in performance descriptors from the construct progression?

The evidence may not be specific to one child, but may describe skills for a group. These skills are aligned to key words in the progression or other construct materials OR are examples of skills highlighted in the progression.

For example, “While solving addition problems after being taught a strategy to do so, students used ten frame and counters to solve. When faced with something difficult in the activity they looked at a sample problem.” (Problem Solving). Or “during read aloud, two children recognized their reading did not make sense” (Reading Comprehension).

Number of pieces of evidence that are skill-specific

Child-Specific When looking at all evidence for an assigned child, does the evidence describe any key skills or performance descriptors that can be attributed to that child? The teacher may use the child’s name, a pronoun or singular verbs in her/his evidence.

The evidence may not be specific to a skill on the progression, but clearly identifies that the teacher is observing a single child.

For example, “child opened the front cover” or “Tina reflected on her choice to take the blocks from Gina…” or “Malik offered another solution to keep the rice from spilling over”

If a teacher has copied the same evidence for every child in the class and it is not clear that the skill was related to the child being coded, mark the evidence is not child specific. For example, “Today I placed the students in a situation with their group to see how they would react to the problem” does not provide enough detail to be considered child-specific.

If a teacher has copied the same evidence for every child in the class and it is clear that she intended that all children met the skill level, mark the evidence as child-specific. For example, “The class was to draw a picture following my three- step instructions. All children in the class were able to follow the directions without additional support.”

Number of pieces of evidence that are child specific


Levels Description Code Master Coder Status Summary

If the Evidence, when taken together, is specific enough to assign a Status Summary, then use the Construct Progression to endorse a Status Summary rating. If the evidence, taken together, is not specific enough to draw a conclusion about the Status Summary (e.g., there are not any pieces of evidence that are child-specific or skill-specific), code as 99. If unsure of the code to be given, though the evidence appears to have met some of the criteria above, code as 0.

Coding tips:

• Remember to code each child’s Status Summary based on all ofthe available evidence for that child.

• When looking at multiple pieces of information, it may be useful tolook at the evidence chronologically. This may give a morecomplete picture of how the teacher is seeing the child’s behaviorand skills.

• If a teacher has recorded classroom-wide activities or a summaryof how she/he observed for all her evidence, code this as 99. Theinformation is not specific enough about the child to determine thechild’s placement on the progression. Do not attempt to extrapolateinformation from the preliminary learning status for a child or groupof children.

Status Summary

99 = not specific enough

0 = unsure

9. If a Master Coder Status Summary was assigned, check what Status Summary the teacher

originally endorsed and record it in the coding spreadsheet.

10. Given the teacher evidence in total, does it seem reasonable that he/she may have

assigned this Status Summary?

Code responses:

1 – Yes, believe the teacher was reasonable in his/her assignment of a Status Summary made based on the evidence entered. This could be the case even when the reviewer’s Status Summary is not an exact match to the teacher’s Status Summary.

0 – No, believe the teacher assigned a Status Summary inconsistent with the evidence presented.


Exhibit D.3.2. Internal reliability coefficients for evidence analysis

Evidence analysis reliability coefficients As part of the evidence analysis and to ensure consistency among reviewers, a 10 percent

sample for each construct was coded by two reviewers prior to coding the full sample. Cohen

kappa estimates, shown below, ranged from .85 to 1.00, indicating almost perfect to perfect

agreement among reviewers. Reviewers discussed any disagreements to reach consensus and

clarify coding rules to be used with the remaining 90 percent of the sample.

Internal Reviewer Agreement Coefficients (Cohen κ)

Construct Cohen Kappa (Cohen κ) Book Orientation 1.00

Crossing Midline 0.94

Emotion Expression 1.00

Emotional Literacy 0.96

Emotional Regulation Strategies 0.96

Following Directions 1.00

Grip & Manipulation 0.89

Gross Motor 1.00

Hand Dominance 0.89

Letter Naming 1.00

Mathematical Patterns 0.89

Object Counting 0.94

Print Awareness 0.93

Perseverance 1.00

Problem Solving 0.85

Reading Comprehension 1.00

Vocabulary 1.00

Writing 1.00


Exhibit D.4.1. A profile of teachers participating in the field test (n = 145)

Characteristic N % Mean (SD)

Kindergarten 85 59%

1st grade 22 15%

2nd grade 16 11%

3rd grade 12 8%

Mixed or Multi-grade 9 6%

Other 1 1%

Has paid paraprofessionals or volunteers in the classroom 78 54%

Has computer in the classroom 137 95%

Has access to mobile device or tablet at school 126 87%

Has good internet access in classroom 111 77%

Has spotty internet access in classroom 34 24%

Received PD on formative assessment 73 50%

Already using formative assessment before this project 110 76%

Participated in K-3 Formative Assessment pilot 38 26%

Gender (Female) 142 98%

Education Level: Master’s degree or higher 60 41%

Years of teaching experience 14.5 (10.3)

American Indian/Alaskan native 0 0%

Asian 5 3.4%

Black/African American 0 0%

Native Hawaiian/Other Pacific Islander 1 0.7%

White 138 95.2%

Unknown/Missing 1 0.7%

Hispanic/Latino Origin 7 4.8%


Revised August 11, 2016

Exhibit D.5.1. Training guidance for administrators

Recommendations for Administrators Supporting Teachers

Help teachers create a positive classroom cultureo Promote a culture that encourages conversation, dialogue and listening.o Investigate “quiet” classrooms. Who is doing the talking/listening?o Model good questioning, probing and listening.

Support teachers’ efforts for using developmentally appropriate practiceso Ask teachers to think about the types of learning formats they use (e.g., whole group, small group,

centers/stations). Is there a balance? Do the learning formats they use provide opportunities forstudents to interact with each other and the materials in the classroom?

o Offer an opportunity for teachers to reflect on their current instructional practices and consider how theymight enhance what they’re doing to increase active participation in engaging learning experiences.

o Allow opportunities for teachers to try new practices.o Arrange for teachers to visit colleagues who successfully use various learning formats and instructional

strategies.

Support intentional planning and decision making by providing opportunities for teachers to work andlearn together

o Put it on the agenda (PLC Meetings, Staff Meetings) to talk through the evidences and the process.o Participate in on-site or online Professional Learning Communities (PLCs) that discuss observation-based

data collection and data trends.o Provide planning time for pilot teachers to meet with others (e.g., grade specific, cross-grade, support

staff).o Establish protocols for collaboration and identify expected outcomes.

Check-in with teacherso Establish a communication protocol - who, how, when.o Determine a focus and make time to visit classrooms, talk with teachers, and ask questions.o Use your own observation data and faculty input to determine staff development opportunities.o Capture positive happenings and celebrate the learning experiences.

Support technology effortso Provide strong wireless connection and access to a computer in the classroom.o Arrange access to a tablet/smartphone for collecting evidence.o Encourage teachers to use the App on their phone/tablet to collect evidences.o Supply contact information for tech support (local and Teaching Strategies).

Consider adjusting time, policies and/or personnel to provide additional supporto Consider schedule adjustments.o Reallocate time on agendas for current meetings to allow specific time for work and collaboration.o Consider removing other expectations.o Offer additional support in the classroom.o Think about how other staff could support this process (e.g., instructional coaches, ELL, EC).


Exhibit D.7.1. Construct performance for each validity claim

AtL

Validity Claims

Boo

k O

rient

atio

n

Prin

t Aw

aren

ess

Lette

r Nam

ing

Follo

win

g D

irect

ions

Writ

ing

Rea

ding

C

ompr

ehen

sion

Voc

abul

ary

Obj

ect C

ount

ing

Mat

h P

atte

rns

Pro

blem

Sol

ving

Han

d D

omin

ance

Grip

& M

anip

ulat

ion

Cro

ssin

g M

idlin

e

Gro

ss M

otor

Em

otio

n E

xpre

ssio

n

Em

otio

nal L

itera

cy

Em

otio

n R

egul

atio

n S

trate

gies

Per

seve

ranc

e

Claim 1.1 Proper sequence

% teachers reporting skills in right order 99 99 99 98 97 99 95 95 95 95 99 99 99 92 94 93 94 99

Frequency distribution (K only) - Smooth or Uneven S U S S S U S U U U S S S U U S S U

Skill IRT factor score means in order (K-3) - Yes or Possible problem (small Ns/skills too close) Y Y Y Y Y Y Y P P Y Y Y Y P Y Y Y Y

Claim 1.3 K entry performance levels

% teachers agreeing with expert on demonstrating level (at cutpoint or higher) 76 76 70 1 85 76 98 17 10 6 90 72 87 56 60 74 100 98

% children placed at or above demonstrating level after K entry (expected pattern) 97 96 96 45 96 90 98 83 66 58 98 95 94 92 88 94 98 99

Claim 1.2 Sensitive to growth

% teachers reporting construct captured growth well 98 96 93 96 92 90 84 92 86 88 99 97 96 91 84 85 84 91

Higher mean placements in winter than fall (Consistent, Inconsistent pattern) C C C C C C C C C C C C C C C C I I

Higher mean placements at higher grade level (K-1, 1-2, 2-3) (Consistent, Inconsistent pattern) na C C I C C I C C I na I I C C C I I

Claim 1.7 Useful for K-3 Instruction

% teachers reporting construct helpful to instruction (very, somewhat) 66 75 73 75 81 76 70 77 68 79 60 68 63 51 64 64 65 73

Claim 1.5 Teacher confidence

% teachers confident in accurately placing child on progression 99 97 95 98 95 93 83 96 88 89 97 92 93 94 74 75 71 89

Claim 1.4 Fidelity of implementation: Reliability of status summary

% children with evidence specific enough to code 78 74 78 80 84 56 69 78 79 55 84 84 74 66 66 45 40 33

% children status summary consistent with teacher evidence 82 89 88 86 95 87 71 66 75 79 90 90 100 96 81 64 86 85

% children teacher's status summary matches master coding 79 85 88 79 86 78 67 59 66 72 84 74 96 83 76 40 55 60

Claim 1.6 Incorporate assessment into classroom routines

% teachers reporting they could assess in daily classroom actvities 92 90 81 99 96 93 78 81 54 83 99 98 84 46 78 77 76 97

% teachers reporting construct easy or very easy to assess 98 97 99 86 93 84 66 95 82 60 98 94 89 95 24 31 22 54

ASSE

SSM

ENT

CONT

ENT

ASSE

SSM

ENT

PRO

CESS

Domains Language & Communication Cognitive Physical/Motor Social-Emotional

85-100

70-84 Good

1-69 Poor/Unsure

Very Good

Construct Performance


Appendix E National Experts Supporting Project Activities Exhibit E.1. National domain experts who supported different project activities

EAG K-3 Formative Assessment: Final Report September 2017 E-1

Exhibit E.1. National domain experts who supported different project activities

Expert Title & Affiliation Domain (Construct) Project Activities

Dr. Stephen Sanders Professor, Physical Education University of South Florida

Physical/Motor Development X X

Dr. Jill Sells Developmental Consultant Seattle, WA

Physical/Motor Development X

Dr. Laura J. Colker Child Development Consultant Washington, DC Cognitive Development X

Dr. Darryl Greenfield Professor of Psychology & Pediatrics University of Miami

Cognitive Development (Science) X

Dr. Douglas Clements

Professor of Early Childhood Learning University of Denver

Cognitive Development (Mathematics) X

Dr. David Dickinson Professor, Dept. of Teaching & Learning, Peabody College Vanderbilt University

Language Development & Communication

X

Dr. Susan Neuman Professor of Early Childhood and Literacy Education New York University

Language Development & Communication

X X

Dr. C. Cybele Raver Vice Provost for Research and Faculty Affairs & Early Childhood Researcher New York University

Social-Emotional Development; Approaches to Learning

X X



Dr. Clancy Blair Professor of Cognitive Psychology New York University

Approaches to Learning; Social-Emotional Development (Problem Solving, Following Directions)

X X

X

Dr. Marilou Hyson Early Childhood Development & Educational Consultant Stockbridge, MA

Approaches to Learning Social-Emotional Development

X X

Dr. Mary Louise Hemmeter

Professor, Department of Special Education Vanderbilt University

Social-Emotional Development X X

Dr. Arthur Baroody

Professor Emeritus, Department of Curriculum & Instruction University of Illinois at Urbana-Champaign

Cognitive Development (Object Counting) X X

Dr. Howard Goldstein

Professor, Communication Sciences & Disorders University of South Florida

Language Development & Communication (Vocabulary)

X

Dr. Dorothy Strickland

Professor of Reading Rutgers, The State University of New Jersey

Language Development & Communication (Reading Comprehension, Writing)

X

Dr. Bethany Rittle-Johnson

Associate Professor, Psychology and Human Development Vanderbilt University

Cognitive Development (Math Patterns) X X

Dr. Lauren Little

Assistant Professor The University of Kansas Medical Center Occupational Therapy Education Department

Physical/Motor Development X X



Dr. Karen McFadden

Assistant Professor, Dept. of Early Childhood Education Brooklyn College, The City University of New York

Social-Emotional Development (Perseverance)

X X

Dr. Brian Kissel Associate Professor, Reading Strand University of North Carolina Charlotte College of Education

Language Development (Writing, Book Orientation, Print Awareness, Letter Naming)

X X

Dr. Jean Vintinner Clinical Assistant Professor University of North Carolina Charlotte College of Education

Language Development (Reading Comprehension, Vocabulary, Problem Solving)

X X

Dr. Emily Frye

Assistant Professor Dept. of Psychological & Brain Sciences College of Arts & Sciences Indiana University

Cognitive Development (Math Patterns) X
