effects of the strong kids curriculum on students at...

EFFECTS OF THE STRONG KIDS CURRICULUM ON STUDENTS AT RISK FOR

INTERNALIZING BEHAVIORS

by

Marenda H. Brown

A thesis submitted to the faculty of

Brigham Young University

In partial fulfillment of the requirements for the degree of

Master of Science

Department of Counseling Psychology and Special Education

Brigham Young University

December, 2006

BRIGHAM YOUNG UNIVERSITY

GRADUATE COMMITTEE APPROVAL

of a thesis submitted by

Marenda H. Brown

This thesis has been read by each member of the following graduate committee and by majority vote has been found to be satisfactory. _____________________________ _________________________________ Date Michelle Marchant, Chair _____________________________ _________________________________ Date Ellie Young _____________________________ _________________________________ Date Gordon Gibb _____________________________ _________________________________ Date Paul Caldarella

BRIGHAM YOUNG UNIVERSITY

As chair of the candidate’s graduate committee, I have read the thesis of Marenda H. Brown in its final form and have found that (1) its format, citations, and bibliographical style are consistent and acceptable and fulfill university and department style requirements; (2) its illustrative materials including figures, tables, and charts are in place; and (3) the final manuscript is satisfactory to the graduate committee and is ready for submission to the university library. __________________________ __________________________________ Date Michelle Marchant Chair, Graduate Committee Accepted for the Department __________________________________ Tina T. Dyches Graduate Coordinator Accepted for the College __________________________________ K. Richard Young Dean, McKay School of Education

ABSTRACT EFFECTS OF THE STRONG KIDS CURRICULUM ON STUDENTS AT RISK FOR

INTERNALIZING BEHAVIORS

Marenda H. Brown

Department of Counseling Psychology and Special Education

Master of Science

Many students are faced with obstacles that can impede their ability to learn.

When obstacles are observable, teachers, parents and schools can identify and provide

services to these students. Students with less observable obstacles, such as internalizing

behaviors, are more difficult to identify and provide services for. Previous studies suggest

that students with internalizing behavior problems are not easily identifiable but still

require support and interventions to be successful in society. This study examined the

effects of the Strong Kids curriculum, developed for enhancing social and emotional

skills, on students at risk for internalizing behavior problems. Students received

instruction in a small group setting. Pre-, post- and follow-up assessments measured

student’s internalizing behaviors and their knowledge of emotional and social skills. The

results of this study suggest that the Strong Kids curriculum may be effective in reducing

internalizing symptoms of students at risk for internalizing behaviors.

ACKNOWLEDGMENTS

As I reflect on the journey of the past years, it is difficult, in the matter of a few

paragraphs, to acknowledge all of those who have been instrumental in this process. My

thanks and appreciation go out to the Special Education department and specifically to

the faculty. The courses and support offered by the faculty helped me to learn how to

conduct research and increase my knowledge and ability as a special educator. I would

like to thank the members of my committee, Drs. Ellie Young, Gordon Gibb and Paul

Caldarella for their support and encouragement.

Additionally, I extend my appreciation to the BYU-PBS Initiative. I have learned

so much by being a research assistant and working with the knowledgeable staff. Their

expertise and assistance helped to ensure the successful implementation of this study. To

my committee chair, Dr. Michelle Marchant, I express my gratitude for her continuous

mentoring and encouragement. Thank you Dr. Marchant for your patience, understanding

and friendship.

Throughout this process, my family has been a great support and strength to me.

Specifically my father, Garth, and sister, Serena, offered insight and understanding of the

intricacies of a research study and thesis. My family is too large to mention everyone by

name but my two sisters, Trina and Liz, and my mother, Janice, were an invaluable

support to me in this endeavor. In addition to encouraging me, they gave of their time to

watch my daughter.

Finally, I never could have completed this thesis without the support of my

husband, Sam, and my daughter Adelyn. Addie is my motivation not only to be a good

mother but a good teacher and researcher as well. Sam is the steadiness in my life. His

unwavering love and faith in my abilities is the greatest support I could ever have. Thank

you to all who have taught, supported and influenced me during this process.

TABLE OF CONTENTS

CHAPTER Page

I. INTRODUCTION………………………………………………………… 1 II. REVIEW OF THE LITERATURE……………………………………….. 4

Identification of Students…………………………………………………. 4 School Setting as a Delivery Option……………………………………… 6 Current Models of Interventions in Schools……………………………… 7

Positive Behavior Support ……………………………………………….. 10 Universal Interventions……………………………………………. 10 Secondary Interventions ………………………………………….. 10 Tertiary Interventions……………………………………………… 11 Social and Emotional Skills ………………………………………………. 14 Purpose ……………………………………………………………………. 18 Research Question ………………………………………………………… 18 Social Validity Research Questions ………………………………………. 18

III. METHOD…………………………………………………………………. 19

Setting …………………………………………………………………….. 19 Participants………………………………………………………………… 20 Selection…………………………………………………………… 20 Screening…………………………………………………………... 21 BYU-PBS Initiative Staff and School Psychologists ……………... 23 Observers………………………………………………………….. 24 Materials…………………………………………………………………… 24 Dependent Variable and Measures………………………………………… 25 Independent Variable and Measures………………………………………. 27 Lesson One………………………………………………………… 28 Lesson Two and Three…………………………………………….. 28 Lesson Four……………………………………………………….. 28 Lesson Five………………………………………………………… 29 Lesson Six and Seven……………………………………………… 29 Lesson Eight……………………………………………………….. 29 Lesson Nine…………………………………………………………29 Lesson Ten…………………………………………………………. 29 Lesson Eleven……………………………………………………… 30 Lesson Twelve………………………………………………………30 Data Collection…………………………………………………………….. 30 Experimental Design and Conditions……………………………………… 31 Data Analysis ………………………………………………………………31

viii

Treatment Fidelity…………………………………………………………. 31 Social Validity………………………………………………………………32 IV. RESULTS………………………………………………………………….. 34

Descriptive Statistics and Mean Comparisons……………………………... 34 TRF Results…………………………………………………………34 ISSC and Knowledge Test Results………………………………… 40 SSBD Participants vs. Recommended Participants……………………….. 44 Social Validity Results…………………………………………………….. 47 Perception of Need………………………………………………… 47 Teacher Perception of Outcome…………………………………… 48 Student Perceptions of Outcome…………………………………… 49 Instructor Recommendations for Curriculum……………………….49 V. DISCUSSION……………………………………………………………… 51

Implications of Strong Kids for Schools……...…………………………….51 Limitations ………………………………………………………………… 55 Recommendations for Further Research…………………………………… 57 REFERENCES…………………………………………………………….. 60

APPENDIXES Appendix A: Informed Consent Forms……………………………………..67 Appendix B: SSBD Scores………………………………………………… 73 Appendix C: Strong Kids Unit Tests and Answer Key……………………. 74 Appendix D: Treatment Fidelity Checklists……………………………….. 81 Appendix E: Strong Kids Instructor Training Information………………… 83 Appendix F: Anticipatory Sets…………………………………………….. 87 Appendix G: Social Validity Questionnaires ………………..…………….. 89 Appendix H: Clinical, Borderline and Normal TRF Student Scores………. 93 Appendix I: Data Coding Key…………………………………………….. 96 Appendix J: Data Analysis………………………………………………… 97

ix

LIST OF TABLES

Table Title Page

1. Demographics of Student Participants…………………………….. 21 2. TRF Descriptive Statistics…………………………………………. 35 3. T-test Comparison of TRF Means…………………………………. 38 4. ISSC and Knowledge Test Descriptive Statistics………………….. 41 5. T-test Comparison of ISSC and Knowledge Test………………….. 41 6. Pre and Post Assessment Scores of Students Who

Moved Before Follow-up…………………………………………... 44 7. SSBD vs. Recommended Student Participant Demographics………45 8. Significant Changes Between SSBD and Recommended

Student Participants…………………………………………………46 9. Student Perceptions of Outcome…………………………………… 49

x

xi

LIST OF FIGURES

Figure Title Page

1. TRF Internalizing Pre-test, Post-test and Follow-up Means………. 36 2. TRF Externalizing Pre-test, Post-test and Follow-up Means……… 37 3. TRF Total Pre-test, Post-test and Follow-up Means………………. 37 4. TRF Internalizing Scores………………………………………….. 39 5. TRF Externalizing Scores…………………………………………. 39 6. TRF Total Problems Scores……………..…………………………. 40 7. ISSC Pre-test, Post-test and Follow-up Means…………………….. 42 8. Strong Kids Knowledge Test Pre-test, Post-test and

Follow-up Means……………………………………………………43

1

CHAPTER I

INTRODUCTION

The primary purpose of schools is to assist students in their learning endeavors.

Because of the interconnection between social-emotional and academic outcomes

(Walker, Ramsey, & Gresham, 2004), children who do not develop social and emotional

skills are likely to experience lower levels of peer acceptance, have lower levels of self-

esteem and self-confidence, and have fewer successful academic skills (Merrell &

Walters, 1998). Developing appropriate social and emotional skills is important for all

children and youth, especially those identified with or at-risk for emotional or behavior

disorders--the latter being often under-identified (Walker, et al., 2004; Merrell & Walters,

1998).

Approximately 20% of children suffer from a mental health disorder, the most

common being anxiety disorders, followed by disruptive and mood disorders (Power,

2003). Of those children, approximately half have diagnosable psychological disorders

and the other half have emotional and behavioral problems that are less severe and

therefore not classified as disorders (Dumas & Nilsen, 2003). Many children with mental

health disorders do not receive services. According to the U.S. Department of Health and

Human Services (1999), fewer than 50% of children with mental health disorders receive

treatment. The U.S. Department of Education found that during the 2000-2001 school

year, 11.5% of students were identified as having a disability, of which 8% were

identified with an emotional or behavior disorder. The total percentage of students

identified with emotional behavior disorders was approximately 0.9%. Based on

population surveys, 5 to 6% of school age students are eligible to receive special

2

education services under the classification of emotional disability. According to

Kauffman (2001) about 2% of students are classified as emotionally disturbed.

Consequently, many children that could benefit from emotional services do not receive

them.

Children who do not develop appropriate social and emotional skills fail to

develop these skills for two reasons—deficits in either acquisition or performance

(Elksnin & Elksnin, 2006). Acquisition deficits are common in students who never learn

a behavioral expectation and are therefore unable to meet that expectation. An example of

an acquisition deficit is, a student yelling out the teacher’s name in order to get their

attention, because he/she does not know how to appropriately acquire it. If a student has a

performance deficit, however, the student knows what the expectation is but chooses not

to comply. An example of a performance deficit is a student who knows how to

appropriately acquire the teacher’s attention but chooses rather to yell out the teacher’s

name. Educators recognize that the prevention of acquisition and performance deficits is

necessary to help children develop appropriate social and emotional skills. However,

various obstacles keep students at-risk for emotional or behavioral problems from

receiving preventive services (Kauffman, 1999).

Many students who are at-risk for or identified as having Emotional Behavioral

Disorders (EBD) do not receive services because only 2% of the 5-6% who have EBD

problems are accurately identified (Kauffman, 2001). Kauffmann (1999) suggests that if

schools are to meet the needs of students with emotional and behavioral disorders the

number of students identified with emotional and behavioral disorders must increase

considerably. One reason that many students may not be identified is because of a

3

concern that students will be stigmatized and labeled as having a disability. A second

reason for under-identification is that students who need emotional skills training are

often placed in special education classrooms because of their low academic achievement-

-thus receiving academic services but not needed emotional or behavioral services

(Elksnin & Elksnin, 2006). Walker, Nishioka, Zeller, Severson, and Feil (2000)

recommend universal screening as a way to identify more students in need of services for

emotional behavior problems. In order to successfully educate students, schools must be

prepared and equipped to both identify and provide services for children with emotional

and behavioral problems, so that these students are empowered in their efforts to learn

(Braden, DiMarino-Linnen, & Good, 2001).

4

CHAPTER II

REVIEW OF THE LITERATURE

Identification of Students

There are a variety of risk factors that threaten the emotional health of children.

Howard and Dryden (1999) note that the classic environmental risk factors have

remained constant for more than 50 years, and include poverty, parental unemployment,

parental criminality, large family size, and poor parenting techniques. Merrell (2001)

adds that strained family relationships, family conflict and poor communication skills

contribute to rates of depression among children. Poor attachment patterns between

children and parents are linked with anxiety problems, along with high rates of stress and

exposure to highly stressful events (Merrell, 2001). Thus, it is evident that perpetual

exposure to multiple risk factors, such as those mentioned above, contributes to

establishing and maintaining emotional and behavioral deficits in children.

Children who demonstrate emotional and behavioral problems can be categorized

as demonstrating either externalizing or internalizing problems. Children with

externalizing behavior problems have been the focus of most intervention research

because their problems are usually disruptive and visible, and therefore easy to identify

by teachers and parents, unlike internalizing behaviors (Merrell & Walters, 1998).

Externalizing behaviors may include: yelling, hitting, spitting, kicking, swearing, biting

and fighting (McConville & Cornell, 2003; Shechtman, 2000). Less research has been

done with children that exhibit internalizing problems.

Internalizing behavior problems are not easily observable because they are not

usually manifest as measurable behaviors (Achenbach & Rescorla, 2001; Gresham &

5

Kern, 2004). Internalizing disorders include depression, dysthymic disorder, specific and

simple phobias, post-traumatic stress disorder, school refusal, selective mutism,

separation anxiety, social phobia, and generalized anxiety disorder (Stahl & Clarizio,

1999). Merrell defines internalizing disorders as problems that usually stem from within

an individual and are maintained from and within the individual (Merrell, 2001).

According to Merrell, there are four types of internalizing disorders: depression, anxiety,

social withdrawal, and somatic or physical problems (Merrell). Because these behaviors

are not as observable as externalizing behavior problems, students with internalizing

problems do not receive adequate services because they are not identified as requiring

assistance (Merrell). The problems that emerge in such students are poor peer

relationships, peer rejection, low self esteem, and future employment troubles (Elksnin &

Elksnin, 2006).

Because of the nature of internalizing behaviors, identification of students at risk

for internalizing problems can pose unique challenges for schools. For example, few

instruments exist for identifying elementary students with internalizing behavior

problems (Merrell, Blade, Lund & Kempf, 2003). The Systematic Screening for Behavior

Disorders (SSBD) is one of the few. Walker, Cheney, Stage, and Blum (2005) compared

students identified by office referral data to those identified using the SSBD. Office

referral data identified only 12 students who were at risk for emotional behavior

disorders; whereas, using the SSBD, 11 additional students were identified. Furthermore,

the SSBD assisted the authors in identifying 12 students who were at risk for

internalizing behavior problems that were not identified via office referral data. These

results indicate that using school-wide screening methods, such as the SSBD, can aid in

6

identifying more students with both externalizing and/or internalizing behavior problems.

Better identification methods will assist schools in providing services to more students

who demonstrate the need.

School Setting as a Delivery Option

In order to meet the needs of children and youth with emotional and behavioral

deficits, it is recommended that effective interventions be made readily accessible

(Walker, Ramsey & Gresham, 2004). Elksnin and Elksnin (2006) explain that

historically, social and emotional instruction took place at home, passing from generation

to generation. Today, families typically move more frequently and do not have access to

their extended family social support system; therefore, fewer children receive social and

emotional skills instruction from family members as in times past. With the magnitude

and complexity of mental health needs in children and adolescents, parents struggle to

adequately address their children’s individual needs, especially when they are isolated

from extended family. Additional concerns that parents confront in meeting their

children’s needs include insurance issues, limited knowledge on how and where to access

resources, and fear of stigmatization (Merrell, 2001). If effective interventions are not

readily accessible, parents and students may struggle to address emotional and behavioral

deficits.

One setting where children can access emotional and social services is schools.

Schools provide an environment rich in opportunities for social and emotional

development (Miller, Brehm & Whitehouse, 1998). Although academics are a primary

focus in schools, classroom and non-classroom experiences are made up of social and

emotional interactions (Elksnin & Elksnin, 2006). Thus, the school setting is prime for

7

rendering services to students’ with emotional behavior problems. Roeser (2001)

suggests that children will be more responsive to receiving emotional and social services

in school, as compared to a clinic, because it is a familiar environment. Students with

emotional and behavioral problems require support in order to be successful academically

and socially. Schools are an ideal context to offer that support.

Current Models of Interventions in Schools

The school community offers an array of service and delivery options, as well as

personnel, who can provide support to students who are at-risk for emotional or

behavioral disorders. Most schools have access to school psychologists or counselors

who are specifically trained to work with such a population. Small group and individual

counseling sessions, family training, and parental support are commonly used by school

psychologists to intervene with students with behavior problems (Dumas & Nilsen,

2003).

Strategies that have been recognized as effective in addressing behavior problems

in schools include token economies, response cost, time out, precision requests, self-

monitoring and peer tutoring (Cook, Landrum, Tankersley & Kauffman, 2002). School

counselors and psychologists use these interventions to respond to students who display

behavior problems. Herman, Merrell, and Reinke (2004) suggest that the role of school

psychologists should be more preventative, rather than reactionary. One way that school

psychologists can be more proactive is by focusing on preventing internalizing behavior

problems. Researchers have studied the effectiveness of various programs in addressing

the social and emotional needs of students.

8

Besley (1999) used the coaching model to teach anger management to two fourth-

grade male students. In this model, the counselor used direct instruction to model

appropriate ways to deal with anger during counseling sessions. After counseling sessions

were completed, the researcher met with the teacher to discuss the model of promoting

appropriate behavior. The researcher then planned times to be in the classroom to help

support the students. This approach allowed students to both learn and practice social

skills in the classroom, where students are faced with opportunities to interact with

students and teachers.

The teacher in this study recognized the benefits of having students receive anger

management counseling in the classroom. According to direct observations and responses

from structured questionnaires and interviews, both target students made improvements.

For example, one student began to use the coaching model on his own after three weeks

of coaching, and acquired no office referrals for the remainder of the year. Furthermore,

the same student reported that his behavior was different and that he had an increased

ability to control his anger. The other student retained more outward problems with

anger management but was able to identify strategies to use when situations provoked his

anger.

The Second Step program was designed to promote social competence in first

grade students. Using direct instruction as well as role plays, games, discussions, and

storybooks, the Second Step program teaches conflict resolution, problem solving skills,

anger management techniques and impulse management strategies. In one experiment,

the Second Step program was instructed in a large group format and was found to reduce

9

the aggression of five of the six target students. This study offers support for addressing

the social needs of first grade students (Renicks, 1999).

MacDonald, Chowdhury, Dabney, Wolpert and Stein (2003) used social skills

instruction with seven children between 8 and 11 years old. Over a six-week period,

participants met in a small group, once a week for 90 minutes, to receive social skills

instruction. The specific areas of focus for instruction were understanding feelings,

developing friendships, and communication skills. These skills were taught using role

play activities, art activities, and small group discussions.

Pre- and post-intervention parent checklists were used to measure the perceived

effect of the MacDonald et al. (2003) study. According to the results of parent checklists,

three children moved from outside the normal range of ideal behavior to within normal

range of ideal behavior. Additionally, teachers reported that two children moved into the

normal range of behavior, according to social behavior checklists. Furthermore, parents

of six children reported improved social skills such as eye contact, listening, taking turns,

compromising, initiating conversations, and staying on topic. Based on the parent and

teacher checklists, improvements were maintained at a three month follow-up

assessment.

As demonstrated in the aforementioned studies, counselors and psychologists fill

an important role in providing interventions that address the needs of students at risk for

emotional and behavioral problems in schools and clinic settings. As psychologists and

counselors take a more proactive role in this effort, they must be supported. One way to

support these professionals is for the entire school to work to proactively prevent

problems through a positive behavior support initiative.

10

Positive Behavior Support

School-wide positive behavior support (PBS) has been shown to proactively meet

the social and emotional needs of students (Kern & Manz, 2004; Safran & Oswald,

2003). PBS is a process of multi-level interventions to address academic, behavioral and

social needs of all students (Lane & Beebe-Frankenberger, 2004). Through PBS,

educators are trained to use empirically validated practices to provide interventions to

address the needs of all students in schoolwide, classroom, non-classroom, and individual

settings (Lewis & Sugai, 1999a). Lewis and Sugai outline the basic hierarchy of positive

behavior support as consisting of universal, secondary, and tertiary levels. The intent of

this hierarchy is to use positive strategies to promote positive behavior of students in the

four aforementioned settings.

Universal interventions. The goal of universal-level interventions is to prevent

behavior problems and academic problems. Schoolwide interventions help prevent

behavior problems and academic problems by creating clear expectations and

consistency. Examples of schoolwide interventions include academic programs, social

skills instruction, schoolwide rules including positive reinforcement, and a clear

discipline hierarchy. Between 80% and 90% of students will respond to schoolwide

interventions and thus will not require additional intervention (Lane & Beebe-

Frankenberger, 2004).

Secondary interventions. Secondary interventions address problems or reverse

harm that has already occurred. Interventions at the secondary level are more intensive

then primary interventions. Students who receive secondary interventions are often

grouped together in ability levels for performance and acquisition. Secondary

11

interventions usually include focused instruction groups. Social skills, academic, and

anger management groups are examples of secondary-level focused instruction groups

(Lane & Beebe-Frankenberger, 2004). The goal of secondary level interventions is to

remediate and prevent problems from escalating to the tertiary level.

Tertiary interventions. Tertiary interventions focus individual students identified

with emotional and behavivoral problems. Examples of tertiary interventions include

functional behavior assessments, behavior intervention plans, individualized instruction,

and home-school interventions (Lane & Beebe-Frankenerger, 2004). Because tertiary

interventions are individual and student-focused, the resources available, the

environment, and the needs of the student should be considered when planning them

(Dunlap & Hieneman, 1999). Typically, these interventions are provided to students who

are identified with emotional and behavioral problems.

The implementation of PBS within schools has successfully and positively

influenced the social and emotional skills of thousands of students throughout the world

(Carr et al., 1999; Lewis & Sugai, 1999a). Research has been conducted to demonstrate

the effectiveness of the application of PBS principles on school communities (Safran &

Oswald, 2003). Within school settings, PBS methods are typically designed to prevent

externalizing behaviors at the universal level or to intervene with individual students who

demonstrate externalizing behavior problems at the secondary and tertiary level.

An example of one PBS study that was a universal playground level intervention

was developed by Lewis, Powers, Kelk, and Newcomer (2002). Specifically, the authors

observed students’ physical and verbal aggression, such as hitting, insulting, and

misusing equipment. Using a universal playground intervention, in which behavior

12

expectations and a group contingency were implemented, the authors found a decrease in

problem behaviors among K-6 graders on the playground (Lewis et al.).

In addition to the effects of implementing PBS on playground behavior, research

has shown positive effects on behavior throughout the school environment. Luiselli,

Putnam, Handler, and Feinberg (2005) studied the effects of implementing a schoolwide

PBS initiative in an elementary school, which included forming a behavior support team,

ensuring a standard way to manage data, and creating positive behavior expectations.

Additionally, teachers and administrators used a token economy system and positive

reinforcement slips that were entered into monthly drawings. Implementation of the

schoolwide PBS program was shown to be effective at decreasing office referrals and

suspensions while increasing academic performance.

Powers (2003) examined the effects of direct instruction of social skills, in small

groups, to students with high rates of behavior problems. In two elementary schools, 19

participants, grades three through six, received small-group social skills instruction

during a 16-week period. One school had implemented PBS strategies for five years

while the other school did not have PBS strategies in place. Student outcomes were

measured by direct observation in the small group setting, classroom and playground.

Additionally, teachers and parents of participants completed questionnaires designed to

measure change. This study provided evidence that social skills training and behavior

management strategies are effective in addressing problem behaviors in the school

setting. Comparison between the schools showed that the students made and maintained

greater gains in social skill acquisition in the school where PBS strategies were in place

verses the school without PBS strategies.

13

The tertiary level of support within the PBS framework focuses on interventions

at the individual level. Scott, Nelson, Liaupsin, Jolivette, Christle, & Riney (2002)

conducted a literature review of PBS strategies with at-risk and adjudicated youth in

alternative educational settings. These youth were at high risk for school failure. Scott et

al. recommended using a Functional Behavior Assessment (FBA) to identify the

appropriate interventions as an effective method of addressing their behavior problems

(Scott et al., 2002). Interventions recommended include mentoring programs and check in

systems to promote positive behavior.

These outcomes and recommendations associated with studies, such as those

conducted by Lewis, Luiselli, Scott, and their respective colleagues, offer support for

using the PBS model in schools. Lewis and his colleagues’ findings indicated that PBS

principles are effective with the general population in a targeted, non-classroom setting

(Lewis et al., 2002). Luiselli et al.’s (2005) findings offered support for implementing

whole-school PBS strategies to decrease problem behavior and increase academics. Scott

et al.’s (2002) review of the literature and subsequent recommendations render support

for applying PBS strategies schoolwide and with a more targeted population--those who

have externalizing behaviors-- to prevent more serious behavior problems from

developing. To date, limited research efforts have focused on assessing the use of PBS

principles with students at the secondary level of interventions, especially those who are

at-risk for internalizing behavior problems.

Based on Power’s (2003) statistic that 20% of children suffer from mental health

disorders, there is a significant need for the identification and treatment of children who

display these disorders. Using public schools as the context for delivering the PBS model

14

is logical because it is a setting that is accessible to those who serve children and youth.

Merrell (2001) purports that although there are advantages to using school-wide and

class-wide interventions to provide all students access to social and emotional skill

training, some students will require individualized interventions. Methods for

accomplishing this worthy objective are described in the following section of this paper.

Social and Emotional Skills

Results from various studies show that active treatments -- any treatment that

addresses problem behavior—are more effective in reducing symptoms of emotional

behavior problems than no treatment (Compton, Burns, Egger, & Robertson, 2002). In

fact, researchers suggest that the treatment type for depressed children does not matter as

much in reducing symptoms as whether the children received any treatment (Compton et

al.). Treatments can be fairly simple, though no two will be alike. Some, such as

cognitive-behavioral interventions, attempt to teach children strategies to help them cope

effectively. Treatments are most commonly delivered in groups, time-limited, and

organized sessions (Compton et al.).

In an extensive literature review, Compton et al. (2002) reviewed the use of

cognitive-behavioral therapy (CBT) to address the needs of children with internalizing

behavior problems. CBT strategies used in the studies were typically group-based

treatments delivered by adults and most often a therapist in an outpatient clinic. Children

who received CBT interventions reported decreased symptoms characteristic of

internalizing disorders as well as increased coping skills. Furthermore, CBT reduced

negative affects, improved attendance rates, and increased confidence in coping with

stressful situations. These results suggest that simple, research-based techniques may

15

yield positive results for the overall quality of the child’s social and emotional welfare.

These strategies have promise for meeting the needs of students who are identified as

needing secondary level interventions within the PBS model.

The Oregon Resiliency Project has invested in developing CBT strategies that are

research based to promote the social and emotional learning of children and youth.

Merrell has developed curricula, Strong Teens and Strong Kids, specific to this purpose.

Strong Teens is a curriculum that is designed to promote social and emotional learning in

adolescents who are in grades 9 to 12. The content of Strong Teens is specific to concerns

relevant to high school and adolescents (Merrell, Carrizales, & Feuerborn, 2004b).

Strong Kids is a partially scripted curriculum to promote social and emotional learning

resiliency with children who are in grades 4 to 8. The content of Strong Kids is specific to

the concerns of upper elementary and middle school students (Merrell, Carrizales, &

Feuerborn, 2004a). Various studies have investigated the effectiveness of these curricula

on children and youth. Some of the reported outcomes are discussed below.

Merrell, Juskelis, Tran and Buchanan (under review) conducted three pilot studies

to test the efficacy of the Strong Kids and Strong Teens curriculum. The results of the

three studies are described below.

The first study took place in a public junior high school with 65 participants from

seventh and eighth grade. The Strong Kids curriculum was taught by study-skills class

teachers, one lesson each week for 12 weeks to students functioning in the general

education curriculum. These students were not identified with emotional/social disorders

or with disabilities. In this study, participants completed pre-and post-assessments, a 10-

item internalizing symptoms checklist and a 20-item social-emotional knowledge test.

16

The outcomes from this study demonstrated that student’s knowledge of healthy social-

emotional behavior increased from pre-to-post-assessment. Additionally, students

reported fewer internalizing problem symptoms after receiving the Strong Kids

curriculum. This study was conducted with general education students and offers support

for using Strong Kids as a universal intervention within the PBS model.

A second study was conducted in a high school with fourteen participants in

grades 9 to 12. These participants were identified with emotional disabilities and

qualified for special education services. Strong Teens lessons were taught once a week

over a 12 week period. Participants completed pre-and post-assessments, a 35-item social

and emotional knowledge test and a 35-item emotional problem symptom checklist. This

study showed statistically significant increases in the participant’s knowledge of healthy

social-emotional behavior and decreases in self-reported internalizing symptoms. These

findings suggest that Strong Teens is an effective intervention for students who require

tertiary level interventions within the Positive Behavior Support model.

A third study examined the effects of the Strong Kids curriculum on 120 fifth

grade students drawn from the general education population. Students in five classrooms

received the instruction of the curriculum once a week for 12 weeks. Students completed

the 20-item knowledge test and 10-item Internalizing Symptoms Scale for Children

(ISSC) as pre and post measures. This study found statistically significant gains as

measured by the 20-item knowledge test, but no significant changes in self-reported

internalizing problems as reported by the 10-item ISSC after instruction of the

curriculum.

17

In addition, Gueldner, Tran, Buchanan, and Merrell (2006) studied the impact of

the Strong Kids curriculum on fourth and eighth grade students in two public schools.

The experimental eighth grade group consisted of seven females and the control group

consisted of three females and four males. The fourth grade treatment group consisted of

three males and four females and the control group consisted of four males and three

females. All students were drawn from the general education population. Eighth grade

students were recruited for participation in a leadership class and fourth grade students

were recruited by the school psychologist. All students completed pre-and post-

assessments, the Strong Kids Symptoms Questionnaire, the Strong Kids Content Test and

the ISSC. Both groups of participants that received the instruction of the Strong Kids

curriculum showed statistically significant increases in scores on the Strong Kids Content

Test. Both treatment groups also reported less problem emotions after instruction. Again

this study also supports the effects of the Strong Kids curriculum with fourth and eighth

grade students.

Merrell et al. (under review) and Gueldner et al. (2006) recommended that further

studies be conducted using Strong Kids with populations who have not previously been

targeted. Endeavors should be taken to investigate the effects of the curriculum on

students who are at-risk, but not identified with emotional behavioral problems of all age

groups, including elementary age. Specifically, researchers should consider assessing the

effectiveness of Strong Kids and Strong Teens on the social and emotional behavior of

students who are at-risk for internalizing behavior problems at all grade levels, including

those in upper elementary grades.

18

Purpose

The purpose of this study was to investigate the effectiveness of using emotional

resiliency and social training to build emotional and social resiliency skills in fourth and

fifth grade students identified by the Systematic Screening for Behavior Disorders

(SSBD), or the school behavior team as students who are at risk for internalizing

behavior problems.

Research Question

What are the effects of the Strong Kids curriculum on social and emotional skills

of fourth and fifth grade students identified by the SSBD as emotionally at risk for

internalizing behaviors?

Social Validity Research Questions

What are the perceptions of the teachers in respect to (a) the need for

social/emotional training in schools and (b) the benefits of the curriculum on student

behavior?

What are the perceptions of the participating students in respect to the benefits of

the curriculum on their own behavior?

19

CHAPTER III

METHOD

Setting

This study was conducted at three elementary schools in two school districts in

central Utah. School A had been implementing a schoolwide PBS program for the past

three years and had a population of 695 students. Of these students, 86% were Caucasian,

12% Hispanic, and 2% were from other ethnic groups. The student to teacher ratio was

1:18. School A made adequate yearly academic progress as measured by federal

requirements and with a national percentile of 50, scored in the 66th percentile for reading

and the 71st percentile for math, as measured by the Stanford Achievement Test Series,

9th edition (SAT9). Furthermore, approximately 75 students were served for social and

emotional needs during the 2005-2006 school year. Services were delivered by the

school psychologist and were usually administered in small group or individual

counseling sessions. As a preventative measure, the school psychologist presented four

lessons targeting bullying and emotions in each classroom.

School B had been implementing school wide PBS for the past four years and had

a population of 524 students. Of these students, 79% were Caucasian, 18% Hispanic, and

3% were from other ethnic groups. The student to teacher ratio was 1:21. School B also

made adequate yearly academic progress as measured by federal requirements and scored

in the 57th percentile for reading and the 65th for math, as measured by the SAT9. At

school B during the 2005-2006 school year, approximately 10 students received either

small group or individual counseling. The school psychologist and classroom teachers

worked together to develop behavior plans and monitor the progress of each student.

20

School C has been implementing school wide positive behavior support for 1 year

and had a population of 613 students. Of these students, approximately 88% were

Caucasian, 9% Hispanic, and 3% were from other ethnic groups. School C also made

adequate yearly academic progress as measured by federal requirements and scored in the

40th percentile for reading and the 45th percentile for math as measured by the SAT9.

During the 2005 - 2006 school year, 6 students at this school received psychological

services as part of their individual education plans, and approximately 25 other students

were referred for small group counseling or parent consultation with the school

psychologist. Teachers of an additional 35 students consulted with the school

psychologist. Additionally, the school psychologist presented one lesson each month, in

every class, to promote pro-social behavior.

In this study, students participated in Strong Kids instruction in small group

setting consisting of 5 – 10 members. The instructors at schools A and C instructed the

Strong Kids curriculum in a conference room around an oval table. The students sat

around the table while the instructors stood at the head of the table and used a white

board mounted on the wall. The instructors did not use the overhead projector but

students were seated close enough to look at handouts without projecting them. At school

B, the instruction took place in an empty kindergarten classroom. Participants sat around

a kidney table for the lessons.

Participants

Selection. Twenty-two students, from the third, fourth and fifth grades,

participated in this study. The target students were identified via the following process:

screening for emotional and social deficits, selection by the school behavior team, and

21

obtaining parental consent for participation (See Appendix A). Table 1 shows the

demographics of the participants by school.

Table 1 Demographics of Student Participants School N Gender Ethnicity Grade Selection

M F Caucasian Hispanic 3rd 4th 5th SSBD Rec

A 10 5 5 9 1 0 7 3 5 5 B 5 1 4 5 0 0 3 2 3 2 C 7 4 3 7 0 2 0 5 4 3

Note. Rec = recommended for participation.

Screening. In order to identify potential participants, researchers used the

Systematic Screening for Behavior Disorders (SSBD). The SSBD is a screening

instrument that helps teachers identify students who exhibit both internalizing and

externalizing behavior problems (Walker & Severson, 1992). It has been normed and

validated for use with grades one through six. Several studies indicate that the SSBD is a

reliable instrument for identifying students with potential behavior disorders (Kelley,

1998; Walker et al., 1990; Zlomke & Spies, 1998).

In this study, research and school staff used the SSBD to identify students who

were at risk for emotional and behavioral problems and therefore in need of secondary-

level intervention, according to the PBS model. First, fourth and fifth grade teachers were

asked to identify 5-10 students in their class who displayed the most extreme

internalizing behaviors by listing their names and list them in order of most severe

behaviors to least. Next, teachers completed a 33-item critical events checklist and a 23-

item combined frequency index for the first three students each teacher listed. Based on

22

the instructions for administrating the SSBD and specified modifications for identifying

students at the PBS secondary level, students who scored one or higher on the Critical

Events Scale moved onto the next level of identification in which teachers completed the

Adaptive Behavior Scale in reference to the target student. For students with scores of 41

or less on the Adaptive Behavior Scale, teachers completed the Maladaptive Behavior

Scale. Students with a score of 14 or more on the Maladaptive Behavior Scale were

identified as possible participants for this study.

After using the screening process described above, fewer student participants

were identified than anticipated. In order to increase the number of student participants,

students with SSBD scores that were close to the cutoff scores were recommended for

participation by the school behavior team. Even after this, the number of student

participants was still lower than anticipated, so the school behavior team recommended

possible participants to the author. At school A, 10 target students were identified: 5 via

the SSBD processes, 4 whose SSBD scores did not meet, but were close to, the cutoff

score, and 1 that did not have a SSBD score but was recommended by the behavior team.

At school B, 5 total students were identified: 3 via the SSBD processes, 1 whose SSBD

score did not meet, but were close to, the cutoff score, and 1 that did not have a SSBD

score but was recommended by the behavior team. At school C, 7 total students were

identified: 4 via the SSBD processes and 3 that did not have SSBD scores but were

recommended by the behavior team. (See Appendix B for student SSBD scores.) Finally,

a parental consent form, detailing the Strong Kids curriculum and nature of this study,

was sent to the parents of all possible participants.

23

BYU-PBS Initiative staff and school psychologists. This study was conducted by

the author, Brigham Young University-Positive Behavior Support Initiative (BYU-PBS

Initiative) staff, a team of school coordinators and research staff who implement the PBS

model in schools, and 3 school psychologists. In order to promote school involvement

and ownership, the Strong Kids lessons were taught by both a BYU-PBS Initiative staff

member and the school psychologist. BYU-PBS Initiative is a public school partnership

that seeks to secure environments that promote learning and support for all students

through the use of the PBS model.

The school psychologist at school A was a female with a BA in psychology and a

candidate for a specialist degree in school psychology, currently completing her

internship at this particular school. Also at school A was a female BYU-PBS Initiative

instructor. She had a BS in elementary education, experience teaching second grade in a

Utah public school, and had been employed with BYU-PBS Initiative for 4 years.

At school B, instruction was delivered by a female school psychologist with a

bachelors degree in Human Development and an Educational Specialist degree in School

Psychology. She held a current Utah School Psychology certificate and the study took

place during her first year of working as a school psychologist. The BYU-PBS Initiative

instructor at school B was a male with a bachelors degree in Spanish and a special

education teaching certificate to teach students with mild to moderate disabilities. He was

pursuing a masters degree in Information Systems Management and had worked with

BYU-PBS Initiative for five years.

School C’s school psychologist was a female with a masters degree in School

Counseling and Psychology. She had been practicing for five years as a school

24

psychologist and is a member of the district’s autism team. The BYU-PBS Initiative

instructor at school C was a male with a bachelor of science degree in Behavioral Science

with an emphasis in psychology and a masters degree in education with an emphasis on

counseling. He worked for approximately 7 years in a youth detention facility with

children ages 10-19. Additionally, he was a probation officer for two years working

specifically with juveniles ages 13-18.

Observers. There were two reliability observers for this study. Reliability

observations were to check for treatment fidelity. The reliability observer at school A was

a female BYU-PBS Initiative Project Specialist with a bachelor of science degree in child

development, a masters degree in educational psychology, and 18 years of teaching

experience. The reliability observer at school B and school C was also a female BYU-

PBS Initiative research staff member with a bachelor of science degree in psychology,

with minors in statistics and editing. She had been admitted to a masters program in

school psychology and worked with BYU-PBS Initiative for one year.

Materials

The materials necessary for this study included a small group setting (tables or

desks and chairs), overhead projector, screen, overhead markers, transparencies, white

board and markers, pencils, and some edible reinforcers. This study required copies of the

SSBD, the 10-item version of the ISSC and the 20-item Strong Kids Knowledge Test

(Merrell, Carrizales, & Feuerborn, 2004b). Additionally, the Teacher’s Report Form

(TRF) was used (Achenbach& Rescorla, 2001).

Dependent Variable and Measures

25

The dependent variables for this study were the emotional resiliency and social

skills of fourth and fifth grade students identified as at-risk for internalizing behavior

problems. Some examples of emotional resiliency include understanding feelings,

thinking clearly and looking at situations with a positive view. Examples of social skills

are interactions with peers and teachers such as asking a question, following directions

and initiating a conversation. The emotional resiliency and social skills of participants

were measured via pre- and post-assessments by the student participants and general

education teachers.

Once selected for the study, student participants completed a short version of the

ISSC–a self-questionnaire (Merrell & Walters, 1998). The ISSC is a 48-item self-report

measure of depression, anxiety and related affective and cognitive symptoms. It is

normed for students in grades three to six. After an extensive literature review, the

developers of the ISSC worked to compile possible items on the checklist for content,

readability and redundancy. The ISSC addresses four general domains of internalizing

disorders: depression, anxiety, somatic complaints, and social withdrawal (Merrell &

Walters, 1998).

Sanders, Merrell, and Cobb (1999) studied the validity of the ISSC. Thirty-five

boys and 5 girls, all African American, completed the ISSC. Results indicated that the

ISSC was accurate in differentiating between general education students and students

identified as having social and emotional disorders (Sanders et al.).

The short version of the ISSC (see Appendix C), consists of 10 items that are

particularly sensitive to the subject matter covered in the Strong Kids curriculum. For

each item, the respondent has the option to choose between never true, hardly ever true,

26

sometimes true and often true. The 10-item ISSC was used in a study of the Strong Kids

curriculum (Merrell, Juskelis, Tran & Buchanan, under review) and is included as part of

the Strong Kids curriculum. It has reliability levels for research, administrative and

screening of .70 to .80. Additionally, the 10-item ISSC has demonstrated adequate

convergent validity coefficients, from .70 to .88, with established self-report measures

including the Children’s Depression Inventory and the full length ISSC.

The 20-item Knowledge test (see Appendix C) is part of the Strong Kids

curriculum, and was developed by members of the Oregon Resiliency Project research

team for use as a pre- and post-measure with the Strong Kids curriculum. It measures

both students’ knowledge and understanding of healthy social-emotional behavior. The

Knowledge test is a self-report measure for students in fourth through eighth grades.

The knowledge test consists of 5 true and false questions, 15 multiple choice questions,

and has been found to be a sensitive measure associated with the Strong Kids

curriculum. The knowledge test has an internal consistency reliability of .60 to .70

(Merrell, Carrizales, & Feuerborn, 2004a).

The general education teachers of the participants were asked to complete the

Teacher’s Report Form (TRF) of the Child Behavior Checklist (Achenbach & Rescorla,

2001). The TRF was completed as a pre- and post-assessment. The TRF is a 113-item

checklist that is normed for children ages 6-18 (Achenbach & Rescorla, 2001). The items

on this checklist are statements that can be answered with not true, somewhat or

sometimes true, and very true or often true. The TRF was selected for this study because

teacher responses are divided into categories that measure internalizing, externalizing and

total behavior problems.

27

Because the TRF has been standardized an average score is 50 with a standard

deviation of 10. Each subscale has specific cutoffs for borderline and clinical scores.

Generally, scores close to 50 represent students that fall into the average range, scores

that are close to 60 are usually in the borderline range, and scores that are close to 65 are

usually in the clinical range (Achenbach & Rescorla, 2001).

Independent Variable and Measures

The independent variable was the instruction of the Strong Kids (Merrell,

Carrizales, & Feuerborn, 2004a) curriculum. There are 12 lessons in the Strong Kids

curriculum. The lesson plans are partially scripted and highly structured. Each lesson

lasted approximately 45 to 50 minutes. Lessons were designed to be very similar in

format and style as well as repetitive.

The lessons were taught by a BYU-PBS Initiative instructor and the school

psychologist at each school over a six-week period during the winter of the 2005/2006

school year with two lessons being taught per week. A treatment fidelity checklist was

completed by a reliability observer who recorded the components of the lesson as they

were instructed during four of the twelve lessons (see Appendix D).

To ensure that all BYU PBS Initiative staff members and school psychologists

presented the Strong Kids lessons in their entirety, the researcher conducted training

sessions on Strong Kids instruction at each school across three sessions. The researcher

taught one lesson of the curriculum and presented all the materials using direct

instruction techniques including defining skills and stating expectations. The researcher

checked for understanding by having the BYU-PBS Initiative instructors and school

psychologists complete a written assessment about the instruction of the curriculum (see

28

Appendix E). Each school psychologist and BYU-PBS Initiative instructors completed

the assessments with 100% accuracy.

Finally, the researcher explained that treatment fidelity checklists would be

completed by an observer during four lessons. The researcher provided copies of the

checklists for the school psychologists and BYU-PBS Initiative instructors so they would

be familiar with the criteria the observer would use to evaluate each lesson. A treatment

fidelity checklist was also completed by an observer during the Strong Kids instruction

training sessions to ensure all school psychologists and BYU-PBS Initiative instructors

received the same training.

Lesson one. Lesson one focused on emotional strength training. This was an

opportunity for the students to be introduced to the curriculum. They learned some

important terms related to the materials including, “. . . emotion, self-esteem, depression

and anxiety” (Merrell et al., 2004a, p.4). Expectations of appropriate behaviors during

instruction time were also discussed.

Lessons two and three. The second and third lessons focused on the topic of

understanding feelings. The purposes of these lessons were to improve vocabulary,

awareness and resiliency of students emotional understanding. Lesson two focused on

understanding one’s own feelings; lesson three builds upon that by teaching how to apply

an understanding of one’s own feelings to real life situations.

Lesson four. Lesson four taught students about anger and how to deal with it.

During this lesson, students learned a six-step anger model to help them cope with

feelings of anger. The two basic concepts involved in learning to deal with anger are “(1)

29

that anger is a normal emotion, and (2) that anger serves the important function of

protection and motivation in our lives” (Merrell et al., 2004a, p. 5).

Lesson five. In lesson five the focus is on learning how to understand other

people’s feelings. The students learned that it is imperative to understand the feelings of

others in order to solve conflict. Students had opportunities to practice compassion and

empathy towards others.

Lessons six and seven. The ability to think clearly is the emphasis of lessons six

and seven. These lessons are “. . . designed to help students recognize positive and

negative thought patterns and how they contribute to our moods, choices, and actions in

positive and negative ways” (Merrell et al., 2004a, p. 6).

Lesson eight. Lesson eight emphasized the importance of thinking positively.

Positive thinking has the power to redirect negative and pessimistic feelings into more

positive and productive thought patterns.

Lesson nine. During lesson nine, students learned strategies for resolving conflict

and solving people-problems. Students learned a problem solving model and had

opportunities to discuss and practice compromising, deal-making and brainstorming

according to scenarios in the lesson.

Lesson ten. Letting go of stress is the focus of lesson ten. This lesson provides an

opportunity for students to discuss things that cause stress in their lives. Students learned

how to identify when they experience stress in their lives. They also learned ways to relax

and face that stress in a healthy manner.

Lesson eleven. During this lesson students spent time setting goals to change

behavior and increase positive activities. There is a six-step process the students were

30

taught so they know how to set and achieve goals. They also learned that when they are

engaged in positive activities they will feel happier and have higher self-esteem.

Lesson twelve. Lesson twelve focused on “finishing up”. The focus of the final

lesson is ending on a positive note. This will be a time for the students to celebrate the

things they have learned and the goals they have set and accomplished.

One modification was made to the recommended presentation format.

Anticipatory sets designed to introduce the topic and engage the students were added to

each lesson (see Appendix F). Most anticipatory sets involved some physical activity.

The reason for including these anticipatory sets was to create an interactive activity that

would get the students attention.

Based on the suggestions for teaching Strong Kids in the instruction manual, a

homework completion reinforcement program was also used. At each lesson, participants

who completed their homework assignment from the previous lesson wrote their name on

a slip and instructors then held a drawing for two or three participants. The chosen

participants were then able to choose a small candy.

Data Collection

Data were collected from pre-test, post-test and follow-up assessments of the

following measures. Students completed the 10-item ISSC and the 20-item Strong Kids

Knowledge Test during the first and last lessons of the Strong Kids curriculum and

additionally at follow up four to six weeks after the last lesson. Students completed the

measures in the same room described in the setting section in a small-group pull out

setting. General education teachers completed the TRF each time the students completed

31

the student measures. Teachers received the TRF in their faculty boxes and returned the

completed form to the school psychologist.

Experimental Design and Conditions

A pre-test-post-test design was used in this study. All students completed the pre-

test measures prior to receiving the Strong Kids curriculum. After completing the pre-

tests, all students then received the treatment, the instruction of the Strong Kids

curriculum. After instruction, post-test measures were completed. Four to Six weeks after

the conclusion of the instruction of the Strong Kids curriculum all measures were

collected one additional time. The follow-up assessments were conducted to see if

participants maintained change after the instruction of the Strong Kids curriculum

Data Analysis

Data and demographics collected from the pre-test, post-test and follow-up

measures were analyzed using a t-test both within and between groups using SPSS, a

statistical program designed for use with the social sciences (Nie, 2005) (p < .05). Using

a t-test allows researchers to determine if differences between groups are statistically

significant (Salkind, 2000).

Treatment Fidelity

To ensure that all lessons were taught accurately and in their entirety, a reliability

observer attended four lessons and completed a treatment-fidelity checklist to ensure the

treatment was stable and validity was ensured. See Appendix D for examples of the

treatment fidelity checklists. At school A, the reliability observer attended and observed

lessons one, four, six and nine. Each instructor was observed two times. During observed

32

lessons, the observer marked an “x” in the “yes” or “no” column to indicate how the

instructor taught each section of the lesson.

The reliability observer at school B observed lessons two, four, six and eight.

Lesson eight was instructed by a substitute instructor from school C because both of the

instructors at School B were unavailable. Each section of lessons six and eight were

checked with a “yes” indicating that each component of the lessons was instructed to the

participants. During lesson two, the instructor did not conduct the follow-up discussion

after handout 2.3. During lesson four, the instructor did not define anger management in

section III. Additionally, during lesson four, the instructor ran out of time for the role-

plays, section VIII, and closure, section IX. All other components of observed lessons

were instructed in their entirety.

At school C, the reliability observer was able to attend and observe lessons three,

six and eight. A fourth observation time was cancelled and could not be made up. Each

section of the three observed lessons was instructed in their entirety and participants

received all components of each lesson.

Social Validity

The overarching goal of social validity is to ensure that the research endeavor is

one of social importance. Social validity has a three-pronged purpose: First, is the

research addressing a problem that is commonly accepted as a problem and does it have a

commonly accepted goal? Second, are the interventions accepted as appropriate for

addressing the problem? Third, will the research produce results that will be acceptable to

society (Bailey & Burch, 2002)? As a way to collect information pertaining to the social

validity of this project, the researcher collected data using four questionnaires at the

33

conclusion of the instruction of the curriculum. The first questionnaire probed teachers

regarding their perceptions about the need for training in emotional and social resiliency.

This questionnaire addressed the first social validity purpose. The second and third

questionnaires measured teachers and students perceptions of outcomes from

participation in the Strong Kids study. Teachers and students were asked about changes

they may have noticed since the student’s completion of the curriculum. These

questionnaires addressed the third social validity purpose. The fourth questionnaire was

completed by the school psychologists and BYU-PBS Initiative staff members who

instructed the curriculum. This questionnaire addressed the second social validity purpose

(see Appendix G).

34

CHAPTER IV

RESULTS

This study examined the effects of the Strong Kids curriculum on students at-risk

for internalizing behavior problems. Students received instruction in a small group

setting. Pre-, post- and follow-up assessments measured student’s internalizing behaviors

and their knowledge of emotional and social skills. The following section details the

results from the pre-, post- and follow-up measures. The descriptive statistics from the

TRF, ISSC and Knowledge Test will be reviewed along with the results of the t-tests to

compare means. These results provide further support for the use of the Strong Kids

curriculum with students at risk for internalizing behaviors.

Descriptive Statistics and Mean Comparisons

Student participants completed the 10-item internalizing symptom checklist and

20-item knowledge test. Teachers completed the TRF. These measures were completed

as a pre-test before the instruction of the Strong Kids curriculum, as a post-test after

instruction and as a follow-up measure 4 to 8 weeks after the conclusion of instruction.

Table 2 provides the main descriptive statistics for the TRF internalizing problems,

externalizing problems and total problems.

TRF results. The results of this study showed a decrease in internalizing,

externalizing and total problems scores. The internalizing pre-test mean was 63.27, while

the post-test internalizing mean was 62.36, and the follow-up internalizing mean was

56.95. There was no statistically significant (p < .05) decrease between the pre and post

internalizing mean scores but there were highly statistically significant (p < .05)

decreases between the pre and follow-up means, and the post and follow-up means.

35

These statistically significant changes in the desired direction indicate that students

internalizing scores decreased after the instruction of the Strong Kids curriculum.

Table 2

TRF Descriptive Statistics

Measure Mean Standard Deviation N

Internalizing Pre-test 63.71 9.49 22

Internalizing Post-test 62.36 6.91 22

Internalizing Follow-up 56.95 7.16 20

Externalizing Pre-test 60.08 9.37 22

Externalizing Post-test 59.00 7.81 22

Externalizing Follow-up 56.05 8.42 20

Total Pre-test 63.75 8.56 22

Total Post-test 62.45 7.43 22

Total Follow-up 59.05 8.20 20

The externalizing pre-test mean was 60.08, and the post-test externalizing mean

was 59.00, while the follow-up mean was 56.05. Thus, there was no significant change

from the externalizing pre-test mean and the post-test mean. There were significant

decreases between the post-test and follow-up externalizing means. There were also

significant decreases between the pre-test and follow-up externalizing means. The

36

significant decreases suggest that students externalizing behavior problems decreased

after the instruction of the Strong Kids curriculum (p < .05).

The pre-test total problems mean was 64.25, the post-test mean for total problems

was 63.35, and the follow-up mean for total problems was 59.05. There was not a

statistically significant change between the pre-test total mean and the post-test total

mean. There were highly statistically significant decreases in problems between the post-

test total mean and the follow-up total mean. There were also highly statistically

significant decreases in problems between the pre-test total mean and the follow-up total

mean (p < .05).

Teachers rated the student participants using the TRF as a pre-test, post-test and

follow-up assessment. Figures 1, 2 and 3 illustrate the mean changes for internalizing

problems, externalizing problems and total problems. Student participants demonstrated

decreases in internalizing, externalizing and overall total problems. These decreases

further support the use of the Strong Kids curriculum. Table 3 outlines the t-test

comparison of means and shows the significant changes.

63.708362.3636

56.95

50

52

54

56

58

60

62

64

66

Pre-test Post-test Follow-up

Norm

aliz

ed T

Sco

res

Int Pre-testInt Post-testInt Follow-up

Figure 1. TRF internalizing pre-test, post-test and follow-up means.

37

60.083359

56.05

50

52

54

56

58

60

62

64

66


Norm

aliz

ed T

Sco

res

Ext Pre-testExt Post-testExt Follow-up

Figure 2. TRF externalizing pre-test, post-test and follow-up means.

63.7562.4545

59.05

50

52

54

56

58

60

62

64

66


Norm

aliz

ed T

Sco

res

Tot Pre-testTot Post-testTot Follow-up

Figure 3. TRF total pre-test, post-test and follow-up means.

38

Table 3

T-test Comparison of TRF Means

Standard Mean Standard Error Measures Difference Deviation Mean t

Internalizing Problems

Pre-test and Post-test .90 6.49 1.38 .657

Pre-test and Follow-up 6.50 5.77 1.29 5.04***

Post-test and Follow-up 4.75 5.39 1.20 3.94***

Externalizing Problems

Pre-test and Post-test .82 5.22 1.11 .73

Pre-test and Follow-up 3.95 6.73 1.51 2.62*

Post-test and Follow-up 3.00 4.66 1.04 2.88**

Total Problems

Pre-test and Post-test 1.23 5.23 1.11 1.10

Pre-test and Follow-up 5.20 5.90 1.32 3.94***

Post-test and Follow-up 3.30 3.66 .82 4.04***

Note. 95% Confidence Interval *p

39

normal scores on the TRF pre-test, post-test and follow-up. Before the instruction of the

Strong Kids curriculum, only five student participants scored in the normal range. After

the instruction of the Strong Kids curriculum on the follow-up TRF assessment, 14

student participants scored in the normal range.

13

10

45

32

4

9

14

0

2

4

6

8

10

12

14

16

INT Pre-test INT Post-test INT Follow-up

ClinicalBorderlineNormal

Figure 4. TRF internalizing scores.

65

3

54

3

11

1314

0

2

4

6

8

10

12

14

16

EXT Pre-test EXT Post-test EXT Follow-up


Figure 5. TRF externalizing scores.

40

13

11

6

23

4

78

10

0

2

4

6

8

10

12

14

TOT Pre-test TOT Post-test TOT Follow-up


Figure 6. TRF total problems scores.

The results of the TRF showed statistically significant (p < .05) decreases in

student participants’ problem behaviors. Teachers reported decreases in student

internalizing, externalizing and total problem behaviors after participants completed the

Strong Kids curriculum.

ISSC and knowledge test results. The mean score for the pre-test 10-item ISSC

was 16.15. The mean score for the post-test 10-item ISSC was 14.65. The follow-up

mean 10-item ISSC score was 14.21. There were statistically significant decreases

between the ISSC pre-test and post-test. There were also statistically significant decreases

between the ISSC pre-test mean and the follow-up mean. There were not statistically

significant changes between the post-test and follow-up ISSC means (p < .05). Table 4

provides demographic information for the ISSC and the knowledge test. Table 5 and

Figure 7 illustrate the decrease in symptoms as reported by student responses on the ISSC

pre-test, post-test and follow-up test.

41

Table 4

ISSC and Knowledge Test Descriptive Statistics

Measure Mean Standard Deviation N

ISSC Pre-test 15.91 4.52 22

ISSC Post-test 14.40 4.67 22

ISSC Follow-up 14.21 4.43 17

Knowledge Pre-test 11.17 3.69 22

Knowledge Post-test 13.64 4.20 22

Knowledge Follow-up 13.44 3.78 18

Table 5

T-test Comparison of ISSC and Knowledge Test

Standard Mean Standard Error Measures Difference Deviation Mean t ISSC

Pre-test and Post-test 1.71 3.53 .77 2.23* Pre-test and Follow-up 1.94 3.65 .89 2.19* Post-test and Follow-up .44 4.10 .99 .44

Knowledge Test

Pre-test and Post-test -2.7 4.88 1.04 -2.62* Pre-test and Follow-up -2.61 5.01 1.18 -2.21* Post-test and Follow-up -.72 2.72 .64 -1.13 Note. 95% Confidence Interval *p

42

15.913

14.409114.2059

13

13.5

14

14.5

15

15.5

16

16.5


Mea

n Sc

ores

ISSC PretestISSC Post-testISSC Follow-up

Figure 7. ISSC pre-test, post-test and follow-up means.

The mean score for the pre-test knowledge test was 10.91. The mean post-test

knowledge score was 13.64. Finally the follow-up knowledge test mean score was 13.44.

There were significant gains between the pre-test and post-test and the pre-test and

follow-up test. There were no significant increases between the post-test mean and the

follow-up mean. Student knowledge increased from pre-test to post-test and maintained

knowledge of content from post-test to follow-up with no significant changes. Student

participants increased in knowledge of content relating to the Strong Kids curriculum

between the pre-test and post-test and maintained knowledge gains between post-test and

follow-up. Figure 8 illustrates the gains in knowledge of content as measured by the

Strong Kids Knowledge test.

43

11.1667

13.6364 13.4444

0

2

4

6

8

10

12

14

16


Mea

n Sc

ores

Knowledge Pre-testKnowledge Post-testKnowledge Follow-up

Figure 8. Strong kids knowledge test pre-test, post-test and follow-up means.

Two students moved before the follow-up assessment. Table 6 outlines the pre

and post scores for those two students. Student number eight had an increase in teacher

and student reported internalizing problems from pre-test to post-test. This student

showed changed in the opposite direction demonstrating increased internalizing behavior

problems. Student 14 showed decreases in teacher and student reported internalizing

problems from pre-test to post-test. Student 14 had scores that were in the desired

direction with decreases in internalizing behaviors and increases in knowledge.

44

Table 6

Pre and Post Assessment Scores of Students Who Moved Before Follow-up

Student ID Test TRF*-INT TRF*-EXT TRF*-TOT ISSC** Knowledge

8 Pre-test 52 57 50 19 14

8 Post-test 68 62 68 20 13

14 Pre-test 71 59 64 17 7

14 Post-test 70 55 65 12 9

Note. *TRF scores are standardized scores. **ISSC and knowledge scores are raw scores.

SSBD Participants vs. Recommended Participants

Because some student participants were identified using the SSBD and others

were recommended for participation, separate data analyses were run on the participants

identified by the SSBD compared to those recommended for the study.

Participants identified by the SSBD showed statistically significant decreases

from the ISSC pre-test mean to ISSC post-test mean. There were significant decreases

from the Internalizing pre-test mean to post-test mean and highly significant decreases.

Table 7 outlines the demographic information for the group of student participants

identified by the SSBD and those recommended for participation in the study. Table 8

outlines the statistically significant outcomes for student participants identified by the

SSBD vs. those that were recommended. For data coding key see Appendix I. For

complete data analysis information see Appendix J.

45

Table 7

SSBD vs. Recommended Student Participant Demographics

SSBD Recommended Standard Standard Scale Mean Deviation n Mean Deviation n TRF Int Pre-test 66.83 4.84 12 60.58 11.98 10

TRF Int Post-test 64.00 5.89 12 60.40 7.82 10

TRF Int Follow-up 58.33 7.44 12 54.88 6.62 8

TRF Ext Pre-test 60.08 6.25 12 56.33 10.67 10

TRF Ext Post-test 61.58 7.06 12 55.90 7.88 10

TRF Ext Follow-up 57.42 7.89 12 54.00 9.30 8

TRF Tot Pre-test 67.92 6.97 12 59.58 8.17 10

TRF Tot Post-test 65.00 7.06 12 59.40 6.70 10

TRF Tot Follow-up 60.83 8.91 12 56.38 6.65 8

ISSC Pre-test 16.25 4.04 12 15.55 5.16 10

ISSC Post-test 13.67 5.10 12 15.30 4.16 10

ISSC Follow-up 13.89 4.04 9 14.56 5.08 8

Knowledge Pre-test 12.00 2.76 12 10.33 4.40 10

Knowledge Post-test 13.58 4.96 12 13.70 3.34 10

effects of the strong kids curriculum on students at...

Documents