experience corps social-emotional learning evaluation€¦ · experience corps is affecting...

ASSOC IAT E S

Experience CorpsSocial-Emotional LearningEvaluation

Experience CorpsSocial-Emotional LearningEvaluationAugust 30, 2019Submitted to:AARP Foundation Experience Corps601 E Street NWWashington, DC 20004

Submitted by:Allan Porowski, Michele de Mars, Emilie Kahn-Boesel, David RodriguezAbt Associates6130 Executive BoulevardRockville, MD 20852

ASSOC IAT E S

C O N T E N T S

Abt Associates Experience Corps SEL Evaluation August 30, 2019 ▌ii

CONTENTS Executive Summary ..................................................................................................................... iii 1. Introduction and Background ..........................................................................................1

2. Study Overview ..................................................................................................................5 2.1. Recruitment and Sample ............................................................................................5 2.2. Design and Programming of Survey ..........................................................................5 2.3. Survey Administration .................................................................................................6 2.4. Response Rates .........................................................................................................6

3. Analysis ..............................................................................................................................7

4. SEL Outcomes ...................................................................................................................9 4.1. Overall .........................................................................................................................9 4.2. Student Subgroups .................................................................................................. 10 4.3. Variation by Competency Area ................................................................................ 11 4.4. DESSA-mini ............................................................................................................. 15

5. Academic Outcomes ...................................................................................................... 16 5.1. Reading (SSIS-SEL) ................................................................................................ 16 5.2. Motivation to Succeed (SSIS-SEL) ......................................................................... 17

6. Recommendations and Conclusions ........................................................................... 19 6.1. Challenges and Successes ..................................................................................... 19 6.2. Future Implementation ............................................................................................. 20 6.3. Conclusion ............................................................................................................... 21

7. References ....................................................................................................................... 22

Appendix A: Supporting Literature .......................................................................................... 23 A.1. School- and After-School-Based SEL Interventions ................................................. 23 A.2. Mentoring Programs .................................................................................................. 24

Appendix B: Item-Level Findings from the DESSA ................................................................ 26

Appendix C: Comparison of Pre-Post Sample with All Respondents .................................. 28

E X E C U T I V E S U M M A R Y

Abt Associates Experience Corps SEL Evaluation August 30, 2019 ▌iii

Executive Summary

Building on AARP Foundation Experience Corps’ prior work related to social-emotional learning (SEL)and the growing body of literature supporting school-based SEL-focused interventions, AARP Foundation (with support from the Corporation for National and Community Service) engaged Abt Associates to conduct an evaluation of SEL outcomes in two AmeriCorps-funded Experience Corpsprograms during the 2018-19 school year. As part of the evaluation, the Abt team reviewed several SEL measurement tools and identified a SEL-focused behavior rating system, the Devereux Student Strengths Assessment (DESSA), which is well aligned to the Experience Corps model and program components.The DESSA provides an overall social-emotional composite score as well as scores for eight individualdomains of SEL competence: Self-Awareness, Social-Awareness, Self-Management, Goal-Directed Behavior, Relationship Skills, Personal Responsibility, Decision Making, and Optimistic Thinking.

The Abt team administered the DESSA to 72 teachers in nine schools within the two local programs, as well as two questions from the Social Skills Improvement System–Social and Emotional Learning Edition(SSIS-SEL) that assess reading performance and motivation. Teachers were invited to complete thesurvey once in November 2018 and again in May 2019.

The findings of the SEL evaluation, which focused on the 101 students with valid pretest and posttest data, demonstrate that the DESSA Composite scores increased significantly between fall 2018 and spring 2019. In addition, on every subscale of the DESSA, students showed statistically significant improvements from fall to spring, with the strongest improvements found on the Personal Responsibility,Relationship Skills, and Decision Making scales. Moreover, students showed statistically significant improvements on the SSIS-SEL items (reading performance and motivation to succeed).

Although this evaluation did not include a comparison group to establish what would have happened in the absence of Experience Corps, the Abt team was able to compare Experience Corps students to DESSA national norms. Students in Experience Corps started the school year at much higher risk of developing social-emotional problems relative to these national norms and ended the year close to thesenational norms. Reductions by almost 50% were observed in the DESSA Composite’s Need for Instruction range between fall and spring, which indicated a substantial transformation of students’ SEL skills.

As part of this evaluation, the DESSA-mini – a shorter version of the DESSA that uses a subset of eightof the 72 questions in the full DESSA – was also administered. Findings from the DESSA-mini showedimprovement in SEL, but these improvements were not statistically significant. Although the DESSA-mini was shown to be highly correlated to the DESSA Composite, it did not appear to be sufficientlysensitive in capturing what were otherwise substantial positive improvements in SEL.

Given these promising preliminary findings, the Abt team recommends that AARP Foundation continue to measure students’ social-emotional development across the full Experience Corps network to further understand the mechanisms by which the Experience Corps program is affecting such development. Although the full DESSA may be too burdensome to administer to all Experience Corps students, future data collection options are discussed to capture SEL outcomes and ensure that this key benefit of theExperience Corps program is being monitored and recognized.

E X E C U T I V E S U M M A R Y

Abt Associates Experience Corps SEL Evaluation August 30, 2019 ▌iii

Executive Summary

Building on AARP Foundation Experience Corps’ prior work related to social-emotional learning (SEL)and the growing body of literature supporting school-based SEL-focused interventions, AARP Foundation (with support from the Corporation for National and Community Service) engaged Abt Associates to conduct an evaluation of SEL outcomes in two AmeriCorps-funded Experience Corpsprograms during the 2018-19 school year. As part of the evaluation, the Abt team reviewed several SEL measurement tools and identified a SEL-focused behavior rating system, the Devereux Student Strengths Assessment (DESSA), which is well aligned to the Experience Corps model and program components.The DESSA provides an overall social-emotional composite score as well as scores for eight individualdomains of SEL competence: Self-Awareness, Social-Awareness, Self-Management, Goal-Directed Behavior, Relationship Skills, Personal Responsibility, Decision Making, and Optimistic Thinking.

The Abt team administered the DESSA to 72 teachers in nine schools within the two local programs, as well as two questions from the Social Skills Improvement System–Social and Emotional Learning Edition(SSIS-SEL) that assess reading performance and motivation. Teachers were invited to complete thesurvey once in November 2018 and again in May 2019.

The findings of the SEL evaluation, which focused on the 101 students with valid pretest and posttest data, demonstrate that the DESSA Composite scores increased significantly between fall 2018 and spring 2019. In addition, on every subscale of the DESSA, students showed statistically significant improvements from fall to spring, with the strongest improvements found on the Personal Responsibility,Relationship Skills, and Decision Making scales. Moreover, students showed statistically significant improvements on the SSIS-SEL items (reading performance and motivation to succeed).

Although this evaluation did not include a comparison group to establish what would have happened in the absence of Experience Corps, the Abt team was able to compare Experience Corps students to DESSA national norms. Students in Experience Corps started the school year at much higher risk of developing social-emotional problems relative to these national norms and ended the year close to thesenational norms. Reductions by almost 50% were observed in the DESSA Composite’s Need for Instruction range between fall and spring, which indicated a substantial transformation of students’ SEL skills.

As part of this evaluation, the DESSA-mini – a shorter version of the DESSA that uses a subset of eightof the 72 questions in the full DESSA – was also administered. Findings from the DESSA-mini showedimprovement in SEL, but these improvements were not statistically significant. Although the DESSA-mini was shown to be highly correlated to the DESSA Composite, it did not appear to be sufficientlysensitive in capturing what were otherwise substantial positive improvements in SEL.

Given these promising preliminary findings, the Abt team recommends that AARP Foundation continue to measure students’ social-emotional development across the full Experience Corps network to further understand the mechanisms by which the Experience Corps program is affecting such development. Although the full DESSA may be too burdensome to administer to all Experience Corps students, future data collection options are discussed to capture SEL outcomes and ensure that this key benefit of theExperience Corps program is being monitored and recognized.

S E C T I O N 1 : I n t r o d u c t i o n a n d B a c k g r o u n d

Abt Associates Experience Corps SEL Evaluation August 30, 2019 ▌1

1. Introduction and Background

AARP Foundation Experience Corps engages older adults to serve as literacy tutors and mentors to children in grades K-3 who are not reading at grade level, aiming to enhance both students’ academic outcomes and social-emotional development. In November 2017, AARP Foundation engaged Abt Associates to conduct an AmeriCorps-funded evaluation of Experience Corps students’ social-emotional development. This is designed to build on Experience Corps’ previous work in social-emotional learning, assess participating students’ SEL outcomes, and to better understand the mechanisms by which Experience Corps is affecting students’ social-emotional development. Experience Corps’ approach to SEL outcomes is rooted in the Collaborative for Academic, Social, and Emotional Learning (CASEL) framework focused on five key competencies: Self-Awareness, Self-Management, Social Awareness, Relationship Skills, and Responsible Decision-Making. As part of the initial stages of the evaluation, the Abt team worked closely with Experience Corps staff to create a theory of change that identifies Experience Corps program activities that support these competencies.

Exhibit 1. Theory of change

Using this theory of change (Exhibit 1), the study team selected the Devereux Student Strengths Assessment (DESSA) because it addresses all five CASEL competencies, in addition to including academic, goal-oriented and strengths-based questions that map to the Experience Corps model. The DESSA is a norm-referenced behavior rating scale that takes approximately 10 minutes to complete per student and includes 72 questions across eight SEL competencies (Exhibit 2). It provides an overall social-emotional composite score, in addition to scores for eight individual domains of SEL competence. There is also a shorter version of the DESSA (DESSA-mini) that uses a subset of eight of the 72 questions included in the full DESSA. As part of this evaluation, AARP Foundation was also interested in

5 Key Competencies Of SEL

Experience Corps Activities Supporting Competency



1. Introduction and Background

AARP Foundation Experience Corps engages older adults to serve as literacy tutors and mentors to children in grades K-3 who are not reading at grade level, aiming to enhance both students’ academic outcomes and social-emotional development. In November 2017, AARP Foundation engaged Abt Associates to conduct an AmeriCorps-funded evaluation of Experience Corps students’ social-emotional development. This is designed to build on Experience Corps’ previous work in social-emotional learning, assess participating students’ SEL outcomes, and to better understand the mechanisms by which Experience Corps is affecting students’ social-emotional development. Experience Corps’ approach to SEL outcomes is rooted in the Collaborative for Academic, Social, and Emotional Learning (CASEL) framework focused on five key competencies: Self-Awareness, Self-Management, Social Awareness, Relationship Skills, and Responsible Decision-Making. As part of the initial stages of the evaluation, the Abt team worked closely with Experience Corps staff to create a theory of change that identifies Experience Corps program activities that support these competencies.

Exhibit 1. Theory of change

Using this theory of change (Exhibit 1), the study team selected the Devereux Student Strengths Assessment (DESSA) because it addresses all five CASEL competencies, in addition to including academic, goal-oriented and strengths-based questions that map to the Experience Corps model. The DESSA is a norm-referenced behavior rating scale that takes approximately 10 minutes to complete per student and includes 72 questions across eight SEL competencies (Exhibit 2). It provides an overall social-emotional composite score, in addition to scores for eight individual domains of SEL competence. There is also a shorter version of the DESSA (DESSA-mini) that uses a subset of eight of the 72 questions included in the full DESSA. As part of this evaluation, AARP Foundation was also interested in

5 Key Competencies Of SEL

Experience Corps Activities Supporting Competency



exploring the feasibility of scaling up SEL measurement nationwide by administering the DESSA, DESSA-mini, individual DESSA subscales, or some other assessment.

Exhibit 2. Descriptions of DESSA scales

Source: LeBuffe, Shapiro, & Naglieri, 2014

These eight DESSA scales were based in part on the CASEL framework described above. Five of the eight DESSA scales align to the five key competencies of the CASEL framework. Two competencies in the CASEL framework were subdivided into separate scales in the DESSA. The DESSA subdivided CASEL’s Responsible Decision Making competency into Decision Making and Personal Responsibility subscales because the CASEL framework’s definition included both how an individual made a decision and their sense of personal responsibility. The DESSA also subdivided CASEL’s Self-Management competency into Goal-Directed Behavior and Self-Management subscales since self-management can be



exploring the feasibility of scaling up SEL measurement nationwide by administering the DESSA, DESSA-mini, individual DESSA subscales, or some other assessment.

Exhibit 2. Descriptions of DESSA scales

Source: LeBuffe, Shapiro, & Naglieri, 2014

These eight DESSA scales were based in part on the CASEL framework described above. Five of the eight DESSA scales align to the five key competencies of the CASEL framework. Two competencies in the CASEL framework were subdivided into separate scales in the DESSA. The DESSA subdivided CASEL’s Responsible Decision Making competency into Decision Making and Personal Responsibility subscales because the CASEL framework’s definition included both how an individual made a decision and their sense of personal responsibility. The DESSA also subdivided CASEL’s Self-Management competency into Goal-Directed Behavior and Self-Management subscales since self-management can be



focused both on managing one’s own emotions and behaviors.1 The DESSA includes one additional scale that is new: Optimistic Thinking. According to developers, optimistic thinking was originally thought of as a character trait rather than a social-emotional skill, so it was not included in the CASEL framework; however, the CASEL framework now includes optimistic thinking in the definition of Self-Awareness. According to the developer, the Optimistic Thinking scale is well established in the literature on resilience (LeBuffe et al., 2014). Exhibit 3 provides a mapping of the CASEL framework to the DESSA scales.

Exhibit 3. Alignment of DESSA to the CASEL framework

Source: Paul LeBuffe

The DESSA has strong psychometric properties. Internal consistency (as measured by Cronbach’s Alpha) is reported in the DESSA manual as .99 for the DESSA Composite when teachers provide ratings. Internal consistency of the individual scales is also very strong:

• Decision Making: .92 • Goal-Directed Behavior: .93 • Optimistic Thinking: .89 • Personal Responsibility: .92 • Relationship Skills: .94 • Self-Management: .92 • Self-Awareness: .89 • Social Awareness: .91

1 Personal communication with Paul LeBuffe, August 27, 2019.



focused both on managing one’s own emotions and behaviors.1 The DESSA includes one additional scale that is new: Optimistic Thinking. According to developers, optimistic thinking was originally thought of as a character trait rather than a social-emotional skill, so it was not included in the CASEL framework; however, the CASEL framework now includes optimistic thinking in the definition of Self-Awareness. According to the developer, the Optimistic Thinking scale is well established in the literature on resilience (LeBuffe et al., 2014). Exhibit 3 provides a mapping of the CASEL framework to the DESSA scales.

Exhibit 3. Alignment of DESSA to the CASEL framework

Source: Paul LeBuffe

The DESSA has strong psychometric properties. Internal consistency (as measured by Cronbach’s Alpha) is reported in the DESSA manual as .99 for the DESSA Composite when teachers provide ratings. Internal consistency of the individual scales is also very strong:

• Decision Making: .92 • Goal-Directed Behavior: .93 • Optimistic Thinking: .89 • Personal Responsibility: .92 • Relationship Skills: .94 • Self-Management: .92 • Self-Awareness: .89 • Social Awareness: .91

1 Personal communication with Paul LeBuffe, August 27, 2019.



In addition, an abbreviated form of the DESSA, called the DESSA-mini, is available from the developer. This assessment includes eight items of the 72 items from the DESSA Composite. The median internal consistency of the DESSA-mini is reported to be .92 (RAND Education Assessment Finder, 2018).



In addition, an abbreviated form of the DESSA, called the DESSA-mini, is available from the developer. This assessment includes eight items of the 72 items from the DESSA Composite. The median internal consistency of the DESSA-mini is reported to be .92 (RAND Education Assessment Finder, 2018).

S E C T I O N 2 : S t u d y O v e r v i e w


2. Study Overview

2.1. Recruitment and Sample

Experience Corps Minneapolis/St. Paul and Experience Corps Boston (Revere) were selected for the SEL evaluation in consultation with Experience Corps national staff and according to selection criteria. Selection criteria included (1) AmeriCorps-funded programs’ consideration of the local program’s integration of Experience Corps’ SEL framework, (2) feasibility of obtaining district approval, (3) regional variation, and (4) Experience Corps strategy (with priority given to local programs using a sustained tutoring model that increases student exposure to Experience Corps programming). The total sample in Minneapolis/St. Paul and Revere includes 72 teachers across nine schools — five charter schools in Minneapolis/St. Paul and four public schools in Revere. Generally, students in Minneapolis/St. Paul received one-on-one tutoring while students in Revere participated in small group tutoring. There are 272 students in sustained tutoring included in the sample. There are 182 students with completed pre-tutoring surveys and 166 students with completed post-tutoring surveys. The DESSA has strict rules for the treatment of missing items,2 which further limited our sample to 147 valid pretest observations and 125 valid posttest observations. The final analytic sample includes 101 students with both completed pre- and post-tutoring surveys.

2.2. Design and Programming of Survey

The teacher survey consisted of the full DESSA and two questions from the Social Skills Improvement System–Social and Emotional Learning Edition (SSIS-SEL) focused on reading proficiency and motivation to learn. The survey was programmed in SurveyGizmo, an online software tool for survey programming and administration. Survey links were individualized for each teacher and prepopulated with the teacher’s information. Each teacher received one survey link per student receiving sustained tutoring and participating in the evaluation. The surveys were programmed so that if a teacher was reporting on two students, the link to the survey for the second student was embedded in the completion page of the survey for the first student. This was designed to reduce teacher burden and increase the response rate.

Each survey question was multiple choice and included five possible response options. The 72 DESSA questions were divided into eight pages. On each page, teachers were prompted, “In the last 4 weeks that you spent with this student, how often did the student…,” and statements such as, “remember important information?” were listed. The five response options were “Never,” “Rarely,” “Occasionally,” “Frequently” and “Very Frequently.” The last page of the survey included the two SSIS-SEL questions. The first SSIS-SEL question was, “In terms of grade-level expectations, this child’s skills in reading are…”; the second question was, “This child’s overall motivation to succeed academically is…” The response options for both questions were, “Highest 15%,” “Next Highest 20%,” “Middle 30%,” “Next

2 Specifically, there can be no more than three items left blank on the entire DESSA, and no more than one item left blank on any individual scale; otherwise, the entire DESSA assessment is considered invalid.

S E C T I O N 2 : S t u d y O v e r v i e w


Lowest 20%” and “Lowest 15%.” Respondents who did not answer a question were reminded to do so by SurveyGizmo, but they were not required to answer any of the questions.

2.3. Survey Administration In November 2018 and again in May 2019, teachers from the two participating Experience Corps programs were invited to complete surveys about the SEL and academic behaviors of their students participating in sustained tutoring.

Teachers were sent survey invitations by Abt staff on the date determined by local Experience Corps program staff to be most convenient for each program. The emails included the links for each student on whom the teacher was asked to report (up to four students), and the links automatically allowed the teacher to navigate from one student to the next. In weeks following each launch date, Abt staff sent reminders to teachers who had not started the survey or who had partially completed it. The survey remained open for four to six weeks. The individualized survey links allowed teachers to complete the survey in multiple sittings as desired.

Abt staff were available to answer questions from teachers throughout the administration period and directed questions to local program staff when needed. Abt provided local program staff with updates on the response rates, and local program staff in turn encouraged teachers to complete the survey.

2.4. Response Rates In the fall, across both Experience Corps programs, 21% of teachers did not start the pre-tutoring survey, 18% partially completed the pre-tutoring survey, and 61% completed the pre-tutoring survey. In the spring, across both Experience Corps programs, 38% of teachers did not start the post-tutoring survey, 10% partially completed the post-tutoring survey, and 53% completed the post-tutoring survey.

In the fall, across both Experience Corps programs, 32% of students were not rated on the DESSA or SSIS-SEL, 2% were partially rated on the DESSA and SSIS-SEL, and 65% were fully rated on both the DESSA and SSIS-SEL. In the spring, across both Experience Corps programs, 39% of students were not rated on the DESSA or SSIS-SEL and 61% were fully rated on the DESSA and SSIS-SEL.

Exhibit 4. Teacher and Experience Corps student-level survey response rates

S E C T I O N 3 : A n a l y s i s


3. Analysis

The DESSA is designed to provide a comprehensive assessment of students’ social-emotional competencies. It is scored by adding raw scores from each item in a scale. Because the DESSA has a different number of items in each scale, individual scale scores cannot be directly compared. Instead, the DESSA provides two types of scores:

1. T-scores: The T-score is a standard score, scaled to have a mean of 50 and a standard deviation of 10. These T-scores range from 28 to 72, and differences in T-scores from pretest to posttest have the same interpretation throughout the range. For example, a five-point change in T-scores from 35 to 40 represents the same magnitude difference in a five-point change in T-scores from 60 to 65. Because of the ease and technical accuracy of their interpretation, we present T-scores whenever we describe pre-post change. The DESSA T-Scores can be transformed into three categorical measures:

— Need/Typical/Strength: The DESSA T-scores can be transformed to a categorical value with an intuitive explanation. A T-score between 28 and 40 indicates a “Need for Instruction,” since these students are at risk for exhibiting or developing social-emotional problems. T-scores between 41 and 59 are described as “Typical,” indicating that students will likely benefit from universal strategies designed to promote social-emotional competence. T-scores between 60 and 72 are classified as “Strength,” indicating that these students’ social-emotional competencies should be supported and sustained. The DESSA has been scaled so that, on average, approximately 16% of students nationwide can be expected to score in the Need for Instruction range, 68% can be expected to score in the Typical range, and 16% can be expected to score in the Strength range.

— Significant Change: The DESSA provides benchmarks to transform an individual student’s pre-post change in T-scores to a significance level. The benchmark for the achievement of a significant outcome depends on both the value of the pretest T-score and the scale being measured. In the analysis below, we present significance levels of aggregate pre-post change using traditional methods (paired T-tests) and significance levels of individual student changes using this DESSA benchmark.

— Clinical Meaningfulness of Change: The DESSA manual provides an additional analysis of posttest T-scores to measure the clinical meaningfulness of students’ improvement. This expands upon the Significant Change benchmark by splitting it up into two categories: “favorable” and “optimal” outcomes. A “favorable” outcome is the result of a significant change in score, but with a posttest T-score below 60. These students showed a significant improvement, but remained in the “Need for Instruction” or “Typical” categories mentioned above. A score is described as “optimal” when there is a significant change in score, and the posttest T-score is 60 or above, also known as the “Strength” category. The DESSA recommends this approach for program evaluation, as the ideal outcome for a program is to have students display statistically significant improvements and ultimately move into the “Strength” category within the various DESSA competencies. This benchmark is used to describe the magnitude of individual-level change; results for this benchmark are presented in Exhibit 11.

S E C T I O N 3 : A n a l y s i s


2. Percentile scores: The DESSA also provides percentile scores, which indicate the percentage of students in the norming sample3 who earned the same or lower raw score. Because the underlying distribution of DESSA scores is on a normal curve, a given pre-post change will have much different interpretations at different places on the percentile distribution (e.g., a five-point change from the 90th to 95th percentile is a far greater change than it is from the 50th to 55th percentile). Because these percentile scores have different meanings at different points in the distribution, we do not use them in calculations. Instead, whenever we compare DESSA findings to norms, we use the Need/Typical/Strength categories that are derived from T-scores.

The DESSA provides an overall composite score that provides an overall indication of a child’s social-emotional competencies. The DESSA also includes eight items that can be added together to produce a DESSA-mini score. We explored the eight-item DESSA-mini results’ alignment to the full DESSA. If the DESSA-mini could accurately capture both the scores and pre-post change relative to the full DESSA, then it may represent a valuable SEL measure for AARP Foundation that could be implemented with low burden.

3 A norming sample is the sample of students used to establish national averages. These samples are critical for interpreting DESSA findings because there are no right or wrong answers on the assessment. The DESSA norming sample consisted of 2,500 children who are representative of the U.S. population with respect to gender, race, ethnicity, region and poverty status. Because DESSA compares individual students to national norms, it is considered to be a norm-referenced assessment. This is in contrast to a criterion-referenced assessment, which assesses abilities by posing questions that have right or wrong answers.

S E C T I O N 4 : S E L O u t c o m e s


4. SEL Outcomes

4.1. Overall

Exhibit 5 presents the overall DESSA Composite scores, DESSA-mini scores, and DESSA scale scores from fall 2018 and spring 2019. Of the 101 students who had valid scores in both the fall and spring, their DESSA Composite T-scores increased by 4.72 points, which represents an effect size of .47. This change from fall to spring was statistically significant (p<.0001), as measured by a paired t-test. The DESA-mini score, however, did not show a statistically significant improvement from fall 2018 to spring 2019 (p<.16). It is evident from the comparison of DESSA Composite and DESSA-mini scores that the difference in scores was most pronounced in the fall; spring 2019 DESSA and DESSA-mini scores were much closer to one another.

On every subscale of the full DESSA, we observed a statistically significant improvement from fall 2018 to spring 2019 (p<.01). The strongest positive changes were in Personal Responsibility (+8.23 points), Relationship Skills (+6.40 points), and Decision Making (+4.82 points).

Exhibit 5. DESSA Composite and individual scale scores showed significant improvements from fall 2018 to spring 2019 for Experience Corps students

Measure n Fall Spring Mean Change p

Mean SD Mean SD

DESSA Composite 101 45.31 9.83 50.03 10.00 +4.72 <.0001

DESSA-mini 101 49.92 11.28 51.25 10.20 +1.33 .1608

Scale Scores

Decision Making 101 46.44 8.71 51.26 9.79 +4.82 <.0001

Goal-Directed Behavior 101 46.27 10.01 50.22 10.11 +3.95 <.0001

Optimistic Thinking 101 42.94 11.26 46.01 10.56 +3.07 .0028

Personal Responsibility 101 40.07 9.61 48.30 11.84 +8.23 <.0001

Relationship Skills 101 45.76 8.77 52.16 9.27 +6.40 <.0001

Self-Management 101 48.81 9.20 51.00 9.50 +2.19 .0084

Self-Awareness 101 48.25 10.64 50.84 9.30 +2.59 .0022

Social Awareness 101 48.85 9.28 51.51 10.35 +2.66 .0047

Source: Fall and spring DESSA scores; only students with valid fall and spring data are included.



Exhibit 6. The percentage of Experience Corps students who were classified in need on the DESSA social-emotional composite dropped by nearly half from fall 2018 to spring 2019

As noted above, the DESSA’s T-scores can be transformed into a categorical measure of need. The fall 2018 DESSA revealed that 38% of students had a Need for Instruction, meaning that they were at risk of developing or exhibiting social-emotional problems. By spring 2019, 20% of students remained in need. Considering that the national norm in the Need for Instruction category is 16%, it is evident that students who received Experience Corps tutoring improved substantially, from collectively being a high-risk group to converging to national averages within one year. The percentage of students in the Strength category on the DESSA Composite more than doubled, from 7% to 17%, also converging to national norms.

4.2. Student Subgroups

Exhibit 7 provides a breakdown of pre-post change in DESSA Composite scores by grade level. Kindergarteners in the sample improved the most, with an improvement of 6.5 points on the DESSA Composite T-score. First graders showed the second-best improvement (+4.6 points), followed by third graders (+4.4 points) and then second graders (+3.7 points). Students in earlier grades seemed to gain the



most on the DESSA Composite, even though kindergarteners began the school year with the highest DESSA Composite scores and therefore had the least room for improvement.4

Exhibit 7. Improvements in social-emotional composite scores were higher in early elementary grades K-1 relative to grades 2-3

n=98; Results for three students in grade 4 were masked because cell sizes were less than 5. The sample included 27 kindergarteners, 22 first graders, 33 second graders and 16 third graders.

4.3. Variation by Competency Area

Exhibit 8 presents the percentage of students in each DESSA category (i.e., Need for Instruction, Typical, Strength) for each subscale of the DESSA. The percentage of students needing instruction dropped on every subscale of the DESSA, with particularly strong declines in need on Personal Responsibility, Relationship Skills, and Decision Making. Likewise, the percentage of students indicating a strength on each subscale improved from the fall to spring, with the exception of Self-Management.

4 We also explored subgroup findings by tutoring model. Although there was a statistically significant difference at baseline between students who received one-to-one tutoring in Minneapolis/St. Paul and students who received small-group tutoring in Revere, both programs had a similar rate of improvement on the DESSA Social-Emotional Composite. There is not enough information to make inferences about the relative effectiveness of one-to-one tutoring versus sustained small-group tutoring.



Exhibit 8. The percentage of Experience Corps students needing instruction dropped on every subscale of the DESSA Composite

n=101; percentages may not add to 100% due to rounding

Exhibit 9 presents a breakdown of individual student performance on the DESSA Composite from fall 2018 to spring 2019. For the DESSA Composite and each of its subscales, the percentage of students who exhibited “significant positive change” by DESSA standards is presented, along with the percentage of students who exhibited a significant negative change and the percentage of students who did not experience a significant change in either direction. Over half (53%) of students experienced a significant positive change on the DESSA Composite from fall 2018 to spring 2019. The percentage of students



experiencing significant positive change was highest for Personal Responsibility and Relationship Skills, and lowest for Self-Awareness. The thresholds for significant improvement are much higher for individual subscales, since there is more measurement error in those scales than in the DESSA Composite. For example, the threshold for a “significant positive change” on the DESSA Composite is an improvement of +3 points (T-score), but the threshold for a “significant positive change” on the Self-Awareness scale is between +7 and +12 points, depending on the value of the pretest (LeBuffe et al., 2014, p. 104).

Exhibit 9. Over half of Experience Corps students experienced a significant positive change on the DESSA Composite from fall 2018 to spring 2019




Exhibit 10. Over half of Experience Corps students made a favorable or optimal change on the DESSA Composite


Exhibit 10 presents the distribution of individual change in DESSA findings by the clinical meaningfulness of change, as defined by DESSA developers. Over half of students in the pre-post sample (n=101) experienced a “favorable” or “optimal” change on the DESSA Composite. More specifically, about 40% of students experienced a clinically favorable outcome and about 11% experienced a clinically optimal outcome on the DESSA Composite score from fall 2018 to spring 2019. Students experienced the highest amount of clinically optimal change within the Personal Responsibility and Decision Making competencies, and the lowest within the Self-Awareness competency. Students who experienced a significant positive change and scored in the “Strength” category in the fall for any of the DESSA competencies are not included in the DESSA manual definition of clinically meaningful change.



4.4. DESSA-mini

As noted in Exhibit 5, improvements in DESSA-mini T-scores from fall 2018 to spring 2019 were not statistically significant. Even though the DESSA-mini showed a positive movement in T-scores, from 49.92 in the fall of 2018 to 51.25 in spring 2019, this finding did not reflect the substantial positive improvements in Experience Corps students’ social-emotional functioning observed elsewhere on both the DESSA Composite and all DESSA subscales.

There was a very strong correlation observed between the DESSA Composite and DESSA-mini in the fall of 2018 (r=.90103; p<.0001) and in the spring 2019 administration (r=.93438; p<.0001). It remains unclear why the DESSA-mini proved to be a less sensitive measure of change than the DESSA Composite. One possible explanation was the spread in scores between the DESSA Composite and the DESSA-mini in the fall 2018 administration: DESSA-mini scores were 4.61 points higher on the T-score (nearly half a standard deviation) relative to the DESSA Composite. The spring administration provided a much closer spread between DESSA Composite and DESSA-mini scores (1.22 points).

S e c t i o n 5 : A c a d e m i c O u t c o m e s


5. Academic Outcomes

5.1. Reading (SSIS)

Exhibit 11 presents the first of two items from the Social Skills Improvement System-Social and Emotional Learning (SSIS-SEL) scale. This exhibit presents teacher-reported ratings of student reading performance relative to grade-level expectations, and uses the same sample as the DESSA analysis to facilitate comparisons across these assessments. In fall 2018, nearly half of Experience Corps students (46%) were rated as having reading performance in the lower 35th percentile relative to grade-level expectations. By spring 2019, about a third of students (33%) remained in the bottom 35th percentile. The fall-to-spring change on this SSIS-SEL measure shows a consistent movement from lower to higher percentile ranks across the categories. Changes between fall and spring were statistically significant (p<.0001).

Exhibit 11. Experience Corps students’ performance in reading improved significantly relative to grade-level expectations, according to teachers

n=101; differences between fall and spring distributions are statistically significant (p>.0001)



5.2. Motivation to Succeed (SSIS)

Exhibit 12 presents teacher-reported percentile ranks of Experience Corps students’ motivation to succeed relative to grade-level expectations. In fall 2018, 30% of students were placed by teachers in the bottom 35th percentile based on their motiviation, while this figure dropped to 20% by spring 2019. This fall-spring change in motivation to succeed was statistically significant (p<.0001). Although the shift in motivation to succeed between fall 2018 and spring 2019 was not as substantial as students’ reading performance, it was also clear from comparing these two SSIS-SEL items that Experience Corps students’ motivation to succeed outpaced their reading performance, which would naturally make tutoring the appropriate intervention.

Exhibit 12. Experience Corps students’ motivation to succeed improved — and was ranked higher by teachers than their academic performance

n=101; differences between fall and spring distributions are statistically significant (p>.0001)



To examine the predictors of performance on the DESSA Composite, spring DESSA scores were modeled controlling for fall DESSA Composite scores, grade level, tutoring type, teacher-reported performance in reading from the SSIS-SEL, and teacher-reported motivation to succeed from the SSIS-SEL (Exhibit 13). The only factor — besides the fall pretest — that emerged as a significant predictor of DESSA performance in the spring was a student’s motivation to succeed. This linkage between motivation and social-emotional development should be investigated further, and suggests that improving motivation may be the key to Experience Corps’ success in improving social-emotional outcomes.

Exhibit 13. Motivation to succeed emerged as a significant predictor of social-emotional improvement

Measure DF Type III SS F P DESSA Composite: Fall 1 566.92 9.40 .0029 Grade Level 4 283.30 1.17 .3277 Type of Tutoring (one-to-one vs. small group) 1 4.99 0.08 .7742 Teacher-Reported Reading Performance: Fall 2018 4 6.02 0.02 .9988 Teacher-Reported Motivation to Succeed: Fall 2018 4 610.53 2.53 .0461

Dependent variable: Spring 2019 DESSA Composite; R2 = 0.481942

S e c t i o n 6 : R e c o m m e n d a t i o n s a n d C o n c l u s i o n s


6. Recommendations and Conclusions

6.1. Challenges and Successes

Findings from the Experience Corps SEL Evaluation were compelling: statistically significant improvements were observed on the DESSA Composite and on all eight social-emotional subscales of the DESSA during the 2018-19 school year. The strongest positive changes in subscale results were observed in Personal Responsibility, Relationship Skills and Decision Making. These findings logically align to our understanding of the Experience Corps model, which provides classrooms with a volunteer tutor — a positive adult presence who can monitor behavior, form relationships with students, serve as a mentor, and tutor students in reading. Substantial improvements in Personal Responsibility were observed on items related to students remembering information, handling their belongings with care, serving an important role at home or school, and following rules (see Appendix A for item-level findings). The extra attention and support students receive from volunteer tutors provides a source of accountability, supervision, and opportunities to practice reading where failure comes without consequences. Improvements in Relationship Skills were largely driven by items assessing students’ behaviors in greeting people in a polite way, attracting positive attention from adults, and making suggestions or requests in a polite way. Improvements in these items may align well with the Experience Corps model, as students receive positive attention from volunteer tutors and students may be inclined to be more polite to unfamiliar, external adults as compared to a classroom teacher. Improvements in Decision Making were largely driven by students being able to learn from experience and show the ability to decide between right and wrong. Again, these may to be aligned well with the Experience Corps model, as volunteer tutors guide students in their comprehension of words and meanings of passages.

Relative to national norms, students in Experience Corps started the school year at higher risk of developing social-emotional problems and ended the year close to national norms. Reductions in Need for Instruction by almost 50% in a single year indicated a substantial transformation of students’ SEL skills. Reading skills and motivation to succeed, as measured by the SSIS-SEL, also improved significantly. In particular, higher motivation to succeed was linked to improvements in reading performance. Although the DESSA does not have items or a scale that explicitly mentions motivation, the Experience Corps annual survey data collection has established that the majority of Experience Corps students improve in motivation over the course of the school year, and teachers attribute at least part of this change to the presence of volunteer tutors. If this is true, Experience Corps would be providing the foundation for both enhanced reading performance and social-emotional development — which will pay dividends throughout a student’s academic career.

Findings from the DESSA-mini showed improvement in SEL, but these improvements were not statistically significant. Although the DESSA-mini was shown to be highly correlated to the DESSA Composite, it did not appear to be sufficiently sensitive in capturing what were otherwise substantial positive improvements in SEL.

The analytic sample used in this analysis (n=101) represents a substantial drop-off in students for which at least some DESSA information was provided. Altogether, there were 172 students who had valid pretest or posttest data according to the DESSA’s strict missing data requirements. We compared DESSA Composite and DESSA subscale T-scores for the full sample (n=145 in fall 2018; n=127 in spring 2019) with the pre-post sample at each data collection time point (Appendix B). All differences between the analytic sample and the full sample were not statistically significant. We therefore do not consider the drop-off in the sample to represent a threat to the validity of the findings presented in this report.

Readers should keep in mind that these results were obtained by a single-group pretest-posttest design. Because a comparison group was not available to establish what would have happened in the absence of Experience Corps implementation, these results should be considered to provide cautiously optimistic



evidence about the relationship between Experience Corps tutoring and students’ social-emotional development. However, the DESSA-established national norms did provide a means of comparison, and our observation of findings moving substantially relative to these national norms provides assurance that the changes observed in this report were likely attributable to Experience Corps.

6.2. Future Implementation

Given that the 72-item DESSA needs to be completed by teachers (or parents) — individually on each student — the administration of the DESSA on a class-wide basis seems to be prohibitively expensive in terms of human, financial and time resources. The logical alternative to collecting the full DESSA would be to administer the DESSA-mini; however, this assessment proved to be less sensitive than the full DESSA in capturing improvements in students’ SEL functioning. AARP Foundation should consider several alternative approaches to collecting SEL data:

1. Administer the DESSA-mini to all Experience Corps students: Although we have found that the DESSA-mini was not sufficiently sensitive to change in SEL outcomes, it nonetheless showed improvements in students’ social-emotional learning. Moreover, the DESSA-mini was also highly correlated to the DESSA Composite scores (with a correlation of .90 on the pretest and .93 on the posttest). The DESSA-mini also has strong psychometric properties, and is therefore still worthy of consideration despite its challenges with sensitivity. In addition, the DESSA-mini was also administered to a larger number of students as part of the SIF impact evaluation, so the Abt team will be able to look carefully at those results across this larger sample.

2. Administer the full DESSA to a random sample of students: Random sampling could substantially reduce the cost of administering the DESSA, especially if random sampling was limited to one or two students per classroom. Still, adding additional burden to teachers, even 10 minutes per student as required by the DESSA, may compromise Experience Corps’ relationships with teachers.

3. Administer a subset of individual DESSA subscales to students: Given that we know Experience Corps students showed the greatest improvements in DESSA’s Personal Responsibility, Relationship Skills and Decision Making subscales, these may be best aligned to the Experience Corps model and worthy of tracking in the future. Even if a few subscales were chosen and burden was reduced by half, the five minutes per student needed to assess social-emotional development may still prove to be prohibitively burdensome to teachers.

4. Administer a subset of DESSA-mini items: Since the most substantial positive changes in social-emotional functioning were found in Personal Responsibility, Relationship Skills, and Decision Making, individual items under these three competencies could be administered to all Experience Corps students. This would have two major disadvantages: (1) We already know that the DESSA-mini is less sensitive to positive change (at least in this evaluation) than the DESSA Composite, so breaking down the DESSA-mini may diminish the sensitivity of SEL measurement even further, and (2) the reconstituted composite measure would have to be assessed for internal consistency and reliability — and it likely will not perform as well as the DESSA-mini. Still, if AARP Foundation wishes to measure SEL for all of its students, this may be the only feasible option to keep burden to an absolute minimum.

5. Find or develop a new SEL measure: The DESSA, SSIS-SEL, and Behavioral and Emotional Rating Scale Second Edition (BERS-2) all claim to have administration times of 10 minutes per student. Although these full instruments would be prohibitively expensive to administer for all Experience Corps students, AARP Foundation could customize a SEL measure that is tailored to the Experience Corps model. This would involve substantial cost, as the new measure would need



to be field tested. Still, it may provide AARP Foundation an opportunity to align a SEL measure to its theory of change instead of relying on broad measures that are likely less sensitive in measuring change brought about by the Experience Corps model.

6.3. Conclusion

Given the substantial improvements observed in students’ social-emotional development, it will be important for Experience Corps staff to both understand and track these data in the future. These data can be used to inform training materials, session structure guidance, and general policy. Given that Experience Corps’ annual surveys have shown that tutoring provides reciprocal benefits to tutors in addition to students, AARP Foundation should also consider assessing social-emotional measures of its volunteer tutors in the future.

S E C T I O N 7 : R e f e r e n c e s


7. References

DeWit, D. W., DuBois, D., Erdem, G., Larose, S., & Lipman, E. L. (2016). The role of program-supported mentoring relationships in promoting youth mental health, behavioral and developmental outcomes. Prevention Science, 17(5), 646-657.

DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12(2), 57-91.

Durlak, J. A., Weissberg, R. P. (2007). The impact of after-school programs that promote personal and social skills. Chicago, IL: CASEL.

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta‐analysis of school‐based universal interventions. Child Development, 82(1), 405-432.

LeBuffe, P.A., Shapiro, V.B., & Naglieri, J.A. (2014). Devereux Student Strengths Assessment: A measure of social-emotional competencies of children in kindergarten through eighth grade. Fort Mill, SC: Aperture Education.

Taylor, R. D., Oberle, E., Durlak, J. A., & Weissberg, R. P. (2017). Promoting positive youth development through school-based social and emotional learning interventions: A meta-analysis of follow-up effects. Child Development, 88, 1160-1167.

RAND Education Assessment Finder. (2018) Devereux Student Strengths Assessment-Mini (DESSA-mini). Retrieved from: https://www.rand.org/education-and-labor/projects/assessments/tool/2009/devereux-student-strengths-asessment-mini-dessa-mini.html

A P P E N D I X A : S u p p o r t i n g L i t e r a t u r e


Appendix A: Supporting Literature

Experience Corps engages older adults to serve as literacy tutors and mentors to at-risk students in grades K-3, aiming to enhance both academic outcomes and students’ social-emotional development. Although Experience Corps’ primary intent is to increase the reading proficiency of elementary-age students, the presence of a tutor provides social-emotional support and a source of mentorship for students. Given the overlap of tutoring, mentoring, and social-emotional support provided by the Experience Corps model, we have reviewed literature on both school and after-school based social-emotional learning (SEL) interventions and mentoring-specific programs.

A.1. School- and After-School-Based SEL Interventions

We reviewed two large scale meta-analyses that have been conducted over the last decade that focused on school-based SEL interventions.5 The first of the two (Durlak, Weissberg, Dymnicki, Taylor & Schellinger, 2011) focused on three independent variables: the intervention format, the inclusion of recommended practices for skill development, and the occurrence of implementation problems. The recommended SAFE — Sequenced, Active, Focused, Explicit — practices are described as including connected and coordinated sets of activities to achieve objectives (sequenced), active forms of learning (active), at least one component devoted to developing personal or social skills (focused), and targeting specific SEL skills (explicit) (Durlak et al., 2011, p. 408). The majority of included programs were classroom-based, delivered by teachers (53%) or non-school personnel (21%), and over half were implemented with elementary-age students (56%) (Durlak et al., 2011).

The outcomes were broken down into six categories including social-emotional skills, attitudes toward self and others, positive social behaviors, conduct problems, emotional distress, and academic performance. The results of the first meta-analysis show the mean effect sizes for all six outcomes as statistically significant, with SEL skills showing the largest improvement.6 The effect size for SEL skills was strong for teacher-led classroom-based interventions, and it was even higher for classroom-based interventions led by non-school staff.7 Moreover, programs following SAFE practices showed significant effects for all outcomes, whereas programs not coded as including SAFE practices only showed significant effects for three outcomes (not including SEL skills). In addition, programs that experienced implementation problems showed significant effects for only two outcomes (not including SEL skills), whereas programs that did not experience implementation problems showed significant effects for all outcomes (Durlak et al., 2011).

The second meta-analysis (Taylor, Oberle, Durlak, & Weissberg, 2017) demonstrated similar results. Each program included in this meta-analysis had to target at least one of the five CASEL competency domains, and 89% of the programs were rated as having SAFE practices. Most of the programs were classroom-based and included structured group lessons of approximately 30-45 minutes. Outcomes were measured using similar categories, with the addition of substance abuse as a seventh outcome. Similarly, there was improvement in all seven outcomes, with SEL skills as the second highest.8 In addition, the 31

5 Durlak, Weissberg, Dymnicki, Taylor, & Schellinger (2011) includes 213 interventions and 270,034 students (K-12) and Taylor, Oberle, Durlak, & Weissberg (2017) includes 82 interventions and 97,406 students (K-12).

6 The mean effect sizes for all six outcomes range from .22 to .57 with SEL skills at .57.

7 The effect size for SEL skills is .62 for class by teacher programs and .87 for class by non-school-personnel programs.

8 The mean effect sizes ranged from .13 to .33, with SEL skills at .23 — which was second to academic performance at .33.



interventions focused on childhood (age 5-10) had stronger effects than interventions delivered to older students (Taylor et al., 2017).9

The results of these meta-analyses, especially effects on social-emotional development, align well to the Experience Corps model. First, the Experience Corps model is focused on elementary-age students, which Taylor et al. (2017) showed to be correlated with higher effect sizes across outcomes. The Experience Corps program also includes components outlined by the SAFE practices described by Durlak et al. (2011), as the tutoring sessions follow a prescribed session structure, include opportunities for students to take an active role in their own learning, focus on components such as students’ reflection of success, and map to the development of specific SEL skills. Beyond this, the Experience Corps program uses implementation monitoring, through the use of session logs, to ensure that tutors are implementing the model with fidelity and that student progress is being documented. In addition, although sustained small-group and one-to-one tutoring is usually conducted outside the classroom (or on the perimeter of the classroom), it is being delivered by non-school staff during the school day, which may speak to the higher effect sizes found by Durlak et al. (2011) for classroom-based interventions delivered by non-school staff.

There has also been literature linking after-school programs (ASPs) that include SAFE practices to positive outcomes, including self-confidence, self-esteem and positive social behaviors. CASEL conducted a meta-analysis of 73 ASPs that are designed to promote personal and social skills. They found that programs that used SAFE practices demonstrated significant positive results on all measured outcomes, while programs that did not use SAFE practices did not demonstrate significant positive results on any outcome (Durlak & Weissberg, 2007).10 Again, these SAFE practices can be clearly seen in the Experience Corps model, with a focus on sequential leveled-reader systems and/or curricula and designated time for students to actively contribute to reader-centered conversations that are designed to strengthen reading comprehension, critical thinking and vocabulary. Moreover, the structured tutoring session guide prescribes time focused on skill building related to reading fluency and quality talk and provides opportunities for reflection and feedback.

A.2. Mentoring Programs

As described in Rhodes’ model of mentoring (DuBois, Portillo, Rhodes, Silverthorn, & Valentine, 2011), mentoring relationships can be very powerful for youth in how they develop a strong interpersonal foundation, which in turn contributes to internal development, including social-emotional development. By modeling positive relationships, mentors can demonstrate that positive relationships with adults are possible, and therefore have an effect on how youth perceive and understand relationships more broadly. The Experience Corps tutoring sessions allow for students to engage in positive and supportive relationships with tutors through social and educational experiences, which in turn may affect how these students engage with others in their lives. Moreover, mentees can learn how to better understand and express their emotions through communication with mentors. For example, the Experience Corps tutoring session structure allows students to reflect on their reading successes, which is an opportunity for students to both process and communicate their emotions.

DuBois et al. (2011) conducted a meta-analysis focused on 73 youth mentoring programs. Findings showed multiple program variables affect the outcomes of youth participating in mentoring programs, including social/relational and psychological/emotional outcomes. For the purpose of this meta-analysis, mentoring is defined as “a program or intervention intended to promote positive youth outcomes via

9 The 31 included interventions focused on childhood (age 5-10) had a mean effect size of .27, which is higher than those focused on early adolescence (.12) and adolescence (.18).

10 The mean effect sizes ranged from .24 to .35 for programs that used evidence-based approaches.



relationships between young persons (18 years old and younger) and specific non-parental adults (or older youth) who are acting in a nonprofessional helping capacity” (DuBois et al., 2011, p. 66). These variables include programs serving youth with greater levels of risk (both individual and environmental risk), programs with mentors who have backgrounds that are closely aligned with the program’s goals, and programs that include a teaching role for mentors. In the Experience Corps model, there is evidence of all three of these components, in that youth are selected for the program on the basis of their reading level and mentors are specifically focused on teaching these youth how to read. In addition, Experience Corps does attract some adult volunteers with backgrounds in education and youth development.

DeWit, DuBois, Erdem, Larose, & Lipman (2016) also discuss the importance of adult mentors in improving youths’ relationships, self-esteem, emotion regulation, and stress management. The authors examined self-report data from 859 youth in Big Brothers/Big Sisters programs and found that youth in both long-term mentoring relationships and continuous mentoring relationships for less than 12 months had better behavioral and social outcomes than youth who were never mentored. Although the Big Brothers/Big Sisters mentoring model differs from the Experience Corps model, there are some parallels in that Experience Corps includes one-to-one mentoring relationships for some participating students, Experience Corps mentors undergo training by Experience Corps program staff, youth participating in the Experience Corps program generally stay in the program for the full school year, and there is an emphasis on mentor-mentee relationship building within each tutoring session.

A P P E N D I X B : I t e m - L e v e l F i n d i n g s


Appendix B: Item-Level Findings From the DESSA

Fall 2018 Spring 2019

Variable Label N Mean Std Dev N Mean Std Dev Decision Making

DESSA_37 37. Follow the example of a positive role model? 101 2.68 0.81 101 2.74 0.88

DESSA_39 39. Accept responsibility for what she/he did? 100 2.58 0.93 101 2.75 0.91 DESSA_42 42. Show good judgement? 101 2.68 0.82 101 2.78 0.83 DESSA_52 52. Seek advice? 101 2.23 1.02 101 2.49 0.93 DESSA_65 65. Learn from experience? 101 1.68 1.33 100 2.75 0.82 DESSA_66 66. Follow the rules of a trusted adult? 101 2.90 0.76 100 2.88 0.83

DESSA_68 68. Show the ability to decide between right and wrong? 101 1.59 1.33 100 2.80 0.83

DESSA_69 69. Use available resources (people or objects) to solve a problem? 101 2.45 0.89 101 2.55 0.82

Goal-Directed Behavior

DESSA_03 3. Keep trying when unsuccessful? 101 2.41 0.96 99 2.56 0.87 DESSA_09 9. Take steps to achieve goals? 101 2.41 0.95 101 2.70 0.84 DESSA_12 12. Try to do her/his best? 101 1.57 1.34 101 2.85 0.78

DESSA_13 13. Seek out additional knowledge or information? 101 2.41 1.05 101 2.60 1.01

DESSA_14 14. Take an active role in learning? 101 2.50 0.98 101 2.63 0.90 DESSA_15 15. Do things independently? 101 2.50 1.02 101 2.71 0.95

DESSA_18 18. Ask to take on additional work or responsibilities? 101 1.93 1.17 101 2.22 1.11

DESSA_26 26. Show creativity in completing a task? 101 2.27 0.94 101 2.57 0.86 DESSA_29 29. Seek out challenging tasks? 101 1.90 1.15 101 2.18 1.03 DESSA_33 33. Work hard on projects? 101 2.56 0.97 100 2.74 0.86 Optimistic Thinking

DESSA_02 2. Carry herself/himself with confidence? 101 2.47 0.83 101 2.70 0.82 DESSA_05 5. Say good things about herself/himself? 101 2.47 0.92 99 2.67 0.78 DESSA_07 7. Speak about positive things? 101 2.65 0.97 101 2.82 0.80

DESSA_10 10. Look forward to classes or activities at school? 101 2.73 0.85 101 2.86 0.75

DESSA_16 16. Say good things about his/her classmates? 101 2.50 0.90 101 2.76 0.80 DESSA_30 30. Say good things about the future? 101 2.27 0.98 101 2.57 0.78

DESSA_36 36. Express high expectations for himself/herself? 101 2.23 1.00 101 2.50 0.88

Personal Responsibility

DESSA_01 1. Remember important information? 101 1.41 1.35 101 2.63 0.86 DESSA_04 4. Handle his/her belongings with care? 101 1.84 1.23 101 2.78 0.88 DESSA_06 6. Serve an important role at home or school? 101 1.70 1.36 100 2.72 0.77 DESSA_20 20. Encourage positive behavior in others? 101 2.36 1.04 98 2.58 0.92

DESSA_21 21. Prepare for school, activities, or upcoming events? 100 2.46 0.98 100 2.75 0.97

DESSA_23 23. Do routine tasks or chores without being reminded? 101 2.54 1.02 100 2.69 0.98

DESSA_24 24. Act as a leader in a peer group? 101 2.09 1.10 101 2.28 1.01 DESSA_28 28. Get things done in a timely fashion? 100 2.40 1.05 101 2.56 0.99

DESSA_32 32. Show care when doing a project or school work? 101 2.70 0.93 101 2.79 0.85

DESSA_35 35. Follow rules? 101 1.75 1.29 101 2.87 0.80 Relationship Skills

DESSA_38 38. Compliment or congratulate somebody? 101 2.45 0.96 100 2.64 0.86 DESSA_40 40. Do something nice for somebody? 101 2.75 0.94 101 2.90 0.84

A P P E N D I X B : I t e m - L e v e l F i n d i n g s


Fall 2018 Spring 2019

Variable Label N Mean Std Dev N Mean Std Dev DESSA_45 45. Show appreciation of others? 101 2.55 0.90 101 2.82 0.75 DESSA_47 47. Greet a person in a polite way? 101 1.80 1.30 100 2.95 0.73 DESSA_50 50. Attract positive attention from peers? 101 2.45 0.91 100 2.72 0.84 DESSA_55 55. Express concern for another person? 101 2.55 0.93 101 2.70 0.84 DESSA_61 61. Attract positive attention from adults? 101 1.48 1.34 101 2.79 0.84

DESSA_64 64. Make a suggestion or request in a polite way? 101 1.76 1.33 101 2.78 0.78

DESSA_70 70. Offer to help somebody? 101 2.65 0.99 101 2.73 0.88 DESSA_71 71. Respond to another person's feelings? 101 2.64 0.83 101 2.87 0.80 Self-Management

DESSA_43 43. Pay attention? 101 2.59 0.94 101 2.67 0.88 DESSA_44 44. Wait for her/his turn? 101 2.87 0.81 100 2.94 0.83

DESSA_46 46. Focus on a task despite a problem or distraction? 101 2.27 1.04 101 2.48 0.99

DESSA_48 48. Act comfortable in a new situation? 101 2.55 0.78 101 2.64 0.74 DESSA_51 51. Perform the steps of a task in order? 101 2.64 0.92 101 2.84 0.91 DESSA_53 53. Think before he/she acted? 101 2.51 0.84 101 2.58 0.90

DESSA_54 54. Pass up something he/she wanted, or do something he/she did not like, to get 101 1.97 0.93 100 2.08 0.92

DESSA_56 56. Accept another choice when his/her first choice was unavailable? 101 2.61 0.81 99 2.57 0.87

DESSA_60 60. Stay calm when faced with a challenge? 101 2.61 0.88 101 2.58 0.86 DESSA_67 67. Adjust well to changes in plans? 101 2.78 0.77 101 2.82 0.77

DESSA_72 72. Adjust well when going from one setting to another? 100 1.82 1.34 100 2.85 0.77

Self-Awareness

DESSA_41 41. Make accurate statements about events in her/his life? 101 2.67 0.86 101 2.82 0.79

DESSA_49 49. Teach another person to do something? 101 2.18 1.06 99 2.47 0.90

DESSA_57 57. Ask questions to clarify what he/she did not understand? 101 2.39 1.10 99 2.57 1.00

DESSA_58 58. Show an awareness of her/his personal strengths? 101 2.24 0.85 101 2.50 0.80

DESSA_59 59. Ask somebody for feedback? 101 2.05 0.98 101 2.21 0.91 DESSA_62 62. Describe how he/she was feeling? 101 2.39 0.92 101 2.65 0.83 DESSA_63 63. Give an opinion when asked? 101 2.58 0.89 101 2.84 0.76 Social-Awareness

DESSA_08 8. Cope well with insults and mean comments? 101 2.15 0.94 101 2.36 0.88 DESSA_11 11. Get along with different types of people? 101 1.93 1.25 101 2.88 0.89 DESSA_17 17. Act respectfully in a game or competition? 101 2.87 0.80 101 2.91 0.85 DESSA_19 19. Respect another person's opinion? 101 2.71 0.83 100 2.74 0.91 DESSA_22 22. Contribute to group efforts? 101 2.60 1.00 101 2.69 0.85 DESSA_25 25. Resolve a disagreement? 100 1.99 0.89 101 2.11 0.98 DESSA_27 27. Share with others? 101 2.76 0.85 101 2.79 0.84 DESSA_31 31. Cooperate with peers or siblings? 101 2.76 0.76 99 2.87 0.82

DESSA_34 34. Forgive somebody who hurt or upset her/him? 101 2.61 0.80 101 2.69 0.86

A P P E N D I X C : S e n s i t i v i t y A n a l y s i s


Appendix C: Comparison of Pre-Post Sample With All Respondents

Experience Corps Students

with Valid Baseline and Follow-Up Data

All Respondents

Scale/Measure n Mean SD n Mean SD Difference p Fall 2018 DESSA Composite 101 45.31 9.83 147 45.46 10.46 -0.15 .9062 Decision Making 101 46.44 8.71 147 46.48 8.97 -0.04 .9718 Goal-Directed Behavior 101 46.27 10.01 147 46.48 10.91 -0.21 .8745 Optimistic Thinking 101 42.94 11.26 147 43.63 12.65 -0.69 .6586 Personal Responsibility 101 40.07 9.61 147 39.80 9.60 0.27 .8259 Relationship Skills 101 45.76 8.77 147 45.85 8.83 -0.09 .9385 Self-Management 101 48.81 9.20 147 48.98 10.19 -0.17 .8947 Self-Awareness 101 48.25 10.64 147 48.82 11.75 -0.57 .6941 Social Awareness 101 48.85 9.28 147 49.00 10.54 -0.15 .9090 Spring 2019 DESSA Composite 101 50.03 10.00 125 49.73 9.78 0.30 .8197 Decision Making 101 51.26 9.79 125 51.04 9.49 0.22 .8661 Goal-Directed Behavior 101 50.22 10.11 125 49.70 9.83 0.52 .7001 Optimistic Thinking 101 46.01 10.56 125 45.61 10.22 0.40 .7724 Personal Responsibility 101 48.30 11.84 125 48.02 11.62 0.28 .8619 Relationship Skills 101 52.16 9.27 125 51.92 9.03 0.24 .8455 Self-Management 101 51.00 9.50 125 50.74 9.30 0.26 .8338 Self-Awareness 101 50.84 9.30 125 50.44 9.23 0.40 .7462 Social Awareness 101 51.51 10.35 125 51.50 10.09 0.01 .9937

AARP Foundation601 E Street, NWWashington, D.C. 20049www.aarpfoundation.org

@AARPFoundationwww.facebook.com/AARPFoundation

AARP Foundation works to end senior poverty by helping vulnerable older adults build economic opportunity and social connectedness. As AARP’s charitable affiliate, we serve AARP members and nonmembers alike. Bolstered by vigorous legal advocacy, we spark bold, innovative solutions that foster resilience, strengthen communities and restore hope.

experience corps social-emotional learning evaluation€¦ · experience corps is affecting...

Documents