LEARNING FROM THE PAST: Using the 2009 Learn and Serve cluster

evaluation to develop the field’s understanding of service-learning

implementation and outcomes

Paul Baumann, Ph.D. Director, National Center for Learning and Citizenship

Shelley Billig, Ph.D. Vice President, RMC Research Corporation, Denver

Susan Abravanel Vice President of Education, Youth Service America (YSA)

Teri DaryCo-Chair, National Coalition for Academic Service-Learning

2012 IARSLCE Annual ConferenceBaltimore, MD

September 25, 2012

INTRODUCTION

Session Goals• Provide an overview of three years of findings

for three participants in the 2009 LSA cohort (NCLC, YSA, Wisconsin DPI)

• Begin to examine how implementation strategy might play a role in program outcomes

• Begin to develop cross-cutting findings.

LSA 2009 COHORTLSA 2009 cohort initially included a number of states and national programs:

• States: AZ, HI, IA, IL, LA, MI, MN, OH, and WI• National Programs: NCLC, YSA (focus on STEM, middle schools, high-poverty schools)

Purpose: Implement and evaluate service-learning and Pre-K 12 schools

Cohort members agreed to use common quasi-experimental evaluation design and use the same survey scales to measure a set of core common outcomes:• Academic engagement• Academic performance• Dropout prevention• Acquisition of 21st century skills

Coordinated evaluation would help to address several shortcomings of S-L research (Billig, 2001; 2008; 2011; Bradley, 2005):• Lack of methodological rigor: Quasi-experimental or experimental design necessary to

establish causality and determine effect sizes• Lack of adequate sample: Large, diverse sample necessary to establish more generalizable

findings across states and various student populations.

RESEARCH QUESTIONS

1. What was the impact of service-learning participation on students?:

a. Academic engagement; educational aspirations; acquisition of 21st century skills; civic dispositions; interest and abilities in STEM; and interest in careers that require STEM skills

b. Academic achievement, dropout rates, attendance, and discipline referral rates

2. What program design characteristics influenced outcomes?3. What is the relationship between each programs’ theory of action

and implementation strategy, and the student outcomes for that program?

4. What conclusions, if any, can we draw by looking across the evaluation findings of the three programs?

METHODOLOGY

• LSA sub-grantees from each state were required to participate if they received a certain level of funding.

• Each recruited a matched-comparison site for every two programs (e.g., one middle school site for every two middle school sites served).

• Matches were based on demographics (roughly the same SES and racial/ethnic percentages) and achievement scores (scores on the state assessment in ELA and math) coming to the school year.

METHODOLOGY

• In year one, a retrospective pre/post-student survey was used; in years two and three, a traditional pre/post-student survey was used.

• Teachers and community members answered a post-only survey.

• Achievement data were collected from the sites or state, including test scores in language arts, math and science, attendance, disciplinary data, and graduation/dropout rates.

SCHOOLS OF SUCCESSNational Center for Learning and Citizenship

Theory of Action:• Guide schools from policy to action, and action to policy

o Encourage high-quality practice and sustainability through policy and district/building leadership

o Encourage policy and district/building leadership through models of high-quality practice.

Expectation for Change:• Changes to district/school policy (broadly conceived)• District/schools will begin new or grow existing service-learning

programs• Increase middle school students’ engagement in STEM related

classes.

SCHOOLS OF SUCCESSNational Center for Learning and Citizenship

Context:• Nine middle schools

o CA, KY (3), MI, MS, NY (2), PAo Two Charter Schools (MI, PA)o Three KY schools, all in one districto Mixture of rural and urban

• All high poverty ( > 50% FARMS)• Some high performing, some low performing • Several experienced with service-learning (CA, PA); the

remainder beginning with service-learning.

SCHOOLS OF SUCCESSNational Center for Learning and Citizenship

Implementation Strategy:• Role of leadership

o Recognition of leadership at multiple levels of the schoolo Worked primarily with leaders from district/building level, as well as key teacher leaders.

• Professional developmento Three days of training annuallyo Focused both on implementation (how to do) as well as policy (how to encourage, lead,

fund, etc.).

• Sustainabilityo Focus on five elements that support sustainability:

Vision and leadership Curriculum and assessment Community and school partnerships Professional development Continuous improvement.

SCHOOLS OF SUCCESS FINDINGS

Service-Learning Quality Mean

SD

Link to Curriculum 3.40 .45Partnerships 3.23 .50Meaningful Service 3.54 .46Youth Voice 3.33 .61Respect for Diversity 3.25 .63Reflection 3.20 .47Progress Monitoring 3.08 .34

Teacher Perceptions of Service-Learning Quality in Their Classrooms (N = 6)

STUDENT RATINGS OF QUALITYService-Learning Quality N Mean SDOverall quality 285 3.02 .57I feel that my service-learning activities were meaningful. 253 3.13 .68My teacher made sure we linked service-learning activities to classroom subjects such as English, math, and/or science.

256 3.18 .72

The skills that I learned from my service-learning activities will be useful to me in the future.

246 3.11 .75

My service-learning activities were important to me. 248 3.02 .74I helped make decisions about my service-learning activities. 255 3.02 .70I helped come up with ideas for my service-learning activities. 261 3.00 .80

During my service-learning activities, we were expected to show respect for other people’s opinions.

255 3.22 .68

My class was asked to identify specific things we had learned during our service-learning activities.

241 3.00 .74

My class talked about several different ways to solve neighborhood or community problems.

237 2.85 .90

IMPACTSService-Learning Comparison

NPostsurvey

Mean NPostsurvey

Mean SignificanceAcademic Engagement 311 2.87 54 2.84 .673

Academic Competence 311 3.46 54 3.50 .533

Aspirations 299 3.62 54 3.74 .104

21st Century Skills 303 3.11 54 3.08 .603

Community Engagement 286 2.62 51 2.58 .523

STEM Content Area Interest and Abilities

292 2.95 53 2.94 .894

STEM Career Interest 274 2.72 52 2.67 .633

TEACHER PERCEPTION OF IMPACTSImpact area: N Mean SDImprovement in academic performance. 6 3.17 .41Increased engagement in school. 6 3.17 .75Improved school attendance. 6 3.00 .63Improved social skills. 6 3.33 .52Increased awareness of community issues. 5 3.40 .55Increased sense of belonging to the community. 6 3.33 .52Increased commitment to working on social or

community problems.5 3.40 .55

Increased belief that they can make a difference. 5 3.20 .84Increased ability to work with others with diverse

backgrounds.6 3.33 .52

Increased respect for others. 6 3.33 .52Increased empathy and desire to help. 5 3.40 .55Increased problem-solving skills. 6 3.33 .52Increased leadership skills. 6 3.17 .75Increased ability to collaborate with others. 5 3.40 .55Improved ability to express ideas. 5 3.40 .55Improved career skills. 5 3.40 .55

STUDENT PERCEPTIONS OF SKILLS ACQUIRED

N=319

Student Perceptions of Skills Acquired Through Participation in Service-Learning (N = 319)

Tutoring skills

Career awareness

Science skills

Reading skills

Writing skills

0 20 40 60 80 100 120 140 160

32

80

87

101

105

123

133

135

138

Number of Respondents

MODERATORS OF OUTCOME

Moderator Outcomes Moderated df F SignificanceStudent Ratings of Program Quality

Academic Engagement

1, 278 12.309 .001**

Academic Competence

1, 278 12.934 .000**

21st Century Skills 1, 273 7.047 .008**

Community Engagement

1, 264 5.163 .024*

*p < .05, **p < .01.

Teachers and classes in middle schools

19 states most seriously challenged by HS graduation rates

≥50% free/reduced lunch demographic

Target:

YSA STEMESTER OF SERVICE

Training and resources

Unique partnership opportunities

Ongoing personalized consultation

Networking

Rigorous evaluation component

WHAT IS A STEMESTER OF SERVICE?

YSA Support:

Focus on incorporating science, technology, engineering or mathematics content learning

Service project targeting environmental or disaster preparedness issues

Career and workforce readiness component

A continuous Semester of Service™ strategy

Content:

WHAT IS A STEMESTER OF SERVICE?

19

SEMESTER OF SERVICE™ Integrated service and learning activities

12-to-14 week “semester”

National/international service days

Design based on the “IPARD/C” model

www.YSA.org/resources

Measure

Service-Learning Students Comparison Students

Signif

Effect Size

Cohen’s d)N

PresurveyMean

PostsurveyMean N

PresurveyMean

PostsurveyMean

Academic Engagement

1,049 2.95 2.99 348 2.86 2.82 .033* .19

Academic Competence

1,048 3.45 3.46 348 3.42 3.40 .297 --

21st Century Skills

1,030 3.09 3.14 343 3.09 3.07 .044* .11

Civic Dispositions

1,013 3.12 3.15 337 3.02 3.05 .841 --

Aspirations 1,003 3.48 3.50 326 3.50 3.52 .946 --STEM interest in courses

1,007 3.03 3.06 341 2.97 2.86 .000***

.18

STEM career 935 2.88 2.91 310 2.89 2.76 .000***

.18

4-point scale where 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly Agree. *p < .05, ***p < .001.

YSA FINDINGS

Student Perceptions of Skills Acquired Through Participation in Service-Learning (N = 1,055)

Tutoring

Career Awareness

Work Experience

Computer

Job

Reading

Writing

Math

Science

0 5 10 15 20 25 30 35 40 45 50

9.2

18.9

26.5

29.5

31.8

34.3

34.6

40.6

44.3

Percentage

Skill

s

YSA FINDINGS

ModeratorOutcomes Moderated F Significance

Effect Size (Cohen’s d)

Student Ratings of Program Quality

(Positive Moderator) Academic Engagement

243.16 .000*** 1.04

(Positive Moderator) Academic Competence

178.89 .000*** .91

(Positive Moderator) Aspirations 74.23 .000*** .59

(Positive Moderator) 21st Century Skills

253.01 .000*** 1.06

(Positive Moderator) Civic Dispositions

285.34 .000*** 1.13

(Positive Moderator) STEM Interest in Courses

162.12 .000*** .86

(Positive Moderator) STEM Interest in Career

119.626 .000*** .72

Moderators of Outcomes: Student Ratings of Program Quality ***p < .001.

[1]

YSA FINDINGS

You can see from the previous slides that high-quality service-learning is very effective… but regular service-learning has a significant but weak effect.

Most of our field has a .03 effect size… much too low.

How can we improve?

YSA FINDINGS

Measure

Service-Learning Students Comparison Students

Signif

Effect Size

Cohen’s d)N

PresurveyMean

PostsurveyMean N

PresurveyMean

PostsurveyMean

Academic Engagement

1,049 2.95 2.99 348 2.86 2.82 .033* .19

Academic Competence

1,048 3.45 3.46 348 3.42 3.40 .297 --

21st Century Skills

1,030 3.09 3.14 343 3.09 3.07 .044* .11

Civic Dispositions

1,013 3.12 3.15 337 3.02 3.05 .841 --

Aspirations 1,003 3.48 3.50 326 3.50 3.52 .946 --STEM interest in courses

1,007 3.03 3.06 341 2.97 2.86 .000***

.18

STEM career 935 2.88 2.91 310 2.89 2.76 .000***

.18

4-point scale where 1 = Strongly Disagree, 2 = Disagree, 3 = Agree, and 4 = Strongly Agree. *p < .05, ***p < .001.

Student Perceptions of Skills Acquired Through Participation in Service-Learning (N = 1,055)

Tutoring

Career Awareness

Work Experience

Computer

Job

Reading

Writing

Math

Science

0 5 10 15 20 25 30 35 40 45 50

9.2

18.9

26.5

29.5

31.8

34.3

34.6

40.6

44.3

Percentage

Skill

s

ModeratorOutcomes Moderated F Significance

Effect Size (Cohen’s d)

Student Ratings of Program Quality

(Positive Moderator) Academic Engagement

243.16 .000*** 1.04

(Positive Moderator) Academic Competence

178.89 .000*** .91

(Positive Moderator) Aspirations 74.23 .000*** .59

(Positive Moderator) 21st Century Skills

253.01 .000*** 1.06

(Positive Moderator) Civic Dispositions

285.34 .000*** 1.13

(Positive Moderator) STEM Interest in Courses

162.12 .000*** .86

(Positive Moderator) STEM Interest in Career

119.626 .000*** .72

Moderators of Outcomes: Student Ratings of Program Quality ***p < .001.

[1]

WISCONSIN DEPARTMENT OF PUBLIC INSTRUCTION

Theory of Action:• Create infrastructure of support

o Build capacity for quality and sustainability through district infrastructure and leadership

o Support quality practice through cascade professional development.

Expectation for Change:• Districts will infuse service-learning as a strategy to achieve

district goals• District will build an infrastructure of support and leadership

to guide practice in local schools.

WISCONSIN DEPARTMENT OF PUBLIC INSTRUCTION

Context:• Up to 10 districts funded per year for a maximum of 6

yearso Up to $20,000 grants per yearo Fundable expenses must support primarily infrastructure,

leadership, curriculum integration, and professional development.

• Geographic and demographic diversity• Collaborative network of support.

SCHOOLS OF SUCCESSNational Center for Learning and Citizenship

Implementation Strategy• Leadership

o Build capacity for leadership at teacher, school, and district levelso Increase capacity for teachers to serve as leaders within their school/district and

at the state level• Professional development

o Initial training by DPI during first semester of fundingo Ongoing professional development through in-person and online environments

• Sustainabilityo All grants must demonstrate progress on five elements that support

sustainability Vision and leadership Curriculum and assessment Community and school partnerships Professional development Continuous improvement

Wisconsin Findings

• Will add this week.

Academic Engagement (7 items) GRADES 3-5 Pre sd Post sd Dif .003

Service-Learning 3.43 0.56 3.26 0.54 -0.17 Comparison 3.25 0.48 3.25 0.45 0.00

Academic Competence (7 items)

.002

Service-Learning 3.19 0.51 3.13 0.48 -0.06 Comparison 3.14 0.51 3.11 0.48 -0.03

Community Attachment (3 items)

.009*

Service-Learning 3.07 0.65 3.01 0.62 -0.06 Comparison 3.00 0.64 2.96 0.62 -0.04

Civic Skills (6 items) .011*Service-Learning 3.06 0.59 3.07 0.55 0.01 Comparison 3.08 0.56 3.01 0.56 -0.07

Civic Dispositions (4 items)

.001

Service-Learning 3.32 0.55 3.31 0.59 -0.01

Comparison 3.33 0.51 3.30 0.53 -0.03

N

Beginning of Year

End of Year

Average Difference g

2Mean SD Mean

SD

Academic Engagement 2.87 0.48 2.89 0.52 0.02 .001

Academic Competence 3.16 0.52 3.14 0.54 -0.02 .001

Community Attachment 2.44 0.51 2.50 0.53 0.06.014***

Civic Dispositions 2.95 0.56 3.01 0.58 0.06.015***

Respect for Diversity 3.26 0.58 3.34 0.53 0.08.019***

Civic Skills: Communication 2.94 0.52 3.00 0.49 0.06

.012***

Civic Skills: Leadership ( 2.93 0.62 3.00 0.60 0.07.015***

Civic Skills: Problem Solving 2.87 0.51 2.93 0.50 0.06

.013***

Service-Learning Engagement --- --- 2.89 0.69 --- ---

Service-Learning Quality ( --- --- 2.87 0.55 --- ---

2.502.44

2.382.41

1

2

3

4

Pretest Posttest

Service-Learning

Comparison

Changes Over Time in Service-Learning andComparison Students Ratings of Community Attachment (N = 1,174)

Frequency PercentageReading Skills 332 33.3

Writing Skills 387 38.8

Math Skills 225 22.5

Science Skills 203 20.3

Computer Skills 280 28.1

Tutoring Skills 165 16.5

Career Awareness 404 40.5

Job Skills 488 48.9

Work Experience 590 59.1

Percentage of Service-Learning StudentsReporting Skill Gains, Grades 6-12 (N = 823)

QUESTIONS FOR DISCUSSION

• What program design characteristics influenced outcomes?

• What is the relationship between each programs’ theory of action and implementation strategy, and the student outcomes for that program?

• What conclusions, if any, can we draw by looking across the evaluation findings of the three programs?

• What other questions do you have for us or for discussion here or in the future?

THANK YOU!

• To continue this conversation, please use the conference proceedings.

• Please contact us if you have additional comments or questions.– Paul Baumann: pbaumann@ecs.org– Shelley Billig: billig@rmcdenver.com– Susan Abravanel: sabravanel@ysa.org – Teri Dary: teri.dary@dpi.state.wi.us