Getting to Social Emotional Learning Measurement – Challenges and Opportunities

Panel Discussion

October 7, 2016 | Tukwila Community Center

• Moderator: Julie Petrokubi, Ph.D., Senior Advisor, Youth Development & Evaluation, Education Northwest Panelists: - Neil Naftzger, Principal Researcher, American Institutes for Research (AIR) - Jessica Beaver, Ph.D., Senior Research Scientist, Seattle Public Schools - Roslyn Kagy, GEAR UP Program Manager, Highline Public Schools

SEL Measurement: Youth Surveys

SEL surveys hold great potential to…

Provide formative data related to multi-dimensional constructs such as school engagement and postsecondary readiness

Clarify connection between youth development programs and school-day learning

Offer opportunities for youth voice

SEL Measurement: Youth Surveys

SEL measurement pose new dilemmas

Confusion about what and how to measure

Limitations of self-report data

Early state of measure development, especially for formative purposes

Concerns about use for accountability purposes

Introductions and overview

Brief presentations by panelists

Panel discussion

What are the challenges and emerging recommendations on using data to support social emotional learning growth?

Session Plan

Findings from Efforts to Validate

the Youth Motivation,

Engagement and Beliefs Survey

Road Map Region SEL Symposium

Neil Naftzger

Principal Researcher

October 2016

Copyright © 2016 American Institutes for Research. All rights reserved.

AMERICAN INSTITUTES FOR RESEARCH

Motivation, Engagement, and Beliefs Survey

• Initially developed by YDEKC

• Modified for use for the statewide evaluation of the 21st CCLC

program in Washington

• Three types of scales:

– Items pertaining to youth’s sense of belonging and engagement in programming

– Items pertaining to youth’s sense of how they may have been affected by participation

in the program

– Items pertaining to how youth reported functioning at present when taking the survey

on a series of areas related to positive youth development

» Academic identity

» Positive mindsets

» Self-management

» Interpersonal skills

7

AMERICAN INSTITUTES FOR RESEARCH

Validation Efforts

• Functions well psychometrically

• Survey scores found to be related to school-related outcomes in

the manner hypothesized

• Demonstrates capacity to detect change over time:

– Youth scoring lower on the survey at baseline improve more

– More positive experiences in programming associated with more growth

– More self-reported program impact associated with more growth

– Lingering concerns about regression to the mean

8

AMERICAN INSTITUTES FOR RESEARCH

Real Change vs. Regression to the Mean

9

Scatterplot of Time 1 Score on the Mindsets Scale and Change from Time 1 to Time 2

AMERICAN INSTITUTES FOR RESEARCH

Interpreting Results

10

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

AI - 2015 AI - 2016 Mindset - 2015 Mindset - 2016 SM - 2015 SM - 2016 Inter - 2015 Inter - 2016

Not at all true Somewhat true Mostly true Completely true

Distribution of Youth Across Rating Scale Categories by Scale and Sample Year:

Scales Related to Youth Outcomes

Neil Naftzger

630-649-6616

nnaftzger@air.org

1120 East Diehl Road, Suite 200

Naperville, IL 60563-4899

General Information: 630-649-6500

www.air.org

11

Photos by Susie Fitzhugh

YDEKC/Road Map SEL Symposium

10.7.2016

SPS Initiatives and SEL

Every Student. Every Classroom. Every Day. 13

Race & Equity

MTSS/

Whole Child

Approach

Reducing Discipline

Disproportionality

Positive

Relationships

High Quality, Culturally

Responsive Pedagogy

Trauma-Informed

Practices

Social-Emotional

Learning

Revised items, new constructs, improved scales

2015-16 Student Survey

Every Student. Every Classroom. Every Day. 14

Every Student. Every Classroom. Every Day. 15

Panorama Portal

Every Student. Every Classroom. Every Day. 16

Every Student. Every Classroom. Every Day. 17

Every Student. Every Classroom. Every Day. 18

School Climate

(23 items)

• Healthy Community

• Belonging

• Classroom Environment

• Safety

Whole Child Success Factors

(12 items)

• Learning Mindset

• Social Emotional Learning

Classroom Instruction

(11 items)

• Student Motivation and Inclusion

• Pedagogical Effectiveness

Survey Analysis

Every Student. Every Classroom. Every Day. 19

Survey Analysis

• Do the students survey responses group by construct as planned? – YES, for the most part. – Confirmatory Factor Analysis found some item and construct

refinement is needed. – Significant developmental differences in the meaning of constructs. – Significant correlation amongst constructs (r=0.61-0.98).

• Next steps will ask: – Are there differences in the meaning or level of constructs by race,

gender and language spoken at home? – To what extent is there a relationship between SEL, climate,

instructional practices and important outcomes such as academic achievement and behavioral health? Does this relationship differ by school or region?

1. Data Usage:

-District Tri-Day

-Interim survey administration, intensive data walks in 50 schools

2. Triangulation with staff and family surveys, student voice focus groups

3. Revised student survey for spring 2017

Data Usage and Refinement Efforts for 2016-17

Every Student. Every Classroom. Every Day. 21

• 75 Questions, 4 sections

• 15-20 minutes to answer

• Delivered online

• Preloaded with student identifiers, demographics, and EWI

• Open for one month

• Schools must meet 85% completion

About the Survey

22

How do you move beyond GEAR UP requirements and tell a dynamic & honest story?

1. Post Secondary Aspirations & Expectations

2. College & Career

3. School Climate

4. Non Cognitive

1. Academic Behaviors

2. Future Orientation

3. Growth Mindset

4. Perseverance/Grit

5. Sense of Belonging

Designing a survey to tell your story

23

Do students trust adults?

24

•Data Dashboards

•Spotlights

•Advisory Related Questions

•Meta Cognitive Scales

•Coach cohort – pre and

post surveys

•Mythbusters

•INSIGHT teams

Evolution of the tool…

25

26

Success Coaches

27

• 12 Coaches

• Under 30

• Bilingual

• Reflective of Community

• 80 hours of orientation

• Focus on race, class, gender,

power, oppression

• Identify students at risk of

falling off track

• immediate coaching &

intervention

• coordinate afterschool

programming

• family engagement

Success Coaches

28

• Served 977 students in 2015-16

• Taught weekly MetaCognitive workshops to 392

students during summer credit recovery

• Supported & strengthened after school programs

(tutoring, clubs, credit recovery)

• Measurements: Attitude (interest, attachment,

educational aspirations) & Behavior (attendance)

Interventions: Student Success

Coaches

29

Student Success Coach Interventions

30

Student participates in at least

one goal setting session

Student receives 20 interventions

● Coaching, tutoring, one-on-

one, class visits

Student makes at least one non

academic connection

● Church, sports, internship

Student attends at least one

college or career visit

● Equitable access

Student & Coach have at least

one connection with key adult

Pre and Post Survey Completed

Success Coaches - Results

31

Success Coaches

32

3 data snapshots

Last page (top supports)

Anecdotal quote

Success Coaches

33

Open Ended Questions:

34

• “working with a coach this year helped me evolve as a person emotionally, mentally and

spiritually. I never imagines I would be as close as I am to my coach, and create such a fruitful

relationship that I feel will last for most of my adult life”

• My coach helped me fix bad relationships with teacher &

learn how to communicate better

• My coach helped me get more involved in my school

and my community. I feel more connected

My coach taught me how to deal with tough things

Happening at home but still get things done at the

Same time & stay on track

the skills my coach taught me go way beyond the

classroom. She helped me forgive, forget, and

Become a better person

Discussion