the future of the naep survey questionnaires: …€“advanced psychometric analyses to identify...

Copyright © 2015 by Educational Testing Service.

The Future of the NAEP Survey Questionnaires: Measuring what Matters

for Student Achievement in the U.S.

1

Jonas P. BertlingDirector, NAEP Survey Questionnaires

Educational Testing Service

NCSA 2015 Symposium “How NAEP is leading assessments into the future”June 23, 2015 | San Diego, CA


Overview

• Future NAEP Survey Questionnaires will include additional noncognitive and opportunity-to-learn modules

• Why are noncognitive factors relevant?

• Implementation strategy: 3 key innovations

• What have we done so far?

• Importance of these innovations to ensure that NAEP stays relevant

2

Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation


NAEP measures what students in the U.S. know and can do



NAEP measures what students in the U.S. know and can do

4

How are students in the Nation doing in key subject-areas?

How do subgroups perform?

Why do students succeed?

What can we do to improve student learning outcomes?

How do students approach learning situations and utilize learning opportunities?

How do learning opportunities differ across the Nation?

How are students in the Nation doing in terms of other outcomes than achievement?

Noncognitive factors

Social Emotional Learning (SEL)

Character

PersonalityMindsets, Essential Skills and Habits

Inter- and Intrapersonal Skills

Soft Skills

Nonacademic skills

Take Action

Cognitive Assessment

Noncognitive Assessment

How can NAEP help stakeholders to take action?



Survey Questionnaires data puts achievement results into context…

5

…and can add value to the Nation’s Report Card by capturing important additional

variables.



Opportunity-to-Learn (OTL) Factors:

(COGNITIVE ASSESSMENT)(SURVEY QUESTIONNAIRES)

“Noncognitive” Student Factors:

Achievement

Cognitive Ability*

Domain-general factors

(“Core”)

Subject-specific factors

*not measured with the NAEP survey questionnaires

NAEP Contextual Variables

OTL At School

“Noncognitive” Student Factors

Students’ exposure to opportunities to acquire relevant

knowledge and skills, both at school and outside of school.

Skills, strategies, attitudes, and behaviors that are distinct from

content knowledge and academic skills (e.g., motivational and self-

regulatory variables)

OTL Outside of School

Basic Demographics


new areas of focus


Why Noncognitive Factors?

• Related to important life outcomes (e.g., achievement, health, well-being, citizenship behavior)

• Noncognitive factors are about as predictive for many achievement outcomes as cognitive factors

• More malleable than cognitive factors

• Can have positive impact on cognitive development

• Important workforce readiness skills

See e.g., Farrington et al. (2012); Heckman & Kautz (2014), Kautz et al. (2015) for comprehensive reviews of the research literature on noncognitive factors

7



3 Key Innovations for the NAEP Survey Questionnaires

Expand topics to include more contextual factors critical for student

achievement

Improve measurement and reporting of contextual factors

Improve questionnaire design for TBA to maximize validity while keeping student

burden low

I.

II.

III.



Alignment with Implementation Recommendations from The Governing Board

Expand topics to include more contextual factors critical for student

achievement

Improve measurement and reporting of contextual factors

Improve questionnaire design for TBA to maximize validity while keeping student

burden low

I.

II.

III.

“NAEP reporting should be enriched by greater use of contextual data derived from background or non-cognitive questions asked of students, teachers, and schools”

-National Assessment Governing Board (2012, p. 2)



Key Challenges & Implementation to Date

• Which factors to measure?

• How to measure them?

• How to keep student burden low?

0

5

10

15

20

25

30

35

NAEP PISA TIMSS/PIRLS

Minutes

Subject-Specific Core

Item DevelopmentPre-testing in Cognitive Labs

Large-scale Pilot Testing

Operational Administration



11




2014

Considerations for Selection of Areas for Development:1. Mandated reporting categories

2. Not intrusive

3. Clearly related to student achievement

4. Malleable, actionable

5. Can be measured with questionnaires

6. Clearly tied to (subject-specific) framework

7. Based on established/validated in research and/or other ILSAs

Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:



12




2014

1. Review of Literature (Research, Policy Frameworks, Other Media)

2. Consider different measurement approaches

3. Consultation and reviews with advisory groups and key stakeholders including:Questionnaire Standing Committee, Subject-area Standing Committees, NAEP Principals Panel, NAEP State Coordinators, NAEP Design and Analysis Committee



http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=pUSCAEvMCKNlKM&tbnid=pH9dSfkD5A3lOM:&ved=0CAUQjRw&url=http://www.parade.com/62123/jmarquez/your-better-school-day-book-list/&ei=QUOMUviPBMjdsATE34D4Bw&bvm=bv.56753253,d.cWc&psig=AFQjCNGM05tblZ3RAPc88pjGJ9W9S3V0bg&ust=1385010328498871

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=pUSCAEvMCKNlKM&tbnid=pH9dSfkD5A3lOM:&ved=0CAUQjRw&url=http://www.parade.com/62123/jmarquez/your-better-school-day-book-list/&ei=QUOMUviPBMjdsATE34D4Bw&bvm=bv.56753253,d.cWc&psig=AFQjCNGM05tblZ3RAPc88pjGJ9W9S3V0bg&ust=1385010328498871


Questionnaire Standing Committee: Current Members

Roger TourangeauWestat

Stanley PresserUniversity of Maryland

Angela DuckworthUniversity of Pennsylvania

Leslie RutkowskiIndiana University Bloomington

Hunter GehlbachHarvard University

Rob SantosUrban Institute

Gerunda HughesHoward University

Norbert SchwarzUniversity of Michigan

David Kaplan University of Wisconsin-Madison

Jonathon StoutLock Haven University

Henry LevinTeachers College, Columbia University

Akane ZushoFordham University



14




Noncogs

Domain-general (Core)

OTL Outside of School

OTL At School

“Noncognitive” Student Factors

Basic Demographics

Subject-specific

12 1

23

3

4

56

7

5

4

1. Grit2. Desire for learning &

Growth Mindset

1. Self-Efficacy/Confidence2. Achievement Goals3. Interest & Enjoyment

3. Socioeconomic status & family academic support

4. Technology Use5. School Climate

OTL At School

4. Out-of-school Activities5. Resources for Learning

and Instruction (incl. technology)

6. Organization of Instruction

7. Teacher Preparation (incl. noncognitive factors and familiarity with technology)

5 Core Modules Additional Subject-specific modules

OTL Outside of

School


2014

“Balanced” reporting approach based on indices

AND select single questions

Indices allow for more robust reporting based on

multiple questions on the same topicJonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation


• All new and revised items go through rigorous pre-testing before administered in any pilot or operational assessment

• One-on-one cognitive interviews with trained interviewers

– Structured probing with both general and item-specific probes

– How do students understand items? What are difficulties? How can items be improved? How do alternative versions compare?




2014 2014-2015




• Coglabs provided basis for data-informed revisions of response options for more clarity and consistency across items

• Coglabs helped with language simplification

• Platform for test of new item formats (e.g. vignettes for potential future use)

• Respondents preferred items with more (e.g., 5 or 6) response options allowing for more differentiated choices than simple yes/no questions

• Eye tracking study showed that 4th grade students can answer and enjoy answering matrix questions




2014 2014-2015

Key Lessons learned from cognitive interviews:





Heat map from NAEP SQ Eye-Tracking Study


• Large-scale digital based pilot will inform decision for operational

• Innovations:– “Spiraling” design to select best questions: Not every question

answered by every student to reduce burden

– Use of timing data in analysis to build final forms

– Advanced psychometric analyses to identify and validate indices

– Item version comparisons to select best measurement approach

– Variation of item sequences to control for potential biases




2014 2014-2015


2016 2017



Summary• Future NAEP Survey Questionnaires will include additional

noncognitive and opportunity-to-learn modules;

• Noncognitive factors are important achievement predictors, malleable, and potential additional outcomes;

• Implementation strategy focuses on (1) new topics, (2) better measurement, and (3) improved design;

• Item development and pre-testing for new 2017 modules successfully completed; large-scale pilot will follow in 2016.

19

These innovations will ensure that NAEP stays relevant as policy interests shift from monitoring student literacy in core subjects to promoting lifelong learners who are able and eager to face the demands and challenges of a truly global society.



Questions? Comments?

20

ETS NAEP Survey Questionnaires Team Members

Jared Anthony

Thank you!

[email protected]

Debby Almonte Laurie StaubJan Alegre Ryan Whorton


the future of the naep survey questionnaires: …€“advanced psychometric analyses to identify...

Documents