redesigning national school surveys coverage and stratification improvement using multiple datasets...

Redesigning National School Surveys

Coverage and Stratification Improvement using Multiple Datasets

March 2014

FEDCASIC 2014Prepared for:

2

Authors:William Robb, Ronaldo Iachan, Kate Flint, Alice Roberts, Lee HardingICF International

Contact: richard.l.harding@icfi.com

3

Talk Outline

To date three national school surveys have used a single, common source for the sampling frame.– National Youth Risk Behaviors Survey– National Youth Tobacco Survey– National School Health Policies and Programs

We presents results from moving to a frame sourced from multiple files.– Make use of data available from the National Center for Education Statistics– Goal is to explore increases in coverage– Balanced with operational efficiency

The Talk– Describe the studies and data sources– Describe the frame build process– Present results regarding coverage and duplication

4

Research Questions

The motivation for building a frame from a combination of sources is to increase coverage, and decrease the risk of coverage bias. We also wanted to explore operational issues.

Specifically, the research questions were:

What is the increase in coverage?

What is the rate of duplicate school entries?

Will the combined frame affect validation and recruitment efforts?

Is there an impact on the estimates?

5

The Studies – Sample Design

SHPPS

National Scope

Elementary, Middle, and High Schools

PSU (districts) stratified by Locale Code

Second Stage: Schools

6

The Studies – Sample Design

YRBS/NYTS

National Scope

High Schools (YRBS/NYTS)

Middle Schools (NYTS)

PSU (counties) stratified by Urbanicity/Ethnicity

Second Stage: Schools

Third Stage: Students

7

Validation

For these studies, we do not replace non-responding schools. We do,

however, replace schools that are found to be ineligible.

Our recruiters contact schools to confirm that

– The school itself is eligible

– All sampled grades are present

– The students are not a “pull-out” population from other schools

School and personnel information is stored in a contact management system,

which provides tracking and support for validation, recruitment, and fielding.

8

Data Sources for Sampling Frame

To Date: Single MDR File

The “MDR” files have been used as a single source for the frame up to this point.

Single source for public and private school data, including– Updated contact information,

including personnel names.– Enrollment and Ethnicity counts

incorporated from NCES

Piloted: Combined with NCES Files

For the 2014 NYTS & SHPPS piloted a combined frame built from NCES and MDR files

Two additional files:– Common Core Data for public

schools– Private School Data for Private

Schools

QED = Quality Education Data’ MDR = “Market Data Retrieval”; NCES = National Center for Education Statistics; CCD = Common Core Data; PSS = Private School Survey

9

Processing Description

Schools– Screening for geography, type– Record level matching based on:• Public: NCES ID (96.0% ), Address (0.8% ), Phone (3.2% )• Private: Address (100%)– Screen for eligible schools

Districts–Match schools to 18,474 CCD districts –Match schools to 13,823 MDR districts– Combined frame included 15,665 districts

10

Processing Summary

Public Schools

Existing Both Sources 51,749 84.0%

Existing MDR Only 3,905 6.3%

Added CCD Only 5,988 9.7%

Total 61,642 Private Schools

Existing Both Sources 13,995 57.0%

Existing MDR Only 3,936 16.0%

Added PSS Only 6,607 26.9%

Total 24,538

11

Coverage of Schools Increase by School Type

Schools - All Schools Schools - High Schools Schools - Middle Schools0

10

20

30

40

50

Coverage Percent Increase

Non-Public Public

12

Coverage of Student Increase by School Type

Students - All Schools Students - High Schools Students - Middle Schools0

5

10

15

20

25

Coverage Percent Increase

Non-Public Public

13

Coverage Increase by Geography

14

Duplicates Schools (NYTS)

Possible Duplicates No Duplicates/No Unmatched No Duplicates/New Zip Code0

5,000

10,000

15,000

20,000

25,000

Duplicate Potential

Not Public Public

15

Ineligible Schools (SHPPS)

Existing Both Sources Existing MDR Only Added CCD/PSS0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

Percent Ineilgible

Not-Public Public

16

Looking Ahead

In process, for proceedings publication:

Assess school ineligibility rates for NYTS

Add elementary schools (SHPPS) to file processing and coverage assessment

Finish duplication assessment via record review

After Fielding

Assess impact on estimates

Preliminary Findings

Increases in coverage, primarily among small and private schools

Increases in ineligibility schools identified during validation

17

Thank You – and stay tuned!

Contact Author: Lee HardingRichard.L.Harding@icfi.com

(301) 572-0524