an empirical study to examine whether monetary incentives improve 12 th grade reading performance

Confidential and Proprietary. Copyright © 2007 by Educational Testing Service. All rights reserved.

An Empirical Study to An Empirical Study to Examine Whether Monetary Examine Whether Monetary

Incentives Improve 12Incentives Improve 12thth Grade Reading PerformanceGrade Reading Performance

Henry BraunHenry Braun Irwin Kirsch Irwin Kirsch Kentaro Kentaro YamamotoYamamoto

Boston CollegeBoston College ETS ETS ETS ETS

Presented at the PDII ConferencePresented at the PDII Conference

Princeton, NJPrinceton, NJ

October 3, 2008October 3, 2008


Overview• What is NAEP? What is NAEP?

• Why this study?Why this study?

• What were the design criteria?What were the design criteria?

• How were they operationalized?How were they operationalized?

• Do monetary incentives make a difference?Do monetary incentives make a difference?

• If so, which ones, how much and for whom?If so, which ones, how much and for whom?

• How robust are the findings?How robust are the findings?

• What are the implications?What are the implications?


The National Assessment of The National Assessment of Educational ProgressEducational Progress

• Large-scale national/state surveys of academic Large-scale national/state surveys of academic achievement achievement begun in 1969begun in 1969

• Tests students in grades 4, 8, 12Tests students in grades 4, 8, 12

• Subjects: Reading, Mathematics, Science, Subjects: Reading, Mathematics, Science, Geography, Geography, Civics, History, etc.Civics, History, etc.

• NAEP (“The Nation’s Report Card) provides a NAEP (“The Nation’s Report Card) provides a snapshot of snapshot of student achievement overall, by student achievement overall, by state and by various state and by various subgroupssubgroups


1212thth Grade NAEP Grade NAEP

• National sample onlyNational sample only

• Lower participation rates than grades 4 Lower participation rates than grades 4 and 8and 8

• Concerns about levels of motivation/effortConcerns about levels of motivation/effort

• Undergoing expansion to state levelUndergoing expansion to state level


Study RationaleStudy Rationale

• Increasing reliance on low-stakes large-scale assessment surveys Increasing reliance on low-stakes large-scale assessment surveys for education policyfor education policy

• Issues relate to both national and international LSASsIssues relate to both national and international LSASs

• In the U.S., NAEP is the only source of nationally comparable data In the U.S., NAEP is the only source of nationally comparable data on student achievement that can be used for state-level on student achievement that can be used for state-level comparisonscomparisons

• Under NCLB, 4Under NCLB, 4thth and 8 and 8thth grade NAEP play an expanded role in grade NAEP play an expanded role in monitoring state-level resultsmonitoring state-level results

• Strong interest in expanding role of 12Strong interest in expanding role of 12thth grade NAEP grade NAEP

• National Commission on 12National Commission on 12thth grade NAEP (2004) grade NAEP (2004) recommendationsrecommendations– Redesign to report on student readinessRedesign to report on student readiness– Expand to state levelExpand to state level– Increase participation and motivationIncrease participation and motivation


Design CriteriaDesign Criteria

Goal:Goal: To estimate the effects of different To estimate the effects of different monetary incentives monetary incentives on student on student performance on 12performance on 12thth grade NAEP grade NAEP

– Internal validityInternal validity

– External validityExternal validity

– Adequate powerAdequate power


LiteratureLiterature

• ExperimentsExperiments– Focus on mathematicsFocus on mathematics– O’Neil et al (NAEP items)O’Neil et al (NAEP items)– Baumert et al (PISA items)Baumert et al (PISA items)

• PsychologyPsychology– Intrinsic vs. extrinsic motivationIntrinsic vs. extrinsic motivation

• Behavioral EconomicsBehavioral Economics– Monetary incentives can workMonetary incentives can work– Participants must be cognizant of incentivesParticipants must be cognizant of incentives


Study Features Study Features

• Focus on NAEP readingFocus on NAEP reading• Randomized trial for internal validityRandomized trial for internal validity

– Prepared detailed implementation protocolPrepared detailed implementation protocol– Employed experienced administrative staffEmployed experienced administrative staff

• External validity (i.e., link directly to NAEP)External validity (i.e., link directly to NAEP)– Used released NAEP materialsUsed released NAEP materials– Followed NAEP administrative and data processing Followed NAEP administrative and data processing

proceduresprocedures– Carried out NAEP-like psychometric and statistical Carried out NAEP-like psychometric and statistical

analysesanalyses• Heterogeneous school sampleHeterogeneous school sample• Large study for sufficient power to detect effectsLarge study for sufficient power to detect effects


Study Design: IncentivesStudy Design: Incentives

• Control: Standard NAEP instructionsControl: Standard NAEP instructions

• Incentive 1: Standard NAEP instructions + Incentive 1: Standard NAEP instructions + Promise of a $20 gift card at conclusion of Promise of a $20 gift card at conclusion of sessionsession

• Incentive 2: Standard NAEP instructions + $5 Incentive 2: Standard NAEP instructions + $5 Gift card + $15 for a correct answer to each of Gift card + $15 for a correct answer to each of two questions to be chosen at random at the two questions to be chosen at random at the conclusion of sessionconclusion of session


Study Design: Incentives (2)Study Design: Incentives (2)

• All students in both incentive conditions were All students in both incentive conditions were asked to select Target or Barnes & Noble for the asked to select Target or Barnes & Noble for the gift card and to indicate their preference on a gift card and to indicate their preference on a sign-up sheetsign-up sheet

• Students in all three conditions actually received Students in all three conditions actually received $35 gift cards at the end of the sessions$35 gift cards at the end of the sessions

• Students were informally debriefed before Students were informally debriefed before leavingleaving


Study Design: InstrumentationStudy Design: Instrumentation

• Mapping to the NAEP Reading Framework (3 Mapping to the NAEP Reading Framework (3 contexts)contexts)– * * Reading for literary experience (35%)Reading for literary experience (35%)– * Reading for information (45%)* Reading for information (45%)– Reading to perform a task (20%)Reading to perform a task (20%)

• Assembling test bookletsAssembling test booklets– 2 reading blocks + background questionnaire2 reading blocks + background questionnaire– Each reading block consists of a passage and a set Each reading block consists of a passage and a set

of associated questionsof associated questions– Each block is expected to take 25 minutesEach block is expected to take 25 minutes– Blocks vary with respect to the total number of Blocks vary with respect to the total number of

questions and the proportions of multiple choice, questions and the proportions of multiple choice, short answer and extended response questionsshort answer and extended response questions


Booklet Booklet DesignDesign

Booklet

Total Number of questions

Multiple-choice

Essay questions(NAEP:

>50 Percent)

Forming a general

understanding (NAEP:

50 Percent)

Making reader /

text connections

(NAEP: 15 Percent)

Examining content and

structure(NAEP:

35 Percent)

AB / BA 20 6 14 (70 Percent)

8(40 Percent)

4(20 Percent)

8(40 Percent)

CD / DC 20 7 13 (65 Percent)

12(55 Percent)

3(14 Percent)

7(32 Percent)

AC / CA 22 9 13(59 Percent)

10(45 Percent)

4(18 Percent)

8(36 Percent)

BD / DB 18 4 14(78 Percent)

10(50 Percent)

3(15 Percent)

7(35 Percent)


Survey Design: Background Survey Design: Background QuestionnaireQuestionnaire

• Items drawn from operational questionnaireItems drawn from operational questionnaire• Two sets of itemsTwo sets of items

– Set ISet I• Demographics and parental educationDemographics and parental education• Home environmentHome environment• School absencesSchool absences

– Set IISet II• Reading practicesReading practices• Future educational expectationsFuture educational expectations• Level of effortLevel of effort


Study Design: Sample SelectionStudy Design: Sample Selection

• Power analysis indicated need for a sample of 60 Power analysis indicated need for a sample of 60 schools with 60 students per school (20 per schools with 60 students per school (20 per condition in each school)condition in each school)

• Worked with NAEP state coordinators and Westat Worked with NAEP state coordinators and Westat to obtain a (final) convenience sample of 59 schoolsto obtain a (final) convenience sample of 59 schools

• Student recruitment was carried out using Student recruitment was carried out using standard NAEP methods (but no special incentives)standard NAEP methods (but no special incentives)

• Number of participating students was lower than Number of participating students was lower than targettarget


Student Response Rates by StateStudent Response Rates by State

StateNumber of

Schools

Percentage of Response Rate

by State

Florida 10 66.8

Massachusetts 9 23.1

Michigan 26 60.2

Mississippi 8 78.2

New Jersey 2 40.0

New Mexico 4 45.8

Wyoming 5 57.1

Overall 59 56.0


AdministrationAdministration

• Random samples of 12Random samples of 12thth graders invited to graders invited to participateparticipate

• In each school students randomly allocated In each school students randomly allocated to the three to the three conditionsconditions

• Fall (not spring) administrationFall (not spring) administration• Sessions in a school were simultaneous or Sessions in a school were simultaneous or

consecutive to consecutive to eliminate possibility of eliminate possibility of contaminationcontamination

• Limited accommodationsLimited accommodations• No make-up sessionsNo make-up sessions


Student Response Rates by ConditionStudent Response Rates by Condition

Session IDSession ID

ContrControl ol

GroupGroupIncentiIncenti

ve 1ve 1IncentiIncenti

ve 2ve 2 TOTALTOTAL

Number Number Students to be Students to be AssessedAssessed 1,5521,552 1,5651,565 1,5461,546 4,6634,663

Number Number Students Students Actually Actually AssessedAssessed 835835 884884 893893 2,6122,612

(Number (Number Actually Actually Assessed / Assessed / Number to be Number to be Assessed)Assessed) 53.853.8 56.556.5 57.857.8 56.056.0


Data Preparation: Data Preparation: Scoring and Item AnalysisScoring and Item Analysis

• Scoring was conducted by NCS/PearsonScoring was conducted by NCS/Pearson• Preliminary item analysis held no surprises: Preliminary item analysis held no surprises:

Differences by condition inDifferences by condition in– Proportions correct Proportions correct – Percentage of omitted itemsPercentage of omitted items

•Highest for extended CR items Highest for extended CR items – Percentage of off-task responsesPercentage of off-task responses

•Generally very small (<< 1%)Generally very small (<< 1%)– Percentage of items not reachedPercentage of items not reached

•Particularly high for last CR itemParticularly high for last CR item


Average Item Proportions Correct Average Item Proportions Correct by Item TYPE and Incentive by Item TYPE and Incentive

ConditionCondition

N Control Group

Incentive 1

Incentive 2

13Multiple

choice .62 .63 .62

18CR-

Dichotomous .55 .59 .60

10CR-

Polytomous .52 .54 .57


Data Preparation: Data Preparation: Scaling, Conditioning and LinkingScaling, Conditioning and Linking

• Scaling by subscaleScaling by subscale– Fit item characteristic curves to dataFit item characteristic curves to data– Compare to archival resultsCompare to archival results– Estimate three-group modelEstimate three-group model– Reasonable fitReasonable fit

• ConditioningConditioning– Combine cognitive data with ancillary data from Combine cognitive data with ancillary data from

questionnairesquestionnaires– Obtain posterior score distribution for each Obtain posterior score distribution for each

studentstudent– Generate “plausible values”Generate “plausible values”

• LinkingLinking– Linear transformation to the NAEP scaleLinear transformation to the NAEP scale– Construct composite reporting scaleConstruct composite reporting scale


Effect Sizes by Subscale, Effect Sizes by Subscale, Item parameters Based on Study Data Item parameters Based on Study Data

OnlyOnly

Incentive 1

to Control

Incentive 2

to Control

Literacy 0.08 0.12

Information

0.15 0.26

Effect Sizes by Subscale, Effect Sizes by Subscale, Item parameters Based on Archival Item parameters Based on Archival

DataData

Incentive 1 to Control

Incentive 2

to Control

Literacy 0.08 0.13

Information

0.15 0.25


Selected ResultsSelected Results

• Effects of incentives range from 3 to 5 points on Effects of incentives range from 3 to 5 points on the NAEP scale overallthe NAEP scale overall

• Male-female differences relatively stableMale-female differences relatively stable

• White-Black and White-Hispanic differences White-Black and White-Hispanic differences grow grow somewhat larger under incentivessomewhat larger under incentives

• Effects of incentives generally positive for Effects of incentives generally positive for subgroupssubgroups

• Estimates reasonably robustEstimates reasonably robust


Comparison of Effects by Comparison of Effects by Incentive ConditionIncentive Condition

Deviation from Control by Percentile

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

0 20 40 60 80 100

percentile

deiv

iati

on

on

NA

EP

read

ing

sca

le

Obs Incentive 1

Obs Incentive 2

Smooth Inc 1

Smooth Inc 2


HOW IMPORTANT WAS IT TO DO WELL ON THIS TEST

Control Group

Incentive 1

Incentive 2

N 155 122 98 COL Percent 19.0 14.1 11.1 Not very important MEAN 287.0 292.4 288.7 N 369 348 345 COL Percent 45.3 40.3 39.2 Somewhat important MEAN 291.6 293.0 296.3 N 205 286 294 COL Percent 25.2 33.1 33.4 Important MEAN 289.9 294.2 297.7 N 86 108 144 COL Percent 10.6 12.5 16.4 Very important MEAN 282.6 291.2 295.1


HOW HARD DID YOU TRY ON THIS TEST COMPARED TO OTHER TESTS

Control Group

Incentive 1

Incentive 2

N 306 232 212 COL Percent 37.6 26.8 24.0 Tried not as much MEAN 285.2 286.8 286.9

N 440 550 578 COL Percent 54.0 63.5 65.5 Tried about as much MEAN 295.7 298.2 300.4

N 53 60 74 COL Percent 6.5 6.9 8.4 Tried harder MEAN 269.8 280.4 290.1

N 16 24 18 COL Percent 2.0 2.8 2.0 Tried much harder MEAN 258.5 271.8 276.5


Study Statistics by Incentive and GenderStudy Statistics by Incentive and Gender

Control Group

Incentive 1

Incentive 2 TOTAL

MaleMEAN 286.7 289.3 292.2 289.4

FemaleMEAN 291.5 295.4 296.8 294.7

TOTALMEAN 289.2 292.6 294.7 292.3

Control Group

Incentive 1

Incentive 2 TOTAL

MaleMEAN 286.7 289.3 292.2 289.4

FemaleMEAN 291.5 295.4 296.8 294.7

5

5

5

5 6 6


Study Statistics by Incentive and Study Statistics by Incentive and Race/EthnicityRace/Ethnicity

Control

Incentive 1

Incentive 2

TOTAL

White, not HispanicMEAN 296.8 302.9 302.7 300.9

Black, not HispanicMEA

N 276.4 274.3 278.6 276.4

HispanicMEA

N 277.1 274.1 280.2 277.3 Control

Incentiv

e 1Incentive

2TOTA

L

White, not HispanicMEAN 296.8 302.9 302.7 300.9

Black, not HispanicMEA

N 276.4 274.3 278.6 276.4

HispanicMEA

N 277.1 274.1 280.2 277.3

6

2

3

20 28 24


STUDY STATISTICS BY CONDITION, GENDER, AND RACE/ETHNICITY

MALE

Male Control Male

Incentive 1 Male

Incentive 2 N 224 246 263 COL Percent 57.0 63.2 64.3 White, not Hispanic MEAN 293.6 299.2 299.6

N 104 101 93 COL Percent 26.5 26.0 22.7 Black, not Hispanic MEAN 273.6 272.1 274.8

N 50 31 36 COL Percent 12.7 8.0 8.8 Hispanic MEAN 281.1 261.7 279.4

N 12 9 14 COL Percent 3.1 2.3 3.4 Asian/Pacific Islander MEAN 298.6 297.2 302.9

N 3 2 1 COL Percent 0.8 0.5 0.2 American Indian/Alaska Native MEAN 273.5 318.3 296.5 N 0 0 2 COL Percent 0 0 0.49 Other MEAN 0 0 289.4


FEMALE

Female Control

Female Incentive 1

Female Incentive 2

N 269 302 298 COL Percent 62.1 62.8 61.3 White, not Hispanic MEAN 299.4 305.5 305.4

N 114 119 112 COL Percent 26.3 24.7 23.1 Black, not Hispanic MEAN 279.0 276.2 281.9

N 37 47 57 COL Percent 8.6 9.8 11.7 Hispanic MEAN 271.6 281.4 280.7

N 8 10 16 COL Percent 1.9 2.1 3.3 Asian/Pacific Islander MEAN 295.7 288.6 297.1

N 3 1 2 COL Percent 0.7 0.2 0.4 American Indian/Alaska Native MEAN 281.5 264.5 294.5

N 2 2 1 COL Percent 0.5 0.4 0.2 Other MEAN 307.6 306.1 325.6

Study Statistics by Condition, Gender, and Race/Ethnicity


STUDY STATISTICS BY CONDITION, GENDER, AND ELIGIBILITY FOR NATIONAL SCHOOL LUNCH PROGRAM

MALE

Male Control Male

Incentive 1 Male

Incentive 2 N 277 274 297 COL Percent 71.2 71.5 73.3 Not eligible MEAN 290.1 295.7 297.3

N 104 103 104 COL Percent 26.7 26.9 25.7 Free/reduced price MEAN 280.2 273.8 279.4

FEMALE

Female Control Female

Incentive 1 Female

Incentive 2 N 303 341 353 COL Percent 70.3 70.9 72.8 Not eligible MEAN 296.3 301.4 300.6

N 119 133 127 COL Percent 27.6 27.7 26.2 Free/reduced price MEAN 280.3 281.0 287.3


MALE

Male Control Male

Incentive 1 Male

Incentive 2 N 43 35 38 COL Percent 11.2 9.2 9.4 Did not finish high school MEAN 280.8 278.6 276.3

N 89 93 90 COL Percent 23.2 24.4 22.3 Graduated high school MEAN 282.5 285.9 284.2

N 105 97 101 COL Percent 27.3 25.4 25.1

Some education after high school

MEAN 288.0 292.1 294.4

N 132 142 160 COL Percent 34.4 37.2 39.7 Graduated college MEAN 292.3 295.0 303.6

N 15 15 14 COL Percent 3.9 3.9 3.5 I don't know MEAN 265.2 283.7 278.5

Study Statistics by Condition, Gender, and Mother’s Education Level


FEMALE

Female Control

Female Incentive 1

Female Incentive 2

N 55 72 67 COL Percent 12.9 15.0 14.0

Did not finish high school

MEAN 280.3 283.7 284.4

N 107 100 127 COL Percent 25.1 20.9 26.6 Graduated high school MEAN 285.1 287.0 292.5

N 120 121 124 COL Percent 28.1 25.3 25.9

Some education after high school

MEAN 298.2 296.5 301.2

N 135 172 157 COL Percent 31.6 35.9 32.9 Graduated college MEAN 298.8 306.2 305.4

N 10 14 3 COL Percent 2.3 2.9 0.6 I don't know MEAN 265.0 275.4 296.2

Study Statistics by Condition, Gender, and Mother’s Education Level


MALE Male Control

Male Incentive 1

Male Incentive 2

N 160 166 187 COL Percent 41.3 43.3 46.3 None MEAN 292.6 291.3 295.6

N 144 152 152 COL Percent 37.2 39.7 37.6 1-2 days MEAN 285.6 288.8 291.7



N 9 4 5 COL Percent 2.3 1.0 1.2 >10 days MEAN 255.4 277.8 268.4

Study Statistics by Condition, Gender, and Number of Days Absent From School Last Month


FEMALE

Female Control

Female Incentive 1

Female Incentive 2

N 148 166 156 COL Percent 34.7 34.6 32.5 None MEAN 296.7 300.0 301.7




N 6 5 4 COL Percent 1.4 1.0 0.8 >10 days MEAN 284.7 294.8 298.6

Study Statistics by Condition, Gender, and Number of Days Absent From School Last Month


Study Statistics by Condition, Gender, and Frequency of Reading for Fun on Own Time

MALE

Male Control Male

Incentive 1 Male

Incentive 2

N 126 110 103 COL Percent 32.7 28.8 25.8 Never or hardly ever MEAN 280.3 287.8 287.4

N 116 115 127 COL Percent 30.1 30.1 31.8 Once or twice a month MEAN 284.5 285.8 289.7

N 84 89 99 COL Percent 21.8 23.3 24.8 1 or 2 times a week MEAN 293.0 287.2 293.2

N 59 68 70 COL Percent 15.3 17.8 17.5 Almost every day MEAN 294.1 303.9 309.9


FEMALE

Female Control

Female Incentive 1

Female Incentive 2

N 77 80 96 COL Percent 18.3 16.9 20.1 Never or hardly ever MEAN 285.3 285.8 290.9

N 150 136 147 COL Percent 35.6 28.8 30.8 Once or twice a month MEAN 286.2 292.9 295.5

N 106 120 122 COL Percent 25.1 25.4 25.6 1 or 2 times a week MEAN 293.1 289.6 297.3

N 89 137 112 COL Percent 21.1 29.0 23.5 Almost every day MEAN 307.2 308.9 308.2

Study Statistics by Condition, Gender, and Frequency of Reading for Fun on Own Time


Sensitivity Analysis (1)Sensitivity Analysis (1)

• Although treatment groups were determined Although treatment groups were determined randomly, there were differences in various randomly, there were differences in various characteristics that characteristics that might have contributed might have contributed to the estimated treatment effects.to the estimated treatment effects.

• We ran an ANOVA adjusting NAEP scores for We ran an ANOVA adjusting NAEP scores for a number of demographic and home a number of demographic and home environment characteristics, as well as environment characteristics, as well as students’ reading habits.students’ reading habits.

• The ANOVAs were run separately for males The ANOVAs were run separately for males and females.and females.

• They yield adjusted least squares means that They yield adjusted least squares means that can be compared to the raw means.can be compared to the raw means.


TREATMENT EFFECTS BY INCENTIVE CONDITION AND GENDER

MALE Male

Control Male

Incentive 1 Male

Incentive 2 Unadjusted 0 2.5 5.5 Adjusted 0 1.0 4.2

FEMALE Female Control

Female Incentive 1

Female Incentive 2

Unadjusted 0 3.9 5.3 Adjusted 0 2.9 5.4


Sensitivity Analysis (2)Sensitivity Analysis (2)

Impact of “leverage” groups was examined by identifying those Impact of “leverage” groups was examined by identifying those subgroups with the largest positive effect (Incentive 2) and a subgroups with the largest positive effect (Incentive 2) and a large enough sample size to rule out sampling fluctuations.large enough sample size to rule out sampling fluctuations.

(i) Male, White, Absent more than 3 days in the last month(i) Male, White, Absent more than 3 days in the last month [Effect ~3x larger than overall effect for males] [Effect ~3x larger than overall effect for males]

[95/802][95/802] Removing this group would reduce effect of Incentive 2 Removing this group would reduce effect of Incentive 2 by ~25%.by ~25%.

(ii) Female, Hispanic, Not ELL(ii) Female, Hispanic, Not ELL [Effect ~3x larger than overall effect for females] [Effect ~3x larger than overall effect for females]

[82/919][82/919] Removing this group would reduce effect of Incentive 2Removing this group would reduce effect of Incentive 2

by ~13%.by ~13%.


SummarySummary

• Data clearly indicate that the design criteria for this Data clearly indicate that the design criteria for this study were metstudy were met

• Monetary incentives improve NAEP reading performanceMonetary incentives improve NAEP reading performance

• Type of incentive makes a differenceType of incentive makes a difference by reporting subgroupby reporting subgroup by quantileby quantile


CaveatsCaveats

• Fall rather than Spring administrationFall rather than Spring administration

• Represented two of the three NAEP subscales Represented two of the three NAEP subscales

• Lower student participation rate than in operational NAEPLower student participation rate than in operational NAEP

• Subgroup sample sizeSubgroup sample size

• Relationship of the sample to the NAEP populationRelationship of the sample to the NAEP population


ImplicationsImplications

• 1212thth grade NAEP results should be interpreted cautiously grade NAEP results should be interpreted cautiously

• Expansion of 12Expansion of 12thth grade NAEP ought to wait on policy grade NAEP ought to wait on policy action on incentivesaction on incentives

• Measuring reading as NAEP does may be problematic in the Measuring reading as NAEP does may be problematic in the current context current context

• In modifying NAEP cognitive instruments (e.g. for readiness), In modifying NAEP cognitive instruments (e.g. for readiness), the administrative setting should be taken into accountthe administrative setting should be taken into account

an empirical study to examine whether monetary incentives improve 12 th grade reading performance

Documents