favored career choices: in rank order

Real and Imaginary Concerns in Assessing

College Readiness

Nathan R. Kuncel

Department of Psychology

Favored Career Choices: In Rank Order

1. Jedi Knight2. Star Fleet Captain3. Batman4. Playboy Millionaire5. Psychology Professor6. Evolutionary Biologist7. Economist8. Brilliant but Misunderstood Novelist

Jedi Knight Innate heritable ability, unrelated to social class, focused through mentoring and intense training, high self control

Star Fleet Captain Born but seasoned leaders who succeed through

managed team effectiveness, charismatic and bold

Batman Single formative childhood experience that motivates massive skill investment, augmented by tremendous family wealth, intense and disagreeable

Professor of Psychology Masochistic enthusiasm for studying nearly intractable research questions, fairly friendly

Stories We Tell About Achieving Success

• Predictable versus Unpredictable

• Book Smarts versus Street Smarts

• General versus Specific Aptitudes

• Social Class Dependent versus Not

• Talent versus Effort

• Linear versus Asymptotic

Stories about Tests– Tests don’t even predict grades well– Tests don’t add anything to prediction– Tests don’t predict anything other than grades– Cognitive tests only predict academic outcomes – Cognitive tests are only proxies for SES– Beyond a certain point, scores don’t matter

Stories about Traditional Alternatives– Letters of Recommendation– Personal Statements– Interviews

Frightening Cautionary Tales about the Future– Faking– Bias– Unreliability

Assertion: Tests Don’t Even Predict

Grades Well

What does “well” mean and how do we quantify it?

Zombie Plague

• Plague sweeps the world

• Plague transforms 60% of infected people into horrifying and mindless monsters.

A Partial Treatment Is Developed!

Zombie Live

No Treatment 60% 40%

Treatment 40% 60%

What is the correlation?

Tests and Grades

• Berry and Sackett (2009) examined SAT-grade r’s in a sample of 165,000 students from 41 schools

• Corrected for restriction of range using school-specific applicant pools

• Obtained individual course grades for each course, and computed validity for each course (148,072 validity coefficients)– This removes influence of student choice of courses– Estimate r between SAT and common course portfolio

Berry and Sackett (2009) results

SAT-First Year College GPA

r

observed .36

corrected for range restriction

.46

corrected for course difficulty

.55

Story:Tests Don’t Add Anything

to Prediction

Bridgeman, Burton, & Cline (2009)

Percent Earning a 3.8 or Higher Graduate GPA

Story: Tests only predict 1st year grades

• Paper submitted to American Psychologist: “As is well known, the SAT predicts nothing but first year grades” [with no citation]

Evidence Across All Admissions Tests: Kuncel & Hezlett (2007) Science

• 7 Admissions Tests– GRE-T– GRE-S– MCAT – LSAT– GMAT– MAT– PCAT

• 8 Outcomes– 1st Year GPA– Graduate GPA– Faculty Ratings– Degree Attainment– Citation Counts– Research Productivity– Quals./Comps. Exams– Licensing Exams

Results are based on over 600,000 students across over 3,000 independent samples

Kuncel & Hezlett (2007). Science, 315, 1080-1081.

If the critics were right….

Book Smarts vs. Street Smartsor

Academic vs. Practical Intelligenceor

Book Learning vs. Common Sense

Book Smarts vs. Street Smarts

• Fundamental argument is that different types of intelligence are needed in academic versus real world contexts

• It is argued that academic contexts require an “academic intelligence” or book smarts

• It is my belief that these arguments are based (in part) on an overly narrow conceptualization of the tasks that comprise academic performance

Academic versus Practical TasksAcademic

• Formulated by others• Well-defined• Presents complete

information• Single method for

obtaining answer• Not embedded in

ordinary experience• Little or no intrinsic

interest

Practical• Requires problem

recognition and formulation• Ill-defined• Requires information

seeking• Multiple acceptable

solutions• Multiple paths to solution• Embedded in ordinary

experience• Requires motivation and

personal involvement

Things Students Do Before Taking a Test

• Determine study objectives and methods

• Manage goal conflicts

• Coordinate work with other classmates

• Seek additional information or materials

• Handle school related finances

• Negotiate with peers and faculty

• Avoid counterproductive school behaviors

• Structure effective communications

A Model of Undergraduate Student Performance Dimensions• Traditional Classroom

Success• Written and Oral

Communication• Personal Discipline• Resolving Goal Conflicts• Studying and Learning

Proficiency• Sustained Goal Directed

Effort

• Interactive Learning and Team Performance

• Administration• Interpersonal Proficiency• Non-Classroom

Performance• Development of Life Goals

and Values

Kuncel (2002); Kuncel, Campbell, Hezlett, & Ones (2001)

Cross-Situational Validity: A Direct Test

• However, we still lack a direct test: – Examine the validity of a single test developed

for academic settings but used in both academic and work settings

– Ideally we would also establish the relationship between this test and other cognitive ability measures

• Unfortunately, a single ability measure is rarely used for both personnel selection and educational admissions decisions

• With one notable exception….

MAT – Academic Performance

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Time to Finish

Degree Attainment

ResearchProductivity

ComprehensiveExams

Faculty Ratings

1st Year Grad. GPA

Grad. GPA

Kuncel, Hezlett, & Ones (2004) JPSP

MAT – Transitional Variables and Creativity

0 0.1 0.2 0.3 0.4 0.5 0.6

InternshipPracticum Ratings

Student TeachingRatings

Conseling WorkSamples

Creativity Ratings

Career PotentialRatings

Conseling PotentialRatings

Kuncel, Hezlett, & Ones (2004) JPSP

MAT – Work Criteria

0 0.1 0.2 0.3 0.4 0.5 0.6

JobPerformance

CounselingPerformance

Ed. Admin.Performance

Member Prof.Org.

Kuncel, Hezlett, & Ones (2004) JPSP

Story: Predictive Power of Cognitive Ability

Tests is an artifact of SES

Its Storytime…

• In the interest of truth in advertising, the SAT should simply be called a “wealth test” - Guiner (Undated)

• “The SAT merely measures the size of student’s houses” - Kohn (2001)

• “Only thing the SAT predicts well now is socioeconomic status” – Colvin (1997)

Recent Claims in the American Psychologist

• “…SAT I scores lose any ability to predict freshman year grades if the regression analyses control for socioeconomic status” (p. 100) – Crosby, Iyer, Clayton, and Downing (2003)

• “…SAT scores used for college admissions do not predict freshman year grades when socioeconomic status is controlled” (p. 1023) – Biernat (2003)

• Atkinson and Geiser (2009) claim that SAT coefficients are “decisively diminished” when SES and HSGPA are controlled.

Consistent SES FindingsSource Sample rtest-grade rtest-grade.SES

U California 77,000 .37 .34

Sackett, Kuncel, et al. 2009a 155,000 .35 .33

Sackett, Kuncel, et al. 2009b 17,000 .37 .36

Moderate observed

relationship

Does not go to zero when

controlling for SES

University of California System Data

HSGPA .30

SAT I .22

Family Income .03

Parents Education .05

“Ok, but I know someone who had really high test scores and they didn’t do as well as another friend who had lower test scores. So scores only matters to a certain degree, right?”

More is also better at school

• Arneson and Sackett (2009)

1.5

2

2.5

3

3.5

4

600 700 800 900 1000 1100 1200 1300 1400 1500 1600

SAT Score

Ad

just

ed F

resh

man

GP

A

Coward & Sackett (1990)• 174 studies on the relationship between

intelligence and job performance• Studies used the 9 scale GATB (General

Aptitude Test Battery)• Total sample size across all studies was a

substantial 36,614• Found overwhelming support for a linear

relationship between ability and job performance

• The commonly held notion appears to be incorrect

Traditional Supplementary Predictors

• Letters of Recommendation

• Personal Statements

• Interviews

00.10.20.30.40.5

GPA

Gra

d.Sc

hool

Facu

ltyR

atin

g

Deg

ree

Com

plet

ion

Res

earc

hPr

oduc

tivity

00.10.20.30.40.5

GPA

Grad

.Sc

hool

Facu

ltyRa

ting

Degr

eeCo

mpl

etio

n

Rese

arch

Prod

uctiv

ity

Letters GRE

Result suggest no incremental validity for letters, save for degree completion. Note that all letters were structured and quantitative.

Kuncel, Vanelli, & Ones (2009)

Cooper-Murphy, Klieger, Borneman, & Kuncel (2007) College and University

Personal Statements

Admissions Interview: Results

N k 80% cred.

3540 37 0.12 0.11 0.04 .06 to .17

2641 23 0.15 0.1 0.04 .10 to .21

262 5 0.24 0.25 0.21 -.03 to .51

231 3 0.13 0.07 0 .13 to .13

robs SDobs SDr

Grade Point Average in Graduate/Medical School

Medical Clinical and Internship Performance Rating

Graduate School Performance Rating (non Medical)

Medical Board Examinations

Yet There is Good Stuff Out There

Study Habits, Skills, and Attitudes

Crede & Kuncel (2007) Perspectives in Psychological Science

N k r

Aggregate Measures 18,517 107 .33

Study Skills 24,547 87 .25

Study Habits 23,390 102 .23

Study Attitudes 7,211 37 .26

0

0.1

0.2

0.3

0.4

0.5

SHSA SAT HSGPA

Class Attendance: Being There

Crede, Roch, & Kieszczynka (2010) Review of Educational Research

N k r

Attendance 21,164 68 .41

Personality Predictors of Graduate Student Performance

Rigdon & Kuncel, 2010

N k robs SDobs SDrGraduate GPA

Emotional Stability 429 6 .24 .12 .03

Achievement 1,645 7 .16 .08 .04

Task Performance

Autonomy 518 3 .16 .02 .00

Ach. Via Indep. 132 3 .27 .12 .00

Qualifying Exam

Ach. Via Indep. 156 2 .26 .00 .00

Obtain Ph.D.

Emotional Stability 1,180 3 .09 .10 .08

Conscientiousness 1,559 5 .08 .09 .07

Cautionary Tales

• Faking and Coaching– Most alternatives that have been proposed would

be highly susceptible to deliberate faking or test preparation coaching

• Personality Assessments– Other ratings

• Study habits, attitude, and skill inventories– Placement and guidance

• Biodata– Verifiable content

• Situational judgments tests

Fairness and Bias

• Extensive research on standardized test scores for bias

• Social class has also been scrutinized.

• Data for letters of recommendation and personal statements in academic settings is thinner.

• A summary of current knowledge on the next slide…

New Alternatives Should be Subjugated to Extensive Study for Bias Before Implementation