jack buckley national center for education statistics (nces) washington, dc october 2013

1

Highlights from the Program for the International Assessment of Adult

Competencies (PIAAC), 2012

Jack BuckleyNational Center for Education Statistics (NCES)Washington, DCOctober 2013

What is PIAAC? International large-scale assessment administered in 2011-12 in 23

countries 16- to 65-year-olds, non-institutionalized, residing in the country,

irrespective of nationality, citizenship, or language status Laptop computer or paper-and-pencil:

In the U.S., 80% took the computer tests and 15% took the paper-and-pencil tests.

Assessment subjects: Literacy, Numeracy, and Problem Solving in Technology-Rich

Environments Conducted in English in the U.S.:

Background survey in English or Spanish. About 4% could not complete the questionnaire because of language difficulties or learning or mental disabilities, and 1% could not complete it for other reasons.

2

3

General patterns of U.S. results

Below international average in all subject areas Ranked better in literacy than in numeracy or problem

solving in technology-rich environments Higher percentage at low proficiency levels than

international average Percentages of top performers similar to or slightly

lower than international average, depending on the subject

Performance gap between young and older population smaller than the average gap internationally

4

Participating countries

4

2012 2015

AustraliaAustriaBelgiumCanadaCyprusCzech RepublicDenmarkEstoniaFinlandFranceGermanyIreland

ItalyJapan Korea, Rep ofNetherlandsNorwayPolandSlovak RepublicSpainSwedenUnited Kingdom United States

ChileGreece IndonesiaIsraelLithuaniaNew ZealandSingaporeSloveniaTurkey

5

What PIAAC reports

Average Scores: Reported on a scale of 0-500 for all domains.

Proficiency Levels: Reported as the percentages of adults scoring at six performance levels in literacy and numeracy and at four performance levels in problem solving in technology-rich environments.

6

Literacy proficiency levels

Locate single piece of information in familiar texts.

Read relatively short digital, print or mixed texts to locate single text.

Make matches between text and information that may require low level para-phrasing and drawing low-level inferences.

Identify, interpret, or evaluate one or more pieces of information and often require varying levels of inference.

Perform multiple-step operations to integrate, interpret, or synthesize information from complex texts, and may require complex inferences.

Integrate information across multiple, dense texts; construct syntheses, ideas or points of view; or evaluate evidence based arguments.

Below Level 1

Level 1Level 2

Level 3

Level 4

Level 5

7

Literacy example item

Below Level 1: Election results The stimulus is a report of the results of a union election. It consists of several brief paragraphs and a simple table identifying the three candidates and the number of votes they received. The test taker is asked to identify which candidate received the fewest votes. To do this, the test taker must simply compare the number of votes that each candidate received. The word “votes” appears only in the question and in the table. Therefore, the task consists of recognizing this direct relationship between the two to infer the answer.

8

Literacy example item

Level 4: Library search The stimulus displays the results of a bibliographic search from a simulated library website. The test taker is asked to identify a book suggesting that the claims made both for and against genetically modified foods are unreliable. To do this, the test taker needs to read the title and description of each book included in the search results. Many pieces of distracting information are present. The necessary information must be inferred from the statement that the author “describes how both sides in this hotly contested debate have manufactured propaganda, tried to dupe the public and . . . [text ends].”

9

Numeracy proficiency levels

Perform basic tasks: counting, arithmetic operations with whole numbers.

Perform one-step tasks: count; sort; arithmetic operations; understanding simple percent (ex. 50%).

Perform 2 or more calculations, simple measurement; spatial representation; estimation; and interpret simple tables, graphs.

Understand & work with mathematical patterns, proportions, basic statistics expressed in verbal or numerical form.

Perform analysis, complex reasoning, statistics and chance; spatial relationships; and communicat-ing well-reasoned explanations for answers.

Understand complex abstract mathema-tical and statistical ideas, embedded in complex texts, draw inferences; arguments or models; justify, reflect on solutions or choices.

Below Level 1

Level 1

Level 2

Level 3

Level 4

Level 5

10

Numeracy example item

Below Level 1: Price tag The stimulus for this item consists of four supermarket price tags. The tags identify the product, the price per pound, the net weight, the date packed, and the total price. The test taker is asked to indicate the item that was packed first by simply comparing the dates on the price tags.

11

Numeracy example item

Level 4: Education level The stimulus for this item consists of two stacked-column bar graphs presenting the distribution of the Mexican population by years of schooling for men and women separately. The y axis of each graph is labeled “percentage” and includes grid lines for 0, 20, 40, 60, 80, and 100%. The x axis is labeled “year” and presents data for 1960, 1970, 1990, 2000, and 2005. A legend identifies three categories of schooling: “more than 6 years of schooling,” “up to 6 years of schooling,” and “no schooling.” The test taker is asked to indicate what percentage of men in Mexico had more than 6 years of schooling in 1970, choosing from a pull-down menu that has 10 response categories: “0–10%,” “10–20%,” and so on.

12

Problem solving in technology-rich environments proficiency levels

Tasks are well-defined involving use of only one function within a generic interface.

Tasks require little or no navigation, and only a few steps to access information for solving the problem. There are few monitoring demands.

Tasks require some navigation across pages and applications for solving the problem. Evaluating the relevance, some integration and inferential reasoning may be needed.

Task may involve multiple steps and operators, navigation across pages and applications. There are typically high monitoring demands, and evaluation of relevance and reliability of information.

Below Level 1

Level 1

Level 2

Level 3

13

Problem Solving in technology-rich environments example item

Level 1: Party invitations This task involves sorting e-mails into pre-existing folders in an inbox. An e-mail interface is presented with five e-mails in an inbox. The test taker is asked to place the e-mails, which are responses to a party invitation, into folders to keep track of who can and cannot attend a party. The task is performed in a single environment, and the goal is explicitly stated. Solving the problem requires a relatively small number of steps and does not demand a significant amount of monitoring across a large number of actions.

14

Problem Solving in technology-rich environments example item

Level 3: Meeting rooms This task involves managing requests to reserve a meeting room on a particular date using a reservation system. The task presents two applications: an e-mail interface (with a number of e-mails requesting reservations stored in an inbox) and a web-based reservation tool that allows the user to assign rooms to meetings at certain times. Successfully completing the task involves taking into account multiple constraints (i.e., the number of rooms available and existing reservations). These constraints generate a conflict (one of the demands for a room reservation cannot be satisfied), which has to be resolved by issuing a standard message to decline one of the requests.

15

U.S. PIAAC Findings

16

U.S. average literacy score (270) lower than the international average (273)

Lower than in 12 countries:Japan, Finland, Netherlands, Australia, Sweden, Norway, Estonia, Flanders-Belgium, Czech Republic, Slovak Republic, Canada, Republic of Korea

Not significantly different than in 5 countries:England and Northern Ireland- U.K., Denmark, Germany, Austria, Cyprus

Higher than in 5 countries:Poland, Ireland, France, Spain, Italy

17

Below level 1 range: 0-175Level 1 range: 176-225Level 2 range: 226-275Level 3 range: 276-375Level 4/5 range: 376-500

Median

Seven countries had higher percentages of adults reaching the highest proficiency level (4/5) in literacy

Median

18

Higher proportion of U.S. adults at the bottom levels of literacy

Lower than in 12 countries:Japan, Finland, Netherlands, Republic of Korea, Estonia, Flanders-Belgium, Australia, Sweden, Poland, Czech Republic, Germany, Austria

Not significantly different than in 8 countries:Denmark, Slovak Republic, Canada, Norway, France, Ireland, Cyprus, England and Northern Ireland- U.K.

Higher than in 2 countries:Spain, Italy

U.S. 16- to 24-year-olds below international average in literacy, rank lower than 16- to 65-year-olds overall

19

20

Only oldest U.S. adults outperformed the international average in literacy

*p < .05. U.S. average score is significantly different from PIAAC international average.

55-6545-5435-4425-3416-24

55-6545-5435-4425-3416-24

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300

263*266

273*275*

272*

255268

279284

279

21

Least educated adults below the international average in literacy


Graduate or professional degree

Bachelor's degree

Associate's degree

High school credential

Below high school


Bachelor's degree

Associate's degree


Below high school

Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300 350

310

298

283

262*

230*

307

298

285

271

246

22

Employed adults in the U.S. had lower average literacy scores than their peers internationally


Employed

Unemployed

Out of the labor force

Employed

Unemployed


Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300

274*

260

257

277

265

261

23

U.S. White adults had higher average literacy scores than either Black or Hispanic adults

*p < .05. Average score is significantly different from White average.

White

Black

Hispanic

Other

0 50 100 150 200 250 300

283

244*

233*

272*

24

U.S. gaps in literacy scores larger than international average by parental education and nativity status

25

In literacy, U.S. gaps larger by educational attainment and skill level of job, but similar to international average by income and employment status

26

U.S. gaps in literacy scores similar to international average by gender, smaller by age, and larger by health status

27

U.S. average literacy score in 2012 not significantly different from 2003, but lower than in 1994

*p < .05. Average score is significantly different from PIAAC.

1994 2003 2012245

250

255

260

265

270

275

280

285

IALS:273*

ALL:268

PIAAC:270

28

Summary of literacy findings:

Lower overall literacy scores than international average

Higher percentage of low performers than international average

Gaps between less advantaged socio-economic groups and more advantaged peers higher in U.S. than internationally

No change in overall U.S. literacy scores since 2003

U.S. average numeracy score (253) lower than the international average (269)

Lower than in 18 countries:Japan, Finland, Flanders-Belgium, Netherlands, Sweden, Norway, Denmark, Slovak Republic, Czech Republic, Austria, Estonia, Germany, Australia, Canada,Cyprus, Republic of Korea, England and Northern Ireland- U.K., Poland

Not significantly different than in 2 countries:Ireland, France

Higher than in 2 countries:Italy, Spain

29

30

Fifteen countries had higher percentages of adults reaching the highest proficiency level (4/5) in numeracy

Below level 1 range: 0-175Level 1 range: 176-225Level 2 range: 226-275Level 3 range: 276-375Level 4/5 range: 376-500

Median

31

Higher proportion of U.S. adults at the bottom levels of numeracy

Lower than in 21 countries:Netherlands, Finland, Japan, Flanders-Belgium, Republic of Korea, Austria, Estonia, Sweden, Czech Republic, Slovak Republic, Germany, Denmark, Norway, Australia, Poland, Canada, Cyprus, France, Ireland, England and Northern Ireland- U.K., Spain

Not significantly different than in 1 country:Italy

U.S. 16- to 24-year-olds below international average in numeracy, rank lower than 16- to 65-year-olds overall

32

33

U.S. adults in all age groups below international average in numeracy


55-65

45-54

35-44

25-34

16-24

55-65

45-54

35-44

25-34

16-24

Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300

247*

250*

258*

260*

249*

253

265

275

279

271

34

U.S. adults at every education level below the international average in numeracy



Bachelor's degree

Associate's degree


Below high school


Bachelor's degree

Associate's degree


Below high school

Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300 350

302*

287*

267*

243*

204*

308

296

283

268

237

35

Employed adults in the U.S. had lower average numeracy scores than their peers internationally


Employed

Unemployed


Employed

Unemployed


Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300

260*

236*

232*

275

256

253

36

U.S. White adults had higher average numeracy scores than either Black or Hispanic adults

*p < .05. Average score is significantly different from White average.

White

Black

Hispanic

Other

0 50 100 150 200 250 300

268

212*

215*

258*

37

U.S. gaps in numeracy scores larger than international average by parental education, not different by nativity status

38

In numeracy, U.S. gaps larger by educational attainment, income, and skill level of job, but similar to international average by employment status

39

In numeracy, U.S. gap similar to international average by gender, smaller by age, and larger by health status

40

U.S. average numeracy score in 2012 lower than in 2003

*p < .05. Average score is significantly different from PIAAC.

2003 2012235

240

245

250

255

260

265

270

275

ALL:262*

PIAAC:253

41

Summary of numeracy findings:

Lower overall numeracy scores than international average

Higher percentages of low performers than international average

Regardless of educational level or gender, U.S. adults lower than international average

Lower U.S. numeracy scores than in 2003

U.S. average problem solving in technology-rich environments score (277) lower than the international average (283)Lower than in 14 countries:Japan, Finland, Australia, Sweden, Norway, Netherlands, Austria, Denmark, Czech Republic, Republic of Korea, Germany, Canada, Slovak Republic, Flanders-Belgium

Not significantly different than in 4 countries:England and Northern Ireland- U.K., Estonia, Ireland, Poland

42

Eight countries had higher percentages reaching the highest proficiency level (3) in problem solving in technology-rich environments

43

Below level 1 range: 0-240Level 1 range: 241-290Level 2 range: 291-340Level 3 range: 341-500

Median

44

Higher proportion of U.S. adults at the bottom levels of problem solving in technology-rich environments

Lower than in 14 countries:Republic of Korea, Finland, Sweden, Netherlands, Japan, Flanders-Belgium, Czech Republic, Norway, Australia,Germany, Austria, Canada, Denmark, Estonia

Not significantly different than in 4 countries:England and Northern Ireland- U.K., Slovak Republic, Poland, Ireland

U.S. 16- to 24-year-olds below international average in problem solving in technology-rich environments

45

46

Only oldest U.S. adults outperformed the international average in problem solving in technology-rich environments

*p < .05. U.S. average scores are significantly different from PIAAC international average.

55-65

45-54

35-44

25-34

16-24

55-65

45-54

35-44

25-34

16-24

Unite

d St

ates

PIAA

C In

tern

ation

al a

vera

ge

0 50 100 150 200 250 300 350

267*

271

279*

283*

285*

259

272

285

295

295

47

Least educated adults below the international average in problem solving in technology-rich environments



Bachelor's degree

Associate's degree


Below high school


Bachelor's degree

Associate's degree


Below high school

Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300 350

301

296

282

268*

259*

304

299

286

278

269

48

Employed adults in the U.S. had lower average problem solving in technology-rich environments scores than their peers internationally


Employed

Unemployed


Employed

Unemployed


Unite

d St

ates

PIAA

C in

tern

ation

al a

vera

ge

0 50 100 150 200 250 300

279*

271*

272*

285

279

278

49

U.S. White adults had higher average problem solving in technology-rich environments scores than either Black or Hispanic adults

*p < .05. Average scores are significantly different from White average.

White

Black

Hispanic

Other

0 50 100 150 200 250 300

285

252*

255*

276*

50

In problem solving in technology-rich environments, U.S. gaps similar to international average by educational attainment, income, employment status, and skill level of job

51

In problem solving in technology-rich environments, U.S. gaps similar to international average by gender and health status, but smaller by age

52

Summary of problem solving in technology-rich environments findings:

Lower overall problem solving in technology-rich environments scores than international average

Higher percentages of low performers than international average

Gaps between youngest and oldest age groups smaller in U.S. than internationally

53

U.S. PIAAC Findings Summary:U.S. PIAAC Findings Summary: Lower overall scores than international average in all

subjects Higher percentages of low performers than

internationally Larger gaps between less advantaged and more

advantaged peers in literacy and numeracy, but not in problem solving in technology-rich environments

Relatively lower performance of young adults and those with high school education or less

Relatively higher performance of older adults in literacy and problem solving in technology-rich environments

55

For more information

Contact:Eugene Owen

NCES202-502-7422

[email protected]

PIAACPIAAC at NCES:

http://nces.ed.gov/surveys/piaac/

mailto:[email protected]

http://nces.ed.gov/surveys/piaac/

jack buckley national center for education statistics (nces) washington, dc october 2013

Documents