s. white 8-04 1 the health literacy component (hlc) of the 2003 naal the health literacy component...

1 S. White 8-04

The Health Literacy

Component (HLC)

of the 2003 NAAL

The Health Literacy

Component (HLC)

of the 2003 NAAL

Sheida White

Project Officer

National Assessment of Adult Literacy

(NAAL)

Sheida White

Project Officer


(NAAL)

An Overview of the

2003 National

Assessment of Adult

Literacy (NAAL)

An Overview of the

2003 National

Assessment of Adult

Literacy (NAAL)

Sheida White

Project Officer


(NAAL)

Sheida White

Project Officer


(NAAL)

A webcast presentationSeptember 8, 2004

2 S. White 8-04

Background, key features, and major goals

Performance of literacy tasks

Skills required to perform literacy tasks

Assessment design and administration

Data analysis and reporting

Topics covered by this presentationTopics covered by this presentation

3 S. White 8-04

Background, Key Features, and Major Goals

4 S. White 8-04

In 1985 and 1992, the National Center for Education Statistics (NCES) conducted adult literacy assessments that focused on real-world tasks

The 2003 National Assessment of Adult Literacy (NAAL) builds on and expands previous NCES adult literacy assessments

NAAL is the first assessment of the nation’s progress in adult literacy since the 1992 National Adult Literacy Survey (NALS)

NAAL backgroundNAAL background

5 S. White 8-04

Is based on in-person interviews of approximately 18,000 Americans age 16 and older

Provides data for the United States as a whole, individual states choosing to participate in a state-level assessment, and the nation’s prison population

Emphasizes the use of printed everyday materials (such as bills, prescriptions, newspapers) needed to function adequately in one’s environment

Provides data on background characteristics, performance of everyday literacy tasks, and skills underlying task performance

Key features of NAALKey features of NAAL

6 S. White 8-04

Describe the status of adult literacy

Report on trends between 1992 and 2003

Identify relationships between literacy and selected characteristics of adults

Provide new information about skills underlying adult literacy, including basic reading skills of the least-literate adults

Facilitate use of NAAL data by diverse audiences, including policymakers, researchers, and educators

Major goalsMajor goals

7 S. White 8-04

Performance of Literacy Tasks

8 S. White 8-04

NAAL’s task-based literacy definitionNAAL’s task-based literacy definition

Literacy is the ability to use

printed and written infor-

mation [prose, document,

and quantitative] to function

in society, to achieve one’s

goals, and to develop one’s

knowledge and potential

S. White 8-04

9 S. White 8-04

Definitions of three key literacy areasDefinitions of three key literacy areas

10 S. White 8-04

Measurement of prose, document, and quantitative literacy

Measurement of prose, document, and quantitative literacy

NAAL’s prose literacy, document literacy, and quantitative literacy scale scores are comparable to those of the 1992 NALS

Scores are based on completion of items that are all Classified as prose, document, or quantitative Open-ended to represent everyday tasks

Placed before stimulus materials to set a purpose for reading

Most items require Short written responses

Searching the text for specific information

11 S. White 8-04

Topics of NAAL stimulus materialsTopics of NAAL stimulus materials

12 S. White 8-04

Sample prose item used in 1992Sample prose item used in 1992

Vitamin E (tocopherol)—helps protect red blood cells.May aid the circulatory system and counteract the agingprocess. Best sources: wheat germ, whole grains, eggs, pea-nuts, organ meats, margarine, vegetable oils, green leafy veg-etables.

S. White 8-04

13 S. White 8-04

Sample document item used in 1992Sample document item used in 1992

S. White 8-04

14 S. White 8-04

Sample quantitative item used in 1992Sample quantitative item used in 1992

S. White 8-04

15 S. White 8-04

NAAL includes 28 health-related items (out of 153 items)

All participants receive some health-related items

All health-related items are also classified as prose, document, or quantitative items

The health-related items

Are included with other items when calculating the prose, document, and quantitative scale scores

Are used to calculate a separate health literacy score, based solely on health-related items

A new health literacy score based on health-related items

A new health literacy score based on health-related items

16 S. White 8-04

NAAL’s Health Literacy Component was developed in response to a request by the U.S. Department of Health and Human Services (HHS)

NAAL focuses on one key aspect of health literacy—the ability to use printed and written materials to accomplish a broad range of health-related tasks

NAAL’s health-related tasks include Clinical tasks, having to do with the health care itself Preventive tasks, having to do with healthy habits and

prevention of illness

Navigation tasks, having to do with bureaucratic demands to get to the health care

Development and focus of the NAAL Health Literacy Component

Development and focus of the NAAL Health Literacy Component

17 S. White 8-04

Skills Required to Perform Literacy Tasks

18 S. White 8-04

Purpose of the Adult Literacy Supplemental Assessment (ALSA)

Purpose of the Adult Literacy Supplemental Assessment (ALSA)

The 1992 NALS provided little information about the literacy abilities of the least-literate adults, who were not able to complete the assessment

The purpose of the new Adult Literacy Supplemental Assessment (ALSA) is to get a clearer picture of the skills of the least-literate adults

ALSA is intended only for adults who would not be capable of meaningful participation in the “main NAAL” assessment, because they lack the skills to perform even the easiest tasks with a high degree of success

These adults take ALSA instead of the main NAAL

19 S. White 8-04

Selection of ALSA participantsSelection of ALSA participants

All NAAL participants take seven easy core screening items

Interviewers score participants’ answers using the scoring guides provided on their computers

Computer selects ALSA participants using an empirically derived algorithm that predicts very low performance

The algorithm predicts which participants would get a total of five items or fewer correct (including the core items)

Five items, spread across the three NAAL scales, would not be sufficient to allow accurate estimates of performance on any of the scales

20 S. White 8-04

Important features of ALSAImportant features of ALSA

ALSA instructions and responses are given orally and can be in either English or Spanish

ALSA materials are

Contextualized (with logos, pictures, etc.)

Familiar (as measured by background questions)

Tangible (e.g., food boxes, medicine bottles, utility bills)

ALSA tasks, although simple and contextualized, still require participants to read at least parts of the words

21 S. White 8-04

The need for additional data about skill deficitsThe need for additional data about skill deficits

In addition to the 6% of adults who lacked the skills to complete the 1992 NALS, an additional 15% of adults were able to perform only the easiest NALS tasks (classified as “Level 1” tasks)

For adults with low scores, more information is needed about the nature of the skill deficits

Lack of basic (word-level) reading skills?

Lack of fluency in basic skills?

Lack of higher level literacy skills?

22 S. White 8-04

Range of skill deficits and their potential implications

Range of skill deficits and their potential implications

23 S. White 8-04

New data on the skills of all adultsNew data on the skills of all adults

The Fluency Addition to NAAL (FAN) provides new data on the skills of all adults

After completing either the main NAAL or ALSA, all participants take FAN, which

Consists of timed oral reading tasks

Describes the oral reading skills of adults (e.g., words read correctly per minute)

Provides a measure of basic (word-level) reading skills

FAN scores can be compared with main NAAL scores at various performance levels, ALSA scores, and health literacy scores

24 S. White 8-04

FAN oral reading materials and associated measures

FAN oral reading materials and associated measures

Pseudoword lists, consisting of possible but nonoccurring English forms (e.g., “wike”), provide a measure of adults’ ability to “decode” (or identify the sounds of) words with which they are not familiar

Word lists, consisting of English words arranged in increasing order of difficulty, provide a measure of adults’ ability to recognize familiar words (often referred to as “sight words”) as well as to decode

Text passages, consisting of 150–200 words each, provide a measure of adults’ ability to read words in connected texts

25 S. White 8-04

Initial measures and scoring challenges for FANInitial measures and scoring challenges for FAN

Initial FAN target measures are

Total words read aloud—whether correctly or not—per unit of time*

Words read correctly per unit of time*

Words read correctly as a percentage of total words read

Challenges in scoring FAN data include ensuring

That correctness can be measured reliably

That speakers of nonstandard varieties of English are not unfairly penalized

*Unit of time: 60 seconds for text passages or 20 seconds for lists

26 S. White 8-04

Key similarities and differences between FAN and ALSA

Key similarities and differences between FAN and ALSA

Similarities include the following:

Both measure basic reading skills

Both have oral instructions in either English or Spanish

Both require reading words and connected text

Differences include the following:

Only ALSA permits the use of compensatory strategies, while FAN does not provide any nonlinguistic clues (such as pictures)

ALSA connected texts are fewer, easier, and shorter (12 sentences each in ALSA, compared with 150200 words each in FAN)

27 S. White 8-04

The functionality of word-level skills and higher level literacy skills

The functionality of word-level skills and higher level literacy skills

NAAL defines literacy as the ability to use printed and written information

As defined by NAAL, all literacy is functional; therefore, NAAL does not differentiate between “literacy” and “functional literacy”

Certain skills, including word-level reading skills, are needed to successfully perform NAAL tasks

The ability to read words is a functional skill, but not the only one

In addition to word-level skills, NAAL recognizes six types of literacy skills

28 S. White 8-04

The six types of literacy skillsThe six types of literacy skills

NAAL literacy skillsNAAL literacy skills

Language comprehension

skills

Understanding the structure and meaning of sentences

Text search skills

Searching textefficiently

Inferential skills

Drawing appropriate text-based inferences

Application skills

Identifying the calculations required to solve quantitative problems

Computation identification

skills

Computation performance

skills

Performing any required calculations

Using newly searched, inferred, or computed information to accomplish a variety of goals

29 S. White 8-04

Analysis of main NAAL data in terms of underlying skills

Analysis of main NAAL data in terms of underlying skills

Recently developed multidimensional analysis methods will be applied to 2003 main NAAL data in order to

Characterize the proficiency of adults in terms of the NAAL literacy skills

Characterize effects of task demands and written materials on the difficulty of literacy tasks

30 S. White 8-04

Assessment Design and Administration

31 S. White 8-04

Selection of the national sample for NAAL Selection of the national sample for NAAL

32 S. White 8-04

Key features of the NAALKey features of the NAAL sampleKey features of the NAALKey features of the NAAL sample

Adults age 16 and over living primarily in households

Nationally representative sample of 10,006 adults, augmented by

Samples from the six states that participated at the state level (an additional 7,166 adults)

Prison sample of inmates (about 1,000 more adults)

Incentive payment of $30 to increase representativeness of the sample and response rate

Oversampling of Blacks and Hispanics

33 S. White 8-04

Matrix samplingMatrix sampling

The main NAAL has a total of 153 items (including the 7 core screening items that are given to all participants and used to select ALSA participants)

Matrix sampling—administering only a portion of the non-core items to each respondent—ensures broad coverage while limiting respondent burden

Each respondent takes 1 of 26 booklets

Each booklet includes 3 of 13 blocks of items (for about 40 items per booklet, including the core items)

Each block is spiraled in the 26 booklets (i.e., paired with every other block)

34 S. White 8-04

Allowing comparisons with the 1992 NALS while introducing new features

Allowing comparisons with the 1992 NALS while introducing new features

1992 assessment

13 blocks total

6 from 1985

7 new for 1992

2003 assessment

13 blocks total

6 from 1992

7 new for 2003

Six of the blocks from 1992 are used again in 2003.

Seven new (2003) blocks replace all six of the blocks from 1985 plus one of the blocks from 1992.

S. White 8-04

35 S. White 8-04

The seven blocks that were newly created for 2003

Are similar to the 1992 blocks in terms of the distribution of skills required by tasks

Have approximately the same average task difficulty as the 1992 blocks

Are spiraled with the 1992 blocks in the 26 booklets

Item response theory will be used to link the 1992 and 2003 scales using the tasks common to both assessments

Allowing comparisons with the 1992 NALS while introducing new features (continued)

Allowing comparisons with the 1992 NALS while introducing new features (continued)

36 S. White 8-04

New performance levels for 2003New performance levels for 2003

In 1992, five performance levels were developed by grouping together tasks of similar difficulty

The 1992 levels proved to have some limitations Scores that are the cut points between levels vary

depending on the measure of task difficulty used No distinctions are made within the lowest level of

performance (Level 1)

For 2003, new performance levels are being developed for the main NAAL by the National Academy of Sciences

Goals of the new levels include increased clarity and better differentiation among adults at the lower end of the scale

37 S. White 8-04

Main NAAL ALSA

Background questionnaire 28 min 28 min

Core screening items 8 min 8 min

Main NAAL or ALSA assessment 39 min 25 min

FAN 14 min 12 min

Total 89 min 73 min

Respondents may take as long as they need to perform main NAAL or ALSA tasks, but are encouraged to move to a new task when stuck or frustrated

NAAL total average assessment timeNAAL total average assessment time

38 S. White 8-04

The assessment is conducted in the participant’s home

The assessment is administered one on one

All participants receive additional time to complete the main NAAL or ALSA if they need it

The background questionnaire is administered orally in either English or Spanish

Instructions for the core screening items and for ALSA are given orally in either English or Spanish

Participants with a native language other than English or Spanish may take the core screening items even if they cannot complete the background questionnaire

Accommodations for adults with special needsAccommodations for adults with special needs

39 S. White 8-04

Expanded NAAL background questionnaireExpanded NAAL background questionnaire

For 2003, the background questionnaire was expanded to better reflect

Economic and technological developments in the past decade

The background of low-performing adults

Health-related activities of adults

Literacy-related activities of parents and their children

40 S. White 8-04

Characteristics of the NAAL scoring rubrics:Statistical considerations

Characteristics of the NAAL scoring rubrics:Statistical considerations

Responses to each item are analyzed for

Frequency with which a particular response is given

Interrater reliability in scoring responses

P-value (percentage of the population responding correctly)

Biserial (and polyserial) correlation between each response (and partially correct response) and the respondent’s overall score

Range findings for delineation of the range of responses received

Differential item functioning (DIF) in favor of certain groups

41 S. White 8-04

Seek evidence that adults can indeed use printed materials

Allow for partially correct responses if the information provided is still useful in accomplishing the task

Score most responses as either correct or incorrect, since often a partially completed task is not an accomplished task

Allow for writing errors as long as the overall meaning of the response is correct

Distinguish responses providing specific information from those providing generalizations

Development of the NAAL scoring rubrics:Content considerations

Development of the NAAL scoring rubrics:Content considerations

42 S. White 8-04

Data Analysis and Reporting

43 S. White 8-04

AM software has been developed by the American Institutes for Research (AIR), with NCES funding

The new software will reduce the amount of time required to analyze data by several months

The software will allow the technical report and the Data File Users Manual to be produced on a more timely schedule than in 1992

The software will also make it easier for states and researchers to do their own data analyses to supplement the information in the state and national reports

New software for data analysis New software for data analysis--

44 S. White 8-04

Average scores and distributionsAverage scores and distributions

Average scale scores and score distributions will beprovided

For the population as a whole

For population groups

But not for individuals

45 S. White 8-04

Inappropriateness of individual scoresInappropriateness of individual scoresInappropriateness of individual scoresInappropriateness of individual scores

No respondent takes all of the assessment

It would not be appropriate to calculate individual scores based on part of the assessment because

The individual scores would be unreliable (a reliable or consistent measure requires many items)

The individual scores would not be a valid representa-tion of the domain of adult literacy (valid representation requires many types of tasks, genres, etc.)

46 S. White 8-04

Trend analysisTrend analysis

Results from the 1992 NALS and the 2003 main NAAL will be compared in terms of

Average scale scores for prose, document, and quantitative literacy

Performance levels set by the National Academy of Sciences

P-values (the percentage of the population giving the correct answer to each item)

47 S. White 8-04

New release of items and p-valuesNew release of items and p-values

About 90 assessment items used in 1992 assessment will soon be released

All released items will be available on the NAAL website (http://nces.ed.gov/naal)

Each item’s p-value will be published

This will be the first publication of p-values for 1992 assessment items

P-values will be published for all items used in 1992—even items that are not released

48 S. White 8-04

Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #1


49 S. White 8-04



50 S. White 8-04

New types of results in 2003New types of results in 2003

Health literacy scale scores measuring the performance of American adults on health-related tasks

Skill-based analyses inferring the literacy skills associated with the ability to perform main NAAL tasks

ALSA scores providing information on the literacy skills and deficits of America’s least-literate adults

FAN oral reading and basic skills data providing information about the skills of all adults

51 S. White 8-04

If you would like to receive NAAL information tailored to your interests, simply fill out a short online form at http://www.edpubs.org/list/NAAL

On the form, you can indicate up to 16 areas of interest Adult basic education Adult secondary education Citizenship/civics education College prep Computer literacy Correctional education Early childhood/children’s literacy ESL/ESOL (English as a second

language)

NAAL mailing listNAAL mailing list

Family literacy Health literacy Homeless education Learning disabilities Mathematical literacy Native language literacy (i.e., literacy in Spanish) Policy and legislation Workplace literacy

52 S. White 8-04

Sheida WhiteNAAL Project Officer

(202) [email protected]

Andrew KolstadSenior Technical Adviser


Websitehttp://nces.ed.gov/naal

Sheida WhiteNAAL Project Officer


Andrew KolstadSenior Technical Adviser


Websitehttp://nces.ed.gov/naal

National Center for Education StatisticsNational Center for Education Statistics

s. white 8-04 1 the health literacy component (hlc) of the 2003 naal the health literacy component...

Documents