Minimizing Differencesthrough Cross-Agency

Development of Measures

Julie D. Weeks, Ph.D.National Center for Health Statistics

2012 FCSM Statistical Policy SeminarWashington, D.C.December 4-5, 2012

The ACS Disability Measure:A Case Study

The development of the six-question ACS disability measure can serve as a model for future collaborations.

•Starting early in the data collection life-cycle;

•Involving a broad spectrum of stakeholders;

•Understanding the disparate needs for the data;

•Using existing models and conceptualizations as guides;

•Joint efforts in testing, including cognitive testing and analyses using other data sources, to inform development;

•Coordinated implementation and research using resulting data; and

•Promotion to wider federal and data user communities.

Origin of theACS Disability Subcommittee

The 2000-2007 ACS disability questions were the same as those administered as part of the 2000 Census long form.

In 2002, the Interagency Committee for the ACS created a Disability Subcommittee to ensure that the ACS disability questions would meet the needs of data users.

Subcommittee charge:•Review the existing disability question set;•Recommend changes or improvements to provide a more accurate estimate of the population with disabilities.

Resulting recommendations had to be made in time fora) 2006 Survey Content Test, and b) implementation in the 2008 ACS.

The ACS Disability Subcommittee

• NCHS

• Census

• OMB

• BLS

• OASPE

• AoA

• DoE

• OSERS

• NSF

• VA

• HUD

• NIDDR

• NIMH

• NCD

• SSA

• HIS

• AHRQ

• FCC

• DoJ

Legislative Mandatesfor Disability Data

The ACS must provide information on disability in order to satisfy agencies with statutory or programmatic requirements for disability data. A review of legislative mandates associated with a wide range of agencies revealed that data on disability was used by agencies:

• in the distribution of benefits or the provision of opportunities,

• for programmatic needs regarding the overall welfare of the disabled population.

Agencies and organizations with legislative mandates included:

VA, HUD, SSA, AoA, DOC, DoEd, NIH, CMS, OMH, HIS, DOJ, DOT, EEC, FCC, NIDRR, NSF, NCD, ACF.

Measurement ObjectivesBased on a legislative review, programmatic needs for data and the constraints of the ACS format, the subcommittee determined that:

• “equalization of opportunity” would be the primary measurement objective for the disability questions

• questions would be designed to identify persons who, because of their limitations in functioning, are at risk for discrimination or lack appropriate opportunity for participation in all aspects of social life

• one measure of severe disability would be included to identify those in need of assistance to maintain independence (for purposes of service provision planning).

Equalization of Opportunities Adopted as Purpose of Measurement

•Seeks to identify all those at greater risk than the general population for limitations in activity or participation

•Disability as a demographic

0

30

60

90

Pro

port

ion (

%)

Nondisabled Disabled

Percent of persons with HS diploma

Functioning DomainsQuestion development began by focusing on the functional areas most commonly associated with limitations in participation.

Based on analyses conducted on large national datasets, including NHIS, MEPS and SIPP, the functioning domains that were found to best predict the total population estimate of disability, including those identified by mental health problems:

•Vision

•Hearing

•Mobility

•Cognitive functioning

•Self care

•Independence getting outside the home

Testing the New ACS QuestionsTesting of the new question set was required to understand if the new questions:

a) would capture the concepts chosen,b) were more reliable than the old ones, andc) would perform equally well across administration modes.

• Cognitive Testing: June 2004 – February 2005, 5 rounds of interviews conducted (n=69) at NCHS and Census, 3 data collection modes (face-to-face, telephone, PAPI self-admin)

• Field Testing: 2006 ACS Content Test

• 62,900 residences, 48 states

• split ballot: ½ sample is “control” with ACS production version and ½ sample is “test” with modified content

• initial mail-out December 2005, mail returned January 2006, non-response follow-up February 2006

Content Test Results

Based on the criteria set out before the test, the revised questions used in the test interviews performed better than the control or original questions.

•Reliability in the test version was equal to or better for all disability items and the summary “with a disability / no disability” measure.

•Nonresponse in the test version was lower for all disability items and the summary “with a disability / no disability” measure.

ACS Disability Questions as a Basic Standard for Federal Surveys

Surveys conducted by other Federal agencies that have adopted the rationale and specific domains of the ACS instrument:

•CPS•SIPP•NHIS•NCVS•FCC •AHS•Education

Increases the standardization of disability questions and data across surveys.

Provides an opportunity for crosswalk analyses standard disability items across surveys.

Same Questions:Similar or Different Numbers

Using the same questions across surveys does not guaranteed that the same prevalence estimates will be generated.

Issues that account for different numbers:• Questions asked at the family level or person level

• Focus of survey - health, employment, crime, or housing

• Mode of data collection

A coordinated approach is needed to understand the differences and appropriately present data to the public

Prevalence of Disability by Survey: 2009

Disability Item Ages

NHIS

CPS ASEC†

(%)

Family- Style(%)

Person-Style(%)

ACS(%)

Vision difficulty 1+ 1.7 2.1 2.4 2.2

Hearing difficulty 1+ 3.1 3.9 4.1 3.4

Ambulatory difficulty 5+ 6.9 6.6 7.2 6.9

Cognitive difficulty 5+ 3.5 3.9 4.7 4.8

Self-care difficulty 5+ 2.0 1.8 2.2 2.6

Independent living diff 15+ 4.0 4.7 5.1 5.4

Any disability 1+ 11.6 12.0 13.1 12.1

†NOTE: CPS data are for population 16 years and over.

Prevalence of Any Disabilityby Survey and Age: 2009

Age

NHIS

CPS ASEC†

(%)Family- Style

(%)Person-Style

(%)ACS(%)

1-4 --- 0.3 0.7 0.8

5-14 --- 3.7 5.7 5.6

15-34 4.1 5.3 5.7 5.1

35-64 10.0 12.3 14.1 12.7

65-74 22.2 28.6 27.6 26.0

75+ 43.3 51.9 53.5 50.9

†NOTE: CPS data are for population 16 years and over.

Prevalence of Disabilityby Survey and Race/Ethnicity: 2009

Race/Ethnicity

NHIS

CPS ASEC†

(%)Family- Style

(%)Person-Style

(%)ACS(%)

Hispanic 7.7 7.4 8.9 8.5

Non-Hispanic white 12.3 12.9 14.4 13.0

Non-Hispanic black 13.3 14.1 13.2 13.9

Non-Hispanic Asian 5.9 6.3 6.7 6.6

Non-Hispanic Other 15.3 15.5 17.1 13.2

†NOTE: CPS data are for population 16 years and over.

* CPS and ACS cover persons 16-64; NHIS covers persons 18-64.

Development of theACS Disability Measure:

Pros and Cons

• Reduced data collection flexibility (lengthy process);

• Resource-intensive process (staff, time, testing costs, survey costs);

• Multiple stakeholders increase costs (logistics);

• Negotiation is difficult;

• Compromises in each step of measure development;

• Comparability (inter- versus intra-agency);

• Post-data collection coordination prolongs the demands and needs for resources (including data release, production of methodological and analytic documentation, research, and outreach).

The New ACS Disability MeasuresFor sample persons 1 year of age and older:

1. Is this person deaf or does he/she have serious difficulty hearing?2. Is this person blind or does he/she have serious difficulty seeing?

For sample persons 5 years of age and older:

3. Because of a physical, mental, or emotional condition, does this person have serious difficulty concentrating, remembering or making decisions?

4. Does this person have serious difficulty walking or climbing stairs?5. Does this person have difficulty dressing or bathing?

For sample persons 15 years of age and older:

6. Because of a physical, mental, or emotional condition, does this person have difficulty doing errands alone such as visiting a doctor’s office or shopping?