1

Identifying and Selecting Measures for Health Disparities Research

Anita L. Stewart, Ph.D.University of California, San Francisco

Clinical Research with Diverse CommunitiesEPI 222, SpringApril 10, 2008

2

Selecting Measures for Your Own Study: The Problem

You are beginning a study You know the concepts (variables) of interest Question:

Which measure of ________ should I use?» A popular measure» One that a colleague used successfully» Create your own

3

Inappropriate Measures can Result in:

Conceptual inadequacy – Measuring wrong concept for your study

Poor data quality (e.g. missing data) Poor variability Poor reliability and validity Inability to detect true associations

– e.g., no measured change in outcome when change occurred

4

Two Types of Considerations in Selecting Measures

Contextual - factors unrelated to specific measurement tools– Characteristics of target population– Goals of research– Practical constraints

Psychometric - properties of measures within your context

5

Basic Steps in Selecting Appropriate Measures

1. Specify context 2. Define concept for your study3. Locate potential measures for consideration4. Review potential measures for:

a) conceptual match to your definition b) adequate psychometric properties in target

group5. Pretest potential measures in your target group6. Choose best ones based on pretest results OR

adapt if necessary to address problems

6

1. Specify Context

Research question and how concept fits research (outcome, predictor, covariate)

Nature of target population (health, age, SES, race/ethnicity, literacy)

Practical constraints (time, personnel, budget, respondent burden)

7

Step 2: Define Concept For Your Study Define concept from your perspective,

taking into account your…– study questions– target population

For outcomes, describe:– how intervention or independent variables

might affect it– specific types of changes you expect

8

Step 3. Locate Potential Measures

Identify candidate measures for all concepts

Redundancy OK for now Do NOT develop your own questions

unless it is absolutely necessary

9

Locating Measures

For major dependent or independent variables– Multi-item measures with known

psychometric properties For background variables and covariates

– Standardized survey measures

10

Locating Measures

Compendia Organizations and research centers Government agencies National and state surveys Large research studies relevant to your

topic Universities and individual researchers

11

Handout: Locating Measures for Health Disparities Research

To link to websites, need to log on to CADC

http://medicine.ucsf.edu/cadc/cores/measurement/resourcescode.html

12

Locating Measures: Compendia

Specific measures of various concepts are compiled, reviewed, listed, or otherwise provided– Books

» Many books review and critique various measures

– Web» A few websites

13

Compendia by Web

Health and Psychosocial Instruments (HaPI)– Over 120,000 instruments

– http:/gateway.ovid.com/

– Need ovid account – UCSF has one if use computer at work (or vpn) – no need to enter password

14

Compendia by Web

National Cancer Institute website Health behavior constructs: theory,

measurement, and research– Reviews concepts and measures of

constructs such as perceived control, social support, and perceived vulnerability

http://dccps.cancer.gov/brp/constructs/index.html

15

Locating Measures: Organizations and Research Centers

Some organizations and research centers specialize in measurement and provide public access websites – RAND

– Michigan Diabetes Research and Training center

16

RAND Health Program

Measures, scoring manuals, and citations Specialty:

– Quality of care, patient satisfaction– Health-related quality of life

» Generic and disease specific

– Medical Outcomes Study (MOS) measures

www.rand.org/health/ (surveys and tools)

17

RAND Measures of Maternal, Child, and Adolescent Health

Pediatric asthma symptom scale Pediatric quality of life inventory

18

Michigan Diabetes Research and Training Center: Survey Instruments

Diabetes Care Profile (DCP) Diabetes History (DMH) Diabetes Knowledge Test (DKT) Diabetes Attitude Scale (DAS-3) Diabetes Empowerment Scale (DES) Michigan Neuropathy Screening

Instrument (MNSI)

www.med.umich.edu/mdrtc/survey/

19

Inter-University Consortium for Political and Social Research

Maintains archive of social science data Membership-based organization – over

500 member colleges/universities– UCSF is a member

Can search website using keywords to locate studies, data, and questionnaires

www.icpsr.umich.edu/

20

Ottawa Health Decision Centre

Patient and physician decision aidshttp://www.ohri.ca/centres/DecisionAids/default.asp

Patient measures, e.g., decisional conflict, decisional regret, stage of decision making, decision self-efficacyhttp://decisionaid.ohri.ca/eval.html

21

Commonwealth Fund Surveys

Health insurance Medicare Health care quality, patient centered care,

underserved populations Child health development, care of the elderly State health policy, international health policy

www.commonwealthfund.org/surveys/

22

Commonwealth Survey of Physicians

2006 International Health Policy Survey of Primary Care Doctors – Use of information technology

– Access to care

– Availability of financial incentives

– Chronic care management

23

Commonwealth Fund Quality of Care Survey

2001 Health Care Quality Survey Racial and ethnic differences in patient

experiences with health care– focus on quality measures such as patient–

physician communication and access to care 

24

Commonwealth Fund Surveys on Policy

“Modern Healthcare" Health Care Opinion Leaders Survey: Assessing SCHIP– State Children’s Health Insurance Program

Health Care Opinion Leaders Survey: Assessing Health Care Experts' Views on Medicaid and Its Future

25

Block Dietary Data Systems

Now known as NutritionQuest– Official source of Block Food Frequency

Questionnaire Measures of nutrition, dietary intake, and

physical activity. Questionnaire design and dietary analysis

services to nutritional epidemiologists and public health researchers

www.nutritionquest.com/index.htm

26

Locating Measures: Government Agencies

Several federal and state government agencies provide measures for use in health and health disparities research– Information about measures (e.g.

clearinghouse)

– Actual measures

27

Agency for Healthcare Research and Quality (AHRQ)

Medical Expenditure Panel Survey (MEPS) – www.ahrq.gov/data/

National Quality Measures Clearinghouse Consumer Assessment of Health Plans

Survey (CAHPS)– www.ahrq.gov/qual/

28

Department of Veterans Affairs

National Survey of Veterans, 2001www.va.gov/vetdata/surveyresults/index.htm

Survey of Veteran Enrollees’ Health and Reliance Upon VA, 2003www.va.gov/vetdata/healthcare/index.htm

Both include measures of demographics and socioeconomic status, military background, health, health insurance, and VA and non-VA benefits usage.

29

National Cancer Institute

Special initiatives on measures Health Information National Trends

Survey (HINTS)– Compiled cancer screening questions,

identified best ones, conducted extensive pretesting

http://hints.cancer.gov/

30

National and State Surveys

Population surveys Tend to have single-item measures rather than

multi-item scales– Good for “standardized” survey items

Some exceptions– e.g., National Health Interview Survey

administered a 6-item scale of psychological distress (K6 scale) published by Kessler et al.

» Pratt LA, Advance Data, March 30, 2007 (No. 382)

31

Centers for Disease Control and Prevention

Behavioral Risk Factor Surveillance System– Surveys of health behaviors

www.cdc.gov/brfss/questionnaires/index.htm

32

California Health Interview Survey (CHIS)

A behavioral risk surveillance survey modeled after the CDC BRFSS– Numerous languages

www.chis.ucla.edu/– Go to “Methodology – review

questionnaires”

33

CDC: National School-Based Youth Risk Behavior Survey (YRBS)

Survey conducted every other year– random national sample of youth in grades 9-12

– Most states conduct survey Measures of substance use, risky sexual

behaviors, diet, physical activity, overweight

www.cdc.gov/HealthyYouth/yrbs/index.htm

34

National Center for Health Statistics (NCHS)

Surveys and data collection systems Can download

– Any survey or portion of survey See handouts for class 10

– Summary of all surveys

– Instructions for downloading

35

NCHS: Sample of Population Surveys

National Health Interview Survey National Health and Nutrition

Examination Survey National Survey of Family Growth National Maternal and Infant Health

Survey Longitudinal Studies of Aging (LSOA)

www.cdc.gov/nchs/express.htm

36

NCHS National Health Care Surveys: Surveys of Physicians

Family of provider-based surveys Provide objective, reliable information

about– organizations and providers

– services rendered

– patients they serve

www.cdc.gov/nchs/nhcs.htm

37

Example: National Ambulatory Medical Care Survey (NAMCS)

Samples MDs engaged in office-based patient care– Samples visits of those MDs

Measures of physician variables including practice characteristics

38

National Health Care Surveys: Types of Measures

Factors that influence use of health care resources

Quality of health care, including safety Disparities in health care services

provided to population subgroups in the U.S.

39

National Health Care Surveys: Ambulatory and Hospital Care

Physician officesNational Ambulatory Medical Care Survey (NAMCS)

Emergency and outpatient hospital departments National Hospital Ambulatory Hospital Medical Survey (NHAMCS)

Ambulatory surgery facilities National Survey of Ambulatory Surgery (NSAS)

Inpatient hospital departmentsNational Hospital Discharge Survey (NHDS)

40

Locating Measures: Large Research Studies

Many large-scale, multi-center and longitudinal studies have developed and used measures on health-related topics

Increasingly, they are posting these on “study” websites

41

Study of Women’s Health Across the Nation (SWAN)

Physical, biological, psychological, and social changes of women during their middle years

Questionnaire can be downloaded

www.edc.gsph.pitt.edu/swan/public

42

Sacramento Area Latino Study on Aging (SALSA) Study

NIA funded longitudinal study of Latinos in the Sacramento region

Started in 1996– each person followed for up to 5 years

http://sitemaker.umich.edu/salsa.study/home

43

Locating Measures: Universities and Individual Researchers

Individual investigators who specialize in measurement and some universities have begun to post measures for public use

Some of these include large research studies posted on university websites

44

University of Michigan Institute for Social Science Research

Two studies on antecedents and consequences of retirement (with NIA)– Health and Retirement Study– Asset and Health Dynamics Among the Oldest Old

(AHEAD) Measures of health insurance, savings, and

economic well-being including life course patterns of wealth accumulation and consumption

45

Dr. James Sallis, Ph.D., San Diego State University

Conducts research with children and parents, provides nearly 50 measures in English and Spanish

2003 Report: Behavioral and Environmental Interventions to Promote Youth Physical Activity and Prevent Obesity  

www-rohan.sdsu.edu/faculty/sallis/measures.html

46

Duke University Center for Demographic Studies

National Long Term Care Survey (with NIA) Prevalence and patterns of functional

limitations, medical conditions, recent medical problems, use of health care, and housing and neighborhood characteristics

www.nltcs.aas.duke.edu/index.htm

47

Tool Kit of Instruments to Measure End of Life Care (TIME)

Instruments to measure “patient-focused, family centered” care

Annotated bibliographies of instruments that measure outcomes of care at the end of life

www.chcr.brown.edu/pcoc/toolkit.htm

48

TIME Annotated Bibliographies Available:

Quality of life Pain and other

symptoms Emotional and

cognitive symptoms Advance care

planning

Spirituality Continuity of care Caregiver well-being Grief and

bereavement and more

49

Roadmap K12 Data Resource Center

Links to over 2 dozen national and state surveys

Provides overview for each: domains, time frame, population, scope, sample size, and contacts

www.epibiostat.ucsf.edu/courses/RoadmapK12/PublicDataSetResources/

50

Locating Measures: Finding Authors of Measures

Published research using measure you are interested in– Unpublished measures often described in

methods– Authors may provide measures

51

Step 4: Review Potential Measures for

Conceptual appropriateness & relevance– in your study– in target group

Psychometric adequacy in target group or groups

Practicality Acceptability

– To respondents and interviewers

52

Conceptual Relevance

Example: you are interested in reports of perceived discrimination in the health care setting

In reviewing measures of discrimination, most are about– Discrimination over the lifecourse– Discrimination in various life settings (work,

school) Not relevant for your purpose

53

Concept Depicted as a Measurement Model

Measurement model:– the dimensional structure of a measure

– how the items related to the construct Can be depicted as a list or visually

54

Measurement Models

Physical Functioning (4 items) Psychological Distress (7 items)

55

Measurement Model (List format)

Physical Functioning defined in terms of:– Walking

– Climbing stairs

– Bending

– Reaching

56

Measurement Model (Visual format)

Physical Functioning

WalkingClimbing

StairsBending Reaching

57

Measurement Model (List format)

Psychological distress– Depression

» Sad» Lost interest» Can’t get going

– Anxiety» Restless» Nervous

58

Measurement Model (Visual format)

Psychological Distress

Depression Anxiety

SadLost

interestCan’t

get goingRestless Nervous

59

Psychometric Adequacy for Your Study

In samples similar to your target group:– good variability– low percent of missing data– good reliability– good validity

As an outcome for planned intervention– responsive, sensitive to change in similar

population– able to detect expected magnitude of change

60

Good Variability

All (or nearly all) scale levels are represented

Distribution approximates bell-shaped normal

No floor or ceiling effects– Scores “bunched” at either end

61

Reliability

Extent to which an observed score is free of random error

Population-specific; reliability increases with:– sample size– variability in scores (dispersion)– a person’s level on the scale

62

Reliability Coefficient

Typically ranges from .00 - 1.00 Higher scores indicate better reliability Types of reliability tests

– Internal-consistency– Test-retest– Inter-rater– Intra-rater

63

Internal Consistency Reliability: Cronbach’s Alpha Requires multiple items measuring same

construct Extent to which items measure same

construct (same latent variable) It is a function of:

– Number of items– Average correlation among items– Variability in your sample

64

Minimum Standardsfor Internal Consistency Reliability

For group comparisons (e.g., regression, correlational analyses)– .70 or above is minimum – .80 is optimal– above .90 is unnecessary

For individual assessment (e.g., treatment decisions)– .90 or above (.95) is preferred

JC Nunnally, Psychometric Theory 3rd ed, McGraw-Hill, 1994

65

Validity

Does a measure (or instrument) measure what it is supposed to measure?

And…Does a measure NOT measure what it is NOT supposed to measure?

66

Validation of Measures is an Iterative, Lengthy Process

Validity is not a property of the measure– validity is a property of a measure for

particular purpose and sample– validation studies for one purpose and

sample may not serve another purpose or sample

Accumulation of evidence:– Different samples– Longitudinal designs

67

Three Major Forms of Measurement Validity

Content Criterion Construct

68

Construct Validity Basics

A process of answering the following questions:

What is the hypothesis? What are the results? Do the results support (confirm) the

hypothesis?

69

Construct Validity: NOTE

Sometimes the hypothesis is that the measure will NOT be correlated with certain other measures, or will be less correlated with some than with others

THUS, observing a low or non-significant correlation can confirm construct validity

70

Limited Data on Psychometric Properties of Many Measures

Not easy to find this information Many studies do not report any

psychometric properties

– Assume the properties from original study carry over

71

Review Measures for Practicality

Method of administration appropriate for your study

Costs of administration within study resources Scoring rules clearly documented Measure available at cost you can afford You are allowed to adapt it if necessary Translations available if needed

72

Practical - Scoring

Know ahead of time how to score items– Count of “correct” answers? Summated scale?

Weighted? Are scoring instructions or computer scoring

programs available? Can scoring programs be purchased from

developers? Do you have a scoring codebook?

73

Review Measures for Availability of Translations if Needed

If you need the questionnaire in another language, are there translations available?– Official (published and tested)– Unofficial (by some other researcher)

If not, you have to conduct translations– Use state-of-the-art methods

74

Review Measures for Acceptability

Acceptability to target population – respondent burden (length, time needed,

distress)

– culturally sensitive Acceptability to interviewers

– interviewer burden

– amount of training needed

75

Respondent Burden

Perceived burden– a function of item difficulty, distress due to

content, perceived value of survey, expectations of length

– as important as “time” burden

76

5. Pretest Potential Measures in Your Target Population

Select best measures for all concepts in your conceptual framework– existing instrument in its entirety

– subscales of relevant domains (e.g., only those that meet your needs)

77

Pretest Pretesting essential for priority measures (e.g.,

outcomes) Pretest is to identify:

– problems with method of administration – unacceptable respondent burden– problems with questions or response choices– words and phrases that do not mean what you

intended to target population

78

Summary

Methods for selecting/reviewing measures described here are “ideal”

Apply these methods to your most important measures– e.g., outcomes, key independent variables

Keep learning– Good, appropriate measures remain the

foundation of excellent research

79

Homework

See handout for class 2 homework Complete rows 1-13 in matrix

– Use form posted on the website Email responses by Monday AM to

anita.stewart@ucsf.edu