Measuring Outcomes to Evaluate Dementia Programs and Interventions

Strategies, Challenges, and Benefits

Michael Lepore, PhDStephanie Hughes, MPP

Elizabeth Blount, MA

Part of the National Alzheimer’s and Dementia Resource Center webinar series sponsored by the Administration for Community Living.

www.rti.orgRTI International is a registered trademark and a trade name of Research Triangle Institute.

Measuring Outcomes to Evaluate Dementia Programs and Interventions

Strategies, Challenges and Benefits

Michael Lepore, PhDStephanie Hughes, MPP

2

National Alzheimer’s and Dementia Resource Center Technical Assistance Webinar

January 30, 2019

Overview of Learning Objectives

• Identify multiple methods for collecting outcome data

• Describe risks of using outcome data collected from proxies instead of from people with dementia

• Describe the benefits of using validated outcome measures

• Provide resources for identifying validated measures

• Describe considerations in selecting a measure(s)

What are Measurable Outcomes?

A “measurable outcome” is an observable end-result that describes how a particular intervention benefits consumers

– It may demonstrate functional status, mental well-being, knowledge, skill, attitude, awareness or behavior

Outcomes from projects targeting individuals living with dementia and their caregivers include, but are not limited to:

– change in quality of life of the person living with dementia

– change in unmet needs

– change in stress and/or burden

– change in caregiver self-efficacy

– change in the responsiveness or cost-effectiveness of a service delivery system

A measurable outcome is different than a measurable “output”, such as: the number of clients served; the number of training sessions held; or the number of service units provided.

Why Measure Outcomes

Outcome measurement results can be used to:

– Decide if you should continue doing what you are doing

– Decide whether and how to change what you are doing

– Encourage participation and support from others, including program participants and funders

Sample Outcome Measure

6

Target population

Intervention Output Goal Outcome Goal Outcome Measure

Outcome Measurement

Persons with dementia who live alone

Care of People with Dementia in their Environments (COPE)

COPE provided to 100 persons with dementia who live alone

Improve quality of life of persons with dementia who live alone

Quality of Life in Alzheimer’s Disease (QOL-AD)

Implement QOL-AD pre-COPE and 30-day post-COPE

Common Outcome Measurement Methods (1/2)

Surveys/Questionnaires: Ask participants questions and record their answers

– Questions may be open- or closed-ended Closed-ended questions (e.g., yes/no) provide quantitative data Open-ended questions usually generate qualitative data, but answers can

be coded to yield quantitative data

Observation: Directly observe operations and activities– Useful when the required information can be obtained by watching and

listening– Can answer questions about whether a program is being delivered and

operated as planned and can better understand both the situation and context

Interviews: Conversation between interviewer and person who can provide input on issue being evaluated

– Follow a script or guide, which may include probing questions – Questions are usually open-ended and data are usually qualitative, but

quantitative data may be collected

Common Outcome Measurement Methods (2/2)

Focus groups: Small-group discussions – Moderator uses a question guide to introduce topics of interest and

probe for discussion– Data are usually qualitative but a short quantitative survey can be

conducted as part of the focus group to obtain quantitative information

Secondary data: Use existing data sources (e.g., medical records)– Abstract outcome data from secondary data sources – Data are usually quantitative and standardized but not tailored to the

evaluation

Capacity to Participate in Data Collection (1/2)

Some people with dementia, such as those in early to moderate stages of Alzheimer’s disease, have capacity to participate in data collection (Karlawish, 2008)

However, dementia is progressive, so capacity of people with dementia to participate declines over time (Moye, Karel, Gurrera, & Azar, 2006)

– Whether an individual is capable of decision making depends on whether he or she can communicate relatively consistently, understand basic information about choices, evaluate the implications of different choices, and rationally comprehend the risks and benefits associated with different options (Karel et al., 2010; Woods & Pratt, 2005).

– Because these capacities fluctuate over time and different decisions require different levels of capacity, a diagnosis of dementia—or even a specific score on a cognitive test—may not be the only criterion for determining decision-making capacity (Woods & Pratt, 2005).

Capacity to Participate in Data Collection (2/2)

There is no “gold standard” for determining capacity (Mitty, 2012)

Several decisional capacity assessment tools exist that can aid researchers in determining capacity among people with dementia (Cacchione, 2011).

– Evaluation to Sign Consent (Resnick et al., 2007)

– Hopemont Capacity Assessment Interview (Edelstein, 1999)

– Capacity to Consent to Treatment Interview (Marson, Ingram, Cody, & Harrell, 1995)

– MacCarthur Competence Assessment Tool-Treatment (Grisso & Applebaum, 1998).

Limited decisional capacity among individuals with dementia also may be indicated by difficulties with naming and delayed memory (Moye et al., 2006)

Proxies: Pros and Cons

Engaging proxies, such as caregivers or family members, is a valuable option for data collection when the person with dementia does not have capacity to participate or provide informed consent

– However, proxy decision-makers commonly make research enrollment decisions based on what they think will maximize the well-being of the person with dementia, not a substituted judgment of what the person with dementia would decide (Karlawish et al., 2008)

– Data provided by proxies commonly differ from data provided by people with dementia (Clarke & Keady, 2002; Sands, Ferreira, Stewart, Brod, & Yaffe, 2004).

People with dementia have been found to have higher hopes for their quality of life than their caregivers do for them (Thorgrimsen et al., 2003)

Selecting an Appropriate Outcome Measure

Does it measure the right outcome(s)?

Is it reliable?

Is it valid?

What are data collection considerations?

Measuring the Intended Outcomes

• Stakeholder agreementDevelop a clear logic

model

• Choose one or more outcome(s)

• Subtle variationsGet specific

• Intensity of intervention• Timing of data collection

Confirm the outcome(s) is realistic

Reliable and Valid Measures

• Reliability – “The extent to which a measure produces the same results when used repeatedly to measure the same thing”

• Validity – “The extent to which a measure actually measures what it is intended to measure.”From Evaluation: A Systematic Approach by Rossi, Lipsey and Freeman

Valid but not reliable

Reliable but not valid

Measures the intended outcome

Saves time

Adds credibility to results

May allow some comparability

Benefits of Using Existing Validated Measures

Obtaining Validated Measures

• National Alzheimer’s and Dementia Resource Center• Evaluating Dementia Services and Supports: Instrument Resource

Listhttps://nadrc.acl.gov/node/70

• 73 instruments• 25 domains – e.g. burden, formal supports, cognitive

functioning, risk assessment, unmet needs, behavioral symptoms, quality of life

• Easy to access• Scoring, administration• Permissions

Considerations in Selecting a Measure

• Who is collecting the data?• What level of expertise is needed?• What training is needed?

• Who is responding?• Person with dementia, caregiver, professional • Cognitive capacity

• Other considerations• Mode – Phone, in person, electronic, paper• Burden – Number of questions, amount of time required• Flow – When and how it will fit into the intervention

process

Frequency of Data Collection

Frequency Typical Use Pros ConsOne time Very brief

interventions Least resource intensive

Little respondent time required

Respondent self-reporting introduces bias

Pre/post Training sessions, measuring knowledge change

Allows measure of change

Relatively little time required

Only measures immediate impact

Pre/post with follow-up

More intensive interventions that impact well-being over time or involve practice change

Allows measure of impact over time

Resource-intensive

References Cacchione, P.Z. (2011). People with dementia: Capacity to consent to research participation. Clinical Nursing Research, 20, 223-

227. Clarke, C.L, & Keady, J. (2002). Getting down to brass tacks: A discussion of data collection with people with dementia. In H.

Wilkinson (Ed.), The perspectives of people with dementia: Research methods and motivations (pp. 25-46). London, UK: Jessica Kingsley Publishers.

Edelstein, B. (1999). Capacity assessment interview manual and scoring guide. Morgantown, WV: West Virginia University Press. Grisso, T., & Applebaum, P.S. (1998). Assessing competence to consent to treatment: A guide for physicians and other health

professionals. New York, NY: Oxford University Press. Hughes, S. J., Walberg, D., Lepore, M., Gould, E., & Walsh, E. G. (2018). Evaluating dementia services and supports: Instrument

resource list. Administration on Aging, Administration for Community Living. Karlawish, J. (2008). Measuring decision making capacity in cognitively impaired individuals. Neurosignals, 16, 91-98. Karel, M.J., Gurrera, R.J., Hicken, B., & Moye, J. (2010). Reasoning in the capacity to make medical decisions: The consideration

of values. Journal of Clinical Ethics, 21(1), 58-71. Marson, D. C., Ingram, K. K., Cody, H. A., & Harrell, L. E. (1995). Assessing the competency of patients with Alzheimer’s disease

under different legal standards. Archives in Neurology, 52, 949-954 Mitty, E.L. (2012). Decision-making and dementia. The Hartford Institute for Geriatric Nursing, New York University, College of

Nursing. Moye, J., Karel, M.J., Gurrera, R.J., & Azar, A.R. (2006). Neuropsychological predictors of decision-making capacity over 9

months in mild-to-moderate dementia. Journal of General Internal Medicine, 21, 78-83. Resnick, B., Gruer-Baldani, A.L., Pretzer-Aboff, I., Galik, E., Buie, V.C., Russ, K., & Zimmerman, S. (2007). Reliability and validity

of the Evaluation to Sign Consent Measure. The Gerontologist, 47, 69-77. Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage Publications, Inc. Sands, L.P., Ferreira, P., Stewart, A.L., Brod, M., & Yaffe, K. (2004). What explains differences between dementia patients’ and

their caregivers’ ratings of patients’ quality of life? The American Journal of Geriatric Psychiatry, 12(3), 272-280. Thorgrimsen, L., Selwood, A., Spector, A., Royan, L., de Madariaga Lopez, M., Woods, R.T., & Orrell, M. (2003). Whose quality

of life is it anyway? The validity and reliability of the Quality of Life-Alzheimer’s Disease (QoL-AD) scale. Alzheimer Disease & Associated Disorders, 17(4), 201-208.

Woods, B., & Pratt, R. (2005). Awareness in dementia: Ethical and legal issues in relation to people with dementia. Aging andMental Health, 9(5), 423-429.

More information

Michael Lepore, mlepore@rti.org

Stephanie Hughes, shughes@rti.org

National Alzheimer’s and Dementia Resource Center, nadrc.acl.gov Evaluating Dementia Services and Supports: Instrument

Resource List https://nadrc.acl.gov/node/70

Measuring Outcomes to Evaluate Dementia Programs and Interventions: Strategies, Challenges, and Benefits

Elizabeth BlountJanuary 30, 2019

• Evidence-based information, coaching and support program for adults with chronic health conditions and their caregivers

• Telephonic/e-mail delivery method

• Personalized and tailored coaching

• Provides: ongoing support, help finding practical solutions to present concerns, future planning/prevention, and adjusts to changes in care

• Service delivery period determined by licensed site

BRI CARE CONSULTATION: PROGRAM OVERVIEW

Key Components of Program

• Initial Assessment: gradual identification of present or future concerns, both medical and non-medical

• Action Plan: evolving set of simple and achievable tasks to address concerns step-by-step, begins in first call

• Maintenance and Support: regularly maintained contacts to monitor care situation (more frequent as needed) and respond to any changes

BRI CARE CONSULTATION: PROGRAM OVERVIEW

Program Development History

• Nine different research demonstrations/projects over 2 decades

• Program demonstration in Georgia:– 2010-2013– Funded by Administration on Aging/Administration for Community

Living– 3 phases:

• Original 3 Year Grant Demonstration: 2010-2013 • Atlanta Regional Commission (ARC) Pilot Project: 2013-2014 • BRI Care Consultation Statewide Expansion: 2013-present

BRI CARE CONSULTATION: PROGRAM OVERVIEW

• Challenges identified by Georgia Division of Aging Services (GDAS) and Area Agencies on Aging (AAAs) with Georgia Alzheimer’s Disease or Related Dementias (ADRD) families:

– Growing waiting lists – Underutilized formal services – Programs serving both caregiver and ADRD care recipient– Effective service abilities using face-to-face methods

• BRI Care Consultation program chosen because:– High efficiency level: cost effectiveness of telephonic model – Ability to contact care consultant – Care Consultation Information System (CCIS) guided program

delivery – Effective in wide range of agencies

PROGRAM IMPLEMENTATION HISTORY IN GEORGIA

• Program delivered at three AAAs: – Atlanta Regional Commission – Heart of Georgia Altamaha– Legacy Link

• Eligibility criteria for program participation included:– Care receivers with/without a formal dementia diagnosis– Care receiver living in the community, with a primary caregiver

(living in the home or elsewhere) – Care receiver living in community without an identified primary

caregiver– Ability to read and speak English

ORIGINAL GRANT DEMONSTRATION

• Instruments utilized to measure outcomes same as original studies that tested efficacy of program

• Measure outcomes established for project:

– Enrollment– Attrition– Caregiver Depression – Caregiving Confidence or self-

efficacy – Caregiver Social Isolation – Caregiver Health Strain

– Caregiver Needs Met-including total needs met and eight categories of needs, including:• Informal help• Dementia diagnosis• Daily tasks• Respite • Legal/financial • Support• Service access• Living arrangements

ORIGINAL GRANT DEMONSTRATION: OUTCOMES

• Results:– Enrollment and Attrition:

• Enrolled- exceeded overall goal• Attrition-slightly higher than national average

– Depression:• Decrease in caregivers of persons with high behavior

problems– Caregiver Confidence or self-efficacy:

• Increase in total sample and caregivers of persons with high behavior problems

ORIGINAL GRANT DEMONSTRATION: RESULTS

• Results:– Social Isolation

• Decrease in caregivers of persons with high behavior problems and high cognitive impairment

– Health Strain• Decrease in caregivers of persons with high behaviors

problems– Needs Met

• Increase in total sample

ORIGINAL GRANT DEMONSTRATION: RESULTS

• Goals identified for expansion:– Maximize the use limited resources– Enhance sustainability of aging network through fee-for-

service options– Infuse evidence-based programs throughout the aging

network – Improve coordination between formal and informal care– Improve the focus on prevention and person-centered

care– Improve availability of services in rural areas

STATEWIDE EXPANSION: GOALS

• Continued scaling of the program in Georgia:• Implementation Sites:

• 9 direct, 3 referring • Ongoing Training and Technical Assistance Provided:

• RCI: providing additional, comprehensive, tailored • BRIA: overall, part of standard licensing agreement

• Families Served:• No longer serving only ADRD families

• Private Pay/Fee-for-Service Model Development • Data Collection:

• Continually through Care Consultation Information System (CCIS)

STATEWIDE EXPANSION: WHERE WE ARE NOW

Bass, D. M., Clark, P.A., Looman, W.J., McCarthy, C.A., & Eckert, S. (2003). The Cleveland Alzheimer’s managed care demonstration: Outcomes after 12 months of implementation. Gerontologist, 43(1), 73-85.

Bass, D., Easom, L., Primetica, B., & Holloway, C. (2015). Reflections on Implementing the Evidence-Based BRI Care Consultation with RCI in Georgia. Generations, 39(4), 49–56.

Clark, P. A., Bass, D. M., Looman, W.J., McCarthy, C.A., & Eckert, S. (2004). Outcomes for patients with dementia from the Cleveland Alzheimer’s Managed Care Demonstration. Aging and Mental Health, 8(1), 40-51.

Easom, L. & Holloway, C. (2015). Georgia BRI Care Consultation (Unpublished grant final report). Rosalynn Carter Institute for Caregiving, Americus, GA.

Kohout, F.J., Berkman, L.F., Evans, D. A., et al (1993). Two shorter forms of the CES-D (Center for Epidemiological Studies Depression) depression symptoms index. J Aging Health, 5:179-93.

Sterling, E. W. (2014). An Analysis of Case Management and Care Consultation in the Atlanta Area (Unpublished report). Sterling Results, LLC, Marietta, GA.

REFERENCES

Social media:

@rcicaregiving

@benrose1908

@georgiadepartmentofhumanservices

Website:

www.rosalynncarter.org

www.benrose.org

https://aging.georgia.gov

KEEP UP WITH US

Thank You!

Contact information

Michael Lepore, mlepore@rti.org

Stephanie Hughes, shughes@rti.org

Elizabeth Blount, elizabeth.blount@gsw.edu