evaluating community-based initiatives may 3, 2005 12:00 noon cst holly ruch-ross, scd david keller,...
TRANSCRIPT
Evaluating Community-based Initiatives
May 3, 2005 12:00 noon CSTMay 3, 2005 12:00 noon CST
Holly Ruch-Ross, ScDHolly Ruch-Ross, ScD
David Keller, MD FAAPDavid Keller, MD FAAP
Thomas Young, MD FAAPThomas Young, MD FAAP
May 3, 2005 American Academy of Pediatrics
Declaration of Vested Interest Holly Ruch-Ross has no interests Holly Ruch-Ross has no interests
to declareto declare David Keller has no interests to David Keller has no interests to
declaredeclare Thomas Young has no interests to Thomas Young has no interests to
declaredeclare
May 3, 2005 American Academy of Pediatrics
Presentation Objectives
At the completion of the presentation, the At the completion of the presentation, the listener will be able to:listener will be able to:
1.1. Identify the distinguishing Identify the distinguishing characteristics of evaluation research characteristics of evaluation research and the benefits of conducting and the benefits of conducting evaluation.evaluation.
2.2. Write clear objectives.Write clear objectives.
3.3. Identify significant evaluation challenges Identify significant evaluation challenges and strategies to address them.and strategies to address them.
May 3, 2005 American Academy of Pediatrics
Defining Evaluation
ResearchResearch: to conduct a careful, patient, : to conduct a careful, patient, systematic, diligent inquiry or systematic, diligent inquiry or examination in some field of knowledge, examination in some field of knowledge, to establish facts or principles; to to establish facts or principles; to laboriously or continuously search after laboriously or continuously search after truth.truth.
Evaluate: Evaluate: to determine the worth of; to to determine the worth of; to find the amount or value of; to appraise.find the amount or value of; to appraise.
May 3, 2005 American Academy of Pediatrics
Defining Evaluation
Evaluation is Evaluation is action action research, intended research, intended to provide information that is to provide information that is usefuluseful for:for: Program development and Program development and
improvementimprovement Program replication Program replication Resource allocationResource allocation Policy decisions.Policy decisions.
May 3, 2005 American Academy of Pediatrics
Reasons to Evaluation Checking Your Process:Checking Your Process: Are you doing Are you doing
what you said you would do?what you said you would do? Determining Your Impact:Determining Your Impact: Are you having Are you having
the desired effect in the target population?the desired effect in the target population? Creating a Fan Base:Creating a Fan Base: Can you generate Can you generate
information and evidence to share with information and evidence to share with funders and other stakeholders?funders and other stakeholders?
Replication Justification:Replication Justification: Is there evidence Is there evidence to support replication of this program?to support replication of this program?
Effective Documentation:Effective Documentation: Can you collect Can you collect information to support your program and information to support your program and meet funder or other requirements?meet funder or other requirements?
May 3, 2005 American Academy of Pediatrics
The Evaluation Cycle
STARTSTART
Implement program and begin to collect evaluative data
Review data. Are you doing what you planned? Having the
intended effect?
Adjust program; Refine evaluation
Plan program and evaluation
May 3, 2005 American Academy of Pediatrics
Types of Evaluation
ProcessProcess
Is the program Is the program being being implemented the implemented the way it was way it was designed?designed?
OutcomeOutcome
Is the program Is the program having the having the intended effect?intended effect?
May 3, 2005 American Academy of Pediatrics
Process Evaluation Information Needs Describe the program and Describe the program and
implementation, who participates in implementation, who participates in the program, what services are the program, what services are received.received.
Information such as number served, Information such as number served, patient characteristics, number of patient characteristics, number of contacts with a program, number of contacts with a program, number of trainings, number of referrals.trainings, number of referrals.
May 3, 2005 American Academy of Pediatrics
Outcome Evaluation Information Needs Detect whether the intervention Detect whether the intervention
made a difference, what changes made a difference, what changes can be measured (knowledge, can be measured (knowledge, attitude, behavior, health status, attitude, behavior, health status, incidence, prevalence)incidence, prevalence)
Longer term outcomes may need Longer term outcomes may need to be assessed using shorter term to be assessed using shorter term indicators. indicators.
May 3, 2005 American Academy of Pediatrics
Outcomes and Indicators: Examples
OutcomeOutcome IndicatorIndicatorPrevent pregnancyPrevent pregnancy Contraceptive Contraceptive
compliancecompliance
High school graduationHigh school graduation School attendance, GPASchool attendance, GPA
Reduce disease Reduce disease incidenceincidence
Immunization rateImmunization rate
Reduce clinical Reduce clinical depressiondepression
CES-D scoreCES-D score
Prevent dental cariesPrevent dental caries Plaque indexPlaque index
May 3, 2005 American Academy of Pediatrics
Goals and Objectives
Goal: broad statement of what the Goal: broad statement of what the program would like to accomplish program would like to accomplish for a specific target population.for a specific target population.
Objective: measurable step Objective: measurable step toward the achievement of a goal.toward the achievement of a goal. WhoWho will do will do WhatWhat by by WhenWhen
May 3, 2005 American Academy of Pediatrics
A GOOD OBJECTIVE IS SMART: SSpecificpecific MMeasurableeasurable AAchievablechievable RRealistic for the programealistic for the program TTime specificime specific
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Finding the Right Tools Tools need to measure the Tools need to measure the right construct. right construct. Ideally, tools should be pre-existing and well-Ideally, tools should be pre-existing and well-
established (valid, reliable and standardized).established (valid, reliable and standardized). Tools must be appropriate for the target Tools must be appropriate for the target
population (in terms of age, culture, population (in terms of age, culture, language, literacy, other issues).language, literacy, other issues).
Tools must be easy to administer in the Tools must be easy to administer in the setting.setting.
When existing tools are used, they must be When existing tools are used, they must be readily available, affordable, and supported readily available, affordable, and supported by the author.by the author.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Designing Your Own Tools Adapt an existing tool to be more Adapt an existing tool to be more
appropriate for target populationappropriate for target population Review literatureReview literature Talk to other granteesTalk to other grantees Talk to others with ideas about what Talk to others with ideas about what
you should ask about: experts, staff, you should ask about: experts, staff, recipients of servicesrecipients of services
Pilot test tools with representatives of Pilot test tools with representatives of the target population.the target population.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Using Qualitative Data Gain insight into feelings, attitudes, Gain insight into feelings, attitudes,
opinions and motivations.opinions and motivations. Study selected issues in depth and Study selected issues in depth and
detail.detail. Gather the broadest response possible Gather the broadest response possible
without predetermined categories.without predetermined categories. Gain rich information about a small Gain rich information about a small
number of people and cases.number of people and cases. Put a human face on the program.Put a human face on the program.
May 3, 2005 American Academy of Pediatrics
Most Common Qualitative Data Collection Strategies
In-depth interview: usually one-on-one In-depth interview: usually one-on-one structured conversation with a person structured conversation with a person who has important information about who has important information about the program.the program.
Focus group: professionally facilitated, Focus group: professionally facilitated, focused discussion among selected focused discussion among selected individuals about program issues. Not individuals about program issues. Not all discussion groups are focus groups, all discussion groups are focus groups, but they may still be useful.but they may still be useful.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Data Collection The sophistication of data collection should be The sophistication of data collection should be
appropriate for the scale of the project.appropriate for the scale of the project. Plan data collection up front, including who, Plan data collection up front, including who,
what and when.what and when. Have a system in place for tracking Have a system in place for tracking
participants (particularly if follow up is participants (particularly if follow up is planned).planned).
Identify the staff person responsible for data Identify the staff person responsible for data handling.handling.
Protect participant confidentiality.Protect participant confidentiality. Do not collect information that you will not Do not collect information that you will not
use.use.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Data Management and Analysis Budget for expenses associated with data Budget for expenses associated with data
entry and analysis.entry and analysis. Have a strategy for data management up Have a strategy for data management up
front. Begin data entry immediately.front. Begin data entry immediately. Remember that data analysis should Remember that data analysis should
follow directly from the questions you are follow directly from the questions you are trying to answer about your intervention.trying to answer about your intervention.
Know what comparative information may Know what comparative information may be available.be available.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Technology Involve technology as early as possible Involve technology as early as possible
in the planning process.in the planning process. Keep it as simple as possible.Keep it as simple as possible. Choose technology that will be around Choose technology that will be around
at least five years, and that is at least five years, and that is compatible with your current system.compatible with your current system.
Make certain that your choices match Make certain that your choices match the technical support that is available the technical support that is available to you.to you.
May 3, 2005 American Academy of Pediatrics
Evaluation Challenges: Getting Help From an Evaluator Specific evaluation training and Specific evaluation training and
applied research experienceapplied research experience Experience in a human service Experience in a human service
settingsetting Professional perspective and Professional perspective and
methodological orientation methodological orientation Interpersonal skillsInterpersonal skills Self interest (i.e., can he/she put Self interest (i.e., can he/she put
yours first?!)yours first?!)
May 3, 2005 American Academy of Pediatrics
Presentation objectives
At the completion of the At the completion of the presentation, the listener will be presentation, the listener will be able to:able to:
Describe a CATCH ProjectDescribe a CATCH Project Describe project’s evaluation planDescribe project’s evaluation plan Describe project’s data collection Describe project’s data collection
and management techniquesand management techniques
May 3, 2005 American Academy of Pediatrics
Keys to Successful Evaluation Develop objectives that directly Develop objectives that directly
contribute to your goals for contribute to your goals for projectproject
Spend time and thought on Spend time and thought on objectives, evaluation will followobjectives, evaluation will follow
Objectives must beObjectives must be RelevantRelevant MeasurableMeasurable Feasible/doableFeasible/doable
May 3, 2005 American Academy of Pediatrics
CATCH Project Description
CATCH Implementation Grant 2003CATCH Implementation Grant 2003 Goal: To improve screening, Goal: To improve screening,
provider knowledge and access to provider knowledge and access to mental health care for children and mental health care for children and families in a primary care and families in a primary care and continuity clinic settingcontinuity clinic setting
This is broad goal for a small grantThis is broad goal for a small grant We had to develop objectives that We had to develop objectives that
were relevant, measurable, and were relevant, measurable, and doable.doable.
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives Process Objective 1: 80% Process Objective 1: 80%
Implementation of Bright Futures Implementation of Bright Futures mental health screening: PEDS, mental health screening: PEDS, Ped Symptom Checklist, Ped Symptom Checklist, Edinburgh Postnatal Depression Edinburgh Postnatal Depression ScreenScreen
Evaluation Method: Chart audit Evaluation Method: Chart audit Findings: Chart audit documented Findings: Chart audit documented
successful implementation of successful implementation of screenings in 90% charts screenings in 90% charts
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives Process Objective 2: To provide Process Objective 2: To provide
120 mental health consultant 120 mental health consultant visitsvisits
Evaluation Method: A data file Evaluation Method: A data file was developed and completed by was developed and completed by the Mental Health consultant to the Mental Health consultant to document the number and type of document the number and type of mental health encounters.mental health encounters.
Findings: 297 MH sessionsFindings: 297 MH sessions
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives Process Objective 3: To provide Process Objective 3: To provide
case management of mental health case management of mental health referralsreferrals
Evaluation Method: Data base Evaluation Method: Data base developed with Excel to track developed with Excel to track referrals, patient contactsreferrals, patient contacts
Findings: 83 children had outside Findings: 83 children had outside referrals and case management referrals and case management documented.documented.
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives Outcome Objective 1: To improve Outcome Objective 1: To improve
provider confidence in managing provider confidence in managing behavioral problems in children.behavioral problems in children.
Evaluation Method: Development of a Evaluation Method: Development of a pre/post survey of the providers in pre/post survey of the providers in clinic on confidence in management of clinic on confidence in management of common behavior problems (no existing common behavior problems (no existing survey tool found)survey tool found)
Findings: Confidence increased from Findings: Confidence increased from 40% to 67% in 6 months. Adequate MH 40% to 67% in 6 months. Adequate MH resources improved from 54% to 95%.resources improved from 54% to 95%.
May 3, 2005 American Academy of Pediatrics
Developing CATCH Objectives
Outcome Objective 2: Identify Outcome Objective 2: Identify resources to sustain mental health resources to sustain mental health providerprovider
Evaluation: Was funding found to Evaluation: Was funding found to continue mental health projectcontinue mental health project
Findings: Primary Care Center Findings: Primary Care Center funding, grants, and partnership funding, grants, and partnership with community mental health with community mental health agency continued and expanded agency continued and expanded MH services.MH services.
May 3, 2005 American Academy of Pediatrics
Summary of CATCH Evaluation
Evaluations are different from scientific Evaluations are different from scientific researchresearch
Good objectives lead to good evaluationsGood objectives lead to good evaluations If possible find reliable tools to measure If possible find reliable tools to measure
outcomes, if not use creative common outcomes, if not use creative common sense.sense.
Accurately track data pre-grant to the endAccurately track data pre-grant to the end Keep it relevant but simple, with larger Keep it relevant but simple, with larger
follow-up grants hire an evaluator!follow-up grants hire an evaluator!
May 3, 2005 American Academy of Pediatrics
Presentation objectives
At the completion of the At the completion of the presentation, the listener will be presentation, the listener will be able to:able to:
Describe a Healthy Tomorrows Describe a Healthy Tomorrows ProjectProject
Describe project’s evaluation planDescribe project’s evaluation plan Describe project’s data collection Describe project’s data collection
and management techniquesand management techniques
May 3, 2005 American Academy of Pediatrics
Healthy Tomorrows Project Description:Family Advocates of Central Massachusetts A legal-medical collaborativeA legal-medical collaborative
Focused advocacy on legal issues likely to Focused advocacy on legal issues likely to affect child health outcomesaffect child health outcomes
Three objectives:Three objectives: Patient identification and referral from Patient identification and referral from
practicespractices Provider training on legal issues and how Provider training on legal issues and how
they affect healththey affect health Advice and counsel for patients and Advice and counsel for patients and
families in needfamilies in need
May 3, 2005 American Academy of Pediatrics
A medical-legal collaborativeLegal Assistance Corporation Legal Assistance Corporation
of Central Massachusettsof Central Massachusetts and five practices in and five practices in Central MassachusettsCentral Massachusetts
• WorcesterWorcester • Pediatric Primary CarePediatric Primary Care• Family Health CenterFamily Health Center
• WebsterWebster• South County PediatricsSouth County Pediatrics
• MilfordMilford• Milford PediatricsMilford Pediatrics
• FitchburgFitchburg• CHC Family Health CHC Family Health
CenterCenter
May 3, 2005 American Academy of Pediatrics
Focused Advocacy
• Housing stabilityHousing stability (e.g. lead poisoning, (e.g. lead poisoning, homelessness, mold and allergens)homelessness, mold and allergens)
• Financial securityFinancial security (e.g. disability benefits, (e.g. disability benefits, food stamps, Medicaid)food stamps, Medicaid)
• Dignity and safetyDignity and safety (e.g. immigration status, (e.g. immigration status, domestic violence)domestic violence)
• Access to servicesAccess to services (e.g. medical, dental, (e.g. medical, dental, mental health, special education services)mental health, special education services)
May 3, 2005 American Academy of Pediatrics
Model of Intervention
Medical Partner Legal Partner
Medical Legal Collaborative
Family
Health
Nutrition
Income
Housing
Health Care
Education Childcare
Disability
Child Protection
Family Law
Immigration
Child Health
May 3, 2005 American Academy of Pediatrics
Process Evaluation
Based on program objectives:Based on program objectives: Are the practices identifying and referring Are the practices identifying and referring
patients in need of legal services to the patients in need of legal services to the program?program?
Are we able to train medical providers on Are we able to train medical providers on legal issues and how they affect health?legal issues and how they affect health?
Have we provided advice and counsel for Have we provided advice and counsel for patients and families in need of legal patients and families in need of legal services?services?
Underlying question:Underlying question: What can we do better?What can we do better?
May 3, 2005 American Academy of Pediatrics
Process Objective 1: Are the practices identifying and referring patients in need of legal services to the program? Findings in Year 1: Findings in Year 1:
76 referrals76 referrals Came from all five Came from all five
sites.sites. Usually associated Usually associated
with trainings.with trainings. Office hours did not Office hours did not
generate referrals.generate referrals. Lessons learned:Lessons learned:
Time with providers Time with providers = more referrals.= more referrals.
Screening instrument Screening instrument needed.needed.
0
5
10
15
20
25
30
1stQuarter
2ndQuarter
3rdQuarter
4thQuarter
Observed Target
May 3, 2005 American Academy of Pediatrics
Process Objective 2: Are we able to train medical providers on legal issues and how they affect health? Year 1: 42 trainingsYear 1: 42 trainings
Webster: 9Webster: 9 Worcester FHC: 7Worcester FHC: 7 Worcester Peds Assoc: 15Worcester Peds Assoc: 15 Fitchburg: 6Fitchburg: 6 Milford: 5Milford: 5
FeedbackFeedback ““Code card is great”Code card is great” ““More time on cases”More time on cases”
Lessons learned:Lessons learned: Providers want practical Providers want practical
toolstools Providers want case-based Providers want case-based
trainingtraining
11 22 33 44 55
Values?Values? 00 00 00 77 99
Useful?Useful? 00 00 00 55 1212
MaterialsMaterials?? 00 00 11 22 1212
Trainer?Trainer? 00 00 00 55 1111
Satisfaction?
May 3, 2005 American Academy of Pediatrics
Process Objective 3: Have we provided advice and counsel for
patients and families in need of legal services? Findings: 58 cases closed Findings: 58 cases closed
for 50 clients. for 50 clients. Counsel and advice in 21 Counsel and advice in 21
cases. cases. Brief service in 26 cases.Brief service in 26 cases. Full representation in 10 Full representation in 10
cases.cases. Referred 1 clients to Referred 1 clients to
another agency.another agency. Lessons learned:Lessons learned:
Case mix more varied than Case mix more varied than expected based on needs expected based on needs assessment.assessment.
Less full representation Less full representation needed than we expected.needed than we expected.
0
5
10
15
20
25
30
35
HousingStability
Dignity/Safety
Observed Expected
May 3, 2005 American Academy of Pediatrics
Outcome Objectives: ???
Medical Partner Legal Partner
Medical Legal Collaborative
Family
Health
Nutrition
Income
Housing
Health Care
Education Childcare
Disability
Child Protection
Family Law
Immigration
Child Health
System Outcome
Health Outcome
Legal Outcome
Practice Outcome
May 3, 2005 American Academy of Pediatrics
Legal Outcome: Evaluation MethodsWas the case resolved in Was the case resolved in
favor of the client?favor of the client?
Is that client better off Is that client better off than before in terms than before in terms of:of:
• Housing stabilityHousing stability• Financial securityFinancial security • Dignity and safetyDignity and safety • Access to servicesAccess to services
From Nasdor, Moving Beyond Funder-Driven From Nasdor, Moving Beyond Funder-Driven Outcome Measures.Outcome Measures.
LACCM databaseLACCM database
LACCM outcome toolLACCM outcome tool
May 3, 2005 American Academy of Pediatrics
Legal Outcome: Problems
What is a case?What is a case?• Need to understand “jargon” of legal Need to understand “jargon” of legal
services.services.
What is an outcome?What is an outcome?• Need to integrate public health and legal Need to integrate public health and legal
datadata• Need an outcome for “brief intervention”Need an outcome for “brief intervention”
Validity of outcome measure?Validity of outcome measure?• Need to assess inter-observer reliabilityNeed to assess inter-observer reliability
May 3, 2005 American Academy of Pediatrics
Practice Outcome
How have the How have the attorneys changed attorneys changed their practice?their practice?
How have the How have the physicians changed physicians changed their practice?their practice?
Is there a more Is there a more collaborative collaborative relationship?relationship?
May 3, 2005 American Academy of Pediatrics
Practice Outcome: Problems Counting: Counting:
No ready made databaseNo ready made database Evaluation:Evaluation:
Utility of self-reported commentsUtility of self-reported comments Data entryData entry
Next steps: Next steps: Commitment to changeCommitment to change Focus groups on practice changeFocus groups on practice change
May 3, 2005 American Academy of Pediatrics
Health Outcome
What is health?What is health? Individual or Individual or
group?group? Compared to Compared to
what?what? Measured how?Measured how? What about the What about the
confounders?confounders?
May 3, 2005 American Academy of Pediatrics
System Outcome
Changes in policyChanges in policy Changes in the Changes in the
lawlaw Changes in Changes in
legislationlegislation Changes in the Changes in the
balance of powerbalance of power
May 3, 2005 American Academy of Pediatrics
Evaluation: Lessons Learned Build evaluation into the program Build evaluation into the program
designdesign Cross-disciplinary communication Cross-disciplinary communication
is essential to evaluating is essential to evaluating multidisciplinary collaborationsmultidisciplinary collaborations
Measure the easy stuff first, but Measure the easy stuff first, but don’t give up on the hard stuffdon’t give up on the hard stuff
May 3, 2005 American Academy of Pediatrics
Where to find more information… For an audio file of the call including For an audio file of the call including
question and answer period, go toquestion and answer period, go to http://www.aap.org/commpeds/resources/teleconferences.http://www.aap.org/commpeds/resources/teleconferences.htmhtm
To submit additional questions, please To submit additional questions, please email email [email protected]@aap.org. Your . Your question will be forwarded to one of the question will be forwarded to one of the teleconference speakers.teleconference speakers.
Please fill out the evaluation form and Please fill out the evaluation form and fax back to Anne Gramiak at 847-434-fax back to Anne Gramiak at 847-434-8000. A form must be completed to 8000. A form must be completed to receive continuing education credits.receive continuing education credits.
May 3, 2005 American Academy of Pediatrics
Program Staff Contact Information
CATCHCATCH Lisa Marth: Lisa Marth: lmarthlmarth@@aapaap.org.org Kathy Kocvara: Kathy Kocvara: kkocvarakkocvara@@aapaap.org.org Aleksandra Stolic: Aleksandra Stolic: astolicastolic@@aapaap.org.org
Healthy Tomorrows Healthy Tomorrows Nicole Miller: Nicole Miller: nmillernmiller@@aapaap.org.org Anne Gramiak: Anne Gramiak: agramiakagramiak@@aapaap.org.org
CPTICPTI Alanna Bailey: Alanna Bailey: [email protected]@aap.org Trisha Calabrese: Trisha Calabrese: [email protected]@aap.org
Medical HomeMedical Home Lauri Levin: [email protected] Levin: [email protected]